I’m using the Intel RealSense2 D435 and Nuitrack Pro.
I know that the Nuitrack provides skeleton(joints) data and the joint has Real/Proj data.
I found that DepthSensor.ConvertProjToRealCoords(…), DepthSensor.ConvertRealToProj(), etc.
But these APIs return values not equal to Joint’s Real/Proj values.
How to get the APIs that convert Joint’s Real to Proj, Proj to Real exactly?
Thanks.
Hi Jylee,
No sure exactly what you mean - the conversions do what they are intended to do.
Take for example:
tdv::nuitrack::Vector3 worldPos = { hand.rightHand->xReal, hand.rightHand->yReal , hand.rightHand->zReal };
tdv::nuitrack::Vector3 screenProjection = _depthSensor->convertRealToProjCoords(worldPos.x, worldPos.y, worldPos.z);
worldPos returns the position of the right hand in the 3D world space based on the position of the hand relative to the sensor
screenProjection returns the position of the right hand in 2D space relative to the apparent viewport of the sensor - SO where the hand is relative to its place on the viewing screen - which is also its effective position in the depthmap
assuming the hand position from the example above … we get a screenProjection result of
Cheers
Westa
Hi Westa,
I’m sorry that I didn’t explain exactly.
I’m using the SkeletonData.
I tried to convert each coordinate.
The results are as follows.
[None] P2R(0.000, 0.000, 0.000) - JR(0.000, 0.000, 0.000) = (0.000, 0.000, 0.000)
[None] R2P(NaN, NaN, 0.000) - JP(0.000, 0.000, 0.000) = (NaN, NaN, 0.000)
[Head] P2R(-1277.688, 958.266, 1699.000) - JR(84.612, 810.830, 1699.973) = (-1362.300, 147.436, -0.973)
[Head] R2P(341.179, 37.042, 1699.973) - JP(0.533, 0.077, 1699.973) = (340.646, 36.965, 0.000)
[Neck] P2R(-1275.432, 956.574, 1696.000) - JR(96.189, 661.922, 1696.352) = (-1371.621, 294.652, -0.352)
[Neck] R2P(344.128, 73.961, 1696.352) - JP(0.538, 0.154, 1696.352) = (343.591, 73.807, 0.000)
[Torso] P2R(-1291.976, 968.982, 1718.000) - JR(80.162, 234.022, 1718.336) = (-1372.138, 734.960, -0.336)
[Torso] R2P(339.851, 182.048, 1718.336) - JP(0.531, 0.379, 1718.336) = (339.320, 181.669, 0.000)
[Waist] P2R(-1276.936, 957.702, 1698.000) - JR(85.396, -19.249, 1698.605) = (-1362.332, 976.951, -0.605)
[Waist] R2P(341.393, 244.822, 1698.605) - JP(0.533, 0.510, 1698.605) = (340.859, 244.312, 0.000)
[LeftCollar] P2R(-1279.192, 959.394, 1701.000) - JR(92.304, 558.191, 1701.681) = (-1371.496, 401.203, -0.681)
[LeftCollar] R2P(343.081, 100.420, 1701.681) - JP(0.536, 0.209, 1701.681) = (342.545, 100.211, 0.000)
[LeftShoulder] P2R(-1283.704, 962.778, 1707.000) - JR(328.640, 556.927, 1707.013) = (-1612.344, 405.851, -0.013)
[LeftShoulder] R2P(401.922, 101.171, 1707.013) - JP(0.628, 0.211, 1707.013) = (401.294, 100.960, 0.000)
[LeftElbow] P2R(-1290.472, 967.854, 1716.000) - JR(443.845, 237.194, 1716.226) = (-1734.317, 730.660, -0.226)
[LeftElbow] R2P(430.046, 181.190, 1716.226) - JP(0.672, 0.377, 1716.226) = (429.374, 180.813, 0.000)
[LeftWrist] P2R(-1236.327, 927.245, 1644.000) - JR(442.042, -75.672, 1644.660) = (-1678.369, 1002.917, -0.660)
[LeftWrist] R2P(434.368, 259.578, 1644.660) - JP(0.679, 0.541, 1644.660) = (433.690, 259.038, 0.000)
[LeftHand] P2R(-1214.518, 910.889, 1615.000) - JR(441.319, -201.227, 1615.940) = (-1655.837, 1112.115, -0.940)
[LeftHand] R2P(436.211, 292.988, 1615.940) - JP(0.682, 0.610, 1615.940) = (435.529, 292.378, 0.000)
[LeftFingertip] P2R(0.000, 0.000, 0.000) - JR(0.000, 0.000, 0.000) = (0.000, 0.000, 0.000)
[LeftFingertip] R2P(NaN, NaN, 0.000) - JP(0.000, 0.000, 0.000) = (NaN, NaN, 0.000)
[RightCollar] P2R(-1279.192, 959.394, 1701.000) - JR(92.304, 558.191, 1701.681) = (-1371.496, 401.203, -0.681)
[RightCollar] R2P(343.081, 100.420, 1701.681) - JP(0.536, 0.209, 1701.681) = (342.545, 100.211, 0.000)
[RightShoulder] P2R(-1291.976, 968.982, 1718.000) - JR(-138.299, 563.402, 1718.264) = (-1153.678, 405.581, -0.264)
[RightShoulder] R2P(285.751, 100.477, 1718.264) - JP(0.446, 0.209, 1718.264) = (285.305, 100.267, 0.000)
[RightElbow] P2R(-1288.216, 966.162, 1713.000) - JR(-252.862, 243.344, 1713.347) = (-1035.354, 722.819, -0.347)
[RightElbow] R2P(257.201, 179.564, 1713.347) - JP(0.402, 0.374, 1713.347) = (256.799, 179.190, 0.000)
[RightWrist] P2R(-1228.054, 921.041, 1633.000) - JR(-234.373, -66.927, 1633.345) = (-993.681, 987.968, -0.345)
[RightWrist] R2P(258.941, 257.436, 1633.345) - JP(0.405, 0.536, 1633.345) = (258.537, 256.900, 0.000)
[RightHand] P2R(-1203.990, 902.992, 1601.000) - JR(-226.954, -191.440, 1601.239) = (-977.036, 1094.432, -0.239)
[RightHand] R2P(259.689, 290.874, 1601.239) - JP(0.406, 0.606, 1601.239) = (259.283, 290.268, 0.000)
[RightFingertip] P2R(0.000, 0.000, 0.000) - JR(0.000, 0.000, 0.000) = (0.000, 0.000, 0.000)
[RightFingertip] R2P(NaN, NaN, 0.000) - JP(0.000, 0.000, 0.000) = (NaN, NaN, 0.000)
[LeftHip] P2R(-1316.793, 987.595, 1751.000) - JR(220.285, -115.987, 1751.521) = (-1537.078, 1103.582, -0.521)
[LeftHip] R2P(373.516, 268.178, 1751.521) - JP(0.584, 0.559, 1751.521) = (372.933, 267.619, 0.000)
[LeftKnee] P2R(-1203.990, 902.992, 1601.000) - JR(243.911, -630.621, 1601.282) = (-1447.901, 1533.613, -0.282)
[LeftKnee] R2P(384.816, 407.579, 1601.282) - JP(0.601, 0.849, 1601.282) = (384.215, 406.730, 0.000)
[LeftAnkle] P2R(-1203.990, 899.230, 1601.000) - JR(243.911, -1115.668, 1601.282) = (-1447.901, 2014.898, -0.282)
[LeftAnkle] R2P(384.816, 536.473, 1601.282) - JP(0.601, 1.118, 1601.282) = (384.215, 535.356, 0.000)
[LeftFoot] P2R(0.000, 0.000, 0.000) - JR(0.000, 0.000, 0.000) = (0.000, 0.000, 0.000)
[LeftFoot] R2P(NaN, NaN, 0.000) - JP(0.000, 0.000, 0.000) = (NaN, NaN, 0.000)
[RightHip] P2R(-1311.529, 983.647, 1744.000) - JR(-51.778, -115.041, 1744.851) = (-1259.751, 1098.688, -0.851)
[RightHip] R2P(307.373, 268.055, 1744.851) - JP(0.480, 0.558, 1744.851) = (306.893, 267.497, 0.000)
[RightKnee] P2R(-1197.974, 898.480, 1593.000) - JR(-56.155, -629.992, 1593.906) = (-1141.819, 1528.472, -0.906)
[RightKnee] R2P(305.009, 408.187, 1593.906) - JP(0.477, 0.850, 1593.906) = (304.532, 407.336, 0.000)
[RightAnkle] P2R(-1197.974, 894.737, 1593.000) - JR(-56.155, -1115.040, 1593.906) = (-1141.819, 2009.776, -0.906)
[RightAnkle] R2P(305.009, 537.678, 1593.906) - JP(0.477, 1.120, 1593.906) = (304.532, 536.557, 0.000)
[RightFoot] P2R(0.000, 0.000, 0.000) - JR(0.000, 0.000, 0.000) = (0.000, 0.000, 0.000)
[RightFoot] R2P(NaN, NaN, 0.000) - JP(0.000, 0.000, 0.000) = (NaN, NaN, 0.000)
where JP is Joint’s Proj, JR is Joint’s Real,
P2R is DepthSensor.ConvertProjToRealCoords(joint.Proj),
R2P is DepthSensor.ConvertRealToProjCoords(joint.Real)
Hi - Still not sure exactly what you are trying to achieve. Can you maybe provide some code that shows your working understanding.
See below code that demonstrates that the math as it currently works.
NOTE: that if you are trying to round trip the values and looking for a perfect result - it will never happen - all this math is floating point based - so there will always be some deviation in the numbers.
// SAMPLE CODE SHOWING THE WORKINGS
// starting with any nuitrack skeletal joint
const auto& worldPos = aJoint.real; // this is the 3D space
const auto& projectedPos = aJoint.proj; // this is the projection onto 2D screen
// nuitrack stores projection cords in a normalised range 0.0 to 1.0 - remap this to depthmap dimensions
int lX = projectedPos.x * 640; // 640 wide by default
int lY = projectedPos.y * 480; // 480 height by default
tdv::nuitrack::Vector3 projectedPos2 = _depthSensor->convertRealToProjCoords(worldPos.x, worldPos.y, worldPos.z);
tdv::nuitrack::Vector3 worldPos2 = _depthSensor->convertProjToRealCoords(lX, lY, projectedPos.z);
The results of a right-hand joint with initial values of:
worldPos = {x=124.292992 y=57.2119293 z=737.492126 }
projectedPos = {x=0.650191307 y=0.407822788 z=737.492126 }
convert normalised projectedPos values to screen scale
lX = 416
lY = 195
note that rounding causes some small deviation this is unavoidable with floating point and screen positions that are integers … BUT the round trip is within floating point tolerances using the functions that pass discreet values for each.
projectedPos2 = {x=416.122437 y=195.754944 z=737.492126 }
The conversion in the other direction suffers round issues a little more - since all the parameters of the function are Integers
worldPos2 = {x=124.051834 y=58.1492996 z=737.000000 }
Also - FYI - the finger-tips are not valid in the current system - which is a shame - BUT that is what it is.
BUT at the end of the day - one would have to ask what sort of workflow you are looking for?
Westa
Hi,
I’m using C# language.
nuitrack.DepthSensor m_depthSensor;
nuitrack.SkeletonData m_skeletonData;
:
nuitrack.Skeleton skeleton = m_skeletonData.Skeletons[0];
nuitrack.Vector3 vp, vr;
foreach (nuitrack.Joint joint in skeleton.Joints)
{
vr = m_depthSensor.ConvertProjToRealCoords(joint.Proj);
string sLine = string.Format(
“[{0}] P2R{1} - JR{2} = ({3:F3}, {4:F3}, {5:F3})”,
joint.Type.ToString(),
vr.ToStr(),
joint.Real.ToStr(),
vr.X - joint.Real.X,
vr.Y - joint.Real.Y,
vr.Z - joint.Real.Z);
sw.WriteLine(sLine);
Console.WriteLine(sLine);
vp = m_depthSensor.ConvertRealToProjCoords(joint.Real);
sLine = string.Format(
"[{0}] R2P{1} - JP{2} = ({3:F3}, {4:F3}, {5:F3})",
joint.Type.ToString(),
vp.ToStr(),
joint.Proj.ToStr(),
vp.X - joint.Proj.X,
vp.Y - joint.Proj.Y,
vp.Z - joint.Proj.Z);
sw.WriteLine(sLine);
Console.WriteLine(sLine);
}
Thanks
Have a look at the code I provided in the previous post.
The values in joint.Proj are currently not directly transmutable using ConvertProjToRealCoords()
joint.x and joint.y contain normalised float values between 0.0 and 1.0 - and joint.x contains depth
To pass these values into ConvertProjToRealCoord() you need to convert joint.x and joint.y to actual screen axis values … that is multiply both by their corresponding depthmap width or depthmap height.
Also - FWIW the parameter types internally used bu ConvertProjToRealCoord() are all integers - which does result in some small rounding errors.
Westa
Thanks, I solved. ^^