Depth value is not OK

Nuitrack SDK Unity3D
1

Hello @olga.kuzminykh

This may concern you because I think you wrote the point cloud tutorial.

I’m trying to do some image processing and after that i found the object position in RGB image. I use OpenCvSharp in Unity. I found the object position in terms of pixel position and they are (x,y).

Once I found the object position I want to get it’s depth value using NuiTrack API.

My implementation is as follows.

if (objectFound == true)
{
if (ax > 20 && ax < 620 && ay > 20 && ay < 460)
{
Cv2.PutText(matRGBA, “Tracking Object”, new Point(50, 50), HersheyFonts.HersheySimplex, 1, new Scalar(0, 255, 0), 1, LineTypes.Link8, false);

                        for (int q = ay - offset; q < ay + offset; q++)
                        {
                            for (int p = ax - offset; p < ax + offset; p++)
                            {

                                //change of position of the point (cube) in Z (depth)
                                newPos = NuitrackManager.DepthSensor.ConvertProjToRealCoords(ax, ay, depthFrame[ay, ax]).ToVector3();

                                sumPos = sumPos + newPos;

                            }
                        }
                        finalPos = sumPos / ((ax + offset - ax + offset) * (ay + offset - ay + offset));

                        Debug.Log(" aX: " + ax + " aY: " + ay + " wX: " + finalPos.x + " wY: " + finalPos.y + " wZ: " + finalPos.z + "depth is: "+ depthFrame[ay, ax]);

                    }
                }

I have enabled 640x480 mode for RGB frame. and I believe the same is also applied to Depth frame. I’m using PointCloud template from Unity example section.

The problem is that the extracted depth is not OK. I gives something like 760-800 and then suddenly jumps to 1300mm.

what could be the reason?

Why is it jumping?

Am I missing something?

BR,
Prasanna

2

Hi Prasanna,

The problem is in the lines:

newPos = NuitrackManager.DepthSensor.ConvertProjToRealCoords(ax, ay, depthFrame[ay, ax]).ToVector3();

sumPos = sumPos + newPos;

You have to replace ax with p and ay with q and also do a null check. Try to use this code instead:

int counter = 0;
for(...)
for(...)
{
    if (depthFrame[q, p] > 0)
    {
        newPos = NuitrackManager.DepthSensor.ConvertProjToRealCoords(p, q, depthFrame[q, p]).ToVector3();

        sumPos = sumPos + newPos;
       counter = counter + 1;
    }
}

finalPos = sumPos / counter;
1 Like