Depth2ColorRegistration problems

[panos]

Hi, I try to get the segmented RGB users with the background removed in c# and .net (not in unity) with Astra+ camera. The same effect that was done with the Coordinate Mapping of the kinect SDK. I found out that setting Depth2ColorRegistration to true does something like that but with 2 problems.

1. (and most important) The image that I get is cut around. It is smaller than it should be. Its like having a smaller FoV which would be fine if it covered the whole image. Both RGB and Depth are set to 640x480 resolution but when I render the image, it covers a smaller part of the texture than the RGB alone with the same resolution. This is strange, is there a way to fix that?

2. When I set Depth2ColorRegistration to true it seems that the RGB image is scaled down to the resolution of the Depth image. Is there a way to have segmented RGB users with the 1080p resolution of the RGB camera? With kinect this was possible with coordinate mapping. Can nuitrack do something like that?

Thanks

[vadim.bazhin]

Dear @panos,

1. Nuitrack generates a user segment based on the Depth image.
   And indeed, in order for the user’s mask to begin to match the RGB image, the Depth2ColorRegistration option should be enabled.
   The Depth2ColorRegistration option converts the Depth image to the RGB camera coordinate system.
   During this operation, some insignificant part of the original information shown in the Depth image is lost, but each pixel of the Depth image begins to match the RGB image.
   But the fact is that the Astra+ sensor initially has a Depth camera with smaller FOV than an RGB camera.
   So in an RGB image near the borders, you really won’t get a user segment.

   Could you please send us a screenshot or a short screen recording of the problem (so we can make sure that the behavior you describe is not expected)?

2. Is there a way to have segmented RGB users with the 1080p resolution of the RGB camera?

   All you need to do is enable the Depth2ColorRegistration option and set the RGB camera resolution to 1080p.

   When I set Depth2ColorRegistration to true it seems that the RGB image is scaled down to the resolution of the Depth image.

   Do you notice that the quality of the RGB image or its resolution is decreasing?

   The Depth2ColorRegistration option only affects the Depth image, and after enabling it, you should start getting a Depth image with exactly the same resolution as the RGB image.

Looking forward to your response.

[panos]

Hi, thanks for the assistance.

1. I upload a screenshot with the problem. I ve marked with red rectangles the areas where the image does not contain any information. I dont have this problem on the same areas - same texture, when I disable Depth2ColorRegistration and on the raw RGB image.
   

   

2. Where exactly should I set the RGB image to be 1080p? I experimented with some values in the config file but it didnt seem to work. I get a 640x480 image. I check the resolution by printing the width and the height from the image I get.

[irakli]

Hello @panos,

Where exactly should I set the RGB image to be 1080p?

You can use the API to set the image resolution. See the example for C#: nuitrack_csharp_device_api_sample

[panos]

Hello

I used the device API to set the resolution of the depth camera to 640x480 and the color to 1920x1080 but when I have Depth2ColorRegistration set to true, the depth camera switches to match the RGB resolution (1920x1080) and the frame rate becomes very slow. When Depth2ColorRegistration is false, each camera gets the correct resolution. Is there a way around that issue?

Thanks

[irakli]

Hello @panos

the depth camera switches to match the RGB resolution (1920x1080) and the frame rate becomes very slow.

Check that you have selected the 1920x1080 and 30 fps mode

[panos]

Hello, yes it’s on 30 fps mode. I believe the processing of the high resolution depth image is what making it slow since it slows down (a lot) both the image stream and the skeletal tracking.

[vadim.bazhin]

Hi @panos,

@panos

I used the device API to set the resolution of the depth camera to 640x480 and the color to 1920x1080 but when I have Depth2ColorRegistration set to true, the depth camera switches to match the RGB resolution (1920x1080) and the frame rate becomes very slow.

Actually it’s unexpected and we have different behavior on our side (depth stream resolution needs to change to 640x360 automatically).

1. What version of Nuitrack are you using?
2. Your Astra+ sensor looks like this, right?
3. Could you include a minimally reproducible code example where you have this problem? (This will help us to solve this issue faster)

Looking forward to your response.

[vadim.bazhin]

@panos

1. I upload a screenshot with the problem. I ve marked with red rectangles the areas where the image does not contain any information. I dont have this problem on the same areas - same texture, when I disable Depth2ColorRegistration and on the raw RGB image.

Empty areas of this size are expected at the top and bottom. As I said before, this is due to the fact that the vertical FOV of the depth camera is smaller than the vertical FOV of the color camera of the Astra+ sensor.

For example, with the Kinect v2 sensor, you did not have this issue because the vertical FOV of the depth camera was greater than that of the color camera:

If the absence of such empty areas at the top and bottom is important to you in this case, please pay attention to the vertical FOV values of the cameras when choosing a sensor model.

[panos]

Hi,

1. I am using nuitrack v0.37.23
2. Yes, that one
3. This is the code I used. It works as expected when Depth2ColorRegistration is false. When its true, it changes the resolution of the Depth frame to match RGB so the depth frame becomes 1920x1080.

public NuiTrackManager()
        {
            try
            {
                Nuitrack.Init("");
                SelectSensor();
            }
            catch (System.Exception e)
            {
                Log.WriteLine("Cannot initialize Nuitrack. " + e.Message);
                throw e;
            }
        }

        private void SelectSensor()
        {
            List<NuitrackDevice> devices = Nuitrack.GetDeviceList();
            if (devices.Count == 0)
            {
                Log.WriteLine("No cameras found");
                return;
            }

            NuitrackDevice device = devices[0];
            Log.WriteLine("Sensor used: " + device.GetInfo(DeviceInfoType.DEVICE_NAME));
            device.SetVideoMode(StreamType.COLOR, GetVideoMode(device.GetAvailableVideoModes(StreamType.COLOR), StreamType.COLOR));
            device.SetVideoMode(StreamType.DEPTH, GetVideoMode(device.GetAvailableVideoModes(StreamType.DEPTH), StreamType.DEPTH));
            Nuitrack.SetDevice(device);
        }

        private VideoMode GetVideoMode(List<VideoMode> modes, StreamType stream)
        {
            int bWidth = 640;
            if (stream == StreamType.DEPTH)
                bWidth = 640;
            else if (stream == StreamType.COLOR)
                bWidth = 1920;
            Console.WriteLine(stream);
          
            VideoMode vMode = modes.Where(m => m.width == bWidth).OrderByDescending(m => m.fps).First();
            Console.WriteLine(vMode.width + " " + vMode.height + " " + vMode.fps);
            return vMode;
        }

4. I scaled the image as a workaround so its ok now. As an optimization, I would prefer if the output image had no blank areas and it was sized accordingly so that it does not contain areas that are always blank.

Thanks!

[vadim.bazhin]

Dear @panos,

Could you please add the “OrbbecSDKDepthProviderModule.Depth.Scaling” option with value “false” in your %NUITRACK_HOME%\data\nuitrack.config file? Like this:

It will solve this issue:

but when I have Depth2ColorRegistration set to true, the depth camera switches to match the RGB resolution (1920x1080) and the frame rate becomes very slow.

Looking forward to your feedback.

[panos]

@vadim.bazhin

OrbbecSDKDepthProviderModule

Hi, I tried it and now I do get the resolutions I set, but the RGB does not map on the Depth frame as its should with Depth2ColorRegistration set to true.

[vadim.bazhin]

Hi @panos,

Are you using data from UserFrame to cut a background?
And after adding the “OrbbecSDKDepthProviderModule.Depth.Scaling” option with value “false” you get different resolution of UserFrame and RGB, right?
Could you please try resizing a UserFrame to an RGB resolution on your side and only then using a UserFrame to cut the background? This should help.

[panos]

Hi, yes I am using UserFrame. How to resize it? I believe it gets its size by the depth frame and I dont see an option in the API for resizing it.

Thanks

[vadim.bazhin]

Dear @panos,

To get resized UserFrame (result), first convert it to Bitmap (userFrameBitmap) type and then use this code:

Does it work for you?

[vadim.bazhin]

Hi @panos,

How are you? Has your issue been solved?
Would be great if you could provide some reply/feedback, we will be ready to help.
Thanks.