Depth2ColorRegistration problems

Hi, I try to get the segmented RGB users with the background removed in c# and .net (not in unity) with Astra+ camera. The same effect that was done with the Coordinate Mapping of the kinect SDK. I found out that setting Depth2ColorRegistration to true does something like that but with 2 problems.

  1. (and most important) The image that I get is cut around. It is smaller than it should be. Its like having a smaller FoV which would be fine if it covered the whole image. Both RGB and Depth are set to 640x480 resolution but when I render the image, it covers a smaller part of the texture than the RGB alone with the same resolution. This is strange, is there a way to fix that?

  2. When I set Depth2ColorRegistration to true it seems that the RGB image is scaled down to the resolution of the Depth image. Is there a way to have segmented RGB users with the 1080p resolution of the RGB camera? With kinect this was possible with coordinate mapping. Can nuitrack do something like that?

Thanks

Dear @panos,

  1. Nuitrack generates a user segment based on the Depth image.
    And indeed, in order for the user’s mask to begin to match the RGB image, the Depth2ColorRegistration option should be enabled.
    The Depth2ColorRegistration option converts the Depth image to the RGB camera coordinate system.
    During this operation, some insignificant part of the original information shown in the Depth image is lost, but each pixel of the Depth image begins to match the RGB image.
    But the fact is that the Astra+ sensor initially has a Depth camera with smaller FOV than an RGB camera.
    So in an RGB image near the borders, you really won’t get a user segment.

    Could you please send us a screenshot or a short screen recording of the problem (so we can make sure that the behavior you describe is not expected)?

  2. Is there a way to have segmented RGB users with the 1080p resolution of the RGB camera?

    All you need to do is enable the Depth2ColorRegistration option and set the RGB camera resolution to 1080p.

    When I set Depth2ColorRegistration to true it seems that the RGB image is scaled down to the resolution of the Depth image.

    Do you notice that the quality of the RGB image or its resolution is decreasing?

    The Depth2ColorRegistration option only affects the Depth image, and after enabling it, you should start getting a Depth image with exactly the same resolution as the RGB image.

Looking forward to your response.

Hi, thanks for the assistance.

  1. I upload a screenshot with the problem. I ve marked with red rectangles the areas where the image does not contain any information. I dont have this problem on the same areas - same texture, when I disable Depth2ColorRegistration and on the raw RGB image.

  2. Where exactly should I set the RGB image to be 1080p? I experimented with some values in the config file but it didnt seem to work. I get a 640x480 image. I check the resolution by printing the width and the height from the image I get.

Hello @panos,

Where exactly should I set the RGB image to be 1080p?

You can use the API to set the image resolution. See the example for C#: nuitrack_csharp_device_api_sample

Hello

I used the device API to set the resolution of the depth camera to 640x480 and the color to 1920x1080 but when I have Depth2ColorRegistration set to true, the depth camera switches to match the RGB resolution (1920x1080) and the frame rate becomes very slow. When Depth2ColorRegistration is false, each camera gets the correct resolution. Is there a way around that issue?

Thanks