Aligning virtual character with RGB stream aka Mixed Reality Capture

[korzen303]

Hi, I have just bought Nuitrack and so far so good.

I would like to mix RGB stream using Color Frame Canvas prefab with Render Mode set to Screen Space - Camera. Now I would like to match the character in scene, avatar driven by Nuitrack skeletal tracking with this image in a way they match. Basically, I would like to put cloths simulated in Unity onto video image of people. How can I do such a thing?

Thanks!

[olga.kuzminykh]

Hi Przemysław,

To do this, you have to change some parameters of the camera on the scene:

1. Set the camera to (0,0,0) and rotate it by 180 degrees(0,180,0);
2. If the model size is incorrect, change FOV of the camera;
3. Please use the model with direct mapping of joints. Such model is used on our RiggedModel2 scene. Make sure that your Canvas doesn’t overlap the model.

You can find the sample project here.

[korzen303]

Hi Olga thanks, it seems that you need to first match the position and tilt of real camera in Unity and than fine tune the FOV.

[adammarcwilliams]

Hi Olga,

I’m creating something similar where I need joints in 3D space to line up with the person on the RGB feed in the back ground but instead of controlling an avatar we’ll be pinning things to them.

I’ve made sure I’m orienting the joints in the same way you do in your example and followed the rest of the advice above but I’m having a real struggle getting them to line up.

I’ve also noticed the head joint doesn’t rotate when I turn my head which is a problem since the app I’m making needs to pin armour and a helmet to the user. Is this correct behaviour? If so I assume my best way around this would be to use the skeleton head joint for the position of the helmet and use your face tracking API to get the face angle to rotate it.

I’m jumping ahead though there, because if I can’t get the skeleton joints to line up with the user in the RGB feed it’s not going to work anyway.

I’ve ran the demo project you posted above with the Unity Chan avatar and get the same problem, even after tweaking the camera position/rotation/FOV to try and get it to line up. Anything I do that gets the wrist joints placed wide enough apart to line-up, results in all the other joints then being out of place.

I feel like the points are never going to match properly because the image is being stretched to fill the screen. Is that the case or have you had this demo working perfectly? Maybe I need to scale the skeletons parent object by the same scale the image is being stretched?

If you have any advise you could give to solve this problem I’d be very grateful.

I’m using a Realsense D415 with this if that makes a difference.

Thanks for your help

Adam

[olga.kuzminykh]

Hi Adam,

It seems that you didn’t turn on the registration of depth and RGB frames.

Nuitrack Knowledge Base

You can align depth and RGB maps in one of two ways:

By editing nuitrack.config :

1. Open nuitrack.config (from %NUITRACK_HOME%/data)
2. Find the section “DepthProvider” and set “Depth2ColorRegistration” to “true” .

By using Nuitrack API (insert the following line to your code):

(C++) nuitrack::setConfigValue("DepthProvider.Depth2ColorRegistration", "true");
(C#) Nuitrack.SetConfigValue("DepthProvider.Depth2ColorRegistration", "true");

[adammarcwilliams]

Hi Olga,

Thanks very much, I’ll give that a try.

[aalmada]

Hi all,
I’m also trying to align the skeleton with the RGB image in Unity but I’m not able to get it right.
I’m using a RealSense D435i depth-sensing camera.
I have the ‘Depth2ColorRegistration’ in ‘nuitrack.config’ set to ‘true’.
I set the camera horizontal FOV to 69.4 and I’m using a RawImage as described in the “Displaying Skeletons on an RGB Image” tutorial.
I’m using the ‘Assets/NuitrackSDK/Tutorials/Avatar Animation/Scripts/RiggedAvatar.cs’ script that I think is the one used in ‘RiggedModel2.scene’.
I’m using a model from MakeHuman with a CMU compliment rig and rendered using a ‘Skinned Mesh Renderer’.
The skeleton is working fine, it’s just the alignment that I’m not getting right. Any more suggestions?

[iosif.semenyuk]

There can be several problems:

1. Background stretching (for Image UI, you need to enable Preserve Aspect https://docs.unity3d.com/2018.4/Documentation/Manual/script-https://docs.unity3d.com/2018.4/Documentation/Manual/script-Image.html, https://docs.unity3d.com/2018.4/Documentation/Manual/script-Image.html)
2. Perhaps the issue is connected to positioning in RiggedAvatar.cs (try to use the calibration script from Nuitrack SDK at startup of your project to align the skeleton)
3. The set FOV of the camera does not match the FOV of the sensor.

Also please take a look at our tutorial (see the section Direct Mapping).

[korzen303]

Maybe it would be better if you guys could please provide a working sample already calibrated, for example, for RealSense cameras…

[aalmada]

@iosif.semenyuk Thank you for the reply

1. The RGB camera is 1920 × 1080 and the output window is 3840x2160. They have the same exact ratio 1.77(7).
2. I’m using the “T Pose Calibration” and “Calibration Info” modules. Should I use others?
3. I now noticed I’m using RGB camera FOV (69.4) but I think I should be using the depth camera FOV (86). This still doesn’t fix it.
4. This is all based on the “Direct Mapping” tutorial. I took several looks already.

[iosif.semenyuk]

RGB and Depth FOV are different for different sensor models, and Nuitrack doesn’t have access to FOV of a sensor. Unfortunately, we don’t have such tutorials (for a 3D model).

[korzen303]

Yeah, but the RGB to D registration (option for RS cameras) should solve this, right?

[iosif.semenyuk]

Yes, this should align depth frames and color frames, however, it’s still necessary to fine-tune a sensor and a model to match it with the background. We’ll consider making a tutorial on aligning RGB and Depth.

[renderLux]

Have you made a tutorial yet?

[iosif.semenyuk]

No, we haven’t made this tutorial yet.

[Stepan.Reuk]

Updated the “Point cloud” tutorial

16. In order to improve the resulting point cloud, we recommend you to turn on depth-to-color registration because a depth map doesn’t accurately match an RGB image, therefore, they should be aligned. To turn on depth-to-color registration, open nuitrack.config from the folder <nuitrack_home>\data and set DepthProvider.Depth2ColorRegistration to true.

https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/Unity_Point_Cloud.md

[renderLux]

@renderLux

Have you made a tutorial yet?

Have you made a tutorial yet?

[EugeneVasiliev]

At the moment, we are preparing a tutorial “Rendering world object behind user”.
In this tutorial, you will see how to recreate a real-world environment in Unity with the correct depth map, as well as the ability to interact with objects on the stage ( AR ).

Scenes with examples from the tutorial will be available in the latest version of the Nuitrack SDK.

Subscribe to the YouTube channel so you don’t miss the video.

[renderLux]

You been working on this tutorial for a year, when will be done?

[EugeneVasiliev]

The video is ready and is in the final stage of editing.
It remains to do voice-over and translation for subtitles.