UNITY - Skeleton tracking and Depth not working correctly

Nuitrack SDK Unity3D
1

We tested out the Nuitrack AI SDK with INTEL RealSense Depth Camera D435, We tested out on multiple test subject but we are having some issues with the skeleton tracking and the depth,

The skeleton tracking sometime is difficult to track and when it does track the 3D model leg are somewhat not align correctly. The upper body seem to be fine but below body the leg area the tracking is not 100%, sometime its okay sometimes its not.

The depth as we can see here have a lot of noise and this make the tracking for skeleton hard due to lost of body features, and its hard as well when I’m trying to used the hand gesture because it does not detect the feature of my hand.

Some example of the Skeleton tracking and the Depth :

  1. Test 01
  2. Test 02
  3. Test 03
  4. Test 04

as you can see on test 1 - 3 we manage to track the skeleton but the 3d model leg its not align correctly, we followed the model rigged “Unity Chan” provided in the sample, and at first we thought we miss some step, but when we check back and run the sample provided in the Nuitrack sample, the same thing happen. The leg is not straighten, it bend inside a bit and sometime it just break the leg outward.

and for test number 4, the skeleton cannot detect the leg at all, it can only detect the knee area, but the knee area is at a wrong place.

Below is the SDK, pc spec and camera I used on development,

SDK:
Nuitrack AI SDK

Camera :
INTEL RealSense Depth Camera D435

Computer :
Dell G3 15 3500 Gaming Laptop
Windows 10 Home Single Language(64bit)
NVIDIA® GeForce® GTX 1650 Ti 4GB GDDR6
12GB DDR4-2933MHz, 2x4G
512GB M.2 PCIe NVMe Solid State Drive

So from all of the example above, is there a way for me to improve the skeleton tracking and fix the 3d model leg so it does not bend inward and break. Is there any documentation regarding on how to rig the 3d model so it can be used properly with Nuitrack skeleton tracking.

2

Hello @hizral
Looks really weird. Could you record a bag-file with something like this?

image (5)
image (5)1287×947 302 KB

3

Hello, i am also facing the same issue with Skeleton Tracking Depth, not Working correctly, The skeleton tracking sometimes is difficult to track and when it does track the 3D model leg are somewhat not aligned correctly. The upper body seems to be fine but below body the leg area the tracking is not 100%, sometimes it’s okay sometimes it’s not.

Please refer the video for more information.

SDK: Nuitrack
Camera: Intel RealSense Lidar Camera L515
Computer: Dell G15 5511 Gaming Laptop, Windows 11, NVIDIA GeForce RTX 3050, Ram 24GB,
i5 intel Core,

Please let me know if you need any other Information.

So from all of the example above, is there a way for me to improve the skeleton tracking and fix the 3d model leg so it does not bend inward and break. Is there any documentation regarding on how to rig the 3d model so it can be used properly with Nuitrack skeleton tracking.

4

Hello @Siddhesh

  1. The same question as in the previous message (you can record in Intel Realsense Viewer)

  1. Also check this recomendations nuitrack-sdk/General_preparations.md at master · 3DiVi/nuitrack-sdk · GitHub

  2. documentation regarding on how to rig the 3d model

https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/Unity_Avatar_Animation.md

5

Hello @Stepan.Reuk

  1. I have Checked this recommendation
  2. I have made my Avatar Animation following your 2. > documentation regarding on how to rig the 3d model

but still facing the same issue is there any other way to fix this if so please provide looking forward to reply

Thank you

7

@Siddhesh

  1. The same question as in the previous message (you can record in Intel Realsense Viewer)

Can you record .bag file? Or just record video from window of nuitrack_sample.exe (Program files\Nuitrack\nuitrack\nuitrack\bin\nuitrack_sample.exe)

9

Hello @Stepan.Reuk,
Sorry, for the late reply Facing some issue with my Intel Lidar Camera
ill send you the record video from window of nuitrack_sample.exe today or as soon as possible,
apologize for any inconvenience.

Thank you

1 Like
10

As discussed before, Please Follow the below link and check the video recording from nuitrack_sample.exe

Thank you

11

@Siddhesh
Tracking looks correct, I think it’s just that the legs sometimes go beyond the lower limit of the sensor’s field of view.

General advice: Make sure that the user with a margin will be within the limits of visibility of the sensor (especially from the bottom and top). The sensor can be moved away and slightly tilted, for example

Unity tip: The avatar’s legs look a little more aesthetically pleasing if you use indirect mapping nuitrack-sdk/Unity_Avatar_Animation.md at master · 3DiVi/nuitrack-sdk · GitHub

P.S. Flashing at the end is something wrong with the sensor?

12

@Siddhesh
Has your issue been solved? Do you have any other questions?

13

No, the issue is still there and I have noted that when the nuitrack sample is not flashing the body tracking works smoothly but when it’s flashing the body tracking is not working properly
Is there any other way to avoid the flashing at the end (The Flashing which you have also noted in the video),

Is there any specific 3.0 USB cable you can suggest for the sensor, if so please share its link

14

@Siddhesh I have tried different sensors and wires and have never seen such flashing. These are probably problems with a specific sensor (there may also be a problem in the usb connector or wire, if possible, then try different combinations).

15

Can you suggest any sensor and USB 3.0 cables that work perfectly or any combinations, if so please share its link, Thank you.

16
17

If you have any more questions, do not hesitate to create a new topic

18