Is nuitrack AI enabled by default in the unity sdk? if you have a licence for it. How can i tell if the AI portion is working. Is there any differences in the way you must develop your application, or is it all just handled for you.
Hello @DrDoob
For Nuitrack AI to work, you need to have a license and configure these values in the config (…\Nuitrack\nuitrack\nuitrack\data\nuitrack.config):
-
DepthProvider.Depth2ColorRegistration=true -
Skeletonization.Type=CNN_HPE
Working in Unity with Nuitrack AI and regular Nuitrack is no different. From the point of view of body tracking, only the algorithm used within Nuitrack itself changes. The interaction through the API does not change in any way.
Please let me know if you have any other questions.
Hi,
I have done what you suggested about changing the config to activate Skeleton AI and this is the result of that:
And also I do not see any effects of Skeleton AI on it.
What should I do right now for fixing it?
Thank you for your time in advance.
And this is the output before applying the changes in the config:
@Motahar hi, it looks like bug is presented only in Nuitrack AI, we will try to reproduce and resolve promptly. Please specify do you use joints positions or orientations for avatar animation?
cc @Ravil @Stepan.Reuk
Hi,
I created RiggedAvatar.cs according to your documentation in https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/Unity_Avatar_Animation.md, the Direct Mapping section. In ProcessSkeleton function it updates both of position and rotation of the model:
https://github.com/MotMans/NuitrackUnity/blob/main/RiggedAvatarAdvanced.cs
Also without AI, it is really deformed and inaccurate as can be seen in the second video, and I do not know how I can improve it.
Hi there,
I’m sorry to pull this thread up again, but I have a very similar problem.
I used the non-ai Version before and thought that I would get more stability and better tracking overall with the AI version. Sadly, the oppsite is the case. If I run the nuitrack_sample.exe for testing these are the results:
- without AI the tracking is pretty stable and all joints are there most of the time. In my own software, the directly mapped 3d models behave like they should
- with AI the tracking is unstable and of poor quality. Wrist and ankle joints are lost frequently causing my 3d models to deform badly.
The environmental circumstances are the same for both scenarios.
Can you please have a look at this? I really need the tracking to be smooth and stable and want to use the advantages of the AI version.
Thanks
Hello,
I have the same problem and the same request.
Thank you in advance.
Hello,
I tried your idea, but the problem remained and it deformed the avatar even more.
I am using the Nuitrack SDK with Orbbec Astra Pro in a blank project on Unity 2020.3.23f1.
I am sending three videos showing how Nuitrack SDK works for me without AI, with AI but with Rigged Avatar component and with AI with Nuitrack Avatar component. I don’t know how to upload videos here, so I attach a link below to my Google Drive. Can you please have a look at this? Or I can upload these videos in another way, if I should.
Thank you in advance for your help.
Let’s check the following:
- Your Avatar is not fully configured (you need to add joints for Collar, see the examples in NuitrackSDK/Avatar/3D Avatar/Prefabs)
- Make sure that the model is placed in the T-pose (can be configured via the
Avatarcomponent) - We don’t fully understand the problem yet, could you record a video with RGB (add the NuitrackSDK/Frame/Prefabs/RGB Frame prefab to the scene)?
- Do you use additional scripts to manage your avatar?
Waiting for your reply.
Hello,
I followed the steps you described. I attach a link below to new video:
Ad.1. I added joints for Collar to my model as I showed at the end of the video.
Ad.2. My model is in T-pose.
Ad.3. I tried to add RGB_Frame prefab to my scene, but I got only blank white image. I don’t quite understand how to set this prefab. Can you explain this to me?
Ad.4. I don’t use any additional scripts.
Thank you in advance for your help.
It looks very strange…
Prefab with RGB should work out of the box (check whether the module is enabled in the NuitrackManager -> Modules -> ColorModule component)
Are there similar problems with NuitrackSDK/Avatar/3D Avatar/Prefabs/UnityChanAvatar prefabs?
Could you send us a model of your character so that we can check it and identify the problem?
Thank you for your answer.
The ColorModule component wasn’t enabled. I attach an image with this component enabled and RGB_Frame prefab on the scene.
I have similar problems with UnityChanAvatar prefabs, like on video attached below. It is better than on my model, but not good.
Nuitrack AI with UnityChanAvatar prefab
I also attach model of my character like you said:
We found that NuitrackAvatar does not handle some models correctly.
We offer a temporary fix, we deployed the main object (Humanblendrig31(2)) and the metarig joint by 180 degrees and this solved your problem.
I attach the corrected test package and video for verification.
It is also recommended to disable AI for smoother operation of the skeleton.
We will fix these issues in the next update.
Hello,
Thank you for your help.
I attach a link below with the two videos, which was recorded using scene from your package: the first one using my Orbbec Astra Pro and the second one using KinectV2 for comparison. I don’t know why, but for me it doesn’t work as well as for you.
Link to videos mentioned above
I am looking forward to the new version of Nuitrack SDK.
Try to move away from the wall.
The skeleton is built on the basis of a depth map and if the difference between you and the surface behind is small, then this can cause problems with the skeleton.
For Orbbec sensors, avoid sunlight hitting you.
Learn more about preparing your environment to work with the sensor.
Check if the AI is disabled (also if you have changed the configuration).
Let us know if the problem persists.
it`s right,thank you!!!
1 Like
Hi @Stepan.Reuk,
For the latest version of Nuitrack, should we still need to apply these 2 changes to enable AI if we have bought Nuitrack AI?
Thanks