Head and hand rotation not tracked using unreal plugin

Nuitrack SDK Unreal Engine
1

Hi Nuitrack,

I am investigating the use of you software for bigger interactive deployment, as a replacement to MediaPipe. Without saying too much, we are making something that would falls into the realm of your fitting room demo. So that is where I have started :slight_smile:

I just installed the Unreal plugin, the only change I have made is fixed the material domain of M_ClothRenderTarget_Mat that you use to render the clothing in ClothWidgetBlueprint.

After that, I am facing the following issues:

  1. The head and hand rotation of the mannequin does not seem to work.
  2. The mannequin does not align with my body on the camera.
  3. The software seems to loose tracking completely when only the upper body is visible.

Hoping that you can help me out :slight_smile:

Regards
Storm

2

Hello @Storm

Sorry for late reply

  1. The next update will add head rotation (based on facetracking) (example of work https://www.youtube.com/watch?v=2TXNZjz0-yM). Also, could you describe in more detail how you want to use hands rotation in your project? (rotation of the wrist, fingers, or something else)
  2. Now it may require making several manual adjustments to ensure that there is a match (You need to set the camera’s field of view in BP_ClothAvatar to be the same as that of your sensor) But in one of the upcoming updates, this will be automated (Soon).
  3. Body tracking in classic mode works best if the body is fully visible, and it should also work if the legs (the part of the body below the waist) are not visible, as long as the joint “waist” is visible.
    image
    image1920Ă—1080 67.3 KB
3

Hi Stepan,

All good :slight_smile:

  1. That sounds awesome! I have to demo the Nuitrack prototype for the team on next monday (the 10th). Would you happen to know when the new update is out / can i possibly get beta access?

The demo you sent looks awesome. Is that using the classical, or AI solver. I am a little confused as to which would be best for my usecase.

  1. We where thinking about having the character hold things, but we do not need it for the demo. In the future, just having a transform at the palm of the hand, or somehow being able to derive it. But again, the demo will focus on upper body. It’s a portrait app. Which leads me to the last question.

  2. How would you approach making a portrait app if the whole body has to be visible? Does the AI model work better here? I had some issues with the AI version not aligning with the camera.

Regards
Storm

4

Updated :exclamation:

  1. Update plugin and update Nuitrack to 0.38.2
  2. Enable in nuitrack.config (C:\Program Files\Nuitrack\nuitrack\nuitrack\data) in Faces.ToUse
    image
  3. Open Anim Blueprint ABP_Unreal_Default_Nuitrack. Enable Rotation for neck_02
    image
    image1837Ă—570 184 KB
  4. Open AvatarMannequin level and Play.

Keep in mind that the head rotation in this approach is taken from Facetracking and has its own specifics, for example, the face should always be clearly visible, not too tilted and not too far away

The demo you sent looks awesome. Is that using the classical, or AI solver. I am a little confused as to which would be best for my usecase.

Classical mode. At the moment, it is recommended to use it in 3D cases. (but it is always better to check both modes for your projects)

For portrait applications, the camera is usually rotated 90/270 degrees.
And for Unreal edit nuitrack.config “Rotation Angle” → 90 or 270
image
For unity, you can simply select the desired angle on the NuitrackManager component.

5

Awesome!

There seem to be some issues in the GetFaceAngles function inside BP_MannequinUE5. It looks like you are using some JSON library that is not bundled with the plugin? Im in UE5.5 btw.

image
image1398Ă—625 80.1 KB

Also, can you make sure the dressing room demo works before updating. It looks like they are using diffrent avatars (BP_AvatarMannequinBlueprint, not BP_MannequinUE5)

Edit: It seems that the JSON issue is due to you not marking “Json Blueprint Utilities” as a dependency in the uplugin file. You need to add the following to the NuitrackPlugin.uplugin file.

	"Plugins": [
		{
			"Name": "JsonBlueprintUtilities",
			"Enabled": true
		}
	]

Do you have a place to submit pull requests for future issues?

1 Like
6

The dressing room demo still does not work.

7

I got the dressing room to “mostly” work by assinging the BP_MannequinUE5 to the AvatarMannequinBlueprint Child Actor in BP_ClothAvatar.

The next issues is that the joints does not seem to scale to fit the person. In other words, It should use the positions of the joints, not only rotations, otherwise the character will be oversized for the user.

Edit: I was able to get the joints postioned after some tweaks to your blueprint and anim bp. The head is still far from accurate.

1 Like