How can I get the lowest possible latency?

Nuitrack SDK
1

Hi there, I’m using Nuitrack in Unity for body tracking in a VR app. I wanted to know what settings I can configure to get the lowest possible latency?

-Which camera would provide the lowest latency?
-What Settings in the Unity SDK/nuitrack sdk could be configured to reduce latency?
-Any interop optimization that can be done between the camera/SDK/and unity communications?

We are targeting <50ms end-to-end latency in order to achieve immersive body tracking in VR.
So we would need to optimize anything that we can throughout the pipeline.

2

What is your target platform? If it’s Android, you can use one of our products: VicoVR or TVico.
VicoVR allows you to receive SkeletalTracker data via Bluetooth / WiFi with a delay of 33ms (30 FPS) on Android.

3

Hi Iosif,

We are targeting the Oculus Quest, which runs android. Unfortunately, the 33ms you describe is the latency between each frame, but does not describe the total motion-to-photon latency - the time it takes the camera to acquire the depth map, transfer over to pc, any type of frame buffering, skeleton solving, or transfer latencies between each layer.

For example, using the realsense D415 @ 60fps on the Nuitrack Zombie Oculus Rift Demo, shows that it takes 7 frames (116 ms) for the human depth map on the TV to change after the real life person has moved. For comparison, the oculus quest and rift controllers have about 50 ms (3 frames) of motion-to-photon latency.

We suspect that certain depth cameras may have lower latencies. Not sure why TVIco would have a lower latency unless this has been specifically optimized. Please advise on this issue.

4

I would like to revisit this topic because I am currently facing a similar issue and I am trying to understand what the practical lower limit is for latency with Nuitrack.

My current setup is:

  • Intel RealSense D455

  • Nuitrack

  • Godot

  • Intel Core i7-3770 3.40 GHz

  • 16 GB RAM

  • NVIDIA GeForce RTX 3050 6 GB

I already managed to get 3D skeleton tracking working in Godot, but I am measuring something around 240 ms between the person’s real movement and the skeleton response inside the game.

My goal is to build an immersive interactive experience, so I need the skeleton to react as fast as possible.

I would like to understand a few things:

  1. Is ~240 ms within the expected range for a D455 + Nuitrack setup?

  2. Are there recommended settings in Nuitrack or in the RealSense configuration to reduce end-to-end latency as much as possible?

  3. Is there any known difference in latency between using Nuitrack in Godot versus Unity?

  4. In practice, what is the lowest motion-to-skeleton latency you have achieved with Nuitrack on PC?

I am especially interested in reducing total real-world latency, not just increasing FPS.

Any guidance would be greatly appreciated.

5

Hello @VitorMoreira
Could you please check the sensor’s performance in RealSense Viewer (in 2D mode, check the stream statistics to see what the latency is), and then in Nuitrack.exe (click “Try Nuitrack” to see the frame reception time)?
Also, make sure you’re using a USB 3.0 cable.
Are you using a C# wrapper to work with Godot?