Skeleton tracking specs ? 60 - 90 fps?

Nuitrack SDK
1

Hi everyone,

I’ve been using Kinnect 2 with its sdk for years for head position via skeleton tracking for projections projects:
Anamorphosis: https://vimeo.com/223640605
Interactive projection: https://vimeo.com/207528533

It’s great, but the 30fps tracking is a bit limited, so I’m looking for a faster solution. Realsense announces 60-90 fps depth, is there a chance that the NUItrack skeleton tracking could extract skeleton data at such speed ? Has anyone tried 90, and got any information about specs / latency ?

Even Intel doesn’t seems to know :slight_smile:

That would be awesome, thanks !

2

Interesting topic! I previously worked with Kinect and now transferred to Intel D415. Based on my computer configuration (sort of old one), for real-time tracking when the setting is 60fps, it can achieve 60 with few frame loss. However, when it comes to setting with 90fps, the time interval is around 22ms, which is actually around 45fps. Would like to hear about your progress to see whether that my hardware performance is the bottleneck or it is the software issue.

1 Like
3

Oh interesting ! WOuld you say the skeleton tracking feels twice smoother / faster with your current setting ?

Never heard or witnessed anything as good as kinect 1-2 yet, so very curious to hear about your experience !

4

@a.potopahin would you know about the skeleton tracking max framerate, and possible bottleneck (image processing / software ?).

Thanks a lot !

5

Hi Joanie,

Please advise what sensor do you use? With Kinect, you won’t be able to get more than 30 FPS.
We’ve just checked RealSense D415 and D435 - you can set 60 FPS with the resolution of 848x480.
To do this, you have to modify nuitrack.config file (in %NUITRACK_HOME%/data folder in Nuitrack Runtime): add the following lines in the "RealSense2Module.Depth" section:

"FPS": 60,
"RawWidth": 848,
"RawHeight": 480,
1 Like
6

Awesome, I just ordered a D435i, thanks to your message I’m aware I’ll have to wait for intel sdk windows support, and then I’ll try 60fps 848x480 thanks to your config file recommendation.
I can’t wait :slight_smile:

7

Based on Intel configuration, they also offer 848x480 @ 90fps. It does work with Nuitrack, however, I had the performance drop issue as I mentioned above. So I am looking forward to your result.

8

Apparently higher frame rate increases the accuracy of the skeleton position. Though have not yet systematically compared the result with Kinect.

9

We’re running NUITrack with Unity and we’re getting a frame-rate of about 17fps on an i7-4400k. We’ve attempted to add the lines “RawWidth, RawHeight, FPS” as mentioned above, however when we do - we get no output at all. Any ideas?

10

Hi ondatabc,

Please advise:

  1. What sensor do you use?
  2. What resolution do you set?
  3. What Nutirack modules do you use?
  4. What sample do you run?

Please try to run one of our standard samples (for example, nuitrack_csharp_sample) to check FPS. Do you still get only 17 FPS? Also, you can check the resource consumption in Unity Profiler.

11

Thank you for your response Olga,
1.We’re using the realsense 435i and Orbbec Astra Pro - and have the same problem with both sensors.
2. 848x480
3. We’re using the Color and Skeleton Modules
4. Our project is based off the nuitrack face tracking tutorial.

Our real issue is that when we enable the videostream in Unity (color module) our framerate drops to 15fps. Now when we disable the video stream on startup we get a higher framerate.

12
  1. Do you use the latest version of our Unity package?
  2. Do you use the DrawColorFrame script in your project?
    This looks like this:
using UnityEngine;
using UnityEngine.UI;

public class DrawColorFrame : MonoBehaviour
{
    [SerializeField] RawImage background;

    void Start()
    {
        NuitrackManager.onColorUpdate += DrawColor;
    }

    void DrawColor(nuitrack.ColorFrame frame)
    {
        background.texture = frame.ToTexture2D();
    }
}

The previous version of this script is less efficient than the latest one. Please update the script if you use it.

13

Thanks Olga,
I finally managed to test the Nuitrack performance at 60fps (thanks Hiroshi :wave:) and the results are very promising…
I’ll run proper graphs and compare 90 / 60 (hard to benchmark with naked eye) but so far it’s such a great improvement. Thanks !!