Skeletal tracking and depth

1

Dear community,

I’m trying to detect skeletons with a D435i Intel RealSense camera and there have occured some questions I would like to clarify:

  1. Is RGB-data used somehow when skeletal tracking is performed or it relies only on depth-data? Does light conditions affect tracking?
  2. It seems, that adding "BackgroundMode": "static_first_frame", improved the tracking considerably. What exactly does this parameter do and is it only about depth-info or, e.g., brightness?

Thank you,
Valery

2

1.1. Tracking is based only on depth data.
1.2. Bright light can affect tracking quality (e.g. sunlight). Nuitrack processes a depth map from a 3D sensor, and the quality of the map outdoors may be very low due to direct sunlight. We recommend that you test these cases yourself.
2. This parameter allows you to improve tracking, subject to the availability of a static background.

When a user stands too close to a wall, Nuitrack detects his skeleton incorrectly (the user “merges” with the wall). Can I prevent this?
You can try to calibrate the background. Find the “Segmentation.Background” section in the nuitrack.config file and replace the line “BackgroundMode”: “dynamic” with these two lines:
“BackgroundMode” : “static_first_frame”,
“CalibrationFramesNumber”: 20
It turns on a static model for the background.
Wait for CalibrationFramesNumber frames, and only then enter the scene.
You can set these options via Nuitrack API as well:
nuitrack::setConfigValue(“Segmentation.Background.BackgroundMode”, “static_first_frame”);
nuitrack::setConfigValue(“Segmentation.Background.CalibrationFramesNumber”, “20”);
Please note that first 20 frames are used to calibrate the background - make sure that the sensor is not moved during this time and no users enter the frame (there should only be your background in front of a sensor). You can add some kind of a calibration counter that will be displayed in your project, which will help you to understand when the calibration is over.

You can try to remove the background, which can help to solve your issue (please use the latest version of Nuitrack):

  1. Find the section "Segmentation.Background" in the nuitrack.config file and replace the line "BackgroundMode": "dynamic" with these two lines: "BackgroundMode" : "static_first_frame", "CalibrationFramesNumber": 20
    It’ll turn on a static background model.
  2. Wait for CalibrationFramesNumber frames, and only then enter the scene. If you set 20 frames as mentioned above, the background should be static for 20 frames (no users, only background). This is very important - if a user appears in the frame immediately, during calibration, the background won’t be calculated and you won’t see any effect. You have to wait for some time after running the program to let Nuitrack calculate the background.

You can also set these options using Nuitrack API:

nuitrack::setConfigValue("Segmentation.Background.BackgroundMode", "static_first_frame"); nuitrack::setConfigValue("Segmentation.Background.CalibrationFramesNumber", "20");

1 Like
3

I see. Thank you very much!

I have two more questions, intertwined one with another.

  1. Does NuiTrack increment IDs? E.g. a person is tracked, then it is lost for some 4-5 seconds. Then there apperas the second one. Will the second person have ID “2” or it will be “1” too?
  2. Is there any interframe tracking? Will NuiTrack “remember” a person tracked on previous frames?
4

Each segment has an ID but it’s not linked to a specific user. For example, if a person with ID:1 leaves the frame and then enters the frame again, it can be marked as ID:2. But if you want to track just one person who doesn’t leave the frame, segment tracking would be suitable for you.

5

That is actually more misleading than clarifying… I cannot figure out the correspondence between a skeleton, a segment and a user. Furthermore, the term “marked” does not seem clear in this context at all. My experiments showed that the person leaving the frame and entering it again has one and the same id (ID:1), even if the interval between the exit and the entrance is rather long. Could you be more specific, please? What is actually tracked and how if I use SkeletonTracker API?

6

A user is a person presented in a depth frame. Each detected user has information about his/her segment (see the User and UserFrame API classes) and skeleton (see the Skeleton and SkeletonData API classes). The user’s segment (i.e. pixels of a depth map that belong to a user) and skeleton have the same ID (an integer number).

Different users presented in the same frame have different IDs. However, there is no user recognition. So these IDs cannot be used for long-term identification of persons. If a person A with ID=1 leaves the frame and then enters the frame again, his/her ID may change to 2. Usually this happens if another user (person B) enters the frame and is marked with available ID=1.

1 Like