Nuitrack Knowledge Base


#1

Support for Devices and Platforms

Do you support other Linux distributions besides Debian/Ubuntu?

At the moment, there is official support only for Debian/Ubuntu distributions. You can try to install Nuitrack by unpacking the .deb archive and running the postinst script.

Do you have any samples that utilize Java or even Xamarin.Android since the API has .NET bindings?

We have a simple Xamarin.Android sample.
It has no GUI, it prints log messages to an Android console only. However, it shows how to use Xamarin.Android to build apps for VicoVR.

Can I make an Unreal Engine project with Nuitrack on Windows?

At this moment, Nuitrack SDK supports Unreal Engine projects for Android only.
You can find a tutorial about Nuitrack usage in Unreal Engine here.
Unfortunately, projects for Windows are not supported yet.

Does your SDK work with Windows 10 UWP applications?

Current Nuitrack SDK uses native С++ libraries built for Windows x86/x86_64 architecture. Besides it has some 3rd-party dependencies (drivers for depth cameras) which are available only for a desktop platform.

Can I run Nuitrack on Ipad?

Unfortunately, Nuitrack doesn’t run on an Ipad at the moment.

Can I activate Nuitrack in a Docker container?

Unfortunately, it’s not possible to use Nuitrack Pro in Docker yet. We’re working on a solution to fix it.

Can I connect RealSense to USB 2.0 and use it with Nuitrack?

You can connect RealSense via USB 2.0, but in this case you have to set lower resolution for a depth map. To do this, open nuitrack.config (%NUITRACK_HOME%/data), find the section "Realsense2Module" and add the following lines to "Realsense2Module.Depth":
"RawWidth": 640,
"RawHeight": 480,

Can I use Nuitrack with RealSense and Android?

Yes. However, for RealSense with Android read permission for a root folder is required. You can check the permissions using any file manager, for example, ES File Manager: if the “/” folder is empty, then read permissions for the root folder aren’t granted in the current firmware version of your Android device.

Can I run Nuitrack on Mac?

Unfortunately, Mac is not supported at the moment.

Do you have a WPF sample?

You can download a simple WPF app with Nuitrack here. In this app, Nuitrack depth module is created and depth frame timestamp is displayed. Also make sure that target platform is set to x64: Project Settings > Build > Platform Target > x64

Installing and Uninstalling Nuitrack

How to install Nuitrack on my PC?

Please, take a look at Installation Instructions in our online documentation.

How to uninstall Nuitrack on Linux?

To uninstall Nuitrack, use the following command:
sudo dpkg -r nuitrack

Questions about Nuitrack License

I cannot enter the secret key because the Activation button is greyed out!

Make sure that you’ve installed Microsoft Visual C++ Redistributable for Visual Studio (step 3 from our Installation Instructions. This will resolve missing dependencies for the Nuitrack module.

After that, run Nuitrack.exe tool and press ‘Compatibility Test’. You should see the window with the depth map. After a few seconds, the window will close and you’ll be able to enter your activation key.

After finishing the Compatibility Test, you should see a file “e24b…” in the %NUITRACK_HOME%/data folder. If it’s not created, please, check the access permissions for %NUITRACK_HOME%/data folder.

If you use RealSense, please check that it’s connected via USB 3.0 and camera firmware version is 5.8.15 or higher.

I get the error: “Network Error (99) : Connection timed out”.

Please check your network connection as it is required for Nuitrack activation (your device should have access to http://download.3divi.com/).

I updated the software and now I cannot apply my license.

You have to reactivate Nuitrack using the same depth sensor that you used for the first activation. You don’t have to purchase a new license, you can use your activation key.

How can I reactivate the license?

The reactivation process of Nuitrack on a different PC with the same sensor is as follows:

  1. Plug in the sensor and run Nuitrack activation tool;
  2. Click “Compatibility Test”;
  3. Wait until the test is finished and enter the license key;
  4. Click “Get Available Licenses”. The tool will show that you have “0” licenses BUT this means that you have 0 NEW licenses; your old licenses can still be used;
  5. Click “Upgrade to Pro”/“Upgrade to Pro for 1 Year” (depending on your license type).

Please note that you have to use the same sensor (with the same serial number) that you used when activating Nuitrack for the first time.

I enter the license key but receive an error “failed: wrong client key”.

Please try to copy your key from the Activation letter (check your email) and paste it to the “Client key” field. Please note that register matters.

I purchased too many licenses! Can I get them back? Can I deactivate them?

For testing or temporary use, we provide customers with Nuitrack Free Trial

If you purchased too many licenses by mistake and didn’t activate them, please contact our support.

In case you have already activated the sensor, you can not deactivate it as this procedure is not provided. To activate a new sensor, you need to purchase another license.

When registering the order, you can specify the same email address and 1 license will be added to your activation key automatically.

What is the difference between Nuitrack Trial and Pro? Can I use Nuitrack for development?

Nuitrack Trial is for demo purpose only and may not be used for development of applications (Project Content) for commercial use.

Nuitrack Pro may be used by anyone who pays the applicable subscription and / or unit license fees to develop, distribute, sell and offer for sale Project Content. Nuitrack Pro license is required for each copy of Project Content.

To remove the time limit, you need to upgrade the Nuitrack Trial to Nuitrack Pro (via license purchase).

You can download Nuitrack Trial here.

SDK and examples are available here.

Online documentation is available here

I get LicenseNotAcquiredException in my app, though standard Nuitrack work fine.

Make sure that you don’t specify the path to nuitrack.config file in the nuitrack::Init() or Nuitrack.Init function. If you specify the path, the file would contain incorrect path to the license.json file. The license.json should be located at the initial folder (when it was created).

When I try to run the nuitrack_license_tool in Ubuntu I always get "Please run with sudo" and when I run with sudo I get "Please run with "sudo -E"

Please make sure that you re-logged in after Nuitrack installation (see the 3rd step here).
Also please check the value of NUITRACK_HOME: echo $NUITRACK_HOME
If the value is empty, please, execute the following command:
export NUITRACK_HOME=/usr/etc/nuitrack
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/nuitrack
After that, run the following command again:
sudo -E nuitrack_license_tool

I bought the license and didn’t receive the secret key!

The secret key and instructions on license activation are sent automatically to your email after you purchased the license. Please, check your ‘Spam’ folder. If it doesn’t help, please contact us and we will duplicate the activation key for you as soon as possible.

Can I activate two sensors with two licenses on the same PC?

You can activate several licenses for several sensors on one PC. Please note that when you activate the license for one sensor, a new license.json file is created in %NUITRACK_HOME%/data folder. This license (and a file) is linked to the sensor that was used. If you want to activate another sensor, you have to backup the already created file (rename it or move to another folder) and follow the activation process again for another sensor. As a result, another license.json file will be created in %NUITRACK_HOME%/data folder, but it will be linked to your second sensor. So, all in all you’ll have two license files. You can keep these two files in one %NUITRACK_HOME%/data folder BUT you have to use the license that is linked to the currently used sensor (and it should be named license.json).

Nuitrack Features and Functionality

What are minimum specifications for Nuitrack?

There are no strict requirements for hardware performance in Nuitrack. RAM usage is below 200 Mb. There is only one restriction for desktop processors - they should support AVX instructions set.

Does Nuitrack provide coordinate mapping between color frames and depth frames?

Yes, you can align depth and RGB maps in one of two ways:

By editing nuitrack.config:

  1. Open the file data/nuitrack.config
  2. Find the section “DepthProvider” and set “Depth2ColorRegistration” to “true”.

By using Nuitrack API (insert the following line to your code):

(C++) nuitrack::setConfigValue("DepthProvider.Depth2ColorRegistration", "true");
(C#) Nuitrack.SetConfigValue("DepthProvider.Depth2ColorRegistration", "true");

Can I select one of connected sensors?

This feature is in our road map. Currently, Nuitrack starts with one of connected supported sensors: Nuitrack loops over supported cameras and starts with the first detected camera.

Can I use Nuitrack with multiple sensors?

Unfortunately, at the time Nuitrack does not support connection of multiple sensors. Nuitrack works with the first supported sensor connected to a PC. Support for multiple sensor connection will be added in one of the next releases.

Is it possible to get the IR data using your SDK?

Unfortunately, it’s not possible to get the IR data using Nuitrack SDK, only depth and RGB data.

Can I use OpenCV 3.4 with Nuitrack?

Current Nuitrack version is built with OpenCV 2.4. Nuitrack will be upgraded to OpenCV 3.4 in one of the next releases.

How can I set up tracking of multiple users?

You can track up to 6 skeletons by two ways:

  1. Using Nuitrack API: take a look at SkeletonTracker::setNumActiveUsers (C++) or nuitrack.SkeletonTracker.SetNumActiveUsers (C#) method in Nuitrack online documentation

  2. Editing nuitrack.config file: in the section “Skeletonization”, set “ActiveUsers” to the desired value (up to 6 skeletons)

What is the difference between Hand Tracker Module and "JOINT_RIGHT_HAND" from Skeleton Tracker Module?

HandTracker returns 2D (screen) coordinates of a cursor, 3D positions and grab state. This is an add-in module above the skeleton (it uses positions of skeleton’s hands). In this case, 3D position of a hand is the center of mass of the point cloud of a hand.

How can I enable RGB stream for Orbbec Astra Pro?

To enable RGB stream for Astra Pro, you should set "CameraID" option in the nuitrack.config file (usually it’s 0, but if it doesn’t work, try 1, 2, …).

How can I change Depth/RGB resolution for Orbbec Astra Pro / Persee?

You can do that by either editing nuitrack.config file:

  1. Open nuitrack.config file (/usr/etc/data/nuitrack for Ubuntu and %NUITRACK_HOME%/data for Windows);
  2. Find the section "AstraProPerseeDepthProvider".
  3. Add the following lines to the beginning:
    "RGB": {
    "Width": <width>,
    "Height": <height>
    },
    where <width>, <height> is one of supported resolutions.

or by using Nuitrack API:

After Init() and before any Create(), call…

C++:
Nuitrack::setConfigValue(“AstraProPerseeDepthProvider.RGB.Width”, “640”);
Nuitrack::setConfigValue(“AstraProPerseeDepthProvider.RGB.Height”, “480”);

С#:
Nuitrack.SetConfigValue(“AstraProPerseeDepthProvider.RGB.Width”, “640”);
Nuitrack.SetConfigValue(“AstraProPerseeDepthProvider.RGB.Height”, “480”);

To change depth frame resolution follow the instructions above but replace RGB with Depth.

How can I change Depth/RGB resolution for ASUS Xtion2 / Orbbec Astra?

You can do that by either editing nuitrack.config file:

  1. Open nuitrack.config file (/usr/etc/data/nuitrack for Ubuntu and %NUITRACK_HOME%/data for Windows);
  2. Find the section "OpenNIModule".
  3. Add the following lines to the beginning:
    "RGB": {
    "Width": <width>,
    "Height": <height>
    },
    where <width>, <height> is one of supported resolutions.

or by using Nuitrack API:

After Init() and before any Create(), call…

C++:
Nuitrack::setConfigValue(“OpenNIModule.RGB.Width”, “640”);
Nuitrack::setConfigValue(“OpenNIModule.RGB.Height”, “480”);

С#:
Nuitrack.SetConfigValue(“OpenNIModule.RGB.Width”, “640”);
Nuitrack.SetConfigValue(“OpenNIModule.RGB.Height”, “480”);

To change depth frame resolution follow the instructions above but replace RGB with Depth.

How can I change depth resolution for RealSense?

Please, open %NUITRACK_HOME%/data/nuitrack.config file in a text editor. Add "RawWidth", "RawHeight", "FPS" options to the “Realsense2Module.Depth” section, for example:

"Realsense2Module": {
"Depth": {
"RawWidth": 640,
"RawHeight": 480,
"FPS": 30,
"Preset": 5,
"ProcessWidth": 640,
"ProcessHeight": 480,
"ProcessMaxDepth": 5000,
"LaserPower": 1.0
},
...
}

RawWidth=848, RawHeight=480, FPS=30 are default values. You can try to set other compatible values in accordance with RealSense SDK.
"ProcessWidth", "ProcessHeight" is the resolution which raw input stream will be resized into (it will be upscaled/downsampled and cropped to fit aspect ratio if needed). Please note that changing these parameters may lead to performance issues.

How can I disable the RGB stream?

To disable the RGB stream:

  1. Open %NUITRACK_HOME%/data/nuitrack.config in a text editor;
  2. In the "DepthProvider" section, find the “RGB” parameter and set it to "Off".
How can I select a different rotation angle?

You can set the rotation angle in the nuitrack.config file. Open the %NUITRACK_HOME%/data/nuitrack.config in a text editor, find "DepthProvider.RotateAngle" option and then set one of possible values: 0 (default value), 90, 180, 270.

What is the difference between Nuitrack SDK and Astra SDK?

Astra SDK is based on Nuitrack SDK (it uses skeleton tracking from Nuitrack). New features are implemented first in Nuitrack SDK and then in Astra SDK (with a possible delay). Besides, all new features not connected to skeleton tracking (for example, face tracking) are not implemented in Astra SDK.

Can I use the pre-recorded video as input with Nuitrack?

You can use the data recorded in .oni format using NiViewer tool from OpenNI2 SDK. This usage presupposes that the depth sensor should be OpenNI2 compatible, for example, Orbbec Astra cameras.

To run Nuitrack with .oni file, open %NUITRACK_HOME%/data/nuitrack.config file in a text editor and set the FULL path to the .oni file in the "OpenNIModule.FileRecord" section. Please note that Nuitrack Pro license is not compatible with this mode.

How can I limit CPU usage by Nuitrack?

"CPU.MaxLoad" parameter in nuitrack.config file limits the CPU usage by Nuitrack libraries. By default, this value is 0.67. You can modify this parameter, however, the decreased value may lead to frame skips.

How can I increase the tracking quality with RealSense cameras?

Starting from Nuitrack 0.23.1, two post-processing filters are available using nuitrack.config file: in the Realsense2Module section, you can find the SpatialFilter and DownsampleFactor (corresponds to the decimation filter in RealSense SDK). For example, if you set DownsampleFactor to 2, the noise level in the depth map will be reduced according to the Intel recommendations.

Is it possible to define a tracking area with the SDK so that only people inside that area are tracked and all other are ignored?

Nuitrack isn’t designed for thresholded depth maps. However, you could try the following:

  1. open %NUITRACK_HOME%/data/nuitrack.config file in a text editor;
  2. add this line

"BoxCutter":{"x":"0","y":"0","z":"0","width":"1000","height":"4000","depth":"3000","alpha":"0","beta":"-1.5"},

as the first line to the "Segmentation" section.

Notes about the 2nd step:

  • depth map will be thresholded by height/2 mm (only the depth values [0…height/2] will be processed);
  • “beta” parameter is the BoxCutter’s rotation angle in radians along X-axis of a sensor. So the box above should have following dimensions after rotation: “width” x “depth” x “height/2” (X, Y, Z order here).
We are interested in face detection and recognition. Does Nuitrack support that?

Nuitrack supports face tracking, you can read more about this feature in our online documentation. You can take a look at our Unity tutorials: Face Tracking with Nuitrack and Creating Animated Emojis with Nuitrack. You can also take a look at our product 3DiVi Fase SDK.

When a user stands too close to a wall, Nuitrack detects his skeleton incorrectly (the user “merges” with the wall). Can I prevent this?

You can try to calibrate the background. Find the "Segmentation.Background" section in the nuitrack.config file and replace the line "BackgroundMode": "dynamic" with these two lines:
"BackgroundMode" : "static_first_frame",
"CalibrationFramesNumber": 20
It turns on a static model for the background.
Wait for CalibrationFramesNumber frames, and only then enter the scene.
You can set these options via Nuitrack API as well:
nuitrack::setConfigValue("Segmentation.Background.BackgroundMode", "static_first_frame");
nuitrack::setConfigValue("Segmentation.Background.CalibrationFramesNumber", "20");
Please note that first 20 frames are used to calibrate the background - make sure that the sensor is not moved during this time and no users enter the frame (there should only be your background in front of a sensor). You can add some kind of a calibration counter that will be displayed in your project, which will help you to understand when the calibration is over.

Can I determine whether a user is standing in front of the camera or backwards with Nuitrack?

Nuitrack API doesn’t provide such function. However, you can use Face Tracking to implement this feature: compare face id with skeleton id - if they’re the same, then the person is facing the sensor, if not (face id is not recognized), the person is standing backwards. Learn more about face tracking with Nuitrack.

Can I track 1 joint instead of all joints?

You can change the number of tracked users but the number of tracked joints is always the same.

Can I adjust the laser power of RealSense cameras?

Yes, you can set the laser power of your sensor in nuitrack.config. Find the line "LaserPower" in the "Realsense2Module" section and set the desired value from 0 (min) to 1 (max).

Can I use Nuitrack for skeleton tracking if a sensor is placed behind a glass?

The quality of detection and recognition depends on the distance between a sensor and a glass and also on the glass width. If the sensor is adjacent to a thin glass, it may prevent distortion. Please note that glare and reflections from the glass seriously affect the detection and tracking quality.

How can I check Nuitrack version?

To see the Nuitrack version, please run Nuitrack Activation Tool (Nuitrack.exe). The version is displayed at the bottom of the window.

Can I set the maximum tracking distance?

You can set the maximum skeleton tracking distance in nuitrack.config (%NUITRACK_HOME%/data), find and edit the following value:

Skeletonization": {
"MaxDistance": 4000,

}

"4000" means the distance in millimeters.

Can I use Nuitrack for face tracking only?

If you use Nuitrack, a face is detected only after a user’s skeleton is detected. For face tracking we can recommend you our product 3DiVi Face SDK.

What do USGH letters mean? They’re displayed when I run Nuitrack samples.

“USGH” are Nuitrack modules:

U - User
S - Segment
G - Gestures
H - Hands

Time next to “U, S, G, H” means the working time of each module.

Issues while Setting Up and Running Nuitrack

When I run Nuitrack samples, I get the following error: Start nuitrack c11 WARNING: Can not load library module:C:\nuitrack/middleware/NuitrackModule.dll ERROR: Empty factory for DepthProvider terminate called after throwing an instance of 'tdv::nuitrack::TerminateException' what(): Empty factory for DepthProvider

Usually, this issue is caused by unresolved dependencies. Please check that you’ve installed Microsoft Visual C++ Redistributable (for Windows).

I use “Nuitrack.NuitrackMode.DEBUG” for Nuitrack initialization but the sample does not run!

Please do not use "Nuitrack.NuitrackMode.DEBUG" as it is only used for Wi-Fi connection between PC and a sensor.

Can I hide the “error messages” displayed at Nuitrack start?

We’ve already hidden some of these warnings in our latest release. However, we cannot hide some messages because they come not from Nuitrack, but from sensor drivers (such as "Can't find video camera!", etc.).

nuitrack_ni_sample doesn’t work with my RealSense!

nuitrack_ni_sample shouldn’t work with RealSense cameras as it works only with the sensors with OpenNI support.

I measure Skeleton Tracking FPS but the results are not stable!

We recommend you to disable RGB stream or downgrade the resolution of the RGB stream to qVGA for more stable FPS measurement.

I encounter IndexOutOfRangeException!

Try to:
Open the file %NUITRACK_HOME%/bin/OpenNI2/Drivers/SenDuck.ini
Add the section [Image] and a line “Resolution”:
[Image]
Resolution=1

When I run the sample, I get an error: _"System.DllNotFoundException: 'Unable to load DLL 'libnuitrack': The specified module could not be found. (Exception from HRESULT:0x8007007E)'"_

This error is caused by mixing 32-bit and 64-bit libraries. Please make sure you’re using the 64-bit version of Nuitrack and a 64-bit compiler. Also set "Target platform" to x64 in your project’s properties.

I cannot open Nuitrack sample as a root user.

We provide automatic settings for a non-root user. As for a root user, you have to set environment variables manually. Please, execute these commands as a root user:

export NUITRACK_HOME=/usr/etc/nuitrack
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/nuitrack

Sample SDK projects cannot execute without libnuitrack.dll. Where is libnuitrack.dll?

You have to install the Nuitrack runtime component. Please, follow the installation instructions.

When I run nuitrack_c11_sample, I get the errors: Start nuitrack c11 ... ERROR: fail to detect depth camera! Press any key to EXIT the program...

These error messages indicate that LIPS SDK is installed in your system. Please uninstall LIPS SDK as it conflicts with Nuitrack SDK.

When I run nuitrack_ni_gl_sample (on Windows), I get an error: glut.h: No such file or directory

Please download glut.zip archive with 64-bit versions of libraries and follow the steps below:

  1. Install header files:
  • copy \Include\gl folder to "C:\Program Files (x86)\Windows Kits\10\Include\10.0.16299.0\um" (note that 10.0.16299.0 is Windows 10 SDK version, it can differ for your system)
  1. Install export libraries:
  • copy \Lib\x86\glut32.lib to "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.16299.0\um\x86"
  • copy \Lib\x64\glut32.lib to "C:\Program Files (x86)\Windows Kits\10\Lib\10.0.16299.0\um\x64"
  1. Install dynamic libraries:
  • copy \bin\x86\glut32.dll to C:\Windows\System32
  • copy \bin\x64\glut32.dll to C:\Windows\SysWOW64

If you’re using x64 compiler, you have to use "C:/Program Files/OpenNI/Lib64/openNI64.lib",
If you’re using x32 compiler, you have to use "C:/Program Files(x86)/OpenNI/Lib/openNI.lib".

I use RealSense with a long cable and Nuitrack but I have some FPS issues.

As far as we know, long cables cannot provide necessary capacity due to USB 3.0 specifications. Please refer to the following issue.

Faces are not tracked!

By default, face tracking in Nuitrack is turned off. To turn on this function, open the nuitrack.config file and set “Faces.ToUse” and “DepthProvider.Depth2ColorRegistration” to true.

Issues with Nuitrack and Unity

I cannot build my Unity project with Nuitrack!

Make sure that you installed Nuitrack correctly following our Installation Instructions.

Also make sure that you have installed the msi packages from the downloaded Nuitrack archive. You need these packages installed on your computer to resolve missing dependencies for the Nuitrack module.

Try to run Basic Unity Sample from Nuitrack unitypackage to check whether Nuitrack works fine or not: import NuitrackSDK.unitypackage to your Unity project and run “BasicSkelet” from the "Scenes" folder.

I get “DllNotFoundException: libnuitrack” error!

Make sure that NUITRACK_HOME and PATH environment variables are set correctly and for all users. You have to restart Unity after setting these variables.
In Unity, select “Nuitrack SDK/Nuitrack/NuitrackAssembly/nuitrack.net.dll” and tick "Any Platform" in the “Inspector” tab.

When I run Nuitrack samples, my console log is filled with errors stating that "The type or namespace name `nuitrack' could not be found. Are you missing an assembly reference?".

Please, try to select Nuitrack SDK/Nuitrack/NuitrackAssembly/nuitrack.net.dll (in Unity) and tick "Any Platform" in the “Inspector” tab. This issue may occur in case this value is unticked.

I'm getting this error when starting the Unity demo scenes: Nuitrack native libraries initialization error: INIT_NUITRACK_NOT_SUPPORTED

This error occurs at start of a Unity project with Nuitrack, however, it does not indicate any particular error. So don’t worry, this is a regular case and everything should work fine.

Questions about Nuitrack and TVico

Can I transfer depth data, user masks and skeleton data using TVico and PC?

For Linux and Windows desktops, there is a limitation in the current Nuitrack version: only skeleton data can be transferred via wireless connection.

If you connect TVico to an Android device, you can transfer all data (depth, user masks, skeleton).

Can I connect TVico via Ethernet or Wi-Fi?

You can use either Ethernet or Wi-Fi for connection. For more information about connecting TVico to an Android device/PC, take a look at our TVico USer Guide.

What is the difference between TVico and Orbbec Persee?

TVico is an interactive Android computer developed jointly by Orbbec and 3DiVI Inc. It is a Persee camera supplied with Nuitrack middleware and Nuitrack perpetual license. You can use TVico to transmit depth data, user masks and skeleton data to other devices using Wi-Fi. Android, Windows and Linux platforms are supported.

Nuitrack Documentation

Where can I find Nuitrack documentation?

You can find Nuitrack online documentation here.

Is there documentation for nuitrack.config file? I want to change all the parameters!

Unfortunately, there is no documentation for nuitrack.config file at the moment as it is not intended to be edited in most cases. It includes internal parameters for modules and experimental features.


Deauthorize license D435
LicenseNotAccquiredException
Frame rate is low using SDK on Android platform
Finetuning gesture tracking Intel RealSense D435
#2

#10