Getting "tdv::nuitrack::ModuleNotInitializedException" in C++ app with Intel RealSense D415

gino-alink · July 19, 2018, 12:46am

I have this code:

int main(int argv, const char** argc)
{
    // Initialize Nuitrack
    try
    {
        Nuitrack::init();
    }
    catch (const Exception& e)
    {
        std::cerr << "Can not initialize Nuitrack (ExceptionType: " << e.type() << ")" << std::endl;
        return EXIT_FAILURE;
    }

    // Create HandTracker module, other required modules will be created automatically.
    auto handTracker = HandTracker::create();
    // Connect onHandUpdate callback to receive hand tracking data
    handTracker->connectOnUpdate(onHandUpdate);

    // Release Nuitrack
    try
    {
        Nuitrack::release();
    }
    catch (const Exception& e)
    {
        std::cerr << "Nuitrack release failed (ExceptionType: " << e.type() << ")" << std::endl;
        return EXIT_FAILURE;
    }
}

When I run it, I get a “ModuleNotInitializedException” error.

That code is an exact copy from the nuitrack_console_sample app.
The sample app works OK with no problems/errors.
But the same code fails on my own trial app.

I don’t use a custom config, same as the sample app.
I’ve also tried different *Tracker modules but all fails with the same exception.

I’ve checked the project settings/dependencies, and they both match.
The only thing I can think of is that I’m still using a TRIAL copy/license, since I’m still checking if we can use Nuitrack for our purposes.

Does the TRIAL license not work outside of the sample app?
Or is there something else?
There isn’t really much documentation whatsoever as to the reason for “ModuleNotInitializedException”.

Westa · July 19, 2018, 2:58am

Try starting with creating a DepthSensor - it is the basis of all the other engines

use this to verify that a frame is being captured.

_depthSensor = DepthSensor::create();

The userTracker is the next stage in the requirements tree - add it next and test it

_userTracker = UserTracker::create();

The skeletonTracker is not a direct requirement of the hand tracker according to the schematics - BUT in the default nuitrack.config - the association is turned on by default. Test it is there next.

_skeletonTracker = SkeletonTracker::create();

Once all these are running - then I would test the handTracker.

Westa

gino-alink · July 19, 2018, 3:19am

Thanks for the reply.

I tried the codes you mentioned.
I still get the same “ModuleNotInitializedException” error

Those are the same codes from the nuitrack_gl_sample app, right?
I checked that sample app and it’s working OK as-is.
I then tried all the different *Sensor and *Tracker modules, but all resulted in the same exception.

I read something in another post, and I already have my system vars set:

NUITRACK_HOME (“E:\Nuitrack\nuitrack-win64\nuitrack”)
PATH (added “E:\Nuitrack\nuitrack-win64\nuitrack\bin” and “E:\Nuitrack\nuitrack-win64\nuitrack\data”)

gino-alink · July 19, 2018, 3:21am

We also tried on an Ubuntu 18.x machine.
We’re getting the same results.

We’re thinking now it’s a device issue…?

Westa · July 19, 2018, 3:32am

A couple of things to check …

Are you environment variables set globally or for a single user? They need to be set globally.

Have you rebooted your computer since you set the environment variables? Without a restart they wont likely be seen by a dynamic library.

There is no technical reason that you need to set the \data folder in the path. The default nuitrack.config should find all the files it needs based on the NUITRACK_HOME variable. Provided that you have not edited the config file.

OK - SOOOO - there is also a couple of very weird known issues with the Intel Realsense sensors.
There are a number of reports that the d415 does not always connect correctly to the USB port - and when it doesnt connect correctly - it winds up running in a very degraded performance state. This degraded state is NOT compatible with the nuitrack system.

ALSO - there is an issue we have seen with the d435 which results in the sensor failing to connect to nuitrack more than once without requiring a hard reset - LITERALLY disconnecting the d415 is one workaround we have found - the other involves doing a hardware reset using the realsense sdk - BEFORE starting nuitrack.

Start by doing some tests with the realsense viewer - plugin the sensor and look at the depth settings - if you cannot select above 640x480 - then your device is running in degraded USB 2.0 mode. (NOTE - it may still SAY its USB 3.0 - this is a bug in the realsense sdk current) This will cause issues with nuitrack - which needs to connect at higher raw resolutions.

Westa

gino-alink · July 19, 2018, 5:38am

Thank you very much for the reply.

I checked and it’s correctly set as a global/system var.
And I did restart my machine after updating.

I tried doing a Hardware Reset using the RealSense Viewer app.
I also tried literally unplugging - re-plugging the device.
Same error.

It’s not running in degraded mode since I can set above 640x480 (i.e. 1280x720) and I can use all the RealSense features normally. It’s also showing USB 3.2 (and it’s detected correctly in Device Manager).

FWIW, these are my RealSense details:

Device Model: D415
Firmware: 05.09.14.00 (this seems to be the latest from the download center)
SDK: v2.12.0

The RealSense SDK isn’t the latest (v2.13.0) but my understanding is that Nuitrack does not depend/use the RealSense SDK anyway, since it should connect directly to the camera.

gino-alink · July 19, 2018, 6:42am

OK, I found the problem: Nuitrack does not work with RealSense SDK v2.12.0.
After downgrading the RealSense SDK to v2.11.1 then using the downgraded realsense.dll, the Exception went away and I can use the APIs normally now.

In the Nuitrack sample apps, it seems the bundled realsense.dll was also built from the same v2.11.1 release.
That’s why it was working.

I may have missed some documentation on supported versions somewhere… but that’s the case right now.

Westa · July 19, 2018, 11:00pm

Actually - nuitrack ONLY works with the realsense sdk - that is what the realsense2.dll file is in the nuitrack bin folder. It is how the nuitrack system communicates with the realsense sensor.

Its been a bone of contention for some time that the nuitrack sdk does not keep itself in sync with the latest realsense sdk - which makes some of our other development works more complicated.

Westa

gino-alink · July 20, 2018, 12:00am

Yeah, I also realized this yesterday.
We have to use the realsense.dll bundled with Nuitrack.
Otherwise Nuitrack won’t work.

I asked Nuitrack directly about this and they said:

We’ve used RealSense SDK v.2.11.1 because this SDK version is recommended by Intel: https://realsense.intel.com/intel-realsense-downloads/#firmware

We regularly update the realsense.dll bundled with Nuitrack but not for all RealSense SDK releases.

So, looks like we’re stuck with the bundled v2.11.1 if we want to use Nuitrack.

Still, thanks for all the help @Westa

Westa · July 20, 2018, 12:19am

FWIW - Intel are now internally recommending new versions for a number of hardware bug related reasons - specially with the d435 - but the intel website takes a lot of time to catch up with the internal development.

Once intel updates their public recommendations im sure nuitrack will advance their core.

Westa

ChrisBao · September 3, 2018, 9:58am

Hi,
Westa, I bought a D415 today, I test the device and found that the quality of rgb image is really bad，it looks like the image was smoothed by a gaussian kernel，is that true？？

Westa · September 3, 2018, 11:15pm

It depends a lot on how you are looking at the image.

If you are looking using the realsense viewer - you see a direct stream of the RGB image.

If you are looking at a frame using nuitrack - then you will be seeing a processed frame.
By default - the size of the RGB frame is downsized by nuitrack to 640x480 which would have the effect of softening the image.

We have found for the 415 that the best approach is to manually set all the frame info for the best results.

// Realsense Depth Module - force to 1280x720 @ 30 FPS
Nuitrack::setConfigValue(“Realsense2Module.Depth.Preset”, “5”); // this set to 3 sometimes helps as well
Nuitrack::setConfigValue(“Realsense2Module.Depth.RawWidth”, “1280”);
Nuitrack::setConfigValue(“Realsense2Module.Depth.RawHeight”, “720”);
Nuitrack::setConfigValue(“Realsense2Module.Depth.ProcessWidth”, “1280”);
Nuitrack::setConfigValue(“Realsense2Module.Depth.ProcessHeight”, “720”);
Nuitrack::setConfigValue(“Realsense2Module.Depth.FPS”, “30”);

// Realsense RGB Module - force to 1280x720 @ 30 FPS
Nuitrack::setConfigValue(“Realsense2Module.RGB.RawWidth”, “1280”);
Nuitrack::setConfigValue(“Realsense2Module.RGB.RawHeight”, “720”);
Nuitrack::setConfigValue(“Realsense2Module.RGB.ProcessWidth”, “1280”);
Nuitrack::setConfigValue(“Realsense2Module.RGB.ProcessHeight”, “720”);
Nuitrack::setConfigValue(“Realsense2Module.RGB.FPS”, “30”);

Westa

Westa · September 3, 2018, 11:23pm

Intel has some good details on the presets as well which is worth looking at:

wESTA