Realsense Rosbag playback results in Unhandled exception

Nuitrack SDK
1

When trying to create a Nuitrack instance using a Realsense recording, tdv::nuitrack::SkeletonTracker::create() results in an exception in the RealSense.dll.

Unhandled exception at 0x00007FFC6F769980 (RealSense.dll) in *.exe: Stack cookie instrumentation code detected a stack-based buffer overrun.

I have just reinstalled NuiTrack and applied for a new trial license. I overrwrote the old dll in my build directory, so everything should be up to date. The recording is working as well.

Here is the important part of the stack trace:

image
image743×53 2.48 KB

Following this are a couple calls to libnuitrack.dll (4 calls), NuitrackModule.dll (14 calls) and RealSense.dll (11 calls).

I also tried setting DeviceHardwareReset to true, to no avail.

Here is the initialisation code:

tdv::nuitrack::Nuitrack::init();

tdv::nuitrack::Nuitrack::setConfigValue("Realsense2Module.FileRecord", (m_RecordingDirectory / recordingPath).string()); // <-- The path is correct I checked

// Create Tracker
m_SkeletonTracker = tdv::nuitrack::SkeletonTracker::create(); // <--- This is where the exception occurs

try {
	tdv::nuitrack::Nuitrack::run();
}
catch (const tdv::nuitrack::Exception& e) {
	mp_Logger->log("Error running Nuitrack: " + std::to_string(*e.what()), Logger::Priority::ERR);
}

I also checked all environment variables and restarted my computer multiple times.

Thank you for any help that you may offer.

2

I was wondering if anyone had ever gotten this kind of issue before? I have now tried with an ONI file recorded using an Orbbec camera and I have the same issue (just with a different dll).

Any help would be greatly appreciated.

3

Hi @LeoPohl

Have you tried using official examples with these recordings?

4

I have tried using the exact code from the examples with an adjusted config file and I ran into the same error. However, I did not build it using cmake. I rather linked it through visual studio and copied the necessary dlls into the bin folder, I dont know if that makes a difference.

5

Hi @LeoPohl, following up on our email thread, could you please attach a minimal reproducible example for us to diagnose ?
This is a only way-to-go, as we are unable to allocate resources to reproduce from scratch (currently it isn’t reproducible on our side).

6

Hi @TAG,

Thanks for coming back to me. I created a minimal project in which I included a sample recording. I checked that the file is okay. As explained earlier, I did not use cmake to build the project, but rather included the library and the header file in visual studio. Additionally I copied all dll from the bin folder in the nuitrack install into the build directory.

I think that the issue is not in the code that I wrote but rather the way I setup the project. Therefore I created a git repository including the project setup. LeonardoPohl/NuitrackBagSample · GitHub
I added the nuitrack-sdk directly since submodule wasnt working for some reason.

7

Hi,

were you able to reproduce my issue with the setup that I provided?

8

I have now also reproduced the issue in the exact setup as mentioned in the example. I shouldve done that earlier but was busy in the mean time. The only code that I have changed is change this:

auto devices = Nuitrack::getDeviceList();
Nuitrack::setDevice(devices[0]);

to:

Nuitrack::setConfigValue("Realsense2Module.FileRecord", "H:\\Recordings\\Session_2023-02-08T22.21.33_Realsense_Camera_0.bag");

And I am getting the same error as in my project. The file is working I tested it with the realsense sdk as well as in the realsense viewer.

9

I have now also tried with python and unity to no avail. Is there anything I am missing for the file playback? I also tried to record a new recording using the intel realsense viewer but I am getting the same results. For python I used the sample code nuitrack-sdk/PythonNuitrack-beta at master · 3DiVi/nuitrack-sdk · GitHub in the PyNuitrack Usage section.

Without change it works as intended, but once I changed


devices = nuitrack.get_device_list()
for i, dev in enumerate(devices):
	print(dev.get_name(), dev.get_serial_number())
	if i == 0:
		#dev.activate("ACTIVATION_KEY") #you can activate device using python api
		print(dev.get_activation())
		nuitrack.set_device(dev)

to

recording_dir = "H:/Recordings"
recording_name = "Session_2023-02-08T22.50.59_Realsense_Camera_0.bag"
print(f"Playback from {recording_dir}/{recording_name}")
nuitrack.set_config_value("Realsense2Module.FileRecord", f"{recording_dir}/{recording_name}")

but then it crashes at nuitrack.create_modules() without any warning.

I also tried to use the unity implementation. Again the same thing happened. I tried without changing anything and it worked fine (the first project tutorial). And once I selected a file in ‘Use Record file’ it crashed unity completely.

If any of the two are easier to fix than the cpp version I would be happy. I am trying to compute the skeleton for multiple recordings and I could do that in a different programming language but I would prefer either python or cpp since I am more familiar with those rather than Unity.

10

I have been thinking of an alternative. I was thinking that it might be possible to store the skeleton in file during the original recording. I noticed that streaming and calculating the skeleton is possible. However, I would rather not rework the recorder that I have developed too much. Is there a way to manually set the depth pointer and color image so I don’t run into synchronisation issues with the recording?

11

I have now tried to run Nuitrack at the same time as I do the recording and noticed that it is not possible, since I have two separate conflicting calls to the Realsense API. So synchronisation is not my only issue. :smiley:
Do you know of any way around this?

12

I have now implemented a version which just Nuitrack and not Realsense. This seems to work, however at quite a low framerate but the deadline is coming closer so I will stick with this implementation for the time being.

13

Hi @LeoPohl

Looks like it’s a bug on our end. The fix was alredy implemented and will be released this week.

In the mean time you can use this workaround:
After:

Nuitrack::setConfigValue("Realsense2Module.FileRecord", "H:\\Recordings\\Session_2023-02-08T22.21.33_Realsense_Camera_0.bag");

add:

auto devices = Nuitrack::getDeviceList();
Nuitrack::setDevice(devices[0]);

This should do the trick.

14

Thank you for your response. I am glad that it is fixed.

However, I have already worked around the issue by recording the skeleton while recording the camera stream. I will share it, maybe someone might find it useful or interesting even though it is an extreme edge case which probably no one would ever encounter since there are better solutions to this problem.

I found that the main influence of the framerate was storing the frames during the stream. But since I am only aiming at recording 300 - 600 frames each recording I can store everything in memory (it costs around 4GB in just frames) and then store the frames using opencv afterwards. The recordings are quite huge compared to the more optimised ros-bag format coming in at around 2GB for 300 frames but they are human readable and therefore uncompressed. With this setup I can reach around 15 to 20 frames per second for the recording. I also store all skeleton data in a json for future use.

I think this thread can be closed now.

Thank you again for your answer.

15