Multiple Sensors / Testing the Device API

Nuitrack SDK Reporting Software Issues
1

Hello everyone,

I’m currently looking into the multiple camera / device API feature introduced in v0.33.0.

As of today, even though this feature seems to be quite new, Nuitrack seems to be the only product advertising this … while Intel and Orbbec allow to connect multiple devices just fine (Multiple Color+Depth), Intel doesn’t have any Bodytracking (besides the totally different Cubemos SDK which does ML only) and Astra does Bodytracking for a single sensor only.

And as we have a usecase where we’d need three sensors + bodytracking on one machine, the timing for this feature’s arrival seemed just perfect.
Not sure if this would actually still be feasible as each Nuitrack process would eat up some of the system’s ressources, but I’m ignoring this possibility for now.

So far I haven’t been able to make it work, and I even tried three different implementations which all showed the same erratic behaviour.
I’m running SDK v0.35.5 + latest matching runtime off github.
These are the things I tried:

  1. Opening the activation tool twice, opening a different sensor on each
  2. nuitrack_csharp_device_api_sample shipped with SDK
  3. I added the device API to the vvvv implementation (link is missing because I can only add two links as a new user) … vvvv is a visual programming environment extensively used in the creative industries.

I have two Orbbec Astra and an Intel RealSense D435.

  1. When having just 1 Astra and 1 RealSense connected, it was somehow working …
    When opening the Astra first, the RealSense second, the Astra stops working once the RealSense opens.
    But I can restart the Astra and then have both working in parallel.
    Not ideal, but it’s something.

  2. When trying to discard the RealSense and using the two Astra instead, one always stops working when the second opens.
    Restarting the other camera just results in the other device stopping.

Seems like an issue with Astra only, I don’t have a second RealSense at hand at the moment.
Astra seemed to give me more stable results for the body tracking though.

I could always pair each sensor with a dedicated i.e. Latte Panda and stream everything to the main application over network, but I had hoped Nuitrack might have solved this by now and I was eager to try it. It would have been a more elegant solution, keeping the involved hardware components to a minimum.

Sharing my results here, maybe I got something wrong / someone ran into the same issues before and can advise here.

Johannes

2

I have been working multiple Astra Orbbec Pros for a project and I ran into the same problems you describe. I created a similar setup using just Astra’s SDK. While multiple sensors work there, the Color streams for any cameras hooked up beyond the first one do not work and show a black screen. Depth frames and skeleton tracking work as expected.

Back in Nuitrack, I tried disabling the color stream based on information from the FAQ (“How can I disable the RGB stream?”) and this fixed the problem for the 2nd sensor. Of course this is a problem if you still need the Color camera. I don’t know if this is a recent issue with the Astra sensors because there is a sample in the Astra SDK that displays views from multiple sensors, color and depth, but I have not been able to replicate the screenshot in their user guide in my testing.

Hope this helps.