Nuitrack + RealSense D435i + Oculus Quest

Nuitrack SDK Unity3D
1

Hello everyone,

I’m trying to implement Skeleton Tracking with a RealSense D435i depth camera connected via USB-C to an Oculus Quest VR headset, using the Nuitrack SDK for Unity.

So far I have personally tested and confirmed that the depth camera and the RealSense SDK is compatible with the Oculus Quest, as is documented here and here.

For this skeleton tracking project though, I attempt to skip completely the RealSense SDK and use the Nuitrack SDK instead, without luck so far. The skeleton does not show up, the RealSense sensor is not initialized (no IR pattern projected) and the logcat output is this:

InitNuitrackLibraries() starts.
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)

Nuitrack callback: onInitFailure
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)

Exception: UnityEngine.AndroidJavaException: java.lang.NullPointerException: Attempt to read from field 'android.content.pm.ServiceInfo android.content.pm.ResolveInfo.serviceInfo' on a null object reference
at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <549ed8055ed14737ac3b841d26182fec>:0
at UnityEngine.AndroidJNISafe.CallStaticVoidMethod (System.IntPtr clazz, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <549ed8055ed14737ac3b841d26182fec>:0
at UnityEngine.AndroidJavaObject._CallStatic (System.String methodName, System.Object[] args) [0x00036] in <549ed8055ed14737ac3b841d26182fec>:0
at UnityEngine.AndroidJavaObject.CallStatic (System.String methodName, System.Object[] args) [0x00001] in <549ed8055ed14737ac3b841d26182fec>:0
at NuitrackLoader.InitNuitrackLibraries () [0x00045] in <9d623a2bde8a41b9b8be63a499144ed4>:0
.
.
// Unrelated Oculus stuff //
.
.
Unable to find libnuitrack
Unable to find nuitrack
DllNotFoundException: libnuitrack
at (wrapper managed-to-native) nuitrack.NativeImporter.nuitrack_InitializeFromConfig(string)
at nuitrack.NativeNuitrack.Init (System.String config) [0x00000] in <e994b21777124ea9bce8f91cd774f163>:0
at nuitrack.Nuitrack.Init (System.String config, nuitrack.Nuitrack+NuitrackMode mode) [0x00029] in <e994b21777124ea9bce8f91cd774f163>:0
at NuitrackManager.NuitrackInit () [0x0005f] in <9d623a2bde8a41b9b8be63a499144ed4>:0
at NuitrackManager.OnApplicationPause (System.Boolean pauseStatus) [0x0004f] in <9d623a2bde8a41b9b8be63a499144ed4>:100:

Apparently something is failing when the Nuitrack.init() java function is called from the NuitrackLoader.cs script at the beginning of the game, triggering the NuitrackCallback.onInitFailure() callback.

When I printed the NuitrackLoader.initState variable inside the onInitFailure callback, I got the value INIT_NUITRACK_MANAGER_NOT_INSTALLED , which is to be expected, since onInitSuccess(), which is the only function that changes the initState to INIT_OK is never called.

What could be preventing onInitSuccess from running ? What is the meaning of this

java.lang.NullPointerException: Attempt to read from field ‘android.content.pm.ServiceInfo android.content.pm.ResolveInfo.serviceInfo’ on a null object reference

null reference exception?

Any help from the Nuitrack dev team would be greatly appreciated !!

Extra Notes:

  • I am working with the First Project > BasicSkeleton scene.
  • I have included the android .so libraries in my project (libmiddleware.so, libnuitrack.so and libOpenNI.so) and placed them inside Assets>NuitrackSDK>Nuitrack>Nuitrack Assembly, alongside the default plugins coming with the Unity package.
  • I have sideloaded the Nuitrack.apk on my Oculus Quest, but this doesn’t seem to make things better.
  • I build the project for Android 7.1 (API Level 25) since this is the one supported by the Quest.
  • I am using Unity 2019.1.4f
  • The project runs fine when targeting Windows, and connecting the D435i to the PC
2

btw, Nuitrack already worked with wireless Android VR headsets, but setup was different:

  1. Use Android box TVico www.tvico.io instead of RealSense. TVico can send tracking data wirelessly via WiFi to any device (“Wireless” case).
  2. VicoVR apk installed on Android VR headset as explained in documentation for the “Wireless” case
  3. Android application is developed using Nuitrack Unity plugin, that receives data from VicoVR apk installed on the same headset.
    Videos:
    https://youtu.be/Re-G5i5RCRw
    https://youtu.be/hDGNY0fJTFc
    more on 3DiVi channel https://www.youtube.com/user/3divicompany/videos
3

Hey @Dmitow, It’s very good to know about TVico as an alternative, and thank you for the links! However, in this particular case, due to size/power/portability requirements the hardware of choice has to be the RealSense, so I can’t circumvent the problem, I have to solve it directly.

4

Hello,
could anyone from the Nuitrack developer team assist me on this please?

5

Hi George,

Unfortunately, it’s not possible to use Nuitrack in this case (with RealSense and Oculus Quest) because TVico.apk cannot be used with RealSense. The only solution for now would be TVico (with TVico.apk installed) + Android + Oculus Quest (as @Dmitow mentioned before).

6

Hi Olga,

and thanks a lot for your reply and your contributions in the forums in general!
Just to clarify, I am not attempting to send data to the Quest wirelessly via WiFi. The particular requirements of our project dictate a sensor that is small enough to be attached onto- and powered by the headset itself.

Therefore, I am directly connecting the RealSense to the Quest’s USB-C port. I have confirmed that this type of connectivity is valid and operational, as I am able to acquire the RGB+D frames from RealSense in an Android Unity app, using the RealSense SDK.

What I am really after though, is the middleware for extracting skeleton data from RGB+D images, and as far as I understand, as of today, Nuitrack is the most robust way of doing that on Android, without entering the field of PoseNet / OpenPose AI solutions.

So far, treating the Quest as just another Android device, has worked for me without problems. Including the necessary .aar or .so Android plugins, and initializing the necessary native Java classes from Unity, along with the correct Android manifest does the trick.

If indeed, the procedure for running Android Nuitrack apps, does not work for the Quest specifically, I would be very interested in understanding why, or where exactly the bottleneck is.

It’s a little hard for me to accept that after overcoming issues such as drivers, power requirements and data transfer rates, I am now stuck on the middleware, which theoretically feeds on image data and should be device agnostic.

7

Sorry for misunderstanding.

  1. Please advise what is Android OS version on Oculus Quest?
  2. What RealSense SDK version do you use?
  3. Please install Nuitrack.apk to Oculus Quest and try to activate the trial license (you can generate a trial key in your personal account). As you run the activation tool, click the “Compatibility Test” button to check Nuitrack operation. And also please send us the adb logcat after you perform the test, it would be very useful for us.

Support for RealSense on Android was added in Nuitrack v0.26.0. However, we’ve only checked Nuitrack with D415 and D435. We’re currently waiting for D435i to test it. That’s why we ask you to test trial Nuitrack with this sensor.

By the way, this step is not required and shouldn’t be performed:

8

I am currently away from my dev environment, so I can directly answer 1. and 2. and will get back to you later with the logcat output.

  1. Android 7.1 (API Level 25)
  2. Realsense SDK 2.22

Thanks for clarifying about the native libraries.

9

Hi,

First of all, thank you @George.Adamon It is a huge help to follow your work on Quest.

I am also working on the same issue. I have tried to run RealSense D435i with Nuitrack SDK in Unity. But it crashes. I wonder have you got any further to succeed.

10

Hi,

I am also working on this issue.

I installed Nuitrack.apk on my Oculus Quest and ran the Activation Tool. However, “Compatibility Test” did not work.

  • After clicking “Compatibility Test”, the dialog of camera permission is displayed.
  • After, the screen remains black and the depth map is not displayed.

the logcat output (filtter Tag=NUITRACK) is this:

onInitSuccess
RGB device VID: 11205, PID: 1281
WARNING: AstraProPerseeDepthProvider: No permissions for RGB device VID: 11205, PID: 1281
WARNING: KinectDepthProvider: No permissions for device (VID: 1118, PID: 686)
DevicesList size: 0
ERROR: Can't create DepthSensor module
AstraProPerseeDepthProvider: Can't create RGB Stream (can't open video capture)
KinectDepthProvider: Can't init context
OpenNI2DepthProvider: Can't open device (	DeviceOpen using default: no devices found
)
OpenNIDepthProvider: Can't create OpenNI DepthGenerator (OpenNI Status: Can't create any node of the requested type!)
Realsense2DepthProvider: Can't enumerate Realsense2 camera
Can't create DepthSensor module
AstraProPerseeDepthProvider: Can't create RGB Stream (can't open video capture)
KinectDepthProvider: Can't init context
OpenNI2DepthProvider: Can't open device (	DeviceOpen using default: no devices found
)
OpenNIDepthProvider: Can't create OpenNI DepthGenerator (OpenNI Status: Can't create any node of the requested type!)
Realsense2DepthProvider: Can't enumerate Realsense2 camera
RGB device VID: 11205, PID: 1281
WARNING: AstraProPerseeDepthProvider: No permissions for RGB device VID: 11205, PID: 1281
WARNING: KinectDepthProvider: No permissions for device (VID: 1118, PID: 686)
DevicesList size: 0
ERROR: Can't create DepthSensor module
AstraProPerseeDepthProvider: Can't create RGB Stream (can't open video capture)
KinectDepthProvider: Can't init context
OpenNI2DepthProvider: Can't open device (	DeviceOpen using default: no devices found
)
OpenNIDepthProvider: Can't create OpenNI DepthGenerator (OpenNI Status: Can't create any node of the requested type!)
Realsense2DepthProvider: Can't enumerate Realsense2 camera
Can't create DepthSensor module
AstraProPerseeDepthProvider: Can't create RGB Stream (can't open video capture)
KinectDepthProvider: Can't init context
OpenNI2DepthProvider: Can't open device (	DeviceOpen using default: no devices found
)
OpenNIDepthProvider: Can't create OpenNI DepthGenerator (OpenNI Status: Can't create any node of the requested type!)
Realsense2DepthProvider: Can't enumerate Realsense2 camera

Notes:

  • Android 7.1.1 (Headset Version: 333700.3780.0)
  • Connecting D435 (Not i)
  • TVLauncherGo is used to run Nuitrack.apk.
11

Hi Tatsuya,

According to the logcat output, your sensor isn’t detected. Please make sure that you connect your sensor via USB 3.0 (try different ports). If a sensor is detected, you should see a pop-up window asking for access to USB.

By the way, what Nuitrack version do you use?

12

Hi Olga

thank you for your answer.

The pop-up window requesting access to the USB device is displayed correctly. However, after that, the screen remains blacked out.
Unfortunately, Oculus Quest has only one port, so you can’t try other ports.

By the way, what Nuitrack version do you use?

Nuitrack app version is “0.28.0”.

13

If possible, try to check Nuitrack on other Android devices (a tablet or a smartphone, for example) with Android 7. Also please note that read permission for a root folder on an Android device is required. You can check the permissions using any file manager, for example, ES File Manager: if the “/” folder is empty, then read permissions for the root folder aren’t granted in the current firmware version of your Android device.