Evaluating the best way to achieve VR hand and gesture tracking, I was experimenting with Leam Motion with mixed results.
Interested to know if there are experiences into this kind of usage with Nuitrack and what kind of sensor might more suited for the job.
There is no integration with LeapMotion. The 3DiVi company has a VR solution for Android devices called a TVico (which is an Orbbec Persee depth sensor bundled with Nuitrack SDK). You can find more information about this product at the website https://tvico.io. Several VR applications are also available on the website
1 Like
Thanks @iosif.semenyuk
We are not using an Android yet a Windows platform.
Any solution for this using Nuitrack and any suitable sensor.
Hi davide445,
only skeleton data can be transferred to Windows or Linux desktop PC when using TVico or VicoVR device. There is no VR case support for “Nuitrack on PC + USB camera” set at this moment.
@mikhail.khayrislamov ok for skeleton if this can be for the hand only.
But if it’s not possible to get it trough a 3d camera on Windows this is a no go on our side.
could you please describe your use case in more details?
- Do you need to transfer hand tracking data from Windows PC (with connected USB sensor) to a HMD?
- What data do you need: wrist positions, hand clenching (grab), smth else?
- What are the main components? Windows PC (USB sensor + your application), HMD (client which receives tracking data), …
Nuitrack could provide hand tracking info (please refer to Hand Tracker module documentation page). However the way you could proceed data depends on your use case.
Looking forward to your reply.
- we just need hand tracking. The development and runtime platform it’s based on Windows, so on Windows we need to catch the data coming from the sensor mounted on the HMD and track the fingers position so to provide our application fingers position and gesture recognition.
- we need basically three kind of recognized gestures: touching an object (so just tracking the position to understand if you are crossing the object boundary), grab an object and activate a button (for doing some action). Being our users totally inexperienced nor interested in technology we need this in the most reliable way. So no tracking loss due to occlusion, no jitters, no wrongly bent fingers just since they are not well tracked, fast track recovery if the hands are out for sight, etc
- Windows PC (with attached via USB the tracking sensor, running our app), HMD (with attached the tracking sensor), probably a treadmill for locomotion sensing
What appear to me strange is searching on YT “Nuitrack hand tracking” your really didn’t find any example at all, so understood there is the capability but just wondering if anyone is using it.
Also discovered this thread
And if Nuitrack doesn’t provide any out of the box hand tracking but just access to the data needing an extensive custom developed post filtering this is really not for us. But maybe I misunderstood or this is not current status.
If I understood correctly you need to run Nuitrack on Windows PC and combine tracking data from 2 sources: USB sensor attached to PC (touching, grad) and tracking sensor mounted on the HMD (finger positions). If yes, you could firstly test free Nuitrack Trial version for Windows by running compiled sample nuitrack_sample.exe or Unity 3D-sample allModulesScene from Nuitrack SDK. Refer to https://nuitrack.com/ to get Nuitrack runtime installer (v0.30.0) and Nuitrack SDK (v1.5.0).
One more note is Nuitrack has no finger tracking module (finger tracking is in our roadmap, but so far without an exact implementation date).
@mikhail.khayrislamov sorry for not being clear. We need to track hands, using a sensor on the HMD.
Since currently we have only the Leap Motion and this is not supported from NuiTrack, want to ask what of the supported HMD-mountable sensors you suggest us to purchase to have a test, so to maximize the tracking accuracy and robustness.
Unfortunately, support for Leap Motion is not the only problem. Tracking of fingers/hands, including the case with a camera mounted on HMD, is a different task. Current composition of Nuitrack algorithms doesn’t cover this case. Nevertheless, tracking of fingers is in our road map but the exact time frame hasn’t been determined yet (please note that the case with a camera mounted on HMD is not the main one).
1 Like
@mikhail.khayrislamov good to know, at this stage any suitable alternative you can suggest for testing?
unfortunately, camera-on-HMD case isn’t supported yet. Thus reliable accuracy level couldn’t be guaranteed even for supported USB-cameras.