HI Jake,
There is really no simple answer to these questions. It has a lot to do with your usage case.
What level of accuracy you require. What level of repeat-ability you require. What level of detail you require.
But for what its worth - with the d435 - 848x480 is the highest recommended pixel width - this is basically the hardware resolved width … Any wider setting is giving you a digitally interpolated result from the sensor - which according to intel results in higher RMS.
Have a look on the realsense community forum - there is a lot of detail on the performance of each sensor and a lot of explanations starting to be forthcoming from intel about the limitations of the d435 - it would have been nice to see these before we all purchased d435s because they looking useful on paper.
The way that the nuitrack system works is by using algorithms (AI like) to recognise that there is a body in the field of view - and then uses further algorithms to calculate where parts of the skeleton for that body “likely are”.
I use the term LIKELY ARE - since its all a set of approximations based on the knowledge of anatomy and human movements and inverse kinematics type math. For this all to work optimally - you need access to reliable and repeatable depth data.
As the errors increase in the depth data - the accuracy of the reporting of the location of the points on the skeleton becomes less accurate from frame to frame which leads to jitter and jerky movements.
Intel uses a lot of post processing to attempt to clean up their data - and remove holes and such – but this also has the potential to actually increase the number of frame to frame “errors” in the sense that a post processed cleaned up hole in one frame - may not be a hole in the next frame - as instead reports a depth - which can result in increased jitter in the tracking.
This is further compounded at contact points such as the floor and occlusion points for example where the hand occludes the elbow or the shoulder.
In the case of contact points such as the floor - the errors result in the reported location of the ankle and foot demonstrating quite high jitter and inaccuracy - as the exact location of the contact point between the foot and the floor become blurred from frame to frame
If you examine the depth data around the floor / foot contact point - with an RMS error of almost a centimeter - that means the floor and the foot location are kind of blending together by up to half an inch - so you are getting a lot of guesswork errors I guess.
As opposed to the D415 at the same distance which would likely be around 3.5-4.5mm without doing the math.
The other side of all this is the virtual size of a PIXEL of depth at this distance - since we are told that the physical hardware can only optimally work at 848x480 for the d435 but with a significantly wider field of view - this means that the effective size of each depth point is much larger for the D435 over the D415. Intel is telling us that this resolves to approximately HALF the level of detail for the same point in space for the D435 over the D415 for any given distance - once you get out past around 1m to 1.5m
The lower level of detail combined with the higher RMS compounds the poor quality of the data being handed to Nuitrack - with the end result being less than optimal tracking - at least for all of our internal usage cases.
For example - we require solid and accurate repeatable tracking of the entire skeleton - and especially the hands - regardless of where they are positioned. Right now at 2m for our needs the skeleton reporting is not reliable or accurate enough with the D435 for what we would consider commercial use.
And frankly I doubt that you will ever get results approaching those of a marker based system with the current d435 sensor technology - though Im very happy to be proved wrong on that if someone can show us a set of settings that works accurately and reliably and repeatably.
FWIW the d415 is getting close - but its still not TimeOfFlight which still feels like the holy grail point - who knows maybe if nuitrack manages to get their LiPS ToF sensor implementation working we may start to get close to a marker based system.
Westa
