I’m looking to use a compute shader to build mesh data from a DepthFrame’s RenderTexture. In order to do this I was hoping I could do the conversion from the raw integer values from the RenderTexture on the compute side and perform a calculation (lerp? … or is it non-linear?) using some sort of min/max depth values. Is there a way to do this? I know there’s the ConvertProjToRealCoords functionality on the CPU side… what’s happening in that function? Can it be easily translated over to compute in the way I described? Could someone share the internals of that function or a prescribed method of performing the depth-space to camera/world-space conversion?
Oh, awesome - yes! I scoured some of your unity SDK package and found a solution - very useful - thanks!
For anyone trying to implement this stuff in the future I’ve distilled down the hlsl code for their reference:
// Convert texture coordinates to NDC space (-1 to 1)
// Converts texture coordinates (UV) to Normalized Device Coordinates (NDC).
inline float2 TextureToNDC(uint2 texCoord) {
// Convert texture coordinates to NDC space (-1 to 1)
return float2(
(texCoord.x + 0.5) / _depthTextureWidth * 2.0f - 1.0f,
(texCoord.y + 0.5) / _depthTextureHeight * 2.0f - 1.0f
);
}
// Calculate the size of a texel in view space at the given depth - useful
// for drawing a reasonably sized quad from a point cloud.
inline float2 CalculateTexelSizeInViewSpace(uint2 texCoord, float depth) {
float vFovTan = tan(_cameraVerticalFOV * 0.5);
float hFovTan = vFovTan * _cameraAspectRatio;
// Calculate size of one texel at this depth
float texelWidth = (2.0 * hFovTan * depth) / _depthTextureWidth;
float texelHeight = (2.0 * vFovTan * depth) / _depthTextureHeight;
return float2(texelWidth, texelHeight);
}
// Uses the camera's field of view and aspect ratio to convert NDC
// coordinates and depth to view space positions.
// This also accounts for the perspective projection of the camera.
// The resulting positions will be in the camera's local space, where:
// > +X is right
// > +Y is up
// > +Z is forward (depth direction)
inline float3 NDCToViewSpace(float2 ndc, float depth) {
// Calculate view space position using perspective projection
float vFovTan = tan(_cameraVerticalFOV * 0.5);
float hFovTan = vFovTan * _cameraAspectRatio;
return float3(
ndc.x * hFovTan * depth, // X = depth * tan(hFov/2) * ndc.x
ndc.y * vFovTan * depth, // Y = depth * tan(vFov/2) * ndc.y
depth // Z = depth
);
}
inline float GetDepth(int2 texCoord) {
float depthValue = _depthTexture[texCoord].x;
return lerp(_sensorDepthRange.x, _sensorDepthRange.y, depthValue);
}
inline float3 GetViewSpacePosition(int2 texCoord, out float depth) {
depth = GetDepth(texCoord);
float2 ndc = TextureToNDC(texCoord);
return NDCToViewSpace(ndc, depth);
}
inline float3 GetViewSpacePositionWithDepth(int2 texCoord, float depth) {
float2 ndc = TextureToNDC(texCoord);
return NDCToViewSpace(ndc, depth);
}
The sensor depth range will need to be [0, maxSensorDepth] where the maxSensorDepth is found in the DepthToTexture component. This will also map the skeleton exactly to the viewspace position if you make the avatar/skeleton component a child of the camera.
I actually had one more question, it’s relatively unrelated; but I’m looking to upgrade my depth camera (I currently have a 400 series intel realsense and I’m not particularly enthralled by it - it’s accuracy is pretty terrible and it fails fast in pretty much any light. Are there any “staff pick” for favourite depth cameras that work with nuitrack? I’m hoping for something with better precision in the 0.5-3m range. I was looking at some of the Orbbec products and heard some favourable things - just wondering if you guys have any preferences / favourites for what you think works best with your SDK.
Hi @Callum, currently we recommend Structured Light lineup from Orbbec for a generic indoor usage - Astra (S/Pro), Astra+, Astra 2.
It the most time-proven option (has the same stable technology as the first Kinect).
ToF and stereo sensors are also good, but have a number of caveats / limitations, and their support in Nuitrack isn’t the most complete.