Face Rectangles & Unity RawImage

Nuitrack SDK Unity3D
1

Hello - A little rusty w/ Unity, never needed to crop a raw image or draw bounding boxes. Also, very new to NuiTrack. Is there a tutorial to show how to use the rectangle data from the face tracker w/ Unity UI? How to use that to draw a bounding box over the color frame, how to crop it for facial recognition, etc. I’m tinkering with that now, will post back if I solve it on my own. Thank you!

2

Hello @BillDozer42

We have a tutorial where the rectangle data for the face is used. There, using this data, the image is stretched https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/Unity_Face_Tracking.md

You can also see how the scripts work in Assets\NuitrackSDK\Faces\Scripts. And make a minimal example using them:
1 Create new Scene
2 Add NuitrackScripts prefab, enable “use Face Tracking” tick (and must be enabled “Depth2Color registration” tick, for improved accuracy)
3A. For one face: Create Canvas and drag the Face Frame under this canvas
3B. For 1-8 faces: Drag Face Canvas on Scene

3

Thanks for that - I did go through those and I can get a handle on the rect. In my example, I am trying to crop the head from a raw image. This may or may not be a more unity-centric problem (Unity version 2020.3.16f1). In a very simple scene, with just a canvas, 2 raw images (one w/ a texture from a local graphic), and the other the cropped output - everything works fine:

    Texture2D originalTexture = (Texture2D)sourceImage.mainTexture;

    Texture2D croppedTexture = new Texture2D(50, 50, TextureFormat.RGBA32, false);

    var pixels = originalTexture.GetPixels(25, 25, 50, 50);

    croppedTexture.SetPixels(pixels);
    croppedTexture.Apply();

    cropImage.texture = croppedTexture;

However, when I run similar code against a raw image that is receiving the color frame, the pixels I receive from GetPixels are all gray… I can’t seem to crop & copy the pixels of just the face from the texture received from the color frame from nui track.

I did have to tick ‘Read/Write Enabled’ in the import settings of the local texture in my sample that works. Not sure how to do that for the texture coming in from color frame, but then again I don’t get the same error as I did for my local image, which prompted me to tick that box in the first place.

Maybe a more specific question/request here is - how do I crop the Texture2D that I receive from the color frame? In the below sample, the pixels are also all gray. Must be some quirk of pulling the pixels from the texture generated by ToTexture2D()? I see some notes about ToRenderTexture, I’ll try that angle as well.

void DrawColor(nuitrack.ColorFrame frame)
{
    var texture = frame.ToTexture2D();

    background.texture = texture;  //this works fine

    Texture2D croppedTexture = new Texture2D(300, 200, TextureFormat.RGBA32, false);

    var pixels = texture.GetPixels(100, 100, 300, 200); //this is all gray

    croppedTexture.SetPixels(pixels);

    croppedTexture.Apply();

    cropImage.texture = croppedTexture;
}

Hope this makes sense - I’ll keep pecking at it…

4

Your example is correct, but we also found a problem in NuitrackSDK.

If you just need to output the texture of the user’s face without further processing on the CPU, then you can use this example (it works very fast, the texture will be inverted).

FaceTexture.cs 
using UnityEngine;
using UnityEngine.UI;

using NuitrackSDK.Frame;

public class FaceTexture : MonoBehaviour
{
    [SerializeField, Range(0, 5)] int userID = 0;
    [SerializeField] RawImage rawImage;

    void Update()
    {
        nuitrack.ColorFrame frame = NuitrackManager.ColorFrame;

        if (frame == null)
            return;

        JsonInfo jsonInfo = NuitrackManager.NuitrackJson;

        if (jsonInfo == null || jsonInfo.Instances.Length <= userID)
            return;

        Face face = jsonInfo.Instances[userID].face;

        if (face.rectangle == null)
            return;

        int faceX = (int)(frame.Cols * Mathf.Clamp01(face.rectangle.left));
        int faceY = (int)(frame.Rows * Mathf.Clamp01(face.rectangle.top));
        int faceWidth = (int)(frame.Cols * Mathf.Clamp01(face.rectangle.width));
        int faceHeight = (int)(frame.Rows * Mathf.Clamp01(face.rectangle.height));

        Texture2D rgbTexture = frame.ToTexture2D();
        Texture croppedTexture = new Texture2D(faceWidth, faceHeight, TextureFormat.RGBA32, false);

        Graphics.CopyTexture(rgbTexture, 0, 0, faceX, faceY, faceWidth, faceHeight, croppedTexture, 0, 0, 0, 0);

        rawImage.texture = croppedTexture;
    }
}
The essence of the problem

The detected problem is that ToTexture2D in certain cases does not contain data on the CPU side, only on the GPU, and the GetPixels method requests data on the CPU. It should be understood that the texture simultaneously stores two copies in memory for the CPU and GPU.

To learn more about the details of the Texture2D memory implementation in Unity, see this.

We will fix the behavior of ToTexture2D in the next update.

5

Excellent - this works great for me - thank you!

Will the incoming fix for ToTexture2D be even faster than this?

6

The fix will work about as fast as the comparable proposed fix.

Please note that if you do not expect further processing of the image, but only its display, it is better to use the ToTexture method, which will choose the most productive conversion method for your platform (ToRenderTexture or ToTexture2D).

ToRenderTexture is faster in most cases, since all calculations are performed on the GPU and there is no copying of data to memory for the CPU.

7

Thank you! We’re good to go now.