Nuitrack, Unreal 4, and PC Problems

I’ve been trying to get Nuitrack to work in Unreal 4.20 for a little over 2 weeks now and have made very little progress. The provided example given is for Android and while the functionality for unreal itself is looking sound, I cannot say the same for anyone wanting to use the library for windows machines.

I’m on a 64bit machine using an Intel RealSense 435 in USB 3 mode. I’ve gotten the example programs and code for OpenGL and the like to work after finding out the hard way that the library works when I had the camera in USB 3 and only after deleting the OpenNI folder in nuitrack’s bin folder (it crashes with the folder included, a bug?).

Now since I know the environment variables are correct and that the example programs work, I can at least put that aside and focus on the Unreal side of things. In there I’ve got something very similar to the example provided for Android, creating a third-party plugin that only really contains the include and lib folders I need for a 64bit program.

the Build.cs file is as follows:

using UnrealBuildTool;
using System.IO;

public class NuitrackDev : ModuleRules
{
    private string NuitrackPath
    {
        get { return Path.GetFullPath(Path.Combine(ModuleDirectory, "../../Thirdparty/Nuitrack/")); }
    }

    public NuitrackDev(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
	
		PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore" });
        
        Definitions.Add("NUITRACK_NO_EXCEPTIONS");
        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            PublicIncludePaths.Add(Path.Combine(NuitrackPath, "include"));
            PublicLibraryPaths.Add(Path.Combine(NuitrackPath, "lib", "win64"));

            PublicAdditionalLibraries.Add("nuitrack.lib");
            PublicAdditionalLibraries.Add("middleware.lib");

            Definitions.Add("WITH_NUITRACK");
        }
    }
}

Cool thing is, with that and the proper includes in the project file itself, Unreal actually compiles! And better still, UE4 loads all the appropriate dll files when Nuitrack::init() is called. But that is where I’m stuck at. Unreal crashes when the init function is called.

Here is an output of the last bit that is relevant:

2018.10.31-22.43.42:119][105]LogTemp: Warning: ANuitrackDevGameModeBase::BeginPlay()
[2018.10.31-22.43.42:120][105]LogTemp: Warning: Nuitrack::init() CALLING...
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\middleware\NuitrackModule.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\OpenNI2.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_calib3d.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Unloaded 'C:\nuitrack\bin\libopencv_calib3d.dll'
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_features2d.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\Program Files\OpenNI\Bin64\OpenNI64.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_imgproc.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_highgui.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_ml.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Unloaded 'C:\nuitrack\bin\libopencv_highgui.dll'
'UE4Editor.exe' (Win32): Unloaded 'C:\nuitrack\bin\libopencv_ml.dll'
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_objdetect.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_video.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\realsense2.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_flann.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\mfreadwrite.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\winusb.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\mfcore.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_calib3d.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_ml.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\nuitrack\bin\libopencv_highgui.dll'. Module was built without symbols.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\msvfw32.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\avifil32.dll'. Symbol loading disabled by Include/Exclude setting.
'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\avicap32.dll'. Symbol loading disabled by Include/Exclude setting.
Unhandled exception at 0x00007FFE92FEE57E (ucrtbase.dll) in UE4Editor.exe: Fatal program exit requested.

The program '[14428] UE4Editor.exe' has exited with code 0 (0x0).

I have no way of knowing why it is crashing here and even using a try-catch gets me nothing as it seems the init function or something within it is forcing my program to exit completely.

Can anyone help me on this? I know most people on here are using Unity, but I don’t see why this should not work for Unreal as well. I feel like I’m only one roadblock away from actually getting it to work.

As usual, I figured it out on my end.

For anyway trying to get Nuitrack to work in Unreal, know that you cannot run the editor through Visual Studio – especially in debug mode. The software will crash if it tries to run the debugger through the library.

Now to figure out how to reliably convert Nui’s orientation matrix to Unreal’s Rotator.

@JZharay Could you share what your fix was and any insight into performance now that you have it working? I’m looking to build a motion tracked experience in unreal and am currently considering Nuitrack, but it seems like it may be more trouble than it’s worth, especially since I already have Kinect v2 working. Would love your input.

Honestly I am going to recommend you don’t bother. Getting Nuitrack to work with unreal requires you do what they did in Unity and calculate bone angles relative to their parent bones. Its a huge hassle and not to mention buggy affair due to having to figure out what the proper orientation is from whatever they use (I’m assuming Unity’s) to Unreal’s. There is no documentation or explanation as to how they get it working, even in Unity, and I feel like I’m reinventing the wheel every time I get a failed result. I understand they got it easier with Unity and its plug and play environment, but the projects I work on strictly center on Unreal and I don’t have the luxury to waste any more time on something that Kinect4Unreal does a much better job of.

But if you do want to go down the rabbit hole, one bit of advice I can give is that the matrix used in Nuitrack is verticle. So X is [0, 3, 6], Y is [1, 4, 7], and Z is [2, 5, 8]. Also, as far as i can tell, the correct way to get a conversion from Nuitrack’s orientation to Unreal is to make a matrix such as FMatrix mTemp(-Y, Z, -X,FVector::ZeroVector). That was pulled from what I got so far (not sure if that was correct one though), but once I figured out I had to do the whole “relative to parent” conversion, I just moved on to something else.

That above took me nearly 3 months off and on to figure out without any help from Nuitrack and their lack documentation.