Is possible to integrate custom deep learning model into the Sdk?

Nuitrack SDK Other Integrations
1

Hi,

I want to know if it is possible to integrate a custom deep learning model into your sdk. For example if one of the models provided for skeletal tracking is not good enough for my use case I need to do a custom training.

I want to know if this is possible?

Thanks

2

Hi @eric,

Thank you for your request.
Could you kindly elaborate a little bit on the use-case - if skeletal tracking isn’t good enough why Nuitrack could be useful at all ?
In case you have a better algorithm/model - why it can’t be used directly in this case?

3

Hi @TAG ,

For example: If I need to detect some more joints in the body and the I want to integrate this using C# I need to create all the code that is doing the inference in C++ and then some C# C++ interoperability.

If Nuitrack has already tested this pipeline instead of developing my custom pipeline (C++ C# ecc) I can modify the weights of the model and then pass some more parameters from C++ and C#. This, I supposed, could be the same for any type of integration that you have Unreal Unity ecc…

This will be much faster instead of creating a custom pipeline.

Of course the model trained is in python.

4

Thank you, it makes more sense now.
Do you have a commercial project which crucially requires this capability ?

Actually there are large number of AI cloud/edge deployment platforms on the market currently which provides all types of integrations and custom model import capabilities.

5

Yes I have, but as you correctly pointed out there are some AI cloud/edge options.

I wanted to know if nuitrack has this possibility because it has a plugin already developed for different types of camera and a plugin for unreal if i have understood correctly. That’s why I have asked.

In any case, thanks for the answer.

6

@eric Sorry for late reply,
anyway we feel the need to provide additional comments.

These are correct.

  1. Nuitrack (being kind of old / mature software) has plugins already developed for the most depth sensors on the market (this list is incomplete as Nuitrack is licensed by some sensor vendors as a white-label product).
  2. Nuitrack has an UE4/5 plugins

In general, we are always interested in improving our tracking engines (instead of replacing them) and have a dedicated tools for that purpose.

We really didn’t thought about custom model integration, as the processing pipeline is always custom for different CNN models and Nuitrack has more non-trivial internals outside the model (model is 10% of the processing for AI, classical tracking engine doesn’t have “a model” at all).

Just out of curiosity / understanding of possibilities - what particular network/CNN architecture did you suppose to use/deploy/integrate ? (it would be great if you could kindly share this info)

7

Hi sorry for late answer. I have tested different deep learning software just for skeletal detection.

Yolov8
Mediapipe
YoloPose

Mediapipe is the most complete solution in my opinion, but has some drawback. Using it in C++ on windows is not easy.

The other solutions are not ready to use at least for a production environment in my opinion, but if you need to improve the model you can with custom training.

Other tool that I have tested is open pose,but not in depth. I think nuitrack is the best option for production, but the fact that is not possible to improve the pipeline is a bit blocking me to adopt, because if I cannot get the required specification I have not understood well how to improve the output.

As i said above I have a custom dataset and for some specific case my trained model is working better than mediapipe for example.