Training codes

Great work. is the code in model.py used for training the onnx inference models? any chance to release the training codes?

[emilianavt]

Thank you. The code in model.py is was used for training, but I still need to update it for the current version of the models. One it is updated, I will update this issue.

I do not plan to release the full training code at this point.

[sasanasadiabadi]

are all the models (face detection, landmarking and gaze detection) based on mobilenet-v3?

[emilianavt]

Yes. They are all (except for the optional, pretrained retinaface model) basically heatmap regression with a mobilenet-v3 backend. The gaze tracking model works basically exactly like the landmark one, just for a single landmark.

Face detection is a bit special. That model outputs a heatmap, radius map and a maxpooled version of the heatmap that is used for decoding the output.

Because the landmarking is quite robust with respect to face size and orientation, the face detection model can get away with outputting only very rough bounding boxes.

[sasanasadiabadi]

Great thanks. I was wondering if its possible to share the pytorch pre-trained weights as well. I'm trying to run the codes in opencv dnn module instead of onnxruntime. current onnx model seems to be not compatible with dnn module.

[emilianavt]

I have now updated the model definitions in model.py to match the currently used models.

I have also uploaded the pytorch weights here. My previous attempts at getting the models to work using opencv's dnn module weren't successful, but if you manage to get them to run, I would be very interested in hearing about it!

[sasanasadiabadi]

Thanks for sharing the files. I was able to convert the models and run them in opencv's dnn module. fps seems to be quite similar with both onnxruntime and dnn module. I will update you when the inference code is complete.

[emilianavt]

Thank you for the update!

I'm curious to know which format you converted the models to for use with the dnn module.

[sasanasadiabadi]

I converted the pytorch weights to onnx and using cv2.dnn.readNetFromONNX() with opencv 4.3 version I could run the inference (no other change to your original code). But the outputs of dnn module and onnxruntime are a little bit different with the same preprocessed input.

I have uploaded the converted weight here.

[emilianavt]

Thank you, I will give it a try using your converted models.

My first guess about the difference is that it might have something to do with the Upsample layers. The way I use them is apparently only fully supported with ONNX opset 11, which many inference engines do not seem to support yet.

[sasanasadiabadi]

yeah the problem was because of align_corners=True in nn.Upsample layer. dnn module is not supporting it yet somehow. therefore needs to set False for inference in dnn module. I will try to find a fix.

[sasanasadiabadi]

Finally solved. Thanks for pointing out the problem. with align_corners=True, converting from pytorch with onnx opset 11 and re-building opencv master (4.3.0-dev), the dnn module returns similar predictions as onnxruntime. I will re-upload the weights.

[emilianavt]

Nice, thank you for the updates! I tried using the models you previously posted with the dnn module, but got an error. I assume those already needed a more recent version than 4.2.0.

[emilianavt]

With the current OpenCV master branch, I also succeeded in loading the opset11 models (prior to optimization using onnxruntime). For the full landmark model, I get a pure inference time of around 13ms using onnxruntime with full optimization enabled and around 20ms for OpenCV's DNN module. The results are are practically identical.

[sasanasadiabadi]

Great!
However the landmarking looks very robust to various occlusion and illumination types, I think the pupil detection can be improved. As you are not planning to release the training codes, could you share some reference on the data preparation of the pupil model? you mentioned that the landmarking and pupil networks are basically same, but it seems the data pre-processing is quite different.