Combine bottom-up model with centroid model

[yitongxia-1180]

Dear developers,

I am trying to create a workflow on my local sever that runs a centroid model first to locate the mouse in a frame, then crop the image accroding to user's demand(e.g. 256 pixel x 256 pixel), the cropped image is centered on the mouse. Then, use a bottom-up model to run inference on this cropped image. Of course, this bottom-up model needs to be trained on cropped images too.

The reason why I want to switch the centered_instance model to bottom-up model in a traditional top-down workflow is that, I want the model to learn the geometric relationship between points, or the edges. I want the predition could respect the physical law of a mouse's body, and not make mistakes like identifying the head as the butt.

Also, I want to speed up the process in bottom-up inference, in which the model needs to search the whole frame to locate each body part, and this is really time- consuming. By providing the model with cropped small images, this process may be considerably speeded up.

In addition, the bottom-up model sometimes makes mistakes like, assigning two incomplete skeletons to a single animal, like I have described in #2529. I want to limit this mistake by using a centroid model to locate the mouse first, so in each cropped image, there is only one mouse(or ask the model to predict on the mouse at the center, considering sometimes the cropped image may include other mice on the edge).

Yet I have not succeed on this customized workflow, but I really think that this is very practical and valuable, that I hope the community could have a discussion and maybe add this feature to sleap in the future.

Apologize again for my poor English. As a phd student, I have soaked myself in training models with sleap for months. I really hope sleap could be better and better everyday!

[tom21100227]

Hi Tongxia,

Those are all great ideas! I would push back a little on the centroid + bottomup part. The difference between bottom-up are centered instance is just an additional output called PAFs, which associates nodes with each other. I could see how that helps a bit with associating different parts of the same instance but it would not help a lot with speed. and with a centroid model it might actually hurt accuracy during times where the mice body is partially occluded.

I would like to highlight a feature from sleap-nn, TensorRT/ONNX export if you're looking to speed up your inference: https://nn.sleap.ai/latest/guides/export/ I think this could help with the speed aspect, and we're actively working on other feature that might help with chimera / partial instances!

Good luck with your PhD!
Best,
Tom