Extract occluded predictions from analysis file?

[dataframing]

Hi all,

First of all, thank you to the maintainers for such an amazing package. It's truly impressive!

I'm working on a standard top-down multiple-mouse tracking dataset. I've labeled frames, trained a few iterations of models, run inference + tracking on a full input video, and exported the project as an H5 file for data analysis. However, I noticed that in the exported analysis file's locations dataset (the tracks H5 file key), any predictions that are occluded are considered missing (i.e., np.nan). However, in the GUI, I can still see the occluded predictions if I focus on the particular instance at that frame.

Is it intentional that the exported analysis file encodes occluded predictions as np.nan? I figure that if the GUI is able to render where the occluded prediction is located, then it must be somewhere in the .slp file, so I was hoping it's recoverable from within the exported H5 file.

For what it's worth, occluded node predictions are usually quite far off from the actual node. It's not so much a matter of precision, but it would help data provenance if we (when we ultimately process the data into a more manageable pandas.DataFrame) have a binary was_occluded_prediction column.

As a follow-up question, do you generally suggest marking occluded nodes as visible when labeling a frame to train a model? Or is there value in informing the model that a particular node is occluded (i.e., improve its performance on future occluded points?)

Apologies if I've missed any similar discussions on this topic — I've looked through the issues and could only really find #600 that appeared somewhat relevant.

[roomrys]

Hi @dataframing,

The occluded predictions are intentionally stored as np.nans as SLEAP has assigned a very low confidence score to that node, i.e. the location of that node is uncertain. Any node that is listed as np.nan can be taken as was_occluded_prediction which generally goes hand in hand with a low confidence score (after the model has been thoroughly trained).

In general, if you have a very good guess as to where the node is located, then it may be worth labeling the node as SLEAP takes into account both the visual and spatial information associated with the label. However, if you are uncertain as to where the node is located, then it is best not to give false training data.

Let us know if you have any follow-up questions.

Thanks,
Liezl

[talmo]

I'll just add to this that the sleap-track CLI has an option to set the --peak_threshold which determines the minimum confidence before we replace it with NaN. If you set this to 0, you'll always get the coordinates back, even if they're very low confidence.

[dataframing]

That all sounds great! Thank you @roomrys and @talmo 🙏🏻