Triangulating coordinates following camera calibration

[lochlan-w]

Hi! First off, I just wanted to say that this is a great tool - very quick to implement, and I like the visualizations of the calibration results.

I had a question regarding triangulation of point coordinates after extracting the camera intrinsics/extrinsics. For context, we're working with animal pose data (extracted from other tools like DeepLabCut, or manually-labelled coordinates of objects of known sizes), where we already have the coordinates in both csv and h5 format. I admit I haven't looked too far into your code, but I was wondering if there was a section you think would be helpful in triangulating the 3D coordinates of the keypoints that we already have (i.e. not necessarily integrating DLC into Caliscope, but combining the output of both tools)?

We've done some manual triangulation before, but it would be nice to know if there's a usable section of code that we could implement to semi-manually triangulate 2D points from multiple views ourselves. Existing tools seem to require an integration of pose estimation AND triangulation, whereas we're primarily interested in just the triangulation step.

If you think it might just be easier for us to manually triangulate the points with some matrix multiplication in our own code, or would like further explanation on anything here, please let me know!

[mprib]

Hi @lochlan-w,

The workflow you are describing is definitely quite doable. There is a test (test_xy_to_xyz.py) that I'm currently cleaning up in a new branch that might give you a sense of things. Rolling your own script with that test as a guide is one option.

As another option, I'm planning to add in a CLI to triangulate data with a properly formatted xy data and a calibrated config.toml (the purpose of the latest branch). I may also make some tweaks to this new branch to attempt to speed up the triangulation. If you have any sample data+config.toml to work with I'm happy to use that to facilitate development and troubleshoot the inevitable unexpected things that pop up.

You can look to the xy_HOLISTIC file here to see how the data should be structured. If you have the point data already time aligned (via the sync index) then the frame_time and frame_index shouldn't matter.

Mac

[mprib]

@lochlan-w and @aruna-ram,

I have just merged in some changes that will allow a straightforward scripting of the triangulation. It is documented here:

https://mprib.github.io/caliscope/triangulation_api/

This is something where I see opportunities to make the calculations proceed much more quickly, and it could also be a good place to dive into gap filling and smoothing strategies, so please keep me in the loop on your experience if you use this.

[lochlan-w]

Thanks for the update! Will be nice to see how well we can integrate it into the pipeline of calibration > triangulation.

Regarding 2D smoothing/gap-filling prior to triangulation, I've had good experience using MATLAB's modified Akima interpolation to fill in gaps. But the base Akima method (which I think also has an existing implementation in Python?) also performs quite well. Both aim to minimize overshooting but prevent overly aggressive interpolation on missing gaps, which can produce nice results (- see here). I have no idea how/whether this method applies to 3D interpolation, but it could be applied to the 2D tracks from each port prior to triangulation. And if this was a feature you were considering adding into the pipeline, could allow users to pick between a variety of interpolation options (with the option to specify the gap fill threshold)?

Regarding smoothing, I have less experience implementing myself, but a (Extended) Kalman filter seems to be the go-to for 2D smoothing, although I'm less certain how straightforward it would be to integrate into this pipeline.

[mprib]

@lochlan-w,

Thank you for the information about Akima. I found the scipy analog and it seems like a very doable inclusion. I've opened an issue here so I can be reminded of it.

The gap-filling / smoothing question is a pretty big can of worms to open and I'm about to pass into a busy month which will keep my mind on other things. Long run I definitely plan to dive into this, but if it's something you have an interest in trying to implement, PRs are welcome!

Regardless, please let me know how things go with the vanilla triangulation.

Best,

Mac

[aruna-ram]

@mprib Amazing! Will test it out and keep you updated.