Roadmap for Open3d and future of Legacy vs Tensor api

[saurabheights]

Hi,

I wanted to get some idea on the future development of Open3d and was wondering if there is a roadmap set. This could be useful in directing open source development effort in a more meaningful direction.

Also, is there any plan in removing Legacy vs Tensor api, i.e. dual apis like o3d.geometry and o3d.t.geometry for a single api. Abstracting out the implementation details away and providing a single api to users will reduce the complexity. However, deprecating api would require a more concrete plan, which brings to my original question of any available roadmap/vision for open3d.

Thank you in advance.

[andy1122]

Hi @saurabheights

I was also looking into the code of the open3d, but i notice something interesting. Where the tensor implementations are based on the eigen based implementations. As you can see in this pull request:

#5330

Here the HiddenPointRemoval function is using the legacy(eigen) version to implement the tensor version, same goes for the other functions as per my understanding.

I am not sure, if they plan to remove one or other. Because from the way some of the above tensor api are implemented. They are completely dependent on the eigen versions.

Let me know, what do you think about this. This was just my observation. Btw, i am a complete beginner to open3d so my understanding can be wrong at places. Feel free to correct me at any place.

[saurabheights]

Hi @andy1122 , that is fine. Reimplementation efforts is also quite substantial, so using wrapper over legacy api is fine. However, IMO there should be a single api to same functionality. For e.g. o3d.t.geometry.Pointcloud() is giving the user same thing as o3d.geometry.Pointcloud(), aka a Pointcloud. Having double code, means twice replication, both for users and developers. P.S. Removal of one of these data structures, wont be so easy because the change will be backward incompatible.

Its questions and thoughts like this which made me ask the broader question - Whats the roadmap?

[andy1122]

I agree, atleast from the user POV it really makes no difference to get the same result from two different functions. However, there is going to be some difference in the performance for some specific function based on tensor api. Do you know, if someone has already done some benchmarking for the tensor vs eigen based functions. And if some of the functions are just wrapped over legacy api, then i am not sure if there is going to be some performance improvement or not. Which essentially kills the purpose of creating two different libraries.

Once we reach the point, where we see the clear difference from the performance POV, and check other tradeoffs as well. Then it will be possible to get rid off one of the libraries.

[ssheorey]

The broad roadmap is to replace eigen based functionality with Tensor based functionality throughout the library. This allows Open3D to take advantage of GPUs (and other future accelerators). The Tensor API is also more flexible in some ways (e.g. generic point cloud attributes). For CPU specific versions, some operations may be slower or faster compared to the Eigen API. As @saurabheights mentioned, Eigen is great well optimized library, so it's not trivial to improve upon it. GPU versions are significantly faster.

Our immediate goal is to provide feature parity between the tensor and legacy APIs so that the legacy API can be deprecated. Finally, (after deprecation notices for at least a couple of versions) o3d.t.geometry will simply become o3d.geometry. This is being accomplished either through re-implementation or through wrapping Eigen APIs depending upon complexity and potential for speedup when switching to the Tensor API. Afterward, we can continue optimizing the tensor functions after API is stabilized.

In parallel, we are continuing to add new functionality, bug fixes and enhancements. If progress seems slow, that is because we have a small team who only work on Open3D part-time. We would really encourage everyone to contribute to Open3D in any way (PRs, issues, discussions, help,....).

[YoshuaNava]

@ssheorey a lot of time has passed since your message. Happily, a lot of the features you mentioned have become a reality, and they are useful to develop new features at the frontier of 3D data processing.

However, there is a key feature that I couldn't find within the current version of Open3D: In-place operations for Tensor structures, starting with rigid-body transformations, and going into filters that down-sample or augment data. This feature would be an enabler for deploying applications developed with Open3D on mobile robots, where copying data can increase latency and consume resources that other sub-components may need.

Besides that, I'm aware that Open3D implements a memory pool that covers both RAM and VRAM, and allows persisting the data in memory for as long as the object lifetime. As a roboticist, I often work with two different hardware profiles:

• Development PC: x86 CPU, NVIDIA GPU, separate RAM and VRAM, and
• Deployment SoC: NVIDIA Jetson with arm64 CPU and embedded GPU, unified memory model.

Is there any documentation on how to successfully deploy on both types of platforms? If not I would be happy to jump in and support any efforts in this direction.

Thank you in advance.

Best,
Yoshua