Depth maps

[godnpeter]

I'm curious about whether there are any future plans in including the depth maps of the provided dataset.

[Axi404]

Future plans may include scaling the dataset to longer-horizon tasks and adding dual-arm data. Our data engine can already render depth information easily; the open question is whether depth is actually necessary for the community. Curious to hear your thoughts.

[godnpeter]

Adding per-frame depth maps would make this dataset significantly more useful. Since you already provide accurate camera intrinsics and extrinsics, paired depth maps would allow the community to reconstruct high-quality point clouds with minimal additional work.

Right now, there are relatively few open robot-trajectory datasets that support point-cloud research. When depth/point clouds do exist, they’re often limited by noisy real-world depth sensors or by simulations that aren’t visually/physically convincing.

InternManip is in a great position to fill this gap: the visuals are highly photorealistic, and the depth could enable clean, consistent 3D reconstructions. Making depth maps public would unlock a lot of downstream work (3D perception, mapping, grasping, 6D pose, etc.) and likely broaden adoption of the dataset.

[Axi404]

Sounds good! I’ll prepare those features in the next update. The exact release date may depend on my workload, but it should be ready within about two weeks.

[godnpeter]

@Axi404 Thank you for agreeing in support this new feature! Super excited about this :)

[godnpeter]

Dear @Axi404 ,
Actually, I think that you guys already included the depth map videos!!

However, what we need is metric depths in order to convert these to point clouds and I don't think that this dataset is metric depth. They seem to be normalized in some sort of manner and I am having trouble converting these to metric depth.

Could you help me out?

[godnpeter]

I think that these are normalized and we don't have the proper materials to convert these into metric depths in the parquet files. Are there any suggestions in which I can convert these to metric depth?

[Moon1x21]

I’m also following this issue. At the moment, I can only obtain normalized depth, while I’m looking for a way to recover metric depth. Are there any updates on this?

[Axi404]

Hi,
Could you please provide a sample video file so that I can try decoding it on my side?
I'm not sure whether this data is publicly available or intended for external use. Thanks!

Dear @Axi404 , Actually, I think that you guys already included the depth map videos!!

However, what we need is metric depths in order to convert these to point clouds and I don't think that this dataset is metric depth. They seem to be normalized in some sort of manner and I am having trouble converting these to metric depth.

Could you help me out?

[godnpeter]

https://github.com/user-attachments/assets/61506acc-7649-46e2-a479-babdc6816eff
https://github.com/user-attachments/assets/aa9b437e-9ae0-44c4-bb4d-56353580458f

I'm sending you the videos for each view for episode 0!

[godnpeter]

Just to give you all the info, we not only have depth maps, but also have semantic maps as well.

I would be super duper really grateful if you can give me some guidance in converting this to metric depth!!

I'm currently trying to make this work, but wasn't able to gain much progress :(

[Axi404]

Sorry for that, and we found it seems not a good idea to save encoded depth map with mp4 format, which cause data corruption.
The best way is to do is to upload it again :(

decoded_output.mp4

[Axi404]

Maybe we should save raw depth image in mp4 format or save encoded depth image in png format

[godnpeter]

Well I was assuming that all we actually additionally need to convert that normalized depth into metric depth is the minimum and maximum distance. I guess that we require something more to convert the currently included normalized depth video into metric depth?

[Axi404]

Yes. And perhaps we need a better way to save raw uint16 depth image into video, it is not that hard and i can handle it.