Skip to content

Add idx format #21289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Add idx format #21289

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0c83604
Add idx format
SantaMcCloud Nov 13, 2025
06d30c5
Update datatypes_conf.xml.sample
SantaMcCloud Nov 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions lib/galaxy/config/sample/datatypes_conf.xml.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1195,6 +1195,7 @@
<datatype extension="rd" type="galaxy.datatypes.binary:Binary" mimetype="application/octet-stream" display_in_upload="true" subclass="true" description="Rdeval read sketch"/>
<datatype extension="safetensors" type="galaxy.datatypes.binary:Safetensors" mimetype="application/octet-stream" display_in_upload="true" description="A simple format for storing tensors safely (as opposed to pickle) and that is still fast (zero-copy)" description_url="https://huggingface.co/docs/safetensors/index"/>
<datatype extension="spatialdata.zip" type="galaxy.datatypes.binary:SpatialData" mimetype="application/octet-stream" display_in_upload="true" description="A data framework that comprises a FAIR storage format and a collection of python libraries for performant access, alignment, and processing of uni- and multi-modal spatial omics datasets" description_url="https://github.com/scverse/spatialdata"/>
<datatype extension="deacon.idx" type="galaxy.datatypes.binary:Binary" subclass="true" mimetype="application/octet-stream" display_in_upload="true" description="Binary index format which is compatible with Deacon (a tool coded in rust)."/>
Copy link
Member

@wm75 wm75 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<datatype extension="deacon.idx" type="galaxy.datatypes.binary:Binary" subclass="true" mimetype="application/octet-stream" display_in_upload="true" description="Binary index format which is compatible with Deacon (a tool coded in rust)."/>
<datatype extension="deacon.idx" type="galaxy.datatypes.binary:Binary" subclass="true" mimetype="application/octet-stream" display_in_upload="true" description="Binary index format which is compatible with Deacon."/>

Copy link
Member

@wm75 wm75 Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another question is how to handle index versioning. Deacon has changed its index format multiple times already and they're only at version 0.13 of the tool. You probably have to define a dedicated class for this that inspects the file and stores the format version as metadata.

Copy link
Contributor Author

@SantaMcCloud SantaMcCloud Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that the version is stored somewhere in the metadata in the file istself. I did check with hexasum if there is an header but i didnt check the content. So what would be a good idea for this? write an siffer?

Copy link
Contributor

@bernt-matthias bernt-matthias Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just ask upstream about versioning and plans for the format. Based on this we can make a better decision.

Copy link
Contributor Author

@SantaMcCloud SantaMcCloud Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bede/deacon#70

Copy link

@bede bede Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, index format used to be tied to the hash table implementation, and the format was not as dense as possible, leading to some churn. As mentioned in bede/deacon#70, format version 3 stores raw 2bit k-mers, meaning hash table implementation could be changed in a future Deacon version for faster performance without breaking the index and requiring a new format.

As of 0.13.0 I consider Deacon more or less feature complete, and 1.0.0 may come soon. I do not anticipate needing to change the index format again.

And yes, the deacon index info shows the index version. Index versions are validated at runtime and fail loudly.

Let me know if you have any further questions, I'm excited to see Galaxy integration.

Copy link
Contributor Author

@SantaMcCloud SantaMcCloud Nov 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for both of your comments! This should help us to find a solution to implement deacon and the index format to galaxy!

</registration>

<sniffers>
Expand Down
Loading