Skip to content

[ENH] Dynamically pull BIDS schema for auto entities and extractions. - #380

Merged
arnaudbore merged 30 commits into
UNFmontreal:devfrom
SamGuay:pull_schema
Jul 29, 2026
Merged

[ENH] Dynamically pull BIDS schema for auto entities and extractions.#380
arnaudbore merged 30 commits into
UNFmontreal:devfrom
SamGuay:pull_schema

Conversation

@SamGuay

@SamGuay SamGuay commented Jun 12, 2026

Copy link
Copy Markdown
Member

This PR makes dcm2bids even more linked to the official BIDS specification by deriving entity_table_keys and auto_entities from the schema instead of hard‑coding them. This means dcm2bids can work with any of the BIDS version available so far. I manually tested 1.9.0 up to 1.11.1.

It adds configurable BIDS schema version handling (including aliases and a bundled fallback), with caching and better offline behavior reusing version_check.json. Tests cover schema loading (online, cached, and bundled), as well as the derived defaults. Full disclosure: I wasn’t sure how to robustly simulate offline behavior in tests, so those tests were auto‑generated, but they match the scenarios I tested manually.

I also improved how update version check is done through a new caching mechanism in tmp_dcm2bids/version_check.json as well.

@SamGuay
SamGuay requested a review from arnaudbore June 12, 2026 11:24
@SamGuay

SamGuay commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

I was afraid fetching v1.9.0 on-the-fly over the web in the GHA would cause issue 🫠. might have to using entities from bundled schema instead.

@codecov-commenter

codecov-commenter commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.63923% with 113 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.85%. Comparing base (41808dd) to head (59ea7dc).

Files with missing lines Patch % Lines
dcm2bids/utils/schema.py 78.43% 44 Missing and 14 partials ⚠️
dcm2bids/utils/tools.py 56.41% 41 Missing and 10 partials ⚠️
dcm2bids/cli/dcm2bids.py 60.00% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #380      +/-   ##
==========================================
- Coverage   76.61%   75.85%   -0.76%     
==========================================
  Files          15       17       +2     
  Lines        1180     1557     +377     
  Branches      202      264      +62     
==========================================
+ Hits          904     1181     +277     
- Misses        223      299      +76     
- Partials       53       77      +24     
Flag Coverage Δ
pytest 75.85% <72.63%> (-0.76%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@SamGuay

SamGuay commented Jun 12, 2026

Copy link
Copy Markdown
Member Author
  • TODO: Update the changes in the documentation and update the tutorial because of dir not required anymore 🫠

@SamGuay
SamGuay marked this pull request as draft June 12, 2026 16:24
SamGuay added 12 commits June 30, 2026 13:00
Had some of them auto-generated as I didn't know how to fake no internet connection.
Had to adjust config for auto_extraction since dir is NOT required as per BIDS v1.9.0
sys.args was more fragile, now if a user explicitly asks for a version, it will be used even if it is the same as the default "bundled" one
@SamGuay
SamGuay marked this pull request as ready for review June 30, 2026 19:03

@arnaudbore arnaudbore left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, a couple of comments but looks great.

Comment thread dcm2bids/cli/dcm2bids.py Outdated
Comment thread dcm2bids/cli/dcm2bids_scaffold.py Outdated
Comment thread tests/data/config_test_auto_extract.json Outdated
Comment thread dcm2bids/utils/utils.py Outdated
Comment thread dcm2bids/cli/dcm2bids.py Outdated
Comment thread dcm2bids/cli/dcm2bids.py Outdated
Comment thread dcm2bids/cli/dcm2bids_scaffold.py Outdated
@SamGuay

SamGuay commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review @arnaudbore. I also pushed the documented function in addition to a modification to the beginner's tutorial

@arnaudbore arnaudbore left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comment, please do a quick pep8 and we will see what copilot has to say.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces BIDS-schema–driven defaults into dcm2bids by adding a schema loader/cache layer and updating defaults/docs/tests so entity ordering and “auto-entities” can be derived from the official BIDS schema (with support for aliases like stable/latest and a bundled fallback).

Changes:

  • Added dcm2bids.utils.schema to download/cache/load BIDS schema JSON (including bundled default schema support).
  • Replaced hard-coded DEFAULT.entityTableKeys / DEFAULT.auto_entities with schema-derived values, and added a new --bids_version CLI flag.
  • Added extensive tests for schema loading/caching and updated docs/config examples to reflect the new schema-driven behavior.

Reviewed changes

Copilot reviewed 16 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
tests/test_schema.py Adds unit tests for schema loading, caching behavior, and schema-derived defaults.
tests/data/config_test_auto_extract.json Updates test config to include custom_entities: "dir" for fmap/epi.
pyproject.toml Excludes bundled schema JSON from codespell scanning.
docs/tutorial/first-steps.md Updates tutorial text and examples for --auto_extract_entities and custom_entities.
docs/how-to/use-advanced-commands.md Documents new --bids_version flag and caching/reproducibility guidance.
docs/how-to/create-config-file.md Fixes a minor doc typo/formatting issue.
dcm2bids/utils/utils.py Switches DEFAULT entity ordering/auto-entities to schema-derived defaults.
dcm2bids/utils/tools.py Adds cached version-check mechanism and version comparison helpers.
dcm2bids/utils/schema.py New module implementing schema URL building, download, caching, derived defaults, and CLI schema resolution.
dcm2bids/utils/schema_data/init.py Adds packaged data access helper for bundled schema JSON.
dcm2bids/utils/logger.py Minor formatting/whitespace fix.
dcm2bids/sidecar.py Minor whitespace/formatting fix in config validation area.
dcm2bids/participant.py Fixes grammar in validation error messages (“contain” vs “contains”).
dcm2bids/dcm2bids_gen.py Minor formatting/whitespace changes (constructor args formatting).
dcm2bids/cli/dcm2bids.py Adds --bids_version argument and schema pre-load call; updates --version output.
dcm2bids/cli/dcm2bids_scaffold.py Routes version check through the new cached check_latest(log_dir=...).
dcm2bids/cli/dcm2bids_helper.py Routes version checks through the new cached check_latest(log_dir=...).
dcm2bids/acquisition.py Updates docstring to reference schema-derived entity ordering; minor whitespace.
.github/workflows/tests.yml Minor whitespace change.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dcm2bids/utils/utils.py Outdated
Comment thread dcm2bids/utils/schema.py Outdated
Comment thread dcm2bids/utils/schema.py Outdated
Comment thread dcm2bids/cli/dcm2bids.py
Comment thread dcm2bids/cli/dcm2bids.py Outdated
Comment thread tests/test_schema.py
Comment thread tests/test_schema.py
Comment thread dcm2bids/utils/schema.py Outdated
Comment thread dcm2bids/utils/schema.py
@arnaudbore
arnaudbore merged commit fd1608c into UNFmontreal:dev Jul 29, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants