Conversation
mattjala
requested review from
fortnern,
glennsong09,
hyoklee,
jhendersonHDF,
lrknox and
vchoi-hdfgroup
as code owners
July 10, 2026 21:47
Contributor
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a data-corruption risk when writing filtered chunked datasets to files whose configured “size of sizes” (via H5Pset_sizes()) is smaller than 8 bytes, by detecting and erroring out on filtered chunk sizes that do not fit in the on-disk encoded field width.
Changes:
- Add an overflow check in
H5D_CHUNK_ENCODE_SIZE_CHECKto ensure filtered chunk sizes fit within the file’s configured “size of sizes” for v5 chunk layout messages and for the single-chunk index in any layout version. - Add new dataset tests covering round-trip correctness for small “size of sizes” values and explicit overflow detection on write/flush.
- Document the fix in the release changelog.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/dsets.c | Adds new tests for filtered chunk size encoding with size of sizes set to 2/4/8, including an overflow-failure case. |
| src/H5Dchunk.c | Extends the filtered chunk size encoding check to honor the file’s configured “size of sizes” field width. |
| release_docs/CHANGELOG.md | Notes the bug fix and the prior silent truncation behavior in release documentation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
github-actions
Bot
removed request for
glennsong09,
jhendersonHDF,
lrknox and
vchoi-hdfgroup
July 10, 2026 22:08
hyoklee
approved these changes
Jul 13, 2026
hyoklee
previously approved these changes
Jul 21, 2026
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
mattjala
force-pushed
the
small_chunked_sizes
branch
from
July 21, 2026 19:02
fb03e41 to
dbb0582
Compare
hyoklee
requested changes
Jul 21, 2026
hyoklee
previously approved these changes
Jul 22, 2026
hyoklee
previously approved these changes
Aug 6, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
hyoklee
approved these changes
Aug 6, 2026
hyoklee
left a comment
Member
There was a problem hiding this comment.
Copilot's suggestion for CHANGELOG.md:
### Fixed silent truncation of filtered chunk sizes when the file "size of sizes" is small
For all chunk index types in version‑5 chunk layout messages, and for the single chunk index in any version of the chunk layout message, the on‑disk size of a filtered chunk is encoded in a fixed‑width field equal to the file's "size of sizes" (set via `H5Pset_sizes()`). The encode check previously assumed this field was always 8 bytes. When the size of sizes was set to 2 or 4, an oversized chunk's encoded size could be silently truncated (for example, 160000 encoded in 2 bytes becomes 160000 & 0xFFFF = 28928), corrupting the chunk. The library now verifies that a filtered chunk's size fits within the file's size of sizes and reports an error at write time.
Fixes GitHub issue #6023
Contributor
|
This pull request has had no activity for 30 days and has been marked stale. Push a commit or comment to keep it open, or it will be flagged for maintainer review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For all chunk index types in version-5 chunk layout messages, and for the single chunk index in any version of the chunk layout message, the on-disk size of a filtered chunk is encoded in a fixed-width field equal to the file's "size of sizes" (set via H5Pset_sizes()). The encode check assumes this field is always 8 bytes, so when the size of sizes is set to 2 or 4 an oversized chunk has its encoded size silently truncated (e.g. 160000 encoded in 2 bytes becomes 160000 & 0xFFFF = 28928), silently corrupting the chunk.
The library now verifies that a filtered chunk's size fits in the file's size of sizes and reports an error at write time instead, via a check in the H5D_CHUNK_ENCODE_SIZE_CHECK macro (src/H5Dchunk.c), which runs from H5D__chunk_file_alloc() before any chunk storage is allocated.
The in-range case, when the filtered chunk is small enough to be encoded by a small size of sizes, is tested by
test_chunk_size_of_sizes, and the case where a filtered chunk is too large to be encoded is tested bytest_chunk_size_of_sizes_overflow.Resolves #6023