Skip to content

Detect filtered chunk size overflow with small size of sizes - #6518

Open
mattjala wants to merge 7 commits into
HDFGroup:developfrom
mattjala:small_chunked_sizes
Open

mattjala wants to merge 7 commits into
HDFGroup:developfrom
mattjala:small_chunked_sizes

Conversation

@mattjala

@mattjala mattjala commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

For all chunk index types in version-5 chunk layout messages, and for the single chunk index in any version of the chunk layout message, the on-disk size of a filtered chunk is encoded in a fixed-width field equal to the file's "size of sizes" (set via H5Pset_sizes()). The encode check assumes this field is always 8 bytes, so when the size of sizes is set to 2 or 4 an oversized chunk has its encoded size silently truncated (e.g. 160000 encoded in 2 bytes becomes 160000 & 0xFFFF = 28928), silently corrupting the chunk.

The library now verifies that a filtered chunk's size fits in the file's size of sizes and reports an error at write time instead, via a check in the H5D_CHUNK_ENCODE_SIZE_CHECK macro (src/H5Dchunk.c), which runs from H5D__chunk_file_alloc() before any chunk storage is allocated.

The in-range case, when the filtered chunk is small enough to be encoded by a small size of sizes, is tested by test_chunk_size_of_sizes, and the case where a filtered chunk is too large to be encoded is tested by test_chunk_size_of_sizes_overflow.

Resolves #6023

@mattjala mattjala added the Component - C Library Core C library issues (usually in the src directory) label Jul 10, 2026
Copilot AI review requested due to automatic review settings July 10, 2026 21:47
@mattjala mattjala added the Component - Testing Code in test or testpar directories, GitHub workflows label Jul 10, 2026
@github-project-automation github-project-automation Bot moved this to To be triaged in HDF5 - TRIAGE & TRACK Jul 10, 2026
@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Checklist

This PR touches the following areas. Each needs a sign-off
from its listed owners before merging.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a data-corruption risk when writing filtered chunked datasets to files whose configured “size of sizes” (via H5Pset_sizes()) is smaller than 8 bytes, by detecting and erroring out on filtered chunk sizes that do not fit in the on-disk encoded field width.

Changes:

  • Add an overflow check in H5D_CHUNK_ENCODE_SIZE_CHECK to ensure filtered chunk sizes fit within the file’s configured “size of sizes” for v5 chunk layout messages and for the single-chunk index in any layout version.
  • Add new dataset tests covering round-trip correctness for small “size of sizes” values and explicit overflow detection on write/flush.
  • Document the fix in the release changelog.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
test/dsets.c Adds new tests for filtered chunk size encoding with size of sizes set to 2/4/8, including an overflow-failure case.
src/H5Dchunk.c Extends the filtered chunk size encoding check to honor the file’s configured “size of sizes” field width.
release_docs/CHANGELOG.md Notes the bug fix and the prior silent truncation behavior in release documentation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/dsets.c Outdated
@github-actions
github-actions Bot requested a review from hyoklee July 14, 2026 14:20
hyoklee
hyoklee previously approved these changes Jul 21, 2026
mattjala and others added 3 commits July 21, 2026 13:58
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment thread release_docs/CHANGELOG.md Outdated
@github-project-automation github-project-automation Bot moved this from To be triaged to In progress in HDF5 - TRIAGE & TRACK Jul 21, 2026
@github-actions
github-actions Bot requested a review from hyoklee July 22, 2026 14:36
hyoklee
hyoklee previously approved these changes Jul 22, 2026
hyoklee
hyoklee previously approved these changes Aug 6, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@hyoklee hyoklee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's suggestion for CHANGELOG.md:

### Fixed silent truncation of filtered chunk sizes when the file "size of sizes" is small

For all chunk index types in version‑5 chunk layout messages, and for the single chunk index in any version of the chunk layout message, the on‑disk size of a filtered chunk is encoded in a fixed‑width field equal to the file's "size of sizes" (set via `H5Pset_sizes()`). The encode check previously assumed this field was always 8 bytes. When the size of sizes was set to 2 or 4, an oversized chunk's encoded size could be silently truncated (for example, 160000 encoded in 2 bytes becomes 160000 & 0xFFFF = 28928), corrupting the chunk. The library now verifies that a filtered chunk's size fits within the file's size of sizes and reports an error at write time.

Fixes GitHub issue #6023

@hyoklee hyoklee added this to the Backlog milestone Aug 28, 2026
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

This pull request has had no activity for 30 days and has been marked stale. Push a commit or comment to keep it open, or it will be flagged for maintainer review.

@github-actions github-actions Bot added the stale label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component - C Library Core C library issues (usually in the src directory) Component - Testing Code in test or testpar directories, GitHub workflows stale

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

Test setting size of sizes to less than 8 with chunked datasets

3 participants