Support local iceberg table with parquet files to local validation and e2es

[akshaychitneni]

Currently data cache is integrates static iceberg table on s3. This issue tracks support for local iceberg table for local validation of cache and also add support to e2e

[akshaychitneni]

/good-first-issue
/area data-cache

[google-oss-prow]

@akshaychitneni:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

Details

In response to this:

/good-first-issue
/area data-cache

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Goku2099]

Thanks for labeling this as a good-first-issue. I can start by exploring local Iceberg validation and propose a small initial change (docs or tests) to get things moving.

[modichika]

Hey @akshaychitneni i would like to take this.

[Snehadas2005]

Hi @akshaychitneni, I would like to work on this issue. I recently completed a PR for the Trainer notebooks, and I am interested in contributing to the data-cache infrastructure.

I have analysed the Rust codebase in pkg/data_cache/ and the existing Docker configuration. My objective is to implement a local file-system provider to support Iceberg/Parquet validation locally and integrate these changes into the E2E testing suite.

Could you please assign this to me? Thank you!

[RONAK-AI647]

/assign