GH-31869: [Python][Parquet] Implement external key material features in Python #48009
Conversation
pfparsons
requested review from
AlenkaF,
raulcd,
rok and
wgtmac
as code owners
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
github-actions
bot
added
Component: Parquet
Component: C++
Component: Python
awaiting review
pfparsons
changed the title
|
|
1 similar comment
|
|
|
|
|
Hi @raulcd would you mind taking a look at this? |
raulcd
left a comment
raulcd
left a comment
There was a problem hiding this comment.
Thanks for the PR @pfparsons !
I am not too familiar with the Parquet encryption side of things but I'll try to take a look. I am running CI at the moment. Maybe @EnricoMi @rok @adamreeve who have worked on Parquet Encryption in the past can take a look
github-actions
bot
added
awaiting changes
There was a problem hiding this comment.
Hi @pfparsons, thanks for the contribution! There are some minor formatting issues that the pre-commit script is reporting in https://github.com/apache/arrow/actions/runs/19077444221/job/54553464743?pr=48009, but the other CI failures are unrelated and are failing on main at the moment.
You can install pre-commit and run it locally to reproduce these errors, see https://pre-commit.com/. You can use something like pre-commit run --from-ref main --to-ref feature/31869 to run it on changes already committed.
I've left some comments with suggested changes, but this mostly looks really good to me. I'm not super familiar with cython though.
Co-authored-by: Adam Reeve <[email protected]>
github-actions
bot
added
awaiting change review
adamreeve
left a comment
adamreeve
left a comment
There was a problem hiding this comment.
This looks good to me thanks @pfparsons. I've just left a couple of minor comments.
EnricoMi
left a comment
EnricoMi
left a comment
There was a problem hiding this comment.
LGTM, just a minor comment on the default value of a string parameter.
Changed defaults for get_{en,de}cryption_properties() parquet_file_path
args from empty string to None
Removed unsed custom_kms_conf arg from
test_external_key_material_rotation
|
|
github-actions
bot
added
awaiting changes
pfparsons
force-pushed
the
feature/31869
branch
from
c7cb646 to
fe0a715
Compare
|
|
adamreeve
changed the title
|
|
|
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 5112de2. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 137 possible false positives for unstable benchmarks that are known to sometimes produce them. |
4 participants
Rationale for this change
Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality.
What changes are included in this PR?
This PR enables external key material for parquet encryption from PyArrow:
Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++
Exposes the rotate_master_keys method of CryptoFactory
Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test.
Are these changes tested?
Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store and added a test for CryptoFactory.rotate_master_keys.
Are there any user-facing changes?