[SPARK-52516][SQL] Don't hold previous iterator reference after advancing to next file in ParquetPartitionReaderFactory #51528
+36
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viirya
changed the title
There was a problem hiding this comment.
It seems that this causes many Parquet-related test case failures somehow. Could you double-check the test cases, @viirya ?
|
cc @peter-toth |
|
Sure, let me check it. |
…n ParquetPartitionReaderFactory
viirya
force-pushed
the
fix_iter_callback
branch
from
7f2023d to
d39dd1b
Compare
|
@dongjoon-hyun It should be fixed now. Thanks. |
3 tasks
peter-toth
approved these changes
Jul 17, 2025
szehon-ho
approved these changes
Jul 17, 2025
| this.iter = iter | ||
| } | ||
|
|
||
| def execute(): Unit = { |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This patch adds only one task completion listener for closing iterators in
ParquetPartitionReaderFactory, instead of adding separate one for each file iterator.Why are the changes needed?
For each file iterator, currently we add one task completion listener used to update closing the iterator when the task is finished. In the listener, the iterator reference is held. So if the file is normally exhausted, the reference cannot be released early. It is a problem especially if the reference are heavy as reported by apache/iceberg#13297.
Similar to #51503, we don't need set up a listener for each file iterator. Thus, we can set up just one listener for all iterator. Once we advance to next file, we can update the update target to new iterator.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.
Was this patch authored or co-authored using generative AI tooling?
No