Skip to content

MAPREDUCE-7508.FileInputFormat can throw ArrayIndexOutofBoundsException because of some concurrent execution. #7859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from
+3 −0
Open

MAPREDUCE-7508.FileInputFormat can throw ArrayIndexOutofBoundsException because of some concurrent execution. #7859

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
53a1844
MAPREDUCE-7508.FileInputFormat can throw ArrayIndexOutofBoundsExcepti…
liangyu-1 Aug 7, 2025
d2ba001
reformat code
liangyu-1 Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .../hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,9 @@ public InputSplit[] getSplits(JobConf job, int numSplits)
} else {
blkLocations = fs.getFileBlockLocations(file, 0, length);
}
if (blkLocations.length == 0){
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contributions. Just wonder why it will meet length != 0 but blkLocations.length == 0, some corner case that lead inconsistent metadata of NameNode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Hexiaoqiao thanks for review.

It is a rare corner case, only happens when using spark streaming monitors a path where the upstream system sometimes starts two identical tasks that attempt to create and write to the same HDFS file simultaneously. This can lead to conflicts where a file is created and written to twice in quick succession.

When Spark scans for files, it uses the FileInputFormat.getSplits() method to split the file. The first step in getSplits is to retrieve the file's length. If the file length is not zero, the next step is to get the block locations array for that file. However, if the two upstream programs rapidly create and write to the same file (i.e., the file is overwritten or appended to almost simultaneously), a race condition may occur:

The file's length is already non-zero, but calling getFileBlockLocations() returns an empty array because the file is being overwritten or is not yet fully written.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Got it. Make sense to me. +1 to check in.
BTW, the root cause here is invoke listStatus to get file status and invoke another interface getFileBlockLocations to get block location, but file has changed between this steps, right? If it is true, is it proper to use blkLocations only as condition at L364 rather than file.length which could be not the correct result here? Thanks again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Got it. Make sense to me. +1 to check in. BTW, the root cause here is invoke listStatus to get file status and invoke another interface getFileBlockLocations to get block location, but file has changed between this steps, right? If it is true, is it proper to use blkLocations only as condition at L364 rather than file.length which could be not the correct result here? Thanks again.

Thanks for your review. It is a good solution to use blkLocations only as condition at L364, if blkLocations array is empty, file.length is also empty, this will not effect the later opretions.

continue;
}
if (isSplitable(fs, path)) {
long blockSize = file.getBlockSize();
long splitSize = computeSplitSize(goalSize, minSize, blockSize);
Expand Down