Skip to content

MPI Read in Batches#22

Merged
c-bebop merged 13 commits intomainfrom
mpi_read_in_batches
Mar 16, 2025
Merged

MPI Read in Batches#22
c-bebop merged 13 commits intomainfrom
mpi_read_in_batches

Conversation

@c-bebop
Copy link
Collaborator

@c-bebop c-bebop commented Mar 11, 2025

This enables an application to read in batches of edges in parallel using our MPI Graph I/O code.

@c-bebop c-bebop requested a review from domischweisgut March 11, 2025 13:49
@c-bebop c-bebop self-assigned this Mar 11, 2025
Copy link
Collaborator

@domischweisgut domischweisgut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you read the files as in the test, everything is fine. If not, I think you have to look at the batch read function for MPI again.

@c-bebop
Copy link
Collaborator Author

c-bebop commented Mar 16, 2025

If you read the files as in the test, everything is fine. If not, I think you have to look at the batch read function for MPI again.

I've addressed the issue of potentially reading past EOF by tracking the count of read in edges along with parameters for batch size and edge size in bytes using ReadBatch struct. Doing so will guarantee that we never read more edge than actually present in the binary file.

Now, instead of living with the limitations of the MPI implementations, we offer an API that is safe from reading past EOF. I've also removed the offset functionality since it's not used nor tested.

@c-bebop c-bebop merged commit 817bc3c into main Mar 16, 2025
8 of 9 checks passed
@c-bebop c-bebop deleted the mpi_read_in_batches branch March 16, 2025 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants