Parallelize task instance gather and support --gather on Modal #199
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t take longer to gather
Previously, the script would fail if `git commit` was attempted with no changes. This was observed in cases like `Automattic__mongoose.5f57a5bb` where the applied patch resulted in no tracked changes. Now, we check `git status --porcelain` before committing and skip the instance if no changes are detected.
for more information, see https://pre-commit.ci
Codecov Report✅ All modified and coverable lines are covered by tests.
... and 4 files with indirect coverage changes 🚀 New features to boost your workflow:
|
…ateless git ls-remote
- Switch from per-task clones to per-worker persistent repositories. - Reduces clone operations from O(tasks) to O(workers) (e.g. 1400 -> 17). - Eliminates file locking race conditions. - Total gather time for Javascript is now ~5 minutes (bottlenecked by math.js).
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
8cefa3c to
e42a5e2
Compare
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
98898a8 to
302b73e
Compare
for more information, see https://pre-commit.ci
AlienKevin
force-pushed
the
kevin/bug-gen-gather
branch
from
52ba582 to
f7f68cb
Compare
for more information, see https://pre-commit.ci
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces parallel processing to the task instance gathering phase to significantly improve performance for large datasets and adds support for the gather phase in the Modal workflow.
Key Changes:
Before this PR, large repos like math.js with >800 task instances timed out after 20 minutes due to slow, sequential git branch creation and git push. After this PR, large repos finish in minutes.
ProcessPoolExecutorto process task instances in parallel, utilizing multiple cores.repo_name_pid_subfolder) to prevent race conditions during concurrent Git operations.--gatherCLI flag (skipping generation/validation).Question: do we want to fix the FAIL_TO_PASS to PASS_TO_FAIL?:
swe-smith currently uses
FAIL_TO_PASSfor tests that pass before the bug patch but fails afterwards, which inverts the semantic and causes confusion. A more intuitive name would bePASS_TO_FAILso I used this convention in this PR. However, if we are to adopt this new convention, the rest of the code and datasets need to be updated, so I'm not sure whether it's worth it?Resolution: Flip
PASS_TO_FAILtoFAIL_TO_PASSin alignment with SWE-bench naming convention when outputing the task instance jsons.Test command