Open
Conversation
Contributor
|
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
from
5aa70a7 to
294f8fe
Compare
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
from
294f8fe to
fa9b929
Compare
Split individual file downloads into configurable parallel segments using HTTP Range requests, matching aria2c-style parallelism. Adds --download-segments flag (default 16) to control the number of concurrent connections per file. Falls back to sequential download when the server does not support Range requests. Uses std::thread::scope with positioned writes (write_all_at) for thread-safe concurrent file I/O. Each segment validates the server returns 206 and caps reads to the expected length to prevent cross-segment overwrites. Co-Authored-By: Dan Cline <6798349+Rjected@users.noreply.github.com> Amp-Thread-ID: https://ampcode.com/threads/T-019ce82d-96b8-72d6-8cb8-f596f3f7260b
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
from
fa9b929 to
b7e0b65
Compare
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
6 times, most recently
from
785b1ec to
e6d61b8
Compare
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
from
e6d61b8 to
17030b2
Compare
…ed-downloads Amp-Thread-ID: https://ampcode.com/threads/T-019d7865-cc03-73ab-8635-79c149ab6a82 # Conflicts: # crates/cli/commands/src/download/mod.rs
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
4 times, most recently
from
cd7c67b to
42e502d
Compare
Rjected
force-pushed
the
perf/parallel-segmented-downloads
branch
from
42e502d to
c0e34e6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22944 (ports the idea to
main)Splits individual file downloads into configurable parallel segments using HTTP Range requests, matching aria2c-style parallelism. Snapshot downloads that took ~3 hours with a single connection can now complete in ~10 minutes with 16 parallel segments.
Changes
--download-segmentsflag (default 16, configurable per Dan's review on perf: parallel segmented snapshot downloads #22944)parallel_segmented_downloadprobes for Range support with a single-byte GET, then spawnsNthreads viastd::thread::scope(used instead of rayon per Dan's feedback about preferring tokio/async-friendly threading;std::thread::scopeworks well here since the code already runs insidespawn_blocking)FileExt::write_all_atfor thread-safe positioned writes (no shared seek cursor).take(expected_len)to prevent cross-segment overwritesresumable_downloadwhen server doesn't support Range requestsTesting
Existing tests pass. The parallel download path is exercised when
--resumableis used (or via the modular archive pipeline) against any Range-capable server.Co-Authored-By: Dan Cline 6798349+Rjected@users.noreply.github.com
Prompted by: Dan