Skip to content

Use crc-fast v1.4.0 for 2X performance boost #4257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

onethumb
Copy link
Contributor

@onethumb onethumb commented Aug 9, 2025

Motivation and Context

crc-fast v1.4.0 doubles throughput to >100GiBs on modern x86_64 systems (Intel Ice Lake, AMD Zen4, and later) when using Rust 1.89.0+ and its stabilized support for AVX512 intrinsics.

It uses conditional compilation to maintain backwards compatibility and performance for Rust versions prior to 1.89.0.

Description

  • Use latest v1.4.0 of the crc-fast library.
  • Updated the Cargo lock files using ./gradlew aws:sdk:cargoUpdateAllLockfiles.
  • Full release notes.

Testing

Drop-in, non-breaking minor version update. Existing tests pass.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Doubles throughput to >100GiB/s on modern x86_64 systems (Intel Ice
Lake, AMD Zen4, and later).
Used: ./gradlew aws:sdk:cargoUpdateAllLockfiles
@onethumb onethumb requested a review from a team as a code owner August 9, 2025 19:29
@onethumb
Copy link
Contributor Author

onethumb commented Aug 9, 2025

Pinging @landonxjames for review. Thanks! 👍

@rcoh
Copy link
Collaborator

rcoh commented Aug 11, 2025

Consumers don't actually depend on these lockfiles—so this doesn't actually impact anything right? More of a "test that everything builds with 1.4"?

However, this does increase the MSRV beyond policy (for, I think, no downstream benefit?) since builders using newer Rustc will pick these improvements up anyway?

edit: I see that this is happening internally via rustc_version gating.

I think the main reason to avoid merging this, at least for the next few weeks, is that it prevents a consumer from forcing the 1.3.0 dependency in the event they are broken by the 1.4.0 changes.

@onethumb
Copy link
Contributor Author

edit: I see that this is happening internally via rustc_version gating.

I think the main reason to avoid merging this, at least for the next few weeks, is that it prevents a consumer from forcing the 1.3.0 dependency in the event they are broken by the 1.4.0 changes.

Is the "next few weeks" timeline just to give things (Rust 1.89.0, this package, etc) time to shake out? Or is there some other change happening in a few weeks that would impact this? (I'm not in any sort of rush, just trying to make sure this doesn't slip off my radar if it languishes).

FWIW, our GH Actions workflows have good version (1.81, 1.89, stable, nightly) and architecture (x86, x86_64, aarch64, powerpc) coverage, and the internal test coverage is extensive, so I think the odds of breakage are low.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants