Skip to content

fix(benches): bench sequentially #11032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Conversation

yash-atreya
Copy link
Member

Motivation

Currently, the benchmarks run in parallel using the same forge bin on the same machine. This leads to incorrect / slow results due to resource contention.

Solution

  • Run benches sequentially per project
  • Fix CI workflow to run sequentially as well since it's run on a self-hosted runner.

PR Checklist

  • Added Tests
  • Added Documentation
  • Breaking changes

Copy link
Contributor

📊 Foundry Benchmark Results

Generated at: 2025-07-17 14:25:08 UTC

Forge Test

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.3.0-nightly (0af4341 2025-07-17)
Repository stable nightly
ithacaxyz-account 5.12 s 5.15 s
solady 2.99 s 2.87 s

Forge Fuzz Test

Repository stable nightly
ithacaxyz-account 5.45 s 5.20 s
solady 3.08 s 3.00 s
📈 View all benchmark results

Forge Build

Repositories Tested

  1. ithacaxyz/account
  2. Vectorized/solady

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.3.0-nightly (0af4341 2025-07-17)

No Cache

Repository stable nightly
ithacaxyz-account 2.58 s 2.83 s
solady 3.88 s 3.98 s

With Cache

Repository stable nightly
ithacaxyz-account 0.210 s 0.443 s
solady 0.093 s 0.195 s

Forge Coverage

Repositories Tested

  1. ithacaxyz/account

Foundry Versions

  • stable: forge Version: 1.2.3-stable (a813a2c 2025-06-08)
  • nightly: forge Version: 1.3.0-nightly (0af4341 2025-07-17)
Repository stable nightly
ithacaxyz-account 22.03 s 21.98 s

System Information

  • OS: linux
  • CPU: 8
  • Rustc: rustc 1.88.0 (6b00bc388 2025-06-23)

🤖 This comment was automatically generated by the Foundry Benchmarks workflow.

To run benchmarks manually: Go to Actions → "Run workflow"

@yash-atreya yash-atreya marked this pull request as ready for review July 17, 2025 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

4 participants