Skip to content

Make fuzz runtime seconds not iterations #4179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Make fuzz runtime seconds not iterations #4179

wants to merge 2 commits into from
+4 −11

Conversation

@TheBlueMatt
Copy link
Collaborator

No description provided.

@ldk-reviews-bot
Copy link

ldk-reviews-bot commented Oct 28, 2025 •
edited
Loading

I've assigned @wpaulino as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@codecov
Copy link

codecov bot commented Oct 28, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.86%. Comparing base (17f7858) to head (fbdb5c5).
⚠️ Report is 21 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #4179      +/-   ##
==========================================
+ Coverage   88.83%   88.86%   +0.02%     
==========================================
  Files         180      180              
  Lines      137504   137504              
  Branches   137504   137504              
==========================================
+ Hits       122155   122192      +37     
+ Misses      12538    12510      -28     
+ Partials     2811     2802       -9
Flag Coverage Δ
fuzzing 20.90% <ø> (-0.59%) ⬇️
tests 88.70% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TheBlueMatt TheBlueMatt mentioned this pull request Oct 28, 2025
Open
@TheBlueMatt TheBlueMatt force-pushed the 2025-10-fiz-fuzz branch 5 times, most recently from 8bfffb2 to 1ba5def Compare October 28, 2025 02:02
tnull
tnull reviewed Oct 28, 2025
View reviewed changes
fuzz/ci-fuzz.sh
# Because we're fuzzing relatively few iterations, the maximum possible
# compiler optimizations aren't necessary, so we turn off LTO
sed -i 's/lto = true//' Cargo.toml
sed -i 's/codegen-units = 1//' Cargo.toml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #3916 we made some attempts of benchmarking before we removed this line. Do you have a vague number of how much the slowdown would be?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I forgot about that, but its almost certainly a tiny difference in fuzzing performance, for a bit more parallelism while compiling some crates. 🤷‍♂️

@TheBlueMatt
Copy link
Collaborator Author

Oh, weird, I guess bumping the debian version fixed the build even with 1.75, which I thought I tested....anyway, I dropped the MSRV bump for fuzzing cause it appears to be woring now. Now this just reduces the runtime to make it more consistent across jobs.

@TheBlueMatt TheBlueMatt changed the title Fix fuzzing CI and tweak runtime Make fuzz runtime seconds not iterations Oct 28, 2025
@tnull
Copy link
Contributor

tnull commented Oct 29, 2025

Oh, weird, I guess bumping the debian version fixed the build even with 1.75, which I thought I tested....anyway, I dropped the MSRV bump for fuzzing cause it appears to be woring now.

It's failing in CI with a linking error though?

@TheBlueMatt
Use honggfuzz's --run-time arg to limit per-fuzz runtime in CI
fef4f50 
We have some complexity in `ci-fuzz.sh` to limit each fuzzer to a
rough runtime, but `honggfuzz` has a `--run-time` argument that we
can simply use instead, which we do here.
@TheBlueMatt
Disable codegen-units = 1 in CI fuzz job
fbdb5c5 
This now slows us down as we run our fuzz job on a machine with
more than one or two cores.
@TheBlueMatt
Copy link
Collaborator Author

Weird, it seems to be inconsistent, sometimes its fine, sometimes it fails.

@ldk-reviews-bot
Copy link

🔔 1st Reminder

Hey @wpaulino! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@TheBlueMatt
Copy link
Collaborator Author

The fuzz job seems to be pretty consistently passing on the builds on git main so for now i think we should just land this as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tnull tnull tnull left review comments

@wpaulino wpaulino Awaiting requested review from wpaulino

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@TheBlueMatt @ldk-reviews-bot @tnull