Block Delimiter: Harden performance test to reduce flakiness #44896

LiamSarsfield · 2025-08-21T16:55:16Z

Fixes HOG-307: Harden Block_Scanner performance test to further remove flakiness

Proposed changes:

Reduce flakiness of Block_Scanner vs parse_blocks performance test by using more CI-friendly thresholds and fewer iterations.
Tag the test with #[Group('performance')].
Slightly adjust generated test content to keep runs fast and stable.

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

N/A

Does this pull request change what data or activity we track or use?

No

Testing instructions:

Verify the test passes and demonstrates Scanner performance advantage

- Centralize tunables in class constants for maintainability - Use competitive benchmark with paired measurements and warmups - Implement trimmed median aggregation for both time and memory - Relax assertions (1.15x → 1.10x ratio, 64KB memory tolerance) - Add retry mechanism for borderline results with graceful fallback - Fully qualify global functions to resolve linter errors - Preserve early-exit advantage with larger test fixture (2050 blocks) - Add always-on correctness test without performance assertions

github-actions · 2025-08-21T16:58:43Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Review, ...).
✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

Follow this PR Review Process:

Ensure all required checks appearing at the bottom of this PR are passing.
Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
You can use GitHub's Reviewers functionality to request a review.
When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

Jetpack plugin:

The Jetpack plugin has different release cadences depending on the platform:

WordPress.com Simple releases happen as soon as you deploy your changes after merging this PR (PCYsg-Jjm-p2).
WoA releases happen weekly.
Releases to self-hosted sites happen monthly:
- Scheduled release: September 2, 2025
- Code freeze: September 1, 2025

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

github-actions · 2025-08-21T17:01:04Z

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack), and enable the update/block-delimiter/performance-comparision-flakiness-HOG-307 branch.
To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack update/block-delimiter/performance-comparision-flakiness-HOG-307

Interested in more tips and information?

In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
Read more about our development workflow here: PCYsg-eg0-p2
Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

- Rename constants for better readability and understanding of their purpose - Update references in the benchmark tests to match new constant names - Ensure performance assertions are aligned with the new naming conventions

jp-launch-control · 2025-08-21T17:37:36Z

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

- Reduced warmup runs, benchmark iterations, and retry iterations for faster CI execution. - Updated performance assertion thresholds to be more CI-friendly, ensuring no significant regressions. - Added checks for system load to skip tests under high load conditions, enhancing reliability. - Adjusted test content generation to maintain early-exit advantage while improving execution speed.

- Adjusted performance assertion thresholds to ensure the scanner does not become significantly slower. - Removed redundant comments and streamlined the test logic for clarity. - Enhanced CI-friendliness by simplifying performance checks and maintaining consistent assertions.

- Changed assertion from assertGreaterThan to assertGreaterThanOrEqual to enhance the precision of performance tests. - This adjustment aims to ensure that the scanner's performance meets the defined minimum time ratio threshold without being overly strict.

dilirity

Tests pass - nice! From my limited tests knowledge, code makes sense as well 😅

I would suggest that anything benchmark related (that's actually code that helps with the benchmark, not actual tests) be moved in a separate folder (like lib or something).

That way the Block_Scanner_Test will only have tests in it.

kraftbj · 2025-08-22T15:12:22Z

Can you attach this to an Linear issue?

With the conversation on pc9hqz-3Fn-p2 , it does lead me to question if we should tweak this approach or do something else with it. This PR is fine, once @dilirity's comments are resolved, and I'm curious if we should think about a larger effort to have better benchmark and long-term verification against benchmarks than this test.

- Introduced Performance_Benchmark_Utils class to centralize benchmarking logic and improve test reliability. - Moved performance-related constants and methods from Block_Scanner_Test to the new utility class. - Updated Block_Scanner_Test to utilize the new utility methods for performance assertions and content generation. - Enhanced test structure for clarity and maintainability, ensuring consistent performance evaluation across tests.

…utoloading

LiamSarsfield · 2025-08-25T09:10:13Z

I would suggest that anything benchmark related (that's actually code that helps with the benchmark, not actual tests) be moved in a separate folder (like lib or something).

Good point, I've updated the tests to now have a utility class.

Can you attach this to an Linear issue?

Ah I didn't add the Linear issue within the description before. It is now.

t does lead me to question if we should tweak this approach or do something else with it.

I agree also 100%. I have created this ticket to investigate a performance benchmarking strategy in this general domain.
HOG-313: HACK Week: Establish a Backend Performance Baseline Strategy for Jetpack

dilirity

LiamSarsfield added 2 commits August 21, 2025 17:50

Add changelog entry for performance test improvements

b3b866d

github-actions bot added [Package] Block Delimiter [Status] In Progress [Tests] Includes Tests labels Aug 21, 2025

github-actions bot added the [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. label Aug 21, 2025

changelog

82b6021

github-actions bot added the [Plugin] Jetpack Issues about the Jetpack plugin. https://wordpress.org/plugins/jetpack/ label Aug 22, 2025

LiamSarsfield added the [Type] Enhancement Changes to an existing feature — removing, adding, or changing parts of it label Aug 22, 2025

LiamSarsfield added 2 commits August 22, 2025 08:54

LiamSarsfield requested review from kraftbj, dilirity and a team August 22, 2025 09:02

LiamSarsfield removed the [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. label Aug 22, 2025

LiamSarsfield marked this pull request as ready for review August 22, 2025 09:02

dilirity requested changes Aug 22, 2025

View reviewed changes

LiamSarsfield added 3 commits August 25, 2025 09:35

Update composer.lock for Block Delimiter and dependencies

9cafb6a

Update composer.json and composer.lock to include classmap for test a…

7392240

…utoloading

LiamSarsfield requested a review from dilirity August 25, 2025 09:10

dilirity approved these changes Aug 25, 2025

View reviewed changes

LiamSarsfield merged commit fa125c0 into trunk Aug 25, 2025
68 checks passed

LiamSarsfield deleted the update/block-delimiter/performance-comparision-flakiness-HOG-307 branch August 25, 2025 16:05

github-actions bot added this to the jetpack/15.0 milestone Aug 25, 2025

github-actions bot removed the [Status] In Progress label Aug 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Block Delimiter: Harden performance test to reduce flakiness #44896

Block Delimiter: Harden performance test to reduce flakiness #44896

Uh oh!

LiamSarsfield commented Aug 21, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 21, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 21, 2025 •

edited

Loading

Uh oh!

jp-launch-control bot commented Aug 21, 2025 •

edited

Loading

Uh oh!

dilirity left a comment

Uh oh!

kraftbj commented Aug 22, 2025

Uh oh!

LiamSarsfield commented Aug 25, 2025 •

edited

Loading

Uh oh!

dilirity left a comment

Uh oh!

Uh oh!

Uh oh!

Block Delimiter: Harden performance test to reduce flakiness #44896

Block Delimiter: Harden performance test to reduce flakiness #44896

Uh oh!

Conversation

LiamSarsfield commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes:

Other information:

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

Uh oh!

github-actions bot commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jp-launch-control bot commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Summary

Uh oh!

dilirity left a comment

Choose a reason for hiding this comment

Uh oh!

kraftbj commented Aug 22, 2025

Uh oh!

LiamSarsfield commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dilirity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LiamSarsfield commented Aug 21, 2025 •

edited

Loading

github-actions bot commented Aug 21, 2025 •

edited

Loading

github-actions bot commented Aug 21, 2025 •

edited

Loading

jp-launch-control bot commented Aug 21, 2025 •

edited

Loading

LiamSarsfield commented Aug 25, 2025 •

edited

Loading