[build] record flaky tests on trunk and report weekly offenders to Slack

[titusfortner]

💥 What does this PR do?

• Adds tracking for flaky tests, so random PR failures can be checked against what is known to be flaky
• Adds workflow to document flaky test targets from previous 3 weeks (should catch flakiness of ~7%)
• Flakiness workflow runs weekly and reports to Slack if things need investigation (anything > ~20% flaky)
• Adds daily scheduled ci-rbe.yml workflow run without test cache to get a full run of all targets
• Scheduled ci.yml workflow run is now once a day instead of twice

🔧 Implementation Notes

• Flaky means the target did not pass first time in at least one run but ended green in at least one, so needing a retry every single run reads as 100% flaky; a target that never ended green is broken rather than flaky and is left out.
• Records flakiness from Run Bazel step, Rerun failures with debug step and github rerun failing tests attempts.
• Rate is flaked runs over total runs across 3 weeks — one week gives too few runs to divide by — counted per run rather than per attempt, and reported only when the target also flaked in the last 7 days.
• Reuses rerun-with-debug in jobs to indicate whether to measure flakiness, which means removing it from unit tests where it shouldn't be needed
• ci.yml merges its per-job results into one artifact, so the weekly report reads about 40 files instead of 550 and stays well inside the 1,000 requests/hour token limit.
• Most Selenium trunk merges are between 07:00 and 01:00 UTC, so daily runs moved outside that window.
• Updated Ruby tests so that the smoke tests always run with the regression test suite.

🤖 AI assistance

• AI assisted (complete below)
  • Tool(s): Claude Code (Opus 5)
  • What was generated: the three scripts, the report workflow, the steps added to bazel.yml, ci.yml and ci-rbe.yml, and this description
  • I reviewed all AI output and can explain the change

💡 Additional Considerations

• We can adjust threshold values once we have real data to look at
• We can restore the CI workflow to run 2x/day if we can keep up with the results

🔄 Types of changes

• New feature (CI tooling)

[qodo-code-review]

PR Summary by Qodo

Track Bazel flakes and report weekly offenders to Slack

✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Records uncached Bazel outcomes from scheduled trunk runs as retained artifacts.
• Aggregates three weeks of per-target, per-OS flakiness into weekly reports.
• Alerts Slack when recently flaky targets exceed configurable confidence thresholds.

Diagram

graph TD
  A["Daily trunk CI"] --> B["Bazel jobs"] --> C["Result parser"] --> D[("Job artifacts")] --> E["Result merger"] --> F[("Merged history")] --> G["Weekly report"] --> H["Slack alert"]
  D -. "fallback" .-> G

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Parse Bazel event protocol output

• ➕ Uses structured data instead of human-readable console formatting
• ➕ Can expose richer test-attempt and timing metadata
• ➖ Requires changing Bazel invocations and processing BEP files
• ➖ Rerun correlation and artifact retention are still needed
• ➖ Introduces more implementation and review complexity

2. Adopt a hosted flaky-test service

• ➕ Provides dashboards, trends, ownership, and notifications out of the box
• ➕ Avoids maintaining custom aggregation and reporting scripts
• ➖ Adds vendor cost and external data handling
• ➖ May not model Selenium's Bazel retries and multi-OS targets precisely
• ➖ Requires integration and operational ownership

Recommendation: Use the proposed artifact-based pipeline as an incremental, repository-native solution: it reuses existing rerun data, avoids a new service, and merges artifacts to remain within API limits. Structured Bazel event output would be a worthwhile follow-up if console-summary parsing proves unstable.

Files changed (9) +363 / -12

Enhancement (6) +356 / -5

bazel.ymlCapture and upload scheduled Bazel flakiness records +18/-4

Capture and upload scheduled Bazel flakiness records

• Renames failed-log collection references, makes diagnostic artifact names unique per workflow attempt, and parses eligible scheduled trunk executions into retained JSONL artifacts. Parsing and uploads are non-blocking so observability failures do not fail CI.

.github/workflows/bazel.yml

ci.ymlSchedule daily CI and consolidate flaky artifacts +28/-1

Schedule daily CI and consolidate flaky artifacts

• Changes scheduled CI from twice daily to 04:00 UTC and adds an always-running trunk aggregation job after language suites complete. The job merges per-job records into one 60-day artifact, reducing API requests made by weekly reporting.

.github/workflows/ci.yml

flaky-report.ymlAdd configurable weekly flakiness reporting +62/-0

Add configurable weekly flakiness reporting

• Introduces a Monday workflow that builds a report from retained history using configurable window, recency, sample, and rate thresholds. Scheduled runs notify the Selenium TLC Slack channel when qualifying offenders exist, while manual runs expose the complete report.

.github/workflows/flaky-report.yml

flaky-report.shAggregate artifact history into flakiness reports +143/-0

Aggregate artifact history into flakiness reports

• Downloads recent merged or fallback per-job artifacts, deduplicates artifact selection by workflow run, and calculates target rates per operating system. It emits a Markdown summary and workflow outputs only when recent offenders exceed the configured run-count and unrounded-rate thresholds.

scripts/github-actions/flaky-report.sh

merge-flaky-results.shMerge per-job flakiness records +30/-0

Merge per-job flakiness records

• Concatenates downloaded job-level JSONL files into a run-level artifact and reports whether output is available. Empty downloads are treated as a successful no-op so missing results do not break scheduled CI.

scripts/github-actions/merge-flaky-results.sh

parse-flaky-results.shParse Bazel executions and recovered failures +75/-0

Parse Bazel executions and recovered failures

• Parses non-cached Bazel summaries into one JSONL record per executed target, including passed, failed, retry-recovered, and debug-rerun-recovered states. It records attempts, failures, operating system, and timestamp while adding recovered flakes to the job summary.

scripts/github-actions/parse-flaky-results.sh

Refactor (1) +0 / -0

collect-failed-logs.shGeneralize failed-log collection script naming +0/-0

Generalize failed-log collection script naming

• Renames the failed Bazel test-log collector referenced by the reusable workflow while preserving collection from the final failure list.

scripts/github-actions/collect-failed-logs.sh

Other (2) +7 / -7

ci-rbe.ymlRun daily uncached remote-build CI +3/-1

Run daily uncached remote-build CI

• Adds a 02:00 UTC daily schedule and automatically disables the Bazel test cache for scheduled executions. This ensures the flakiness denominator represents real target executions rather than cached results.

.github/workflows/ci-rbe.yml

ci-ruby.ymlAlways run Ruby smoke coverage +4/-6

Always run Ruby smoke coverage

• Makes smoke tests run alongside the full scheduled matrix, while removing flakiness measurement from unit tests that do not rerun with debug. It also excludes OS-sensitive and Selenium Manager tags from full browser matrices to prevent overlap.

.github/workflows/ci-ruby.yml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

1. Failed reruns appear recovered ✓ Resolved 🐞 Bug ≡ Correctness

Description

The parser assumes every initial failure absent from _run2.txt passed on rerun, but _run2.txt
only contains FAILED summaries. Targets that time out, become incomplete, or emit no summary
because the rerun aborts are therefore recorded as rerun-recovered, corrupting flake rates and
potentially triggering false Slack alerts.

Code

scripts/github-actions/parse-flaky-results.sh[47]

+  comm -23 <(sort "$FAILURES/_run1.txt") <(sort "$FAILURES/_run2.txt") > "$FAILURES/_recovered.txt"

Evidence

The original failure list includes FAILED, TIMEOUT, and INCOMPLETE targets, while the rerun list
records only FAILED targets. The new comm -23 logic treats every missing rerun target as
recovered, so absence is incorrectly equated with a successful rerun.

scripts/github-actions/rerun-failures.sh[9-10]
scripts/github-actions/rerun-failures.sh[41-47]
scripts/github-actions/parse-flaky-results.sh[42-52]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Rerun recovery is inferred from absence in `_run2.txt`, even though that file only records `FAILED` targets. TIMEOUT, INCOMPLETE, interrupted, or otherwise unreported reruns can consequently be marked as recovered.

## Issue Context
Parse rerun output into explicit passed and unsuccessful target sets. Only change a target to `rerun-recovered` when the rerun summary explicitly reports that target as passed; retain the original failure status otherwise.

## Fix Focus Areas
- scripts/github-actions/parse-flaky-results.sh[46-52]
- scripts/github-actions/rerun-failures.sh[41-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Flaky scripts lack tests 📘 Rule violation ☼ Reliability

Description

The new parsing, aggregation, and reporting behavior has no automated tests, despite relying on
multiple output formats, thresholds, and artifact-selection rules. Regressions could silently
produce inaccurate flaky-test reports or Slack notifications.

Code

scripts/github-actions/parse-flaky-results.sh[R33-36]

+  $2 ~ /^FLAKY/ { print $1 "\tretry-recovered\t" $5 "\t" $8 > out; next }
+  $3 != "in" { next }
+  $2 == "PASSED" { print $1 "\tpassed\t0\t1" > out; next }
+  $2 ~ /(FAILED|TIMEOUT|INCOMPLETE)/ {

Evidence

PR Compliance ID 4 requires behavioral changes to include appropriate tests and favors small unit
tests. The cited scripts introduce substantial parsing and reporting behavior, while repository
searches found no automated test or fixture invoking any of them; only their workflow callers exist.

AGENTS.md: Prefer Writing/Updating Tests and Favor Small Unit Tests Over Browser Tests
scripts/github-actions/parse-flaky-results.sh[23-40]
scripts/github-actions/merge-flaky-results.sh[17-29]
scripts/github-actions/flaky-report.sh[79-109]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Add automated tests for the newly introduced flaky-test parsing, merging, and reporting scripts.

## Issue Context
Repository-wide searches found only workflow callers for these scripts and no tests exercising representative Bazel logs, recovered reruns, cached results, artifact deduplication, date windows, or reporting thresholds. Use small fixture-based tests rather than browser or end-to-end tests.

## Fix Focus Areas
- scripts/github-actions/parse-flaky-results.sh[23-75]
- scripts/github-actions/merge-flaky-results.sh[17-29]
- scripts/github-actions/flaky-report.sh[41-143]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Slack threshold too low ✗ Dismissed 🐞 Bug ≡ Correctness

Description

The workflow defaults min-rate to 5, so scheduled runs notify Slack for targets above 5% rather
than the stated approximately 20% offender threshold. This will report ordinary low-rate flakes as
weekly offenders and create substantially more alerts than intended.

Code

.github/workflows/flaky-report.yml[R23-27]

+      min-rate:
+        description: Percent of runs a target must flake in before it is counted
+        required: false
+        type: string
+        default: '5'

Evidence

The workflow input and script both default MIN_RATE to 5, and the significant-target query directly
selects rates greater than that value before setting the Slack output.

.github/workflows/flaky-report.yml[23-27]
.github/workflows/flaky-report.yml[44-52]
scripts/github-actions/flaky-report.sh[128-143]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The scheduled flaky report uses a default minimum rate of 5%, while the PR specifies Slack reporting for offenders above approximately 20%.

## Issue Context
Update the workflow and script defaults consistently so scheduled execution uses the intended offender threshold while preserving manual overrides.

## Fix Focus Areas
- .github/workflows/flaky-report.yml[23-27]
- .github/workflows/flaky-report.yml[44-50]
- scripts/github-actions/flaky-report.sh[25-28]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

Review mode: ⚖️ Balanced: This is a localized behavioral change in flaky-result classification, where parsing and failure-state assumptions can affect CI reporting; it is real risk but not dense or broad enough to warrant extended review.

Tip of the day

💡 Did you know, you can show, collapse, or hide each part of a finding: code, evidence, and all

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit 8e5edd5 ⚖️ Balanced

Results up to commit 23cfc43 ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (1) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Flaky scripts lack tests 📘 Rule violation ☼ Reliability

Description

The new parsing, aggregation, and reporting behavior has no automated tests, despite relying on
multiple output formats, thresholds, and artifact-selection rules. Regressions could silently
produce inaccurate flaky-test reports or Slack notifications.

Code

scripts/github-actions/parse-flaky-results.sh[R33-36]

+  $2 ~ /^FLAKY/ { print $1 "\tretry-recovered\t" $5 "\t" $8 > out; next }
+  $3 != "in" { next }
+  $2 == "PASSED" { print $1 "\tpassed\t0\t1" > out; next }
+  $2 ~ /(FAILED|TIMEOUT|INCOMPLETE)/ {

Evidence

PR Compliance ID 4 requires behavioral changes to include appropriate tests and favors small unit
tests. The cited scripts introduce substantial parsing and reporting behavior, while repository
searches found no automated test or fixture invoking any of them; only their workflow callers exist.

AGENTS.md: Prefer Writing/Updating Tests and Favor Small Unit Tests Over Browser Tests
scripts/github-actions/parse-flaky-results.sh[23-40]
scripts/github-actions/merge-flaky-results.sh[17-29]
scripts/github-actions/flaky-report.sh[79-109]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Add automated tests for the newly introduced flaky-test parsing, merging, and reporting scripts.

## Issue Context
Repository-wide searches found only workflow callers for these scripts and no tests exercising representative Bazel logs, recovered reruns, cached results, artifact deduplication, date windows, or reporting thresholds. Use small fixture-based tests rather than browser or end-to-end tests.

## Fix Focus Areas
- scripts/github-actions/parse-flaky-results.sh[23-75]
- scripts/github-actions/merge-flaky-results.sh[17-29]
- scripts/github-actions/flaky-report.sh[41-143]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Failed reruns appear recovered ✓ Resolved 🐞 Bug ≡ Correctness

Description

The parser assumes every initial failure absent from _run2.txt passed on rerun, but _run2.txt
only contains FAILED summaries. Targets that time out, become incomplete, or emit no summary
because the rerun aborts are therefore recorded as rerun-recovered, corrupting flake rates and
potentially triggering false Slack alerts.

Code

scripts/github-actions/parse-flaky-results.sh[47]

+  comm -23 <(sort "$FAILURES/_run1.txt") <(sort "$FAILURES/_run2.txt") > "$FAILURES/_recovered.txt"

Evidence

The original failure list includes FAILED, TIMEOUT, and INCOMPLETE targets, while the rerun list
records only FAILED targets. The new comm -23 logic treats every missing rerun target as
recovered, so absence is incorrectly equated with a successful rerun.

scripts/github-actions/rerun-failures.sh[9-10]
scripts/github-actions/rerun-failures.sh[41-47]
scripts/github-actions/parse-flaky-results.sh[42-52]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Rerun recovery is inferred from absence in `_run2.txt`, even though that file only records `FAILED` targets. TIMEOUT, INCOMPLETE, interrupted, or otherwise unreported reruns can consequently be marked as recovered.

## Issue Context
Parse rerun output into explicit passed and unsuccessful target sets. Only change a target to `rerun-recovered` when the rerun summary explicitly reports that target as passed; retain the original failure status otherwise.

## Fix Focus Areas
- scripts/github-actions/parse-flaky-results.sh[46-52]
- scripts/github-actions/rerun-failures.sh[41-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Slack threshold too low ✗ Dismissed 🐞 Bug ≡ Correctness

Description

The workflow defaults min-rate to 5, so scheduled runs notify Slack for targets above 5% rather
than the stated approximately 20% offender threshold. This will report ordinary low-rate flakes as
weekly offenders and create substantially more alerts than intended.

Code

.github/workflows/flaky-report.yml[R23-27]

+      min-rate:
+        description: Percent of runs a target must flake in before it is counted
+        required: false
+        type: string
+        default: '5'

Evidence

The workflow input and script both default MIN_RATE to 5, and the significant-target query directly
selects rates greater than that value before setting the Slack output.

.github/workflows/flaky-report.yml[23-27]
.github/workflows/flaky-report.yml[44-52]
scripts/github-actions/flaky-report.sh[128-143]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The scheduled flaky report uses a default minimum rate of 5%, while the PR specifies Slack reporting for offenders above approximately 20%.

## Issue Context
Update the workflow and script defaults consistently so scheduled execution uses the intended offender threshold while preserving manual overrides.

## Fix Focus Areas
- .github/workflows/flaky-report.yml[23-27]
- .github/workflows/flaky-report.yml[44-50]
- scripts/github-actions/flaky-report.sh[25-28]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

Code review by qodo was updated up to the latest commit 8e5edd5