Long Github Actions queue wait times

[aabmass]

cc @open-telemetry/python-maintainers

In the python repos, we're seeing really long queue wait times for the last few days. I think the issue is org wide though. In some cases, jobs are queued for over an hour.

Another case from earlier this week

---

https://www.githubstatus.com/ looks fine. Please let me know if the issue is on our side, and I can try to fix. Thanks!

[trask]

Did some AI assisted digging...

35 episodes in 33 days

Episode = 10-min bucket, ≥5 hosted jobs, p90 wait >10 min, merged within 30 min. Cells: jobs / p90 min, where the p90 links to the job that waited that long. Bold = delayed. - = no hosted jobs in that window. Links are to the p90 job.

episode (UTC)	mins	semantic-conv	python	python-contrib	js	go
06-30 06:20	10	-	-	764 / 27.2	-	-
06-30 15:10	10	-	1093 / 13.2	-	-	83 / 2.5
06-30 16:10	10	-	-	761 / 14.2	-	-
07-07 17:00	10	2 / 7.8	543 / 10.7	-	-	1 / 0.2
07-08 14:10	10	-	-	772 / 13.2	41 / 0.4	34 / 1.6
07-08 15:20	30	30 / 13.2	1087 / 4.7	3032 / 28.4	-	30 / 0.8
07-09 13:00	30	-	-	761 / 25.2	-	-
07-09 15:20	10	16 / 8.4	1423 / 11.1	-	-	-
07-09 16:20	10	-	541 / 10.6	-	-	-
07-10 14:30	10	14 / 9.7	557 / 6.6	756 / 12.9	-	31 / 0.8
07-13 16:40	10	51 / 8.6	-	768 / 12.1	-	31 / 0.2
07-14 13:50	20	-	1689 / 27.2	-	-	-
07-14 14:50	10	15 / 18.5	-	-	-	2 / 0.1
07-14 16:30	50	69 / 21.2	-	-	-	28 / 2.3
07-15 18:10	10	2 / 0.2	-	791 / 11.3	-	-
07-16 14:00	40	1 / 44.2	3399 / 16.9	1581 / 48.4	-	-
07-16 15:30	10	-	1129 / 12.6	789 / 5.0	-	30 / 1.2
07-20 13:30	10	1 / 9.7	565 / 12.2	-	-	-
07-20 14:30	240	193 / 47.2	3348 / 48.4	792 / 64.8	269 / 49.4	363 / 3.0
07-21 12:20	50	1 / 24.0	566 / 6.2	2382 / 26.9	72 / 0.8	-
07-21 18:30	10	13 / 22.8	-	-	-	-
07-22 15:00	30	17 / 42.4	-	809 / 51.0	-	-
07-22 16:20	70	32 / 17.3	1131 / 12.6	795 / 17.3	-	-
07-22 18:30	10	-	-	791 / 11.1	-	-
07-24 16:10	10	-	563 / 14.5	-	-	-
07-27 15:30	10	-	565 / 11.8	-	-	-
07-27 16:20	60	139 / 43.0	1013 / 15.4	-	-	31 / 4.1
07-27 19:00	50	34 / 43.8	2269 / 4.8	2373 / 44.9	-	30 / 2.4
07-27 20:30	20	1 / 16.1	2 / 0.1	3177 / 17.5	-	58 / 0.1
07-30 14:30	10	2 / 4.5	1130 / 12.2	-	-	-
07-30 19:30	30	-	-	1570 / 67.5	-	-
07-31 13:50	10	-	-	774 / 11.6	-	-
07-31 15:20	40	21 / 119.2	1697 / 21.6	775 / 119.2	-	-
07-31 16:40	30	1 / 65.9	1730 / 14.8	1544 / 66.1	-	-
07-31 20:20	10	-	-	1545 / 15.5	-	32 / 1.4

I'll open a ticket with CNCF to see if they can look into it.

Raw data available: queue-wait-raw-data.zip

[aabmass]

To help find a root cause, can someone with Admin access see the Github Actions metrics? https://docs.github.com/en/actions/how-tos/administer/view-metrics#viewing-github-actions-metrics-for-your-organization

I have a hunch that the python and python-contrib repos are enqueueing too many small workflows and eating into the 500 concurrent workflows limit.

[maryliag]

looking for top jobs runs and top workflow runs. I see the python repos with a lot of job runs. For the workflow run is the PR dashboard on top, then a huge gap following but java and collector

[aabmass]

Thanks, I will look into batching up some of the python checks.

For the workflow run is the PR dashboard on top, then a huge gap following but java and collector

IIUC, since open-telemetry/shared-workflows#269 the PR dashboard should be running on the CNCF runners?

[trask]

IIUC, since open-telemetry/shared-workflows#269 the PR dashboard should be running on the CNCF runners?

I'm reverting that in open-telemetry/shared-workflows#278, but have solved the massive number of shared-workflow (PR dashboard) runs in open-telemetry/shared-workflows#271.

[TylerHelmuth]

I'm feeling this pain every day in the NA morning in the helm repo and operator repo. Here is some data for the helm chart repo:

Overall queue time — last 30 days (731 jobs / 503 runs, all ubuntu-latest)

Metric	Value
Mean	6m47s
p50	3s
p90	10m16s
p95	49m44s
p99	2h28m
Max	2h40m
Jobs waiting >30s	175 (23.9%)
Jobs waiting >2m	132 (18.1%)
Jobs waiting >5m	109 (14.9%)

[fractalwrench]

I've also spotted Github Actions delays in the Kotlin repo too