We are rate limited on the GitHub `generate-jitconfig` API

[delan]

Flapping runners due to issues like #43 can cause us to hit GitHub API rate limits.

Jul 12 02:28:16 ci0 monitor-start[2172579]: 2025-07-12T02:28:16.156039Z  INFO monitor::profile: Creating runner runner_id=50603 profile.base_vm_name="servo-ubuntu2204"
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.147353Z  INFO monitor::shell: line=
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.147381Z  INFO monitor::shell: line=Allocating 'base-clone-1.i  0% [                 ]    0 B/s |  512 B  --:-- ETA
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.235766Z  INFO monitor::shell: line=
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.235777Z  INFO monitor::shell: line=Allocating 'base-clone-1.img'                               |  90 GB  00:02
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.236027Z  INFO monitor::shell: line=
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.236030Z  INFO monitor::shell: line=Clone 'ci-servo-ubuntu2204.50603' created successfully.
Jul 12 02:28:19 ci0 monitor-start[2172579]: 2025-07-12T02:28:19.262845Z  INFO monitor::image: Starting guest
Jul 12 02:28:20 ci0 monitor-start[2415863]: Domain 'ci-servo-ubuntu2204.50603' started
Jul 12 02:28:20 ci0 monitor-start[2172579]: 2025-07-12T02:28:20.756554Z  INFO monitor: Received API request request=GithubJitconfig { response_tx: Sender { .. }, remote_addr: RemoteAddr { client_ip: Some(192.168.100.151), real_ip: None } }
Jul 12 02:28:22 ci0 monitor-start[2172579]: 2025-07-12T02:28:22.937265Z  INFO cmd_lib::child: gh: API rate limit exceeded for user ID 251518. If you reach out to GitHub Support for help, please include the request ID DE26:388973:1983D64:1A55618:6871C846 and timestamp 2025-07-12 02:28:22 UTC. (HTTP 403)
Jul 12 02:28:23 ci0 monitor-start[2172579]: 2025-07-12T02:28:23.296252Z ERROR monitor: Monitor thread error report=Running ["gh" "api" "-H" "Accept: application/vnd.github+json" "-H" "X-GitHub-Api-Version: 2022-11-28" "/orgs/servo/actions/runners" "--paginate" "-q" ".runners[]" | "jq" "-s" "."] exited with error; status code: 1 at src/github.rs:83
Jul 12 02:28:23 ci0 systemd[1]: monitor.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 02:28:24 ci0 systemd[1]: monitor.service: Failed with result 'exit-code'.
Jul 12 02:28:24 ci0 systemd[1]: monitor.service: Consumed 12min 35.416s CPU time, 320.7M memory peak, 47.6M incoming IP traffic, 8.1G outgoing IP traffic.
Jul 12 02:28:24 ci0 systemd[1]: monitor.service: Scheduled restart job, restart counter is at 11.

[delan]

we are still rate limited. it might actually be due to the 400 pages of runners (~10000 runners) we have registered:

i suspect this is because the docker image also uses the deprecated runner 2.323.0. in #43, we found that this causes runners to churn as fast as every 40 seconds (SERVO_CI_MONITOR_START_TIMEOUT / target_count) in the non-docker monitor, because we keep trying to register runners that never successfully listen for jobs.

maybe something similar has happened in the docker monitor? @Narfinger @jschwe

[delan]

the fact that we have exactly 400 pages of runners (~10000 runners) is also suspicious. we may need to unregister all of those dead runners before we can register any new runners.

in the non-docker monitor, we unregister runners that hit the SERVO_CI_MONITOR_START_TIMEOUT without successfully listening for jobs. we do this by polling the github api to check if our runners are actually online, and unregistering and destroying runners that are StartedOrCrashed for too long.

does the docker monitor unregister runners that fail to listen for jobs?

[jschwe]

does the docker monitor unregister runners that fail to listen for jobs?

It currently does not! I guess we could make the docker entry point a script which can handle startup failure and unregistering the runner, and as a fallback add something similar to the what nix selfhosted monitor does.

[jschwe]

gh api   -H "Accept: application/vnd.github+json"  \
   -H "X-GitHub-Api-Version: 2022-11-28"   \
/orgs/servo/actions/runners?per_page=100 \
 --jq '.runners[]|select(.status=="offline").id' \
| xargs -I% sh -c \
'{ echo "Removing runner %"; gh api   --method DELETE   -H "Accept: application/vnd.github+json"   -H "X-GitHub-Api-Version: 2022-11-28"   /orgs/servo/actions/runners/%; }'

FYI, this is the script I'm using to clean-up the offline runners. Fetching runners is capped at 100 at once.