Repository navigation
fix: Stop exhausting GitHub API rate limits on busy orgs - #117
Merged
Merged
Conversation
Every completed workflow_job webhook made up to three GitHub API calls with the org's installation token: fetch .sentry/sentry_config.ini, fetch the run, and fetch the workflow. Skipped jobs also fetched the config, only to be dropped afterwards. Busy orgs (getsentry, cyeragit, MetaMask) ran out of their hourly installation budget, and GitHub answered the config fetch with 403 until the window reset, so their remaining jobs were dropped and reported as errors. - Cache each org's DSN for 10 minutes - Cache workflow paths, which every job of a run shares - Bail on skipped jobs before making any API call This brings a traced job down to about one API call, and skipped jobs to none. Fixes SENTRY-GITHUB-ACTIONS-APP-5W Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Addresses review findings on the new caching: - Read the workflow path from the run instead of fetching and caching the workflow. A @cached wrapper holds its cache as a closure variable, so Sentry's captured frame locals sent other orgs' private repo and workflow URLs with any error. This also removes a call per job and the 422s from fetching required workflows' `workflow_url`. - Keep the DSN cache in module globals for the same reason, and key it on (installation id, org), so a DSN is only served to an installation whose token could read that org's config. - Run the skipped-job test in Github App mode, where a token would be minted, and give each org its own DSN in the cache test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f04f751. Configure here.
armenzg
approved these changes
Oct 9, 2026
…aths Addresses review findings: - cli.py still called fetch_dsn_for_github_org without the new installation_id argument, so every CLI run raised a TypeError before ingesting the job. - A run's path can name the repo the workflow lives in and end in a ref (e.g. `org/repo/.github/workflows/foo.yml@refs/heads/main`), as the REST API documents for required workflows. Splitting on "/" then tagged the job with `main` or `foo.yml@main` instead of `foo.yml`. Strip the ref before taking the file name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
billyvg
added a commit
that referenced
this pull request
Oct 9, 2026
Resolves the conflict with #117 in src/sentry_config.py: the request timeout and owner check now live in _fetch_dsn, behind #117's DSN cache. - #117's tests used `other_org`, which isn't a valid GitHub login, so the owner check rejected it. They now use `other-org`. - test_every_request_has_a_timeout no longer registers the workflow fetch, which #117 removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes SENTRY-GITHUB-ACTIONS-APP-5W:
HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/getsentry/.sentry/contents/sentry_config.ini, 16.4M events. It should also stop most of SENTRY-GITHUB-ACTIONS-APP-V: 422s from fetching required workflows.Root cause
The 403 is GitHub's hourly rate limit on the installation, not a missing permission:
getsentryconfig fetches returned 200 about 101k times and 403 about 66k times.cyeragit(135k / 126k) andMetaMask(90k / 7k) show the same mix.getsentry's 66,207 403s.Every completed
workflow_jobwebhook made up to three API calls with the installation token:.sentry/sentry_config.ini.The config fetch also ran for skipped jobs, which are then dropped.
getsentryalone sends about 167k webhooks a day, which is roughly 18k calls/hr if none were rate limited. It gets through about 10.8k calls/hr before GitHub starts refusing. For the rest of each hour, CI jobs are dropped, and each drop is reported here as an error.The issue's first-seen date matches #71 (2025-09-08), which added
logger.exceptionto the fetch. The 403s were probably happening before then and only started reaching Sentry with that change.Changes
(installation id, org).@cacheddecorator. The decorator's wrapper holds the cache as a closure variable, so it ends up in the frame variables Sentry sends with any error. That would put other orgs' cached values into this app's error events.runs["path"]) instead of fetching the workflow. The fetch also fails with 422 for required workflows, whoseworkflow_urlpoints at/actions/required_workflows/{id};pathis correct for those runs too.org/repo/.github/workflows/foo.yml@refs/heads/main), so the ref is stripped before taking the file name.jobArun fixture gains thepathfield that current API responses include.send_tracealready ignored them.cachetools==5.3.2torequirements.txt. The production image already ships it throughrequirements.frozen.txtas a transitive dependency ofgoogle-auth. Newergoogle-authreleases dropped it, so the dev and tox environments didn't have it.sentry_config.inican take up to 10 minutes to apply.A traced job now costs about 1 API call instead of 3, and a skipped job costs none. For
getsentrythat's an estimated ~5.5k calls/hr, about half of what it already gets through before being limited. Token revocation (DELETE /installation/token) returned 204 on all ~587k calls, including during the limited windows, so it isn't blocked by the budget and is unchanged.Testing
pytest: 27 passed, 1 skipped (already skipped before this change). New tests check that:workflowtag is the file name whether or not the path carries a repo and ref.@cacheddecorator, the workflow fetch, an org-only cache key, a skip check after minting the token, or splitting the path without stripping its ref each makes its test fail.cli.pywith GitHub and Sentry mocked out: it now passes the installation id, where the old call raisedTypeError: missing a required argument: 'installation_id'.pre-commit runpasses on all changed files.🤖 Generated with Claude Code