Repository navigation
fix(storage): reuse keep-alive connections in webhook storage - #223
Merged
Merged
Conversation
Each webhook write called requests.post, which opens a new TCP connection and leaves it in TIME_WAIT. At a few hundred writes/s a single conserver pod ran out of ephemeral ports and every write failed with "Failed to establish a new connection", sending the vCons to the storage DLQ. Use one requests.Session per worker process, shared across the short-lived storage threads, with an HTTPAdapter pool sized by the new "pool-size" option (default 16). The session is rebuilt when the pid changes so a forked worker never shares its parent's sockets. Tests: a local HTTP/1.1 receiver sees five sequential writes arrive on one client port; a forked pid gets a fresh session. Both fail when the change is reverted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Webhook storage called
requests.postonce per write, opening a new TCP connection each time and leaving it in TIME_WAIT. On a busy deployment (a few hundred writes/s from one conserver pod) the pod ran out of ephemeral ports. Every write then failed withFailed to establish a new connectionand went to the storage DLQ: hundreds of thousands of vCons in under half an hour.What
requests.Sessionper worker process, shared by the short-lived storage threads (a newThreadPoolExecutorper vCon, so a thread-local session would never be reused). urllib3's pool is thread-safe, and these POSTs carry no cookies.HTTPAdapterpool sized by a newpool-sizeoption (default 16; size it toCONSERVER_VCON_CONCURRENCY).raise_for_status()DLQ propagation.Tests
common/tests/test_webhook_keepalive.py: a local HTTP/1.1 receiver sees five sequential writes arrive on one client port; a changed pid gets a fresh session.requests.postfails the first test; dropping the pid check fails the second.test_webhook_storage_failures.pyis updated to patch the session instead ofrequests.post. All 7 pass.🤖 Generated with Claude Code