Skip to content

feat: support timeouts in the send_request context helper (#2138) - #2164

Draft
aryansk wants to merge 1 commit into
apify:masterfrom
aryansk:fix/2138-send-request-timeout
Draft

feat: support timeouts in the send_request context helper (#2138)#2164
aryansk wants to merge 1 commit into
apify:masterfrom
aryansk:fix/2138-send-request-timeout

Conversation

@aryansk

@aryansk aryansk commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Closes #2138

Summary

SendRequestFunction (used via context.send_request() in request handlers) only accepted url, method, payload and headers, so a handler could not bound how long an extra HTTP call may take. All four HTTP client implementations (HttpxHttpClient, CurlImpersonateHttpClient, ImpitHttpClient, PlaywrightHttpClient) already support a per-request timeout: timedelta | None — this PR just exposes it through the public contract.

Changes

  • src/crawlee/_types.py: add timeout: timedelta | None = None to the SendRequestFunction protocol + docstring.
  • src/crawlee/crawlers/_basic/_basic_crawler.py: thread timeout through the _prepare_send_request_function closure into HttpClient.send_request.
  • tests/unit/crawlers/_basic/test_basic_crawler.py: regression test — context.send_request('/slow?delay=2', timeout=timedelta(milliseconds=100)) raises asyncio.TimeoutError.

Validation

  • pytest tests/unit/crawlers/_basic → 112 passed, 5 skipped (3.10/3.11-only features)
  • ruff check + ruff format --check clean on all touched files

Note: no new dependency; timeout uses the existing timedelta convention across the Python HTTP clients (the JS implementation names it timeoutMillis, but the Python clients are timedelta-based).

Expose the HTTP clients' existing per-request `timeout` support through
`SendRequestFunction` so request handlers can bound how long an extra
HTTP call may take. Adds a regression test against the slow test-server
endpoint.
@Battleplus

Copy link
Copy Markdown

Hi! I've opened a PR that supersedes this one for #2138: #2174.

It implements the same feature (exposing timeout through SendRequestFunction / context.send_request()), and additionally fixes a bug your draft would hit: PlaywrightHttpClient.send_request passes timeout.total_seconds() to Playwright's APIRequestContext.fetch, which expects milliseconds — so any timeout passed on a Playwright crawler would be ~1000x too short. The new PR also adds a regression test for that conversion plus the real end-to-end timeout test.

Feel free to close this draft in favor of #2174, or let me know if you'd prefer to take the changes over. Thanks for the original implementation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support timeouts in send_request

4 participants