Skip to content

enhance http_wrapper to be a better http citizen #1037

Description

@willmurphyscode

There have been a couple of recent instances of automated traffic interception blocking Vunnel from downloading data.

When asking to be allowed to programmatically read the data we need, we want to be able to say that we respect the servers we get that data from.

Concretely:

  • always set a User-Agent header so we identify ourselves (some providers do this today)
  • respect 429s - don't retry until Retry-After or a reasonable default
  • respect Retry-After on 503s
  • respect Retry-After on 403s (I recently learned GitHub will sometimes do this)

Please comment if I missed something. Thanks!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    • Status
      Ready

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions