What to do about the user agent?

[chrisburr]

This is prompted by #general > rattler-build is getting 403 but can curl source file @ 💬

I have no idea what the right way of handling this is. rattler-build should definitely be setting a application specific user agent but that wouldn't actually work in this case. This least bad working option I can come up with for this server is:

$ curl -sI -A "rattler-build/0.57.2 (compatible; conda; python-requests/2.32) reqwest/0.12" https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz
HTTP/2 200
date: Mon, 23 Feb 2026 05:11:18 GMT
content-type: application/x-gzip
content-length: 17056067
server: cloudflare

I have no idea if this is a one-off bad server config or if it could be more widespread. Maybe @matthewfeickert knows if it's practical to contact Argonne to fix thier cloudflare config?

rattler-build attempt

`recipe.yaml`

schema_version: 1

package:
  name: test-download
  version: "0.1.0"

source:
  url: https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz
  sha256: 8d24a38619eb9fd326c7bdf9fdb01466149a0ab7dc3ef1caffda728858bf5a85

build:
  script:
    - echo "Works" >> $PREFIX/test-download.txt

$ rattler-build build --recipe recipe.yaml

 ╭─ Finding outputs from recipe
 │ Found 1 variants
 │
 │ Build variant: test-download-0.1.0-hb0f4dca_0
 │
 │ ╭─────────────────┬────────────╮
 │ │ Variant         ┆ Version    │
 │ ╞═════════════════╪════════════╡
 │ │ target_platform ┆ "linux-64" │
 │ ╰─────────────────┴────────────╯
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: test-download-0.1.0-hb0f4dca_0
 │
 │ ╭─ Fetching source code
 │ │ Fetching source from url: https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz
 │ │
 │ ╰─────────────────── (took 1 second)
 │
 ╰─────────────────── (took 1 second)
Error:   × Failed to download source from url: HTTP status client error (403 Forbidden) for url (https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz)
  ╰─▶ HTTP status client error (403 Forbidden) for url (https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz)

curl with default User-Agent

$ curl -sI https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz | head -3
HTTP/2 200
date: Mon, 23 Feb 2026 05:01:11 GMT
content-type: application/x-gzip

User-Agent comparison

Each test runs: curl -sI -A "..." https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz

• ❌ `` (403)
• ❌ reqwest (403)
• ❌ rattler-build/0.57.2 (403)
• ❌ wget/1.21 (403)
• ❌ Mozilla/5.0 (403)
• ✅ curl/8.0.0
• ✅ python-requests/2.32.5
• ✅ CPython/3.12.12
• ✅ CPython/99.0.0
• ❌ conda/25.11.1 requests/2.32.5 (403)
• ✅ conda/25.11.1 requests/2.32.5 CPython/3.12.12
• ✅ conda/25.11.1 requests/2.32.5 CPython/3.12.12 Linux/5.14.0-611.26.1.el9_7.x86_64 rhel/9.7 glibc/2.34 solver/libmamba conda-libmamba-solver/25.11.0 libmambapy/2.5.0
• ✅ rattler-build/0.57.2 python-requests/2.32.5
• ✅ rattler-build/0.57.2 CPython/3.12.12

403 response is a Cloudflare managed challenge

$ curl -s -A "reqwest" https://web.cels.anl.gov/projects/sigma/downloads/moab/moab-5.6.0.tar.gz | grep -oE "(cdn-cgi/challenge-platform|cType: .managed.|Just a moment)"
Just a moment
cType: 'managed'
cdn-cgi/challenge-platform

[wolfv]

Thank you for the investigation ... I should read up a bit more about the cloudflare challenge, and what the best way to deal with it is.

An idea I had: we could retry and modify the user agent upon retry. Or do it like browsers do and send a number of unrelated "compatibility" strings in the user agent (e.g. send a curl & conda-build user agent with the rattler-build one). We'd only do that for external resources, and keep the real one for repodata requests etc. for counting.

[matthewfeickert]

I have no idea if this is a one-off bad server config or if it could be more widespread. Maybe @matthewfeickert knows if it's practical to contact Argonne to fix thier cloudflare config?

No idea, but it is nice to have other people see that there's an issue with the ANL URL beyond me. cc @vijaysm for if ANL might be able to address things on their end, given discussion in conda-forge/moab-feedstock#112 (comment).

[vijaysm]

Thanks for tagging me. I'll raise an issue at ANL and see if we need to modify server configuration. I could also upload to zenodo as a backup.

[matthewfeickert]

I'll raise an issue at ANL and see if we need to modify server configuration.

@vijaysm While user agent information should get added to rattler-build (this is outside my knowledge base), have you heard back from ANL on this yet and if they had thoughts on adjusting their cloudflare config?

I could also upload to zenodo as a backup.

While I'm a fan of Zenodo in general, the DOIs for the archives won't resolve to something with a version number in the URL, so that would require manual updates for each version bump.

[vijaysm]

I've raised the issue twice to ANL helpdesk and don't have any resolution yet.

If not Zenodo, do you prefer or recommend some other source for provenance?

[matthewfeickert]

I've raised the issue twice to ANL helpdesk and don't have any resolution yet.

Okay, thanks for the update and for asking.

If not Zenodo, do you prefer or recommend some other source for provenance?

I'd recommend uploading to Zenodo regardless, as it is nice to have long-term archival solutions. However, no, I don't have any ideas here if you want to use a currated subset of the full repository as the source distribution over the source repository itself. (For conda-forge if I'm not using a sdist from another package index like PyPI, I prefer to fall back to using the tarball generated from the version control hosting platform over using a source distribution from something less robust.)