Merge pull request #75 from blacklanternsecurity/redirect-cookies

apply cookies from redirect hops to the hops that follow

Author: liquidsec

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## Unreleased
+
+- Cookies set on a redirect hop are applied to the hops that follow it, the way a browser does. What a chain collects lives for that one request, so nothing carries between requests
+- What a chain will hold is capped (4096 bytes per cookie, 50 cookies, 8KB total), so a response that sets hundreds of large cookies can't grow every later hop's `Cookie` header without bound
+- A `Domain` attribute is checked against the Public Suffix List, so `Domain=com`, `Domain=co.uk` or `Domain=github.io` can't be used to carry a cookie onto an unrelated host, and a cookie may only widen within one registrable domain
+- A cookie set in the request's own `Cookie` header always wins: every hop sends it, and a `Set-Cookie` naming it is ignored rather than replacing it, deleting it, or being sent alongside it
+- `redirect_cookies=False` (or `--no-redirect-cookies`) reverts to the previous behavior
+
 ## 0.10.0
 
 - Fix responses being discarded when `Content-Encoding` is declared on an empty body (bodyless redirects, `HEAD`, `304`)

Cargo.lock

@@ -46,6 +46,13 @@ http = "1"
 flate2 = "1"
 brotli = "8"
 
+# Public Suffix List — decides where a cookie's `Domain` stops being allowed
+# to widen. Without it `Domain=com` or `Domain=github.io` is accepted and the
+# cookie crosses to an unrelated host. The list is compiled in, so this costs
+# no network and no runtime lookup; dependabot keeps the version, and with it
+# the list, current.
+psl = "2.1.224"
+
 # Hashing — response fingerprinting (matches BBOT's hash format)
 murmur3 = "0.5"
 

Cargo.toml

@@ -109,6 +109,7 @@ Output is JSON (one object per response), including status, headers, redirect ch
 | `--rate-limit` | Requests per second (batch mode) | unlimited |
 | `-L, --follow-redirects` | Follow redirects | off |
 | `--max-redirects` | Max redirect hops | `10` |
+| `--no-redirect-cookies` | Don't apply cookies a redirect sets to later hops | off |
 | `-t, --timeout` | Request timeout (seconds) | `10` |
 | `--max-body-size` | Max response body (bytes) | 10 MB |
 | `--verify` | Enable TLS cert validation | off |
@@ -232,6 +233,7 @@ All parameters except `url` are optional:
 | `timeout` | `int` | Request timeout in seconds |
 | `follow_redirects` | `bool` | Follow redirects |
 | `max_redirects` | `int` | Max redirect hops |
+| `redirect_cookies` | `bool` | Apply cookies a redirect sets to later hops (default `True`) |
 | `verify_certs` | `bool` | Enable TLS cert validation (default `False`) |
 | `proxy` | `str` | HTTP/SOCKS proxy URL |
 | `no_proxy` | `list[str]` | Hosts that bypass the proxy |
@@ -319,6 +321,40 @@ response = await client.request(
 )
 ```
 
+### Cookies across redirects
+
+When `follow_redirects` is on, a cookie set by one hop is sent on the hops that follow it, the same way a browser does. That's what makes a login or bot-check page work: it hands you a cookie along with the redirect, and the cookie has to be on the next request to count for anything. Without this you'd loop or land back on the same page.
+
+This is **not a session**, and there's no cookie storage behind it. What a chain collects is created when the request starts and dropped when it returns, so nothing carries into the next request and no two requests can see each other's cookies. A batch of 500 URLs runs 500 independent chains, which keeps every result reproducible on its own.
+
+What a chain will hold is capped, since a response can set as many cookies as it likes and every later hop would carry all of them: 4096 bytes per cookie and 50 cookies, which is what RFC 6265 asks a client to support and roughly what browsers allow, and 8KB across the whole chain, which is about where servers stop accepting a header line. Past those, later cookies are dropped and the ones already held are kept, with a line in the debug log saying what went.
+
+Which cookie goes to which hop follows the usual rules (RFC 6265): a cookie with no `Domain` goes back only to the exact host that set it, a `Domain` that doesn't cover the host that sent it is thrown out, `Path` has to match, and `Secure` cookies never go over plain HTTP.
+
+`Domain` also gets checked against the Public Suffix List, because the rules above don't cover it on their own: `Domain=com` does cover the host that set it, so it passes every other check, and then covers every other `.com` the chain visits. A cookie may only widen within one registrable domain, so `auth.example.com` can hand one to `app.example.com`, while `Domain=com`, `Domain=co.uk`, `Domain=github.io` and `Domain=s3.amazonaws.com` are refused. That is what stops a redirect walking a cookie the chain picked up onto an unrelated host. Headers you supply yourself are a different matter: those are sent as given on every hop, including after a redirect to another host, because a header you set is a header we send.
+
+Pass `redirect_cookies=False` (or `--no-redirect-cookies` on the CLI) to turn it off and send only your own headers on every hop.
+
+```python
+# On by default.
+r = await client.request("https://example.com/login", method="POST",
+                         body="user=x&pass=y", follow_redirects=True)
+
+# Off: every hop gets only the headers you supplied.
+r = await client.request("https://example.com/login", follow_redirects=True,
+                         redirect_cookies=False)
+```
+
+**A cookie you set yourself always wins.** If your request carries `Cookie: session=mine`, every hop of that chain sends `session=mine`. A `Set-Cookie` naming a cookie you set is ignored: it can't replace your value, an expiry on it can't delete your value, and the two never go out together as a duplicate pair. Sites reset cookies mid-redirect routinely, and servers disagree about which value to read when a name appears twice (some take the first, some the last), so the only rule that behaves the same everywhere is that what you wrote is what lands on the wire. Cookies the chain sets under *other* names are merged into your `Cookie` header, yours first.
+
+```python
+# Every hop sends session=mine, whatever the site tries to set.
+r = await client.request("https://example.com/start", follow_redirects=True,
+                         headers=[("Cookie", "session=mine")])
+```
+
+Run with `-v` to see it happen: the debug log records both the cookies each hop is given and any `Set-Cookie` that lost to one of yours.
+
 ### Proxy exclusions (`no_proxy`)
 
 `proxy` routes a request through an HTTP or SOCKS5 proxy; `no_proxy` is a per-request list of hosts that bypass it and connect directly — the `NO_PROXY` equivalent. It's accepted by `request()`, `download()`, `raw_connect()`, and `BatchConfig`, and as the repeatable `--no-proxy` CLI flag.

README.md

@@ -769,11 +769,12 @@ async fn dispatch_request(
     uri: &http::Uri,
     config: &RequestConfig,
     log: &DebugLog,
+    redirect_cookies: Option<&str>,
 ) -> Result<SingleResponse, ClientError> {
     // The pooled high-level client populates Host / :authority from the URI
     // itself, so we don't add a Host header here. Adding it would cause
     // duplicate :authority + host in the HTTP/2 HPACK block.
-    let request = build_request(uri, config, false, false)?;
+    let request = build_request(uri, config, false, false, redirect_cookies)?;
     let v = config.verbosity;
 
     debug_record(log, v, 1, "   Request headers:");
@@ -1124,10 +1125,10 @@ async fn dispatch_direct(
     // :authority and :path, and a Host header of our own would duplicate
     // :authority in the HPACK block.
     let request = if h2 {
-        build_request(&request_uri, config, false, false)?
+        build_request(&request_uri, config, false, false, None)?
     } else {
         let use_origin_form = config.request_target.is_none();
-        build_request(&request_uri, config, use_origin_form, true)?
+        build_request(&request_uri, config, use_origin_form, true, None)?
     };
 
     debug_record(log, v, 1, "   Request headers:");
@@ -1191,6 +1192,7 @@ async fn dispatch_forward_proxy(
     target_uri: &http::Uri,
     config: &RequestConfig,
     log: &DebugLog,
+    redirect_cookies: Option<&str>,
 ) -> Result<SingleResponse, ClientError> {
     let proxy_uri: http::Uri = proxy_url.parse().map_err(|e: http::uri::InvalidUri| {
         ClientError::invalid_url(format!("invalid proxy URL: {}", e))
@@ -1228,7 +1230,7 @@ async fn dispatch_forward_proxy(
 
     // Build request with absolute-form URI (SendRequest does NOT normalize it).
     // The low-level http1 sender doesn't auto-populate Host, so add it manually.
-    let request = build_request(target_uri, config, false, true)?;
+    let request = build_request(target_uri, config, false, true, redirect_cookies)?;
     let v = config.verbosity;
 
     debug_record(log, v, 1, "   Request headers:");
@@ -1255,6 +1257,11 @@ fn build_request(
     config: &RequestConfig,
     origin_form: bool,
     manual_host_header: bool,
+    // Cookies picked up from earlier hops of this redirect chain, already
+    // filtered down to the ones that apply to `uri` and to names the caller
+    // didn't set themselves. Merged into the caller's own `Cookie` header
+    // when there is one, so we never send two.
+    redirect_cookies: Option<&str>,
 ) -> Result<hyper::Request<FullBody>, ClientError> {
     // For direct connections (dispatch_direct), use origin-form (path + query only)
     // in the request-line per RFC 7230 §5.3.1. For pooled/client connections,
@@ -1312,11 +1319,35 @@ fn build_request(
         builder = builder.header("Accept-Encoding", "gzip, deflate, br");
     }
 
+    // Emit the caller's headers, folding any redirect-chain cookies into the
+    // first `Cookie` header they supplied. If they supplied none, the
+    // chain's cookies go out as their own header after the caller's.
+    //
+    // The chain's list can't contain a name the caller set, because
+    // `ChainCookies` refuses to store one, so this concatenation never
+    // produces the same
+    // name twice. That matters: duplicate names are read inconsistently
+    // across servers (some take the first, some the last), so a request
+    // carrying both would behave differently depending on the target.
+    let mut merged_cookies = false;
     if let Some(ref custom_headers) = config.headers {
         for (name, value) in custom_headers {
+            if let Some(extra) = redirect_cookies
+                && !merged_cookies
+                && name.eq_ignore_ascii_case("cookie")
+            {
+                merged_cookies = true;
+                builder = builder.header(name.as_str(), format!("{}; {}", value, extra));
+                continue;
+            }
             builder = builder.header(name.as_str(), value.as_str());
         }
     }
+    if let Some(extra) = redirect_cookies
+        && !merged_cookies
+    {
+        builder = builder.header("Cookie", extra);
+    }
 
     let body_bytes = config.body.clone().unwrap_or_default();
     builder
@@ -1666,6 +1697,21 @@ impl HyperClient {
         let mut redirect_chain: Vec<RedirectHop> = Vec::new();
         let mut hops = 0u32;
 
+        // Cookies picked up as we walk this chain. Created here and dropped
+        // when the request returns, so two concurrent requests can never see
+        // each other's cookies and a request's result depends only on its own
+        // inputs. `hop_cookies` is the `Cookie` header for the hop we're about
+        // to make, recomputed per hop because each one may be a different host.
+        // The caller's own cookies are recorded up front so the chain can
+        // never touch them: whatever they put in a `Cookie` header is what
+        // every hop sends.
+        let mut chain_cookies = config.should_forward_redirect_cookies().then(|| {
+            crate::cookies::ChainCookies::with_caller_cookies(crate::cookies::caller_cookie_names(
+                config.headers.as_deref().unwrap_or(&[]),
+            ))
+        });
+        let mut hop_cookies: Option<String> = None;
+
         loop {
             // Decide the connection mode for the *current* target host on every
             // hop, not just the first. A redirect can send the request to a
@@ -1695,9 +1741,16 @@ impl HyperClient {
             };
 
             let resp = if let Some(ref proxy_url) = proxy_url_for_fwd {
-                dispatch_forward_proxy(proxy_url, &uri, config, log).await?
+                dispatch_forward_proxy(proxy_url, &uri, config, log, hop_cookies.as_deref()).await?
             } else {
-                dispatch_request(&cached.as_ref().unwrap().inner, &uri, config, log).await?
+                dispatch_request(
+                    &cached.as_ref().unwrap().inner,
+                    &uri,
+                    config,
+                    log,
+                    hop_cookies.as_deref(),
+                )
+                .await?
             };
             let hop_ms = start.elapsed().as_millis();
             debug_record(log, v, 1, &format!("<- {} ({}ms)", resp.status, hop_ms));
@@ -1741,6 +1794,42 @@ impl HyperClient {
                     peer_ip: hop_peer_ip,
                 });
 
+                // Take this hop's `Set-Cookie` headers, then work out which of
+                // everything collected so far applies to where we're going.
+                // The domain / path / Secure rules are what stop a cookie from
+                // following a redirect onto a host it doesn't belong to.
+                if let Some(chain) = chain_cookies.as_mut() {
+                    let rejected = chain.store(&resp.headers, &uri);
+                    if !rejected.caller_owned.is_empty() {
+                        debug_record(
+                            log,
+                            v,
+                            1,
+                            &format!(
+                                "   Kept the caller's own cookie(s) over a Set-Cookie for: {}",
+                                rejected.caller_owned.join(", ")
+                            ),
+                        );
+                    }
+                    // Say so rather than quietly holding fewer cookies than
+                    // the chain set: a cap nobody can see reads as coverage.
+                    if !rejected.over_limit.is_empty() {
+                        debug_record(
+                            log,
+                            v,
+                            1,
+                            &format!(
+                                "   Cookie limit reached, dropped: {}",
+                                rejected.over_limit.join(", ")
+                            ),
+                        );
+                    }
+                    hop_cookies = chain.header_for(&next_uri);
+                    if let Some(ref c) = hop_cookies {
+                        debug_record(log, v, 1, &format!("   Sending cookies: {}", c));
+                    }
+                }
+
                 uri = next_uri;
                 continue;
             }
@@ -2140,7 +2229,7 @@ mod tests {
     fn test_build_request_auto_host_from_uri() {
         let uri: http::Uri = "http://example.com:8080/path".parse().unwrap();
         let config = RequestConfig::new("http://example.com:8080/path".to_string());
-        let req = build_request(&uri, &config, true, true).unwrap();
+        let req = build_request(&uri, &config, true, true, None).unwrap();
         assert_eq!(req.headers().get("host").unwrap(), "example.com:8080");
     }
 
@@ -2152,7 +2241,7 @@ mod tests {
         // HTTP/2 HPACK block, which some origin servers reject.
         let uri: http::Uri = "http://example.com:8080/path".parse().unwrap();
         let config = RequestConfig::new("http://example.com:8080/path".to_string());
-        let req = build_request(&uri, &config, false, false).unwrap();
+        let req = build_request(&uri, &config, false, false, None).unwrap();
         assert!(req.headers().get("host").is_none());
     }
 
@@ -2161,7 +2250,7 @@ mod tests {
         let uri: http::Uri = "http://example.com:8080/path".parse().unwrap();
         let mut config = RequestConfig::new("http://example.com:8080/path".to_string());
         config.headers = Some(vec![("Host".to_string(), "custom.host".to_string())]);
-        let req = build_request(&uri, &config, true, true).unwrap();
+        let req = build_request(&uri, &config, true, true, None).unwrap();
         // Should only have the custom Host, not auto-derived
         let hosts: Vec<_> = req.headers().get_all("host").iter().collect();
         assert_eq!(hosts.len(), 1);
@@ -2176,7 +2265,7 @@ mod tests {
         let uri: http::Uri = "http://example.com:8080/path".parse().unwrap();
         let mut config = RequestConfig::new("http://example.com:8080/path".to_string());
         config.headers = Some(vec![("Host".to_string(), "custom.host".to_string())]);
-        let req = build_request(&uri, &config, false, false).unwrap();
+        let req = build_request(&uri, &config, false, false, None).unwrap();
         let hosts: Vec<_> = req.headers().get_all("host").iter().collect();
         assert_eq!(hosts.len(), 1);
         assert_eq!(hosts[0], "custom.host");
@@ -2190,7 +2279,7 @@ mod tests {
             ("Host".to_string(), "first.host".to_string()),
             ("Host".to_string(), "second.host".to_string()),
         ]);
-        let req = build_request(&uri, &config, true, true).unwrap();
+        let req = build_request(&uri, &config, true, true, None).unwrap();
         let hosts: Vec<_> = req.headers().get_all("host").iter().collect();
         assert_eq!(hosts.len(), 2);
         assert_eq!(hosts[0], "first.host");
@@ -2201,15 +2290,15 @@ mod tests {
     fn test_build_request_origin_form_strips_authority() {
         let uri: http::Uri = "http://example.com:8080/path?q=1".parse().unwrap();
         let config = RequestConfig::new("http://example.com:8080/path?q=1".to_string());
-        let req = build_request(&uri, &config, true, true).unwrap();
+        let req = build_request(&uri, &config, true, true, None).unwrap();
         assert_eq!(req.uri(), "/path?q=1");
     }
 
     #[test]
     fn test_build_request_absolute_form_preserves_uri() {
         let uri: http::Uri = "http://example.com:8080/path?q=1".parse().unwrap();
         let config = RequestConfig::new("http://example.com:8080/path?q=1".to_string());
-        let req = build_request(&uri, &config, false, false).unwrap();
+        let req = build_request(&uri, &config, false, false, None).unwrap();
         assert_eq!(req.uri().to_string(), "http://example.com:8080/path?q=1");
     }
 
@@ -2219,7 +2308,7 @@ mod tests {
         // Simulate: origin_form=false (as dispatch_direct does when request_target is Some)
         let uri: http::Uri = "http://evil.com/admin".parse().unwrap();
         let config = RequestConfig::new("http://example.com/".to_string());
-        let req = build_request(&uri, &config, false, true).unwrap();
+        let req = build_request(&uri, &config, false, true, None).unwrap();
         assert_eq!(req.uri().to_string(), "http://evil.com/admin");
     }