Test suite failing with race conditions

[ssokolow]

While putting together #1513, I discovered that cargo test would report random tests failing.

The common factor seems to be that they're functional tests and the errors I've observed are either timeout waiting for port or Address already in use or ConnectionRefused, which very much feels like the test are racing with the server they depend on and with each other.

Here are some examples, each from a different run:

failures:

---- auth_file_accepts::case_1 stdout ----
[miniserve stdout] miniserve v0.31.0
[miniserve stderr] Sun, 03 Aug 2025 04:06:19 +0000 [ERROR] Failed to bind server to [::]:33675
[miniserve stderr] Sun, 03 Aug 2025 04:06:19 +0000 [ERROR] caused by: Address already in use (os error 98)
[miniserve stderr] Error: Failed to bind server to [::]:33675
[miniserve stderr] caused by: Address already in use (os error 98)

thread 'auth_file_accepts::case_1' panicked at tests/fixtures/mod.rs:179:13:
timeout waiting for port 33675
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    auth_file_accepts::case_1

test result: FAILED. 5 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.03s

---- bind_ipv4_ipv6::case_2 stdout ----
[miniserve stdout] miniserve v0.31.0
[miniserve stdout] Sun, 03 Aug 2025 04:09:32 +0000 [INFO] starting 12 workers
[miniserve stdout] Bound to [::]:43859
[miniserve stdout] Serving path /tmp/.tmpfnYGhM
[miniserve stdout] Available at (non-exhaustive list):
[miniserve stdout]     http://[::1]:43859
[miniserve stdout] 
[miniserve stdout] Sun, 03 Aug 2025 04:09:32 +0000 [INFO] Actix runtime found; starting in Actix runtime
[miniserve stdout] Sun, 03 Aug 2025 04:09:32 +0000 [INFO] starting service: "actix-web-service-[::]:43859", workers: 12, listening on: [::]:43859

thread 'bind_ipv4_ipv6::case_2' panicked at tests/fixtures/mod.rs:179:13:
timeout waiting for port 43859
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    bind_ipv4_ipv6::case_2

test result: FAILED. 12 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.04s

...and with my PR (#1513) which added a miniscule delay to program startup imposed by performing two network requests (for IPv4 and IPv6) to a "What is my public IP address?" service, validate_printed_urls started failing with this output:

failures:

---- validate_printed_urls::case_2 stdout ----
Error: reqwest::Error { kind: Request, url: "http://[::1]:35471/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:35471, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_5 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:38693/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:38693, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_3 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:35529/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:35529, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_4 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:33055/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:33055, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_1 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:41519/", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:41519, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_6 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:43281/7f86b7", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:43281, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- validate_printed_urls::case_7 stdout ----
Error: reqwest::Error { kind: Request, url: "http://127.0.0.1:41225/prefix", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", 127.0.0.1:41225, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })) }

---- bind_ipv4_ipv6::case_2 stdout ----
[miniserve stdout] miniserve v0.31.0
[miniserve stdout] Sun, 03 Aug 2025 03:53:46 +0000 [INFO] starting 12 workers
[miniserve stdout] Bound to [::]:33347
[miniserve stdout] Serving path /tmp/.tmph1gmL1
[miniserve stdout] Available at (non-exhaustive list):
[miniserve stdout]     http://[::1]:33347
[miniserve stdout] 
[miniserve stdout] Depending on firewall/NAT settings, may be publicly accessible at:
[miniserve stdout]     http://[HAND-REDACTED BECAUSE MY ACTUAL PUBLIC IP]:33347
[miniserve stdout] 
[miniserve stdout] Sun, 03 Aug 2025 03:53:46 +0000 [INFO] Actix runtime found; starting in Actix runtime
[miniserve stdout] Sun, 03 Aug 2025 03:53:46 +0000 [INFO] starting service: "actix-web-service-[::]:33347", workers: 12, listening on: [::]:33347

thread 'bind_ipv4_ipv6::case_2' panicked at tests/fixtures/mod.rs:179:13:
timeout waiting for port 33347
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    bind_ipv4_ipv6::case_2
    validate_printed_urls::case_1
    validate_printed_urls::case_2
    validate_printed_urls::case_3
    validate_printed_urls::case_4
    validate_printed_urls::case_5
    validate_printed_urls::case_6
    validate_printed_urls::case_7

test result: FAILED. 5 passed; 8 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.03s

[svenstaro]

I'm painfully aware of this. :/

I haven't really had time to look at this and would be super grateful if you could try to help sort this out!

[ssokolow]

I'll see if I can find time to familiarize myself with the codebase enough to figure it out.

EDIT: Having not yet looked, my intuition says "race conditions are caused by a lack of synchronization and the observed bugs point to missing synchronization in two places: the collision over port numbers (Address already in use) and the failure to reliably wait for a subprocess to initialize (ConnectionRefused)"... I'll need to check the source to identify what would cause timeout waiting for port 33347 given that's not a standard error message.

[ssokolow]

OK, while I haven't confirmed (by implementing a proposed fix and seeing if it solves it), my first impression is that your port collisions are the sockets version of the TOCTTOU bug that makes faccess a much more niche tool than novices think.

Basically, port_check is a fundamentally broken crate because this code from it...

pub fn free_local_ipv4_port() -> Option<u16> {
    let socket = SocketAddrV4::new(Ipv4Addr::LOCALHOST, 0);
    TcpListener::bind(socket)
        .and_then(|listener| listener.local_addr())
        .map(|addr| addr.port())
        .ok()
}

...is saying "I'm done with this port. You can reuse it." when it drops the socket and then it's the OS's decision how random to be about which port it hands out next time a thread asks for an unused one.

What you instead want to do is do that "Ask for port 0 and then read out what the OS gave you" part inside your init code for your test server, so you don't release the port until you're done with it.

(Basically the same mistake and same fix as when you query a file's existence or permissions before opening it instead of blindly trying to open it and handling the error.)

EDIT: Given that you're using subprocesses, the proper solution to this and probably also to that other race condition which has the appearance of a wait-for-ready race is to give miniserve a command-line option equivalent to Xephyr's -displayfd <fd>, where the miniserve subprocess will take responsibility for picking a random port and then write it to the given file descriptor when it's ready to accept connections. That way, it can use that "Ask for port 0 and the OS will give you a random unused one" trick properly.

EDIT: It's getting late, so I can't make a PR tonight, but we'll see if I can find time for it tomorrow.

[ssokolow]

As for validate_printed_urls, I don't know enough about Actix and Tokio to know exactly what happens when during the initialization, but it seems a very reasonable conclusion that the errors that were made more prevalent by my PR stem from "If I println! after I create_tcp_listener, then waiting on the output of println! will be an adequate synchronization barrier to assume the TCP port is now open" being an incorrect assumption.

Something else for me to look into when I'm not winding down for bed.

I wonder if there needs to be some kind of "flush queued async executor operations" call akin to what needs to happen in certain kinds of GTK or Qt "call this X11-invoking function and then exit so we can be used in shell scripts" code.

[ssokolow]

OK, here's what I've thought of so far for addressing that collision problem:

To avoid the TOCTTOU bug, either the miniserve subprocess is going to need to take an already bound socket as an argument systemfd/listenfd-style or it's going to need to be responsible for asking the OS to pick the random port and report it back to the parent process so there's no point when the socket is released.

However, traditional approaches to either of those either flow through the FromRawFd trait and its unsafe from_raw_fd method (where the invariant that must be preserved is "calling process, which may not be the test suite, doesn't engage in shenanigans") or use UNIX domain sockets (referenced by file path instead of port number) to leverage best practices for avoiding temp filename collision. (Which Win32 doesn't have)

If there's a sufficiently maintained and non-niche crate to provide a portable wrapper around named pipes, then the test runner could create a named pipe, leveraging best practices for avoiding colliding tempfile names, pass it into miniserve via some kind of --random-port[=REPORT_FILE] argument, and have miniserve open the pipe and write 12345\n or whatever the port is to that file and flush it, giving the test runner a cross-process analogue to waiting on a tokio::oneshot.

However, nothing leaps out at me for named pipes, so I think the best solution so far would probably be to do the same thing but with random TCP sockets:

1. The test runner creates a TcpListener on port 0, asking the OS to reserve a random port for it. (We'll call this listener test_listener.)
2. The test runner spawns miniserve and passes the value of test_listener.local_addr().port() via some kind of --random-port[=READY_PORT] argument. (Because the test runner never releases its hold on the listening side of test_listener, a collision is avoided there and, because the test harness never gives the same port number to more than one miniserve, there's no confusion there.)
3. miniserve binds to port 0, once again asking the OS to pick an unused port for it. miniserve will not release the port binding for as long as it runs, so there's no room for TOCTTOU now.
4. Once it's reliably ready to receive connections, miniserve opens the port given as READY_PORT and writes its own listening port number followed by a newline, so that even the most maliciously insane packet fragmentation from hell couldn't confuse where the port number ends.
5. Upon receiving miniserve's listening port via test_listener, the test harness may now proceed, trusting that no race conditions are possible.

Architecturally, it's similar to including the sending side of a tokio::oneshot in a "work order" struct you're dropping into a channel so that your async executor can await some sync code running in a completely separate worker thread pool. (Which I do in the on-demand thumbnailing code for my "miniserve but an image gallery" experiment so I can put a limit on the number of threads and comply with Tokio's recommendations.)

The main thing I still need to look into is how to guarantee that sending a packet will occur after actix-web has finished making the port ready to listen for connections.

EDIT: I'm assuming this is the key line in the actix-web docs:

The Server must be awaited or polled in order to start running

...which means the question is how to inject something into the Tokio executor to run specifically after the Server has finished starting.

[ssokolow]

I've opened actix/actix-web#3714 to request documentation on how to hook a handler on "ready to receive connections" and, if they say "not currently possible", I'll go for a sleep-and-retry loop.

[svenstaro]

Would #935 fix this you think?

[svenstaro]

Also thanks for the in-depth research! I'm definitely very interested in getting this fixed properly. Though I admit, a retry loop might be very effective here. Still holding my breath for a more proper solution.

[ssokolow]

Would #935 fix this you think?

Not really. Having miniserve only listen on UNIX domain sockets during testing would mean you're no longer doing full functional testing on its ability to println! URLs for TCP sockets.

Also, I could be wrong, but I think miniserve currently supports versions of Windows 10 which predate the build when they finally added AF_UNIX support, so tests relying on AF_UNIX wouldn't be able to test compatibility for that.

(I'm aware of that because, among other things, I have the last Win7-compatible version of miniserve in my /srv/retro/Win7 Samba share)

Also, if you go the "miniserve takes an altready bound socket" route, the announcement specifically says that Windows's implementation doesn't support sending open file descriptors over UNIX domain sockets.

Also thanks for the in-depth research! I'm definitely very interested in getting this fixed properly. Though I admit, a retry loop might be very effective here. Still holding my breath for a more proper solution.

Pending a response/solution to actix/actix-web#3714, I think the least incorrect solution would be to do all the work I described to avoid port collisions, then use a retry loop to wait for the miniserve subprocess to become ready to service requests. (Which would also make it easy to switch to a fully correct solution later.)