Valgrind reveals more file descriptor leaks

[marcbrevoort-cyberhive]

Version

│ │ ├── tokio v1.44.2
│ │ │ └── tokio-macros v2.5.0 (proc-macro)
│ │ └── tokio v1.44.2 ()
│ ├── tokio v1.44.2 ()
│ │ ├── tokio v1.44.2 ()
│ │ ├── tokio-util v0.7.15
│ │ │ └── tokio v1.44.2 ()
│ │ │ ├── tokio v1.44.2 ()
│ │ │ ├── tokio-util v0.7.15 ()
│ │ ├── tokio v1.44.2 ()
│ │ ├── tokio-rustls v0.24.1
│ │ │ └── tokio v1.44.2 ()
│ ├── tokio v1.44.2 ()
│ │ └── tokio v1.44.2 ()
│ │ ├── tokio v1.44.2 ()
│ │ │ ├── tokio v1.44.2 ()
│ │ ├── tokio v1.44.2 ()
│ │ ├── tokio-rustls v0.26.2
│ │ │ └── tokio v1.44.2 ()
│ ├── tokio v1.44.2 ()
│ ├── tokio-rustls v0.26.2 ()
│ ├── tokio-util v0.7.15 ()
│ │ ├── tokio v1.44.2 ()
├── tokio v1.44.2 ()
│ │ ├── tokio v1.44.2 ()
│ │ └── tokio v1.44.2 ()
│ ├── tokio v1.44.2 ()
│ ├── tokio-util v0.7.15 (*)

Platform
Linux mrjb-Inspiron-7501 6.8.0-60-generic #63~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 22 19:00:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Description

Valgrind still reports some file descriptor leaks pointing to Tokio

I have a test test_fd_leaks that reports on file descriptors before/after a function call
(in this case a fairly elaborate end-to-end test).

This is defined as

#[test]
#[serial]
fn test_fd_leaks() {
    const MAX_EXPECTED_LEAKS: u16 = 0;
    let n_before = fd_count(); // function that counts file descriptors currently in use as reported by /proc/self/
    test_send_data_cpq_a2b_ok(); // code that uses tokio
    let n_after = fd_count();
    assert!(n_after - n_before <= MAX_EXPECTED_LEAKS);
}

fd_count is defined as

#[cfg(target_os = "linux")]
pub fn fd_count() -> usize {
    let Ok(fdlist) = std::fs::read_dir("/proc/self/fd") else {
        return 0;
    };
    return fdlist.filter_map(|fd| fd.ok()).count();
}

This is run under valgrind using the following script.

# build the test binary
RUST_LOG=trace RUST_BACKTRACE=1 cargo test --no-run -- --test-threads 1 --nocapture $FILTER 2>&1 |grep Executable|grep lib.rs > fd_target
# find its file name
cat fd_target|sed 's/.*[\(]//;s/[\)].*//' > test_binary
rm fd_target
export TEST_BINARY=$(cat test_binary)

# run test "test_fd_leaks" of the test binary under valgrind
valgrind --track-fds=yes $TEST_BINARY test_fd_leaks

This should not leave behind any file descriptors. However, 12 file descriptors were leaked.
I suspect they were left behind on tokio shutdown.

Valgrind output:

==3488881== Open AF_UNIX socket 16: <unknown>
==3488881==    at 0x1E102A5: fcntl (fcntl.c:25)
==3488881==    by 0x1DD44C0: std::fs::File::try_clone (owned.rs:115)
==3488881==    by 0x192D8D5: tokio::runtime::signal::Driver::new (mod.rs:72)
==3488881==    by 0x198AFE5: tokio::runtime::driver::create_signal_driver (driver.rs:251)
==3488881==    by 0x198AA5F: tokio::runtime::driver::create_io_stack (driver.rs:152)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881==    by 0xA2B2BE: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} (mod.rs:559)
==3488881== Open file descriptor 15:
==3488881==    at 0x1E102A5: fcntl (fcntl.c:25)
==3488881==    by 0x1DDB1A0: std::os::fd::owned::OwnedFd::try_clone (owned.rs:115)
==3488881==    by 0x19C1CF6: mio::sys::unix::selector::Selector::try_clone (epoll.rs:34)
==3488881==    by 0x19C2B95: mio::poll::Registry::try_clone (poll.rs:711)
==3488881==    by 0x1960953: tokio::runtime::io::driver::Driver::new (driver.rs:98)
==3488881==    by 0x198A763: tokio::runtime::driver::create_io_stack (driver.rs:150)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881== Open file descriptor 14:
==3488881==    at 0x1E10654: eventfd (eventfd.c:8)
==3488881==    by 0x19C30F2: mio::sys::unix::waker::Waker::new_unregistered (mod.rs:8)
==3488881==    by 0x19C2F7E: mio::sys::unix::waker::Waker::new (eventfd.rs:27)
==3488881==    by 0x19BFE9A: mio::waker::Waker::new (waker.rs:87)
==3488881==    by 0x196088C: tokio::runtime::io::driver::Driver::new (driver.rs:97)
==3488881==    by 0x198A763: tokio::runtime::driver::create_io_stack (driver.rs:150)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881== Open file descriptor 13:
==3488881==    at 0x1E10583: epoll_create1 (epoll.c:13)
==3488881==    by 0x19C1BD8: mio::sys::unix::selector::Selector::new (mod.rs:8)
==3488881==    by 0x19C2CE7: mio::poll::Poll::new (poll.rs:322)
==3488881==    by 0x19607A8: tokio::runtime::io::driver::Driver::new (driver.rs:95)
==3488881==    by 0x198A763: tokio::runtime::driver::create_io_stack (driver.rs:150)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881==    by 0xA2B2BE: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} (mod.rs:559)
==3488881== Open AF_UNIX socket 12: <unknown>
==3488881==    at 0x1E102A5: fcntl (fcntl.c:25)
==3488881==    by 0x1DD44C0: std::fs::File::try_clone (owned.rs:115)
==3488881==    by 0x192D8D5: tokio::runtime::signal::Driver::new (mod.rs:72)
==3488881==    by 0x198AFE5: tokio::runtime::driver::create_signal_driver (driver.rs:251)
==3488881==    by 0x198AA5F: tokio::runtime::driver::create_io_stack (driver.rs:152)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881==    by 0xA2B2BE: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} (mod.rs:559)
==3488881== Open AF_UNIX socket 11: <unknown>
==3488881==    at 0x1E15A4C: __syscall6 (syscall_arch.h:59)
==3488881==    by 0x1E15A4C: __alt_socketcall (syscall.h:65)
==3488881==    by 0x1E15A4C: socketpair (socketpair.c:8)
==3488881==    by 0x19C391B: mio::sys::unix::uds::pair (mod.rs:8)
==3488881==    by 0x19C2441: mio::sys::unix::uds::stream::pair (stream.rs:24)
==3488881==    by 0x19C3F57: mio::net::uds::stream::UnixStream::pair (stream.rs:56)
==3488881==    by 0x19526C8: <tokio::signal::unix::OsExtraData as tokio::signal::registry::Init>::init (unix.rs:60)
==3488881==    by 0x199CF9B: tokio::signal::registry::globals_init (registry.rs:157)
==3488881==    by 0x191FB80: core::ops::function::FnOnce::call_once (function.rs:250)
==3488881==    by 0x1967BC2: tokio::util::once_cell::OnceCell<T>::do_init::{{closure}} (once_cell.rs:48)
==3488881==    by 0x1963899: std::sync::poison::once::Once::call_once::{{closure}} (once.rs:155)
==3488881==    by 0x313893: std::sys::sync::once::futex::Once::call (futex.rs:176)
==3488881==    by 0x19636F3: std::sync::poison::once::Once::call_once (once.rs:155)
==3488881==    by 0x1967B8F: tokio::util::once_cell::OnceCell<T>::do_init (once_cell.rs:47)
==3488881== Open AF_UNIX socket 10: <unknown>
==3488881==    at 0x1E15A4C: __syscall6 (syscall_arch.h:59)
==3488881==    by 0x1E15A4C: __alt_socketcall (syscall.h:65)
==3488881==    by 0x1E15A4C: socketpair (socketpair.c:8)
==3488881==    by 0x19C391B: mio::sys::unix::uds::pair (mod.rs:8)
==3488881==    by 0x19C2441: mio::sys::unix::uds::stream::pair (stream.rs:24)
==3488881==    by 0x19C3F57: mio::net::uds::stream::UnixStream::pair (stream.rs:56)
==3488881==    by 0x19526C8: <tokio::signal::unix::OsExtraData as tokio::signal::registry::Init>::init (unix.rs:60)
==3488881==    by 0x199CF9B: tokio::signal::registry::globals_init (registry.rs:157)
==3488881==    by 0x191FB80: core::ops::function::FnOnce::call_once (function.rs:250)
==3488881==    by 0x1967BC2: tokio::util::once_cell::OnceCell<T>::do_init::{{closure}} (once_cell.rs:48)
==3488881==    by 0x1963899: std::sync::poison::once::Once::call_once::{{closure}} (once.rs:155)
==3488881==    by 0x313893: std::sys::sync::once::futex::Once::call (futex.rs:176)
==3488881==    by 0x19636F3: std::sync::poison::once::Once::call_once (once.rs:155)
==3488881==    by 0x1967B8F: tokio::util::once_cell::OnceCell<T>::do_init (once_cell.rs:47)
==3488881== Open file descriptor 9:
==3488881==    at 0x1E102A5: fcntl (fcntl.c:25)
==3488881==    by 0x1DDB1A0: std::os::fd::owned::OwnedFd::try_clone (owned.rs:115)
==3488881==    by 0x19C1CF6: mio::sys::unix::selector::Selector::try_clone (epoll.rs:34)
==3488881==    by 0x19C2B95: mio::poll::Registry::try_clone (poll.rs:711)
==3488881==    by 0x1960953: tokio::runtime::io::driver::Driver::new (driver.rs:98)
==3488881==    by 0x198A763: tokio::runtime::driver::create_io_stack (driver.rs:150)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881== Open file descriptor 7:
==3488881==    at 0x1E10583: epoll_create1 (epoll.c:13)
==3488881==    by 0x19C1BD8: mio::sys::unix::selector::Selector::new (mod.rs:8)
==3488881==    by 0x19C2CE7: mio::poll::Poll::new (poll.rs:322)
==3488881==    by 0x19607A8: tokio::runtime::io::driver::Driver::new (driver.rs:95)
==3488881==    by 0x198A763: tokio::runtime::driver::create_io_stack (driver.rs:150)
==3488881==    by 0x1989AC5: tokio::runtime::driver::Driver::new (driver.rs:48)
==3488881==    by 0x195131D: tokio::runtime::builder::Builder::build_current_thread_runtime_components (builder.rs:1489)
==3488881==    by 0x1951020: tokio::runtime::builder::Builder::build_current_thread_runtime (builder.rs:1457)
==3488881==    by 0x1950F4A: tokio::runtime::builder::Builder::build (builder.rs:911)
==3488881==    by 0xA641EF: reqwest::blocking::client::ClientHandle::new::{{closure}} (client.rs:1229)
==3488881==    by 0xA8E4C5: std::sys::backtrace::__rust_begin_short_backtrace (backtrace.rs:152)
==3488881==    by 0xA2B2BE: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} (mod.rs:559)

"Poison" in a few instances in the output above suggests a panic invalidating locks.

[Darksonn]

It sounds like you did not shut down your Tokio runtime. But even then be aware of #7367.

[marcbrevoort-cyberhive]

From #7367 -

"Signal handlers are process global, so the file descriptor is shared between all runtimes, and can therefore not be closed."

I do not understand - couldn't the file descriptor be wrapped in a reference counter so that the last runtime to be dropped could then (safely) close the shared file descriptor? Is the file descriptor represented by tokio::signal also used outside tokio?

Also forgive me for possibly barking up the tree, as the above dump implicates mostly the reqwest crate - Given how pervasive Tokio is for async implementations, it's highly unlikely every crate dev gets it right, so the more fool-proof it is (hardened against fools like me!) the better. The Principle of Least Astonishment may apply? Several of us seem to get caught out by this one.

[Darksonn]

In your case, you have a bunch of file descriptors that Tokio does close on shutdown. It looks like you are not actually shutting down your runtime.

[marcbrevoort-cyberhive]

I'm really sorry but I don't see how you conclude that - could you enlighten me?

I've got various places where I create a runtime, but to the best of my knowledge each of those is now paired up by an instance of

    if let Some(rt) = rt_handle.take() {
        debug!("tokio runtime stop (1)");
        rt.shutdown_background();
    }

or similar (the number at the end of the log line is an identifier so that the runtime startup/shutdown can be paired up).
(On waiting for shutdown completion: See below).

I did, in fact, come across an instance where that wasn't yet the case - yet after fixing those, the amount of leaked descriptors still shows unchanged at 12 at the end of the run.

So I put together a minimal example,

Cargo.toml

[package]
name = "fd_leaks"
version = "0.1.0"
edition = "2024"

[dependencies]
reqwest = { version = "0.12.19", default-features = false, features = [
    "brotli",
    "blocking",
    "deflate",
    "gzip",
    "json",
    "rustls-tls",
] }
reqwest-eventsource = { version = "0.6.0" }

# tokio = { version = "1.45.1", features = ["full"] }

src/main.rs

pub fn fd_count() -> usize {
    let Ok(fdlist) = std::fs::read_dir("/proc/self/fd") else {
        return 0;
    };
    return fdlist.filter_map(|fd| fd.ok()).count();
}

fn main() {
    println!("fd count before get={:?}", fd_count());
    let Ok(get) = reqwest::blocking::get("https://www.rust-lang.org") else {
        println!("fd count after failed get={:?}", fd_count());  // 4
        return;
    };
    let body = get.text().unwrap_or("".to_string());

    // 4 w/o tokio, 6 if tokio uncommented in Cargo.toml
    // -- even without actually being included in main.rs!
    // presumably these are the 2 FD referred to in #7367
    println!("fd count after successful get={:?}", fd_count());  

}

Note tokio isn't in use in src/main.rs (though may be used by reqwest)
When uncommenting tokio in Cargo.toml, we see fd count after increase by 2, presumably these are the 2 FD referred to in #7367.

So when I set Cargo to use only feature "rt-multi-thread", this shows 4 again instead of 6. So far so good.

then if to main I add

    println!("tokio runtime start::client_query_with_timeout");
    let runtime = match Runtime::new() {
        Ok(r) => r,
        Err(e) => {
            println!("Cannot create runtime, error was: {:?}", e);
            return;
        }
    };

    println!("fd count after create rt={:?}", fd_count());
    runtime.shutdown_background();
    std::thread::sleep(std::time::Duration::from_secs(1));
    println!("fd count wo rt shutdown={:?}", fd_count());

this will see the FD count rise to 7, then drop again to 4. Nice.

Without the sleep, it appears not to drop but that would be 'cause it is still getting around to it (I'm aware that when using sleep inside a runtime, it should be tokio's sleep rather than std::thread::sleep()).

So the minimal example gives some insights, but even when these insights are applied to the big code base (including calls to runtime.shutdown_background()) I'm still out of 12 file descriptors. In code, every runtime creation has a counterpart shutdown; I'm only including the features that are in use, and allow 5 seconds after runtime shutdown before counting file descriptors.

Colour me baffled, what are you seeing that I'm missing? Which runtime is apparently not being shut down, and how could I tell this for myself?

[Darksonn]

When you use reqwest::blocking::get, the reqwest crate will spawn a background thread containing a Tokio runtime:

https://github.com/seanmonstar/reqwest/blob/13487fb60253253e309442671c390682e2d1df18/src/blocking/client.rs#L1225-L1274

You're seeing the extra file descriptors because that runtime is still running after the reqwest::blocking::get call. I think it will shut down automatically shortly, but I don't know reqwest so well.

---

As for runtime.shutdown_background(), that does indeed not wait for the runtime shutdown to finish, so the sleep is necessary to wait for shutdown to complete. You can use drop(runtime) to shut down the runtime in a manner where you wait for shutdown to complete.