Nomad v2.0.4, nomad-driver-podman 0.6.5, Linux amd64.
An ephemeral CI runner task restarts after every job it accepts. Each in-place restart grows the allocation's logmon process by exactly one OS thread and one open fd on the alloc log directory; after 146 restarts it sat at 159 threads / 163 fds. The per-start log FIFO/rotator wiring appears to be recreated on every task start without tearing down the previous generation.
Repro: restart any task in place and watch its logmon - ls /proc/PID/task | wc -l and ls /proc/PID/fd | wc -l grow by one per restart and never shrink.
Expected: flat counts across in-place restarts.
Workaround: plugin-level disable_log_collection = true (logs land in journald via Podman); logmon then stays an inert stub, flat at 17 fds across the same restarts. The task-level logs { disabled = true } is not usable here because nomad-driver-podman does not advertise DisableLogCollection - maybe worth a driver fix too, but the leak is in logmon's per-start wiring itself.
Still present in the 2.0.5 sources as far as I can tell. Happy to test a patch.
Nomad v2.0.4, nomad-driver-podman 0.6.5, Linux amd64.
An ephemeral CI runner task restarts after every job it accepts. Each in-place restart grows the allocation's logmon process by exactly one OS thread and one open fd on the alloc log directory; after 146 restarts it sat at 159 threads / 163 fds. The per-start log FIFO/rotator wiring appears to be recreated on every task start without tearing down the previous generation.
Repro: restart any task in place and watch its logmon -
ls /proc/PID/task | wc -landls /proc/PID/fd | wc -lgrow by one per restart and never shrink.Expected: flat counts across in-place restarts.
Workaround: plugin-level
disable_log_collection = true(logs land in journald via Podman); logmon then stays an inert stub, flat at 17 fds across the same restarts. The task-levellogs { disabled = true }is not usable here because nomad-driver-podman does not advertise DisableLogCollection - maybe worth a driver fix too, but the leak is in logmon's per-start wiring itself.Still present in the 2.0.5 sources as far as I can tell. Happy to test a patch.