Skip to content

Volume plugin validation during runtime refresh causes spurious errors for unrelated operations #28110

Description

@kriansa

Issue Description

I'm the author of podman-volume-stratis, a volume plugin for Stratis filesystems. While doing QA on a real deployment, I noticed that every boot produces error-level log messages about my volume plugin being inaccessible — even though everything works fine.

The errors come from network quadlet services that have nothing to do with volumes. On the first podman invocation after reboot, Runtime.refresh() eagerly validates all volume plugins by sending HTTP requests to their /Plugin.Activate endpoints, regardless of what operation was actually requested.

These are purely cosmetic — as long as I have the plugin as a dependency on the volume quadlets, all volumes mount successfully once the plugin starts. But the error log messages are confusing, create noise in system logs, and are indistinguishable from real failures at a glance. This only affects quadlet services started at boot — normal CLI operations work fine since the plugin is already running by then.

Steps to reproduce the issue

  1. Set up quadlet files where a container depends on a volume (backed by a volume plugin) and a network:
# myapp.network (no dependencies, starts early at boot)
[Network]
NetworkName=myapp
# myapp-data.volume (depends on volume plugin service)
[Unit]
After=my-volume-plugin.service
Requires=my-volume-plugin.service

[Volume]
VolumeName=myapp-data
Driver=my-plugin
# myapp.container (depends on both)
[Install]
WantedBy=default.target

[Container]
ContainerName=myapp
Image=docker.io/library/alpine:latest
Network=myapp.network
Volume=myapp-data.volume:/data:rw
Exec=sleep infinity

The dependency graph is correct: the container waits for both the network and the volume, and the volume waits for the plugin service. There is no ordering issue on the user side.

  1. Reboot the system

  2. The network quadlets have no dependencies, so systemd starts them early. The generated network service runs podman network create, which is the first podman invocation after reboot and triggers Runtime.refresh().

Describe the results you received

The network quadlet triggers a refresh that validates all volume plugins, producing error-level messages for every plugin-backed volume:

Feb 17 15:15:41 atlas systemd[1]: Finished NetworkManager-wait-online.service - Network Manager Wait Online.
Feb 17 15:15:41 atlas systemd[1]: Reached target network-online.target - Network is Online.
[... network dependency fulfilled]
Feb 17 15:15:41 atlas systemd[1]: Starting git-server--main-network.service...
Feb 17 15:15:41 atlas systemd[1]: Starting ingress--main-network.service...
Feb 17 15:15:41 atlas systemd[1]: Starting smtp-server--main-network.service...
Feb 17 15:15:41 atlas systemd[1]: Starting unifi-network--main-network.service...
Feb 17 15:15:41 atlas systemd[1]: Starting ups-exporter--main-network.service...
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume ingress--data uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume unifi-network--db uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume unifi-network--main uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume ups-exporter--data uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume git-server--conf uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
Feb 17 15:15:41 atlas smtp-server--main-network[854]: level=error msg="Volume git-server--data uses volume plugin stratis, but it cannot be accessed - some functionality may not be available: cannot access plugin stratis socket \"/run/podman/plugins/volume-stratis.sock\": stat /run/podman/plugins/volume-stratis.sock: no such file or directory"
[ !!! this is where podman is trying to do a system refresh but volumes aren't ready]
Feb 17 15:15:41 atlas podman[854]: system refresh
Feb 17 15:15:41 atlas podman[863]: network create 997c24ae7fa7279689489eb3e973169231a87f83838acf2578023e74c59013bf (name=ups-exporter--main, type=bridge)
Feb 17 15:15:41 atlas ingress--main-network[842]: ingress--main
Feb 17 15:15:41 atlas ups-exporter--main-network[863]: ups-exporter--main
Feb 17 15:15:41 atlas smtp-server--main-network[854]: smtp-server--main
Feb 17 15:15:41 atlas podman[854]: network create a5c6f387d620feaf47f7fc787d1dd1912269945e025c3ffa1284aadc92fad77b (name=smtp-server--main, type=bridge)
Feb 17 15:15:41 atlas podman[842]: network create a92acc68eab4d6bac1363b6641e9bfdc32a1da2edff676bbb057e4aec8a0ddca (name=ingress--main, type=bridge)
Feb 17 15:15:41 atlas podman[840]: network create 56ea3234a74065cce2f9ace7df76f5862683f8e294d79baeae818a4e604eabd0 (name=git-server--main, type=bridge)
Feb 17 15:15:41 atlas git-server--main-network[840]: git-server--main
Feb 17 15:15:41 atlas podman[862]: network create 8975f121d6b772cf6dbb0c4273fce48d04948e6f8e81bf21d4e5c08e37d7a70d (name=unifi-network--main, type=bridge)
Feb 17 15:15:41 atlas unifi-network--main-network[862]: unifi-network--main
Feb 17 15:15:41 atlas systemd[1]: Finished ups-exporter--main-network.service.
Feb 17 15:15:41 atlas systemd[1]: Finished smtp-server--main-network.service.
Feb 17 15:15:41 atlas systemd[1]: Finished git-server--main-network.service.
Feb 17 15:15:41 atlas systemd[1]: Finished unifi-network--main-network.service.
Feb 17 15:15:41 atlas systemd[1]: Finished ingress--main-network.service.
Feb 17 15:15:41 atlas stratisd[739]: [2026-02-17T15:15:41Z INFO  stratisd::engine::strat_engine::liminal::liminal] Pool with name "podman_vols" and UUID "1e75cfba-6b7f-4376-b022-0437a4e5eb15" set up
Feb 17 15:15:41 atlas systemd[1]: Started stratisd.service - Stratis daemon.
[...plugin dependency fulfilled]
Feb 17 15:15:41 atlas systemd[1]: Starting podman-volume-stratis.service - Podman/Docker Volume Plugin for Stratis Filesystems...
Feb 17 15:15:41 atlas stratisd[739]: [2026-02-17T15:15:41Z INFO  stratisd::stratis::ipc_support::dbus_support] D-Bus API is available
Feb 17 15:15:41 atlas podman-volume-stratis[1367]: time=2026-02-17T15:15:41.858Z level=INFO msg="starting volume plugin" pool=podman_vols mount_path=/mnt socket=/run/podman/plugins/volume-stratis.sock backend=dbus
Feb 17 15:15:41 atlas podman-volume-stratis[1367]: time=2026-02-17T15:15:41.969Z level=INFO msg="listening on socket" path=/run/podman/plugins/volume-stratis.sock
Feb 17 15:15:41 atlas systemd[1]: Started podman-volume-stratis.service - Podman/Docker Volume Plugin for Stratis Filesystems.
[... volume dependency fulfilled]
Feb 17 15:15:41 atlas systemd[1]: Starting git-server--conf-volume.service...
Feb 17 15:15:41 atlas systemd[1]: Starting smtp-server--main.service - SMTP Server Container...
[... then all containers start successfully ...]

Describe the results you expected

A podman network create should not attempt to validate volume plugins. The error messages suggest something is wrong when nothing actually is.

podman info output

host:
  arch: amd64
  buildahVersion: 1.41.8
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.el10.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: almalinux
    version: "10.1"
  eventLogger: journald
  kernel: 6.12.0-124.35.1.el10_1.x86_64
  linkmode: dynamic
  logDriver: journald
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-2.el10.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.0-1.el10.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.16.0
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.el10_0.x86_64
    path: /usr/bin/crun
  os: linux
  security:
    rootless: false
    seccompEnabled: true
    selinuxEnabled: true
  serviceIsRemote: false
plugins:
  volume:
  - local
  - stratis
store:
  graphDriverName: overlay
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.6.0
  Built: 1768780800
  GoVersion: go1.25.3 (Red Hat 1.25.3-1.el10_1.alma.2)
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

N/A

Additional information

Probable root cause

finalizeVolumeSqlite() eagerly loads and validates the volume plugin (including an HTTP roundtrip to /Plugin.Activate) for every plugin-backed volume whenever a volume is read from the database.

This function is called from AllVolumes(), which is called during refresh(). The refresh runs on the first podman invocation after reboot, regardless of what operation was requested.

However, vol.refresh() — the actual per-volume refresh called at line 856 — only re-acquires the volume's lock. It does not use the plugin reference at all. The plugin is only needed for actual volume operations (mount, unmount, create, remove).

Suggested fix

One possible approach would be to defer plugin validation in finalizeVolumeSqlite() — instead of calling getVolumePlugin() eagerly when loading volumes from the database, resolve the plugin lazily on first actual use. This would prevent HTTP requests to plugins during refresh and eliminate the spurious errors.

Does this analysis make sense? I'm happy to help with a fix if you agree with the approach, or if you'd suggest a different direction I'm all ears.

Metadata

Metadata

Assignees

Labels

ContribfestBugs for Contribfestbugweekkind/bugCategorizes issue or PR as related to a bug.triagedIssue has been triagedvolunteers-wantedIssues good for community/volunteer contributions

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions