Skip to content

Add automatic cleanup of warm AI containers on process shutdown #3776

@rickstaa

Description

@rickstaa

Is your feature request related to a problem? Please describe.

Currently, when running docker compose down, the warm containers created by go-livepeer remain running on the machine. These containers persist even after the main Livepeer process is stopped, requiring manual cleanup if the orchestrator or worker node is moved to a new host. Several orchestrators have asked about this feature as it will improve their operations.

Running docker compose down seems to leave the warm containers on the machine. Is there a way to ensure those are also terminated when the go-livepeer process stops?

Describe the solution you'd like

It would be helpful if the go-livepeer binary automatically cleaned up any spawned warm containers upon shutdown, not just during startup.
Possible options include:

  • Trigger cleanup when the Livepeer process receives a shutdown signal.
  • Alternatively, allow child containers to perform a health check that shuts them down when the parent process is no longer running.

Describe alternatives you've considered

  • Current behavior only cleans up on orchestrator startup.
  • Manual cleanup via docker kill or system scripts works but adds operational overhead, especially when moving orchestrators between machines.

Additional context

This would improve operational simplicity and prevent orphaned containers accumulating on nodes after shutdown allowing better orchestration of compute.

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: triagethis issue has not been evaluated yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions