Skip to content

Add hard timeouts for stuck runtime requests #124

Description

@gaelic-ghost

What to add

Add hard timeout handling for stuck runtime requests so generation, playback, queue bookkeeping, and request event streams cannot remain active forever without a terminal state.

Why

A live service request can currently appear to remain running / playing after generated audio has completed. In the observed case, the worker stayed ready and generated audio was retained, but playback still reported an active request and DELETE /requests/{request_id} timed out.

Hard timeouts would keep the service operable when a backend, playback sink, stream, event continuation, or cancellation path fails to finish cleanly.

Suggested behavior

  • Apply bounded timeouts around active generation requests.
  • Apply bounded timeouts around active playback requests.
  • Apply bounded timeouts around request cancellation so cancel routes return promptly.
  • Emit a terminal event when a timeout fires.
  • Include a human-readable timeout message that names the request ID, operation, profile name when available, elapsed time, and likely stuck stage.
  • Clear or quarantine the active queue slot so later requests are not blocked by stale active state.
  • Preserve enough request history to debug what stage timed out.

Related observed failure

  • Request ID: 0217E899-B3D0-49A1-B1E0-55941BED6149
  • Operation: generate_speech
  • Voice profile: swift-signal
  • Generated audio completed, but playback stayed active.
  • Cancel route timed out and the request remained running.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions