Suppress traceback in UI for expected custom Failed flow states returned by users (3.4.22)

[zzstoatzz]

This discussion was created from a Slack thread conversation.

Original Thread: https://prefect-community.slack.com/archives/C04DZJC94DC/p1764064403374339

---

Summary
Users can return a Prefect State directly from a flow, e.g., return Failed(name="custom_failure", message="..." ). In Prefect 3.4.22, when a flow returns a Failed state, the engine retrieves the run result which raises a prefect.exceptions.FailedRun. In deployment/worker execution (subprocess), this exception bubbles to the subprocess boundary where Prefect logs an error with full traceback ("Execution of flow run ... exited with unexpected exception") that is sent to the API. The Prefect UI then shows a large internal traceback even though the failure was intentionally user-declared and not an unexpected engine crash. In contrast, running the same flow locally via flow.serve() typically does not send that boundary traceback to the API/UI, so the UI appears cleaner. This results in inconsistent UX across execution contexts and noisy UI logs for expected terminal failures.

Environment

• Prefect version: 3.4.22
• Execution: deployment on local process work pool
• Repro: Create a flow that returns Failed(name="custom_failure", message="custom failure"), deploy, run; observe UI logs include full traceback from prefect.engine/prefect.flow_runs.runner even though the state is expected.

Why this matters

• Some flows legitimately end in a Failed state by design (e.g., business rules/validation). Users want the run to surface as red with a custom name/message but without implying an unexpected engine crash. The current boundary traceback in the UI is confusing to stakeholders scanning logs and prevents distinguishing expected terminal failures from unexpected crashes.

Current behavior (code references)

• flow_engine.run_flow_sync returns engine.result() by default; failed states cause state.result(raise_on_failure=True) to raise FailedRun (src/prefect/states.py)
• In subprocess execution, the exception is logged with logger.exception at the engine boundary (src/prefect/engine.py), which is sent to API/UI
• flow.serve() differs in where logs are emitted/forwarded, so the UI often does not show this traceback.

Requested enhancement

• Provide a user-facing way to mark user-returned Failed states as "graceful" so that subprocess boundary does not log a traceback to API, e.g.:
  1. An execution/deployment option that sets engine return_type="state" (or equivalent) to avoid calling engine.result() on the final state; or
  2. An option to set raise_on_failure=False for final flow result retrieval when the flow returns a Failed state created by the user; or
  3. A configuration flag (flow decorator or deployment setting) to suppress traceback logging for user-declared terminal Failed states while still logging real unexpected exceptions.

Notes

• We explored logging-level changes, but users do not want to suppress all ERRORs from prefect.engine globally since that hides unrelated critical errors. A targeted, engine-aware approach is preferable.
• Workarounds include custom logging filters on the API handler to drop only the boundary error for matching FailedRun messages, but this requires logging customization per environment and feels brittle.

Closing
It would be great to align the UX between flow.serve() and deployments, allowing expected custom Failed states to be clearly represented without large internal tracebacks in UI logs.

---

This discussion was automatically created by the Marvin bot to preserve valuable community insights.