Add destination-level toggle controls

[ruslandoga]

Summary

• Adds a persisted enabled flag for destinations and exposes it through the backend API and Backends LiveView index/show controls.
• Excludes disabled destinations from attached-source and drain-rule routing, explicit ingestion, alert delivery, counters, backpressure checks, source workers, and consolidated pipelines; alert destinations are rechecked on the primary immediately before external dispatch.
• Reconciles lifecycle from primary state using an ID-only, per-backend serialized path: application writes fan out once, WAL events reconcile locally after companion cache records are invalidated, startup happens before cache exposure, and disable invalidation happens before shutdown.
• Makes backend cache misses primary-backed. A cache-wide checkpoint guards in-flight reads, while exact-key and returned-record generations protect committed read-through, warmer, and gossip values; stale readers use compare-and-delete so they cannot remove a newer fill.
• Treats post-commit lifecycle failures as convergence failures: the persisted toggle succeeds, the failure is logged, and cluster/WAL/periodic reconciliation can retry it.
• Uses separate create and partial-update OpenAPI request schemas, so PATCH {"enabled": false} is documented without requiring name, type, or config.

Deployment

1. Run the migration before deploying the application release.
2. Deploy the application to every node before anyone uses the toggle controls or writes enabled through the API. Mixed-version nodes do not have the new reconciler or disabled-routing guards.
3. After the rollout completes, the controls can be used normally; WAL reconciliation and periodic workers provide idempotent convergence.

On PostgreSQL 11+, adding this constant DEFAULT true NOT NULL column uses the fast-default path and does not rewrite existing rows. The statement should therefore be quick, though it still briefly takes an ACCESS EXCLUSIVE table lock.

Related fixes

• Cache invalidation cleanup: #3864 — remove the unreachable Cachex.del/2 key
• Supervisor child selection: #3865 — select the requested child from Supervisor.which_children/1 with Enum.find_value/2

Those pre-existing fixes are isolated for independent review. Their overlapping diffs can be removed from this PR after they land and this branch is rebased.

Validation

• 361 focused tests and 1 doctest across backend lifecycle/cache/routing, WAL/gossip, alerting, API schema/controller, and LiveView files: 0 failures (3 excluded)
• mix format --check-formatted
• mix compile --warnings-as-errors
• mix credo (no priority issues; existing design suggestions only)

Linear: O11Y-2315

[ruslandoga]

Deployment notes

Keeping this PR as the end-state reference; production should roll out in three phases:

1. Migration only: add enabled BOOLEAN NOT NULL DEFAULT true. PostgreSQL 11+ uses the constant-default fast path, though the statement still needs a brief exclusive lock.
2. Dormant logic: deploy the schema, routing/supervisor guards, reconciliation, and cache handling everywhere, but do not accept API writes or show the UI toggle. Mixed-version code must preserve/version the cluster RPC contract and treat cached backend structs without enabled as enabled.
3. Activation: expose API and LiveView writes only after every node runs the logic release.

The explicit enabled: true argument creates a new cache-key variant alongside the legacy key. Coexistence is safe but causes temporary cold misses and duplicate entries; keep eviction for both variants. Backend-ID busting clears non-empty lists, while exact per-source deletion is required for cached empty lists when changing false → true. Optionally wait one backend-cache TTL (~20 minutes) after the logic rollout; rollback only to a logic-aware release while any destinations are disabled.

[ruslandoga]

Scope checkpoint

This PR has grown because it moved from “add a destination toggle” to “guarantee cluster-wide, race-free lifecycle transitions under replica lag, cache gossip, WAL delivery, and process failures.”

The product contract determines the appropriate scope:

• Simple routing toggle: disabled destinations are excluded from new routing. Existing backend processes may remain alive but idle.
• Strong pause/resume: disabling immediately stops delivery everywhere, while enabling cannot expose a destination until every required pipeline is ready.

The current implementation has been converging on the second contract. That is what pulled in cache generations, tombstones, WAL ordering, RPC reconciliation, locks, retries, and supervisor lifecycle changes.

If the simple contract is sufficient, the PR can be reduced substantially:

• Keep the migration, schema, API/OpenAPI, and UI.
• Filter disabled destinations at the actual routing and dispatch boundaries.
• Leave backend children and consolidated pipelines running but idle.
• Remove synchronous lifecycle reconciliation, cluster RPC orchestration, worker rewrites, and most cache-protocol changes.
• Keep chore: remove redundant backend cache invalidation #3864 and fix: stop the requested backend child #3865 as separate fixes.
• Document that propagation follows the existing cache/WAL semantics.

If an immediate, hard cluster-wide cutoff is required, the current complexity is real, but it should be presented and split as distributed lifecycle infrastructure rather than treated as part of a small UI toggle.

No pruning decision has been applied yet; this comment records the scope decision we should make before continuing.