Cycles Event Server is the asynchronous processing tier for the Cycles stack. It consumes Redis-backed work streams produced by cycles-server, persists operational records, and performs the out-of-band work that should not block the runtime reservation path: webhook fan-out and CyclesEvidence signing.
It currently runs two independent workers:
-
Webhook delivery — consumes Cycles events and delivers them to subscriber endpoints using signed, at-least-once HTTP delivery. Deliveries include HMAC-SHA256 signatures, retry with exponential backoff, delivery-attempt tracking, automatic subscription disablement after repeated failures, and encrypted storage for webhook secrets and custom headers.
-
CyclesEvidence signing — consumes evidence source records emitted by
cycles-server, builds acycles-evidence/v0.1envelope, signs it with Ed25519, and stores the signed envelope by content address.cycles-serverthen serves the signed evidence atGET /v1/evidence/{id}. The evidence private signing key is isolated in this service;cycles-serverdoes not need access to it. See CyclesEvidence signing and the identity enablement runbook.
The event server is not the runtime budget authority. It does not reserve, commit, release, or enforce budget. Its role is to turn Cycles state changes into durable, auditable delivery and identity artifacts: webhook deliveries, replayable event history, delivery diagnostics, trace-correlated records, and signed evidence envelopes.
Specs: cycles-governance-admin-v0.1.25.yaml for events, webhooks, delivery records, and replay · cycles-evidence-v0.1.yaml for the evidence envelope.
cycles-server-admin cycles-server (runtime)
│ tenant/budget CRUD, │ reservation ops,
│ api_key/policy lifecycle │ budget thresholds,
│ │ rate spike detection
│ │
└──────────┐ ┌────────────┘
▼ ▼
EventService.emit() → save event + find matching subscriptions
WebhookDispatchService → create PENDING delivery + LPUSH dispatch:pending
│
▼
Redis ──BLMOVE──► cycles-server-events (DispatchLoop)
│
├── DeliveryHandler: load delivery + event + subscription
├── WebhookUrlGuard: delivery-time SSRF re-validation against
│ config:webhook-security (blocked → permanent FAIL, no retry)
├── SubscriptionRepository: decrypt signing secret (AES-256-GCM)
├── WebhookTransport: HTTP POST with HMAC-SHA256 signature
├── On success: mark SUCCESS, reset consecutive failures
├── On failure + retries left: exponential backoff → RETRYING
├── On failure + retries exhausted: FAILED + increment consecutive failures
├── Ack: LREM dispatch:processing only after state/retry is durable
└── On consecutive failures >= threshold: subscription → DISABLED
Event sources (per spec source field): cycles-admin, cycles-server, expiry-sweeper, anomaly-detector
| Concern | Admin / Runtime Servers | Events Service |
|---|---|---|
| Workload | Synchronous CRUD + reservation ops | Asynchronous delivery, variable latency |
| Scaling | Scale with API traffic | Scale with webhook volume |
| Failure isolation | Servers stay responsive during delivery backlog | Delivery retries don't block API |
| Concurrency | Multiple instances | Multiple instances safe (BLMOVE claim + ack) |
Alongside webhook delivery, this service is the signing tier for CyclesEvidence — tamper-evident, content-addressed audit envelopes for every budget decision (see the concept docs and cycles-evidence-v0.1.yaml).
cycles-server (producer) cycles-server-events (signer)
emits {decide|reserve|commit|release|error} EvidenceWorker (reliable queue: BLMOVE)
source record + computes evidence_id ──LPUSH──► ├── CyclesEvidenceEnvelopeBuilder: build cycles-evidence/v0.1
(returns cycles_evidence on the response) ├── recompute evidence_id; CROSS-CHECK vs producer's
│ stamped id → dead-letter on drift
evidence:pending ├── EnvelopeSigner: Ed25519 sign (PRIVATE KEY lives here)
└── EvidenceStore: SET evidence:envelope:<id> ◄── cycles-server
serves GET /v1/evidence/{id}
Why the signer is in this tier: the expensive work (JCS canonicalization + Ed25519 signing) is asynchronous and must not block a reservation, and the private signing key is isolated to this service — cycles-server only ever holds the public identity (it reproduces the evidence_id content hash synchronously, never signs). The worker recomputes the id and dead-letters on mismatch, so producer/worker config drift fails closed.
Configure it: the shared public identity (EVIDENCE_SERVER_ID, EVIDENCE_SIGNING_SIGNER_DID) plus this service's private key (EVIDENCE_SIGNING_PRIVATE_KEY_HEX) — full provisioning, coherence rules, and verification steps are in docs/evidence-identity-enablement.md. If EVIDENCE_SERVER_ID is blank, the evidence signer is disabled and does not claim or dead-letter source records.
# From cycles-server-admin directory
docker compose -f docker-compose.full-stack.yml upServices: Redis (6379), Admin (7979), Runtime Server (7878), Events worker (9980 management; 7980 internal/no API)
REDIS_HOST=localhost REDIS_PORT=6379 java -jar target/cycles-server-events-*.jar| Variable | Default | Description |
|---|---|---|
REDIS_HOST |
localhost | Redis hostname |
REDIS_PORT |
6379 | Redis port |
REDIS_PASSWORD |
(empty) | Redis password |
WEBHOOK_SECRET_ENCRYPTION_KEY |
(empty) | AES-256-GCM key for signing secret encryption (base64-encoded 32 bytes). If empty, secrets stored/read as plaintext (backward compatible). |
dispatch.pending.timeout-seconds |
5 | BLMOVE blocking timeout (seconds) |
DISPATCH_PROCESSING_RECOVERY_IDLE_MS |
120000 | Minimum age before an in-flight dispatch:processing delivery is considered stale and requeued. Keep above the max expected webhook HTTP + Redis write time. |
dispatch.retry.poll-interval-ms |
5000 | Retry queue poll interval (ms) |
RETRY_BATCH_SIZE |
100 | Max retries to requeue per poll cycle |
dispatch.http.timeout-seconds |
30 | HTTP request timeout for webhook delivery |
dispatch.http.connect-timeout-seconds |
5 | HTTP connect timeout |
MAX_DELIVERY_AGE_MS |
86400000 | Maximum delivery age (ms). Deliveries older than this after events service outage are auto-failed instead of delivered stale. Default: 24 hours. |
EVENT_TTL_DAYS |
90 | Redis TTL for event:{id} keys (days). Spec: "90 days hot." |
DELIVERY_TTL_DAYS |
14 | Redis TTL for delivery:{id} keys (days). |
RETENTION_CLEANUP_INTERVAL_MS |
3600000 | How often to trim expired ZSET index entries (ms). Default: 1 hour. |
CYCLES_METRICS_TENANT_TAG_ENABLED |
false | Include tenant IDs as Prometheus metric labels. Keep false in production unless per-tenant drill-down is worth the cardinality. |
JAVA_OPTS |
(empty) | Extra JVM options appended after image defaults by the Docker entrypoint. |
EVIDENCE_SERVER_ID |
(empty) | CyclesEvidence server_id. Blank disables the evidence signer. When set, it must be byte-identical to cycles-server's value (incl. the /v1 base) or the worker's id cross-check dead-letters. See the enablement runbook. |
EVIDENCE_SIGNING_SIGNER_DID |
(empty) | Public Ed25519 key (64 hex), the public half of EVIDENCE_SIGNING_PRIVATE_KEY_HEX, identical to cycles-server's value. |
EVIDENCE_SIGNING_PRIVATE_KEY_HEX |
(empty) | Secret — Ed25519 seed (64 hex) this service signs with; lives only here. Required when EVIDENCE_SERVER_ID is set unless ephemeral dev mode is explicitly allowed; setting only one of the signing pair fails startup. |
EVIDENCE_ALLOW_EPHEMERAL_SIGNING_KEY |
false | Development-only escape hatch. When true and evidence is enabled without a signing pair, generates an ephemeral key that will not verify across restarts. Keep false in production. |
openssl rand -base64 32The same key must be configured in both cycles-server-admin and cycles-server-events. Admin encrypts secrets on write; events decrypts on read.
1. Client creates subscription (POST /v1/admin/webhooks)
└── optionally provides signing_secret, or admin auto-generates one (whsec_...)
2. Admin stores encrypted secret in Redis
└── webhook:secret:{subscriptionId} = AES-256-GCM(secret, WEBHOOK_SECRET_ENCRYPTION_KEY)
└── Returns plaintext secret to client ONCE in WebhookCreateResponse (never again)
3. Events service reads + decrypts secret on each delivery
└── CryptoService.decrypt(redis.get("webhook:secret:{id}"))
└── Backward compatible: plaintext secrets (no "enc:" prefix) returned as-is
└── Fail closed: encrypted secrets without the correct key are not delivered unsigned
4. PayloadSigner computes HMAC-SHA256(JSON payload, decrypted secret)
└── Sent as X-Cycles-Signature: sha256=<hex> header
5. Webhook receiver verifies signature using their copy of the secret
The X-Cycles-Signature header contains sha256=<hex> where <hex> is the HMAC-SHA256 of the raw JSON request body using the subscription's signing secret as the key.
Why HMAC? Without it, anyone who discovers a webhook URL can send fake events. HMAC proves both identity (shared secret) and integrity (body hash). Same standard used by GitHub, Stripe, and Slack webhooks.
Verification example (Python):
import hmac, hashlib
def verify(body: bytes, secret: str, signature: str) -> bool:
expected = "sha256=" + hmac.new(
secret.encode(), body, hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature)| Header | Value | Description |
|---|---|---|
Content-Type |
application/json |
Always JSON |
X-Cycles-Signature |
sha256=<hex> |
HMAC-SHA256 of body (if signing secret configured) |
X-Cycles-Event-Id |
evt_abc123... |
For deduplication (at-least-once delivery) |
X-Cycles-Event-Type |
budget.exhausted |
Event type for routing |
User-Agent |
cycles-server-events/0.1.25.22 |
Service identifier |
X-Cycles-Trace-Id |
<32-hex-lowercase> |
W3C trace-id (spec v0.1.25.27) — always present |
traceparent |
00-<trace-id>-<16-hex-span>-<flags> |
W3C Trace Context v00 — always present. <flags> preserves upstream sampling when WebhookDelivery.traceparent_inbound_valid=true (spec v0.1.25.28), else 01 |
X-Request-Id |
<request-id> |
Originating HTTP request id — present when event.request_id is populated |
| Custom headers | Per subscription | From WebhookSubscription.headers map |
Custom headers are additive only. Reserved delivery headers such as Content-Type,
User-Agent, X-Cycles-*, X-Request-Id, and traceparent are ignored if a
subscription tries to set them.
Default: 5 retries with exponential backoff (1s, 2s, 4s, 8s, 16s), capped at 60s max delay.
| Setting | Default | Range | Description |
|---|---|---|---|
max_retries |
5 | 0-10 | Retries after initial failure (6 total attempts) |
initial_delay_ms |
1000 | 100-60000 | Delay before first retry |
backoff_multiplier |
2.0 | 1.0-10.0 | Multiplier per retry |
max_delay_ms |
60000 | 1000-3600000 | Maximum delay cap |
Auto-disable: after disable_after_failures (default 10) consecutive delivery failures, the subscription status is set to DISABLED. Reset to 0 on any successful delivery.
PENDING ──HTTP 2xx──► SUCCESS (reset consecutive_failures)
│
└──non-2xx──► RETRYING ──retry──► SUCCESS
│
└──max retries exceeded──► FAILED
│
└──consecutive >= threshold──► subscription DISABLED
| Key | Type | Written By | Read By | Description |
|---|---|---|---|---|
dispatch:pending |
LIST | Admin (LPUSH), Events retry scheduler (LPUSH) | Events (BLMOVE) | Delivery IDs awaiting processing |
dispatch:processing |
LIST | Events (BLMOVE) | Events (LREM/recovery) | In-flight delivery IDs claimed but not yet acked |
dispatch:processing:claimed_at |
ZSET | Events (ZADD/ZREM) | Events recovery | Claim timestamps used to recover only stale in-flight deliveries |
dispatch:retry |
ZSET | Events (ZADD) | Events (ZRANGEBYSCORE) | Retry queue (score = timestamp) |
delivery:{id} |
STRING | Admin (SET) | Events (GET/SET) | Delivery record JSON |
event:{id} |
STRING | Admin (SET) | Events (GET) | Event record JSON |
webhook:{id} |
STRING | Admin (SET) | Events (GET/SET) | Subscription JSON |
webhook:secret:{id} |
STRING | Admin (SET, encrypted) | Events (GET, decrypts) | AES-256-GCM encrypted signing secret |
Multiple events service instances can safely claim from the same dispatch:pending list. Each claim atomically moves one delivery ID to dispatch:processing; the worker removes it from processing only after delivery, subscription, and retry state is durable. Recovery is age-gated: only entries that remain in dispatch:processing longer than DISPATCH_PROCESSING_RECOVERY_IDLE_MS are moved back to pending, so rolling deploys do not requeue another live replica's active delivery.
| Key | TTL | Cleanup |
|---|---|---|
event:{id} |
90 days (configurable) | Auto-expire via Redis EXPIRE |
delivery:{id} |
14 days (configurable) | Auto-expire via Redis EXPIRE |
events:{tenantId}, events:_all |
N/A (ZSET) | Hourly trim via RetentionCleanupService |
deliveries:{subId} |
N/A (ZSET) | Hourly trim via RetentionCleanupService |
dispatch:pending |
Self-draining | Claimed by BLMOVE into dispatch:processing |
dispatch:processing |
Self-draining | Acked by LREM; stale entries recovered to pending after the idle window |
dispatch:retry |
Self-draining | Entries move to pending when ready |
If cycles-server-events is not running or not deployed:
- Admin and runtime servers are unaffected — event emission is fire-and-forget, wrapped in try-catch, never blocks the API response
- Events and deliveries accumulate in Redis —
event:{id}keys (90-day TTL),delivery:{id}keys (14-day TTL),dispatch:pendinglist grows - Redis memory is bounded — TTLs ensure keys auto-expire even if never consumed
- When the events service restarts:
- Stale deliveries (older than
MAX_DELIVERY_AGE_MS, default 24h) are immediately marked FAILED — they won't be delivered late - Orphaned in-flight deliveries in
dispatch:processingare recovered todispatch:pendingafterDISPATCH_PROCESSING_RECOVERY_IDLE_MS - Fresh deliveries are processed normally via BLMOVE claim + ack
- RetentionCleanupService trims orphaned ZSET index entries hourly
- Stale deliveries (older than
- No data loss for events — event records persist in Redis for 90 days regardless of delivery status
As of admin-spec v0.1.25.16, six tenant-scoped webhook REST endpoints on cycles-server-admin accept both ApiKeyAuth and AdminKeyAuth:
GET /v1/webhooks, GET/PATCH/DELETE /v1/webhooks/{id}, POST /v1/webhooks/{id}/test, GET /v1/webhooks/{id}/deliveries.
Admin-initiated updates record actor_type=admin_on_behalf_of in audit metadata (vs api_key for tenant-initiated).
No functional impact on this service — cycles-server-events reads subscriptions from Redis directly and does not call those admin HTTP endpoints. Noted here for observability and ops awareness.
| Category | Count | Types |
|---|---|---|
budget |
16 | created, updated, funded, debited, reset, reset_spent, debt_repaid, frozen, unfrozen, closed, threshold_crossed, exhausted, over_limit_entered, over_limit_exited, debt_incurred, burn_rate_anomaly |
reservation |
5 | denied, denial_rate_spike, expired, expiry_rate_spike, commit_overage |
tenant |
6 | created, updated, suspended, reactivated, closed, settings_changed |
api_key |
6 | created, revoked, expired, permissions_changed, auth_failed, auth_failure_rate_spike |
policy |
3 | created, updated, deleted |
system |
5 | store_connection_lost, store_connection_restored, high_latency, webhook_delivery_failed, webhook_test |
webhook |
6 | created, updated, paused, resumed, deleted, disabled |
budget.reset_spent (v0.1.25.6, admin-spec v0.1.25.18) is emitted for billing-period rollovers and is distinct from budget.reset (which is a ceiling resize that preserves spent). Consumers can route these separately. The payload's spent_override_provided flag indicates whether spent was explicitly supplied (migration / proration / correction) vs defaulted to 0 (routine rollover).
Pluggable transport interface. Currently implements webhook (HTTP POST).
public interface Transport {
String type();
TransportResult deliver(Event event, Subscription subscription, String signingSecret, Delivery delivery);
}Spring Actuator endpoints run on a separate management port (9980). This worker has no public HTTP API; do not publish 7980 on an ingress. Keep 9980 on an internal-only ClusterIP / network and scrape from there.
| Endpoint | Description |
|---|---|
GET :9980/actuator/health/liveness |
Process liveness check |
GET :9980/actuator/health/readiness |
Readiness check, including Redis PING |
GET :9980/actuator/health |
Aggregate health |
GET :9980/actuator/info |
Build info (version, artifact) |
GET :9980/actuator/prometheus |
Prometheus-format metrics for scraping |
Prometheus scrape config example:
scrape_configs:
- job_name: cycles-server-events
metrics_path: /actuator/prometheus
static_configs:
- targets: ['localhost:9980']Override the management port via the MANAGEMENT_PORT env var if 9980 collides.
In addition to Spring Boot's auto-emitted http_server_requests_seconds (which covers the actuator endpoints, not the outbound webhook traffic), this service exposes eight domain-level meters under the cycles_webhook_* namespace — seven counters plus one latency timer. Operators can alert on fleet-wide failure rates, stale-delivery backlogs, subscription auto-disables, and payload-validator warnings without grepping logs.
Full metric inventory, tag semantics, ready-to-paste Prometheus alert rules, SLO definitions, dashboard queries, and an incident playbook live in OPERATIONS.md.
The webhook POST body is the full event JSON. Null fields are omitted.
{
"event_id": "evt_abc123",
"event_type": "budget.exhausted",
"category": "budget",
"timestamp": "2026-04-01T12:00:00Z",
"tenant_id": "t_xyz789",
"source": "runtime",
"data": {
"budget_id": "bdg_001",
"current_balance": 0,
"limit": 10000
},
"actor": {
"type": "api_key",
"key_id": "key_abc",
"source_ip": "10.0.1.42"
},
"correlation_id": "req_def456"
}# Build and run unit tests (293 unit tests, 95%+ line coverage enforced by JaCoCo)
mvn verify
# Run all tests including integration (requires Docker for Testcontainers Redis)
mvn verify -Pintegration-tests
# Run
REDIS_HOST=localhost REDIS_PORT=6379 java -jar target/cycles-server-events-*.jarCHANGELOG.md— release notes for downstream consumers (Docker / JAR / operators)OPERATIONS.md— operator runbook: metrics inventory, alert recipes, SLOs, incident playbookAUDIT.md— engineering history, audit posture, and cross-repo drift notes- Sibling services (same conventions, dashboards carry over):
cycles-server— runtime reservation + budget authoritycycles-server-admin— admin plane (tenants, budgets, webhooks, API keys)
Apache License 2.0