Skip to content

Add native Prometheus serving metrics endpoint - #32

Open
remer wants to merge 1 commit into
brontoguana:mainfrom
remer:feat/native-prometheus-metrics
Open

Add native Prometheus serving metrics endpoint#32
remer wants to merge 1 commit into
brontoguana:mainfrom
remer:feat/native-prometheus-metrics

Conversation

@remer

@remer remer commented Aug 22, 2026

Copy link
Copy Markdown

Adds a non-queued GET /metrics endpoint to the Rust HTTP server so telemetry remains scrapeable during long inference.

Exposed signals:

  • exact model-worker running and scheduler waiting gauges
  • prompt/generation token counters
  • exact prefix-query and reused-prefix token counters
  • active logical KV tokens and context occupancy
  • TTFT, inter-token latency, and end-to-end latency histograms
  • terminal outcomes (stop, length, tool_calls, error, abort)
  • RAM session-cache occupancy, hit/miss/eviction, and save/restore counters

The common serving metrics use the established vllm: names for collector compatibility; Krasis-specific occupancy/session metrics use the krasis_ namespace. The endpoint snapshots metrics under a short mutex and formats after releasing it. Per-token TPOT and logical-KV updates use one lock acquisition.

Semantics:

  • TTFT starts at HTTP admission and includes scheduler wait.
  • TPOT is the interval between consecutive generated tokens.
  • logical KV occupancy counts tokens actually represented in active sequence state; it is intentionally not the percentage of the preallocated KV buffer.
  • prefix hits are a token numerator and prefix queries are full prompt-token denominator.
  • a request guard records early exits as errors and clears running/KV gauges on every path.

Validation:

  • cargo check --release --lib passes with CUDA enabled
  • optimized extension build imports under Python 3.13
  • live smoke covered non-streaming completion, streamed length completion, exact progressive-prefix reuse, terminal counters, histogram counts, and a 10 ms /metrics scrape while generation was active
  • no credentials, hostnames, addresses, local paths, or request contents are included in this change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant