Support New Origin Type: Druid

[Raboo]

Would be awesome if there was support for druid.

[jranson]

Hi Elias, thanks for the suggestion. I will make sure we get it on the 2022 Roadmap - it needs a revamp!

[jranson]

I've finally had a chance to do the analysis on this. Apache Druid is a very good match for Trickster's TSDB acceleration, and is one we should have added support for a long time ago. I think we can get this included in the upcoming 2.1 release.

@houyuwushang would you like to implement on this feature? I will add more structured specifications for it to this issue the next 24 hours, so i recommend waiting on those to land before proceeding with implementation.

[houyuwushang]

Yes, I'd be happy to take this on. I'll wait for your design specs before getting started.

[jranson]

Apache Druid Backend - Feasibility + Implementation Plan

Goal: add druid as a fully delta-cacheable timeseries provider. This covers the native JSON query API (POST /druid/v2/), which is the best-case scenario for Trickster: time ranges live in a structured intervals JSON field (no SQL/text parsing needed) and the step lives in a structured granularity field. Druid SQL (POST /druid/v2/sql) is out of scope for time series acceleration and should be OPC-cached (it is a read-only endpoint, no need to parse the SQL to filter for SELECTs), while /druid/v2/sql/task should be proxy-only. Everything else on the API surface is ObjectProxyCache or plain proxy.

Feasibility summary (verified against Druid docs, Aug 2026)

• Native queries are JSON objects POSTed to /druid/v2/. Time range: "intervals": ["2024-01-01T00:00:00Z/2024-01-02T00:00:00Z", ...] (ISO-8601 start/end strings; array may contain multiple disjoint ranges). Not URL params - but structured JSON, so SetExtent is a JSON unmarshal → replace intervals → re-marshal. No AST parser dependency.
• Step: the granularity field. Three forms: - simple strings: second, minute, five_minute, ten_minute, fifteen_minute, thirty_minute, hour, six_hour, eight_hour, day, week, month, quarter, year, plus all and none - {"type":"duration","duration":<ms>,"origin":...} - exact ms buckets - {"type":"period","period":"PT1M","timeZone":...,"origin":...} - ISO-8601 period, calendar/DST-aware
• Druid aligns result-row timestamps to granularity boundaries itself, so extent normalization to the step is native behavior. The intervals array should be excluded from the query for the purposes of deriving cache key.
• Delta-eligible: fixed-width granularities (simple grains second→day when UTC-aligned; duration type; period type with UTC/no-DST fixed periods). Not delta-eligible → OPC fallback (canOPC=true from ParseTimeRangeQuery): all (one bucket spanning the whole interval), none (raw ms), calendar grains (week/month/quarter/year), period granularities with non-UTC timeZone, multi-element intervals arrays (TimeRangeQuery has a single Extent).
• Delta-cacheable query types (response rows are independent per time bucket, so cached+fresh merge is safe): timeseries ([{timestamp, result:{...}}]), groupBy ([{version, timestamp, event:{...}}] - dimensions become series tags), topN ([{timestamp, result:[rows]}] - each bucket's ranking is self-contained).
• OPC-only read queries: scan (raw rows, no granularity), search, segmentMetadata, datasourceMetadata, timeBoundary (short TTL - the answer moves as data arrives), any GET metadata endpoints (/druid/v2/datasources..., /status).
• Cache-key hazard: clients (Druid console, Grafana/grafadruid, Turnilo) commonly set context.queryId (random UUID per query) and other per-request context keys. These MUST be stripped from the canonical statement or every request is a cache miss. Strip at minimum: queryId, sqlQueryId, priority, timeout, queryDeadline. Keep semantic keys (skipEmptyBuckets, grandTotal) in the key.
• Realtime ingestion mutates recent buckets → rely on the existing backfill-tolerance machinery (bft); default it generously (e.g. 60s+) since Druid realtime tasks answer for the head of the timeline before segments publish.

Implementation of native query API, delta cache

File set mirrors influxdb/clickhouse: pkg/backends/druid/{druid.go, routes.go, handler_query.go, handler_proxy.go, health.go, url.go, parsing.go, observability.go, stubs.go} + pkg/backends/druid/model/{model.go, unmarshal.go, marshal.go}. Structural template: InfluxDB Flux (JSON body, tokenized statement, body rewritten in SetExtent - pkg/backends/influxdb/flux/flux.go), with the ClickHouse plan discipline (analyze once → immutable plan on trq.ParsedQuery → render per shard; DPC calls SetExtent once per miss range, so the plan must be concurrency-safe and never mutated).

1. Registration: add druid const + Names/Values + supportedTimeSeries in pkg/backends/providers/providers.go; add NewClient to pkg/backends/providers/registry/registry.go. Client is type Client struct { backends.TimeseriesBackend } per the existing pattern.
2. parsing.go - ParseTimeRangeQuery(*http.Request): - accept only POST with application/json body on the query path; unmarshal into a minimal typed struct (queryType, dataSource, intervals, granularity, context, plus json.RawMessage passthrough for everything else - never lossily round-trip fields we don't model). - gate on queryType ∈ {timeseries, groupBy, topN}; else return canOPC=true. - resolve granularity → trq.Step (map simple grains to durations; duration type direct; period type via fixed-width check) and origin → trq.Phase; non-fixed → canOPC=true. - single interval only → trq.Extent; multi-interval → canOPC=true. - canonical statement for the cache key: re-marshal the parsed query with intervals replaced by a placeholder token, context stripped of per-request keys, and map keys in deterministic order. Store the parsed struct on trq.ParsedQuery, original body on trq.OriginalBody. - honor trq.CacheKeyElements for dataSource so per-datasource metrics/keying work.
3. url.go - SetExtent: take the plan from trq.ParsedQuery (fail closed with an error if absent, per the ClickHouse pattern), render intervals: [start/end) - note Druid intervals are half-open end-exclusive, so render extent.End + step as the end bound to include the final bucket, mirroring influxql's SetTimeRange(start, end.Add(step)) - and write the re-marshaled JSON via request.SetBody. Do not mutate the shared plan.
4. model/ - modeler over dataset.DataSet (do NOT invent a bespoke Timeseries; only prometheus predates DataSet): - WireUnmarshaler per queryType: timeseries → one series, fields from result keys; groupBy → series keyed by dimension values as tags, values from event; topN → series keyed by the dimension value, one series per distinct dim value (rows per bucket). - WireMarshaler re-emits the native response shape for the queryType (round-trip fidelity for the fields we carry). Cache side uses the stock dataset.MarshalDataSet/UnmarshalDataSet.
5. routes.go - DefaultPathConfigs: - /druid/v2 (exact, POST) → DPC query handler (which itself falls back to OPC/proxy via the canOPC path). - /druid/v2/sql (prefix, POST) → OPC. Requires body-aware cache keying (CacheKeyParams can't see a JSON body - confirm OPC key derivation hashes the body for POST; if not, that's a prerequisite fix). - /druid/v2/sql/task (prefix, POST) → proxy. - /druid/v2/datasources (prefix, GET) → OPC. - /status/health → health; / → proxy.
6. health.go - default health check: GET /status/health (broker/router returns true).
7. observability.go - analysis/rewrite-failure metrics mirroring pkg/backends/clickhouse/observability.go, with stable reason codes for every canOPC/reject path (unsupported-query-type, multi-interval, non-fixed-granularity, context-parse, etc.).
8. Fast-forward: This feature should return ErrFastForwardDisabled-equivalent stub. We may revisit later using a same-query, head-bucket-only interval if warranted, but currently out of scope.
9. Tests: table-driven parse tests (every granularity form, multi-interval, context stripping, each queryType), SetExtent render tests including end-exclusivity, model round-trip tests from captured real Druid responses (wikipedia quickstart datasource), DPC integration test per the clickhouse/influxdb pattern. Maintain >=93% coverage.
10. Docs: docs/supported-origin-types.md (or current equivalent), example config, README provider table.

Explicitly out of scope

• ALB time-series-merge provider support (prometheus-only today).
• Native ingestion/management APIs (/druid/indexer/...) - proxy only, never cached.
• scan/search delta caching. Revisit scan later: it has intervals and ordered raw rows, similar to raw InfluxQL selects, but no granularity - would need the raw-data (step=0) treatment if that path is ever wanted.
• Fast Forward
• Accelerating SQL dialect endpoint