Skip to content

Commit 980faef

Browse files
authored
Redis-backed session cache for cross-replica model affinity (#879)
* add pluggable session cache with Redis backend * add Redis session affinity demos (Docker Compose and Kubernetes) * address PR review feedback on session cache * document Redis session cache backend for model affinity * sync rendered config reference with session_cache addition * add tenant-scoped Redis session cache keys and remove dead log_affinity_hit - Add tenant_header to SessionCacheConfig; when set, cache keys are scoped as plano:affinity:{tenant_id}:{session_id} for multi-tenant isolation - Thread tenant_id through RouterService, routing_service, and llm handlers - Use Cow<'_, str> in session_key to avoid allocation when no tenant is set - Remove unused log_affinity_hit (logging was already inlined at call sites) * remove session_affinity_redis and session_affinity_redis_k8s demos
1 parent 128059e commit 980faef

15 files changed

Lines changed: 1534 additions & 725 deletions

File tree

config/plano_config_schema.yaml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -441,6 +441,25 @@ properties:
441441
minimum: 1
442442
maximum: 10000
443443
description: Maximum number of session-pinned routing cache entries. Default 10000.
444+
session_cache:
445+
type: object
446+
properties:
447+
type:
448+
type: string
449+
enum:
450+
- memory
451+
- redis
452+
default: memory
453+
description: Session cache backend. "memory" (default) is in-process; "redis" is shared across replicas.
454+
url:
455+
type: string
456+
description: Redis URL, e.g. redis://localhost:6379. Required when type is redis.
457+
tenant_header:
458+
type: string
459+
description: >
460+
Optional HTTP header name whose value is used as a tenant prefix in the cache key.
461+
When set, keys are scoped as plano:affinity:{tenant_id}:{session_id}.
462+
additionalProperties: false
444463
additionalProperties: false
445464
state_storage:
446465
type: object

0 commit comments

Comments
 (0)