fix(rag): JWKS validation blocks userinfo-based RBAC for Duo access tokens

[sriaradhyula]

Problem

The rag-server's comprehensive RBAC system (merged in #672) is fully implemented — it fetches user identity from the OIDC userinfo endpoint, extracts email and groups, and maps them to roles (admin/ingestor/viewer) with Redis caching. None of this works in prod because JWKS validation fails before userinfo is ever called.

Auth flow in rbac.py (require_authenticated_user):

1. validate_token() via JWKS       ← FAILS for Duo access_token
2. fetch_userinfo() → email/groups ← never reached
3. map groups → role               ← never reached
4. cache in Redis                  ← never reached

Why JWKS fails

Duo issues access tokens signed with a key whose ID (ce989860..., a 64-char hex SHA-256) is not published in Duo's public JWKS for the configured client. This is expected — access tokens are intended for Duo's own resource servers, not for third-party JWKS validation. The rag-server's attempt to validate them via JWKS is architecturally incorrect for this token type.

Consequences in prod (plat-prod-use2-1)

• Every caipe-ui → rag-server REST call returns 401 Unauthorized
• Knowledge Bases tab is completely broken for all users
• The supervisor works only because it uses MCP (no auth header → trusted network grants admin)
• PR fix(ui): skip Bearer token for internal rag-server URLs #1127 worked around this by stripping the auth header entirely for internal URLs — but this bypasses the RBAC system and gives all UI users the same TRUSTED_NETWORK_DEFAULT_ROLE regardless of their actual group membership

Root Cause

validate_token() in auth.py is a hard prerequisite for the userinfo fetch. There is no fallback: if JWKS validation fails, the request is rejected with 401 immediately.

# rbac.py — require_authenticated_user()
provider, access_claims = await auth_manager.validate_token(token)  # hard fail
# ...
userinfo = await auth_manager.fetch_userinfo(token, provider)       # never reached

Correct Fix

In auth.py, when JWKS validation fails, fall back to the userinfo endpoint directly. If the OIDC provider's userinfo endpoint accepts the token and returns valid claims, the token is implicitly valid — the provider has verified it server-side.

async def validate_token(self, token: str) -> Tuple[OIDCProvider, Dict[str, Any]]:
    for provider in self.providers.values():
        try:
            claims = await provider.validate_token(token)
            return provider, claims
        except JWTError:
            # JWKS failed — try userinfo fallback (handles opaque/non-JWKS-validatable tokens)
            try:
                userinfo = await provider.fetch_userinfo(token)
                return provider, userinfo
            except Exception:
                continue
    raise JWTError("Token validation failed for all providers")

With this fix:

• Duo access_token → JWKS fails → userinfo succeeds → email + groups returned
• Group-to-role mapping works: user in caipe-admins → admin, others → viewer
• Per-user RBAC is fully restored
• PR fix(ui): skip Bearer token for internal rag-server URLs #1127 can be reverted (the auth header should be forwarded, not stripped)

Related

• Implement RBAC for RAG Retrieval Chunk Scopes #682 — RBAC for RAG Retrieval Chunk Scopes (depends on per-user identity working)
• feat: RAG RBAC #672 — feat: RAG RBAC (implemented the userinfo + group mapping, blocked by this issue)
• PR fix(ui): skip Bearer token for internal rag-server URLs #1127 — workaround that should be reverted once this is fixed
• PR fix(rag): prioritize JWT auth over trusted network #815 — fix(rag): prioritize JWT auth over trusted network

[Abhishek9639]

Hi @sriaradhyula,
I’d like to work on this issue. I’ll investigate the JWKS validation flow in validate_token() and explore a proper handling strategy for Duo access tokens so that userinfo-based RBAC can work as expected.
Could you please assign this to me?
Thanks