Merge pull request #1843 from are-ces/lcore-1426-byok-config-refactoring

LCORE-1426: RAG Config refactoring

Author: tisnik

docs/devel_doc/openapi.json

@@ -37,9 +37,36 @@ Both strategies can be enabled independently via the `rag` section of `lightspee
 
 For **runtime-created** vector stores (`POST /v1/vector-stores`), configure
 [`vector_store`](#configure-dynamic-vector-store-providers) instead of
-`byok_rag`. BYOK registers static corpora with a fixed `vector_db_id`; dynamic
+`rag.byok.stores`. BYOK registers static corpora with a fixed `vector_db_id`; dynamic
 providers only declare capacity (provider id, storage, default embeddings).
 
+### Inline RAG chunk flow
+
+```mermaid
+flowchart TD
+    subgraph Sources["Source Fetching"]
+        B1["BYOK Store 1"] --> BPool
+        B2["BYOK Store 2"] --> BPool
+        BN["BYOK Store N"] --> BPool
+        BPool["BYOK Pool\ncapped at rag.byok.max_chunks"]
+        OKP["OKP (Solr)\ncapped at rag.okp.max_chunks"]
+    end
+
+    BPool --> Pool["Merged Pool\n(all chunks, sorted by score)"]
+    OKP --> Pool
+
+    Pool --> Decision{Reranker\nenabled?}
+
+    Decision -->|Yes| Rerank["Cross-Encoder Rerank\n+ BYOK score boost"]
+    Decision -->|No| Cut
+
+    Rerank --> Cut["Top K cut\nrag.retrieval.inline.max_chunks"]
+
+    Cut --> Context["Final Inline RAG Context"]
+```
+
+Each BYOK store is queried in parallel, and the merged BYOK results are capped at `rag.byok.max_chunks` total. OKP fetches up to `rag.okp.max_chunks`. Together these form the reranking pool. If the reranker is enabled, the full pool is reranked with a cross-encoder and BYOK score boosts are applied. The result is capped at `rag.retrieval.inline.max_chunks`.
+
 The **Embedding Model** is used to convert queries and documents into vector representations for similarity matching.
 
 > [!NOTE]
@@ -63,32 +90,28 @@ Use the [`rag-content`](https://github.com/lightspeed-core/rag-content) reposito
 Download a local embedding model such as `sentence-transformers/all-mpnet-base-v2` by using the script in [`rag-content`](https://github.com/lightspeed-core/rag-content) or manually download and place in your desired path.
 
 > [!NOTE]
-> The embedding model can also be downloaded automatically at first start-up (which will be slower). In the `byok_rag` section of `lightspeed-stack.yaml`, specify a supported model name as `embedding_model` instead of a local path. The model will be downloaded to the `~/.cache/huggingface/hub` folder.
+> The embedding model can also be downloaded automatically at first start-up (which will be slower). In the `rag.byok.stores` section of `lightspeed-stack.yaml`, specify a supported model name as `embedding_model` instead of a local path. The model will be downloaded to the `~/.cache/huggingface/hub` folder.
 
 ---
 
 ## Configure BYOK Knowledge Sources
 
-> [!WARNING]
-> **Deprecated in 0.7.0**: The top-level `byok_rag`, `rag`, `okp`, and `reranker` sections
-> are deprecated. In 0.7.0, all RAG-related configuration is unified under a single `rag`
-> section: stores move to `rag.byok.stores` (with `backend` instead of `rag_type`),
-> retrieval strategies move to `rag.retrieval.inline`/`rag.retrieval.tool`, OKP moves to
-> `rag.okp`, and the reranker moves to `rag.retrieval.inline.reranker`.
-> See the [v0.7.0 Migration Guide](migrations/v0.7.0.md) for full details and examples.
 
-BYOK knowledge sources are configured in the `byok_rag` section of `lightspeed-stack.yaml`. The required configuration is automatically generated at startup when using `make run`, `make run-stack`, `docker-compose`, or library mode — no manual enrichment is needed.
+
+BYOK knowledge sources are configured in the `rag.byok.stores` section of `lightspeed-stack.yaml`. The required configuration is automatically generated at startup when using `make run`, `make run-stack`, `docker-compose`, or library mode — no manual enrichment is needed.
 
 ### FAISS example
 
 ```yaml
-byok_rag:
-  - rag_id: custom-index
-    rag_type: inline::faiss
-    embedding_model: sentence-transformers/all-mpnet-base-v2  # or path to local model
-    embedding_dimension: 768
-    vector_db_id: vs_8c94967b-81cc-4028-a294-9cfac6fd9ae2                                    # Generated by rag-content during index creation
-    db_path: <path-to-vector-index>                            # e.g. /home/USER/vector_db/faiss_store.db
+rag:
+  byok:
+    stores:
+      - rag_id: custom-index
+        backend: faiss
+        embedding_model: sentence-transformers/all-mpnet-base-v2  # or path to local model
+        embedding_dimension: 768
+        vector_db_id: vs_8c94967b-81cc-4028-a294-9cfac6fd9ae2                                    # Generated by rag-content during index creation
+        db_path: <path-to-vector-index>                            # e.g. /home/USER/vector_db/faiss_store.db
 ```
 
 Where:
@@ -117,17 +140,19 @@ Each pgvector-backed table follows this schema:
 > The `vector_store_id` (e.g. `rhdocs`) is used to point to the table named `vector_store_rhdocs` in the specified database, which stores the vector embeddings.
 
 ```yaml
-byok_rag:
-  - rag_id: pgvector-example
-    rag_type: remote::pgvector
-    embedding_model: sentence-transformers/all-mpnet-base-v2
-    embedding_dimension: 768
-    vector_db_id: rhdocs  # becomes PostgreSQL table 'vector_store_rhdocs'
-    host: ${env.POSTGRES_HOST}
-    port: ${env.POSTGRES_PORT}
-    db: ${env.POSTGRES_DATABASE}
-    user: ${env.POSTGRES_USER}
-    password: ${env.POSTGRES_PASSWORD}
+rag:
+  byok:
+    stores:
+      - rag_id: pgvector-example
+        backend: pgvector
+        embedding_model: sentence-transformers/all-mpnet-base-v2
+        embedding_dimension: 768
+        vector_db_id: rhdocs  # becomes PostgreSQL table 'vector_store_rhdocs'
+        host: ${env.POSTGRES_HOST}
+        port: ${env.POSTGRES_PORT}
+        db: ${env.POSTGRES_DATABASE}
+        user: ${env.POSTGRES_USER}
+        password: ${env.POSTGRES_PASSWORD}
 ```
 
 > [!NOTE]
@@ -302,21 +327,23 @@ The OKP (Offline Knowledge Portal) Solr Vector IO is a read-only vector search p
 
 ```yaml
 rag:
-  inline:
-    - okp               # inject OKP context before the LLM request
-  tool:
-    - okp               # expose OKP as the file_search tool
-
-okp:
-  rhokp_url: ${env.RH_SERVER_OKP}   # OKP base URL (env var or literal URL)
-  offline: true         # true = use parent_id for source URLs (offline mode)
-                        # false = use reference_url (online mode)
+  retrieval:
+    inline:
+      sources:
+        - okp               # inject OKP context before the LLM request
+    tool:
+      sources:
+        - okp               # expose OKP as the file_search tool
+  okp:
+    rhokp_url: ${env.RH_SERVER_OKP}   # OKP base URL (env var or literal URL)
+    offline: true         # true = use parent_id for source URLs (offline mode)
+                          # false = use reference_url (online mode)
 ```
 
-Set `rhokp_url` to the base URL of your OKP server. Use `${env.RH_SERVER_OKP}` to read the URL from the environment; when omitted or empty, a default from the application constants is used.
+Set `rhokp_url` to the base URL of your OKP server under `rag.okp`. Use `${env.RH_SERVER_OKP}` to read the URL from the environment; when omitted or empty, a default from the application constants is used.
 
 > [!NOTE]
-> When `okp` is listed in `rag.inline` or `rag.tool`, Lightspeed Stack automatically enriches
+> When `okp` is listed in `rag.retrieval.inline.sources` or `rag.retrieval.tool.sources`, Lightspeed Stack automatically enriches
 > the underlying configuration at startup with the required `vector_io` provider and `registered_resources`
 > entries for the OKP vector store. No manual registration is needed.
 
@@ -342,14 +369,15 @@ curl -sX POST http://localhost:8080/v1/query \
 
 **Query Filtering:**
 
-To further filter the OKP context, set the `chunk_filter_query` field in the `okp` section of
+To further filter the OKP context, set the `chunk_filter_query` field in the `rag.okp` section of
 `lightspeed-stack.yaml`. Filters follow the OKP key:value format and are applied as a static
 `fq` parameter on every OKP search request.
 
 ```yaml
-okp:
-  rhokp_url: ${env.RH_SERVER_OKP}
-  chunk_filter_query: "product:*openshift*"
+rag:
+  okp:
+    rhokp_url: ${env.RH_SERVER_OKP}
+    chunk_filter_query: "product:*openshift*"
 ```
 
 Per-request filtering is also available on all inference endpoints via request field **`solr`**: `mode` (`semantic`, `hybrid`, or `lexical`) and `filters` (key:value format). Legacy payloads that omit `mode`/`filters` and send filter key:value pairs at the top level still work with `mode` set to `hybrid`. 
@@ -368,28 +396,24 @@ Example:
 
 **Prerequisites:**
 
-- The OKP server must be running and accessible at the URL given in `okp.rhokp_url` (or `${env.RH_SERVER_OKP}`).
+- The OKP server must be running and accessible at the URL given in `rag.okp.rhokp_url` (or `${env.RH_SERVER_OKP}`).
   For instructions on how to pull and run the OKP image, visit: https://github.com/lightspeed-core/lightspeed-providers/lightspeed_stack_providers/providers/remote/solr_vector_io/solr_vector_io/README.md
 
 
 **Chunk volume:**
 
-> [!WARNING]
-> **Deprecated in 0.7.0**: The chunk limit constants below are replaced by configurable
-> fields in `lightspeed-stack.yaml` (`rag.byok.max_chunks`, `rag.okp.max_chunks`,
-> `rag.retrieval.inline.max_chunks`, `rag.retrieval.tool.max_chunks`).
-> See the [v0.7.0 Migration Guide](migrations/v0.7.0.md) for details.
 
 OKP and BYOK scores are not directly comparable (different scoring systems), so
-`score_multiplier` (a BYOK-only concept) does not apply to OKP results. To control
-the number of retrieved chunks, set the constants in `src/constants.py`:
+`score_multiplier` (a BYOK-only concept) does not apply to OKP results. However, when
+the reranker is enabled, it normalizes scores across sources using a cross-encoder model.
+To control the number of retrieved chunks, configure `max_chunks` in `lightspeed-stack.yaml`:
 
-| Constant | Value | Description |
-|----------|-------|-------------|
-| `INLINE_RAG_MAX_CHUNKS` | 10 | Hard upper bound on the final merged inline RAG chunks (BYOK + OKP) delivered to the LLM |
-| `OKP_RAG_MAX_CHUNKS` | 5 | Fetch hint for OKP (Inline RAG); controls how many chunks enter the reranking pool |
-| `BYOK_RAG_MAX_CHUNKS` | 10 | Fetch hint for BYOK stores (Inline RAG); controls how many chunks enter the reranking pool |
-| `TOOL_RAG_MAX_CHUNKS` | 10 | Max chunks retrieved via Tool RAG (`file_search`); independent from `INLINE_RAG_MAX_CHUNKS` |
+| Config path | Default | Description |
+|-------------|---------|-------------|
+| `rag.retrieval.inline.max_chunks` | 10 | Hard upper bound on the final merged inline RAG chunks (BYOK + OKP) delivered to the LLM |
+| `rag.okp.max_chunks` | 5 | Fetch limit for OKP (Inline RAG); controls how many chunks enter the reranking pool |
+| `rag.byok.max_chunks` | 10 | Fetch limit for BYOK stores (Inline RAG); controls how many chunks enter the reranking pool |
+| `rag.retrieval.tool.max_chunks` | 10 | Max chunks retrieved via Tool RAG (`file_search`); independent from inline max_chunks |
 
 **Limitations:**
 
@@ -399,9 +423,7 @@ the number of retrieved chunks, set the constants in `src/constants.py`:
 
 # Complete Configuration Reference
 
-To enable RAG functionality, configure the `byok_rag` and `rag` sections in
-your `lightspeed-stack.yaml`. Add `vector_store` when you also need
-runtime `POST /v1/vector-stores` capacity.
+To enable RAG functionality, configure the `rag` section (including `rag.byok.stores` and `rag.retrieval`) in your `lightspeed-stack.yaml`. Add `vector_store` when you also need runtime `POST /v1/vector-stores` capacity.
 
 Below is an example of a working `lightspeed-stack.yaml` configuration with:
 
@@ -420,14 +442,6 @@ service:
   port: 8080
   auth_enabled: false
 
-byok_rag:
-  - rag_id: ocp-docs
-    rag_type: inline::faiss
-    embedding_model: sentence-transformers/all-mpnet-base-v2
-    embedding_dimension: 768
-    vector_db_id: vs_3a7f9b2e-45dc-4e1a-b8f2-1c9d0e3f5a6b
-    db_path: /home/USER/lightspeed-stack/vector_dbs/ocp_docs/faiss_store.db
-
 # Optional: capacity for runtime POST /v1/vector-stores (not a static corpus)
 vector_store:
   default_provider: example
@@ -440,18 +454,24 @@ vector_store:
         path: /home/USER/lightspeed-stack/vector_dbs/example/faiss_store.db
 
 rag:
-  inline:
-    - ocp-docs
-  tool:
-    - ocp-docs
+  byok:
+    stores:
+      - rag_id: ocp-docs
+        backend: faiss
+        embedding_model: sentence-transformers/all-mpnet-base-v2
+        embedding_dimension: 768
+        vector_db_id: vs_3a7f9b2e-45dc-4e1a-b8f2-1c9d0e3f5a6b
+        db_path: /home/USER/lightspeed-stack/vector_dbs/ocp_docs/faiss_store.db
+  retrieval:
+    inline:
+      sources:
+        - ocp-docs
+    tool:
+      sources:
+        - ocp-docs
 ```
 
-BYOK providers and registered resources are generated at startup from
-`byok_rag`. Dynamic providers and create defaults are generated from
-`vector_store` during unified synthesis. Embedding models for
-those providers are registered automatically when needed. Inference models
-and providers must still be configured separately (for example in your
-baseline / profile `run.yaml`).
+BYOK providers and registered resources are generated at startup from `rag.byok.stores`. Dynamic providers and create defaults are generated from `vector_store` during unified synthesis. Embedding models for those providers are registered automatically when needed. Inference models and providers must still be configured separately (for example in your baseline / profile `run.yaml`).
 
 ---
 

docs/user_doc/byok_guide.md

@@ -34,40 +34,45 @@ quota_handlers:
   scheduler:
     # scheduler ticks in seconds
     period: 10
-byok_rag:
-  - rag_id: ocp-docs           # referenced in rag.inline / rag.tool
-    rag_type: inline::faiss
-    embedding_model: sentence-transformers/all-mpnet-base-v2
-    embedding_dimension: 768
-    vector_db_id: vs_123       # Vector store ID (from index generation)
-    db_path: /tmp/ocp.faiss
-    score_multiplier: 1.0      # Weight for this vector store's results (Inline RAG only)
-  - rag_id: knowledge-base     # referenced in rag.inline / rag.tool
-    rag_type: inline::faiss
-    embedding_model: sentence-transformers/all-mpnet-base-v2
-    embedding_dimension: 768
-    vector_db_id: vs_456       # Vector store ID (from index generation)
-    db_path: /tmp/kb.faiss
-    score_multiplier: 1.2      # Weight for this vector store's results (Inline RAG only)
-
 # RAG configuration
 rag:
-  # Inline RAG: context injected before the LLM request from the listed sources
-  # List rag_ids from byok_rag, or 'okp' to include OKP
-  inline:
-    - ocp-docs
-    - knowledge-base
-    - okp
-  # Tool RAG: LLM can call file_search on demand to retrieve context
-  # List rag_ids from byok_rag, or 'okp' to include OKP
-  # Omit to disable tool RAG
-  tool:
-    - ocp-docs
-    - knowledge-base
+  byok:
+    max_chunks: 10               # Max total chunks across all BYOK stores
+    stores:
+      - rag_id: ocp-docs         # Referenced in retrieval.inline / retrieval.tool
+        backend: faiss
+        embedding_dimension: 1024
+        vector_db_id: vs_123     # Llama-stack vector_store_id
+        db_path: /tmp/ocp.faiss
+        score_multiplier: 1.0    # Weight for this vector store's results (Inline RAG only)
+      - rag_id: knowledge-base   # Referenced in retrieval.inline / retrieval.tool
+        backend: faiss
+        embedding_dimension: 384
+        vector_db_id: vs_456     # Llama-stack vector_store_id
+        db_path: /tmp/kb.faiss
+        score_multiplier: 1.2    # Weight for this vector store's results (Inline RAG only)
+
+  # OKP provider settings (only used when 'okp' is listed in retrieval sources)
+  okp:
+    offline: true                # true = use parent_id for source URLs, false = use reference_url
+    max_chunks: 5                # Max chunks fetched from OKP
+    # Additional Solr filter query applied to every OKP search request.
+    # Use Solr boolean syntax
+    # chunk_filter_query: "product:*ansible* AND product:*openshift*"
 
-# OKP provider settings (only used when 'okp' is listed in rag.inline or rag.tool)
-okp:
-  offline: true    # true = use parent_id for source URLs, false = use reference_url
-  # Additional Solr filter query applied to every OKP search request.
-  # Use Solr boolean syntax
-  # chunk_filter_query: "product:*ansible* AND product:*openshift*"
+  retrieval:
+    # Inline RAG: context injected before the LLM request from the listed sources
+    # List rag_ids from byok stores, or 'okp' to include OKP
+    inline:
+      sources:
+        - ocp-docs
+        - knowledge-base
+        - okp
+      max_chunks: 10             # Cap on merged inline result
+    # Tool RAG: LLM can call file_search on demand to retrieve context
+    # List rag_ids from byok stores, or 'okp' to include OKP
+    tool:
+      sources:
+        - ocp-docs
+        - knowledge-base
+      max_chunks: 10             # Tool RAG limit

docs/user_doc/rag_guide.md

@@ -33,18 +33,6 @@ conversation_cache:
     ssl_mode: disable
     gss_encmode: disable
 
-#byok_rag:
-#  - rag_id: ocp_docs
-#    rag_type: inline::faiss
-#    embedding_dimension: 1024
-#    vector_db_id: vector_byok_1
-#    db_path: /tmp/ocp.faiss
-#  - rag_id: knowledge_base
-#    rag_type: inline::faiss
-#    embedding_dimension: 384
-#    vector_db_id: vector_byok_2
-#    db_path: /tmp/kb.faiss
-
 quota_handlers:
   sqlite:
       db_path: quota.sqlite

examples/lightspeed-stack-byok-okp-rag.yaml

@@ -24,7 +24,7 @@
     RAGInfoResponse,
     RAGListResponse,
 )
-from models.config import Action, ByokRag
+from models.config import Action, RagStore
 from utils.endpoints import check_configuration_loaded
 
 logger = get_logger(__name__)
@@ -112,7 +112,7 @@ async def rags_endpoint_handler(
         raise HTTPException(**response.model_dump()) from e
 
 
-def _resolve_rag_id_to_vector_db_id(rag_id: str, byok_rags: list[ByokRag]) -> str:
+def _resolve_rag_id_to_vector_db_id(rag_id: str, byok_rags: list[RagStore]) -> str:
     """Resolve a user-facing rag_id to the llama-stack vector_db_id.
 
     Checks if the given ID matches a rag_id in the BYOK config and returns
@@ -178,7 +178,7 @@ async def get_rag_endpoint_handler(
 
     # Resolve user-facing rag_id to llama-stack vector_db_id
     vector_db_id = _resolve_rag_id_to_vector_db_id(
-        rag_id, configuration.configuration.byok_rag
+        rag_id, configuration.configuration.rag.byok.stores
     )
 
     try: