feat: Add per-model URL overrides for NVIDIA embed and rerank models - #18154
feat: Add per-model URL overrides for NVIDIA embed and rerank models#18154mkaaad wants to merge 1 commit into
Conversation
nvidia/embed-qa-4 and two rerank models expose full per-model URLs under ai.api.nvidia.com that the base + suffix assembly cannot reach, so they now carry explicit url overrides in the preset config.
dosubot
Bot
added
size:XS
📝 WalkthroughWalkthroughThe NVIDIA model catalog now includes two reranking models and one embedding model. Each entry defines its model type and NVIDIA provider endpoint. ChangesNVIDIA model catalog
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
conf/models/nvidia.json (1)
185-185: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winKeep NVIDIA embedding endpoints synchronized.
conf/models/nvidia.jsonis used by the Go endpoint resolver, but PythonNvidiaEmbedstill hard-codes the endpoint fornvidia/embed-qa-4. The current URLs match, but future catalog changes will not update Python requests. Propagate the model-specific URL to Python and honor it, or add a consistency test for both mappings. Keep the catalog field because the Go resolver uses it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@conf/models/nvidia.json` at line 185, Update the Python NvidiaEmbed implementation for nvidia/embed-qa-4 to use the model-specific URL from the shared NVIDIA catalog instead of hard-coding the endpoint, while preserving the url field in conf/models/nvidia.json for the Go resolver. Ensure both mappings remain synchronized for future catalog changes.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@conf/models/nvidia.json`:
- Around line 184-188: Add the content_length property with value 512 to the
nvidia/embed-qa-4 model entry alongside its existing model_types metadata,
ensuring the Go model catalog exposes NVIDIA’s maximum input limit.
---
Nitpick comments:
In `@conf/models/nvidia.json`:
- Line 185: Update the Python NvidiaEmbed implementation for nvidia/embed-qa-4
to use the model-specific URL from the shared NVIDIA catalog instead of
hard-coding the endpoint, while preserving the url field in
conf/models/nvidia.json for the Go resolver. Ensure both mappings remain
synchronized for future catalog changes.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: afc3b4b3-ef28-48a6-8c79-97040abbb40a
📒 Files selected for processing (1)
conf/models/nvidia.json
| "name": "nvidia/embed-qa-4", | ||
| "url": "https://ai.api.nvidia.com/v1/retrieval/nvidia/embeddings", | ||
| "model_types": [ | ||
| "embedding" | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- candidate configuration ---'
sed -n '150,205p' conf/models/nvidia.json
printf '%s\n' '--- model-limit references ---'
rg -n -C 3 'max_tokens|max_tokens' conf internal rag | head -n 240
printf '%s\n' '--- NVIDIA adapter/configuration references ---'
rg -n -C 4 'embed-qa-4|nvidia/embeddings|NVIDIA' . -g '*.go' -g '*.py' -g '*.json' | head -n 300Repository: infiniflow/ragflow
Length of output: 31269
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- configuration schema and loader ---'
sed -n '1,135p' conf/model.md
rg -n -C 5 'content_length|model_config|model.*max_tokens|8192' rag api conf internal -g '*.py' -g '*.go' -g '*.json' | head -n 320
printf '%s\n' '--- embedding model construction and call path ---'
sed -n '790,875p' rag/llm/embedding_model.py
rg -n -C 5 'content_length|tokenizer|token.*limit|max_tokens' rag api -g '*.py' | head -n 320Repository: infiniflow/ragflow
Length of output: 48833
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- all runtime references to the new and legacy fields ---'
rg -n -C 4 --glob '*.py' --glob '*.json' --glob '*.go' --glob '*.ts' --glob '*.js' \
'content_length|max_output|max_tokens' . | head -n 500
printf '%s\n' '--- model configuration loading ---'
rg -n -C 6 'conf/models|all_models|model.*config|model_types|model_type' api common rag -g '*.py' | head -n 500
printf '%s\n' '--- embedding truncation helpers and Base implementation ---'
rg -n -C 8 'class Base|def _batched_encode|truncate_to|num_tokens_from_string|def truncate' rag/llm/embedding_model.py common -g '*.py'Repository: infiniflow/ragflow
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- model metadata fields in the owning services ---'
rg -n -C 8 'max_tokens|content_length|max_output' api/db/services/llm_service.py api/db/joint_services/tenant_model_service.py api/db/db_models.py rag/llm/embedding_model.py rag/llm/__init__.py 2>/dev/null || true
printf '%s\n' '--- embedding base implementation ---'
rg -n -C 12 'class Base|def _batched_encode|def encode|truncate_to' rag/llm/embedding_model.py | head -n 260
printf '%s\n' '--- model catalog import/merge paths ---'
rg -n -C 8 'models/.*json|all_models|json.load|model_types' api common tools -g '*.py' | grep -E 'models|all_models|model_types|content_length|max_tokens' | head -n 320Repository: infiniflow/ragflow
Length of output: 45932
🌐 Web query:
NVIDIA NIM nvidia/embed-qa-4 maximum input tokens 512 NV-Embed-QA documentation
💡 Result:
The NVIDIA NIM model nvidia/embed-qa-4 has a maximum input token length of 512 tokens [1][2]. Please note that this specific model has been deprecated in the NVIDIA NIM service, and users are advised to transition to other available models [3]. For current embedding models, such as nvidia/nv-embedqa-e5-v5, the maximum sequence length is also 512 tokens [4][5], whereas other supported models in the NeMo Retriever Embedding NIM suite may support significantly larger token limits, such as 4096 or 8192 tokens [4][6][7]. If your input text exceeds the maximum token length, you can use the truncate parameter (available as START, END, or NONE) to manage how the input is processed [8][9][10].
Citations:
- 1: https://docs.api.nvidia.com/nim/reference/nvidia-embed-qa-4
- 2: https://docs.api.nvidia.com/nim/re/reference/nvidia-embed-qa-4
- 3: https://build.nvidia.com/nvidia/embed-qa-4
- 4: https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/support-matrix.html
- 5: https://docs.api.nvidia.com/nim/re/reference/nvidia-nv-embedqa-e5-v5
- 6: https://docs.nvidia.com/nim/nemo-retriever/text-embedding/2.3.0/support-matrix.html
- 7: https://docs.nvidia.com/nim/nemo-retriever/text-embedding/2.2.0/support-matrix.html
- 8: https://docs.api.nvidia.com/nim/re/reference/nvidia-nv-embedqa-e5-v5-infer
- 9: https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/reference.html
- 10: https://docs.api.nvidia.com/nim/re/reference/nvidia-llama-3_2-nv-embedqa-1b-v1-infer
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- NVIDIA model reconciliation and runtime defaults ---'
sed -n '930,1030p' api/apps/services/provider_api_service.py
sed -n '1088,1120p' api/apps/services/provider_api_service.py
sed -n '270,375p' api/db/joint_services/tenant_model_service.py
printf '%s\n' '--- references to provider model catalogs ---'
rg -n -C 3 'conf/models|models_dir|all_models\.json|provider.*json' . -g '*.py' -g '*.go' -g '*.sh' -g '*.yml' -g '*.yaml' | head -n 320
printf '%s\n' '--- deterministic configuration/runtime probe ---'
python3 - <<'PY'
import json
from pathlib import Path
catalog = json.loads(Path("conf/models/nvidia.json").read_text())
entry = next(m for m in catalog["models"] if m["name"] == "nvidia/embed-qa-4")
print("catalog_entry_keys:", sorted(entry))
print("catalog_max_tokens:", entry.get("max_tokens"))
print("catalog_content_length:", entry.get("content_length"))
source = Path("api/apps/services/provider_api_service.py").read_text()
print("nvidia_merge_defaults_to_8192:", 'model["max_tokens"] = _to_int(model.get("max_tokens"), 8192)' in source)
embed = Path("rag/llm/embedding_model.py").read_text()
start, end = embed.index("class NvidiaEmbed"), embed.index("class LmStudioEmbed")
nvidia = embed[start:end]
print("nvidia_client_truncation:", '"truncate": "END"' in nvidia)
print("nvidia_client_side_truncate_to:", "truncate_to=" in nvidia)
print("nvidia_model_name_remapped:", '"NV-Embed-QA"' in nvidia)
PYRepository: infiniflow/ragflow
Length of output: 36469
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- Go model metadata and context-limit resolution ---'
rg -n -C 8 'ContentLength|content_length|MaxTokens|max_tokens|ResolveModelContextLength|truncate' internal/entity internal/service -g '*.go' | head -n 420
printf '%s\n' '--- NVIDIA Go driver ---'
sed -n '1,240p' internal/entity/models/nvidia.go
printf '%s\n' '--- provider model configuration types ---'
rg -n -C 8 'type .*Model|ModelConfig|ProviderConfig|ContentLength|MaxOutput' internal/entity/models -g '*.go' | head -n 360Repository: infiniflow/ragflow
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- provider model structs and JSON decoding ---'
rg -n -C 12 'type Model struct|type ListModelResponse|ContentLength|MaxOutput|MaxTokens' internal/entity/models/*.go | head -n 260
printf '%s\n' '--- embedding model construction ---'
rg -n -C 12 'GetEmbeddingModel|EmbeddingModel\\{|MaxTokens:' internal -g '*.go' | head -n 320
printf '%s\n' '--- context resolver implementation ---'
sed -n '3588,3655p' internal/service/model_service.goRepository: infiniflow/ragflow
Length of output: 21940
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- exact embedding model type and construction ---'
rg -n -C 10 'type EmbeddingModel|EmbeddingModel struct|MaxTokens int|MaxTokens:' internal -g '*.go' | head -n 260
printf '%s\n' '--- embedding model provider methods ---'
rg -n -C 10 'GetEmbeddingModel|EmbeddingModel' internal/service internal/dao internal -g '*.go' | head -n 360
printf '%s\n' '--- catalog model type definitions ---'
rg -n -C 8 'ContentLength.*json|MaxOutput.*json|ListModelResponse struct|Model struct' internal/entity/models internal/dao -g '*.go' | head -n 300Repository: infiniflow/ragflow
Length of output: 50376
Declare the provider input limit.
NVIDIA documents nvidia/embed-qa-4 with a 512-token maximum input. Add "content_length": 512 so the Go model catalog exposes this limit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@conf/models/nvidia.json` around lines 184 - 188, Add the content_length
property with value 512 to the nvidia/embed-qa-4 model entry alongside its
existing model_types metadata, ensuring the Go model catalog exposes NVIDIA’s
maximum input limit.
Source: Coding guidelines
1 participant
No description provided.