RAGProof 1.0.0 — the first public release.
A test harness that scores any RAG pipeline on retrieval quality, groundedness,
citation accuracy and prompt-injection resistance, and fails CI when quality
regresses.
Highlights
- Deterministic-first metric families: retrieval (precision/recall/MRR/nDCG),
generation (groundedness, citation validity/support, relevance, completeness),
robustness (injection resistance, abstention, overrefusal). - Provider-agnostic LLM judge (OpenRouter/OpenAI/Ollama/Anthropic), calibrated
against human-scored fixtures, with a content-addressed response cache. - CI quality gate with per-metric thresholds, bootstrap confidence intervals so
judge noise does not fail builds, JUnit output, and a distinct exit-code
contract (0 pass / 1 gate fail / 2 execution error / 3 config error). - Dataset generation from a corpus with answerability verification, plus
hash-verified frozen datasets. - A local, read-only dashboard control panel (
pip install 'ragproof[ui]'). - Reusable GitHub Action and a Dockerfile.
Proven on a real production RAG system. Run against DOC-007-AI over a
100-case dataset: groundedness 0.997, citation support 1.000, and a real,
gate-failing prompt-injection finding. See the case study in the README.
Install
pip install ragproof
Verified: 256 tests on a 3-OS x 3-Python matrix, mypy --strict clean.