Contextual Precision over-penalizes overlapping chunks in financial-document RAG

[Ruthwik-Data]

Bug Summary

When evaluating a financial‑doc RAG pipeline, ContextualPrecisionMetric treats near‑duplicate overlapping chunks as distinct low‑quality retrievals. This under‑reports retrieval quality for pipelines that use overlap to preserve context around table/section boundaries — a standard practice for dense financial documents.

The core issue: increasing chunk overlap from 0% to 10–20% (which improves answer quality) makes the eval score worse — not because retrieval degraded, but because the metric doesn’t consolidate overlapping chunks that cover the same relevant span.

Redundancy ≠ Irrelevance.

Pipeline Setup

• Domain: Public‑company filings (10‑K, 10‑Q, earnings transcripts)
• Chunking: Fixed‑size ~512–1024 tokens, 10–20% overlap (to avoid cutting tables/footnotes at boundaries), via LangExtract
• Embeddings: Ollama local embeddings (nomic-embed-text)
• Vector store: Supabase + pgvector (top‑k retrieval, no dedup, k=5–10)
• Eval LLM: Ollama (local — llama3 / deepseek-r1)
• Orchestration: n8n → Supabase → dbt (analytics layer)
• Evaluation: DeepEval RAG metrics comparing input, retrieval_context, actual_output, expected_output

Reproduction

Query: “What was total revenue for FY 2023, and how did it change YoY?”

Retrieved chunks (k=5):

• Chunk 1: Revenue table from Consolidated Statements of Operations
• Chunk 2: Overlapping — repeats revenue rows + a few lines above
• Chunk 3: Overlapping — same rows + footnotes below
• Chunks 4–5: MD&A narrative explaining YoY revenue change

Result: Answer is factually correct, fully grounded, uses only the correct filing. But context_precision reports ~0.3–0.5 instead of near 1.0.

Why:

• Each overlapping chunk that doesn’t add unique coverage gets scored as a separate “miss”
• Multiple windows over the same relevant span are penalized as if they were off‑topic retrievals

from deepeval import evaluate
from deepeval.metrics import ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import DeepEvalBaseLLM
from langchain_openai import ChatOpenAI

class OllamaLocal(DeepEvalBaseLLM):
    def __init__(self, model_name="llama3", base_url="http://localhost:11434/v1/"):
        self.model = ChatOpenAI(
            model_name=model_name,
            openai_api_key="ollama",
            base_url=base_url,
            temperature=0,
        )

    def generate(self, prompt):
        return self.model.invoke(prompt).content

    def get_model_name(self):
        return "ollama-local"

    async def a_generate(self, prompt):
        return self.generate(prompt)

local_model = OllamaLocal()

test_case = LLMTestCase(
    input="What was total revenue for FY 2023, and how did it change year-over-year?",
    actual_output="Total revenue was $12,483.2M, up 13.4% year-over-year.",
    expected_output="In FY 2023, total revenue was $12,483.2M, up 13.4% compared to prior year.",
    retrieval_context=[
        "Chunk 1: Net revenues: $12,483.2M (FY2023), $11,009.7M (FY2022), $9,204.1M (FY2021). Product revenue: $7,841.0M. Service revenue: $4,642.2M.",
        "Chunk 2: Total net revenues $12,483.2M, $11,009.7M, $9,204.1M. Cost of revenues: $5,214.3M. Gross profit: $7,268.9M. [Footnote 3: Revenue recognized under ASC 606]",
        "Chunk 3: [Footnote 3] Revenue is recognized when performance obligations are satisfied. Variable consideration is estimated using the expected value method.",
        "Chunk 4: Revenue increased 13.4% year-over-year, driven by a 15.2% increase in product revenue and 10.8% growth in services. North America contributed 61% of total revenue.",
        "Chunk 5: Management attributes YoY growth to enterprise expansion and new product launches in Q3 FY2023. FX headwinds reduced reported revenue by approximately $340M.",
    ],
)

metric = ContextualPrecisionMetric(threshold=0.7, model=local_model)
result = evaluate([test_case], [metric])
# Score: ~0.3–0.5 despite all chunks being from the correct section```

***

## Expected Behavior  
When all retrieved chunks come from the correct document section and the answer is fully grounded, overlapping chunks should not drag precision down the same way genuinely irrelevant chunks do.  


## Possible Approaches  
1. **Chunk similarity consolidation** — optionally cluster near‑duplicate chunks (by text similarity or shared span coverage) and treat them as a single retrieval unit for precision calculation.  
2. **Configurable redundancy tolerance** — a parameter like `deduplicate_context=True` or `max_redundant_hits=N` so users can tell the metric not to penalize overlapping windows from the same section.  
3. **Doc‑aware precision** — if all top‑k chunks are from the same document + section containing the answer span, weight precision accordingly rather than strict per‑chunk scoring.  

If the current per‑chunk behavior is intentional, guidance in the docs on recommended chunking strategies for DeepEval’s metrics (especially for dense domains like financial filings) would help a lot.  

Happy to provide a full minimal repro with synthetic 10‑K text and controlled overlap levels.  

***

## Environment  
- **deepeval:** 3.9.5  
- **Python:** 3.14.3  
- **OS:** macOS  
- **Vector store:** Supabase + pgvector  
- **Eval LLM:** Ollama (llama3 / deepseek‑r1 via `http://localhost:11434/v1/`)  
***

[Whatsonyourmind]

The core issue is that the precision metric treats each chunk as an independent Bernoulli trial (relevant/irrelevant), but overlapping chunks violate the independence assumption. When chunks share >50% content, they are effectively the same retrieval counted multiple times, inflating the denominator without adding to the numerator.

Mathematical framing: Contextual precision is precision@k = |relevant_chunks| / k. With overlapping chunking, you might retrieve 3 chunks that all cover the same relevant span. Correct precision = 1.0 (you found the answer). Reported precision = 1/3 = 0.33 (only one chunk is labeled relevant since the others are redundant).

A principled fix — coverage-based deduplication before scoring:

1. Compute pairwise text overlap between all retrieved chunks (Jaccard similarity on token n-grams, or character-level overlap ratio)
2. Cluster chunks where overlap > threshold (e.g., 0.5)
3. For each cluster, count as a single retrieval unit with relevance = max(relevance of any chunk in cluster)
4. Compute precision on the deduplicated set

In your example, chunks 1-3 would cluster together (all from the revenue table), giving 3 retrieval units: {revenue cluster, chunk 4, chunk 5}. If all are relevant, precision = 3/3 = 1.0.

Immediate workaround (preprocessing before evaluation):

from difflib import SequenceMatcher

def dedup_context(chunks, threshold=0.5):
    clusters = []
    for chunk in chunks:
        merged = False
        for cluster in clusters:
            if any(SequenceMatcher(None, chunk, c).ratio() > threshold for c in cluster):
                cluster.append(chunk)
                merged = True
                break
        if not merged:
            clusters.append([chunk])
    return [max(c, key=len) for c in clusters]

test_case.retrieval_context = dedup_context(test_case.retrieval_context)

The overlap threshold should be configurable — for financial docs with dense tables, 10-20% overlap between chunks is standard practice to avoid cutting tables at boundaries.

[Ruthwik-Data]

Thanks a lot for the detailed explanation and the dedup_context suggestion – the Bernoulli framing is spot on, and clustering near‑duplicate chunks definitely helps in many cases. I think your solution is partially correct, but it doesn’t fully capture the failure mode I’m running into with overlapping financial‑doc chunks.

The core issue is that we’re still measuring precision purely at the chunk level, while the pipeline is effectively operating at the span/section level. In realistic 10‑K setups, I can have several overlapping chunks that all correctly cover the same answer‑supporting region (same table rows / same MD&A paragraph). Even if I cluster on raw text similarity, this can still misbehave when:

• chunks share the same answer span but are not highly similar (e.g., table vs nearby narrative), or
• I want to treat “same doc section + overlapping coverage of the answer span” as a single retrieval unit, regardless of minor textual differences.

To make the problem more concrete, here’s a minimal sketch of the current behavior vs what I think we actually want:

from deepeval import evaluate
from deepeval.metrics import ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase

# Simplified example: 3 overlapping chunks around the same answer span
retrieval_context = [
    "Chunk 1: Total revenue FY2023: $12,483.2M. FY2022: $11,009.7M.",
    "Chunk 2: FY2023 total revenue was $12,483.2M (13.4% YoY growth).",
    "Chunk 3: Revenue increased 13.4% YoY, driven by product growth.",
]

test_case = LLMTestCase(
    input="What was total revenue for FY 2023, and how did it change YoY?",
    actual_output="Total revenue was $12,483.2M, up 13.4% year-over-year.",
    expected_output="In FY 2023, total revenue was $12,483.2M, up 13.4% compared to prior year.",
    retrieval_context=retrieval_context,
)

metric = ContextualPrecisionMetric(threshold=0.7, model="gpt-4.1")
result = evaluate([test_case], [metric])

print("Contextual precision (raw chunks):", metric.score)
# In practice, this often ends up closer to ~0.3–0.5 instead of ~1.0,
# because only 1 chunk is labeled "relevant" and the others are treated as redundant.

From a RAG‑user perspective, all 3 chunks above are good retrievals – they’re just multiple windows around the same answer span. But the metric still behaves like:

• relevant_chunks = 1
• k = 3
• precision@k ≈ 0.33

So even with deduplication based purely on text similarity, we’re still fundamentally counting “how many chunks” instead of “how many answer‑supporting regions” we retrieved.

What I’ve been experimenting with is a more coverage‑aware / section‑aware variant, where we collapse chunks by the region they cover before scoring. Very roughly:

from collections import defaultdict
from typing import List, Dict

class RetrievedChunk:
    def __init__(self, text: str, doc_id: str, section_id: str, covers_answer: bool):
        self.text = text
        self.doc_id = doc_id          # e.g. filing ID
        self.section_id = section_id  # e.g. "10-K:Consolidated_Statements_Of_Operations"
        self.covers_answer = covers_answer  # ground-truth label for eval

def cluster_by_section(chunks: List[RetrievedChunk]) -> List[List[RetrievedChunk]]:
    clusters: Dict[tuple, List[RetrievedChunk]] = defaultdict(list)
    for ch in chunks:
        key = (ch.doc_id, ch.section_id)
        clusters[key].append(ch)
    return list(clusters.values())

def section_level_precision(chunks: List[RetrievedChunk]) -> float:
    clusters = cluster_by_section(chunks)
    # A cluster is relevant if ANY chunk in it covers the answer span
    relevant_clusters = sum(
        1 for c in clusters if any(ch.covers_answer for ch in c)
    )
    k = len(clusters)
    return relevant_clusters / k if k > 0 else 0.0

In this framing:

• We treat “all overlapping windows around the same revenue table / MD&A section” as one retrieval unit, not 3 separate Bernoulli trials.
• Precision is computed over these units, so if the model retrieved three overlapping windows from the correct section and nothing irrelevant, the score is 1.0, which matches the actual retrieval quality for this domain.

Your clustering idea is very close to this, but I think the key is to let users define the clustering dimension (e.g., (doc_id, section_id) or span IDs) rather than only clustering on raw text similarity. That would make it possible to plug domain‑aware signals (like “same table section”) into ContextualPrecision without building a whole custom metric from scratch.

[Whatsonyourmind]

@Ruthwik-Data You're right — text similarity clustering is the wrong abstraction for this. The correct unit of retrieval is the section (or span), not the chunk. Your cluster_by_section approach is the cleaner formulation.

The mathematical distinction is important:

• Text dedup (my suggestion): clusters on sim(chunk_i, chunk_j) > threshold. Fails when chunks cover the same answer span but have different surface forms (table vs. narrative in your example).
• Section-level (your suggestion): clusters on (doc_id, section_id). Captures the semantic unit regardless of text similarity. Correct for structured docs like 10-Ks.

Your framing generalizes cleanly. The metric becomes:

section_precision@k = |relevant_sections| / |retrieved_sections|

where a section is relevant if ANY chunk in it covers the answer span. This is equivalent to computing precision on the image of the retrieval function under the section mapping, rather than on the raw output.

The key API design insight: the clustering dimension should be user-defined because it's domain-dependent. For financial docs it's (doc_id, section_id). For legal docs it might be (contract_id, clause_number). For code it might be (file, function_name). The metric framework should accept an optional group_by function that maps chunks to retrieval units:

metric = ContextualPrecisionMetric(
    threshold=0.7,
    model="gpt-4.1",
    group_by=lambda chunk: (chunk.metadata["doc_id"], chunk.metadata["section_id"])
)

With no group_by, it falls back to treating each chunk independently (current behavior). With group_by, it clusters first, then computes precision on the clustered set. This is backward-compatible and handles your case correctly.

One edge case to handle: when chunks from DIFFERENT sections both cover the answer (e.g., revenue is mentioned in both the financial statements and the MD&A), those should remain separate retrieval units — both are genuinely relevant, independently retrieved.

[Ruthwik-Data]

I think we now have a pretty crisp formulation of both the underlying problem and a concrete direction for a fix.

To summarize the proposed solution for the DeepEval team:

1. The real retrieval unit in many RAG setups (especially financial 10‑Ks) is a section/span, not an individual chunk. Overlapping windows around the same table or paragraph are multiple views of the same retrieval decision, and treating them as independent Bernoulli trials is what makes ContextualPrecision under‑report quality.

2. The metric should therefore compute precision over grouped retrieval units, where a unit is defined by a domain‑specific key such as (doc_id, section_id) for finance, (contract_id, clause_number) for legal, (file, function_name) for code, etc.

3. API‑wise, this falls out naturally as an optional group_by hook on the metric:

metric = ContextualPrecisionMetric(
    threshold=0.7,
    model="gpt-4.1",
    group_by=lambda chunk: (chunk.metadata["doc_id"], chunk.metadata["section_id"])
)

• If group_by is not provided, the metric behaves exactly as it does today (per‑chunk precision).
• If group_by is provided, the metric first clusters chunks by group_by(chunk), marks each cluster as relevant if any member supports the answer span, and then computes precision on these clusters (sections) instead of raw chunks.

This effectively turns ContextualPrecision into:

section_precision@k = |relevant_sections| / |retrieved_sections|

with the desired behavior for overlapping‑chunk RAG: three windows around the same revenue table count as one highly relevant retrieval unit, not “1 relevant + 2 redundant errors”. Different sections that both genuinely support the answer remain separate units and can both be relevant.

I’m happy to help prototype a group_by implementation (plus tests using the overlapping 10‑K example) if the DeepEval team is open to exploring this direction.

[Whatsonyourmind]

@Ruthwik-Data That's an excellent summary — clean formulation that the DeepEval team can act on.

One implementation detail worth calling out: the group_by parameter should probably default to None (current behavior) so it's fully backwards-compatible. When set to "section", the grouping logic only needs to run in the precision denominator — recall is unaffected since it's anchored to the expected output, not the retrieval set.

For the section detection heuristic, a simple approach that works well on 10-Ks: if consecutive chunks share >60% token overlap (measured by Jaccard on whitespace-tokenized sets), they belong to the same section. This avoids needing structural metadata from the chunking pipeline while still catching the table/paragraph duplication pattern you described.

Looking forward to seeing this land.