Skip to content

poc-results.md

Latest commit

 

History

History
100 lines (73 loc) · 3.83 KB

poc-results.md

File metadata and controls

100 lines (73 loc) · 3.83 KB
Raw

Proof of Concept: Structured Ontology vs Markdown + RAG

Date: 2026-03-07 Runs: 48 (3 runs × 8 tasks × 2 modes) Judges: GPT-5.4 (primary), Claude Opus 4.5 Judge agreement: 83.3% | Average confidence: 0.927

Setup

Two systems answered the same business reasoning questions about a real company expansion project:

Mode A (Markdown) Mode B (Right Reasons)
Knowledge source Project docs in Markdown files Structured ontology in Dolt via MCP
Access method Full docs loaded into context window SQL queries through MCP tools
Agent Same model, same questions Same model, same questions

Headline results

Metric Mode A (Markdown) Mode B (Right Reasons) Delta
Entity recall 0.514 0.976 +90%
Critical recall 0.572 1.000 +75%
Reasoning score 1.96 / 5 4.33 / 5 +121%
Reasoning variance 1.457 0.472 3× more stable
Latency 284.6s 183.8s -35%
Token usage 418,144 393,872 -6%
Answer length 9,052 chars 4,268 chars -53%

The "why?" question result

The sharpest finding: on "why did we make this decision?" questions, Markdown scored 0% entity recall across all 6 runs. Right Reasons scored 100% across all 6 runs.

This is not a marginal improvement. It is a deterministic failure vs deterministic success.

Markdown-based agents "know about" topics narratively, but cannot trace the explicit links between decisions, goals, and beliefs. The ontology encodes those links as structured relations, making them reliably retrievable.

The surprising tradeoff

Conventional assumption: more tool calls = more latency.

Actual result:

Mode B: 25.9 tool calls, 183.8s latency
Mode A:  9.5 tool calls, 284.6s latency

More tool calls, lower latency. Mode A pays upfront by loading all Markdown into context (418k tokens) and scanning linearly. Mode B pays in small, targeted queries. Per-step reasoning is cheaper even with more steps.

Pairwise comparison

Both judges evaluated answers head-to-head:

Wins
Mode B (Right Reasons) 20
Tie 4
Mode A (Markdown) 0

Mode A won zero head-to-head comparisons.

Task breakdown

Task type Mode A recall Mode B recall Key finding
Implementation context 0.370 0.903 Mode A misses capabilities, integrations, constraints
Impact analysis 0.876 1.000 Most competitive task type for Mode A
"Why?" questions 0.000 1.000 Deterministic failure vs success
Human summaries 0.810 1.000 Mode B is 2× more concise with +48% fact coverage

Hypotheses

Hypothesis Criterion Result Verdict
H1 — Token efficiency ≥20% reduction -6% overall; -24.5% on implementation tasks Partially confirmed
H2 — Accuracy ≥15% recall improvement +90% entity recall, +75% critical recall Confirmed
H3 — Explainability ≥1 point reasoning improvement +121% (1.96 → 4.33) Confirmed
H4 — Readability ≥4/5 readability score 3.79 (GPT-5.4), 4.13 (Opus 4.5) Partially confirmed

Limitations

  • Prompts not optimized. Neither mode used tuned prompts. Mode B results are achieved with minimal instructions.
  • No manual scoring. LLM-as-judge only (2 independent models, high agreement). Human evaluation would strengthen H4.
  • 3 runs per task. Sufficient for pattern detection, but the absolute 0% vs 100% result on "why?" questions should be validated at larger N.
  • Read-only test. Write/update workflows not measured.

Experiment config

{
  "primary_judge": "gpt_5_4",
  "judges": ["claude_opus_4_5", "gpt_5_4"],
  "runs_per_task": 3,
  "tasks_total": 8,
  "systems": ["markdown_context", "dolt_mcp"],
  "temperature": 0,
  "schema_version": "1.1"
}