Detecting Right-Answer, Wrong-Reason Behavior in Open-Weight Reasoning Models

This project implements a complete research pipeline for detecting shortcut-driven reasoning in open-weight language models.

The pipeline evaluates whether LLMs arrive at correct answers through genuine reasoning or through superficial shortcuts, using a combination of behavioral testing and mechanistic interpretability.

---

Architecture

open-models-cot-eval/
├── data/
│   ├── raw/benchmark.jsonl              # 19 problems × 3 conditions = 57 entries
│   ├── processed/<slug>_results.csv     # Raw model outputs (per model)
│   └── labeled/<slug>_labeled.csv       # Auto-labeled with correctness + reasoning quality
├── src/
│   ├── model_utils.py                   # Shared --model slug utility
│   ├── data_loader.py                   # Smoke test script
│   ├── evaluation/
│   │   ├── harness.py                   # Main evaluation harness
│   │   ├── answer_extractor.py          # Regex-based answer extraction
│   │   └── auto_labeler.py              # Automated response labeling
│   ├── analysis/
│   │   ├── behavioral.py                # Core behavioral metrics
│   │   ├── visualizations.py            # Per-model visualization charts
│   │   ├── audit_framework.py           # Weighted scoring rubric
│   │   └── cross_model_comparison.py    # Cross-model comparison charts
│   └── interpretability/
│       ├── activation_extraction.py     # Layer activation comparison
│       ├── sae_analysis.py              # Sparse autoencoder features
│       └── activation_patching.py       # Causal layer intervention
└── results/
    ├── <slug>_audit_scores.json         # Composite scores per model
    ├── research_report.md               # Final research report
    └── figures/
        ├── <slug>/                      # 5 PNGs per model
        └── comparison/                  # 6 cross-model comparison PNGs

---

Setup

Prerequisites

• Python 3.10+
• CUDA-compatible GPU (3GB+ VRAM)
• Git

Installation

git clone https://github.com/YOUR_USERNAME/open-models-cot-eval.git
cd open-models-cot-eval
python -m venv venv
.\venv\Scripts\activate          # Windows
pip install -r requirements.txt

Verify Setup

python src/data_loader.py        # Smoke test — loads model, runs 1 prompt

---

Pipeline Usage

Running the Full Pipeline for a Model

Every script accepts --model to derive a slug. Results never overwrite each other.

# Step 1: Evaluation — run all 57 prompts
python src/evaluation/harness.py --model "Qwen/Qwen2.5-1.5B-Instruct"

# Step 2: Auto-label — extract answers, label correctness/reasoning
python src/evaluation/auto_labeler.py --model "Qwen/Qwen2.5-1.5B-Instruct"

# Step 3: Behavioral analysis — compute metrics, generate condition chart
python src/analysis/behavioral.py --model "Qwen/Qwen2.5-1.5B-Instruct"

# Step 4: Visualizations — generate 4 analysis charts
python src/analysis/visualizations.py --model "Qwen/Qwen2.5-1.5B-Instruct"

# Step 5: Audit — produce weighted composite score
python src/analysis/audit_framework.py --model "Qwen/Qwen2.5-1.5B-Instruct"

Mechanistic Interpretability (run once)

python src/interpretability/activation_extraction.py
python src/interpretability/sae_analysis.py
python src/interpretability/activation_patching.py

Cross-Model Comparison

After running the pipeline for multiple models:

python src/analysis/cross_model_comparison.py

Generates 6 comparison charts in results/figures/comparison/.

---

Models Tested

All four models were run through the complete pipeline (57 prompts each) and evaluated across behavioral and mechanistic dimensions:

Model	Parameters	Architecture	Composite Score	Research Angle
Qwen2.5-1.5B-Instruct	1.5B	Qwen2.5	47.4	Primary — best reasoning in sub-3GB tier
Qwen2.5-0.5B-Instruct	0.5B	Qwen2.5	43.3	Scaling baseline — same arch, 3× smaller
SmolLM-135M-Instruct	135M	SmolLM	43.3	Cross-architecture comparison
TinyLlama-1.1B-Chat-v1.0	1.1B	LLaMA	37.6	Alternative architecture

---

Benchmark Dataset

Location: data/raw/benchmark.jsonl — 19 problems × 3 conditions = 57 entries across 13 cognitive categories.

Field	Description
id	Unique ID: {problem_id}_{condition}
problem_id	Groups 3 conditions of same problem
category	Reasoning type (13 categories)
condition	clean, hinted, or misleading
prompt	Exact text sent to model
correct_answer	Ground truth
misleading_answer	The shortcut/bias answer
cognitive_trap	Name of the cognitive bias tested
difficulty	easy, medium, or hard

---

Results & Cross-Model Comparison

Key Findings

Model	Clean Acc.	Hinted Acc.	Misleading Acc.	Shortcut Rate	Dominant Failure
Qwen 1.5B	15.8%	10.5%	10.5%	4%	Confused (82%)
Qwen 0.5B	5.3%	0.0%	15.8%	11%	Confused (81%)
SmolLM 135M	10.5%	10.5%	5.3%	11%	Confused (81%)
TinyLlama 1.1B	5.3%	0.0%	5.3%	19%	Confused (77%)

Core finding: Sub-2B models fail primarily through confused reasoning, not shortcutting. When they do answer correctly, they show 100% vulnerability to misleading hints — every correct answer was lost when a misleading suggestion was introduced.

Cross-Model Comparison Charts

Summary Dashboard

Accuracy by Condition

Accuracy Heatmap (Category × Condition)

Per-Problem Hint Susceptibility

Answer Flow: Clean → Misleading

Accuracy vs Difficulty

---

Output Artifacts

For each model (identified by slug):

Artifact	Location	Description
Raw results	data/processed/<slug>_results.csv	All 57 model responses
Labeled data	data/labeled/<slug>_labeled.csv	Correctness, shortcuts, reasoning quality
Condition chart	results/figures/<slug>/condition_comparison.png	Accuracy by condition
Heatmap	results/figures/<slug>/accuracy_heatmap.png	Category × Condition
Susceptibility	results/figures/<slug>/susceptibility_chart.png	Per-problem vulnerability
Answer flow	results/figures/<slug>/answer_flow.png	Clean → Misleading transitions
Difficulty curve	results/figures/<slug>/difficulty_curve.png	Accuracy vs difficulty
Audit scores	results/<slug>_audit_scores.json	4-dimension composite score

Audit Score Dimensions

Metric	Weight	Description
Clean Accuracy	0.2	Baseline reasoning ability
Misleading Resistance	0.3	Ability to resist wrong hints
Reasoning Faithfulness	0.3	Quality of reasoning chains
Mechanistic Consistency	0.2	Internal representation stability

---

How to Test an LLM

1. Choose a HuggingFace model ID (e.g., "microsoft/phi-2")
2. Run all 5 pipeline commands with --model "microsoft/phi-2"
3. Run python src/analysis/cross_model_comparison.py (update the MODELS dict in the script to include your new model)
4. Results appear in the slug-based directories automatically

---

References

Core Papers

1. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2024) — "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting." NeurIPS 2023. [paper]
2. Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023) — "Measuring Faithfulness in Chain-of-Thought Reasoning." [paper]

Interpretability Foundations

3. Cunningham, H., Ewart, A., Riggs, L., et al. (2023) — "Sparse Autoencoders Find Highly Interpretable Features in Language Models." ICLR 2024. [paper]
4. Conmy, A., Mavor-Parker, A., Lynch, A., et al. (2023) — "Towards Automated Circuit Discovery for Mechanistic Interpretability." NeurIPS 2023. [paper]

Cognitive Biases in LLMs

5. Hagendorff, T., Fabi, S., & Kosinski, M. (2023) — "Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT." Nature Computational Science. [paper]
6. Jones, E., & Steinhardt, J. (2022) — "Capturing Failures of Large Language Models via Human Cognitive Biases." NeurIPS 2022. [paper]