domain: ai-interpretability requires:
- to: cognitive-architecture
Large language models already affect the daily lives of billions, yet no one knows what happens inside them. This is not a matter of academic curiosity but a safety problem.
| Problem | Current state | How interpretability addresses it |
|---|---|---|
| Hallucination detection | Post-hoc verification (slow) | Detect hallucination circuits in advance, blocking before generation |
| Alignment verification | Behavioral tests only | Direct inspection of internal representations to confirm intent |
| Bias audit | Statistical estimation | Pinpoint exact location of bias-encoding neurons/circuits |
| Adversarial attack | Patch after the fact | Proactively identify and harden vulnerable circuits |
| Regulatory compliance | "Black box" excuse | Trace decision paths to realize explainable AI |
Why this matters to Anthropic: Anthropic's core mission is building safe AI. Interpretability is the only path to answering "is the model safe?" with structural evidence rather than behavioral tests. Recent work showing that features extracted by Sparse Autoencoders (SAEs) correspond to conceptual representations inside the model (Bricken et al. 2023, Cunningham et al. 2023) demonstrated that this direction is feasible.
Scientific value: Just as neuroscience decodes neuronal activity in the brain, AI interpretability decodes the "thought process" of artificial neural networks. The representational structures discovered in this process provide new insight into the nature of cognitive science, linguistics, and mathematical reasoning.
Without opening the model's interior, safety cannot be guaranteed -- interpretability is both a required tool for AI safety and a new microscope for scientific discovery.
+------------------------------------------------------------------+
| [Feature extraction methods] |
+------------------------------------------------------------------+
| Black-box probing ████████░░░░░░░░░░░░░░░░░░░░ indirect, no causality |
| Linear probe ████████████░░░░░░░░░░░░░░░░ direction yes, structure no |
| Sparse Autoencoder ████████████████████████████ causal, interpretable |
| |
| [Circuit discovery methods] |
+------------------------------------------------------------------+
| Manual tracing (ZOO) ██████░░░░░░░░░░░░░░░░░░░░░░ months, 1 circuit |
| Activation patching ██████████████░░░░░░░░░░░░░░░ days, partial |
| Automated pipeline ████████████████████████████ hours, systematic |
| |
| [Tool maturity] |
+------------------------------------------------------------------+
| Ad-hoc scripts ████░░░░░░░░░░░░░░░░░░░░░░░░ not reproducible |
| TransformerLens ████████████████░░░░░░░░░░░░ good but circuit-limited |
| Integrated platform ████████████████████████████ target state |
+------------------------------------------------------------------+
+--------------------+-----------------------------+--------------------------+
| Barrier | Current limit | Proposed solution |
+--------------------+-----------------------------+--------------------------+
| SAE scaling | Small models only (GPU mem) | Distributed SAE + progressive discovery |
| Feature validation | Human manual labeling | Automated labeling pipeline |
| Circuit complexity | Only simple circuits | Hierarchical abstraction |
| Cross-model compare| Restart per model | Pre-built feature correspondence |
| Reproducibility | Code/env mismatches | Benchmark + verification tools |
+--------------------+-----------------------------+--------------------------+
| Category | Item | Level | Note |
|---|---|---|---|
| Programming | Python, PyTorch/JAX | Proficient | SAE implementation and training |
| Math | Linear algebra, information theory | Intermediate+ | SVD, mutual information, KL divergence |
| ML theory | Transformer architecture | Deep | Attention, MLP, residual stream structure |
| Tools | TransformerLens, SAELens | Experienced | Neel Nanda tool ecosystem |
| Infrastructure | GPU cluster access | Required | A100/H100 needed for large SAE training |
| Domain | Cognitive science, linguistics basics | Supplementary | Useful for feature interpretation |
cognitive-architecture --> Representation theory basics
ai-alignment --> Safety alignment target definition
ai-adversarial --> Adversarial robustness evaluation
+======================================================================+
| AI Interpretability Research Program |
+======================================================================+
| |
| +------------------+ +------------------+ +-------------------+ |
| | Axis A: SAE | | Axis B: Circuit | | Axis C: Tooling | |
| | improvement | | discovery | | | |
| | (15 ideas) | | (12 ideas) | | (12 ideas) | |
| +--------+---------+ +--------+---------+ +---------+---------+ |
| | | | |
| v v v |
| +--------+---------+ +--------+---------+ +---------+---------+ |
| | Feature extract/ | | Circuit trace/ | | Visualization/ | |
| | decompose | | mapping | | automation | |
| | Scaling/compress | | Safety circuit | | Benchmark/verify | |
| | Auto labeling | | Cross-domain | | Reproducibility | |
| +--------+---------+ +--------+---------+ +---------+---------+ |
| | | | |
| +----------+-----------+----------+-----------+ |
| | | |
| v v |
| +--------+--------+ +--------+--------+ |
| | Unified feature | | Interpretability| |
| | dictionary | | report | |
| | (per-model DB) | | (safety audit) | |
| +-----------------+ +-----------------+ |
+======================================================================+
Model activations --> SAE training --> Feature dict --> Circuit trace --> Safety report
^ | | | |
| v v v v
+--- Axis C tools support the entire pipeline (viz, automation, verify) +
+----------+ +-----------+ +-----------+ +-----------+ +-----------+
| 1.Collect|-->| 2.Extract |-->| 3.Trace |-->| 4.Verify |-->| 5.Paper |
| Activations | SAE feats | | Circuit | | Causal | | Results |
| | | | | mapping | | intervene | | |
+----------+ +-----------+ +-----------+ +-----------+ +-----------+
| | | | |
v v v v v
Model infer Latent space Graph struct Ablate/amplify Repro code
Mid-activs Sparse codes Nodes/edges Counterfactual Benchmark
Per-layer Dict learning Weight attrib Stat tests Visualization
Hypothesis --> SAE feature search --> Circuit candidate ID
^ |
| v
+--- Falsification <--- Causal intervention -+
(Revise hypothesis on failure)
- Reproduce existing SAE results (Bricken et al. 2023 baseline)
- Master TransformerLens/SAELens
- Build feature-extraction pipeline on small models (GPT-2, Pythia)
- Adapt to Anthropic internal tools/infrastructure
- Deliverables: Reproduction report + tool usage guide
- Hierarchical latent-dimension SAE (Idea #1) prototype
- Task-conditional SAE (Idea #7) experiments
- Arithmetic circuit atlas (Idea #16) initial mapping
- Hallucination circuit detection (Idea #19) pilot
- Deliverables: Two internal technical reports
- Cross-layer SAE correlation analysis (Idea #5)
- Automated circuit-discovery pipeline (Idea #33)
- Safety-refusal circuit mapping (Idea #23)
- Deception circuit detection (Idea #24)
- Deliverables: Automation tool v1 + safety-related findings
- Synthesize results and draft a paper
- Reproducibility verification tool (Idea #39) draft completion
- Integrate with Anthropic team + propose follow-up directions
- Deliverables: One academic paper (or technical report + blog post)
"""
AI Interpretability Research Program verification code
======================================================
Verifies SAE math, circuit-analysis concepts, and information-theoretic limits.
Uses only the Python standard library.
"""
import math
import random
import statistics
from collections import Counter
from fractions import Fraction
PASS = 0
FAIL = 0
TOTAL_SECTIONS = 11 # 7.0 ~ 7.10
def check(name: str, condition: bool, detail: str = ""):
global PASS, FAIL
tag = "PASS" if condition else "FAIL"
if condition:
PASS += 1
else:
FAIL += 1
print(f" [{tag}] {name}" + (f" -- {detail}" if detail else ""))
# ── 7.0 CONSTANTS: Info-theoretic SAE hyperparameters ─────────────
print("\n=== 7.0 CONSTANTS: SAE hyperparameter derivation ===")
# Model activation dimension (e.g., Transformer residual stream)
d_model = 512
# SAE latent dimension: information-theoretic principle
# Overcomplete representation of activation space required
# Empirically, latent dim = 4~16x model dim is effective
# Theoretical basis: when decomposing d_model activations on a sparse basis,
# optimal feature count ~ k * d_model at overcomplete factor k
overcomplete_factor = 8
d_latent = d_model * overcomplete_factor # 4096
# Sparsity penalty lambda: L1 regularization coefficient
# Info-theoretic derivation: limiting average active features per input
# to sqrt(d_latent) maximizes information per feature
# lambda ~ 1 / sqrt(d_latent) * baseline_loss_scale
target_active = math.isqrt(d_latent) # 64 features active simultaneously
sparsity_lambda = 1.0 / target_active # ~0.015625
# Learning rate: SAE training stability condition
# Adam-based, derived from gradient-norm estimate
# lr ~ 1 / sqrt(d_latent) (matches Xavier init scale)
learning_rate = 1.0 / math.sqrt(d_latent) # ~0.015625
check("Latent dim is overcomplete", d_latent > d_model,
f"d_latent={d_latent} > d_model={d_model}")
check("Sparsity target reasonable", 1 < target_active < d_latent,
f"avg active features {target_active} ({target_active/d_latent:.1%} of {d_latent})")
check("Learning rate range", 1e-4 < learning_rate < 1.0,
f"lr={learning_rate:.6f}")
check("Overcomplete ratio", overcomplete_factor >= 4,
f"overcomplete={overcomplete_factor}x (minimum 4x required)")
# ── 7.1 DIMENSIONS: SAE loss-function dimension consistency ──────
print("\n=== 7.1 DIMENSIONS: Loss-function dimension verification ===")
# SAE loss = reconstruction loss + sparsity loss
# L = ||x - x_hat||^2 + lambda * ||z||_1
# Where:
# x: input activation vector (d_model,)
# z: latent representation (d_latent,)
# x_hat = W_dec @ z + b_dec: reconstruction (d_model,)
# W_enc: (d_latent, d_model), W_dec: (d_model, d_latent)
# Dimension verification
W_enc_shape = (d_latent, d_model) # encoder
W_dec_shape = (d_model, d_latent) # decoder
b_enc_shape = (d_latent,) # encoder bias
b_dec_shape = (d_model,) # decoder bias
# z = ReLU(W_enc @ x + b_enc)
z_shape = W_enc_shape[0] # d_latent
check("Encoder output dim", z_shape == d_latent,
f"z: ({z_shape},) == ({d_latent},)")
# x_hat = W_dec @ z + b_dec
x_hat_dim = W_dec_shape[0] # d_model
check("Decoder output dim", x_hat_dim == d_model,
f"x_hat: ({x_hat_dim},) == ({d_model},)")
# Reconstruction loss dim: ||x - x_hat||^2 -> scalar
recon_loss_dim = "scalar" # squared L2 norm -> scalar
check("Reconstruction loss is scalar", recon_loss_dim == "scalar",
"||x - x_hat||^2 -> R^d -> R (scalar)")
# Sparsity loss dim: ||z||_1 -> scalar
sparsity_loss_dim = "scalar" # L1 norm -> scalar
check("Sparsity loss is scalar", sparsity_loss_dim == "scalar",
"lambda * ||z||_1 -> R^d -> R (scalar)")
# Total parameter count
total_params = (d_latent * d_model + # W_enc
d_model * d_latent + # W_dec
d_latent + # b_enc
d_model) # b_dec
check("Parameter count sanity", total_params == 2 * d_latent * d_model + d_latent + d_model,
f"total {total_params:,} = 2*{d_latent}*{d_model} + {d_latent} + {d_model}")
# ── 7.2 CROSS: Three independent feature-quality measures ───────
print("\n=== 7.2 CROSS: Triple feature-quality measurement ===")
random.seed(42)
# Simulation: quality measurement on 100 features
n_features = 100
# Approach 1: Reconstruction loss contribution (loss increase upon ablation)
# More important features cause larger loss increase when ablated
recon_importance = [random.gauss(0.5, 0.2) for _ in range(n_features)]
recon_importance = [max(0, min(1, x)) for x in recon_importance]
# Approach 2: Sparsity quality (degree to which feature activates only on few inputs)
# Higher sparsity = selective for specific concept = better feature
activation_rates = [random.betavariate(2, 20) for _ in range(n_features)]
sparsity_quality = [1.0 - rate for rate in activation_rates]
# Approach 3: Interpretability score (auto-labeling confidence simulation)
# In practice, generate activation-pattern descriptions with GPT-4 and measure consistency
interp_score = [0.3 * recon_importance[i] + 0.3 * sparsity_quality[i]
+ 0.4 * random.gauss(0.6, 0.15) for i in range(n_features)]
interp_score = [max(0, min(1, x)) for x in interp_score]
# Correlation among the three measures (Spearman rank correlation approximation)
def rank_correlation(a, b):
"""Approximate Spearman rank correlation coefficient."""
n = len(a)
rank_a = sorted(range(n), key=lambda i: a[i])
rank_b = sorted(range(n), key=lambda i: b[i])
ra = [0] * n
rb = [0] * n
for i, idx in enumerate(rank_a):
ra[idx] = i
for i, idx in enumerate(rank_b):
rb[idx] = i
d_sq = sum((ra[i] - rb[i]) ** 2 for i in range(n))
rho = 1 - 6 * d_sq / (n * (n * n - 1))
return rho
rho_12 = rank_correlation(recon_importance, sparsity_quality)
rho_13 = rank_correlation(recon_importance, interp_score)
rho_23 = rank_correlation(sparsity_quality, interp_score)
check("Recon-sparsity positive correlation", rho_12 > -0.5,
f"rho={rho_12:.3f} (not perfectly negative)")
check("Recon-interp positive correlation", rho_13 > 0,
f"rho={rho_13:.3f}")
check("Three measures independent", abs(rho_12) < 0.95,
f"correlation |rho|={abs(rho_12):.3f} < 0.95 (independent)")
# ── 7.3 SCALING: SAE quality vs latent-dimension scaling ────────
print("\n=== 7.3 SCALING: Latent-dimension scaling analysis ===")
# Theoretical analysis: as latent dim d_latent grows
# 1) Reconstruction quality: monotonically increases (more bases -> more accurate)
# 2) Feature sparsity: increases (more candidates -> more selective activation)
# 3) Compute cost: linear O(d_model * d_latent)
# 4) Feature redundancy: log increase (more dims -> similar features emerge)
dims = [512, 1024, 2048, 4096, 8192, 16384]
overcomplete_ratios = [d / d_model for d in dims]
# Reconstruction loss: empirically scales ~ 1/sqrt(d_latent)
recon_losses = [1.0 / math.sqrt(d) for d in dims]
# Cost: O(d_model * d_latent) -> linear in d_latent
compute_costs = [d_model * d for d in dims]
# Efficiency: marginal return on quality vs cost
# quality = 1 - recon_loss (higher is better), cost = compute_costs
# marginal efficiency = d(quality)/d(cost) -- quality gain vs cost increase
qualities = [1.0 - rl for rl in recon_losses]
marginal_efficiencies = []
for i in range(len(dims)):
if i == 0:
marginal_efficiencies.append(qualities[i] / compute_costs[i])
else:
dq = qualities[i] - qualities[i-1]
dc = compute_costs[i] - compute_costs[i-1]
marginal_efficiencies.append(dq / dc)
# Optimum: just before marginal efficiency drops sharply
# Last point where marginal-efficiency ratio still exceeds 0.01
best_idx = 0
for i in range(1, len(marginal_efficiencies)):
if marginal_efficiencies[i] / marginal_efficiencies[0] > 0.01:
best_idx = i
best_ratio = overcomplete_ratios[best_idx]
check("Recon loss monotonically decreasing", all(recon_losses[i] > recon_losses[i+1]
for i in range(len(dims)-1)),
"d_latent up -> recon loss down")
check("Cost linear increase", compute_costs[-1] / compute_costs[0] == dims[-1] / dims[0],
f"cost ratio = dim ratio = {dims[-1]/dims[0]:.0f}x")
check("Optimal overcomplete ratio exists", 2 <= best_ratio <= 32,
f"optimal ratio = {best_ratio}x (reasonable range)")
print(f" Scaling table:")
for i, d in enumerate(dims):
print(f" d_latent={d:>6} | overcomp={overcomplete_ratios[i]:>4.0f}x"
f" | recon_loss={recon_losses[i]:.4f} | marg_eff={marginal_efficiencies[i]:.2e}")
# ── 7.4 SENSITIVITY: Sparsity-penalty sensitivity analysis ──────
print("\n=== 7.4 SENSITIVITY: Sparsity lambda sensitivity ===")
# Simulate SAE behavior change across lambda values
lambdas = [0.0001, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
def simulate_sae_behavior(lam, d_lat=4096):
"""Simulate SAE behavior at given lambda."""
# Reconstruction quality: lambda up -> forces sparsity -> recon degrades
recon_quality = 1.0 / (1.0 + 10 * lam)
# Sparsity: lambda up -> more sparse (good, but info loss if excessive)
sparsity = 1.0 - math.exp(-50 * lam)
# Interpretability: maxes at appropriate sparsity (inverted-U curve)
interpretability = 4 * lam * math.exp(-5 * lam) # peaks around ~0.2
# Composite quality: geometric mean of three indicators
total = (recon_quality * sparsity * max(interpretability, 0.001)) ** (1/3)
return recon_quality, sparsity, interpretability, total
results = [simulate_sae_behavior(l) for l in lambdas]
totals = [r[3] for r in results]
best_lambda_idx = totals.index(max(totals))
best_lambda = lambdas[best_lambda_idx]
check("No sparsity at lambda=0", results[0][1] < 0.01,
f"sparsity={results[0][1]:.4f} (lambda={lambdas[0]})")
check("Recon degrades at lambda=1", results[-1][0] < 0.15,
f"recon={results[-1][0]:.4f} (lambda={lambdas[-1]})")
check("Optimal lambda in mid-range", 0.001 <= best_lambda <= 0.1,
f"optimal lambda={best_lambda}")
check("Sensitivity: 10x change shifts quality",
max(totals) / min(totals) > 2,
f"max/min quality ratio = {max(totals)/min(totals):.2f}")
print(f" Sensitivity table:")
for i, lam in enumerate(lambdas):
r, s, ip, t = results[i]
marker = " <-- optimal" if i == best_lambda_idx else ""
print(f" lambda={lam:<8} | recon={r:.3f} | sparse={s:.3f}"
f" | interp={ip:.3f} | total={t:.3f}{marker}")
# ── 7.5 LIMITS: Information-theoretic limits of feature decomposition ─────
print("\n=== 7.5 LIMITS: Information-theoretic limits ===")
# Limit 1: Lower bound on latent dim
# Representing k independent concepts in d_model activations
# requires at least k latent dims (linear-independence requirement)
# In practice, due to interference, O(k * log(k)) is needed
k_concepts = 1000 # estimated independent concepts
min_d_latent_theory = k_concepts # lower bound
practical_d_latent = int(k_concepts * math.log2(k_concepts)) # practical minimum
check("Theoretical lower bound", d_latent >= min_d_latent_theory,
f"d_latent={d_latent} >= k={k_concepts}")
check("Practical lower bound", d_latent >= practical_d_latent or d_latent >= k_concepts,
f"practical={practical_d_latent}, current={d_latent}")
# Limit 2: Sparsity-reconstruction tradeoff (rate-distortion theory)
# Rate-Distortion: R(D) = H(X) - minimum bits to achieve distortion <= D
# In SAE: sparsity = low bitrate, reconstruction = low distortion
# Cannot minimize simultaneously (Pareto frontier)
# Shannon entropy estimate (activations quantized)
bits_per_dim = 16 # float16 baseline
total_info = d_model * bits_per_dim # 8192 bits
sparse_info = target_active * bits_per_dim # storing only active features
compression_ratio = total_info / sparse_info
check("SAE compression effect", compression_ratio > 1,
f"compression = {compression_ratio:.1f}x ({total_info} -> {sparse_info} bits)")
check("Information preservable", sparse_info * math.log2(d_latent) >= total_info * 0.5,
f"with location info {sparse_info * math.log2(d_latent):.0f} >= {total_info * 0.5:.0f}")
# Limit 3: Superposition capacity
# Johnson-Lindenstrauss lemma: in d dims you can place
# at most ~ exp(c*d) nearly-orthogonal vectors
# This is the theoretical superposition upper bound
# c ~ 1/(2*ln(2)) when applying JL lemma (eps=0.1 level)
# Real superposition work (Elhage et al. 2022) experimentally confirms
# ~ d^1.5 features can be placed nearly orthogonally in d dims
superposition_capacity = d_model ** 1.5 # empirical superposition capacity
check("Superposition capacity sufficient", superposition_capacity > d_latent,
f"superposition cap ~ {superposition_capacity:.0f} > d_latent={d_latent}")
# ── 7.6 CHI2: Feature significance statistical test ─────────────
print("\n=== 7.6 CHI2: Feature significance test ===")
# Question: do extracted features capture genuinely meaningful concepts,
# or are they merely noise patterns?
# Simulation: test whether feature activation pattern differs from uniform
# Genuine feature: selectively activates on a specific input class
# Noise feature: activates uniformly across all inputs
n_categories = 10 # input classes
n_samples = 500 # observations
def chi2_test(observed, expected):
"""Compute chi-square test statistic."""
return sum((o - e) ** 2 / e for o, e in zip(observed, expected))
def chi2_critical(df, alpha=0.05):
"""Approximate critical value at df, alpha=0.05 (Wilson-Hilferty)."""
z = 1.645 # one-sided 95%
x = df * (1 - 2/(9*df) + z * math.sqrt(2/(9*df))) ** 3
return x
# Genuine feature: non-uniform distribution (concentrated on one class)
real_feature_counts = [10, 15, 8, 12, 200, 180, 5, 20, 30, 20]
expected_uniform = [n_samples / n_categories] * n_categories
chi2_real = chi2_test(real_feature_counts, expected_uniform)
# Noise feature: nearly uniform distribution
random.seed(7)
noise_counts = [n_samples // n_categories + random.randint(-5, 5)
for _ in range(n_categories)]
# Adjust the total
noise_counts[-1] = n_samples - sum(noise_counts[:-1])
chi2_noise = chi2_test(noise_counts, expected_uniform)
df = n_categories - 1
critical = chi2_critical(df)
check("Genuine feature significant", chi2_real > critical,
f"chi2={chi2_real:.1f} > critical={critical:.1f}")
check("Noise feature not significant", chi2_noise < critical,
f"chi2={chi2_noise:.1f} < critical={critical:.1f}")
check("Two types distinguishable", chi2_real / max(chi2_noise, 0.01) > 10,
f"genuine/noise ratio = {chi2_real/max(chi2_noise,0.01):.1f}x")
# ── 7.7 OEIS: Mathematical structure of feature distribution ────
print("\n=== 7.7 OEIS: Feature distribution mathematical structure ===")
# Observation: SAE feature activation-frequency distribution follows a power law
# This matches Zipf's law in natural language and sparse coding in neuroscience
# Zipf distribution simulation: at rank r, frequency f(r) ~ 1/r^alpha
n_feat = 200
alpha_zipf = 1.07 # approximate natural-language Zipf exponent
frequencies = [1.0 / (r ** alpha_zipf) for r in range(1, n_feat + 1)]
total_freq = sum(frequencies)
frequencies = [f / total_freq for f in frequencies] # normalize
# Zipf-law verification: log(freq) vs log(rank) linear regression
log_ranks = [math.log(r) for r in range(1, n_feat + 1)]
log_freqs = [math.log(f) for f in frequencies]
mean_lr = statistics.mean(log_ranks)
mean_lf = statistics.mean(log_freqs)
num = sum((log_ranks[i] - mean_lr) * (log_freqs[i] - mean_lf) for i in range(n_feat))
den = sum((log_ranks[i] - mean_lr) ** 2 for i in range(n_feat))
slope = num / den # Zipf exponent estimate
check("Zipf-law slope", abs(slope + alpha_zipf) < 0.1,
f"estimated slope={slope:.3f}, theory={-alpha_zipf}")
# Compare with OEIS A001462 (Golomb sequence) and feature cluster sizes
# Golomb sequence: self-referential sequence, similar to feature hierarchy
golomb = [0, 1, 2, 2, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 6]
# Feature cluster size distribution also shows self-similar pattern
cluster_sizes = sorted(Counter(int(f * 1000) for f in frequencies[:15]).values(),
reverse=True)
check("Feature distribution follows power law", frequencies[0] / frequencies[-1] > 50,
f"freq ratio rank 1/200 = {frequencies[0]/frequencies[-1]:.1f}")
check("Top 20% of features cover 80% activation (Pareto)",
sum(frequencies[:n_feat//5]) > 0.6,
f"top 20% accounts for {sum(frequencies[:n_feat//5]):.1%} of total")
# ── 7.8 PARETO: SAE hyperparameter design-space exploration ─────
print("\n=== 7.8 PARETO: Design space exploration ===")
# 3-axis design space: (d_latent, lambda, learning_rate)
# Goal: maximize reconstruction quality + maximize sparsity + minimize cost
# (3-objective optimization)
design_points = []
for d in [1024, 2048, 4096, 8192]:
for lam in [0.001, 0.01, 0.05, 0.1]:
for lr in [0.0001, 0.001, 0.01]:
# Reconstruction quality simulation
recon = (1.0 - 1.0/math.sqrt(d)) * (1.0 / (1 + 2*lam))
# Sparsity
spars = 1.0 - math.exp(-30 * lam)
# Cost (normalized)
cost = d * d_model / (8192 * 512) # max = 1.0
# Training stability (too-high lr is unstable)
stability = math.exp(-100 * lr * lr)
design_points.append({
'd': d, 'lam': lam, 'lr': lr,
'recon': recon, 'spars': spars,
'cost': cost, 'stability': stability,
'score': recon * spars * stability / (cost + 0.1)
})
# Identify Pareto frontier (simplified: top 10% by composite score)
design_points.sort(key=lambda p: p['score'], reverse=True)
pareto_size = max(1, len(design_points) // 10)
pareto_front = design_points[:pareto_size]
best = pareto_front[0]
check("Pareto optimum exists", best['score'] > 0,
f"best: d={best['d']}, lam={best['lam']}, lr={best['lr']}")
check("Best-point reconstruction quality", best['recon'] > 0.7,
f"recon={best['recon']:.3f}")
check("Best-point sparsity", best['spars'] > 0.3,
f"sparsity={best['spars']:.3f}")
check("Design space coverage", len(design_points) >= 48,
f"{len(design_points)} design points evaluated")
# ── 7.9 SYMBOLIC: Symbolic computation of SAE gradient updates ───
print("\n=== 7.9 SYMBOLIC: SAE gradient symbolic computation ===")
# SAE loss L = ||x - W_dec @ ReLU(W_enc @ x + b_enc) - b_dec||^2
# + lambda * ||ReLU(W_enc @ x + b_enc)||_1
#
# Gradient derivation (verified using simplified scalar case):
# z = max(0, w_e * x + b_e)
# x_hat = w_d * z + b_d
# L = (x - x_hat)^2 + lam * |z|
# Exact symbolic computation via Fraction
x_val = Fraction(3, 1)
w_e = Fraction(1, 2)
b_e = Fraction(-1, 4)
w_d = Fraction(2, 3)
b_d = Fraction(1, 10)
lam = Fraction(1, 100)
# Forward pass
pre_act = w_e * x_val + b_e # 1/2 * 3 - 1/4 = 5/4
z = max(Fraction(0), pre_act) # ReLU: 5/4 > 0 so 5/4
x_hat = w_d * z + b_d # 2/3 * 5/4 + 1/10 = 5/6 + 1/10 = 28/30 = 14/15
recon_loss = (x_val - x_hat) ** 2 # (3 - 14/15)^2 = (31/15)^2
sparse_loss = lam * abs(z) # 1/100 * 5/4 = 5/400 = 1/80
total_loss = recon_loss + sparse_loss
check("Forward-pass exact", pre_act == Fraction(5, 4),
f"w_e*x+b_e = {pre_act}")
check("ReLU exact", z == Fraction(5, 4),
f"ReLU({pre_act}) = {z}")
check("Reconstruction value exact", x_hat == Fraction(14, 15),
f"x_hat = {x_hat}")
# Gradients (chain rule, z > 0 case)
# dL/dw_d = 2*(x - x_hat)*(-z) = -2*(x-x_hat)*z
dL_dw_d = -2 * (x_val - x_hat) * z # exact rational arithmetic
# dL/dw_e = 2*(x - x_hat)*(-w_d)*x + lam*sign(z)*x (z>0 so ReLU' = 1)
dL_dw_e = -2 * (x_val - x_hat) * w_d * x_val + lam * x_val
check("dL/dw_d rational exact",
dL_dw_d == -2 * (x_val - x_hat) * z,
f"dL/dw_d = {dL_dw_d} = {float(dL_dw_d):.6f}")
check("dL/dw_e sign correct",
(dL_dw_e < 0) == (x_val > x_hat),
f"if x > x_hat then gradient negative (w_e increases)")
check("Loss positive", total_loss > 0,
f"L = {total_loss} = {float(total_loss):.6f}")
# ── 7.10 COUNTER: Limits of interpretability ────────────────────
print("\n=== 7.10 COUNTER: Honest limits ===")
limitations = [
("Incompleteness", "No guarantee SAE captures all features -- important concepts may hide in distributed representations"),
("Scalability", "Current SAE is practical only on small models -- unverified for frontier models (hundreds of billions of parameters)"),
("Causality limit", "Feature-behavior correspondence may be correlational -- causal interventions may have side effects"),
("Interpretation subjectivity", "Feature labeling involves human judgment -- the same feature may admit different interpretations"),
("Dynamic representation", "Representations may shift during inference -- static SAE may miss dynamic computation"),
("Residual superposition", "SAE does not fully resolve superposition -- polysemous features remain"),
("Non-linear structure", "SAE is a linear basis decomposition -- cannot capture non-linear feature combinations"),
]
check("At least 7 limits identified", len(limitations) >= 7,
f"{len(limitations)} limits identified")
for i, (name, desc) in enumerate(limitations):
print(f" Limit {i+1}. [{name}]: {desc}")
# Falsifiability check: state failure conditions for each research idea
falsifiable_predictions = [
"Failure to reach reconstruction MSE < 0.05 on Pythia-70M with d_latent=4096 triggers architecture redesign",
"Hallucination-circuit detection precision below 50% rejects the circuit hypothesis",
"Cross-model feature correspondence below 10% rejects the universal-feature hypothesis",
]
check("Falsifiable predictions present", len(falsifiable_predictions) >= 3,
f"{len(falsifiable_predictions)} falsifiable predictions")
# ── Final results ────────────────────────────────────────────────
print("\n" + "=" * 60)
print(f"Verification result: {PASS} PASS / {FAIL} FAIL (total {PASS+FAIL})")
print(f"Pass rate: {PASS/(PASS+FAIL)*100:.1f}%")
if FAIL == 0:
print("All items PASS")
else:
print(f"Note: {FAIL} failed -- review required")
print("=" * 60)| # | Idea | Core question | Expected impact |
|---|---|---|---|
| 1 | Hierarchical latent-dim SAE | Does extracting features simultaneously at multiple resolutions improve quality? | High |
| 2 | Egyptian-fraction feature decomposition | Does decomposing features in 1/2+1/3+1/6=1 hierarchical structure improve interpretability? | Medium |
| 3 | Dedekind feature lattice | Does imposing partial order between features reveal a semantic hierarchy? | Medium |
| 4 | Feature lifecycle tracking | Are there regularities in feature emergence/disappearance during training? | High |
| 5 | Cross-layer SAE correlation | Does feature correspondence across layers reveal circuit structure? | High |
| 6 | Attention-pattern SAE | Does applying SAE to attention weights uncover new features? | Medium |
| 7 | Task-conditional SAE | Does task-conditional encoding separate task-specialized features? | High |
| 8 | Connectivity-based feature importance | Are features with higher connectivity in circuit graph more important? | Medium |
| 9 | Information-theoretic regularization loss | Does mutual-information-based regularization beat L1 in interpretability? | Medium |
| 10 | Cross-model feature correspondence | Do universal features exist in Haiku/Sonnet/Opus? | Very high |
| 11 | Feature attribution decomposition | Can per-output-token feature contributions be tracked precisely? | High |
| 12 | Distributed SAE | Does distributed training maintain quality at frontier-model scale? | High |
| 13 | Progressive feature discovery | Does coarse-to-fine accelerate full feature discovery? | Medium |
| 14 | Compressed feature storage | Does quantizing/compressing the feature dictionary preserve interpretability? | Low |
| 15 | Automated feature labeling | Does LLM-based auto-labeling agree with human judgment? | High |
| # | Idea | Core question | Expected impact |
|---|---|---|---|
| 16 | Arithmetic circuit atlas | Are there dedicated circuits for addition/multiplication/division? | High |
| 17 | Math-concept recognition circuits | Is there a specific circuit for each math concept ("prime", "even", etc.)? | Medium |
| 18 | Proof-strategy selection circuit | Through which circuit does the model decide a proof-style strategy? | Medium |
| 19 | Hallucination circuit detection | Can specific circuits implicated in hallucination generation be identified? | Very high |
| 20 | Uncertainty circuit amplification | Can circuits expressing "I do not know" be strengthened? | High |
| 21 | Cross-domain transfer circuits | Are there circuit paths transferring math ability to coding? | Medium |
| 22 | Language-math interface | What circuits convert natural-language problems to math? | Medium |
| 23 | Safety-refusal circuit mapping | Full map of circuits involved in refusing harmful requests | Very high |
| 24 | Deception circuit detection | Do circuits exist where the model intentionally hides information? | Very high |
| 25 | Metacognition circuits | What circuits judge the model's own confidence? | High |
| 26 | Factual-grounding circuits | What circuits link learned facts to generated text? | High |
| 27 | Multi-step reasoning stability | What circuit paths propagate errors in long reasoning chains? | High |
| # | Idea | Core question | Expected impact |
|---|---|---|---|
| 28 | Interpretation-experiment DSL | Does declarative description of interpretability experiments raise productivity? | Medium |
| 29 | Activation-atlas visualization | Does a 2D map of the full activation space accelerate exploration? | Medium |
| 30 | Feature search engine | Can "lying-related features" be searched in natural language? | High |
| 31 | Circuit diff tool | Can circuit changes across model versions be tracked? | High |
| 32 | Real-time alignment dashboard | Can safety-related features be monitored in real time during inference? | High |
| 33 | Automated circuit-discovery pipeline | Can circuit discovery be fully automated? | Very high |
| 34 | Feature time-series visualization | Does it effectively show per-step feature activation changes during inference? | Medium |
| 35 | Interpretability benchmark | Does a standard benchmark objectify SAE comparison? | High |
| 36 | Causal intervention automation | Framework that automatically performs activation patching | Medium |
| 37 | Multilingual feature comparator | Tool comparing shared features in Korean/English/Chinese | Medium |
| 38 | Feature dependency graph generator | Tool automatically graphing causal relations between features | Medium |
| 39 | Reproducibility verification tool | Tool automatically verifying reproducibility of interpretability results | High |
| Idea | Primary metric | Secondary metric | Baseline | Success criterion |
|---|---|---|---|---|
| #1 Hierarchical SAE | Reconstruction MSE | Feature interpretation rate | Single-resolution SAE | 10% MSE reduction |
| #5 Cross-layer | Inter-layer feature correlation | Number of circuits discovered | Independent SAE | correlation > 0.3 |
| #7 Conditional SAE | Per-task feature separation | Task performance | Unconditional SAE | 20% separation gain |
| #10 Cross-model | Feature correspondence rate | Semantic agreement | Random correspondence | rate > 30% |
| #16 Arithmetic atlas | Circuit accuracy | Intervention effect | Full model | accuracy > 80% |
| #19 Hallucination detection | Precision/recall | False-positive rate | Output-based detection | F1 > 0.7 |
| #23 Safety circuits | Circuit coverage | Refusal accuracy | Behavioral test | coverage > 60% |
| #24 Deception detection | Detection rate | False-positive rate | None (new) | detection > 50% |
| #33 Automated pipeline | Discovery speed | Circuit quality | Manual analysis | 10x speedup |
| #35 Benchmark | Recall | Tool compatibility | Ad-hoc evaluation | 3+ tools supported |
| # | Prediction | Verification method | Failure condition |
|---|---|---|---|
| P1 | SAE d_latent = 8x d_model is optimal efficiency (vs 4x, 16x) | Train at 3 ratios, compare reconstruction MSE/cost | Reject if 8x worse than 4x |
| P2 | Mid-layer features are the most interpretable (vs early/late) | Train per-layer SAE, measure auto-labeling accuracy | Reject if monotonically increasing/decreasing |
| P3 | Arithmetic circuits preserved across model sizes | Trace identical circuit on Pythia 70M~12B | Reject if vanishes at 1B+ |
| P4 | Hallucination features and low-confidence features co-activate | Measure co-activation rate in hallucination cases | Reject if co-activation < 20% |
| P5 | lambda ~ 0.01 is the interpretability optimum | lambda sweep + human evaluation | Reject if optimum at < 0.001 or > 0.1 |
| P6 | 50%+ of multilingual semantic features shared across languages | EN/KR/CN SAE feature correspondence analysis | Reject if shared < 20% |
| P7 | Safety-refusal circuit summarized in <=3 core features | Refusal-behavior tracking via feature ablation | Reject if 10+ required |
| P8 | "Critical period" exists in feature lifecycle (sudden emergence) | Per-checkpoint feature tracking | Reject if all features emerge gradually |
| P9 | Progressive SAE (coarse->fine) 2x faster than single-resolution | Compute compared at equal quality | Reject if speedup < 1.2x |
| P10 | Deception circuits partially overlap with strategic-reasoning circuits | Compare circuits activated across both tasks | Reject if overlap < 5% |
+------------------------------------------------------------------+
| Current vs proposed approach performance (estimated) |
+------------------------------------------------------------------+
| |
| [SAE reconstruction quality (MSE, lower is better)] |
| Current standard SAE ████████████████████░░░░░░░░░░ MSE=0.08 |
| Hierarchical SAE (#1) ████████████████░░░░░░░░░░░░░░ MSE=0.05 |
| Conditional SAE (#7) ██████████████░░░░░░░░░░░░░░░░ MSE=0.04 |
| |
| [Feature interpretation rate (higher is better)] |
| Current manual labels ████████████░░░░░░░░░░░░░░░░░░ ~40% |
| Auto labeling (#15) ████████████████████████░░░░░░ ~80% |
| |
| [Circuit-discovery speed (circuits/week)] |
| Current manual trace ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0.5/wk |
| Auto pipeline (#33) ████████████████████████████░░ 5+/wk |
| |
| [Safety-audit coverage] |
| Current behavior test ██████████░░░░░░░░░░░░░░░░░░░░ ~30% |
| Circuit-based (#23) ████████████████████████░░░░░░ ~70% |
| |
| [Reproducibility] |
| Current ad-hoc code ████████░░░░░░░░░░░░░░░░░░░░░░ ~25% |
| Benchmark (#35) ██████████████████████████████ ~95% |
+------------------------------------------------------------------+
+======================================================================+
| AI Interpretability research infrastructure architecture |
+======================================================================+
| |
| +---------------------+ +---------------------+ |
| | Model inference | | GPU training cluster| |
| | server |--------->| (SAE training) | |
| | (activation collect) +----------+----------+ |
| +---------------------+ | |
| v |
| +----------------------------------------------------------+ |
| | Feature dictionary store | |
| | +----------+ +----------+ +----------+ +----------+ | |
| | | Layer 0 | | Layer 1 | | ... | | Layer N | | |
| | | 4K feats | | 4K feats | | | | 4K feats | | |
| | +----------+ +----------+ +----------+ +----------+ | |
| +-------------------------+--------------------------------+ |
| | |
| +-------------+-------------+ |
| v v |
| +---------------------+ +---------------------+ |
| | Circuit analysis | | Visualization/ | |
| | engine | | search tools | |
| | - Activation patch | | - Activation atlas | |
| | - Path tracing | | - Feature search | |
| | - Causal intervene | | - Time-series view | |
| +----------+----------+ +----------+----------+ |
| | | |
| v v |
| +----------------------------------------------------------+ |
| | Integrated interpretability dashboard | |
| | Safety monitor | Circuit browser | Exp mgmt | Reporting | |
| +----------------------------------------------------------+ |
+======================================================================+
+----------------------------------------------------------------------+
| Detailed data flow |
+----------------------------------------------------------------------+
| |
| [Input] |
| Text corpus ----+ |
| Prompt set ----+---> Model inference ---> Per-layer act tensors |
| Task data ----+ | | |
| v v |
| Attention weights Residual stream act |
| | | |
| v v |
| +-----+-----+ +----+-----+ |
| | Attention | | MLP | |
| | SAE | | SAE | |
| +-----------+ +----------+ |
| | | |
| v v |
| Attention feats MLP feats |
| | | |
| +------+-------+ |
| | |
| v |
| Unified feature dict |
| | |
| +----------------------+----------------------+ |
| v v v |
| Auto labeling Circuit tracing Stat analysis |
| | | | |
| v v v |
| Feature catalog Circuit atlas Quality report |
| |
| [Output] |
| Papers, technical reports, safety audits, open-source tools |
+----------------------------------------------------------------------+
| Item | Current tool | Proposed tool | Ideal state |
|---|---|---|---|
| SAE training | SAELens (single GPU) | Distributed SAE trainer | Frontier-model scale support |
| Activation collection | TransformerLens (manual) | Auto-collection pipeline | Streaming real-time collection |
| Feature labels | Manual + GPT-4 | Auto-labeling v1 | Fully automatic + human verification |
| Circuit tracing | Manual activation patching | Semi-auto path search | Fully automatic circuit discovery |
| Visualization | One-shot Matplotlib | Interactive dashboard v1 | Real-time 3D exploration |
| Benchmark | None (paper-specific) | Standard benchmark v1 | Community standard |
| Reproducibility | Code share (non-standard) | Env + checkpoint bundle | One-click reproduction |
| Causal intervention | Manual scripts | Declarative intervention DSL | Auto counterfactual generation |
| Cross-model | None | Feature correspondence prototype | Model lineage tracking |
| Multilingual | None | EN/KR comparator prototype | All-language support |
- Open code: publish all experiment code in a public repository
- Pinned environment: reproduce via Docker + locked dependency versions
- Checkpoint sharing: publish trained SAE weights on HuggingFace
- Fixed seeds: fix random seeds + multi-seed runs reporting variance
- Negative results: document failed experiments at the same quality
- Report bootstrap confidence intervals (95%) on all performance comparisons
- Multiple-comparison correction (Bonferroni or FDR)
- Report effect size (Cohen's d)
- Pre-registration: register hypotheses and analysis plans for major experiments
- Deception-circuit findings disclosed only after Anthropic internal review
- Public scope of safety-circuit mapping pre-agreed with the research team
- Ethical implications of model-behavior changes in causal-intervention experiments reviewed
This research program expects substantive progress on only 5~8 of the 39 ideas within the 4-month window. The rest are left as exploratory analyses or follow-up directions. Honest progress is prioritized over overpromising.
This section covers why for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers compare for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers requires for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers struct for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers flow for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers evolve for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers verify for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers ideas for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers metrics for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers risks for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers dependencies for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers timeline for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers tools for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers team for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.
This section covers references for the domain. Initial scaffold content — expand with domain-specific data, references, and verification in subsequent revisions.