Recent LLM optimization research actively explores Layer Looping and Pruning (e.g., MobileLLM, Layer Skip, ShortGPT) to balance performance and inference speed. The key question is: "Which layers are most critical to model performance (Layer Importance)?"
Existing methods have critical limitations on quantized models:
-
Gradient-based measurement failure: In continuous models,
$|∂L/∂w|$ intuitively yields layer importance. However, INT4/INT8 quantized models are non-differentiable, forcing approximations like STE (Straight-Through Estimator) that do not accurately represent weight importance in discrete space. - Activation-based measurement limitations: Cosine similarity-based activation measurements only infer "can this layer be removed (redundancy)?" — they do not directly prove "how critical is this layer's computation?"
The methodological roots of this project draw inspiration from the "observation in discrete space" step of the Wave Function Collapse (WFC) algorithm. However, the core metric proposed — Q-PSA (Quantized Perturbation Sensitivity Analysis) — does not follow WFC's collapse or propagation logic. It is independently defined as a discrete-space sensitivity analysis that performs direct perturbation on quantization grids with statistical scoring.
WFC's collapse/propagation is a process of constructing a solution, while Q-PSA is a process of measuring the stability of an already-constructed solution (quantized weights).
- Direct perturbation observation: Alter a layer's weights to other permitted values within the quantization grid and measure the resulting Perplexity (PPL) change.
- Sensitivity-based importance scoring:
- High Sensitivity (high importance): Even small weight changes cause large PPL shifts — the layer must perform its computation precisely (deterministic state).
- Low Sensitivity (low importance): Weight changes to other quantization values cause minimal PPL change — the layer is relatively less important or redundant.
The resulting Q-PSA Score enables accurate selection of Layer Looping and Pruning candidates.
- Target models: GGUF format small models for fast iteration
- Phase 1 (pipeline validation): Qwen2.5 0.5B (Q4_K_M)
- Phase 2 (scale validation): Qwen2.5 1.5B or Llama-3.2 1B (Q4_K_M)
- Phase 3 (practicality validation): Llama-3 8B (Q4_K_M)
- Evaluation dataset: WikiText-2
- Metric: Perplexity (PPL)
- Environment: Identical hardware, identical runtime versions, cache cleared before each run
Perturbation procedures are fixed upfront to ensure reproducibility and fairness.
Full weight enumeration is computationally infeasible. We use stratified sampling with statistical significance:
- Proportional sampling per sub-matrix within each Transformer Block:
- Attention: Q, K, V, O projections
- FFN: gate, up (down excluded — see §7)
-
$n$ samples per sub-matrix →$N = n \times (\text{number of sub-matrices})$ per layer- Phase 1 initial value:
$n = 200$ (6 sub-matrices → ~1,200 per layer) - Sample size adequacy validated via convergence analysis in Phase 1
- Phase 1 initial value:
- Adjacent level shift (±1 quantization level): Change to the nearest permitted value on the quantization grid
- Single-weight perturbation: One weight at a time
- Bidirectional measurement: Both +1 and -1 directions to eliminate directional bias
⚠️ Known limitation: Single-weight perturbation cannot capture interaction effects between weights. Phase 1 starts with this approach; small-group perturbation (2–4 simultaneous) may be explored in Phase 2 if needed.
- Per perturbation:
$\Delta PPL = PPL_{perturbed} - PPL_{baseline}$ - Per-layer Q-PSA Score: record both mean and variance of
$\Delta PPL$ across sub-matrices- High mean → sensitive layer (important)
- High variance → internally heterogeneous layer (requires further analysis)
| Phase | Model | Layers | Samples/Layer | Forward passes | Est. time* |
|---|---|---|---|---|---|
| 1 | 0.5B | 24 | ~1,200 | ~57,600 | After profiling |
| 2 | 1.5B | 28 | ~1,200 | ~67,200 | After profiling |
| 3 | 8B | 32 | ~1,200 | ~76,800 | After profiling |
* Single forward pass time measured at Phase 1 start to estimate total time
Optimization strategy:
- Measure PPL on a fixed subset of WikiText-2 (e.g., first 1,000 tokens) instead of the full dataset
- Subset adequacy validated by correlation analysis with full-dataset PPL (target: r > 0.95)
1. Baseline PPL measurement → verify: original model PPL recorded
2. Single forward pass profiling → verify: total experiment time estimable
3. Subset size adequacy validation → verify: subset PPL vs full PPL correlation > 0.95
4. Full-layer perturbation scoring → verify: all layer scores computed
5. Convergence analysis (sample size adequacy) → verify: rank stability within tolerance
6. Layer ranking → verify: ranking table complete
7. Baseline comparison experiments → verify: all baselines run under identical conditions
The most direct validation of Q-PSA rankings: Remove the top-$K$ least-sensitive (least important) layers via pruning and compare PPL retention across methods.
-
$K = 1, 2, 4, 8$ (adjusted by model size) - Metric:
$PPL_{pruned} / PPL_{base}$ (lower is better)
Select the top-$K$ most-sensitive (most important) blocks for layer looping and compare performance.
- Base Model: Original model PPL (lower bound)
-
Random Selection: Random
$K$ layers (10 runs, mean ± std) - Activation Norm: L2 norm of layer outputs
- Block Influence (BI) Score: ShortGPT method — cosine similarity between layer input and output
- Taylor Importance (STE): Gradient-based approximation — quantitative test of the "inaccuracy" claim
- Q-PSA (proposed): This project's method
Verify whether Q-PSA produces different layer rankings from existing methods:
- Compute Spearman's ρ between all method pairs
- High ρ → same conclusions as existing methods → weak differentiation
- Low ρ → different perspective → determine which is more accurate via pruning/looping results
All comparisons record the following metrics:
| Metric | Description |
|---|---|
| PPL | Core performance metric |
| Scoring time | Computational cost per method |
| Peak memory usage | Practicality assessment |
| Layer ranking | Cross-method comparison |
Questions that must be answered in Phase 1. Negative results trigger a direction change.
- Differentiation: Does Q-PSA produce significantly different layer rankings from existing methods (BI score, Activation Norm)?
- Kill: Spearman's ρ > 0.9 (nearly identical rankings → no methodological differentiation)
- Accuracy: If rankings differ, does Q-PSA-based pruning actually defend PPL better?
- Kill: No PPL retention advantage over existing methods
- Practicality: Is the scoring time reasonable compared to existing methods?
- Kill: >100× slower than existing methods (impractical even with accuracy advantage)
- New metric for quantized model optimization: Moving beyond inaccurate continuous-space approximations (STE) to an intuitive, robust evaluation metric optimized for discrete space.
- Practical computational cost: Discarding WFC's heavy collapse/propagation loop in favor of observation-inspired statistical scoring within realistic time budgets.
- Successful pivot from T-WFC: Overcoming the fundamental limitations of prior research (T-WFC) where discrete-space training could not outperform continuous-space optimizers (SGD), and repositioning the idea toward LLM compression R&D.
- Independence assumption: Single-weight perturbation ignores weight interactions. The validity of this assumption in real models requires experimental confirmation.
- Scale transfer: Conclusions from small models are not guaranteed to hold at larger scales. Mitigated through phased scale-up validation.
ffn_down.weightexclusion: llama.cpp's CPU backend repacks Q4_K tensors on certain layers into an optimizedq4_K_8x8layout. Writing to repacked tensors viaggml_backend_tensor_setcauses a crash.ffn_down.weightis currently excluded from perturbation targets. This means Q-PSA scores only 6 of 7 sub-matrices per layer. Mitigation options: disable CPU_REPACK at build time, or use GPU-only tensor access.