There are two leversto adopt independently or together:
- MLA (Multi-head Latent Attention) → compresses the KV cache to a low-rank latent → huge VRAM + bandwidth savings, minor quality hit after a short recovery fine-tune.
- DSA (DeepSeek Sparse Attention) kernels → token-level sparsity at prefill/decoding → big FLOP/latency wins; no weight changes (inference-time).
Qwen-1.7B today uses dense decoder attention with GQA + RoPE (and optional sliding window in configs).
There are two leversto adopt independently or together:
Qwen-1.7B today uses dense decoder attention with GQA + RoPE (and optional sliding window in configs).