Design ModernBERT-Pro, an enhanced encoder-only transformer incorporating Multi-head Latent Attention (MLA) for global layers, fine-grained Mixture-of-Experts FFN layers, Mamba-2 hybrid blocks for linear-complexity local mixing, and a multi-stage curriculum pretraining pipeline. The target is a ~350M-param model (with ~2.5x effective FLOPs via spar
-
Updated
May 25, 2026 - Python