Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Cerebras
Argues stochastic depth deserves a return to LLM pretraining: with tuned layer distribution and schedules, models match or beat baseline validation loss while cutting training FLOPs by up to 25%, and the dropped-layer structure unlocks early exit, layer skipping, and self-speculative decoding for up to 1.5x faster inference. Backed by 2,400+ runs from 271M to 8.2B parameters on Cerebras CS-3.
Why it matters
Layer dropout had disappeared from modern pretraining recipes; ICML 2026-published results at this scale make it a practical lever for both training cost and post-hoc inference speedups on the same checkpoint.
Importance: 2/5
Industrial-scale study (2,400+ runs) with a practical training-cost lever