NCP-ArchPreview: latent-space language models at 8.9B via Next Concept Prediction

The Intern-NCP team scaled a latent-space language model to 8.9B parameters trained on 5.73T tokens, the largest such demonstration to date. On top of next-token prediction it adds Next Concept Prediction: a product-quantized concept vocabulary built from hidden states, a Concept Module predicting future multi-token concepts, and feedback of those predictions to guide token-level generation, trained end-to-end. It matches OLMo-3-7B's final pretraining loss with 51.3% of the tokens and beats it by 2.45 points macro-average (+5.99 GSM8K) after full pretraining; the 17M-parameter VQ module alone enables cheap domain adaptation and improves speculative-drafting accepted length by 4.17%.