Low-Bit N-Gram Embeddings for Language Models

Research official 1 src. ~1 min

The paper quantizes the n-gram, Engram-style embedding tables now used by open-weight models, noting that DeepSeek-V4.1-Flash spends 196B parameters on its Engram memory. Across nanochat-, Llama- and GPT-2-family models trained from scratch, four-bit tables cut roughly 75% of table memory while keeping over 99% of the validation-loss gain, and reinvesting the saved memory in larger tables beats the original full-precision tables.

Why it matters

Turns a 196B-parameter frontier memory table into a roughly 50B one at no quality cost.

Importance: 3/5

Directly usable recipe for a component that dominates parameter count in current MoE models.

efficiency quantization moe architecture paper

Sources

official Low-Bit N-Gram Embeddings for Language Models — arXiv