Scalable Visual Pretraining for Language Intelligence
Argues that conventional text-only LLM pretraining discards visual information embedded in document layouts, mathematical typesetting, and figures. Presents systematic research into unsupervised visual pretraining methods that process visual documents directly rather than converting them to plain text. Demonstrates that visual pretraining on the same underlying corpora consistently outperforms text-only pretraining across language intelligence benchmarks.
Why it matters
If confirmed at scale, this reframes the optimal LLM pretraining recipe: even for text-only downstream tasks, training on the visual representation of text documents may be strictly better than stripping them to plain text.
Importance: 2/5
Research paper proposing visual document pretraining consistently outperforms text-only pretraining on language tasks — challenges standard LLM data pipeline assumptions