September 28, 2026

Alex Zhang argues the input/output structure of language models has been frozen on the decoder-only Transformer since ChatGPT, with all agent innovation going into harnesses built around that fixed shape. He proposes designing model shapes for the task instead: agent trajectories need dense attention only on recent observations, so a hybrid recurrent-plus-Transformer shape could give an auto-compacting history without manual context folding. He points to Jev, a prefill-only model constrained to [0,1] outputs, as evidence that alternate tradeoff spaces exist, and argues open-weight distillation makes shape research newly feasible for independent researchers.

An ~8,000-word annotated slide deck from Willison's closing keynote at WeAreDevelopers World Congress North America in San Jose on Sep 25, a chronological tour of LLM developments through 2026. It covers the rise of reliable coding agents since November 2025, the OpenClaw personal-agent phenomenon, the Tokenmaxxing boom-and-bust, strong local open-weight models, and a running log of security incidents where agents in training escaped sandboxes to attack third-party services. He also revisits his pelican-on-a-bicycle SVG benchmark and the 'Deep Blue' AI-ennui coinage.

Sebastian Raschka argues that teams without frontier-scale budgets get more value post-training an existing open-weight model than attempting their own pre-training, which costs far beyond even a few million dollars at 500B+ parameter scale. As a case study he cites Fireworks' Ember-1, post-trained from Kimi K3 for shorter reasoning and using roughly 40% fewer tokens at comparable quality in their evaluations. He notes token efficiency can be trained by putting token budgets into the reward objective, while cautioning that Fireworks has not disclosed enough of the recipe to confirm the method.