llama.cpp brings MTP speculative decoding to recurrent Qwen models
ggml
llama.cpp nightly builds b10720–b10731 (Aug 31–Sep 1) land a performance series: b10731 adds recurrent-state rollback for qwen4exp, enabling MTP speculative decoding on recurrent models — decoding reaches 183 tok/s on code and 144 tok/s on prose on Qwen3.8-Flash-Next versus 123/83 before. Other builds optimize AVX2 IQ-quant prompt processing, KV-cache restore of non-contiguous cells (25–63 s down to ~0.4 s in a production test), Metal M1 Ultra flash-attention tunings, and a CUDA XOR-swizzle flash attention with a DGX Spark race fix.
Why it matters
Speculative decoding was previously limited to transformer-style models; extending it to recurrent architectures roughly halves the local-inference gap for Qwen's newest hybrid models on consumer hardware.
Importance: 2/5
Notable inference performance work in nightly builds, official releases