NVIDIA reports faster hybrid-model inference with ReplaySSM

NVIDIA says ReplaySSM, developed with Princeton's Dao AI Lab, increases throughput for hybrid AI models and is integrated into TensorRT-LLM, vLLM and FlashInfer. The method also runs in production in GLM-5.3-Flash and Kimi-K3. Instead of rewriting a model's full fixed-size state after every token, ReplaySSM stores a checkpoint and recent inputs. This reduces memory traffic and lets speculative decoding discard rejected draft tokens without restoring full state snapshots. At batch size 256, NVIDIA measured decode steps running 1.20× to 1.47× faster across four models. Speculative throughput reached 1.85× and 2.10× vLLM's throughput on two models at batch size 512. Tests at batch size 1 showed no speedup.