Original title: You Could Have Come Up With Kimi Delta Attention
Article
Starting from causal softmax attention, the article recasts the core computation as linear attention by removing normalization, yielding a fixed-size state matrix that stores summed key-value outer products. It then shows why plain linear writes are problematic: they accumulate instead of overwrite, so new keys can distort old memory. DeltaNet fixes this by adding only the prediction error for the current key, scaled by a learned factor, and the article links this correction to a one-step online gradient update on reconstruction loss. The update is a rank-one outer-product change that acts locally in key space. Gated DeltaNet adds a global retention gate applied before prediction and correction, enabling exponential forgetting of stale memory but at a single rate for all channels. Kimi Delta Attention generalizes the gate to a per-channel vector, producing independent decay per key dimension and creating a diagonal-plus-low-rank transition while keeping the same key-to-value state shape. The post then treats execution as a scheduling question rather than a new formula: fused recurrent kernels decode quickly with one-token stateful steps and low latency, while chunkwise kernels improve throughput for training and long prefill by converting intra-chunk dependencies into matrix operations and triangular solves. Both paths are mathematically equivalent, and the article emphasizes they differ mainly in performance tradeoffs between latency and arithmetic intensity.
Readers respond positively to the explicit notation section and math/physics toggle, arguing that uniform notation reduces friction in machine-learning papers and makes shapes easier to reason about. Several commenters explicitly reject the titular tease, noting they could not derive the method themselves and appreciating the detailed derivation quality. A few appreciate the braket explanation and request context for why it was chosen, while others remain skeptical of the framing and of claims of originality. One substantive critique is that the method is more of an incremental combination of familiar linear-attention ideas than a fundamental breakthrough, and that attention’s historical impact was driven by practical parallelism and hardware assumptions rather than this variant alone. Another comment adds a historical computation angle, comparing it to LSTM-style ideas and arguing that fully unrolled approaches were mostly impossible before modern compute scales. Some readers also mention that modern ML and Transformer data-flow is still hard to parse and ask whether the article clarifies real-time execution across steps. Overall, the discussion values educational clarity and notation consistency but is divided on whether the approach is innovative or mostly a reassembly of known pieces.