Moonshot AI's Kimi Team proposes replacing transformer residual connections with a depth-wise attention mechanism — claiming a 1.25x compute advantage with negligible overhead.
On March 15, 2026, Moonshot AI's Kimi Team published a technical report that takes aim at one of the most foundational and least-questioned components of modern neural network design: the residual connection. In doing so, the Beijing-based lab has opened a debate over whether the architectural scaffolding underlying virtually every large language model in production today — from GPT to Gemini to Claude — has been quietly limiting their potential for nearly ten years.
The paper introduces Attention Residuals, a mechanism that replaces the standard "add everything equally" accumulation of layer outputs with a learned, selective process. In a conventional transformer, each successive layer receives the sum of all prior layer representations — an ever-growing pile in which early signal can be diluted and no layer has any means of reaching back to preferentially retrieve what came before. The Kimi Team argues this is not a minor inefficiency. It is a structural flaw analogous to the sequential compression problem that limited recurrent neural networks before transformers displaced them.
Their solution is conceptually elegant: apply the same attention mechanism that transformers already use across tokens in a sequence, but apply it instead across depth — across layers. Each layer now computes a weighted combination of all preceding layer outputs, with weights that are learned during training. The model can, in effect, decide: for this particular computation, which prior layer's representation matters most?
The added parameter cost is deliberately minimal — one small learned vector and one normalization operation per layer. The team reports inference latency overhead of less than 2% and training overhead under 4% when using pipeline parallelism, a figure critical for production viability.
For large-scale deployment, the team developed a variant called Block Attention Residuals, which avoids the memory pressure of attending to every individual prior layer. Instead, layers are grouped into blocks and cross-layer attention operates over block-level summaries. The team found that approximately eight blocks captures most of the benefit at a fraction of the cost — a practical compromise that bridges theoretical appeal and engineering reality.
The benchmark results are the claim most likely to draw scrutiny. Tested on the Kimi Linear architecture — a mixture-of-experts model with 48 billion total parameters but only 3 billion activated — the method matched the performance of a baseline model trained with 25% more compute. That 1.25x compute efficiency figure was validated across five model scales, and the team reports the method outperformed all prior residual-improvement proposals, which either showed no gain or required prohibitively high memory costs.
The underlying diagnosis the paper offers — that standard PreNorm transformers suffer from hidden-state dilution, where individual layer contributions shrink as a proportion of the total as depth increases — is not new as an observation. What is new is the proposed remedy, and the claim that a lightweight attention mechanism over depth is sufficient to reverse it. Training diagnostics cited in the paper show more bounded hidden-state magnitudes and more evenly distributed gradients across layers under the new scheme, suggesting the architecture is doing what it claims structurally, not merely performing better on benchmarks by accident.
The paper arrives at a commercially significant moment for Moonshot AI. The company is currently in the process of raising up to $1 billion in funding at a reported valuation of $18 billion. The Attention Residuals work extends a line of architectural research built on the Kimi Linear architecture, released in October 2025, and follows a rapid product cadence: Kimi K2 in July 2025, Kimi K2.5 in January 2026.
Whether the findings replicate at the frontier scales used by the largest American and European labs — models orders of magnitude larger than the 48B tested — remains the central open question. Architecture papers that prove out at mid-scale have historically faced harder tests at the extreme end of the compute curve. The field will be watching. If the compute efficiency gains hold, the implication is not incremental: it would mean that a core structural assumption baked into a decade of AI development was leaving measurable performance on the table the entire time.