The Hidden Geometry of Broken Attention: Spikes, Sinks, and What They Cost You

If you’re building on top of large language models — quantizing them, running inference at scale, or trying to understand why certain efficiency tricks break — two phenomena have probably bitten you without you fully understanding why. Massive activations and attention sinks are among the most practically disruptive quirks of transformer internals, and a new paper systematically unpacks their anatomy and causal relationship for the first time.

What We’re Talking About

Massive activations are a quantization engineer’s nightmare: a handful of tokens develop extreme outlier values in a small subset of hidden-state channels. We’re not talking about values being 2–3× larger than average — in some models, these outliers are orders of magnitude beyond the norm, concentrated in just a few dimensions and a few tokens. This blows up the dynamic range that INT8 or INT4 quantization has to cover, degrading model quality dramatically unless you apply special per-channel or per-token scaling hacks.

Attention sinks are the phenomenon identified in the StreamingLLM line of work: certain tokens — typically the initial token(s) in a sequence, often the BOS token — attract a disproportionate share of attention mass across many layers and heads, regardless of whether those tokens are semantically relevant to the query. In a model generating the 500th token of a document, attention heads may still be routing significant probability mass back to token 0 even when it’s just a <s> delimiter with no meaningful content.

The suspicious part: these phenomena frequently co-occur, and they frequently involve the same tokens. That convergence is not a coincidence, but prior work left the why largely unexplored.

The Causal Question

The natural hypotheses are (1) massive activations cause attention sinks, (2) attention sinks cause massive activations, or (3) both are downstream of some shared upstream mechanism in how transformers learn to route information. Getting this wrong matters. If you suppress massive activations to fix your quantization pipeline, do you inadvertently disrupt the attention sink structure? If you modify attention patterns to implement sliding-window or streaming inference, are you breaking something the residual stream depends on?

The paper attacks this through systematic ablations — intervening on one phenomenon while measuring the other, isolating which tokens and channels are involved, and tracing the functional consequences.

Three Characters in the Title

The paper’s title is doing real work. “The Spike, the Sparse, and the Sink” names three distinct structural features that need to be understood separately before their interaction makes sense:

• The Spike refers to the extreme magnitude outliers in activation values — the sharp, localized anomalies that define massive activations.
• The Sparse refers to the highly localized nature of both phenomena: a small number of tokens across a small number of channels or heads. Neither effect is diffuse; both are pathologically concentrated.
• The Sink is the attention sink itself — the gravitational pull certain tokens exert on attention distributions.

By decomposing the phenomena this way, the authors can ask targeted questions: Is the sparsity of the spike what makes a token become a sink? Does becoming a sink reinforce the spike through repeated residual accumulation?

Why This Matters for Practitioners

Quantization. The dominant source of quality degradation in aggressive LLM quantization is outlier activations. Methods like SmoothQuant, LLM.int8(), and GPTQ each deal with this differently, but they’re all working around the same underlying pathology. Understanding whether massive activations are a cause of attention sink behavior — or merely correlated — tells you whether eliminating them is safe or whether it will cascade into broken attention patterns.

Efficient inference and KV cache compression. Sliding-window attention and streaming inference (e.g., StreamingLLM) preserve attention sink tokens specifically because evicting them collapses generation quality even when they’re semantically irrelevant. If the causal story is that these tokens carry load-bearing information in the residual stream because of their activation magnitudes, then KV cache compression schemes need to account for this, not just treat sink tokens as a heuristic special case.

Model editing and interpretability. Massive activations have been linked to specific behaviors: certain tokens act as “register” positions that accumulate global context. If the co-occurrence with attention sinks is causal rather than incidental, interventions aimed at understanding or redirecting information flow need to account for both phenomena simultaneously.

What to Watch For

The clearest immediate implication is for anyone building quantization or inference optimization pipelines: treat massive activations and attention sinks as a coupled system, not independent problems. Suppressing one without understanding its effect on the other is likely to produce regressions that are hard to diagnose because they’ll show up as subtle generation quality degradation rather than obvious failures.

More broadly, this kind of mechanistic anatomy work is increasingly necessary as the field moves from “transformers work, deploy them” to “transformers work in specific ways that we need to understand to build reliably on top of them.” The spike, the sparse, and the sink are not bugs — they appear to be emergent structures that trained models depend on. The question is learning their grammar well enough to work with them deliberately.