Muir makes a striking claim: a 1.34B-parameter linear-attention model trained at 4K can process a continuous 10M-token stream with a fixed 25 MB recurrent state and no perplexity drift. That result matters. It does not establish a general 10M-token context window.
The difference is measurement scope. The paper evaluates language-model stability at 10M tokens, exact passkey retrieval at up to 1M, multi-hop and document extraction mostly at short lengths, and training throughput from 2K to 32K on eight TPU7x chips. It also uses different Muir readout variants for the strongest streaming and single-key results.
The useful reading is therefore not “linear attention solved 10M context.” It is that Muir presents a credible mechanism for keeping a fixed-size state useful over much longer streams, while exposing a real trade-off between storing one distant association and reading several associations at different lags.
The claim needs four scorecards
| Question | Strongest reported evidence | What it does not establish |
|---|---|---|
| Does the recurrent state remain numerically and statistically stable? | Three continuous FineWeb-Edu streams of 10,485,760 target tokens; dynamic-readout Muir keeps the lowest perplexity through 10M | Arbitrary fact retrieval from anywhere in the stream |
| Can it retrieve one exact distant value? | Static-readout Muir reaches 90.0% zero-shot and 99.4% after a 64K cooldown on 1M-token S-NIAH-1 | Multi-hop reasoning or broad document comprehension at 1M |
| Can it combine information at different lags? | Full dynamic-readout Muir leads the tested multi-hop, multi-needle, and six-task extractive-recall comparisons | The same behavior at 1M or 10M lengths |
| Is it faster? | At 1.34B on eight TPU7x chips, Muir matches GDN-2 and crosses the tested Transformer near 11K training length | GPU inference latency, serving throughput, or a production cost result |
These distinctions are not caveats added after the fact. They are the architecture.
Why fixed state changes the problem
Softmax attention keeps token-addressable keys and values. Its inference cache grows with sequence length. Muir, like other gated linear-attention systems, compresses history into a fixed-size matrix state. The paper reports 25 MB of float32 recurrent state across 24 layers for its 1.34B model. Its comparison Transformer would require a 2.1 TB bfloat16 KV cache at 10M tokens under the paper's configuration.
Compression removes linear KV-cache growth, but it also removes the guarantee that every earlier token remains separately addressable. New key-value writes share the same finite state, so associations interfere. The system must decide what to decay, what to erase, what to write, and how to read rows whose timescales and norms have diverged.
This is the same boundary that matters when a KV cache becomes runtime state, but Muir changes the storage contract. A KV cache retains token-indexed state and pays a growing memory bill. Muir pays a fixed memory bill and accepts a learned interference problem.
Muir reallocates precision where state is sensitive
GDN-2 gives channel-wise decay and separate erase and write gates full expressive capacity. Muir's authors argue that this budget is misallocated.
Their trained-model probes find that state updates are about 13 times more sensitive to write perturbations than erase perturbations. Key activation and normalization concentrate energy in a small subset of channels, while value writes affect all value channels. In one intervention, freezing the write gate degrades validation NLL 14.5 times more than freezing the erase gate in both GDN-2 and Muir.
Muir therefore compresses decay and erase more aggressively, keeps more capacity for writes and readout, and reinvests the saved parameters in a wider feed-forward network. At fixed model budget, its FFN is up to 30.4% wider than GDN-2's in the reported configuration.
The second problem appears at read time. Slow-decaying state rows accumulate much more energy than fast rows. An unweighted query lets those high-energy rows dominate, which encourages stronger write-time erasure to suppress interference. That erasure shortens the memory horizon.
Muir adds a matched-filter readout gain. It reweights rows according to their learned dynamics, allowing the model to reduce erasure and preserve associations longer. This is a precision-allocation argument, not merely another attention layer: spend parameters on the gates and channels that change the state most, then correct energy imbalance when reading it.
The 10M result is a state-stability test
The length-extrapolation protocol is more careful than the headline suggests. The authors train at 4,096 tokens on 100B FineWeb-Edu tokens. They then evaluate 64 contiguous 128K documents and three continuous streams, each containing 10,485,760 target tokens. The 10M streams concatenate long documents and strip internal EOS markers so recurrent state accumulates without reset.
The dynamic-readout model maintains the lowest perplexity across the stream and avoids the drift seen in the static-readout variant. This is meaningful evidence that its state and readout remain usable under long, uninterrupted accumulation.
Perplexity is still a local next-token metric. Flat perplexity says that the growing history does not degrade average prediction quality under this protocol. It does not show that the model can retrieve any named fact from token 100,000 or combine evidence placed millions of tokens apart.
That separation also explains why context compression needs a decision-equivalence test. A compressed state can preserve predictive quality while losing a rare detail needed for one later decision.
Exact recall belongs to a different variant and length
The paper's strongest passkey result uses static readout. On RULER S-NIAH-1, a seven-digit value is hidden at one of ten depth bins and queried at the end. Each of eleven context lengths from 1K to 1M uses 160 examples.
Static-readout Muir keeps the full write-gate rank and reaches 90.0% zero-shot recall at 1M tokens. After a 300-step, 0.315B-token cooldown at 64K, it reaches 99.4%. That is a strong single-association result at 256 times the original training length.
Full Muir spends half of that write rank on token-dependent dynamic readout. It performs better when a task must select associations from several lags, but its single-passkey recall tapers from 64K onward. The paper reports near-perfect average passkey recall only through 32K for this dynamic variant.
The trade-off is coherent:
- Static readout preserves more capacity for writing one distant association and applies one stable channel filter.
- Dynamic readout changes the filter per query token, which helps multi-hop, multi-needle, and real document extraction where relevant spans occur at different distances.
- A fixed parameter budget prevents maximizing both at once in the tested design.
This is why the 10M streaming result and the 1M passkey result should never be merged into one context-window number.
Throughput is a hardware-specific training result
The throughput experiment uses 1.34B models on eight TPU7x chips at a fixed 524,288 tokens per step. Both Muir and GDN-2 use the authors' boundary-pivoted JAX chunkwise kernel. The Transformer uses TPU Splash Attention.
Muir and GDN-2 are roughly equal at 4K, around 179K tokens per second. Transformer remains faster through 8K and the curves cross near 11K. At 16K and 32K, Muir is reported as 1.64 and 2.74 times faster than Transformer. Muir also matches GDN-2 while carrying a wider FFN and dynamic readout.
This result shows that the proposed kernel can make the recurrence competitive on wide-matrix TPU hardware. It does not answer decode latency, GPU kernel maturity, batching behavior, memory fragmentation, quantized inference, or serving cost. Those require a separate deployment benchmark.
The kernel work is central, not plumbing. The paper shows that a naive chunkwise decay factorization overflows on 25% of heads in one stress test. Its boundary-pivoted construction avoids unsafe prefix division and keeps the trained systems in the reported stable conditioning regime. An architecture that is linear on paper still needs a numerically stable mapping to actual accelerators.
A practical evaluation contract
Before replacing attention layers with Muir or a similar recurrence, freeze four test groups.
- Streaming stability: contiguous domain text, no artificial resets, tail NLL and state statistics over the longest production horizon.
- Exact retention: single and multiple values at controlled depths, including early-context failures and overwrite pressure.
- Task utility: extraction, multi-hop reasoning, code navigation, and agent decisions whose evidence spans different lags.
- System performance: prefill and decode separately, target GPUs or accelerators, real batch sizes, quantization, failure recovery, and state-transfer cost.
Report static and dynamic readout as different configurations. Compare against a strong Transformer or hybrid baseline under the same parameter, data, token order, kernel maturity, and hardware budget. A flat perplexity curve cannot substitute for recall, and a passkey cannot substitute for task utility.
What the evidence supports today
As of 8 November 2026, Muir is a research architecture supported by a detailed preprint and a Google Research publication page. The reported experiments span 166M to 1.34B parameters, 100B training tokens, mechanistic probes, long-stream tests, and a hardware-specific kernel study. The paper does not link a Muir weights release, a dedicated implementation repository, or an independent reproduction.
The current evidence supports three conclusions. Sensitivity-aware gate allocation is a plausible way to improve a fixed recurrent state. Matched-filter readout exposes a measurable trade-off between distant single-key retention and multi-lag adaptability. Constant-state language modeling can remain stable far beyond training length under the authors' protocol.
Production replacement is a later claim. It requires released artifacts, independent reruns, larger models, hybrid configurations, GPU inference measurements, and application-level long-context tasks.
FAQ
Did Muir prove a 10M-token context window?
It proved stable language-model perplexity across three continuous 10M-token streams for the tested 1.34B model. Exact passkey retrieval was tested to 1M tokens, and broader recall tasks used shorter contexts.
Why does static readout beat dynamic readout on passkeys?
Static readout keeps a wider write gate and needs only one channel filter for a single query lag. Dynamic readout spends capacity on changing the filter across query tokens, which helps multi-lag tasks but reduces single-key write capacity under the fixed budget.
Is 25 MB the full inference memory footprint?
No. It is the reported float32 recurrent state across 24 layers. Model weights, activations, runtime buffers, local convolutions, batching, and serving infrastructure remain outside that number.
Is Muir faster than Transformer?
In the reported training test, it crosses the TPU Splash-Attention Transformer near 11K and is 2.74 times faster at 32K. This does not establish GPU inference or production serving performance.