KV Cache as an Agent Runtime: What Changes

KV-cache manipulation can let pretrained language models observe, reason, and respond concurrently without changing their weights. Yandex's AsyncReasoning experiments demonstrate a working mechanism for thinker, writer, and new-input streams; its broader agent-runtime design remains an engineering direction with an SGLang implementation still in progress. The shift matters because inference state becomes a control plane that now needs scheduling, isolation, provenance, and recovery.

Reading time: 8 minutes · Word count: approximately 1,500

TL;DR

• A KV cache can become shared, multi-view inference state rather than a flat token history.
• AsyncReasoning uses positional transformations to let thinking and writing progress concurrently without retraining.
• Reported latency gains are benchmark-specific and trade against accuracy when writing advances too early.
• Yandex's multi-stream agent runtime and Doom demo are preliminary, not a released production stack.
• Writable inference state needs ownership, visibility, versioning, replay, and reclamation rules.

Three evidence levels

The phrase KV cache as an agent runtime combines results at different maturity levels.

First, the AsyncReasoning preprint provides a concrete implementation and experiments. It splits inference into user-input, private-thinking, and public-response streams, rearranges their logical positions, and periodically lets the model decide whether the writer should pause.

Second, the Yandex research post generalizes that mechanism into an architectural position: cache blocks can act like shared memory, attention views like access mappings, and mode changes like scheduling events. It describes named streams for perception, reasoning, speech, actions, tool calls, and tool results.

Third, Yandex says it is developing a continuously updated Qwen3.5 Doom agent and a broader SGLang implementation. Those are current work and planned publication, so they should be treated as prototypes rather than established production capabilities.

The paper demonstrates a mechanism. The blog proposes a runtime abstraction. The production system remains unfinished.

From append-only cache to multiple logical views

During standard autoregressive inference, a transformer stores attention keys and values for prior tokens so it does not recompute the whole prefix at every step. The cache normally follows one logical order: prompt, reasoning, answer.

AsyncReasoning breaks that single view into blocks. The thinker should see the prompt, the writer's current output, and then its own thoughts. The writer should see the prompt, current thoughts, and then its own output. Physically copying and re-encoding the blocks would erase much of the efficiency benefit.

For models using rotary positional embeddings (RoPE), attention depends on relative positions. The implementation stores a cache block once in block-local coordinates and transforms the query differently for each logical view. The same physical block can therefore appear at different positions to the thinker and writer.

This changes the cache's role:

• Before: an optimization that avoids recomputing a fixed prefix.
• After: reusable state whose ordering, visibility, and update timing can be controlled by the inference engine.

No weight update is required. The runtime changes how existing execution state is exposed.

Concurrent thinking still needs synchronization

The writer cannot safely outrun the thinker on every task. AsyncReasoning periodically asks the model whether its private thoughts are ahead of the public response. If the answer indicates sufficient progress, both streams continue; otherwise the writer pauses.

This policy produces an explicit accuracy-latency tradeoff. The paper reports up to an 80-fold reduction in time to the first non-thinking token and up to a 12-fold reduction in total user-perceived delay across its tested settings. The abstract limits the first-response claim more concretely: from minutes to five seconds or less.

Those are best-case results, not a universal service-level objective. In one Qwen3-32B MATH-500 comparison on an A100, time to first public token fell from 346.50 to 4.23 seconds and total user-silence delay from 346.99 to 28.66 seconds, while accuracy moved from 0.79 to 0.76. The experiments used Qwen3 and GPT-OSS variants on selected math, commonsense, safety, and partial-input benchmarks. A more aggressive continue bias reduced delay but sometimes reduced accuracy because the writer answered too early. Smaller models lost more accuracy under mode switching.

The scheduler is therefore part of model behavior. A production system needs to version and evaluate the pause policy alongside the model and prompt.

New input can enter while reasoning continues

The paper also tests partial prompts revealed as shards. New information is inserted into prompt, thinker, and writer blocks after decoding starts. When shards arrive early enough, evaluated models approach the accuracy of receiving all information upfront. Accuracy declines as updates arrive later, and the current method does not handle information arriving after generation has completed.

This is the key agent-runtime capability. A camera frame, tool result, or user correction no longer has to wait for the current reasoning turn to finish. The inference engine can make it visible to selected streams and continue from persistent state.

That advantage comes with a new correctness problem: which outputs were generated before an update, which reasoning saw it, and which actions became stale? The cache needs lineage, not only speed.

The minimum runtime contract

Treating cache state as shared memory requires at least seven controls.

Add quotas and isolation at the physical memory layer. A long-lived cache can leak information across requests, retain sensitive tool results, or consume GPU memory without bound if ownership and deletion are vague.

Safety becomes concurrent too

AsyncReasoning's HarmBench experiment illustrates both promise and risk. On Qwen3-32B, the paper reports an attack success rate of 6.5% for non-thinking, 12.5% for standard thinking, and 10.0% for default AsyncReasoning. A safety-specific prompt reduced the reported rates to 0% for synchronous thinking and 0.5% for AsyncReasoning. In that setting, asynchronous safety reasoning reduced median time to first token from 32.3 to 22.2 seconds and total delay from 45.3 to 35.0 seconds while preserving 0.93 accuracy on MATH-500.

These are 200 HarmBench validation samples evaluated with an LLM judge. They show that a background safety stream can help in one setup; they do not establish a general guardrail. The same experiment also shows that additional reasoning can increase harmful compliance without an explicit safety policy.

A runtime should treat safety as an independent decision stream with authority to pause or revoke public output and actions. It should also log which tokens were released before the decision, because a late block cannot retract an irreversible side effect.

What belongs in the model, harness, and runtime

The emerging stack has three state domains:

• Model weights: slow-changing capabilities and learned behavior.
• Inference runtime: active token representations, visibility, concurrency, and scheduling.
• Agent harness: durable task state, tools, permissions, checkpoints, external memory, and business workflow.

Moving interaction into the inference runtime can cut latency and avoid repeatedly serializing context through the harness. It does not eliminate the harness. Tool authorization, durable records, retries, human approval, and external side effects remain outside attention memory.

The clean interface is event-based. The harness submits a typed observation or tool result; the runtime records its provenance, exposes it to authorized streams, and returns outputs tagged with the state version they observed. Before an action executes, the harness checks that its version is current and its permission is valid.

Frequently asked questions

What is a KV cache in an LLM?

It stores attention keys and values from prior tokens so the model can generate the next token without recomputing the full prefix. The new work uses those stored states as reusable blocks with multiple logical views.

Does modifying the KV cache retrain or fine-tune the model?

No. AsyncReasoning changes inference-time positions, visibility, and scheduling. The model weights remain unchanged.

Is Yandex's agent runtime available today?

The AsyncReasoning paper has a reference implementation. The broader named-stream runtime, SGLang integration, and continuous Doom system are described as ongoing work in the September 2026 post.

Can KV-cache state replace long-term Agent memory?

No. It is active inference state optimized for model execution. Durable memory needs semantic records, access policy, versioning, retrieval, and lifecycle management outside the cache.

What is the main production risk?

An output or action may be based on stale, unauthorized, or cross-request state. Isolation, state-version checks, replay logs, and side-effect gates are prerequisites for deployment.

References

• Yandex Research: The KV cache as an agent runtime
• AsyncReasoning paper, version 3
• Yandex AsyncReasoning reference implementation
• Hogwild! Inference
• SGLang
• Latent workspace vs external Agent memory
• Agent sandbox persistence model

The practical next step is to implement one interruptible two-stream workflow and log every state transition. Measure stale-output rate, accuracy, latency, memory growth, and cross-request isolation before calling the cache a runtime.