DeepInfer Deterministic Inference: An Output Contract

DeepInfer turns deterministic inference into a feature with explicit conditions. Its evidence includes batch-invariant kernels, fixed sampling state, and tests that compare outputs across batch compositions. This article converts those claims into an output contract that defines what must match, which variables may change, which hardware path is covered, and where the open-source release stops short of a production service.

Reading time: 10 minutes | Length: about 2,100 words

TL;DR

• DeepInfer's public release supports DeepSeek-V4.1-Flash and describes batch-invariant inference plus deterministic sampling.
• Deterministic sampling currently requires the NVIDIA path, DEEPINFER_BATCH_INVARIANT=1, variable-length MTP disabled, and a fixed seq_seed.
• The repository includes a strong batch-invariance test: 64 sequences run in a batch and then individually, with token IDs, log probabilities, top-token IDs, and top-token probabilities compared exactly.
• That test does not establish cross-hardware, cross-compiler, cross-tensor-parallel, or cross-version identity. Those belong in separate acceptance cells.
• The released system is an offline inference engine. A complete service still needs a KV-cache system, prefill/decode disaggregation, and replica load balancing; the repository also lacks a paired cost table for deterministic mode.

Determinism has four different meanings

Teams often collapse several guarantees into one word. That creates false confidence before any framework comparison begins.

Temperature zero only addresses part of the sampling policy. It does not force GPU reductions, attention kernels, routing, or batching to follow the same numerical path. A small score difference can change the winning token at a close boundary, and the new token changes every later decoding step.

Thinking Machines Lab's analysis frames batch invariance as the key missing property in many serving systems. Official vLLM and SGLang documentation now expose related modes. DeepInfer enters an active engineering field rather than inventing determinism from scratch. Its value lies in the specific contract and implementation it ships for the DeepSeek stack.

What DeepInfer actually promises

The DeepInfer repository calls itself the framework behind DeepSeek's large-scale production services, while describing the open-source release as offline inference with core serving features.

Its README makes two relevant claims:

1. Batch-invariant inference keeps model results identical across different batch compositions. It is enabled with DEEPINFER_BATCH_INVARIANT=1 and currently marked NVIDIA-only.
2. Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. It requires batch-invariant mode, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed. This path is also marked NVIDIA-only.

The conditions are part of the feature. Removing any of them changes the claim.

The public support matrix is narrow at launch:

• model: DeepSeek-V4.1-Flash;
• accelerators: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs;
• deterministic and batch-invariant flags: documented for NVIDIA only;
• runtime: Linux, Python 3.12, with CUDA 13 on NVIDIA or CANN 9.2 on Ascend.

This prevents a common reading error. DeepInfer supports Ascend as an accelerator, but the README does not promise that deterministic sampling works on Ascend.

The repository test is stronger than a feature badge

The most useful evidence sits in test_batch_invariant.py, not in the feature list.

The test constructs 64 random token sequences with varied prompt lengths, runs them together, and then runs each prompt individually. It compares four outputs:

• generated token IDs;
• token log probabilities;
• top-token IDs;
• top-token log probabilities.

The individual replay receives the token sequence from the batched result as a golden path. This lets the test compare the same decoding trajectory, including the reported probabilities, rather than allowing an early sampling divergence to make every later position incomparable. The assertion requires exact tuple equality.

That is a meaningful verification interface. It checks more than whether the final text happens to look the same.

The test also defines its scope:

• one parameterized model case;
• expert parallel size 8;
• a fixed maximum batch of 64 sequences;
• generated token and probability outputs under the repository's configured environment.

Because the single-request replay supplies the batched token sequence as golden_token_ids, this test directly checks numerical invariance along a fixed decoding path. It does not independently demonstrate that unconstrained sampling with a fixed seq_seed repeats the same natural generation across multiple runs. The repository's sampler tests check statistical behavior rather than seeded replay. The code therefore provides strong evidence for a batch-invariant computation path, while the broader deterministic-sampling claim still needs a separate deployment test.

DeepInfer separately includes a vision-encoder batch-invariance test that compares image embeddings with torch.equal across repeated and mixed image shapes. This shows that the project treats batch invariance as a component-level property, not only an end-to-end demo.

Repository tests are primary evidence for implementation intent. A green upstream test does not prove that a local deployment, a different commit, or an unsupported topology preserves the same property. Production acceptance still needs to run on the actual stack.

The public GitHub Actions view currently shows security and dependency workflows rather than a hardware pytest record for these model tests. The test source is inspectable; a public run artifact on the claimed accelerator is not part of the available evidence.

Convert the feature into an acceptance matrix

A deterministic mode should ship with a matrix of controlled changes. Each row defines the invariant, the variable, and the comparison rule.

The first four rows exercise batch invariance. The later rows test broader reproducibility.

Research on determinism across tensor-parallel sizes is a useful warning: batch-invariant computation does not automatically make different parallel decompositions identical. Compiler versions, kernel selection, collective ordering, and hardware may introduce other paths.

The contract should also name the comparison level. Exact token equality is appropriate for user-visible replay. Exact log-probability equality is stronger and useful for debugging or reinforcement-learning pipelines. Bitwise tensor equality is stronger again and can impose a larger implementation cost. A vague requirement such as "same answer" cannot distinguish semantic similarity from numerical reproducibility.

Asynchronous scheduling and determinism solve different problems

DeepInfer enables asynchronous scheduling by default to overlap CPU batch scheduling with device execution. It also provides DSpark with confidence-scheduled verification and variable draft lengths.

These features address utilization and decoding speed. They do not create determinism by themselves. The README explicitly disables variable-length MTP for deterministic sampling. That is a concrete example of a product tradeoff becoming part of the output contract.

The right benchmark therefore has at least two axes:

1. performance envelope: prefill throughput, decode throughput, time to first token, inter-token latency, and tail latency;
2. reproducibility envelope: token equality, score equality, batch invariance, and supported stack variations.

Publishing only the first axis turns determinism into an unchecked label. Publishing only the second hides the capacity cost. DeepInfer's repository includes reproducible performance scripts and Pareto charts measured on four SM100 GPUs for prefill and 64 for decode, but it does not publish paired deterministic-on versus deterministic-off results. Each deployment needs to measure that delta with its own workload.

This is the same reason inference tuning should start with workload shape. A coding agent with long repeated prefixes has a different bottleneck from short independent chat requests. Determinism can move the bottleneck again by changing kernel choices or speculative decoding behavior.

Offline determinism is one layer of production reproducibility

DeepInfer's README lists three components required for a complete inference service but absent from the release:

• a KV-cache system;
• vision-encoder, prefill, and decode disaggregation;
• load balancing across data-parallel replicas for prefill and decode.

These are operationally significant. A load balancer can route two identical requests to replicas with different model or kernel versions. A cache layer can reuse state produced under an older configuration. A disaggregated service can cross multiple hardware and software boundaries before emitting a token.

The production contract should therefore attach an environment fingerprint to every replay-sensitive request:

model checkpoint + tokenizer + DeepInfer commit
+ kernel/operator versions + compiler/runtime
+ accelerator model + parallel topology
+ deterministic flags + sampling parameters + seed
+ cache lineage + serving replica

When outputs differ, this fingerprint turns a generic nondeterminism report into a bounded comparison.

It also exposes a second source of variability in DeepInfer's evaluation pipeline. The benchmark documentation separates generation from scoring. AIME and GPQA can use a deterministic rule-based judge, while the default LLM judge remains subject to its own sampling. HLE supports only the LLM judge. Deterministic generation and deterministic evaluation are separate claims.

A release gate for deterministic inference

Before a team enables deterministic mode in production, require five artifacts:

1. Frozen corpus: real request shapes, including long prompts, mixed lengths, structured outputs, and multimodal cases where applicable.
2. Acceptance matrix: every supported batch, scheduler, topology, and upgrade variation with an explicit equality rule.
3. Cost report: throughput, latency, memory, and capacity measured with the mode on and off.
4. Environment fingerprint: enough information to reproduce the exact inference path.
5. Drift gate: a failed cell blocks release or records an approved compatibility break.

This turns determinism from a configuration flag into a maintained product property. The feature is real only for the cells that continue to pass.

Frequently asked questions

What is batch-invariant LLM inference?

It means a request keeps the same declared result when the size, order, or contents of the surrounding batch change. The declaration should specify whether equality covers tokens, scores, or internal tensors.

Why can temperature zero still produce different outputs?

Temperature controls sampling behavior. Numerical results can still change when GPU kernels, reduction order, batch composition, parallelism, or software versions change. A small score change can flip a token decision.

How do I enable deterministic sampling in DeepInfer?

The current README requires DEEPINFER_BATCH_INVARIANT=1, DEEPINFER_ENABLE_MTP_VARLEN=0, and a fixed seq_seed in SamplingParams. The documented path is NVIDIA-only.

Does DeepInfer deterministic inference work on Huawei Ascend?

DeepInfer supports Ascend 950 NPUs, but its README labels batch-invariant inference and deterministic sampling as NVIDIA-only. The official documentation does not currently promise the same mode on Ascend.

Does batch invariance guarantee identical output across tensor-parallel sizes?

No general guarantee follows from batch invariance alone. Tensor-parallel topology changes computation and communication order. It requires its own acceptance test and an explicit support statement.

What is the performance cost?

DeepInfer does not publish a paired deterministic-on versus deterministic-off cost table in the repository. Measure throughput, tail latency, and memory on the target workload rather than importing a number from another engine.

Next action

Before enabling deterministic mode, run one frozen request through four cells: alone, inside a mixed batch, after a process restart, and under the target scheduler load. Compare token IDs and log probabilities, record the environment fingerprint, and measure the latency and throughput delta. The result defines the first verified cell of the deployment contract.

References

• DeepSeek DeepInfer repository
• DeepInfer batch-invariance test
• DeepInfer deployment Q&A
• DeepInfer evaluation documentation
• vLLM batch invariance
• SGLang deterministic inference
• Thinking Machines Lab: Defeating Nondeterminism in LLM Inference
• Deterministic inference across tensor-parallel sizes
• Microsoft Research: LLM-42 Verified Speculation