OpenAI Jalapeño Benchmark Audit: Full-Stack Inference

OpenAI's first Jalapeño results support a strong claim: its custom inference system delivered more throughput per watt and lower latency than the compared NVIDIA systems across three public models. The deeper result is architectural. Jalapeño turns chip, memory, network, compiler, model, and serving software into one optimization loop, while the current benchmark leaves long-context agent workloads and production economics for the next test.

Reading time: 7 minutes · About 1,500 words

TL;DR

• Jalapeño led the disclosed InferenceX comparisons on GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T.
• The headline range is 1.5 to 1.9 times more peak throughput per watt and 1.7 to 3.6 times lower end-to-end latency.
• These are fixed 8k/1k, single-token-prediction runs normalized by published package TDP.
• SemiAnalysis verified the runs in OpenAI's lab, while explicitly noting that it did not run the full suite or see AgentX results.
• The durable advantage, if it survives broader tests, is a feedback loop across the whole inference stack.

Put the headline numbers back into their test cells

OpenAI published three comparisons on August 25, 2026. All use the public InferenceX benchmark, nominal 8k input and 1k output sequences, and single-token prediction.

The source tables also report substantially lower time between tokens and much higher throughput at the comparison system's best previous interactivity point. Those numbers show the shape of the serving curve. Jalapeño keeps more throughput while preserving a responsive user experience.

Power normalization deserves equal attention. OpenAI divided performance by each accelerator's published package thermal design power, using 700 W for Jalapeño, 1,200 W for GB200, and 1,400 W for GB300. OpenAI says Jalapeño's sustained measured power stayed at or below 550 W on the tested workloads. The charts therefore use a conservative denominator for Jalapeño, but they still compare package ratings rather than identical wall-power measurements for complete systems.

The right conclusion is specific: Jalapeño produced a better latency and package-TDP-normalized throughput envelope in the disclosed configurations. That is already important. Fleet economics, facility power, yield, reliability, and utilization require more evidence.

Why a chip result is really a systems result

Language-model inference changes bottlenecks during one request. Prefill processes the prompt and tends to be compute-heavy. Decode emits tokens sequentially and tends to be constrained by memory bandwidth. Model parallelism introduces communication. KV-cache placement determines how much state moves across memory and links.

Optimizing one phase can move waiting somewhere else. A very fast compute unit can sit idle while model state crosses a link. A large memory pool can still lose to poor scheduling. A high peak token rate can disappear at the latency target an interactive agent needs.

OpenAI says Jalapeño was designed across the chip, memory, network, compiler, runtime, and rack-scale system. It can explicitly place model state and keep the KV cache local while activating different resources for each phase. A large network domain keeps more of a workload inside one connected system.

This is the key shift: the accelerator becomes part of a control loop rather than a standalone component.

The loop can observe real serving workloads, identify whether compute, memory, communication, or scheduling is limiting performance, and change hardware and software together. OpenAI's companion full-stack strategy article extends that loop from chips and models to products, demand, and capacity allocation.

The engineering trade: broad balance over one perfect phase

Jalapeño uses one general inference architecture across the disclosed models. SemiAnalysis reports that the benchmark used single-token prediction, without speculative decoding or prefill-decode disaggregation. That matters because several competing systems reach their best points with more specialized serving techniques.

OpenAI's choice appears to favor a fungible resource pool. The same system can handle changing ratios of prompt processing, cache reads, cache writes, and output generation. This sacrifices some theoretical efficiency available from permanently separating prefill and decode hardware, while reducing the risk that a fixed fleet mix becomes wrong as workloads change.

That trade fits agent infrastructure. Agent traffic is bursty, multi-turn, cache-heavy, and sensitive to serial latency. A balanced pool can be easier to schedule and operate than several narrowly optimized pools. The current data shows that the balance works on fixed sequences. It does not yet prove the same result under a realistic agent session graph.

What SemiAnalysis verified, and where verification stops

The benchmark has stronger provenance than a vendor slide alone. InferenceX publishes recipes, logs, artifacts, driver and framework versions, token counts, latency records, and power telemetry for normal public runs. SemiAnalysis says its team went to OpenAI's lab and verified the disclosed Jalapeño InferenceX runs with OpenAI engineers.

The same report states two limits. The figures were provided by OpenAI, and SemiAnalysis did not run the full InferenceX suite. It also had not seen AgentX results.

AgentX is the more demanding next gate because it replays long, multi-turn coding sessions with shared prefixes, pauses, subagent fan-out, and repeated KV-cache reuse. Its median reconstructed request is far longer than an 8k prompt. The scenario also stresses routers, cache policy, scheduling, and tail latency over a one-hour profiling window.

This creates a clean evidence ladder:

1. Working silicon exists and ran three public model families.
2. Fixed-sequence InferenceX results show a strong serving envelope.
3. Independent observers verified those runs in the lab.
4. Long-context AgentX, public raw artifacts, production stability, and economics remain open gates.

The next benchmark should test the feedback loop

The most valuable follow-up would test more than raw speed.

• AgentX results: long contexts, multi-turn sessions, prefix reuse, and subagent bursts.
• Measured system power: wall power for accelerator, host, memory, networking, and cooling under the same service-level objective.
• Quality parity: task-quality checks across every optimization mode, especially when speculative decoding or lower precision is introduced.
• Fleet behavior: tail latency, failure recovery, degraded nodes, software upgrades, and utilization over weeks.
• Economics: silicon yield, rack cost, networking, operations, depreciation, and useful tokens per dollar.
• Portability: performance across changing model architectures without a bespoke rewrite for every release.

A result that survives those gates would support a stronger thesis. OpenAI would own a first-party path that improves through workload feedback while keeping external suppliers as credible alternatives. That can increase bargaining power, reduce serving cost, and shorten the loop from model design to deployed performance.

The control loop is the strategic asset

Jalapeño's first benchmark is impressive because a first-generation custom system reached the Pareto frontier across three public models. Its strategic importance comes from something harder to copy than a single chart.

OpenAI can observe how frontier models behave in products, redesign the serving stack around those workloads, change the chip and network, and feed the result back into model and product decisions. The chip is one actuator inside that loop.

For infrastructure buyers, the practical lesson is simple: evaluate the system at the user experience and workload you actually operate. Peak tokens per chip remain useful, but matched latency, power, cache behavior, reliability, and completed-task cost decide whether an inference platform creates value.

Next action: treat every new accelerator claim as a test matrix. Freeze the model, precision, sequence shape, quality target, latency target, and power boundary before comparing the numbers.

FAQ

Is OpenAI Jalapeño faster than NVIDIA Blackwell?

In the three disclosed 8k/1k InferenceX comparisons, Jalapeño delivered higher package-TDP-normalized throughput and lower latency than the listed GB200 or GB300 configurations. The public evidence does not yet establish superiority across every model, serving method, long-context workload, or total-cost boundary.

What is InferenceX?

InferenceX is an open, continuously updated inference benchmark from SemiAnalysis. It measures throughput, interactivity, latency, energy, and cost across real accelerators and serving stacks, with public recipes and run artifacts.

Why does AgentX matter?

AgentX replays long, multi-turn agent sessions with shared prefixes and subagent fan-out. Those patterns stress KV caches, routers, scheduling, and tail latency in ways that a fixed 8k/1k request cannot.

What does full-stack inference mean?

It means optimizing the model, compiler, serving runtime, chip, memory, networking, and rack as one system. The goal is to improve completed user work under latency, power, reliability, and cost constraints.

References

• OpenAI. Jalapeño's first results show industry-leading speed and efficiency in AI inference, August 25, 2026.
• OpenAI. The full stack behind abundant intelligence, August 25, 2026.
• SemiAnalysis. OpenAI Jalapeño: Better Than Nvidia Blackwell, August 25, 2026.
• InferenceX. About and reproducibility.
• InferenceX. AgentX methodology.