How to Benchmark Qwen3.8-27B for Real Workloads

A Qwen3.8-27B speed result describes a system, not a model. The measured system includes a particular weight artifact, quantization, inference engine, cache state, context shape, reasoning setting, task, hardware topology, and token-counting method. This guide turns those variables into a benchmark contract, then uses two first-hand deployments to show why one tokens-per-second number cannot support a buying decision.

Reading time: 7 minutes · About 1,500 words

TL;DR

• Freeze the exact model artifact, quantization, engine build, prompts, output limits, cache state, and reasoning mode.
• Report aggregate throughput, per-stream decode, time to first token, quality, failures, and completed-task cost together.
• A 2x RTX 5090 test scaled from 268 tok/s at one request to 962 tok/s at eight, while per-stream decode fell from 286 to 141 tok/s.
• That curve is useful for the tested NVFP4 and SGLang stack; it is not a universal Qwen3.8 hardware ranking.
• Benchmark the workload users actually run, then observe it in production long enough to find the next bottleneck.

The model name identifies only one layer

The official Qwen3.8-27B model card describes a 27B dense vision-language model with a native 262,144-token context window. Its hybrid layout mixes Gated DeltaNet blocks with gated attention, and its deployment docs show SGLang, vLLM, and TokenSpeed as supported engines.

Several behavior controls change the workload before hardware sees it:

• Thinking is enabled by default.
• reasoning_effort supports xhigh, medium, and low.
• preserve_thinking is enabled by default and keeps reasoning history across turns.
• Thinking and non-thinking modes use different recommended sampling settings.
• Static YaRN can extend context, while Qwen warns that it may reduce short-context performance.

The model card also makes a crucial systems point: lower reasoning effort can speed up one turn yet increase failures, retries, total latency, and token use in multi-turn agent tasks. A benchmark that measures only decode speed can reward a configuration that completes less useful work.

The artifact identity matters just as much. BF16, FP8, NVFP4, GGUF, GPTQ, and other quantizations can come from different publishers, calibration data, kernels, and conversion versions. Treat Qwen3.8-27B as a family name. Record the repository, revision or digest, quantization method, and any draft model as part of the test subject.

One useful concurrency curve, with a narrow scope

A first-hand 2x RTX 5090 deployment report tested an NVFP4 Qwen3.8-27B artifact with tensor parallelism across two cards, a vendor SGLang build, and DFlash2 support. The concurrency test used the same math prompt, disabled thinking, set max_tokens=1024, and increased concurrent requests from one to eight.

The report defines aggregate throughput as all generated tokens divided by wall-clock time, including prefill. Per-stream decode excludes prefill. That distinction explains why the one-request aggregate number is slightly lower than pure decode.

This curve supports a practical conclusion for that machine: eight simultaneous streams still receive fast interactive decoding while total throughput grows. It also shows why the word fast needs a workload. A single user may value 286 tok/s. A small agent pool may value 962 aggregate tok/s with 141 tok/s per stream. A batch pipeline may accept lower interactivity for more total throughput.

It does not establish that NVFP4 is always better than FP8, that SGLang always beats vLLM, or that two 5090 cards beat every alternative. Those comparisons require matched artifacts, engines, contexts, quality tests, and power boundaries.

A second system reveals different bottlenecks

An independent NVIDIA Developer Forum deployment measured one and two DGX Spark systems with NVFP4, SGLang, DFlash2, and tensor parallelism over RoCE.

The author reported 52 to 61 tok/s for single-Spark code generation and 87 tok/s for dual-Spark, while a repeated 16k prefix reached first token in roughly 0.44 seconds on one Spark and 0.74 seconds on two. The dual system improved decode while adding communication latency to cached turns.

The same report documents eight compatibility fixes around reasoning-effort mapping, structured output, multi-turn history, cache accounting, and streaming token IDs. These are part of performance because a fast server that breaks the client workflow has zero useful throughput for that request.

The two reports should not be merged into a leaderboard. They use different hardware, interconnects, software builds, tasks, and measurement procedures. Together they demonstrate a more valuable rule: hardware speed, cache behavior, protocol compatibility, and task quality must be measured as one workflow.

Use a benchmark contract, not a spreadsheet of anecdotes

Freeze the following fields before running comparisons.

Run at least four workload cells:

1. Cold single-turn requests to expose model load, prefill, and compilation costs.
2. Warm repeated-prefix turns to measure cache behavior.
3. Concurrent independent requests to map throughput and per-user interactivity.
4. Multi-turn task completion with tools and quality checks to measure the real outcome.

Repeat each cell enough times to report a distribution. Separate warmup from the measurement window. Count actual model tokens rather than streaming events because speculative decoders can emit several tokens in one event.

Measure quality and retries beside speed

Quantization and speculative decoding need a quality gate. Select tests that represent the deployment: executable coding tasks, schema-valid extraction, tool-call completion, long-context retrieval, or domain evaluation. Freeze decoding settings and run every candidate against the same inputs.

The decision metric should be cost per accepted task, not tokens per second alone:

accepted-task cost =
  (compute + energy + operator time + retry cost) / accepted tasks

For an agent workflow, also track turns to completion, tool failures, repeated reasoning, and human repair time. A lower reasoning setting that creates more retries can lose even when every individual response arrives faster.

Quantization labels need their own skepticism. Two files called INT4 can use different group sizes, calibration sets, outlier handling, kernels, and accumulation precision. Compare artifacts, not labels.

Move from benchmark to production observation

A short benchmark identifies a candidate configuration. Production observation decides whether it fits.

Run the chosen stack for one to two weeks and collect request-shape distributions, cache hit rate, queue depth, TTFT, decode rate, errors, retries, energy, and accepted-task outcomes. Then identify the actual constraint.

If users already receive 141 tok/s while agents wait on tools, more decode speed has little value. If a long context exhausts KV cache, memory becomes the priority. If schema failures cause retries, protocol and decoding controls are the bottleneck. If utilization stays low, elastic capacity may beat owned hardware.

The model is only one replaceable component. The durable asset is a benchmark and observability loop that can retest the next quant, engine, or GPU without changing the definition of success.

Next action: save the benchmark contract with every result. A number without its configuration and quality gate belongs in a conversation, not a procurement decision.

FAQ

What is a good Qwen3.8-27B tokens-per-second result?

It depends on the artifact, engine, hardware, context, cache state, reasoning mode, concurrency, and task. Define an interactivity target and accepted-task target first, then compare configurations under that contract.

Should I benchmark with thinking enabled?

Benchmark the mode used in production. For mixed workloads, run separate thinking and non-thinking cells. Qwen's model card recommends different sampling parameters and warns that lower reasoning effort can increase retries.

Can I compare NVFP4, FP8, and INT4 results from different posts?

Use them as discovery signals. A defensible comparison needs matched tasks, quality checks, engine versions, hardware boundaries, and measurement methods. Quantization labels alone do not establish equivalence.

Which metrics matter most for an agent server?

Track time to first token, inter-token latency, aggregate throughput, cache hit rate, tail latency, task success, retries, tool failures, and cost per accepted task. Agent users experience the entire loop.

References

• Qwen. Qwen3.8-27B model card.
• Qwen. Qwen3.8 official repository.
• Yage. 2x RTX 5090 Qwen3.8-27B production test, August 24, 2026.
• NVIDIA Developer Forums. Qwen3.8-27B NVFP4 on single and dual DGX Spark, August 2026.
• SGLang. Qwen3.8-27B RTX 5090 cookbook verification issue.