DeepSeek V4.1 Flash on eight NVIDIA B300 GPUs

Executive summary

• The best configuration was four independent model replicas, each spread across two GPUs with TP=EP=2. With 32 continuously active users, it sustained 3,467.6 output tokens/s and 130.8 complete answers/minute during a ten-minute measurement window.
• Median time to the visible answer was 8.90 seconds, p95 was 25.23 seconds, and median end-to-end latency was 12.76 seconds. The benchmark included the model's reasoning tokens, not only the visible answer.
• All three eight-GPU layouts completed the benchmark without an error. The selected TP2 deployment later completed a one-hour soak with 8,015 successful responses, zero errors and zero container restarts.
• Four TP2 replicas used considerably more GPU memory than wider TP4 or TP8 instances. At peak load, the tightest GPU still retained 15.2 GiB of VRAM headroom.

These results measure serving infrastructure, not the factual quality of model answers.

Hardware and serving stack

We tested DeepSeek V4.1 Flash on one DGX B300 server. The machine had eight NVLink-connected NVIDIA B300 SXM6 GPUs, each with 275,040 MiB of memory, two Intel Xeon 6776P CPUs and 2 TiB of system RAM.

The checkpoint was pinned to revision dba1be0a40aa45a94ad051997016db3960a90277. SGLang reported an FP8 checkpoint, FP8 E4M3 KV cache, the DeepSeek V4 attention backend and the FlashInfer MXFP4 MoE runner. Requests used the streamed OpenAI-compatible chat API through a local gateway, so the results exclude public-network and browser latency.

The gateway kept every conversation on the same replica. This matters because a follow-up can reuse the prefix cache only when it reaches the worker holding that conversation's KV state.

Deployment layouts

Every candidate used all eight GPUs, but divided them differently:

The TP4 and TP8 runs were non-speculative baselines. The TP2 row was the tuned production candidate, so this is a comparison of deployable profiles rather than a topology-only microbenchmark.

The selected profile can be summarized as follows:

{
  "replicas": 4,
  "gpus_per_replica": 2,
  "tensor_parallel_size": 2,
  "expert_parallel_size": 2,
  "context_length": 1048320,
  "chunked_prefill_size": 4096,
  "max_total_tokens_per_replica": 8388608,
  "max_running_requests_per_replica": 16,
  "cuda_graph_max_batch_size": 16,
  "kv_cache_dtype": "fp8_e4m3",
  "speculative_decoding": "DSpark block 3",
  "hierarchical_cache": true,
  "strict_thinking": true
}

The model retains its native 1,048,576-position table. We serve 1,048,320 positions because speculative decoding needs a 256-position safety margin. Prompt and generated tokens share this limit. HiCache and pinned host Engram tables were enabled, and prefill ran in 4,096-token chunks.

Workload and request sizes

The main workload simulated 32 users chatting continuously without think time. Each session contained six natural question-and-answer turns about a public-domain book excerpt. After six turns, the client started a new session with a unique early prefix, making its first request cold. Later turns retained the document and preceding visible answers, allowing prefix-cache reuse.

The initial prompt mix was:

Responses had a maximum allowance of 32,768 tokens. Natural responses stopped before that limit; the winning ten-minute run averaged about 1,594 generated tokens per completed request, including reasoning. Only the visible answer was added to the next turn's conversation history.

We report both raw time to first token and time to the visible answer. DeepSeek can emit reasoning before its final answer, so raw TTFT alone understates the delay experienced by a reader.

Ten-minute results at 32 concurrent users

Throughput is calculated only from requests fully completed inside the 600-second launch window. Requests still in flight were drained and recorded separately.

The four-replica layout delivered 42.2% more output throughput than two TP4 replicas and 111.2% more than one TP8 instance. It also improved visible-answer latency. All three profiles kept an average of essentially 32 requests in flight and returned zero failed requests.

For the selected TP2 profile, latency remained fairly stable across the five initial context sizes:

These medians include cached follow-ups. Cold first turns are intentionally visible in the tail latency and in the separate controlled test below.

Why four TP2 replicas are not twice as fast as two TP4 replicas

Four model replicas do not imply twice the throughput. A TP4 worker distributes each forward pass over twice as many GPUs, while TP2 creates more independent schedulers and more copies of the model. Wider parallelism therefore recovers some of the apparent replica disadvantage.

At the same time, TP4 and TP8 must coordinate more ranks for tensor and expert-parallel operations. The extra collective communication and synchronization reduce the benefit of adding GPUs to one request. With 32 users, four TP2 workers also receive only about eight active conversations each, compared with sixteen per TP4 worker and all 32 on the TP8 worker.

Finally, DSpark block 3 was enabled on the tuned TP2 profile but disabled on the original TP4 and TP8 baselines. The result answers the operational question of which tested configuration serves this workload best. It does not isolate the effect of parallel width from speculative decoding.

Cold prompts and prefix caching

We also ran a controlled suite with 32 concurrent requests, the same input-size distribution and exactly 512 generated tokens per request. Each input was repeated three times, producing 96 requests. Kernel warm-up was excluded from the measurements.

On the selected four-TP2 deployment:

Prefix reuse increased fixed-work throughput by 6.3 times and reduced median TTFT by 94.2%. This is why session-aware routing is part of the deployment rather than an optional optimization.

VRAM, utilization and power

More replicas trade memory capacity for throughput. The telemetry below covers each ten-minute chat run and includes all eight GPUs.

No GPU reported an ECC error. The selected profile fits, but its 15.2 GiB worst-case reserve is much less forgiving than TP4 or TP8. Any increase in CUDA graph capture, token capacity or static memory allocation needs to be measured rather than assumed safe.

Testing 64 clients on two TP4 replicas

After the main comparison, we enabled DSpark block 3 on the two-TP4 layout and ran two shorter, 180-second tests directly after deployment. Because these runs started cold and used a shorter window, they should be compared with each other, not directly with the warmed ten-minute table.

Doubling the client count increased output throughput by 29.3%, not 100%. Median visible-answer latency rose by 49%, while p95 nearly doubled. The workers reached 31 of 32 running slots and their waiting queues reached 12 requests, but the token pool used only about 21% of its capacity. Compute and scheduling contention, not VRAM, set the limit.

One-hour validation and production choice

We retained four TP2 replicas with DSpark block 3. In a subsequent one-hour run, the 3,600-second measurement window completed 7,983 answers, sustained 3,549.2 output tokens/s and 133.05 answers/minute, and recorded zero errors. Median visible-answer start was 8.97 seconds, p95 was 23.31 seconds, and median end-to-end latency was 12.84 seconds.

Including the 20.4-second drain, the deployment returned 8,015 successful responses. All responses stopped normally, and the workers and gateway remained healthy with no container restart.

For this mixed long-context workload, independent TP2 replicas produced the best combination of throughput and latency. TP4 retains much more memory headroom and may be preferable when individual conversations need larger simultaneous KV allocations. TP8 is the simplest layout, but it leaves too much of this server's multi-user throughput unused.