Executive summary
- One NVIDIA B300 delivered 285 output tokens/s for a single short request and 972 tokens/s at concurrency eight. More concurrency added almost no throughput and mainly increased queueing.
- The first response over exactly 800,000 input tokens started after 71.4 seconds. A natural cached follow-up started after 1.59 seconds, although controlled runs varied from 1.62 to 11.15 seconds.
- The main load suite completed 1,788 requests without an error, preemption, or out-of-memory event. We retained an 8,192-token scheduler batch and a 1,048,576-token context limit for production.
These numbers measure infrastructure performance, not model answer quality.
What we tested
We deployed DeepSeek V4 Flash with vLLM 0.25.0 on one NVIDIA B300 SXM6 with 275,040 MiB of memory. The model used mixed FP4/FP8 weights, an FP8 KV cache, and DSpark speculative decoding with seven draft tokens. The client sent streamed, OpenAI-compatible requests directly on the server, so the results exclude public-network and web-interface latency.
Where useful throughput tops out
For a short workload with 1,024 input and 256 output tokens, concurrency eight gave the best balance between throughput and responsiveness.
| Concurrency | Output tokens/s | TTFT p50 / p95 | E2E p50 |
|---|---|---|---|
| 1 | 285 | 89 / 90 ms | 863 ms |
| 4 | 698 | 109 / 286 ms | 1,398 ms |
| 8 | 972 | 119 / 494 ms | 1,941 ms |
| 16 | 1,030 | 2,051 / 2,436 ms | 3,769 ms |
Moving from eight to sixteen concurrent requests added only 6% throughput, while median time to first token rose from 119 ms to 2.05 seconds. With open-loop arrivals, 2 requests/s was stable and queueing appeared at 3 requests/s. On a decode-heavy workload with 4,096 input and 2,048 output tokens, concurrency eight reached 1,695 output tokens/s.
Legal document with repeated follow-ups
We measured a typical workflow using a long court decision and a Czech question about the Constitutional Court's reasoning. For the stress test, we repeated the document until the formatted input contained exactly 800,000 tokens. A follow-up then reused the same document and the previous answer.
| Scenario | Input tokens | Prefix-cache hit | TTFT | End-to-end latency |
|---|---|---|---|---|
| Initial question | 800,000 | 0% | 71.40 s | 78.42 s |
| Cached follow-up | 801,622 | 99.80% | 1.59 s | 6.15 s |
| Same follow-up without cache | 801,622 | 0% | 71.56 s | 79.31 s |
In this natural run, caching reduced time to first token by 97.8%. It should not yet be treated as a fixed SLO. Across three cache-isolated repetitions, the hit rate stayed near 99.78%, while TTFT ranged from 1.62 to 11.15 seconds. The request queue remained empty, and the difference occurred inside vLLM's prefill stage.
What we kept
Raising the scheduler batch from 8,192 to 16,384 tokens made cold prefill 2.7% faster, but the complete two-question workflow was 0.8% slower. It also reduced KV-cache capacity by 31.6%. We therefore retained the 8,192-token batch and only raised the context limit to the model's native 1,048,576 tokens.
The model occupied 156.32 GiB and vLLM allocated 81.17 GiB to the KV cache. At least 20.72 GiB remained free during the 800K test. The GPU reached 100% utilization, 1,006 W of its 1,100 W power limit, and 67 °C. There is memory headroom, but a cold prefill leaves effectively no compute headroom on one GPU.
Production recommendations
Follow-ups must reach the same live cache for their conversation or document. Multiple replicas therefore need session-aware routing or a persistent, shared KV cache. The observed cache-latency variance also deserves retesting on newer vLLM releases before defining an SLO.
When the application can avoid resending the entire document, a stateful session or retrieval over stored content is preferable. Genuine 800K requests require a gateway that accepts at least 4 MiB request bodies, streams responses, and allows more than 100 seconds of upstream time.
With a second NVLink-connected B300, our first comparison would be tensor and expert parallelism against two independent replicas with strict session affinity. Separate prefill and decode nodes become more compelling under mixed concurrent traffic. A single long conversation still uses both stages sequentially and requires a large KV-state transfer between GPUs.