Compare model versions

Diff

Before

The MTP0 measurements below were collected at NInfer revision
[`f08597d`](https://github.com/Neroued/ninfer/commit/f08597d6eaafce5b875934aaa85854fcd5426df8),
and the MTP3 measurements at revision
[`32c9881`](https://github.com/Neroued/ninfer/commit/32c9881b6783949df4999422a764b3dcaa111b13).
Both campaigns used one NVIDIA GeForce RTX 5090, CUDA 13.1 compile/runtime, CUDA driver API 13.3,
stochastic sampling, INT8 group-64 KV, CUDA Graphs, a 1,024-token prefill chunk, and prefix reuse
disabled. MTP0 used no speculative backend and a 262,144-token context limit; MTP3 used a
131,072-token per-request context limit and three draft tokens.

After

Measured on September 28–29, 2026 with NInfer revision
[`7f6aafed`](https://github.com/Neroued/ninfer/commit/7f6aafedb5f20200def820cfe51ab81c09c20eeb),
one RTX 5090, driver 617.14, and CUDA 13.4 compile/runtime/driver API.
These serving runs use FP8 E4M3 row-256 KV, CUDA Graphs, a 1,024-token prefill chunk, disabled
prefix reuse, and temperature 0.6 / top-p 0.95 / top-k 20 / min-p 0 / presence penalty 1.0 /
frequency penalty 0. MTP0 has a 262,144-token context ceiling; MTP3 uses 131,072 tokens per
request, three draft tokens, the optimized proposal head, and automatic shared KV capacity.

Diff

Before

The fixed corpus contains three long-reasoning and twelve cross-scenario fixtures with five seeds
each, for 75 requests. Every concurrency point starts a fresh server and uses the same shuffle seed
and ordered HTTP send sequence. C=1 is the serial single-request corpus. Makespan includes prefill,
decode, workload transitions, and final drain.

| C | Requests | Decode tokens | Makespan | Requests/s | Corpus decode (tok/s) | Avg batch | MTP acceptance | Speedup |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 | 75 | 752,160 | 4,670.27 s | 0.0161 | 161.1 | 1.00 | 60.8% | 1.00× |
| 2 | 75 | 739,951 | 2,510.78 s | 0.0299 | 294.7 | 1.98 | 59.2% | 1.86× |
| 4 | 75 | 713,384 | 1,647.74 s | 0.0455 | 432.9 | 3.29 | 58.0% | 2.83× |
| 8 | 75 | 723,602 | 2,164.90 s | 0.0346 | 334.2 | 2.36 | 57.6% | 2.16× |

All 300 requests completed without a request, CUDA, or out-of-memory failure. C=4 gives the
shortest complete-corpus makespan. C=8 is limited by memory pressure, which makes its
complete-corpus result slower than C=4. Sampling is stochastic, so the fixed prompts and seeds do
not imply token-identical continuations across concurrency-specific numerical routes; exact
decode-token totals are shown above.

After

Each C is one complete 75-request corpus, with three reasoning and twelve cross-scenario fixtures,
five seeds per fixture, and a fixed shuffled send order. Makespan includes prefill, decode,
admission waits, transitions, and drain; actual output lengths vary.

| C | Requests | Computed prefill tokens | Decode tokens | Makespan (s) | Requests/s | Corpus prefill (tok/s) | Corpus decode (tok/s) | Avg batch | MTP acceptance |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 75 | 15,460 | 693,701 | 4,115.22 | 0.0182 | 3.8 | 168.6 | 1.00 | 59.6% |
| 2 | 75 | 15,460 | 709,989 | 2,394.23 | 0.0313 | 6.5 | 296.5 | 1.90 | 58.9% |
| 4 | 75 | 15,460 | 722,202 | 1,640.17 | 0.0457 | 9.4 | 440.3 | 3.13 | 58.5% |
| 8 | 75 | 15,460 | 708,589 | 1,443.37 | 0.0520 | 10.7 | 490.9 | 3.52 | 59.5% |

All 300 requests completed without request, CUDA, or allocation errors. At C=1/2/4/8,
automatic KV capacity is 131,072 / 262,144 / 253,632 / 225,024 tokens. C=8 has average batch 3.52
and up to five waiting requests in the sampled intervals. The resident MTP3 weights occupy
19.729 GiB, and the workspace arena is 243.3 MiB.

Diff

Before

Not present in the earlier snapshot.

After

Values are arithmetic mean ± sample standard deviation over five fixed seeds per fixture.

| Prompt tokens | Samples | Prefill phase (tok/s) | Server TTFT (ms) | Decode phase (tok/s) |
|---|---|---|---|---|
| 7,680 | 5 | 12,819.1 ± 16.8 | 602.7 ± 1.1 | 74.1 ± 0.3 |
| 64,512 | 5 | 8,658.2 ± 44.8 | 7,487.4 ± 37.7 | 68.2 ± 0.2 |
| 130,048 | 5 | 6,198.5 ± 19.6 | 21,055.7 ± 67.2 | 62.5 ± 0.3 |
| 260,096 | 5 | 4,016.4 ± 10.4 | 64,910.0 ± 171.8 | 53.4 ± 0.6 |

Diff

Before

The C=1 point supplies five samples for each fixture. Values are arithmetic mean ± sample standard
deviation from server phase timings and speculative counters.

| AIME 2026 fixture | Completion tokens | Decode phase (tok/s) | MTP acceptance | MTP tokens/round |
|---|---:|---:|---:|---:|
| Problem 1 | 1,465.4 ± 417.3 | 195.2 ± 4.6 | 76.0% ± 2.4% | 3.28 ± 0.07 |
| Problem 15 | 65,414.4 ± 271.9 | 151.4 ± 2.0 | 56.2% ± 1.1% | 2.69 ± 0.03 |
| Problem 30 | 50,023.4 ± 14,839.1 | 167.5 ± 23.7 | 64.6% ± 14.9% | 2.94 ± 0.45 |

After

The C=1 corpus supplies these phase statistics, with five samples per reasoning fixture.

| Fixture | Samples | Completion tokens | Decode phase (tok/s) | MTP3 acceptance | MTP3 tokens/round |
|---|---|---|---|---|---|
| `long_decode_aime26_01` | 5 | 1,559.4 ± 727.2 | 207.4 ± 3.6 | 77.2% ± 1.8% | 3.32 ± 0.05 |
| `long_decode_aime26_15` | 5 | 65,536.0 ± 0.0 | 161.7 ± 3.8 | 57.2% ± 2.1% | 2.72 ± 0.06 |
| `long_decode_aime26_30` | 5 | 37,978.0 ± 7,474.4 | 170.0 ± 2.3 | 59.9% ± 1.5% | 2.80 ± 0.05 |

Diff

Before

Each category contains three fixtures and five seeds per fixture, for 15 samples.

| Category | Decode phase (tok/s) | MTP acceptance | MTP tokens/round |
|---|---:|---:|---:|
| Code | 194.3 ± 6.1 | 76.4% ± 3.9% | 3.29 ± 0.12 |
| Story | 126.1 ± 10.9 | 37.4% ± 5.8% | 2.12 ± 0.17 |
| Translation | 192.3 ± 11.9 | 75.0% ± 6.5% | 3.25 ± 0.19 |
| Structured output | 219.8 ± 8.6 | 90.8% ± 5.1% | 3.72 ± 0.15 |

See the
[full methodology and results](https://github.com/Neroued/ninfer/blob/master/docs/performance/qwen3.8-27b.md)
for metric definitions and the exact reproduction command.

After

Each category pools three fixtures × five seeds. Values are mean ± sample standard deviation.

| Category | Samples | Decode phase (tok/s) | MTP3 acceptance | MTP3 tokens/round |
|---|---|---|---|---|
| Code | 15 | 205.3 ± 9.5 | 76.8% ± 5.1% | 3.30 ± 0.15 |
| Story | 15 | 132.6 ± 13.1 | 37.6% ± 7.1% | 2.13 ± 0.21 |
| Translation | 15 | 202.9 ± 12.3 | 75.2% ± 6.6% | 3.26 ± 0.20 |
| Structured | 15 | 231.7 ± 10.6 | 90.6% ± 5.6% | 3.72 ± 0.17 |

All five AIME 15 samples reach the 65,536-token output budget. The C=1 corpus contains
40 stop-token and 35 output-limit results; all are retained in the statistics. These measurements
do not score answer accuracy or task completion.

The [full results and reproduction commands](https://github.com/Neroued/ninfer/blob/master/docs/performance/qwen3.8-27b.md)
also cover DFlash2 K=7, MTP3 decode saturation, and completion outcomes.

Diff

Before

Each value is the arithmetic mean ± sample standard deviation over five fixed seeds after server
warm-up.

| Prompt tokens | Prefill phase (tok/s) | Server TTFT (ms) | Decode phase (tok/s) |
|---:|---:|---:|---:|
| 7,680 | 8,340.4 ± 13.0 | 931.6 ± 1.6 | 71.2 ± 0.1 |
| 64,512 | 5,297.9 ± 259.2 | 12,281.1 ± 561.5 | 65.7 ± 0.8 |
| 130,048 | 3,544.7 ± 25.3 | 36,853.5 ± 259.4 | 59.6 ± 0.9 |
| 260,096 | 2,203.1 ± 13.4 | 118,354.8 ± 717.2 | 52.9 ± 2.3 |

After

Not present in the later snapshot.