Compare model versions

Diff

Before

> ### ⚠️ Read this before comparing any number here to a discrete GPU
>
> Every measurement on this card is from an **AMD Ryzen AI MAX+ 395 "Strix Halo"** — an
> **integrated GPU with unified memory**. There is no discrete VRAM. The Radeon 8060S addresses
> ordinary system RAM through the **GTT aperture**, and the box's 128 GB is **shared between CPU
> and GPU**.
>
> That is the entire reason a 98 GiB model runs here at all: the GPU can reach system memory, so
> capacity is enormous. The trade is **bandwidth** — roughly 215 GB/s measured, against ~1 TB/s on
> a high-end discrete card. So expect big-model *capacity* that a 24 GB dGPU cannot touch, and
> per-token *speed* well below one.
>
> This is also why the numbers below are reported as **GTT resident**, not "VRAM used" — on this
> hardware those are the same pool, and `nvidia-smi`-style VRAM intuitions do not transfer.

> ### ⛔ ngram-mod at ≥64K can wedge the GPU
>
> `--spec-type ngram-mod` is the right spec-decode for this family on ROCm, and it is fine at 32K.
> At 64K and above, a context-checkpoint restore desyncs the QSA indexer cache from the KV
> caches. The next kernel then wedges the SDMA queue — `Ring sdma0 reset failure`, D-state
> llama-server, `kill -9` useless, hard power-off to recover. Field report by
> [@liusecret](https://huggingface.co/liusecret) on
> [`Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF#6`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6).
>
> **Workaround (no new binary required):**
> ```
> -ctxcp 0 -cpent -1
> ```
> `--ctx-checkpoints 0` plus `--checkpoint-every-n-tokens -1`. If no checkpoints are created,
> the restore path never runs. Keep ngram-mod on ROCm; do not use it on Vulkan/gfx1151
> (acceptance collapses to 0). A fork that serialises `mem_idx` is the real fix — until that
> binary is in your hands, use the flags.

STRIX is the quality tier: same attention handling as STRIX\_LEAN, but the PLE table goes to Q8_0 and token embeddings to Q6_K. 25 GiB more on disk than FAST and — measured — not one byte more in GPU memory. Quantized directly from my own BF16 conversion of the release weights.
`general.file_type` = **105** (`Q4_0_ROCMFP4_STRIX`). **5.51 bpw, 113.47 GiB.**

Read straight out of the GGUF headers of the files in this repo:

| tensor group | type |
|---|---|
| MoE expert weights (`ffn_*_exps`, 144) | `TYPE_101` (ROCmFP4, 4.251 bpw) |
| shared expert (`ffn_*_shexp`, 144) | `TYPE_101` |
| attention (`attn_*`, 120) | half `TYPE_100`, half `TYPE_101` |
| `per_layer_token_embd.weight` (PLE, 51.2B params) | `Q8_0` — 50.66 GiB |
| `token_embd.weight` | `Q6_K` |
| `output.weight` (lm head) | `Q6_K` |
| norms / biases | `F32` |

After

Heading only; no text in this section.

Diff

Before

Not present in the earlier snapshot.

After

The `qwen4exp` MTP graph shipped with a broken combiner (it mean-pooled the hyper-connection
streams), so `--spec-type draft-mtp` acceptance sat near **0.36** and gave no real speedup. That is
now **fixed** — this repo ships [`qwen4exp-mtp-graph.patch`](./qwen4exp-mtp-graph.patch); apply it to
the [`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX) fork and rebuild
(`git apply qwen4exp-mtp-graph.patch` before `cmake --build`).

Pair the model with a Flash-Next MTP head from
[`kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF).
**Measured** on the Uncensored FAST (imatrix) build with the **Q8_0 head** (`mtp-Qwen3.8-Flash-Next-Q8_0.gguf`) at
**short context (`-c 2048`)**: **acceptance 0.94**, **31.80 tok/s** vs **24.9 tok/s** no-draft
(**+27.7%**), warm 160-token completion, `cache_prompt:false`. The graph fix and the heads are shared
across the Flash-Next family, but this tier's own MTP speed has not been measured, and the Q6_K / Q4
heads were not benchmarked. The head only *proposes* draft tokens; the main model verifies every one,
so your output is unchanged.

```
llama-server -m <the first shard in this repo>.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp \
  --spec-draft-n-min 2 --spec-draft-n-max 4 --n-gpu-layers-draft 99 \
  -ngl 999 -fa on -np 1 -c 32768 --jinja
```
`-np 1` is required with `draft-mtp`.

> ⛔ **Keep speculative decoding at ≤32K context for now — this includes `draft-mtp`.** It restores
> context checkpoints whenever drafts are rejected, which is the same path as the ≥64K GPU wedge
> field-reported in [`Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF#6`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6), and it has only been tested at short context. The build recipe on
> this card does not include a fix. For full 262K context, run without `-md` / `--spec-type`.


> ### ⚠️ Read this before comparing any number here to a discrete GPU
>
> Every measurement on this card is from an **AMD Ryzen AI MAX+ 395 "Strix Halo"** — an
> **integrated GPU with unified memory**. There is no discrete VRAM. The Radeon 8060S addresses
> ordinary system RAM through the **GTT aperture**, and the box's 128 GB is **shared between CPU
> and GPU**.
>
> That is the entire reason a 98 GiB model runs here at all: the GPU can reach system memory, so
> capacity is enormous. The trade is **bandwidth** — roughly 215 GB/s measured, against ~1 TB/s on
> a high-end discrete card. So expect big-model *capacity* that a 24 GB dGPU cannot touch, and
> per-token *speed* well below one.
>
> This is also why the numbers below are reported as **GTT resident**, not "VRAM used" — on this
> hardware those are the same pool, and `nvidia-smi`-style VRAM intuitions do not transfer.

> ### ⛔ ngram-mod at ≥64K can wedge the GPU
>
> `--spec-type ngram-mod` is the right spec-decode for this family on ROCm, and it is fine at 32K.
> At 64K and above, a context-checkpoint restore desyncs the QSA indexer cache from the KV
> caches. The next kernel then wedges the SDMA queue — `Ring sdma0 reset failure`, D-state
> llama-server, `kill -9` useless, hard power-off to recover. Field report by
> [@liusecret](https://huggingface.co/liusecret) on
> [`Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF#6`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6).
>
> **Workaround (no new binary required):**
> ```
> -ctxcp 0 -cpent -1
> ```
> `--ctx-checkpoints 0` plus `--checkpoint-every-n-tokens -1`. If no checkpoints are created,
> the restore path never runs. Keep ngram-mod on ROCm; do not use it on Vulkan/gfx1151
> (acceptance collapses to 0). A fork that serialises `mem_idx` is the real fix — until that
> binary is in your hands, use the flags.

STRIX is the quality tier: same attention handling as STRIX\_LEAN, but the PLE table goes to Q8_0 and token embeddings to Q6_K. 25 GiB more on disk than FAST and — measured — not one byte more in GPU memory. Quantized directly from my own BF16 conversion of the release weights.
`general.file_type` = **105** (`Q4_0_ROCMFP4_STRIX`). **5.51 bpw, 113.47 GiB.**

Read straight out of the GGUF headers of the files in this repo:

| tensor group | type |
|---|---|
| MoE expert weights (`ffn_*_exps`, 144) | `TYPE_101` (ROCmFP4, 4.251 bpw) |
| shared expert (`ffn_*_shexp`, 144) | `TYPE_101` |
| attention (`attn_*`, 120) | half `TYPE_100`, half `TYPE_101` |
| `per_layer_token_embd.weight` (PLE, 51.2B params) | `Q8_0` — 50.66 GiB |
| `token_embd.weight` | `Q6_K` |
| `output.weight` (lm head) | `Q6_K` |
| norms / biases | `F32` |