Compare model versions

Diff

Before

> ### 🔧 Runtime: build the ROCmFPX fork below
> Stock `llama.cpp` will not load this file. You need **both** the **`qwen4exp`** architecture
> **and** the ROCmFP4 tensor types in one tree. Upstream
> [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) has the ROCmFP4 types but
> not `qwen4exp`. Our fork has both:
>
> **[`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX)** — a fork of `charlie12345/ROCmFPX`, branch `main`.
>
> ```bash
> git clone https://github.com/kingjones30/ROCmFPX.git
> cd ROCmFPX
> cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
> cmake --build build --target llama-server llama-quantize -j$(nproc)
> ```
>
> Verified 2026-08-27 on gfx1151: clean clone → **0 build errors** → `llama-server` loads a
> `qwen4exp` ROCmFP4 GGUF from this family and generates coherent text.

After

> ### 🔧 Runtime: build the ROCmFPX fork below
> Stock `llama.cpp` will not load this file. You need **both** the **`qwen4exp`** architecture
> **and** the ROCmFP4 tensor types in one tree. Upstream
> [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) has the ROCmFP4 types but
> not `qwen4exp`. Our fork has both:
>
> **[`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX)** — a fork of `charlie12345/ROCmFPX`, branch `main`.
>
> ```bash
> git clone https://github.com/kingjones30/ROCmFPX.git
> cd ROCmFPX
> cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
> cmake --build build --target llama-server llama-quantize -j$(nproc)
> ```
>
> ⚠️ **Apply the bundled fix patches before `cmake`:** `qwen4exp-qsa-checkpoint-fix.patch`
> always, plus `qwen4exp-mtp-graph-fork.patch` if you want `--spec-type draft-mtp` on this
> clone. Full steps further down.
>
> Verified 2026-08-27 on gfx1151: clean clone → **0 build errors** → `llama-server` loads a
> `qwen4exp` ROCmFP4 GGUF from this family and generates coherent text.

Diff

Before

The `qwen4exp` MTP graph shipped with a broken combiner (it mean-pooled the hyper-connection
streams), so `--spec-type draft-mtp` acceptance sat near **0.36** and gave no real speedup. That is
now **fixed** — this repo ships [`qwen4exp-mtp-graph.patch`](./qwen4exp-mtp-graph.patch); apply it to
the [`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX) fork and rebuild
(`git apply qwen4exp-mtp-graph.patch` before `cmake --build`).

Pair the model with a Flash-Next MTP head from
[`kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF).
**Measured** on the Uncensored FAST (imatrix) build with the **Q8_0 head** (`mtp-Qwen3.8-Flash-Next-Q8_0.gguf`) at
**short context (`-c 2048`)**: **acceptance 0.94**, **31.80 tok/s** vs **24.9 tok/s** no-draft
(**+27.7%**), warm 160-token completion, `cache_prompt:false`. The graph fix and the heads are shared
across the Flash-Next family, but this tier's own MTP speed has not been measured, and the Q6_K / Q4
heads were not benchmarked. The head only *proposes* draft tokens; the main model verifies every one,
so your output is unchanged.

```
llama-server -m <the first shard in this repo>.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp \
  --spec-draft-n-min 2 --spec-draft-n-max 4 --n-gpu-layers-draft 99 \
  -ngl 999 -fa on -np 1 -c 32768 --jinja
```
`-np 1` is required with `draft-mtp`.

> ⛔ **Keep speculative decoding at ≤32K context for now — this includes `draft-mtp`.** It restores
> context checkpoints whenever drafts are rejected, which is the same path as the ≥64K GPU wedge
> field-reported in [`Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF#6`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6), and it has only been tested at short context. The build recipe on
> this card does not include a fix. For full 262K context, run without `-md` / `--spec-type`.


> ### ⚠️ Read this before comparing any number here to a discrete GPU
>
> Every measurement on this card is from an **AMD Ryzen AI MAX+ 395 "Strix Halo"** — an
> **integrated GPU with unified memory**. There is no discrete VRAM. The Radeon 8060S addresses
> ordinary system RAM through the **GTT aperture**, and the box's 128 GB is **shared between CPU
> and GPU**.
>
> That is the entire reason a 98 GiB model runs here at all: the GPU can reach system memory, so
> capacity is enormous. The trade is **bandwidth** — roughly 215 GB/s measured, against ~1 TB/s on
> a high-end discrete card. So expect big-model *capacity* that a 24 GB dGPU cannot touch, and
> per-token *speed* well below one.
>
> This is also why the numbers below are reported as **GTT resident**, not "VRAM used" — on this
> hardware those are the same pool, and `nvidia-smi`-style VRAM intuitions do not transfer.

> ### ⛔ ngram-mod at ≥64K can wedge the GPU
>
> `--spec-type ngram-mod` is the right spec-decode for this family on ROCm, and it is fine at 32K.
> At 64K and above, a context-checkpoint restore desyncs the QSA indexer cache from the KV
> caches. The next kernel then wedges the SDMA queue — `Ring sdma0 reset failure`, D-state
> llama-server, `kill -9` useless, hard power-off to recover. Field report by
> [@liusecret](https://huggingface.co/liusecret) on
> [`Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF#6`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6).
>
> **Workaround (no new binary required):**
> ```
> -ctxcp 0 -cpent -1
> ```
> `--ctx-checkpoints 0` plus `--checkpoint-every-n-tokens -1`. If no checkpoints are created,
> the restore path never runs. Keep ngram-mod on ROCm; do not use it on Vulkan/gfx1151
> (acceptance collapses to 0). A fork that serialises `mem_idx` is the real fix — until that
> binary is in your hands, use the flags.

STRIX is the quality tier: same attention handling as STRIX\_LEAN, but the PLE table goes to Q8_0 and token embeddings to Q6_K. 25 GiB more on disk than FAST and — measured — not one byte more in GPU memory. Quantized directly from my own BF16 conversion of the release weights.
`general.file_type` = **105** (`Q4_0_ROCMFP4_STRIX`). **5.51 bpw, 113.47 GiB.**

Read straight out of the GGUF headers of the files in this repo:

| tensor group | type |
|---|---|
| MoE expert weights (`ffn_*_exps`, 144) | `TYPE_101` (ROCmFP4, 4.251 bpw) |
| shared expert (`ffn_*_shexp`, 144) | `TYPE_101` |
| attention (`attn_*`, 120) | half `TYPE_100`, half `TYPE_101` |
| `per_layer_token_embd.weight` (PLE, 51.2B params) | `Q8_0` — 50.66 GiB |
| `token_embd.weight` | `Q6_K` |
| `output.weight` (lm head) | `Q6_K` |
| norms / biases | `F32` |

After

The `qwen4exp` MTP graph shipped with a broken combiner (it mean-pooled the hyper-connection
streams), so `--spec-type draft-mtp` acceptance sat near **0.36** and gave no real speedup. That is
now **fixed** — this repo ships [`qwen4exp-mtp-graph.patch`](./qwen4exp-mtp-graph.patch); apply it to
the [`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX) fork and rebuild
(`git apply qwen4exp-mtp-graph.patch` before `cmake --build`).

Pair the model with a Flash-Next MTP head from
[`kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF`](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF).
**Measured** on the Uncensored FAST (imatrix) build with the **Q8_0 head** (`mtp-Qwen3.8-Flash-Next-Q8_0.gguf`) at
**short context (`-c 2048`)**: **acceptance 0.94**, **31.80 tok/s** vs **24.9 tok/s** no-draft
(**+27.7%**), warm 160-token completion, `cache_prompt:false`. The graph fix and the heads are shared
across the Flash-Next family, but this tier's own MTP speed has not been measured, and the Q6_K / Q4
heads were not benchmarked. The head only *proposes* draft tokens; the main model verifies every one,
so your output is unchanged.

```
llama-server -m <the first shard in this repo>.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp \
  --spec-draft-n-min 0 --spec-draft-n-max 1 --n-gpu-layers-draft 99 \
  -ngl 999 -fa on -np 1 -c 32768 --jinja
```
`-np 1` is required with `draft-mtp`.

⚠️ **Updated 2026-09-17 — re-download if you pulled it earlier.** `qwen4exp-mtp-graph.patch` now
carries the `models.h` and `llama-model.cpp` hunks it needs. The previous version applied cleanly but
**failed to compile** (`'graph_mtp' was not declared in this scope`). The bundled patch matches the
build steps on this card; for the other build path use `qwen4exp-mtp-graph-d3ca537.patch` (if you build `charlie12345/ROCmFPX` @ `d3ca537` + the arch patch), also bundled here.

**Measured plain vs `draft-mtp`** — median of 3 per cell, one binary, greedy, `cache_prompt:false`,
256 generated tokens, `-c 2048`, Q8_0 head, Uncensored STRIX_LEAN-imatrix weights, gfx1151 / ROCm 7.2.4
(2026-09-17):

| workload | plain | `--spec-draft-n-max 4` | `--spec-draft-n-max 1` |
|---|---|---|---|
| reasoning | 23.91 | 30.94 (+29%, acc 0.680) | **31.94 (+34%, acc 0.945)** |
| JSON output | 23.99 | 28.31 (+18%, acc 0.597) | 27.24 (+14%, acc 0.758) |
| code | 24.09 | 21.56 (**−10%**, acc 0.422) | **26.80 (+11%, acc 0.711)** |
| long-document summary | 23.80 | 20.36 (**−14%**, acc 0.352) | **24.14 (+1%, acc 0.641)** |

⭐ **Use `--spec-draft-n-max 1`.** It did not lose a single workload here, and it wins most where the
next token is predictable. `n-max 4` pays for four draft forward passes per step, so it only wins when
acceptance is high (reasoning, JSON) and is a genuine loss on code and long-document work. MTP also
costs prefill speed, because the draft head processes the prompt too. The older **+27.7%** figure came
from one reasoning-shaped prompt — it holds for that shape, not universally, so measure your own.

> ✅ **Depth:** with the bundled checkpoint fix applied, `draft-mtp` is verified from 2K to
> 128K — see the box further down for what was measured and what is still open.


> ### ⚠️ Read this before comparing any number here to a discrete GPU
>
> Every measurement on this card is from an **AMD Ryzen AI MAX+ 395 "Strix Halo"** — an
> **integrated GPU with unified memory**. There is no discrete VRAM. The Radeon 8060S addresses
> ordinary system RAM through the **GTT aperture**, and the box's 128 GB is **shared between CPU
> and GPU**.
>
> That is the entire reason a 98 GiB model runs here at all: the GPU can reach system memory, so
> capacity is enormous. The trade is **bandwidth** — roughly 215 GB/s measured, against ~1 TB/s on
> a high-end discrete card. So expect big-model *capacity* that a 24 GB dGPU cannot touch, and
> per-token *speed* well below one.
>
> This is also why the numbers below are reported as **GTT resident**, not "VRAM used" — on this
> hardware those are the same pool, and `nvidia-smi`-style VRAM intuitions do not transfer.

> ### ✅ Depth: `draft-mtp` is fixed and measured (2026-09-17)
>
> The ≥64K wedge came from context-checkpoint restores leaving the QSA indexer cache (`mem_idx`) out
> of the checkpoint. The fix ships here as
> [`qwen4exp-qsa-checkpoint-fix.patch`](./qwen4exp-qsa-checkpoint-fix.patch) — it overrides
> `state_write` / `state_read` on `llama_memory_hybrid_idx`. Apply it with the build steps on this
> card **even if you never use speculative decoding**.
>
> With it applied, `--spec-type draft-mtp` ran clean from **2K to 128K** on gfx1151: 8 depth rungs,
> **864 context-checkpoint restores** (2 of them prompt-cache rollbacks at 64K), **0 GPU faults**,
> coherent output at every depth. Measured 2026-09-17 on Ryzen AI MAX+ 395 / ROCm 7.2.4 with the
> Uncensored STRIX_LEAN-imatrix weights + `mtp-Qwen3.8-Flash-Next-Q8_0.gguf`, `-c 262144`,
> `--spec-draft-n-max 4`, default context checkpoints. That 128K run used my own fork tree; the exact
> build steps on this card were verified to 16K.
>
> ⚠️ **Still open:** `--spec-type ngram-mod` at ≥64K has **not** been retested with the patch — the

21 more lines not shown.

Diff

Before

You need **two** things in one tree: the `qwen4exp` architecture and the ROCmFP4 tensor types.
Neither side has both — `charlie12345/ROCmFPX` has the ROCmFP4 types but no `qwen4exp`, and the
upstream qwen4exp work has no ROCmFP4. **The patch that combines them ships in this repo:**
[`qwen4exp-on-rocmfpx-d3ca537.patch`](./qwen4exp-on-rocmfpx-d3ca537.patch) (156 KB, 25 files).

```bash
git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize -j$(nproc)
```

Verified from a clean clone: applies without conflicts, compiles with zero errors, and the
resulting `llama-server` loads these GGUFs and generates.

**What the patch adds** — the pieces people miss when copying files by hand:

| file | why |
|---|---|
| `src/llama-memory-hybrid-idx.{cpp,h}` | **new** — the Qwen Sparse Attention indexer needs its own memory class |
| `src/models/qwen4exp.cpp` | **new** — the arch implementation (picked up by the `models/*.cpp` GLOB) |
| `conversion/qwen4exp.py` | **new** — conversion support |
| `src/llama-{arch,batch,hparams,kv-cache,model,model-loader,model-saver,quant}.{cpp,h}` | interdependent — cherry-picking individual files will not work |

Conversion note: the entry point is the **`conversion/`** package, not `convert_hf_to_gguf.py`,
which rejects `Qwen4ExpForConditionalGeneration`. The 51B-parameter PLE table will also OOM a
naive conversion that builds an F32 intermediate — cast each shard straight to BF16 and write
positionally.

After

You need **two** things in one tree: the `qwen4exp` architecture and the ROCmFP4 tensor types.
Neither side has both — `charlie12345/ROCmFPX` has the ROCmFP4 types but no `qwen4exp`, and the
upstream qwen4exp work has no ROCmFP4. **The patch that combines them ships in this repo:**
[`qwen4exp-on-rocmfpx-d3ca537.patch`](./qwen4exp-on-rocmfpx-d3ca537.patch) (156 KB, 25 files).

```bash
git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
# both fixes ship in this repo — apply them before configuring:
curl -LO https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-qsa-checkpoint-fix.patch
git apply qwen4exp-qsa-checkpoint-fix.patch      # checkpoint safety at >=64K: apply this always
curl -LO https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-mtp-graph.patch
git apply qwen4exp-mtp-graph.patch               # only if you want --spec-type draft-mtp
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize -j$(nproc)
```

Verified from a clean clone: applies without conflicts, compiles with zero errors, and the
resulting `llama-server` loads these GGUFs and generates.

**What the patch adds** — the pieces people miss when copying files by hand:

| file | why |
|---|---|
| `src/llama-memory-hybrid-idx.{cpp,h}` | **new** — the Qwen Sparse Attention indexer needs its own memory class |
| `src/models/qwen4exp.cpp` | **new** — the arch implementation (picked up by the `models/*.cpp` GLOB) |
| `conversion/qwen4exp.py` | **new** — conversion support |
| `src/llama-{arch,batch,hparams,kv-cache,model,model-loader,model-saver,quant}.{cpp,h}` | interdependent — cherry-picking individual files will not work |

Conversion note: the entry point is the **`conversion/`** package, not `convert_hf_to_gguf.py`,
which rejects `Qwen4ExpForConditionalGeneration`. The 51B-parameter PLE table will also OOM a
naive conversion that builds an F32 intermediate — cast each shard straight to BF16 and write
positionally.