Compare model versions

Diff

Before

base_model:
- Qwen/Qwen3.8-27B
tags:
- vllm
- llm-compressor
- qwen3_5
- compressed-tensors
- int4
- conversational
- fp8
license: apache-2.0
pipeline_tag: image-text-to-text
library_name: transformers

After

language:
- en
- zh
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- int4
- vllm
- llm-compressor
- compressed-tensors
base_model: Qwen/Qwen3.8-27B

Diff

Before

Not present in the earlier snapshot.

After

- **Model Architecture:** Qwen3_5ForConditionalGeneration
  - **Input:** Text / Image
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** INT4
  - **Activation quantization:** None
- **Release Date:** 2026-08-17
- **Version:** 1.0
- **Model Developers:** RedHatAI

This model is a quantized version of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B).
It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

Diff

Before

Not present in the earlier snapshot.

After

This model was evaluated on GSM8K Platinum, IFEval, MMLU-Pro, MATH-500, GPQA Diamond, and AIME 2025 using [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness) and [lighteval](https://github.com/neuralmagic/lighteval), all served with vLLM (OpenAI-compatible API). Each benchmark was run with 3 seeds (1234, 2345, 3456; 8 seeds for AIME 2025) and the results averaged; recovery is computed against the BF16 model.

Diff

Before

| Eval | Qwen/Qwen3.8-27B (BF16) | RedHatAI/Qwen3.8-27B-INT4 | Recovery (INT4/BF16) |
| --- | --- | --- | --- |
| gsm8k_platinum (strict-match) | 95.75% | 96.77% | 101.07% |
| ifeval (prompt-level strict) | 92.24% | 91.93% | 99.67% |
| mmlu_pro_chat (exact-match) | 84.46% | 83.45% | 98.81% |
| math_500 (pass@1) | 83.73% | 83.33% | 99.52% |
| gpqa_diamond (pass@1) | 89.23% | 87.88% | 98.49% |
| aime25 (avg@1, 8 seeds) | 95.42% | 94.17% | 98.69% |

After

Not present in the later snapshot.

Diff

Before

Not present in the earlier snapshot.

After

Heading only; no text in this section.

Diff

Before

This model was obtained by quantizing the weights of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) to int4, ready for inference with vLLM.
Weights are quantized to int4 with a group size of 128 and the KV Cache has been quantized to FP8. Only the weights of the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). Vision tower and outputhead layers are kept in their original precision.

After

This model was obtained by quantizing the weights of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) to INT4 data type while keeping activations in original precision, with FP8 KV cache, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.

Only the weights of the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor).

Diff

Before

Not present in the earlier snapshot.

After

Heading only; no text in this section.

Diff

Before

```bash
vllm serve RedHatAI/Qwen3.8-27B-INT4
  --tensor-parallel-size 1
  --enable-auto-tool-choice
  --tool-call-parser qwen3_coder
  --reasoning-parser qwen3
  --mm-encoder-tp-mode data
  --max-model-len 69632
  --gpu-memory-utilization 0.9
```

After

```bash
vllm serve RedHatAI/Qwen3.8-27B-INT4 \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data \
  --max-model-len 69632 \
  --gpu-memory-utilization 0.9
```

Diff

Before

Not present in the earlier snapshot.

After

This model was created by applying [LLM Compressor](https://github.com/vllm-project/llm-compressor) with calibration samples from open-perfectblend, using AWQ smoothing and the W4A16 GPTQ scheme, exported in compressed-tensors format.

<details>

```python
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.8-27B"

# Load model.
with load_context(Qwen3_5ForConditionalGeneration):
    model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)


recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
            r"re:.*lm_head",
            "re:.*embed_tokens$",
            r"re:.*linear_attn\.in_proj_a$",
            r"re:.*linear_attn\.in_proj_b$",
        ],
        kv_cache_scheme={
            "num_bits": 8,
            "type": "float",
            "symmetric": True,
            "strategy": "tensor",
            "dynamic": False,
            "observer": "static_minmax",
        },
    ),
]

NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 4096

ds = load_dataset(
    "mlabonne/open-perfectblend",
    split=f"train[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.shuffle(seed=42)

ROLE_MAP = {"human": "user", "gpt": "assistant"}


def preprocess_function(example):
    messages = [
        {
            "role": ROLE_MAP.get(msg["from"], msg["from"]),
            "content": [{"type": "text", "text": msg["value"]}],
        }
        for msg in example["conversations"]
    ]
    return processor.apply_chat_template(
        messages,
        tokenize=True,
        return_dict=True,
        add_generation_prompt=False,
        processor_kwargs={
            "return_tensors": "pt",
            "padding": False,
            "truncation": True,
            "max_length": MAX_SEQUENCE_LENGTH,
            "add_special_tokens": False,
        },
    )

27 more lines not shown.

Diff

Before

Not present in the earlier snapshot.

After

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>Qwen/Qwen3.8-27B (BF16)</th>
      <th>RedHatAI/Qwen3.8-27B-INT4</th>
      <th>Recovery</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td rowspan="2"><b>Instruction Following</b></td>
      <td>IFEval (0-shot, prompt-level strict)</td>
      <td>92.24%</td>
      <td>91.93%</td>
      <td>99.67%</td>
    </tr>
    <tr>
      <td>MMLU-Pro (exact-match)</td>
      <td>84.46%</td>
      <td>83.45%</td>
      <td>98.81%</td>
    </tr>
    <tr>
      <td rowspan="4"><b>Reasoning</b></td>
      <td>GSM8K Platinum (strict-match)</td>
      <td>95.75%</td>
      <td>96.77%</td>
      <td>101.07%</td>
    </tr>
    <tr>
      <td>MATH-500 (pass@1)</td>
      <td>83.73%</td>
      <td>83.33%</td>
      <td>99.52%</td>
    </tr>
    <tr>
      <td>GPQA Diamond (pass@1)</td>
      <td>89.23%</td>
      <td>87.88%</td>
      <td>98.49%</td>
    </tr>
    <tr>
      <td>AIME 2025 (pass@1)</td>
      <td>95.42%</td>
      <td>94.17%</td>
      <td>98.69%</td>
    </tr>
  </tbody>
</table>

Diff

Before

This model is a quantized version of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B).

After

Not present in the later snapshot.

Diff

Before

```python
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.8-27B"

# Load model.
with load_context(Qwen3_5ForConditionalGeneration):
    model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)


recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
            r"re:.*lm_head",
            "re:.*embed_tokens$",
            r"re:.*linear_attn\.in_proj_a$",
            r"re:.*linear_attn\.in_proj_b$",
        ],
        kv_cache_scheme={
            "num_bits": 8,
            "type": "float",
            "symmetric": True,
            "strategy": "tensor",
            "dynamic": False,
            "observer": "static_minmax",
        },
    ),
]

NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 4096

ds = load_dataset(
    "mlabonne/open-perfectblend",
    split=f"train[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.shuffle(seed=42)

ROLE_MAP = {"human": "user", "gpt": "assistant"}


def preprocess_function(example):
    messages = [
        {
            "role": ROLE_MAP.get(msg["from"], msg["from"]),
            "content": [{"type": "text", "text": msg["value"]}],
        }
        for msg in example["conversations"]
    ]
    return processor.apply_chat_template(
        messages,
        tokenize=True,
        return_dict=True,
        add_generation_prompt=False,
        processor_kwargs={
            "return_tensors": "pt",
            "padding": False,
            "truncation": True,
            "max_length": MAX_SEQUENCE_LENGTH,
            "add_special_tokens": False,
        },
    )


ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

22 more lines not shown.

After

Not present in the later snapshot.

Diff

Before

Run once per seed (1234, 2345, 3456). Final reported values are averages.

After

Not present in the later snapshot.

Diff

Before

```
lm_eval \
  --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args model=Qwen/Qwen3.8-27B,max_length=69632,base_url=http://127.0.0.1:8002/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized
_requests=False,tokenizer_backend=None,timeout=3600 \
  --num_fewshot 0 \
  --seed <SEED> \
  --gen_kwargs do_sample=True,temperature=1.0,top_p=0.95,top_k=20,seed=<SEED>,max_gen_toks=32000 \
  --apply_chat_template
```

After

Not present in the later snapshot.

Diff

Before

```
lm_eval \
  --model local-chat-completions \
  --tasks ifeval \
  --model_args model=Qwen/Qwen3.8-27B,max_length=69632,base_url=http://127.0.0.1:8006/v1/chat/completions,num_concurrent=16,max_retrie
s=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600 \
  --num_fewshot 0 \
  --seed <SEED> \
  --gen_kwargs do_sample=True,temperature=1.0,top_p=0.95,top_k=20,seed=1234,max_gen_toks=32000 \
  --apply_chat_template
```

After

Not present in the later snapshot.

Diff

Before

```
lighteval endpoint litellm \
  runs/20260818_110422_qwen3.8-27b-int4_full_ifeval-math-aime-gpqa/configs/litellm_aime25_seed1234.yaml \
  'aime25|0' \
  --save-details
```

After

Not present in the later snapshot.

Diff

Before

```
lighteval endpoint litellm \
  runs/20260818_110422_qwen3.8-27b-int4_full_ifeval-math-aime-gpqa/configs/litellm_math_500_seed1234.yaml \
  'math_500|0' \
  --save-details
```

After

Not present in the later snapshot.

Diff

Before

```
lighteval endpoint litellm \
  runs/20260818_110422_qwen3.8-27b-int4_full_ifeval-math-aime-gpqa/configs/litellm_gpqa_diamond_seed1234.yaml \
  'gpqa:diamond|0' \
  --save-details
```

After

Not present in the later snapshot.