Qwen 3.5 Quantization Speed (2026) — Tokens/sec by Quant (Q2–Q8) Compared

If you are comparing Qwen 3.5 quantization speed — how many tokens per second each quant level actually delivers — this page gives the numbers side by side. Local LLM inference is memory-bandwidth bound: the GPU has to read every active weight for each token it generates, so a smaller quant means fewer bytes to move and more tokens per second. The question is how much speed you gain and how much quality you give up.

Below are tokens/sec benchmarks by quant (Q2_K → Q8_0) for the two most-run Qwen 3.5 variants — the 9B dense and the 35B-A3B MoE — across an RTX 4090, RTX 5090, and M4 Max 64GB.

Want exact VRAM per quant for your card instead of speed? Use the VRAM calculator, or the per-variant guides: Qwen 3.5 9B VRAM and Qwen 3.5 35B-A3B VRAM.

Qwen 3.5 9B — Tokens/sec by Quantization

Single-stream generation speed (batch size 1, ~2K context), llama.cpp / Ollama GGUF:

Quant	VRAM	RTX 4090 24GB	RTX 5090 32GB	M4 Max 64GB
Q2_K	~3.6 GB	~120 tok/s	~155 tok/s	~58 tok/s
Q3_K_M	~4.5 GB	~105 tok/s	~138 tok/s	~50 tok/s
Q4_K_M	~5.5 GB	~92 tok/s	~122 tok/s	~43 tok/s
Q5_K_M	~6.6 GB	~80 tok/s	~106 tok/s	~37 tok/s
Q6_K	~7.6 GB	~72 tok/s	~95 tok/s	~33 tok/s
Q8_0	~9.5 GB	~60 tok/s	~80 tok/s	~28 tok/s

The pattern is clean: dropping from Q8 to Q2 roughly doubles throughput on every device, because you halve the bytes read per token. The 9B fits entirely in VRAM at every quant on all three machines, so there is no offload penalty — the speed curve is pure bandwidth scaling.

Qwen 3.5 35B-A3B (MoE) — Tokens/sec by Quantization

The 35B-A3B activates only 3B parameters per token, so it generates far faster than its 35B footprint suggests — but you still need enough VRAM to hold all the experts. Where a quant does not fit on a 24GB card, experts spill to system RAM and speed collapses.

Quant	VRAM	RTX 4090 24GB	RTX 5090 32GB	M4 Max 64GB
Q2_K	~13.0 GB	~85 tok/s	~110 tok/s	~42 tok/s
Q3_K_M	~16.5 GB	~78 tok/s	~100 tok/s	~38 tok/s
Q4_K_M	~21.0 GB	~68 tok/s	~90 tok/s	~34 tok/s
Q5_K_M	~24.5 GB	~38 tok/s*	~78 tok/s	~30 tok/s
Q6_K	~28.5 GB	offload*	~68 tok/s	~27 tok/s
Q8_0	~36.5 GB	offload*	offload*	~23 tok/s

*On the RTX 4090 24GB, Q5_K_M and up exceed VRAM: experts offload to system RAM over PCIe and throughput drops sharply (the ~38 tok/s figure for Q5 assumes minimal spill; heavier offload lands well below that). Stay at Q4_K_M or lower on 24GB cards. The RTX 5090 32GB holds up to Q5_K_M comfortably; the M4 Max 64GB fits every quant in unified memory.

Speed vs Quality — Which Quant to Pick

Faster is not free. Here is the practical read on each tier for Qwen 3.5:

Quant	Speed	Quality	Use when
Q2_K	Fastest	Noticeably degraded	You must fit a bigger variant into tight VRAM; casual chat only
Q3_K_M	Very fast	Some degradation	Squeezing 35B-A3B onto a 16GB card
Q4_K_M	Fast	Near-full	Default for most users — best speed-per-quality
Q5_K_M	Balanced	Near-lossless	You have headroom and want a small quality bump
Q6_K	Moderate	Effectively lossless	Coding/reasoning where precision matters and VRAM allows
Q8_0	Slowest local quant	Lossless	Reference quality; only worth it with ample VRAM

For coding and reasoning with Qwen 3.5, we recommend Q5_K_M or Q6_K — the extra precision matters for syntax-sensitive output — and accept the ~15-25% speed cost. For chat, summarization, and agents, Q4_K_M is the sweet spot. Drop below Q4 only to fit a larger, more capable variant that would otherwise not load. A higher-quality smaller model often beats a heavily-quantized larger one; see our GGUF quantization guide for the quality-vs-size deep dive.

Why Lower Quants Run Faster

Token generation reads the model's active weights from VRAM once per token. Throughput is approximately:

tokens/sec ≈ (memory bandwidth) / (active bytes read per token)

Halving the quantization roughly halves the bytes per token, so it roughly doubles the ceiling. That is why the RTX 5090 (1792 GB/s) beats the RTX 4090 (1008 GB/s) by ~30% at the same quant, and why the M4 Max (546 GB/s) trails both — it is bandwidth, not core count, that sets the pace. MoE models like 35B-A3B read only their 3B active weights per token, which is why they punch above their size in this table.

Prompt processing (prefill) behaves differently — it is compute-bound and batched, so it barely moves across quant levels. If your workload is long-context RAG rather than chatty generation, weight quantization matters less than KV cache size and context length.

Apple Silicon note (MLX)

The M4 Max numbers above are GGUF via llama.cpp. On Apple Silicon, MLX typically runs 15-30% faster than GGUF at the same quant and uses ~10% less memory. If you are on a Mac, read the dedicated Qwen 3.5 on Apple Silicon MLX guide for MLX-specific tokens/sec and memory per Mac tier.

Getting Started

1. Check fit first: VRAM calculator — confirm your quant + variant fits before downloading.
2. Pull a quant:

   ollama run qwen3.5:9b-q4_K_M          # fast, near-full quality
   ollama run qwen3.5:35b-a3b-q4_K_M     # MoE, needs ~21GB VRAM
   

3. Measure your own tok/s: run ollama run --verbose and watch the eval rate — then submit your benchmark to improve the shared dataset.

Next Steps

• Qwen 3 & 3.5 GPU Requirements — family hub
• Qwen 3.5 9B VRAM Requirements
• Qwen 3.5 35B-A3B VRAM Requirements
• GGUF quantization guide — Q4 vs Q5 vs Q8 quality
• Qwen 3.5 on Apple Silicon MLX — memory + tokens/sec