Qwen3 8B
Alibaba Qwen · 8.2B parameters · 32,768 token max context · Model card
Memory needed
| Quant | Weights | Total @ 8K ctx |
|---|---|---|
| Q3_K_M | 4.0 GB | 6.1 GB |
| Q4_K_M | 5.0 GB | 7.1 GB |
| Q5_K_M | 5.8 GB | 8.0 GB |
| Q6_K | 6.8 GB | 9.0 GB |
| Q8_0 | 8.7 GB | 11.0 GB |
| F16 | 16.4 GB | 18.9 GB |
Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.
Can my hardware run it?
| Device | Verdict | Best quant | Est. tok/s (Q4) |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 | Yes | F16 | ~223 |
| NVIDIA GeForce RTX 4090 | Yes | F16 | ~143 |
| AMD Radeon RX 7900 XTX | Yes | F16 | ~142 |
| NVIDIA GeForce RTX 5080 | Yes | Q8_0 | ~140 |
| NVIDIA GeForce RTX 5070 Ti | Yes | Q8_0 | ~136 |
| NVIDIA GeForce RTX 3090 Ti | Yes | F16 | ~132 |
| NVIDIA GeForce RTX 3090 | Yes | F16 | ~122 |
| NVIDIA GeForce RTX 4080 SUPER | Yes | Q8_0 | ~114 |
| NVIDIA GeForce RTX 3080 10GB | Yes | Q6_K | ~107 |
| NVIDIA GeForce RTX 4070 Ti SUPER | Yes | Q8_0 | ~102 |
| NVIDIA GeForce RTX 5070 | Yes | Q8_0 | ~98.1 |
| Apple M5 Max (40-core GPU, 128GB) | Yes | F16 | ~92.2 |
| Apple M2 Ultra (76-core GPU, 192GB) | Yes | F16 | ~72.5 |
| Apple M3 Ultra (80-core GPU, 512GB) | Yes | F16 | ~70.8 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Yes | Q8_0 | ~69.9 |
| Apple M4 Max (40-core GPU, 128GB) | Yes | F16 | ~63.9 |
| NVIDIA GeForce RTX 3060 12GB | Yes | Q8_0 | ~58.1 |
| Apple M5 Pro (20-core GPU, 64GB) | Yes | F16 | ~51.0 |
| Apple M3 Max (40-core GPU, 64GB) | Yes | F16 | ~51.0 |
| NVIDIA GeForce RTX 4060 Ti 8GB | Yes | Q5_K_M (4K ctx) | ~49.1 |
| Apple M1 Max (32-core GPU, 64GB) | Yes | F16 | ~47.0 |
| NVIDIA DGX Spark | Yes | F16 | ~44.0 |
| Apple M4 Pro (20-core GPU, 48GB) | Yes | F16 | ~39.0 |
| Apple M4 (10-core GPU, 24GB) | Yes, usable | Q8_0 | ~18.5 |
Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).