NVIDIA GeForce RTX 4090 × DeepSeek R1 Distill Qwen 32B
Can the RTX 4090 run DeepSeek R1 Distill Qwen 32B?
Yes
Yes. The RTX 4090 can run DeepSeek R1 Distill Qwen 32B at up to Q4_K_M (needs ~22.4 GB at 4K context of 22.8 GB usable), at an estimated ~35.8 tokens/sec at Q4_K_M.
Speed is estimated from memory bandwidth calibrated with a measured 186.21 tok/s Llama 2 7B Q4_0 result (source). No direct public benchmark of this exact pair is recorded yet.
By quantization (8K context)
| Quant | Memory needed | Fits? | Est. tok/s |
|---|---|---|---|
| Q3_K_M | 19.4 GB | ✅ | ~44.5 |
| Q4_K_M | 23.4 GB | ✅ at 4K ctx | ~35.8 |
| Q5_K_M | 27.0 GB | ❌ | — |
| Q6_K | 30.8 GB | ❌ | — |
| Q8_0 | 38.8 GB | ❌ | — |
| F16 | 70.5 GB | ❌ | — |
Fastest hardware for DeepSeek R1 Distill Qwen 32B
- NVIDIA GeForce RTX 5090 — ~55.7 tok/s at Q4_K_M
- AMD Radeon RX 7900 XTX — ~35.4 tok/s at Q4_K_M
- NVIDIA GeForce RTX 3090 Ti — ~32.9 tok/s at Q4_K_M
- NVIDIA GeForce RTX 3090 — ~30.4 tok/s at Q4_K_M
- Apple M5 Max (40-core GPU, 128GB) — ~23.1 tok/s at Q4_K_M
No suitable GPU? Rent one by the hour on Vast.ai or RunPod (affiliate links).
Get new models & benchmarks in your inbox
No spam. Unsubscribe anytime by replying to any email.