NVIDIA DGX Spark × DeepSeek R1 Distill Llama 70B
Can the DGX Spark run DeepSeek R1 Distill Llama 70B?
Barely
Yes. The DGX Spark can run DeepSeek R1 Distill Llama 70B at up to Q8_0 (needs ~80.7 GB of 115 GB usable), at an estimated ~5.1 tokens/sec at Q4_K_M.
Speed is estimated from memory bandwidth calibrated with a measured 57.21 tok/s Llama 2 7B Q4_0 result (source). No direct public benchmark of this exact pair is recorded yet.
By quantization (8K context)
| Quant | Memory needed | Fits? | Est. tok/s |
|---|---|---|---|
| Q3_K_M | 38.9 GB | ✅ | ~6.4 |
| Q4_K_M | 47.6 GB | ✅ | ~5.1 |
| Q5_K_M | 55.3 GB | ✅ | ~4.3 |
| Q6_K | 63.5 GB | ✅ | ~3.8 |
| Q8_0 | 80.7 GB | ✅ | ~2.9 |
| F16 | 149 GB | ❌ | — |
Fastest hardware for DeepSeek R1 Distill Llama 70B
- Apple M5 Max (40-core GPU, 128GB) — ~10.7 tok/s at Q4_K_M
- Apple M2 Ultra (76-core GPU, 192GB) — ~8.4 tok/s at Q4_K_M
- Apple M3 Ultra (80-core GPU, 512GB) — ~8.2 tok/s at Q4_K_M
- Apple M4 Max (40-core GPU, 128GB) — ~7.4 tok/s at Q4_K_M
- Apple M5 Pro (20-core GPU, 64GB) — ~5.9 tok/s at Q4_K_M
No suitable GPU? Rent one by the hour on Vast.ai or RunPod (affiliate links).
Get new models & benchmarks in your inbox
No spam. Unsubscribe anytime by replying to any email.