AMD Radeon RX 7900 XTX × Llama 3.1 8B Instruct
Can the RX 7900 XTX run Llama 3.1 8B Instruct?
Yes
Yes. The RX 7900 XTX can run Llama 3.1 8B Instruct at up to F16 (needs ~18.4 GB of 22.8 GB usable), at an estimated ~145 tokens/sec at Q4_K_M.
Speed is estimated from memory bandwidth (no measured benchmark for this device). No direct public benchmark of this exact pair is recorded yet.
By quantization (8K context)
| Quant | Memory needed | Fits? | Est. tok/s |
|---|---|---|---|
| Q3_K_M | 5.9 GB | ✅ | ~181 |
| Q4_K_M | 6.9 GB | ✅ | ~145 |
| Q5_K_M | 7.7 GB | ✅ | ~124 |
| Q6_K | 8.7 GB | ✅ | ~107 |
| Q8_0 | 10.6 GB | ✅ | ~82.9 |
| F16 | 18.4 GB | ✅ | ~44.0 |
Fastest hardware for Llama 3.1 8B Instruct
- NVIDIA GeForce RTX 5090 — ~229 tok/s at Q4_K_M
- NVIDIA GeForce RTX 4090 — ~147 tok/s at Q4_K_M
- NVIDIA GeForce RTX 5080 — ~143 tok/s at Q4_K_M
- NVIDIA GeForce RTX 5070 Ti — ~139 tok/s at Q4_K_M
- NVIDIA GeForce RTX 3090 Ti — ~135 tok/s at Q4_K_M
No suitable GPU? Rent one by the hour on Vast.ai or RunPod (affiliate links).
Get new models & benchmarks in your inbox
No spam. Unsubscribe anytime by replying to any email.