NVIDIA GeForce RTX 4060 Ti 8GB × Llama 3.2 3B Instruct
Can the RTX 4060 Ti 8GB run Llama 3.2 3B Instruct?
Yes
Yes. The RTX 4060 Ti 8GB can run Llama 3.2 3B Instruct at up to Q8_0 (needs ~5.2 GB of 7.6 GB usable), at an estimated ~126 tokens/sec at Q4_K_M.
Speed is estimated from memory bandwidth calibrated with a measured 63.86 tok/s Llama 2 7B Q4_0 result (source). No direct public benchmark of this exact pair is recorded yet.
By quantization (8K context)
| Quant | Memory needed | Fits? | Est. tok/s |
|---|---|---|---|
| Q3_K_M | 3.3 GB | ✅ | ~156 |
| Q4_K_M | 3.7 GB | ✅ | ~126 |
| Q5_K_M | 4.1 GB | ✅ | ~107 |
| Q6_K | 4.5 GB | ✅ | ~92.5 |
| Q8_0 | 5.2 GB | ✅ | ~71.8 |
| F16 | 8.3 GB | ❌ | — |
Fastest hardware for Llama 3.2 3B Instruct
- NVIDIA GeForce RTX 5090 — ~300 tok/s at Q4_K_M
- NVIDIA GeForce RTX 4090 — ~300 tok/s at Q4_K_M
- NVIDIA GeForce RTX 3090 — ~300 tok/s at Q4_K_M
- NVIDIA GeForce RTX 3090 Ti — ~300 tok/s at Q4_K_M
- NVIDIA GeForce RTX 5080 — ~300 tok/s at Q4_K_M
No suitable GPU? Rent one by the hour on Vast.ai or RunPod (affiliate links).
Get new models & benchmarks in your inbox
No spam. Unsubscribe anytime by replying to any email.