NVIDIA GeForce RTX 3060 12GB × Mistral Small 3.1 24B
Can the RTX 3060 12GB run Mistral Small 3.1 24B?
No
No. Mistral Small 3.1 24B needs at least ~14.2 GB, but the RTX 3060 12GB has ~11.4 GB usable. You would need CPU offloading (much slower) or more memory.
By quantization (8K context)
| Quant | Memory needed | Fits? | Est. tok/s |
|---|---|---|---|
| Q3_K_M | 14.2 GB | ❌ | — |
| Q4_K_M | 17.1 GB | ❌ | — |
| Q5_K_M | 19.8 GB | ❌ | — |
| Q6_K | 22.5 GB | ❌ | — |
| Q8_0 | 28.4 GB | ❌ | — |
| F16 | 51.6 GB | ❌ | — |
Fastest hardware for Mistral Small 3.1 24B
- NVIDIA GeForce RTX 5090 — ~76.2 tok/s at Q4_K_M
- NVIDIA GeForce RTX 5080 — ~59.5 tok/s at Q3_K_M
- NVIDIA GeForce RTX 5070 Ti — ~57.8 tok/s at Q3_K_M
- NVIDIA GeForce RTX 4090 — ~48.9 tok/s at Q4_K_M
- NVIDIA GeForce RTX 4080 SUPER — ~48.5 tok/s at Q3_K_M
Rent a GPU instead
Mistral Small 3.1 24B doesn't fit here? Rent a cloud GPU with enough VRAM by the hour and run it there.
Vast.ai — cheap marketplace GPUs · RunPod — on-demand GPU pods
Affiliate links: we may earn a commission at no extra cost to you.
Get new models & benchmarks in your inbox
No spam. Unsubscribe anytime by replying to any email.