Mistral Nemo 12B

Mistral AI · 12.2B parameters · 131,072 token max context · Model card

Memory needed

QuantWeightsTotal @ 8K ctx
Q3_K_M5.9 GB8.3 GB
Q4_K_M7.4 GB9.8 GB
Q5_K_M8.7 GB11.1 GB
Q6_K10.1 GB12.5 GB
Q8_013.0 GB15.5 GB
F1624.4 GB27.3 GB

Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.

Can my hardware run it?

DeviceVerdictBest quantEst. tok/s (Q4)
NVIDIA GeForce RTX 5090YesF16~150
NVIDIA GeForce RTX 4090YesQ8_0~96.2
AMD Radeon RX 7900 XTXYesQ8_0~95.3
NVIDIA GeForce RTX 5080YesQ8_0 (4K ctx)~94.1
NVIDIA GeForce RTX 5070 TiYesQ8_0 (4K ctx)~91.4
NVIDIA GeForce RTX 3090 TiYesQ8_0~88.5
NVIDIA GeForce RTX 3090YesQ8_0~81.7
NVIDIA GeForce RTX 4080 SUPERYesQ8_0 (4K ctx)~76.7
NVIDIA GeForce RTX 3080 10GBYesQ4_K_M (4K ctx)~72.2
NVIDIA GeForce RTX 4070 Ti SUPERYesQ8_0 (4K ctx)~68.4
NVIDIA GeForce RTX 5070YesQ5_K_M~65.9
Apple M5 Max (40-core GPU, 128GB)YesF16~62.0
Apple M2 Ultra (76-core GPU, 192GB)YesF16~48.7
Apple M3 Ultra (80-core GPU, 512GB)YesF16~47.6
NVIDIA GeForce RTX 5060 Ti 16GBYesQ8_0 (4K ctx)~47.0
Apple M4 Max (40-core GPU, 128GB)YesF16~42.9
NVIDIA GeForce RTX 4060 Ti 8GBYesQ3_K_M (4K ctx)~41.0
NVIDIA GeForce RTX 3060 12GBYesQ5_K_M~39.1
Apple M5 Pro (20-core GPU, 64GB)YesF16~34.3
Apple M3 Max (40-core GPU, 64GB)YesF16~34.3
Apple M1 Max (32-core GPU, 64GB)YesF16~31.6
NVIDIA DGX SparkYesF16~29.6
Apple M4 Pro (20-core GPU, 48GB)YesF16~26.2
Apple M4 (10-core GPU, 24GB)Yes, usableQ8_0~12.5

Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).

Rent a GPU instead

Don't own the hardware for Mistral Nemo 12B? Rent a cloud GPU with enough VRAM by the hour and run it there.

Vast.ai — cheap marketplace GPUs  ·  RunPod — on-demand GPU pods

Affiliate links: we may earn a commission at no extra cost to you.