Mistral Small 3.1 24B

Mistral AI · 24B parameters · 131,072 token max context · Model card

Memory needed

QuantWeightsTotal @ 8K ctx
Q3_K_M11.7 GB14.2 GB
Q4_K_M14.5 GB17.1 GB
Q5_K_M17.1 GB19.8 GB
Q6_K19.8 GB22.5 GB
Q8_025.5 GB28.4 GB
F1648.0 GB51.6 GB

Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.

Can my hardware run it?

DeviceVerdictBest quantEst. tok/s (Q4)
NVIDIA GeForce RTX 5090YesQ8_0~76.2
NVIDIA GeForce RTX 5080YesQ3_K_M~59.5
NVIDIA GeForce RTX 5070 TiYesQ3_K_M~57.8
NVIDIA GeForce RTX 4090YesQ6_K~48.9
NVIDIA GeForce RTX 4080 SUPERYesQ3_K_M~48.5
AMD Radeon RX 7900 XTXYesQ6_K~48.4
NVIDIA GeForce RTX 3090 TiYesQ6_K~45.0
NVIDIA GeForce RTX 4070 Ti SUPERYesQ3_K_M~43.2
NVIDIA GeForce RTX 3090YesQ6_K~41.5
Apple M5 Max (40-core GPU, 128GB)YesF16~31.5
NVIDIA GeForce RTX 5060 Ti 16GBYesQ3_K_M~29.7
Apple M2 Ultra (76-core GPU, 192GB)YesF16~24.8
Apple M3 Ultra (80-core GPU, 512GB)YesF16~24.2
Apple M4 Max (40-core GPU, 128GB)YesF16~21.8
Apple M5 Pro (20-core GPU, 64GB)Yes, usableQ8_0~17.4
Apple M3 Max (40-core GPU, 64GB)Yes, usableQ8_0~17.4
Apple M1 Max (32-core GPU, 64GB)Yes, usableQ8_0~16.1
NVIDIA DGX SparkYes, usableF16~15.0
Apple M4 Pro (20-core GPU, 48GB)Yes, usableQ8_0~13.3
Apple M4 (10-core GPU, 24GB)BarelyQ4_K_M~6.3
NVIDIA GeForce RTX 5070No——
NVIDIA GeForce RTX 4060 Ti 8GBNo——
NVIDIA GeForce RTX 3060 12GBNo——
NVIDIA GeForce RTX 3080 10GBNo——

Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).

Rent a GPU instead

Don't own the hardware for Mistral Small 3.1 24B? Rent a cloud GPU with enough VRAM by the hour and run it there.

Vast.ai — cheap marketplace GPUs  ·  RunPod — on-demand GPU pods

Affiliate links: we may earn a commission at no extra cost to you.