Gemma 3 12B

Google · 12.2B parameters · 131,072 token max context · Model card

Memory needed

QuantWeightsTotal @ 8K ctx
Q3_K_M5.9 GB10.1 GB
Q4_K_M7.4 GB11.6 GB
Q5_K_M8.7 GB13.0 GB
Q6_K10.1 GB14.4 GB
Q8_013.0 GB17.4 GB
F1624.4 GB29.2 GB

Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.

Can my hardware run it?

DeviceVerdictBest quantEst. tok/s (Q4)
NVIDIA GeForce RTX 5090YesF16~150
NVIDIA GeForce RTX 4090YesQ8_0~96.2
AMD Radeon RX 7900 XTXYesQ8_0~95.3
NVIDIA GeForce RTX 5080YesQ6_K~94.1
NVIDIA GeForce RTX 5070 TiYesQ6_K~91.4
NVIDIA GeForce RTX 3080 10GBYesQ3_K_M (4K ctx)~89.8
NVIDIA GeForce RTX 3090 TiYesQ8_0~88.5
NVIDIA GeForce RTX 3090YesQ8_0~81.7
NVIDIA GeForce RTX 4080 SUPERYesQ6_K~76.7
NVIDIA GeForce RTX 4070 Ti SUPERYesQ6_K~68.4
NVIDIA GeForce RTX 5070YesQ5_K_M (4K ctx)~65.9
Apple M5 Max (40-core GPU, 128GB)YesF16~62.0
Apple M2 Ultra (76-core GPU, 192GB)YesF16~48.7
Apple M3 Ultra (80-core GPU, 512GB)YesF16~47.6
NVIDIA GeForce RTX 5060 Ti 16GBYesQ6_K~47.0
Apple M4 Max (40-core GPU, 128GB)YesF16~42.9
NVIDIA GeForce RTX 3060 12GBYesQ5_K_M (4K ctx)~39.1
Apple M5 Pro (20-core GPU, 64GB)YesF16~34.3
Apple M3 Max (40-core GPU, 64GB)YesF16~34.3
Apple M1 Max (32-core GPU, 64GB)YesF16~31.6
NVIDIA DGX SparkYesF16~29.6
Apple M4 Pro (20-core GPU, 48GB)YesF16~26.2
Apple M4 (10-core GPU, 24GB)Yes, usableQ8_0~12.5
NVIDIA GeForce RTX 4060 Ti 8GBNo——

Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).

Rent a GPU instead

Don't own the hardware for Gemma 3 12B? Rent a cloud GPU with enough VRAM by the hour and run it there.

Vast.ai — cheap marketplace GPUs  ·  RunPod — on-demand GPU pods

Affiliate links: we may earn a commission at no extra cost to you.