Gemma 3 4B
Google · 4.3B parameters · 131,072 token max context · Model card
Memory needed
| Quant | Weights | Total @ 8K ctx |
|---|---|---|
| Q3_K_M | 2.1 GB | 4.1 GB |
| Q4_K_M | 2.6 GB | 4.6 GB |
| Q5_K_M | 3.1 GB | 5.1 GB |
| Q6_K | 3.5 GB | 5.6 GB |
| Q8_0 | 4.6 GB | 6.6 GB |
| F16 | 8.6 GB | 10.8 GB |
Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.
Can my hardware run it?
| Device | Verdict | Best quant | Est. tok/s (Q4) |
|---|---|---|---|
| NVIDIA GeForce RTX 5090 | Yes | F16 | ~300 |
| NVIDIA GeForce RTX 4090 | Yes | F16 | ~273 |
| AMD Radeon RX 7900 XTX | Yes | F16 | ~270 |
| NVIDIA GeForce RTX 5080 | Yes | F16 | ~267 |
| NVIDIA GeForce RTX 5070 Ti | Yes | F16 | ~259 |
| NVIDIA GeForce RTX 3090 Ti | Yes | F16 | ~251 |
| NVIDIA GeForce RTX 3090 | Yes | F16 | ~232 |
| NVIDIA GeForce RTX 4080 SUPER | Yes | F16 | ~217 |
| NVIDIA GeForce RTX 3080 10GB | Yes | Q8_0 | ~205 |
| NVIDIA GeForce RTX 4070 Ti SUPER | Yes | F16 | ~194 |
| NVIDIA GeForce RTX 5070 | Yes | F16 | ~187 |
| Apple M5 Max (40-core GPU, 128GB) | Yes | F16 | ~176 |
| Apple M2 Ultra (76-core GPU, 192GB) | Yes | F16 | ~138 |
| Apple M3 Ultra (80-core GPU, 512GB) | Yes | F16 | ~135 |
| NVIDIA GeForce RTX 5060 Ti 16GB | Yes | F16 | ~133 |
| Apple M4 Max (40-core GPU, 128GB) | Yes | F16 | ~122 |
| NVIDIA GeForce RTX 3060 12GB | Yes | F16 | ~111 |
| Apple M5 Pro (20-core GPU, 64GB) | Yes | F16 | ~97.3 |
| Apple M3 Max (40-core GPU, 64GB) | Yes | F16 | ~97.2 |
| NVIDIA GeForce RTX 4060 Ti 8GB | Yes | Q8_0 | ~93.6 |
| Apple M1 Max (32-core GPU, 64GB) | Yes | F16 | ~89.7 |
| NVIDIA DGX Spark | Yes | F16 | ~83.9 |
| Apple M4 Pro (20-core GPU, 48GB) | Yes | F16 | ~74.4 |
| Apple M4 (10-core GPU, 24GB) | Yes | F16 | ~35.4 |
Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).