Llama 3.1 8B Instruct

Meta · 8B parameters · 131,072 token max context · Model card

Memory needed

QuantWeightsTotal @ 8K ctx
Q3_K_M3.9 GB5.9 GB
Q4_K_M4.8 GB6.9 GB
Q5_K_M5.7 GB7.7 GB
Q6_K6.6 GB8.7 GB
Q8_08.5 GB10.6 GB
F1616.0 GB18.4 GB

Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.

Can my hardware run it?

DeviceVerdictBest quantEst. tok/s (Q4)
NVIDIA GeForce RTX 5090YesF16~229
NVIDIA GeForce RTX 4090YesF16~147
AMD Radeon RX 7900 XTXYesF16~145
NVIDIA GeForce RTX 5080YesQ8_0~143
NVIDIA GeForce RTX 5070 TiYesQ8_0~139
NVIDIA GeForce RTX 3090 TiYesF16~135
NVIDIA GeForce RTX 3090YesF16~125
NVIDIA GeForce RTX 4080 SUPERYesQ8_0~117
NVIDIA GeForce RTX 3080 10GBYesQ6_K~110
NVIDIA GeForce RTX 4070 Ti SUPERYesQ8_0~104
NVIDIA GeForce RTX 5070YesQ8_0~101
Apple M5 Max (40-core GPU, 128GB)YesF16~94.5
Apple M2 Ultra (76-core GPU, 192GB)YesF16~74.3
Apple M3 Ultra (80-core GPU, 512GB)YesF16~72.6
NVIDIA GeForce RTX 5060 Ti 16GBYesQ8_0~71.7
Apple M4 Max (40-core GPU, 128GB)YesF16~65.5
NVIDIA GeForce RTX 3060 12GBYesQ8_0~59.6
Apple M5 Pro (20-core GPU, 64GB)YesF16~52.3
Apple M3 Max (40-core GPU, 64GB)YesF16~52.3
NVIDIA GeForce RTX 4060 Ti 8GBYesQ5_K_M (4K ctx)~50.3
Apple M1 Max (32-core GPU, 64GB)YesF16~48.2
NVIDIA DGX SparkYesF16~45.1
Apple M4 Pro (20-core GPU, 48GB)YesF16~40.0
Apple M4 (10-core GPU, 24GB)Yes, usableF16 (4K ctx)~19.0

Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).

Rent a GPU instead

Don't own the hardware for Llama 3.1 8B Instruct? Rent a cloud GPU with enough VRAM by the hour and run it there.

Vast.ai — cheap marketplace GPUs  ·  RunPod — on-demand GPU pods

Affiliate links: we may earn a commission at no extra cost to you.