Qwen2.5 Coder 32B Instruct

Alibaba Qwen · 32.8B parameters · 131,072 token max context · Model card

Memory needed

QuantWeightsTotal @ 8K ctx
Q3_K_M16.0 GB19.4 GB
Q4_K_M19.9 GB23.4 GB
Q5_K_M23.4 GB27.0 GB
Q6_K27.1 GB30.8 GB
Q8_034.8 GB38.8 GB
F1665.6 GB70.5 GB

Approximate GGUF sizes from bits-per-weight; totals include fp16 KV cache and ~0.8 GB overhead.

Can my hardware run it?

DeviceVerdictBest quantEst. tok/s (Q4)
NVIDIA GeForce RTX 5090YesQ6_K (4K ctx)~55.7
NVIDIA GeForce RTX 4090YesQ4_K_M (4K ctx)~35.8
AMD Radeon RX 7900 XTXYesQ4_K_M (4K ctx)~35.4
NVIDIA GeForce RTX 3090 TiYesQ4_K_M (4K ctx)~32.9
NVIDIA GeForce RTX 3090YesQ4_K_M (4K ctx)~30.4
Apple M5 Max (40-core GPU, 128GB)YesF16~23.1
Apple M2 Ultra (76-core GPU, 192GB)Yes, usableF16~18.1
Apple M3 Ultra (80-core GPU, 512GB)Yes, usableF16~17.7
Apple M4 Max (40-core GPU, 128GB)Yes, usableF16~16.0
Apple M5 Pro (20-core GPU, 64GB)Yes, usableQ8_0~12.8
Apple M3 Max (40-core GPU, 64GB)Yes, usableQ8_0~12.7
Apple M1 Max (32-core GPU, 64GB)Yes, usableQ8_0~11.8
NVIDIA DGX SparkYes, usableF16~11.0
Apple M4 Pro (20-core GPU, 48GB)Yes, usableQ6_K~9.8
NVIDIA GeForce RTX 5080No——
NVIDIA GeForce RTX 5070 TiNo——
NVIDIA GeForce RTX 5070No——
NVIDIA GeForce RTX 5060 Ti 16GBNo——
NVIDIA GeForce RTX 4080 SUPERNo——
NVIDIA GeForce RTX 4070 Ti SUPERNo——
NVIDIA GeForce RTX 4060 Ti 8GBNo——
NVIDIA GeForce RTX 3060 12GBNo——
NVIDIA GeForce RTX 3080 10GBNo——
Apple M4 (10-core GPU, 24GB)No——

Speeds estimated from memory bandwidth (calibrated per device with a measured llama.cpp benchmark where available).

Rent a GPU instead

Don't own the hardware for Qwen2.5 Coder 32B Instruct? Rent a cloud GPU with enough VRAM by the hour and run it there.

Vast.ai — cheap marketplace GPUs  ·  RunPod — on-demand GPU pods

Affiliate links: we may earn a commission at no extra cost to you.