Methodology

Memory: weights = parameters × bits-per-weight ÷ 8 (Q4_K_M ≈ 4.85 bpw, Q8_0 ≈ 8.5, F16 = 16), +3% for embeddings/metadata, + fp16 KV cache at 8K context (2 × layers × KV heads × head dim × 2 bytes × tokens), + 0.8 GB runtime overhead.

Usable memory: 95% of GPU VRAM; 75% of Mac unified memory (macOS default GPU working-set limit).

Speed: token generation is memory-bandwidth-bound: tok/s ≈ efficiency × bandwidth ÷ active weight bytes. Efficiency is calibrated per device from a measured llama.cpp Llama 2 7B Q4_0 result (sources: llama.cpp CUDA scoreboard (Llama 2 7B Q4_0, tg128, no FA); llama.cpp Apple Silicon benchmarks (Llama 2 7B Q4_0, tg128)). MoE models use active parameters. Devices with no measurement use the average efficiency of their class. All pair speeds are labeled estimates.

Specs: GPU VRAM/bandwidth from vendor spec pages; Mac bandwidth from Apple specs as listed in the llama.cpp thread; model params/context/architecture from Hugging Face model cards.