Leaderboard

Measured llama.cpp text-generation speed (tg128) on Llama 2 7B Q4_0 — the standard community benchmark. Every number links to its source.

#	Device	Memory	Bandwidth	tok/s	Tok/s per $ (MSRP)	Source
1	NVIDIA GeForce RTX 5090	32 GB	1792 GB/s	290.02	14.5 /$100	CUDA thread
2	NVIDIA GeForce RTX 4090	24 GB	1008 GB/s	186.21	11.6 /$100	CUDA thread
3	NVIDIA GeForce RTX 5080	16 GB	960 GB/s	181.99	18.2 /$100	CUDA thread
4	NVIDIA GeForce RTX 5070 Ti	16 GB	896 GB/s	176.85	23.6 /$100	CUDA thread
5	NVIDIA GeForce RTX 3090 Ti	24 GB	1008 GB/s	171.19	8.6 /$100	CUDA thread
6	NVIDIA GeForce RTX 3090	24 GB	936 GB/s	158.16	10.6 /$100	CUDA thread
7	NVIDIA GeForce RTX 4080 SUPER	16 GB	736 GB/s	148.33	14.8 /$100	CUDA thread
8	NVIDIA GeForce RTX 3080 10GB	10 GB	760 GB/s	139.65	20.0 /$100	CUDA thread
9	NVIDIA GeForce RTX 4070 Ti SUPER	16 GB	672 GB/s	132.26	16.6 /$100	CUDA thread
10	NVIDIA GeForce RTX 5070	12 GB	672 GB/s	127.54	23.2 /$100	CUDA thread
11	Apple M5 Max (40-core GPU, 128GB)	128 GB	614 GB/s	119.92	—	Metal thread
12	Apple M2 Ultra (76-core GPU, 192GB)	192 GB	800 GB/s	94.27	—	Metal thread
13	Apple M3 Ultra (80-core GPU, 512GB)	512 GB	800 GB/s	92.14	—	Metal thread
14	NVIDIA GeForce RTX 5060 Ti 16GB	16 GB	448 GB/s	90.94	21.2 /$100	CUDA thread
15	Apple M4 Max (40-core GPU, 128GB)	128 GB	546 GB/s	83.06	—	Metal thread
16	NVIDIA GeForce RTX 3060 12GB	12 GB	360 GB/s	75.57	23.0 /$100	CUDA thread
17	Apple M5 Pro (20-core GPU, 64GB)	64 GB	307 GB/s	66.33	—	Metal thread
18	Apple M3 Max (40-core GPU, 64GB)	64 GB	400 GB/s	66.31	—	Metal thread
19	NVIDIA GeForce RTX 4060 Ti 8GB	8 GB	288 GB/s	63.86	16.0 /$100	CUDA thread
20	Apple M1 Max (32-core GPU, 64GB)	64 GB	400 GB/s	61.19	—	Metal thread
21	NVIDIA DGX Spark	128 GB	273 GB/s	57.21	1.4 /$100	CUDA thread
22	Apple M4 Pro (20-core GPU, 48GB)	48 GB	273 GB/s	50.74	—	Metal thread
23	Apple M4 (10-core GPU, 24GB)	24 GB	120 GB/s	24.11	—	Metal thread
24	AMD Radeon RX 7900 XTX	24 GB	960 GB/s	—	—	none found

Benchmarks use different llama.cpp builds (CUDA scoreboard: recent builds; Apple: build 8e672ef, Nov 2023, except M5 rows on c1d0e7a/60081bb), so cross-platform comparison is approximate.

No suitable GPU? Rent one by the hour on Vast.ai or RunPod (affiliate links).