NVIDIA RTX 4090 24GB

NVIDIA’s flagship consumer GPU, two generations on this site: RTX 5090 (32GB GDDR7, 1,792 GB/s, 575W, $1,999, 2025) and RTX 4090 (24GB GDDR6X, 1,008 GB/s, 450W, $1,599 MSRP, 2022). Same architecture story: the highest memory bandwidth per dollar of any consumer card, and decode speed to match.

What it does well:

• Raw decode speed: the 5090’s 1,792 GB/s makes it the fastest single-card tok/s on this site for models that fit in 32GB - 32B dense at 8-bit, 70B at 3-bit, large MoE at 1-2 bit.
• Ecosystem: CUDA means every runtime runs first and best here - llama.cpp, Ollama, vLLM, exllamav2, and the newest quant formats.

Where it falls short: VRAM, not bandwidth, is the ceiling - 32GB cannot hold 70B at 4-bit with real context. Street prices on both cards have run well above MSRP since launch.

Run it locally: GGUF quants via Ollama or llama.cpp, EXL2/EXL3 via TabbyAPI. The modeldex results rank checkpoints by fit and estimated tok/s.