Self-hosting open-weight models removes per-token metered billing, eliminates third-party prompt retention, and keeps codebases offline—provided you size weights, KV caches, and memory bandwidth correctly before downloading a 40 GB checkpoint. Use this command center to compute exact VRAM footprints across NVIDIA RTX, Apple Silicon, and AMD Strix Halo rigs, compare Ollama, llama.cpp, MLX, vLLM, and SGLang, copy working CLI launch flags, and check hardware break-even against cloud APIs.
Total memory equals quantized weight size (parameters × bits per weight / 8) plus context KV cache plus ~1–1.5 GB of runtime compute buffers. For a 32B model at Q4_K_M (~18.4 GiB) with a 32K FP16 context, 24 GB VRAM fits tightly unless you enable Q8_0 KV caching to halve context RAM.
Autoregressive decoding at batch size 1 loads every active parameter weight through GPU or unified RAM once for each generated token. Decode speed in tokens/sec tracks Memory Bandwidth (GB/s) divided by active weight GB—making GDDR7 (RTX 5090 ~1,792 GB/s), HBM3, M3 Ultra (819 GB/s), and sparse MoE checkpoints dramatically faster.
Choose Ollama or LM Studio for fast single-developer setup, Apple MLX for lowest-latency native execution on M-series Macs, raw llama-server for precise KV-cache quant and grammar flags across CUDA/Vulkan/CPU, EXL2 for maximum consumer NVIDIA speed, and vLLM or SGLang when serving concurrent multi-user API traffic on Linux GPUs.