Deploy fast, private open-weight models on consumer GPUs, Apple Silicon, Strix Halo, or server hardware without VRAM OOMs.
Every lab runs completely client-side with copyable prompts and browser checkpoints.
Curriculum Progress
0 of 4 modules completed (0%)
Local memory footprint equals quantized model weights plus dynamic context KV cache plus ~1 GB runtime scratchpad. GGUF K-quants store mixed precision with block scale factors (so Q4_K_M averages ~4.8 bpw rather than exactly 4.0 bpw), while FP16 KV cache scales linearly with layers, KV heads, and active context length.
Check whether Qwen3-Coder-30B-A3B (Q4_K_M) + 32,768 tokens fits cleanly in 24 GB VRAM: 1. Weights = 30.5B * 4.8 bpw / 8 ~= 17.0 GiB 2. FP16 KV cache @ 48 layers x 4 GQA heads x 128 dim = ~3.0 GiB (drops to ~1.5 GiB at Q8_0) 3. Runtime compute buffer ~= 1.0 GiB Total with Q8_0 KV = ~19.5 GiB -> safely fits 24 GB with ~4.5 GiB margin.
Which two settings save gigabytes of VRAM when running long 32K+ contexts locally?
Calculate GGUF weights + KV cache, match RTX / Apple Silicon / Strix Halo rigs, and compare Ollama, llama.cpp, MLX & vLLM.
Compare self-hosting an LLM on your own GPU against a cloud API, including hardware, power, and break-even.
Match hidden-gem GitHub repos (<3.5k ★) for compression, routing, MCP code maps, and edge serving.
Browse stored pricing, context windows, and benchmark data for 60+ models.
Yes. Every calculation uses interactive presets in /local-llms and /tools/local-vs-cloud covering consumer 8 GB GPUs, Apple Silicon Macs, Strix Halo mini-PCs, and datacenter nodes.
Yes. Module 3 walks through exact tradeoffs and CLI flags for each engine depending on platform and concurrency.
Use /local-llms for weight/KV sizing and launch flags, /tools/local-vs-cloud for financial break-even, and /libraries?category=local-edge-inference for open-source routing tools.