The right local AI hardware is whatever holds your model's weights plus its working memory with room left over, and the arithmetic is more specific than the marketing suggests. For most people building an offline AI rig in 2026, that means 32 GB of system RAM, an NVIDIA card with at least 12 GB of VRAM or an Apple Silicon chip with unified memory, and a fast SSD. This article gives you the actual formulas, the thresholds we enforce in shipping code, and the failure modes we hit getting there.
We build and ship an offline AI system that has to run correctly on hardware we do not control — laptops from 2018, gaming desktops, Apple Silicon Macs, mini PCs in field deployments. Everything below comes out of making that work. The numbers are not benchmarks from a lab. They are thresholds in production code that decide what loads on your machine.
The one formula that matters for LLM hardware requirements
Almost every "how much VRAM do I need" answer online is wrong because it stops at the weights. The real requirement is:
Memory needed = weights + KV cache + compute buffers
Skip any of the three terms and you will buy a card that cannot load the model you bought it for. Here is each term, with real numbers from models we ship.
Term 1: Weights
This is the file on disk, and it is the only term most people count. At Q4_K_M quantization — the practical default for local inference — the shipped sizes look like this:
| Model | Parameters | Quantization | Weights on disk |
|---|---|---|---|
| Llama 3.1 Instruct | 8B | Q4_K_M | ~4.9 GB |
| Qwen2.5 Instruct | 14B | Q4_K_M | ~9.0 GB |
| Qwen3.6 A3B (mixture-of-experts) | 35B total / ~3.6B active | Q4_K_M | ~22.1 GB |
| Qwen2.5-VL (vision) | 3B + projector | Q4_K_M + f16 | ~1.9 GB + 1.34 GB |
A rough rule for any model at Q4_K_M: weights in GB ≈ parameters in billions × 0.63. A 70B model at Q4_K_M lands near 42 GB, which is why 70B is a workstation conversation and not a laptop one.
Term 2: KV cache — the silent killer
The KV cache stores the attention keys and values for every token in your context window. It scales linearly with context length, and it is where people get ambushed.
We hit this hard. Our vision model shipped with a 128k-token default context. At roughly 36 KB per token for that architecture, the KV cache alone was about 4.6 GB — before the weights, before the image batch, before the compute buffers. On 16 GB machines it out-of-memoried during allocation. Every single image request failed.
The fix was one flag. Capping the context to 4096 tokens cut the KV cache to roughly 0.15 GB — a thirty-fold reduction — and our evaluation set showed zero quality loss: 200 out of 200 on organism classification, 91 out of 91 on toxic fungus identification. Same weights, same model, same answers, one thirtieth of the memory.
The lesson generalizes: your context window is a hardware purchase. Before you buy a bigger card to run a longer context, check whether you actually need the longer context. Most retrieval and conversational workloads do fine at 4096.
Term 3: Compute buffers and runtime overhead
Prompt evaluation, batch processing, and intermediate activation tensors all need scratch space. This term is architecture-dependent and hard to compute precisely, so we budget it empirically. In our vision runtime we reserve 6 GB of overhead on top of weights and projector — that constant exists because we measured the real allocation failures, not because it is theoretically derivable.
For text-only inference at modest context, 1 to 2 GB of overhead is a reasonable planning figure. Do not budget zero.
Quantization: what to pick and what you give up
Quantization is the highest-leverage decision in local AI hardware planning, because it changes your memory requirement by a factor of four without changing your machine.
| Format | Bits/param | Size vs f16 | Quality | Use when |
|---|---|---|---|---|
| f16 / bf16 | 16 | 100% | Reference | You are training or evaluating, not deploying |
| Q8_0 | ~8.5 | ~53% | Effectively lossless | Memory is free and precision is critical |
| Q6_K | ~6.6 | ~41% | Very close to f16 | You have headroom and want margin |
| Q5_K_M | ~5.7 | ~36% | Slightly better than Q4 | The next size up does not fit |
| Q4_K_M | ~4.9 | ~31% | Small, rarely noticeable loss | Default. Start here. |
| Q3 / Q2 | 2–3.5 | ~15–22% | Visible degradation | Desperation only |
The most important consequence: a larger model at Q4 almost always beats a smaller model at Q8 for the same memory budget. A 14B at Q4_K_M (~9 GB) outperforms an 8B at Q8_0 (~8.5 GB) on essentially every task. Spend your memory on parameters, not on precision.
GPU: what VRAM tier actually buys you
A GPU is the single biggest speed lever in local AI hardware. Offloading model layers to VRAM moves you from tens of seconds per answer to single digits. But VRAM is a hard wall — when it is full, it is full.
| VRAM | Example cards | Realistically runs | What to expect |
|---|---|---|---|
| 6 GB | GTX 1660 Ti, RTX 2060, RTX 3050 | 8B at Q4, partial offload of larger | Real speedup on 8B; 14B spills to CPU |
| 8 GB | RTX 3060 Ti, RTX 4060 | 8B fully; 14B partial | The practical entry point for GPU inference |
| 12 GB | RTX 3060 12GB, RTX 4070 | 14B nearly full offload | Best value tier for a local LLM rig |
| 16 GB | RTX 4060 Ti 16GB, RTX 4080 | 14B fully, comfortably | Fast and roomy for mainstream models |
| 24 GB | RTX 3090, RTX 4090 | 14B trivially; 30B-class MoE fully | The enthusiast ceiling |
| 32–48 GB | RTX 5090, workstation cards | 30B-class comfortably, 70B partial | Diminishing returns per dollar |
Why reported free VRAM is frequently a lie
Here is the non-obvious finding that cost us the most debugging time, and it will save you a card purchase.
On Windows and Linux, the Vulkan-based VRAM query routinely reports the sum of multiple memory heaps — device-local VRAM plus host-visible system memory — as "free VRAM." The number it gives you is not the number you can actually allocate as dedicated video memory.
Concretely: a GTX 1660 Ti with 6 GB of physical VRAM reported over 10 GB free. It passed a naive 1.2× headroom check against a 9 GB model, began loading, and then hard-failed during tensor allocation. Not a graceful degradation — a crash.
So our runtime uses two different headroom multipliers depending on platform:
-
Apple Metal: 1.2× headroom. Metal exposes
recommendedMaxWorkingSetSize, which is an honest number. A tight margin is safe. - Vulkan / NVIDIA on Windows and Linux: 2.0× headroom. We require twice the model size in reported free VRAM before committing to full GPU offload, precisely because we do not trust the report.
At 2.0×, a 24 GB card passes for a 9 GB model. A 10 to 16 GB card does not, and drops to partial offload — which is the correct outcome for that hardware anyway. If you are sizing a card yourself, apply the same discipline: plan for your model to fit in half your card's advertised VRAM, not all of it.
What partial offload actually means
When the whole model will not fit, the runtime splits it: some transformer layers on the GPU, the rest on the CPU. If a model has 48 layers and 38 fit in VRAM, you get roughly 79% of the compute accelerated and the remaining 21% running at CPU speed — and the slow part gates the whole pipeline, because every token passes through every layer.
Partial offload is worth having, but the returns are steeply non-linear. We accept it as a good outcome above roughly 80% of layers. Below that, we only promote a machine to the larger model if it clears three additional gates: at least 16 GB of system RAM, at least 4 CPU cores, and a live calibration run measuring at least 8 tokens per second. Eight tokens per second is not arbitrary — a typical detailed answer is around 600 tokens, so 8 tok/s is 75 seconds of generation, which is the edge of tolerable. Below that we drop to a smaller model rather than make the user wait.
That is a useful acceptance test for your own rig: measure tokens per second on a real prompt, and if you are under 8, change something.
Apple Silicon: unified memory and the ceiling nobody mentions
Apple Silicon is the most underrated local AI hardware available, for one structural reason: unified memory. There is no separate VRAM pool and no transfer over PCIe. The GPU addresses the same physical RAM the CPU does, at high bandwidth. A 32 GB M-series Mac can put far more model into GPU-accessible memory than a 12 GB discrete card, in a fanless laptop drawing a fraction of the power.
Two caveats, and the second one is important enough that it should change what you buy.
Caveat one: your macOS version decides whether the GPU is used at all. On macOS 13 (Ventura) and earlier, local inference runtimes fall back to CPU only. On macOS 14 (Sonoma) and later, Metal acceleration is available and performance improves substantially. The upgrade is free. If you are on Ventura with capable hardware, that upgrade is the cheapest speedup you will ever get.
Caveat two: the Metal wired-memory limit is roughly 65 to 75 percent of your unified RAM. Not all of it is available to the GPU for model residency. This is why a 24 GB Mac — which looks like it should comfortably hold a 22.1 GB model — cannot. Sixty-five to seventy-five percent of 24 GB is 16 to 18 GB, which is less than 22.1 GB, so it fails. We exclude 24 GB Macs from our Expert tier for exactly this reason.
Practical guidance for Apple Silicon:
- 16 GB: comfortable with 8B; 14B works with GPU offload.
- 24 GB: excellent for 14B. Do not expect 30B-class residency.
- 32 GB: the real threshold for 30B-class models. This is the sweet spot.
- 64 GB+: everything up to and including 70B at Q4.
CPU-only inference and the mixture-of-experts loophole
Conventional wisdom says a big model without a GPU is unusable, and for dense models that is true. A dense 32B model on CPU generates roughly 2 to 4 tokens per second. A 600-token answer takes three to five minutes. Nobody will use that twice.
Mixture-of-experts architectures break the assumption, and this is the most important development in local AI hardware in years. In an MoE model, only a fraction of the parameters activate for any given token. The 35B model we ship has about 3.6B active parameters per token. That splits the requirement in two:
- Residency is a 35B problem. All 22.1 GB of weights must sit in RAM, because any expert might be needed for the next token.
- Compute is a 3.6B problem. Only the selected experts do arithmetic.
The result: roughly 30 tokens per second on CPU alone, and well past 100 with GPU acceleration. Thirty tokens per second is genuinely usable — a 600-token answer in twenty seconds, no graphics card involved.
This reshapes what you should buy. If a mixture-of-experts model is on your list, spend your money on RAM, not on a GPU. A mini PC with 32 GB or 64 GB of RAM and no discrete graphics will run a 30B-class MoE model at conversational speed. That is a fundamentally different and cheaper machine than the gaming rig people assume they need. It is also the reason a low-power, fanless, solar-friendly box is a viable AI server — a point we develop in building an off-grid AI server.
For CPU inference, prioritize in this order: memory capacity first, memory bandwidth second, core count third, clock speed last. Our own floor for the MoE tier is 32 GB of RAM and 8 cores. Below either, we do not attempt it.
What about NPUs?
Neural processing units now ship in Apple's M-series, Qualcomm Snapdragon X, Intel Core Ultra, and recent AMD parts. The honest assessment for 2026: NPUs are excellent for small, fixed models — wake words, background blur, on-device transcription — and are not yet the primary path for general LLM inference in the llama.cpp-class tooling most local AI runs on. Support is improving, but as of today, do not buy a machine for its TOPS number expecting it to accelerate your local LLM. Buy RAM and a real GPU, or buy Apple Silicon and use Metal.
Offline AI rig recommendations at four budgets
| Tier | Spec | Model class | Expected throughput |
|---|---|---|---|
| Entry | Any 2020+ laptop, 16 GB RAM, integrated graphics | 8B at Q4_K_M | ~5–12 tok/s; answers in 30–60 s |
| Solid | Ryzen 7 / Core i7, 32 GB RAM, RTX 4060 8GB — or Apple Silicon 24 GB on macOS 14+ | 14B at Q4_K_M | ~20–40 tok/s; answers in 10–20 s |
| Serious | 8+ cores, 32–64 GB RAM, RTX 4070 Ti / 4080 12–16GB — or Apple Silicon 32 GB | 14B fully offloaded; 30B-class MoE | ~40–80 tok/s; answers in under 10 s |
| Workstation | 12+ cores, 64–128 GB RAM, RTX 4090 / 5090 24–32GB | 30B-class fully; 70B partial | 100+ tok/s; effectively instant |
| Do not buy | Under 8 GB RAM · pre-2016 CPUs · Chromebooks · "AI-ready" 8 GB machines sold on TOPS | — | Swapping, crashes, or minutes per answer |
One line summary if you want a single recommendation: 32 GB of RAM and a 12 GB NVIDIA card, or a 32 GB Apple Silicon Mac on macOS 14 or later. That configuration runs everything most people will actually use, and it does not require a workstation budget.
Storage: the term everyone forgets
Storage does not affect how fast your model answers. It affects how long you wait before it answers anything at all, because the weights must be read from disk into RAM on every cold start.
- NVMe SSD: a 9 GB model loads in a few seconds.
- SATA SSD: 15 to 30 seconds.
- USB 3.x: roughly 30 to 90 seconds for a 5 GB model. Perfectly workable.
- USB 2.0: two minutes or more. Check your port — full-size USB-A ports on otherwise modern machines are often still 2.0.
- Spinning disk: avoid.
Once resident in RAM, storage speed becomes completely irrelevant for the rest of the session. This is why the practical advice for field use is to launch your offline AI before you need it, not after. We covered the full startup sequence in why your computer matters more than the drive.
How to test your machine before you trust it
Specs predict. Measurement decides. Run this on the machine you actually intend to use:
- Time the cold load. Launch from powered-off, stopwatch until the model is ready. Write the number down.
- Time a medium answer. Ask a real question that needs a few hundred words. Write that down too.
- Compute tokens per second. Roughly, output words × 1.3 ÷ seconds. Under 8 tok/s means change something — smaller model, more offload, or more RAM.
- Watch memory during generation. If you are swapping to disk, no other tuning will help. Reduce context or model size.
- Repeat with everything else open. Your real-world state is not a freshly rebooted machine with nothing running.
Do this once, keep the numbers, and you will never be surprised by your own hardware.
Frequently asked questions
How much VRAM do I need to run a 13B or 14B model?
At Q4_K_M a 14B model is about 9 GB of weights. Add 1 to 2 GB of KV cache at a 4096-token context plus compute buffers, so 11 to 12 GB is the working requirement. Because reported free VRAM is unreliable on Windows, plan for a 16 GB card for full offload, or a 12 GB card with near-full offload.
Can I run a local LLM without a GPU?
Yes. Dense models under 8B run acceptably on a modern CPU, and mixture-of-experts models change the equation entirely — a 35B MoE with 3.6B active parameters reaches roughly 30 tokens per second on CPU alone, provided you have the RAM to hold all 22 GB of weights.
Is Apple Silicon better than NVIDIA for local AI?
For memory capacity per dollar and for power efficiency, yes — unified memory lets a 32 GB Mac hold models a 12 GB discrete card cannot. For raw throughput on models that fit in VRAM, a high-end NVIDIA card is faster. Apple Silicon also requires macOS 14 or later for GPU acceleration at all.
How much RAM do I need for local AI?
8 GB is the floor and only for small models with everything else closed. 16 GB is the practical minimum for a good experience. 32 GB is the recommended target and the hard requirement for 30B-class mixture-of-experts models. 64 GB and up serves 70B-class work.
Does an NPU help run LLMs locally?
Not meaningfully yet. NPUs accelerate small fixed-function models well, but general LLM inference in the widely-used local runtimes still targets CPU, CUDA, and Metal. Do not choose a machine based on its TOPS rating expecting faster local AI chat.
What is the best budget offline AI rig?
A used or current-gen mini PC or desktop with 32 GB of DDR5 and a 12 GB RTX card gives you full 14B offload and headroom for mixture-of-experts models. If you do not need a GPU, a 32 GB fanless mini PC running an MoE model on CPU is cheaper, quieter, and draws far less power.
Does quantization hurt answer quality?
Q4_K_M loses a small, generally imperceptible amount versus 16-bit. The important trade-off is different from what people assume: for a fixed memory budget, a larger model at Q4 beats a smaller model at Q8 on nearly every task. Spend memory on parameters.
Why does my model crash instead of running slowly when memory is tight?
Tensor allocation is all-or-nothing. If the runtime believes there is enough VRAM — often because the driver over-reports free memory by summing heaps — it commits to the allocation and fails hard rather than degrading. That is exactly why a 2.0× headroom rule on non-Apple platforms is worth enforcing.
Where this leaves you
Local AI hardware planning comes down to four decisions: pick a quantization (Q4_K_M), cap your context deliberately (4096 unless you have a reason), size RAM against weights plus KV cache plus overhead, and treat reported free VRAM as an optimistic estimate rather than a fact.
If you would rather skip the sizing exercise entirely, that is what we built. Off-Grid AI Core ships three model tiers on one bootable USB and picks between them automatically using the exact thresholds described in this article — Speed on constrained machines, Balanced on mainstream hardware, and Expert on anything with 32 GB of RAM. Nothing compiles, nothing downloads, and every answer cites a source document you can open. Domain expansion packs add medical, veterinary, electrical, mechanical, tactical, and agricultural depth on top of the core library.
New to running models locally? Start with how to run an offline AI LLM without internet access. Want to see it identify things from photographs on the same hardware? See offline vision AI that cites sources instead of guessing.
