To run an offline AI LLM, you need three things on local storage: the model weights, an inference runtime that can execute them, and enough RAM to hold the weights while they run. Once those are in place, a local AI chat session works exactly the same with the network cable pulled as with it plugged in — no API key, no account, no signal. This guide covers the four practical routes to get there, the step-by-step process for doing it yourself, and the one thing almost every offline LLM setup gets wrong.
What "offline AI LLM" actually means
The phrase gets used loosely, so it is worth being precise. A large language model is a file — a large matrix of numbers called weights. On its own it does nothing. A runtime loads those weights into memory and performs the arithmetic that turns your question into a response. Neither step needs the internet. The internet is only involved because most people encounter AI through a service that happens to run those steps on someone else's computer.
When you run an LLM offline, you move both steps onto your machine. The weights sit on your drive. The runtime executes on your CPU or GPU. Your question never leaves the building, and no part of the process requires a connection.
There are three separate things people conflate when they say "offline AI," and keeping them apart will save you a lot of disappointment:
- Offline inference — the model runs locally. This is the easy part, and it is what most tutorials cover.
- Offline knowledge — the model actually knows the thing you need. A general-purpose model was trained on the open internet, which means its knowledge is broad, dated, and unevenly reliable in the specific domains where being wrong is expensive.
- Offline verification — you can check the answer against a source without going online to do it. This is the part virtually nobody solves, and it is the part that matters most when you cannot call anyone to confirm.
A stock local LLM setup gives you the first. This guide will get you there, and then be honest about the second and third. If you want the category defined from first principles before you build anything, start with what is air-gapped AI.
Why cloud AI fails exactly when you need it most
Cloud AI is excellent right up until one of these happens:
- The grid goes down. No power at the tower means no cell data, and your home internet died with the first outage.
- You have no signal. Backcountry, basements, steel buildings, most of rural America, and roughly seventy percent of the planet's landmass.
- The service changes. Models get deprecated, pricing changes, terms of service get rewritten, and features you built a habit around disappear on someone else's schedule.
- The content is sensitive. Legal work, medical questions about a real person, proprietary engineering, security research. Every prompt you send to a hosted service is a disclosure to a third party.
- You are rate limited or logged out at the moment you needed one specific answer.
None of these are hypothetical. They are the ordinary failure modes of a system whose critical dependency lives somewhere else. An offline AI LLM removes the dependency. That is the entire value proposition, and it is a big one.
The four ways to run an LLM offline, honestly compared
There is no single right answer here. There is a real trade between how much time you want to spend and how much you need to trust the result.
| Route | Setup effort | Ongoing maintenance | Works offline day one? | Cites its sources? | Tamper-evident? |
|---|---|---|---|---|---|
| llama.cpp + GGUF by hand | High — compile or fetch binaries, learn the flags | You own every update | Only if you downloaded everything first | No | Only if you hash it yourself |
| Ollama / LM Studio | Low — installer, then pull a model | Low, but the pull step needs internet | Only after a successful online first run | No | No |
| Container stack (Docker + a web UI) | Medium to high | Highest — images, volumes, versions | Only with a pre-pulled image cache | Depends what you bolt on | No |
| Pre-built signed appliance | None — plug in and launch | None required; updates optional | Yes | Yes | Yes — refuses to run if altered |
The first three routes are the same product with different amounts of wrapping: a general-purpose model, running locally, answering from whatever it absorbed during training. The fourth is a different category, and the difference is covered further down.
How to run an LLM offline yourself, step by step
If you want to build it, build it. This is the actual sequence, and none of it is difficult — it is just unforgiving about order of operations.
Step 1: Choose a model sized to your machine, not your ambition
Model names carry a parameter count: 3B, 8B, 14B, 70B. More parameters generally means better reasoning and better instruction-following, and it definitely means more memory. The parameter count is the first filter, because a model that does not fit in RAM does not run at any speed.
Good starting points by machine class:
- 8 GB RAM: a 3B to 4B model. Usable for summarizing and simple Q&A.
- 16 GB RAM: a 7B or 8B model comfortably; a 14B model with GPU help or patience.
- 32 GB RAM: 14B comfortably, and this is where mixture-of-experts models in the 30B class become viable.
- 64 GB+ RAM: essentially anything short of the frontier open-weight models.
Step 2: Choose a quantization, and choose Q4_K_M
Raw model weights ship at 16 bits per parameter, which makes an 8B model roughly 16 GB. Quantization compresses those weights to fewer bits, trading a small amount of quality for a large amount of memory. The formats you will see are named like Q4_K_M, Q5_K_M, Q6_K, and Q8_0.
Q4_K_M is the default recommendation across the local AI world for a good reason: it cuts memory roughly four-fold against 16-bit while the measured quality loss stays small enough that most users cannot detect it in ordinary use. It is what we ship on every model tier in Off-Grid AI, and it is what you should start with. Go up to Q5 or Q6 only if you have memory to spare and a task that is genuinely sensitive to precision.
Step 3: Do the memory arithmetic before you download 20 GB
The mistake almost everyone makes is assuming the file size is the memory requirement. It is not. Your actual requirement is:
weights + KV cache + compute buffers + whatever your operating system is already using
The KV cache is working memory that scales with your context window, and it can be enormous. We hit this directly: our vision model shipped with a 128k default context, which produced roughly a 4.6 GB KV cache and caused hard out-of-memory failures on 16 GB machines. Capping the context to 4096 tokens dropped that to about 0.15 GB with no measurable quality loss on our evaluation set. Same weights, same model, thirty-fold difference in memory pressure — from one flag.
Practical rule: budget the weight file size, plus 1 to 2 GB for a modest context, plus 4 GB for your operating system. If that total exceeds your installed RAM, pick a smaller model. When a machine runs out of RAM it starts swapping to disk, and a model that would have loaded in sixty seconds can take ten minutes — or crash outright.
Step 4: Download everything while you still have internet
This sounds obvious and it is the single most common failure. An offline AI setup is only offline-capable after a successful online setup. Before you go anywhere, pull down:
- The model weights file, in full — verify the byte count matches what the source lists.
- The runtime binaries for your platform and architecture.
- Any tokenizer or projector files the model requires as companions.
- The documentation, as a local file. You will not be able to search the web for a flag you forgot.
Step 5: Verify the hash
Run a SHA-256 checksum against the downloaded weights and compare it to the published value. A truncated or corrupted multi-gigabyte download often fails in ways that look like a software bug rather than a bad file, and you do not want to diagnose that in a situation where you cannot re-download. This takes thirty seconds and it is the difference between a verified payload and a hopeful one.
Step 6: First run, with the context size set deliberately
Load the model and set the context window explicitly rather than accepting the default. Defaults are chosen by model publishers to showcase capability, not to fit your hardware. A 4096-token context is plenty for conversational use and for most retrieval workloads, and as shown above it costs a fraction of the memory.
Expect the first load to be slow. The weights have to be read from storage into RAM, which on a fast SSD takes seconds and from a USB 3.x drive takes roughly thirty to ninety seconds for a 5 GB model. On a USB 2.0 port it can exceed two minutes. Once loaded, the model stays resident and every subsequent question is fast — storage speed stops mattering entirely.
Step 7: Prove it is actually offline
Do not take anyone's word for this, including ours. Turn on airplane mode, or physically unplug the ethernet and disable Wi-Fi, and then use the tool for ten minutes. If it works, it is offline. If any part of it stalls, errors, or silently degrades, something in your stack was reaching out. For a stricter check, watch outbound connections with your firewall or a network monitor during a session and confirm nothing leaves. We go deeper on what a genuinely zero-egress design looks like in how privacy-focused offline AI works without internet.
The part a DIY local LLM setup will not solve
Follow those seven steps and you will have a working local AI chat. It will be private, it will be free to run, and it will work with the network unplugged. It will also have a specific and serious weakness: it does not know anything in particular, and it will not tell you when it is guessing.
A general-purpose model produces the most statistically plausible continuation of your prompt. Under normal conditions that is remarkably useful. Under pressure — a question with a number in the answer, a dosage, a wire gauge, a purification ratio, a torque spec — plausible and correct are not the same thing, and the model has no mechanism to tell you which one it just produced. It has no citation. It has no source. It has no way to refuse.
That gap is why Off-Grid AI is built differently. It is not a chatbot with a survival prompt attached. The architecture inverts the usual arrangement:
- Retrieval first, generation second. Every question searches a curated corpus of 685 vetted source documents, indexed into 35,799 individually cited chunks. The model is only permitted to reason over what retrieval returned — it is not permitted to answer from training data.
- Every claim carries a citation. Answers come with a sources panel, and each citation opens the underlying document. You can check the work without going online to do it.
- It refuses. When retrieval does not surface sufficient grounding, the system declines rather than improvising. Incomplete is an acceptable state. Confidently wrong is not.
- It is cryptographically verifiable. Every corpus file is SHA-256 hashed, the version manifest is Ed25519 signed, and the runtime verifies integrity at startup. If anything on the drive has been altered, it refuses to start rather than serving you modified knowledge.
- It is deterministic. Same corpus, same model, same question produces the same answer. That is a testable property, and we test it.
The system also ships three model tiers and selects between them automatically based on what your hardware can actually run: Speed (an 8B model) for constrained machines, Balanced (a 14B model) for mainstream laptops and desktops, and Expert (a 35B mixture-of-experts model) for machines with 32 GB of RAM or more. You do not configure any of it. For a deeper look at how that selection works, see our guide to running a 32B-class model offline even without a GPU.
If you want an unconstrained local AI chat sandbox alongside the cited system — for brainstorming, drafting, and general conversation where citations are not the point — that ships too, deliberately walled off from the verified side. We wrote about why that separation matters in AI Playground: a fully offline local AI chat sandbox, kept honestly separate.
A quick hardware reality check
Whichever route you take, response speed is a function of your machine, not the model file. The same weights on a fast machine and a slow one differ by an order of magnitude or more in time-to-answer.
The short version: 16 GB of RAM is the practical floor for a good experience, 8 GB works if you close everything else, and below 8 GB is not worth attempting. An Apple Silicon Mac on macOS 14 or later gets GPU acceleration through Metal and unified memory, which makes it disproportionately good at this workload. A Windows machine with an NVIDIA card and 8 GB or more of VRAM will offload most of the work to the GPU and answer in seconds. A CPU-only machine still works — it just takes tens of seconds instead of single digits.
The full breakdown, including the VRAM arithmetic and why reported free VRAM is frequently a lie on Windows, is in our companion piece on the best local AI hardware for running LLMs. If you are planning something more permanent, see building an off-grid AI server with solar power.
Frequently asked questions
Can I run an LLM offline with no internet at all?
Yes. Once the model weights and runtime are on local storage, inference requires no network access whatsoever. The only step that needs internet is the initial download — which is why a pre-loaded device works from the moment you plug it in, and a DIY setup does not.
Does ChatGPT work offline?
No. ChatGPT, Claude, Gemini, and every other hosted assistant run on remote servers and require an active connection for every message. There is no offline mode. Running AI without internet means running a different, locally-executed model.
How much RAM do I need to run an LLM offline?
Budget the quantized weight file size, plus 1 to 2 GB for the KV cache at a 4096-token context, plus about 4 GB for your operating system. In practice: 8 GB of RAM runs a 3B to 4B model, 16 GB runs an 8B comfortably, and 32 GB opens up 14B and 30B-class mixture-of-experts models.
What is GGUF and why does everyone use it?
GGUF is the single-file model format used by llama.cpp and the tools built on it. It packages weights, metadata, and tokenizer into one portable file that runs on CPU, NVIDIA GPUs, AMD GPUs, and Apple Metal without recompiling anything. That portability is why it dominates local AI.
Is a local AI chat actually private?
If the model runs locally and nothing phones home, yes — your prompts never leave the machine. Verify rather than assume: run the tool in airplane mode, and watch outbound connections during a session. Some "local" front-ends still send telemetry.
Can an offline LLM be trusted for medical or safety questions?
A general-purpose local model, no. It will produce fluent, confident, unsourced text and give you no signal about whether it is right. For safety-critical use you need a system that retrieves from vetted sources, cites every claim, and refuses when the grounding is insufficient. That is a different architecture, not a different prompt.
How do I know my offline AI files have not been tampered with?
Hash them. A SHA-256 checksum compared against a published value proves the file is byte-identical to what was published. Stronger still is a cryptographic signature over the manifest, verified at startup, so the system refuses to run in a modified state rather than trusting you to check manually.
What is the fastest way to get a working offline AI LLM?
A pre-built device. Every step above — model selection, quantization, memory sizing, download, hash verification, runtime configuration — is done in advance, signed, and verified at boot. You plug it in and ask a question.
Getting started
If you want to learn the stack, build it yourself. The seven steps above are the whole job, and the understanding you get from doing it is worth the afternoon.
If you want an offline AI LLM that works the day it arrives, cites every claim to a real document, refuses rather than guesses, and verifies its own integrity before it will run, that is what we built. Off-Grid AI Core ships as a bootable USB for macOS, Windows, and Linux, with domain expansion packs for medical, veterinary, electrical, mechanical, tactical, and agricultural work.
Either way, do the airplane-mode test before you rely on it. Knowing what your setup actually does — and how fast it does it on your specific machine — is the part that matters when the connection is gone.
