# How to Run an LLM Locally (2026 Guide) > Run a large language model on your own Mac or PC. A practical guide to hardware, tools, quantization, setup, using it for coding, and whether it's worth it. - Published: 2026-07-22 - Author: Samy BEN SADOK - Canonical: https://geotoolbox.ai/blog/run-llm-locally --- Running an LLM locally used to mean a weekend of compiling code and reading GPU forums. In 2026 it takes one command and about fifteen minutes, and it works on the laptop you already own. This guide covers the parts that actually trip people up: whether your hardware can handle it, which tool to install, what those cryptic model file names mean, how to wire a local model into your code editor, and the honest answer to whether any of it is worth the effort. It is for developers and marketers who want a private, offline model of their own, without a per-token bill. You do not need a background in machine learning, just a willingness to run a command or so. ## What Running an LLM Locally Actually Means A local LLM is an open-weight language model that runs entirely on your own computer instead of calling a hosted service like ChatGPT or Claude. You download the model file once, and from then on every prompt is answered by your own hardware. Nothing is sent to a vendor, nothing is logged on someone else's server, and it keeps working with the Wi-Fi off. That is the whole appeal, and also the whole catch. A model running on your laptop is not a free copy of ChatGPT. It is a different tool that happens to share the same interface. It excels at private, repetitive, and offline work, and it is genuinely useful for learning [how these models actually work](https://geotoolbox.ai/blog/how-ai-models-think). It will not match a frontier cloud model on the hardest reasoning or on very long documents. Set that expectation early and everything that follows makes sense. Get it wrong, and you will spend an evening fighting your GPU only to conclude that local models are "dumb." They are not dumb. They are smaller, and you are the one deciding how small. You have a few decisions to make: what your machine can handle, which tool to install, and which model file to download. The rest of this guide walks through them in order. ## Can Your Computer Run One? Memory is the gate. Before download size or GPU speed, the question is whether the model fits, because a model that fits in fast memory runs many times quicker than one that spills over into slower memory. Three numbers matter, and only one applies to you: - **VRAM** is the memory on a discrete graphics card (an NVIDIA or AMD GPU). On a Windows or Linux desktop, this is the number that counts. - **System RAM** is your main memory. A model can run here on the CPU alone, but slowly. - **Unified memory** is Apple Silicon's trick: on an M-series Mac, the CPU and GPU share one pool, so a Mac with plenty of unified memory can load models that would need an expensive discrete GPU on a PC. This is why a mid-range Mac is often the smoothest place to start. A rough rule at the common Q4_K_M quantization (explained below): budget roughly half a gigabyte for the weights of each billion parameters, then add headroom for the runtime and your conversation. A 7-billion-parameter model needs roughly 5 GB all in, so an 8 GB card handles it with room to spare; an 8B model fits too, with less headroom for long chats.
Your memoryLargest model (Q4_K_M)What to expect
8 GB RAM or VRAM7B to 8BRoughly 30 to 50 tokens/sec on a GPU; single digits, and slow, on CPU only
16 GB RAM or VRAM13B to 14BThe everyday sweet spot; roughly 20 to 40 tokens/sec on a GPU
24 GB VRAM (RTX 3090/4090)up to 32BNear the best quality you can run locally; roughly 15 to 30 tokens/sec
48 GB+ (2x24GB, or a large-memory Mac)up to 70BHighest local quality; slower, roughly 8 to 15 tokens/sec, and worth it for the size
Those speeds are rough and assume the model runs on a GPU or Apple Silicon; the exact number depends on your specific chip and the model. Interactive chat is comfortable once you are into the double digits of tokens per second, and the largest models trade speed for quality by design. One counterintuitive point: capacity beats raw speed. An older 24 GB RTX 3090 will out-run a newer 16 GB RTX 4080 on a large model, because the 3090 keeps the whole model in VRAM while the 4080 has to offload part of it to system RAM and pays the speed penalty. No GPU at all? You can still run small models on the CPU, but expect roughly 3 to 8 tokens per second on a 7B model, which is fine for a background task and painful for a live chat. ## Pick Your Tool Half a dozen apps all claim to "run local LLMs," which is where most people stall. Sort them by the job you actually have, not by feature lists. The real split is command line versus graphical interface, and beginner versus builder.
ToolInterfaceBest forWorth knowing
OllamaCommand line + APIDevelopers who want a model behind an APIOne command to install and run; serves an OpenAI-compatible endpoint by default
LM StudioDesktop GUIBeginners who want to point and clickSearches and downloads models in-app; no terminal needed
JanDesktop GUIPrivacy-first usersOpen source, fully offline, a clean ChatGPT-style window
GPT4AllDesktop GUIAbsolute beginners on modest hardwareRuns well CPU-only; chatting in a couple of minutes
llama.cppCommand line / engineTinkerers who want full controlThe engine most other tools are built on, including Ollama
A couple more are worth naming even though they are not starting points. [Open WebUI](https://docs.openwebui.com/) puts a ChatGPT-style browser interface on top of Ollama when you outgrow the terminal. [vLLM](https://github.com/vllm-project/vllm) is a serving engine for when a model needs to handle many users at once: Ollama is tuned for one person on one machine, while vLLM is built to batch many simultaneous requests, so a busy shared server is a different problem from running a model on your own desk. If you are not sure, the short version: install [Ollama](https://docs.ollama.com/quickstart) if you are comfortable in a terminal or want to build on it, and [LM Studio](https://lmstudio.ai/) if you would rather never see one. Both are free, both run the same underlying models, and you can switch later without losing anything. If leaving no trace is the whole point, Jan and GPT4All lean hardest into privacy and are built to run fully offline by default. ## Understand the Model Files (Quantization and GGUF) Go to download a model and you hit a wall of cryptic file names: the same model in versions labeled Q2_K, Q4_K_M, Q5_K_M, Q8_0. This is the step that stops people cold, and it is simpler than it looks. Those labels are **quantization** levels. A model's weights are originally stored at high precision (16 bits each). Quantization rounds them to fewer bits to shrink the file and speed up inference. Four-bit quantization cuts the memory footprint by roughly 75 percent while losing only a small amount of quality, usually in the low single digits of a percent. That trade is what makes an 8-billion-parameter model fit on a laptop at all. **GGUF** is the standard file format these quantized models come in, used by Ollama, LM Studio, llama.cpp, and Jan. When you grab a model from [Hugging Face](https://huggingface.co/), you are looking for its GGUF version. The default you want, in almost every case, is **Q4_K_M**. It is the accepted sweet spot: small enough to fit, good enough that you will not notice the compression. Step up to Q5_K_M or Q6_K if you have memory to spare and want a little more quality. Avoid Q2 and Q3 unless you are desperate to squeeze a model onto too little memory, because that is where output visibly degrades. Which model, though? That is a separate decision with its own moving parts, and the open-weight landscape shifts monthly. Rather than chase version numbers, start with a well-known family and a size your hardware can hold: Meta's Llama, Alibaba's Qwen (including its Qwen Coder variants), Google's Gemma, Mistral, or DeepSeek. For a current shortlist by use case, see our guide to the [best open-source LLMs](https://geotoolbox.ai/blog/best-open-source-llms), and note that several of the strongest options are [open-weight families like Qwen and DeepSeek](https://geotoolbox.ai/blog/chinese-ai-models-compared) you can run entirely offline. The rule that ages well: pick a family, pick a size that fits, download the Q4_K_M file. ## Run Your First Model in 15 Minutes With Ollama, the whole process is short, and most of the fifteen minutes is the model downloading in the background. 1. **Install Ollama.** Download it from [ollama.com](https://ollama.com/download) and run the installer. It sets up a background service and adds the `ollama` command to your terminal. 2. **Pull and run a model.** In a terminal, type `ollama run llama3.2`. Ollama downloads the model on the first run, then drops you into an interactive chat right there in the terminal. Swap `llama3.2` for any model in its [library](https://ollama.com/library); a 3B model is a safe first pull on modest hardware. 3. **Chat.** Type a message and press enter. That is a large language model answering from your own machine, no account and no internet required.
![A local LLM data flow: your prompt goes to a runner, then the model in your computer's memory, and the answer returns, with nothing sent to the cloud.](/blog/run-llm-locally/local-llm-data-flow.png)
With a local model, the whole loop stays on your machine. Nothing is sent to a cloud provider.
Prefer a familiar window over a terminal? Point [Open WebUI](https://docs.openwebui.com/) at Ollama and you get a browser-based, ChatGPT-style interface with chat history and document upload for asking questions over your own files, all still running locally. LM Studio users get the same thing built in. First-run downloads are large. A small model is a few gigabytes, a 30B model can be twenty or more, and a 70B file alone can top 40 GB. Keep several gigabytes free for a small model and considerably more for the bigger tiers, and use an SSD, not a spinning drive. Run `ollama list` to see what you have pulled and `ollama rm ` to clear the ones you are done with. ## Use Your Local Model for Coding This is where a local model earns its keep, and it is the part most guides skip. The key is that Ollama and LM Studio expose an **OpenAI-compatible API** on your machine. Anything that can talk to OpenAI can talk to your local model instead by pointing at a different address. Ollama serves that endpoint at `http://localhost:11434/v1`. Because the request never leaves your computer, the API key is not checked, so any placeholder value works. For **Cursor** or the **Continue** extension in VS Code and JetBrains, the setup is the same idea in both: add a model provider, set the base URL to your local endpoint, and choose a model you have pulled (a Qwen Coder model is the usual pick for this). [Continue](https://www.continue.dev/) has native Ollama support in its config, so it is often the smoothest path to a local coding assistant. For a wider set of options here, we keep a running list of [open-source tools for Claude Code](https://geotoolbox.ai/blog/claude-code-open-source-tools). **Claude Code has one gotcha worth knowing before you waste an hour on it.** Claude Code speaks Anthropic's message format, not OpenAI's, so it does not use the same `/v1` endpoint as Cursor or Continue. Ollama exposes a separate Anthropic-compatible endpoint for exactly this, and the simplest path is to let it connect everything for you with `ollama launch claude`. If you set it up by hand instead, export these variables and add them to your shell profile so they persist: `ANTHROPIC_BASE_URL` pointed at your local Ollama, `ANTHROPIC_AUTH_TOKEN` set to any value, and `ANTHROPIC_API_KEY` set to an empty string. Do not skip that last one: a cloud Anthropic key already sitting in your environment can otherwise override the local setup, which is the exact dead end this section is about. See [Ollama's documented Claude Code integration](https://docs.ollama.com/integrations/claude-code) for the full walkthrough. Be honest with yourself about the result. A local 7B to 32B coding model is not GPT-5 or Claude Opus. It handles autocomplete, boilerplate, and private codebases you cannot paste into a cloud tool, and it costs nothing per token after the hardware. Treat it as a strong second seat for privacy-sensitive or offline work. ## Why Is It Slow? Fixing Common Problems The most common complaint is speed, and the cause is almost never a weak GPU. It is silent offload. When a model does not fully fit in VRAM, the runner quietly moves part of it into system RAM, which is far slower, so generation crawls. You do not need new hardware. Pick a model or quantization that fits. Diagnose it in one command. Run `ollama ps` while a model is loaded and look at the processor column: if it says anything other than 100% GPU, part of the model is on the CPU. From there: - **Pick a smaller model or quant** so the whole thing fits in memory. - **Lower the context length.** A longer context holds more of the conversation but eats memory and slows generation; along with [settings like temperature](https://geotoolbox.ai/blog/ai-temperature), it is one of the main levers you control. - **Keep the model resident** so it is not reloaded on every request. A realistic baseline: interactive chat feels fine above about 15 tokens per second and sluggish below 5. If a model that should fit your memory, say 7B to 32B, is crawling in single digits on a capable GPU, you are almost certainly offloading. The largest 70B-class models are slower by design and are the exception. The other classic failure is **the GPU not being used at all on Windows**, especially inside WSL2. The usual mistake is installing NVIDIA drivers *inside* WSL2, which breaks the passthrough. The host Windows driver already provides GPU access to WSL2, so do not install a second one there. AMD support through ROCm is thinner than NVIDIA's; if you are on AMD or fighting WSL, a Mac with Apple Silicon is the least painful path by a wide margin. Local runners also default to a fairly small context window, which is why a local model can seem to "forget" earlier in a conversation than ChatGPT does. You can raise it, with Ollama's `num_ctx` setting or by choosing a longer-context model, but a bigger context uses more memory and slows generation. It is a trade-off, not a free upgrade. That trade-off has a sharp edge worth watching: the **context cliff.** A model that fits comfortably for a short chat can run out of memory when you paste a long document, because a bigger context grows the memory it needs. This is also why the 32B and 70B ceilings in the hardware table assume short-to-moderate context; a very long prompt eats into the headroom those models need. If a model crashes only on long inputs, that is why. Trim the input or raise your memory headroom. ## Is Running an LLM Locally Worth It? For privacy, offline access, bulk or repetitive work, and learning, yes. As a cheaper drop-in for a frontier model on your toughest tasks, no. The cost case is real but conditional. If you already run a capable GPU or a high-memory Mac, local inference is effectively free after electricity, and for heavy or bulk use it undercuts a per-token API bill quickly. If you would have to buy hardware specifically for this, a one-time GPU in the few-hundred-dollar range competes with tens of dollars a month in [a ChatGPT subscription or the API](https://geotoolbox.ai/blog/chatgpt-pricing) only once your usage is high enough. For a light, occasional user, paying for the cloud is cheaper and less hassle. Buy hardware for the privacy and control, not to save a few dollars. Where local wins outright is the work you cannot or should not send to a vendor: confidential client files, proprietary code, anything under a policy that forbids pasting into cloud tools, and any task you need to run with no internet at all. It is also the best way to build real intuition for how these systems behave. That intuition pays off directly if you work in marketing or SEO. The open-weight models you can now run on a laptop are increasingly the same ones answering questions about brands inside AI search, so running one locally is a cheap way to watch how they read and summarize content. ## Frequently Asked Questions ### How much RAM do you need to run an LLM locally? Around 8 GB runs 7B to 8B models at Q4_K_M quantization, 16 GB is the comfortable everyday range and handles 13B to 14B, and 24 GB or more opens up models in the 32B class. On a discrete GPU the relevant number is VRAM; on an Apple Silicon Mac it is unified memory. A rough guide is about half a gigabyte for the weights of each billion parameters, plus headroom for the runtime and your conversation. ### Do I need a GPU to run an LLM locally? No, but it helps a lot. A modern CPU with enough RAM can run small models at roughly 3 to 8 tokens per second, which is fine for background tasks and slow for live chat. A GPU, or an Apple Silicon Mac's unified memory, is what makes local models feel responsive. ### Is it worth running LLMs locally? It is worth it for privacy, offline use, bulk or repetitive jobs, and learning. It is not worth it as a cheaper replacement for a frontier model on complex reasoning, where a cloud model still wins. If you already own capable hardware, it costs nothing per token; if you would buy hardware just to save money, the math only works at high usage. ### What laptop can run an LLM locally? Any laptop with 16 GB of memory can run useful 7B to 8B models. Apple Silicon MacBooks (M-series) are especially strong because their unified memory lets the GPU use a large shared pool, so a high-memory Mac punches well above a PC laptop with the same total memory. ### Is running an LLM locally actually private? Yes, in the sense that prompts are processed on your machine and are not sent to a model vendor. A couple of caveats: some apps still make network calls for updates or optional web search, and if you expose the local server to your network, other devices could reach it. Keep the server bound to your own machine and disable optional cloud features if privacy is the point. ### Which local model should I start with? Start with a well-known family at a size your memory can hold rather than chasing the newest version number: a small Llama or Qwen model at Q4_K_M is a safe first pull, and a Qwen Coder model is a good default for coding. Once one is running, try a larger size from the same family to feel the quality and speed trade-off. ## Where to Go From Here The fastest way to understand local models is to run one. Install Ollama or LM Studio, pull a small model that fits your memory, and give it a real task tonight. Fifteen minutes later you will know more about what these systems can and cannot do than any benchmark chart will tell you. Once you have seen how a model reads and summarizes text, the natural next question for anyone doing marketing or SEO is whether the AI systems built on these models can actually find and cite your own site. That is the problem geotoolbox is built for. Start by checking whether AI crawlers can reach your pages at all with our free [AI crawler checker](https://geotoolbox.ai/tools/ai-crawler-checker). ## Sources - Ollama Quickstart and Docs - Ollama, 2026 - `docs.ollama.com/quickstart` - Running local models with Claude Code - Ollama, 2026 - `docs.ollama.com/integrations/claude-code` - Hugging Face model hub - Hugging Face - `huggingface.co` - LM Studio - Element Labs - `lmstudio.ai` - Continue (local AI code assistant) - Continue - `continue.dev` - llama.cpp inference engine - ggml-org, GitHub - `github.com/ggml-org/llama.cpp`