Running an LLM locally means downloading an open model and running it entirely on your own computer—no API, no cloud, no per-token cost—so your data never leaves your machine and you can work offline. With tools like Ollama, getting a capable model running takes a single command. This guide covers why you'd run a model locally, the tools that make it easy, the hardware that determines what you can actually run, quantization, and the honest tradeoffs against cloud APIs.
Why run an LLM locally
Running models on your own machine solves several problems that cloud APIs can't:
- Privacy. Your data never leaves your computer—no prompts, no documents, nothing sent to a third party. For sensitive or regulated work, this alone justifies local models.
- Offline access. No internet required, so a local model works on a plane, in a secure facility, or anywhere connectivity is unreliable.
- No cost or rate limits. After the hardware you already own, there's no per-token bill and no throttling. You can experiment endlessly, which is invaluable for learning and prototyping.
- Control and learning. You can run many different open models, use fine-tuned or specialized ones, and see how they behave up close.
The honest boundary: local models are ideal for a single developer or user—experimentation, private tasks, offline work, prototyping. They're not how you serve an application to thousands of users; that's a different discipline covered in deploying LLMs in production, where you serve many concurrent requests on dedicated hardware. Local is your personal workshop, not your production factory.
The tools that make it easy
Running models locally used to require serious effort. Today, a few tools make it nearly trivial.
Ollama is the most popular and the simplest starting point. It handles downloading, managing, and serving models, and it exposes a local API for your code. Getting started is genuinely one command:
- Install Ollama from ollama.com (macOS, Windows, and Linux are all supported).
- Pull and run a model with
ollama run llama3—it downloads the model and drops you straight into a chat. - Use it from code via Ollama's local HTTP API (at
localhost:11434), so you can build applications against the model running on your own machine. - Swap models freely with
ollama run mistral,ollama run phi3, and so on.
LM Studio is the best option if you prefer a graphical app over the terminal—it gives you a model browser, a chat interface, and a local API server in a friendly UI. Under the hood, both tools rely on llama.cpp, the highly optimized C++ inference engine that made local LLMs practical on ordinary hardware; you can use it directly for maximum control, though most people don't need to. Models themselves come from Hugging Face, the main hub for open models, typically in the GGUF format that llama.cpp and Ollama use. Other tools worth knowing include Jan and GPT4All. For nearly everyone, though, the right answer is: start with Ollama.
The hardware reality: memory is everything
The single factor that decides what you can run locally is memory—specifically, whether the model fits in your GPU's VRAM (video memory) or, on some systems, in regular RAM. If a model doesn't fit, it either won't run or crawls unusably slowly.
A useful rule of thumb: at 4-bit quantization (explained below), a model needs roughly 0.5–0.6 GB of memory per billion parameters, plus some overhead. So an 8-billion-parameter model needs about 5–6 GB, while a 70-billion-parameter model needs around 40 GB. That maps to hardware like this:
| Model size (4-bit) | Approx. memory needed | Runs comfortably on |
|---|---|---|
| ~3B | ~2 GB | Almost anything, even a modest laptop |
| ~8B | ~5–6 GB | Most modern laptops and GPUs (8GB+), Apple Silicon |
| ~13B | ~8–10 GB | A mid-range GPU or a 16GB+ Mac |
| ~34B | ~20 GB | A 24GB GPU (RTX 3090/4090) or a 32GB+ Mac |
| ~70B | ~40 GB | A high-end or multi-GPU setup, or a 64GB+ Mac |
A few hardware notes matter. A dedicated GPU is the fast path—an RTX 4090 or 3090 (24GB) runs 8B models effortlessly at roughly 50–150 tokens per second and handles larger ones with heavier quantization. Apple Silicon has a real advantage: its unified memory lets the GPU use system RAM, so a Mac with 32–64GB can run bigger models than most standalone PC GPUs. Running on CPU alone works but is slow. Before downloading anything, check your available memory and pick a model that fits.
Quantization and choosing a model
Quantization is the technique that makes local LLMs practical. It compresses a model's weights from higher precision (like 16-bit) to lower precision (commonly 4-bit), which dramatically shrinks the memory the model needs and speeds it up, at the cost of a modest quality reduction. It's the reason an 8-billion-parameter model that would need ~16GB at full precision fits comfortably in ~5–6GB.
The usual sweet spot is 4-bit quantization—large savings for small quality loss. In the GGUF format, you'll see named levels like Q4_K_M (a popular balance); lower levels like 2-bit save more memory but degrade quality more noticeably. Tools like Ollama pick sensible defaults, so you rarely need to think hard about this, but knowing it exists explains how big models fit small machines.
On which model to run, two principles apply. First, smaller often wins locally: a compact model runs faster, fits more hardware, and frequently handles a specific task as well as a big one—the tradeoffs between small and large models are worth understanding, and locally the case for small is even stronger. Match the model to your actual need rather than grabbing the biggest one that fits, following the same logic as choosing an LLM for your use case. Second, set your quality expectations honestly: capable as open local models are, they generally trail frontier cloud APIs like the leading GPT and Claude models. Test candidates on your own task using real evaluation metrics rather than assuming a model is good enough because it runs.
Local vs cloud: the honest tradeoffs
Deciding between local and cloud comes down to what you're optimizing for.
Local wins on privacy (data stays on your machine), offline capability, zero per-token cost, no rate limits, and full control. Cloud APIs win on raw capability (access to the largest, most powerful models), zero hardware requirements, effortless scaling to many users, and speed from data-center GPUs. The realistic summary: local models are excellent for private, offline, cost-free development and for tasks a smaller model handles well, while cloud APIs remain the choice when you need frontier-level quality or must serve real scale.
A cost nuance is worth naming: "local" isn't truly free—you've paid for the hardware and you pay for electricity—but the marginal cost per request is effectively zero, which is powerful for heavy experimentation. When you move from personal use toward serving an application, the path forks again: you either adopt a managed API or graduate to running open models on your own infrastructure, the domain of self-hosting AI models, where controlling spend becomes its own discipline of inference cost optimization. Local development is often the first step on that road—prove it works on your machine, then decide how to serve it.
Common mistakes to avoid
- Choosing a model too big for your hardware. If it doesn't fit in memory, it won't run or will be painfully slow. Check your available memory against the model size first.
- Ignoring quantization. Running a model at full precision when a 4-bit version would fit and barely lose quality wastes memory and speed. Use quantized models.
- Expecting frontier quality from a small local model. Local open models are impressive but generally trail the best cloud APIs. Calibrate expectations to the model's size.
- Using local for high-scale serving. A laptop serving one user is not a production system. For many concurrent users, use proper serving infrastructure.
- Fighting raw llama.cpp as a beginner. You don't need to compile and configure the engine by hand. Start with Ollama or LM Studio.
- Grabbing the biggest model that fits. A smaller model is often faster and just as good for your task. Right-size rather than maximize.
- Forgetting local isn't cost-free. Hardware and electricity are real costs; local just shifts them off the per-token meter.
Frequently asked questions
What does running an LLM locally mean? It means downloading an open model and running it entirely on your own computer, rather than calling a cloud API. The model's computation happens on your CPU or GPU, so no data leaves your machine, there's no per-token cost, and it works offline. Tools like Ollama and LM Studio make this straightforward, letting you run capable open models on ordinary consumer hardware.
What hardware do I need to run LLMs locally? Mainly enough memory to fit the model. A small 8-billion-parameter model needs roughly 5–6 GB at 4-bit quantization, running on most modern laptops or an 8GB+ GPU, while a 70B model needs around 40 GB and serious hardware. A dedicated GPU is fastest, Apple Silicon's unified memory handles larger models well, and CPU-only works but is slow. Check available memory before choosing a model.
What's the easiest way to run an LLM locally?
Ollama is the simplest. Install it, then run a single command like ollama run llama3, which downloads the model and starts a chat—and it also exposes a local API for your code. If you prefer a graphical interface over the terminal, LM Studio offers a friendly app with a model browser and chat. Both handle the complexity for you.
Are local LLMs as good as ChatGPT or Claude? Generally not at the frontier. Open models you can run locally are capable and improving fast, but the largest cloud models still lead on raw quality and reasoning. For many specific tasks, a good local model is more than sufficient, and its privacy and cost advantages can outweigh the quality gap. Evaluate on your actual task rather than assuming either way.
Is running LLMs locally free? There's no per-token or subscription cost, which makes heavy experimentation effectively free at the margin. But it isn't truly free—you've invested in the hardware capable of running the model, and it draws electricity while working. Compared to cloud APIs, local shifts cost from an ongoing usage bill to upfront hardware, which pays off for frequent use and privacy-sensitive work.
The takeaway
Running LLMs locally gives you private, offline, cost-free access to capable open models on hardware you already own—and with Ollama or LM Studio, getting started is genuinely a one-command affair. The key constraints to internalize are that memory determines what you can run, quantization is what makes large models fit small machines, and local open models trade some frontier quality for privacy and control. Your next step is to install Ollama and run an 8B model like Llama 3 today—it fits almost any modern machine—because a few minutes of watching a model respond entirely on your own hardware is the fastest way to understand what local LLMs can and can't do.