Unusually for this blog, we have almost nothing to sell you here. We are Geonode, we sell proxies, and running a language model on your own machine requires precisely none of that. No proxies, no bandwidth, no account. The adjacency is one step removed: local models are frequently deployed with retrieval over your own data, and if that data comes from the public web, someone has to collect it. That is our end of the pipeline and it is genuinely separate from the model. So read this as a guide written by people with no stake in which model you pick, which is a rarer position in this category than it should be.
What "Local" Buys You and What It Costs
Worth being concrete, because both sides of this are routinely overstated.
What you gain. Data never leaves your machine, which for regulated work is not a preference but a requirement. No per-token cost, so heavy or experimental usage is free after hardware. No rate limits and no dependency on somebody else's uptime. Complete version stability — the model does not change under you, which matters enormously if you have tuned prompts against its behaviour. And the ability to fine-tune on your own data without shipping it anywhere.
What you pay. Capability, mostly. The best local models in 2026 are genuinely good and are still behind the frontier hosted models on hard reasoning. Hardware, which is a real capital cost. Speed, unless you have bought serious hardware. Your own time on setup, quantisation choices and integration. And electricity, which is not nothing for a machine under sustained load.
The framing that leads people astray is treating this as a cost comparison. If you send a few hundred thousand tokens a month to a hosted API, local models will not save you money — the hardware costs more than years of that usage. Local wins on privacy, control and stability, or on very high volume. If none of those apply to you, the economics do not either.
The Hardware Question Decides Everything Else
Before looking at any model, work out your VRAM. Everything else follows from it.
| Available memory | What runs comfortably | Realistic expectation |
|---|---|---|
| 8 GB | 4B–8B models, quantised | Good for summarisation, extraction, simple chat |
| 12–16 GB | 12B–14B quantised, MoE models with small active sets | Solid general assistant, decent coding help |
| 24 GB | 30B-class MoE, 32B dense quantised | Genuinely capable across most everyday tasks |
| 48 GB | 70B quantised | Close to hosted mid-tier quality |
| 80 GB | 120B-class MoE at native quantisation | The top of what a single card does |
Two things change this arithmetic in your favour.
Mixture-of-experts architectures. These have a large total parameter count but activate only a fraction per token. gpt-oss-120b has 117 billion total parameters and 5.1 billion active; OpenAI's model card states it fits "into a single 80GB GPU (like NVIDIA H100 or AMD MI300X)" using MXFP4 quantisation of the expert weights. gpt-oss-20b is 21 billion total, 3.6 billion active, and runs "within 16GB of memory". You get the quality benefits of a larger model at the memory and speed cost of a much smaller one.
Apple Silicon unified memory. On a Mac, system memory is GPU memory. A machine with 64 GB of unified memory runs models that would require a card most people do not own. Throughput is lower than a discrete GPU of equivalent capacity, but the capacity ceiling is far higher for the money.
Also budget for context. The model weights are not the whole story — the KV cache grows with context length and can consume several gigabytes on long inputs. A model that fits at 4K context may not fit at 128K.
The Models Worth Running in 2026
Qwen3 is the most useful family for local deployment, primarily because it spans the whole size range with consistent behaviour. The Hugging Face collection lists 0.6B, 1.7B, 4B, 8B, 14B, 32B dense models plus the 30B-A3B and 235B-A22B mixture-of-experts variants, with quantised builds in FP8, GGUF, AWQ, GPTQ and MLX formats.
The Qwen3-8B model card documents the features that matter: Apache 2.0 licence, 32,768 tokens of native context extending to 131,072 with YaRN scaling, and "support of 100+ languages and dialects". Its distinguishing feature is a switch between thinking mode for reasoning-heavy work and non-thinking mode for efficient dialogue, in one model.
One practical detail from that card that people miss: sampling settings matter, and the recommended values differ by mode — temperature 0.6, top-p 0.95, top-k 20 for thinking; temperature 0.7, top-p 0.8, top-k 20 otherwise. Greedy decoding is explicitly discouraged in thinking mode. If a model is performing worse than you expected, check your sampling parameters before blaming the model.
Gemma 4 from Google is the notable release for anyone who was previously put off by Gemma's licensing, because it is now Apache 2.0. The model card lists five sizes — E2B, E4B, 12B, 26B A4B and 31B — with "a 128K context window, while the medium models support 256K", multilingual support in "over 140 languages", and text and image input with audio supported on the E2B, E4B and 12B models. Native audio input in a small open-weight model is unusual and worth knowing about if that is your use case.
gpt-oss from OpenAI covers the two ends. The 20B fits a 16 GB machine; the 120B fits a single 80 GB card. Both are Apache 2.0, described on the model cards as a "Permissive Apache 2.0 license: Build freely without copyleft restrictions or patent risk". The 20B is positioned for "lower latency, and local or specialized use cases", with agentic work, function calling and code execution among the stated applications.
DeepSeek-R1 remains the reference for open reasoning models and is MIT licensed, which the model card notes "support commercial use, allow for any modifications and derivative works". For local use the distilled versions matter more than the full model: Qwen-based distills at 1.5B, 7B, 14B and 32B, and Llama-based at 8B and 70B, all fine-tuned on "800k samples curated with DeepSeek-R1". The 14B and 32B distills are the practical picks for consumer hardware.
Llama remains the most-downloaded family by a wide margin — Ollama's library shows llama3.1 at 119.1M pulls and llama3.2 at 82.1M, ahead of deepseek-r1 at 92.2M and gemma3 at 40M. Popularity means the best tooling, the most community fine-tunes and the most troubleshooting material, which is a genuine advantage independent of raw capability.
And do not overlook embedding models. nomic-embed-text sits third in Ollama's library at 84.2M pulls, which tells you how much local LLM usage is actually retrieval over private documents rather than chat.
Licences Matter More Than Benchmarks
This is the section that saves people from an expensive mistake, and it is routinely omitted from model comparisons.
"Open weights" is not one thing. The models above fall into three groups:
Apache 2.0 — Qwen3, Gemma 4, gpt-oss, the Qwen-based DeepSeek distills. Permissive, commercially usable, no field-of-use restrictions, patent grant included. If you are shipping a product, this is what you want.
MIT — DeepSeek-R1 itself. Equally permissive, explicitly supporting commercial use, modification and derivative works.
Custom community licences — the Llama family, and by inheritance the Llama-based DeepSeek distills, which carry the Llama 3.1 and Llama 3.3 licences respectively. These permit a great deal but are not Apache or MIT: they carry acceptable-use policies, attribution requirements, and conditions that activate at large user counts. Usually fine. Not automatically fine, and worth reading before you build a product on one.
The Gemma 4 move to Apache 2.0 is significant precisely because Gemma previously used a custom licence. If you evaluated the family before and ruled it out on licensing grounds, that reason no longer applies.
Two practical points. First, check the licence of the specific model you are pulling, not the family — the DeepSeek distills are the clearest example, where different distills of the same release carry different licences depending on their base model. Second, if you are fine-tuning, read what the licence says about derivative works and about naming, because some require the derivative to carry the original name.
The Tools: Ollama, llama.cpp, LM Studio, vLLM
| Tool | Best for | Interface | Trade-off |
|---|---|---|---|
| Ollama | Getting started, local development | CLI + HTTP API | Less control over inference details |
| llama.cpp | Maximum control, unusual hardware | CLI + server | You configure everything yourself |
| LM Studio | Non-technical users, experimentation | GUI | Less suited to automation |
| vLLM | Serving multiple users | HTTP API | Needs proper GPU hardware |
Ollama is the right default for most people. One command to pull a model, an OpenAI-compatible HTTP endpoint, sensible quantisation defaults. The ceiling is that when you want to tune inference parameters precisely you will eventually reach past it.
llama.cpp is what Ollama is built on, and using it directly gives you full control over quantisation, context handling, GPU layer offloading and CPU threading. It is also the best-supported path for unusual hardware — older cards, CPU-only, Apple Silicon.
LM Studio is a desktop application with model discovery and a chat interface. If the person using the model is not a developer, this is the answer.
vLLM is a serving system rather than a local tool, built for throughput with continuous batching. If you are running a model for a team rather than for yourself, this is the tier you move to, and it expects real GPU hardware.
A useful pattern: develop against Ollama's OpenAI-compatible endpoint, and if you later need to serve properly, move to vLLM behind the same interface. Your application code does not change.
Quantisation Without the Hand-Waving
Quantisation reduces the numerical precision of the weights so the model needs less memory. It is how a model that nominally needs 60 GB runs on a 24 GB card.
| Format | Size vs FP16 | Quality | Use when |
|---|---|---|---|
| FP16/BF16 | 100% | Reference | You have memory to spare |
| Q8 | ~50% | Near-indistinguishable | You have room and want maximum quality |
| Q5_K_M | ~35% | Very good | A sensible balance |
| Q4_K_M | ~28% | Good, mild degradation | The common default |
| Q3 and below | ~20% | Noticeable degradation | Only when nothing else fits |
The rule that holds up in practice: a larger model at heavier quantisation usually beats a smaller model at lighter quantisation. A 32B model at Q4 will typically outperform a 14B at Q8 in the same memory budget. The exception is at the extreme end — below Q3, degradation becomes severe enough that the trade reverses.
Note also that MXFP4, used for the gpt-oss expert weights, is a case where quantisation is part of the model's design rather than something applied afterwards. That is why the memory figures on those model cards are as low as they are.
Test on your own workload rather than trusting a table, including this one. Quantisation degrades different capabilities unevenly, and a level that is invisible for summarisation may be obvious for code generation.
Realistic Expectations Versus the Frontier
Setting these correctly avoids most disappointment.
Where local models are genuinely competitive: summarisation, extraction, classification, translation, straightforward code completion, drafting, retrieval-augmented question answering over your own documents. For all of these, a well-chosen 14B–32B model is fine, and the difference against a frontier hosted model is small enough that you would struggle to notice it.
Where the gap is still real: long multi-step reasoning, complex agentic work, large codebases requiring genuine understanding, tasks needing broad and current world knowledge. The gap has narrowed considerably. It has not closed.
Where local models win outright: anything where the data cannot leave your infrastructure, and anything at high enough volume that per-token pricing dominates. These are not capability arguments and they are frequently the decisive ones.
One more expectation to calibrate: speed. On consumer hardware a 32B model produces tokens at a rate that is fine for a chat interface and slow for anything batch. If you are processing ten thousand documents, measure throughput before committing to an architecture.
When You Should Just Use an API
The section that argues against the premise.
When your volume is low. A few hundred thousand tokens a month costs very little on a hosted API and will never justify a GPU purchase. Buying hardware to avoid a small bill is a poor trade unless the hardware has other uses.
When you need frontier capability. If the task genuinely requires the strongest available reasoning, local models are not there yet and pretending otherwise wastes weeks.
When you do not have the hardware. Running a 7B model on an 8 GB card because that is what you have, and then concluding local models are not good, is a common and avoidable disappointment. The model you can run is not the model you read about.
When maintenance is a cost you cannot absorb. Models update, tooling changes, quantisation formats evolve. A hosted API is somebody else's operational problem.
When you are prototyping. Build against an API, establish that the product works, then evaluate whether moving local is worth it. The reverse order means solving hardware problems before you know whether the idea is any good.
And the case with no ambiguity: if your constraint is that data must not leave your premises, none of the above applies. Local is not a preference in that situation, and the question is only which model fits your hardware.
People Also Ask
What is the best local LLM in 2026?
There is no single answer, but Qwen3 is the most useful family for local deployment because it spans 0.6B to 235B with consistent behaviour and an Apache 2.0 licence. Match the size to your VRAM: 8B for 8–16 GB, the 30B-A3B mixture-of-experts model for 24 GB, gpt-oss-120b if you have an 80 GB card.
How much VRAM do I need to run a local LLM?
8 GB runs useful 4B–8B models quantised. 16 GB covers 12B–14B comfortably, or gpt-oss-20b which is documented to run within 16 GB. 24 GB opens up 30B-class models. Mixture-of-experts architectures shift this in your favour because only a fraction of parameters activate per token.
Are local LLMs as good as ChatGPT or Claude?
Not on the hardest reasoning tasks, no. For summarisation, extraction, classification, translation and retrieval over your own documents, a good 14B–32B local model is close enough that the difference rarely matters. The gap is narrowing but has not closed.
Which local LLMs can I use commercially?
Qwen3, Gemma 4 and gpt-oss are Apache 2.0. DeepSeek-R1 is MIT. All permit commercial use without field-of-use restrictions. The Llama family uses custom community licences that permit a great deal but carry conditions worth reading. Check the specific model rather than the family — DeepSeek's Qwen-based distills are Apache 2.0 while its Llama-based distills carry Llama licences.
What is the easiest way to run an LLM locally?
Ollama for developers — one command to pull a model, and an OpenAI-compatible HTTP endpoint. LM Studio if you want a graphical interface and no command line. Both handle quantisation and hardware detection for you.
Does quantisation hurt quality?
Somewhat, and unevenly. Q8 is near-indistinguishable from full precision. Q4_K_M is the common default with mild degradation most people do not notice. Below Q3 it becomes obvious. As a rule, a larger model at Q4 beats a smaller model at Q8 in the same memory budget.
Can I run a local LLM on a Mac?
Yes, and often better than on a comparably priced PC, because Apple Silicon's unified memory is available to the GPU. A 64 GB Mac runs models that would need a card most people do not own. Throughput is lower than a discrete GPU, but the capacity ceiling is much higher per pound.
Do I need proxies or special networking for local LLMs?
No. The model runs on your machine and makes no network requests. Networking only enters the picture if you are collecting web data to retrieve over, which is a separate part of the pipeline entirely.
Wrapping Up
Pick by memory first, licence second, benchmark third. That ordering is the opposite of how most comparisons are written and it is the one that produces a working system.
Your VRAM determines which models are even candidates, and mixture-of-experts architectures have made that constraint considerably less binding — 117 billion parameters on a single 80 GB card, or 21 billion within 16 GB, were not realistic propositions long ago. Licence determines whether you can ship what you build, and the shift of Gemma 4 to Apache 2.0 means the permissive tier now covers most of the strong options. Benchmarks come last because within a size class the differences are small and your specific workload will disagree with the leaderboard anyway.
The honest summary of where this stands: for privacy-constrained work, for high volume, and for anything where you need the model to stop changing under you, local is now a straightforwardly good option rather than a compromise. For occasional use at frontier capability, it still is not, and a hosted API remains the sensible answer.