Local models are finally worth running

Running a language model on your own hardware used to be a demonstration rather than a decision. The models that fit on consumer machines were noticeably worse, the tooling was unpleasant, and the honest recommendation was to use an API and get on with your work.

Two things changed. Small models got substantially better at the narrow tasks most work actually consists of, and unified-memory machines made a large amount of fast memory available at a price that is not absurd.

Key takeaways

• —Memory bandwidth, not raw compute, is what determines whether a model runs usably.
• —Unified-memory machines are cheap in memory and slow in compute — which suits inference and not training.
• —Local wins on privacy, latency and fixed cost; it still loses on frontier capability.

Bandwidth is the specification that matters

The intuition most people bring from gaming — that a fast GPU is what matters — is wrong for inference, and it is why so many local setups disappoint.

Generating a token requires reading the model's weights out of memory. Every token. A 20GB model at 10 tokens per second is moving 200GB per second, and that number is a memory bandwidth figure, not a compute one. Compute sits mostly idle waiting for weights to arrive.

This explains the two things that confuse people:

• A machine with a modest GPU and very fast unified memory outperforms a faster GPU with less memory, once the model no longer fits in the smaller card. The moment any part of a model spills to system RAM, throughput collapses by an order of magnitude.
• Adding a second GPU rarely helps unless it lets the model fit entirely in VRAM. It is the fitting that matters, not the doubling.

The practical rule: find the quantized size of the model you want to run, add roughly 25% for context, and buy at least that much fast memory. Everything else is secondary.

What actually runs well locally

The honest split, after a year of these being usable:

Genuinely good locally. Classification, extraction, summarization of documents you already have, rewriting, code completion within a file, and anything you would otherwise send a frontier model that does not need frontier reasoning. This is most of the traffic in most production systems — a point worth reading alongside the case for routing to smaller models.

Workable with patience. Longer-form drafting, multi-file code work, and agentic loops. They run; they are slower, and the quality gap is real without being disqualifying.

Not there. Hard reasoning, long-context synthesis across many documents, and anything where a subtle error is expensive. Frontier models remain meaningfully better and it is not close.

The three sensible configurations

A Mac with unified memory. The Mac mini M4 is the least expensive way into serious local inference, provided it is specified with enough memory — the base configuration is not the machine you want, and memory cannot be added later. Unified memory means the GPU addresses all of it, so a 32GB machine runs models a 16GB discrete GPU cannot touch. Bandwidth is good rather than exceptional, which caps token rate but is entirely adequate.

A desktop with a discrete GPU. Higher bandwidth, therefore faster tokens, and the only route if you intend to fine-tune rather than only run inference. The constraint is VRAM: consumer cards top out well below what unified-memory machines offer, and the price per gigabyte is far worse.

A machine you already own. Worth trying before buying anything. Any recent laptop runs a small model at a usable rate, and an afternoon establishes whether local inference fits how you actually work — which is the question the hardware cannot answer.

The reasons that are not about performance

Nobody chooses local inference for raw capability. The arguments that hold up are different ones.

Data that cannot leave. For legal or contractual reasons, some material cannot be sent to a third party. This is a hard constraint rather than a preference, and it is the most common genuine reason to run locally.

Latency. No network round trip. For interactive work — completion while typing, live transcription — the difference is felt rather than measured.

Fixed cost. A machine is capital; an API is a per-token bill that scales with usage. For steady heavy use the crossover arrives within a year. For bursty use it never does, and the API is simply cheaper.

It keeps working. No rate limit, no deprecated model, no pricing change, no outage.

What to check before you buy

• The quantized size of the specific model you intend to run, plus about 25%.
• Memory bandwidth, in GB/s. This predicts token rate better than any other single number.
• Whether memory is upgradeable. On unified-memory machines it is not, so the purchase decision is permanent.
• Whether you need to train or only to run. Training changes the answer entirely toward discrete GPUs.
• Whether your actual workload is one of the things that runs well locally. Test on hardware you already own first.