Every model you have ever used through an API can also run on hardware you control. The question is not whether it works — it does — but which model fits your memory, which runtime serves it, and what has to change the moment a second person wants to use it.
Irv CassioReference guideBeginner to enterpriseAbout 18 min read
TL;DR · 90 seconds
The whole guide in one paragraph
Start with Ollama. Install it, pull Qwen3.6-35B-A3B, and you have a working local model in about five minutes — it is a Mixture-of-Experts design with ~35B total parameters that activates only ~3B per token, so it answers like a small model and knows like a large one, and it fits in roughly 21GB at Q4. When you want a coding agent rather than a chat partner, the obvious move is Qwen3.8-27B — dense, Apache 2.0, 262K context, and the strongest open model on vendor coding benchmarks. On Apple Silicon, measure before you switch: on an M5 Max the sparse 3.6 scored higher on executed code tests and finished the same suite 5.5–6.5x faster. The 27B earns its keep as a vision and long-context specialist rather than as the everyday coding agent. Hugging Face is where everything else lives — three million models, and now the home of llama.cpp and the GGUF format too. Under the hood the runtime matters: llama.cpp runs anywhere, MLX is 20–87% faster on Apple Silicon for models under 14B, and vLLM is what you serve from — roughly 20x Ollama's throughput under concurrency. Put Open WebUI or AnythingLLM in front for people, and LiteLLM in front for developers: it is where per-person keys, budgets and per-request cost and latency logging live. Then point your existing tools at it — Claude Code through an Anthropic-compatible gateway, the Codex CLI through OSS mode — and the agent you already know keeps working, on your hardware. The through-line: the model is the easy part; the gateway is where local AI becomes something an organisation can actually run.
Easiest first stepOllama · ~5 minutes
Best general starterQwen3.6-35B-A3B
Memory at Q4_K_S~21 GB
Best local coding modelQwen3.8-27B · Apache 2.0
Its context window262,144 tokens
MLX lead under 14B+20% to +87%
vLLM vs Ollama, concurrent793 vs 41 tok/s
Models on Hugging Face~3 million
01 · Start hereAll users
Ollama, in about five minutes. Then never think about model files again.#
There is a version of this guide that starts with quantisation formats and GPU offload layers. This is not that guide. Install Ollama, pull one model, and you are running. Everything else in these nine sections is an answer to a problem you do not have yet.
# macOS: brew install ollama — or download from ollama.com
ollama pull qwen3.6:35b-a3b
ollama run qwen3.6:35b-a3b
That is the whole first step. Ollama handles the download, picks a sensible quantisation for your hardware, and gives you two things at once: an interactive chat session in the terminal, and an OpenAI-compatible API on http://localhost:11434/v1 that a surprising number of tools already know how to talk to.
Why Ollama first, specifically
It switches models for you. Ask for a different model and Ollama loads it, unloading what it no longer needs. No restart, no config file, no manual memory arithmetic. This is the single biggest quality-of-life difference from running a raw inference server.
It has a catalogue. Roughly 4,500 models in the library — Qwen, Llama, DeepSeek, Kimi, Mistral, Gemma, Phi — each with pre-built quantisations, so "which file do I download" stops being a question.
It wires up coding assistants.ollama launch installs and configures popular assistants against the model you choose, which is how most people first point a real agent at local inference.
It has an escape hatch upward. Cloud models carry a :cloud suffix and run on Ollama's servers using the same CLI and API. When a task genuinely needs a model larger than your machine, you change a tag rather than a toolchain.
The one habit to build now: unloading a model is not the same as removing it. An unloaded model is still callable, and the next thing that calls it reloads it — at its default context length, which can be far larger than you intended. On a shared box that is how one forgotten model quietly starves everything else of GPU memory. ollama rm is the command that actually retires something.
Which model — and what it actually costs in memory. #
Two models cover almost everyone starting out. Qwen3.6-35B-A3B for general use, because its architecture makes it feel far faster than its size. Qwen3.8-27B when you want a coding agent, because it is the strongest local model for that work right now and its licence lets an enterprise actually deploy it.
Qwen3.6-35B-A3B — the one to start with
The A3B in the name is the whole story. It is a Mixture-of-Experts model: about 35 billion total parameters, of which roughly 3 billion are activated for any given token. You pay for 35B in memory and get something close to 3B in speed. For a first local model — where the most common disappointment is "it works but it is unbearably slow" — that trade is exactly the right one.
Memory: roughly 21GB at Q4_K_S, or about 14GB at Q3_K_S if you are tight. A 16GB card with 32GB of system RAM behind it can run the Q4. Released April 16, 2026.
Qwen3.8-27B — the one for coding agents
Released August 14, 2026 under Apache 2.0, dense rather than MoE, with a vision encoder and a 262,144-token context window. Alibaba's reported figures put it at 61.7 on SWE-bench Pro, 90.3 on LiveCodeBench v6, 84.3 on OSWorld-Verified and 70.7 on CoWorkBench. It fits on a single GPU, and the licence means the weights can be inspected, modified and hosted entirely behind your own controls.
That last point is not a footnote. A permissive licence is the difference between "we experimented with a local model" and "legal signed off on running this in production."
The first four bars are weights. The fifth is the same model with a long context window attached — which is the number that surprises people when they move from a laptop to a server.
The KV cache is the line item nobody budgets for. Model weights are a fixed cost you can look up. The key-value cache — the memory holding the conversation itself — scales with context length and with how many requests you serve at once. A 27B model whose weights are 30GB can comfortably need twice that to serve a 262K window to several people. If you size a GPU purchase on weights alone, you will buy the wrong GPU. One exception worth knowing before you over-buy a Mac: that arithmetic assumes a runtime which pre-allocates the window — llama.cpp and vLLM both do. MLX allocates the cache lazily, so declaring a 262K context costs almost nothing until a conversation grows into it. Measured on an M5 Max: two models both loaded at 262K held 53.83GB of weights while only 21.1GB was wired. On Apple Silicon the KV cache is a bill you are charged as you use it, not one you pay at load.
How it compares to the rest of the field
Meta's Muse Glimmer-30B (released August 10, 2026) is the natural comparison, and it is genuinely close. On LLM Stats' composite it sits at 35.3 against Qwen3.6-35B-A3B's 32.6, and of the ten benchmarks reported for both, Glimmer wins seven — AIME 2026, CharXiv-R, Humanity's Last Exam, MCP Atlas, SkillsBench, SWE-Bench Pro and SWE-Bench Verified. Qwen takes GPQA, MMMU-Pro and OmniDocBench 1.5.
But the architectures are not comparable on paper alone. Glimmer is dense — all 32.3B parameters fire for every token. Qwen3.6-35B-A3B fires about 3B. On a benchmark table that difference is invisible; on your machine it is the difference between a model you use and a model you open once. Benchmarks measure what a model knows, not what it costs you to ask.
Qwen3.6-35B-A3B
Qwen3.8-27B
Muse Glimmer-30B
Architecture
MoE · ~3B active
Dense 27B
Dense · 32.3B active
Released
Apr 16, 2026
Aug 14, 2026
Aug 10, 2026
Licence
Open weights
Apache 2.0
Meta terms
Context
Long
262,144
Long
Memory at Q4
~21 GB
~17 GB
~18–20 GB
Best at
General chat, speed
Coding, agents, vision
Reasoning benchmarks
Pick it when
It is your first model
An agent will drive it
Benchmarks are the brief
Treat every figure above as vendor-reported. Alibaba published Qwen's numbers, Meta published Glimmer's, and the composite scores come from third-party aggregators running vendor-supplied configurations. Independent replication of 2026 local-model benchmarks is thin. The only benchmark that settles anything is your own workload, run twice.
Hugging Face is where the other three million models live. #
Ollama's catalogue is curated, which is exactly what you want on day one and exactly what constrains you by month three. Hugging Face is the index for everything else — around three million models, half a million datasets, and roughly 18 million developers publishing to it.
You go there for three reasons: a model Ollama has not packaged, a specific quantisation someone else already made and measured, or a fine-tune aimed at your domain. In practice you will be looking for one of two file formats.
GGUF — the quantised format llama.cpp reads, and therefore what Ollama and LM Studio consume. This is the format for "run it on whatever hardware I have." Community quantisers publish a ladder of sizes for most popular models within days of release.
MLX — the Apple Silicon format. Same weights, packed for the unified-memory path. If you are on a Mac, look for an MLX build before a GGUF one.
Safetensors — the full-precision originals. This is what vLLM pulls when you serve a model properly, and what you quantise from if you are making your own.
LM Studio will search, download and run both GGUF and MLX builds directly from the Hub, which makes it the most pleasant way to browse Hugging Face without leaving a GUI.
Two ownership changes worth knowing before you standardise on the Hub. Hugging Face acquired ggml.ai in February 2026, bringing llama.cpp and the GGUF format itself in-house — so the index and the dominant local runtime now share an owner. And on September 3, 2026, Nvidia announced a definitive agreement to acquire Hugging Face for $12.93 billion, expected to close in the first half of 2027 subject to regulatory approval. Nothing has changed for users yet. But "the neutral open-model hub" and "owned by the company selling the GPUs" are different propositions, and an enterprise standardising on the Hub should have a mirror strategy that does not depend on that question resolving well.
llama.cpp, MLX, vLLM. Three answers to three different questions.#
Ollama is not a runtime — it is a manager wrapped around one. Underneath, three engines do the actual work, and the right one depends entirely on whether you are serving yourself, serving a Mac, or serving other people.
llama.cpp — the one that runs anywhere
The C/C++ engine behind Ollama, LM Studio and most desktop tooling. Its virtue is reach: CPU-only, mixed CPU/GPU, Nvidia, AMD, Apple, a Raspberry Pi. If a model can run on your hardware at all, llama.cpp is usually why. It is also the origin of GGUF, and since February 2026 it has been developed inside Hugging Face.
MLX — the one for Apple Silicon
Apple's array framework, built around the fact that on an M-series chip the CPU and GPU share the same physical memory. Tensors move between them with no copy, which removes an overhead that frameworks designed for discrete GPUs cannot avoid.
The measured effect is large where it matters most. MLX leads llama.cpp by 20 to 87 percent for models under 14B, and by 10–20% above that. On a Mac mini M4 Pro running Qwen3-Coder-30B-A3B, one comparison recorded ~130 tok/s on MLX against 43 tok/s for Ollama's llama.cpp backend — roughly 3x. Measured first-hand on an M5 Max in September 2026, each model alone at the full 262K context and one serving slot: Qwen3.6-35B-A3B 8-bit decodes at 78.6 tok/s with a 224ms median time-to-first-token, against 27.2 tok/s for Qwen3.8-27B at 4-bit and 17.5 tok/s at 8-bit — the sparse-versus-dense gap, not a quantisation gap. The gap narrows above 27B, where memory bandwidth becomes the binding constraint for everyone. For the same 4-bit model MLX is 5–10% more memory-efficient, which on a fixed-RAM Mac directly translates into a larger model fitting.
Two data points tell you where this is heading: Apple's January 2026 research on the M5's per-core Neural Accelerators reported time-to-first-token 4.06x faster than M4 on Qwen3-14B-4bit, and Ollama announced it was switching to MLX in March 2026. On a Mac, MLX is no longer the enthusiast option — it is the default path.
vLLM — the one you serve from
Everything above assumes one person at a time. vLLM assumes the opposite. Two architectural choices carry it: PagedAttention, which manages the KV cache the way an operating system manages virtual memory, and continuous batching, which slots a new request into the running batch the moment a slot frees rather than waiting for the batch to finish.
The gap is a function of concurrency, not raw speed. For a single user Ollama is competitive and starts in about three seconds against vLLM's nine. The advantage appears the moment requests overlap — which is what agents do constantly.
llama.cpp / Ollama
MLX
vLLM
Use it for
One person, any hardware
One person, a Mac
Many people or many agents
Hardware
Everything
Apple Silicon only
Nvidia-class GPU
Model format
GGUF
MLX
Safetensors, FP8/AWQ
Cold start
~3 s
Fast
~9 s
Concurrency
Weak
Weak
The entire point
Operational cost
None
None
Real — it is a service
The pattern most teams land on is not a choice but a sequence: prototype on Ollama against the OpenAI-compatible surface, then move the same calls to vLLM when traffic justifies running a service. Because both speak the same API, that migration changes a base URL and nothing else.
A terminal chat session proves the model works. It does not get anyone else using it. Three tools cover the gap, and they are complementary rather than competing — most real deployments end up running two of them.
LM Studio — the desktop testbench
A polished Mac and Windows app for browsing Hugging Face, downloading GGUF or MLX builds, and trying them side by side. It also exposes a local OpenAI-compatible server and ships developer SDKs, with no usage caps. It gives you one thing a plain OpenAI-shaped listing cannot: /api/v0/models reports whether a model is loaded, at what context length, on which engine and at which quantisation. Where your governance story depends on knowing exactly what answered a request, that endpoint is the difference between verifying and assuming. This is the tool for evaluating models — the fastest way to answer "is this one better than that one for my task" without touching a config file.
Open WebUI — the shared front end
A ChatGPT-style web interface for a team. Multi-user, document upload, a large extension ecosystem, and by a clear margin the most active community of the self-hosted chat front ends. If the goal is "give the department a private ChatGPT," this is the default answer.
AnythingLLM — the document one
Purpose-built for retrieval over your own documents, with workspaces, an embedded vector store and swappable embedding models. It treats Ollama, LM Studio, LocalAI and cloud providers all as interchangeable backends, so it does not lock you to a runtime. In an August 2026 comparison over a 5,047-page corpus it came out ahead on production reliability — best citations, persistent workspaces, and the lowest hallucination rate of the tools tested at 6%.
The shape that worksLM Studio on the engineer's laptop for evaluation · Open WebUI for the organisation · AnythingLLM when the question is "what do our documents say"
The chat front end is where your security review will actually land. These tools hold sessions, conversation history and often document contents. Bind them to loopback and put a reverse proxy with TLS in front — never expose the application port directly. A self-hosted chat UI speaking plain HTTP on a corporate LAN sends every login password and every conversation in the clear, and it is the single most common finding in a first local-AI deployment.
LiteLLM: one endpoint, real keys, real observability. #
This is the section that separates a demo from a deployment. An inference server has no authentication — it assumes it is on loopback talking to one trusted process. The moment a second person uses it, you need to know who is calling, cap what they spend, revoke them without restarting inference, and be able to answer "what happened" afterwards. That is a gateway's job.
LiteLLM is the open-source one most teams land on. It presents a single OpenAI-compatible endpoint across 140+ providers and nearly 1,900 models, and — the part that matters here — it does not care whether the model behind it is local or cloud. Self-hosted, all data stays in your infrastructure, and it brings per-person virtual keys, budget enforcement, RBAC and audit logging, with per-request token usage, cost and latency logged to Langfuse or any OpenTelemetry pipeline.
Everything reaches the gateway; nothing reaches an inference server directly. The 10ms the gateway adds buys you identity, revocation and a log — there is no performance argument for routing around it.
The four rules that make this work
Bind every inference server to loopback. If vllm or ollama is reachable from the network, the gateway is decorative — anyone can skip it. This is not a hardening step to do later; it is the thing that makes the gateway mean anything.
One live key per person, self-service. Issue keys from a small portal tied to your directory, and make issuing a new key revoke the old one. Otherwise forgotten keys accumulate, stay valid for their full lifetime, and nobody is watching them. Store the gateway's token id, never the key itself — a lost key gets replaced, not recovered.
Secrets in a mode-600 environment file, never in the unit file. Systemd unit files are world-readable by default. A master key inline in one is a master key readable by every account on the box.
Capture the configuration in a repo, and check it for drift. Configuration that lives only on the machine turns any rebuild into archaeology. Snapshot it, review the diff, and if the diff shows something you did not do, find out who did.
One developer's settings can be everyone's outage. A real failure worth internalising: a single coding-agent session grew past the server's context limit and resent the same oversized request every few minutes for more than a day. Each rejection sent the gateway down a secret-redaction code path with a catastrophic-regex bug, pinning its event loop and holding 35GB of memory. Result: the gateway was unreachable for every user and every tool until an administrator restarted it. Shared inference means shared blast radius — set the client-side context limits in §07, and alert on repeated identical rejections.
Point Claude Code — or the Codex CLI — at your own model. #
This is the payoff. You do not have to learn a new agent to use a local model. The coding CLI you already use can be redirected at a model you host, keeping its skills, its tool use and its workflow, while the tokens stay on your hardware. The two CLIs get there by noticeably different routes.
The Codex CLI — the easy one
Codex speaks the OpenAI API, which is exactly what every local server already serves, so there is no translation layer to go wrong. The supported path is OSS mode:
# the shortest route — Ollama does the wiring
ollama launch codex
# or select a local provider directly
codex --oss -m qwen3.8:27b
# or set it once in ~/.codex/config.toml
model = "qwen3.8:27b"
base_url = "http://localhost:11434/v1/"
wire_api = "responses"
Two requirements are not negotiable. The model must support tool calling — Codex uses tools for everything it does, and a model without them fails immediately rather than degrading. And it needs at least a 32K context window; below that the agent spends its budget re-reading itself.
Claude Code — worth the extra steps
Claude Code speaks the Anthropic Messages API and nothing else, so it needs something in front that accepts /v1/messages. LiteLLM in front of vLLM is the common shape — and recent vLLM serves /v1/messages natively, so the gateway can pass the request through rather than translating it. Three environment variables start it:
Claude Code refuses any model id without "claude" or "anthropic" in it
The check runs client-side, before a socket is opened. Your gateway never receives the request and logs absolutely nothing, so every server-side investigation comes up empty. It is not a picker filter — it is a hard block on starting a turn.
The fix is a gateway alias whose id contains the required substring, pointing at the same backend, with the honest name in display_name. The word "claude" in that alias is a client compatibility requirement, not a claim about what is running — and it is worth saying that out loud in your own documentation, because the next engineer will read that id and draw the wrong conclusion.
The five settings that turn a working session into a reliable one
Setting
Why it matters
ANTHROPIC_DEFAULT_HAIKU_MODEL
Background work — conversation titles, summaries — resolves the haiku alias, not your main model. Leave it unmapped and your gateway is asked for a Claude model it has never heard of. The session works, titles silently never appear. Map the sonnet and opus aliases too.
CLAUDE_CODE_MAX_OUTPUT_TOKENS
Sent as max_tokens on every turn. It is also the only guard against a reasoning model spiralling inside its own thinking block, because reasoning counts against it. Measured on an M5 Max: Qwen3.8-27B at 4-bit spent 602 seconds and 63,727 characters of reasoning on a CSV-parsing task, hit a 16,384-token ceiling and returned nothing at all, while its own 8-bit build answered the same task correctly in 6,808 tokens. Set this too low and a thinking model returns empty; leave it unset and it can burn ten minutes producing nothing.
CLAUDE_CODE_MAX_CONTEXT_TOKENS
How big the window is. For an unrecognised model id Claude Code guesses, and a wrong guess is what produced the outage in §06. Set it to the server's limit minus your output ceiling minus a couple of percent of margin.
CLAUDE_CODE_AUTO_COMPACT_WINDOW
When to compact. It is capped at the window Claude Code believes in, so on its own it cannot correct a wrong belief — set both, to the same number.
CLAUDE_CODE_EFFORT_LEVEL
Pin it. Some open models accept only a subset of the effort ladder, and a value the chat template rejects returns a 500 on the very first turn that reads exactly like a broken gateway.
Put these in the env block of ~/.claude/settings.json rather than your shell — a GUI app does not inherit a login shell, and the settings file beats shell exports, so setting both and editing only one produces a change that does nothing for reasons that are invisible. Better still, scope it per repository with a project-level .claude/settings.json, so one codebase goes to the local model while everything else stays on your subscription. That is usually what you actually want.
Why the context window is the setting to get right. Claude Code recovers from an over-long conversation only when it recognises the "too long" error. An error worded differently — because it came from vLLM through a gateway rather than from Anthropic — does not trigger compaction, so the client resends the same oversized request indefinitely. Setting the window correctly prevents the session from ever reaching the ceiling; a compatibility hook at the gateway that rewrites the error is the belt to that suspenders.
Toolbelt
ClaudeSwitch — flipping between cloud and local from the menu bar
A macOS menu-bar utility that moves Claude Code between the Anthropic cloud and a model you run yourself — LM Studio, Ollama, or a LiteLLM proxy — for both the CLI and the desktop app, in one click. It owns a defined set of keys in settings.json and removes them cleanly on the way back, probes the destination for what it can actually serve rather than trusting configuration, and includes a compatibility relay for gateways that have not yet been patched for Claude Code's request shape.
Three harnesses that assume a local model is normal. #
The agent harness — the loop around the model that calls tools, manages context and decides what to do next — used to be an afterthought. In 2026 it is a category, and the notable thing about all three of these is that pointing them at a self-hosted endpoint is a supported first-class path rather than a hack.
DeepSeek Harness (dsh)
Open-sourced August 13, 2026, and the reception was without precedent: 22,000 GitHub stars in the first 90 minutes, 100,000 within 48 hours, 155,000 within five days, and past 172,000 in the first week. The design idea is that everything is a plugin — the interface, the tool calls, even the agent loop itself — rather than a fixed feature set you configure around.
Why it belongs in a local-models guide: a harness whose agent loop is replaceable is a harness you can adapt to a model's actual quirks. Every open model has them — a tool-call format that drifts, an effort ladder with missing rungs, a reasoning block that needs bounding. With a fixed harness you work around those in the gateway. With a pluggable one you fix them where they belong. That is the argument for it; the counter-argument is that a plugin surface that wide is also an attack surface that wide, and the project is weeks old.
OpenClaw and Hermes Agent
Both support self-hosted models, by different routes. OpenClaw integrates with Ollama directly, which makes it the lower-friction choice for a fully offline setup — you pick a local model the way you would pick a cloud one. Hermes Agent takes any OpenAI-compatible endpoint, which means it works with Ollama, vLLM, LM Studio or a LiteLLM gateway without knowing the difference, and the same agent can run locally, in a container, on a server or in a cloud sandbox.
Read that difference as the trade it is: OpenClaw gives you a shorter path to running; Hermes gives you one configuration that survives moving the model. If the model will eventually live behind a gateway — and in an enterprise it will — the endpoint-agnostic route is the one that does not need redoing.
Worth stating plainly: a harness pointed at a local model still runs commands on your machine, still reads your files, and still acts on whatever text comes back from the model. Running the model locally removes a data-egress problem. It does not remove the need for the same permission boundaries, approval gates and sandboxing you would want around a cloud-backed agent — and an open model is, if anything, easier to steer with injected instructions than a heavily post-trained frontier one.
The rollout ladder. Each rung adds one thing that was optional on the one below.#
Most local-AI efforts fail not at the model but at the transition between rungs — usually by skipping straight from a laptop to a shared server without adding the things a shared server requires. The ladder is short, and each step has exactly one new obligation.
Most efforts fail at the transition between rungs, not at the model — usually by taking rung three's hardware without rung three's obligations.
What rung four actually requires
TLS everywhere, including internally. A gateway on plain HTTP sends every virtual key across the LAN on every request, and the admin UI sends the master key at login. An internal CA is a weekend of work and closes the whole category — publish the fingerprint, and tell people that a certificate warning after that means stop, not click through.
Identity you already have. Bind key issuance to the corporate directory. A directory account is the authorisation; no ticket queue, no approval step, no shadow list of who has access.
Revocation that works without a restart. This is the functional test of whether you have a gateway or just a proxy.
An audit trail with cost attached. Per-request token usage, cost and latency per key. Without it you cannot answer the two questions leadership will ask — what is this costing, and is anyone actually using it.
An inventory of credentials the agents themselves hold. The privileged service is usually not the model — it is the tooling around it. Credential stores that appear in no documentation are the most common serious finding in a mature local-AI deployment, and they live in skill directories and service configs rather than anywhere a scanner is pointed.
Configuration in a repository, checked for drift. A config repo that silently diverges from the running box is worse than no repo, because someone will one day rebuild from it.
The honest cost picture
Local inference is not free; it moves the cost from a variable line to a fixed one. You buy the GPU, you power it, and someone owns the pager. What you get in exchange is not primarily savings — it is that the data never leaves, the model never changes underneath you, and the per-token meter stops running. Those three properties are why regulated workloads, high-volume agent loops and anything touching customer records end up here regardless of the arithmetic.
Questions to take into the room
Which specific workloads are we moving local — and is the reason data residency, cost, or latency? The answer changes the hardware.
Who carries the pager when the GPU box is down and it is the whole company's coding assistant?
What is our mirror strategy for model weights if the Hub's ownership changes what is available?
Are we running one shared model, or per-team models — and do we have the memory headroom for the second answer?
What is the rollback? If the local model is worse at a task than the cloud one, how quickly can a developer switch back, and does that switch leave a record?
The model is the easy part. Everything that makes local AI survivable in an organisation — keys, revocation, TLS, budgets, a log — is infrastructure you would have needed anyway.The actual lesson from running one of these
Every number in one table
Figures as reported at the time of writing, September 18, 2026. Vendor benchmark claims are marked as such; local-model benchmarks in 2026 have thin independent replication. The rows marked measured were added September 20, 2026 from a three-model run on an M5 Max — first-party, executed tests, not vendor-reported.
Three ways a local benchmark lies to you. All three were hit in a single night of measuring, and each produced a clean, publishable-looking number rather than an error. One: a token ceiling is not a budget for a thinking model. Set max_tokens to 4,096 and the model can spend all of it inside its reasoning block and return empty content, scoring a flat zero that measures your harness rather than the model. Record finish_reason and treat length as void, not as failure. Two: two quantisations of one model can land under a single key, so lms load warns “2 models match… loading the first one” and serves 8-bit under a 4-bit label. Pin the quantisation and read it back from the server before the run counts. Three: serving slots default high, and a model loaded at four parallel slots is not comparable with one loaded at one. Verify quantisation, context length and slot count from the server's own listing, every run.