Local Models

Local Models#

A local model runs on hardware the operator controls. The trade against frontier APIs is real: smaller models, more setup, slower on a laptop, but no data leaves the box. For sensitive corpora, classified workflows, air-gapped labs, or anywhere vendor logging is unacceptable, local is the only option.

Runtimes#

  • Ollama, ollama run llama3.1 and you have a local server with an OpenAI-compatible API. Easiest entry point; packages models, weights, and the runtime as a single artifact.

  • llama.cpp, the standard CPU/GPU inference engine for GGUF-quantised models. Ollama wraps it; llama-server runs it directly. Lowest overhead.

  • vLLM, production-grade serving. Continuous batching, high throughput, OpenAI-compatible. For single-machine GPU inference at scale.

  • TGI (Text Generation Inference, Hugging Face), alternative production server. Integrates with the HF model hub.

  • MLX, Apple Silicon native. Best performance per watt on a MacBook.

  • Text Generation WebUI / LM Studio, desktop GUIs for experimentation; not for production.

Model families#

  • Llama (Meta), broad ecosystem, many fine-tunes. llama-3.x for general use; code-llama derivatives for code.

  • Qwen (Alibaba), strong multilingual; qwen2.5-coder for code.

  • Mistral / Mixtral, mistral-small, mixtral-8x7b. Mixtral’s MoE structure runs faster than dense models its size.

  • Gemma (Google), compact; good per-parameter quality.

  • Phi (Microsoft), small models trained for reasoning; phi-3-mini runs on a laptop.

  • DeepSeek, open-weight reasoning models; deepseek-r1 family.

Quantisation#

Quantisation trades precision for size. The format names are GGUF tags from llama.cpp:

  • Q8_0, 8-bit; near-lossless. Default if you can fit it.

  • Q5_K_M, 5-bit; common quality/size sweet spot.

  • Q4_K_M, 4-bit; the standard for laptops. Noticeable quality drop on tasks that need precision (code, math).

  • Q3, Q2, aggressive; quality degrades fast. Avoid for operator workflows.

A 7B model at Q4_K_M fits in ~5 GB; a 70B model at Q4_K_M needs ~40 GB. RAM, VRAM, or unified memory; the runtime decides where it lives.

When local is the right choice#

  • Sensitive data. Targeting data, captured material, internal reports. The OPSEC argument outweighs the quality gap.

  • Offline / air-gapped. Field deployments, isolated labs.

  • High volume / low cost. Bulk classification, embedding generation, log enrichment over millions of rows. Frontier API bills add up; local hardware amortises.

  • Latency-sensitive. Local inference avoids round-trip to a vendor; useful for interactive UIs.

When frontier wins#

  • Hard reasoning. Frontier models still outperform open-weight models on multi-step reasoning, complex code, long context.

  • Long context. Frontier APIs offer 200k-1M+ token windows; local stacks struggle past 32k without specialized setup.

  • Tool use. Frontier tool-use is more reliable; local tool use needs careful prompting and often a fine-tune.

Operational practice#

  • Pin the model and the quantisation. Capture ollama show / llama-server --version output in the ops record.

  • Health-check the server. Local servers crash; wrap in systemd with restart-on-failure.

  • Track tokens/sec. It is the only useful capacity metric.

  • Embeddings locally. nomic-embed-text, bge-m3 run on CPU; no reason to send embeddings to a vendor for sensitive corpora.

References#

Contents