Self-Hosted AI Coding: A Practical Guide to Privacy-First Development with Open Source Tools

You can run capable coding models entirely on your own hardware today using Ollama or llama.cpp for inference, an open model such as Qwen2.5-Coder for completions and chat, and an editor extension like Continue to wire it into VS Code or JetBrains. The trade-off is real: you exchange frontier-model quality and zero setup for full data control and predictable costs.

Why bother when hosted models are better?

Let me be direct about the quality gap first, because pretending it does not exist helps nobody. As of early 2026, the best open-weight coding models you can realistically run on a single workstation trail Claude, GPT-class, and Gemini models on complex, multi-file agentic tasks. If your daily work is "refactor this service across twelve files and update the tests," a hosted frontier model with an agent harness will do it better, faster, and with fewer dead ends.

Self-hosting wins in four specific situations:

• Contractual or regulatory constraints. Defense, healthcare, and some financial clients prohibit source code leaving controlled infrastructure, full stop. Vendor enterprise agreements with zero-retention clauses help, but for some contracts "we send code to a third party" is disqualifying regardless of the retention policy.
• Air-gapped or low-connectivity environments. A local model works on a plane, in a SCIF, or on a factory floor with no reliable uplink.
• Cost predictability at scale. Heavy agentic use of hosted APIs can run hundreds of dollars per developer per month. A one-time hardware spend amortizes differently, and for high-volume, low-complexity tasks (autocomplete, commit messages, boilerplate) local inference is effectively free after purchase.
• Latency for autocomplete. Tab completion needs to feel instant. A local model on adequate hardware returns first tokens in well under 200 milliseconds with no network round trip, which is hard to match consistently from a remote API under load.

If none of those apply to you, a hosted tool is probably the right call, and you should treat local models as a fallback or a learning exercise rather than your primary setup.

What hardware do I actually need to run a coding model locally?

This is where most guides get vague, so here are concrete tiers based on what I run and what colleagues run:

Tier 1: Autocomplete only (any recent machine). Small models in the 1.5B to 7B parameter range handle fill-in-the-middle completion well. A 3B model quantized to 4 bits needs roughly 2 to 3 GB of memory and runs acceptably on CPU, though a modest GPU or an Apple M-series chip makes it feel native. This tier is genuinely useful and cheap to reach.

Tier 2: Chat and code review (16 to 24 GB of VRAM or unified memory). Models in the 14B to 32B range are where local chat becomes worth using. Qwen2.5-Coder-32B at Q4_K_M quantization occupies roughly 19 to 20 GB, which fits on a single RTX 4090 (24 GB) or comfortably in a MacBook Pro with 36 GB or more of unified memory. On a 4090 you can expect generation speeds in the range of 30 to 40 tokens per second, which is fine for chat but slower than hosted APIs. On Apple silicon, expect roughly half that unless you are on an Ultra-class chip.

Tier 3: Agentic work (48 GB plus, often multi-GPU). Larger open models and long-context agent loops need serious memory, because KV cache grows with context length. A 32B model with a 32K token context can push past 24 GB once the cache fills. This is dual-GPU or Mac Studio territory, and the money often exceeds several years of a hosted subscription. Be honest with yourself about whether the privacy requirement justifies it.

A caveat that bites people: quantization is not free. Q4 quantization is the standard compromise, but on coding tasks I see a noticeable increase in subtle errors (off-by-one logic, wrong standard library calls) compared to Q8 or full precision of the same model. If you have the memory, run Q5_K_M or Q6_K for chat models and reserve Q4 for autocomplete, where speed matters more than deep reasoning.

The core stack: inference server, model, editor integration

Inference: Ollama or llama.cpp

Ollama is the pragmatic default. It manages model downloads, handles quantization variants, exposes an OpenAI-compatible API on localhost:11434, and keeps models loaded in memory between requests. Getting a coding model running takes one command:

1ollama pull qwen2.5-coder:32b
2ollama run qwen2.5-coder:32b

Under the hood, Ollama builds on llama.cpp, which you can also run directly via llama-server if you want finer control over context length, GPU layer offloading, and sampling parameters. I use llama.cpp directly when I need a specific quantization or a flash-attention flag Ollama does not expose, and Ollama for everything else because its lifecycle management saves real time.

If you are serving a team rather than one machine, look at vLLM instead. Its continuous batching gives dramatically better throughput when multiple developers hit the same GPU server concurrently. For a single user, vLLM is overkill and its startup and memory pre-allocation behavior is annoying on a workstation.

Editor integration: Continue

Continue is the most mature open-source editor extension for VS Code and JetBrains that works cleanly with local endpoints. A minimal config pointing chat at a 32B model and autocomplete at a small fast model looks like this:

1models:
2 - name: qwen-coder-chat
3 provider: ollama
4 model: qwen2.5-coder:32b
5 roles: [chat, edit]
6 - name: qwen-coder-fim
7 provider: ollama
8 model: qwen2.5-coder:1.5b
9 roles: [autocomplete]

That split matters. Do not point autocomplete at your big chat model. Completion requests fire on nearly every keystroke pause, and a 32B model will either lag or evict itself from memory fighting with the chat model. A dedicated 1.5B or 3B model trained with fill-in-the-middle support is the correct tool, and the quality difference from a larger model in that role is smaller than you would expect because completion is a local, short-horizon task.

Tabby is a reasonable alternative if you want a self-hosted server with team features (shared deployment, usage analytics) rather than per-developer setup.

How close can local models get to Copilot or Claude for real work?

Set expectations by task type:

Autocomplete: near parity. Single-line and short multi-line completion from a local FIM model is honestly hard to distinguish from GitHub Copilot for mainstream languages. This is the easiest win in the whole self-hosted story.

Chat and explain-this-code: good. A 32B coding model answers "why does this async handler deadlock" or "write a pytest fixture for this" competently. It falls behind on obscure libraries, very recent framework versions, and questions requiring long chains of reasoning. Knowledge cutoffs hurt more locally because you cannot rely on the vendor silently updating the model.

Multi-file agentic editing: the weak spot. Agent workflows amplify model weaknesses because errors compound across steps. A hosted frontier model might complete a cross-cutting refactor in one shot where a local 32B model takes three attempts and leaves a broken import. Open-source agent harnesses can point at local endpoints, and it works, but budget your patience. I keep agentic tasks on hosted models when the code is not sensitive, and drop to local only when policy demands it.

Failure modes to watch for locally:

• Context overflow silently degrades output. Local servers often truncate from the front of the prompt when you exceed the configured context window, so the model loses your system instructions first. Set num_ctx (Ollama) explicitly and monitor prompt sizes.
• Wrong chat template ruins a good model. If a model produces rambling or self-interrupting output, check that your server applies the correct prompt template for that model family before blaming the weights.
• Thermal throttling on laptops. Sustained generation on a MacBook can halve token throughput after a few minutes. Fine for chat, painful for agents.

Licensing and compliance are not automatically solved

Self-hosting removes the data-transmission problem but not the licensing one. Read the model license before deploying at work. Qwen2.5-Coder ships under Apache 2.0, which is clean for commercial use. Llama-family models carry Meta's community license with specific conditions. Some "open" models restrict commercial use entirely. Your compliance team will ask, and "it was on Hugging Face" is not an answer.

Also decide what "privacy-first" means operationally. If your editor extension has telemetry enabled, or your local server logs prompts to disk on a machine that syncs to cloud backup, you have quietly rebuilt the exposure you were avoiding. Continue lets you disable telemetry in config; do it, and check where Ollama stores logs on your platform.

A sensible adoption path

Do not try to replace your hosted tooling in one weekend. Here is the sequence I recommend and have used with teams:

Week one: Install Ollama, pull a 1.5B or 3B coding model, and wire it into Continue for autocomplete only. This gives you a real daily-use win with almost no hardware requirement, and it teaches you the operational basics (memory pressure, model loading, template quirks) on a low-stakes task.

Week two: Add the largest chat model your hardware runs at Q5 or better. Use it alongside your hosted tool for two weeks and notice which questions it answers well and which it fumbles. Most engineers land on a natural split: local for quick lookups and code explanation, hosted for hard design questions.

Month two: If you have a genuine compliance driver, pilot a shared vLLM deployment on a GPU server for the team, measure concurrent throughput under real load, and only then talk about decommissioning hosted access for the sensitive repositories. If you do not have that driver, stop here: a hybrid setup where local handles autocomplete and hosted handles agentic work is the best cost-quality-privacy balance most teams can reach in 2026, and there is no shame in it.

The engineers who get burned are the ones who buy the GPU first and figure out the workflow later. Start with the smallest model that does useful work, prove the workflow, then scale hardware to match demonstrated need.