Monthly Archives: August 2026

Do we really need big models in 2026?

Posted on by

I have a theory: for roughly 90% of what people use ChatGPT, Claude, or Gemini for day-to-day, a 7B to 32B local model is not just “good enough.” It is the right tool. The big models can be overkill, and the companies selling them know it.

You open ChatGPT to write an email, summarize a document, debug a function, format  JSON, explain an error message, or draft a blog post.You are paying $20 to $50 per million tokens for a trillion-parameter model to do work a 32 billion parameter model can do just as well.

Anthropic and OpenAI know this. They have both quietly built smaller model tiers and are actively routing users toward them. Meanwhile, local models like Qwen 3.5 and Gemma 4 have crossed a capability threshold that makes them genuinely competitive for most everyday tasks.

What People Actually Use LLMs For

The 90% hypothesis is simple. Most LLM usage falls into a handful of structured, well-defined tasks that need language understanding, pattern matching, and instruction following. None of these require emergent reasoning or frontier-level knowledge.

Writing and editing emails, messages, documents. Code generation and debugging for standard tasks. Summarization of articles, meetings, or long documents. Translation and language learning. Data formatting. Explaining concepts and answering general questions. Creative writing, brainstorming, planning.

These are the things you are probably doing right now.

The 10% that needs big models is real. Complex multistep reasoning with many constraints. Novel research and mathematical proof. Creative work requiring genuine originality. Tasks requiring the latest knowledge combined with deep reasoning. Extremely long context like entire codebases or legal documents.

Notice, however, what that list does not include. It does not include writing an email. It does not include formatting JSON. Furthermore, it does not include explaining an error message. The big models are optimized for the last 1% of capability, not the 90% of everyday tasks.

OpenAI and Anthropic Are Quietly Routing You to Smaller Models Already

Here is what the companies selling frontier models are doing. They are admitting, through their pricing and routing decisions, that smaller models handle most work.

Anthropic recommends Haiku 4.5 at $1 per million input tokens and $5 per million output tokens for routine tasks. Their API documentation literally says to start with Opus 5 for complex work,but use Haiku for everything else. Haiku is the default recommendation for the vast majority of API calls.

OpenAI positions GPT-5.6 Luna at $0.20 per million input tokens and $1.20 per million output tokens as the cost-sensitive workhorse. They call it the nano tier. Deliberately smaller. Deliberately cheap.

Both companies’ SDKs offer an automatic fallback to smaller models based on task complexity. Both offer adaptive thinking where the model decides whether it needs deep reasoning or a quick answer.

The subtext is clear: bigger is not better for most tasks. The routing is a business decision. Cheaper inference costs for them. However, it validates an instinct you probably already had: the small models handle the bulk of real work.

If the biggest companies in AI are routing you to smaller models for 90% of tasks, why send those requests over the internet at all?

The Capability Numbers: What 32B Models Actually Do

The benchmarks from Google DeepMind’s Gemma 4 release in April 2026 are what make this argument concrete. Not adjectives. Numbers.

Reasoning and knowledge benchmarks:

Benchmark Gemma 4 31B Gemma 4 26B (MoE, 4B active) Gemma 4 12B
MMLU Pro 85.2% 82.6% 77.2%
AIME 2026 (math) 89.2% 88.3% 77.5%
GPQA Diamond (science) 84.3% 82.3% 78.8%
BigBench Extra Hard 74.4% 64.8% 53.0%
MMMLU 88.4% 86.3% 83.4%

Coding benchmarks:

Benchmark Gemma 4 31B Gemma 4 26B (MoE) Gemma 4 12B
LiveCodeBench v6 80.0% 77.1% 72.0%
Codeforces ELO 2150 1718 1659
HLE (no tools) 19.5% 8.7% 5.2%

Vision benchmarks:

Benchmark Gemma 4 31B Gemma 4 26B (MoE) Gemma 4 12B
MMMU Pro 76.9% 73.8% 69.1%
MATH-Vision 85.6% 82.4% 79.7%

Source: Google DeepMind, Gemma 4 blog post on Hugging Face, April 2026. https://huggingface.co/blog/gemma4

On Chatbot Arena (the LMSYS blind human preference tests), the estimated Elo scores tell the same story:

  • Gemma 4 31B: ~1452
  • Gemma 4 26B MoE (4B active): ~1441
  • GPT-4o: ~1480
  • Claude Haiku: ~1380
  • GPT-4o-mini: ~1350

Source: Chatbot Arena Elo ratings, LMSYS. https://lmarena.ai

The 26B MoE model with only 4B active parameters scores within 30 points of GPT-4o on blind human preference tests. A 31B model scoring 85.2% on MMLU Pro is in the same ballpark as models 10 to 50 times larger. The 26B MoE model with a Codeforces ELO of 1718 is competitive with GPT-3.5-tier coding assistants.

These are not almost there numbers. These are actually good numbers.

Where Models Get Good Enough

The performance curve is not linear. Going from 7B to 32B gives you roughly 80% of the capability gain from going to 100B+. Going from 32B to 100B gives you maybe 10 to 15% more.

Microsoft’s Phi series research demonstrated that 3B to 14B models can outperform much larger models on instruction-following tasks when trained on high-quality synthetic data rather than scraped web data. The instruction-tuning gap is real. A well-trained 7B model can match a poorly-trained 70B model on most tasks.

Gemma 4’s 26B MoE model (4B active) proves that model architecture matters more than raw parameter count. You do not need to run all parameters to get their performance. The training data quality of Gemma 4 and Qwen 3.5, which use curated, deduplicated, high-signal datasets, gives them a disproportionate advantage over older models trained on scraped web data.

Based on the benchmark data, here is where the thresholds actually sit:

  • 7B models: Good for simple tasks. Formatting, basic code, short summaries, translations.
  • 13B to 32B models: Good for most tasks. Complex code, long documents, multistep reasoning, creative writing.
  • 70B+ models: Needed for frontier reasoning, novel research, extremely complex multi-constraint tasks.
  • 100B to 1.8T models: Needed for the 1% edge cases. However, even these models are being routed to smaller tiers for
    routine work.

The Advantages Nobody Talks About

Latency

A local 32B model on an RTX 4090 produces its first token in 50 to 200 milliseconds. API models take 500 milliseconds to 3 seconds depending on load and tier. For IDE integrations, interactive workflows, and real-time assistance, this difference is transformative. You stop waiting for the model. The model keeps up with you.  However, as we will see below, the trade off for faster response is slower throughput.

The Privacy Problem: You Are Training Your Competitors

When you send a prompt to a commercial provider, here is what actually happens:

  1. Your prompt leaves your network and travels through the public internet.
  2. It lands on company servers, often in data centers you cannot identify.
  3. It is processed by a cluster of GPUs across multiple machines.
  4. It may be cached, logged, and stored for quality assurance.
  5. It may be used to train future model versions.
  6. It may be accessible to third-party subprocessors.

Each big company has a privacy policy. However, over and over we have seen where companies do not always abide by their privacy policies. You can look at the reports of abuse by law enforcement of “private” security cameras an example. When companies get caught violating their own privacy policies, sometimes there are consequences, sometimes there are none. Many companies also have you agree to terms where the company can change the terms and conditions at any time, and you just have to accept it.

The real-world implications are concrete:

  • A startup sends its product roadmap to a commercial LLM for feedback. Now it is in their training data.
  • A law firm drafts confidential contracts with another. Now it is in their logs.
  • A healthcare organization processes patient notes with a commercial LLM. Now  private medical information could be accessed.
  • A financial firm analyzes trading strategies online. Now it is accessible to the company and any number of subprocessors.

With a local model, zero data leaves your machine. No subprocessors. No third-party vendors. No cloud infrastructure. No data retention policies to read and worry about. No compliance framework to justify to your legal team. No opt-out checkbox to find and click. Your prompts are processed by your GPU, in your data center, under your control. If you are in healthcare, finance, defense, or any regulated industry, this is not a nice-to-have. It is a requirement.

Reliability

No rate limits. No service degraded pages. No API version changes breaking your integration. Works offline. Works when the internet goes down.

Customization

Fine-tune on your codebase, your style, your domain. Control the full stack: model, tokenizer, generation parameters. No vendor lock-in.

You Still Get the Familiar Browser-like Experience

There are already multiple free and open source interfaces that you can use with local LLMs. You do not have to use a terminal to run a local model. There are polished, feature-rich desktop apps that look and feel like what you are already used to, just with your own models running locally.

Jan (jan.ai) has 43.9k GitHub stars and over 4 million downloads. It calls itself an “open-source ChatGPT replacement” that runs 100% offline. It is a native desktop app for macOS, Windows, and Linux via Flatpak. One-click model downloads directly from Hugging Face. Built-in tool-calling, web search, and vision support. Clean, modern UI that feels like ChatGPT but does not require an account. Supports multiple backends including llama.cpp, MLX for Apple Silicon, and Transformers. Active development with weekly releases, currently at v0.8.4.

Source: https://github.com/janhq/jan, https://jan.ai

AnythingLLM has 64.5k GitHub stars. It is described as “the all-in-one AI app” and lets you chat with your own documents, use AI agents, and supports multi-user setups. Built-in vector database and document pipelines for PDF, DOCX, and text files. No-code AI agent builder for automating complex workflows. Dynamic model routing that automatically switches between local and cloud models based on rules you define. Automatic memory, where the app remembers important information about you or your workspace. Scheduled tasks and recurring prompts with full agent capabilities. Multi-user support with per-user access controls. Desktop app available for Mac, Windows, and Linux. Browser extension for searching and chatting from anywhere. Image generation built in via the /img command.

Source: https://github.com/Mintplex-Labs/anything-llm

oobabooga/textgen (formerly text-generation-webui) has 47.5k GitHub stars. It is the original open-source local LLM GUI, now rebranded as TextGen Desktop app with chat mode, generation mode, and API mode. Chat and generation features include attaching images for visual understanding and uploading documents to chat about their contents. Multiple backends: llama.cpp, ExLlamaV3, Transformers, TensorRT-LLM. OpenAI and Anthropic-compatible API, so you can use it as a local drop-in replacement. Tool-calling support where models can call custom functions including web search, page fetching, and math. Training and image generation tabs for fine-tuning LoRAs and generating images. Portable builds for Linux, Windows, and macOS with CUDA, Vulkan, ROCm, and CPU-only options. One-minute setup: download, unzip, double-click the executable, and a window opens.

Source: https://github.com/oobabooga/textgen

You do not have to choose between privacy and convenience. These tools give you a browser-like experience with the same familiar chat interface, the same markdown rendering, the same copy-paste workflow, but with your own models running entirely on your machine. No accounts. No subscriptions. No data leaving your computer.

Where the Big Models Still Win

I want to be honest about where frontier models still pull ahead.

Complex multistep reasoning. Opus 5 and Sol still lead on tasks requiring many constraints, novel logic chains, or mathematical proof. If your task is “solve this unsolved math problem” or “design a new algorithm,” you need the big models.

Long context. 1M+ context windows are hard to match locally without massive VRAM. Reading an entire legal document or codebase and finding a specific clause is still an API advantage.

Multimodal input. Vision, audio, and video understanding are API-exclusive for now, though Gemma 4 is closing this gap fast with image, audio, and video support.

Continuous improvement. API models get better every week. Local models are frozen at download. The solution is periodic re-downloads, since new model versions come out monthly, or using hosted local model providers.

Ease of use. An API call is simpler than setting up a GPU. Not everyone has an RTX 4090 sitting around.

Speed. Unless you happen to have a huge system with multiple 48 GB GPUs and gigabytes of system RAM, local LLMs will be slower. You WILL wait longer for results. The number of tokens per second that the local models generate will generally be much lower than the enterprise models.  However, in some cases, it is not insanely slower. My Intel Arc B70 with 32 GB of VRAM runs MoE modelsat perfectly acceptable speeds.

The right mental model is tool selection, not replacement. Use API models for tasks requiring frontier-level reasoning, one-off tasks where setup cost matters more than per-call cost, and workloads needing 1M+ context windows. Use local models for high-volume repetitive tasks, sensitive data workflows, latency-sensitive applications, and any task you will call the model more than 500 times per day.

How to Get Started

How to Serve a Model

Today there are several ways to run a local model. Llama.cpp is a well-known tool that offers a lot of capabilities. You can do things like offload model layers to system RAM to run larger models than your VRAM would normally allow at the cost of speed. You can also compress your key-value cache for extra memory savings.

Ollama is another well-known server. It does not allow as much customization as llama.cpp, but it is very simple to use. You can run a command like ollama serve to run a server, ollama run to run a model, and so on.

VLLM is an up-and-coming server method that focuses on high-throughput and memory-efficient inference and operation. It offers features that have not yet been included in the others.

Hardware Requirements

  • 8B models: 8 GB VRAM. Any modern GPU, even integrated graphics.
  • 32B models: 16 to 24 GB VRAM.
  • 70B models: 48 GB+ VRAM.
  • CPU-only: runs anything, but 10 to 50x slower.

Frequently Asked Questions

Can a 32B parameter model really replace GPT-4 or Claude?

Not for every task. For code generation, documentation, summarization, and routine reasoning, the gap is small enough that most teams will not notice the difference. On Chatbot Arena blind human preference tests, Gemma 4 26B (4B active) scores within 30 points of GPT-4o. My own testing with OpenCode has shown me that Ornith 1.0 35B
MoE can code just as well as the commercial models for my tasks.

Do I need a GPU to run local LLMs?

You can run smaller models (7B to 13B) on CPU, but it will be slow, much slower. A consumer GPU with 16 GB or more VRAM, like an RTX 4090 or RTX 4080 Super, gives great performance for 32B models. Smaller (1 to 7B parameter) models are being actively researched and several can run in system RAM at acceptable speeds for things like using an AI agent.

How do local models compare to Claude Haiku or GPT-4o-mini?

Qwen 3.5-32B and Gemma 4-26B score competitively on most benchmarks. Gemma 4 26B MoE scores 82.6% on MMLU Pro compared to roughly 77% for GPT-4o-mini, and 77.1% on LiveCodeBench. They are in the same tier.

Is it legal to run these models locally?

Qwen 3.5 uses the Apache 2.0 license, which is fully open for commercial use. Gemma 4 also uses Apache 2.0. Both are truly open.

What about model updates? Won’t local models get stale?

That is a fair concern. The solution is periodic re-downloads, since new model versions come out monthly.

What is the smallest model I should start with?

Try Qwen 3.5-14B or Gemma 4 E4B (4.5B effective). They run on any modern laptop and handle roughly 70% of everyday tasks well.

The Bottom Line

You are not giving up capability by going local. You are choosing the right tool for the job.

The big models are real engineering achievements. They are also overkill for the tasks you actually use them for, and the companies selling them know it. They have built smaller tiers, routed you to them, and called it a business decision. Call it what it really  is: a confirmation that the capability threshold has been crossed.

Try one local model this week. Run a side-by-side comparison with your current API model. You might be surprised by how much you do not need the biggest model in the room.

Links: