Understanding Quantization

If you try to run an uncompressed, full‑precision open‑source model on a consumer machine, you will almost instantly run out of memory. Quantization is the secret math that makes local AI possible for everyday users.

It compresses a model’s weights so it uses a fraction of the VRAM while retaining nearly all of its original intelligence.

📉 Bits and Brains: How It Works

AI models are collections of numbers called weights. In their native form, these weights are stored as high‑precision 16‑bit floating‑point numbers (FP16).

Quantization maps those precise 16‑bit numbers down to smaller formats — 8‑bit, 4‑bit, or even 3‑bit integers.

Think of it like saving a massive image file:

• FP16 (Uncompressed): A raw, uncompressed photo. Perfect, but huge.
• Q4 (Quantized): A highly optimized JPEG. If you zoom in with a magnifying glass, you might notice tiny artifacts — but to the naked eye, it looks identical and is 75% smaller.

Quantization is how you fit a “giant brain” into a normal laptop.

🔠 Common 2026 Quantization Formats

When downloading models via LM Studio, Ollama, or vmlx, you’ll see shorthand extensions attached to model names. Here’s what they mean.

The GGUF Standard (Ollama + llama.cpp + LM Studio)

GGUF is the standard format used by Ollama, LM Studio, and llama.cpp. It allows your computer to split a model across GPU VRAM and system RAM if needed. Other engines (like vmlx and ExLlamaV2) use GPTQ or AWQ instead (different format, but similar intent).

• Q4_K_M (4‑bit Medium):
  The global standard for everyday use. Fastest token speeds, drastically reduced VRAM, and only minor reasoning loss.

• Q5_K_M (5‑bit Medium):
  The “sweet spot” for text and coding. Recovers most of the model’s intelligence while still saving huge amounts of VRAM.

• Q8_K_M (8‑bit):
  Near‑native accuracy. Use this only if you have VRAM to spare and need maximum precision for math or technical coding.

Importance Matrix (IQ) Quants

These formats compress less‑critical layers more aggressively while preserving core reasoning layers.

• IQ4_XS
  Ultra‑small footprint with surprisingly strong reasoning. Ideal for laptops and 8–12GB GPUs.

• IQ3_M
  Extremely compact. Works best on large models (70B+) and MoE architectures.

🧮 Which Quant Should YOU Choose?

Your hardware determines the best quantization format. Use this table as your quick reference.

Additional Notes

• MoE models (Gemma 4‑26B‑A4B, Qwen 4 MoE, Llama 5 MoE)
  Perform extremely well at Q4_K_M and often tolerate IQ3_M without major degradation.
  MoE routing layers typically remain higher precision automatically.

• Large models tolerate compression better than small ones.
  A 70B model at IQ3_M often outperforms a 7B model at Q5_K_M.

• VRAM fit is king.
  A smaller quantized model running fully in VRAM will always outperform a larger model spilling into slow system RAM.

🧰 Quantization Across Hosting Tools

Different engines support different quant formats:

• Ollama: GGUF only
• LM Studio: GGUF + GPTQ
• vmlx: GPTQ + AWQ
• ExLlamaV2: GPTQ only

This is why downloads sometimes look different depending on your tool.

For a full breakdown of hosting tools, see:
👉 Local AI Hosting Tools

⚖️ The Trade-Off: Speed vs. Smarts

When choosing a quant, follow this simple loop:

1. Fit the model entirely in VRAM.
   This is the single biggest performance factor.

2. Start with Q4_K_M.
   If you have headroom, move up to Q5_K_M or Q8_K_M.

3. Use IQ quants for laptops or tight VRAM budgets.

4. Use Q8_K_M only when you truly need maximum precision.

Next Steps

• Return to the master setup:
  👉 Local AI Setup for Busy Humans

• Explore the architectural alternative:
  👉 Mixture of Experts (MoE) Explained

• Choose your hosting tools:
  👉 Local AI Hosting Tools

• Pick your model:
  👉 Advanced AI Models for Power Users