Reducing Model Size Without Losing Accuracy: Quantization

A practical deep dive into model quantization — the precision-ladder trick that takes a 14GB model down to 3.5GB, what you lose, what you keep, and the code to measure both.

Six months ago I was trying to put a 13B-parameter model on a client's on-premise box. Not a GPU rack — a single production server with 32GB of RAM and no CUDA. In FP16 the model alone needed 26GB, which left almost nothing for the context, the OS, and the application sharing the same machine. The client, a document-processing company in India, wanted the whole thing to fit, and my first instinct — before any architecture conversation — was the same one you will have: shrink the model.

I tried a smaller model first and the accuracy dropped hard on their legal-document task. Then I tried quantization, and the numbers changed completely. The same 13B model in 4-bit fit in 6.5GB, ran fast enough on CPU, and lost a fraction of a percent on their eval set. The client was skeptical until they saw the before-and-after numbers, and honestly, so was I. Quantization is the single highest-leverage technique for shrinking models, and it is badly misunderstood.

In this article I am going to show you what quantization actually does, the precision-ladder taxonomy, the difference between the main techniques (INT8, GPTQ, GGUF, AWQ, QLoRA), real code to quantize and evaluate, and the failure modes I have collected in production.

What quantization actually does

A neural network's weights are stored as floating-point numbers. A 7B model in FP32 uses 4 bytes per weight — 28GB. In FP16 it is 2 bytes per weight — 14GB. Quantization is the act of mapping those continuous values onto a smaller set of discrete values so each weight needs fewer bits — 8 bits (INT8), 4 bits (INT4), or even less.

The reason it does not destroy accuracy is subtle: most weights in a trained network are redundant. The distribution of weights is concentrated around small values, the network is heavily over-parameterized, and small perturbations to individual weights barely move the output. Quantization exploits exactly that slack. It replaces precision with range: instead of representing 0.0001234567 exactly, you represent "roughly 0.00012," and the network shrugs.

But "roughly" is doing a lot of work. Where you place the discrete values — the quantization grid — is the entire art, and that is where the taxonomy comes in.

The math, in one line. For a given precision, you compute a scale s and (for asymmetric) a zero point z, then map each float weight w to round(w / s) + z. The grid of representable values is whatever s and z define, and the entire design space is deciding those two numbers — globally, per tensor, or per channel. The quality of a quantizer is measured by how small the reconstruction error w - dequantize(quantize(w)) is across the real weight distribution, weighted by how much each weight matters to the output.

The taxonomy: how quantizers differ

Every quantization technique you will read about is a combination of four design choices.

Post-training quantization (PTQ) vs. quantization-aware training (QAT). PTQ takes a trained model and converts it after the fact. It is fast, requires no retraining, and works well down to INT8, with care at INT4. QAT injects the quantization error into the training loop — the model learns to be robust to low precision. More expensive, but it recovers most of the accuracy that naive PTQ loses at 4-bit. The rule of thumb: start with PTQ; reach for QAT only when PTQ eats more accuracy than you can afford.

Symmetric vs. asymmetric. Symmetric quantization centers the range at zero, so the zero point is exactly 0. It is simple and efficient, and it works well when the weight distribution is roughly symmetric around zero — which is true for many layers. Asymmetric uses an offset (the "zero point") so the range can hug a lopsided distribution. Slightly more expensive, meaningfully better on skewed layers like activations.

Per-tensor vs. per-channel. Per-tensor quantization uses one scale for a whole tensor — cheap, but a single outlier weight forces a coarse grid for everything. Per-channel quantization gives each channel its own scale, which preserves far more accuracy for a small overhead. It is why modern INT8 pipelines almost always go per-channel.

Calibration. PTQ needs a calibration set — a few hundred examples that represent the data the model will actually see. The quantizer runs the model, records activation ranges, and sets the grid from what it observes. Calibration data quality is the #1 silent killer of quantized models, and I will come back to it in the failure modes.

The precision ladder: what the numbers mean

Here is the mental model I use, with a 7B model as the running example:

The modern serving stack is a ladder you climb one rung at a time: train or fine-tune in BF16, serve in INT8 when you have a GPU with INT8 kernels, and drop to INT4 when memory or CPU is the constraint. Each rung trades a little accuracy for a lot of memory and speed.

The four techniques you will actually meet

INT8 (via ONNX Runtime or TensorRT). The safe default on GPUs. INT8 kernels are well supported, the accuracy hit is usually tiny, and you can often go per-channel with calibration and barely notice a difference. This is where I tell beginners to start.

GPTQ. A post-training method that quantizes to 4-bit by solving a per-layer optimization problem — it minimizes the reconstruction error of the layer outputs given the weight rounding. The result is a dense 4-bit model with strong accuracy. GPTQ is best when you control the calibration set and want one accurate 4-bit artifact.

GGUF. The format, not a quantizer — a container used by llama.cpp that stores weights in various quantization schemes (Q4_K_M, Q5_K_M, and so on) optimized for CPU inference. GGUF is the format you download from Hugging Face when you want to run a model locally on a laptop or a CPU box. The K-quants (Q4_K_M etc.) mix high- and low-bit blocks so important groups keep more precision.

AWQ. Activation-aware weight quantization. Instead of minimizing weight reconstruction error, AWQ protects the small fraction of weights that matter for the activations that produce the output. It scales down the weights tied to important activation channels before quantizing. On many models AWQ beats GPTQ at the same bit-width, especially in low-data or domain-specific settings.

QLoRA. The fine-tuning cousin: you keep the base model in 4-bit and train a set of small low-rank adapters on top, using the 4-bit weights only as a frozen anchor. This is not a serving trick — it is a way to fine-tune a model on one GPU that could never hold the full-precision version.

A working example: 4-bit loading and evaluation in Python

Let me show you the two things you will actually do: load a model in 4-bit, and measure whether the quantization hurt. Measuring is the part most tutorials skip, and it is the part that keeps you honest.

First, load with bitsandbytes:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-7b-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=dict(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_quant_type="nf4",      # normalized float 4 — good default
        bnb_4bit_use_double_quant=True,
    ),
    device_map="auto",
)

Second, the part that matters — an eval harness. You compare the full-precision model and the quantized model on the same fixed set of examples and look at three numbers: token-level accuracy, output length, and a semantic score. If you only check loss, you will miss that a 4-bit model can lose its formatting behavior while keeping its loss almost identical.

from datasets import load_dataset
import torch

eval_set = load_dataset("your-task-set", split="test").select(range(200))

def evaluate(model, tokenizer, samples):
    correct, total = 0, 0
    for s in samples:
        prompt = s["input"]
        gold = s["label"]
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
        with torch.inference_mode():
            out = model.generate(**inputs, max_new_tokens=64)
        pred = tokenizer.decode(out[0][inputs.input_ids.shape[1]:])
        correct += int(normalize(pred) == normalize(gold))
        total += 1
    return correct / total

fp16_acc = evaluate(fp16_model, tokenizer, eval_set)
int4_acc = evaluate(model, tokenizer, eval_set)
print(f"FP16 accuracy: {fp16_acc:.3f} | INT4 accuracy: {int4_acc:.3f}")
print(f"Memory: FP16 {size_gb(fp16_model):.1f}GB -> INT4 {size_gb(model):.1f}GB")

That is the whole discipline: quantize, evaluate on your real task, and compare. The day you skip the eval is the day a 4-bit model silently starts mangling your domain's terminology and nobody notices until a customer does.

Production reality: the failure modes

Bad calibration data is the silent killer. I quantized a model for a legal client with a generic calibration set, and the model started renaming court documents. The calibration data must look like production data — same domain, same format, same distribution of topics. Re-run calibration on a sample of the actual workload, not on the demo examples you grabbed from the model card.

Unstructured low-bit losses are real but subtle. A 4-bit model rarely collapses. It drifts. It becomes over-confident, loses edge cases, and misplaces terminology. If your task has zero tolerance for drift — medical codes, contract clauses — you may not be able to afford 4-bit at all, and you should measure that before you build on it.

Integer kernels are not universal. An INT8 model only speeds up on hardware that actually has INT8 kernels (most modern GPUs). On plain CPU, INT8 inference may be no faster than FP16 — the speedup comes from the kernel, not from the smaller file. Measure latency on the target hardware, not the marketing page.

Compatibility churn. GPTQ files need matching kernels, GGUF needs the right llama.cpp version, and a quantized file from one framework often does not run in another. Pin your versions. A working 4-bit model is worthless if your serving stack drifted two versions and silently falls back to slow FP32.

Small models quantize worse. A 13B model survives 4-bit much better than a 1.5B model. When your model is small, the redundancy slack is thinner, and quantization eats proportionally more of it. For small models, prefer INT8 or a careful QAT pass.

The "two quantizations" trap. A model has weights and activations, and a surprising amount of accuracy loss comes from quantizing the activations, not the weights. Some stacks quantize weights to INT4 but keep activations in FP16 (like QLoRA-style serving), and that combination is far more stable than quantizing both. If your eval shows an unexplained drop, check whether the activations are also being rounded — and if so, whether you even need that.

What about mixed precision and stacking with pruning?

Two questions I get asked constantly deserve short answers.

Can I mix precisions? Yes, and you usually should. Mixed-precision serving keeps some layers at a higher bit-width and drops the rest lower. The greedy version — measure each layer's sensitivity to quantization on a calibration set, then give the sensitive layers more bits and the robust ones fewer — recovers most of the accuracy of full FP16 at most of the memory savings of INT4. The only catch is that mixed-precision formats need kernel support on your runtime, which narrows your serving options. When the runtime supports it, it is the best accuracy-per-byte you can buy.

How does quantization relate to pruning? They compose, and the order matters. Quantization compresses each weight's precision; pruning removes weights entirely. You can prune to 40% sparsity and then quantize to INT4 and get the benefit of both — the memory roughly halves twice. But do it in the right order: prune first, because pruning changes the weight distribution and invalidates a quantizer calibrated on the dense model. I cover pruning in a companion piece, but the short version is: prune, retrain, then quantize. Never the reverse.

When NOT to quantize

• When accuracy is untestable. If you cannot build an eval set that reflects production, you cannot verify quantization. Do not quantize blind.
• When you have GPU headroom. If the model fits in memory and latency is acceptable, quantization buys you nothing but risk. I have seen teams quantize "because it is modern" and then debug phantom accuracy bugs for a week.
• When the domain is low-resource. Specialized terminology, unusual formatting, and languages with sparse data make the model lean harder on its weights — exactly the redundancy quantization removes.

The practitioner checklist

• Confirm the model fits in memory at FP16/FP32 before touching quantization
• Build an eval set from real production data (200+ samples)
• Record FP16 baseline: accuracy, latency, memory
• Start with INT8 (per-channel, calibrated) and measure the delta
• Drop to INT4 (GPTQ/AWQ/GGUF) only if the INT8 accuracy holds and memory still pinches
• Calibrate on data that matches production — never the demo examples
• Test on the actual target hardware (CPU vs GPU kernels differ)
• Pin quantizer and runtime versions in your lockfile
• Re-run the eval on every model version you serve
• Document the accepted accuracy loss as a decision, not an accident

The takeaway

The client's 13B model runs on that 32GB box today in 4-bit, and the legal eval is within a fraction of a percent of the full-precision version. Quantization did not make the model smaller by making it dumber; it made it smaller by removing the precision it never needed in the first place.

The method is always the same: climb the ladder one rung, measure on real data, and stop at the first rung that fits. Fourteen gigabytes to 3.5 is a lovely headline. The eval harness is the part that keeps it honest.

*Gulshan Yad

Quantization Trade-Offs

Quantization involves a trade-off between model size and accuracy. As the bitwidth of the weights and activations decreases, the model size reduces, but the accuracy may also decrease. This trade-off is not always linear, and the optimal bitwidth may depend on the specific model and dataset.

Quantization Methods for Weights

There are several methods for quantizing weights, including uniform, k-means, and learned quantization. Uniform quantization involves dividing the dynamic range of the weights into equal intervals, while k-means quantization involves clustering the weights into k clusters. Learned quantization involves training a separate model to predict the optimal quantization points for the weights.

Quantization Methods for Activations

Activation quantization can be performed using methods like ternary quantization, where activations are represented as -1, 0, or 1. This can significantly reduce the model size, but may also reduce the accuracy.

Quantization-Aware Training (QAT)

Quantization-aware training (QAT) involves training the model with quantized weights. This can help to improve the accuracy of the quantized model and reduce the impact of quantization on the model's performance.

Post-Training Quantization (PTQ)

Post-training quantization (PTQ) involves applying quantization after the model has been trained. This can be a simpler and more efficient approach than QAT, but may not offer the same level of accuracy.

Quantization for Edge Devices

Quantization is particularly useful for models deployed on edge devices, where memory and computational resources are limited. By reducing the model size, quantization can help to improve the performance and efficiency of edge devices.

Quantization and Model Interpretability

Quantization can make it more challenging to interpret model weights and activations, as the reduced precision may obscure underlying relationships and patterns. This can be a challenge for developers who need to understand and debug the model's behavior.

Key Takeaways

• Quantization reduces model size by representing weights and activations as smaller integers, preserving the original dynamic range.
• Weight quantization methods include uniform, k-means, and learned quantization, each with varying trade-offs between accuracy and model size.
• Activation quantization can be performed using methods like ternary quantization, where activations are represented as -1, 0, or 1.
• Quantization-aware training (QAT) and post-training quantization (PTQ) are two approaches to reduce model size without sacrificing accuracy.
• QAT involves training the model with quantized weights, while PTQ applies quantization after training has completed.