Clicked Gallery

What is Quantization?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

The 4-bit quantization cut memory use enough to run the model on a single consumer GPU.

The reader highlighted one word in the docs. Clicked explained the technical term “quantization” in simple terms:

Explained in three depths

●○○

Overview

▾

Quantization shrinks an AI model by storing its numbers with less precision. Each number takes less memory, so the model runs on smaller, cheaper hardware. Done well, quality barely drops.

●●○

Detail

▾

A model is billions of numbers, called weights, normally stored at 16 bits each. Quantization re-encodes them at 8 or 4 bits, so a 70-billion-weight model drops from about 140 gigabytes to roughly 35 at 4 bits. The full version doesn't fit even on a data-center GPU, while the shrunken one squeezes onto a single high-end consumer graphics card. The trick works because weights cluster in narrow ranges, so a well-chosen coarse scale loses little: benchmark scores typically dip a few percent at 8 bits and stay usable at 4. Below that it gets rough, and 2-bit versions are often noticeably worse. Speed usually improves as well: the slow part of running a model is fetching all those numbers from memory, and smaller numbers mean less to fetch. This is the main reason local AI exists at all, letting people run yesterday's frontier models on gaming hardware.

●●●

Analogy

▾

Shrinking a photo to send it. The original file is huge; saving it at a lower quality makes the file a tenth of the size, and on a phone screen nobody can tell. Zoom in far enough and you will find the smudges, but for everyday viewing the picture does its job.

AI explanations may contain errors · Not professional advice

Same facts, different vibe — Slang mode 😎

●○○

Overview

▾

Quantization is putting a model on a diet: same brain, numbers stored sloppier. Half or a quarter of the memory, and it barely notices. 😎

●●○

Detail

▾

A model is just billions of numbers, and stock ones burn 16 bits per number. Requantize at 4 bits and a 70B model shrinks from about 140 GB to about 35, which is the line between not fitting on a data-center GPU and running on your gaming rig. The numbers huddle in tight ranges anyway, so coarse rounding costs shockingly little: a few percent on benchmarks at 8-bit, still solid at 4. Push to 2-bit and the wheels start wobbling. Bonus: it usually runs faster, since the slow part is fetching numbers, not crunching them. This one trick is basically why local AI is a hobby instead of a fantasy. 😎

●●●

Analogy

▾

MP3s. Nobody stores albums as raw studio files; you keep a tenth of the data and your earbuds can't tell the difference. Audiophiles swear they hear the loss, blind tests say barely, and meanwhile the whole library fits in your pocket.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Quantization maps model parameters from higher-precision floating-point representations to lower-bit-width formats, reducing memory footprint and bandwidth requirements at inference. Post-training quantization and quantization-aware training trade implementation cost against accuracy retention, with degradation generally modest at 8-bit and increasingly significant below 4-bit.

Want Clicked to explain terms like “quantization” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required

More from the gallery

How to actually read a research paper

Skim smart, break the jargon wall, and keep what you learned — one session, done right.

What is Stochastic Gradient Descent (SGD)?

Millions of dials, one wrongness score — how models actually learn, tiny step by tiny step.

What is Backpropagation?

How one final error becomes a precise adjustment for millions of settings — Goal Seek at scale.

What is Technical Debt?

Shortcuts and aging systems that bill the future — principal, interest, and repayment.

What is Retrieval-Augmented Generation (RAG)?

Make the AI answer from your documents — retrieval first, generation second, receipts included.

What are Vector Embeddings?

Meaning turned into coordinates — how machines find what's similar without matching a single word.