Clicked Gallery

What is Model Distillation?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineers used model distillation to shrink the flagship system into a phone-sized assistant.

The reader highlighted one word in the docs. Clicked made the technical term “model distillation” easy to understand:

Explained in three depths

●○○

Overview

▾

Model distillation is training one AI model on another model's answers instead of on raw data. A big teacher model answers huge numbers of examples, and a student model learns to reproduce those answers, picking up most of the skill. It is used to shrink big models into cheap fast ones, and to boost a model using a stronger rival's outputs.

●●○

Detail

▾

The teacher's finished answers become the student's textbook. Run a large model across enormous numbers of prompts, collect its outputs, sometimes including how confident it was across the options, and train the student to reproduce them. Studying finished judgments turns out to be a far more concentrated lesson than studying the raw internet. The classic purpose is compression: the student often keeps a large share of the teacher's ability at a tenth of the size, which is how flagship-level AI ends up running on phones. That differs from quantization, which squeezes the same model into coarser numbers rather than training a new one. The second purpose makes the headlines: improving your own model by distilling a stronger rival's, since a competitor's model can be distilled through nothing but its public answers. That is why AI labs ban the practice in their terms of service and accuse each other of raising students on their teachers.

●●●

Analogy

▾

An apprentice who learns from the master chef's finished decisions instead of from years of cooking school. She does not re-taste ten thousand ingredient combinations; she studies what the master actually plated across ten thousand orders, what got salted, what got sent back, what got paired. The master spent a career learning from raw ingredients; the apprentice learns from the master's answers. The strange result is real: most of the skill transfers without any of the original journey.

AI explanations may contain errors · Not professional advice

Same facts, different vibe — Slang mode 😎

●○○

Overview

▾

Distillation is training a model on another model's answers: the big teacher answers a mountain of prompts, the student learns to match them. Used to shrink your own giant into something phone-sized, or to level up your model off a stronger rival's outputs. Most of the skill, a fraction of the cost. 😎

●●○

Detail

▾

The teacher's answers are the curriculum. Blast a big model with a mountain of prompts, save everything it says, sometimes with its confidence spread across the options, and train your student to match. Skipping the raw internet and learning straight from finished judgments is a shortcut that often keeps most of the ability at a tenth of the size. Use one: shrink your own flagship until it fits on a phone, and no, that is not quantization, which squeezes the same model into coarser numbers instead of raising a new one. Use two, the spicy one: level up YOUR model by distilling someone else's, through nothing but its public answers. Which is why labs write bans into their terms of service and accuse each other of raising students on their teachers, and the receipts are just API responses. 😎

●●●

Analogy

▾

Learning chess by studying a grandmaster's published games instead of grinding a million games yourself. Every move and every choice laid out across thousands of games: you study the decisions, not the decade behind them. Within a year you play a decent imitation of their style. The grandmaster is annoyed, because you built it entirely off games they made public, and no rule says studying someone's games is cheating.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Model distillation is the training of a student model to reproduce the outputs, and often the output distributions, of a teacher model, serving both compression, retaining much of the teacher's capability at reduced size and inference cost, and capability transfer between models. Distillation performed against third-party models via their public interfaces raises contractual and intellectual-property disputes and is commonly prohibited in providers' terms of service.

Want Clicked to explain terms like “model distillation” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required

More from the gallery

What is Quantization?

Storing a model's numbers less precisely — half the memory, barely any loss.

What is Fine-Tuning?

Training a finished model a little more on your own data, and how that differs from RAG.

What is a Transformer (in AI)?

The design behind the T in GPT: every word attends to every other word, all at once.

What is Stochastic Gradient Descent (SGD)?

Millions of dials, one wrongness score — how models actually learn, tiny step by tiny step.

Overfitting vs. Underfitting: What's the Difference?

Memorized the answers vs. never opened the book — and the score gap that exposes both.

What is a Context Window?

How much text an AI can hold in mind at once — and why long chats forget their start.