What is RLHF (Reinforcement Learning from Human Feedback)?

RLHF, or reinforcement learning from human feedback, is the training step where people judge a model's answers and the model is adjusted toward the ones they preferred. It is what turns a system that predicts text into one that answers questions helpfully. It is also why models are polite, why they refuse certain requests, and why they sometimes agree with you when you are wrong.

A freshly trained model has learned exactly one skill: given some words, guess which words usually come next. That is not the same as being useful. Ask an early one how to fix a flat tyre and it might reply with a list of similar questions, because that is what often follows a question online. Writing rules for what a good answer looks like fails immediately, since nobody can define helpful precisely enough to code it, and more text does not help because the internet holds few examples of an ideal assistant. So the job goes to people. Raters pick the better of two candidate answers, thousands of times, and those choices train a second model to score answers the way they did. The main model is then fine-tuned to earn high scores from that scorer. The catch is that it learns what raters approved of rather than what is true, so it can drift toward flattery, refuse harmless requests that merely resemble risky ones, and inherit the assumptions of whoever did the rating.

Teaching someone to cook by having them taste, rather than handing them recipes. You cannot write down what delicious means in a way that survives contact with a real kitchen, so instead they cook, you taste, and you say this one, more salt, less time. After enough rounds they develop judgment no recipe could have given them. But they have learned your palate specifically, so if you happen to love salt, you have trained a cook who oversalts and is quietly certain that is correct.