Reinforcement Learning from Human Feedback (RLHF) is a method used to train AI models to produce responses that better match human preferences.
The process usually has three main stages:
- Supervised fine-tuning: A pretrained language model is trained on examples of high-quality human-written responses. This gives the model a useful starting point for following instructions.
- Human preference data: Humans compare multiple responses to the same prompt and indicate which responses they prefer. These comparisons are used to train a reward model, which learns to predict which outputs people are likely to prefer.
- Reinforcement learning: The language model generates responses, and the reward model scores them. An optimization algorithm, traditionally PPO (Proximal Policy Optimization), adjusts the model so that it tends to produce higher-reward responses while keeping it reasonably close to the original model.
The process is repeated over many examples, gradually encouraging behaviors such as following instructions, being helpful, and producing clearer responses.
However, RLHF is not simply “teaching an AI what is true.” The reward model learns patterns from human judgments, so biases, inconsistent ratings, or poorly designed feedback can affect the resulting model.
In simple terms, RLHF combines pretrained knowledge, human preference comparisons, and reinforcement learning to steer an AI model toward responses that humans generally prefer.
