Reinforcement Learning from Human Feedback
Aussi : RLHF, Reinforcement Learning from Human Feedback, Human preference fine-tuning
A training method that fine-tunes AI models using human preference judgments, aligning their outputs with what people find helpful, safe, and high quality.
What It Is
Reinforcement Learning from Human Feedback (RLHF) is a technique for aligning machine learning models, especially large language models, with human preferences. Instead of relying only on raw training data, RLHF incorporates human judgments about which model outputs are better, then uses those judgments to steer the model toward more desirable behavior.
RLHF combines two fields: supervised learning (learning from labeled examples) and reinforcement learning (learning from a reward signal). The key insight is that humans are often better at *comparing* two answers than at writing the perfect answer from scratch, so RLHF turns human preferences into a reward that the model can optimize.
Why it matters
Models trained purely to predict text can be fluent yet unhelpful, evasive, or unsafe. RLHF helps close the gap between what a model *can* produce and what users actually *want*. It is a major reason modern conversational assistants feel cooperative, follow instructions, and refuse clearly harmful requests.
- Alignment: outputs better match human intent and values.
- Quality: responses become more helpful, concise, and relevant.
- Safety: reduces toxic, biased, or dangerous content.
How it works in practice
RLHF typically follows three stages:
1. Supervised fine-tuning (SFT): humans write or curate example responses, and the base model is fine-tuned on them.
2. Reward model training: humans rank or compare multiple model outputs for the same prompt. These comparisons train a separate reward model that scores how good any response is.
3. Policy optimization: the main model is fine-tuned with a reinforcement learning algorithm (commonly PPO) to maximize the reward model's score, often with a constraint that keeps it close to the original model.
Concrete Example
Suppose a chatbot is asked, "Explain inflation to a beginner." The model generates four answers. Human reviewers rank them from best to worst, favoring clear, accurate, jargon free responses. The reward model learns that pattern. During optimization, the chatbot is nudged to produce answers resembling the top ranked ones. Over many prompts, it consistently gives clearer, more helpful explanations.
Limitations
RLHF inherits human biases, can be costly to label, and may reward confident sounding but wrong answers if reviewers are not careful. Variants like RLAIF (using AI feedback) and direct preference optimization aim to reduce these costs.