RLHF (Reinforcement Learning from Human Feedback) is a technique for fine-tuning language models by combining reinforcement learning with human feedback. Instead of optimizing solely on static data, the model learns to generate responses that people consider more useful, safe, or coherent. It is one of the key methods that allowed large pre-trained models to be converted into conversational assistants like ChatGPT.
The process usually consists of three phases:
- Supervised fine-tuning: the model is trained with examples of responses written or validated by humans.
- Reward model: annotators compare and rank several responses, and with those preferences, a model is trained to predict which output will be liked more.
- Reinforcement optimization: the main model is adjusted to maximize the reward, usually with algorithms like PPO.
Its importance lies in the fact that it aligns the model's behavior with human expectations that are difficult to encode in rules. As a nuance, it depends heavily on the quality and biases of the annotators, and it is costly, which is why alternatives like DPO or the use of AI to generate feedback have emerged.