Fine-tuning GPT-2 from human preferences
In 2019, OpenAI researchers led by Daniel M. Ziegler published 'Fine-Tuning Language Models from Human Preferences,' the foundational paper that introduced reinforcement learning from human feedback (RLHF) for natural language tasks. Using GPT-2 as the base model, the team demonstrated that human preference judgments — collected through pairwise comparisons — could effectively fine-tune language models across four tasks: stylistic continuation with positive sentiment, physically descriptive continuation, and summarization on the TL;DR and CNN/Daily Mail datasets. The paper's most influential finding was that models trained with 60,000 human comparisons learned to copy whole sentences verbatim from source text, revealing how labeler heuristics can shape model behavior — a lesson that remains central to modern alignment research.


