Instruction following

A pretrained model continues text. It does not, in itself, do what you ask. The ability to follow an instruction is added afterwards, and for one vendor the recipe is published step by step.

OpenAI's InstructGPT paper names three steps: collect demonstrations from human labelers and train a supervised policy on them; collect comparisons between model outputs and train a reward model on those rankings; then optimise the model against that reward model¹. The result is why this path became standard rather than simply making the pretrained model bigger. In the paper's 2022 human evaluation, outputs from the 1.3 billion parameter instruction-tuned model were preferred to outputs from the 175 billion parameter GPT-3, a model one hundred times larger². The same pattern is confirmed at a later scale: OpenAI reports that GPT-4's post-training alignment improved measures of factuality and adherence to desired behaviour³.

The documentation is not symmetrical between vendors. Anthropic describes pretraining and human-feedback techniques at a high level, and no citable source breaks out its instruction tuning as a distinct stage the way the InstructGPT paper does. That is a gap in what is published, not evidence of a different mechanism, and this page does not fill it.

What instruction following actually optimises matters more than the recipe. The paper states that a tuned model will follow the user's instruction "even if that could lead to harm in the real world"². Compliance with what was asked is the target. Being right, and the request being a good idea, are not the same target. The paper also reports an alignment tax: tuning for instruction following caused performance degradations on some public test sets relative to the base model². And whose instructions count is a human choice. The labelers were primarily English-speaking, and the paper says outright that they are not representative of everyone the deployed model affects².

The boundary: tuning does not make the model reliable. The paper's own summary is that InstructGPT "still makes simple mistakes", including accepting instructions with false premises and over-hedging on plain questions². So the practical reading is narrow: the target of this training is compliance with the instruction, and an instruction being followed is not the same as the resulting answer being correct or safe.