Examples beat adjectives

Telling a chatbot to be more concise, more professional, more creative describes a result. Pasting one short example of the output you want is a specimen of it. Both vendors recommend the second, without claiming it outperforms the first. Anthropic asks for three to five relevant, diverse examples wrapped in tags, saying they improve accuracy and consistency¹. OpenAI describes including a handful of input and output pairs as a way to steer a model toward a new task instead of retraining it². The phrase in this page's title is this volume's shorthand and not wording either vendor uses.

The measurement for the handful-of-examples case is old. In the 2020 GPT-3 paper that first measured the technique, the 175-billion-parameter model scored 64.3 percent on closed-book TriviaQA with no examples and 71.2 percent with a few³. That model has been superseded and nothing newer was located for the same comparison. The 2024 measurements that do exist concern a different quantity. On Gemini 1.5 Pro, prompts built from many model-generated worked examples reached 83 percent average success on Big-Bench Hard against 72.1 percent for human-written chain-of-thought prompts⁴. The same study separately measured translation with 997 examples in the prompt, improving over a one-shot baseline on Bemba and Kurdish⁴, and pushed accuracy on a different benchmark past a reported figure for another model using a 125-example prompt⁴. Vendors recommend three to five examples; these separate results test hundreds to thousands.

Why examples work is the surprising part. In the most-cited controlled experiment, replacing the answers inside the examples with randomly chosen wrong ones cost only zero to five percentage points of accuracy, while removing the input-and-answer structure entirely performed close to or worse than no examples at all⁵. What examples mainly supply is the shape of the reply and the range of answers in play, not correct pairings for the model to copy.

The boundary is that more stops helping, and how much depends on the task and the model. In the same 2024 study, summarization quality peaked at about fifty examples and then deteriorated⁴, and in a separate study one vision model gained nothing from long example prompts that substantially helped two others⁶. Stacking examples on faith is not a reliable path to a better reply.