Sampling and divergent runs

Run the same prompt twice against the same repository and you can get two different pieces of work. This is not a setting you have failed to find. Anthropic states that even at the lowest temperature "the results will not be fully deterministic", and on models released after Opus 4.6 the knob is gone rather than discouraged: any temperature other than the default is rejected outright.¹ No vendor documents which sampling settings the coding agents actually use, and none exposes temperature specifically, so any specific temperature figure quoted for one of these products is unknown rather than documented.

One experiment measured the mechanism, on one open model and serving stack, and found it was not the sampler. Thinking Machines Lab ran one prompt a thousand times through an open model at temperature zero and got eighty distinct completions; the first hundred and two tokens were identical in every run, and the split happened at the next token, where 992 runs went one way and 8 went another.² Their diagnosis is that server load makes batch size vary unpredictably, and that common kernels are not batch-invariant, so a request's arithmetic depends on what happened to be batched alongside it. Underneath sits floating-point non-associativity: adding 0.1 to a very large number and then subtracting that number gives zero, while subtracting first gives 0.1.² Determinism was achievable with batch-invariant kernels, at roughly a 60 percent slowdown.²

The figure traces the measured split after an identical prefix, showing that the same prompt and settings can still produce divergent runs.

<style>
  .div-wrap { font-family: system-ui, sans-serif; color: currentColor; }
  .div-wrap svg { width: 100%; height: auto; }
  .trunk, .branch { fill: none; stroke: currentColor; stroke-width: 2; }
  .branch-b { opacity: 0.55; }
  .tick { stroke: currentColor; stroke-width: 1.2; stroke-dasharray: 4 4; opacity: 0.7; }
  .cap { font-size: 14px; fill: currentColor; opacity: 0.8; }
  .lab { font-size: 15px; fill: currentColor; }
  .head { font-size: 16px; font-weight: 600; fill: currentColor; }
  .dot { fill: currentColor; }
  .dot-a { animation: run-a 4.2s linear infinite; }
  .dot-b { animation: run-b 4.2s linear infinite; }
  @keyframes run-a {
    0%   { transform: translate(0px, 0px); opacity: 0; }
    6%   { opacity: 1; }
    45%  { transform: translate(230px, 0px); }
    72%  { transform: translate(500px, -52px); opacity: 1; }
    100% { transform: translate(500px, -52px); opacity: 1; }
  }
  @keyframes run-b {
    0%   { transform: translate(0px, 0px); opacity: 0; }
    6%   { opacity: 1; }
    45%  { transform: translate(230px, 0px); }
    72%  { transform: translate(500px, 52px); opacity: 1; }
    100% { transform: translate(500px, 52px); opacity: 1; }
  }
  @media (prefers-reduced-motion: reduce) {
    .dot-a, .dot-b { animation: none; }
  }
</style>
<div class="div-wrap">
  <svg viewBox="0 0 700 230" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="One prompt sampled twice: the runs share an identical prefix and then split at one token">
    <title>Two runs of one prompt, diverging at a sampling step</title>
    <text x="16" y="26" class="head">One prompt, sampled twice</text>
    <text x="16" y="46" class="cap">Same request, same settings, one thousand runs measured</text>
    <path class="trunk" d="M 40 140 L 270 140" />
    <line class="tick" x1="270" y1="80" x2="270" y2="200" />
    <text x="270" y="72" class="cap" text-anchor="middle">the split</text>
    <path class="branch" d="M 270 140 L 540 88" />
    <path class="branch branch-b" d="M 270 140 L 540 192" />
    <circle class="dot dot-a" cx="40" cy="140" r="6" />
    <circle class="dot dot-b" cx="40" cy="140" r="6" />
    <text x="40" y="168" class="cap">identical prefix</text>
    <text x="548" y="84" class="lab">992 runs</text>
    <text x="548" y="102" class="cap">one continuation</text>
    <text x="548" y="188" class="lab">8 runs</text>
    <text x="548" y="206" class="cap">another continuation</text>
    <text x="16" y="222" class="cap">Eighty distinct completions in total. Nothing in the request changed between runs.</text>
  </svg>
</div>

Figure: At temperature zero, all runs shared 102 tokens before the next token split 992 to 8. Other vendors describe the same residue. OpenAI's seed parameter is documented as a best-effort attempt at deterministic sampling, explicitly not a guarantee, and their own measurement shows the average distance between repeated responses shrinking rather than reaching zero.³ Adaptive thinking adds a second, independent source: the model decides how much thinking to spend per request, so that budget also moves between runs.⁴ Vendors treat the variation as a measurement problem in their own results, averaging a benchmark over ten trials rather than reporting one.⁵ The documented practical response is process, not configuration: checkpoints at every turn, and rewinding conversation, code or both when a run goes somewhere you do not want.⁶

The limits of this page are worth holding onto. The thousand-run experiment used one open model on one serving stack; no equivalent measurement exists for a commercial endpoint, and no vendor confirms or denies batch-invariance in its own serving. Nobody has published how far two runs of the same coding task diverge in what they actually change. Divergence is documented; its size on your work is not.