Eigma
EN 中 DE
Why AI's Builders Asked to Be Restrained
Contents Why Nobody Slows Down Alone

Saying What You Want

Why Nobody Slows Down Alone

In September 2026, Amodei's pacing proposal treated coordination as an obstacle rather than an automatic consequence of shared concern. It described escalating levels of difficulty for cooperation with authoritarian governments. It also said that some coordination among laboratories in democratic countries would be legally challenging without government support.1 The proposal therefore contained its own warning: wanting collective restraint does not make unilateral restraint attractive.

A laboratory may care about safety and still fear that a competitor will continue racing. If slowing sacrifices competitive position while producing little shared protection, continued development can look like the less damaging choice. RAND modeled advanced AI competition as resembling a prisoner's dilemma, in which fear that another actor will cut corners can pressure even a safety-conscious actor to keep pace.2 This is a structural explanation, not a claim that every participant is insincere.

A counter-analysis on the Effective Altruism Forum argues that a Stag Hunt is a better model. In that framing, mutual cooperation and mutual defection can both be stable. When actors trust one another enough, cooperation may become the more attractive outcome rather than a heroic sacrifice.3 The disagreement matters because one model emphasizes the pull toward racing, while the other emphasizes conditions under which coordinated restraint could hold.

The figure illustrates why competitive pressure can sustain racing despite safety concerns, while mutual restraint avoids the competitive loss imposed on an actor who slows alone.

<svg viewBox="0 0 590 380" xmlns="http://www.w3.org/2000/svg" role="img" aria-labelledby="payoff-title payoff-desc" style="font-family: system-ui, sans-serif">
  <title id="payoff-title">Illustrative payoff sketch for mutual and unilateral restraint</title>
  <desc id="payoff-desc">A qualitative matrix showing that mutual restraint avoids unilateral competitive loss, while racing by both actors preserves competitive pressure.</desc>
  <text x="570" y="24" text-anchor="end" font-size="14" fill="currentColor">illustrative</text>
  <text x="385" y="50" text-anchor="middle" font-size="18" font-weight="700" fill="currentColor">Other actor</text>
  <text x="260" y="82" text-anchor="middle" font-size="16" fill="currentColor">Restrains</text>
  <text x="465" y="82" text-anchor="middle" font-size="16" fill="currentColor">Races</text>
  <text x="18" y="170" font-size="16" font-weight="700" fill="currentColor">Actor restrains</text>
  <text x="18" y="290" font-size="16" font-weight="700" fill="currentColor">Actor races</text>
  <rect x="150" y="100" width="205" height="115" fill="none" stroke="currentColor" stroke-width="2"/>
  <rect x="355" y="100" width="205" height="115" fill="none" stroke="currentColor" stroke-width="2"/>
  <rect x="150" y="215" width="205" height="115" fill="none" stroke="currentColor" stroke-width="2"/>
  <rect x="355" y="215" width="205" height="115" fill="none" stroke="currentColor" stroke-width="2"/>
  <text x="252" y="145" text-anchor="middle" font-size="16" font-weight="700" fill="currentColor">Mutual restraint</text>
  <text x="252" y="175" text-anchor="middle" font-size="14" fill="currentColor">Both avoid unilateral loss</text>
  <text x="457" y="145" text-anchor="middle" font-size="16" font-weight="700" fill="currentColor">Unilateral restraint</text>
  <text x="457" y="175" text-anchor="middle" font-size="14" fill="currentColor">Restrainer loses position</text>
  <text x="252" y="260" text-anchor="middle" font-size="16" font-weight="700" fill="currentColor">Other restrains alone</text>
  <text x="252" y="290" text-anchor="middle" font-size="14" fill="currentColor">Actor gains position</text>
  <text x="457" y="260" text-anchor="middle" font-size="16" font-weight="700" fill="currentColor">Mutual racing</text>
  <text x="457" y="290" text-anchor="middle" font-size="14" fill="currentColor">Safety effort is pressured</text>
  <text x="355" y="360" text-anchor="middle" font-size="14" fill="currentColor">Qualitative incentives, not measured payoffs</text>
</svg>

Figure: Unilateral restraint costs position, whereas mutual restraint avoids that loss and mutual racing keeps safety effort under pressure. Schelling's concepts of focal points and credible commitments clarify what cooperation requires. A focal point gives actors a salient option around which to coordinate. A credible commitment constrains an actor's future choices, making its promise more trustworthy to others.4 Common standards might provide a focal point, while enforceable constraints might support commitment. Neither mechanism guarantees trust.

The boundary is that these models diagnose incentives; they do not establish which model perfectly describes frontier AI competition. RAND's race framing and the Stag Hunt counter-analysis disagree about how trapped the actors are. Amodei's proposal identifies possible coordination steps, but it does not show that legal authority, credible commitments, or sufficient trust already exist.1

References

  1. [1]We Must Pace the Frontier
  2. [2]RAND report on competitive dynamics toward advanced AI
  3. [3]The AI race is not a prisoner's dilemma
  4. [4]Academic discussion of The Strategy of Conflict
Quizzes

  1. Two safety-conscious labs both prefer mutual restraint but fear losing position if only one slows. Enforceable constraints can help them escape this trap by creating a ______.

    • credible commitment
    • focal point
    • specification proxy
    • unchecked race

    A credible commitment limits future discretion, giving rivals a stronger reason to believe that an actor will keep its promise.

  2. Which comparison captures how the Stag Hunt counter-analysis differs from the prisoner's-dilemma framing?

    • The prisoner's-dilemma framing emphasizes pressure to race; the Stag Hunt framing allows sufficient trust to stabilize mutual restraint.
    • The prisoner's-dilemma framing centers on legal barriers; the Stag Hunt framing centers on government authorization for restraint.
    • The prisoner's-dilemma framing relies on focal points; the Stag Hunt framing favors unilateral pledges over credible commitments.
    • The prisoner's-dilemma framing starts from shared safety priorities; the Stag Hunt framing explains racing through different safety priorities.

    Both framings recognize strategic pressure, but the Stag Hunt analysis gives sufficient trust a larger role in making mutual cooperation stable.

  3. A coordination trap can arise even when competing actors genuinely care about safety.

    • True
    • False

    Fear that a rival will continue racing can make unilateral restraint costly without requiring dishonesty or indifference to safety.

Comments

No comments yet. Start the conversation.