Meaning as position

Ask what a word means to one of these systems and the honest answer is a position. A word is given a list of numbers, and how close two of those lists are is what stands for how related the two words are. A product sold for exactly this purpose states the rule plainly: the distance between two of these lists measures their relatedness, with smaller distances meaning greater similarity.¹

Nobody types the positions in. The founding paper on word vectors learned them by prediction. One of its two designs predicts a word from the words around it, the other predicts the neighbours from the word.² Position falls out of which words turn up near which, the property the paper reports as "similar words tend to be close to each other".² The space also holds relationships as directions and not only as closeness, which is what the paper's arithmetic on king, man, woman and queen demonstrates.² This part is old and cheap: those vectors were trained on 1.6 billion words in less than a day, in a 2013 paper.² The mechanism belongs to the literature. OpenAI does not publish how it trains the embedding models it sells, so it stays attributed to the literature and not to that product.

Two objects must be kept apart, because merging them is the usual mistake. The first is the static table just described, one fixed list of numbers per token, only the input stage inside a transformer and carrying nothing about where in the sentence a token sits, which is why order is added separately.³ The second is what the model computes after that: self-attention relates the positions of a sequence to each other,³ so what the model holds for a token is a function of the other tokens present. Later work made the contrast explicit by modelling polysemy, one word carrying different internal representations in different contexts,⁴ with BERT documenting this conditioning through all layers.⁵

The boundary. The map is a static word table and nothing more. A transformer does start by converting each incoming token to exactly this kind of fixed starting vector,³ so the table is consulted, but that vector is not the model's finished, context-aware representation, and not a dictionary the model looks up as the last word on meaning. Anthropic's interpretability work, inside a deployed model, finds something geometric but spread out: each concept spans many neurons and each neuron serves many concepts.⁶ There is no cell holding a word. The animation below draws that flat map, one word settling near its neighbours; read it as that fixed starting picture, not as a dictionary a chat model consults, since computation moves beyond it the moment it begins.

<style>
  .wmap{position:relative;height:264px;border:1px solid rgba(120,130,160,0.45);border-radius:10px;font-family:system-ui,sans-serif;color:#3a4151;overflow:hidden}
  @media (prefers-color-scheme:dark){.wmap{color:#e6e9f2;border-color:rgba(160,170,200,0.45)}}
  .w{position:absolute;font-size:15px;padding:2px 6px;border-radius:6px;background:rgba(130,150,200,0.16);white-space:nowrap}
  .mover{left:330px;top:70px;background:rgba(130,150,200,0.34);animation:travel 9s ease-in-out infinite}
  @keyframes travel{0%,10%{left:40px;top:196px}45%,100%{left:330px;top:70px}}
  .hint{position:absolute;right:12px;top:10px;font-size:12px;opacity:0.75}
  .cap{position:absolute;left:12px;bottom:8px;font-size:12px;opacity:0.75}
</style>
<div class='wmap'>
  <span class='w' style='left:300px;top:38px'>king</span>
  <span class='w' style='left:398px;top:80px'>man</span>
  <span class='w' style='left:326px;top:116px'>woman</span>
  <span class='w' style='left:40px;top:150px'>tokenization</span>
  <span class='w' style='left:132px;top:196px'>encoding</span>
  <span class='w' style='left:44px;top:234px'>annoyingly</span>
  <span class='w mover'>queen</span>
  <span class='hint'>queen settles next to king</span>
  <span class='cap'>Illustrative. A static table of word positions, not the inside of a chat model.</span>
</div>

Figure: Related words occupy neighbouring positions, and queen settles beside king; this is the input table the model starts from.