Learning from text at scale

The ability to answer you is built in two stages, and the companies that build these systems describe the two stages very unequally.

The first stage is pretraining. The model is fed a very large amount of text and trained to predict the next token in a document, where a token is a chunk of text a word or shorter. OpenAI describes GPT-4 as a Transformer-style model pre-trained to do exactly that, on "publicly available data (such as internet data) and data licensed from third-party providers"¹. Anthropic describes its own first stage the same way, as pretraining on large diverse data to acquire language capabilities through methods like word prediction, and says it cleans and filters that text using deduplication and classification². OpenAI's disclosure under California law puts the scale in the broadest terms available: its systems are developed on datasets containing trillions of tokens of text, image, audio and audiovisual content, with no specific datasets or proportions named³.

Pretraining alone does not produce an assistant. It produces something that continues text. Anthropic describes a second stage, post-training, where human feedback techniques are applied separately from pretraining to elicit helpful, harmless and honest responses². Fluency and general competence come from the first stage. The human-feedback techniques applied in the second stage are documented as a distinct part of training, separate from next-token prediction.

The diagram below lays out the two stages in order, pretraining then post-training, and what each is documented to add, because the two labelled boxes look symmetrical even though the disclosure behind them is not.

graph TD
  A["Large text mix"] --> B["Pretraining"]
  B -->|"adds language and patterns"| C["Base model: continues text"]
  C --> D["Post-training"]
  D -->|"adds human-feedback shaping"| E["The assistant you chat with"]
  B -.-> F["Sources given only<br/>as broad categories"]
  D -.-> G["Methods named,<br/>the mix not published"]

Figure: Pretraining and post-training in sequence; the second stage is documented in far less detail than the first.

The second stage is far less disclosed than the first. OpenAI declined to give details of GPT-4's architecture, training compute, hardware or dataset construction, citing the competitive landscape and the safety implications of large models¹. Stanford's 2025 transparency index records Anthropic marking its public-dataset list, its licensed sources, the compensation for them, and the language and domain composition of its data as proprietary and not disclosed⁴.

The judgement stops at the level of stages. The two-stage description, pretraining then separately applied human-feedback techniques, is how Anthropic describes its own process²; OpenAI documents pretraining in comparable depth but does not offer the same two-stage account of what follows it. Neither company publishes the proportions of its text mix, and no primary source gives a total corpus size for any current model. What is documented is the broad category labels each vendor gives its sources; exact datasets, proportions and stage-specific composition are what remain undisclosed.