Search, files and tools

Left alone, a reply is generated from the model's weights. For web search, a tool changes that path in a documented way: the model itself decides, for that request, whether to reach for the tool, and a developer can also force the choice¹². Attached files and other tools follow their own paths, described below, and are not the same mechanism.

The diagram below traces what changes for a search-connected reply: whether the words come from the weights alone or from the weights plus retrieved text the reply can point back to.

graph LR
  Q[Your question] --> D{Search used for this request?}
  D -->|No| W[From weights only]
  D -->|Yes| T[Search text into prompt]
  W --> R[Reply, nothing to cite]
  T --> C[Reply can cite that text]

Figure: search changes where an answer's words come from, not whether the answer can still be wrong.

Vendors document three kinds of tool. Web search reaches current web content and returns citations, and the choice to search is the model's unless a developer forces it: Anthropic lists recent events, current prices and organisations that may have changed as reasons to search, and established facts, mathematics and coding concepts as things to answer directly¹. OpenAI documents the same per-request choice, with search context capped at 128 thousand tokens and a setting that lets a developer force a search². Files arrive in two shapes that behave differently. An indexed file is cut into overlapping chunks of about 800 tokens, embedded, and only the chunks a search matches come back, ten by default, so the model still generates the reply from selected source material rather than returning the file itself³. A plainly attached PDF instead has its text and its page images put into the prompt whole, which can increase token usage⁴. Code execution runs Python in a sandbox and is documented to support running code iteratively, processing uploaded files, and generating certain files and graphs that plain text generation does not produce⁵.

Measurements exist, and they are not measurements of the shipped consumer features. They come from research harnesses on models from 2022 and 2023. Supplied with search results, GPT-4 went from 28.6 to 75.6 percent on a benchmark of questions whose answers change, and from 12.0 to 77.1 percent on the fastest-changing subset⁶. Generating and executing Python rather than reasoning in prose moved word problems with large numbers from 23.1 to 61.2 percent⁷. Neither vendor publishes a before-and-after figure for its own tools, and nothing was located for file search specifically.

The boundary is that a tool changes where an answer's words come from without auditing them. File search is documented to return only the chunks a query matched, not the whole document³, and what comes back, matched or generated, is still text the model produced.