OpenAI's API: From Chat Completions to Responses

Most posts in this series are about something that lives inside a model or underneath one. This one is about the thin layer on top: the HTTP surface OpenAI gives you to talk to its models, and the way that surface has rebuilt itself three times in five years. Four distinct APIs have accumulated over that stretch, each layered in without fully retiring the last.

Once you see the pattern behind the evolution, the framing around "Responses is agent-native" starts to look like concrete engineering. Every generation of the API pulled more of the agent loop off your machine and onto OpenAI's servers.

Provider APIs are where mental models meet HTTP

By the time a turn reaches api.openai.com, the model is a sealed box behind a URL. You don't get to touch the KV cache, the sampler, the chat template. What you do get is a request schema and a response schema, and those schemas have opinions. They decide which parts of the agent loop are your problem and which parts are OpenAI's.

The throughline I want you to carry is this: every new generation of the OpenAI API is a server-side state migration. In Completions you held everything. In Chat Completions you still held almost everything but the roles got named. In Assistants, OpenAI started holding conversations and files for you in exchange for a complicated state machine. In Responses (March 2025), they kept the server-side state but collapsed the state machine back into one endpoint that actually feels like the single-shot call of Chat Completions.

The bar at the top of the hero animation is the whole plot. Watch where the green bar grows. That's the migration.

Prereqs

This post sits late in arc 11 and leans on earlier posts pretty hard. You don't need to re-read them, but the intuitions you should already have:

• Function calling as structured generation from function-calling. You know that tool_calls in a response are JSON blobs the model emits, not the model running code. You know the host is the thing that actually dispatches to a Python function.
• The agent loop from the-agent-loop. The model -> runtime -> tool -> resume cycle is in your head, and the word "runtime" means "whoever is in charge of the loop."
• Transcript formats from transcript-formats. You know that OpenAI's wire shape isn't Anthropic's isn't Harmony, and that the provider API you call is just one surface onto a model's underlying transcript format.

If any of those feel fuzzy, the rest of this post will mostly make sense but the misconceptions section won't land.

The minimal formalism: a request is a JSON blob

Stripping all four APIs to their wire level, a request is always "here are some messages or some text, here are maybe some tools you can call, please give me back a next thing." The differences are in the shape of the JSON.

The Chat Completions endpoint takes a messages array: a list of objects with a role (system, user, assistant, tool) and a content field, plus an optional tools array of JSON-schema function definitions. It returns a choices[0].message object that either has content (an answer) or tool_calls (a list of intents).

The Responses endpoint takes a shorter input (a string or a list of typed content items) and an optional conversation id that points at a server-held history. Its tools array accepts both user-defined function tools and built-in tool types like web_search, file_search, and computer_use. It returns an output array where each item has a type: message, function_call, web_search_call, reasoning, and so on.

The two shapes look similar but aren't. One is a transcript you rebuild on every call. The other is a short delta against state the server already has.

POST /v1/chat/completions
{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are helpful." },
    { "role": "user", "content": "Weather in Tokyo?" },
    { "role": "assistant", "tool_calls": [ ... ] },
    { "role": "tool", "tool_call_id": "...", "content": "..." }
  ],
  "tools": [
    { "type": "function", "function": {
        "name": "get_weather",
        "parameters": { ... }
    } }
  ]
}
 
// response: choices[0].message.tool_calls

POST /v1/responses
{
  "model": "gpt-4o",
  "conversation": "conv_abc123",
  "input": "Weather in Tokyo?",
  "tools": [
    { "type": "web_search" },
    { "type": "file_search", "vector_store_ids": [...] },
    { "type": "function", "name": "get_weather",
      "parameters": { ... } }
  ],
  "reasoning": {
    "effort": "medium"
  }
}
 
// response: output[] — a list with
//   output_text, function_call, web_search_call,
//   reasoning, ...

The right column is what you're looking at if you start a new OpenAI project in 2026. The left column is what a lot of codebases have been living in since 2023. Both still work. They are not the same object, and knowing which one you're on matters every time you read the docs.

The server-side state migration, literally

Let me be concrete about what "more state on the server" means, because it's the single most useful way to hold the four generations in your head. The diagram below lists the pieces of a running agent conversation down the left, and which side owns each piece across the four generations along the top.

A few things to pull out of this.

The system prompt and the message history are the first to migrate. In Chat Completions you resend the entire message history on every call. The model doesn't need you to remind it what was said; the server does, because the server is stateless. Assistants fixed this by introducing server-side threads. Responses kept threads under a new name, conversation, and made them easier to use.

Tool schemas migrated later and only partially. In Responses you still send your user-defined function schemas on each call, because those are part of your app's code and OpenAI has no business storing them. But the built-in tool types (web_search, file_search, computer_use) are effectively schemas OpenAI already knows about. You just say {"type": "web_search"} and the server fills in the rest.

Reasoning traces never existed on the client side. For o1, o3, o4-mini and their descendants, the chain of thought is on the server by construction, because it's used as input to the final answer and (on OpenAI specifically) never surfaced verbatim; you get a short summary and a server-held reasoning id to reference on the next turn. Other providers make different calls here (Anthropic returns signed thinking blocks, Gemini returns an opaque thoughtSignature), but on OpenAI the full trace stays on the server. That's why the "reasoning trace" row is blank for the first two generations and solid green for the last two: the concept didn't exist, and when it did, it was never going to be client-held.

The handle to take away: "server-side state migration." It's the one phrase that predicts almost every change across the four generations. If you know a conversation now has an id, you already know the messages don't need to be resent. If you know reasoning is server-held, you already know why you can't just edit o3's thoughts.

A short history, because each generation solved a real problem

The easy version of this story is "OpenAI kept shipping new APIs." What actually happened is each generation was invented to fix the thing that broke in the one before it.

Completions (2020) gave you a prompt string in and a completion string out. There were no roles, no messages, and no tools. You built a chatbot by manually concatenating "User: hello\nAssistant:" and hoping the model stopped at the right newline. Tool calling didn't exist, and the "chatbot" concept wasn't baked into the surface. This is why old OpenAI playground screenshots look like you're writing a screenplay. The endpoint still exists for legacy and for base models, but it's been effectively frozen since 2023.

Chat Completions (March 2023) named the roles. system, user, assistant, and later tool became first-class. You stopped concatenating role prefixes and started sending an array of {role, content} objects. The model was fine-tuned to honor that structure. Then in June 2023, OpenAI added function_call (renamed to tool_calls that November) and the modern agent loop was born. This endpoint is the workhorse of the whole ecosystem. Every LangChain, every LlamaIndex, every homegrown script written between 2023 and 2025 is calling /v1/chat/completions. Almost every other provider's primary endpoint is shaped like it. It is the 2026 default if you don't have a specific reason to use something else.

Assistants API (beta, late 2023) was the first attempt at server-side state. It introduced assistants (a persistent config bundle), threads (server-held message histories), runs (an async state machine for executing a turn), messages, and tools including a code interpreter and file search. On paper this was a great idea. In practice the state machine was complicated, the latency was high because you were polling a run's status instead of reading a streaming response, and file handling had its own separate API. A lot of developers tried it and went back to Chat Completions. Assistants is deprecated as of 2026 in favor of Responses, with a migration guide that's mostly a set of find-and-replaces.

Responses API (March 2025) kept the good parts of Assistants and threw out the rest. The state machine collapses back to a single endpoint that returns when it's done (with streaming available). Threads became conversation objects. Built-in tools (web_search, file_search, computer_use) are now first-class tool types. Reasoning got its own field. The response is a typed output[] array instead of a flat message. It's the one API OpenAI is actively evolving.

Each generation added one primitive the previous one lacked: roles, then tool calls, then server-side threads, then first-class built-ins and reasoning. The direction is always toward more loop-state living on the server.

What you have to reason about

The JSON shape is a small part of what you deal with when you're shipping against this API. The rest is the runtime behavior of the endpoint itself: rate limits, streaming, model selection, and (for o-series) how much hidden reasoning you're paying for.

Rate limits are per-model and come in two flavors, RPM (requests per minute) and TPM (tokens per minute). The TPM limit counts input plus output tokens, and for o-series models it counts reasoning tokens too. If you're running a production agent with Responses, TPM is usually the limit you hit first. Build in retry-with-backoff and an alternative model fallback.

Streaming works on both Chat Completions and Responses, though the shape of the events differs. Chat Completions streams deltas on choices[0].delta. Responses streams a typed event stream where each event says what part of the output is getting updated (response.output_text.delta, response.function_call_arguments.delta, and so on). The Responses stream is noisier but more useful for a UI, because you can distinguish "model is thinking" from "model is writing" from "model is emitting a tool call."

Model selection in 2026 roughly decomposes into three tiers. GPT-5 is the current frontier default: a unified model that handles both general chat and reasoning behind a single reasoning.effort knob, and the natural starting point for net-new work. The earlier-generation GPT-4o family (including gpt-4.1) is still around as a fast, cheap non-reasoning workhorse, and the legacy o-series (o3, o4-mini) still exists for codebases that pinned to it before GPT-5 landed. Below all of that sits GPT-5-mini for cost-sensitive bulk work. A reasonable rule of thumb: start with GPT-5 at reasoning.effort: medium, drop to low or to GPT-5-mini when the task is simple and volume is high, bump to high when the task has steps the model needs to check its own work on.

Reasoning models add one more dial. You pass a reasoning.effort of low, medium, or high, and the model spends proportionally more hidden tokens thinking before it answers. The bars below are illustrative (I made the numbers up; real benchmarks vary by task), but the shape is real.

Medium is the practical default. Low is great when you have a reasoning model but a simple turn and you don't want to pay for overthinking. High is for problems where you'd rather wait fifteen seconds than get a confidently wrong answer.

The built-in tools

This is the piece of Responses that makes it genuinely different, not just a cleaner Chat Completions. In Chat Completions, every tool has to be a function you defined, with a JSON schema, and your code has to run it. In Responses, there are a handful of tool types where the server runs the tool for you.

{
  "type": "web_search_call",
  "id": "ws_01",
  "status": "completed",
  "results": [
    { "url": "https://...", "title": "..." }
  ]
}

The important bit: the protocol for the model is the same. It still emits a tool call intent into its transcript. It still reads the tool result from a follow-up turn. The difference is that for built-in types, OpenAI's infrastructure sits in the middle and actually runs the search or fetches the page. You don't get a function_call back that you have to dispatch; you get a web_search_call with the results already populated.

This is where the "tool_calls are model-emitted intent, the host runs the code" handle from the function-calling post gets a little complicated. For built-ins, OpenAI is the host. For user-defined functions, you still are. A production agent on Responses typically runs both: a few built-ins that the server handles, plus custom functions for anything that touches your own systems. The model doesn't care which is which; it just picks a tool and emits a call.

Worked example

Let me actually walk through a tool-calling turn in both APIs, end to end, so the diff becomes concrete. The task: a user asks "What's the weather in Tokyo?" and the model has one function, get_weather(city), that returns a string.

On Chat Completions, your code builds the full messages array from scratch on every call. First call: send [system, user] plus the tools array. The response comes back with choices[0].message.tool_calls = [{id: "call_01", function: {name: "get_weather", arguments: '{"city": "Tokyo"}'}}]. Your code parses that, runs get_weather("Tokyo"), gets "18C, cloudy", and makes a second call. This time messages is [system, user, assistant_with_tool_calls, tool_result]. The response comes back with choices[0].message.content = "Tokyo is 18C and cloudy.". You did two HTTP calls and rebuilt the full history twice.

On Responses, you start a conversation (either by passing conversation="auto" to let the server create one, or by calling client.conversations.create() up front). First call: send input="What's the weather in Tokyo?", conversation="conv_...", and the tools array. The response comes back with output = [{type: "function_call", ...}]. Your code dispatches it, then makes a second call with input=[{type: "function_call_output", call_id: "call_01", output: "18C, cloudy"}]. You do not resend anything else. The server already has the prior turn. The response comes back with output = [{type: "message", content: [{type: "output_text", text: "Tokyo is 18C and cloudy."}]}].

Two HTTP calls in both cases, but the second request on Responses carries one item instead of four. The animation below walks the Responses version step by step.

client.responses.create(
  model="gpt-4o",
  conversation="conv_abc123",
  input="Weather in Tokyo?",
  tools=[{
    "type": "function",
    "name": "get_weather",
    "parameters": {...}
  }]
)

The thing I want to register: the agent loop is still the agent loop. model -> tool_call -> your code -> tool_result -> resume, the cycle from the agent-loop post. What changed is which side remembers the earlier turns. In Chat Completions you remember them. In Responses you hand OpenAI a conversation id and they remember for you. Every other surface-level difference follows from that.

Implementation sketch

Here is the full thing in the official openai Python SDK. Same weather tool, same user question, first in Chat Completions and then in Responses. I've kept both versions as close to apples-to-apples as I can make them.

# pip install openai
from openai import OpenAI
import json
 
client = OpenAI()
 
def get_weather(city: str) -> str:
    # pretend this hits a weather API
    return "18C, cloudy"
 
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
 
# --- Chat Completions: you hold the history ---
messages = [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Weather in Tokyo?"},
]
 
r1 = client.chat.completions.create(
    model="gpt-5", messages=messages, tools=tools
)
msg = r1.choices[0].message
messages.append(msg)  # the assistant's tool_calls turn
 
for call in (msg.tool_calls or []):
    args = json.loads(call.function.arguments)
    out = get_weather(**args)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": out,
    })
 
r2 = client.chat.completions.create(
    model="gpt-5", messages=messages, tools=tools
)
print(r2.choices[0].message.content)
# Tokyo is 18C and cloudy.

# --- Responses: the server holds the history ---
# For function tools, Responses expects a flat shape
# (no nested "function": {...}).
resp_tools = [{
    "type": "function",
    "name": "get_weather",
    "description": "Get current weather for a city.",
    "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"],
    },
}]
 
conv = client.conversations.create()
 
r1 = client.responses.create(
    model="gpt-5",
    conversation=conv.id,
    input="Weather in Tokyo?",
    tools=resp_tools,
)
 
# Dispatch any function_call items. Built-ins would already be done.
follow_up = []
for item in r1.output:
    if item.type == "function_call":
        args = json.loads(item.arguments)
        out = get_weather(**args)
        follow_up.append({
            "type": "function_call_output",
            "call_id": item.call_id,
            "output": out,
        })
 
r2 = client.responses.create(
    model="gpt-5",
    conversation=conv.id,
    input=follow_up,   # just the tool outputs; no history resend
    tools=resp_tools,
)
print(r2.output_text)
# Tokyo is 18C and cloudy.

The Chat Completions version pushes every prior turn back onto the wire on the second call; the Responses version pushes only the tool output. The Responses version has a conv.id floating around that is the server's receipt for the running conversation. And both versions dispatch the function call the same way: json.loads(args) then call the Python function. The tool-loop cycle is unchanged. What changed is where the conversation lives.

If you're starting something new in 2026, I'd write the Responses version by default, add tools=[{"type": "web_search"}] when you want live information, and only fall back to Chat Completions if some library you depend on hasn't migrated yet. If you already have a Chat Completions codebase that works, don't rewrite it for its own sake. The migration surface area is not small.

Misconceptions

"The Responses API is just Chat Completions with a different URL." It isn't, though the marketing can make it sound that way. Three things are meaningfully different. Conversations are server-held objects you reference by id instead of messages[] you resend. The output array is typed and contains items for tool calls, reasoning, and built-in tool results that have no Chat Completions equivalent. And built-in tools (web_search, file_search, computer_use) are tool types the server runs, not schemas you have to implement. If you treat Responses as a drop-in for Chat Completions, you'll miss most of what it actually adds.

"tool_calls means the model ran my code." This came up in the function-calling post, but it's worth repeating because the API shape can hide it. A tool_calls field in a Chat Completions response, or a function_call item in a Responses output, is a model-emitted intent. The model produced a JSON blob that says "I would like you to run this function with these arguments." Your code, the host, is the thing that actually dispatches the call, runs the function, and sends the result back. The model never touches your database, your filesystem, or the internet. The one place this gets fuzzy is built-in tools, where OpenAI is the host and their infrastructure does touch the internet for you. But that's their server running their code, not your model running yours.

"Assistants is still the right place to start for a stateful chatbot." Not in 2026. Assistants is in deprecation mode; new state-holding agent work goes to Responses. The only reason to touch the Assistants API now is to maintain code that was written against it, and even then the migration to Responses is mostly mechanical.

"Reasoning is always better." It's better at tasks that benefit from hidden planning. It's slower and pricier on everything else. On pure extraction, classification, formatting, or summarization, GPT-5 at reasoning.effort: low (or a non-reasoning model like GPT-4o) will match high effort at a fraction of the cost and latency. Reach for more reasoning effort when the task has steps the model needs to check its own work on, like math, multi-file code edits, or nested tool use.

What's next

This post covered the OpenAI provider surface: the four generations of its API, what each one added, and the throughline that ties them together ("server-side state migration"). The practical takeaway is that Chat Completions is the 2026 workhorse for single-shot calls, Responses is where net-new agent work lives, and Assistants is quietly on the way out. The underlying agent loop didn't change; what changed is who holds its state.

The next post covers Anthropic's API: Messages, Thinking, and Context Tools, which takes the same turn we just walked through and shows how it looks on a provider that bet on different abstractions for the same set of problems.

Additional reading (and watching)

• OpenAI. (2025). New tools for building agents: Responses API, Agents SDK, and built-in tools. The March 2025 announcement introducing Responses, the Agents SDK, and the first batch of built-in tools (web search, file search, computer use).
• OpenAI. Chat Completions API reference. Canonical spec for /v1/chat/completions, including the messages, tools, and tool_calls shapes.
• OpenAI. Responses API reference. Canonical spec for /v1/responses, including the input, conversation, and typed output items.
• OpenAI. (2024). Learning to Reason with LLMs (o1 system card). Introduces the reasoning-model family and explains why the chain-of-thought is hidden by construction.
• OpenAI. Legacy Completions API. The /v1/completions endpoint, preserved for base models and legacy workloads; no tool calling, no roles.
• OpenAI. (2023). Function calling and other API updates. The June 2023 post that introduced function calling to Chat Completions, later renamed to tool calling.
• OpenAI. Assistants API migration guide. Covers the Assistants-to-Responses migration path; Assistants is deprecated, Responses is the replacement.
• OpenAI. Rate limits. RPM and TPM accounting, including how reasoning tokens are counted against TPM for o-series models.
• OpenAI. Models overview. The 2026 model lineup, including GPT-5 (the unified frontier default) and GPT-5-mini, along with the earlier-generation GPT-4o / GPT-4o-mini and legacy o-series (o3, o4-mini), with their context windows and pricing tiers.
• OpenAI. OpenAI Agents SDK. Python SDK built on top of Responses; canonical reference for what an "agent-native" API looks like in use.
• OpenAI. MCP support in the Responses API. Documents {"type": "mcp", "server_url": "..."} tool entries and the Responses runtime's MCP handling.
• OpenAI. (2025). Computer-Using Agent (CUA) system card. The model and safety card for the computer_use tool and the ChatGPT Operator product it powers.