Google's API: Gemini, Grounding, and the ADK

Coming from OpenAI's or Anthropic's APIs, the first thing to reset when you read Gemini's docs is the assumption that a "request" is a list of text messages with image or tool blocks bolted on. In Gemini, a request is a list of parts, and a part can be text, an inline image, an audio clip via the Files API, a video, a function call, a function response, or a model-generated "thought." Same array, same shape. The modalities don't live in separate lanes.

That one design choice propagates through the whole API, and it's the thing worth carrying away from this post. Once you see that parts[] is the universal container, grounding metadata, thought signatures, function calls, and the ADK all fit into the same mental model without extra scaffolding.

The problem Gemini's API is actually solving

OpenAI's Chat Completions and Anthropic's Messages both started from the same place: a sequence of user/assistant turns with text in each one. Images, tool use, and reasoning arrived later and had to find room inside a transcript format that wasn't originally designed for them. Anthropic's content blocks and OpenAI's move to the Responses API are both, in large part, attempts to catch up to a world where a single turn needs to carry text plus a PDF plus a screenshot plus the model's own reasoning plus some function calls.

Gemini's API didn't start from text. It started from the Gemini technical report's framing of the model itself as a native multimodal system, jointly trained across text, code, image, audio, and video rather than a text model with vision bolted on top. The request format reflects that. You hand the model a contents[] array; each entry has a role and a parts[] list; each part carries one chunk of one modality. The model reads the parts in order and doesn't care which is which.

I'll call this multimodal-native vs multimodal-bolted-on and use it as a handle for the rest of the post. The other providers are catching up by adding blocks; Gemini was shaped around the array from day one.

Prereqs

This sits in Arc 11, so I'm assuming you've already picked up the vocabulary from earlier posts:

• Function calling as structured generation from the function-calling post. You know that a tool call is the model emitting a JSON object that matches a schema.
• The agent loop from the-agent-loop. You know the model → runtime → tool → resume cycle.
• Transcript formats from transcript-formats. You know that different providers encode the same conceptual turn differently.
• MCP from the MCP post, for the "tools live outside the model" intuition.

None of that is being replaced here. Gemini expresses the same ideas in its own shapes, and those shapes are what this post is about.

generateContent

The whole API comes down to one method. The client calls generateContent on a model, hands it a request, and gets a response. Strip away streaming and batching and you're left with:

A content is { role, parts }. A part is one of a small, fixed set of shapes: { text }, { inlineData }, { fileData }, { functionCall }, { functionResponse }, and { thought, text, thoughtSignature }. A candidate has its own content.parts[] and a finishReason, plus optional groundingMetadata if grounding was enabled and the model actually used it.

I think the mental picture to hold is: everything the client sends lives in contents[].parts[], and everything the model sends back lives in candidates[].content.parts[]. Same container, both directions.

{
  "contents": [{
    "role": "user",
    "parts": [

    ]
  }]
}

Small binaries (under a few MB) travel as inlineData with base64 bytes and a mimeType. Larger files get uploaded separately via the Files API and referenced as fileData with a fileUri. The model sees both the same way; the choice is purely about how the bytes get to the server.

Streaming, safety, grounding, two surfaces

Once the request lands, three behaviors are worth internalizing because they will show up the first time you debug anything.

Streaming uses generateContentStream. The response comes back as a sequence of partial candidates, each one a cumulative update to the same parts[] array. Text parts arrive as token chunks; function calls arrive as a single complete part; thought parts stream independently from the final answer. SSE over HTTP, same as OpenAI's streaming, different framing than Anthropic's content-block-delta model.

Safety filters run on both input and output. Every response includes promptFeedback and, per candidate, a safetyRatings array with four categories (harassment, hate, sexually explicit, dangerous) at four severity levels. A finishReason: "SAFETY" means the candidate got stopped mid-generation and you should inspect the ratings rather than retry blindly. You can tune thresholds per request via safetySettings, within limits that tighten on Vertex compared to AI Studio.

Grounding with Google Search is a built-in tool the model can decide to call. Enable it by adding { google_search: {} } to the tools list and the model may, at its own discretion, issue search queries before answering. When it does, the response's groundingMetadata carries the queries, the chunks that came back, and per-span support scores linking the answer to specific chunks. I'll walk through the payload in a second.

The other runtime subtlety is that there are two places you can send this request. The same model IDs (gemini-3-pro, gemini-3-flash, and friends) serve from AI Studio (the developer-API endpoint) and from Vertex AI (the Cloud Platform endpoint). Different auth, different billing, different guardrail surface.

A common debugging story: "I can call the model from AI Studio but not from Vertex" usually means an IAM permission, not a broken model. The google-genai SDK papers over the difference at the constructor level, but the operational stories underneath are genuinely different.

Grounding metadata, in the response

Grounding is where I think Google's API offers the clearest thing the other providers don't match natively. You flip a flag in the tools array and the model gets read access to Google Search; answers come back with attribution structured enough that you can render citations without parsing the text.

The payload you care about, slightly simplified, looks like:

{
  "candidates": [{
    "content": { "parts": [{ "text": "As of early 2026, ..." }] },
    "groundingMetadata": {
      "webSearchQueries": ["CRISPR base editor clinical trials 2026", ...],
      "groundingChunks": [
        { "web": { "uri": "https://nejm.org/...", "title": "..." } },
        ...
      ],
      "groundingSupports": [
        { "segment": { "startIndex": 17, "endIndex": 68, "text": "..." },
          "groundingChunkIndices": [1],
          "confidenceScores": [0.87] }
      ],
      "searchEntryPoint": { "renderedContent": "<div>...Google Search suggestions...</div>" }
    }
  }]
}

The segment fields let you highlight the exact character range of the answer that was supported by a given chunk; that's what makes inline citation bubbles cheap to render. The searchEntryPoint.renderedContent is an HTML blob Google's terms require you to show when you display grounded results (it contains the search suggestions chip). And the confidenceScores are the grounding system's estimate of how well a chunk supports a span, not the model's own uncertainty about the answer. That's an easy thing to misread the first time through the response.

This is the handle I'd keep: grounding metadata travels with the response. The model doesn't just tell you the answer; the runtime tells you which pieces came from where. You can render it, store it for audit, or throw it away. It's there either way.

Thought signatures, and how they differ

Starting with the 2.5 family and carrying through into 3.x, Gemini returns its extended reasoning inside the same parts[] array as everything else. A thought part looks like a text part with thought: true, and function-call or thought parts can carry a thoughtSignature field, an opaque token that encodes the internal reasoning state. The signature is the interesting one.

The three big providers each solve the "how does the model resume its own reasoning on the next turn without us having to read all of it back" problem differently. The comparison is the clearest way to see what Gemini chose.

{
  "parts": [
    {
      "thought": true,
      "text": "Let me check the FDA approvals..."
    },
    {
      "thoughtSignature": "AbC9...opaque...xYz"
    },
    {
      "text": "As of early 2026, the FDA has ..."
    }
  ]
}

[
  {
    "type": "thinking",
    "thinking": "Let me check the approvals...",
    "signature": "EqoB...crypto-signed...Q=="
  },
  {
    "type": "text",
    "text": "As of early 2026, ..."
  }
]

[
  {
    "type": "reasoning",
    "id": "rs_abc123",
    "summary": [
      { "type": "summary_text",
        "text": "Checked 2026 FDA approvals." }
    ]
  },
  {
    "type": "message",
    "content": [{ "type": "output_text",
                  "text": "As of early 2026..." }]
  }
]

So. Gemini gives you a thought signature, which is opaque. You treat it as a bag of bytes you pass back verbatim on the next turn; the model uses it to hydrate whatever internal state the reasoning produced, and tampering with it just fails. Anthropic gives you signed thinking blocks: you can read the reasoning text, but the block carries a cryptographic signature, and if you modify the text you have to strip the signature first and the server will refuse to rebuild the model's state from it. OpenAI's Responses API takes a third path: you don't get the reasoning tokens at all in most models; you get a short summary, and you reference the full server-side reasoning by an id on your next turn.

Those are three different answers to the same underlying question: what do we let developers do with the model's private scratch space? Gemini says, "here's an opaque handle, pass it back." Anthropic says, "here are the words; we signed them so you can't lie about them." OpenAI says, "we'll hold on to the words; you just reference them." None of them is obviously right. They reflect three different bets on where audit, portability, and provider control should sit.

Worked example

Let's put it together. I'll walk through a single generateContent call that registers a custom tool, turns on Google Search grounding, and asks a question whose answer needs both. The whole interaction is one request/response pair plus a follow-up to hand the function result back.

The user asks: "What's the stock price of Alphabet right now, and how does that compare to the 2026 CRISPR therapy coverage we talked about?" The model decides it needs two things: a live stock price from a tool I registered, and some grounded context from search. It uses both.

The request has two tools declared:

{
  "tools": [
    {
      "functionDeclarations": [{
        "name": "get_stock_price",
        "description": "Get the current stock price for a ticker.",
        "parameters": {
          "type": "OBJECT",
          "properties": { "ticker": { "type": "STRING" } },
          "required": ["ticker"]
        }
      }]
    },
    { "google_search": {} }
  ],
  "contents": [{ "role": "user", "parts": [{ "text": "What's the stock price..." }] }]
}

The first response comes back with a functionCall part and a groundingMetadata block. The model already did the search to fill in the CRISPR context, and it's asking me to run the stock-price tool:

{
  "candidates": [{
    "content": {
      "role": "model",
      "parts": [
        { "thought": true, "text": "I need a live stock price and some 2026..." },
        { "thoughtSignature": "AbC9...opaque..." },
        { "functionCall": { "name": "get_stock_price", "args": { "ticker": "GOOGL" } } }
      ]
    },
    "finishReason": "STOP",
    "groundingMetadata": {
      "webSearchQueries": ["2026 CRISPR therapy FDA approval"],
      "groundingChunks": [{ "web": { "uri": "https://fda.gov/...", "title": "..." } }]
    }
  }]
}

I run the tool, get 183.42, and send it back as a functionResponse part on a new turn. Critically, I also send the thoughtSignature back in the same parts[], which is what lets the model pick up from where it left off in its reasoning:

{
  "contents": [
    { "role": "user",  "parts": [{ "text": "What's the stock price..." }] },
    { "role": "model", "parts": [
        { "thoughtSignature": "AbC9...opaque..." },
        { "functionCall": { "name": "get_stock_price", "args": { "ticker": "GOOGL" } } }
    ]},
    { "role": "user", "parts": [
        { "functionResponse": {
            "name": "get_stock_price",
            "response": { "price": 183.42, "currency": "USD" }
        }}
    ]}
  ]
}

The final response is a text part stitched from the tool result and the grounded context. The full grounding metadata (chunks, supports, search entry point) rides along on the final candidate.

Two things to notice. First, tools are either custom (a functionDeclarations entry you define) or built-in (google_search, code_execution, url_context) and they share the same tools array. You can mix them. Second, the thought signature is the only piece of the response you must echo back to keep multi-turn reasoning coherent; the rest of the returned parts[] you can trim as you like.

Implementation sketch

Here's the whole thing end-to-end in the google-genai SDK. Thirty-something lines and it exercises parts, function declarations, grounding, and the thought-signature replay.

# pip install google-genai
from google import genai
from google.genai import types
 
client = genai.Client()  # reads GOOGLE_API_KEY; add vertexai=True for Vertex
 
def get_stock_price(ticker: str) -> float:
    """Pretend this is a real market-data call."""
    return {"GOOGL": 183.42, "AAPL": 224.10}.get(ticker, 0.0)
 
tools = [
    types.Tool(function_declarations=[types.FunctionDeclaration(
        name="get_stock_price",
        description="Get the current stock price for a ticker.",
        parameters={
            "type": "OBJECT",
            "properties": {"ticker": {"type": "STRING"}},
            "required": ["ticker"],
        },
    )]),
    types.Tool(google_search=types.GoogleSearch()),  # built-in grounding tool
]
 
contents = [types.Content(role="user", parts=[types.Part.from_text(
    "What's GOOGL's price right now, and how does it compare to "
    "the 2026 CRISPR therapy coverage we discussed?"
)])]
 
# Turn 1: model may emit a function_call + thought signature, and fire search.
resp = client.models.generate_content(
    model="gemini-3-pro",
    contents=contents,
    config=types.GenerateContentConfig(tools=tools, thinking_config=types.ThinkingConfig(include_thoughts=True)),
)
 
# Replay the model's parts (carries thoughtSignature) before appending the tool result.
contents.append(resp.candidates[0].content)
 
# Execute the function call the model requested.
fc = next(p.function_call for p in resp.candidates[0].content.parts if p.function_call)
result = get_stock_price(**fc.args)
 
contents.append(types.Content(role="user", parts=[types.Part.from_function_response(
    name=fc.name, response={"price": result, "currency": "USD"},
)]))
 
# Turn 2: model finalizes, still grounded.
final = client.models.generate_content(
    model="gemini-3-pro", contents=contents,
    config=types.GenerateContentConfig(tools=tools),
)
print(final.text)
print(final.candidates[0].grounding_metadata.web_search_queries)

You have to append resp.candidates[0].content to contents before adding the function response. That's what carries the thought signature forward. Drop it and the model starts the next turn from a cold state, which at best wastes the thinking budget and at worst produces a noticeably worse answer.

The ADK: when the loop is more than one call

Everything above is one-shot. For actual agents (multi-step tool use, memory across sessions, evaluation harnesses) Google ships the Agent Development Kit, which open-sourced in April 2025 and is now the canonical framework for building agents with Gemini.

ADK is model-agnostic in principle (you can point it at Anthropic or OpenAI via adapters) but it's shaped around Gemini's idioms. An agent is a Python class, usually an LlmAgent, with tools attached as Python functions or AgentTool wrappers. A Runner owns sessions; a MemoryService persists state across them; a set of callbacks lets you hook in at every step of the loop.

The piece worth highlighting is that ADK accepts MCP servers as tool providers out of the box. Stand up a local MCP server (the MCP post walks through this), wrap it in MCPToolset, hand it to the agent, and every tool the server exposes is available to Gemini under the same functionDeclarations shape the model was trained on. The $N + M$ collapse from the MCP post applies here too: you don't write a Gemini-shaped version of a GitHub integration, you point ADK at the GitHub MCP server.

ADK also ships an evaluation harness, a built-in web UI for stepping through agent traces, and first-class deployment to Cloud Run and Vertex's Agent Engine. Not all of that is load-bearing for a typical project. The part that earns its keep is the tool-orchestration story: ADK handles the model-runtime-tool-resume loop, handles injecting function responses as parts, and handles keeping thought signatures wired through the right places, all of which is the kind of plumbing you really don't want to reinvent per project.

Misconceptions

"Gemini's multimodal support is just vision with extra modalities." What makes multimodal-native different isn't the list of supported types. It's that all of them live in the same parts[] array and get read in sequence. You can put a frame of video between two text parts and ask the model a question that spans them. You can upload an audio file and follow it with an image and then a text question about both. The other providers can increasingly accept these payloads, but the design still leaks in subtle ways (separate content-block types, separate tool paths, separate pricing dimensions). In Gemini it's all one array.

"Thought signatures are Gemini's version of reasoning.id." They look similar and they solve a similar problem, but they're not the same object. OpenAI's reasoning.id is a server-side pointer; the reasoning itself lives on OpenAI's servers and you reference it by id. Gemini's thoughtSignature is an opaque token that travels over the wire with each turn; the server uses it to reconstruct the prior reasoning state. The nearest cousin is actually Anthropic's signed thinking, minus the readable text. Knowing which provider you're talking to changes what you have to store, what you have to forward, and what breaks if you drop it.

"Grounding is just RAG with Google Search." Grounding is a runtime behavior the model invokes, not a retrieval step you run and stuff into the prompt yourself. The model decides when to search, picks the queries, reads the chunks, and emits structured attribution back in the response. You don't get to pick the chunks, and you don't get to see them before the model does. If you want RAG-style control (custom retriever, custom rerank, custom chunk selection) you should still build RAG on top of Gemini; grounding is a different animal, better when you want live attribution without running your own pipeline.

"The ADK is a framework you have to buy into for any Gemini work." It really isn't. You can build production agents on google-genai directly and a couple hundred lines of loop code. ADK starts paying for itself once you need memory that outlives a session, multi-agent routing, evaluation harnesses, or managed deployment to Agent Engine. For a single-turn generateContent call, which covers the large majority of production usage, it's overkill.

What's next

This post covered Google's API as one object: a parts[] array that carries text, inline bytes, uploaded files, function calls, function responses, and the model's own thoughts; a response that carries the same kinds of parts plus groundingMetadata when search was used; two endpoints (AI Studio and Vertex) that ride the same SDK; and ADK as the framework for when a single call isn't enough. The handle to carry: multimodal-native, not multimodal-bolted-on, and parts[] as the one container everything fits inside.

The next post covers open-weight models in practice: how Llama, Mistral, Qwen, Gemma, and gpt-oss actually differ once you leave the hosted APIs behind, and what their license, format, and tooling stories mean for a production deployment in 2026.

Additional reading (and watching)

• Gemini Team, Google. (2023–2024). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805. The technical report introducing Gemini as a natively multimodal model family, trained jointly across text, code, image, audio, and video.
• Google. Gemini API overview and the google-genai Python SDK. The canonical reference for generateContent, parts[], the Files API, and the 2025-era unified SDK that replaced google-generativeai.
• Google. Grounding with Google Search and Vertex AI: Grounding. Describes the google_search tool, the groundingMetadata response shape, and the searchEntryPoint display requirement.
• Google. Vertex AI Generative AI documentation. The enterprise deployment surface (IAM, logging, Model Armor guardrails, and Model Garden) that shares the Gemini model family with AI Studio.
• Google. Gemini thinking and Thought signatures. The opaque-handle design for preserving model reasoning across turns, introduced with Gemini 2.5 and carried into 3.x.
• Anthropic. Extended thinking. Claude's signed-thinking-block design, the closest analog to Gemini's thought signatures but with visible reasoning text and a cryptographic signature over the block.
• OpenAI. Responses API: reasoning items. The reference-by-id approach to multi-turn reasoning: reasoning content stays server-side, clients reference it by id on follow-up turns.
• Google. Agent Development Kit documentation and the ADK GitHub repository. Open-sourced April 2025; the canonical framework for building multi-step Gemini agents with tools, memory, evaluation, and Cloud Run / Agent Engine deployment.
• Google. ADK: MCP tools. How to consume MCP servers from an ADK agent. Confirms the $N + M$ collapse carries from the MCP post into the ADK world.
• Google. Vertex AI Model Garden. The unified Vertex catalog (Gemini alongside Claude, Llama, Mistral, Mixtral, and partner models) under one IAM and billing surface.