The Complete Google ADK Guide

This is part one of three. It covers everything you need to build a working agent with Google's Agent Development Kit, not a teaser. By the end you will have installed ADK, created an agent that answers questions, given it a tool it can call, watched every decision it made in a browser-based inspector, stored facts across turns, routed work through a graph of steps, and driven the whole thing from your own Python script. Mid-level and Senior take the same topics further; nothing here is wasted.

Each section ends with a Try it task. Do them as you go. Agents are one of those topics where reading produces a comfortable illusion of understanding that collapses the first time a model calls your function with arguments you did not expect.

A note on versions before we start, because it matters more here than with most tools. This guide is written against ADK 2.10.0 for Python, released in September 2026. ADK 2.0 arrived in May 2026 and replaced a large part of the 1.x execution model, so a great deal of the tutorial material you will find online — blog posts, conference talks, Stack Overflow answers, and the memory of any chatbot you ask — describes 1.x and will quietly mislead you. Wherever 1.x did something differently, this guide says so explicitly.

What ADK is, and the problem it solves

The Agent Development Kit is Google's open-source framework for building, testing and deploying agents: programs where a language model is allowed to decide what happens next. You give the model an instruction, a set of functions it is permitted to call, and a conversation; the model chooses whether to answer directly, call one of your functions, call another function with the first one's result, or hand the conversation to a different agent entirely. ADK is the machinery that runs that loop safely, records what happened, and lets you deploy the result.

To see why that machinery is worth having, picture building it yourself. You start with a single call to a model API: a prompt in, text out. That is easy. Then you want the model to be able to look something up, so you describe a function in the prompt and ask the model to reply with JSON when it wants to call it. Now you need to parse that JSON, validate it, call the real function, format the result, append it to the conversation, and call the model again. Then you need a cap on how many times that can loop, because a model that keeps asking for the same lookup will otherwise bill you forever. Then you want the conversation to survive a server restart, so you need storage. Then a file the user uploaded needs to be available three turns later, so you need somewhere to keep blobs. Then someone asks why the agent said something strange on Tuesday, and you discover you logged almost nothing useful.

Every one of those problems is generic. None of them is about your actual product. ADK's proposition is that it solves all of them, consistently, with components you can swap: the loop, the function-calling plumbing, the call cap, conversation storage, file storage, long-term memory, tracing, evaluation and deployment. What you write is the interesting part — the instructions, the tools, and the shape of the workflow.

Two design decisions shape everything that follows, and they are worth noticing now rather than discovering later.

ADK is code-first. An agent is an ordinary Python object you construct in an ordinary Python file. A tool is an ordinary Python function. There is no visual builder you must use and no domain-specific language you must learn, which means your agent is reviewable in a pull request, testable with pytest, and diffable in git log like anything else you write. A YAML form exists for configuring agents, but the normal path is code.

ADK is model-agnostic and deployment-agnostic. It is built by Google and optimised for Gemini models running on Google Cloud, and that is the smoothest path, but it is not a lock-in. ADK supports Anthropic's Claude models, OpenAI models, Oracle's OCI Generative AI service, and through LiteLLM a very long list of other providers including locally-hosted models served by Ollama. Likewise an ADK agent can run on your laptop, in a container anywhere, on Cloud Run, on Kubernetes, or on Google's fully managed Agent Runtime. For readers working with Gulf or Egyptian employers, that flexibility is not a detail: data-residency rules frequently decide which model endpoint and which region you are allowed to use, and a framework that treats the model as a swappable component is much easier to get past a security review than one that assumes a single provider.

The agent loop, drawn once

Before any code, fix the shape of the thing in your head. One user message produces one invocation, and an invocation is a loop.

The arrow from the tool back to the model is the part that makes this an agent rather than a chatbot. The model is not called once; it is called repeatedly until it decides it has enough to answer. Each pass around the loop costs a model call and therefore money and latency, and nothing in the model guarantees it will ever stop. That is why ADK caps the loop: by default an invocation may make at most 500 model calls, after which it raises an error rather than continuing. You will almost certainly never approach 500 legitimately. If you hit it, you have a bug, and the cap has just saved you from paying for it.

Everything the loop does is recorded as a stream of events. A user message is an event. A model response is an event. A request to call a tool is an event, and the tool's result is another. A change to stored state is an event. Events are immutable and ordered, which is what makes the whole system debuggable: when an agent behaves strangely, you do not guess, you read the event list for that invocation and see exactly which tool returned what.

Hold on to one more idea, which is where ADK 2.x differs most from what you may have read. Agents are not the only things that can be a step in the loop. In 2.x, an agent, a plain Python function, a tool and a whole nested pipeline are all nodes, and a pipeline is a graph of nodes with edges between them. The class BaseAgent is itself a subclass of BaseNode. This sounds abstract until you need a step that does no reasoning at all — reformat this JSON, write this row to the database — at which point being able to drop a plain function into the pipeline, with no model call and no cost, is obviously the right answer.

The nouns you need: agent, tool, session, runner

Four words carry most of the weight. Learn them precisely, because they are used loosely in conversation and precisely in the API.

An agent is the model-driven unit. The class you will use almost always is LlmAgent, and Agent is simply an alias for it, so the two names mean the same thing. Its important fields are name (an identifier, used in traces and in delegation), model (a model identifier string), instruction (the system prompt that tells it what to do), description (one sentence saying what this agent is for, which matters because it is what other agents read when deciding whether to hand work over), and tools (the list of functions it may call). The default model in ADK 2.10 is gemini-3.5-flash, so you can leave model out, though being explicit is better practice.

A tool is, in the simplest and most common case, a plain Python function. You do not subclass anything or register anything. You write a function with type-annotated arguments and a docstring, pass it in the tools list, and ADK wraps it as a FunctionTool automatically, generating the schema the model needs from your annotations and the description from your docstring. That mechanism has a consequence worth saying out loud: your docstring is production code. It is the only explanation the model gets of what your function does and when to call it. A vague docstring produces a tool the model calls at the wrong moments, and a missing type hint produces arguments of the wrong type.

A session is one conversation thread. It is identified by three things together — app_name, user_id and a session id — and it holds the ordered list of events plus a key-value scratchpad called state. Sessions are created and loaded by a SessionService, which is the first component most people swap: InMemorySessionService for tests, a SQLite file for local development, a real database in production.

A runner is the orchestrator. You hand it a user message and it does everything else: loads the session, drives the loop, calls the model, executes tools, persists events, and yields those events back to you as they happen. It is constructed with a required keyword-only session_service, plus either an agent, a node, or an App object. App is the top-level container for a deployed agent and carries cross-cutting configuration such as plugins; you do not need it to get started.

Two smaller nouns will appear often enough to introduce now. An artifact is a named, versioned binary blob — a PDF, an image, a generated chart — stored by an ArtifactService rather than stuffed into state. Memory is long-term recall across sessions, served by a MemoryService, and is a different thing from state, which lives only within one conversation. Beginners conflate these constantly; the distinction is simply scope.

Installing ADK and checking the setup

ADK for Python requires Python 3.10 or later. Check before anything else, because the failure if you are on an older interpreter is a confusing dependency-resolution error rather than a clear message.

Always install into a virtual environment. Agent frameworks pull in a substantial dependency tree — the Google GenAI SDK, FastAPI, Uvicorn, Starlette, Pydantic, OpenTelemetry, SQLAlchemy's async pieces and more — and you do not want that mixed into your system Python.

python3 -m venv .venv
source .venv/bin/activate
pip install google-adk

On Windows the activation step differs. In Command Prompt:

python -m venv .venv
.venv\Scripts\activate.bat
pip install google-adk

In PowerShell:

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install google-adk

If PowerShell refuses to run the activation script, the fix is a one-off policy change: Set-ExecutionPolicy -Scope CurrentUser RemoteSigned. If you prefer uv, uv venv && uv pip install google-adk does the same job considerably faster.

Now verify, and verify in two ways, because the two checks fail differently and tell you different things.

adk --version

adk, version 2.10.0

python -c "import google.adk; print(google.adk.__version__)"

The first proves the command-line tool is on your PATH. The second proves the library is importable from the interpreter you are actually using. If adk --version works but the import fails, you have activated one environment and installed into another — the single most common setup problem, and worth ten seconds of checking before you debug anything else. pip show google-adk is a third option and prints the version alongside the install location, which is useful precisely when you suspect that mismatch.

ADK ships a core install plus a long list of optional extras, and this is deliberate: the full set of integrations would be an enormous install for someone who wants one agent and one tool. The extras available in 2.10 include a2a, agent-identity, all, antigravity, benchmark, bigquery-analytics, community, daytona, db, dev, docs, e2b, eval, extensions, gcp, livekit, mcp, mongodb, oci, openai, otel-gcp, redis, slack, test, toolbox and tools. You install them with bracket syntax, and in zsh — which is the default shell on macOS — you must quote the brackets or the shell will try to glob them:

pip install "google-adk[eval,gcp]"

The three you are most likely to need early are db (database-backed sessions, which pulls in SQLAlchemy), eval (the evaluation tooling behind adk eval) and gcp (Vertex AI, Cloud Storage and Agent Runtime features). ADK is good about telling you when an extra is missing: reaching for a database session service without it produces the exact message The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db], which is a refreshingly actionable error.

Giving the agent credentials

ADK does not ship a model. It needs credentials for one, and there are two ways in. Choosing between them is the step where most first attempts stall, so be deliberate.

The quick path is Google AI Studio: you obtain an API key and set two environment variables in a file named .env inside your agent's folder, which the ADK command-line tools load automatically.

GOOGLE_GENAI_USE_ENTERPRISE=0
GOOGLE_API_KEY=your-key-here

The production path is Vertex AI, also presented in the documentation as Gemini Enterprise, which authenticates with Google Cloud credentials instead of a key:

GOOGLE_GENAI_USE_ENTERPRISE=1
GOOGLE_CLOUD_PROJECT=my-project
GOOGLE_CLOUD_LOCATION=us-central1

followed by one command to establish Application Default Credentials on your machine:

gcloud auth application-default login

Note the variable name carefully, because this is a trap laid by every tutorial written before mid-2026. The flag used to be called GOOGLE_GENAI_USE_VERTEXAI and took values like TRUE. Since ADK 2.3 the correct name is GOOGLE_GENAI_USE_ENTERPRISE. The old name still works through a deprecation fallback, but it emits the warning GOOGLE_GENAI_USE_VERTEXAI is deprecated, please use GOOGLE_GENAI_USE_ENTERPRISE instead. If you see that warning, rename the variable in your .env; do not set both.

The reason these two paths are not interchangeable is worth internalising. The flag tells the GenAI client which service to talk to, and the credential must match. An API key with the enterprise flag set to 1, or Cloud credentials with it set to 0, produces an authentication failure that looks like a bad key rather than like a misconfiguration — typically API key not valid, 400 INVALID_ARGUMENT or 403 PERMISSION_DENIED. When you see any of those, check the flag against the credential type first, before you regenerate a key you did not need to regenerate.

For the rest of this guide, either path works. Use AI Studio if you want to be running in two minutes; use Vertex if you are on a work machine where a Cloud project already exists and loose API keys would not survive review.

Your first agent, step by step

ADK expects a specific folder layout, and getting it wrong is the most common first-run failure. Make a parent folder — call it agents — and let ADK create the agent inside it.

mkdir agents
cd agents
adk create my_agent

adk create asks which model and which backend you want, then writes four files: .env, .gitignore, __init__.py and agent.py. You can skip the prompts with flags: --model, --api_key, --project and --region. There is also a --type flag offering code or config; leave it at the default code, because the config variant is explicitly marked experimental and not ready for use.

The generated agent is exactly this:

from google.adk.agents.llm_agent import Agent

root_agent = Agent(
    model='gemini-3.5-flash',
    name='root_agent',
    description='A helpful assistant for user questions.',
    instruction='Answer user questions to the best of your knowledge',
)

and my_agent/__init__.py contains one line:

from . import agent

Three things in that tiny file are load-bearing and will each bite you if changed carelessly. The variable must be called root_agent, because that is the name ADK's loader looks for. The __init__.py must import the agent module, or the loader cannot reach inside the package. And the agent lives in a folder whose name is the app name, inside a parent directory you run the tools from.

Now run it. The simplest form is an interactive terminal chat:

adk run my_agent

Note the working directory: you are in agents, the parent, and you pass the agent's folder name. For a single question with no chat loop, pass the query as a second argument:

adk run my_agent "what can you do?"

If instead you see this, the layout is wrong:

No root_agent found for 'my_agent'. Searched in 'my_agent.agent.root_agent',
'my_agent.root_agent' and 'my_agent/root_agent.yaml'.

The message is unusually helpful: it lists the three places it looked. In order of likelihood, the cause is that you ran the command from inside my_agent instead of from agents, that __init__.py is empty or missing, or that you renamed the variable. A related error, A 'root_agent' was found for 'my_agent' but it is not a ... 'root_agent', means the name exists but holds something that is not an agent, node or App — commonly a function you forgot to call.

adk run has flags worth knowing on day one. --save_session writes the conversation to a file when you exit, optionally with --session_id ID; --resume FILE.json picks a saved session back up; --replay FILE.json re-runs a recorded set of inputs, which makes bug reports reproducible. --state '{"k":"v"}' seeds the session's state before the first turn. --in_memory skips local persistence entirely. --timeout 30s bounds a run. --log_level debug turns on the detail you need when the agent is doing something inexplicable, and --jsonl emits machine-readable output for scripting.

Giving the agent a tool

An agent without tools can only talk. The moment it can call a function, it can act — and that is the whole point. Here is a complete agent with one tool.

from google.adk.agents.llm_agent import Agent

SHIPPING_DAYS = {"AE": 2, "SA": 3, "EG": 5, "KW": 3}

def estimate_delivery_days(country_code: str) -> dict:
    """Estimate delivery time in business days for a destination country.

    Use this whenever the user asks how long an order will take to arrive.

    Args:
        country_code: Two-letter ISO country code, uppercase, such as "AE" or "EG".
    """
    days = SHIPPING_DAYS.get(country_code.upper())
    if days is None:
        return {"status": "unsupported", "country_code": country_code}
    return {"status": "ok", "country_code": country_code.upper(), "days": days}

root_agent = Agent(
    model="gemini-3.5-flash",
    name="shipping_agent",
    description="Answers questions about order delivery times.",
    instruction=(
        "You help customers with delivery questions. "
        "When asked how long delivery takes, call estimate_delivery_days with the "
        "two-letter country code. If the tool reports status 'unsupported', say we "
        "do not ship there yet and do not guess a number."
    ),
    tools=[estimate_delivery_days],
)

Read that function again as the model sees it. The first docstring line is the summary that tells the model what the tool is for. The second line is an explicit trigger condition, which is far more effective than hoping the model infers one. The Args block explains the exact format expected, which is why the model sends "AE" rather than "United Arab Emirates". The country_code: str annotation becomes the parameter's type in the schema the model is given.

Four habits make tools behave well, and all four are visible above.

Return a dictionary, not a bare string. The model reads the result as data. A dict with a status key lets the model distinguish success from failure without parsing prose, and lets you add fields later without breaking anything.

Make failure a value, not an exception. estimate_delivery_days returns {"status": "unsupported"} instead of raising. The model can then say something sensible. An uncaught exception is for genuine bugs, not for "the user asked about a country we do not serve".

Say what to do with each outcome in the instruction. The line "do not guess a number" exists because a model with no number and a desire to be helpful will invent one. Instructions that cover the failure path are what separate a demo from something you would show a customer.

Keep one tool to one job. A tool that takes a mode argument and does three different things is three tools wearing a coat, and the model will pick the wrong mode.

One more detail about instructions, and it is a recent change. ADK substitutes session state into instructions using {braces}: if state holds customer_tier, then writing {customer_tier} in the instruction injects its value, and {customer_tier?} makes it optional so a missing key does not error. As of 2.10, ${var} and backslash-escaped \{var} are deliberately left exactly as written rather than substituted. If you are following an older tutorial that uses ${var} and wondering why nothing is injected, that is why.

The dev UI, and reading what the agent actually did

Terminal chat is fine for a smoke test and useless for understanding behaviour. ADK bundles a local web interface that shows you the inside of every invocation.

adk web --port 8000

Run it from the parent directory — agents, not my_agent — because it presents a dropdown of every agent folder it can find. It binds to 127.0.0.1 on port 8000 by default. Open the address it prints, pick your agent, and talk to it.

The chat panel is the least interesting part. The panels that matter are Events and Trace. Events lists every event in the invocation in order: your message, the model's decision, the function call with its exact arguments, the function's exact return value, and the final response. When your agent does something baffling, the answer is almost always visible here within thirty seconds, and the usual revelation is that the model called your tool with an argument you had not considered. Trace shows the same invocation as timed spans, so you can see which model call or which tool consumed the latency. There are also panels for session state and for stored artifacts, and an Eval tab for running saved test cases.

Useful flags: --reload_agents picks up edits to your agent files without a restart, which turns the edit-instruction-and-retry cycle from twenty seconds into two. --logo-text and --logo-image-url let you rebrand it for a demo. --no-reload disables the file watcher, which you need on Windows in some configurations.

adk web --port 8000 --reload_agents

By default, local runs persist their state next to your agent: sessions go into <agents_dir>/<agent>/.adk/session.db, a SQLite file, and artifacts into <agents_dir>/<agent>/.adk/artifacts. Memory is in-process only. This is why a conversation can survive restarting adk web, and also why a stale .adk/session.db can cause schema errors after you upgrade ADK across a major version. For a local development database the fix is simply to delete the file and start again; for anything you care about, there is a proper migration command, covered below.

State, output_key, and remembering things

Agents need to remember. ADK gives you three distinct mechanisms, and using the wrong one is a classic beginner mistake, so separate them clearly.

State is a key-value scratchpad attached to a session. It is the right place for the small facts this conversation has established: the order number the user mentioned, the language they prefer, the result of step two. Keys carry an optional prefix that sets their scope, and the prefixes are the whole design:

The app: row deserves a warning sign. It is shared by every user of the application. Writing one user's data into an app: key is a data leak, not a bug in your logic, and it is the kind of mistake that is trivially easy to make and extremely awkward to explain afterwards.

The simplest way to write state is the output_key field on an agent. Set it, and the agent's final text is saved into state under that key automatically:

summariser = Agent(
    model="gemini-3.5-flash",
    name="summariser",
    description="Condenses a support ticket into two sentences.",
    instruction="Summarise the user's issue in at most two sentences.",
    output_key="ticket_summary",
)

A later agent can then read it in its own instruction with {ticket_summary}, and that pair — output_key writing, {braces} reading — is how most multi-step ADK pipelines pass data along without any plumbing code.

Tools write state too. A tool function can declare a ToolContext parameter, and ADK will supply it; the context exposes state for reading and writing, and save_artifact and load_artifact for files.

Artifacts are the second mechanism: named, versioned binary blobs — an uploaded PDF, a generated image — stored by an ArtifactService rather than embedded in state. Putting a base64 image into state technically works and is a bad idea, because state is loaded with every turn.

Memory is the third: recall that crosses session boundaries. It is served by a MemoryService — InMemoryMemoryService for development, Vertex AI Memory Bank or Vertex RAG in production — and the agent reaches it through the built-in load_memory and preload_memory tools. The distinction from state is purely one of scope: state is this conversation, memory is everything this user has ever told you.

For local development you get persistence for free, since sessions land in .adk/session.db. To be explicit, or to point somewhere else, the service URI flags work on run, web and api_server alike. --session_service_uri accepts memory://, sqlite://<path>, or a full SQLAlchemy URL. There is a rule about those URLs that catches everyone exactly once: ADK's database session service uses SQLAlchemy's async layer, so the URL needs an async driver. postgresql+asyncpg://... works; a bare postgresql://... does not, and produces Database related module not found for URL or Invalid database URL format or argument. The async spellings are sqlite+aiosqlite, postgresql+asyncpg, mysql+aiomysql and mariadb+asyncmy. You also need the db extra installed.

Workflows: graphs instead of fixed pipelines

Not every step should be decided by a model. When you know the order — classify, then route, then format — encoding that order in code is cheaper, faster and far more predictable than writing an instruction that begs a model to follow steps.

In ADK 2.x the construct for this is Workflow: a graph of nodes connected by edges. This is the biggest change from 1.x and the place where old tutorials will most reliably mislead you, so here is the honest version. ADK 1.x offered three orchestrator classes — SequentialAgent, ParallelAgent and LoopAgent — for running steps in order, in parallel, and in a loop. In 2.x all three still import and still run, but they emit a deprecation warning that says exactly what to do:

DeprecationWarning: SequentialAgent is deprecated in favor of Workflow and will be
removed in a future version. Workflow cannot yet be used as an LlmAgent sub-agent.

Learn Workflow. Recognise the other three when you meet them in old code, and know they are on the way out.

A workflow is a Workflow object with a list of edges. Here is a complete, runnable triage graph:

from google.adk import Workflow, Event
from google.adk.workflow import DEFAULT_ROUTE

def classify(node_input: str):
    return Event(route="BUG" if "crash" in node_input else "OTHER", output=node_input)

def bug(node_input: str):
    return Event(message=f"bug handler got: {node_input}")

def other(node_input: str):
    return Event(message="other handler")

root_agent = Workflow(
    name="triage",
    edges=[
        ("START", classify),
        (classify, {"BUG": bug, DEFAULT_ROUTE: other}),
    ],
)

Four mechanics are on display, and they are the whole of the graph model.

START is the entry point. The edge ("START", classify) says the first node to run is classify. A tuple with more than two entries, such as ("START", a, b, c), is shorthand for a chain: a, then b, then c.

Data flows through node_input and output. The first node receives the user's text as its node_input argument. Whatever a node puts in its event's output becomes the next node's node_input. No shared mutable blackboard is required for the common case.

A route chooses the edge. When a node returns an Event with route="BUG", ADK looks at that node's outgoing edge dictionary and follows the "BUG" key. DEFAULT_ROUTE is the fallback for anything unmatched. A route may also be a list of strings, in which case execution fans out to several nodes at once.

Nodes are not only functions. An Agent can be a node, a nested Workflow can be a node, and so can a tool. The plain functions above cost nothing to run, which is the point: classify here is deterministic string matching, not a model call. Replace it with an Agent the day the classification genuinely needs judgement, and the graph around it does not change.

For fan-in, JoinNode waits for several predecessors to finish before continuing. max_concurrency on the workflow bounds how many nodes run in parallel. Individual nodes accept a timeout, which raises NodeTimeoutError when exceeded, and a retry_config built with RetryConfig(max_attempts=..., initial_delay=..., backoff_factor=...).

Running an agent from your own code

The CLI is for development. In an application you construct the Runner yourself and consume its events. Here is the complete pattern.

import asyncio
from google.adk import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

from my_agent.agent import root_agent


async def main():
    runner = Runner(
        app_name="my_app",
        agent=root_agent,
        session_service=InMemorySessionService(),
    )
    session = await runner.session_service.create_session(
        app_name="my_app", user_id="u1"
    )

    message = types.Content(role="user", parts=[types.Part(text="How long to EG?")])
    async for event in runner.run_async(
        user_id="u1", session_id=session.id, new_message=message
    ):
        if event.is_final_response():
            print(event.content.parts[0].text)


asyncio.run(main())

Several details in those twenty lines are exactly the things people get wrong on a first attempt.

session_service is keyword-only and required. There is no default, and that is deliberate: where conversations are stored is an architectural decision, not something a framework should guess. Alongside it, the runner takes app_name plus one of agent, node or app. If you want the in-memory defaults with less typing, InMemoryRunner(agent=root_agent, app_name="my_app") wires them up for you — good for tests, never for production.

Create the session before you run, or opt out explicitly. Calling run_async with a session id that does not exist raises SessionNotFoundError with the message Session <id> not found.. If you would rather sessions appear on demand, pass auto_create_session=True to the Runner. Note that since 2.9 InMemorySessionService also raises this on appending to an unknown session, where it previously discarded the event silently — strictly better, and a change that surfaced some long-standing bugs in people's code.

The message must be a real types.Content with at least one part. Passing nothing produces ValueError: No parts in the new_message. or A new message is required for a new invocation..

run_async is an async generator and yields every event, not just the answer: partial streaming chunks, tool calls, tool results, state changes. That is a feature — it is how you build a UI that shows "looking up delivery times…" — but it means you must filter. event.is_final_response() is the public helper for "this is the answer". A synchronous runner.run(...) also exists for scripts, and since 2.8 it re-raises agent errors rather than swallowing them.

Configuration, and the errors you will actually hit

Most ADK configuration is environment variables and command-line flags rather than a config file, which is convenient once you know the names. A few flag conventions first: ADK's flags are snake_case with underscores — --session_service_uri, --log_level, --save_session — with a handful of hyphenated exceptions such as --reload/--no-reload, --provider-args, --logo-text and --allow-unsafe-unpickling. When a flag "does not exist", check the separator before anything else.

The environment variables worth knowing early:

Now the errors. Here are the ones a beginner meets in their first week, with what each one actually means.

No root_agent found for '<name>'. Searched in '<name>.agent.root_agent', '<name>.root_agent' and '<name>/root_agent.yaml'. — the layout. You are inside the agent folder instead of its parent, or __init__.py does not import agent, or the variable is not called root_agent.

Agent not found: '<name>'. No matching directory or module ... — the app name in your URL or command does not match a folder. GET /list-apps on the API server lists the valid names. Since 2.5 an unknown app_name returns a sensible 404 rather than a 500.

Max number of llm calls limit of `500` exceeded (LlmCallsLimitExceededError) — the loop ran away. Two agents transferring to each other forever, or a tool whose error message makes the model retry indefinitely. Treat it as a bug to fix, not a limit to raise. If you genuinely need more headroom, RunConfig(max_llm_calls=N) or ADK_MAX_LLM_CALLS will give it, and a value of zero or less means unlimited with a warning — which you should essentially never want.

Session <id> not found. and Session with id <id> already exists. — the two halves of session lifecycle. Create it first, or auto_create_session=True; and do not call create_session with an id that is already taken when you meant get_session. One small gotcha from 2.10: the session id "user" is now reserved and rejected.

The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db] — a missing extra, and the message tells you which. The same pattern applies for gcp, eval, a2a and the rest.

Model <name> not found. — you passed a bare model string that is not a Gemini model. Non-Google models need their wrapper class and their extra: LiteLlm("openai/gpt-...") from google.adk.models.lite_llm, or Claude/AnthropicLlm for Anthropic, or the OpenAI integration in google.adk.integrations.openai with pip install "google-adk[openai]".

google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED — model quota or rate limiting, not your code. Back off and retry; ADK has retry plugins and, since 2.9, automatic failover to a backup model.

A warning that the database is using the legacy v0 schema, or schema errors about missing node_info or output columns, means a session database created by an older ADK. The 2.0 event schema added fields. The command is adk migrate session --source_db_url ... --dest_db_url ..., and for a throwaway local database, deleting .adk/session.db is faster.

Putting it all together

One small project that uses everything above. A support triage agent: it classifies an incoming message deterministically, routes technical issues to an agent with a diagnostic tool, sends everything else to a general agent, and stores the outcome in state.

from google.adk import Workflow, Event
from google.adk.agents.llm_agent import Agent
from google.adk.workflow import DEFAULT_ROUTE

KNOWN_ISSUES = {
    "login": "Clear your browser cookies for the site and sign in again.",
    "upload": "Files must be under 25 MB. Larger uploads are rejected.",
}

def lookup_known_issue(topic: str) -> dict:
    """Look up the standard remedy for a known product issue.

    Call this before answering any technical question.

    Args:
        topic: A single lowercase keyword for the problem area, such as
            "login" or "upload".
    """
    remedy = KNOWN_ISSUES.get(topic.lower())
    if remedy is None:
        return {"status": "not_found", "topic": topic}
    return {"status": "ok", "topic": topic.lower(), "remedy": remedy}

def classify(node_input: str):
    """Deterministic triage. No model call, no cost."""
    text = node_input.lower()
    technical = any(word in text for word in ("error", "crash", "login", "upload"))
    return Event(route="TECHNICAL" if technical else "GENERAL", output=node_input)

technical_agent = Agent(
    model="gemini-3.5-flash",
    name="technical_agent",
    description="Diagnoses product errors using the known-issue list.",
    instruction=(
        "You are a support engineer. Always call lookup_known_issue first with a "
        "single keyword. If status is 'ok', give the remedy in your own words. "
        "If status is 'not_found', say the issue is not in the known list and ask "
        "for the exact error message. Never invent a remedy."
    ),
    tools=[lookup_known_issue],
    output_key="triage_outcome",
)

general_agent = Agent(
    model="gemini-3.5-flash",
    name="general_agent",
    description="Handles billing, account and general enquiries.",
    instruction=(
        "You handle non-technical support questions in at most three sentences. "
        "If the question is clearly a product fault, say it needs technical support."
    ),
    output_key="triage_outcome",
)

root_agent = Workflow(
    name="support_triage",
    edges=[
        ("START", classify),
        (classify, {"TECHNICAL": technical_agent, DEFAULT_ROUTE: general_agent}),
    ],
)

Walk through what happens when someone writes "I get an error when I upload a file". classify runs first as a plain function: it matches "error" and "upload", returns route="TECHNICAL", and passes the text through as output. ADK follows the "TECHNICAL" edge to technical_agent, which receives the text and, following its instruction, calls lookup_known_issue("upload"). The tool returns a dict with status: "ok" and a remedy. The model reads that and composes an answer. Because output_key is set, the final text is written to state["triage_outcome"], where any later step or any dashboard you build can read it.

Now notice what the structure buys you. The classification costs nothing, because it is string matching rather than a model call, and it is deterministic, so it is trivially unit-testable with plain pytest and no model at all. The expensive model calls happen only on the branch that needs judgement. The two agents have genuinely distinct description values, so if you later turn this into a delegating multi-agent system the model has something real to choose between. And the hallucination risk is addressed in the one place it can be: the instruction says explicitly what to do when the tool finds nothing.

Run it with adk web, send both kinds of message, and read the Events panel for each. On the technical branch you will see five events — your message, the classify node's routed event, the function call with topic: "upload", the function result, and the final response. On the general branch you will see three. That difference, visible in the event list, is the graph doing its job.

What you can now do, and what comes next

You can install ADK and verify it properly, configure either credential path and recognise the errors when they do not match, scaffold and run an agent, write tools whose docstrings actually steer the model, read an invocation's event list instead of guessing, persist facts in state at the right scope, build a graph that routes work between deterministic functions and model-driven agents, and drive the whole thing from your own async code. That is a genuinely useful agent, and you understand why each piece is there.

Several things were named and deliberately left shallow, and they are the natural next steps. Multi-agent delegation through sub_agents and transfer_to_agent, where the model chooses which specialist handles a request based on their descriptions. Callbacks — before_model_callback, before_tool_callback and their siblings — which let you inspect, modify or block a step, and are the main hook for guardrails. Plugins, which are bundles of callbacks applied globally to an App. Evaluation: eval sets, adk eval, and AgentEvaluator in pytest, which is how you stop agent quality from regressing silently when you change a prompt. MCP, the Model Context Protocol, which lets your agent consume tools from external servers through McpToolset and expose its own through to_mcp_server. And deployment: adk deploy cloud_run, adk deploy gke, adk deploy agent_engine for Google's managed Agent Runtime, and adk deploy docker for any container host.

Two warnings to carry forward. First, never run production behind in-memory services with more than one replica — sessions would be split across instances, and on Cloud Run or Kubernetes, where the agent directory is often read-only, ADK can silently fall back to in-memory storage unless you set service URIs explicitly. Second, the dev server is unauthenticated by design; real deployments need your own authentication layer, and the API server does not verify that the user_id in a request belongs to the caller.

For neighbouring tools in this catalogue, the obvious next reads are LangGraph for a different and instructive take on graph-based agent orchestration, Langfuse for tracing and evaluating agent runs across frameworks, MCP for the tool protocol ADK speaks, and Vertex AI for the managed platform ADK deploys onto most smoothly. If your agent will do retrieval, RAGAS covers measuring whether it retrieves well.

Then read Mid-level, which takes the execution model apart far enough that you can predict ADK's behaviour rather than observe it, and covers production services, testing, callbacks and the integration surface properly.

Sources

• ADK documentation home
• Installation
• Get started with Python
• What changed in ADK 2.0
• Graphs
• Graph routes
• Graph data handling
• Command-line reference
• Session state
• Safety and security
• Deploy
• Release notes
• Python API reference
• adk-python CHANGELOG
• google-adk on PyPI

This is what to have ready when Google ADK comes up in an interview at the beginner level. It assumes you have worked through the detailed guide. The goal is not to recite the API but to sound like someone who has actually run an agent, watched it misbehave, and understood why.

One piece of context sets the tone for everything below. ADK changed substantially between version 1.x and 2.0, which went generally available in May 2026, and most written material online still describes 1.x. An interviewer who works with ADK knows this. Being the candidate who says "that was the 1.x model; in 2.x it's a graph of nodes" is an unusually cheap way to look current.

The thirty-second answer

"Google ADK is an open-source framework for building agents — programs where a language model decides what to do next by calling functions you give it. It handles the loop of model call, tool call, model call again, plus conversation storage, tracing, evaluation and deployment, so you only write the instructions, the tools and the shape of the workflow. It's code-first Python, optimised for Gemini and Google Cloud but model-agnostic: it also supports Claude, OpenAI and anything behind LiteLLM."

If they want more, add one sentence about the version: "Since 2.0 the execution engine is graph-based — agents, plain functions and tools are all nodes in a workflow graph — which replaced the old sequential, parallel and loop agent classes."

Do not open with Google Cloud. ADK's appeal is that it is not tied to it, and candidates who lead with the cloud story often reveal they have only read the marketing page.

The model, in four nouns

Have these four ready as crisp one-liners, because "explain the main concepts" is the standard second question.

Agent. A model plus an instruction plus a list of tools it may call. The class is LlmAgent, and Agent is an alias for it. Key fields: name, model, instruction (what this agent should do), description (one sentence other agents read when deciding whether to delegate), tools, and output_key.

Tool. A capability the model may invoke. In the common case it is a plain Python function: ADK wraps it automatically and builds the schema from your type hints and the description from your docstring. That means the docstring is production code, not documentation — a point worth making explicitly, because it signals you have actually debugged one.

Session. One conversation thread, identified by app_name, user_id and a session id together. It holds the ordered event list and a key-value scratchpad called state. It is created and loaded by a SessionService.

Runner. The orchestrator. You give it a user message; it loads the session, drives the loop, calls the model, executes tools, persists events and yields events back. Constructed with a required keyword-only session_service plus an agent, a node or an App.

Then one more sentence that ties them together: "An invocation is one user turn — everything from the message arriving to the final answer — and it can involve many model calls, many tool calls and several agents."

The commands to have ready

You will not be asked to recite flags, but five commands should come out without hesitation.

pip install google-adk      # needs Python 3.10+
adk --version               # prints: adk, version 2.10.0
adk create my_agent         # scaffolds .env, .gitignore, __init__.py, agent.py
adk run my_agent            # interactive terminal chat
adk web --port 8000         # local dev UI, run from the PARENT directory

Know what adk web is for, because that is the follow-up. Its value is not the chat box; it is the Events panel, which lists every event in an invocation — your message, the model's decision, the exact tool call arguments, the exact tool return value, the final answer — and the Trace panel, which shows the same run as timed spans. Say that you debug agents by reading the event list rather than by guessing, and you will sound like a practitioner.

Also know that it is development-only. Its own help text says the endpoints are unauthenticated. The headless equivalent is adk api_server, with the same warning.

Two more worth having: adk eval for running saved evaluation sets, and adk deploy with its four subcommands — cloud_run, gke, agent_engine and docker. One flag convention is worth a sentence because it shows familiarity: ADK's flags are snake_case with underscores, like --session_service_uri, with a few hyphenated exceptions such as --reload/--no-reload.

The minimal agent, from memory

Be able to write this on a whiteboard:

from google.adk.agents.llm_agent import Agent

root_agent = Agent(
    model='gemini-3.5-flash',
    name='root_agent',
    description='A helpful assistant for user questions.',
    instruction='Answer user questions to the best of your knowledge',
)

with my_agent/__init__.py containing from . import agent. Then say what is load-bearing: the variable must be named root_agent, the __init__.py must import the agent module, and the agent folder sits inside a parent directory you run the tools from. Those three facts are the cause of the single most common first-run error, so volunteering them is worth more than the code itself.

The programmatic version is the other thing worth being able to sketch:

runner = Runner(app_name="my_app", agent=root_agent,
                session_service=InMemorySessionService())
session = await runner.session_service.create_session(app_name="my_app", user_id="u1")
async for event in runner.run_async(user_id="u1", session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text="hi")])):
    if event.is_final_response():
        print(event.content.parts[0].text)

Three things to point out while writing it: session_service is required and keyword-only, because where conversations live is an architectural choice the framework refuses to guess; you must create the session first unless you pass auto_create_session=True; and run_async yields every event, so is_final_response() is how you pick out the answer.

The classic traps, and why they happen

Interviewers at this level probe for scar tissue. These are the ones that produce it.

Running from the wrong directory. adk web and adk run want the parent of the agent folder. Run them from inside it and you get No root_agent found for '<name>', which helpfully lists the three places it searched. Related causes: an empty __init__.py, or a variable named something other than root_agent.

Runaway loops. An invocation is capped at 500 model calls by default, and exceeding it raises LlmCallsLimitExceededError with the message that the limit of 500 was exceeded. The right answer to "how do you fix that?" is not "raise the limit". It is: find the loop. Two agents transferring back and forth, or a tool whose error message makes the model keep retrying. You can override with RunConfig(max_llm_calls=N) or ADK_MAX_LLM_CALLS, and a good habit is to set the cap low during development so loops fail fast and cheaply.

The deprecated environment variable. The flag that chooses Vertex AI over AI Studio was GOOGLE_GENAI_USE_VERTEXAI; since 2.3 it is GOOGLE_GENAI_USE_ENTERPRISE. The old name still works but warns. Mentioning this unprompted is a strong signal, because it is the difference between someone who ran ADK recently and someone who read a 2025 blog post.

Credential and flag mismatch. An AI Studio key with the enterprise flag set to 1, or Cloud credentials with it set to 0, fails with API key not valid, 400 INVALID_ARGUMENT or 403 PERMISSION_DENIED — errors that look like a bad key. Check the flag against the credential type before regenerating anything.

Synchronous database URLs. ADK's database session service uses SQLAlchemy's async layer, so postgresql:// fails and postgresql+asyncpg:// works. The async spellings are sqlite+aiosqlite, postgresql+asyncpg, mysql+aiomysql, mariadb+asyncmy. You also need the db extra, and ADK tells you so: The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db].

Vague docstrings. Because the tool schema is generated from your function's type hints and docstring, a vague docstring produces a tool the model calls at the wrong times with the wrong arguments. The fix is a docstring with an explicit trigger condition and an Args block specifying exact formats.

Using the deprecated orchestrators. SequentialAgent, ParallelAgent and LoopAgent still import and run in 2.x, but warn. The replacement is Workflow.

Common interview questions

"What is ADK and when would you use it?" Use the thirty-second answer, then give a shape: you reach for ADK when a task needs a model to decide something and then act — look things up, combine results, hand off to a specialist — rather than produce a single piece of text. For a one-shot prompt, a plain model API call is cheaper and simpler, and saying so demonstrates judgement.

"Walk me through what happens when a user sends a message." One user message starts one invocation. The runner loads the session and sends the message plus the instruction plus the tool schemas to the model. The model either answers or asks to call a tool. If it asks, ADK runs your function and sends the result back. That repeats until the model answers or the call cap trips. Everything along the way — message, model response, tool call, tool result, state change — is recorded as an ordered, immutable event, which is what makes the run debuggable afterwards.

"How does a Python function become a tool the model can call?" You pass it in the agent's tools list and ADK wraps it as a FunctionTool. The parameter schema comes from your type annotations and the description from your docstring. Then add the practical part: return a dict rather than a bare string so the model can read it as data, make expected failures a status field rather than an exception, and state in the instruction what the agent should do for each outcome — otherwise a model with no data and a desire to help will invent some.

"What's the difference between state, artifacts and memory?" State is a key-value scratchpad on the session: small facts about this conversation. Artifacts are named, versioned binary blobs — files, images — held by an ArtifactService rather than stuffed into state. Memory is long-term recall across sessions, served by a MemoryService, reached through the load_memory and preload_memory tools. The distinction is scope, and state's prefixes make that explicit: no prefix is session-scoped, user: spans that user's sessions, app: is shared by every user of the app, and temp: lives only for the current invocation and is never persisted. If you can add that app: is shared by all users and therefore must never hold per-user data, you have answered a security question they did not ask yet.

"How do you run several steps in a fixed order?" A Workflow: a graph of nodes with edges. ("START", classify) sets the entry point, a longer tuple is a chain, and a dictionary edge like (router, {"BUG": handler, DEFAULT_ROUTE: other}) routes on the route value a node sets on its event. A node's output becomes the next node's node_input. Nodes can be agents, plain Python functions, tools or nested workflows. Then the version point: this replaced SequentialAgent, ParallelAgent and LoopAgent, which are deprecated in 2.x and warn when used.

"Why would you use a plain function as a node instead of an agent?" Because it costs nothing and is deterministic. Classification by keyword match, reformatting, writing a row — none of that needs a model. Pushing deterministic work out of the model saves money, removes latency, removes a source of nondeterminism, and makes that step unit-testable with plain pytest and no network. Reserve model calls for the steps that genuinely need judgement.

"How do you debug an agent that gives a wrong answer?" Read the event list for that invocation, in adk web's Events panel or by printing the events from run_async. You will see the exact arguments the model sent to each tool and the exact values they returned, and the cause is usually visible immediately: the model called the wrong tool, or called the right one with an argument format you had not anticipated. Then say what you change — the docstring, the instruction, or the tool's return shape — rather than "I'd adjust the prompt".

"How would you deploy this?" Not with adk web, which is unauthenticated and development-only. The options are adk deploy cloud_run, adk deploy gke, adk deploy agent_engine for Google's managed Agent Runtime, or adk deploy docker for any container host. Then the one production fact a beginner should know: never run in-memory services behind more than one replica, because sessions would be split across instances. Use a database or a managed session service, and set the service URIs explicitly, because on Cloud Run or Kubernetes the agent directory is often read-only and ADK can silently fall back to in-memory storage.

"Does ADK lock you into Gemini?" No. It is optimised for Gemini and the default model is gemini-3.5-flash, but Claude, OpenAI and the hundred-odd providers behind LiteLLM are supported, including locally served models through Ollama. Non-Google models need their wrapper class and the matching extra; a bare unrecognised model string fails with Model <name> not found.. This is a useful answer to have for employers with data-residency constraints, which is a live concern for many Gulf and Egyptian organisations.

What separates a strong beginner answer from a weak one

At this level nobody expects architectural depth. What they are listening for is whether you have run the thing. Four habits make the difference, and they are learnable in an afternoon.

Name the version and the change. A weak answer describes ADK generically. A strong one says "in 2.x" or "since 2.3" and attaches the claim to a release. You do not need the whole changelog; four facts carry almost all the credibility — graph workflows replaced the three orchestrator classes in 2.0, GOOGLE_GENAI_USE_ENTERPRISE replaced GOOGLE_GENAI_USE_VERTEXAI in 2.3, the default model is gemini-3.5-flash, and the current Python release is 2.10.

Quote the error, not the feeling. "It didn't find my agent" is forgettable. "It printed No root_agent found, and listed the three module paths it searched, so I knew immediately I was one directory too deep" is the answer of someone who reads output. Error messages are the cheapest possible evidence of hands-on work, and ADK's are unusually good, so use them.

Reach for determinism before prompting. When asked how to make an agent more reliable, the weak instinct is to improve the prompt. The strong instinct is to ask which part of the task did not need a model at all, move it into a plain function node, and shrink the surface the model can get wrong. Prompt improvements are the second move, not the first.

Say what you would measure. Beginners rarely mention evaluation, so mentioning it stands out. You do not need to know the eval format in detail; it is enough to say that agent behaviour drifts whenever a prompt, a tool or a model version changes, that ADK ships eval sets and an adk eval command plus a pytest entry point for exactly that reason, and that you would pin the ADK version and run the eval set in CI before upgrading — because several recent minor releases contained real behaviour changes.

A fifth habit is negative: do not overclaim. ADK has a large surface — multi-agent delegation, callbacks, plugins, MCP, A2A, live audio streaming, prompt optimisation, four deployment targets. Saying "I have built single agents with tools and a routing workflow, and I have read about the multi-agent and evaluation pieces but not used them" is a better answer than a vague gesture at all of it. Interviewers probe claims, and a confident boundary is far more reassuring than an unconvincing breadth.

Sixty-second self-test

Answer out loud, without looking.

1. Name the four core nouns and define each in one sentence.
2. What two files does a minimal agent folder need, and what must each contain?
3. From which directory do you run adk web, and what error tells you that you got it wrong?
4. What is the default per-invocation model-call cap, and what is the right response to exceeding it?
5. Which environment variable chooses Vertex AI over AI Studio, and what was it called before?
6. Where does a tool's parameter schema come from?
7. What is the difference between instruction and description?
8. What replaced SequentialAgent, ParallelAgent and LoopAgent?
9. What do the user:, app: and temp: state prefixes scope to?
10. Why must a database session URL use a driver like asyncpg rather than the default?
11. What does run_async yield, and how do you pick out the answer?
12. Why is adk web unsuitable for production?

If any answer takes longer than five seconds, go back to that section of the detailed guide. Numbers 3, 4, 5 and 9 are the ones most likely to come up, because each one is a mistake the interviewer has watched someone make.

These are the mistakes people actually make in their first fortnight with Google ADK, each with the symptom you will see, the reason it happens, and the fix. Then the habits worth adopting on day one, and a checklist to run before you show anyone your agent.

Everything here is written against ADK 2.10 for Python. A general note that explains an unusual number of the problems below: ADK 2.0 arrived in May 2026 and changed a great deal, and most tutorial material online — including whatever a chatbot remembers — still describes 1.x. When something you copied does not work, "this is 1.x advice" is a genuinely high-probability explanation.

The mistakes beginners actually make

Running the commands from inside the agent folder

Symptom. No root_agent found for 'my_agent'. Searched in 'my_agent.agent.root_agent', 'my_agent.root_agent' and 'my_agent/root_agent.yaml'.

Cause. adk run and adk web take the agent's folder name and expect to be run from its parent. If you are inside my_agent and type adk run my_agent, ADK looks for my_agent/my_agent/, which does not exist.

Fix. Keep a parent directory — agents/ — with one folder per agent inside it, and always run the tools from the parent. If the layout is right and you still see this, check the other two causes: my_agent/__init__.py must contain from . import agent, and the variable in agent.py must be named exactly root_agent. A variant message, A 'root_agent' was found ... but it is not a ... 'root_agent', means the name exists but holds the wrong kind of object — most often a function you forgot to call, or a module.

Copying GOOGLE_GENAI_USE_VERTEXAI out of an old tutorial

Symptom. Everything works but you see GOOGLE_GENAI_USE_VERTEXAI is deprecated, please use GOOGLE_GENAI_USE_ENTERPRISE instead on every run.

Cause. The variable that chooses between Vertex AI and AI Studio was renamed in ADK 2.3. The old name still functions through a deprecation fallback.

Fix. Rename it in your .env to GOOGLE_GENAI_USE_ENTERPRISE. Do not set both. adk create writes the new name for you, so the clearest signal that a tutorial predates mid-2026 is that it writes the old one.

Mismatching the credential and the backend flag

Symptom. API key not valid, 400 INVALID_ARGUMENT, or 403 PERMISSION_DENIED — all of which look like a bad key.

Cause. The flag tells the GenAI client which service to call, and the credential has to match. An AI Studio key with GOOGLE_GENAI_USE_ENTERPRISE=1, or Cloud credentials with =0, fails at authentication.

Fix. Check the flag against the credential type before you regenerate anything. AI Studio means =0 plus GOOGLE_API_KEY. Vertex means =1 plus GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION and gcloud auth application-default login. People waste a surprising amount of time rotating a key that was never the problem.

Installing into one environment and running from another

Symptom. adk --version works but import google.adk fails, or the reverse.

Cause. A virtual environment that was created but not activated, or activated in one terminal tab and not another.

Fix. Verify twice, always: adk --version proves the CLI is on your PATH, and python -c "import google.adk; print(google.adk.__version__)" proves the library is importable from the interpreter you are using. If the two disagree, stop debugging anything else. pip show google-adk prints the install location, which settles it.

Writing a vague tool docstring

Symptom. The agent calls your tool at the wrong moment, or calls it with "United Arab Emirates" when it needs "AE".

Cause. ADK generates the tool schema from your type annotations and the tool description from your docstring. That text is the only explanation the model receives. A one-word docstring and an unannotated argument give the model nothing to work with.

Fix. Write the docstring as an instruction to a colleague who cannot see your code. First line: what it does. Second line: when to call it. Then an Args block giving the exact expected format of each argument.

Returning a bare string from a tool, and raising on expected failures

Symptom. The agent paraphrases an error message to the user as if it were an answer, or an ordinary "not found" case crashes the invocation.

Cause. The model reads a tool's return value as data. A bare string gives it nothing structured to branch on, and an exception removes its chance to respond at all.

Fix. Return a dict with a status key. Make expected outcomes values — {"status": "not_found", "topic": topic} — and reserve exceptions for genuine bugs. Then close the loop in the instruction by saying what to do with each status, including an explicit prohibition on guessing.

Letting the model do work that did not need a model

Symptom. An agent that is slow, expensive, and occasionally gets a trivial step wrong for no discoverable reason.

Cause. Classification, reformatting and record-writing were put inside an instruction instead of in code.

Fix. In ADK 2.x a plain Python function is a first-class node in a Workflow. It costs nothing, never varies, and is unit-testable without a network. Move the deterministic steps out and reserve model calls for judgement. This single habit improves cost, latency, reliability and testability at once, which is rare.

Using the deprecated orchestrator classes

Symptom. DeprecationWarning: SequentialAgent is deprecated in favor of Workflow and will be removed in a future version. Workflow cannot yet be used as an LlmAgent sub-agent.

Cause. You followed a 1.x tutorial. SequentialAgent, ParallelAgent and LoopAgent still import and run in 2.x, which is exactly why the mistake persists.

Fix. Use Workflow with edges. ("START", a, b, c) is a chain; a dictionary edge routes on the route value a node sets on its event, with DEFAULT_ROUTE as the fallback. Note the second sentence of the warning, which is a real current limitation worth remembering: a Workflow cannot yet be used as an LlmAgent sub-agent.

Forgetting DEFAULT_ROUTE

Symptom. Some messages produce no response at all, and the event list simply stops after the routing node.

Cause. Routing is exhaustive matching. A route value with no matching edge has nowhere to go.

Fix. Always include DEFAULT_ROUTE in a routing dictionary. It is the else branch, and the same reasoning applies.

Swallowing exceptions inside a node

Symptom. Transient failures are never retried, even though you configured retry_config.

Cause. A broad except Exception inside a node means ADK never learns the node failed, so its automatic retry machinery does not engage.

Fix. Let genuine exceptions propagate and let the framework handle retries. Related, and worth knowing before it bites: since 2.9 a node that failed runs again when an invocation resumes, rather than replaying as if it had completed. Node bodies with side effects must therefore be idempotent. Both of these are invisible in testing and expensive in production.

Using a synchronous database URL

Symptom. Database related module not found for URL '<redacted>' or Invalid database URL format or argument.

Cause. ADK's database session service uses SQLAlchemy's asyncio layer, so the URL needs an async driver.

Fix. Use sqlite+aiosqlite, postgresql+asyncpg, mysql+aiomysql or mariadb+asyncmy, and install the db extra. If the extra is missing you will get a message that names it precisely: pip install google-adk[db].

Forgetting to quote extras in zsh

Symptom. zsh: no matches found: google-adk[eval,gcp].

Cause. zsh, the default shell on macOS, treats square brackets as a glob pattern.

Fix. Quote them: pip install "google-adk[eval,gcp]".

Running run_async without creating a session

Symptom. Session <id> not found. (SessionNotFoundError).

Cause. You passed a session id that was never created. Since 2.9 this also fires in InMemorySessionService when appending to an unknown session, where it previously discarded the event silently.

Fix. Call create_session first, or pass auto_create_session=True to the Runner, or use --auto_create_session on the API server. The mirror-image error, Session with id <id> already exists., means you called create_session when you meant get_session. One small 2.10 detail: the session id "user" is reserved and rejected.

Treating the call cap as the problem

Symptom. Max number of llm calls limit of `500` exceeded (LlmCallsLimitExceededError).

Cause. The loop ran away — two agents transferring to each other, or a tool whose error text makes the model retry forever.

Fix. Fix the loop, not the limit. Read the event list and you will see the repeating pair immediately. RunConfig(max_llm_calls=N) and ADK_MAX_LLM_CALLS exist, and a value of zero or less means unlimited with a warning, which you should essentially never want.

Exposing the dev server

Symptom. None, until it matters.

Cause. adk web and adk api_server are unauthenticated by design. The help text says so: run them on a trusted network only.

Fix. Keep them on 127.0.0.1, which is the default. Do not port-forward them out of a cluster, and do not deploy with --with_ui and call it production. Restrict CORS with --allow_origins and put real deployments behind your own authentication.

Putting per-user data in app: state

Symptom. One user sees another user's details.

Cause. The app: state prefix is shared by every user of the application. It is for shared configuration.

Fix. Use no prefix for this conversation, user: for that user across their sessions, and temp: for values that should not be persisted past the current invocation. Note also that the API server does not verify user_id — your own auth layer must map the authenticated principal to it and stop anyone choosing someone else's.

Habits worth adopting on day one

Read the Events panel before forming a theory. Start adk web, send the message, and open Events. You will see the exact arguments the model sent and the exact value your function returned. Nine times in ten the cause is right there, and it is usually not what you assumed. Guessing at prompts without reading the event list is the most expensive habit in this field.

Run adk web --reload_agents. It picks up edits to your agent files without a restart, turning the edit-and-retry cycle from twenty seconds into two. Over an afternoon of instruction tuning that compounds into real time.

Set the call cap low while developing. ADK_MAX_LLM_CALLS=5 turns a runaway loop from an expensive mystery into a two-second error. Raise it when the agent legitimately needs more.

Pin the version. Write google-adk==2.10.0 in your requirements file. ADK ships a minor release roughly every two weeks, and 2.6, 2.7, 2.9 and 2.10 each contained behaviour changes. ADK also publishes constraints files for fully reproducible installs. Upgrade on purpose, after reading the release notes' breaking-changes section.

Treat docstrings as code in review. When you review a teammate's tool, read the docstring the way the model will: does it say when to call this, and does it pin the argument formats? A docstring change is a behaviour change, so it deserves the same attention as a logic change.

Write the failure path into the instruction. For every tool, state what the agent should do when it returns nothing useful, including an explicit instruction not to invent an answer. Models fill gaps helpfully and confidently, and this one sentence is the cheapest hallucination control available.

Unit-test the deterministic nodes. Any plain-function node is testable with pytest in milliseconds, with no model and no network. If you have pushed classification and formatting out of the model, you now have real tests on the parts that should never vary.

Name agents and their descriptions as if a stranger must choose between them. instruction is read by this agent's model; description is read by other agents' models when they decide whether to hand work over. Two agents whose descriptions could be swapped without anyone noticing will be confused for each other the moment you add delegation, and the symptom — work going to the wrong specialist — looks like a model problem rather than a one-line text problem.

Learn the flag convention once. ADK's command-line flags are snake_case with underscores: --session_service_uri, --save_session, --log_level, --reload_agents. A handful are hyphenated instead — --reload/--no-reload, --provider-args, --logo-text, --allow-unsafe-unpickling. When a flag appears not to exist, check the separator before you go looking for a version difference.

Reach for --log_level debug and --replay instead of print statements. adk run --log_level debug surfaces the detail you need when behaviour is inexplicable, and --save_session followed by --replay FILE.json makes a bug reproducible for a colleague, which is far more useful than a description of what you saw.

Keep .env out of git and out of production. adk create writes a .gitignore and warns that secrets live in .env. Honour both. In production, use Secret Manager or workload identity, never a key baked into an image.

Delete .adk/session.db when local sessions act strange. Local runs persist sessions to <agents_dir>/<agent>/.adk/session.db and artifacts alongside. After a major upgrade a stale database can produce schema errors about missing fields; for a throwaway dev database, deleting it is the fastest fix. For data you care about, adk migrate session --source_db_url ... --dest_db_url ... is the proper route.

Choose the deployment services deliberately. Never run in-memory services behind more than one replica; sessions would be split across instances. On Cloud Run or Kubernetes the agent directory is often read-only and ADK can silently fall back to in-memory storage unless you pass explicit service URIs.

The checklist

Run this before you show your agent to anybody.

If every line passes, you have an agent that behaves predictably, fails honestly, costs what you expect, and will not embarrass you in a review. That is a better position than most first agents reach, and it is entirely a matter of these habits rather than of any advanced feature.

This is part two of three. Beginner left you with an agent that runs: a folder, a root_agent, a couple of function tools, adk run and adk web. That is enough to demo. It is not enough to put an agent in front of users, because the questions that decide whether an agent works in production are not "does it answer" but "where does its state live", "what happens on the third retry", "how much did that conversation cost", and "what exactly did the model see".

This level answers those. It takes every Beginner topic further and adds the machinery that only matters once the agent has a second replica, a budget, and someone on call.

Where this picks up

Everything here is verified against ADK 2.10.0 for Python (pip install google-adk, adk --version prints adk, version 2.10.0), which needs Python 3.10 or later. ADK 2.x is a genuinely different runtime from 1.x, so if a tutorial you find online builds sequences out of SequentialAgent, it predates the graph engine and you should read it with suspicion.

Each section opens with the problem it solves and closes with a Try it you can run against a real agent in a few minutes.

The runtime: how an invocation actually executes

Beginner treated the Runner as the thing you call. To predict ADK's behaviour you need its actual job, which is narrower and more interesting than "run the agent".

One invocation is one user turn: everything from receiving a user message to the final response. Inside that turn there may be a dozen model calls, twenty tool calls and three agent transfers. The unit of truth inside an invocation is not a variable, it is the Event. An event is an immutable record of something that happened — a user message, a model response, a tool call, a tool result, a state change, a control signal. The runner's loop is: hand control to the agent or graph, receive events as they are yielded, persist each one through the session service, and pass it on to the caller.

That design has consequences you will feel.

The session is a replayable log, not a mutable object. The session's state dictionary is a projection of the state_delta fields on its events. When ADK restores a session from a database, it is replaying that log. This is why the 2.x breaking-change list includes a blunt instruction: appending directly with context.session.events.append() breaks determinism, and you must yield events instead. A directly appended event bypasses the persistence and ordering the runner guarantees, so the session you reload is not the session you ran.

The event schema changed in 2.0. Events gained node_info and output fields to support the graph engine. A database-backed session store created by ADK 1.x does not have those columns, and the fix is adk migrate session, not a hand-written ALTER TABLE.

Custom _run_async_impl() overrides are silently ignored. In 1.x, subclassing an agent and overriding that method was the documented escape hatch for custom orchestration. In 2.x the graph engine does not call it. Nothing errors; your code simply never runs. If you inherited a 1.x codebase, this is the first thing to grep for.

The 2.0 shift in one sentence: agents, tools and plain functions are now all nodes in a workflow graph, and BaseAgent subclasses BaseNode. The hierarchical executor that walked a tree of agents is gone.

from google.adk import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

runner = Runner(app_name="support", agent=root_agent,
                session_service=InMemorySessionService())

session = await runner.session_service.create_session(app_name="support", user_id="u1")

async for event in runner.run_async(
        user_id="u1",
        session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text="my order is late")])):
    if event.is_final_response():
        print(event.content.parts[0].text)

Two details in that signature matter at work. session_service is keyword-only and required — there is no implicit default, which is deliberate, because an accidental in-memory default is exactly the bug that bites on the second replica. And Runner.__init__ accepts agent=, node= or app=: you pass agent for a single LlmAgent, node for a Workflow graph, and app for an App that bundles a root agent with plugins and caching configuration. auto_create_session=True saves you the create_session call, which is convenient in scripts and dangerous in a service, because it means a typo'd session ID silently becomes a brand-new empty conversation instead of an error.

Graph workflows: edges, routes and fan-out

The problem Beginner's sub_agents solves is delegation by persuasion: the model reads each sub-agent's description and chooses to transfer. That is right when the branching genuinely needs judgement, and wrong when it does not. A ticket triage step that routes on a classifier's label should not depend on a language model deciding to call transfer_to_agent, because the failure mode is a loop and the cost is tokens.

Workflow is the deterministic alternative. It is a graph: nodes plus edges, where a node may be an agent, a plain function, a tool or another Workflow.

from google.adk import Workflow, Event
from google.adk.workflow import DEFAULT_ROUTE

def classify(node_input: str):
    return Event(route="BUG" if "crash" in node_input else "OTHER", output=node_input)

def bug(node_input: str):
    return Event(message=f"bug handler got: {node_input}")

def other(node_input: str):
    return Event(message="other handler")

root_agent = Workflow(
    name="triage",
    edges=[("START", classify),
           (classify, {"BUG": bug, DEFAULT_ROUTE: other})],
)

Read that edges list carefully, because the whole API is in it. START is the entry point, and the first node receives the user's text as node_input. An edge written as a tuple of nodes — ("START", a, b, c) — is a chain: a runs, then b, then c. An edge written as a node plus a dictionary is conditional routing: the node sets route on the event it returns, and the matching key decides which edge fires. DEFAULT_ROUTE is the fallback, and since 2.10 it can fan out to several nodes rather than just one.

Data moves by one rule that is easy to forget: a node's output becomes the next node's node_input. If classify had returned only a route and no output, the downstream handlers would have received nothing. Routing and data are separate channels on the same event.

A route may also be a list of strings, and that is how you fan out: every matching edge fires, and Workflow.max_concurrency bounds how many run at once. To bring the branches back together you need a JoinNode, which waits for several predecessors before continuing. Without a join, the branches simply end.

Every node, whatever its kind, shares a set of fields: name, description, rerun_on_resume, wait_for_output, retry_config, timeout, input_schema, output_schema and state_schema. Workflow adds edges, max_concurrency and graph on top of those, which is what makes nesting work — a Workflow is itself a node with a timeout and a retry policy, so a sub-graph can be retried as a unit.

Three constraints to design around. First, SequentialAgent, ParallelAgent and LoopAgent still import and still run, but they log a deprecation warning whose exact text is worth knowing because it also documents the one gap: SequentialAgent is deprecated in favor of Workflow and will be removed in a future version. Workflow cannot yet be used as an LlmAgent sub-agent. Second, that gap is real — if your architecture needs a model to transfer into a deterministic multi-step pipeline, today you wrap the pipeline differently (Workflow as Tool, added in 2.4) rather than attaching it as a sub-agent. Third, since 2.5 ADK validates that the old orchestrators are not used inside Workflow graphs, so a half-migration fails fast rather than behaving oddly.

State scopes, and which ones survive what

Beginner used state as a scratchpad and instruction templating to read it. At work the question becomes: who else can see this value, and how long does it live? ADK answers with key prefixes, and getting them wrong is how tenant data leaks.

The app: row is the dangerous one. It is shared by every user of the application, so a tenant identifier, a customer name or a document ID written there is visible to the next caller. The isolation keys in ADK are app_name, then user_id, then session_id, in that order; app: state deliberately sits above all of them.

State changes travel as state_delta on an event's actions, which is why the recommended ways to write state all funnel through an event. From a tool you write tool_context.state['key'] = value and ADK turns that into a delta on the tool-result event. From a node you set it on the event you yield. From the runner you can seed a turn with run_async(..., state_delta={...}), which is the clean way to inject request-scoped facts such as the authenticated user's tier without putting them in the prompt text.

Instruction templating reads state back, and this behaviour changed in 2.10, which is the single most likely cause of a prompt regression after an upgrade. {key} is substituted from state. {key?} is substituted if present and left empty if not. {artifact.name} injects an artifact's text. But as of 2.10, ${key} and backslash-escaped \{key} are deliberately left as written instead of being filled from state. If your prompts used ${} because it looked like shell or JavaScript interpolation, they silently stopped working.

from google.adk.agents.llm_agent import Agent

agent = Agent(
    model='gemini-3.5-flash',
    name='order_agent',
    description='Answers questions about a specific customer order.',
    static_instruction=(
        "You are a support agent for an online retailer. "
        "Never reveal internal cost fields. Answer in the customer's language."
    ),
    instruction=(
        "The customer's locale is {user:locale?}. "
        "The order under discussion is {current_order_id}. "
        "Literal braces like ${not_state} are left alone since ADK 2.10."
    ),
    output_key='last_answer',
)

Two fields there deserve naming. static_instruction is a stable prefix, placed where it can be cached, which matters for cost; the templated instruction is the part that changes per turn. output_key saves the agent's final text into state['last_answer'], which is the normal way to pass an agent's answer to a later node without parsing events.

For structured hand-offs, prefer input_schema and output_schema — Pydantic models — over asking the model nicely for JSON. Since 2.10 there is also opt-in function-tool argument validation and coercion, gated behind ADK_ENABLE_FUNCTION_TOOL_ARG_VALIDATION=1, which is worth switching on in staging to find tools whose type hints and real arguments disagree.

Failure handling: retries, timeouts, idempotence and resumption

Agents fail in ways request handlers do not. A model returns a 429, a tool's upstream API times out, a node half-finishes after writing a row. ADK gives you per-node controls for all three, plus a resumption model — and the resumption model has a sharp edge that changed in 2.9.

RetryConfig is a node field: RetryConfig(max_attempts, initial_delay, max_delay, backoff_factor, jitter, exceptions). Setting max_attempts=0 means no retries. timeout is also a node field, and a node that exceeds it raises NodeTimeoutError — either raise the timeout or split the work, because a timeout that fires mid-write is the worst of both outcomes.

from google.adk import Workflow
from google.adk.workflow import RetryConfig, node

@node(
    name="charge_customer",
    retry_config=RetryConfig(max_attempts=3, initial_delay=1.0,
                             backoff_factor=2.0, jitter=True),
    timeout=30,
    rerun_on_resume=False,
)
def charge_customer(node_input: str):
    # Must be idempotent: a failed attempt reruns on resume since 2.9.
    ...

Now the edge. In ADK 2.9, workflow node resumption changed so that a node which failed runs again on resume; before, it replayed as though it had completed. That is better semantics and a breaking change for anyone whose nodes have side effects. A node that posts a payment, sends an email or inserts a row must be idempotent — keyed on an invocation-derived identifier so a second attempt is a no-op. The rerun_on_resume field is the per-node lever when rerunning genuinely is not safe.

Resumption itself comes from App(resumability_config=ResumabilityConfig(is_resumable=True)). With it, an invocation interrupted by a crash or by a human-in-the-loop pause can pick up where it stopped, replaying workflow nodes from the event log. This is the mechanism behind the HITL features: tool confirmation (require_confirmation / request_confirmation), the request_input tool added in 2.2 for asking a human a clarifying question, and long-running tools via LongRunningFunctionTool. All three work by suspending an invocation and resuming it later, which only makes sense if the event log is the truth — see the first section.

Above the node level there are two global guards. max_llm_calls on RunConfig defaults to 500 per invocation, overridable with the ADK_MAX_LLM_CALLS environment variable; exceeding it raises LlmCallsLimitExceededError with the message Max number of llm calls limit of 500 exceeded. A value of zero or less means unlimited, with a warning, and you should treat that as a production anti-pattern: the limit is the thing standing between a transfer loop and a four-figure bill. And since 2.9 ADK can perform automatic model failover to backup models, which pairs well with ReflectAndRetryModelPlugin (2.6) and ReflectAndRetryToolPlugin for the cases where the right answer to a bad response is to try again with feedback rather than to give up.

Tools, callbacks and the places to put a policy

A tool at this level is not just a Python function with a docstring. It is the point where an agent touches the real world, which makes it the point where policy belongs.

ADK gives you four kinds of interception, and choosing the right one is most of the skill:

1. Inside the tool Validate arguments against ToolContext and state. This is where business rules live: "this user may only refund their own orders". It is ordinary Python and it is testable.
2. Tool callbacks before_tool_callback sees the arguments before the call. Return a dict to block the call and hand that dict back to the model as the result; return None to allow it. after_tool_callback can rewrite a result, and on_tool_error_callback catches failures.
3. Model callbacks before_model_callback screens the request — prompt-injection checks, PII redaction, refusals that should never cost a token. Returning content or an LlmResponse short-circuits the model call entirely.
4. Plugins Callback bundles registered once on the App or Runner so they apply to every agent. LoggingPlugin, AutoTracingPlugin, the Model Armor guardrail plugin (2.8), BigQueryAgentAnalyticsPlugin. This is where cross-cutting policy belongs, because per-agent callbacks drift.

The short-circuit semantics are the part people get wrong. A before callback that returns a value replaces the step; a before callback that returns None lets it proceed. So a guardrail is a callback that returns something only when it wants to intervene.

def block_bulk_refunds(tool, args, tool_context):
    """before_tool_callback: deny, or return None to allow."""
    if tool.name == "issue_refund" and args.get("amount", 0) > 500:
        return {"error": "Refunds over 500 require a human approver.",
                "escalate": True}
    return None

agent = Agent(
    model='gemini-3.5-flash',
    name='refund_agent',
    description='Handles refund requests within policy limits.',
    instruction='Help the customer. Follow refund policy exactly.',
    tools=[issue_refund],
    before_tool_callback=block_bulk_refunds,
)

Beyond plain functions, the tool types you will actually reach for: AgentTool wraps an agent as a tool, and the docs recommend mode='single_turn' for it, because a chat-mode sub-agent used as a tool is a loop waiting to happen. McpToolset consumes an MCP server, which is how you share one tool implementation across ADK, Claude and anything else that speaks the protocol — see the MCP guide for the protocol itself. OpenAPIToolset(spec_str=..., spec_str_type="json") turns an existing REST specification into tools, which is often the fastest honest path to integrating an internal service. to_mcp_server (2.5) goes the other way and serves your ADK agent as an MCP server. LongRunningFunctionTool is for work that outlives a turn.

Two cautions. Tool output is untrusted input: ADK 2.8 through 2.10 added fencing for relayed agent output and for server-supplied MCP tool descriptions, and 2.10 stopped executing code found in model reasoning — but your own UI still has to escape model output, or you have shipped an XSS. And McpToolset over stdio on Windows has hit NotImplementedError from asyncio subprocess handling; run adk web --no-reload, or use WSL.

Services: where state lives when there are two replicas

This is the section that decides whether your agent survives contact with production, and it has exactly one headline: never run in-memory services behind more than one replica.

ADK's serving process is stateless — a FastAPI app plus the Runner. All durable state sits in pluggable services:

Each is selectable by URI on adk run, adk web, adk api_server and every adk deploy subcommand:

adk api_server agents/ \
  --host 0.0.0.0 --port 8000 \
  --session_service_uri "postgresql+asyncpg://adk:$PGPASSWORD@db:5432/adk" \
  --artifact_service_uri "gs://my-agent-artifacts" \
  --memory_service_uri "rag://my-corpus-id" \
  --allow_origins "https://app.example.com" \
  --otel_to_cloud

--session_service_uri accepts memory://, sqlite://<path>, any SQLAlchemy URL, or agentengine://<id>. --artifact_service_uri takes gs://<bucket>, memory:// or file://<path>. --memory_service_uri takes rag://<corpus_id>, agentengine://<id> or memory://.

The async-driver rule catches almost everyone once. DatabaseSessionService uses SQLAlchemy's asyncio layer, so the URL needs an async driver: sqlite+aiosqlite, postgresql+asyncpg, mysql+aiomysql, mariadb+asyncmy. A bare postgresql:// cannot be used, and the error you get is Database related module not found for URL '<redacted>' or Invalid database URL format or argument. SQLAlchemy itself arrives with the db extra; without it the message is explicit: The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db].

One pleasant exception: if an Agent Platform environment is detected, 2.10 auto-configures session and memory against Agent Platform Sessions and Memory Bank, so the managed runtime does the right thing without flags.

Artifacts are the service people skip and then need: named, versioned binary blobs — a generated PDF, an uploaded screenshot — scoped to a session, or to a user via the user: filename prefix. You read and write them with tool_context.save_artifact and load_artifact. Since 2.6 artifacts are namespaced by app, which was a breaking change: paths from earlier versions do not resolve. Memory is the other one: long-term, cross-session recall through a MemoryService, which agents reach via the load_memory and preload_memory tools rather than by reading it directly.

A note for readers with data-residency obligations, which is most teams in the Gulf and Egypt: the location of your state is now a configuration line, not an architectural property. GOOGLE_CLOUD_LOCATION pins the Vertex region, your --session_service_uri pins the database, and --artifact_service_uri pins the bucket. If a contract says conversation transcripts stay in a particular jurisdiction, those three values are what an auditor should be reading, and in-memory fallback means the transcript is wherever the pod happened to run.

Serving: the API surface and the deploy targets

adk web is a development tool and says so: its help text reads "This server is intended for local development. Its endpoints are unauthenticated...". For anything else, adk api_server runs the same FastAPI application headlessly, defaulting to host 127.0.0.1 and port 8000.

Its REST surface is small enough to memorise:

GET    /list-apps
POST   /apps/{app}/users/{user}/sessions/{session}     # body = initial state
GET    /apps/{app}/users/{user}/sessions/{session}
DELETE /apps/{app}/users/{user}/sessions/{session}
POST   /run
POST   /run_sse

{
  "app_name": "support",
  "user_id": "u1",
  "session_id": "s1",
  "new_message": { "role": "user", "parts": [{ "text": "where is my order?" }] },
  "streaming": false
}

/run_sse is the streaming variant and the one a UI should use. The flags worth knowing on api_server: --url_prefix /adk to mount behind a path, --allow_origins for CORS (literal values, or a regex: prefix), --with_ui to serve the dev UI alongside the API, --auto_create_session, --a2a for Agent2Agent endpoints, --extra_plugins my.mod.Plugin to register plugins without touching code, --enable_features / --disable_features for named feature flags such as JSON_SCHEMA_FOR_FUNC_DECL,PROGRESSIVE_SSE_STREAMING, --eval_storage_uri gs://bucket, and --trigger_sources pubsub,eventarc which registers /apps/{app_name}/trigger/* endpoints for event-driven invocation, hardened with --trigger_oidc_audience and --trigger_oidc_service_accounts.

Read the security note in its own help text as a requirement, not a disclaimer: "This server's endpoints are unauthenticated. Run it on a trusted network only, and put it behind your own authentication and authorization layer." There is also a specific multi-tenancy consequence: the API server does not authenticate user_id. Whatever string the client sends becomes the isolation key. Your gateway must map the authenticated principal to user_id and reject attempts to pick someone else's.

Four deploy paths, in increasing order of control:

# Managed: Agent Runtime on Agent Platform (formerly Vertex AI Agent Engine)
adk deploy agent_engine --project my-proj --region us-central1 \
  --display_name "Support agent" --otel_to_cloud my_agent

# Serverless containers
adk deploy cloud_run --project my-proj --region us-central1 \
  --service_name support --session_service_uri "sqlite+aiosqlite:///./sessions.db" \
  my_agent -- --no-allow-unauthenticated --min-instances=2

# Kubernetes
adk deploy gke --project my-proj --region us-central1 \
  --cluster_name agents --service_type ClusterIP my_agent

# A local container, same flags as cloud_run minus project and region
adk deploy docker my_agent

Everything after -- on cloud_run is passed through to gcloud run deploy, which is how you set authentication and minimum instances. --service_type on GKE defaults to ClusterIP, which is usually what you want behind an ingress; if you are putting an ADK service into a cluster you already run, the Kubernetes guide covers the ingress and secret plumbing around it.

Watch the deprecated agent_engine flags, because old scripts carry them and the help text now says they will be removed: --staging_bucket ("no longer required or used"), --trace_to_cloud, --adk_app, --adk_app_object, --env_file, --requirements_file, --absolutize_imports and the import-validation flags. Configuration moved to .agent_engine_config.json and tracing to --otel_to_cloud. Using --agent_engine_id updates an existing instance; omitting it creates a new one, which is a quietly expensive way to misread a flag.

When the CLI's assumptions do not fit, build your own container around the same app factory:

from google.adk.cli.fast_api import get_fast_api_app

app = get_fast_api_app(
    agents_dir="/app/agents",
    session_service_uri="postgresql+asyncpg://adk:pass@db:5432/adk",
    allow_origins=["https://app.example.com"],
    web=False,
)

Two deployment details that cost people an afternoon: --adk_version on deploy defaults to your local version, so a developer on 2.8 deploying from their laptop pins production to 2.8; and reload is not supported on Cloud Run, so use --no-reload in containers. Since 2.10 generated Dockerfiles use exec-form RUN/CMD, reject multi-line environment values, and exclude the local .adk folder from the image — which also means adk deploy no longer ships your dev session.db, a fix from 2.6 worth knowing if you are on an older version.

Observability: traces, metrics and token spend

An agent without tracing is not debuggable, because the interesting failures are "the model called the wrong tool with plausible arguments" and those leave no stack trace.

ADK ships OpenTelemetry instrumentation using GenAI semantic conventions. Spans include invoke_agent, call_llm, execute_tool and workflow spans, carrying gen_ai.* attributes, and 2.2 added native gen_ai.client.* metrics. If you already run an OTel collector, the OpenTelemetry guide covers the pipeline side; ADK's job is to emit.

Three facts to carry:

Durations changed from milliseconds to seconds in 2.3. Any dashboard or alert threshold written against 1.x or early 2.x is off by a factor of a thousand, and the direction of the error means latency alerts silently stop firing.

Content capture is opt-in. Recording prompts and responses in traces is deliberately not the default, and the docs advise caution in production. Transcripts in a tracing backend are a data-residency and privacy surface, so treat enabling it as a decision with a reviewer, not a debugging convenience you forget to turn off.

Export is a flag. --trace_to_cloud sends traces to Cloud Trace; --otel_to_cloud sends traces and logs, and needs the otel-gcp extra for google.adk.telemetry.google_cloud. For third-party backends, ADK's integrations page lists OTel-based options including Langfuse, Arize/Phoenix and W&B Weave — the Langfuse guide is the right companion if you want prompt-level analytics rather than infrastructure traces.

In code, your levers are RunConfig.telemetry for per-request configuration, AutoTracingPlugin (2.2) to instrument without touching each agent, LoggingPlugin for console output while developing, and DebugLoggingPlugin which redacts credentials. BigQueryAgentAnalyticsPlugin, from the bigquery-analytics extra, writes events to BigQuery for analysis — note that 2.7 removed pyarrow from the gcp extra, so this plugin now needs its own extra installed explicitly.

The cost dimension deserves its own metric, and 2.8 added per-invocation and per-workflow token-spend telemetry. Pair it with the eval efficiency metrics from 2.10 — duration, tokens, model calls — and you have the two numbers that actually move: what a conversation costs today, and what a change did to that cost.

Evaluation: making "it got better" a number

Prompt changes have no compiler. The only honest way to know a change helped is to run it against cases with expected behaviour, and ADK treats that as a first-class workflow rather than a bolt-on.

Evaluation data comes as eval sets (*.evalset.json) or test files (*.test.json). Criteria include tool-trajectory match — did the agent call the right tools in the right order — response match via ROUGE, LLM-judge scoring, rubric-based grading, safety, hallucination checks, user simulation, and since 2.10 the efficiency metrics.

pip install "google-adk[eval]"

adk eval_set create agents/support/agent.py regression
adk eval_set add_eval_case agents/support/agent.py regression

adk eval agents/support/agent.py regression.evalset.json \
  --print_detailed_results --eval_storage_uri gs://my-evals

# a subset, by case id
adk eval agents/support/agent.py regression.evalset.json:eval_1,eval_2

import pytest
from google.adk.evaluation import AgentEvaluator

@pytest.mark.asyncio
async def test_support_agent_regressions():
    await AgentEvaluator.evaluate(
        agent_module="agents.support",
        eval_dataset_file_path_or_dir="tests/regression.evalset.json",
    )

One 2.10 behaviour change makes this trustworthy: AgentEvaluator.evaluate and evaluate_eval_set now raise ValueError when zero eval cases run. Previously a wrong path or a typo'd case filter passed silently, which is the worst possible failure for a quality gate — a green CI run that tested nothing. If you upgraded and your eval job suddenly fails, the likely truth is that it was never running.

Do not mix file paths and eval-set IDs in one adk eval invocation. adk test [FOLDER] [--rebuild] runs pytest over agent test JSON. And for regression testing that does not depend on a live model, adk conformance record and adk conformance test capture and replay an interaction, which is the right tool for "did this refactor change behaviour" as opposed to "is the agent good".

In CI the shape that works is: pin the exact ADK version and use the published constraints file (constraints-3.11.txt / constraints-3.12.txt, which since 2.6 add a four-day supply-chain buffer), run adk eval on a small fast set for every pull request, run the full set plus adk conformance test before a version bump, and fail on the efficiency metrics as well as the quality ones so a prompt that doubles token use cannot merge quietly. If your pipelines live in GitHub Actions, the GitHub Actions guide covers the caching and secret handling around this.

Cost and context engineering

Cost in an agent is model tokens, and tokens are mostly context: the instruction, the history, the tool schemas and the tool results re-sent on every call. A ten-turn conversation with an unmanaged history re-sends the first turn ten times.

The levers, roughly in order of how much they return for the effort:

from google.adk import App
from google.adk.apps import EventsCompactionConfig, ContextCacheConfig

app = App(
    name="support",
    root_agent=root_agent,
    plugins=[],
    events_compaction_config=EventsCompactionConfig(
        compaction_interval=10,
        overlap_size=2,
        token_threshold=8000,
        event_retention_size=50,
    ),
    context_cache_config=ContextCacheConfig(),
)

The structural point behind the table: a classifier node that only needs the latest message should not receive the whole conversation. include_contents='none' on that agent is often a larger saving than any model swap, and it also makes the node more reliable, because a classifier distracted by history is a classifier that drifts. The same reasoning favours graph routing over model-chosen transfer — the deterministic branch costs zero tokens.

The default model is now gemini-3.5-flash (LlmAgent.DEFAULT_MODEL), with gemini-live-2.5-flash-native-audio for live audio. Change it globally with LlmAgent.set_default_model(...) or per run with adk run --default_llm_model. ADK is model-agnostic: LiteLlm(model="openai/gpt-...") from google.adk.models.lite_llm reaches a hundred-plus providers including Ollama and vLLM (the LiteLLM guide covers that layer), Claude / AnthropicLlm for Anthropic, and since 2.10 OpenAI lives at google.adk.integrations.openai behind the openai extra, with google.adk.labs.openai still re-exporting it. Passing a bare unknown model name raises Model <name> not found. — the registry only resolves Gemini names directly.

One 2.10 trap on OpenAI: OpenAIResponsesLlm ignores thinking_config and warns; use OpenAIGenerateContentConfig.effort instead.

Finally, know what the platform itself costs. Agent Runtime is paid beyond a no-cost tier, Cloud Run and GKE bill normal compute, and Cloud Trace, Cloud Logging and BigQuery analytics are all line items. A tracing configuration that captures full message content on every request is an observability bill as well as a privacy decision.

Debugging beyond the dev UI

The dev UI's Events and Trace tabs are excellent and local-only. Here is the triage order for the failures you will actually hit, with the exact messages.

It will not start. No root_agent found for '<name>'. Searched in '<name>.agent.root_agent', '<name>.root_agent' and '<name>/root_agent.yaml'. has four causes in practice: the folder layout is wrong, __init__.py is missing its from . import agent, the variable is not called root_agent, or you ran adk web from inside the agent folder instead of its parent. Since 2.8 a related message reports a type mismatch when root_agent exists but is not an agent, node or App. Agent not found: '<name>'. No matching directory or module ... means the app name in the URL does not match a folder; GET /list-apps shows the valid names, and an unknown app_name has returned 404 rather than 500 since 2.5.

It loops. Max number of llm calls limit of 500 exceeded is LlmCallsLimitExceededError, and the cause is almost always tool ping-pong or a transfer loop between peer agents. Resist raising the limit first. Read the trace, find the repeating span pair, and fix the instruction or the tool design; disallow_transfer_to_parent and disallow_transfer_to_peers exist precisely for transfer loops.

Sessions misbehave. Session <id> not found. is SessionNotFoundError, and since 2.9 InMemorySessionService raises it when you append an event to an unknown session instead of silently discarding the event — so a class of bug that used to manifest as mysteriously missing history now fails loudly. Session with id <id> already exists. is AlreadyExistsError from calling create_session with an existing ID; use get_session or omit the ID. Note that session_id "user" became a reserved value and is rejected in 2.10.

Credentials. 400 INVALID_ARGUMENT, API key not valid or 403 PERMISSION_DENIED usually means the backend flag and the credential type disagree. The flag is GOOGLE_GENAI_USE_ENTERPRISE — 0 for an AI Studio API key, 1 for Vertex with Application Default Credentials plus GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION, followed by gcloud auth application-default login. If you see GOOGLE_GENAI_USE_VERTEXAI is deprecated, please use GOOGLE_GENAI_USE_ENTERPRISE instead, your .env predates 2.3; the old variable still works with a warning, and adk create now writes the new one.

Quota. google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED is rate limiting. Back off, use ReflectAndRetryModelPlugin or the 2.9 model failover, request quota, or reduce calls.

Upgrade fallout. Missing node_info or output columns after moving to 2.x means the session schema needs adk migrate session --source_db_url ... --dest_db_url ...; for a dev database, deleting .adk/session.db is faster. The warning The database is using the legacy v0 schema, which uses Pickle ... points at the same command, and --allow-unsafe-unpickling should only ever be used on a database you fully trust. Test suites running with -W error started failing in 2.10 because legacy live-audio modules under google.adk.flows.llm_flows now emit DeprecationWarning; import from google.adk.live.

Empty or malformed input. ValueError: No parts in the new_message. and A new message is required for a new invocation. mean exactly what they say — send types.Content(role="user", parts=[types.Part(text=...)]). ValueError: User message cannot contain function calls. is a security guard from 2.5: a client may not forge function-call parts.

Two more worth knowing: NodeTimeoutError when a node exceeds its timeout, and InvocationNotFoundError (2.10) when rewinding to an invocation ID that does not exist.

Putting it all together

Here is a production-shaped support agent that uses the pieces from every section: a deterministic triage graph, a model-driven specialist, a guardrail callback, a durable session store, plugins for observability, and compaction for cost.

from google.adk import App, Workflow, Event
from google.adk.agents.llm_agent import Agent
from google.adk.apps import EventsCompactionConfig, ResumabilityConfig
from google.adk.plugins import AutoTracingPlugin, LoggingPlugin
from google.adk.workflow import DEFAULT_ROUTE, RetryConfig, node


@node(name="triage", timeout=10,
      retry_config=RetryConfig(max_attempts=2, initial_delay=0.5))
def triage(node_input: str):
    """Deterministic routing: no model call, no loop risk."""
    text = node_input.lower()
    if any(w in text for w in ("refund", "charge", "invoice")):
        return Event(route="BILLING", output=node_input)
    if any(w in text for w in ("crash", "error", "broken")):
        return Event(route="BUG", output=node_input)
    return Event(route="GENERAL", output=node_input)


def block_bulk_refunds(tool, args, tool_context):
    if tool.name == "issue_refund" and args.get("amount", 0) > 500:
        return {"error": "Refunds over 500 need a human approver."}
    return None


billing = Agent(
    model="gemini-3.5-flash",
    name="billing_agent",
    description="Handles refunds, charges and invoices.",
    static_instruction="You are a billing specialist. Never reveal internal cost fields.",
    instruction="Customer locale: {user:locale?}. Order: {current_order_id?}.",
    tools=[issue_refund, lookup_invoice],
    before_tool_callback=block_bulk_refunds,
    output_key="billing_answer",
)

bug = Agent(
    model="gemini-3.5-flash",
    name="bug_agent",
    description="Collects reproduction details for engineering.",
    instruction="Collect steps to reproduce, version and platform. Be brief.",
    output_key="bug_answer",
)

general = Agent(
    model="gemini-3.5-flash",
    name="general_agent",
    description="Answers general product questions.",
    instruction="Answer from the product documentation only.",
    include_contents="none",
    output_key="general_answer",
)

support_graph = Workflow(
    name="support",
    edges=[
        ("START", triage),
        (triage, {"BILLING": billing, "BUG": bug, DEFAULT_ROUTE: general}),
    ],
    max_concurrency=2,
)

root_agent = App(
    name="support",
    root_agent=support_graph,
    plugins=[AutoTracingPlugin(), LoggingPlugin()],
    events_compaction_config=EventsCompactionConfig(
        compaction_interval=10, overlap_size=2, token_threshold=8000),
    resumability_config=ResumabilityConfig(is_resumable=True),
)

# local: durable sessions, so restarts do not lose conversations
adk web --port 8000 \
  --session_service_uri "sqlite+aiosqlite:///./sessions.db"

# CI: the gate that must stay green
adk eval agents/support/agent.py tests/regression.evalset.json --print_detailed_results
adk conformance test

# deploy: explicit state, no dev UI, authenticated
adk deploy cloud_run --project my-proj --region us-central1 \
  --service_name support \
  --session_service_uri "postgresql+asyncpg://adk:$PGPASS@/adk?host=/cloudsql/..." \
  --artifact_service_uri "gs://my-agent-artifacts" \
  --otel_to_cloud \
  agents/support -- --no-allow-unauthenticated --min-instances=2

Trace the decisions through that file. Triage is a function node, so the branch is free, testable and cannot loop. The billing agent carries the only dangerous tool and the only guardrail, which keeps the policy next to the risk. general runs with include_contents='none', because a documentation answer does not need the history. The App holds the cross-cutting concerns — tracing, logging, compaction, resumability — so none of the three agents has to know about them. The deploy line names its session and artifact services explicitly, which is the difference between a working service and the silent in-memory fallback, and --no-allow-unauthenticated means the unauthenticated API surface is not public.

What is deliberately not in there: no SequentialAgent, no _run_async_impl, no ${} templating, no --staging_bucket, no --with_ui, and no in-memory session service. Each of those is a 1.x habit or a deprecated flag that still runs today and will not run forever.

Where you are now

You can predict ADK's behaviour instead of guessing at it. You know that an invocation is a stream of persisted events, that the session is a projection of that log, and that the two rules protecting it are "yield, do not append" and "let exceptions propagate". You can build a graph with routed branches, fan-out and a join, and you know why a deterministic route beats a model-chosen transfer for anything a classifier can decide. You know the four state scopes and which one leaks across tenants. You can place a policy at the right one of four interception points. You know the async-driver rule, the silent in-memory fallback and the unauthenticated user_id, which are the three things that turn a working demo into an incident. You can read the common error messages without searching for them, run an eval gate that fails honestly, and name the cost levers in order.

What is left is ownership: architecture and failure modes under load, which component breaks first, the trust and supply-chain model, multi-tenancy with hard isolation, upgrade and migration strategy across a fleet of agents, governance, incident playbooks, and the judgement call about where ADK stops being the right tool. That is the Senior level. If you are heading there, two neighbours are worth reading alongside it: the MCP guide for the tool-sharing protocol that decides how much of your integration work is reusable, and the Langfuse guide for the evaluation and analytics layer that sits above ADK's own tracing.

Sources

• ADK documentation home — https://google.github.io/adk-docs/ now redirects here
• ADK 2.0 overview and breaking changes
• Graphs and routes
• Graph data handling and human input
• Dynamic graphs
• Agent collaboration
• Command-line reference
• Session state
• Safety and security
• Deployment overview, Agent Runtime, Cloud Run, GKE
• Observability and traces
• Release notes
• Python API reference
• adk-python CHANGELOG and releases
• google-adk on PyPI

Mid-level ADK interviews are not vocabulary tests. The interviewer assumes you can write an Agent with a tool; what they want to know is whether you have run one in production. Almost every question has the same shape: here is a symptom, what do you check first, and what trade-off are you making when you fix it.

These answers are written against ADK 2.10.0 for Python. ADK 2.x replaced the hierarchical agent executor with a graph engine, so a candidate fluent in SequentialAgent orchestration is saying they last used ADK in 2025.

The framing that scores well

Three habits separate strong answers from acceptable ones, regardless of the question.

Name the mechanism, not the symptom. "Sessions were being lost" is a report. "In-memory session service behind two replicas, so the load balancer sent the user to an instance that had never seen the conversation" is an answer.

Say what you would measure. ADK gives you OpenTelemetry spans (invoke_agent, call_llm, execute_tool), per-invocation token-spend telemetry since 2.8, and eval efficiency metrics since 2.10. Reaching for those beats reasoning in the abstract.

Volunteer the trade-off. Deterministic routing costs flexibility, compaction costs fidelity, caching costs invalidation complexity. Naming the cost unprompted is the clearest signal that you have shipped rather than read.

Scenario: the agent lost a user's conversation

"Users report that the agent forgets what they said one message ago, but only sometimes. It never happens locally. What do you check first?"

Replica count, then the session service URI, in that order.

The ADK serving process is stateless: a FastAPI app plus the Runner, with all durable state in pluggable services. If the session service is in-memory, each replica holds its own conversations, and "only sometimes" is exactly what you see when a load balancer spreads requests across instances. The reason it never happens locally is that locally there is one process, and ADK persists to <agents_dir>/<agent>/.adk/session.db by default.

The second half of the answer is the part that impresses, because it explains why this happens to teams who thought they had configured a database. On Cloud Run or Kubernetes the agents directory is typically read-only, and ADK silently falls back to in-memory services rather than failing. So a deployment that omitted --session_service_uri does not error; it quietly becomes stateless. The fix is an explicit URI — postgresql+asyncpg://..., agentengine://<id>, or Firestore — and the environment variables ADK_FORCE_LOCAL_STORAGE=1 / ADK_DISABLE_LOCAL_STORAGE=1 exist to make the choice explicit rather than inferred.

Add the driver rule, because it is the next error you hit: DatabaseSessionService uses SQLAlchemy's asyncio layer, so the URL needs an async driver (postgresql+asyncpg, sqlite+aiosqlite, mysql+aiomysql, mariadb+asyncmy). A bare postgresql:// produces Database related module not found for URL '<redacted>', and SQLAlchemy comes from the db extra.

What a weak answer misses: it blames the model's context window, or suggests putting the history in the prompt.

Scenario: the bill tripled overnight

"Token spend tripled after a release that only changed prompts. Where do you look?"

Start with calls per invocation, not tokens per call. The most common cause of a step change in agent cost is not a longer prompt; it is a loop. ADK caps this with max_llm_calls, default 500 per invocation, and when it trips you get LlmCallsLimitExceededError with the message Max number of llm calls limit of 500 exceeded. But a loop that settles at forty calls instead of four never trips the limit and costs ten times as much, silently.

The two loop shapes to name: tool ping-pong, where the model calls a tool, dislikes the result and calls it again; and transfer loops between peer agents. disallow_transfer_to_parent and disallow_transfer_to_peers exist for the second. The diagnostic is the trace: find the repeating call_llm / execute_tool span pair.

If calls per invocation are flat, look at context. Tokens are mostly history, instructions and tool schemas re-sent on every call. The levers, in order of return: EventsCompactionConfig; ContextCacheConfig for Gemini context caching, which has a 4096-token minimum; static_instruction to make the stable prefix cacheable; include_contents='none' on agents that do not need history; cheaper models for classifier sub-agents; and RunConfig.service_tier, added in 2.10. Measure with the 2.8 token-spend telemetry and the 2.10 efficiency metrics, so a cost regression fails CI rather than appearing on an invoice.

What a strong answer covers that a weak one misses: that a prompt change can triple cost without being longer, by changing how often the model decides to call a tool. The prompt is a control surface over the loop, not just over the words.

Scenario: a node ran twice and charged the customer twice

"After we enabled resumability, a payment was taken twice. Explain."

This is an ADK 2.9 behaviour change and it is the single best question for separating readers from operators. In 2.9, a workflow node that failed runs again on resume; before 2.9, it replayed as though it had completed. The new behaviour is correct — a failed node genuinely did not finish — but it means any node with side effects must be idempotent.

The fix has three parts. Make the side effect idempotent, keyed on something derived from the invocation so a second attempt is a no-op upstream. Use the node's rerun_on_resume field where rerunning genuinely cannot be made safe. And separate the risky write into its own node with a tight timeout and an explicit RetryConfig, rather than burying it inside a node that also does retrieval and formatting.

A good answer connects this to the related rule: a broad except Exception inside a node disables automatic retries, because ADK's retry logic fires on exceptions that propagate. A node that swallows its failure and returns a polite message looks successful, so nothing retries and nothing alerts. "Let exceptions propagate and configure RetryConfig" is a one-line principle that covers a lot of ground.

Trade-off: Workflow routing versus model-chosen transfer

"When do you use a graph edge and when do you let the model transfer to a sub-agent?"

The honest rule: use a graph edge when the decision does not require judgement, and a transfer when it does.

A Workflow is a graph of nodes with edges. A node returns an Event carrying a route string, and the matching edge fires; DEFAULT_ROUTE is the fallback, and since 2.10 it can fan out to several nodes. A route can also be a list, which is how you fan out, with Workflow.max_concurrency bounding parallelism and a JoinNode for fan-in. The decision is made by your code, so it is unit-testable, costs no tokens, and cannot loop.

Model-chosen transfer works differently: an LlmAgent with sub_agents=[...] exposes transfer_to_agent, and the model picks from each sub-agent's description. That is the right tool when the routing signal is genuinely linguistic and you cannot enumerate the cases — but it costs a model call per decision, depends on prompt wording, and is a known source of loops.

So: route a ticket by keyword or classifier output with an edge. Route an ambiguous, open-ended request that could belong to three specialists with a transfer. And mention the current constraint, because it shows you have hit it: the deprecation warning on SequentialAgent states it directly — Workflow cannot yet be used as an LlmAgent sub-agent. If you need a model to hand off into a deterministic pipeline, Workflow-as-Tool (2.4) is the route, not sub_agents.

Weak answer tell: treating sub_agents as the general multi-agent mechanism and graphs as an optimisation. In 2.x it is the other way round — the graph is the execution engine, and BaseAgent subclasses BaseNode.

Trade-off: managed runtime, Cloud Run, or GKE

"We need this agent in front of customers next month. Where do you deploy it?"

Frame it as what you want to stop owning.

Agent Runtime on Agent Platform — the product the docs now use, formerly Vertex AI Agent Engine, still deployed with adk deploy agent_engine and still addressed as agentengine:// — is fully managed and auto-scaling, and brings managed sessions, Memory Bank, a sandbox and telemetry with it. Since 2.10, when an Agent Platform environment is detected, sessions and memory auto-configure. You stop owning state infrastructure. You accept the platform's pricing beyond its no-cost tier and its region set, which matters if a contract pins data residency.

Cloud Run via adk deploy cloud_run is serverless containers: you own the session database and the artifact bucket, you pass flags through to gcloud run deploy after a -- (so -- --no-allow-unauthenticated --min-instances=2), and you get normal compute billing. It is the right default for a team that already runs Cloud Run services.

GKE via adk deploy gke is for teams needing cluster-level control, open models, or an existing ingress; --service_type defaults to ClusterIP. adk deploy docker builds the same container locally.

A strong answer adds two operational notes. adk deploy --adk_version defaults to your local version, so deploying from a laptop pins production to whatever that developer has installed — pin it in CI. And --with_ui ships the development UI, whose own help text says its endpoints are unauthenticated; it does not belong in production.

Scenario: the prompt stopped reading state after an upgrade

"We went from 2.9 to 2.10 and the agent stopped knowing the customer's name, which we injected from state. Nothing else changed."

This is the 2.10 instruction-templating change. {var} is still substituted from state. ${var} and backslash-escaped \{var} are now left as written rather than filled. Teams that wrote ${customer_name} because it reads like shell or JavaScript interpolation had working prompts that silently became literal text.

The answer then widens correctly: this is why you read the "Behavior changes" section of the release notes before bumping, and why minor versions deserve the same care as majors here. Python ships a minor roughly every two weeks, and 2.6, 2.7, 2.9 and 2.10 each carried real behaviour changes — app-namespaced artifacts, pyarrow leaving the gcp extra, node rerun on resume, templating. The guard rail is an eval set plus adk conformance test in CI against a pinned version.

Know the state scopes too, since that is the likely follow-up: no prefix is session scope, user: is all sessions of that user within the app, app: is all users of the app, and temp: lives only for the current invocation and is never persisted. Tenant data in app: state is a leak, not a bug.

Scenario: CI is green but the agent is worse

"Your eval job passes on every pull request, and users say quality dropped. What is your first hypothesis?"

That the eval job is testing nothing.

This is a real ADK 2.10 change and a good one: AgentEvaluator.evaluate and evaluate_eval_set now raise ValueError when zero eval cases run. Before 2.10 a wrong dataset path or a typo'd case filter passed silently, which is the worst possible failure mode for a quality gate. So the first check is whether the job is actually loading cases — and if you are on 2.10 and your eval job suddenly started failing after an upgrade, the truthful reading is that it was never running.

Second hypothesis: the criteria do not measure what regressed. ADK's eval criteria include tool-trajectory match, response match via ROUGE, LLM-judge scoring, rubric-based grading, safety, hallucination and user simulation. A suite that only checks response text will not notice that the agent stopped calling the lookup tool and started guessing. Third: the gate has no efficiency dimension, and the 2.10 metrics — duration, tokens, model calls — let you fail a prompt change that doubled cost even though quality held.

Mention adk conformance record / adk conformance test too. Conformance is replay-based: it answers "did this refactor change behaviour" without a live model, a different and cheaper question than "is the agent good".

Scenario: a tool did something it should not have

"An agent issued a refund it had no business issuing. Where should the control have been?"

Name all four interception points and then choose, because the choice is the answer.

In-tool validation against ToolContext and state is where business rules belong — "this user may only refund their own orders" is ordinary Python and belongs in ordinary Python. A before_tool_callback sees the arguments before the call and blocks by returning a dict, which becomes the tool result the model sees; returning None allows the call. A before_model_callback screens the request itself, which is where prompt-injection checks and PII redaction go, and returning content or an LlmResponse short-circuits the model call entirely. Plugins — LoggingPlugin, AutoTracingPlugin, the Model Armor guardrail plugin from 2.8 — are callback bundles registered once on the App or Runner, which is where cross-cutting policy belongs because per-agent callbacks drift.

For the refund case specifically: the amount limit belongs in the tool or a before_tool_callback, and anything genuinely irreversible should also require human-in-the-loop confirmation — ADK supports tool confirmation (require_confirmation / request_confirmation), the request_input tool added in 2.2, and long-running tools, all built on the resumability machinery.

Finish with the trust model. Tool output is untrusted input. ADK 2.8 to 2.10 added fencing for relayed agent output and server-supplied MCP tool descriptions, and 2.10 stopped executing code found in model reasoning — but your UI still has to escape model output, and code execution still needs a real sandbox (Agent Engine, Cloud Run sandbox, ContainerCodeExecutor, E2B, Daytona), hermetic and with no network.

Scenario: one user read another user's data

"A tenant saw another tenant's order details. Walk me through the possibilities."

Two mechanisms, and you should name both.

First, state scope. The isolation keys are app_name, then user_id, then session_id. app:-prefixed state deliberately sits above all of them and is shared by every user of the app, so tenant data written there is visible to the next caller. Artifacts are namespaced by app since 2.6 and scoped to the caller since the 2.4 and 2.5 security fixes, and delete_session has enforced ownership since 2.2 — so if you are on an older version, upgrading is part of the fix.

Second, and more likely: the API server does not authenticate user_id. Whatever string the client sends becomes the isolation key. The api_server help text is explicit that its endpoints are unauthenticated and must sit behind your own authentication and authorization layer. So the real question is whether your gateway maps the authenticated principal to user_id, or whether it forwards a client-supplied value. If a user can edit a request body and change user_id, you do not have multi-tenancy, you have a naming convention.

The hard-isolation answer, when the requirement demands it: separate apps, separate projects, or separate databases, plus user-auth OAuth so tools act with the user's own token rather than a shared service account. ADK's identity model distinguishes agent-auth, where tools act as the service account and everyone shares access, from user-auth, where the agent can only do what the user could.

What strong answers have in common

Looking across all of these, the pattern is consistent enough to prepare for.

A strong answer names the ADK noun. Event, node, route, service, callback, plugin, invocation. Someone who says "the orchestration layer" is paraphrasing; someone who says "the event's state_delta" has read the code.

A strong answer knows what changed recently and why it matters. The graph engine replacing the hierarchical executor, _run_async_impl being silently ignored, GOOGLE_GENAI_USE_VERTEXAI becoming GOOGLE_GENAI_USE_ENTERPRISE, node rerun on resume, templating in 2.10, eval raising on zero cases. These are not trivia — each one is a production incident someone has already had.

A strong answer distinguishes defaults from decisions. max_llm_calls defaults to 500, the default model is gemini-3.5-flash, content capture in traces is off, CLI telemetry is off, durations became seconds in 2.3. Knowing which values you inherited and which you chose is most of what operating a framework means.

A weak answer treats the model as the explanation for every behaviour, proposes prompt changes for infrastructure problems, or describes features without their costs.

Sixty-second self-test

Answer out loud. Hesitating on any of these is a reading list, not a failure.

Where does session state live, and what happens with two replicas and no --session_service_uri? Which driver prefixes does DatabaseSessionService require? What are the four state prefixes and which one crosses users? What is the default max_llm_calls, which error fires, and which two loop shapes cause it? What changed about failed-node resumption in 2.9? Why does except Exception inside a node break retries? What does a node's output become downstream, and what is route for? Which templating forms still substitute from state in 2.10? What does AgentEvaluator.evaluate now do when zero cases run? Which variable replaced GOOGLE_GENAI_USE_VERTEXAI? What does everything after -- on adk deploy cloud_run do? Who authenticates user_id? What are the four places a guardrail can live, and which returns a dict to block? Which unit did span durations change to in 2.3? Name three cost levers and what each costs you.

Answer those without hedging and you can hold a mid-level ADK conversation. What remains — fleet upgrades, hard tenant isolation, incident playbooks, and when to choose something else — is the Senior level.

These are the mistakes that cost mid-level ADK users a day each, the practices that return the most for the least effort, and a set of recipes you can paste into a real project. Everything is verified against ADK 2.10.0 for Python.

The ordering is deliberate. The errors come first because you are probably here because of one. The practices are ranked by leverage, not by how interesting they are. The recipes assume you already know what you want and just need the exact command.

The errors that actually happen

In-memory services behind more than one replica

Symptom: users lose their conversation intermittently. Never reproducible locally.

Cause: the serving process is stateless and all durable state lives in services. In-memory session state is per-process, so each replica holds its own conversations.

Why it happens even when you configured a database: locally ADK persists to <agents_dir>/<agent>/.adk/session.db. On Cloud Run or Kubernetes that directory is usually read-only, and ADK silently falls back to in-memory rather than failing. A deploy that forgot --session_service_uri does not error.

Fix: always pass explicit URIs in any non-local deployment. Use ADK_FORCE_LOCAL_STORAGE=1 or ADK_DISABLE_LOCAL_STORAGE=1 to make the choice explicit, and remember --use_local_storage cannot be combined with explicit URIs.

A synchronous database URL

Symptom: Database related module not found for URL '<redacted>' or Invalid database URL format or argument.

Cause: DatabaseSessionService uses SQLAlchemy's asyncio layer. A bare postgresql:// has no async driver.

Fix: postgresql+asyncpg://, sqlite+aiosqlite://, mysql+aiomysql:// or mariadb+asyncmy://, and install the driver. If SQLAlchemy itself is missing you get the explicit message The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db].

A broad except that silently disables retries

Symptom: a flaky tool never recovers, and nothing appears in your error metrics.

Cause: ADK's automatic retry logic fires on exceptions propagating out of a node. A node that catches everything and returns a polite message looks successful.

Fix: let exceptions propagate. Configure RetryConfig(max_attempts=..., initial_delay=..., backoff_factor=..., jitter=True) on the node. RetryConfig(max_attempts=0) means no retries, explicitly.

A non-idempotent node after enabling resumability

Symptom: duplicate charges, duplicate emails, duplicate rows — only in conversations that were interrupted.

Cause: ADK 2.9 changed node resumption so a failed node runs again on resume. Before 2.9 it replayed as though completed.

Fix: make side-effecting nodes idempotent, keyed on an invocation-derived identifier. Where that is impossible, set rerun_on_resume=False on the node. Keep the risky write in its own small node with a tight timeout.

Overriding _run_async_impl in 2.x

Symptom: your custom orchestration code does not run. No error, no warning.

Cause: in 2.x the graph engine does not call _run_async_impl. Custom overrides are silently ignored.

Fix: express the logic as nodes, or as a workflow-aware custom BaseNode. Grep for the method name when you inherit any 1.x codebase.

context.session.events.append()

Symptom: history that reloads differently from how it ran. Missing events after a restart.

Cause: the session is a projection of the persisted event log. Appending directly bypasses the runner's persistence and ordering, which breaks determinism.

Fix: yield events.

${var} in instructions after upgrading to 2.10

Symptom: the agent stopped knowing values you injected from state. The prompt now contains literal ${customer_name}.

Cause: since 2.10, ${var} and backslash-escaped \{var} are left as written. Only {var} is substituted.

Fix: use {var}, or {var?} for optional. {artifact.name} injects an artifact's text.

Tenant data in app: state

Symptom: one user sees another's details.

Cause: app:-prefixed state is shared by all users of the app, deliberately above the user_id isolation key.

Fix: session scope (no prefix) for conversation facts, user: for per-user values, temp: for intravocation scratch that is never persisted, and app: only for genuinely global configuration.

A client-supplied user_id

Symptom: a user changes a request body and reads someone else's session.

Cause: the API server does not authenticate user_id. Its help text says the endpoints are unauthenticated and must sit behind your own auth layer.

Fix: your gateway maps the authenticated principal to user_id and rejects any client-supplied value. For hard isolation use separate apps, projects or databases, plus user-auth OAuth so tools act with the user's own token.

Hitting the call limit instead of fixing the loop

Symptom: Max number of llm calls limit of 500 exceeded (LlmCallsLimitExceededError).

Cause: tool ping-pong, or transfer loops between peer agents.

Fix: read the trace for the repeating call_llm / execute_tool pair. Fix the instruction or the tool design. Use disallow_transfer_to_parent and disallow_transfer_to_peers for transfer loops. Raising max_llm_calls is triage, not a fix; a value of zero or less means unlimited, with a warning, and should never ship.

A green eval job that tests nothing

Symptom: CI passes while quality drops.

Cause: before 2.10, a wrong dataset path or typo'd case filter made AgentEvaluator.evaluate pass silently.

Fix: upgrade — 2.10 raises ValueError when zero eval cases run. If your eval job started failing after the upgrade, it was never running.

Credential and backend mismatch

Symptom: 400 INVALID_ARGUMENT, API key not valid, or 403 PERMISSION_DENIED.

Cause: GOOGLE_GENAI_USE_ENTERPRISE disagrees with the credential type.

Fix: 0 with GOOGLE_API_KEY for AI Studio; 1 with GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION plus gcloud auth application-default login for Vertex. If you see GOOGLE_GENAI_USE_VERTEXAI is deprecated, please use GOOGLE_GENAI_USE_ENTERPRISE instead, rename the variable — the old one still works but warns.

Schema errors after a major upgrade

Symptom: missing node_info or output columns, or the warning The database is using the legacy v0 schema, which uses Pickle ....

Cause: the 2.0 Event schema added fields.

Fix: adk migrate session --source_db_url ... --dest_db_url .... Delete .adk/session.db for a dev database. Use --allow-unsafe-unpickling only on a database you fully trust.

The practices that pay off most, in order

1. Pass explicit service URIs everywhere except your laptop. This single habit removes the most expensive class of ADK bug. Treat an unset --session_service_uri in a deployment manifest as a failing review comment.

2. Pin the exact version and use the published constraints file. ADK ships a Python minor release roughly every two weeks, and 2.6, 2.7, 2.9 and 2.10 each contained real behaviour changes. constraints-3.11.txt and constraints-3.12.txt have been published since 2.6 with a four-day supply-chain buffer. Also pin --adk_version in CI, because adk deploy otherwise defaults to whatever the deploying machine has installed.

3. Prefer a graph edge over a model-chosen transfer for any decision a classifier could make. It costs no tokens, is unit-testable, and cannot loop. Keep sub_agents and transfer_to_agent for genuinely ambiguous linguistic routing.

4. Run an eval set in CI, with tool-trajectory assertions and efficiency metrics. Response-text checks miss the most common regression, which is an agent that stops calling a tool and starts guessing. The 2.10 efficiency metrics let a cost regression fail the build.

5. Turn on tracing before you need it. AutoTracingPlugin on the App, --otel_to_cloud or --trace_to_cloud in deployment. Note that span durations changed from milliseconds to seconds in 2.3, so audit any threshold written against an older version. Leave content capture off unless someone has signed off on transcripts leaving your boundary.

6. Put cross-cutting policy in plugins, not per-agent callbacks. Per-agent callbacks drift as the agent count grows. --extra_plugins my.mod.Plugin lets you register one without touching code.

7. Make the unchanging part of your prompt static_instruction. It is the cheapest cost lever, because it makes the prefix cache-stable. Pair with ContextCacheConfig (Gemini needs 4096 tokens minimum).

8. Set include_contents='none' on agents that do not need history. Usually a bigger saving than a model swap, and it makes classifiers more reliable by removing the distraction.

9. Run your test suite with -W error. Deprecations in ADK are advance notice of removals. 2.10 added DeprecationWarning to the legacy live-audio modules under google.adk.flows.llm_flows; import from google.adk.live.

10. Treat tool output as untrusted input. 2.8 to 2.10 added fencing for relayed agent output and MCP tool descriptions, and 2.10 stopped executing code found in model reasoning — but escaping model output in your UI is still your job, and code execution still needs a real sandbox.

Practice cards

Work through these on a real agent. Each one is fifteen minutes and leaves you with knowledge you cannot get from reading.

Operational recipes

Durable local development

pip install "google-adk[db,eval,gcp]"

adk web agents/ --port 8000 \
  --session_service_uri "sqlite+aiosqlite:///./sessions.db" \
  --artifact_service_uri "file://./artifacts" \
  --log_level debug

Run adk web from the parent of the agent folder. Running it from inside produces No root_agent found for '<name>'. Searched in '<name>.agent.root_agent', '<name>.root_agent' and '<name>/root_agent.yaml'. — which has the same four causes every time: wrong directory, missing from . import agent in __init__.py, the variable not named root_agent, or the wrong working directory.

A headless server with real state and CORS

adk api_server agents/ \
  --host 0.0.0.0 --port 8000 \
  --session_service_uri "postgresql+asyncpg://adk:$PGPASSWORD@db:5432/adk" \
  --artifact_service_uri "gs://my-agent-artifacts" \
  --memory_service_uri "agentengine://1234567890" \
  --allow_origins "https://app.example.com" \
  --extra_plugins my_pkg.plugins.PolicyPlugin \
  --otel_to_cloud \
  --no-reload

Use --url_prefix /adk when mounting behind a path, and regex: in --allow_origins when you need a pattern. The endpoints remain unauthenticated: put them behind IAM, IAP or a gateway.

Driving it from a client

curl -sN -X POST http://localhost:8000/run_sse \
  -H 'Content-Type: application/json' \
  -d '{"app_name":"support","user_id":"u1","session_id":"s1",
       "new_message":{"role":"user","parts":[{"text":"where is my order?"}]},
       "streaming":true}'

GET /list-apps lists valid app names. Sessions are created with POST /apps/{app}/users/{user}/sessions/{session}, whose body is the initial state, and the same path serves GET and DELETE. session_id "user" is reserved and rejected since 2.10.

Cost guardrails in code

from google.adk import App
from google.adk.agents.run_config import RunConfig
from google.adk.apps import ContextCacheConfig, EventsCompactionConfig

run_config = RunConfig(max_llm_calls=60)

app = App(
    name="support",
    root_agent=root_agent,
    events_compaction_config=EventsCompactionConfig(
        compaction_interval=10, overlap_size=2, token_threshold=8000,
        event_retention_size=50),
    context_cache_config=ContextCacheConfig(),
)

ADK_MAX_LLM_CALLS sets the same limit from the environment. Measure the effect with the per-invocation and per-workflow token-spend telemetry added in 2.8.

The CI gate

jobs:
  agent-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install "google-adk[eval]==2.10.0" -c constraints-3.12.txt
      - run: python -W error -m pytest tests/ -q
      - run: adk eval agents/support/agent.py tests/smoke.evalset.json --print_detailed_results
      - run: adk conformance test

Run a small fast eval set per pull request and the full set plus conformance before a version bump. Select individual cases with file.evalset.json:eval_1,eval_2, and do not mix file paths and eval-set IDs in one invocation.

Deploying without shipping the dev UI

adk deploy cloud_run --project my-proj --region us-central1 \
  --service_name support --app_name support --port 8000 \
  --session_service_uri "postgresql+asyncpg://adk:pass@/adk?host=/cloudsql/my-proj:us-central1:adk" \
  --artifact_service_uri "gs://my-agent-artifacts" \
  --otel_to_cloud --adk_version 2.10.0 \
  agents/support -- --no-allow-unauthenticated --min-instances=2

Everything after -- goes to gcloud run deploy. Omit --with_ui. On Agent Engine, avoid the deprecated flags whose help text says they will be removed: --staging_bucket, --trace_to_cloud, --adk_app, --adk_app_object, --env_file, --requirements_file and --absolutize_imports. Configuration lives in .agent_engine_config.json, extra code goes in with --extra_packages, and --agent_engine_id is the difference between updating an instance and creating a new one.

Secrets

.env is for local development only; adk create warns that secrets live there and writes a .gitignore. In production use Secret Manager — Cloud Run --set-secrets / --update-secrets, with roles/secretmanager.secretAccessor on the runtime service account — and prefer Application Default Credentials or workload identity over API keys. Since 2.8, adk deploy --env logs variable names rather than values.

Migrating a 1.x agent

pip install -U google-adk                 # 2.x
# or stay on the old line deliberately:
pip install "google-adk~=1.0"

adk migrate session --source_db_url sqlite:///old.db --dest_db_url sqlite:///new.db

The checklist: replace SequentialAgent, ParallelAgent and LoopAgent with Workflow edges; delete _run_async_impl overrides; replace context.session.events.append() with yield; rename GOOGLE_GENAI_USE_VERTEXAI to GOOGLE_GENAI_USE_ENTERPRISE; migrate session databases; change ${var} to {var}; move OpenAI imports to google.adk.integrations.openai with the openai extra; and re-check artifact paths, which became app-namespaced in 2.6. The deprecation warning on the old orchestrators also documents the remaining gap: Workflow cannot yet be used as an LlmAgent sub-agent.

The checklist

1. State is explicit Every non-local deployment names its session and artifact service URIs, with async DB drivers, and no deployment relies on the local-storage fallback.
2. Versions are pinned An exact ADK version plus the constraints file, and --adk_version set in CI rather than inherited from a laptop.
3. Loops are bounded max_llm_calls set deliberately below the 500 default, transfer restrictions where peers could ping-pong, and node timeouts on anything that calls out.
4. Failures are honest No broad except inside nodes, RetryConfig where retries make sense, and side-effecting nodes idempotent because 2.9 reruns failed nodes on resume.
5. Policy sits at the right layer Business rules in tools, argument checks in before_tool_callback, input screening in before_model_callback, cross-cutting rules in plugins, confirmation on anything irreversible.
6. Tenancy is enforced outside ADK The gateway sets user_id, no tenant data in app: state, and separate apps or databases where isolation must be hard.
7. Quality is measured An eval set in CI with tool-trajectory and efficiency criteria, conformance replay before version bumps, and tests run with -W error.
8. Cost is visible Token-spend telemetry on, static_instruction for the stable prefix, compaction configured, and include_contents='none' wherever history is not needed.
9. The dev surface is not public No --with_ui in production, --allow_origins restricted, and the API behind IAM, IAP or a gateway.
10. Upgrades are read, not assumed The release notes' behaviour-change section reviewed before every bump, because minor versions here carry real changes.

The theme running through all ten: ADK's defaults are tuned for a developer on a laptop, and almost every production problem is a default that nobody converted into a decision. Go through the list once per service and the framework becomes predictable.

Three more errors worth recognising

A bare non-Gemini model name

Symptom: Model <name> not found.

Cause: the registry resolves Gemini names directly; everything else needs a provider class.

Fix: LiteLlm(model="provider/model") from google.adk.models.lite_llm, Claude / AnthropicLlm, or the OpenAI classes in google.adk.integrations.openai — and install the matching extra. In 2.10 the error message itself also tells you how to reach a third-party model. One 2.10 note on OpenAI: OpenAIResponsesLlm ignores thinking_config and warns, so use OpenAIGenerateContentConfig.effort instead.

MCP stdio servers on Windows

Symptom: NotImplementedError from an asyncio subprocess when McpToolset launches a stdio server.

Cause: the Windows event-loop policy does not support subprocess transports under the reloader.

Fix: run adk web --no-reload, or work in WSL. ADK 2.6 and 2.8 fixed the related Windows path and glob-expansion problems, so upgrading is worth trying first.

BigQuery analytics stopped installing

Symptom: BigQueryAgentAnalyticsPlugin fails on a missing pyarrow after an upgrade.

Cause: 2.7 removed pyarrow from the gcp extra — a deliberate breaking change.

Fix: pip install "google-adk[bigquery-analytics]". The same message pattern applies to every optional feature: ADK names the extra it needs in the error, so read the message rather than guessing the package.

A pattern across all three: ADK's errors are unusually specific about the fix, naming the extra, the command or the replacement class. When something fails, read the full message before searching, because about half the time the message is the answer.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, and adds the work that only appears when ADK is your responsibility rather than your tool: the runtime other teams deploy onto, the storage their sessions land in, the token bill their agents generate, and the pager that rings when a graph starts looping at three in the morning.

Up to this point the questions have been about making an agent work. From here they are different. What can a tool call reach when the model has been talked into calling it? Which component gives way first when ten teams share one deployment? What breaks when you bump a minor version, given that ADK shipped behaviour changes in 2.6, 2.7, 2.9 and 2.10? And who may say "this agent is ready for production", on what evidence?

Everything here is verified against Google ADK 2.10.0 for Python, the reference implementation. Where the other SDKs diverge in ways that matter to a platform owner, I say so.

Where this picks up

I start with architecture, because every judgement that follows — security, scale, cost, upgrade risk — is a consequence of where the state lives.

The architecture you now own

An ADK deployment is smaller than it looks. There is a serving process, and there are services. The serving process is a FastAPI application (adk api_server, or your own main.py calling get_fast_api_app) wrapping a Runner. It holds no durable truth. Everything a conversation needs between turns lives in a service you chose and configured: a SessionService for events and state, an ArtifactService for binary blobs, a MemoryService for cross-session recall, and a CredentialService for tokens the agent obtained on a user's behalf.

The Runner is the orchestrator. You construct it with exactly one of agent=, node= or app=, and session_service is a required keyword argument. App is the top-level container when you want plugins, compaction, context caching or resumability:

from google.adk import Runner
from google.adk.apps import App
from google.adk.sessions import DatabaseSessionService
from google.adk.artifacts import GcsArtifactService
from google.adk.plugins.logging_plugin import LoggingPlugin

app = App(
    name="support_triage",
    root_agent=root_agent,
    plugins=[LoggingPlugin()],
)

runner = Runner(
    app=app,
    session_service=DatabaseSessionService(
        db_url="postgresql+asyncpg://adk:secret@10.0.0.5/adk_sessions",
    ),
    artifact_service=GcsArtifactService(bucket_name="adk-artifacts-prod"),
)

Three details in that snippet are the whole architecture. The database URL uses postgresql+asyncpg, because DatabaseSessionService runs on SQLAlchemy's asyncio layer and a bare synchronous driver cannot be used — a plain postgresql:// URL fails with Invalid database URL format or argument. SQLAlchemy arrives with the db extra; without it you get The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db]. And the artifact service is GCS, because the file service writes to a directory only one replica can see.

The exception is Agent Runtime on Agent Platform — the product the docs used to call Vertex AI Agent Engine. When ADK detects an Agent Platform environment it auto-connects sessions and memory to Agent Platform Sessions and Memory Bank, so the same code has a different durability story depending on where it runs. Teach both names: the CLI command is still adk deploy agent_engine and service URIs are still agentengine://.

The graph engine, and what 2.0 changed underneath you

ADK 2.0 went GA on 2026-05-19 and replaced the hierarchical agent executor with a graph-based execution engine. Agents, tools and plain functions are now nodes in a workflow graph; BaseAgent subclasses BaseNode. If you inherited a 1.x codebase, this is the single largest source of latent bugs in your estate, because several of the changes fail quietly.

The new top-level exports are Agent, Context, Event, Runner and Workflow, and google.adk.workflow adds BaseNode, Node, FunctionNode, JoinNode, Edge, START, DEFAULT_ROUTE, RetryConfig, NodeTimeoutError and the node decorator. A graph is a name plus a list of edges:

from google.adk import Workflow, Event
from google.adk.workflow import DEFAULT_ROUTE, RetryConfig

def classify(node_input: str):
    return Event(route="BUG" if "crash" in node_input else "OTHER", output=node_input)

def bug(node_input: str):
    return Event(message=f"bug handler got: {node_input}")

def other(node_input: str):
    return Event(message="other handler")

root_agent = Workflow(
    name="triage",
    edges=[
        ("START", classify),
        (classify, {"BUG": bug, DEFAULT_ROUTE: other}),
    ],
    max_concurrency=4,
)

START is the entry point, the first node receives the user's text as node_input, and a node's output becomes the next node's node_input. ("START", a, b, c) is a plain chain. A node paired with a dict is conditional routing on the route the node set on its Event; a route may be a list, which fans out, and since 2.10 DEFAULT_ROUTE can fan out too. JoinNode is the fan-in. Nodes share the fields where most of your platform policy ends up living: rerun_on_resume, wait_for_output, retry_config, timeout, input_schema, output_schema and state_schema.

The four 2.x behaviour changes a platform owner must be able to recite:

1. Custom _run_async_impl() overrides are silently ignored by the new runtime. A 1.x custom agent that did real work in that method now does nothing, with no error. Grep your estate for it.
2. context.session.events.append() breaks determinism. Events must be yielded. Appended events are not part of the replay the graph engine depends on, so resumption and conformance replay diverge from the live run.
3. A broad except Exception inside a node disables automatic retries. Swallowing the exception tells the engine the node succeeded. Let exceptions propagate and let RetryConfig decide.
4. The Event schema gained node_info and output. Database-backed session stores need a schema change, applied with adk migrate session. Skipping it produces schema errors about the missing columns.

SequentialAgent, ParallelAgent and LoopAgent still import and run, but their deprecation warning names the constraint you will hit: SequentialAgent is deprecated in favor of Workflow and will be removed in a future version. Workflow cannot yet be used as an LlmAgent sub-agent. That second sentence is a real limitation — an LlmAgent router with sub-agents cannot simply take a Workflow as one of them. Restructure so the graph is the top level and agents are nodes inside it. ADK 2.5 added validation that old orchestrators are not used inside graphs, so the styles will not quietly mix.

Which component breaks first

Scaling an agent platform is not like scaling a web service, because the expensive, rate-limited, non-deterministic component is not yours. Here is the order in which things actually give way, from most to least likely, based on where ADK's own limits sit.

Model quota goes first. google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED is the most common production failure in any ADK estate, and it is not a bug in your code. A single agent invocation can make many model calls, and RunConfig.max_llm_calls defaults to 500 per invocation — a ceiling, not a budget. Three mitigations exist in ADK itself: back off and retry with ReflectAndRetryModelPlugin, use the automatic model failover to backup models added in 2.9, and reduce call volume with better tool design. Everything else is a quota request.

Then the loop. Max number of llm calls limit of 500 exceeded, raised as LlmCallsLimitExceededError, means an agent is ping-ponging: a tool that keeps returning something the model wants to re-ask about, or two agents transferring to each other. Raising the limit with RunConfig(max_llm_calls=N) or ADK_MAX_LLM_CALLS converts a crash into a very expensive crash. A value of zero or less means unlimited, with a warning, and should be prohibited by policy in your organisation. The right fix is always upstream: tighten instructions, give the tool a terminal result, or set disallow_transfer_to_parent and disallow_transfer_to_peers on agents that should not hand control back.

Then the session store. DB session writes take per-session in-process locks, so a single hot session is serialised inside one replica, and concurrent writers across replicas depend entirely on the database. Long conversations are the pathology here: every turn appends events, and the event list is what gets loaded and re-serialised. Event compaction is the remedy, and I treat it as a scaling control rather than a cost control.

Then concurrency inside the graph. Workflow.max_concurrency bounds how many nodes run in parallel, and RunConfig.tool_thread_pool_config bounds the tool thread pool. Fan-out through a route list is the easiest way to accidentally multiply your model spend and your downstream load by the width of the fan. Set max_concurrency on every graph that fans out, and treat an unset value in review as a finding.

Then timeouts. A node that exceeds its timeout raises NodeTimeoutError. Without timeouts, a hanging MCP server or a slow HTTP tool holds an invocation open indefinitely, consuming a request slot and, on a serverless runtime, billing you for the wait.

State, sessions and the storage decisions behind them

A session is one conversation thread: app_name, user_id, id, state, events and last_update_time. State is a key/value scratchpad whose prefix decides its scope, and this is the single most consequential four-line table in ADK.

The app: prefix is the one that causes data-leak incidents, because it looks like "application state" and behaves like a global variable visible to every user. Make it a blocked pattern in review unless the value is genuinely a configuration constant.

Choosing a session store comes down to three questions. Does it survive a restart, can two replicas see it, and where does the data physically sit? The in-memory service fails the first two; since 2.9 it also raises SessionNotFoundError when you append an event to an unknown session instead of discarding it silently. SQLite via sqlite+aiosqlite survives a restart on one machine and fails the replica test. A SQLAlchemy async database — postgresql+asyncpg, mysql+aiomysql, mariadb+asyncmy — passes all three and is the default answer for self-hosted ADK, as does Agent Platform Sessions via agentengine://. Firestore and Redis are available, the Firestore root collection set by ADK_FIRESTORE_ROOT_COLLECTION.

The residency question gets skipped and then becomes urgent. If you serve a bank in Riyadh or a government entity in Abu Dhabi, the conversation log is customer data: user messages, tool arguments, retrieved documents. So the session database must sit in a region your commitments allow — for self-hosted ADK, a Postgres instance you placed deliberately. And GOOGLE_CLOUD_LOCATION decides where the model call goes: a Gulf deployment with us-central1 in its .env has moved the prompt across a border. Set it per environment, not once in a template that gets copied.

adk deploy cloud_run \
  --project acme-prod --region me-central2 \
  --service_name support-triage \
  --session_service_uri "postgresql+asyncpg://adk@/adk_sessions?host=/cloudsql/acme-prod:me-central2:adk-pg" \
  --artifact_service_uri "gs://acme-adk-artifacts-me" \
  --memory_service_uri "agentengine://1234567890" \
  --otel_to_cloud \
  support_triage \
  -- --no-allow-unauthenticated --min-instances=2

Everything after the bare -- is passed through to gcloud run deploy, which is how you get --no-allow-unauthenticated and --min-instances. Note what is absent: --with_ui. The dev UI does not belong in production.

Artifacts are named, versioned binary blobs scoped to a session or, with the user: filename prefix, to a user, held by an in-memory, file or GCS ArtifactService. ADK 2.6 made file artifacts namespaced by app, a breaking change: paths written by earlier versions are not where 2.6 looks. If artifacts "disappeared" across that upgrade, that is why.

The trust model

ADK's security posture starts from a sentence in its own CLI help, and I quote it to every team I onboard: "This server's endpoints are unauthenticated. Run it on a trusted network only, and put it behind your own authentication and authorization layer." The same is true of adk web, whose help adds that it is intended for local development. There is no built-in login, no tenant check and no rate limit. Everything in front of the Runner is yours to build.

That gives you four obligations at the edge. Authenticate the caller, with Cloud Run --no-allow-unauthenticated and identity tokens, IAP, or a gateway. Restrict origins with --allow_origins, which takes literal origins or a regex:-prefixed pattern. Rate-limit per principal. And map the authenticated principal to user_id yourself — the API server does not authenticate user_id, so a client that can reach /run can claim to be anyone. Trigger endpoints are the exception: they can require Google OIDC through --trigger_oidc_audience and --trigger_oidc_service_accounts.

Inside the agent, the question is whose authority a tool call carries. Agent-auth means tools act as the service account: right when every user has the same entitlements, catastrophic when they do not, because an agent-auth database tool will read another tenant's rows if the model is persuaded to ask. User-auth means OAuth with the user's own token, carried by auth_scheme and auth_credential on tools and toolsets with tokens held by a CredentialService. It costs more to build and it is the correct default for anything touching per-user data. ADK 2.10 also keeps the OAuth2 client secret out of session state.

Then there is prompt injection, which in an agent framework is the primary threat rather than a theoretical one. Any text the model sees can carry instructions: a page fetched by url_context, a document from Vertex AI Search, a tool result, a description from a remote MCP server, another agent's output. ADK has hardened steadily — path-traversal and zip-slip fixes in 2.2, fenced relayed agent output and MCP tool descriptions in 2.8 and 2.10, no code execution from model reasoning in 2.10, SQL-injection guards in the BigQuery and Spanner tools. None of it solves injection. Your defences are layered and yours:

Callbacks are the enforcement point, and the contract matters: a before callback that returns a value — content, an LlmResponse or a dict — short-circuits the step, and None lets it proceed. So before_tool_callback returning a dict is a block, before_model_callback is where input screening lives, and on_tool_error_callback decides whether a failure is retryable. Register anything that must apply everywhere as a plugin on the App or Runner, so a new team cannot forget it. Gemini's SafetySetting and HarmBlockThreshold go through generate_content_config, and the 2.8 Model Armor plugin gives you a managed layer.

Code execution is the sharpest edge. Model-generated code must be sandboxed: the Agent Engine sandbox, the Cloud Run sandbox executor from 2.5, ContainerCodeExecutor (hardened by default since 2.4), E2B or Daytona. The docs are explicit that custom executors must be hermetic, with no network connections, and BuiltInCodeExecutor runs in the provider's environment rather than one you control. Relatedly, GCS tools since 2.9 touch local files only inside local_file_root and refuse local access entirely when it is unset — leave it unset unless an agent truly needs it.

Finally, egress and network. VPC Service Controls perimeters constrain where data can go, private Cloud Build worker pools (--worker_pool, 2.8) keep deployment builds off the public internet, and mTLS is supported across the Google API tools, MCP and the Agent Registry, configured through GOOGLE_API_USE_CLIENT_CERTIFICATE and GOOGLE_API_USE_MTLS_ENDPOINT. And because agent output lands in a browser, escape it: model output in a UI is untrusted content and an XSS vector.

Secrets and the supply chain

.env is a local-development mechanism. adk create writes one and warns in as many words: ⚠️ WARNING: Secrets (like GOOGLE_API_KEY) are stored in .env. It also writes a .gitignore, which is the only reason most teams have not committed a key. In production, secrets come from Secret Manager — on Cloud Run through --set-secrets or --update-secrets, with roles/secretmanager.secretAccessor on the runtime service account. ADK ships Secret Manager and Parameter Manager clients for agents that must fetch one, and since 2.8 adk deploy --env logs env var names and not values.

Prefer Application Default Credentials or workload identity over API keys wherever Vertex is in play. The flag that selects between them is GOOGLE_GENAI_USE_ENTERPRISE, and this is the rename that breaks the most tutorials: GOOGLE_GENAI_USE_VERTEXAI is deprecated as of 2.3. The old variable still works with a fallback and warns GOOGLE_GENAI_USE_VERTEXAI is deprecated, please use GOOGLE_GENAI_USE_ENTERPRISE instead. The source checks the new name first, and adk create now writes GOOGLE_GENAI_USE_ENTERPRISE=0. Put the rename in your platform's environment template and grep for the old name in CI.

# .env for a Vertex / Gemini Enterprise deployment
GOOGLE_GENAI_USE_ENTERPRISE=1
GOOGLE_CLOUD_PROJECT=acme-prod
GOOGLE_CLOUD_LOCATION=me-central2

The supply chain is the half of this section teams skip. ADK publishes constraints-3.11.txt and constraints-3.12.txt, introduced in 2.6, which add a four-day buffer against freshly published malicious releases. Use them:

pip install "google-adk[gcp,db,eval]==2.10.0" \
  -c https://raw.githubusercontent.com/google/adk-python/main/constraints-3.12.txt

Pin the exact version. The extras matter operationally, because a missing one is a runtime failure rather than an install failure: eval for adk eval, db for DatabaseSessionService, gcp for Vertex, GCS and Agent Engine, a2a for Agent2Agent, mcp for MCP toolsets, openai for the OpenAI integration that moved to google.adk.integrations.openai in 2.10, and bigquery-analytics since 2.7 moved pyarrow out of the gcp extra — a breaking change for anyone using BigQueryAgentAnalyticsPlugin. Quote the brackets in zsh.

One more supply-chain lever that is easy to forget: CLI telemetry. adk telemetry enable|disable|status controls it, consent was added in 2.6, and the docs say it is off by default. Set it explicitly in your golden image so that a future default change does not surprise your legal team.

Multi-tenancy

ADK gives you three isolation keys, in order: app_name, then user_id, then session_id. Everything else is policy you implement. Artifacts are namespaced by app since 2.6 and scoped to the caller by the 2.4 and 2.5 security fixes, and delete_session has enforced ownership since 2.2 — but those are guards against bugs, not a tenancy model.

The decision is how hard a boundary you need, and there are three honest answers. Soft tenancy puts every tenant in one app, separated by user_id, relying entirely on your gateway's principal mapping: cheapest, and one bug away from a cross-tenant read. Medium tenancy gives each tenant its own app_name in one deployment, which separates artifacts and makes sessions filterable while the database, service account and model quota stay shared — so one tenant's loop still exhausts another's quota. Hard tenancy means a separate deployment, database and ideally project per tenant, which regulated Gulf customers tend to require contractually and you should price accordingly.

The app: state prefix deserves repeating in this context because it crosses every boundary above except the deployment one. Data written under app: is visible to all users of that app. In a soft-tenancy deployment, that is every tenant.

Cost control as an engineering discipline

Cost in an ADK estate is model tokens, and almost everything else is noise. Agent Runtime is billed beyond a no-cost tier; Cloud Run and GKE are ordinary compute; Cloud Trace, Cloud Logging and BigQuery analytics each cost something. But a single chatty agent with a 4,000-token instruction and a twenty-turn conversation will outspend all of it, because the conversation history is resent on every call.

That observation points at the four levers that matter, in order of leverage.

Event compaction shortens the history. App(events_compaction_config=EventsCompactionConfig(...)) takes compaction_interval, overlap_size, token_threshold, event_retention_size and a summarizer, which lets you run either a sliding window or a token-threshold policy. This is the lever that fixes both the cost and the session-store scaling problem described earlier, which is why I reach for it first.

Context caching makes the unchanged prefix cheap. context_cache_config=ContextCacheConfig(...) enables Gemini context caching, which has a 4,096-token minimum — below that there is nothing to cache — and Anthropic cache breakpoints have been supported since 2.8. Caching only works if the prefix is genuinely stable, which is what static_instruction is for: a cache-friendly prefix separate from the instruction that interpolates state. A team that puts the current timestamp in its instruction has disabled its own cache.

Fewer and cheaper calls. max_llm_calls as a budget rather than a ceiling, a cheaper model per sub-agent (the default is gemini-3.5-flash, and LlmAgent.set_default_model(...) or adk run --default_llm_model change it), include_contents='none' on agents that genuinely do not need history, and RunConfig.service_tier added in 2.10.

Measure, then argue. Since 2.8 ADK emits per-invocation and per-workflow token-spend telemetry, and 2.10 added eval efficiency metrics covering duration, tokens and model calls. That combination is what turns a cost conversation from opinion into engineering: you can show the token cost of a prompt change before it ships, in the same CI run that checks correctness.

from google.adk.apps import App, EventsCompactionConfig, ContextCacheConfig

app = App(
    name="support_triage",
    root_agent=root_agent,
    events_compaction_config=EventsCompactionConfig(token_threshold=8000, overlap_size=2),
    context_cache_config=ContextCacheConfig(),
)

Observability you can be paged with

ADK is instrumented with OpenTelemetry out of the box, following GenAI semantic conventions: spans named for invoke_agent, call_llm and execute_tool, workflow spans, gen_ai.* attributes, and native gen_ai.client.* metrics added in 2.2. One version detail will mislead you if you miss it — agent and tool duration metrics changed from milliseconds to seconds in 2.3. Dashboards and alert thresholds built against 1.x or early 2.x are off by a factor of a thousand.

Export has two switches. --trace_to_cloud sends traces to Cloud Trace; --otel_to_cloud sends traces to Cloud Trace and logs to Cloud Logging, and needs the otel-gcp extra. Standardise on --otel_to_cloud, and note that the old --trace_to_cloud flag on adk deploy agent_engine is deprecated there. For non-Google backends the OTel instrumentation is the integration point — Langfuse, Arize Phoenix, W&B Weave and AgentOps all consume it — which makes Langfuse and OpenTelemetry neighbours rather than competitors. AutoTracingPlugin (2.2) adds spans without code changes, and RunConfig.telemetry configures a single request.

The thing to decide deliberately is what goes into a span. Capturing message content in traces is opt-in, and the docs advise caution in production for good reason: a trace with prompts and tool arguments in it is a copy of your customer conversations in your observability vendor, outside whatever residency and retention promises your session store honours. My default is content capture off in production, on in staging, with a documented, time-boxed exception process for debugging a specific incident.

Four signals are worth alerting on, and they are not the usual four:

1. LlmCallsLimitExceededError rate. A loop that costs money. Page on any sustained non-zero rate.
2. 429 rate from the model provider. Separate provider throttling from your own bugs before debugging the agent.
3. Token spend per invocation, p95. The cost-regression detector.
4. Tool error rate by tool name. Agents degrade silently, so a broken tool shows up as worse answers rather than errors.

BigQueryAgentAnalyticsPlugin, with the bigquery-analytics extra, logs events to BigQuery, which is the right home for the quality questions traces answer badly: which tools get called together, where conversations are abandoned, which routes in a graph never fire. The adk web Trace and Events tabs remain a development tool.

Upgrades and migrations

ADK's Python SDK ships a minor release roughly every two weeks with patches between them, and minor releases have repeatedly contained behaviour changes — 2.6, 2.7, 2.9 and 2.10 all did. An estate that floats on google-adk without a pin will change behaviour without a deploy. The upgrade discipline is therefore not optional, and it has five steps.

Pin the exact version and a constraints file. Read the release notes' "Breaking changes" and "Behavior changes" sections, which ADK now publishes per release — they are short and they are the highest-value ten minutes in the process. Run your eval sets and adk conformance test against the new version in CI. Back up the session database and run adk migrate session where the schema changed. Then roll forward one environment at a time.

Keep --adk_version in mind: adk deploy defaults it to your local version, so a developer deploying from an un-upgraded laptop silently pins production to an older SDK. Set it explicitly in CI.

The 1.x-to-2.x migration is the big one, and it is mostly mechanical once you know the list. Pin ~=1.0 to stay put if you need time (npm install @google/adk@^1.6.0 or go get google.golang.org/adk@v1 for the other SDKs); 1.34.0 was the last 1.x release. To move:

Two version facts shape multi-language estates. Java has no 2.0 line — the latest is 1.10.1, and the docs snippet still shows 1.6.0, so tell teams to take the latest. TypeScript is at 2.1.0 and Go at 2.4.0 on the v2 line, import path google.golang.org/adk/v2, Go 1.25 floor, with Go 1.7.0 still maintained for v1. If your platform standard is "ADK 2.x", your Java teams cannot comply, and you should say so rather than let them discover it.

Finally, a test-suite detail that will bite a careful team: 2.10 added DeprecationWarnings to the legacy live-audio modules under google.adk.flows.llm_flows (audio_cache_manager, audio_transcriber, transcription_manager). Suites running pytest with -W error will fail on upgrade. Move the imports to google.adk.live.

Governance: evals and conformance as a release gate

The question "is this agent ready for production" cannot be answered by a code review, because the thing that changed may be a sentence in a prompt. ADK's answer is evaluation, and the platform owner's job is to make it a gate rather than a notebook.

The material is eval sets (*.evalset.json) and test files (*.test.json), and the criteria available cover more ground than most teams use: tool-trajectory match, response match via ROUGE, LLM-judge, rubric-based scoring, safety, hallucination, user simulation, and the efficiency metrics added in 2.10. Trajectory matching is the one I push hardest, because it is the only criterion that catches an agent that reached the right answer by the wrong route — and the wrong route is where the cost and the security exposure live.

Run them from the CLI, from pytest, or from the adk web Eval tab during development:

pip install "google-adk[eval]==2.10.0"
adk eval support_triage/agent.py \
  evals/triage.evalset.json:refund_case,escalation_case \
  --config_file_path evals/criteria.json \
  --print_detailed_results

Select individual cases with the file.evalset.json:case_a,case_b syntax, and do not mix file paths and IDs in one invocation. adk eval_set create and adk eval_set add_eval_case build sets from the CLI, and generate_eval_cases helps bootstrap them. In pytest, await AgentEvaluator.evaluate(agent_module=..., eval_dataset_file_path_or_dir=...) is the entry point, and 2.10 made an important change: evaluate and evaluate_eval_set now raise ValueError when zero eval cases run, where previously that passed silently. If your CI has been green for months, confirm it was actually running cases.

adk conformance record and adk conformance test give you replay-based regression testing: record a real interaction, then assert that a later version reproduces it. This is the check that catches runtime behaviour changes rather than quality changes, which makes it exactly the right gate in front of an SDK bump. adk test runs pytest over agent test JSON, and adk optimize runs the GEPA prompt optimiser against a sampler config, which belongs in a research loop rather than a release gate.

The governance structure I would put around that: a required eval set per agent with trajectory and safety criteria, a conformance recording refreshed per release, both run in CI on every change to prompts, tools or graph structure, and a published cost-per-invocation number in the pull request. Teams own their evals; the platform owns the gate and the templates.

Incident playbooks

Five incidents account for most ADK pages. Each has a signature and a first move.

The agent is looping and the bill is climbing. Signature: Max number of llm calls limit of 500 exceeded, LlmCallsLimitExceededError, token spend per invocation spiking. First move: drop ADK_MAX_LLM_CALLS to contain the spend, then read one trace to find the ping-pong — it is almost always a tool returning something non-terminal, or two agents transferring to each other. Fix with instructions, a terminal tool result, or disallow_transfer_to_peers. Do not fix it by raising the limit.

Conversations lose their history intermittently. Signature: users report amnesia, and it correlates with replica count. First move: confirm which session service is actually in use. The cause is nearly always the silent in-memory fallback on a read-only agents directory, or an in-memory service behind more than one replica. Set explicit URIs and redeploy.

Session <id> not found. Signature: SessionNotFoundError. Cause: running against a session ID that was never created — and since 2.9 this also fires in InMemorySessionService when appending to an unknown session, where it used to discard the event. First move: call create_session first, or enable auto_create_session on the Runner or --auto_create_session on the API server. Related: Session with id <id> already exists. (AlreadyExistsError) means you passed an existing ID to create_session; use get_session or omit the ID. And note that session_id "user" is reserved and rejected as of 2.10.

Everything returns 429. Signature: google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED. First move: confirm it is provider throttling and not your own retry storm, then lean on model failover (2.9) and ReflectAndRetryModelPlugin while you request quota. A retry loop with no backoff makes this worse, which is one more reason not to swallow exceptions inside nodes.

The deploy came up but no agent is served. Signature: No root_agent found for '<name>'. Searched in '<name>.agent.root_agent', '<name>.root_agent' and '<name>/root_agent.yaml'., or Agent not found: '<name>'. No matching directory or module ..., or since 2.5 a 404 on an unknown app_name. First move: check the directory layout — my_agent/{__init__.py,agent.py} with from . import agent in the init file, the variable named root_agent, and the server started from the parent directory. GET /list-apps shows the names the server actually knows. Since 2.8 a type mismatch reports itself distinctly: A 'root_agent' was found for '<name>' but it is not a ... 'root_agent'.

Where ADK stops

Owning a tool well includes knowing what it is not for, and saying so before a team spends a quarter finding out.

ADK is not a workflow engine. A Workflow graph has retries, timeouts, joins and resumption, which makes it look like one. It is not: it has no scheduler, no cron, no backfill, no cross-run dependency graph, no multi-day durability story of the kind Prefect or Airflow provide. If the work is "run this every night over yesterday's data", that is an orchestrator's job, and the orchestrator can call an ADK agent as one step.

ADK is not where deterministic logic belongs. If the routing can be expressed as an if, write the if. A model call to make a decision a rule could make is slower, more expensive and non-deterministic. The honest test for whether you need an agent at all: is there a step where the right next action genuinely depends on judgement over unstructured input? If not, you want functions and a queue.

ADK's server is not an API platform. It has no authentication, no tenancy, no rate limiting. Everything in front of the Runner is a gateway you build or buy.

ADK is not a model router. LiteLlm gives you many providers through one class, but if your problem is budget routing, key management and fallbacks across providers for an organisation, LiteLLM as a proxy is the dedicated tool and ADK should be its client.

ADK is not an evaluation platform. adk eval is good and it is scoped to agent trajectories and responses. For dataset-centric RAG evaluation, RAGAS is the specialist; for trace analytics and production quality over time, Langfuse is.

The framework comparison, briefly, since you will be asked. ADK's distinctive strengths are the graph runtime with resumption, first-class evaluation and conformance tooling, deep Google Cloud integration, and four language SDKs. LangGraph overlaps most on graph orchestration, with a larger ecosystem and no cloud affinity; CrewAI optimises for role-based teams with less runtime control. Interoperability lowers the stakes: ADK consumes MCP tools through McpToolset, serves itself over MCP with to_mcp_server (2.5), exposes agents over Agent2Agent with to_a2a and consumes remote ones with RemoteA2aAgent.

Running ADK as a platform for other teams

Everything above becomes a product when more than one team uses it. The deliverable is not a document; it is a template, a gate and a set of defaults that are hard to get wrong.

Choose one runtime and one storage default. Agent Runtime on Agent Platform if managed sessions, Memory Bank, sandboxing and auto-scaling are worth the bill and the lock-in. Cloud Run if you want serverless containers and your own database — this is the sweet spot for most organisations. Kubernetes via adk deploy gke if you need open models, GPUs or existing cluster policy, noting that --service_type defaults to ClusterIP. Your own container with get_fast_api_app when you need middleware ADK does not provide. Pick one, document the other three as exceptions, and provide the deploy command as a CI job rather than a README snippet.

Ship a template repository, not instructions. It should contain the agent layout, an App with your mandatory plugins already registered, explicit service URIs per environment, the pinned install line with constraints, an eval set with at least one trajectory and one safety case, a conformance recording, the CI jobs that run both, and the greps for deprecated patterns. A team that clones this gets your platform's guarantees for free; a team that writes from scratch gets none of them.

Make policy code. Anything you would write in a wiki as "always do X" should be a plugin on the App, a before_tool_callback registered globally, a CI check, or a deployment flag your pipeline sets. Guardrails that depend on a team remembering are not guardrails. --extra_plugins my.mod.Plugin lets you inject one at the API-server level, which is how you enforce a logging or Model Armor policy across agents you did not write.

Own the version, not the code. You decide which ADK version is current, you read the behaviour-change notes, you run the conformance suite against the bump, and you roll it out. Teams own their agents' prompts, tools and evals. That division is the whole bargain, and it works because the behaviour changes that break them are not visible from inside their own code.

Publish a cost and quota model. Token spend per invocation per agent, from the 2.8 telemetry, on a dashboard every team can see, plus a per-tenant quota budget enforced at your gateway. Without this, the first expensive agent silently throttles everyone else and nobody knows why.

Define the readiness bar. Mine has seven items: explicit session and artifact URIs; auth in front of the endpoint with a principal-to-user_id mapping you can point at; max_llm_calls set below the default; timeouts on every node that calls out; an eval set and a conformance recording running in CI; traces exported with content capture off; and a named owner. An agent missing any of these is a prototype, whatever it is called in the planning document.

The review checklist

Use this on pull requests and design documents. It is ordered by how often each item is actually wrong.

The complete picture

Put the three levels together and the shape is simple. An agent is a model with tools and instructions. A Workflow is a graph of nodes — agents, functions, tools or nested graphs — whose edges carry the routes the nodes choose. A Runner drives that graph and emits an immutable stream of Events, and an App wraps it with plugins, compaction, caching and resumability. Everything durable lives in a service you selected, and the serving process is a stateless FastAPI application you scale by adding copies.

The senior work is almost entirely about the boundaries of that picture. The edge, because ADK ships no authentication and no tenancy. The storage, because the default is local and the fallback is silent. The authority behind each tool, because a model that can be argued with should not hold unconditional credentials. The token bill, because it dominates cost and responds to four specific levers. The version, because minor releases change behaviour your teams cannot see coming.

If you remember five things from this level, make them these. Set your service URIs explicitly, or you will lose conversations. Map the principal to user_id yourself, or you will leak across tenants. Make failing nodes idempotent, or resume will double-charge someone. Read the behaviour-change notes before every bump, because 2.6, 2.7, 2.9 and 2.10 each shipped one that fails quietly. And put the guardrails in code, because the ones in the prompt are suggestions.

Where the series leaves you

Across three levels you have gone from adk create my_agent to owning a multi-tenant agent platform: the mental model and first agent at Beginner, graphs, tools, sessions, evaluation and deployment at Mid, and architecture, security, cost, upgrades and governance here. You can now read an ADK release note and predict which of your agents it breaks, which is the clearest marker of ownership.

Where to go next depends on which boundary you are standing on. If the gap is production quality and trace analytics, Langfuse is the next guide, and ADK's OpenTelemetry instrumentation plugs straight into it. If it is retrieval quality, RAGAS. If it is the tool layer you want to share across frameworks, MCP. If it is the runtime, Kubernetes and Terraform are the infrastructure this platform sits on. And if it is model access and budget across providers, LiteLLM.

One last habit. ADK moves roughly every two weeks, and this guide is a snapshot of 2.10.0. The release notes are short, the behaviour-change sections are honest, and reading them is twenty minutes a fortnight. In a framework this young, that reading is not maintenance overhead — it is the job.

Sources

• ADK documentation home
• Installation
• Get started with Python
• What's new in ADK 2.0
• Graphs and routes
• Graph data handling and human input
• Dynamic graphs
• Agent collaboration
• Command-line reference
• Session state
• Safety and security
• Deploy, Agent Runtime, Cloud Run, GKE
• Observability and traces
• Release notes
• Python API reference
• adk-python CHANGELOG and releases
• google-adk on PyPI

Senior ADK interviews are not about the API. Nobody will ask you to recite the fields of LlmAgent. They will describe a system, hand you a whiteboard, and listen for whether you reason about state, authority, cost and blast radius — or whether you reach for an agent because agents are interesting. Everything below is framed against Google ADK 2.10 for Python.

What the interviewer is actually testing

Four things, in roughly this order of weight.

Do you know where the state is? An ADK deployment is a stateless FastAPI process plus pluggable services. Candidates who can say that in one sentence, and then name the consequence — in-memory services behind two replicas lose conversations — are immediately in the top quartile.

Do you treat the model as untrusted? Senior agent engineering is mostly about what a persuadable component is allowed to do. If you say "I put the rule in the system prompt", you have failed the question.

Can you talk about cost without hand-waving? Token spend is the bill. If you can name four levers and say which you would pull first, you sound like someone who has paid an invoice.

Do you know when not to use the tool? The strongest signal in any framework interview is a candidate who says "this does not need an agent" at the right moment.

The thirty-second framing

ADK is Google's open-source, model-agnostic agent framework. You write agents as code — Python, TypeScript, Go or Java — and since 2.0 they execute as nodes in a graph: a Workflow with edges, routes, retries, timeouts and resumption. A Runner drives the graph and emits an immutable stream of events, and everything durable lives in services you choose: sessions, artifacts, memory, credentials. It is optimised for Gemini and Google Cloud but runs Claude, OpenAI, LiteLLM and Ollama models, and it ships evaluation and conformance tooling, which most competitors do not.

If they push on what makes it different, give three things: the graph runtime with resumption, first-class evaluation and replay-based conformance testing, and four language SDKs.

System-design prompts

"Design a customer-support agent for a bank in the Gulf, serving two million customers"

Start with the boundary, not the agent. ADK's own CLI help says its endpoints are unauthenticated and must be run on a trusted network behind your own authentication and authorization layer. So layer one is a gateway that authenticates the customer and maps the authenticated principal to user_id server-side. Say that explicitly: the API server does not authenticate user_id, so a client that can reach /run can claim to be anyone. That one sentence is often the whole interview.

Layer two is the runtime. Cloud Run with a minimum instance count, deployed via adk deploy cloud_run, with everything after a bare -- passed through to gcloud run deploy so you can set --no-allow-unauthenticated --min-instances=2. No --with_ui.

Layer three is storage, and here you name the trap before they ask. Sessions and artifacts must have explicit URIs — --session_service_uri postgresql+asyncpg://... and --artifact_service_uri gs://... — because on Cloud Run the agents directory is usually not writable and ADK falls back silently to in-memory services. Note the async driver requirement: DatabaseSessionService is SQLAlchemy-asyncio, so postgresql:// fails and postgresql+asyncpg:// works, with SQLAlchemy arriving through the db extra.

Layer four is residency, which in this scenario is the differentiator. The session store holds customer messages and tool arguments, so it sits in a region your commitments allow, and GOOGLE_CLOUD_LOCATION decides where the model call goes — a Gulf deployment with us-central1 in its environment has moved the prompt across a border.

Layer five is authority. Balance lookups and transaction history are per-customer data, so user-auth OAuth, not agent-auth. Explain the distinction: agent-auth means tools act as the service account and will read another customer's rows if the model is persuaded to ask; user-auth means the agent can do exactly what the authenticated customer could.

Then the budget: max_llm_calls well below the 500 default, timeout on every node that calls a core banking API, max_concurrency on any graph that fans out, event compaction so a long conversation does not grow unboundedly, and static_instruction plus context caching for the stable prefix (Gemini caching needs at least 4,096 tokens to do anything).

Close with the gate: an eval set with trajectory and safety criteria plus a conformance recording, both in CI.

"Ten product teams want to build agents. You own the platform. What do you build?"

The answer is a template, a gate and a default — not a document.

The template repository carries the agent layout, an App with your mandatory plugins registered, explicit per-environment service URIs, a pinned install line with the published constraints file, an eval set with at least one trajectory case, a conformance recording, and the CI jobs that run both. The point to articulate: a team that clones this inherits your guarantees; a team that starts from the quickstart inherits none of them.

The gate is version ownership. You decide the current ADK version, you read the release notes' "Breaking changes" and "Behavior changes" sections, you run conformance against the bump, and you roll it out. Justify it with the fact pattern: 2.6 namespaced artifacts by app, 2.7 moved pyarrow out of the gcp extra, 2.9 changed failed nodes to rerun on resume, 2.10 stopped substituting ${var} in instructions. None of those are visible from inside a team's own code.

The default is one runtime and one storage choice, with the other options documented as exceptions: Agent Runtime on Agent Platform if managed sessions and Memory Bank are worth the bill, Cloud Run for most organisations, GKE when you need open models or existing cluster policy, your own container with get_fast_api_app when you need middleware ADK does not provide.

Then say the thing interviewers wait for: policy must be code. Anything you would write in a wiki as "always do X" becomes a plugin on the App, a globally registered before_tool_callback, a CI check, or a flag your pipeline sets. --extra_plugins my.mod.Plugin lets you inject one at the API-server level into agents you did not write.

"Design multi-tenancy for a SaaS product built on ADK"

Name the isolation keys first: app_name, then user_id, then session_id. Then present three tiers honestly. Soft tenancy — one app, tenants separated by user_id — is cheapest and one gateway bug away from a cross-tenant read. Medium tenancy gives each tenant its own app_name, which separates artifacts and makes sessions filterable, but the database, service account and model quota stay shared, so one tenant's loop still exhausts another's quota. Hard tenancy is a deployment, database and ideally project per tenant, which regulated customers often require contractually.

Then the detail that proves you have read the state documentation: the app: state prefix is visible to all users of the app, so it must never hold tenant or user data. user: is per user within the app, no prefix is session-scoped, and temp: lasts one invocation and is never persisted.

"An agent needs to run model-generated Python over a customer's data"

The answer is a sandbox, and you should name the options: the Agent Engine sandbox, the Cloud Run sandbox executor, ContainerCodeExecutor (hardened by default since 2.4), E2B or Daytona. Add that the docs require custom executors to be hermetic, with no network connections, and that BuiltInCodeExecutor runs in the model provider's environment rather than one you control. Then add the data question: if the code touches customer data, the sandbox's egress and the data's residency are the same conversation.

Incident-response questions

"Your token bill tripled overnight. Walk me through the first ten minutes"

Containment before diagnosis. Set ADK_MAX_LLM_CALLS low to cap per-invocation spend, which converts an unbounded loop into a bounded error you can count. Then read one trace end to end — invoke_agent, every call_llm, every execute_tool — and compare the call count with what you believed the agent did. The cause is almost always a tool returning something non-terminal so the model re-asks, or two agents transferring back and forth; the fix is tighter instructions, a terminal tool result, or disallow_transfer_to_parent and disallow_transfer_to_peers. Finish with the structural answer: per-invocation token-spend telemetry has existed since 2.8, so the real failure was not having a p95 alert on it.

The anti-answer, which interviewers listen for: raising max_llm_calls. That turns a crash into an expensive crash.

"Users say the agent forgets the conversation, but only sometimes"

Say "it correlates with replica count" before they tell you. Then: confirm which session service is actually in use, because the likely cause is either an in-memory service behind more than one replica or the silent local-storage fallback on a read-only agents directory. The fix is explicit URIs. Mention that since 2.9 InMemorySessionService raises SessionNotFoundError on an append to an unknown session rather than discarding the event silently, which makes this class of bug louder than it used to be.

"After upgrading, a workflow charged a customer twice"

This is the 2.9 question, and it has a precise answer: before 2.9, a node that failed replayed on resume as though it had completed; from 2.9 it runs again. Any node body with a side effect must be idempotent. The immediate fix is an idempotency key derived from the invocation, and the structural fix is a review-checklist item plus a test that fails a node once under ResumabilityConfig and asserts the side effect happened exactly once.

"You deployed and the service returns 404 for your agent"

Read the error rather than guessing. No root_agent found for '<name>'. Searched in '<name>.agent.root_agent', '<name>.root_agent' and '<name>/root_agent.yaml'. means the layout is wrong: my_agent/{__init__.py,agent.py} with from . import agent in the init file, a variable actually named root_agent, and the server started from the parent directory. Agent not found: '<name>' or a 404 on an unknown app_name means the name in the request does not match a folder — GET /list-apps shows what the server knows. Since 2.8, a type mismatch reports itself distinctly.

"Everything is returning 429"

google.genai.errors.ClientError: 429 RESOURCE_EXHAUSTED is provider throttling, not your bug. Establish that first, because teams waste hours debugging the agent. Then: model failover to backup models (2.9), ReflectAndRetryModelPlugin for backoff, and a quota request. Add the self-inflicted variant — a retry storm — and connect it back to a code smell: a broad except Exception inside a node both disables ADK's automatic retries and invites your own badly-behaved ones.

"A session database upgrade went wrong"

Schema errors mentioning missing node_info or output columns mean the 2.0 event schema change was not applied: back up, then adk migrate session --source_db_url ... --dest_db_url .... If the warning mentions the legacy v0 pickle schema, the same command moves off it, and --allow-unsafe-unpickling is for trusted databases only. In development, deleting .adk/session.db is faster than migrating.

"A tool started returning wrong answers and nobody noticed for a week"

This question is about observability, and the right answer refuses the obvious one. Agents degrade gracefully: a tool that returns stale or empty results does not raise, so your endpoint stays healthy, your latency is fine, and the answers quietly get worse. So the signal you need is per-tool error and result-shape monitoring rather than HTTP status codes, plus a quality signal from evaluation run on a schedule against production-like cases.

Then name the structural remedy: BigQueryAgentAnalyticsPlugin, with the bigquery-analytics extra, logs events to BigQuery, which is the right place for the questions traces answer badly — which tools get called together, where conversations are abandoned, which routes in a graph never fire. A route that has never fired in production is either dead code or a bug, and you cannot see that in a trace.

Finish with the policy point: a tool is a dependency, and dependencies get health checks. If the team that owns the downstream API does not know an agent is calling it, that is the finding, not the stale data.

How to argue a trade-off to leadership

Three arguments come up repeatedly, and each has a shape that works.

"Why does this cost so much?" Do not explain tokens. Show a number: token spend per invocation for the agent, from the telemetry, and the same number after you added static_instruction, context caching and event compaction. Then name the one thing leadership controls — scope. An agent that answers anything costs more than an agent that answers three questions, because the instruction, the tool list and the history all grow. Offer the trade explicitly: narrower scope, cheaper model per sub-agent, or a higher bill.

"Why can't we just use the system prompt for the rules?" Frame it as authority, not capability. A prompt is a request to a component that can be argued with; a before_tool_callback is a check in code that fails closed. Then make it concrete with the worst case for your business — a refund issued, a record deleted, a customer's data read by another customer — and point out that the control costs about twenty lines. Nobody argues with that framing; plenty argue with "prompt injection is a risk".

"Why do we need to spend a week on evaluation?" Because the thing that changes is a sentence, and code review cannot catch a sentence. ADK gives you trajectory matching, which is the only criterion that catches an agent reaching the right answer by the wrong route — and the wrong route is where both the cost and the security exposure live. Add the 2.10 detail that makes the case stronger: AgentEvaluator.evaluate now raises ValueError when zero cases run, where it used to pass silently, so a suite that was green may never have been running.

A fourth question is coming in most organisations: "should we standardise on ADK?" The honest answer names the constraints. Java has no 2.0 line — the latest is 1.10.1 — so a polyglot estate cannot fully standardise on 2.x today. Python ships a minor release roughly every two weeks, several of which have carried behaviour changes, so someone must own version uptake. And interoperability lowers the stakes anyway: ADK consumes MCP tools, serves itself over MCP, and speaks Agent2Agent in both directions, so the tool layer survives a framework change even if the orchestration does not.

Questions you should ask them

Interviewers read these as seniority signals, and all four are genuinely diagnostic. Which session service is production using, and was it set explicitly? Who owns the ADK version, and what happened at the last bump? Do agents run with agent-auth or user-auth credentials for per-user data? And what gates a prompt change — a review, or an eval suite?

Sixty-second self-test

Answer out loud. Where does an ADK deployment keep state, and what happens on two replicas with the default configuration? What does app: state scope to, and why is that dangerous? Which 2.x change makes a custom _run_async_impl do nothing? What changed in 2.9 about failed nodes on resume, and what does that require of your code? What is the default max_llm_calls, and why is raising it the wrong fix? Name the four cost levers. What does --otel_to_cloud do that --trace_to_cloud does not? Which environment variable replaced GOOGLE_GENAI_USE_VERTEXAI? Why does postgresql:// fail as a session URI? And what single sentence from ADK's own CLI help defines your security obligations?

If any of those needed a pause, that section of the detailed guide is where to go back.

These are the things that only show up once ADK is carrying other people's traffic: the traps that need scale or a version bump to appear, the habits that make platform ownership sustainable, and the checklists I actually use on pull requests and design documents. Verified against Google ADK 2.10 for Python.

Production traps that only appear at scale

The silent in-memory fallback. Local runs persist sessions to <agents_dir>/<agent>/.adk/session.db and artifacts to .adk/artifacts. On Cloud Run or Kubernetes that directory is usually read-only, and ADK falls back to in-memory services instead of failing. Symptom: the deployment is healthy, works perfectly in testing with one instance, and starts losing conversations the moment it scales. Always pass --session_service_uri and --artifact_service_uri explicitly, and treat ADK_FORCE_LOCAL_STORAGE and ADK_DISABLE_LOCAL_STORAGE as debugging tools rather than configuration.

The synchronous database URL. DatabaseSessionService runs on SQLAlchemy's asyncio layer, so postgresql:// fails with Invalid database URL format or argument or Database related module not found for URL '<redacted>'. You need postgresql+asyncpg, mysql+aiomysql, mariadb+asyncmy or sqlite+aiosqlite, and SQLAlchemy itself comes from the db extra — without it the message is explicit: The 'sqlalchemy' package is required to use this feature. Please install it by running: pip install google-adk[db].

The app: state prefix. It looks like application state and behaves like a global visible to every user of the app. In a deployment where tenants share an app_name, anything written under app: crosses the tenant boundary. Block the pattern in review unless the value is a configuration constant.

user_id is not an identity. The API server does not authenticate it. If your gateway forwards a client-supplied user_id, any caller can read any user's sessions. The mapping from authenticated principal to user_id must happen server-side, in code you own.

The 2.9 resume rerun. Before 2.9, a failed node replayed on resume as though it had completed; now it runs again. Every node body with a side effect — a charge, a ticket, a message, a downstream write — must be idempotent or resumption duplicates it. This is the single most expensive quiet change in the 2.x line.

Swallowed exceptions disable retries. A broad except Exception inside a node tells the engine the node succeeded, so RetryConfig never fires. Let exceptions propagate and configure retries declaratively.

Appended events break determinism. context.session.events.append() puts events outside the replay the graph engine depends on, so resumption and conformance replay diverge from the live run. Always yield.

Ignored _run_async_impl. Custom overrides are silently ignored by the 2.x runtime. A 1.x custom agent that did its real work there now does nothing, with no error at all.

Fan-out multiplies everything. A route that returns a list fans out, and since 2.10 DEFAULT_ROUTE can too. Without Workflow.max_concurrency, the width of the fan multiplies your model spend and your downstream load simultaneously.

No timeout means no bound. A hanging MCP server or slow HTTP tool holds an invocation open indefinitely, consuming a request slot and, on serverless, billing you for the wait. Set timeout on every node that calls out and handle NodeTimeoutError.

Raising max_llm_calls is not a fix. The default is 500 per invocation. Max number of llm calls limit of 500 exceeded means a loop; raising the ceiling converts a crash into an expensive crash, and a value of zero or less means unlimited. Prohibit that by policy.

Per-session write locks. DB session writes take per-session in-process locks, so one hot session serialises inside a replica and concurrent writers across replicas depend entirely on the database. Long conversations are the pathology: every turn appends events that get loaded and re-serialised.

BigQuery protected mode is process-local. Since 2.10 its session state lives in process memory, so it neither survives restarts nor works across replicas, and non-SELECT statements must land in the session's anonymous dataset.

Reload in containers. The help says reload is not supported for Cloud Run. Use --no-reload.

Artifacts moved in 2.6. File artifacts became namespaced by app, a breaking change. If artifacts "disappeared" across that upgrade, the paths changed.

pyarrow left the gcp extra in 2.7. BigQueryAgentAnalyticsPlugin users need google-adk[bigquery-analytics].

Templating changed in 2.10. ${var} and backslash-escaped \{var} are now left exactly as written; only {var} is substituted. A prompt that relied on ${var} now reaches the model as literal text — a quality regression with no error anywhere.

Metric units changed in 2.3. Agent and tool durations went from milliseconds to seconds. Dashboards and alert thresholds built earlier are wrong by a factor of a thousand.

-W error test suites break on 2.10. The legacy live-audio modules under google.adk.flows.llm_flows now emit DeprecationWarning. Import from google.adk.live.

Zero eval cases used to pass. Since 2.10, AgentEvaluator.evaluate and evaluate_eval_set raise ValueError when no cases run. If your CI has been green for months, confirm it was ever running anything.

session_id "user" is reserved and rejected as of 2.10.

Platform-ownership habits

Own the version; let teams own the prompts. You decide the current ADK version, read each release's "Breaking changes" and "Behavior changes" sections, run conformance against the bump, and roll it forward one environment at a time. Teams own their agents' instructions, tools and evals. The bargain works because the changes that break them — 2.6, 2.7, 2.9, 2.10 — are invisible from inside their own code.

Pin exactly, and use the constraints file. pip install "google-adk[gcp,db,eval]==2.10.0" -c <constraints-3.12.txt>. The constraints files, published since 2.6, add a four-day buffer against freshly uploaded malicious releases. Also set --adk_version in CI, because adk deploy defaults it to the deploying machine's version — a developer's laptop can silently pin production to an older SDK.

Ship a template, not a wiki page. Agent layout, an App with mandatory plugins registered, explicit per-environment service URIs, the pinned install line, an eval set with a trajectory case, a conformance recording, and the CI jobs that run both. A team that clones it inherits your guarantees; a team that starts from the quickstart inherits none.

Make policy code. Every "always do X" becomes a plugin on the App, a globally registered callback, a CI check, or a flag your pipeline sets. --extra_plugins my.mod.Plugin injects one at the API-server level into agents you did not write. A guardrail that depends on someone remembering is not a guardrail.

Publish the cost model. Per-invocation token spend per agent, from the telemetry available since 2.8, on a dashboard every team can see, with a per-tenant quota budget enforced at your gateway. Without it, the first expensive agent throttles everyone else and nobody knows why.

Keep a written readiness bar. Mine: explicit session and artifact URIs; authentication in front of the endpoint with a server-side principal-to-user_id mapping; max_llm_calls below the default; timeouts on every outbound node; an eval set and conformance recording in CI; traces exported with content capture off; a named owner. Anything missing one of those is a prototype whatever the roadmap calls it.

Decide the trace-content policy once. Capturing message content in traces is opt-in and the docs advise caution in production, because a trace with prompts and tool arguments in it is a copy of your conversations in your observability vendor, outside the residency and retention promises your session store honours. Off in production, on in staging, with a time-boxed exception process.

Say no to agents sometimes. If the routing can be an if, write the if. The honest test: is there a step where the right next action genuinely depends on judgement over unstructured input? If not, functions and a queue are cheaper, faster and deterministic. Saying this early is the most valuable thing a platform owner does.

Know the limits you cannot fix. Java has no 2.0 line (1.10.1 is current), so a polyglot estate cannot fully standardise on 2.x. A Workflow cannot yet be an LlmAgent sub-agent, so some 1.x hierarchies need restructuring rather than substitution. Tell teams before they discover it.

Read the release notes every fortnight. Python ships a minor release about every two weeks. The behaviour-change sections are short and honest, and twenty minutes a fortnight is the cheapest incident prevention available.

Operational recipes

Find the agents burning calls. Set ADK_MAX_LLM_CALLS to a low value in staging for a day. Every agent quietly making dozens of calls per turn surfaces as LlmCallsLimitExceededError, and you end up with an inventory instead of a suspicion.

Measure before optimising context. Record per-invocation token spend for a day, add static_instruction for the stable prefix and a token-threshold compaction policy, then compare. Write the percentage down; it is the number you quote next time someone proposes an agent for a job a function call would do. Context caching only helps above Gemini's 4,096-token minimum and only if the prefix is genuinely stable — a timestamp in the instruction disables your own cache.

Audit session growth. Count events per session in the production database and take the ninety-ninth percentile. If it is in the hundreds, event compaction comes before anything else, because it fixes the cost problem and the store-pressure problem together.

Shadow-mode your guardrails. Add a before_tool_callback to the highest-risk tool that logs every call's arguments to an audit stream and blocks nothing for a week. Reading what the model actually tried to do is consistently more informative than any threat model.

Gate SDK bumps with conformance. adk conformance record a representative interaction, then adk conformance test against the new version in CI. Evals catch quality regressions; conformance catches runtime behaviour changes, which is exactly what a minor ADK release carries.

Grep for the deprecated patterns. One CI job, twenty lines: GOOGLE_GENAI_USE_VERTEXAI, _run_async_impl, session.events.append, app: state writes, SequentialAgent, ParallelAgent, LoopAgent, and ${ inside instruction strings. It will find something on its first run.

Clean up old images. Before 2.6, adk deploy uploaded .adk/session.db, so images built from a developer's laptop shipped that laptop's conversation history. ADK 2.10 also excludes the .adk folder and generates exec-form Dockerfiles. If pre-2.6 images are still running, treat it as a disclosure to remediate.

Set telemetry consent explicitly. adk telemetry enable|disable|status. It is off by default per the docs; setting it deliberately in your golden image means a future default change does not surprise anyone.

Rehearse one incident per quarter. Pick a failure you have written a playbook for — the in-memory fallback, a looping agent, a botched session migration — and actually run it in staging. The playbooks that survive contact are the ones someone has executed once. The common discovery is not that the fix is wrong but that nobody knows which session service production is really using, which is itself the finding.

Keep a one-page inventory. Every agent, its owner, its runtime, its session store, its model, whether it uses agent-auth or user-auth, its eval coverage, and its p95 token spend. Ten rows of this answers more questions in an incident than any amount of documentation, and it makes the gaps visible: the agent with no owner, the one with agent-auth over per-user data, the one whose evals have not changed since it launched.

Make the migration path explicit before the deprecation lands. SequentialAgent, ParallelAgent and LoopAgent still run and warn today; they will not forever. Write the Workflow equivalent for one real agent, publish it as the reference, and set a date. Doing this while the old path still works is an afternoon; doing it after a removal is an outage.

The pull-request checklist

Run this on any change to an agent, its tools, its graph or its prompts.

1. Session and artifact service URIs explicit, with an async DB driver, and no in-memory service behind more than one replica.
2. The principal-to-user_id mapping happens server-side; the client cannot choose its own ID.
3. No app: state holding user or tenant data.
4. Tool credentials scoped per tool, and user-auth OAuth wherever the data is per-user.
5. Destructive tools guarded in code — before_tool_callback validation, allow-lists, require_confirmation — never in the prompt.
6. Node bodies with side effects are idempotent.
7. Exceptions propagate inside nodes; no broad except Exception.
8. Events yielded, never appended.
9. No _run_async_impl override.
10. max_llm_calls set, max_concurrency set on fan-out graphs, timeout on every outbound node.
11. {var} templating, not ${var}; static_instruction used for the stable prefix.
12. Model-generated code runs in a hermetic sandbox with no network access.
13. Secrets from Secret Manager, not .env; GOOGLE_GENAI_USE_ENTERPRISE, not the deprecated name.
14. Install line pinned to an exact version with a constraints file and the needed extras.
15. Eval set updated, with at least one trajectory criterion, and conformance re-recorded if the runtime path changed.
16. No --with_ui, no reload, in any production target.

The design-document checklist

Different questions, asked earlier, where they are cheap to answer.

1. Does this need an agent at all? If the decision is rule-expressible, the answer is no.
2. What is the blast radius of the worst tool call? Name it concretely — a refund, a deletion, a cross-tenant read — then show the control that makes it impossible rather than unlikely.
3. Which tenancy tier? Soft, per-app_name, or a separate deployment. State which, and what it costs if the gateway mapping is wrong.
4. Where does the conversation data physically live? Session store region, model region via GOOGLE_CLOUD_LOCATION, trace content capture, and retention. For Gulf and Egyptian customers this is usually contractual, not advisory.
5. Which runtime, and why not the default? Agent Runtime, Cloud Run, GKE, or your own container with get_fast_api_app.
6. What is the expected cost per invocation, and which of the four levers — compaction, caching, fewer or cheaper calls, service_tier — is already applied?
7. What is the failure behaviour? Which nodes retry, which are idempotent, what happens on resume, what the user sees when the model is throttled.
8. How will we know it degraded? Tool-level error and result monitoring, scheduled evals, and the four alerts that matter: limit-exceeded rate, 429 rate, p95 token spend, and per-tool error rate.
9. What gates a prompt change? If the answer is "review", the design is not finished.
10. Who owns it in six months, and who upgrades the SDK under it?

Where to take this next

If the gap is production quality and trace analytics, Langfuse consumes ADK's OpenTelemetry output directly; so does any OpenTelemetry backend. For retrieval quality, RAGAS. For a tool layer that outlives a framework choice, MCP. For the runtime underneath, Kubernetes and Terraform. And for provider budgets, keys and fallbacks across an organisation, LiteLLM as a proxy with ADK as its client.