The Complete FastAPI for ML Guide

This is part one of three. It covers everything you need to put a machine learning model behind a web API with FastAPI, written for someone who has never built an API before. By the end you can install FastAPI, write endpoints that accept and validate data, load a model once when the server starts, return predictions as JSON, read the error messages the framework gives you, test the whole thing without starting a server, and package it for a colleague. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and an API only becomes real once you have sent it a bad request and watched it answer politely.

The guide was checked against FastAPI 0.142.2, released on 30 September 2026. FastAPI is still a 0.x library, so minor versions can change behaviour. We will point out the places where that matters.

What FastAPI is, and the problem it solves

A trained model is a file and a function. It sits in a notebook, and the only person who can use it is the one who owns the notebook. To make it useful to a mobile app, a website, a colleague's script, or another service, you need a way for other programs to send it inputs and receive outputs over a network. The standard way to do that is an HTTP API: a program that listens on a port, receives requests such as "predict the price for this house", runs your code, and sends back a response, usually in a text format called JSON.

FastAPI is a Python framework for building exactly that. You write ordinary Python functions, add type hints, and FastAPI turns them into web endpoints. It does three jobs for you that you would otherwise write by hand:

• It reads the request and converts the raw text into Python values.
• It validates those values against the types you declared and rejects bad input with a clear error before your code ever runs.
• It writes documentation for your API automatically, including a page where you can try each endpoint from your browser.

The diagram is the whole idea. The middle box is where FastAPI earns its keep, and the rest of this guide fills in each arrow.

It helps to know what came before. Python has had web frameworks for a long time. Flask is small and flexible, but it leaves validation and documentation to you, so every project grows its own handwritten checks. Django is large and includes a database layer and an admin interface, which is more than a model-serving service needs. FastAPI sits in between: small like Flask, but with validation and documentation built in because it reads your type hints.

Under the hood FastAPI is two libraries glued together by a thin layer of its own. Starlette handles the web part: routing, requests, responses, middleware. Pydantic handles the data part: checking that values have the right type and shape. FastAPI runs on ASGI, a standard interface between Python web applications and the servers that run them. ASGI is what lets one server process handle many waiting requests at once, which matters later when we discuss async.

Why does this matter for machine learning in particular? Because a model is only valuable once something else can call it, and because the failures in model serving are mostly boring input failures: a missing field, a string where a number was expected, a list of the wrong length. A framework that rejects those at the door, with an error that tells the caller exactly what was wrong, saves you days of debugging. It also gives you a clear contract. The documentation page that FastAPI generates is the contract between your model and everyone who uses it.

One note for readers in the Gulf and Egypt. If your employer handles personal or regulated data, where the API runs matters as much as how it is written. A prediction endpoint that receives customer records is subject to the same data-residency rules as the database those records came from. Keep that in mind when you reach the deployment sections, because a cloud region is a decision, not a default.

What you need to follow along: Python installed, a terminal, and a text editor. No web experience is assumed. We define each term as it appears.

How the web conversation works

Before writing any FastAPI, you need five terms that every web API shares. They are not specific to FastAPI, and you will use them in every interview and every debugging session.

An HTTP request is a message a client sends to a server. It has a method (the verb), a path (the address of the thing on the server), optional headers (labelled metadata), and an optional body (the payload). The server answers with an HTTP response: a status code, headers, and usually a body.

The methods you need are few. GET asks for something and should not change anything on the server. POST sends data to be processed or stored. There are others (PUT, PATCH, DELETE) but a first model API needs just these two. A prediction can be a GET when the inputs are one or two numbers, and a POST when the inputs are a structured record.

The path may include a path parameter, a variable piece such as /items/42, where 42 identifies one item. It may also carry a query string, the part after the question mark, such as /predict?x=2. Query parameters are for optional or simple settings. Large or structured inputs belong in the body, which for APIs is almost always JSON: a text format made of objects (curly braces with named fields), lists (square brackets), strings, numbers, booleans and null.

Status codes are three-digit numbers that tell the client what happened. You will meet these constantly:

The first digit is the useful part. Codes starting with 2 mean success, 4 mean the caller made a mistake, and 5 mean the server did. When a request fails, that first digit tells you whose problem it is before you read anything else.

Finally, a server is a long-running program that listens on a port, a numbered door on a machine. When you run an API locally you will see an address like http://127.0.0.1:8000. The first part, 127.0.0.1, means "this computer only", and 8000 is the port. Nobody else on the network can reach it, which is exactly what you want while learning.

The mental model: four nouns

FastAPI has a large documentation site, but the core fits in four nouns. Learn these and every example you read will make sense.

Path operation. A function bound to a method and a path through a decorator. The line @app.get("/predict") means: when a GET request arrives at /predict, call the function underneath. The function itself is the path operation function. "Path operation" is FastAPI's name for the pairing of the two. (A decorator is a line starting with @ that wraps the function below it. You do not need to know how decorators work to use this one.)

Parameters. What the request carries in. FastAPI looks at your function's arguments and their type hints and decides where each comes from. An argument that matches a name in the path is a path parameter. A simple-typed argument that does not is a query parameter. An argument whose type is a Pydantic model is read from the request body. Header and cookie parameters exist too, and you declare them explicitly.

Pydantic model. A class that describes the shape of a piece of data: its field names, their types, and which are optional. Models are used for request bodies and for responses. When data arrives, Pydantic checks it against the class, converts what it can (the string "3" becomes the integer 3 where the declared type is int), and raises an error describing anything that does not fit.

Dependency. A function that FastAPI calls for you before your path operation function and whose result is handed in as an argument. Dependencies are how you share things like a loaded model, a database connection, or a login check across many endpoints without copying code. They appear in a later section. For now just remember that you declare them with Depends().

A fifth idea ties the first four together: the application object. You create it once with app = FastAPI(), and it collects every path operation you register. The server is told to run app, and everything else hangs off it.

The order matters for debugging. If the request path does not match any path operation, you get 404. If it matches but the parameters do not validate, you get 422 and your function never runs. Only if both pass does your code execute. When something fails, ask which stage it failed at.

Installing FastAPI and checking the setup

The FastAPI documentation recommends the uv package manager, and plain pip works everywhere. Both approaches are shown. Use whichever you already have.

With uv:

uv init awesome-project --bare
cd awesome-project
uv add "fastapi[standard]"

With pip, inside a virtual environment (a private folder of packages for one project, so projects do not interfere):

python -m venv .venv
source .venv/bin/activate        # Linux and macOS
pip install "fastapi[standard]"

On Windows the activation line is .venv\Scripts\activate in Command Prompt and .venv\Scripts\Activate.ps1 in PowerShell.

Two details trip up almost everyone, so read them slowly.

First, the quotes around fastapi[standard] are not optional on macOS. The default shell there, zsh, treats square brackets as a pattern to match file names. Without the quotes you get zsh: no matches found: fastapi[standard] and nothing installs. The quotes tell the shell to pass the text through untouched.

Second, what [standard] means. It is an extra: a named bundle of optional packages. The plain fastapi package is minimal. fastapi[standard] adds the command-line tool (fastapi dev, fastapi run), the server that runs your app, and other defaults you will want. For learning, always install the standard extra. There is also fastapi[standard-no-fastapi-cloud-cli], the same without the command-line client for the official hosting product, for teams that do not want it.

To check that everything works:

fastapi --version

On a fresh install this prints the version of the command-line tool and of the cloud client. Those are separate numbers from the framework's own, so do not be alarmed that they look different from 0.142.2. To see the framework version, ask Python:

python -c "import fastapi; print(fastapi.__version__)"

If the shell says the fastapi command is not found, there are two usual causes. Either you installed plain fastapi without the standard extra, or your virtual environment is not active. Re-run the activation line and check that your prompt shows the environment name.

Python version: current FastAPI documentation uses modern syntax such as str | None, which needs a reasonably recent Python 3. If you are on an old interpreter, upgrade before you debug anything else. The documentation's Docker example uses Python 3.14, and that is what this guide was run on.

Your first API, step by step

Create a file named main.py in your project folder. This is the whole program:

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
async def root():
    return {"message": "hello"}

Read it line by line, because every part is a pattern you will reuse.

from fastapi import FastAPI brings in the class. app = FastAPI() creates the application object. The name app is a convention, and the tools look for it. @app.get("/") registers the function below as the handler for GET requests at the root path. The function returns a Python dictionary, and FastAPI converts it to JSON automatically. You never call a JSON function yourself.

Now run it:

fastapi dev

The command finds main.py in the current folder, imports app from it, and starts a development server. It prints something like this:

 Starting FastAPI in development mode

 Server started at http://127.0.0.1:8000
    Documentation at http://127.0.0.1:8000/docs

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started server process [67552]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

The exact decoration varies between versions, but three things are stable. The address is http://127.0.0.1:8000. The documentation lives at /docs. And "Application startup complete" tells you the app loaded without errors. Uvicorn is the server program that FastAPI starts for you.

fastapi dev runs in development mode, which watches your files and reloads automatically when you save a change. This is the reason to use it while learning: edit, save, retry, with no restart. It is not for production. For that there is a separate fastapi run, covered later.

Open a second terminal and call the API:

curl http://127.0.0.1:8000/

{"message":"hello"}

curl is a command-line tool that sends HTTP requests. You can also just paste the address into a browser. If neither works, check that the server terminal still shows the "Uvicorn running" line and that you used the same port.

Now the feature that makes FastAPI popular. Open http://127.0.0.1:8000/docs in a browser. You will see an interactive page listing your endpoint. Click it, press Try it out, press Execute, and the page sends a real request and shows you the response. You wrote no documentation. FastAPI built it from your code and its OpenAPI schema, a standard machine-readable description of an API, which you can see raw at /openapi.json. Whenever you add or change an endpoint, this page changes with it, and it can never drift out of date because it is generated.

If fastapi dev complains that it cannot find the app, check three things: that your file is called main.py, that you are in the same folder, and that the variable is called app. You can also pass the path explicitly with fastapi dev main.py.

Inputs: path, query and body parameters

A prediction API takes inputs. FastAPI decides where an input comes from by looking at how you declare the function argument, so this section is really about reading your own function signatures.

Query parameters. Any simple-typed argument that is not in the path becomes a query parameter:

@app.get("/predict")
def predict(x: float, scale: float = 1.0):
    return {"result": x * 42 * scale}

Here x has no default, so it is required. scale has a default, so it is optional. A call to /predict?x=2 returns {"result":84.0}. A call to /predict?x=2&scale=0.5 returns 42.0. The type hint float does real work: FastAPI converts the text "2" to the number 2.0 before your function runs.

Notice the word def rather than async def. We will explain the difference shortly. For now, this is the safe choice for code that calls a model.

Path parameters. A variable piece of the path is declared in curly braces and matched by name:

@app.get("/models/{model_name}")
def get_model(model_name: str):
    return {"model": model_name}

A request to /models/churn calls the function with model_name="churn". Use path parameters to identify which thing, such as which model or which record, and query parameters for settings.

Body parameters. When you declare an argument whose type is a Pydantic model, FastAPI reads it from the JSON body. We build one next section.

What happens when a caller gets it wrong? This is the best part. Call /predict with no x:

{"detail":[{"type":"missing","loc":["query","x"],"msg":"Field required","input":null}]}

with HTTP status 422. And call /predict?x=abc:

{"detail":[{"type":"float_parsing","loc":["query","x"],"msg":"Input should be a valid number, unable to parse string as a number","input":"abc"}]}

Read these as a trained eye does. detail is always a list, because several things can be wrong at once. Each entry has a type (a machine-readable category), a loc (where the problem is: first the source such as query or body, then the field name), a msg (a human sentence), and the input that was received. Entries for rules with limits also carry a ctx with the limit values. You wrote no error handling for any of this. Your function never ran.

Parameters can carry extra rules through Query(), Path() and the like, using the Annotated type form that the current documentation teaches:

from typing import Annotated
from fastapi import Query

@app.get("/search")
def search(q: Annotated[str, Query(min_length=3, max_length=50)]):
    return {"q": q}

Annotated[type, extra] means "this type, with this extra information attached". Older tutorials write q: str = Query(min_length=3) instead. That still appears around the internet, but the documentation now prefers Annotated, so learn it first.

Request bodies and Pydantic models

Real model inputs are rarely one number. A house has an area, a number of rooms, a district. These belong in a JSON body, described by a Pydantic model.

from pydantic import BaseModel, Field


class House(BaseModel):
    area_m2: float = Field(gt=0, description="Living area in square metres")
    rooms: int = Field(ge=1, le=20)
    district: str
    has_parking: bool = False


class Prediction(BaseModel):
    price: float
    model_version: str


@app.post("/predict-house")
def predict_house(house: House) -> Prediction:
    price = 1200 * house.area_m2 + 5000 * house.rooms
    return Prediction(price=price, model_version="demo-1")

A model is a class that inherits from BaseModel, with one annotated attribute per field. area_m2: float says the field is a float and is required. has_parking: bool = False says it is optional with a default. Field(gt=0) adds a rule: greater than zero. ge means greater than or equal and le means less than or equal. The description shows up in the generated docs.

Send a request:

curl -X POST http://127.0.0.1:8000/predict-house \
  -H "Content-Type: application/json" \
  -d '{"area_m2": 120, "rooms": 3, "district": "Maadi"}'

{"price":159000.0,"model_version":"demo-1"}

The -X POST sets the method, -H sets a header telling the server the body is JSON, and -d supplies the body. Because this command has several parts, a mistake in any one produces a confusing error, so type it carefully.

The -> Prediction return annotation does two jobs. It tells FastAPI what shape the response has, so the docs show it, and it makes FastAPI validate and filter the output. If your function accidentally returns extra fields, such as an internal ID or a database password hash, the ones not in Prediction are dropped. This is a cheap safeguard, and it is a good habit to always declare what you return. You may also pass response_model=Prediction in the decorator, which does the same thing and is what you will see in older code.

Now send bad data:

curl -X POST http://127.0.0.1:8000/predict-house \
  -H "Content-Type: application/json" \
  -d '{"area_m2": -5, "rooms": 3, "district": "Maadi"}'

{"detail":[{"type":"greater_than","loc":["body","area_m2"],"msg":"Input should be greater than 0","input":-5,"ctx":{"gt":0.0}}]}

Status 422 again, with loc pointing into the body. A body that is not valid JSON at all, or not an object, is rejected the same way, with a message such as "Input should be a valid dictionary or object to extract fields from".

Three habits make models safer.

1. Put the rules in the model, not the function. A rule written once as Field(gt=0) is enforced for every caller and shown in the docs. A rule buried in an if statement is hidden and easy to forget.
2. Use Pydantic v2 names. FastAPI moved to Pydantic v2 in version 0.100.0. To turn a model into a dictionary use house.model_dump(), and for a JSON string house.model_dump_json(). Older code that calls .dict() or .json() is v1 style and shows deprecation warnings.
3. Optional means | None. Write description: str | None = None for a field that may be absent. The | None is the type; the = None is the default. You need both.

The 400-range errors that you raise yourself use HTTPException, covered in the section on errors below.

Serving a real model: load once with lifespan

This is the section that matters most for machine learning, and the place beginners most often go wrong.

A model must be loaded from disk into memory before it can predict. Loading is slow, often seconds, and for a large model, minutes. If you load it inside the endpoint function, you pay that cost on every request, and a hundred simultaneous callers load a hundred copies. The correct pattern is to load once when the server starts, keep it in memory, and have every request reuse it.

FastAPI gives you a hook for this called lifespan: an asynchronous context manager passed to the application. Code before the yield runs at startup. Code after it runs at shutdown. This is the official documentation's own example, which uses a stand-in function in place of a real model:

from contextlib import asynccontextmanager

from fastapi import FastAPI


def fake_answer_to_everything_ml_model(x: float):
    return x * 42


ml_models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load the ML model
    ml_models["answer_to_everything"] = fake_answer_to_everything_ml_model
    yield
    # Clean up the ML models and release the resources
    ml_models.clear()


app = FastAPI(lifespan=lifespan)


@app.get("/predict")
async def predict(x: float):
    return {"result": ml_models["answer_to_everything"](x)}

Walk through what happens. When the server starts, FastAPI runs lifespan up to the yield. The model goes into the ml_models dictionary. The server then begins accepting requests, and each call to /predict looks the model up in the dictionary: no loading, no waiting. When the server stops, the code after yield runs, which clears the dictionary and lets you close files or connections.

To use a real scikit-learn model that you saved earlier with joblib, replace the fake line:

import joblib

@asynccontextmanager
async def lifespan(app: FastAPI):
    ml_models["house_price"] = joblib.load("models/house_price.joblib")
    yield
    ml_models.clear()


@app.post("/predict-house")
def predict_house(house: House) -> Prediction:
    model = ml_models["house_price"]
    features = [[house.area_m2, house.rooms]]
    price = float(model.predict(features)[0])
    return Prediction(price=price, model_version="2026-10-01")

Two small details deserve attention. The call is wrapped in float(...) because many ML libraries return NumPy numbers, which are not plain Python types, and the response validation can reject or mishandle them. And the features are a list of lists, because scikit-learn expects a batch of rows even when you predict one.

Why a dictionary at module level? It is the simplest thing that works: a place both the lifespan function and the endpoints can reach. In the next sections we see a cleaner way to hand the model to endpoints, and a later guide covers application state.

Watch your startup log. A failing model load now fails at startup, loudly, before any request arrives. That is a feature: a server that starts successfully is a server that has its model. If the file is missing you will see the FileNotFoundError in the terminal and the app will not start, instead of the first user receiving a 500.

A deprecation to know about: older tutorials load models with @app.on_event("startup") and @app.on_event("shutdown"). Those are deprecated in favour of lifespan. Worse, if you pass lifespan= to FastAPI(...), any on_event handlers are silently not called. If you copy old code into a lifespan-based app and your model dictionary is empty, this is the reason.

def or async def: the question every beginner asks

You have now seen both def and async def on endpoints, and the documentation is relaxed about which to use. For model serving the choice has a real consequence, so here is the rule in full.

async def defines a coroutine: a function that can pause while it waits for something slow, such as a network call or a database reply, and let the server work on other requests in the meantime. The pause points are marked with await. The server's single worker handles many requests by switching between them whenever one is waiting.

Plain def defines an ordinary blocking function. FastAPI notices this and runs it in a threadpool: a set of background threads, so the main loop is not held up.

The official rules are short:

• Use async def when the library you call is designed to be awaited, so you write await something().
• Use plain def when the library is not awaitable. Most database drivers and most machine learning libraries are not.
• When unsure, use def.

Now the trap. Calling model.predict(...) is blocking, CPU-heavy work. If you put it inside an async def endpoint without awaiting anything, the server's event loop is occupied for the entire time the prediction runs. Every other request, even a trivial health check, waits. One slow prediction freezes the whole server.

The async def example in the official lifespan snippet is fine because its fake model returns instantly. A real model does not. When you replace the fake with a real one, change the endpoint to def.

A threadpool is not a cure for slow models, though. Python threads share one process, and heavy computation in pure Python code is still limited by a lock called the Global Interpreter Lock. Many numerical libraries release that lock while they compute, which helps, but it is not guaranteed. The documentation's advice for CPU-bound work such as inference is to use multiprocessing. In practice that means running several worker processes or several copies of the service, which a later section covers. For a beginner: use def, measure, and scale out when one process is not enough.

Dependencies: handing things to endpoints

You now have a model in a module-level dictionary and endpoints that reach into it. That works, but it couples every endpoint to a global, and it makes tests awkward. FastAPI's answer is dependency injection: instead of reaching out for what you need, you declare it as a parameter, and FastAPI supplies it.

A dependency is any function. You attach it to an endpoint argument with Depends:

from typing import Annotated

from fastapi import Depends


def get_model():
    return ml_models["house_price"]


Model = Annotated[object, Depends(get_model)]


@app.post("/predict-house")
def predict_house(house: House, model: Model) -> Prediction:
    price = float(model.predict([[house.area_m2, house.rooms]])[0])
    return Prediction(price=price, model_version="2026-10-01")

Read Depends(get_model) as "before running this endpoint, call get_model and pass me whatever it returns as model". Pass the function itself, without parentheses: writing Depends(get_model()) calls it too early and is a classic slip.

Model = Annotated[object, Depends(get_model)] is a type alias: a short name for a long annotation, so every endpoint can write model: Model instead of repeating the whole thing. It is the pattern the documentation uses for shared dependencies.

Why bother? Three reasons that you will appreciate as the project grows.

1. One place to change. If you later load the model from a registry instead of a file, or add a check that the model is ready, you edit get_model and no endpoint changes.
2. Easy testing. Tests can replace get_model with a fake, so they do not need the real model file. That is the subject of the testing section.
3. Sharing logic. Anything several endpoints need, such as checking an API key, parsing pagination parameters, or opening a database session, can be a dependency.

Dependencies can have their own dependencies, forming a small tree, and FastAPI resolves it for you. They can be def or async def in any mix. A dependency can also take query or header parameters, which then appear in the documentation for every endpoint that uses it.

An example that shows the sharing idea is a simple API-key check:

from fastapi import Header, HTTPException


def require_api_key(x_api_key: Annotated[str, Header()]):
    if x_api_key != "change-me":
        raise HTTPException(status_code=401, detail="Invalid API key")


@app.post("/predict-house", dependencies=[Depends(require_api_key)])
def predict_house(house: House, model: Model) -> Prediction:
    ...

The dependencies=[...] list in the decorator is for dependencies that must run but whose return value you do not need. Two things to note. FastAPI converts the argument name x_api_key into the header name x-api-key for you, replacing underscores with hyphens. And the hard-coded key here is only for learning. Real keys come from environment variables or a secret manager, never from source code.

Errors: raising them and reading them

An API that only handles good input is half an API. You will deal with errors in two directions: the ones you raise on purpose, and the ones that reach you uninvited.

Raising errors on purpose. Use HTTPException:

from fastapi import HTTPException


@app.get("/models/{model_name}")
def get_model_info(model_name: str):
    if model_name not in ml_models:
        raise HTTPException(status_code=404, detail="Model not found")
    return {"model": model_name}

raise stops the function immediately, and FastAPI converts the exception into a response with the status you chose and a body of {"detail": "Model not found"}. Pick the status code that matches the situation: 404 when a thing does not exist, 400 for a request that is well-formed but unacceptable (for example a combination of values your model cannot handle), 409 for a conflict such as creating something that already exists.

Reserve error responses for the caller's mistakes. If your code has a bug, you will get a 500 automatically, and you should fix the bug rather than catching it. A common beginner mistake is wrapping everything in try/except and returning 200 with an "error" field. That hides failures from monitoring tools and from callers, which rely on the status code.

Reading errors that arrive. Here is a short field guide, based on what you will actually see.

The most important line is the 500 row. The browser or curl only receives the words "Internal Server Error". The reason is in the terminal where the server runs. Beginners stare at the client and miss the traceback printed a few inches away. The last line of a traceback names the exception, and the lines above it point to your file and line number.

Validation also covers responses. If you declared -> Prediction and your function returns something that cannot be turned into a Prediction, FastAPI raises an error on the server and the caller gets a 500, with the details in the server log. That is a bug in your code, not in the request.

You can also control how a Pydantic model treats unexpected extra fields. By default, extra fields in the body are ignored. Whether to forbid them is a design choice: forbidding catches typos in field names (such as rooom), at the cost of being strict with clients who send harmless extras.

Testing without starting a server

Testing an API by hand with curl stops scaling around the fifth endpoint. FastAPI ships a test client that calls your app directly in Python, with no server and no network.

Install the two packages you need (the test client uses httpx, an HTTP library):

uv add httpx pytest
# or: pip install httpx pytest

Create test_main.py next to main.py:

from fastapi.testclient import TestClient

from main import app

client = TestClient(app)


def test_predict_house_rejects_negative_area():
    response = client.post(
        "/predict-house",
        json={"area_m2": -5, "rooms": 3, "district": "Maadi"},
    )
    assert response.status_code == 422
    assert response.json()["detail"][0]["loc"] == ["body", "area_m2"]

Run it:

pytest

Note that the test function is a plain def, not async, and you do not use await. The client handles that. client.post(...) takes a json= argument and does the encoding and the header for you, which is less error-prone than the long curl command. The response object offers status_code and .json().

One complication specific to this guide's pattern. A bare TestClient(app) does not run the lifespan code, so the model never loads and ml_models["house_price"] raises a KeyError. To run startup and shutdown, use the client as a context manager:

def test_predict_house_works():
    with TestClient(app) as client:
        response = client.post(
            "/predict-house",
            json={"area_m2": 120, "rooms": 3, "district": "Maadi"},
        )
    assert response.status_code == 200
    assert "price" in response.json()

Inside the with block, the lifespan runs on entry and its shutdown half runs on exit.

This is where the dependency from the previous section pays off. The test above needs the real model file. A faster test replaces the dependency with a stub using app.dependency_overrides, a dictionary mapping a dependency function to a replacement:

class StubModel:
    def predict(self, rows):
        return [100.0]


def test_predict_with_stub_model():
    app.dependency_overrides[get_model] = lambda: StubModel()
    try:
        client = TestClient(app)
        response = client.post(
            "/predict-house",
            json={"area_m2": 120, "rooms": 3, "district": "Maadi"},
        )
        assert response.json()["price"] == 100.0
    finally:
        app.dependency_overrides.clear()

Here from main import get_model is also needed at the top. The try/finally guarantees the override is removed even if the assertion fails, so one test cannot leak into the next. Stub tests check your API logic (validation, error codes, response shape) in milliseconds, while a few tests with the real model check that the model loads and predicts. You want both kinds.

Depending on your versions of Starlette and httpx, you may see a deprecation warning about httpx when the test client is imported. In our run with FastAPI 0.142.2 the warning appeared and the tests still passed. A warning is not a failure, but read the warning text when you upgrade, because the underlying test dependency may change.

Running it properly: dev, run, and configuration

So far you have used fastapi dev. It is for development only. When the API runs for other people, use the production command:

fastapi run

fastapi run starts the same app without reload and bound for real use. You can point it at a file and set the port:

fastapi run app/main.py --port 80

It also accepts --workers to start several worker processes and --proxy-headers, used when the app sits behind a proxy that terminates HTTPS. Why would you want several workers? One Python process runs one thing at a time for CPU-heavy work, so on a server with several cores you start several processes. The cost is memory: each worker loads its own copy of the model. Four workers with a 2 GB model use roughly 8 GB. Check your memory before you raise the count.

In a managed environment such as Kubernetes (covered in the Kubernetes guide), the usual practice is the opposite: one process per container, and let the platform run several containers. You do not need that decision yet, only to know that it exists.

Configuration. Your API will need settings: the path to a model file, an API key, a log level. Do not write them into the code. Read them from environment variables, which are named values the operating system gives to a program:

import os

MODEL_PATH = os.environ.get("MODEL_PATH", "models/house_price.joblib")

MODEL_PATH=/data/models/v2.joblib fastapi run

The first argument is the variable name and the second is a default for local development. The same code then works on your laptop, in CI and in production, with only the environment changing. FastAPI's documentation has a page on settings and environment variables built around Pydantic's settings support, which gives you typed, validated settings. It is worth reading once your configuration has more than three values.

Secrets such as API keys never go in the code or the repository. Pass them through the environment or a secret manager. Once a key has been committed to a public repository, treat it as leaked, even after you delete the commit.

A security note that is easy to forget. The interactive docs at /docs are served by default, and they list every endpoint. That is wonderful in development. For a public service, decide deliberately whether to expose them. You can turn them off when creating the app by passing docs_url=None, redoc_url=None and openapi_url=None to FastAPI(...). The parameter names are in the documentation's metadata and URLs page, and this is a mid-level topic, but know that the choice exists.

Finally, a quick word on networking. The address 127.0.0.1 accepts connections only from your own machine. Inside a container you must listen on all interfaces so that the outside world can reach the port. fastapi run handles this for you, which is another reason to use it rather than a raw server command inside a Dockerfile.

Packaging it in a container

Your API works on your machine. To hand it to a colleague, or to run it on a server, the standard tool is a container image: your code, Python and every dependency in one portable package. If containers are new to you, read the Docker guide first, then return here.

First, list your dependencies in requirements.txt. Pin the framework to a minor version, because FastAPI is still 0.x and minor releases may include breaking changes:

fastapi[standard]>=0.142.0,<0.143.0
scikit-learn
joblib

(Pin your ML libraries to the exact versions you trained with. A model saved with one scikit-learn version may not load in another.)

Then a Dockerfile, following the shape in the official deployment documentation:

FROM python:3.14

WORKDIR /code

COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

COPY ./app /code/app

CMD ["fastapi", "run", "app/main.py", "--port", "80"]

Three decisions in these few lines are deliberate.

The requirements.txt is copied and installed before the application code. Docker caches each step, and dependencies change rarely while code changes constantly. With this order, editing your code rebuilds only the last copy step, not the slow install.

The CMD uses the exec form, a JSON list of strings, and not the shell form (CMD fastapi run ... as a bare string). With the shell form, the signals Docker sends to stop the container do not reach your app, which breaks graceful shutdown, so your lifespan cleanup does not run, and in Docker Compose it causes a delay of about ten seconds on every stop. The documentation warns against it explicitly.

And the base image is python. You may find older tutorials that use a tiangolo/uvicorn-gunicorn-fastapi image. That image is deprecated, so build your own as above.

Your project layout for this Dockerfile looks like this:

.
├── Dockerfile
├── requirements.txt
└── app
    ├── main.py
    └── models
        └── house_price.joblib

Build and run it:

docker build -t house-api .
docker run -p 8000:80 house-api

The -p 8000:80 flag maps port 8000 on your machine to port 80 inside the container, which is where the CMD told the app to listen. Then http://127.0.0.1:8000/docs works exactly as before. If the container exits immediately, run docker logs on it and read the last lines: a model path that is wrong inside the container is the usual culprit, because relative paths are resolved from the working directory, /code.

Model files raise a question: should the model be inside the image or fetched at startup? For a small model, copying it in keeps the image self-contained and the deployment simple. For a large one, fetching from storage at startup keeps the image small but makes startup depend on the network. Both are valid, and you will meet this trade-off again in the mid-level guide.

For regional readers: if your data may not leave the country, choose the cloud region your container runs in deliberately, and prefer a region inside the Gulf or Egypt where your provider offers one. The image is portable, but the data it processes is not free to roam.

Putting it all together

Here is one small project that uses everything: a house-price service with validation, a model loaded once, a dependency, an error, and a test. Create the folder house-api with this layout:

house-api/
├── main.py
├── test_main.py
├── train.py
└── models/

First, train.py creates the model file. It uses scikit-learn, which you install with pip install scikit-learn:

import joblib
from sklearn.linear_model import LinearRegression

X = [[50, 1], [80, 2], [120, 3], [200, 5]]
y = [60000, 100000, 159000, 270000]

model = LinearRegression().fit(X, y)
joblib.dump(model, "models/house_price.joblib")
print("saved")

Run mkdir models && python train.py. Now the application:

import os
from contextlib import asynccontextmanager
from typing import Annotated

import joblib
from fastapi import Depends, FastAPI, HTTPException
from pydantic import BaseModel, Field

MODEL_PATH = os.environ.get("MODEL_PATH", "models/house_price.joblib")
ml_models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    ml_models["house_price"] = joblib.load(MODEL_PATH)
    yield
    ml_models.clear()


app = FastAPI(title="House price API", lifespan=lifespan)


class House(BaseModel):
    area_m2: float = Field(gt=0, description="Living area in square metres")
    rooms: int = Field(ge=1, le=20)
    district: str


class Prediction(BaseModel):
    price: float


def get_model():
    return ml_models["house_price"]


Model = Annotated[object, Depends(get_model)]


@app.get("/health")
def health():
    return {"status": "ok", "model_loaded": "house_price" in ml_models}


@app.post("/predict-house")
def predict_house(house: House, model: Model) -> Prediction:
    if house.area_m2 > 2000:
        raise HTTPException(status_code=400, detail="Area outside the trained range")
    price = float(model.predict([[house.area_m2, house.rooms]])[0])
    return Prediction(price=price)

Note the /health endpoint. It costs four lines and every deployment platform you will meet uses something like it to ask "is this service alive?". The 400 check shows a business rule that Pydantic's simple limits cannot express, for example refusing inputs far outside what the model saw in training. A model asked about a 50,000 square metre house will happily return a number, and the number will be wrong.

Now the tests:

from fastapi.testclient import TestClient

from main import app


def test_health():
    with TestClient(app) as client:
        assert client.get("/health").json()["model_loaded"] is True


def test_prediction_is_a_number():
    with TestClient(app) as client:
        r = client.post("/predict-house", json={"area_m2": 100, "rooms": 3, "district": "Maadi"})
    assert r.status_code == 200
    assert isinstance(r.json()["price"], float)


def test_validation_rejects_bad_rooms():
    with TestClient(app) as client:
        r = client.post("/predict-house", json={"area_m2": 100, "rooms": 0, "district": "Maadi"})
    assert r.status_code == 422


def test_out_of_range_area_is_400():
    with TestClient(app) as client:
        r = client.post("/predict-house", json={"area_m2": 5000, "rooms": 3, "district": "Maadi"})
    assert r.status_code == 400

Run pytest, then fastapi dev, open /docs, and try the endpoint by hand. You have built a validated, tested, documented model API in about eighty lines.

What you can now do, and what comes next

You can now explain what an HTTP API is and read status codes by their first digit. You can install FastAPI with the right extra and quote it correctly on a Mac. You can write path operations that take query, path and body parameters, describe inputs with Pydantic models, and let the framework reject bad requests with a precise 422. You can load a model once in lifespan, hand it to endpoints through a dependency, and choose between def and async def with a reason. You can raise your own errors, read a 500 by finding the traceback in the server terminal, test with TestClient including startup code, replace the model with a stub, and package the whole thing in a container with an exec-form CMD.

Three cautions to carry forward. FastAPI is a 0.x library, so pin the minor version and read the release notes' breaking-changes entries before upgrading; version 0.137.0, for example, changed how routers include other routers, which affects code that inspects router.routes. Prefer the current forms: Annotated over default-value Depends, lifespan over on_event, Pydantic v2 names such as model_dump. And remember that FastAPI makes the web layer easy but does not make your model fast or safe on its own. Slow predictions need more processes or a dedicated serving system, and personal data needs the same care it had before it reached an API.

What comes next. The mid-level guide covers how requests are actually processed, routers for organising larger apps, background tasks, middleware, CORS, settings, structured logging, proper async patterns, and running several workers or replicas. The senior guide covers the platform view: scaling, security, observability including the native OpenTelemetry support added in 0.142.0, upgrades and when to choose something else.

If you want to go sideways instead, these guides are natural neighbours. MLflow tracks the experiments and models that you would load into this API. BentoML and Triton are dedicated model-serving tools you can compare with a hand-written FastAPI service, and KServe runs model servers on Kubernetes. Knowing where a plain FastAPI service is enough, and where a specialised server is better, is one of the clearest signs of experience.

Sources

• FastAPI home and documentation: https://fastapi.tiangolo.com/
• Tutorial index (installation, first steps, running): https://fastapi.tiangolo.com/tutorial/
• Release notes: https://fastapi.tiangolo.com/release-notes/
• Lifespan events and the machine learning model example: https://fastapi.tiangolo.com/advanced/events/
• Concurrency and async/await (def versus async def): https://fastapi.tiangolo.com/async/
• Dependencies: https://fastapi.tiangolo.com/tutorial/dependencies/
• Testing: https://fastapi.tiangolo.com/tutorial/testing/
• Deployment overview: https://fastapi.tiangolo.com/deployment/
• Deploy with Docker: https://fastapi.tiangolo.com/deployment/docker/
• About FastAPI versions: https://fastapi.tiangolo.com/deployment/versions/
• FastAPI 0.137.0 release notes on GitHub: https://github.com/fastapi/fastapi/releases/tag/0.137.0

This is the interview companion to the beginner guide. It is written for the first conversations about FastAPI: a screening call, a junior ML engineer interview, or the "how would you serve this model?" question that ends up in almost every MLOps loop. Read the detailed guide first. Here we focus on what interviewers actually ask, how to answer in your own words, and where candidates lose points.

The guide was checked against FastAPI 0.142.2 (30 September 2026).

The thirty-second answer

FastAPI is a Python web framework for building HTTP APIs. You write ordinary functions with type hints, and it validates incoming requests, converts them to Python values, serialises your responses to JSON, and generates interactive documentation automatically. It is built on Starlette for the web layer and Pydantic for validation, and for machine learning it is commonly used to put a trained model behind a prediction endpoint.

If you are asked "what is it?" and have only a breath, say that. Then add one sentence that shows you have used it: "The part I care most about for ML is that bad input is rejected with a 422 before my model code runs, and that I load the model once at startup instead of per request."

The model to carry in your head

Interviewers rarely want a list of features. They want to see that you hold a small, correct model and can reason from it. Carry these four nouns.

A path operation is a function bound to an HTTP method and a path by a decorator such as @app.get("/predict"). Parameters are what the request brings in, and FastAPI decides where each comes from by reading your function signature: path parameters from the URL, query parameters from the part after the question mark, and a Pydantic model argument from the JSON body. A Pydantic model is a class describing the shape and rules of data; it validates and converts what arrives. A dependency is a function whose result FastAPI passes into your endpoint, declared with Depends, used for shared things such as the loaded model or an authentication check.

The request lifecycle follows from that. The path and method are matched first, so a wrong path gives 404 and a wrong method gives 405. Parameters are then validated, so bad input gives 422 and your function never runs. Dependencies are resolved. Only then does your function execute, and a crash inside it gives 500. When an interviewer gives you a symptom, name the stage it failed at. That single habit makes you sound senior.

Commands and snippets to have ready

You are not expected to recite documentation, but a few things should come out of your fingers without hesitation.

Install, with the quotes:

pip install "fastapi[standard]"

Run in development, with auto-reload, and in production mode:

fastapi dev
fastapi run app/main.py --port 80

The minimal ML endpoint with the model loaded once through lifespan. If you can write this on a whiteboard, you have passed the practical core of most beginner interviews:

from contextlib import asynccontextmanager
from fastapi import FastAPI

ml_models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    ml_models["m"] = load_model()   # your loader
    yield
    ml_models.clear()


app = FastAPI(lifespan=lifespan)


@app.get("/predict")
def predict(x: float):
    return {"result": ml_models["m"](x)}

A typed body and a raised error:

from fastapi import HTTPException
from pydantic import BaseModel, Field


class Features(BaseModel):
    area: float = Field(gt=0)
    rooms: int = Field(ge=1)


@app.post("/predict")
def predict(f: Features) -> dict:
    if f.area > 2000:
        raise HTTPException(status_code=400, detail="Out of range")
    return {"price": 1.0}

A test that sends bad input:

from fastapi.testclient import TestClient
from main import app


def test_bad_area():
    with TestClient(app) as client:
        r = client.post("/predict", json={"area": -1, "rooms": 2})
    assert r.status_code == 422

And the container command, in exec form:

CMD ["fastapi", "run", "app/main.py", "--port", "80"]

The classic traps and why they happen

These are the things interviewers probe because they separate people who have run an API from people who have read about one.

Loading the model per request. The model is read from disk inside the endpoint. It works with one request and collapses under load, because loading is slow and every concurrent request loads its own copy. The fix is lifespan: load once at startup.

A blocking model call inside async def. async def endpoints run on the event loop. A CPU-heavy model.predict with nothing to await holds the loop, so every other request waits, including health checks. The fix is plain def, which FastAPI runs in a threadpool, and scaling out with processes for real load.

Shell-form CMD in a Dockerfile. Without the JSON-list form, termination signals do not reach the app, so shutdown is not graceful and lifespan cleanup does not run. It also causes a delay of about ten seconds when stopping under Docker Compose.

on_event combined with lifespan. The old @app.on_event("startup") is deprecated, and when you pass lifespan the on_event handlers are silently not called. The model dictionary stays empty and the first request raises KeyError.

Workers multiply memory. Each worker process loads its own copy of the model. Four workers and a 2 GB model is about 8 GB. Candidates who say "just add workers" without saying this lose points.

Unquoted extras on zsh. pip install fastapi[standard] fails on macOS with zsh: no matches found. It is a shell behaviour, not a FastAPI bug.

Assuming 1.0. FastAPI is still a 0.x library. Minor versions can contain breaking changes, so you pin the minor version and test before upgrading. Version 0.137.0, for example, changed how include_router works, and code that walked router.routes expecting a flat list broke.

Common interview questions

What is FastAPI and why would you use it for ML?

FastAPI is a Python framework for HTTP APIs built on Starlette and Pydantic. For ML it gives you a thin, typed layer over a model: input validation from Pydantic models, automatic OpenAPI documentation at /docs, and a clean way to load the model once with lifespan. I would use it when I need a custom prediction service or when the model needs surrounding logic such as feature lookup, business rules or authentication. If I only needed to serve a standard model format at scale, I would compare it with a dedicated serving tool.

How does FastAPI validate input?

It reads your function signature. Type hints on simple arguments define query or path parameters, and a Pydantic model argument defines the JSON body. Before your function is called, FastAPI parses and validates everything. If something is wrong it returns a 422 with a detail list, where each entry has a type, a loc showing where the problem is (for example ["body", "area_m2"]), a msg and the received input. Your function never runs on invalid data.

What is the difference between 400, 404, 422 and 500?

422 is FastAPI's own validation failure: the request did not match the declared types or rules. 400 is something I raise on purpose through HTTPException when a request is well-formed but unacceptable, for example a value outside the range my model was trained on. 404 means the path or resource does not exist. 500 means my code raised an exception, and the traceback is in the server log, not in the client's response. The first digit tells me whose problem it is: 4xx the caller, 5xx the server.

Where do you load the model, and why?

In the lifespan context manager passed to FastAPI(lifespan=...). Code before the yield runs at startup and loads the model into memory, and code after it releases resources at shutdown. That way the cost is paid once, the server does not accept traffic until the model is ready, and a failed load stops the app from starting instead of producing 500s on the first request. Endpoints receive the model through a dependency or by reading from the shared dictionary.

def or async def for a prediction endpoint?

Plain def. The rule in the documentation is to use async def only when the libraries you call are awaitable, and def otherwise, and when unsure use def. Inference is blocking and CPU-bound, so inside async def it would block the event loop for all requests. With def, FastAPI runs the function in a threadpool. For real concurrency on CPU-bound inference, the documentation points to multiprocessing, which in practice means several worker processes or several replicas.

What is dependency injection in FastAPI, and why does it matter?

A dependency is a function that FastAPI calls before the endpoint and whose return value it passes in, declared as Annotated[Type, Depends(fn)]. I pass the function without calling it. It matters for three reasons: shared logic lives in one place, such as returning the loaded model or checking an API key; dependencies can have sub-dependencies; and tests can swap a dependency for a stub with app.dependency_overrides, so API tests do not need the real model.

How do you test a FastAPI app?

With TestClient from fastapi.testclient, which needs httpx installed, and pytest. Tests are plain def functions with no await. To run the lifespan startup, so the model actually loads, I use the client as a context manager: with TestClient(app) as client. I test validation failures first because they are cheap and important, and I use dependency_overrides with a stub model for fast logic tests, plus a few tests against the real model.

How do you run and package it?

In development I use fastapi dev, which reloads on file changes. For deployment I use fastapi run, in a container built from a Python base image: copy requirements.txt first and install, then copy the code, and use an exec-form CMD such as ["fastapi", "run", "app/main.py", "--port", "80"]. I do not use the old tiangolo/uvicorn-gunicorn-fastapi image because the documentation deprecates it.

What are Annotated and Depends, and why do the docs prefer them?

Annotated[type, extra] attaches metadata to a type, for example Annotated[str, Query(min_length=3)] or Annotated[Model, Depends(get_model)]. The current documentation teaches this form instead of putting Query() or Depends() as a default value. It keeps the default for real defaults, makes type aliases possible, and plays well with type checkers.

What changed in Pydantic v2 that you must know?

FastAPI moved to Pydantic v2 at 0.100.0. The method names changed: model_dump() replaces .dict(), and model_dump_json() replaces .json(). Configuration moved from an inner class Config to model_config. Old tutorials still show the v1 forms, so I check which version a snippet assumes before pasting it.

Is FastAPI production-ready, and what would you add first?

Yes, it is widely used in production, but the framework alone is not a production system. First I would add a health endpoint, settings from environment variables, structured logging and request tracking, authentication suited to the callers, and limits at the proxy for request size. For HTTPS I would terminate at a proxy or load balancer and run the app with --proxy-headers. I would pin the FastAPI minor version, because it is still 0.x.

How do you scale a FastAPI model service?

I measure first. If the host runs the service directly, more worker processes with --workers use more cores, but memory grows with each worker because each loads the model. On Kubernetes or a similar platform the recommended pattern is one process per container and more replicas behind a load balancer, with memory requests and limits set from measured use. For heavy models, a dedicated serving runtime behind or instead of FastAPI may serve better, which is a judgement call to explain, not a rule.

Why might a service be slow even when the endpoint code is simple?

The usual suspects are: a blocking call inside async def stalling the event loop; the model reloaded on each request; too few worker processes for a CPU-bound model; very large input validation or serialisation; and memory pressure causing swapping when too many workers each hold a copy of the model. I would reproduce the problem, check the server logs and timing around the model call first, and only then change anything.

What a strong answer covers that a weak one misses

A weak answer lists features: "it is fast, it has docs, it uses type hints". A strong answer ties each feature to a failure it prevents. Validation prevents silent bad input. Lifespan prevents per-request model loading. A def endpoint prevents a frozen event loop. An exec-form CMD prevents ungraceful shutdown.

A weak answer says "FastAPI is faster than Flask" without qualification. A strong answer knows that the speed claim is about concurrency handling for waiting work, that CPU-bound inference is not made faster by the framework, and that the real gains are validation and documentation.

A weak answer gives the status code and stops. A strong answer says where to look: the 422 body for validation, the server terminal for a 500.

A weak answer pastes a tutorial from three years ago. A strong answer uses lifespan, Annotated, and model_dump, and says why the old forms are deprecated.

A weak answer says "it is stable". A strong answer says "it is a 0.x library, so I pin the minor version and read the release notes".

Sixty-second self-test

Answer these aloud, without scrolling up. If you hesitate on more than two, reread the matching section of the detailed guide.

1. Name the four nouns of FastAPI and say what each one does.
2. What status code does a missing required field produce, and where in the response do you find the field name?
3. Why do you load a model in lifespan and not inside the endpoint?
4. A teammate wrote async def predict with a scikit-learn call inside. What happens under load, and what is the smallest fix?
5. What does the quote in pip install "fastapi[standard]" protect against?
6. Why does TestClient(app) sometimes fail with a KeyError on the model, and how do you fix it?
7. Give two reasons to prefer an exec-form CMD.
8. You set four workers and the machine runs out of memory. Why?
9. A 500 appears in the client. Where do you look first?
10. Why does FastAPI being a 0.x library change how you manage upgrades?

Model answers in one line each: routes, parameters, models, dependencies; 422 and the loc field; once-only cost plus fail-at-startup; the event loop blocks, so switch to def; zsh would treat the brackets as a file pattern; lifespan does not run without the context manager form; graceful shutdown and no Compose delay; each worker holds its own model copy; the server terminal; minor versions can break, so pin and test.

Finally, a note on delivery. Interviewers forgive not knowing a detail and rarely forgive a confident wrong answer. If you are unsure whether something exists, say "I would check the documentation for the exact name, but the idea is..." and give the idea. For FastAPI, which changes quickly, that honesty reads as experience.

You may also be asked to compare FastAPI with other ways of serving models. A short, fair answer: FastAPI is a general web framework, so you write the serving logic yourself, which gives control and costs effort. Dedicated tools such as BentoML or Triton package model serving features such as batching and model management, and KServe runs servers on Kubernetes. The decision depends on the team's size, the model's cost and the required throughput, and saying so is better than naming a winner.

One last preparation tip. Before an interview, build the small house-price project from the detailed guide from memory, with no copy and paste. Run it, break it on purpose with a bad body and a wrong path, and read each error aloud. Candidates who have done this once speak about status codes, lifespan and dependencies with a calm, concrete confidence that no amount of reading produces.

This is the practical companion to the beginner guide. The detailed guide teaches how FastAPI works; this file is about what actually goes wrong in the first weeks and the habits that prevent it. Each mistake below follows the same shape: the symptom you will see, the cause underneath, and the fix. Skim the headings, and come back to a mistake the moment you meet it.

The guide was checked against FastAPI 0.142.2 (30 September 2026).

The mistakes beginners actually make

The install fails with "no matches found"

Symptom. You run pip install fastapi[standard] on a Mac and the shell prints zsh: no matches found: fastapi[standard]. Nothing is installed.

Cause. zsh, the default macOS shell, treats square brackets as a pattern for matching file names, finds no file called that, and refuses to run the command. FastAPI never sees it.

Fix. Quote the whole thing: pip install "fastapi[standard]". Quote it in every note and README you write, so the next person does not hit it.

The fastapi command does not exist

Symptom. fastapi dev says the command is not found, even though you installed FastAPI yesterday.

Cause. One of two things. Either you installed plain fastapi, which is the minimal package without the command-line tool, or your virtual environment is not active in this terminal, so the shell cannot see the packages you installed.

Fix. Check that your prompt shows the environment name, and activate it if not (source .venv/bin/activate on Linux and macOS). Then reinstall with the standard extra: pip install "fastapi[standard]". If you use uv, uv run fastapi dev runs the command inside the project environment without activation.

Editing the code and nothing changes

Symptom. You change the response text, refresh, and see the old answer.

Cause. You started the server with fastapi run, which does not reload, or a second old server is still running on the same port and is the one answering you.

Fix. Use fastapi dev while developing, since it reloads when you save. If it still seems stale, stop everything with Ctrl+C, check that no other terminal is running a server, and start again. The log line when a reload happens is your proof that the change was picked up.

"Address already in use"

Symptom. The server refuses to start and complains the address is already in use.

Cause. Another process, often your own previous run left in a forgotten terminal, owns port 8000.

Fix. Find and stop the old server, or start the new one on another port: fastapi dev --port 8001. Do not keep stacking servers on different ports without closing the old ones, because later you will test against the wrong one.

Loading the model inside the endpoint

Symptom. The first prediction takes several seconds, and every one after is just as slow. Under two or three simultaneous callers the memory climbs and the service crashes.

Cause. The endpoint calls joblib.load(...) or builds the model on every request. Each request pays the loading cost and holds its own copy.

Fix. Load once in lifespan and keep the model in a dictionary or behind a dependency. Code before the yield runs at startup. The side benefit is that a missing model file stops the server from starting, which is a much better failure than an error in front of a user.

A model call inside async def freezes everything

Symptom. While one prediction runs, other requests, including trivial ones like /health, hang until it finishes.

Cause. async def endpoints run on the event loop. A blocking model.predict with no await holds the loop for its full duration, and nothing else can be scheduled.

Fix. Write the endpoint as plain def. FastAPI runs it in a threadpool, so the loop stays free. The documentation's rule is simple: async def only when you call awaitable libraries, def otherwise, and when unsure, def. For real parallelism on CPU-bound inference, plan on several processes or replicas.

Old on_event code that never runs

Symptom. You copied a tutorial that uses @app.on_event("startup") to load a model. The app starts, but the model dictionary is empty and the first request fails with KeyError.

Cause. on_event is deprecated. If you also pass lifespan= to FastAPI(...), the on_event handlers are not called at all, and there is no error telling you so.

Fix. Move the loading code into the lifespan function and delete the on_event handlers. Use only one mechanism in an app.

Calling the dependency instead of passing it

Symptom. An error appears at import time or the endpoint receives the wrong thing when you write Depends(get_model()).

Cause. With the parentheses you call get_model immediately, once, when the module loads, and hand FastAPI its result instead of the function. FastAPI needs the function so it can call it for each request.

Fix. Write Depends(get_model) with no parentheses.

Forgetting that defaults make parameters optional

Symptom. Your model receives None or a default value for a feature the caller forgot, and returns a confident but meaningless prediction. No error appears.

Cause. Somebody added = None or = 0 to the parameter, perhaps to stop a 422 during testing. A parameter with a default is optional, so FastAPI no longer rejects its absence.

Fix. For inputs the model really needs, give no default. Let the 422 happen. Reserve defaults for genuinely optional settings, and if a missing field would corrupt a prediction, it is not optional.

Returning NumPy types from an endpoint

Symptom. You return model.predict(x)[0] and get a serialisation error, an unexpected response shape, or a 500 mentioning a type FastAPI cannot encode.

Cause. ML libraries return NumPy numbers and arrays, which are not plain Python types. Your response model or the JSON encoder may not handle them.

Fix. Convert before returning: float(pred), int(pred) or pred.tolist(). Declare a response model such as Prediction so a type mismatch is caught in your tests rather than by a caller.

Staring at the client when the bug is on the server

Symptom. The client only shows "Internal Server Error". You change the request six times and nothing improves.

Cause. A 500 means your code raised an exception, and the explanation is printed in the terminal where the server runs, not in the HTTP response.

Fix. Look at the server terminal first. The last line of the traceback names the exception, and the lines above point to your file and line number.

Mistaking a 422 for a server problem

Symptom. A teammate reports "the API is broken, it returns 422".

Cause. 422 is FastAPI doing its job: the request failed validation. The server is healthy and your function never ran.

Fix. Read the detail list in the response. The loc entry says where, for example ["body", "rooms"], the msg says why, and the input shows what was received. Fix the request, or if the rule itself is wrong, fix the Pydantic model.

Testing the lifespan with a bare TestClient

Symptom. Your endpoint works under fastapi dev but the test fails with KeyError: 'house_price'.

Cause. client = TestClient(app) alone does not run the lifespan code, so the model was never loaded in the test.

Fix. Use the context manager form, with TestClient(app) as client:, which runs startup on entry and shutdown on exit. Or override the model dependency with a stub using app.dependency_overrides, and remember to clear the overrides afterwards.

Forgetting the test dependencies

Symptom. Importing the test client fails because httpx is missing, or pytest is not found.

Cause. The test client uses httpx, and neither it nor pytest is part of the base FastAPI install.

Fix. Install both: uv add httpx pytest or pip install httpx pytest. You may also see a deprecation warning about httpx when running the tests with current versions. Read it, and note it for the next upgrade. It did not make tests fail in our run.

A shell-form CMD in the Dockerfile

Symptom. docker stop takes about ten seconds, the lifespan shutdown code never runs, and Compose feels slow to restart.

Cause. CMD fastapi run app/main.py without brackets runs the command through a shell, which does not pass termination signals on to your app.

Fix. Use the exec form, a JSON list: CMD ["fastapi", "run", "app/main.py", "--port", "80"].

Four workers, out of memory

Symptom. You add --workers 4 for speed, and the container is killed for exceeding memory.

Cause. Each worker is a separate process, and each one runs lifespan and loads its own copy of the model. Memory use is roughly the model size times the number of workers, plus the framework overhead.

Fix. Measure memory with one worker, then multiply. On Kubernetes, prefer one process per container and more replicas, with limits set from the measurement. Do not raise workers to fix latency before you know whether the model call itself is the bottleneck.

Trusting a tutorial's old syntax

Symptom. Deprecation warnings, or code that uses .dict(), class Config, @app.on_event or x: str = Header().

Cause. Much of the internet predates Pydantic v2 (FastAPI 0.100.0) and the move to Annotated.

Fix. Check the date and the version. Prefer model_dump(), model_config, lifespan and Annotated. When a snippet and the official documentation disagree, the documentation wins.

Upgrading without pinning or testing

Symptom. After a routine pip install --upgrade, something that worked yesterday breaks.

Cause. FastAPI is still 0.x, and minor versions may include breaking changes. For example, 0.137.0 changed how included routers are stored, which affects code that iterates over router.routes.

Fix. Pin a minor range such as fastapi[standard]>=0.142.0,<0.143.0, run your tests, then upgrade deliberately and read the release notes for the breaking-changes entries. The documentation's procedure is: write tests, upgrade, run tests, fix, then pin the new version. Do not pin Starlette yourself, because FastAPI selects it.

Habits worth adopting on day one

Declare the output as well as the input. Always give an endpoint a return annotation such as -> Prediction or a response_model. It documents the contract, validates what you return, and drops fields you did not mean to expose. It costs one line.

Put rules in the Pydantic model. Field(gt=0) and Field(ge=1, le=20) document the valid range, appear in /docs and are enforced for every caller. A rule hidden inside an if is invisible to the people who need it.

Keep a stable /health endpoint. It should return quickly and say whether the model is loaded. Every orchestrator, load balancer and uptime monitor you meet will want something like it.

Version your model output. Return a model_version field. When a prediction looks wrong in a month, knowing which model produced it is half of the investigation.

Reject inputs outside the trained range. Pydantic checks types and limits, not whether your model has ever seen anything like the input. A house of 50,000 square metres returns a number, and the number is meaningless. Add an explicit 400 for out-of-range inputs, and log how often it triggers.

Read settings from the environment. The model path, log level and keys come from environment variables with a sensible default for local development. The same code then runs everywhere, and a bad path fails at startup.

Keep secrets out of the repository. Never commit an API key. If one was ever committed to a public repository, treat it as leaked even after deleting the commit, and rotate it.

Write the sad-path tests first. A test that sends a negative area and expects 422 takes thirty seconds to write and protects the part of the system you did not code yourself.

Use the generated docs as your test client. Keep /docs open while you develop. If something is awkward to try there, it will be awkward for your callers too, which is a design signal.

Pin versions, including the model's libraries. A model saved with one scikit-learn version may not load with another. Put exact versions in requirements.txt for everything the model needs, and pin FastAPI to a minor range.

Think about where the data goes. For employers in the Gulf and Egypt that handle personal or regulated data, the region your API runs in is a compliance decision. Ask early which regions are allowed, and choose the deployment region deliberately, not by default.

Learn one neighbouring tool. FastAPI is the web layer. Understand Docker for packaging, and read about MLflow for where the model file comes from. A service you can build, package and trace back to an experiment is worth more than one you can only run.

Debugging in a sensible order

When something is wrong, do not change code first. Ask these questions in order, and stop at the first one that explains it.

1. Is the server running and listening where you think? Check the terminal for "Application startup complete" and the port.
2. What status code came back, and what is its first digit? 4xx means the request is wrong, 5xx means your code is.
3. If 404 or 405, compare the path and method to your decorator. A trailing slash or a POST sent to a GET route is enough.
4. If 422, read detail: loc, msg, input. The answer is in the response.
5. If 500, read the traceback in the server terminal, bottom line first.
6. If it is slow, check whether a blocking call sits inside async def, whether the model is loaded per request, and how many workers are running.
7. If it works locally but not in the container, read docker logs, then check paths, which are resolved from the container's working directory, and the listening port.
8. If a test fails but the server works, check whether the lifespan ran, whether httpx is installed, and whether a dependency override leaked from another test.

This order works because each step is cheap and each rules out a whole category. Beginners tend to jump to step six or seven because performance and containers feel like the interesting problems, but most real beginner issues are a wrong port, a wrong method or an invalid body.

A short checklist

Before you call an endpoint finished, tick each of these.

• I installed "fastapi[standard]" with quotes, inside a virtual environment.
• I run fastapi dev while developing and fastapi run only for deployment.
• Inputs are described by a Pydantic model with rules, and required fields have no default.
• The output has a return annotation or response_model.
• The model loads once in lifespan, not inside the endpoint.
• The prediction endpoint is plain def, unless I really await something.
• NumPy values are converted to plain Python numbers before returning.
• Out-of-range inputs get a clear 400, with a message the caller can act on.
• There is a /health endpoint that reports whether the model is loaded.
• I have at least one test for a 422, one for a normal prediction using with TestClient(app), and one using a stub model.
• Settings come from environment variables, and no secret is in the code.
• requirements.txt pins FastAPI to a minor range and pins the ML libraries exactly.
• The Dockerfile uses an exec-form CMD, copies requirements.txt before the code, and does not use the deprecated tiangolo/uvicorn-gunicorn-fastapi image.
• I measured memory with one worker before choosing a worker count.
• I know which cloud region the service will run in, and whether my data may be processed there.

If every box is ticked you have a service that is small, honest about its inputs and outputs, and easy for the next person to run. That is a better foundation than any clever optimisation, and it is exactly what the mid-level guide builds on: routers, middleware, background work, proper async patterns and running more than one process.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. If you can already serve a model behind @app.post("/predict"), this level is about making that service predictable under load, testable in CI and boring in production.

Where this picks up

The guide targets FastAPI 0.142, the current stable release at the time of writing. FastAPI is still a 0.x project: the versioning page says a MINOR bump can carry breaking changes and a PATCH bump is a bug fix only. Version 1.0 is reserved for the future and does not exist yet, so treat every minor upgrade as something to test, which a later section returns to. Each section starts with the problem it solves and ends, where it helps, with a Try it you can run on a real project in a few minutes.

What happens between the socket and your function

Beginner treated FastAPI as a decorator that turns a function into an endpoint. To predict its behaviour under load you need a slightly more exact picture, because almost every mid-level production problem is a misunderstanding of who runs what, and where.

FastAPI is an ASGI framework. ASGI is the interface between an async-capable Python web server and an application; the server (Uvicorn, started for you by fastapi run) owns the sockets, parses HTTP, and calls your application as an async callable. FastAPI sits on Starlette, which provides routing, middleware, requests, responses and the lifespan protocol, and on Pydantic, which provides validation and serialisation. FastAPI's own contribution is the glue: it reads your type hints, builds the parameter extraction and validation plan once at startup, and generates the OpenAPI schema from the same information.

The order matters in practice. Middleware wraps everything, so it sees requests that never reach a route, such as a 404 or a rejected body. Parameter parsing and validation happen before your function is called, which is why a malformed request never reaches your model code and why the 422 response is produced without a line of yours. Dependencies are resolved after routing and before the function body; a failing dependency (an HTTPException raised inside one) short-circuits the request in the same way. Finally the return value is validated and serialised against the response type, and that step costs CPU, which matters when you return large structures.

The single most useful consequence is this: each Uvicorn process runs one event loop on one thread, and all of its in-flight requests share it. Concurrency inside a process is cooperative. A coroutine that awaits yields control; a coroutine that computes does not. Everything else in this guide about workers, pools and replicas follows from that one sentence.

async def, def and the threadpool, precisely

The official guidance on the question is short and worth quoting in spirit: use async def when the libraries you call are awaitable and you call them with await; use plain def when they are not; and when unsure, use def. The reason for each half of that rule is what makes it stick.

When you declare an endpoint with async def, FastAPI awaits it directly on the event loop. When you declare it with plain def, FastAPI runs it in an external threadpool and awaits the result, so the event loop stays free to serve other requests while your function runs. That is the whole mechanism. It explains the surprising result that a plain def endpoint calling a blocking function is often better than an async def endpoint calling the same function: the first one blocks a pool thread, the second blocks the entire process.

Model inference is the case that catches teams out. model.predict(x) in scikit-learn, a PyTorch forward pass, an ONNX Runtime session call: none of these are awaitable, and all of them are CPU-bound. Put one inside async def and, for the duration of the call, the process cannot accept a new connection, answer a health check, or finish writing another response. Latency for unrelated endpoints climbs, the Kubernetes liveness probe times out, and the pod is restarted while it is busy doing exactly what you asked.

import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/bad")
async def bad():
    time.sleep(2)          # stands in for model.predict(): blocks the whole event loop
    return {"ok": True}

@app.get("/better")
def better():
    time.sleep(2)          # runs in the threadpool; other requests keep flowing
    return {"ok": True}

Plain def is not a cure-all, though, and the reason is Python's global interpreter lock. A thread running pure-Python code holds the lock, so several threads doing CPU-bound Python work do not run in parallel. Many numerical libraries (NumPy, PyTorch, ONNX Runtime) release the lock inside their native code, so threads do overlap there, but you should measure rather than assume. The official async documentation says that for CPU-bound work such as machine learning you should use multiprocessing in production. In practice that means one of two things: several worker processes or replicas, each with its own copy of the model, or a process pool that your endpoint hands work to.

A third pattern is useful when you have an awaitable edge around a blocking core: an async def endpoint that does the awaitable work (fetching features from a database with an async driver, for instance) and then pushes the blocking call onto a thread with run_in_threadpool, which FastAPI re-exports from fastapi.concurrency.

from fastapi import FastAPI
from fastapi.concurrency import run_in_threadpool

app = FastAPI()

def score(features: list[float]) -> float:
    return sum(features)   # placeholder for a blocking model call

@app.post("/score")
async def score_endpoint(features: list[float]):
    # awaitable I/O could go here, then:
    result = await run_in_threadpool(score, features)
    return {"score": result}

The threadpool is finite: Starlette limits how many threads run concurrently per process, and the limit can be raised through AnyIO's capacity limiter if profiling shows queueing. Past the limit, requests wait. The pool therefore protects you from unbounded thread creation, and it also means that a slow dependency can queue unrelated requests behind it.

Where the model lives: lifespan, state and workers

Beginner showed the lifespan context manager: code before yield runs at startup, code after runs at shutdown, and this is where the model is loaded. Two details at this level determine whether that design survives contact with production.

First, lifespan replaced the older on_event decorators. @app.on_event("startup") and @app.on_event("shutdown") are deprecated, and if you pass lifespan= to FastAPI(...), any on_event handlers are silently not called. A codebase migrated halfway, with a lifespan added and the old startup handler left in place, simply stops running the old handler, and the symptom is a missing connection pool or an empty cache, with no error.

Second, where you keep the loaded object matters for testing. The official example stores the model in a module-level dictionary. That works, and it is also a global that every test shares. A cleaner pattern stores the object on app.state, which Starlette provides for exactly this purpose, and reads it through a dependency, so tests can replace it.

from contextlib import asynccontextmanager
from typing import Annotated

from fastapi import Depends, FastAPI, Request
from pydantic import BaseModel


class Model:
    """Stand-in for a real artifact: joblib, torch, ONNX Runtime session."""
    def predict(self, features: list[float]) -> float:
        return sum(features)


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.model = Model()      # load once per process
    yield
    app.state.model = None         # release


app = FastAPI(lifespan=lifespan)


def get_model(request: Request) -> Model:
    return request.app.state.model


ModelDep = Annotated[Model, Depends(get_model)]


class PredictRequest(BaseModel):
    features: list[float]


class PredictResponse(BaseModel):
    score: float


@app.post("/predict")
def predict(body: PredictRequest, model: ModelDep) -> PredictResponse:
    return PredictResponse(score=model.predict(body.features))

Note three small choices. The endpoint is plain def because the model call blocks. The model is reached through get_model, a function the test suite can override. And the response is a typed model, so the output contract is part of the schema.

Now the multiplier. Every Uvicorn worker process runs its own lifespan, so a service started with --workers 4 loads the model four times. A one-gigabyte model becomes four gigabytes of resident memory before the first request, and on a node with limits this is a common cause of OOMKilled containers that "worked on my laptop with one worker". There is no sharing of Python objects across processes. Some libraries memory-map weights so the operating system shares the pages, which reduces the cost, but you should confirm that for the specific format you use by measuring resident memory per process.

Loading time is the other side of the same decision. If loading takes forty seconds, the service is not ready for forty seconds, and a platform that routes traffic before then returns errors. The fix is not to load lazily on the first request, which only moves the delay onto an unlucky user. The fix is a readiness signal, covered in the Kubernetes section: the pod reports not-ready until the lifespan has finished.

Dependency injection, in full

At Beginner level a dependency was a function you passed to Depends(). At this level you should treat the dependency system as FastAPI's composition mechanism, because it carries your database sessions, model handles, authentication, tenant resolution and pagination, and because it is the main seam for testing.

A dependency is any callable. FastAPI inspects its signature the same way it inspects an endpoint, so a dependency can itself declare query parameters, headers, a body, or further dependencies. That produces a tree: the endpoint depends on a function that depends on a function. FastAPI resolves the tree from the leaves up. Within a single request, if the same dependency appears more than once in the tree, it is called once and its result reused (the default caching is per request, and Depends(..., use_cache=False) opts out). That detail is what makes it safe for both an authentication dependency and a permission dependency to depend on the same "current user" function without hitting the database twice.

Dependencies can be def or async def and you can mix them freely. A plain def dependency runs in the threadpool, which makes it a candidate for the same blocking-call reasoning as endpoints.

The pattern that earns its keep in ML services is the yield dependency, which gives you setup and teardown scoped to one request. The code before yield runs before the endpoint, the yielded value is injected, and the code after runs once the response is on its way.

from typing import Annotated, Iterator
from fastapi import Depends, Header, HTTPException

class Session:
    def close(self) -> None:
        print("session closed")

def get_session() -> Iterator[Session]:
    session = Session()
    try:
        yield session
    finally:
        session.close()           # always runs, even if the endpoint raised

SessionDep = Annotated[Session, Depends(get_session)]

def require_tenant(x_tenant: Annotated[str, Header()]) -> str:
    if x_tenant not in {"acme", "globex"}:
        raise HTTPException(status_code=403, detail="Unknown tenant")
    return x_tenant

TenantDep = Annotated[str, Depends(require_tenant)]

Two habits keep this readable. Use Annotated aliases such as SessionDep and TenantDep, so an endpoint signature reads as a list of capabilities rather than a column of Depends(...) calls. And always pass the function to Depends without calling it: Depends(get_session), not Depends(get_session()). Calling it runs the function once at import time and passes a value where FastAPI expects a callable, and the resulting error points at FastAPI rather than at your mistake.

Dependencies can also be attached without injecting a value, using the dependencies=[Depends(...)] argument on a decorator or a router. This is the right tool for checks that only need to run, such as an authentication gate, a rate limiter, or an audit hook.

The override mechanism is the reason to put the model behind get_model. app.dependency_overrides is a dictionary mapping an original dependency function to a replacement. During a test you set app.dependency_overrides[get_model] = lambda: FakeModel(), run the request, and then clear the dictionary. The replacement can declare its own parameters, and FastAPI resolves them. The next section on testing builds on that.

One distinction deserves care: a dependency is the right place for something that varies per request or per test (a session, a user, a model handle chosen by tenant). It is the wrong place for a one-off computation you could do at import time, and it is a poor place for heavy work that you do not cache, since it runs for every request that uses it.

Request and response models as a contract

A Pydantic model on the request is validation. A Pydantic model on the response is a contract with your callers, and mid-level engineers should treat it with the same care as a database schema, because downstream systems start to depend on it the day it ships.

Declare the response type with a return annotation (-> PredictResponse) or with response_model= on the decorator. FastAPI validates what you return against it, filters out fields that are not declared, and uses it to generate the response schema. The filtering is a security feature as much as a convenience: if your internal object has a password_hash or a raw_features field and the response model does not, it does not leave the service. Use separate models for input and output (UserCreate, UserPublic) rather than one model with a dozen optional fields.

Pydantic v2 is the baseline, and FastAPI moved to it fully at 0.100.0. If you are maintaining older code you will see the v1 spellings: .dict(), .json() and an inner class Config. The current names are model_dump(), model_dump_json() and a model_config attribute. Mixing the two styles in one codebase is a frequent source of confusing deprecation warnings, so migrate a file at a time.

For ML endpoints, validation at the edge does real work that you would otherwise write by hand. Constrain the shape of the input so the model never sees an invalid tensor:

from pydantic import BaseModel, Field, field_validator

class PredictRequest(BaseModel):
    features: list[float] = Field(min_length=4, max_length=4)
    request_id: str | None = None

    @field_validator("features")
    @classmethod
    def finite(cls, v: list[float]) -> list[float]:
        if any(x != x or abs(x) == float("inf") for x in v):
            raise ValueError("features must be finite numbers")
        return v

class PredictResponse(BaseModel):
    score: float
    model_version: str

Three decisions are worth making explicitly. Bound every list and string: an unbounded list[float] lets a caller send ten million numbers and consume memory and CPU before your code looks at it. Reject non-finite values that would propagate NaN through a model and produce NaN output that serialises badly. And return the model version in every response, so that when a metric shifts you can tell which artifact produced it.

Since 0.115.0 FastAPI also lets you declare query, header and cookie parameters as Pydantic models, with an option to forbid extra fields, which is useful when an endpoint has many filter parameters and you want one validated object. Request bodies can forbid unknown fields through the model's configuration as well. Whether to be strict is a judgement about your callers: forbidding extras catches typos early and makes it harder to evolve the API without a coordinated change, while ignoring them is friendlier to clients that send more than you need. Pick one and apply it consistently.

Errors and validation failures you control

FastAPI's default error behaviour is good, and mid-level work is knowing which parts to take over. There are three categories of failure and each has its own handling.

Validation failures produce HTTP 422 with a JSON body whose detail is a list of error objects. Each entry names where the problem was (a loc list such as ["body", "features", 0]), carries a human message and a machine-readable type. Clients can use loc to highlight the offending field. If your callers are other services, do not hide this. If they are public users and you do not want to leak your schema's shape, register a handler for RequestValidationError that returns a flatter body.

Business errors are raised as HTTPException with a status code and a detail. Choose codes with care: 400 or 422 for a bad request, 404 for a missing resource, 409 for a conflict such as a duplicate create, 403 for a permitted identity doing a forbidden thing. These are the codes the official examples use for wrong header token (400), missing item (404) and already-existing item (409). Avoid returning 200 with an "error" field in the body, since proxies, retries, dashboards and alerting all key off status codes.

Unexpected exceptions become a bare 500 with no useful body, which is right for callers and wrong for you. Register a handler that logs the exception with the request ID and returns a generic message. Keep stack traces out of the response.

import logging
from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse

log = logging.getLogger("api")

class ModelUnavailable(Exception):
    pass

def install_handlers(app: FastAPI) -> None:
    @app.exception_handler(ModelUnavailable)
    async def model_unavailable(request: Request, exc: ModelUnavailable):
        return JSONResponse(status_code=503, content={"detail": "Model not ready"})

    @app.exception_handler(RequestValidationError)
    async def validation(request: Request, exc: RequestValidationError):
        log.info("validation failed on %s: %s", request.url.path, exc.errors())
        return JSONResponse(status_code=422, content={"detail": exc.errors()})

A custom exception class for domain failures (ModelUnavailable, FeatureStoreTimeout) is cleaner than raising HTTPException deep inside library code: the inner layers stay unaware of HTTP and one handler maps the exception to a status. A 503 with a Retry-After header is the honest response when the model is not loaded or a downstream dependency is down, and it lets well-behaved clients back off.

Handlers registered for a specific exception class take precedence over the generic one, and HTTPException itself can be customised the same way. Test the handlers: an error path that has never been exercised usually has a bug in it.

Routers, and the 0.137.0 change that surprised people

Past a few endpoints you split the application into APIRouter modules: one for predictions, one for health, one for administration. A router takes a prefix, tags, and dependencies that apply to everything inside it, and you attach it with app.include_router(router). Putting an authentication dependency on an admin router, instead of repeating it per endpoint, removes the commonest source of "we forgot to protect one route".

from fastapi import APIRouter, Depends

def require_key() -> None:
    ...   # check an API key, raise HTTPException(401) if missing

router = APIRouter(prefix="/v1", tags=["predict"], dependencies=[Depends(require_key)])

@router.post("/predict")
def predict(): ...

Version 0.137.0 (June 2026) changed what include_router does, and it is the one breaking change in this series of releases that is likely to affect code you wrote. Before it, including a router cloned its routes into the parent. Now the parent keeps the original APIRouter and APIRoute objects. The consequences are concrete. router.routes is no longer guaranteed to be a flat list of APIRoute; it can hold intermediate objects that contain further routers, forming a tree. Routes added to a router after it was included are reflected, which they were not before. Memory use can drop because routes are not duplicated. The release also adds experimental .matches() and .handle() hooks on APIRouter, which are not a stable API, and its notes describe the change as groundwork for per-router dependencies, exception handlers and middleware later.

What breaks is code that treats routes as a flat list. Custom documentation generators, route-introspection scripts, a health check that enumerates endpoints, a hack that rewrites each route's dependencies after inclusion: all of these may fail or silently skip routes. If you have any, the fix is to stop iterating app.routes for application logic and, where you must walk routes, handle nested objects. Prefer the supported surface (the OpenAPI schema from app.openapi()) over the router internals.

Structure the project so a router imports its own dependencies and the application module imports the routers, never the other way round, to avoid circular imports. A layout that has worked well for ML services is app/main.py for the assembly and lifespan, app/routers/ for HTTP, app/deps.py for shared dependencies, app/schemas.py for Pydantic models and app/ml/ for the framework-free inference code. Keeping inference code free of FastAPI imports means you can unit-test it without a web stack, and reuse it from a batch job.

Testing beyond the first test

Beginner used TestClient to call an endpoint and assert on the response. A service with a model needs a test strategy with layers, and each layer answers a different question.

Unit tests for inference code need no FastAPI at all: given a feature vector, does predict return the right thing? These are fast and numerous. API tests use TestClient and check the HTTP contract: status codes, response shape, validation behaviour, authentication. Integration tests load the real artifact and run a few golden requests; there should be few of them, since they are slow and need the real file.

TestClient is built on httpx, which you must install (uv add httpx pytest, or pip install httpx pytest). Tests are ordinary def functions with no await. The important detail for ML services is the lifespan. A bare TestClient(app) does not necessarily run startup and shutdown; using it as a context manager does:

from fastapi.testclient import TestClient

from app.main import app, get_model

class FakeModel:
    def predict(self, features):
        return 0.5

def test_predict_with_fake_model():
    app.dependency_overrides[get_model] = lambda: FakeModel()
    try:
        with TestClient(app) as client:        # runs lifespan startup and shutdown
            r = client.post("/predict", json={"features": [1, 2, 3, 4]})
        assert r.status_code == 200
        assert r.json()["score"] == 0.5
    finally:
        app.dependency_overrides.clear()

Several practices make this robust. Put the override and cleanup in a pytest fixture so a failing assertion cannot leave the override in place for the next test; a leaked override is a classic cause of tests that pass alone and fail together. Override the model, not the whole endpoint, so the validation and serialisation path is still tested. Where the real lifespan loads a large file, either point it at a tiny fixture artifact through a setting, or override the dependency and skip loading, but write one test that runs the real lifespan so a broken load path is caught.

A contract test compares the generated schema to a committed copy: dump app.openapi() to JSON and diff it in CI. A change to a response model then appears as a diff in a pull request, and a reviewer must approve it. This is cheap and catches the accidental breaking change described earlier.

Test the sad paths deliberately: a missing header, an oversize list, a model that raises, a dependency that times out. They are the paths that run at three in the morning.

Configuration and secrets

A service has settings (a model path, a log level, an allowed-origins list) and secrets (a database password, a signing key). They arrive through the environment, are typed at the edge, and never get baked into an image.

The documented approach is Pydantic Settings, which reads environment variables into a typed class. It is a separate package from Pydantic itself (pydantic-settings), so add it to your requirements. You define the settings once, validate them at startup, and fail fast if something is missing or malformed.

from functools import lru_cache
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="APP_", env_file=".env")

    model_path: str = "/models/current"
    log_level: str = "INFO"
    allowed_origins: list[str] = []
    api_key: str            # required: startup fails if APP_API_KEY is unset

@lru_cache
def get_settings() -> Settings:
    return Settings()

Wrap construction in lru_cache and read the settings through a dependency, Depends(get_settings). The cache means the environment is parsed once, and the dependency means tests can override settings without touching real environment variables.

Pydantic gives you protection that plain os.environ does not: a port set to "eight thousand" fails at startup with a message that names the variable, instead of failing later at the point of use. Mark a setting without a default when the service cannot run without it, so a missing secret crashes the pod during rollout.

Separate configuration that varies per environment (URLs, model path, log level) from secrets. The first can live in a Kubernetes ConfigMap or Helm values; the second belongs in a Secret or an external secrets manager, injected as environment variables or mounted files at run time. Never put a secret in the Dockerfile, in a build argument that ends up in a layer, or in the repository, a rule that applies to every container, as the Docker guide explains in its section on layers. For teams that run a secrets manager, the Vault guide covers how workloads obtain credentials without them living in manifests. The .env file in the example is a developer convenience; production should not rely on one.

Keep the model artifact out of the image where it changes more often than the code. Pulling the model by version at startup, from object storage or a model registry, means a new model needs a configuration change and a restart rather than an image rebuild, and the version you load is a setting you can log. The MLflow guide covers registries and aliases; the pattern here is to resolve one pinned version at startup and report it in responses.

Scaling: processes, replicas and pools

You now know that a process has one event loop and that CPU-bound inference does not parallelise on it. The scaling decision is how to add parallelism, and the official deployment documentation separates two situations.

On a cluster such as Kubernetes, run one process per container and do not use --workers. Let the platform replicate containers and load-balance across them. Replication then has one mechanism, one place to tune, and a failure of one process takes down one replica. On a single server or Docker Compose, --workers 4 is acceptable, and fastapi run app/main.py --port 80 --workers 4 is the documented form.

The arithmetic is the planning tool. Resident memory per process times processes per pod times pods is your memory bill. Throughput per process, measured with a load test using realistic payloads, tells you how many replicas to run for a target request rate. Do not guess at either figure. Measure the cold start time as well, since it determines how fast a new replica can absorb load when autoscaling reacts.

When one request needs more than one core, or when you want blocking inference isolated from the event loop, use a process pool within a worker. The shape is a ProcessPoolExecutor created in the lifespan, with each worker process loading its own model through an initializer, and an async def endpoint that awaits loop.run_in_executor(pool, predict_fn, features).

import asyncio
from concurrent.futures import ProcessPoolExecutor
from contextlib import asynccontextmanager
from fastapi import FastAPI

_model = None

def _init_worker() -> None:
    global _model
    _model = object()              # load the real artifact here, once per pool process

def _predict(features: list[float]) -> float:
    return float(sum(features))    # uses _model in a real service

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.pool = ProcessPoolExecutor(max_workers=2, initializer=_init_worker)
    yield
    app.state.pool.shutdown(wait=True)

app = FastAPI(lifespan=lifespan)

@app.post("/predict")
async def predict(features: list[float]):
    loop = asyncio.get_running_loop()
    score = await loop.run_in_executor(app.state.pool, _predict, features)
    return {"score": score}

The trade-offs are real. The pool isolates the loop from blocking work and caps concurrency at max_workers, which is a deliberate backpressure mechanism, but arguments and results are pickled across a process boundary, so large arrays are expensive to send. It also adds a failure mode: if a pool process is killed (out of memory, for example), the executor can become broken and every later call raises, so handle that case and consider letting the health check fail so the replica restarts. For many teams the simpler answer is more replicas of a plain def service.

Past a certain point, general-purpose Python serving is the wrong tool. When you need dynamic batching, GPU scheduling, or multiple model frameworks behind one endpoint, a dedicated runtime does that better, and FastAPI becomes the thin layer in front: see the Triton guide for model serving on GPUs, the BentoML guide for packaging model services, and Ray Serve for scaling Python serving across a cluster. For large language models, the vLLM guide describes a server built for that workload. Choosing FastAPI is right when the model is small or moderate, the logic around it is custom, and the team wants ordinary Python.

Also bound the work you accept. Without limits, a traffic spike turns into a growing queue and every request times out. Set timeouts at the proxy, size the threadpool or pool to what the hardware can do, and return 503 when you are saturated rather than accepting work you cannot finish.

Containers and Kubernetes

The official Docker documentation gives a short Dockerfile, and the details in it carry weight.

FROM python:3.14
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
COPY ./app /code/app
CMD ["fastapi", "run", "app/main.py", "--port", "80"]

Copying requirements.txt first and installing before copying the source means the dependency layer is cached until the requirements change, which is the same cache rule the Docker guide explains. Pin the Python base image to the version you test with rather than a floating tag, and consider a -slim variant to cut the image size; check that your ML wheels install on it before committing.

The most important line is the last one. Use the exec form of CMD, the JSON array. The shell form wraps your process in a shell, which then becomes PID 1 and does not forward signals. The documentation warns that this causes a ten-second delay on stop or recreate in Docker Compose, and that it breaks graceful shutdown and the lifespan shutdown code, because the application never receives SIGTERM and is killed instead. Your cleanup after yield simply does not run.

The documentation also says the old tiangolo/uvicorn-gunicorn-fastapi base image is deprecated and recommends building your own, which is what this Dockerfile does.

Behind a TLS-terminating proxy or load balancer, pass --proxy-headers, as in fastapi run app/main.py --port 80 --proxy-headers, so the app trusts the forwarded protocol and host headers and generates correct URLs and redirects. Be deliberate about which proxies you trust, because forwarded headers are client-controllable if the app is exposed directly.

On Kubernetes the pieces are a Deployment, a Service and probes. The Kubernetes guide covers the objects; here is how they meet a FastAPI service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scorer
spec:
  replicas: 3
  selector:
    matchLabels: { app: scorer }
  template:
    metadata:
      labels: { app: scorer }
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          image: registry.example.com/scorer:1.4.2
          ports:
            - containerPort: 80
          envFrom:
            - configMapRef: { name: scorer-config }
            - secretRef: { name: scorer-secrets }
          resources:
            requests: { cpu: "1", memory: 2Gi }
            limits: { memory: 2Gi }
          startupProbe:
            httpGet: { path: /healthz, port: 80 }
            periodSeconds: 5
            failureThreshold: 30
          readinessProbe:
            httpGet: { path: /readyz, port: 80 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 80 }
            periodSeconds: 10

Separate the endpoints deliberately. /healthz answers whether the process is alive: it returns 200 as soon as the app can respond and does no downstream calls. /readyz answers whether it can serve traffic: it returns 503 until app.state.model is set and any required dependency is reachable. The startup probe gives a slow model load room to finish before the liveness probe starts judging; without one, a forty-second load on a probe that allows ten seconds is a crash loop. A liveness probe that checks the model or a database turns a downstream outage into a restart storm, so keep it shallow.

Memory is set from measurement. Set the request and limit for memory equal for a model-serving container, so the scheduler places it where the model fits and an out-of-memory kill is predictable. A CPU limit can throttle inference in surprising ways; many teams set a CPU request and omit the limit, which is a decision to make with your platform team.

When the pod is told to stop, Kubernetes sends SIGTERM, then waits for the grace period. With the exec-form CMD, Uvicorn receives the signal, finishes in-flight requests, and runs your lifespan shutdown. Set the grace period longer than your slowest request. A rolling update that kills pods mid-request produces a burst of errors that is easy to prevent.

For packaging the Kubernetes manifests, Helm gives you templating per environment. Use Init Containers for one-off setup such as database migrations, not application startup code, so that three replicas starting together do not race to migrate.

Observability

A service you cannot see is a service you cannot tune. Three signals matter: logs, metrics and traces, plus one that is specific to ML, which is what the model saw and returned.

Logs should be structured, one JSON object per line, and each should carry a request ID. A middleware that reads an incoming X-Request-ID header (or generates one), stores it in a context variable, adds it to every log record and echoes it in the response lets you trace one request through the service and into downstream calls. Log at the edge: method, path, status, duration, model version. Do not log raw request bodies by default; features can be personal data, and for employers in the Gulf and Egypt, data-residency and privacy rules often govern where those logs may be stored.

import time
import uuid
from fastapi import FastAPI, Request

def install_middleware(app: FastAPI) -> None:
    @app.middleware("http")
    async def add_request_id(request: Request, call_next):
        rid = request.headers.get("x-request-id", str(uuid.uuid4()))
        start = time.perf_counter()
        response = await call_next(request)
        elapsed_ms = (time.perf_counter() - start) * 1000
        response.headers["x-request-id"] = rid
        print({"rid": rid, "path": request.url.path,
               "status": response.status_code, "ms": round(elapsed_ms, 1)})
        return response

A caution on this style of middleware: @app.middleware("http") is convenient, but it adds overhead to every request and can interfere with streaming responses and background tasks. For high-throughput services consider a pure ASGI middleware, which is more code and less surprising.

Metrics answer "how is it behaving over time": request rate, error rate, latency percentiles per endpoint, in-flight requests, and ML-specific numbers such as inference time, batch size and the distribution of predictions. Latency measured at the model call separately from latency at the edge tells you whether time is going into inference or into everything around it. Prometheus-format metrics are the common choice and are covered in the Prometheus guide, with dashboards in Grafana. The FastAPI documentation does not prescribe a metrics library, so treat the exporter you pick as a third-party dependency and review it like one.

Traces show one request's path through several services. FastAPI 0.142.0 (29 September 2026) added native OpenTelemetry support (pull request #16403), and 0.142.2 made the app start even if automatic OpenTelemetry configuration fails, so a broken telemetry setup no longer prevents the service from booting. The release notes do not describe how to enable it or which settings it reads, so do not rely on this guide for the configuration: read the pull request and the documentation for the version you run before you adopt it. The OpenTelemetry guide explains the concepts (spans, context propagation, exporters) that transfer regardless of how the integration is switched on. Because the feature is days old at the time of writing, pin the minor version and test it in staging before relying on it.

The ML-specific signal is prediction logging: record a sample of inputs and outputs, with the model version, so you can compare live behaviour to training data and spot drift. Sample, anonymise or hash identifiers, and write to a store with the retention you are allowed. Without this, the first sign of a degraded model is a business metric weeks later.

Security at the edge

A mid-level engineer is not expected to design an identity platform, but is expected not to ship the common holes. The checklist is short, and each item has a reason.

Authenticate with something real. fastapi.security provides OAuth2PasswordBearer, HTTPBearer and API-key helpers for headers, query parameters and cookies. The official tutorial walks through OAuth2 with a password flow and JWT. For anything with real users, validate tokens issued by your identity provider rather than rolling your own login, and check the signature, expiry and audience. Use the helper as a dependency so protected routes declare it and the generated OpenAPI documentation shows the lock icon. For service-to-service calls inside a cluster, a mounted token or mutual TLS at the mesh layer may fit better than a bearer token in every request.

Do not use the token tutorial's sample secrets in production. Signing keys come from your secrets store and are rotated. Password hashing, if you handle passwords at all, should use a maintained library; check the current tutorial for the recommendation instead of copying older posts, since some earlier examples used a package that is no longer maintained.

Limit cross-origin access. CORSMiddleware with an explicit list of allowed origins, never a wildcard together with credentials. CORS is a browser rule: it does not protect your API from a script or another server.

Protect the documentation and schema. The interactive docs and /openapi.json are valuable to an attacker. In a public deployment, turn them off with FastAPI(docs_url=None, redoc_url=None, openapi_url=None) or put them behind authentication.

Limit request size and rate at the proxy. FastAPI does not enforce an upload or body size limit for you; configure it at the ingress or load balancer, along with timeouts and rate limits. This pairs with the bounded list lengths in your models.

Terminate TLS at the proxy and pass --proxy-headers so the app knows it is served over HTTPS. The documentation has an "About HTTPS" page that explains the model.

Keep .pkl and similar files trusted. A pickled model file can execute code when loaded. Load artifacts only from storage you control, verify checksums, and prefer formats that are data-only where your framework supports them.

Run as a non-root user in the container, and keep the image small and scanned. The Trivy guide covers scanning images for known vulnerabilities in CI.

Streaming responses and background work

Not every result fits in one JSON body, and not every task should hold the request open. FastAPI offers two tools with different guarantees.

Streaming returns data as it is produced, using StreamingResponse with a generator. It suits large downloads, server-sent events and token-by-token output from a language model. The caveats are that status and headers are sent first, so an error midway cannot change the status code, and that a client disconnect should stop the generator so you do not keep computing for nobody. A blocking generator runs in the threadpool, an async generator on the loop, so the earlier rules about blocking apply again.

BackgroundTasks runs a function after the response is sent. It is right for small fire-and-forget work such as writing a prediction log record. It is wrong for anything long, anything that must not be lost, and anything CPU-heavy: it runs in the same process, shares the worker's resources, and disappears if the process is killed, with no retry and no status to query.

For batch scoring, retraining, or any job measured in minutes, return an identifier immediately and hand the work to a queue and a separate worker service, then provide a status endpoint. This is the architecture that keeps the API thin. The API validates, enqueues and reports; the worker computes. For orchestrating the heavier work, the Ray guide covers distributed Python execution, and the Kubernetes guide covers running jobs.

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

def write_audit(request_id: str, score: float) -> None:
    ...   # append to a log store; keep this short and idempotent

@app.post("/predict")
def predict(features: list[float], tasks: BackgroundTasks):
    score = float(sum(features))
    tasks.add_task(write_audit, "req-123", score)
    return {"score": score}

Because a background task failure happens after the response, nobody sees it. Log inside the task and alert on those logs.

Upgrades and staying current on a 0.x framework

Because FastAPI is 0.x, upgrade policy is part of engineering the service. The documented procedure is plain: write tests, upgrade, run the tests, fix what broke, then pin the new version. Do not pin Starlette yourself; FastAPI chooses a compatible version, and a manual pin fights it. You can pin Pydantic to a range such as pydantic>=2.7.0,<3.0.0.

Pin FastAPI to a minor range, for instance fastapi[standard]>=0.142.0,<0.143.0. You then receive patch fixes automatically and take minors deliberately. Before each minor bump, read the release notes for the "Breaking Changes" heading, search your code for the affected surface, and run the suite. A recent example worth knowing: 0.142.0 also raised the Starlette version it uses and the minimum Typer version, which matters if you pin either yourself.

Recognise what is deprecated and remove it on your own schedule: on_event handlers in favour of lifespan, Pydantic v1 spellings in favour of the v2 names, and the old combined Uvicorn-Gunicorn base image in favour of your own Dockerfile. The documentation now teaches Annotated for parameters and dependencies, and older tutorials that put Depends() or Header() in default values still run but are no longer the recommended style.

CI/CD for a model service

The build-and-deploy path should make the same artifact flow from a commit to production, as the GitHub Actions guide shows for pipelines in general. For a FastAPI model service, the stages are these.

First, lint and unit-test the framework-free inference code, which is quick. Second, run the API tests with the fake model, plus the OpenAPI contract test. Third, build the image with the cache working, tag it with the commit identifier, and scan it. Fourth, run a smoke test against the built container: start it, wait for /readyz, send one golden request and check the response and the reported model version. Only then push the image, and promote the same image digest through staging to production rather than rebuilding per environment.

The smoke test is the stage people skip and regret. It catches everything the unit tests cannot: a missing file in the image, a wrong CMD, a dependency that installs on your laptop but not in the slim base, a model that fails to load under the container's memory limit. It runs the real startup path.

Putting it all together

Here is how the pieces combine in one service, described as the decisions you would make in order.

You start with the framework-free inference module in app/ml/, tested on its own with a handful of feature vectors. You define the request and response models with bounds on every list, finite-number validation, and a model_version field in the response. You write the lifespan so that it loads the model from a pinned location named by a setting, stores it on app.state, and raises if loading fails. The model is reached through get_model, and settings through a cached get_settings, both of them dependencies.

The endpoint is a plain def function, because inference blocks, and it returns a typed response. Routers group the prediction endpoints under a version prefix with an authentication dependency attached to the router, and a separate router carries /healthz and /readyz. Custom exception handlers turn a missing model into a 503 and log unexpected errors with the request ID. A middleware sets and logs that ID.

Tests override the model with a fake, run the lifespan in one real-artifact test, check the OpenAPI schema against the committed file, and cover the unhappy paths. The Dockerfile copies requirements first, installs with no cache, and ends with an exec-form CMD that runs fastapi run on one process with --proxy-headers if a proxy terminates TLS. The Kubernetes manifest requests equal memory and limit measured from a load test, defines startup, readiness and liveness probes, and sets a grace period longer than the slowest request. Replicas, not workers, provide the parallelism.

The pipeline lints, tests, builds, scans, smoke-tests the container and promotes the same digest. FastAPI is pinned to a minor range and bumped deliberately, with the release notes read first. Logs are structured, metrics include inference time and the model version, and you have read the OpenTelemetry pull request before deciding to use native tracing.

Where you are now

You can explain what the event loop and the threadpool do, and you can predict which calls block the whole process. You can load a model once per process, keep it reachable through a dependency, and replace it in tests. You can shape validation errors, design a response contract, and keep it from changing by accident. You understand the router tree introduced in 0.137.0, the reason workers multiply model memory, and why a cluster should scale by replicas.

You can containerise the service with the right CMD, wire probes that separate liveness from readiness, and configure the service through typed settings. You know the security basics at the edge, the limits of background tasks, and how to pin and upgrade a 0.x framework.

The next level, Senior, treats the service as a platform for a team: failure modes at scale, multi-tenancy, supply-chain trust, governance, migrations and the point at which FastAPI is not the right tool.

Sources

• FastAPI documentation home
• FastAPI tutorial: installation, running, first steps
• FastAPI release notes
• FastAPI deployment overview
• FastAPI in containers with Docker
• About FastAPI versions
• Lifespan events
• Concurrency and async / await
• Dependencies
• Security
• Testing
• FastAPI 0.137.0 release
• FastAPI 0.142.0 release
• Native OpenTelemetry pull request #16403

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than level, in about thirty-five minutes. Mid-level interviews rarely ask what FastAPI is. They give you a symptom or a design choice and watch how you reason. Fast review first, scenarios and trade-offs after. Senior reviews all three.

Where this picks up

Foundations, in one screen

FastAPI is an ASGI web framework built on Starlette for the web layer and Pydantic for validation. You declare endpoints with type hints; it validates input, serialises output, and generates an OpenAPI schema and interactive docs from the same declarations.

Plain def versus async def: async def runs on the event loop; plain def runs in a threadpool. Use async def only when you await something. Blocking code inside async def stalls every request in that process.

Lifespan: an async context manager passed as FastAPI(lifespan=...). Code before yield is startup, code after is shutdown. The older @app.on_event handlers are deprecated, and they are not called at all if you pass lifespan.

Versioning: FastAPI is 0.x. Minor bumps can break things, patch bumps are fixes. 1.0 has not been released.

Scenario questions

"Latency on every endpoint doubles when the prediction endpoint is busy. What do you check first?"

A strong answer starts with the symptom's shape: unrelated endpoints slow down together, which points to something shared in the process, and the shared thing is the event loop. Look at the prediction endpoint's signature. If it is async def and calls a blocking model, that is the cause: the loop cannot serve anything else while predict runs.

The fix is to make the endpoint plain def so FastAPI runs it in the threadpool, or to offload the call with run_in_threadpool, or, if inference is CPU-heavy and Python-bound, to move it to a process pool or more replicas. Then verify: load-test with a mix of endpoints and compare the p99 of the cheap ones before and after.

What a weak answer does: says "add more workers" without finding the cause, or suggests making the model call async by adding the keyword, which changes nothing about blocking. What a strong answer adds: the global interpreter lock, and that many numeric libraries release it so threads can help but you should measure; the official guidance to use multiprocessing for CPU-bound work in production.

"The pod keeps being restarted right after deploy. The logs show it was loading the model."

Check the probes. A forty-second load against a liveness probe that starts at once and allows ten seconds produces a crash loop: Kubernetes kills the pod before it can finish loading. The fix is a startup probe with enough tolerance for a cold start, a readiness probe that returns 503 until app.state.model is set, and a shallow liveness probe that does not depend on the model.

Then ask why loading is so slow, since startup time determines how fast autoscaling helps. Options include fetching the model from a nearby location, using a smaller artifact, and memory-mapped formats. A strong answer also checks the memory limit, because an out-of-memory kill during load looks identical from the outside. kubectl describe pod showing OOMKilled settles it.

"It runs fine with one worker and gets OOM-killed with four."

Each worker process runs its own lifespan and loads its own copy of the model. Memory is per process and nothing is shared automatically. Four workers means four copies. The fix on a cluster is the one the documentation recommends: one process per container and scale with replicas, with the memory request and limit set from measured resident size. On a single server where multiple workers are acceptable, size them against RAM, or use a memory-mapped format and measure whether the pages are shared.

A strong answer mentions that replicas multiply memory too, so the real cost model is memory per process times processes per pod times pods.

"A test passes alone and fails when the whole suite runs."

Suspect leaked global state. In FastAPI the usual culprit is app.dependency_overrides: a test set an override, an assertion failed before the cleanup, and later tests ran with the fake. The fix is to set and clear overrides in a fixture so teardown always runs. The second suspect is a module-level model dictionary or other global populated by one test, which is why keeping the model on app.state and reaching it through a dependency is easier to isolate. A third is a test that used TestClient(app) without the context manager and so relied on, or skipped, the lifespan.

"After upgrading from 0.136 to 0.137, our route-listing script crashes."

0.137.0 changed include_router. It used to clone routes into the parent; it now keeps the original APIRouter and APIRoute objects, so router.routes can contain intermediate objects holding further routers, a tree rather than a flat list. A script that assumes every entry is an APIRoute breaks. The robust fix is to read the generated OpenAPI schema, app.openapi()["paths"], which is a documented output, instead of walking route internals.

A strong answer also describes the upgrade procedure from the documentation: tests first, upgrade, run the tests, fix, then pin. And it proposes the lasting defence: pin the minor range and keep a committed OpenAPI file with a diff test.

"Users report a prediction request returned 200 but the audit log entry is missing."

Look at how the audit write is done. If it is a BackgroundTasks function, it runs after the response in the same process, so a crash, a restart or an exception inside it loses the record silently. Background tasks suit small fire-and-forget work; anything that must not be lost belongs on a queue with a worker that retries. Also check that the task logs its own failures, because nobody sees an exception that happens after the response.

"A client sends a 50 MB JSON body and the service slows for everyone."

FastAPI does not set a body size limit. The proxy or ingress should: set a maximum body size and timeouts there. Inside the service, bound the model fields (max_length on lists and strings) so a huge array is rejected during validation, before it reaches the model. Mention that validation itself costs CPU on a large body, which is another reason to reject early at the edge.

"The docs page is public and security flagged it."

The interactive docs, ReDoc and /openapi.json describe your whole API. For a public deployment, disable them with FastAPI(docs_url=None, redoc_url=None, openapi_url=None) or put them behind authentication. A good answer separates the concerns: hiding the docs is not a substitute for authentication, since an attacker can still call endpoints they guess.

"We need to roll out a new model version with no downtime. How?"

Start from what the service already provides. Because the model version is a setting and is reported in every response, a new version is a configuration change plus a rollout rather than a code release. On Kubernetes, a rolling update replaces pods one at a time, and the readiness probe is what keeps traffic away from a new pod until its model has loaded. The grace period and exec-form CMD ensure old pods finish in-flight requests before they stop.

To limit risk, run the new version on a fraction of traffic first, a canary, and compare error rate, latency and a quality signal against the old one, using the version label on your metrics. Roll back by reverting the setting. A weak answer says "deploy it and watch"; a strong answer names the readiness gate, the canary comparison and the rollback path, and mentions that the same image can carry both versions if the model is fetched at startup.

"A teammate wants to add a global dictionary cache of predictions inside the service. Your view?"

Ask what it is for and what could go wrong. A cache inside a process is per process, so replicas and workers each hold their own, hit rates are lower than expected, and memory grows with the number of distinct inputs unless it is bounded. It must be keyed by the model version, or it serves stale answers after a rollout. It also changes the performance profile in ways that make load tests misleading. If predictions are expensive and inputs repeat, a bounded cache with a size limit, the model version in the key and a metric for the hit rate is reasonable; for sharing across replicas, an external cache is the usual choice, with its own latency and failure modes. The strong answer treats it as a design decision with measurements rather than a quick fix.

Trade-off questions

Workers versus replicas

On a cluster, one process per container and more replicas: one scaling mechanism, failures isolated to one replica, the platform does load balancing, memory limits map to one process. On a single host or Docker Compose, several workers are acceptable and simpler. The cost of workers inside a container is that a memory-limit kill takes all of them, and that the orchestrator cannot see per-process health.

async def with a process pool versus plain def

Plain def is the simplest option and often enough when the numeric library releases the GIL. A process pool gives real parallelism for Python-bound CPU work, isolates the loop, and caps concurrency, at the price of pickling arguments, per-process model copies and a new failure mode when a pool process dies. Choose the pool when measurements show threads do not scale, not before.

FastAPI versus a dedicated model server

FastAPI wins when the model is small to moderate, the logic around it is custom (feature lookups, business rules, several models composed) and the team wants ordinary Python. A dedicated runtime wins when you need dynamic batching, GPU scheduling, multi-framework serving or very high throughput per GPU. The common compromise is FastAPI in front for authentication, validation and orchestration, and a serving runtime behind it. The Triton guide, the BentoML guide, the Ray Serve guide and the vLLM guide describe the alternatives.

Load the model from the image or at startup

Baking the model into the image gives immutability and simple rollbacks, and makes images large and slow to pull. Fetching a pinned version at startup keeps images small and lets a model change without a rebuild, at the price of a startup dependency on storage and a longer cold start. Whichever you choose, pin an exact version and report it in each response.

Strict or permissive request models

Forbidding unknown fields catches client typos early and makes the contract explicit; it also makes it harder for clients to send extra data you do not need and harder to evolve an API gradually. Ignoring extras is friendlier. State your choice and apply it consistently.

Middleware versus dependency

A middleware sees every request, including ones that never match a route, and suits cross-cutting concerns such as request IDs and timing. A dependency is explicit per route or router and suits authentication, tenant resolution and anything the endpoint needs the value of. Router-level dependencies mean a new route cannot forget its authentication.

Commands and snippets to have ready

fastapi dev                                   # development, auto-reload, docs at /docs
fastapi run app/main.py --port 80             # production mode
fastapi run app/main.py --port 80 --workers 4 # several workers on one host
fastapi run app/main.py --port 80 --proxy-headers   # behind a TLS-terminating proxy
pip install "fastapi[standard]"               # quote the brackets in zsh

# Override a dependency in a test, always cleaning up
app.dependency_overrides[get_model] = lambda: FakeModel()
try:
    with TestClient(app) as client:           # runs lifespan
        ...
finally:
    app.dependency_overrides.clear()

CMD ["fastapi", "run", "app/main.py", "--port", "80"]   # exec form, never shell form

What a strong answer covers that a weak one misses

• A cause, not a remedy. Weak answers reach for workers, caches or more hardware. Strong ones name the mechanism: event loop, per-process model copies, global test state, shell-form CMD.
• A way to verify. "I would load-test it before and after", "I would read OOMKilled in kubectl describe", "I would diff the OpenAPI file."
• Production boundaries. Probes that separate liveness from readiness, graceful shutdown, proxy limits, secrets outside the image.
• Honesty about the framework's age. Mentioning that FastAPI is 0.x and that you pin the minor range shows operational maturity.
• Knowing what is deprecated. on_event, Pydantic v1 spellings, the combined Uvicorn-Gunicorn base image, shell-form CMD.
• Recognising the limit. Saying when you would not use FastAPI for the model itself.

Common interview questions

What does Depends do and why is it useful for ML services? It declares that a function's result should be computed by FastAPI and injected, with sub-dependencies resolved as a tree and results cached within one request. In an ML service it gives you a seam: the endpoint asks for a model, and tests or other environments substitute a different one through dependency_overrides.

Why does the shell form of CMD matter? The shell becomes PID 1 and does not forward SIGTERM to the app, so graceful shutdown and the code after yield in lifespan do not run, and Docker Compose waits ten seconds before killing the container, according to the FastAPI Docker documentation.

How do you run startup code, and what is deprecated? Use the lifespan context manager. on_event("startup") and on_event("shutdown") are deprecated, and they are ignored when lifespan is set.

How do you keep a slow model from blocking other requests? Do not call it directly in async def. Use plain def so it runs in the threadpool, offload it with run_in_threadpool, use a process pool, or scale with replicas, depending on measurements.

How would you secure an inference API? Authenticate with tokens from your identity provider using fastapi.security helpers, restrict CORS origins, disable or protect the docs in public deployments, enforce size limits and rate limits at the proxy, terminate TLS at the proxy with --proxy-headers, and keep secrets out of images.

How do you upgrade FastAPI safely? Pin a minor range, read the release notes for breaking changes, run the tests including an OpenAPI diff, upgrade, fix, then pin the new range. Do not pin Starlette yourself.

What changed in 0.137.0? include_router no longer clones routes; it keeps the original router objects, so router.routes can be a tree and code that assumes a flat list may break.

What is new in 0.142? Native OpenTelemetry support arrived in 0.142.0, and 0.142.2 made the app start even if automatic OpenTelemetry configuration fails. Say honestly that you would read the pull request before relying on it, as the release notes do not describe the configuration.

Where does a model belong in the code? Loaded once per process in the lifespan, stored on app.state, and reached through a dependency, with inference code in a module that does not import FastAPI so it can be unit-tested and reused in batch jobs.

How do you observe it? Structured logs with a request ID, metrics for rate, errors, latency and inference time per model version, traces through OpenTelemetry, and sampled prediction logging for drift checks. See the Prometheus guide and the OpenTelemetry guide.

Sixty-second self-test

Answer aloud without looking back.

1. Which of async def and def runs on the event loop, and which runs in the threadpool?
2. Name two things that go wrong with a blocking model call inside async def.
3. Why does --workers 4 multiply memory, and what does the documentation recommend on Kubernetes?
4. What happens to on_event handlers if lifespan is set?
5. How do you substitute a model in a test, and how do you make sure the substitution is undone?
6. What does the shell form of CMD break?
7. What did 0.137.0 change about include_router?
8. Which probe should wait for the model to load, and which should not look at it?
9. Why is BackgroundTasks the wrong place for a record that must not be lost?
10. Where do you put the request size limit, and what do you do inside the model?

If any answer took more than a sentence, go back to the matching section of the detailed guide. Related guides: Docker, Kubernetes and Helm.

Part two of three. At this level your service starts and answers requests, so the problems change character. They are no longer "why won't this run" but "why does latency spike under load", "why was the pod killed" and "why did the upgrade break our script". Start with the error table, then the practices in order of leverage, then practice cards and operational recipes.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

Three of these deserve a closer look because they are easy to misdiagnose.

The event-loop stall. Nothing logs an error and no exception is raised. The evidence is indirect: latency rises across endpoints that share nothing, CPU on the pod is moderate rather than saturated, and the health check times out occasionally. To confirm, run a request to the suspected endpoint while polling a trivial one such as /healthz. If the trivial one stalls for the duration of the heavy one, the loop is blocked. The async def keyword is not what makes code asynchronous; awaiting something is.

Memory that grows with workers. The signal is OOMKilled in kubectl describe pod, or a container that exits with code 137. The cause is arithmetic: the model's resident size times the number of processes, plus the interpreter and libraries per process. Measure it with a single process first, then decide how many you can afford.

The leaked override. Overrides are global to the app object. A failing assertion that skips cleanup leaves the fake in place, and the next test passes for the wrong reason or fails mysteriously. The only reliable defence is a fixture with a teardown.

The practices that pay off most, in order of leverage

These are ordered by how much trouble each one prevents per hour spent.

1. Find and remove blocking calls on the event loop

This single habit prevents the most production incidents. Review every async def endpoint and dependency and ask whether each call inside it is awaited. Anything that is not awaited and takes more than a millisecond or so, a model call, a synchronous HTTP client, a file read of a large file, a synchronous database driver, a time.sleep, belongs in a plain def function or behind run_in_threadpool. When unsure, the documentation's rule applies: use def.

2. Put the model behind a dependency

Load once in the lifespan, store on app.state, hand it out with get_model. You gain a test seam, a single place to add version reporting, and a clean spot to add a per-tenant model cache later. It costs six lines.

3. Pin FastAPI to a minor range and keep an OpenAPI diff test

FastAPI is 0.x, so a minor can break. Pin with a range like fastapi[standard]>=0.142.0,<0.143.0, bump deliberately, read the release notes for "Breaking Changes" first, and let a committed openapi.json and a test that compares it to app.openapi() tell you when the public contract moved. Do not pin Starlette; FastAPI chooses it.

4. Make the response model a contract

Return typed models, separate the internal object from the public one, bound every list and string, and include the model version in each response. This stops accidental data exposure and gives you a handle when a metric shifts.

5. Use exec-form CMD and real probes

CMD ["fastapi", "run", "app/main.py", "--port", "80"]

Together with startup, readiness and liveness probes and a termination grace period longer than your slowest request, this makes deploys uneventful. It is the cheapest reliability work you can do.

6. Scale with replicas on a cluster; use workers on a single host

One process per container, measured memory requests and limits, horizontal scaling. Avoid combining a process manager inside the container with orchestrator-level replication.

7. Type your settings and fail at startup

Use Pydantic Settings in a cached get_settings, make required settings required, and let a missing secret stop the pod during rollout rather than the first request.

8. Test the real startup path once

Fakes make most tests fast, but one test that runs the actual lifespan against a tiny artifact, and one container smoke test in CI that waits for /readyz and sends a golden request, catch the failures fakes hide: missing files, wrong paths, incompatible wheels.

9. Log with a request ID and the model version

Every log line and every response should be traceable. Add a middleware for the ID, include the version, and measure inference time separately from request time.

10. Keep secrets, large artifacts and docs exposure under control

Secrets come from a Secret or manager, never the image. Pull models by pinned version. Turn off /docs and /openapi.json in public deployments or authenticate them.

A diagnostic order for a slow or failing service

When something is wrong in production, resist the urge to change configuration at random. Work from the outside in, and stop at the first layer that explains the symptom.

First, is the process alive and ready? Check pod status, restart counts and the last termination reason. A restart count that climbs right after deploy points at probes or memory. A pod that is running but not ready points at the lifespan: either the model is still loading or loading failed in a way that did not crash the process.

Second, is the event loop free? Poll a trivial endpoint while the slow one runs. If the trivial endpoint stalls, something is blocking the loop, and no amount of scaling the database or the cache will help until you find it. If the trivial endpoint stays quick, the slowness is local to one endpoint or one downstream call.

Third, where does the time go inside the request? This is the reason to record inference time separately from total request time. If total latency is high and inference is low, the time is in validation, serialisation, a dependency, or waiting for a thread in the pool. A large response body is a frequent hidden cost: serialising a big list of floats takes real CPU. If inference time is high, the model or its inputs are the issue, and the question becomes whether input sizes changed, whether a new model version is heavier, or whether the process is competing for cores with its neighbours.

Fourth, is the pool saturated? Plain def endpoints and dependencies share the threadpool. A burst of slow requests can occupy every thread, and the next request waits even though CPU is idle. Symptoms are latency that grows linearly with concurrency and low CPU. The remedies are to reduce the time each request holds a thread, to raise the pool limit if the work is I/O-bound, or to add replicas.

Fifth, is it the platform? CPU throttling from a limit that is too tight, noisy neighbours, slow storage when loading a model, and a proxy timeout that is shorter than your slowest request all look like application problems. Compare the same request against a single replica outside the platform before concluding the code is at fault.

Write down what you changed and what the graphs did. Teams that keep a short note per incident build up a picture of their service that no documentation supplies.

Errors that hide in the data, not the code

Some failures at this level are not exceptions at all, so no error table catches them.

Silent type coercion. Pydantic converts compatible values by default, so the string "3" becomes the integer 3 for an int field. That is helpful for query strings and surprising for a model whose training pipeline distinguished the two. If your contract needs strict types, say so in the model configuration and test it with a request that sends the wrong type.

Feature order and schema drift. A request that sends features as a list relies on position. If the training code reordered columns, the endpoint returns plausible but wrong numbers and no test fails. Prefer named fields in the request model, convert to the model's expected order in one place, and cover that conversion with a test against a known example.

Training-serving skew. The preprocessing in the service differs subtly from preprocessing at training time: a different rounding, a missing default, an unseen category. Share the preprocessing code between both, version it with the model, and compare a sample of live inputs against training statistics.

NaN and infinity. A non-finite value can pass type validation and then propagate through a model. Reject it at the edge, as the detailed guide shows.

Timezone and locale assumptions. Timestamps arriving without an offset are ambiguous. For services used across the Gulf and Egypt, where daylight saving rules differ between countries and have changed over the years, require an explicit offset or UTC and convert in one place.

Stale caches. A module-level cache of model outputs or features keeps serving after the model version changes. Include the model version in every cache key.

Mistakes in how teams run the service

• Debug mode in production. Running fastapi dev in a container because it was convenient. It reloads on file changes and is meant for development.
• No limit on concurrency. The service accepts everything, queues grow, and every request times out together. Prefer to shed load with a 503 at saturation.
• One giant router file. Hundreds of lines in main.py make dependencies hard to reason about. Split by capability and attach shared dependencies to routers.
• Changing the response shape without telling anyone. Callers fail in their own systems days later. Version the path (/v1/), and diff the schema.
• Logging whole payloads. It floods storage and may breach privacy or data-residency commitments. Log identifiers and sizes, and sample content deliberately.
• Treating a pickle as data. Loading an untrusted pickled model runs code. Load only artifacts you produced or can verify.
• Copying old tutorials. Examples using on_event, .dict(), or default-value Depends() still run but teach habits the current documentation has moved past.

Practice cards

Short exercises, each with a prompt and a way to check yourself.

Card 1: Prove the stall

Prompt. Write one endpoint that calls time.sleep(2) in async def and one in plain def. Call each three times in parallel while polling /docs. Check. /docs stalls with the first and stays responsive with the second. Write down the sentence that explains why.

Card 2: Count your copies

Prompt. Print os.getpid() in your lifespan and start with --workers 3. Check. You see three lines. Measure the resident memory of each process and multiply by your planned replica count to estimate the total.

Card 3: Swap the model

Prompt. Write a test that replaces the model with a fake through dependency_overrides, inside a fixture that always clears the override. Check. Deliberately fail an assertion and confirm the next test is unaffected.

Card 4: Run the lifespan in a test

Prompt. Write one test using with TestClient(app) as client: that asserts the readiness endpoint returns 200 after startup. Check. Replace the context manager with a bare TestClient(app) and observe the difference in behaviour for your app.

Card 5: Shape an error

Prompt. Define ModelUnavailable, raise it from a dependency when app.state.model is None, and register a handler that returns 503. Check. A test asserts the status code and body; a second asserts that an unrelated exception still returns a bare 500.

Card 6: Break the shutdown

Prompt. Build your image with shell-form CMD, run it, and stop it. Print a line after yield in the lifespan. Check. The line does not appear and the stop takes ten seconds. Switch to exec form and see the line appear promptly.

Card 7: Detect a contract change

Prompt. Commit openapi.json generated from app.openapi(). Add a field to a response model. Check. Your diff test fails, and a reviewer has to regenerate and approve the file.

Card 8: Survive the router tree

Prompt. Search your repository for app.routes and router.routes. Replace each use that enumerates endpoints with app.openapi()["paths"]. Check. The script yields the same list as before and still works when a router includes another router.

Card 9: Bound the input

Prompt. Give a list field max_length, reject non-finite values with a validator, and send a request that violates each. Check. Both return 422 and the model function is never called; confirm with a counter in a fake.

Operational recipes

Check which version you are running

pip show fastapi | head -n 3
python -c "import fastapi; print(fastapi.__version__)"

Log the value at startup so a deployment can be correlated with a behaviour change.

Upgrade safely

# 1. branch, then loosen the pin
#    fastapi[standard]>=0.142.0,<0.143.0  ->  fastapi[standard]>=0.143.0,<0.144.0
# 2. run the full suite, including the OpenAPI diff test
pytest -q
# 3. read the release notes for the Breaking Changes heading of each minor you skip
# 4. deploy to staging, run the container smoke test, then promote

Adjust the range numbers to the version you are actually moving to.

Run in production mode locally

fastapi run app/main.py --port 8000

Use this, not fastapi dev, to reproduce production behaviour. The development command reloads on change and is not meant for serving traffic.

Reproduce a container problem

docker build -t scorer:test .
docker run --rm -p 8000:80 --memory 2g scorer:test
docker stop scorer

Running with a memory cap reproduces out-of-memory behaviour on your machine. Read the Docker guide for the tools that inspect layers and signals.

Inspect a pod that will not become ready

kubectl describe pod <name>        # look for OOMKilled, probe failures, last state
kubectl logs <name> --previous     # logs from the crashed container
kubectl get events --sort-by=.lastTimestamp

Probe failures and OOMKilled are visible in the describe output. If the previous logs end during model loading, suspect probe timing or memory. The Kubernetes guide covers these commands in more depth.

Check the contract

import json
from app.main import app

with open("openapi.json", "w") as f:
    json.dump(app.openapi(), f, indent=2, sort_keys=True)

Commit the output, and in CI compare a fresh dump to the committed file. Sorting the keys keeps the diff stable.

Smoke test a built image

docker run -d --rm --name smoke -p 8000:80 scorer:test
until curl -fsS localhost:8000/readyz; do sleep 1; done
curl -fsS -X POST localhost:8000/predict \
  -H 'content-type: application/json' \
  -d '{"features":[1,2,3,4]}'
docker stop smoke

The until loop waits for readiness, which is the point: a model that takes thirty seconds to load should not fail the pipeline, and a model that never loads should. Add a timeout around the loop in a real pipeline. See the GitHub Actions guide for wiring it into CI.

Find memory per process

ps -o pid,rss,command -C python        # Linux: RSS in kilobytes per process

On macOS use Activity Monitor or ps -o pid,rss,command -p <pid>. Divide by one thousand for megabytes. Take the reading after startup and again after a few hundred requests, because some libraries allocate lazily.

Turn off the docs for a public service

from fastapi import FastAPI

app = FastAPI(docs_url=None, redoc_url=None, openapi_url=None)

If you still need the schema for CI, generate it in a build step with a separate app factory flag rather than exposing the endpoint.

Habits that keep a service healthy

• Read the release notes on every minor. Five minutes against a possible afternoon of debugging.
• Keep inference code framework-free. It tests faster and a batch job can reuse it.
• Name the model version everywhere. Responses, logs, metric labels.
• Change one thing at a time. Model, code, and dependency upgrades in separate deploys, so a regression has one suspect.
• Look at the graphs after each deploy. Latency percentiles, error rate and memory for the first fifteen minutes.
• Write down the capacity numbers. Requests per second per replica and memory per process, from a load test, in the repository.

A checklist before you call a service ready

• No blocking calls inside async def endpoints or dependencies.
• Model loaded in the lifespan, reached through a dependency, version reported.
• Request and response models bounded and separated from internal objects.
• Exec-form CMD, non-root user, secrets outside the image.
• Startup, readiness and liveness probes, with a grace period longer than the slowest request.
• Memory request equals limit, taken from a measurement.
• Tests: fake-model API tests, one real-lifespan test, an OpenAPI diff, error paths.
• FastAPI pinned to a minor range, with release notes read before each bump.
• Authentication on routers, CORS limited, docs protected, limits set at the proxy.
• Request ID in logs, inference time measured separately.

The senior guide in this series picks up from here with failure modes at scale, multi-tenancy, supply-chain trust and when to stop using FastAPI for the model itself.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when FastAPI is not your tool but the platform other teams build their model services on.

Up to now the questions were about making one service correct and fast. From here they change: what happens when forty services share your conventions, what breaks when you upgrade the framework underneath all of them, who is allowed to call what, and how you notice that a model endpoint is quietly burning money. FastAPI is still a 0.x library (the current stable release when this was written is 0.142.2, from 30 September 2026), so ownership includes accepting that every minor version can carry a breaking change.

Where this picks up

I start with the architecture, because every failure mode in this track is a consequence of one fact: a FastAPI process is a single event loop in a single Python interpreter, holding a model in its own memory.

Architecture and failure modes

A request enters a reverse proxy or load balancer, reaches one Uvicorn-based process started by fastapi run, is parsed and validated by Pydantic, passes through your middleware and dependencies, and finally reaches your path operation function. The framework itself is built on Starlette for the web layer and Pydantic for validation and serialization, and it runs on ASGI. None of that is the expensive part. The expensive part is whatever your function does with the model.

There are five failure modes worth naming, because you will meet all of them and each has a distinct signature.

Event-loop starvation. A blocking CPU call inside an async def function holds the loop. Health checks stop answering, the orchestrator marks the pod unhealthy, restarts it, and the restart sheds the very load that caused the problem, which then comes back. The signature is latency that rises for every endpoint at once, including /health, while CPU on that pod is high. The official guidance is unambiguous: write def when the library you call is not awaitable, and use multiprocessing, which in practice means more processes or replicas, for CPU-bound work such as inference.

Threadpool exhaustion. Plain def path operations run in an external threadpool, which has a finite size. A model call that takes two seconds under load will fill that pool, and new requests queue behind it invisibly. The signature is requests that spend a long time before your function starts. Measure queueing separately from handler time; I come back to this in the observability section.

Memory multiplication. Each worker process loads its own copy of the model. A one-gigabyte model with four workers is four gigabytes before a single request arrives, plus working memory per request. The signature is an out-of-memory kill during a traffic spike rather than at startup, because per-request allocations arrive on top of the resident weights.

Slow start and cold replicas. A replica that takes ninety seconds to load weights is not ready for ninety seconds, and if your readiness probe does not reflect that, the load balancer sends traffic to a process that cannot serve it. The signature is a burst of errors on every deploy and every scale-up.

Unclean shutdown. If the container is started with a shell-form CMD, signals do not reach the server, shutdown code in your lifespan never runs, and the FastAPI Docker docs warn about the ten-second delay on stop and recreate in Docker Compose. The signature is dropped in-flight requests during every rollout.

A platform owner's first job is to make these five modes visible and boring: a template that gets the process model right, probes that mean what they say, and dashboards that show saturation before users report it.

Scaling: which component breaks first

Scaling an inference API is a question of which resource saturates first, and the answer depends on the model, not the framework. Walk the chain in order.

CPU cores per process. Python executes one thread of bytecode at a time per process, and many numerical libraries release the interpreter lock inside their native code. So a single process can sometimes use several cores during a prediction, and sometimes cannot. Do not assume either. Measure with a load test at one process and watch core utilisation. If the process uses roughly one core, you scale by adding processes. If it uses many, adding workers on the same machine causes them to fight over the same cores, and throughput can fall.

Memory per process. This usually breaks before CPU for any model larger than a few hundred megabytes. The official deployment guidance for clusters is one process per container with no --workers flag, letting the cluster replicate and balance. In that model you set memory requests and limits from measured per-container use, and the scheduler packs replicas onto nodes. On a single server or under Docker Compose, --workers 4 is acceptable, and then the multiplication is yours to account for.

# Cluster: one process per container, scale with replicas
fastapi run app/main.py --port 80

# Single host: multiple processes share the machine
fastapi run app/main.py --port 80 --workers 4

The accelerator. If the model runs on a GPU, one process per GPU is the usual rule, because two processes holding the same model on one device double the device memory and contend for it. When a team needs continuous batching, tensor parallelism or quantised kernels, the right answer is often a dedicated serving runtime behind FastAPI rather than reimplementing that inside request handlers. That is an engineering judgement, not an official FastAPI statement, and I treat it fully in the section on where FastAPI stops.

Downstream dependencies. Feature stores, vector databases, and object storage all have connection and rate limits. A hundred replicas each opening ten connections is a thousand connections. The failure appears as sudden timeouts on a dependency that looked healthy. Pool sizes, timeouts, and retry budgets belong in the platform template, not in each team's copy.

The proxy and the network. Request body size, idle timeout, and buffering are set at the proxy. FastAPI does not enforce an upload limit by itself, so an unbounded body can consume memory before your code sees it unless the edge enforces a cap.

To make this concrete, a useful capacity model has three numbers per model service: resident memory per process after warm-up, requests per second per process at your latency target, and the startup time to ready. From those you compute replicas for peak, memory per node, and how fast you can react to a spike. If startup time is two minutes, autoscaling reactively cannot absorb a sudden doubling, so you either keep headroom or scale on a leading indicator such as queue depth.

Autoscaling on CPU alone is misleading for I/O-heavy feature-fetching services and for GPU-bound ones. Prefer a signal tied to the thing that saturates: in-flight requests per replica, or queue wait time. Expose it as a metric and let the autoscaler read it. The Kubernetes guide covers the autoscaling mechanics, and KServe is the usual next step when you want that behaviour packaged for models.

The model lifecycle: lifespan, memory and reloads

The lifespan context manager is where the model is loaded, and the official documentation uses exactly this pattern for machine-learning models: code before yield runs at startup, code after it at shutdown. The older @app.on_event("startup") and @app.on_event("shutdown") decorators are deprecated, and if you pass lifespan= the on_event handlers are not called at all. On a platform, that second fact is a migration hazard: a service that mixes both will silently lose its old startup code the day someone adds a lifespan.

For a platform, the lifespan is not just where one model loads. It is the contract for the readiness of the whole service. A good shared lifespan does four things in order: validate configuration and fail fast with a clear message, load the model artifact from a controlled location, run a warm-up inference so the first real request is not the slowest, and only then let the server accept traffic.

from contextlib import asynccontextmanager
from fastapi import FastAPI

def build_lifespan(load_model, warm_up):
    @asynccontextmanager
    async def lifespan(app: FastAPI):
        model = load_model()          # may be slow: weights, tokenizer
        warm_up(model)                # first call is often the slowest
        app.state.model = model
        yield
        app.state.model = None        # release before the process exits
    return lifespan

Because the factory takes the loader as an argument, each team supplies its own model and the platform supplies the sequence. A service becomes FastAPI(lifespan=build_lifespan(load, warm)), and no team can forget the warm-up.

Reloading a model. Teams will ask to swap a model without restarting. Resist the in-process hot swap unless you must have it. Loading a new model while the old one is still resident briefly doubles memory, and a failed load can leave the process in a state nobody tested. The robust pattern is the one your cluster already provides: roll out new replicas with the new model, let readiness gate traffic, and retire the old ones. If you do need an in-process swap, load the replacement fully, validate it with a smoke input, replace the reference atomically, and only then release the old one.

Where artifacts come from. Loading from a network location at every start couples your availability to that location. Decide per platform whether weights are baked into the image, mounted from a volume, or pulled by an init step, and write the decision down. Baking weights makes images large and rebuilds slow; pulling them makes starts slow and fragile. A common compromise is a versioned artifact in a registry, pulled by an init container onto a shared volume, with the application only ever reading a local path. The MLflow guide covers the registry side of that, and BentoML is an example of a tool that packages the model and its API together.

Graceful shutdown. The code after yield runs only if the process receives a signal and shuts down cleanly. That is why the exec-form CMD ["fastapi", "run", ...] matters: the server is process one in the container and receives the termination signal directly. Pair it with a termination grace period longer than your longest request, and make the readiness probe fail first so the load balancer stops sending new work before the process stops.

FROM python:3.14
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
COPY ./app /code/app
CMD ["fastapi", "run", "app/main.py", "--port", "80"]

This is the shape the official Docker documentation shows. Copying the requirements file before the application code keeps the dependency layer cached. The deprecated tiangolo/uvicorn-gunicorn-fastapi base image should not appear in any template you own; build your own image instead. The Docker guide goes into hardening that image.

The router tree and your shared library

Release 0.137.0, published on 14 June 2026, was a breaking change to how routers compose. Before it, router.include_router(other) cloned the routes of the included router into the parent. Now it keeps the original APIRouter and APIRoute objects, and router.routes is no longer a flat list of APIRoute instances. It can contain intermediate objects that hold further routers, which forms a tree. The release notes describe several consequences: routes added to a router after it has been included are reflected in the parent, memory use can fall, and new experimental .matches() and .handle() hooks exist on APIRouter that are not supported for general use. The notes also say the change is a foundation for per-router dependencies, exception handlers and middleware later; treat that as direction rather than a feature you can use today.

For a platform owner, the practical question is what in your estate assumed a flat list. Typical offenders are:

• custom documentation generators that walk app.routes to list endpoints,
• route-introspection code that builds an access-control table from the registered paths,
• middleware or startup hooks that mutate router.routes to add, remove or wrap routes,
• test helpers that count or filter routes.

Find these with a search for .routes across your repositories before you upgrade past 0.136, and replace them with something that does not depend on the internal structure. The most stable replacement for introspection is the generated OpenAPI schema, which is a public output: it lists every path, method and parameter regardless of how routers are nested. If you need an access-control inventory, derive it from OpenAPI plus a custom extension field rather than from router internals.

from fastapi import FastAPI

def endpoint_inventory(app: FastAPI) -> list[tuple[str, str]]:
    """Public, structure-independent list of (method, path)."""
    schema = app.openapi()
    out = []
    for path, item in schema["paths"].items():
        for method in item:
            out.append((method.upper(), path))
    return sorted(out)

That function survives router refactors because it reads the schema, not the tree. Use it in a test that fails when an endpoint appears or disappears without an accompanying change to an approved list; it doubles as a cheap guard against accidentally exposing an internal route.

Two further rules follow from the change. First, do not mutate router.routes in platform code; if you need cross-cutting behaviour, use dependencies, middleware or an exception handler. Second, include each router once. The 0.142.1 release fixed a bug where endpoints were wrapped repeatedly in included routers, which is a reminder that composition code is a place where regressions hide, so your upgrade tests should exercise nested includes.

For your shared library, a layout that works well is a small core package that owns the application factory, the lifespan factory, the auth dependencies, the error format and the telemetry setup; and a thin per-team package that owns routers and model code. Teams build a router and hand it to the factory.

from fastapi import APIRouter, FastAPI

def create_app(routers: list[APIRouter], lifespan, *, public_docs: bool = False) -> FastAPI:
    app = FastAPI(
        lifespan=lifespan,
        docs_url="/docs" if public_docs else None,
        redoc_url="/redoc" if public_docs else None,
        openapi_url="/openapi.json" if public_docs else None,
    )
    for router in routers:
        app.include_router(router)
    return app

The factory is the single place that decides whether interactive documentation is exposed, so that choice is a platform default and not an accident of each team's copy. Internal environments can switch it on; public ones leave it off.

Dependencies as the extension surface

The dependency system is the mechanism through which a platform adds behaviour to other teams' endpoints without forking their code. Dependencies are functions, can be sync or async and mixed freely, can have sub-dependencies forming a tree, and are declared with Annotated and Depends. You pass the function itself to Depends, never a call of it.

The most valuable platform dependencies are:

• Authentication, which turns a credential into a verified principal.
• Tenant resolution, which turns the principal into a tenant context.
• Quota or rate checking, which rejects work before it reaches the model.
• Model access, which hands the endpoint the right model object.
• Request identity, which attaches a request ID used in logs and traces.

Publishing these as type aliases gives teams one line to adopt each.

from typing import Annotated
from fastapi import Depends, Header, HTTPException, Request

async def current_principal(authorization: Annotated[str | None, Header()] = None):
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="Missing bearer token")
    return await verify_token(authorization.removeprefix("Bearer "))  # your IdP check

def get_model(request: Request):
    return request.app.state.model

Principal = Annotated[dict, Depends(current_principal)]
Model = Annotated[object, Depends(get_model)]

A team endpoint now reads async def predict(body: Body, who: Principal, model: Model), and the platform controls what those two names mean. In tests, app.dependency_overrides replaces them, which is why dependencies beat module-level globals: your contract tests can substitute a fake principal and a stub model without a real identity provider or a real model.

There is a design discipline here. A dependency is a public API of your platform, so changing its signature is a breaking change for every team. Version them: add a new dependency rather than editing the old one, deprecate the old one with a warning and a date, and measure usage before removal. Dependencies with side effects, such as quota decrements, need particular care because FastAPI caches a dependency's result within a single request, which is normally helpful but means you must not rely on a dependency executing twice.

One more consideration is cost. Dependencies run on every request, so a dependency that makes a network call to validate a token or check a quota adds that latency to everything. Cache verification results with a short, bounded lifetime where the security model allows, and make the cache a platform concern so that teams do not each invent one.

Security: the trust model

Start from a trust model, because individual controls only make sense against one. A FastAPI model service has at least four boundaries: the network edge, the identity of the caller, the content of the request, and the artifacts the service loads.

The edge. TLS is normally terminated at a proxy or load balancer, and the application is told that it is behind one with --proxy-headers, so that generated URLs and redirects use the correct scheme. That flag makes the server trust forwarded headers, so it must only be enabled when the proxy is yours and clients cannot reach the application directly; otherwise a client can forge those headers. Restricting which upstream addresses are trusted is possible through a forwarded-allow-IPs setting of the underlying server, but I have not confirmed its exact spelling in the FastAPI CLI, so check the current CLI help before relying on it. Network policy should also ensure the pod is reachable only from the proxy.

The caller. Authentication in the official tutorial path is OAuth2 with JWT through fastapi.security, which provides helpers such as OAuth2PasswordBearer and HTTPBearer. The framework supports the OpenAPI security schemes: API keys in a query string, header or cookie; HTTP bearer, basic and digest; OAuth2 flows including client credentials and authorization code; and OpenID Connect. For a platform serving other internal teams and services, the right choice is almost always to validate tokens issued by your existing identity provider instead of issuing your own. The service verifies signature, issuer, audience and expiry, and reads scopes or roles from claims. It does not store passwords. The tutorial's password-flow example is a teaching device, not a template for production identity.

Be explicit about what the service must never trust. Do not trust a tenant identifier supplied in the request body or a custom header if it can instead be derived from the verified token. Do not trust the client's claim about which model to run without checking that the caller is entitled to it. Do not trust the sizes and shapes of inputs: validate lengths, array dimensions and file sizes in the schema, because a model will happily try to process an enormous payload.

The content. Pydantic validates types, but not intent. Constrain fields with bounds, maximum lengths and enums. For models that accept free text and feed a language model, input is an injection surface; that is covered in the LLM guides, and the point here is that the API layer is where you enforce size and rate limits, not the model. Return errors that do not leak internals: a stack trace in a response body tells an attacker your library versions.

The artifacts. Your service executes whatever is in its dependency tree and loads whatever model file you give it. Many model formats can execute code on load, so a model file from an untrusted source is untrusted code. Load only from locations your platform controls, verify checksums or signatures of artifacts where you can, and prefer data-only formats when your framework supports them. For dependencies, pin versions, use a lock file, scan images, and build from a minimal base. Trivy is the scanner this catalogue covers, and the Docker guide covers signing and provenance.

Secrets and configuration. The documented approach is environment variables read through Pydantic settings. Never bake secrets into the image, and never log them. In Kubernetes, mount them from a secret store or a secret manager; Vault is covered in this catalogue. Treat any setting that disables a security control, such as relaxed CORS, as something that needs a reviewed exception.

CORS is a browser control, not an access-control mechanism. CORSMiddleware with a narrow list of allowed origins protects browsers; it does nothing against a script on a server. Never combine a wildcard origin with credentials.

Multi-tenancy

FastAPI has no built-in tenancy, so your platform must decide what a tenant is and where isolation is enforced. There are three levels, and the right one depends on what you can afford to lose if isolation fails.

Shared process, logical isolation. All tenants hit the same replicas. A dependency resolves the tenant from the verified token, and every data access and every cache key includes the tenant. This is the cheapest and the most dangerous: one missing filter leaks data across tenants, and one heavy tenant slows everyone. It is acceptable when tenants are teams in one company and the data is not sensitive across them.

Shared cluster, separate deployments. Each tenant, or each class of tenant, gets its own deployment, possibly its own model version, with its own resource limits and its own network policy. This costs more memory because models are duplicated, and it isolates both failure and noisy neighbours. It is the right default for external customers or regulated data.

Separate clusters or accounts. The strongest separation, usually driven by data-residency rules. For Gulf and Egyptian employers, some data may be required to stay inside a national or regional boundary, which can force region-specific deployments regardless of what is convenient. Design your deployment artifacts so that the region is a parameter, not a fork.

Within a shared process, the hard problem is per-tenant models. If each tenant has a fine-tuned model, you cannot keep them all resident. Use a bounded cache with least-recently-used eviction, size it from measured memory, and make the cold path explicit: a tenant whose model is not resident pays a load time, and you should return a clear status or queue rather than blocking a worker for a minute.

from collections import OrderedDict
from threading import Lock

class ModelCache:
    def __init__(self, loader, max_models: int):
        self._loader, self._max = loader, max_models
        self._items: OrderedDict[str, object] = OrderedDict()
        self._lock = Lock()

    def get(self, tenant_id: str):
        with self._lock:
            if tenant_id in self._items:
                self._items.move_to_end(tenant_id)
                return self._items[tenant_id]
        model = self._loader(tenant_id)          # slow; outside the lock
        with self._lock:
            self._items[tenant_id] = model
            self._items.move_to_end(tenant_id)
            while len(self._items) > self._max:
                self._items.popitem(last=False)
        return model

The slow load happens outside the lock so one cold tenant does not block the others, at the cost that two simultaneous cold requests for the same tenant may both load; for large models, add a per-key guard. Remember the cache is per process, so with several workers each holds its own set, and the eviction budget must be computed per process.

Fairness matters as much as isolation. A single tenant sending a burst can saturate the threadpool. Enforce per-tenant concurrency and rate limits in a dependency, and shed load early with a clear 429 rather than queueing unboundedly. Queues without bounds turn overload into latency and then into timeouts that clients retry, which multiplies the load.

The 0.137.0 router refactor is described as groundwork for per-router dependencies, which would make tenant scoping cleaner by attaching it to a router rather than to each endpoint. That is future direction; until it ships and you have tested it, attach shared dependencies explicitly.

Observability: OpenTelemetry and what to measure

Release 0.142.0 on 29 September 2026 added native OpenTelemetry support, and 0.142.2 the next day made the application start even if the automatic OpenTelemetry configuration fails. The release notes available when this guide was written do not describe how to enable it or which settings it reads, and I have not verified the configuration surface. Before you build a platform standard around it, read the pull request (number 16403 in the FastAPI repository) and the current documentation, and test it with your exporter. The OpenTelemetry guide covers the concepts that apply either way.

What I can say with confidence is the design stance. Telemetry that is enabled by the framework is only useful if it carries identity. Whatever mechanism you use, make sure each request has a request ID that appears in logs and in spans, and that spans carry a tenant identifier and a model name and version as attributes. Be careful with cardinality: put tenant and model in traces and logs freely, but be selective about metric labels, because a label with thousands of values multiplies time series and cost.

The 0.142.2 behaviour is worth designing around: telemetry failure must never take the service down. That is the right principle for your own instrumentation too. Wrap exporters so that a broken collector degrades to dropped data, not to failed requests.

The signals that matter for a model service, in rough order of value, are these.

1. Request latency by endpoint and status, as a histogram, not an average. Averages hide the tail that users feel.
2. Inference time separately from total time. The difference is queueing, serialization and network. If total rises while inference is flat, the problem is the platform, not the model.
3. In-flight requests and threadpool or queue wait. This is your saturation signal and the best autoscaling input.
4. Process memory, resident and over time. A slow upward slope means a leak or an unbounded cache.
5. Model load time and readiness time, to catch slow starts before they hurt a rollout.
6. Input and output statistics feeding drift detection. The Evidently guide covers what to do with them, and the API layer is where you capture them cheaply.
7. Error classes: validation failures (422), authentication failures, downstream failures and internal errors, kept separate, because a spike in 422s after a client release is a contract problem, not an outage.

Prometheus metrics libraries for FastAPI are third-party, not part of the framework; the Prometheus guide and Grafana guide cover the consuming side. Choose one approach per platform and standardise the metric names, because dashboards and alerts written against one team's names are useless for the next.

Logs should be structured, one event per line, with the request ID, tenant, model version, status and duration. Never log full request bodies by default: they contain personal data and they are large. Log a hash or a size, and offer an explicit, short-lived debug mode with redaction for investigations.

Cost

FastAPI itself contributes very little to the bill; the model and the way you provision it dominate. Cost on a platform has five levers.

Memory multiplication. Workers times model size times replicas is the largest hidden cost. Right-size from measured resident memory, not from a guess, and recheck after each model change, because a new model version is a new memory profile. If you can run one process per container and a measured memory request, the scheduler packs nodes efficiently.

Idle capacity. A model service with steady low traffic still holds its memory. Consolidate small models into shared services where the isolation level allows it, and scale rarely used services down when your startup time permits. A service that takes two minutes to load cannot scale to zero without a user waiting for it.

Accelerators. A GPU held by a service that uses it ten percent of the time is the most expensive waste you will find. Batch requests where latency allows, share a device among models through a serving runtime when appropriate, and measure utilisation, not allocation.

There is no official FastAPI cost guidance beyond the deployment-concepts material on replication and memory, so the numbers are yours to gather. A practical report per service has four columns: monthly infrastructure cost, requests served, cost per thousand requests, and utilisation. The ratio between peak and average load tells you how much of the bill is headroom. Put that report where service owners see it monthly; behaviour changes when the number has a name next to it.

Upgrades and migrations

Because FastAPI is 0.x, the versioning page tells you that minor version bumps can contain breaking changes and patch bumps are bug fixes only. Version 1.0.0 is reserved for the future and does not exist yet. That makes the upgrade policy a core platform deliverable, not an afterthought.

Pin the minor version. Pin with a range whose lower bound is 0.142.0 and whose upper bound excludes 0.143.0, as in the block below. That accepts patch fixes automatically and requires a deliberate act to take a new minor. The documentation's procedure is to write tests, upgrade, run the tests, fix what fails, then pin the new version. Do not pin Starlette yourself; FastAPI selects a compatible version, and the 0.142.0 release moved from Starlette 1.3.1 to 1.6.0 along with a Typer minimum bump to 0.26.1. Pydantic can be pinned with a range too, as in the block below.7.0 up to, but excluding, 3.0.0`.

fastapi[standard]>=0.142.0,<0.143.0
pydantic>=2.7.0,<3.0.0

Read the release notes by heading. Before each minor, read the entries for a breaking-changes heading. For this track, the ones I could confirm are: 0.100.0 (7 July 2023), the full move to Pydantic v2; 0.115.0 (17 September 2024), which added parameter models for query, header and cookie parameters; 0.137.0 (14 June 2026), the router tree; and 0.142.0 (29 September 2026), native OpenTelemetry. I did not retrieve the details of 0.112.0 or of the 0.138 to 0.141 releases, so read them yourself before you plan a jump across that range.

Run a canary cohort. Pick three or four representative services, upgrade them first, and let them run in production for a week. The cohort should include one service with heavy router nesting, one with custom middleware, one with streaming responses, and one with an unusual dependency graph. If the upgrade is clean there, release it to the rest through the shared library.

Own a compatibility test suite. It is a small set of services, or one synthetic service, that exercises every platform contract: the lifespan factory, the auth dependency, the error format, the endpoint inventory, graceful shutdown, and telemetry. Run it on every FastAPI release, including patches, in CI. The goal is to learn about a breaking change from a red build, not from a customer.

# In CI: a scheduled job that tests the next release against your contract suite
pip install "fastapi[standard]>=0.142.0" pytest httpx
pytest tests/platform_contract -q

Run it without the upper bound to learn what the next minor would do to you, while production stays pinned.

Deprecations on your own surface. Remove on_event handlers, deprecated base images, and Pydantic v1 style calls. In Pydantic v2 the method names are model_dump() and model_dump_json(), and configuration is model_config. Teams on old style code should be migrated with a codemod or a lint rule, with a deadline, not left to drift. A lint rule in the shared CI template that fails on on_event is cheap and effective.

Governance: contracts, versions and standards

A platform without governance becomes forty different frameworks that happen to share a name. Governance does not need a committee. It needs a small number of enforced defaults and a clear path to an exception.

The OpenAPI schema is the contract. FastAPI generates it automatically, so use it. Store the schema for every service in version control or an artifact registry, and run a diff on every change that classifies it: adding an optional field is compatible, removing a field or making an optional one required is breaking. Block breaking changes without a version bump. Consumers can generate clients from the schema, so keeping it honest keeps them honest.

API versioning. Choose one scheme, such as a path prefix, and apply it everywhere. With routers it is natural: a v1 router and a v2 router in the same application, sharing model code. Define the deprecation window up front, and use the response header and documentation to announce it. Log the version used by each caller so you know when it is safe to remove the old one.

Error format. Agree one error shape across services. FastAPI returns a 422 with a detail list for validation failures, and HTTPException produces a detail that is a string; clients that handle both differently are a common source of bugs. A platform exception handler that converts all errors to one documented envelope, with a stable machine-readable code, saves every consumer from special cases.

Model version in the response. Return the model name and version in a header or a field. Without it, debugging a bad prediction means guessing which version was serving at the time. With it, a support ticket carries its own evidence.

Exceptions. Every rule needs a path to an exception with an owner and an expiry date. Rules with no exception path get bypassed silently, and then you have no record of who deviated or why.

Incident playbooks

These are the incidents you will see, with the order in which to look. For each, the first move is to stop the damage, the second is to find the cause, and the third is a change that prevents recurrence.

Latency rises on every endpoint, including health checks. Suspect event-loop starvation. Check whether a recent change added blocking code inside an async def, whether a synchronous library was called without being offloaded, or whether inference moved into the loop. Mitigate by rolling back, or by scaling replicas to spread the load. Fix by moving the work to a plain def function or a process pool, and add a test that fails on a blocked loop. Watch for the restart storm: if probes share the loop, the orchestrator may be making it worse.

Memory climbs until pods are killed. Distinguish a leak from a steady footprint. A footprint that jumped after a deploy points to a new model or a new worker count. A slow slope points to an unbounded cache, a growing global list, or a request-scoped object that is retained. Mitigate with a lower request concurrency or more replicas with a memory limit. Fix by bounding the structure and adding a memory alert on slope.

Errors on every deploy and scale-up. Slow start with a readiness probe that is not tied to the lifespan. Confirm that readiness only passes after the model is loaded and warmed. Mitigate with a slower rollout and a minimum ready time. Fix by making readiness reflect the lifespan.

A spike of 422s after a client release. A contract change. Compare the schema against the last known good one and check the validation loc field of the errors to find the offending parameter. Mitigate by restoring compatibility on the server, such as accepting both forms. Fix by adding the schema diff gate to CI.

Cross-tenant data appears in a response. Treat this as a security incident. Stop the affected endpoint, preserve logs, identify which dependency or cache key lacked the tenant, and notify according to your policy. The fix is a tenant-scoped key and a test that issues requests as two tenants and asserts they never see each other's data.

Upgrade broke production. Roll back to the pinned previous minor, which is why the previous version must stay available in your artifact registry. Read the release notes for the breaking-changes entry matching the failure. If it is the 0.137.0 router tree, search for code that touches router.routes.

Shutdown drops requests. Confirm exec-form CMD, a grace period longer than your slowest request, and a readiness probe that fails before termination. Check that the lifespan shutdown code finishes within the grace period.

Write each of these as a short runbook in the same format: symptom, first three checks, mitigation, owner. Run a game day once a quarter where someone injects one of them in a staging environment, and measure how long it takes to find the cause.

Where FastAPI stops

A good platform owner also knows when not to use the tool. FastAPI is an excellent general-purpose HTTP layer around Python code. It is not a model-serving runtime, and pretending otherwise leads to reinventing features badly.

Dedicated inference runtimes. When you need dynamic batching, concurrent model execution on one GPU, model versioning with traffic splitting, or optimised kernels, a runtime built for that does it better. Triton and vLLM are examples in this catalogue. A common and healthy architecture keeps FastAPI as the thin, authenticated, observable front door and sends the heavy computation to such a runtime over the network. Note that this is an engineering judgement, not an official FastAPI recommendation.

Model-serving platforms. If many teams need standard deployment, scaling, canary rollouts and a model registry integration, packaging that in a platform designed for it can be cheaper than building it. KServe does this on Kubernetes, Ray Serve does it for Python workloads with composition and autoscaling, and BentoML packages a model with its service. Each of those can sit alongside FastAPI services in the same estate. Choose by who runs it: a framework that your team already operates well beats a better one it does not.

Long-running and batch work. Request handlers should not run training, large batch scoring or long jobs. FastAPI's BackgroundTasks suit only small fire-and-forget work, since they share the process and die with it. Use a queue and a separate worker service, and return a job identifier that the client polls or receives a callback for.

Heavy distributed compute. If the work is a distributed computation, Ray is the better tool, and FastAPI becomes the entry point that submits work to it.

A decision rule I use: if the question is "how do we expose this function safely over HTTP", it is FastAPI; if the question is "how do we run this model efficiently at scale", it is a serving runtime with FastAPI in front only if you need custom logic. If you cannot say which question you are asking, you are about to build a platform inside a request handler.

Running it as a platform for other teams

The technical choices above only work if teams actually adopt them. Running FastAPI as a platform is mostly a product problem: make the right path the easy path, and make the cost of leaving it visible.

Ship a paved road. A project template that generates a service with the lifespan factory, auth dependency, error envelope, telemetry, Dockerfile, CI pipeline and health endpoints. A new service should be deployable in an hour with no platform involvement. Teams that start from the template inherit every default; the platform's influence over the estate is proportional to how many services began there.

Version the template and the library separately. The library is imported and upgraded; the template is copied and drifts. Keep as much as possible in the library, so a fix reaches everyone through a version bump, and keep the template thin. Track which services are on which library version, and publish a dashboard so that stragglers are visible.

Support model. A channel, an office hour, and a rotation. Record questions, because repeated questions are documentation or template bugs. A platform that answers the same question twice has a defect.

Measure adoption and health. Useful numbers: services on the supported minor, services using deprecated APIs, mean time from release to canary, share of services with a working readiness probe, and percentage of endpoints covered by contract tests. Review them monthly. They tell you where the next piece of platform work should go.

Deprecation is a feature. Announce, warn at runtime, measure usage, remove. When you remove something, do the migration work for the long tail yourself where it is cheap, because a deprecation that waits on forty teams' backlogs never completes.

Security response. When an advisory lands for FastAPI, Starlette, Pydantic or Uvicorn, you need to know within a day which services are affected and be able to roll a patched library version. That requires a software inventory: a list of every service, its image digest and its dependency lock file, queryable. The first time you need it is a bad time to discover you do not have one.

The review checklist

Use this for design documents and pull requests that touch a model service or the platform library.

Process and scaling

• One process per container in a cluster, or a justified worker count on a single host, with memory computed as workers times model size.
• No blocking inference inside async def; plain def, or a process-based path, for CPU-bound work.
• Measured resident memory, requests per second per process and time to ready are recorded.
• Autoscaling uses a saturation signal, not only CPU.

Lifecycle

• Model loaded in lifespan, with warm-up, and no on_event handlers remain.
• Readiness passes only after load; liveness is cheap.
• Exec-form CMD, a grace period longer than the slowest request, and clean release after yield.
• Artifacts come from a controlled location and are verified.

Security

• Tokens validated against the identity provider for signature, issuer, audience and expiry.
• Tenant derived from the verified token, never from the body.
• Docs and OpenAPI endpoints disabled on public deployments.
• CORS origins narrow; proxy headers trusted only behind a proxy you control.
• Secrets from a secret store; none in the image, repository or logs.
• Input sizes and shapes bounded in the schema; body size limited at the proxy.

Contract and governance

• The OpenAPI schema diff is clean, or the version was bumped.
• One error envelope; model name and version returned.
• No code touching router.routes; endpoint inventory test passes.
• Deprecated style, such as .dict() and class Config, removed.

Operations

• Latency histogram, inference time, in-flight requests, memory and error classes exported.
• Request ID in logs and traces; no request bodies logged by default.
• Runbook exists for the three incidents most likely for this service.
• FastAPI pinned to a minor version with an upgrade owner.

Cost

• Replica count and memory request justified by measurement.
• Cost per thousand requests reported to the owning team.

The complete picture

Taken together, the three levels describe a progression. At Beginner you made an endpoint that returned a prediction. At Mid you made it correct under load: the lifespan, the async and threading rules, dependencies and tests. At Senior you made it a product other teams depend on.

The thread through all of it is that FastAPI is deliberately thin. It gives you typed parsing, dependency injection, automatic OpenAPI and a clean async story, and it leaves the hard problems to you: memory is per process, inference is computation that the event loop cannot hide, identity belongs to your identity provider, and a 0.x version number is a promise that things will change. A platform is the set of decisions that absorbs those truths once so that forty teams do not each rediscover them.

The most important habits are the unglamorous ones. Pin the minor. Test the upgrade before it reaches production. Make readiness mean ready. Derive the tenant from the token. Keep the schema as the contract and diff it. Measure memory, saturation and cost per request, and show them to the people who own the service. Write runbooks before the incident. And know where FastAPI stops, so that you send the heavy work to a runtime built for it instead of rebuilding that runtime in a request handler.

Where the series leaves you

You can now design, secure, scale and govern a FastAPI-based serving platform, and argue for those choices in a design review. The sensible next steps are the neighbours in this catalogue. Learn Docker hardening and Kubernetes operations in depth, since that is where your pods actually live. Look at KServe, Ray Serve and BentoML to see what a purpose-built serving layer gives you that a general framework does not. Add OpenTelemetry, Prometheus and Grafana to make the signals above real, and MLflow for the artifact and registry side.

Three things to do this week. Search your repositories for router.routes and on_event. Write down your upgrade policy in one page, including who owns it and how a minor release is canaried. And pick one service, measure its memory, saturation and cost per thousand requests, and put the numbers in front of its owners.

Sources

• FastAPI documentation home
• FastAPI tutorial: installation, running and the first steps
• FastAPI release notes
• FastAPI deployment overview
• FastAPI in containers with Docker
• About FastAPI versions
• Lifespan events
• Concurrency and async / await
• Dependencies
• Security
• Testing
• FastAPI 0.137.0 release
• FastAPI 0.142.0 release

Part three of three. Senior interviews for a platform role rarely ask what a decorator does. They hand you an ambiguous situation and watch how you structure it: what you ask first, what you measure, what you trade away, and how you explain the decision to someone who does not care about Python. This file gives you system-design prompts, incident questions and the leadership arguments, each with a model answer and a note on what separates a strong answer from a weak one.

A reminder on the version: FastAPI is a 0.x library, with 0.142.2 current when this was written, so minor releases can carry breaking changes. Almost every good answer below touches that fact somewhere.

The thirty-second answer

If you are asked "what does owning FastAPI as a platform mean", say this: "FastAPI is a thin, typed HTTP layer on Starlette and Pydantic. It gives you validation, dependency injection and automatic OpenAPI, and it leaves memory, concurrency and identity to me. Owning it as a platform means turning those into defaults: a template with a correct lifespan and probes, shared auth and tenancy dependencies, the schema as a versioned contract, a pinned-minor upgrade policy tested against a contract suite, and telemetry that shows saturation. And it means knowing where to stop and put a serving runtime behind it."

That answer has the shape interviewers want: it names what the tool does, what it refuses to do, and what you build around the gap.

System design: a shared model-serving platform

Prompt: "Twenty teams want to expose models over HTTP. Design the platform."

Start by asking questions, because the right design depends on the answers. How many models and how large? Latency targets? Are callers internal or external? Is any data subject to residency rules? What do teams run on today, and who is on call? Who pays for compute? A candidate who starts drawing boxes before asking these has not done this before.

Then propose layers. At the base, a container image built from a controlled base, running fastapi run in exec form, one process per container, with replicas providing scale. On top, a shared library holding an application factory, a lifespan factory that loads, warms and only then serves, authentication and tenant dependencies, one error envelope, telemetry setup, and a request ID. Teams add routers and model code. A template generates the repository, the Dockerfile, the CI pipeline and the probes. The schema is exported on every build and diffed against the last release.

Cover the questions interviewers probe. On scaling, say you load test one process to get resident memory, throughput at the latency target and startup time, then compute replicas, and scale on in-flight requests or queue wait rather than CPU alone. On isolation, say you start with logical isolation inside a shared deployment for internal teams, and move to separate deployments for sensitive data, with the region as a deployment parameter. On security, say tokens are validated against the existing identity provider, tenants come from the token, documentation endpoints are off in public environments, and artifacts are loaded only from controlled storage. On upgrades, say you pin the minor version and canary a cohort.

A strong answer ends with the boundary: "For teams that need continuous batching or GPU sharing, I would put a dedicated inference runtime behind the FastAPI front door instead of building that into handlers." A weak answer proposes FastAPI for everything, or alternatively proposes a heavyweight serving platform on day one with no reason.

System design: one slow model behind a busy API

Prompt: "A single endpoint runs a two-second model call. Traffic is three hundred requests a second at peak. How many pods?"

Do not guess a number. Say that two seconds per call at one concurrent call per process gives half a request per second per process, so three hundred per second needs six hundred processes if each handles one call at a time, which tells you immediately that either the call can be batched, parallelised across cores, or accelerated. Ask whether the two seconds is CPU time on one core, whether the library releases the interpreter lock, and whether requests can be batched. Then explain that measurement drives the decision: if one process uses several cores during a call, fewer processes with more cores each may serve the same load; if it is GPU-bound, batching is the lever and a runtime with dynamic batching is the likely answer.

Mention memory: six hundred processes times the model size is the real constraint, which is why you would consider a serving runtime that holds one copy of the weights and serves many concurrent requests. Mention headroom for the startup time, since reactive autoscaling cannot absorb a spike if each replica takes two minutes to be ready.

A strong answer converts the vague prompt into measurements and names which of them would change the architecture. A weak answer divides numbers without questioning whether one request per process is the right model.

System design: multi-tenant fine-tuned models

Prompt: "Each of five hundred customers has their own fine-tuned model. Serve them."

The core problem is that five hundred resident models do not fit. Propose a bounded cache of models with least-recently-used eviction in each process, sized from measured memory, and say that the cold path must be explicit: a tenant whose model is not resident pays a load time, so either return a clear status and queue the request, or route tenants consistently to the same replica so caches stay warm. Note that the cache is per process, so total capacity is processes times cache size, and consistent routing improves the hit rate.

Add isolation. The tenant identity is derived from the verified token, every cache key and data access includes it, and a test issues requests as two tenants and asserts nothing leaks. Add fairness: per-tenant concurrency limits and fast 429 responses so a single customer's burst cannot take the threadpool. Add observability: model name, version and tenant in traces, with care about metric label cardinality.

Then give the alternative. If the models are small adapters on a shared base model, serving the base once and swapping adapters is far cheaper than five hundred full models, and that is a job for a runtime designed for it. Say that you would ask how the models differ before committing.

A strong answer finds the memory arithmetic and the cold-start cost on its own. A weak answer keeps a dictionary of all models and does not notice until memory runs out.

System design: zero-downtime upgrade of the framework

Prompt: "FastAPI releases a new minor version. Roll it out to sixty services."

Open with the fact that FastAPI is 0.x and minor versions can contain breaking changes, so this is a managed change, not a dependency bump. Describe the sequence. A scheduled job in CI installs the next release without an upper bound and runs a contract test suite covering the lifespan factory, auth, error envelope, endpoint inventory from the OpenAPI schema, graceful shutdown and telemetry. If it passes, bump the shared library, which pins the minor. Release first to a canary cohort chosen for variety: one with heavily nested routers, one with custom middleware, one streaming, one with an odd dependency graph. Hold for a week, then roll to the rest in waves, with the previous library version retained for rollback.

Give a real example. Release 0.137.0 changed include_router so that the original router and route objects are kept and router.routes can hold a tree rather than a flat list. Code that iterated or mutated router.routes could break. The way to find it is to search for .routes across repositories, and the way to prevent the problem is to use the OpenAPI schema for introspection instead of router internals.

A strong answer includes rollback and the canary composition. A weak answer says "run the tests" without saying whose tests or against what.

Incident response: every endpoint slowed at once

Prompt: "At 14:05 latency rose on all endpoints, including health checks. CPU is high. What do you do?"

Say the signature points to event-loop starvation, because health checks share the loop with the work. First, stabilise: check whether the orchestrator is restarting pods, because restarts under load worsen it, and scale out or roll back to relieve pressure. Second, find the change: what deployed near 14:05, and did it add a synchronous call inside an async def or move inference into the loop? Third, confirm with evidence: compare handler time against total time, look at in-flight requests and the threadpool queue, and sample a stack if you can. Fourth, fix: move blocking work to a plain def function or a process-based path, since the official guidance is def for non-awaitable libraries and multiprocessing for CPU-bound work. Fifth, prevent: add a test or a loop-lag metric that would have caught it, and separate liveness from the model path.

A strong answer begins with stopping the harm and ends with a preventive change. A weak answer jumps to profiling and leaves the system down while it does.

Incident response: memory kills during a traffic spike

Prompt: "Pods are out-of-memory killed only during peak traffic, never at startup."

The clue is the timing. Resident weights are loaded at startup and fit, so the growth is per-request working memory or an unbounded structure. Check memory slope and concurrency: more concurrent requests means more working memory, and if the number of workers or concurrent threads rose, the multiplication is the cause. Check for caches or global lists that grow. Mitigate by limiting concurrency per process with fast refusal, adding replicas with a proper memory limit, or reducing batch sizes. Fix by bounding every cache, setting the memory request from the peak rather than the idle footprint, and alerting on the slope.

A strong answer distinguishes steady footprint from growth and mentions per-request memory. A weak answer raises the limit and moves on.

Incident response: errors on every deploy

Prompt: "Every rollout produces a burst of five-hundred errors for about a minute."

Two causes dominate. The first is readiness that does not reflect the lifespan, so traffic reaches replicas still loading weights. The second is unclean shutdown, so in-flight requests are dropped when old replicas stop. Check the probe definition, the model load time and the container command. A shell-form CMD stops signals reaching the server, which the FastAPI Docker documentation warns about, and your shutdown code after yield never runs. Fix with an exec-form command, a readiness probe that passes only after warm-up, a termination grace period longer than the slowest request, and readiness that fails first at shutdown. Verify with a rollout under load in staging.

A strong answer names both causes and the test that proves the fix.

Incident response: possible cross-tenant leak

Prompt: "A customer says they saw another customer's result."

Treat it as a security incident from the first minute. Stop or restrict the affected endpoint if you can do so safely, preserve logs and traces, and involve whoever owns security response. Then find the path: which dependency resolved the tenant, which cache keys omit the tenant, whether a module-level variable holds request state, and whether a dependency's cached value was shared across requests by mistake. Establish the exposure window and the affected customers with request IDs and tenant attributes in the logs, which is exactly why you log them. Fix with tenant-scoped keys, remove shared mutable state, and add a two-tenant test. Communicate according to policy, with facts and without speculation.

A strong answer prioritises containment and evidence before root cause. A weak answer starts debugging code while the leak continues.

Incident response: the upgrade that broke production

Prompt: "A minor upgrade went out and several services now fail at startup or serve wrong docs."

Roll back to the previous pinned minor immediately, which requires that it still exists in your artifact registry. Then read the release notes for the breaking-change entry that matches. If services fail where they touch router.routes, the 0.137.0 router tree is the suspect. Ask why your canary and contract suite did not catch it: usually the canary lacked a service with the affected pattern, or the contract suite lacked a test for it. Add both. Notice also what the release did not change: because FastAPI selects its Starlette version, you should not have pinned Starlette yourself, and if you did, that is a second finding.

A strong answer includes the process fix, not only the technical one.

Trade-offs to argue

These are the questions where the interviewer plays a skeptical manager. The pattern for every one is the same: state the options, name the cost of each in terms the listener cares about, recommend one, and say what would change your mind.

FastAPI service or a dedicated serving runtime? FastAPI gives you control, a familiar stack and a low barrier for teams. A runtime gives you batching, GPU efficiency and standard rollouts. Recommend FastAPI at the edge and for light models, a runtime for heavy ones, and say that a measured GPU utilisation under about a third, with latency targets that allow batching, is your trigger to move.

One process per container or several workers? In a cluster, one per container, because the orchestrator already does replication and balancing and memory accounting is clearer. On a single host, several workers are fine. The cost of several workers is memory multiplication and noisier failure.

Logical or physical tenant isolation? Logical is cheap and risky; physical is costly and safe. Recommend by data sensitivity and by what a leak costs the business, not by technical preference.

Pin tightly or stay current? Pin the minor, and stay within one or two minors of current. The cost of pinning is falling behind; the cost of floating is surprise breakage. A tested cadence beats either extreme.

Arguing to leadership

You will sometimes need to justify platform work to people who measure money and risk, not latency. Translate each technical point.

For the upgrade policy: "FastAPI has not reached version 1.0, and its own documentation says minor releases can break things. A weekly automated test of the next release against our contract suite costs a few hours of compute and removes the chance that an upgrade becomes an outage. The alternative is to upgrade rarely and in large jumps, which is how teams end up two years behind and unable to take a security fix."

For memory right-sizing: "Each model is copied into every process that serves it. Measuring and right-sizing this typically reduces the number of machines we pay for. I will report cost per thousand requests per service each month so owners can see the effect of their choices."

For the paved road: "Every service built from the template inherits the same security, probes and telemetry, so a fix to the template reaches all of them through a version bump. Every service outside the template is something we must audit by hand."

For a runtime behind the API: "GPUs are the most expensive item in our estate. A runtime designed for batching can serve more requests per GPU. The cost is another component to operate, which I would adopt only for the models where utilisation numbers justify it."

The structure that works is a sentence on the risk in their terms, a sentence on the cost of the fix, and a sentence on what you will report so they can see it worked. Avoid jargon, and never pretend a number you have not measured.

What a strong answer covers that a weak one misses

Strong answers share habits. They ask clarifying questions first. They say what they would measure before saying what they would build. They name failure modes unprompted: event-loop starvation, memory per process, slow start, unclean shutdown. They give the version-specific fact when it matters, for example that on_event is deprecated and ignored once lifespan is set, or that 0.137.0 changed router composition. They separate stopping harm from finding cause from preventing recurrence. They say where FastAPI stops. And they are honest about uncertainty: "I would check the release notes for that" is a better answer than an invented flag.

Weak answers describe features without consequences, scale by adding workers without counting memory, treat async def as a speed switch, and propose building everything in handlers. They also tend to answer with one technology and no trade-off, which signals that the candidate has used the tool but not owned it.

Sixty-second self-test

Answer each aloud in one or two sentences. If you hesitate, reread the matching section of the detailed guide.

1. Why does a blocking call inside async def hurt every endpoint, including health checks?
2. How does memory scale with workers, and why does that decide your process model in a cluster?
3. What did 0.137.0 change about routers, and how do you find code it breaks?
4. What happens to on_event handlers when you pass lifespan?
5. Why must readiness reflect the lifespan, and liveness stay cheap?
6. Where does the tenant identifier come from, and where must it never come from?
7. What is your upgrade policy for a 0.x framework, step by step?
8. Name two signals you would autoscale on instead of CPU.
9. Which work does not belong in a request handler, and where does it go?
10. When would you put a serving runtime behind FastAPI?
11. Which three documents or endpoints should a public deployment not expose by default?
12. What do you promise teams that use your platform, and what inventory lets you keep the promise?

If you can answer eleven of twelve without notes, you are ready for the system-design rounds. For further practice, read about Kubernetes scaling, then compare KServe and Ray Serve as alternatives to what you built, and rehearse the trade-off arguments above against each.

Part three of three. At this level the problems are not syntax. They are an image forty teams deploy, a framework upgrade that lands on a Tuesday, a model that quietly doubled in memory, and a tenant filter that one endpoint forgot. Start with the error table, then the practices, the review checklists and the operational recipes.

The version context: FastAPI is 0.x (0.142.2 current when this was written), so minor releases can break things, and several traps below come straight from that.

Production traps that appear at scale

Everything from Beginner and Mid still applies. These cause incidents rather than failed requests.

Notice how many of these are the same two root causes: confusing waiting with computing, and confusing one process with the service. Teach both to everyone who writes a handler.

Event-loop starvation in practice

The most common serious trap deserves detail, because it hides well in code review. Nothing looks wrong: the function is async def, it awaits a database call, and then calls model.predict(batch) on a normal synchronous library. The await points make it feel asynchronous, but the prediction call holds the loop for its full duration. Under light traffic nobody notices. Under load the loop spends its time inside predictions, and everything else, including your readiness endpoint, waits.

Three habits prevent it. First, a code-review rule: a call that does not need await and takes more than a few milliseconds does not belong in an async def. Second, a default of plain def for any path operation that touches a model, as the official guidance says when a library is not awaitable; FastAPI then runs it in the threadpool. Third, a measurement: export event-loop lag, the delay between when a callback was scheduled and when it ran. A loop-lag metric above tens of milliseconds is an early warning long before users complain.

Memory traps

Memory is the resource that surprises people, because the arithmetic is simple and still gets skipped. A model's resident size times the number of processes is the floor. Working memory per concurrent request sits on top. A new model version is a new memory profile, so a retrain can turn a stable service into one that gets killed.

Practical habits: record resident memory after warm-up in the deployment metadata, set the memory request from the peak under load, and alert on the slope of memory over time, not only on the limit. Bound every cache with a maximum size. Be suspicious of module-level lists and dictionaries that only grow, such as a "recent requests" list kept for debugging. And keep model loading out of import time: code that loads weights at import runs in every worker process and in every tool that imports your module, including tests and documentation builders.

If tenants have their own models, the cache is your memory budget. Size it per process and remember that with several processes the total is multiplied. Evict least-recently-used models and make the cold path an explicit, observable event with its own metric.

Upgrade traps

A 0.x framework means upgrades are a recurring project. The traps are quiet.

Floating ranges. A requirement such as fastapi>=0.100 accepts every future minor, including breaking ones. Pin the minor and let patches flow. The documented procedure is to write tests, upgrade, run them, fix what fails, then pin the new version.

Pinning the wrong thing. Do not pin Starlette yourself; FastAPI selects a compatible version, and the 0.142.0 release moved it from 1.3.1 to 1.6.0. Pinning it independently creates conflicts. Pydantic can be pinned with a range too. The pin recipe below shows both.

Skipping release notes. Read the entries for each minor you cross. Two worth remembering: 0.137.0 changed router composition so router.routes can hold a tree, and 0.142.0 added native OpenTelemetry. Another is 0.142.1, which fixed endpoints being wrapped repeatedly in included routers.

Testing only the happy path. Your tests must include nested routers, custom middleware, streaming, lifespan behaviour and shutdown. A canary cohort should cover the same patterns.

No rollback. Keep the previous library version and image available, and rehearse the rollback.

Deprecated style lingering. on_event, Pydantic v1 names such as .dict() and .json(), the deprecated tiangolo/uvicorn-gunicorn-fastapi base image. These work until the day they do not. A lint rule that fails the build is cheaper than a migration under pressure.

Security traps

Trusting the request for identity. A tenant or user identifier read from a body, a query parameter or a custom header can be forged. Derive identity from the verified token.

Incomplete token validation. Checking only the signature accepts tokens issued for other audiences or by other issuers. Verify signature, issuer, audience and expiry, and read scopes from claims.

Home-grown identity. For a multi-user platform, validate tokens from your identity provider. The official OAuth2 password-flow tutorial is a teaching tool; it is not a design for your production identity.

Proxy headers without a proxy. --proxy-headers makes the server trust forwarded headers. Use it behind a proxy you control, and make sure clients cannot reach the application directly.

Wide-open CORS. CORS protects browsers only. Keep the allowed origins narrow, and never combine a wildcard with credentials.

Public docs. The generated documentation lists every path and schema. Disable it on public deployments.

Unbounded inputs. FastAPI does not set a request size limit by itself. Enforce it at the proxy and bound lengths and array sizes in the Pydantic schema.

Untrusted model files. Some model formats can execute code when loaded. Load only from controlled storage and verify artifacts. The Trivy guide covers scanning images and dependencies, and Vault covers secrets.

Secrets in images and logs. Pass secrets through environment variables from a secret store. Never log them or request bodies by default.

Multi-tenancy traps

Tenancy bugs are quiet because every request still returns a correct-looking answer; it is just somebody else's. The most common cause is a cache or a module-level variable that was written for one tenant and read by another. Python makes this easy: a dictionary at module level is shared by every request in the process, and a function-level default argument that holds a list is shared across calls. When you review code, search for module-level mutable state and ask of each one what would happen if two tenants used it in the same second.

A second trap is scoping the data access but not the model. A tenant is correctly restricted to its own rows, yet the prediction runs on a model trained on everyone's data, or on a model chosen from a request parameter that the tenant is not entitled to. Treat the model choice as an authorisation decision, made from the token and a table the caller cannot edit.

A third trap is fairness. One tenant sending a burst fills the threadpool, and every other tenant sees timeouts although nothing is wrong with their requests. Limit concurrency per tenant, reject excess work immediately with a 429, and keep the limits in a shared dependency so that every service behaves the same way. Test fairness in staging by running one noisy tenant and one quiet tenant and checking the quiet tenant's latency.

The fourth is residency. If a customer contract says data stays in a country or region, a convenient shared deployment in another region is a breach, not an optimisation. Make the region a parameter of your deployment artifacts, record where each tenant lives, and make it impossible for a request to be routed elsewhere by accident.

Observability traps

Telemetry that takes the service down. Instrumentation runs on the request path, so a slow or broken collector can slow or break the service. FastAPI 0.142.2 fixed the case where a failed automatic OpenTelemetry configuration prevented startup, and the principle applies to your own exporters: they must degrade to dropped data, never to failed requests. Test it by pointing the exporter at an address that does not answer.

Trusting a feature you have not read. Native OpenTelemetry arrived in 0.142.0, and its configuration was not described in the release notes available when this guide was written. Read the pull request and the current documentation, and test it with your own collector before you tell teams to rely on it.

Metric label explosions. A label with thousands of values, such as a tenant identifier or a raw URL path with an identifier in it, multiplies the number of time series and the bill. Put high-cardinality values in traces and logs, and keep metric labels to a small, bounded set such as endpoint template, status class and model name.

Only averages. An average latency of two hundred milliseconds can hide a tail of five seconds. Use histograms and look at the ninety-fifth and ninety-ninth percentiles. Record inference time separately from total time, so you can tell a slow model from a slow platform.

No request identity. Without a request ID in every log line and span, an investigation is a guessing game. Generate one at the edge, propagate it, and return it in a response header so a customer can quote it in a ticket.

Logging payloads. Request bodies contain personal data and are large. Log sizes and hashes by default, and provide a short-lived, redacted debug mode for investigations with an owner who approves it. The OpenTelemetry guide and the Prometheus guide cover the instrumentation and the metrics side.

Platform-ownership habits

These are the practices that pay off most, in rough order of leverage.

1. Ship a paved road. A template and a shared library that make the correct lifespan, probes, auth, error format and telemetry the default. One good template outweighs a hundred review comments.
2. Make the schema the contract. Export the OpenAPI schema on every build, store it, and diff it. Classify changes as compatible or breaking, and require a version bump for breaking ones.
3. Own a contract test suite. A small set of tests of every platform promise, run against each new FastAPI release in CI, including patch releases.
4. Keep an inventory. Every service, its image digest, its library version and its lock file, queryable. When an advisory arrives, the first question is always "who is affected".
5. Report cost per thousand requests. Show each owning team its number monthly, with utilisation and the peak-to-average ratio.
6. Write runbooks before incidents. Symptom, three checks, mitigation, owner. Rehearse one each quarter.
7. Version your dependencies and your errors. Add rather than edit a platform dependency, deprecate with a date, and measure usage before removal.
8. Return the model version. In a header or field, so a bad prediction can be traced to the model that produced it.
9. Offer exceptions with owners and expiry. Blanket refusal creates shadow platforms.
10. Rotate upgrade ownership. Knowledge held by one person is an outage waiting for their holiday.

If you have time for only three, do the template, the schema diff and the inventory.

Review checklist: pull requests

Use this on any pull request that touches a model service.

• Is any blocking or CPU-bound call inside an async def?
• Is a model, client or connection created at import time instead of in lifespan?
• Are caches and global structures bounded?
• Does a new endpoint take identity from the token, with tenant scoping on every data access?
• Are input sizes, lengths and array dimensions bounded in the schema?
• Are timeouts set on every downstream call?
• Does the response include the model name and version?
• Is a new dependency added rather than an existing one edited in place?
• Does the OpenAPI diff show only compatible changes, or is the version bumped?
• Are there tests for validation failures and for the auth path, using dependency overrides?
• Does anything read or mutate router.routes?
• Are logs free of secrets and request bodies?

Review checklist: design documents

For a new service or a significant change, the document should answer these.

• Load. Expected requests per second, peak factor, latency target, and how they were measured.
• Resources. Resident memory per process, throughput per process, startup time, and the resulting replica count.
• Process model. One process per container or several workers, and why.
• Isolation. What a tenant is, where isolation is enforced, and what a leak would cost.
• Failure. What happens when the model fails to load, when a dependency is slow, and when the service is overloaded.
• Lifecycle. How the model is delivered, loaded, warmed and replaced, and how readiness reflects it.
• Security. The trust boundaries, token validation, secrets handling, and what is exposed publicly.
• Observability. Which signals exist, which alert, and who is paged.
• Upgrade. Pinned version, canary plan and rollback.
• Cost. Estimated monthly cost and cost per thousand requests.
• Exit. Under what conditions would this move to a dedicated serving runtime such as Triton or KServe.

A document that cannot fill in the load and resources sections has not been designed yet; it has been hoped for.

Operational recipes

Recipe: the pin recipe.

fastapi[standard]>=0.142.0,<0.143.0
pydantic>=2.7.0,<3.0.0

Recipe: find code that depends on router internals.

grep -rn "\.routes" --include="*.py" .

Review each match. Anything iterating over router.routes or app.routes, or adding routes by mutation, needs replacing with the OpenAPI schema or a documented extension point before you move past 0.136.

Recipe: a structure-independent endpoint inventory test.

from app.main import app

APPROVED = {("GET", "/health"), ("POST", "/predict")}

def test_endpoints_match_approved_list():
    paths = app.openapi()["paths"]
    found = {(m.upper(), p) for p, item in paths.items() for m in item}
    assert found == APPROVED

It fails when someone adds an endpoint without approval, and it does not depend on how routers are nested.

Recipe: testing against the next release without changing production.

pip install --upgrade "fastapi[standard]" httpx pytest
pytest tests/platform_contract -q

Run it in a scheduled CI job. Production stays pinned; the job tells you what the next minor would do.

Recipe: a cheap loop-lag probe.

import asyncio, time

async def monitor(record, interval: float = 0.5):
    while True:
        start = time.perf_counter()
        await asyncio.sleep(interval)
        record(time.perf_counter() - start - interval)

Start it as a task in your lifespan and export the recorded value as a metric. A rising value means something is holding the loop.

Recipe: dependency override in a contract test.

from fastapi.testclient import TestClient
from app.main import app
from platform.deps import current_principal

def test_predict_as_tenant_a():
    app.dependency_overrides[current_principal] = lambda: {"tenant": "a"}
    try:
        client = TestClient(app)
        assert client.post("/predict", json={"x": 1.0}).status_code == 200
    finally:
        app.dependency_overrides.clear()

Run the same test as a second tenant and assert that results never cross. If your tests need the lifespan to run, use the client as a context manager and check the current testing documentation for the exact behaviour.

Recipe: a rollback you have rehearsed. Keep the previous library version and image. Once a quarter, deploy the older version to a canary service under load and confirm that it starts, serves and passes the contract suite. A rollback that has never been run is a hypothesis.

A short platform checklist

Print this, or paste it into your runbook.

• FastAPI pinned to a minor; upgrade owner named
• Next-release contract test runs on a schedule
• Exec-form CMD; one process per container in the cluster
• Lifespan loads, warms, then serves; readiness reflects it
• No blocking calls in async def; loop-lag metric exported
• Memory measured; requests and limits set from peak
• Tenant from the token; scoped cache keys; two-tenant test
• Docs and OpenAPI disabled on public deployments
• Schema stored and diffed; one error envelope; model version returned
• Inventory of services, digests and lock files
• Cost per thousand requests reported monthly
• Runbooks written; one rehearsed this quarter

Carry this list into your next design review. As with most platform work, the incidents it prevents are the ones nobody ever has to write up.