The Complete Prefect Guide

This is part one of three, written against Prefect 3.8 (tested on 3.8.7). It assumes you have never used Prefect, and it assumes you can read basic Python. By the end you will have written a workflow, watched it retry a flaky step, cached an expensive one, run it on a schedule, sent it to a worker, and read the dashboard to find out what went wrong when something failed. Mid-level and Senior take the same ideas into production; nothing you learn here is thrown away.

Each section ends with a Try it task. Do them in order. Orchestration is one of those subjects where reading is not enough: the concepts only make sense once you have watched your own flow fail, retry, and recover.

What Prefect is, and the problem it solves

Prefect is a Python library and a service for running your code reliably and seeing what it did. You write ordinary Python functions. You put a decorator on them. From then on, every call is recorded, can be retried when it fails, can be scheduled, and shows up in a web dashboard with its logs and its outcome.

That sounds modest, so it helps to see the problem it removes. Imagine a data team at a retailer in Riyadh. Every morning a script must download yesterday's orders from an API, clean them, load them into a warehouse, and refresh a model's features. The first version is a file called run_daily.py and a cron entry on someone's server. It works for a month. Then the orders API times out one night and the script dies halfway. Nobody knows until the dashboard is empty at nine in the morning. The fix is to log in, read a log file, and rerun the whole thing by hand, including the download that already succeeded. A week later the server is rebooted and cron is silently gone.

Every one of those failures has the same shape. The script did its job, but nothing around it did its job: nothing noticed the failure, nothing retried, nothing remembered which steps had finished, nothing told a human. Workflow orchestration is the name for the layer that does those things. Prefect is one of several orchestrators, and its particular design goal is to add that layer with as little change to your Python as possible.

Read the diagram left to right. Your code stays your code. The decorators make it report to an API. The API keeps the record, and the dashboard is a window onto that record.

What people use it for:

What came before, and where Prefect sits

The oldest answer is cron: a line in a file that starts a script at a chosen time. Cron is reliable at exactly one thing, starting a process. It does not know whether the process succeeded, it cannot retry, it cannot express "run B after A finished", and it keeps no history beyond whatever your script wrote to a file.

The next generation were workflow schedulers such as Apache Airflow (see the Airflow guide). Airflow asks you to describe your workflow as a graph, a DAG, in a file the scheduler parses. That graph is fixed in advance: the tasks and their dependencies are declared, and the scheduler runs them. This is a powerful model and it runs a huge share of the world's data pipelines, but it asks you to think in Airflow's vocabulary, and dynamic behaviour (loop over however many files arrived today) takes extra machinery.

Prefect's bet is different. You write normal Python, with normal loops, conditions and exceptions, and Prefect observes it while it runs instead of asking you to declare a graph first. If you want a step to repeat for each file, you write a for loop or call .map(). If you want to branch, you write an if. The graph of what happened is reconstructed from the run.

A fair comparison includes the other modern orchestrators. Dagster (see the Dagster guide) organises the world around the data assets your pipeline produces. Prefect organises it around the functions that run. Neither is universally better; the choice is usually about how your team prefers to think.

One more piece of history matters because it will save you hours. Prefect 3.0 became generally available in September 2024. Prefect 2.x is now legacy, yet a large amount of blog posts and AI-generated code still teaches the 2.x API: agents, Deployment.build_from_flow, infrastructure blocks. Those are gone in 3.x. This guide teaches only the 3.x way, and whenever an old name would trap you, it says so.

The mental model: four nouns and a server

Prefect has a small vocabulary. Learn these words now and every later section is a variation on them.

A flow is a Python function decorated with @flow. It is the unit you run, schedule and deploy. A flow can call tasks, call other flows, and run any Python you like. Each time a flow is called, Prefect creates a flow run: a record with an ID, a name, parameters, a start time, logs and a final state.

A task is a Python function decorated with @task. It is a smaller unit of work with extra powers: it can be retried, cached, given a timeout, and run concurrently with its siblings. Each call creates a task run. Tasks are where you put the steps that can fail or are expensive.

A state describes where a run is in its life. The everyday ones are Scheduled, Pending, Running, Completed, Failed, Cancelled and Crashed. The two to learn early are the last two failures. Failed means your code raised an exception and retries (if any) are used up. Crashed means something outside your code killed the run: the machine ran out of memory, or a signal stopped the process. Keeping those two apart saves real debugging time, because one is a bug in your code and the other is a problem with the place it ran.

A deployment is a flow plus instructions for running it remotely: where its code lives, what parameters to use, what schedule to follow, and which infrastructure should execute it. You do not need a deployment to run a flow on your laptop. You need one when you want Prefect, rather than you, to start the run.

Behind these nouns sits one more thing: the Prefect API. It is a server, either Prefect Cloud (hosted by Prefect) or one you run yourself, that stores states, logs and schedules. The critical design point is the split between control and execution. The API stores metadata and decides what should run. Your code executes on machines you control, and only metadata goes to the API. In a region with data-residency rules, such as many Gulf and Egyptian employers have, that split matters: your data never has to leave your own infrastructure for Prefect to orchestrate it.

Installing Prefect and checking the setup

Prefect 3.8 needs Python 3.10 to 3.14. Python 3.9 support was dropped in Prefect 3.5, so an old system Python is the most common reason an install fails. Check yours first.

python3 --version

Always use a virtual environment so Prefect's dependencies do not collide with other projects. On Linux and macOS:

python3 -m venv .venv && source .venv/bin/activate
pip install -U prefect

On Windows PowerShell the activation line differs:

py -m venv .venv
.venv\Scripts\Activate.ps1
pip install -U prefect

If you use uv, the equivalent is uv venv && source .venv/bin/activate && uv pip install prefect, or uv add prefect inside a project that has a pyproject.toml. Conda works too. On Windows the official documentation notes you may need to add Python's Scripts folder to your Path before the prefect command is found.

Check that the install worked:

prefect version

The output lists the Prefect version, the API version, the Python version, your operating system, the active profile, and the server type. A fresh install prints something like Server type: ephemeral, which means "no server is configured; Prefect will start a temporary one when needed". Do not worry about that word yet. It is explained in the next section.

There is also a much smaller package, prefect-client, meant for containers that only need to talk to an existing API and do not need the CLI or the server. As a beginner you want the full prefect package.

Where runs are recorded: ephemeral, local server, Cloud

Before writing a flow, settle where its record will go. There are three options and beginners are often confused because the first one is invisible.

Ephemeral mode is the default. With nothing configured, when you run a flow Prefect starts a temporary local API in the background, records the run, and shuts it down. It is convenient for learning because there is nothing to set up, but you cannot browse a dashboard, because the server disappears when your script ends. This is governed by the setting PREFECT_SERVER_EPHEMERAL_ENABLED, which the default ephemeral profile switches on.

A local server is a long-running API you start yourself. It gives you the dashboard and is the right choice for learning everything else in this guide.

prefect server start

Leave that terminal open. It serves the dashboard at http://127.0.0.1:4200 and the API at http://127.0.0.1:4200/api. The local server stores its data in a SQLite file under ~/.prefect by default. In a second terminal, with the same virtual environment active, tell your client where the API is:

prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"

Do not forget the /api ending. Leaving it off is one of the most common beginner mistakes, and it gives confusing errors.

Prefect Cloud is the hosted option. You run prefect cloud login, sign in through the browser, and choose a workspace. Runs are then recorded in Cloud and you use the Cloud dashboard. The free tier is enough to learn on. Code and data still stay on your machine; only metadata goes to Prefect.

prefect cloud login

The key that browser login creates expires after 30 days. For automation such as CI you would create a service account key and log in non-interactively. That belongs to later levels.

Either is fine for this guide. The rest of the text says "the dashboard" and means whichever you chose. Since Prefect 3.8, the self-hosted dashboard is the newer React-based interface by default, so screenshots in older blog posts may look different from what you see.

Your first flow

Create a file named hello.py. The smallest useful Prefect program is one decorated function:

from prefect import flow

@flow(log_prints=True)
def hello(name: str = "MENA"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello()

Run it the way you run any Python script:

python hello.py

You will see log lines in the terminal, roughly: the flow run was created with a random name such as amiable-falcon, the message printed, and the flow run finished in state Completed. Now open the dashboard and click Runs. The run is there, with its name, duration, and logs containing your greeting.

Three details deserve attention. First, log_prints=True tells Prefect to capture print() output as log entries, which is why the greeting appears in the dashboard and not only in your terminal. Without it, prints go to the terminal alone. Second, the flow is called like a normal function, including its parameter. There is no special runner. Third, the flow run got a generated name; you can control it with flow_run_name=, for example @flow(flow_run_name="greet-{name}").

The type hint on name: str is not decoration. Prefect validates the parameters of a flow against its hints, using Pydantic. Call hello(name=123) and Prefect will refuse the run with a validation error before any of your code executes. That early failure is a feature: bad input is caught at the door.

What happens if the function raises? Try it by changing the body to raise ValueError("boom"). The flow run ends in state Failed, the traceback appears in the logs, and the exception is re-raised to your script. A flow fails when an exception propagates out of it, which is the same rule you already know from Python.

Tasks: the steps inside a flow

A flow that only prints is not an orchestrator. The interesting part begins when you split the work into tasks. Here is a small pipeline that downloads data, cleans it and summarises it. It uses a public endpoint so you can run it yourself.

import httpx
from prefect import flow, task

@task(retries=3, retry_delay_seconds=5)
def fetch_users() -> list[dict]:
    response = httpx.get("https://jsonplaceholder.typicode.com/users", timeout=10)
    response.raise_for_status()
    return response.json()

@task
def extract_cities(users: list[dict]) -> list[str]:
    return sorted({u["address"]["city"] for u in users})

@task
def summarise(cities: list[str]) -> str:
    return f"{len(cities)} distinct cities: " + ", ".join(cities)

@flow(log_prints=True)
def city_report():
    users = fetch_users()
    cities = extract_cities(users)
    print(summarise(cities))

if __name__ == "__main__":
    city_report()

Install httpx first with pip install httpx, then run the file. In the dashboard, open the run and look at the task runs: three of them, each with its own state, duration and logs. That is the payoff of decorating. Before, you had one opaque script. Now you can see that the fetch took two seconds and the summary took a millisecond.

The decorator option that matters most is retries. retries=3 means that if the task raises, Prefect runs it again, up to three more times. retry_delay_seconds=5 waits five seconds between attempts. Network calls fail transiently all the time, and this one line converts a three-in-the-morning page into a line in a log. You can also pass a list of delays, such as retry_delay_seconds=[1, 10, 60], for growing waits, or use exponential_backoff from prefect.tasks.

Tasks called inside a flow look like normal function calls, and they are: calling fetch_users() runs it right then and returns its result. Data passes between tasks as ordinary return values and arguments. There is no separate wiring language.

A few other task options appear early: timeout_seconds=30 fails a task that runs too long, name="fetch" and tags=["api"] label it in the dashboard, and log_prints=True works on tasks as well. Tasks are also callable outside a flow in Prefect 3, but as a beginner keep them inside flows.

How do you decide what is a task? Use tasks for units that can fail independently or are worth retrying, caching or timing on their own: network calls, database queries, slow computations, file writes. Do not turn every two-line helper into a task. Each task run is recorded with states and events, and thousands of tiny tasks add overhead without adding insight. A plain function is fine for trivial steps.

Logging and reading states

When something goes wrong you will reach for logs, so set them up properly from the start. There are two ways to write logs that Prefect captures. The quick one is log_prints=True on the decorator, as you have used. The better one for real work is the run logger:

from prefect import flow, task, get_run_logger

@task
def load(rows: list[dict]) -> int:
    logger = get_run_logger()
    logger.info("Loading %d rows", len(rows))
    if not rows:
        logger.warning("Nothing to load")
    return len(rows)

@flow
def nightly():
    load([{"id": 1}, {"id": 2}])

get_run_logger() returns a standard Python logger that is attached to the current flow run or task run, so each message appears in the dashboard against the right run and carries a level: debug, info, warning or error. It only works inside a flow or task. Calling it from plain module code raises MissingContextError; outside of runs, use from prefect.logging import get_logger.

Logs are sent to the API unless you disable that. The default level is INFO, so debug lines are hidden until you change PREFECT_LOGGING_LEVEL.

Now that you have seen states in the dashboard, here is how they fit together. A scheduled run begins as Scheduled. When a process picks it up it becomes Pending, then Running. If a retry is pending it shows AwaitingRetry and then Retrying. It ends in one of the terminal states: Completed, Failed, Cancelled or Crashed. Two states worth recognising: Late marks a run that was supposed to start and did not, typically because nothing was available to run it, and Cached marks a task that did not execute because a saved result was reused. The next section explains where that comes from.

You can also ask for a state rather than an exception. Calling a task with return_state=True returns the State object instead of raising, which lets a flow decide what to do with a failure:

@flow
def tolerant():
    state = risky_task(return_state=True)
    if state.is_failed():
        print("risky_task failed, continuing with a fallback")

Running tasks concurrently with submit and map

So far tasks ran one after another. Calling fetch_users() blocks until it finishes. When you have many independent items, such as a hundred URLs, waiting for each in turn wastes time. Prefect tasks can run concurrently.

The two tools are .submit() and .map(). Both return futures, which are placeholders for results that may not be ready yet.

from prefect import flow, task

@task
def process_customer(customer_id: str) -> str:
    return f"Processed {customer_id}"

@flow(log_prints=True)
def many():
    ids = [f"customer{n}" for n in range(10)]
    futures = process_customer.map(ids)
    results = [f.result() for f in futures]
    print(results[:3])

if __name__ == "__main__":
    many()

.map(ids) launches one task run per item and returns a list of futures. Calling .result() on a future waits for that task and gives you its return value. You can also pass a future straight into another task, and Prefect waits for it automatically before running the downstream task. That is how you build a pipeline where step two starts as soon as its own input is ready.

.submit(x) does the same for a single call. Use it when you want to start a task and keep going, then collect the result later:

future = process_customer.submit("customer1")
other_work = do_something_else()
print(future.result())

What runs these tasks side by side? A task runner. The default is ThreadPoolTaskRunner, which uses threads, good for work that mostly waits on the network. For CPU-heavy Python there is ProcessPoolTaskRunner, and dedicated runners for Dask and Ray live in separate packages. As a beginner, the default is right.

Two small rules prevent the usual stumbles. If a task takes several arguments and only one varies per item, wrap the constant ones in unmapped() from prefect, as in process.map(ids, config=unmapped(cfg)). Without it, .map() will try to iterate the constant too and complain with MappingLengthMismatch or MappingMissingIterable. And in Prefect 3, futures are synchronous: do not await a future from .submit().

Results, persistence and caching

A result is what a task or flow returns. By default Prefect keeps results in memory for the duration of the run, so passing data between tasks works, but it does not save them anywhere after the run is over. That default is deliberate: results can be large or sensitive, and Prefect does not assume you want them stored.

Saving results is called persistence. You turn it on per task or flow with persist_result=True, or globally with PREFECT_RESULTS_PERSIST_BY_DEFAULT=true. Persisted results are written to local storage, which defaults to ~/.prefect/storage in current versions, or to a storage location you choose with result_storage. Teams running work on several machines need shared storage, such as a cloud bucket, because a result saved on one machine's disk cannot be read by another. If you ever see MissingResult, it means you asked for a result that was never persisted.

Caching builds on persistence. If a task has already run with the same inputs, you may not want to run it again. Prefect 3 makes this easy, and gives a surprise to beginners: tasks use a default cache policy made of the task's inputs, its source code, and the flow run ID. Within one flow run, calling the same task with the same arguments reuses the saved result. Across different runs, the run ID changes, so nothing is reused unless you say so.

To reuse results across runs, choose a policy that leaves the run ID out:

from datetime import timedelta
from prefect import flow, task
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(cache_policy=INPUTS + TASK_SOURCE, cache_expiration=timedelta(hours=1))
def expensive_lookup(country: str) -> dict:
    print(f"computing for {country}")
    return {"country": country, "score": len(country)}

@flow(log_prints=True)
def report():
    print(expensive_lookup("Egypt"))
    print(expensive_lookup("Egypt"))

if __name__ == "__main__":
    report()

Run it twice. The first run prints "computing for Egypt" once and shows the second call as Cached. The second run, within the hour, prints no computing line at all, because the saved result is reused. The key is built from the inputs plus the task's own source code, so if you edit the function, the cache is invalidated on purpose. Setting a cache policy turns persistence on for that task automatically.

The policies you meet first are INPUTS, TASK_SOURCE, FLOW_PARAMETERS and NO_CACHE, and you combine them with +. To switch caching off for a task, write cache_policy=NO_CACHE. To force a fresh run while keeping the policy, pass refresh_cache=True.

Parameters and subflows

A flow is a function, so its arguments are its parameters. They are how you reuse one flow for many cases: a date, a country, a dry-run switch. Give them type hints and defaults, and Prefect validates them.

from datetime import date
from pydantic import BaseModel
from prefect import flow

class ReportConfig(BaseModel):
    country: str = "Egypt"
    top_n: int = 5

@flow(log_prints=True)
def daily_report(day: date, config: ReportConfig = ReportConfig()):
    print(f"Report for {day} in {config.country}, top {config.top_n}")

if __name__ == "__main__":
    daily_report(day=date(2026, 9, 30), config=ReportConfig(country="Saudi Arabia"))

Pydantic models as parameters are a good habit, because the deployment UI later renders them as a form and rejects malformed input up front. If the parameters fail validation, the run goes from Pending straight to Failed without ever reaching Running, and you get ParameterTypeError.

A flow can also call other flows. The inner flow is a subflow: it gets its own flow run, linked to the parent in the dashboard. Use subflows when a section of your pipeline deserves its own record, parameters and final state, for instance one subflow per country.

@flow
def load_country(country: str):
    ...

@flow
def load_all():
    for country in ["Egypt", "Saudi Arabia", "UAE"]:
        load_country(country)

The parent waits for each subflow to finish. Prefer tasks for small steps and subflows for meaningful units; a rule of thumb is that if you would want to rerun it by itself from the dashboard, it may be a flow.

You can also set a time limit and retries at the flow level. @flow(retries=2, retry_delay_seconds=30, timeout_seconds=600) reruns the whole flow on failure and fails it if it runs past ten minutes. Flow retries rerun the flow function from the start, so combine them with caching, so the tasks that already succeeded are not repeated.

Serving a flow: your first scheduled run

Everything so far ran when you typed python file.py. To have Prefect start runs on its own, you create a deployment. The quickest way is .serve().

from prefect import flow

@flow(log_prints=True)
def morning_report(country: str = "Egypt"):
    print(f"Good morning from {country}")

if __name__ == "__main__":
    morning_report.serve(
        name="morning-report",
        cron="0 8 * * *",
        parameters={"country": "Egypt"},
    )

Start it with python serve_demo.py while your local server is running. Unlike before, the process does not exit. It stays alive, registers the deployment with the API, and polls for runs. The dashboard now has a Deployments page listing morning-report/morning-report, with a next run scheduled for 08:00.

You do not have to wait until morning. Open a second terminal and start a run by hand:

prefect deployment run "morning-report/morning-report" --param country="Saudi Arabia"

The name format is always flow-name/deployment-name. You will see the serving process pick the run up and execute it, and the run appears in the dashboard with the parameter you passed. You can do the same from the dashboard's Run button, which offers a form for the parameters.

The crucial limitation to understand: a served flow only runs while the serving process is alive. Close the terminal, and the deployment still exists but nothing is listening, so scheduled runs become Late. .serve() is perfect for learning, for a single machine, and for containers you keep running with a process manager. For anything that must survive a reboot or scale across machines, the next sections introduce work pools and workers.

Schedules

A schedule tells Prefect when to create runs for a deployment. Prefect supports three kinds. Cron uses the familiar five-field expression. Interval repeats every N seconds. RRule follows the iCalendar recurrence format and suits irregular patterns such as "first Monday of each month".

With .serve() the shortcuts are cron=, interval= and rrule=. A few examples:

morning_report.serve(name="every-morning", cron="0 8 * * *")
morning_report.serve(name="every-ten-min", interval=600)

For more control, including a timezone, use the schedules argument with the classes from prefect.schedules:

from prefect.schedules import Cron

morning_report.serve(
    name="riyadh-morning",
    schedules=[Cron("0 9 * * MON-FRI", timezone="Asia/Riyadh")],
)

The timezone matters more than beginners expect. Without one, a cron expression is interpreted in UTC, so "8 a.m." would land at 11 a.m. in Riyadh. State the timezone you mean. Prefect keeps cron and RRule schedules at the same local wall-clock time across daylight-saving changes, which is irrelevant in Riyadh or Cairo for most of the year but useful to know if your team spans regions.

A deployment can hold several schedules, each with its own parameters. The scheduler service generates upcoming runs ahead of time, so the dashboard shows future runs as Scheduled. By default it plans up to 100 runs, at most 100 days ahead, for each deployment. You manage schedules from the dashboard, from your code, or from the CLI:

prefect deployment schedule ls "morning-report/morning-report"
prefect deployment schedule pause "morning-report/morning-report" --all

Work pools and workers

.serve() ties execution to one process you started. Real teams want execution to be decoupled from deployment: developers register what should run, and separate infrastructure decides where and how. Prefect does this with work pools and workers.

A work pool is a named bridge between deployments and infrastructure. It has a type (process, docker, kubernetes, and several cloud types) and a template describing how to start a run of that type. A deployment targets a pool. A worker is a small long-running process, started with the CLI, that polls one pool for runs and, for each, creates the infrastructure: a subprocess for a process pool, a container for a docker pool, a Job for Kubernetes. The worker's type must match the pool's type. Workers only make outbound connections to the API, so the machine running them does not need to accept inbound traffic.

The simplest pool type is process: runs execute as plain subprocesses on the machine where the worker runs. Create one and start a worker:

prefect work-pool create my-pool --type process
prefect worker start --pool my-pool

Leave the worker running. Now deploy a flow to that pool. Instead of .serve(), use .from_source(...).deploy(...), which says where the code lives and which pool should run it:

from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="https://github.com/your-org/your-repo.git",
        entrypoint="flows/report.py:morning_report",
    ).deploy(
        name="morning-report-pool",
        work_pool_name="my-pool",
        cron="0 8 * * *",
    )

Replace the repository and entrypoint with your own. The entrypoint has the form path/to/file.py:function_name. The worker, when it gets a run, clones the repository, loads the flow from that file and runs it. That is the big difference from serving: the worker fetches code at run time, so pushing new code to the repository changes what tomorrow's run executes, with no restart.

The docker and kubernetes pool types are the route to containers and clusters; they need pip install "prefect[docker]" or "prefect[kubernetes]" and a little more configuration. See the Docker guide and Kubernetes guide for the underlying tools. A worker will offer to install the matching integration package when it is missing, and you can set the behaviour with --install-policy.

Check on things with:

prefect work-pool ls
prefect work-pool inspect my-pool

Workers send a heartbeat every 30 seconds and are shown offline after missing a few, so the dashboard's Work Pools page tells you at a glance whether anyone is listening. This is the first place to look when runs go Late.

Deploying from a prefect.yaml file

Writing deployments as Python is fine for one flow. Teams with several usually prefer a declarative file committed beside the code: prefect.yaml. You create a starter with:

prefect init

and then deploy with the prefect deploy command, which reads the file. A minimal version:

name: mena-pipelines
prefect-version: 3.8.7

pull:
  - prefect.deployments.steps.git_clone:
      repository: https://github.com/your-org/your-repo.git
      branch: main
  - prefect.deployments.steps.pip_install_requirements:
      requirements_file: requirements.txt
      directory: "{{ git_clone.directory }}"

deployments:
  - name: morning-report
    entrypoint: flows/report.py:morning_report
    parameters:
      country: Egypt
    schedules:
      - cron: "0 8 * * *"
        timezone: Asia/Riyadh
    work_pool:
      name: my-pool

Read the file as three ideas. The pull section lists steps a worker runs to get your code onto the machine before the flow starts: clone a repository, then install its requirements. The deployments list names each deployment, its entrypoint, parameters, schedules and work pool. Double-curly templating such as {{ git_clone.directory }} refers to the output of an earlier step, and {{ $ENV_VAR }} reads an environment variable at deploy time.

Then deploy:

prefect deploy --name morning-report
prefect deploy --all

Without flags, prefect deploy walks you through an interactive wizard that asks for the flow, the pool and a schedule and offers to save the answers to prefect.yaml. That is a gentle way to produce the first file. Keep the file in version control so a deployment is reviewable like any other change.

Variables, secrets and blocks

Real flows need configuration: a bucket name, an API endpoint, and, worse, credentials. Do not hard-code them, and do not commit them. Prefect gives you three places to keep them outside your code.

Variables are named values stored on the server, meant for non-sensitive configuration such as an environment name. They are not encrypted. Set and read them from the CLI or Python:

prefect variable set env_name prod
prefect variable get env_name

from prefect.variables import Variable

Variable.set("env_name", "prod", overwrite=True)
env = Variable.get("env_name", default="dev")

Secrets hold sensitive strings. The Secret block stores its value encrypted on the server and hides it in the UI:

from prefect.blocks.system import Secret

Secret(value="s3cr3t").save("db-password", overwrite=True)
password = Secret.load("db-password").get()

The first line is usually run once, by a person, from a trusted machine. The second is what the flow runs. The password lives on the server, and the flow code only contains its name.

Blocks are the general mechanism: a typed, saved configuration object, such as credentials for a cloud bucket, that you load by name. Integration packages supply block types; for instance prefect-aws provides an S3Bucket block. You register a block type and create documents for it, then in code call .load("name"). Blocks are how a flow gets access to a storage service without knowing the key.

The everyday CLI, grouped by intent

Most of what you will do after setup happens in a handful of commands. Grouped by what you are trying to accomplish:

Check where you are.

prefect version
prefect config view
prefect profile ls

prefect config view shows the active profile's settings and --show-defaults adds the ones you have not changed. A profile is a named bundle of settings stored in ~/.prefect/profiles.toml. The defaults are ephemeral, local, test and cloud. Create one per backend and switch with prefect profile use local.

Find and inspect runs.

prefect flow-run ls
prefect flow-run ls --state Failed
prefect flow-run inspect <ID>
prefect flow-run logs <ID>

Act on runs.

prefect flow-run cancel <ID>
prefect flow-run retry <ID_OR_NAME>
prefect flow-run watch <ID>

Work with deployments.

prefect deployment ls
prefect deployment inspect "flow/deployment"
prefect deployment run "flow/deployment" --param key=value --watch

The --watch flag makes the command wait and stream the run's progress, which is handy in scripts. Since Prefect 3.6.21, many commands also accept --output json, useful when you need to feed the answer to another program.

Manage infrastructure.

prefect work-pool ls
prefect worker start --pool my-pool

Configuration, profiles and where settings come from

Prefect is configured by settings. Each has a name like PREFECT_API_URL and a default. You can change them in several places, and when the same setting appears in more than one place, a fixed order decides the winner.

From strongest to weakest: environment variables, a .env file, a prefect.toml file in your project, the [tool.prefect] table in pyproject.toml, the active profile, and finally the built-in default. The practical rule is that an environment variable always wins, which is what makes containers and CI easy to configure.

The command prefect config set writes to the active profile, and prefect config view --show-sources tells you where each value came from, which is the fastest way to answer "why is it using that URL?".

[api]
url = "http://127.0.0.1:4200/api"

[logging]
level = "DEBUG"

The settings you will meet first:

Reading the common errors

Most early problems fall into a few families. Here is how to read them.

"No Prefect API URL provided. Please set PREFECT_API_URL to the address of a running Prefect server." Your client has no backend and ephemeral mode is off, typically because you are in a container or a custom profile. Run prefect config set PREFECT_API_URL=http://host:4200/api or prefect cloud login, and remember the /api suffix.

"Found incompatible versions: client: X, server: Y. Major versions must match." A Prefect 2 client is talking to a Prefect 3 server, or the other way round. Align the major versions. A client newer than its server can also earn 422 responses, so keep the client at or below the server's version.

401 "Invalid authentication credentials" on Cloud. The API key is missing, wrong or expired. Browser-login keys last 30 days, so a laptop that worked last month can stop working. Log in again.

Runs stuck in Scheduled, then Late. Nothing is picking them up. Check that a worker is running for that pool, that its type matches the pool, that the pool and queue are not paused, and that no concurrency limit is full. Worker logs state plainly when a pool is paused. Work through prefect work-pool inspect and the Work Pools page.

"Unable to start worker. Please ensure you have the necessary dependencies installed..." The worker type needs an integration package. Install it, for example pip install prefect-kubernetes, or use --install-policy always.

Crashed. Not your code's exception. Suspect memory, a killed container or a lost machine. Read the worker logs and the container or pod logs, and raise memory limits if needed.

Script errors at startup such as MissingFlowError or "Script at ... encountered an exception". The entrypoint is wrong (check path/file.py:function spelling), the code was not pulled, or a dependency is missing on the worker. Since 3.7.3 this last one is the most common cause.

ParameterTypeError. The parameters do not match the flow's type hints. Fix the payload. If the deployment's saved parameters no longer match the flow's signature, you will see SignatureMismatchError; redeploy.

A reliable general method: open the failed run, read the last error in its logs, note which state it ended in, and ask whether the cause is your code (Failed), the infrastructure (Crashed), or the plumbing (Late, never started).

Putting it all together

Here is a small end-to-end project that uses most of what you have learned: fetch data with retries, cache an expensive step, run tasks concurrently, log properly, and schedule it. It is a nightly report that fetches posts for several users and counts words.

import httpx
from datetime import timedelta
from prefect import flow, task, get_run_logger
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(retries=3, retry_delay_seconds=[2, 10, 30], timeout_seconds=30)
def fetch_posts(user_id: int) -> list[dict]:
    logger = get_run_logger()
    url = f"https://jsonplaceholder.typicode.com/posts?userId={user_id}"
    response = httpx.get(url, timeout=10)
    response.raise_for_status()
    logger.info("user %s returned %d posts", user_id, len(response.json()))
    return response.json()

@task(cache_policy=INPUTS + TASK_SOURCE, cache_expiration=timedelta(hours=6))
def count_words(posts: list[dict]) -> int:
    return sum(len(p["body"].split()) for p in posts)

@flow(log_prints=True, retries=1, retry_delay_seconds=30)
def nightly_posts(user_ids: list[int] = [1, 2, 3, 4, 5]):
    fetches = fetch_posts.map(user_ids)
    counts = count_words.map(fetches)
    totals = dict(zip(user_ids, [c.result() for c in counts]))
    print(f"Word counts by user: {totals}")
    return totals

if __name__ == "__main__":
    nightly_posts.serve(
        name="nightly-posts",
        cron="0 2 * * *",
        parameters={"user_ids": [1, 2, 3, 4, 5]},
    )

Walk through it, because each line is a lesson from earlier. fetch_posts is a task because it touches the network, so it has retries with growing delays and a timeout. count_words is cached on its inputs and source, so rerunning the flow within six hours skips recomputation. .map() runs the five fetches concurrently, and because count_words.map(fetches) receives futures, each count starts as soon as its own fetch finishes, with no manual waiting. The flow has its own retry for a bad night. The flow is served on a cron schedule for 02:00, and you can also trigger it by hand with prefect deployment run "nightly-posts/nightly-posts".

To make it survive reboots, you would move from .serve() to a work pool and a worker, a prefect.yaml in the repository, and a pull step that installs httpx. That is the same code with a different deployment path, and it is the natural bridge to the Mid-level guide.

Run it, let it complete once, and open the dashboard. You should be able to explain, for this one flow, every row: the flow run, five fetch task runs, five count task runs, and their states, durations and logs.

What you can now do, and what comes next

You can now install Prefect and check the setup, choose between ephemeral mode, a local server and Cloud, turn Python functions into flows and tasks, and watch them in the dashboard. You can retry flaky steps, time out slow ones, cache expensive results, run tasks concurrently with .submit() and .map(), and pass validated parameters. You can serve a flow on a schedule, deploy one to a work pool with a worker, describe deployments in prefect.yaml, store configuration as Variables and Secrets, and read the common error messages well enough to tell a code bug from an infrastructure failure from a missing worker.

What comes next, in the Mid-level guide: Docker and Kubernetes work pools, base job templates and job variables, automations and events, concurrency limits, transactions, artifacts and testing your flows. Senior covers running Prefect as a platform: high-availability servers, security, multi-team setups and cost.

For neighbouring tools in this catalogue, the Airflow guide and Dagster guide show the same problem solved with different philosophies, the MLflow guide pairs naturally with Prefect for tracking the models your flows train, and the Docker guide is the prerequisite for container-based pools.

Sources

• Prefect documentation
• Install Prefect
• Prefect quickstart
• Flows
• Tasks
• States
• Caching
• Deployments
• Schedules
• Work pools
• Workers
• Blocks
• Variables
• Settings and profiles
• Upgrade from Prefect 2 to 3
• Prefect 3.8 release notes
• Prefect releases on GitHub

Part one of three. A twenty-five-minute review of the Beginner Detailed track for Prefect 3.8, followed by the questions interviewers actually ask about it. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

Prefect is a Python workflow orchestrator. You decorate ordinary functions with @flow and @task, and Prefect records every run, retries failures, caches results, runs things on a schedule, and shows it all in a dashboard. The API stores only metadata and state; your code and data run on infrastructure you control.

Then the line that shows you have used it: "a flow fails when an exception propagates out of it, and a run that is Crashed was killed from outside my code, so I debug the code for Failed and the infrastructure for Crashed."

The model

The sentence to have ready: Prefect separates the control plane (Cloud or a self-hosted server that stores state and schedules) from the execution plane (the place your code runs). That separation is why data can stay inside your own network, and why workers only need outbound connections.

The commands to have ready

You should be able to type these without looking them up.

pip install -U prefect
prefect version
prefect server start
prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"
prefect cloud login
prefect flow-run ls --state Failed
prefect flow-run logs <ID>
prefect deployment run "flow/deployment" --param key=value
prefect work-pool create my-pool --type process
prefect worker start --pool my-pool
prefect deploy --name my-deployment

And the Python you should be able to write on a whiteboard:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=5)
def fetch(n: int) -> int:
    return n * 2

@flow(log_prints=True)
def pipeline(items: list[int]):
    futures = fetch.map(items)
    print([f.result() for f in futures])

if __name__ == "__main__":
    pipeline.serve(name="demo", cron="0 8 * * *")

Be ready to explain each line: what retries re-runs, what .map() returns, why .result() is needed, and what serve does to the process.

The classic traps and why they happen

Trap one: the old API. A large amount of web content and AI-generated code targets Prefect 2. It uses prefect agent start, Deployment.build_from_flow, infrastructure blocks and prefect deployment build. All are removed in 3.x. Agents became workers, and deployments are created with flow.serve(), flow.deploy(), flow.from_source() or prefect deploy. Interviewers ask about this to see whether you learned from current documentation.

Trap two: serving versus workers. A served flow runs only while its serving process is alive. If the process exits, schedules still exist but produce Late runs, since nothing is listening. Work pools and workers decouple deployment from execution and survive process restarts when supervised.

Trap three: the missing /api. PREFECT_API_URL must end with /api. Pointing at the bare server address gives confusing failures.

Trap four: default caching. In Prefect 3, tasks use a default cache policy built from inputs, task source and flow run ID. Inside one run, identical calls may be served from cache. Across runs, the ID differs so nothing is reused unless you pick a policy without it. People are surprised twice: once when a task does not rerun, and once when a cache they expected does not persist across runs.

Trap five: retries re-run everything. A retry re-executes the task function from its first line. A side effect before the failing line happens again. Make side effects safe to repeat, and keep tasks small.

Trap six: results are not saved by default. Results live in memory during a run. Persistence must be switched on, and runs that span machines need shared storage. MissingResult is the error when you ask for a result that was never persisted.

Trap seven: dependencies on workers. Since 3.7.3, code pulled by a worker no longer installs its requirements automatically. Either add a pip_install_requirements pull step, bake an image, or opt in with PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=true.

Trap eight: secrets in Variables. Variables are unencrypted. Sensitive values belong in Secret blocks or a proper secret manager.

Trap nine: timezone-free cron. A cron schedule without a timezone is read in UTC. A team in Riyadh expecting 8 a.m. local gets 11 a.m. unless timezone="Asia/Riyadh" is set.

Common interview questions

What problem does an orchestrator solve that cron does not?

Cron starts a process and then knows nothing. An orchestrator records the outcome, retries failed steps, expresses dependencies, remembers which steps finished, keeps history and logs, and alerts people. It turns "a script ran" into "a workflow with a visible, recoverable state".

How is Prefect different from Airflow?

Airflow asks you to declare a graph of tasks in advance, which the scheduler parses and runs. Prefect observes ordinary Python at runtime, so loops, conditions and exceptions are just Python, and the graph is reconstructed from the run. Airflow has a longer history and a huge ecosystem of operators; Prefect has a lighter authoring model. I would say both are valid and the choice depends on team habits and existing investment.

What is the difference between a flow and a task?

A flow is the unit you run, schedule and deploy; a task is a step inside a flow with its own retries, caching, timeout and concurrency. A flow can call tasks, other flows and any Python. I use tasks for things that can fail or be expensive, such as network calls, and plain functions for trivial helpers, because each task run adds recording overhead.

What happens when a task raises an exception?

If it has retries left, Prefect marks it as awaiting retry and runs the whole function again after the delay. When retries are exhausted it ends in Failed. A flow is Failed if an exception propagates out of it, which happens when the task is called directly, or when I read a future's result and it re-raises.

What is the difference between Failed and Crashed?

Failed means my code raised an exception and retries are used up. Crashed means something outside my code ended the run: an out-of-memory kill, a signal, a lost machine. I fix code for the first and look at infrastructure logs and resource limits for the second.

How do you run tasks in parallel?

With .submit() for one call or .map() for many. Both return futures; .result() waits for a value, and passing a future to another task makes it wait automatically. The default task runner is thread-based, good for I/O. Futures are synchronous in Prefect 3, so I do not await them.

What does .map() do with a constant argument?

It tries to iterate it, which raises MappingLengthMismatch or MappingMissingIterable. I wrap constants in unmapped().

What is a deployment, and how do you create one?

A deployment is the server-side description of how to run a flow: entrypoint, parameters, schedules, work pool, code location. I create one with flow.serve() for static infrastructure, flow.from_source(...).deploy(...) for a work pool, or prefect deploy with a prefect.yaml file. Runs are referred to as flow-name/deployment-name.

Explain work pools and workers.

A work pool has a type and a template describing how to start a run of that type, such as a process, a Docker container or a Kubernetes Job. A worker is a small process that polls a pool and starts the infrastructure for each scheduled run. The worker's type must match the pool's type. Workers only make outbound connections, which is good for locked-down networks.

A scheduled run is stuck in Late. What do you check?

Late means it was still Scheduled 15 seconds after its start time, so nothing picked it up. I check that a worker is running for that pool, that its type matches, that the pool and queue are not paused, that no concurrency limit is full, and the worker logs. For served flows, I check the serving process is alive.

How does Prefect caching work?

Caching needs a saved result. A cache policy computes a key; if a stored result exists under that key, the task is Cached and does not run. The default policy combines inputs, task source code and flow run ID. To reuse across runs I use something like INPUTS + TASK_SOURCE with cache_expiration. To disable, cache_policy=NO_CACHE. To force a rerun, refresh_cache=True.

How do you handle secrets and configuration?

Non-sensitive settings go in Variables or environment variables. Secrets go in Secret blocks, which are encrypted on the server, or an external secret manager. Nothing sensitive is committed to the repository. Variables are unencrypted, so never a password.

Where does Prefect run my code, and what does Prefect Cloud see?

My code runs on my infrastructure, on a laptop, worker, container or cluster. Cloud or the self-hosted server sees metadata: states, logs, parameters, schedules. For employers with data-residency concerns this hybrid model lets data stay in-region while orchestration is hosted.

How do you pass data between tasks?

As ordinary arguments and return values. Within a run the result stays in memory. Large data should not flow through parameters: Cloud rejects request bodies over 5 MiB and self-hosted flow parameters default to a 512 KB limit. Pass a reference such as a path or URI instead.

How do you test or debug a flow locally?

Call it like a function, with no server needed, since ephemeral mode records the run automatically. I use log_prints=True or get_run_logger(), and then the dashboard or prefect flow-run logs <ID> to read the run.

What changed from Prefect 2 to 3 that you would warn a colleague about?

Agents are gone in favour of workers; infrastructure blocks are gone in favour of typed work pools with job variables; Deployment.build_from_flow and prefect deployment build are removed; schedules use schedules=[...] or cron=; Pydantic 2 is required; futures are synchronous; and the default task runner is thread-based.

What are profiles and how do they help?

A profile is a named set of settings stored in ~/.prefect/profiles.toml. The defaults are ephemeral, local, test and cloud. I keep one profile per backend and switch with prefect profile use, so I never point a test run at production by accident. prefect config view --show-sources tells me where any given value came from, and environment variables override the profile.

Why might a flow pass locally and fail on a worker?

The environment differs. The usual causes are a dependency missing on the worker (since 3.7.3 pulled code does not auto-install requirements), a secret or environment variable that exists only on my laptop, or a file path that exists only locally. I list what my laptop has that the worker lacks, and fix it with a pull step, an image or a Secret block.

When would you use a subflow instead of a task?

When a section of the pipeline deserves its own flow run, its own parameters and its own final state in the dashboard, for example one subflow per country. The parent waits for it, and I can inspect it separately. For small steps inside a flow I use tasks.

How would you be notified when a flow fails?

The beginner answer is an on_failure hook on the flow that posts to a chat channel or sends an email, called with the flow, the run and the final state. The fuller answer is an Automation that triggers on the failed-run event, which I would learn at the next level. The old flow-run notification policies were folded into Automations in 3.4.

What would you do on day one with Prefect in a new team?

Check the version and whether they use Cloud or a self-hosted server, read how deployments are defined, find the work pools and confirm a worker is online, run an existing flow by hand from the CLI, and read its logs. That orients me before I change anything.

What a strong answer covers that a weak one misses

A weak answer describes Prefect as "a scheduler". A strong one names the control-plane/execution-plane split, knows a run ends in Failed versus Crashed for different reasons, and points out that the serve model dies with its process while workers do not.

A weak answer says "I would add retries". A strong one adds that retries re-run the whole task so side effects must be safe to repeat, and mentions a flow-level retry combined with caching so finished tasks are not redone.

A weak answer recites Prefect 2 commands. A strong one uses only 3.x vocabulary and can say what replaced each removed piece.

A weak answer stops at "the run failed". A strong one walks a method: open the run, read the last error, note the state, decide between code, infrastructure and plumbing, and then use the CLI if the browser is unavailable.

Questions to ask the interviewer

Good questions signal experience. Ask whether they self-host the server or use Cloud and how they handle upgrades; how they manage deployments (Python, prefect.yaml, or infrastructure as code); what their work pools run on; and how failures reach humans today. These show you think about operating the tool, not only writing flows.

Whiteboard exercise

You may be asked to sketch a nightly pipeline. A reliable shape: a flow with a date parameter; a task that extracts with retries and exponential backoff; a task that transforms, cached on inputs; a task that loads, written to be safe to repeat; an on_failure hook that sends a message; a deployment with a cron schedule and timezone on a process or container pool. Say aloud why each choice was made, for example "load is idempotent because a retry will run it twice".

Then extend it on request. To process many files, use .map() with unmapped() for shared configuration. To limit load on a database, mention concurrency limits as something you would learn next. To reduce risk on failure, mention that tasks already finished can be skipped by caching when the flow retries.

Sixty-second self-test

Answer out loud, without looking back.

1. What are the two decorators, and what does each create when called?
2. Name the three places a flow's run can be recorded.
3. What must PREFECT_API_URL end with?
4. What is the difference between Failed and Crashed?
5. What does .map() return, and how do you get values out of it?
6. Why does a task show Cached, and what are two ways to stop it?
7. What happens to a served flow when you close the terminal?
8. What are the three parts of the work-pool picture, and which must match in type?
9. Which command starts a worker, and which replaced prefect agent start?
10. Since 3.7.3, what do you have to do about dependencies when a worker pulls your code?
11. Where does a Variable differ from a Secret block?
12. Why should a cron schedule carry a timezone?

If you stumbled on more than three, re-read the matching sections of the Detailed track, then rebuild the nightly pipeline from memory. If you also want to see the same problem solved with a different philosophy, compare it with the Airflow guide and the Dagster guide.

Part one of three. Most Prefect beginner trouble comes from four confusions: which backend the client is talking to, which version of the API a tutorial teaches, what a retry or a cache actually repeats, and who is supposed to be running the flow. Start with the table, then the longer explanations, the habits, and the checklist.

Common errors at this level

Mistakes beginners actually make

Reading a Prefect 2 tutorial

Symptom. You paste a command such as prefect agent start or call Deployment.build_from_flow, and get an unknown command or an import error.

Cause. Prefect 3 went generally available in September 2024 and removed agents, infrastructure blocks, storage blocks for deployments, and the old deployment build commands. A lot of blog posts and AI answers predate that.

Fix. Use the current mapping: agents are now workers (prefect worker start --pool <pool>), building a deployment is flow.serve(), flow.deploy(), flow.from_source(...) or prefect deploy, infra_overrides is job_variables, and a single schedule= is better written as cron=, interval= or schedules=[...]. When in doubt, check the page on docs.prefect.io and its version.

Thinking the dashboard is empty because nothing ran

Symptom. Your script printed output and ended in Completed, but the dashboard shows no runs.

Cause. With nothing configured, Prefect used ephemeral mode: it started a temporary local API, recorded the run there, and shut it down. The dashboard you opened belongs to a different server, or none exists.

Fix. Run prefect server start, then prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api" in the terminal where you run your flows. Confirm with prefect config view. Only the records made after this appear.

Forgetting /api

Symptom. The client cannot reach the server although the browser can.

Cause. The dashboard is at http://127.0.0.1:4200 but the API lives at http://127.0.0.1:4200/api. The setting must contain the API address.

Fix. Append /api. It is the single most common first-hour error.

Assuming .serve() is permanent

Symptom. Your deployment ran fine while you were at your desk, then scheduled runs piled up as Late overnight.

Cause. A served flow runs inside the process that called serve(). When you closed the laptop or terminal, nothing was listening. The deployment still existed, so runs were still scheduled, but no one executed them.

Fix. For learning, keep the process running. For anything real, either run the serve script under a supervisor (a container with a restart policy, systemd, a process manager) or move to a work pool and a worker. Check the pool page in the dashboard to see whether any worker is online.

Wrong worker for the pool

Symptom. A worker starts, says nothing useful, and runs stay Scheduled.

Cause. Workers and pools have types, and they must match. A docker worker cannot serve a process pool. A paused pool also gives no runs to anyone; the worker says the pool is paused.

Fix. prefect work-pool inspect my-pool shows type and status. Start the worker with --pool my-pool. If the worker asks to install an integration package, let it, or install prefect-docker or prefect-kubernetes yourself.

Missing dependencies on the worker

Symptom. The deployment works locally, but the run on the worker fails immediately with an import error or a script error.

Cause. Workers fetch your code but, since Prefect 3.7.3, do not install its requirements automatically. The worker's environment simply lacks the package.

Fix. Add a pip_install_requirements pull step in prefect.yaml, bake the dependencies into an image for container pools, or set PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=true where the worker runs. Keep a requirements.txt with pinned versions.

Wrong entrypoint

Symptom. MissingFlowError, or a message like "Script at ... encountered an exception".

Cause. The entrypoint must be path/to/file.py:function_name, relative to the repository root after the pull, and the function must be decorated with @flow. A misspelled path or a function name that is not a flow fails.

Fix. Copy the path from the repository, check case and slashes, and confirm the decorator. If the code failed while importing, the real error is earlier in the log.

Retrying something that is not safe to repeat

Symptom. An email is sent three times, a row is inserted twice, or a file is uploaded repeatedly.

Cause. When a task is retried, the whole function runs again from the top. Side effects before the failure point happen again.

Fix. Keep tasks small and focused; separate the risky network call from the side effect. Make writes idempotent, for example by using an upsert or a check-before-write. Retries are best on reads.

Being surprised by default caching

Symptom. A task that you expected to run shows Cached and its output is old, or a cache you expected across runs does not exist.

Cause. Prefect 3's default policy builds the key from inputs, task source and flow run ID. Inside a run, repeated identical calls reuse the result. Across runs the ID changes, so nothing is shared.

Fix. To disable, use cache_policy=NO_CACHE. To force a refresh, refresh_cache=True. To share across runs, choose INPUTS + TASK_SOURCE and set cache_expiration. If you get a HashError, an input cannot be hashed; exclude it with INPUTS - "name".

Expecting results to be saved

Symptom. MissingResult when you fetch a result from a finished run, or a retried flow on another machine cannot find an earlier output.

Cause. Results live in memory unless persistence is turned on, and local disk is invisible to other machines.

Fix. Set persist_result=True where it matters, or PREFECT_RESULTS_PERSIST_BY_DEFAULT=true, and point result_storage at storage every machine can reach.

Mapping a constant

Symptom. MappingLengthMismatch or MappingMissingIterable.

Cause. .map() iterates every argument it is given. A configuration object or a single string will be iterated too.

Fix. Wrap it: process.map(items, config=unmapped(cfg)).

Awaiting futures

Symptom. A TypeError or an odd error when you await task.submit(...).

Cause. In Prefect 3 futures are synchronous. You wait with .result() or .wait().

Fix. Remove the await. Async tasks have their own rules; stay with synchronous flows until you are comfortable.

Putting a secret in a Variable

Symptom. None, until someone sees it. Variables are unencrypted and visible in the workspace.

Cause. The names look alike.

Fix. Store credentials in a Secret block or your platform's secret manager. Variables are for things you would be happy to read aloud in a meeting.

Leaving cron without a timezone

Symptom. The report lands three hours late, or early, every day.

Cause. Cron without a timezone is read as UTC. Riyadh is three hours ahead of UTC.

Fix. Use the schedules argument with Cron("0 8 * * *", timezone="Asia/Riyadh") or the timezone key in prefect.yaml.

Turning everything into a task

Symptom. A flow with hundreds of tiny tasks is slow, and its dashboard is unreadable.

Cause. Every task run records states and events. Overhead is small per task but adds up, and tiny tasks tell you nothing you would act on.

Fix. Make tasks for work that can fail, be retried, cached or timed independently. Use ordinary functions for small transformations inside a task.

Sending large data through parameters

Symptom. A 413 Content Too Large from Cloud or a parameter-size error on a self-hosted server.

Cause. Cloud rejects request bodies over 5 MiB, and self-hosted flow parameters default to 512 KB.

Fix. Pass a path, URI or identifier, and let the flow read the data from storage.

Habits worth adopting on day one

Pin your versions. Put prefect==3.8.7, or the version you tested, in requirements.txt. Keep client and server on compatible versions, with the client no newer than the server.

Use a virtual environment for every project. It stops dependency fights and makes prefect version mean something.

Name things. Give flows, tasks and deployments clear names, and use flow_run_name or task_run_name templates so the dashboard tells a story. city-report helps more than main.

Turn on log_prints=True while learning, and move to get_run_logger() for real work, because levels and structure help when you search logs.

Type-hint your parameters. Prefect validates them, and a bad input fails before any work starts. Use Pydantic models for groups of settings.

Decide what deserves a task. Network calls and slow steps, yes. Trivial helpers, no.

Give every network task retries and a timeout. Something like retries=3, retry_delay_seconds=[2, 10, 30], timeout_seconds=30 is a reasonable starting point for API calls.

Make tasks idempotent. Assume any step may run twice, and ask what happens if it does.

State your timezone. Always, for every schedule.

Keep secrets out of code. Use Secret blocks or a secret manager, and never print a secret to the logs.

Commit prefect.yaml to version control. A deployment is configuration, and configuration is reviewed like code.

Read the state before debugging. Failed points to your code. Crashed points to the machine. Late points to a missing worker. You save time by choosing the right place to look first.

Practise from the terminal. prefect flow-run ls, inspect and logs let you investigate over SSH when no browser is at hand.

Check the date on tutorials. If the page predates September 2024 or mentions agents, it is about Prefect 2.

Use prefect config view --show-sources when a setting will not stick. It names where each value came from, and environment variables always win.

Practice cards

Card 1: the flaky API. Write a flow that calls a public API in a task with three retries. Point it at a bad URL and read the retry entries in the logs. Then fix the URL and compare the two runs in the dashboard.

Card 2: the cache. Write a task that sleeps for three seconds and returns a value. Give it INPUTS + TASK_SOURCE with a one-hour expiry. Run the flow twice. Note the duration difference and identify the Cached state. Edit the task body and observe the miss.

Card 3: fan-out. Use .map() over ten items with a two-second sleep each. Confirm the flow finishes in around two seconds. Add a shared argument with unmapped().

Card 4: serve and die. Serve a flow with an interval schedule, stop the process, and watch runs become Late. Restart it and see what happens to the backlog.

Card 5: a worker. Create a process pool, start a worker, and deploy a flow from a Git repository with from_source. Trigger it with the CLI. Then stop the worker and trigger again, and locate the problem using only prefect work-pool inspect.

Card 6: break it on purpose. For each of these, write down the message you get: a wrong entrypoint, a missing dependency, a bad parameter type, a stopped worker.

Card 7: secret and variable. Store one of each, read both in a flow, and find them in the dashboard. Decide what you would do if the variable held something that turned out to be sensitive.

Card 8: explain it. Explain to a friend, in two minutes and without the word "orchestrator", what Prefect would change about a cron script.

A debugging routine that works

When a run goes wrong, follow the same short routine every time. It is deliberately boring.

First, find the run: in the dashboard's Runs page, or with prefect flow-run ls --state Failed. Second, note its final state. Failed means exception; go to the traceback. Crashed means outside interference; go to the worker or container logs and resource limits. Late or stuck in Scheduled means nothing picked it up; go to the work pool. Cancelled means someone or something asked it to stop.

Third, read the last error in the logs, not the first. A chain of retries produces repeated lines; the final traceback is the one that explains why retries were exhausted. Fourth, reproduce it locally by calling the flow as a function with the same parameters; if it fails there, you have a code bug, and if it passes there, the difference is the environment: dependencies, credentials, files or network. Fifth, change one thing and run again.

The environment difference deserves a special mention because it is the classic beginner trap. Code that works on your laptop and fails on a worker is nearly always a dependency missing on the worker, a secret or environment variable that only exists locally, or a file path that exists only on your machine. List what your laptop has that the worker does not, and the answer is usually in that list.

If you are using Prefect Cloud from behind a company proxy and see certificate errors, do not disable verification. Point SSL_CERT_FILE at the trusted certificate bundle your IT team provides. Disabling TLS checks is a development-only setting.

Before you deploy: the checklist

• I use Prefect 3 documentation and no 2.x commands.
• prefect version shows the version I expect, and the client is not newer than the server.
• PREFECT_API_URL ends in /api, and prefect config view shows the backend I intend.
• Every network call is a task with retries and a timeout.
• Tasks are safe to run twice.
• Parameters have type hints, and large data is passed by reference.
• I know whether each task should be cached, and I chose the policy on purpose.
• Results that must survive the run are persisted to shared storage.
• Schedules state their timezone.
• Something will be running the flow: a supervised serve process, or a worker for a matching pool that I have seen online.
• The worker has my dependencies, through a pull step, an image, or the opt-in setting.
• Secrets live in Secret blocks or a secret manager, not in code, Variables or logs.
• prefect.yaml is committed and reviewed.
• I can tell Failed from Crashed from Late, and I know where to look for each.
• I have triggered a run by hand and watched it complete before trusting the schedule.

Once every box is ticked, continue to the Mid-level track, which covers container and Kubernetes pools, job variables, automations and concurrency. If you want a second orchestrator to compare with, the Airflow guide and Dagster guide are the natural next read, and the Docker guide helps before container pools.

This is part two of three. It picks up where Beginner ended and takes the pieces you already use, flows, tasks, deployments and workers, down to the level where you can predict what Prefect will do before it does it. The guide is written against Prefect 3.8 (tested on 3.8.7, released 25 September 2026).

Where this picks up

Each section explains the problem first, then the mechanism, then ends with a Try it you can run in a few minutes.

What a run really is

Beginner treated a flow run as "my function, but observed". The precise picture is that a run is a row in the API's database with a state history, and your Python process is merely the thing that reports transitions for it. Everything Prefect does, from retries to alerts, is a reaction to a state change on that row.

A state has three parts: a type that is one of a small fixed set, a name that is free to be more specific, and optional data or a message. The types matter because logic keys off them. Scheduled, Late, AwaitingRetry, AwaitingConcurrencySlot and Resuming are all of type SCHEDULED. Pending, Submitting and InfrastructurePending are PENDING. Running and Retrying are RUNNING. Paused and Suspended are PAUSED. Completed, Cached and RolledBack are COMPLETED. Failed and TimedOut are FAILED, and Crashed has a type of its own. Cancelling is non-terminal, while Cancelled, the COMPLETED names, FAILED names and Crashed are terminal.

The distinction you will use most is Failed versus Crashed. Failed means your code raised an exception and Prefect has no retries left. Crashed means something outside your code killed the process: a SIGTERM, a KeyboardInterrupt, an out-of-memory kill, a pod eviction. The consequences differ. A retry policy on a task will not help a Crashed run, because the process that would have retried is gone. A Crashed run is an infrastructure ticket, not a code ticket, and the first place to look is the container or pod status, not the traceback.

A run is marked Late when it is still Scheduled fifteen seconds after its start time. Late is not an error state, it is a symptom: nothing picked the run up. Sections below cover the four usual reasons.

How a flow decides its final state

A flow does not fail because a task failed. It ends Failed only if an exception propagates out of the flow function, if it returns a Failed state, or if it returns an iterable containing one. Otherwise it ends Completed. This surprises people migrating from Prefect 2, where task states influenced the flow. In Prefect 3, if you call a task, catch nothing, and the task raises, the exception propagates and the flow fails. But if you use return_state=True or .submit() and never resolve the future, the failure can be silently dropped.

from prefect import flow, task

@task
def risky(n: int) -> int:
    if n == 3:
        raise ValueError("bad input")
    return n

@flow
def strict():
    futures = risky.map(range(5))
    return futures.result()          # raises on the failed one, so the flow fails

@flow
def tolerant():
    futures = risky.map(range(5))
    states = [f.state for f in futures]      # wait for all, never raises
    return [s for s in states if s.is_failed()] or None

strict fails the flow when it asks for results, because .result() re-raises. tolerant inspects states and decides for itself. Neither is wrong, but you must pick deliberately. The bug to avoid is submitting futures and returning without touching them, then wondering why a green flow hid a red task.

Retries, timeouts and failing well

Retries exist to absorb transient faults: a dropped connection, a rate limit, a pod that was not ready. They are a poor tool for deterministic bugs, because a bug fails identically every time. The skill at this level is to retry only what can recover, and to space attempts so you do not hammer a struggling dependency.

The task decorator accepts retries, retry_delay_seconds and retry_jitter_factor. The delay may be a single number, a list of numbers used in order, or a callable. The bundled helper exponential_backoff from prefect.tasks produces a growing series. Jitter adds randomness so that a hundred mapped tasks that all failed at the same moment do not all retry at the same moment.

from prefect import task
from prefect.tasks import exponential_backoff

def only_transient(task, task_run, state) -> bool:
    exc = state.result(raise_on_failure=False)
    return isinstance(exc, (ConnectionError, TimeoutError))

@task(
    retries=4,
    retry_delay_seconds=exponential_backoff(backoff_factor=10),
    retry_jitter_factor=0.5,
    retry_condition_fn=only_transient,
    timeout_seconds=120,
)
def fetch_features(url: str) -> dict:
    ...

The retry_condition_fn is the piece most teams miss. Without it, a ValueError from bad data retries four times and wastes twenty minutes. With it, only the exception classes you listed are retried and everything else fails on the first attempt. The function receives the task, the task run and the state, and returns a boolean.

A flow has the same retries and retry_delay_seconds, but think about what a flow retry means: the whole function runs again, and every task that is not cached runs again with it. A flow retry is only cheap if your tasks are idempotent or cached. That is the real reason caching and transactions, next, matter: they make retries safe.

timeout_seconds on a task or flow ends the attempt with state TimedOut, which is in the FAILED type, with a message such as Flow run exceeded timeout of N second(s). Set timeouts on anything that calls out over a network. A hung socket with no timeout is a run that stays Running forever, holds a concurrency slot, and never alerts anyone.

If the default number of retries should be different for a whole project, the setting tasks.default_retries (env var PREFECT_TASKS_DEFAULT_RETRIES, default 0) and tasks.default_retry_delay_seconds apply to tasks that do not set their own. Use the setting sparingly: explicit decorator arguments are easier to find when you read the code.

Results, persistence and caching

Caching is where Prefect 3 differs most from what you would guess, so it deserves precision.

Persistence is off unless something turns it on

A task returns a value, and by default Prefect keeps it in memory for the life of the run (cache_result_in_memory is true) but does not write it anywhere. Result persistence is off by default: results.persist_by_default (env var PREFECT_RESULTS_PERSIST_BY_DEFAULT) is false. That is why state.result() on a run fetched later from the API can raise MissingResult, with the message that state data is missing because persistence was disabled and the state was retrieved from the API.

Setting persist_result=True on a task or flow, or setting a cache policy, turns persistence on for it. Where the bytes go is decided by result_storage, which takes a block or a slug string such as "s3-bucket/my-block", or by the default block setting, or, when neither is set, by the local path setting. Since 3.8.6 the local path defaults to $PREFECT_HOME/storage, which is ~/.prefect/storage unless you moved PREFECT_HOME.

Local disk is a trap the moment your flow runs on more than one machine. A task that persisted to worker A's disk cannot be read by a retry that landed on worker B, and a Kubernetes pod's disk vanishes with the pod. For anything that runs on dynamic infrastructure, configure shared storage and name it explicitly.

Serialisation defaults to pickle (cloudpickle). The alternatives are json, compressed/pickle and compressed/json, set per task or flow with result_serializer, or globally with PREFECT_RESULTS_DEFAULT_SERIALIZER. JSON is readable and portable but only handles JSON types. Pickle handles almost anything but ties the file to Python versions and to your code, so do not treat pickled results as a long-term data format. The result_storage_key argument lets you name the file, for example "hello-{parameters[name]}.pickle", which is useful when another process needs to find it.

Cache policies are values you can combine

The default task cache policy, called DEFAULT, builds a key from the task's inputs, its source code, and the flow run ID. The run ID is the reason two separate flow runs do not share a cache by default: the key changes with every run. People then expect caching to work across runs, add nothing, and see no hits. To share across runs you must drop the run ID.

The policies live in prefect.cache_policies: DEFAULT, INPUTS, TASK_SOURCE, FLOW_PARAMETERS and NO_CACHE. They combine with +, and you can exclude parameters with -.

from datetime import timedelta
from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(
    cache_policy=(TASK_SOURCE + INPUTS) - "debug",
    cache_expiration=timedelta(hours=6),
    result_storage="s3-bucket/team-results",
    persist_result=True,
)
def build_table(day: str, debug: bool = False) -> str:
    ...

Reading that policy aloud: the key depends on the task's code and its inputs, but not on the debug parameter, so toggling debug never busts the cache. The key no longer includes the run ID, so a run tomorrow can hit the entry from today, until it expires after six hours. Editing the function body changes the source hash, so a code change correctly invalidates stale results. That last property is the quiet benefit of including TASK_SOURCE: you rarely have to remember to clear a cache after a bugfix.

Two escape hatches exist. cache_key_fn(context, parameters) computes the key yourself, which you need when an input is not hashable. When Prefect cannot hash an input it raises HashError, and the fix is either INPUTS - 'client' for the offending parameter or a custom key function. refresh_cache=True forces a recompute and overwrites the stored entry, which is what a manual "rebuild" parameter should do.

A cache hit shows the state Cached, which is a COMPLETED name. If a task you expected to run shows Cached, you are seeing the default policy. Use cache_policy=NO_CACHE on tasks that must always run, such as a notification or a write to an external system.

Transactions, rollback and idempotency

Retries and caches make repeated execution safe for computation. Transactions make it safe for side effects. A transaction groups tasks so that they succeed or fail as a unit, and gives you a hook to undo work when the group fails.

from prefect import flow, task
from prefect.transactions import transaction

@task
def write_staging(day: str) -> str:
    path = f"s3://lake/staging/{day}.parquet"
    ...
    return path

@write_staging.on_rollback
def remove_staging(txn):
    ...          # delete whatever the task wrote

@task
def publish(path: str) -> None:
    ...

@flow
def daily(day: str):
    with transaction(key=f"daily-{day}"):
        path = write_staging(day)
        publish(path)

If publish raises, the on_rollback hook on write_staging runs and the transaction ends in the RolledBack state, a COMPLETED name, so the cleanup itself counts as handled. The key gives the transaction an identity. On a rerun with the same key, tasks that already committed are not redone, which is how you get idempotency without writing the bookkeeping yourself. The matching on_commit hook runs when the transaction commits, and is the natural place for a final notification.

Transactions do not make an external system transactional. If your rollback hook deletes a file, it is best effort: the hook might fail, or the process might be killed first, which is a Crashed run and no hook at all. Design the side effects themselves to be safe to repeat, for example by writing to a deterministic path and replacing rather than appending. The transaction is the seatbelt, not the brakes.

Concurrency inside a run: runners, futures and mapping

Tasks called normally block. To overlap them you call .submit() for one future or .map() over an iterable for a list of futures. The default task runner is ThreadPoolTaskRunner, which is right for I/O-bound work such as API calls and database queries, and wrong for CPU-bound work because of the interpreter's global lock. For CPU-bound work there is a built-in ProcessPoolTaskRunner (added in the 3.4 series), and the prefect-dask and prefect-ray packages provide DaskTaskRunner and RayTaskRunner for distribution across machines. You set the runner on the flow: @flow(task_runner=ProcessPoolTaskRunner()).

Futures in Prefect 3 are synchronous: do not await task.submit(). Resolve them with .result() or .wait(), with wait and as_completed from prefect.futures, or by passing a future into a downstream task, which resolves it for you and also records the dependency for the run graph.

Mapping has two traps. First, .map() requires iterables of equal length, and raises MappingLengthMismatch if they differ, or MappingMissingIterable if you pass only static arguments. Constants that should not be iterated go inside unmapped(). Second, an unmapped object is shared by reference, not copied, so if a mapped task mutates it, every other mapped task sees the change. Treat unmapped values as read-only.

Finally, think about grain. Every task run is a set of API calls and events. A flow that maps a hundred thousand trivial tasks produces a hundred thousand state histories, which is slow for you and expensive for the server. Batch rows into chunks and map over chunks, or keep trivial steps as plain Python functions inside a task. A rule that works: a task should do enough work that its overhead is lost in the noise, and deserve its own retry.

Work pools, queues and workers

flow.serve() is simple: one long-lived process polls for runs and executes them as subprocesses. Its cost is that the process must always be running, on infrastructure you manage, and every flow shares it. Work pools separate the decision to run from the place where it runs.

A work pool has a type and a base job template. The type decides what is created for each run: a process, a Docker container, a Kubernetes Job, an ECS task, a Cloud Run job, a Vertex AI job or an Azure Container Instance. The template is JSON with two keys, job_configuration, which holds the real infrastructure spec with {{ variable }} placeholders, and variables, a JSON schema describing those placeholders and their defaults. A variable has effect only if job_configuration references it, which is the cause of a common puzzle: you add a variable to the schema, set it on a deployment, and nothing changes because nothing uses it.

There are three families of pool. Hybrid pools need a worker that you run. Push pools, available on Cloud, let Prefect submit to your cloud account directly with no worker. Managed pools run on Prefect-managed or Coiled compute, also Cloud only. This guide concentrates on hybrid pools because they work the same on self-hosted and Cloud.

# See the default template for a type, edit it, then create the pool from it
prefect work-pool get-default-base-job-template --type kubernetes --file k8s.json
prefect work-pool create etl-k8s --type kubernetes --base-job-template k8s.json

# Inspect and operate
prefect work-pool ls --verbose
prefect work-pool inspect etl-k8s
prefect work-pool set-concurrency-limit etl-k8s 20
prefect work-pool pause etl-k8s
prefect work-pool preview etl-k8s --hours 24

work-pool preview shows the runs scheduled to hit the pool in the next window, which is the quickest way to size a pool. pause stops workers from picking up new runs without touching runs already in flight, and it is the right emergency brake during an incident.

Work queues subdivide a pool. Every pool has a default queue. A queue has a priority, a unique positive integer where 1 is the highest, and an optional concurrency limit. When a worker has capacity it serves higher priority queues first. Use queues when one pool must mix urgent and bulk work, for example so that a nightly backfill cannot starve an on-demand request. Queue status is READY if a worker polled within the last minute, NOT_READY otherwise, or PAUSED.

prefect work-queue create urgent --pool etl-k8s --priority 1 --limit 5
prefect work-queue create backfill --pool etl-k8s --priority 10 --limit 2

The worker

A worker is a small polling process. It asks the API for scheduled runs in its pool and queues, and for each one creates the infrastructure named by the pool type. The worker does not run your code, except for the process type. It creates a Job or container that runs it. Workers make only outbound connections to the API, which is why they can live behind a firewall with no inbound rule.

pip install prefect-kubernetes
prefect worker start --pool etl-k8s --work-queue urgent --work-queue backfill \
  --name etl-worker-1 --limit 10 --with-healthcheck

The worker's type must match the pool's type, and the matching integration package must be installed: prefect-docker, prefect-kubernetes, prefect-aws for ECS, prefect-gcp for Cloud Run and Vertex AI, prefect-azure for Container Instances. If the package is missing the worker prompts to install it, governed by --install-policy (always, if-not-present, never or prompt, default prompt). In a container, use never or pre-install, because a prompt has no one to answer it. If you see Unable to start worker. Please ensure you have the necessary dependencies installed, this is the cause.

--limit caps how many runs this worker has in flight. --prefetch-seconds lets the worker start creating infrastructure slightly before the scheduled time, hiding container start latency. The worker sends a heartbeat every 30 seconds by default, and after three missed heartbeats, about 90 seconds, it shows as offline. The worker queries for work every few seconds; the setting worker.query_seconds is documented inconsistently across official pages, so check prefect config view --show-defaults before quoting a number.

Why a run stays Scheduled

When runs go Late, walk this list in order. Is a worker running at all, for this pool? Does its type match? Is the pool or the queue paused, in which case the worker logs The work pool '<name>' is currently paused? Has a concurrency limit, on the pool, queue, worker or deployment, been reached? prefect work-pool inspect and the worker's own log answer the first three in under a minute. The fourth shows as the AwaitingConcurrencySlot state.

Deployments and prefect.yaml

A deployment is the server-side record that says which flow, which code, which infrastructure, which parameters and which schedule. Two paths create one. The Python path calls flow.deploy(...) or flow.from_source(...).deploy(...). The declarative path writes a prefect.yaml and runs prefect deploy. Teams that review infrastructure in pull requests usually prefer the declarative file, because the diff shows what changed.

Prefect 2's prefect deployment build and apply, Deployment.build_from_flow, storage blocks and agents are all gone. If a tutorial uses them it is for the wrong version.

The anatomy of the file

name: churn-pipelines
prefect-version: 3.8.7

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker
      image_name: registry.example.com/ml/churn
      tag: "{{ $GIT_SHA }}"
      dockerfile: auto

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker
      image_name: "{{ build_image.image_name }}"
      tag: "{{ build_image.tag }}"

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/churn

definitions:
  work_pool: &k8s_pool
    name: etl-k8s
    job_variables:
      image: "{{ build_image.image }}"
      env:
        ENVIRONMENT: "{{ prefect.variables.environment }}"

deployments:
  - name: score-daily
    entrypoint: flows/score.py:score
    parameters: { model_stage: Production }
    tags: [churn, daily]
    schedules:
      - cron: "0 5 * * *"
        timezone: "Asia/Riyadh"
    concurrency_limit:
      limit: 1
      collision_strategy: CANCEL_NEW
    work_pool: *k8s_pool

  - name: score-adhoc
    entrypoint: flows/score.py:score
    work_pool:
      <<: *k8s_pool
      work_queue_name: urgent

Read it top to bottom. build runs once at deploy time, on your machine or CI runner, and its step outputs are addressable by id, so {{ build_image.image }} flows into the job variables. push uploads the image. pull runs on the worker at run time, before the flow starts, and is where code arrives. Here the code is baked into the image, so pull only sets the working directory. definitions uses plain YAML anchors to avoid repeating the pool block, and deployments is a list, each with its own entrypoint in the form path/file.py:function.

Templating has four sources. Step outputs are {{ step-id.field }}. Blocks are {{ prefect.blocks.<type-slug>.<name> }}. Variables are {{ prefect.variables.<name> }}. Environment variables are {{ $ENV_VAR }}. The important timing detail: environment templates resolve at deploy time, on the machine running prefect deploy, while block and variable references resolve at run time on the worker. So a {{ $GIT_SHA }} bakes in the commit of the deploying CI job, and a variable reference always reads the latest value when the run starts. Use that distinction to keep secrets out of git: a reference to a block is committed, the secret it resolves to is not.

Code delivery: bake or pull

There are two models for getting code to a run, and the choice drives much else. Bake: build an image containing the code and dependencies, reference it by tag, and promote the same image across environments. Startup is fast and the run is reproducible, at the cost of a build per change. Pull: use an existing image and let a pull step fetch the code, typically git_clone, at run time. Iteration is fast because there is no image build, at the cost of dependency management and a dependence on the repository being reachable at run start.

With pull, one change in 3.7.3 bites: automatic installation of dependencies for pulled code became opt-in. A deployment that used to install requirements.txt on its own now starts without the packages and fails with a ScriptError or ModuleNotFoundError. Either add an explicit pip_install_requirements step after the clone, or set PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=true. The explicit step is better, because the dependency install is then visible in the file.

pull:
  - prefect.deployments.steps.git_clone:
      repository: https://github.com/example/churn.git
      branch: main
      access_token: "{{ prefect.blocks.secret.github-token }}"
  - prefect.deployments.steps.pip_install_requirements:
      directory: "{{ git_clone.directory }}"
      requirements_file: requirements.txt

If you omit pull, prefect deploy generates one for you. Writing pull: [] disables pull steps completely, which is what you want when the image already holds the code. Mixing these up is a classic cause of MissingFlowError and Unexpected exception encountered when trying to load flow.

Deploy commands and the Python path

prefect deploy                                   # interactive, pick a deployment
prefect deploy --name score-daily                # one by name
prefect deploy --all                             # every deployment in the file
prefect deploy -n "score/*"                      # pattern match
prefect --no-prompt deploy --name score-daily    # for CI: never prompt
prefect deployment run score/score-daily --param model_stage=Staging --watch

In CI, use --no-prompt so a missing choice fails the job rather than waiting forever. deployment run accepts --param key=value or a --params JSON string, --start-in, --job-variable key=value to override infrastructure for one run, and --watch to stream until the run finishes. The Python equivalent for a pull deployment is flow.from_source(source=..., entrypoint="flows/score.py:score").deploy(name=..., work_pool_name=...), and for a baked image flow.deploy(..., image=..., build=True, push=True), where build=False and push=False skip each stage.

Parameters and validation

The deployment stores a JSON schema derived from the flow's signature. A run with parameters that do not fit fails as ParameterTypeError and goes from Pending straight to Failed without ever Running. That is helpful, and surprising the first time. If you change a flow's signature and do not redeploy, runs fail with SignatureMismatchError: Function expects parameters [...] but was provided with parameters [...]. The remedy is to redeploy. Use Pydantic models for structured parameters: they validate and give the UI a form.

Schedules, time zones and the scheduler's horizon

A deployment can carry several schedules, each of which is a cron expression, an interval or an RRule, with its own timezone, parameters and active flag. Multiple schedules are useful: the same flow hourly by day and daily by night, with different parameters on each.

Time zones are where schedules go wrong. Cron and RRule schedules keep local wall-clock time, so a 0 9 * * * schedule in Asia/Riyadh fires at 09:00 local all year. Saudi Arabia and most Gulf states do not observe daylight saving, so this matters less there than for a team in Europe or the US, but Egypt's rules have changed, so always set a named zone rather than an offset. Interval schedules behave differently: under 24 hours they follow UTC elapsed time, and intervals of 24 hours or more keep wall-clock time. The day_or flag on cron controls whether day-of-month and day-of-week combine with OR. RRule strings are limited to 6,500 characters and do not support COUNT: use UNTIL.

prefect deployment schedule create score/score-daily --cron "0 5 * * *" --timezone "Asia/Riyadh"
prefect deployment schedule ls score/score-daily
prefect deployment schedule pause score/score-daily --all

The scheduler is a service with a horizon

Schedules are materialised as future runs by a server service, not computed at trigger time. It keeps a run buffer for each deployment controlled by four settings: at most 100 runs ahead (PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS), at least 3 (..._MIN_RUNS), at most 100 days ahead (..._MAX_SCHEDULED_TIME, P100D) and at least one hour (..._MIN_SCHEDULED_TIME, PT1H). Two consequences follow. A every-minute schedule fills its 100-run buffer quickly and is replenished continuously, so pausing a schedule leaves already-created Scheduled runs that you may need to delete. And because runs exist in advance, work-pool preview and the UI can show you what is coming.

If the scheduler service is disabled, for example on API replicas started with --no-services, and no other process runs services, nothing gets scheduled and nothing tells you so. Always run services somewhere.

Events, automations and alerting

Prefect emits an event for each state change, with a name such as prefect.flow-run.Completed, a resource identifying what it is about, optional related resources and a payload. You can emit your own with emit_event from prefect.events, or with prefect event emit on the command line. Events are how Prefect knows anything happened, and events are what automations listen to.

An automation has a trigger and one or more actions. Trigger kinds are reactive (an event happened), proactive (an expected event did not happen within a window), metric-based on Cloud, compound and sequence. Actions include running a deployment, pausing or resuming deployments, work pools, queues and other automations, cancelling, suspending or changing the state of a flow run, sending a notification, and calling a webhook. Templates use Jinja, for example {{ flow_run.name }}.

The proactive trigger deserves special attention, because it detects the failure mode nothing else does: the flow that silently did not run. A reactive alert on Failed never fires if the schedule was paused, the worker was down or the deployment was deleted. A proactive trigger that says "alert me if there is no Completed event for this deployment in 26 hours" catches all of them.

Since 3.4.0 the old flow-run notification policies are gone and became automations. If you want "tell Slack when a run fails", that is an automation with a send-notification action, or a webhook action toward Teams or PagerDuty.

prefect automation ls
prefect automation inspect "Notify on failed scoring" --yaml
prefect automation create --from-file automations/failed-scoring.yaml
prefect automation pause "Notify on failed scoring"

A deployment trigger is an automation defined on a deployment, through the triggers: key in prefect.yaml or the triggers= argument in Python. It keeps the rule with the code that owns it, in version control, instead of as UI state that nobody can review. Chaining is the classic use: start the scoring deployment when the training deployment emits a completion event. Because the server evaluates triggers, it keeps working when the original process has crashed, which hooks cannot promise.

Events have a retention of seven days by default (server.events.retention_period, P7D), which bounds how far back a proactive window or a forensic query can reach on self-hosted servers, and is also the biggest driver of database growth.

Concurrency controls

Prefect has four concurrency mechanisms, and confusing them is a common source of "why is my run waiting". Each answers a different question.

Deployment concurrency takes a limit and a collision_strategy. With ENQUEUE, extra runs wait in AwaitingConcurrencySlot until a slot frees up. With CANCEL_NEW, the new run is cancelled instead. Use ENQUEUE for work that must eventually happen and CANCEL_NEW for "skip this tick if the last one is still going", which is usually right for a frequent schedule with variable duration.

Global concurrency limits are named slot pools you occupy in code with a context manager. They are the general tool: limit simultaneous connections to a database, cap calls to a fragile API, or serialise access to a file across unrelated flows.

from prefect import task
from prefect.concurrency.sync import concurrency, rate_limit

@task
def query_warehouse(sql: str):
    with concurrency("warehouse", occupy=1, timeout_seconds=60):
        ...

@task
def call_llm(prompt: str):
    rate_limit("llm-api", occupy=1)      # the limit needs slot_decay_per_second
    ...

prefect global-concurrency-limit create warehouse --limit 4
prefect gcl create llm-api --limit 10 --slot-decay-per-second 2
prefect gcl ls
prefect concurrency-limit create gpu 2       # tag-based

concurrency() holds slots for the duration of the block, while rate_limit() consumes slots that refill at slot_decay_per_second, so a rate limit only works if the limit has that decay set. Async code imports from prefect.concurrency.asyncio. Slots are leases that are renewed while you hold them, and strict=True fails the run if a lease cannot be renewed, rather than carrying on without the guarantee. A crashed holder does not leak its slot forever: the lease expires and the server's repossessor service reclaims it.

Tag-based limits attach to a tag, as in @task(tags=["gpu"]) plus prefect concurrency-limit create gpu 2. Since 3.4.19 they are backed by a GCL called tag:gpu, so you see them in both places. Two traps: a limit of zero aborts runs immediately rather than queueing them, and tasks blocked on a tag slot retry on an interval of 30 seconds, so a saturated tag adds latency in chunks of that size.

Configuration: settings, profiles and secrets

Prefect's behaviour is governed by settings, each of which has a TOML key and an environment variable name, such as api.url and PREFECT_API_URL. Precedence, highest first, is: environment variables, then a .env file, then prefect.toml, then the [tool.prefect] table in pyproject.toml, then the active profile, then defaults. Knowing the order is the answer to "I set it, why is it ignored": an environment variable in the container always beats the profile you configured on your laptop.

A profile is a named group of settings in ~/.prefect/profiles.toml, created with prefect profile create, selected with prefect profile use, and showing the default profiles ephemeral, local, test and cloud. Profiles are for a person switching between backends. For CI and containers, use environment variables, because they are explicit and leave no state on disk. For project-wide defaults that belong in git, use prefect.toml.

prefect config view --show-sources --show-defaults
prefect config set PREFECT_API_URL="http://prefect.internal:4200/api"
prefect profile create staging --from local
prefect profile use staging

The --show-sources flag is the debugging tool here: it says where each value came from. The failure No Prefect API URL provided. Please set PREFECT_API_URL to the address of a running Prefect server means the client had no URL and ephemeral mode was off. The URL must end in /api. Forgetting the suffix produces confusing 404s.

Variables, blocks and secrets

Three places hold configuration, and each has a job. Variables are named JSON values for non-sensitive settings, limited to 5,000 characters serialised, and not encrypted. Secret blocks hold one encrypted string. Credential blocks, such as AwsCredentials, hold typed fields, with secret fields stored encrypted and shown obfuscated. The rule is simple: nothing sensitive in a Variable, ever, because anyone who can read variables reads it in plain text.

from prefect.blocks.system import Secret
from prefect.variables import Variable

token = Secret.load("github-token").get()
env = Variable.get("environment", default="dev")

In Kubernetes or ECS, you can also inject secrets through job_variables.env from the platform's own secret store rather than through Prefect, which keeps the secret out of Prefect's database altogether. For an EU, Gulf or Egyptian employer with data-residency rules, that is often the requirement: the hybrid model already keeps your data in your own account and sends only metadata to the API, but credentials stored as blocks are metadata too, so decide on purpose where they live.

The 3.5.0 release removed the old system blocks JSON, String and DateTime. Use Variables.

Running the server yourself

The open-source server is one process with an API, a UI and a set of background services, backed by a database. The defaults, SQLite and everything in one process, are fine for a laptop and a small team's first pipelines. They stop being fine when you need multiple API replicas, because SQLite cannot back more than one server, or when run volume grows.

For production, the database is PostgreSQL 14.9 or newer with the pg_trgm extension. Since 3.8.0 Prefect forces READ COMMITTED sessions on it. The connection string is PREFECT_SERVER_DATABASE_CONNECTION_URL, and the pool defaults of 5 plus 10 overflow per process should be checked against Postgres max_connections multiplied by the number of replicas. Use the canonical PREFECT_SERVER_* names. The PREFECT_API_DATABASE_* spellings seen in older posts are legacy aliases.

The services are the scheduler, late-run marker, foreman (worker and queue health), cancellation cleanup, pause expiration, the repossessor for concurrency leases, the task-run recorder, the event persister, the triggers service for automations, and an optional database vacuum. When you run prefect server start, they run in the same process. To scale, split them.

# API replicas (several), no background services
export PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false
prefect server start --host 0.0.0.0 --no-services

# Services (one or more) as a separate process
prefect server services start

# Migration, as a one-off job before rolling the new version
prefect server database upgrade -y

With several processes, in-memory messaging no longer connects them, so events and leases need Redis, through the prefect-redis package: set PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging, the matching cache and causal-ordering settings, the lease storage, and the Redis connection variables. Duplicate scheduled runs in a high-availability setup are a known symptom of several services processes sharing the default in-memory Docket backend, cured by pointing PREFECT_SERVER_DOCKET_URL at Redis. Put a load balancer in front with health checks on /api/health for liveness and /api/ready for readiness, where ready checks the database and returns 503 when it is down. Proxies must pass WebSocket upgrade headers, and behind one you set PREFECT_SERVER_UI_API_URL.

The open-source server has no users and no roles. The options are a shared HTTP basic-auth string (PREFECT_SERVER_API_AUTH_STRING on the server, PREFECT_API_AUTH_STRING on clients), CSRF protection, and whatever your reverse proxy or SSO gateway adds. The server binds to 127.0.0.1 by default. If you need per-user roles, SSO and audit, that is the reason to consider Prefect Cloud, which has workspaces, roles and service accounts. Check its current plan packaging on the pricing page rather than trusting this guide, since it changes.

Upgrades have one rule: keep the client version at or below the server version within a major version. A client newer than its server can receive 422 responses, and a major mismatch fails with Found incompatible versions. Upgrade the server, then workers and images. Back up Postgres first, and run migrations as a separate job. Release notes matter: the notable breaks this year were the opt-in dependency install in 3.7.3 and the new UI default in 3.8.0.

Testing flows

Because flows and tasks are ordinary Python, most of your tests should be ordinary Python tests. A task is a function. A task object exposes the raw function as .fn, so you can test the logic without any orchestration. A flow can simply be called, which runs it against a temporary local server. The layers of a good test suite therefore are: unit tests of the pure functions you wrote under the tasks, a handful of flow-level tests that run the real orchestration, and a deploy check in CI.

import pytest
from prefect.testing.utilities import prefect_test_harness
from flows.score import normalise, score

@pytest.fixture(autouse=True, scope="session")
def harness():
    with prefect_test_harness():
        yield

def test_normalise_task_logic():
    assert normalise.fn([1, 2, 3]) == [0.0, 0.5, 1.0]

def test_flow_returns_failed_state_for_bad_input():
    state = score(model_stage="nope", return_state=True)
    assert state.is_failed()

The harness starts an isolated temporary database so your tests do not touch your real server or pollute its history. Using return_state=True turns an exception into an inspectable state, which lets you assert on failure behaviour without pytest.raises. Write tests for the properties that matter for operations: that the retry condition rejects a ValueError, that a cache key ignores a debug flag, that a rollback hook removes what the task wrote.

Debugging beyond the logs

Start from the state and the message, not from the traceback. prefect flow-run inspect <id> prints the run with its state, and --output json (available since 3.6.21) makes it scriptable. prefect flow-run logs <id> --tail --num-logs 200 reads the end of the log without opening the UI, and prefect flow-run watch <id> follows a run live. prefect flow-run retry <id> reruns a failed run from the CLI.

The table maps the symptom to where the answer usually lives.

Crashes deserve a routine. The worker logs Flow run infrastructure exited with non-zero status code N. Exit code 137 is a SIGKILL, which for containers nearly always means the kernel killed the process for exceeding its memory limit. The fix is a bigger limit or less memory, set through job variables, not more retries. A pod that never reaches Running, stuck with an image pull error, shows InfrastructurePending on the run, and kubectl describe pod explains why.

For corporate networks that intercept TLS, SSL: CERTIFICATE_VERIFY_FAILED from the client is fixed by giving Prefect the right certificate authority file through SSL_CERT_FILE or PREFECT_API_SSL_CERT_FILE. The setting that disables verification exists, PREFECT_API_TLS_INSECURE_SKIP_VERIFY, but it is for development only.

Production patterns and the neighbouring tools

Prefect is an orchestrator, so its value in a stack is what it connects. The patterns below are the ones that show up repeatedly.

Containers and Kubernetes. A Docker work pool starts a container per run, and a Kubernetes pool creates a Job. Build the image with the Docker guide practices: multi-stage, pinned base, tagged by commit. The prefecthq/prefect images are published with tags such as 3-latest and versioned variants, and pinning prefect==3.8.7 in your own image keeps workers and flows aligned with the server. Run the worker as a Deployment in the cluster with the official prefect-worker Helm chart from https://prefecthq.github.io/prefect-helm, which runs non-root with a read-only root filesystem by default. For Kubernetes jobs, set resource requests and limits in the base job template and use finished_job_ttl so completed Jobs are cleaned up. See the Kubernetes guide and the Helm guide for the cluster side.

Infrastructure as code. Pools, variables, automations and deployments are configuration, and configuration belongs in version control. The official Terraform provider prefecthq/prefect manages workspaces, work pools, deployments, blocks, variables and automations. The Terraform guide covers the workflow. A sensible split: Terraform owns pools and their job templates, which change rarely and have blast radius, while prefect.yaml in each project owns deployments, which change with code.

GitOps delivery. Let CI run prefect deploy --all --no-prompt after the image is pushed, authenticating with a service account key in PREFECT_API_KEY on Cloud or the basic-auth string on self-hosted. If your platform team runs Argo CD, it is natural to manage the worker's Helm release that way while the application repositories publish deployments from CI.

ML systems. A typical shape has Prefect orchestrate the steps and delegate the ML bookkeeping to specialised tools. Tracking and registry belong in MLflow: log the run inside a task and pass the run ID downstream as a small string, not the artefact. Data versioning belongs in DVC, and monitoring in Evidently, where a flow computes a drift report and calls create_markdown_artifact or create_table_artifact from prefect.artifacts to put a summary on the run page. Coming from another orchestrator, the official docs include an Airflow migration guide, and the Airflow guide helps with vocabulary: DAG becomes flow, operator becomes task, pool becomes work pool.

Putting it all together

Here is one pipeline that uses most of this guide. A daily scoring flow runs on a Kubernetes work pool at 05:00 Riyadh time. Its deployment limits concurrency to one with CANCEL_NEW, so a slow day cannot pile up. The flow reads features in a retried, timeout-protected task that only retries connection errors. Scoring tasks are cached on task source and inputs, minus a debug flag, with results persisted to an object store block, so a retry after a crash redoes only what is missing. The write to the serving table happens inside a transaction keyed by the date with an on_rollback hook that removes partial output, into a deterministic path. A global concurrency limit protects the warehouse, and a rate limit protects the model API. The run publishes a markdown artifact with row counts and drift numbers.

from datetime import timedelta
from prefect import flow, task, get_run_logger
from prefect.artifacts import create_markdown_artifact
from prefect.cache_policies import INPUTS, TASK_SOURCE
from prefect.concurrency.sync import concurrency
from prefect.tasks import exponential_backoff
from prefect.transactions import transaction

def transient(task, task_run, state) -> bool:
    exc = state.result(raise_on_failure=False)
    return isinstance(exc, (ConnectionError, TimeoutError))

@task(retries=3, retry_delay_seconds=exponential_backoff(backoff_factor=10),
      retry_jitter_factor=0.5, retry_condition_fn=transient, timeout_seconds=300)
def read_features(day: str) -> list[dict]:
    with concurrency("warehouse", occupy=1, timeout_seconds=120):
        ...

@task(cache_policy=(TASK_SOURCE + INPUTS) - "debug",
      cache_expiration=timedelta(hours=12), persist_result=True,
      result_storage="s3-bucket/team-results")
def score_chunk(rows: list[dict], debug: bool = False) -> list[float]:
    ...

@task
def write_scores(day: str, scores: list[float]) -> str:
    path = f"s3://lake/scores/{day}.parquet"
    ...
    return path

@write_scores.on_rollback
def undo_write(txn):
    ...

@flow(log_prints=True, timeout_seconds=3600)
def score(day: str, model_stage: str = "Production") -> str:
    logger = get_run_logger()
    rows = read_features(day)
    chunks = [rows[i:i + 1000] for i in range(0, len(rows), 1000)]
    scored = score_chunk.map(chunks)
    flat = [s for f in scored for s in f.result()]
    with transaction(key=f"score-{day}"):
        path = write_scores(day, flat)
    create_markdown_artifact(f"Scored {len(flat)} rows to {path}", key="score-summary")
    logger.info("done")
    return path

Around it, prefect.yaml holds the deployment, a proactive automation alerts on absence, and CI deploys to staging first.

Where you are now

You can now reason about a run from the API's side, choose retries, caching and transactions deliberately, wire code to infrastructure through pools and prefect.yaml, control concurrency, run a server on Postgres, test flows and debug from states and exit codes.

The Senior level treats Prefect as a platform for other teams: scaling limits, multi-tenancy, security, upgrades, cost and when to choose something else.

Sources

• Prefect documentation
• Install Prefect
• Prefect 3.8 release notes
• Prefect 3.7 release notes
• Flows
• Tasks
• States
• Caching
• Results
• Transactions
• Deployments
• Work pools
• Workers
• Schedules
• Automations
• Global concurrency limits
• Tag-based concurrency limits
• Settings and profiles
• Settings reference
• prefect.yaml
• Running flows with Docker
• Running flows on Kubernetes
• Self-hosted Prefect
• Storing secrets
• Prefect Helm charts
• Prefect pricing

Part two of three. A cumulative review of Beginner and Mid-level material on Prefect 3.8, organised by scenario rather than level, in about thirty-five minutes. Interviewers at this level do not ask what a flow is. They describe a broken pipeline and watch how you narrow it down.

Where this picks up

The model in one screen

The API stores every run and its states. Workers and serve processes connect outbound and never need an inbound port. Code and data stay on your infrastructure, and only metadata goes to the API, whether that is Prefect Cloud or your own server. If you can say that in two sentences and draw the chain above, you have passed the first minute of most interviews.

Scenario questions

"The nightly flow did not run last night and nobody was told. Where do you look?"

A strong answer starts from the run, not from the code, and walks a fixed order. First, did a run exist? Open the deployment in the UI. If no run was created, the schedule is paused or missing, or the scheduler service is not running, which can happen when API replicas were started with --no-services and nobody runs services elsewhere. If a run exists and is Scheduled, then Late after fifteen seconds, nothing picked it up: check that a worker is running for that pool, that its type matches the pool, that neither pool nor queue is paused, and that no concurrency limit is holding it in AwaitingConcurrencySlot. prefect work-pool inspect and the worker's log answer most of this. If the run started but sat in Pending or InfrastructurePending, look at the infrastructure: image pull, pod scheduling, quota.

The second half of the answer is what a weak candidate forgets: why nobody was told. A failure alert is reactive, so it never fires for a run that did not happen. The fix is a proactive automation that alerts when no Completed event has arrived for that deployment within a window slightly longer than the schedule. Saying "I would add alerting on absence, not only on failure" is often the sentence that separates the levels.

"A task fails sometimes with a timeout to an internal API. What do you change?"

Add retries with backoff and jitter, and narrow them. Backoff, through exponential_backoff from prefect.tasks, gives the dependency room to recover. Jitter keeps a hundred mapped tasks from retrying in lockstep. A retry_condition_fn restricts retries to connection and timeout errors, so a ValueError from bad data fails at once instead of burning twenty minutes. A timeout_seconds on the call stops a hung socket from holding the run forever. You also keep the call in its own small task, so the retry repeats a request and not an hour of work. If the API is fragile under load, add a global concurrency limit or rate_limit around it, because retries alone can turn a brownout into an outage.

Mention what retries do not cover: a Crashed run, where the process was killed, is not retried by task retry settings because the retrying process no longer exists.

"The flow is green but the output table is missing rows."

This is the green flow with red tasks. In Prefect 3 a flow ends Failed only if an exception propagates or a Failed state is returned. If tasks were submitted with .submit() or .map() and the futures were never resolved, a failed task need not fail the flow. The answer is to resolve the futures, through .result(), wait, or by passing them downstream, and to decide explicitly which failures are fatal. You would also check whether return_state=True was used and the state ignored. A candidate who reaches for "increase the retries" has missed the point.

"Every run recomputes an expensive task even though I set a cache policy."

Three causes, in order of likelihood. The default policy includes the flow run ID, so separate runs never share a key. The fix is to use TASK_SOURCE + INPUTS, which drops the run ID. Second, the result storage is local disk on an ephemeral pod, so the cached entry is gone with the pod and every run misses, silently. The fix is a shared object store through result_storage. Third, an input changes on every run, such as a timestamp or a client object, so the key never repeats; exclude it with INPUTS - 'name' or write a cache_key_fn. Bonus points for noting that caching requires result persistence, which a cache policy turns on automatically, although persistence is otherwise off by default.

"A deployment that worked last month fails at start with ModuleNotFoundError."

If the code is pulled at run time, think of the 3.7.3 change: automatic dependency installation for pulled code became opt-in. The deployment that used to install its requirements on its own now starts without them. The fix is an explicit pip_install_requirements pull step, which I would prefer because it is visible in prefect.yaml, or setting PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=true. The broader answer is to bake dependencies into the image for production, and to use pull only where fast iteration matters more than reproducibility. Also mention checking for version skew: a worker image upgraded past a pinned dependency can produce the same symptom.

"Runs go Crashed and the retries do nothing."

Crashed is an infrastructure state: SIGTERM, an out-of-memory kill, a pod eviction. Retries on a task cannot help, because the process that would retry is gone. You look at the worker log for Flow run infrastructure exited with non-zero status code N, and at the container or pod status. Exit code 137 is a SIGKILL and usually a memory limit. The fix is to raise the limit through job variables or reduce memory, for example by chunking the data. To survive a crash without redoing work, persist results to shared storage and use caching or a keyed transaction, so a rerun skips finished steps. And alert with an automation, not a hook, because an on_crashed hook runs in the process that was killed.

"Two scheduled runs overlap and corrupt each other's output."

Use deployment concurrency with a limit of one. Then choose the collision strategy on purpose: ENQUEUE if every run must eventually happen, CANCEL_NEW if skipping a tick is acceptable. Also make the output safe to repeat, by writing to a deterministic path and replacing it, because concurrency limits reduce the window for collisions but do not make a write idempotent. If the conflict is on a shared resource across different flows, a global concurrency limit around that resource is the right tool, not a deployment limit.

"Everything is slow and the database is huge."

Two sources dominate. Events, which are kept for seven days by default, are the fastest-growing table, so shorten retention on a busy server. And tiny tasks: each task run records states and events, so a flow that maps a hundred thousand trivial tasks generates enormous traffic. Batch rows into chunks, or keep trivial steps as plain functions. On Postgres, tune autovacuum on high-churn tables and use pg_repack rather than VACUUM FULL. On Prefect Cloud, the same pattern shows up as 429 responses against plan rate limits, and the client's retry with Retry-After only hides it.

"The client says it cannot talk to the server, or gets 422 responses."

Check versions first. The major versions must match, or you get Found incompatible versions. Within a major, a client newer than its server can receive 422 responses, so the rule is client at or below server, upgrading the server first. Then check PREFECT_API_URL, which must end in /api. Then check the profile versus environment precedence with prefect config view --show-sources, because an environment variable in a container overrides the profile on a laptop.

Trade-off questions

Serve versus a work pool

serve is the smallest thing that works: one process, runs as subprocesses, no pool. It fits a single flow on a box you already run, and static infrastructure is sometimes exactly what you want. A work pool gives dynamic, per-run infrastructure with its own resources, isolation between flows, queues with priorities, and scale-to-zero when nothing is running. The price is a worker to operate and a template to maintain. A good answer names the trigger for moving: different flows needing different images or resources, or the serve process becoming a single point of failure.

Bake code into the image or pull it at run time

Baking gives reproducibility and fast startup and lets you promote one immutable image through environments, at the cost of a build on every change. Pulling gives fast iteration with no build, at the cost of dependency management, a network dependency at run start, and the 3.7.3 opt-in install. A common answer is to pull in development and bake for production.

Prefect Cloud versus self-hosted

Self-hosting costs you Postgres, Redis for multi-process setups, upgrades and on-call, and the open-source server has no users or roles, only an optional shared basic-auth string. Cloud removes that operating burden and adds workspaces, roles, service accounts and, on higher plans, SSO and audit, with plan limits on rate and retention. The hybrid model means execution stays in your account either way. Data-residency concerns usually turn on what metadata and credentials you store centrally, not on where code runs. Pricing and plan packaging change, so a good candidate says to check the current page.

Variables versus blocks versus environment variables

Variables are unencrypted JSON for non-sensitive configuration. Secret and credential blocks are encrypted at rest and obfuscated in the UI. Environment variables injected by the platform, for example Kubernetes Secrets through job_variables.env, keep the secret outside Prefect completely. The answer that impresses: never put a secret in a Variable, and prefer the platform's secret store when policy demands it.

Hooks versus automations

Hooks run in the run's process, are defined in code, and are easy to test, but die with the process. Automations are evaluated by the server, survive crashes, can react to the absence of events and can chain deployments, but live in configuration that you must keep in version control. Use hooks for in-process cleanup, and automations for alerting and orchestration across runs.

Task runner choice

Threads suit I/O-bound work and are the default. ProcessPoolTaskRunner suits CPU-bound work on one machine. Dask or Ray runners, from prefect-dask and prefect-ray, suit work that must spread across machines. A strong answer also asks whether the work should be a task at all: if each unit is a millisecond, the orchestration overhead exceeds the work.

What a strong answer covers that a weak one misses

Commands to have ready

prefect version
prefect config view --show-sources
prefect flow-run ls --state-type FAILED
prefect flow-run inspect <id> --output json
prefect flow-run logs <id> --tail --num-logs 200
prefect flow-run watch <id>
prefect work-pool inspect <pool>
prefect work-pool preview <pool> --hours 24
prefect worker start --pool <pool> --with-healthcheck
prefect deploy --all
prefect --no-prompt deploy --name <name>
prefect deployment run <flow>/<deployment> --param k=v --watch
prefect gcl ls
prefect automation ls

Being able to say what each one is for, and what you expect it to show, matters more than reciting flags.

Common interview questions

What is the difference between a flow and a deployment? A flow is code. A deployment is a server-side record of which flow to run, from where, on what infrastructure, with which parameters and schedules. You cannot trigger a flow remotely without one.

What does a worker actually do? It polls the API for scheduled runs in its pool and queues and creates infrastructure for each, such as a container or Kubernetes Job. It sends heartbeats and is shown offline after three missed ones, roughly ninety seconds with the default interval.

What are job variables? Per-deployment or per-run overrides for the variables defined in the pool's base job template, such as image, environment or CPU. They replaced infra_overrides.

How do you pass secrets to a flow running on Kubernetes? Through a Secret block loaded in the flow, or by injecting a Kubernetes Secret as an environment variable through job variables. Not through a Variable, which is unencrypted.

How does a schedule handle daylight saving? Cron and RRule schedules keep local wall-clock time in their named time zone. Interval schedules under twenty-four hours follow elapsed UTC time, and longer intervals keep wall-clock time. Use named zones, not offsets.

What happens if two scheduled runs collide? Whatever the deployment's concurrency settings dictate: no limit means both run, a limit with ENQUEUE queues the second, and CANCEL_NEW cancels it.

How would you test a flow? Unit test the logic through a task's .fn, run a few real flows against the test harness's temporary database, use return_state=True to assert on failures, and run a deploy step against staging in CI so a broken entrypoint fails the build.

How do you upgrade a self-hosted server? Back up Postgres, upgrade the server first, run migrations as a separate job with prefect server database upgrade -y when running several replicas, then upgrade workers and images, keeping the client at or below the server version.

What changed recently that you would warn a team about? Opt-in dependency installation for pulled code in 3.7.3, the new UI becoming the default for self-hosted servers in 3.8.0, and the removal of the old system blocks in 3.5.0. Checking release notes before upgrading is itself a good answer.

How do you stop one team's flows starving another's? Separate work pools or queues with priorities and limits. A follow-up worth making: these are not security boundaries. On the open-source server one server is one tenant.

Sixty-second self-test

1. Name the states of type PENDING and say what puts a run into Late.
2. Give two reasons a cache policy produces no hits.
3. Which part of prefect.yaml runs on the worker, and which at deploy time?
4. List four reasons a run stays Scheduled.
5. When does {{ $ENV_VAR }} resolve, and when does {{ prefect.variables.x }}?
6. What does a proactive trigger catch that a reactive one cannot?
7. What is the version rule between client and server?
8. Why does rate_limit need slot_decay_per_second?

If you can answer all eight without looking, you are ready for the Senior level, which covers running Prefect as a platform. For the layers underneath, see the Docker and Kubernetes guides.

Part two of three. The errors people make once they are past the basics of Prefect 3.8, the practices that pay back most, and recipes you can paste into a real project. Each error has a symptom, a cause and a fix.

Where this picks up

Errors at this level

A green flow hides a failed task

Symptom. The flow is Completed but output is missing. Cause. Futures from .submit() or .map() were never resolved, so nothing raised inside the flow. Fix. Call .result(), use wait, or pass the futures to a downstream task. Decide which failures are fatal, and inspect future.state for the ones that are not.

Retrying a bug

Symptom. A task fails the same way four times and the run takes twenty minutes to report it. Cause. retries with no retry_condition_fn, so a ValueError retries like a timeout. Fix. Add a condition function that returns true only for connection and timeout errors.

Caching never hits

Symptom. An expensive task recomputes on every run. Cause. The default policy includes the flow run ID, or the inputs change each time, or the result storage is ephemeral disk. Fix. Use TASK_SOURCE + INPUTS, exclude volatile parameters with -, and point result_storage at an object store block.

Caching hits when it should not

Symptom. A task that sends an email shows Cached and does not send. Cause. The default policy caches on inputs and source. Fix. Mark side-effect tasks with cache_policy=NO_CACHE, and give a rebuild parameter refresh_cache=True.

MissingResult

Symptom. State data is missing when you read a result from a run you fetched later. Cause. Persistence is off by default, so the value existed only in memory. Fix. persist_result=True with shared result_storage, or PREFECT_RESULTS_PERSIST_BY_DEFAULT=true.

HashError when caching

Symptom. Prefect cannot hash an argument, often a client or connection. Fix. Exclude it, as in INPUTS - 'client', or write a cache_key_fn. Better, create clients inside the task.

Runs go Late and nobody knows why

Symptom. Runs sit Scheduled, then Late. Cause. No worker for the pool, a type mismatch, a paused pool or queue, or a full concurrency limit. Fix. prefect work-pool inspect, read the worker log, and check for the currently paused message. A worker that started without its integration package fails with Unable to start worker; install prefect-kubernetes, prefect-docker or the right package, or set --install-policy.

Crashed is debugged as Failed

Symptom. Hours spent reading a traceback that does not exist. Cause. Crashed means an infrastructure problem. Fix. Read the worker's exited with non-zero status code N line and the container status. Code 137 is usually an out-of-memory kill, so raise the limit through job variables.

Pulled code without its dependencies

Symptom. ModuleNotFoundError or ScriptError at start on a deployment that used to work. Cause. Since 3.7.3, automatic dependency installation for pulled code is opt-in. Fix. Add a pip_install_requirements pull step, or bake dependencies into the image.

Deployment parameters out of date

Symptom. SignatureMismatchError: Function expects parameters [...]. Cause. The flow signature changed and the deployment still holds the old schema. Fix. Redeploy, and make that part of your pipeline.

Variable that does nothing

Symptom. You add a variable to a work pool's template and set it on a deployment, but nothing changes. Cause. job_configuration does not reference it with {{ variable }}. Fix. Reference it in the configuration, or it is ignored.

Secrets in Variables

Symptom. A token visible to every user who can read variables. Cause. Variables are unencrypted. Fix. Use a Secret or credential block, or inject a platform secret through job variables, and rotate what leaked.

Wrong API URL

Symptom. No Prefect API URL provided, or odd 404s. Cause. No URL set, or the /api suffix is missing. Fix. prefect config set PREFECT_API_URL=http://host:4200/api, then prefect config view --show-sources to see which layer wins.

Version skew

Symptom. Found incompatible versions, or 422 responses. Cause. A client newer than its server, or a major mismatch. Fix. Upgrade the server first and keep the client at or below it.

Tag limit of zero

Symptom. Tasks abort immediately. Cause. A tag concurrency limit of 0 aborts rather than queues. Fix. Use a positive limit, or pause the work some other way.

Duplicate scheduled runs on a scaled server

Symptom. Two runs at every tick. Cause. Several services processes share the default in-memory Docket backend. Fix. Set PREFECT_SERVER_DOCKET_URL to a Redis URL.

Enormous parameters

Symptom. 413 Content Too Large on Cloud, or failures past 512 KB on a self-hosted server. Fix. Pass a URI, not the data.

The practices that pay off most, in order of leverage

1. Make side effects idempotent. Deterministic output paths, replace rather than append, and keyed transactions. Every other reliability feature is safer on top of this.
2. Persist results to shared storage. It unlocks caching, cross-machine retries and inspecting results later.
3. Alert on absence. One proactive automation per critical deployment.
4. Set timeouts on everything that touches a network. A hang is the failure nobody sees.
5. Retry narrowly. A condition function, backoff and jitter, on a small task.
6. Put deployments in prefect.yaml and deploy from CI with --no-prompt. Reviewable, repeatable, and a broken entrypoint fails the build.
7. Bake dependencies into images for production. Pull for development speed only.
8. Pin versions. Pin prefect==3.8.7 in images, and keep clients at or below the server.
9. Pick the concurrency mechanism by the question. Pool limit, deployment limit, tag limit or named global limit.
10. Keep tasks meaningful. Enough work to justify a state history and a retry boundary.

Why these practices rank where they do

The ranking above is by leverage, which is the amount of pain removed per hour spent. It is worth understanding the reasoning, because your own pipeline may reorder it.

Idempotency first, because everything else assumes it. Retries, flow retries, reruns after a crash, prefect flow-run retry, a second worker picking up a run after a network partition: all of them execute code more than once. If running twice produces a different result from running once, each of those safety features becomes a way to corrupt data. A deterministic output path and a replace-on-write rule make repetition harmless, and then you can be generous with retries. Teams that skip this step end up turning retries off, which trades one class of incident for another.

Shared result storage second, because it makes recovery cheap. When a flow of twenty tasks crashes at task fifteen, the difference between rerunning fifteen tasks and rerunning one is the difference between a forty-minute delay and a two-minute one. It needs three things together: persisted results, storage that every worker can reach, and a cache key that stays stable across attempts. Miss any one and the recovery quietly degrades to a full rerun. This is also why you should test recovery on purpose: kill a worker mid-run in staging and watch what the rerun skips.

Absence alerts third, because silent failure is the expensive kind. Most teams discover their alerting gap when a business user asks why yesterday's numbers are stale. A proactive automation costs ten minutes and closes the gap for paused schedules, dead workers, deleted deployments and scheduler outages together. Tune the window to the schedule plus a realistic margin, since a window that is too tight pages people for slow days and teaches them to ignore it.

Timeouts fourth, because a hang holds resources. A run that never finishes keeps its concurrency slot, its pod, and its place in the deployment's limit, and with ENQUEUE it blocks every run behind it. A timeout turns an invisible hang into a TimedOut state that your alerts already understand.

Narrow retries, pinned versions and baked dependencies follow because they reduce surprise rather than recover from it. A retry condition stops wasted time. A pinned prefect==3.8.7 stops a worker image from drifting ahead of a server you have not upgraded yet. Baked dependencies stop the 3.7.3 class of failure, where a platform default changes and a previously working deployment breaks at five in the morning.

Habits for the first week on a new project

Start by reading what exists before writing anything. Run prefect config view --show-sources so you know which backend you are talking to, which is how people avoid testing against production by accident. Run prefect work-pool ls --verbose and prefect deployment ls to see the real topology, then open one recent run in the UI and follow its graph from state to logs to the worker that ran it.

Next, find where the secrets are. Look at blocks, variables and any environment injection in the job templates. If you find a token in a Variable, raise it immediately, since every reader of variables can see it. Check that the CI credential is a service account, not a colleague's browser-login key that expires after thirty days and will fail on a weekend.

Then look at the alerts. List the automations with prefect automation ls and ask two questions of each critical deployment: what tells us when it fails, and what tells us when it does not run? If the answer to the second is nothing, you have found your first contribution.

Finally, write the flow's failure story down in one paragraph in the repository. Which tasks are safe to retry, which writes are idempotent, where results live, who gets paged. Writing it exposes the gaps faster than reviewing the code, and it is the paragraph the on-call engineer will read at three in the morning.

Reading a failed run like a checklist

When a run fails, resist the urge to open the log first. Work through four questions in order. What is the final state name and type? Failed, TimedOut and Crashed point at different owners. Did it ever reach Running? A run that goes from Pending straight to Failed failed validation or could not load code, and its log will be short. Where did it run? The worker name and the infrastructure identifier tell you which machine or pod to inspect. And what changed? A new image tag, a changed parameter, a dependency release, an upgraded server or a different profile explain most regressions, and prefect deployment inspect shows the current configuration so you can compare it with what you expected.

Only then read the log, starting from the end. Prefect logs flow and task messages in order, and the error you want is usually the last exception, with earlier warnings giving context. If the log is empty, the process likely died before logging started: think image pull, entrypoint errors or an out-of-memory kill at startup, and look at infrastructure output rather than Prefect's.

Mistakes that only show up with growth

Some problems stay invisible at ten runs a day and appear at a thousand. Task granularity is the clearest: a flow that maps a task per row is fine for a hundred rows and a burden at a million, because each task run records states and events. Batch into chunks, and keep trivial work as plain Python. Logging volume behaves the same way, since every log line a flow emits goes to the API unless you switch that off, and on Cloud the plan's rate limits apply. Keep logs to what a person would read, and route bulk output to files or your log platform.

Retention is the third. Events are kept for seven days by default on a self-hosted server, and the database grows with run volume. Decide your retention deliberately, enable maintenance, and tune Postgres before disk alarms make the decision for you.

Practice cards

Operational recipes

Find out why a run is stuck

prefect flow-run inspect <run-id>
prefect work-pool inspect <pool>
prefect work-pool preview <pool> --hours 6
prefect flow-run logs <run-id> --tail --num-logs 100

Read the state name, then the pool status, then the worker's own output. If the worker is healthy and the pool is not paused, look for a concurrency limit in the run's state message.

Inspect the state of everything that failed today

prefect flow-run ls --state-type FAILED --output json
prefect flow-run ls --state Crashed

The type filter catches Failed and TimedOut together, and the name filter isolates Crashed, which goes to a different owner.

A deploy step for CI

- step:
    name: Deploy flows
    script:
      - pip install "prefect==3.8.7" prefect-docker
      - prefect cloud login --key "$PREFECT_API_KEY" --workspace "$PREFECT_WORKSPACE"
      - prefect --no-prompt deploy --all

The same two commands work in any CI system. On a self-hosted server replace the login line with PREFECT_API_URL and, if you use basic auth, PREFECT_API_AUTH_STRING. The PREFECT_WORKSPACE variable here is your own name for the account/workspace handle.

A retry policy to copy

from prefect import task
from prefect.tasks import exponential_backoff

def transient(task, task_run, state) -> bool:
    exc = state.result(raise_on_failure=False)
    return isinstance(exc, (ConnectionError, TimeoutError))

@task(retries=4, retry_delay_seconds=exponential_backoff(backoff_factor=10),
      retry_jitter_factor=0.5, retry_condition_fn=transient, timeout_seconds=120)
def call_api(url: str) -> dict:
    ...

A cache policy that survives across runs

from datetime import timedelta
from prefect import task
from prefect.cache_policies import INPUTS, TASK_SOURCE

@task(cache_policy=(TASK_SOURCE + INPUTS) - "debug",
      cache_expiration=timedelta(hours=6),
      persist_result=True, result_storage="s3-bucket/team-results")
def expensive(day: str, debug: bool = False):
    ...

Protect a fragile dependency

prefect global-concurrency-limit create warehouse --limit 4
prefect gcl create llm-api --limit 10 --slot-decay-per-second 2

from prefect.concurrency.sync import concurrency, rate_limit

with concurrency("warehouse", occupy=1, timeout_seconds=60):
    ...
rate_limit("llm-api", occupy=1)

Run a worker on Kubernetes

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm repo update
kubectl create secret generic prefect-api-key --namespace=prefect --from-literal=key=<API_KEY>
helm install prefect-worker prefect/prefect-worker --namespace prefect -f values.yaml

The values file names the work pool and, for Cloud, the account and workspace IDs. See the Helm guide for values management and the Kubernetes guide for RBAC and resources.

Scale a self-hosted server

export PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false
prefect server start --host 0.0.0.0 --no-services      # several API replicas
prefect server services start                           # services, separately
prefect server database upgrade -y                      # one-off migration job

Multiple processes need Redis for messaging, and a load balancer health-checking /api/health and /api/ready. Put this into your Terraform or chart values, not into a runbook that nobody reads.

Move a flow from another orchestrator

List what the old DAG did, then map it: the DAG becomes a flow, operators become tasks, sensors become events or schedules with a proactive automation, pools become work pools. Move one pipeline at a time, keep the old one running until the new one has completed enough cycles to trust. The Airflow guide gives the vocabulary and the official docs include an Airflow migration guide.

A checklist before you ship a deployment

• Every network call has a timeout, and only transient errors retry.
• Output paths are deterministic, and the write is inside a keyed transaction if it must be undone.
• Result storage is shared, and named, wherever caching or cross-machine retry matters.
• The deployment has a concurrency limit and a collision strategy chosen on purpose.
• The schedule has a named time zone.
• A proactive automation alerts on absence, and a reactive one on failure.
• Secrets are in blocks or platform secret stores, never Variables.
• Dependencies are in the image, or an explicit pull step installs them.
• prefect --no-prompt deploy runs in CI against staging first.
• prefect==3.8.7 is pinned in the image and the client is not newer than the server.

Where to go next

The Senior level covers running Prefect as a platform: the architecture and its failure modes, scaling limits, the security model, multi-tenancy, cost and upgrades, and when another tool is the better choice. If you are still building foundations, revisit Docker, because most Prefect production problems turn out to be container problems, and the MLflow guide for the tracking half of an ML pipeline.

This is part three of three. It takes every topic from the Beginner and Mid levels one step further, to the point where someone else's pipelines depend on the Prefect you run. The questions change. You are no longer asking how to make a flow retry. You are asking what happens to four hundred scheduled runs when the database fails over, who is allowed to change a work pool's job template, and how you upgrade the server without stopping anyone's night shift.

Everything here is written against Prefect 3.8, tested on 3.8.7. Where a fact depends on your plan or deployment, the text says so, and where it is cheap to check on your own system, it gives you the command.

Where this picks up

I am starting with the architecture, because every failure mode in this guide follows from where state lives and who talks to whom.

Architecture and failure modes

Prefect is a hybrid system by design. A control plane, either Prefect Cloud or a server you host, stores metadata: flow runs, task runs, states, schedules, logs, events and configuration. An execution plane, your workers, serve processes and containers, runs the code. Code and data stay in your environment, and only metadata travels to the API. Workers make outbound connections only, so your execution side needs no inbound ports.

The self-hosted server is one process with several jobs. It serves the API and the UI, and it runs background services: the scheduler that creates future runs, late_runs that marks runs Late, foreman that watches worker and queue health, cancellation_cleanup, pause_expirations, the repossessor that reclaims expired concurrency leases, task_run_recorder and event_persister that write telemetry, and triggers that evaluates automations. A separate db_vacuum service trims old data. By default they all live inside prefect server start, which is exactly right for a laptop and exactly wrong once several people depend on it.

Each component fails differently, and the failure mode is what you need to know in advance.

The database is the single point of failure. If PostgreSQL is down, the API's /api/ready endpoint returns 503, nothing can be scheduled, workers cannot claim runs, and flows that are already running cannot report state. Runs in progress usually keep executing, because your code does not depend on the API to compute, but their state updates stall, and the client retries with backoff. The practical consequence is that your recovery time objective for Prefect is your recovery time objective for Postgres. Managed Postgres with automated failover is the cheapest way to meet it.

The API tier is stateless and therefore boring. Replicas behind a load balancer fail and recover with no special handling, as long as the events messaging layer is shared (more on that shortly).

The services tier fails quietly. If no process is running the scheduler, your API is perfectly healthy and no new runs appear. If two processes run services without a shared coordination backend, you can see duplicated scheduled runs. This is the most common architecture bug in a first self-hosted deployment, and the fix is in the HA section below.

Workers fail in a way that looks like a scheduling problem. A worker that dies leaves its pool with scheduled runs and nobody polling. A worker sends a heartbeat every 30 seconds by default, and is marked offline after missing three, so roughly 90 seconds. Runs that were not yet claimed become Late once they are still Scheduled 15 seconds past their start time. Runs that were already claimed and whose infrastructure died end in Crashed, or are detected by the flow run heartbeat, which since 3.6 defaults to 180 seconds (PREFECT_FLOWS_HEARTBEAT_FREQUENCY).

Flow run infrastructure fails in ways Prefect can only observe. An out-of-memory kill, an evicted pod or an image pull failure are infrastructure events, not code exceptions. Prefect represents them as Crashed, never Failed, and that distinction is the first thing to teach your users, because the fix lives in resource limits and images rather than in retry logic.

Scaling: which component breaks first

For a self-hosted server the order of failure is predictable, and it is worth memorising because it tells you where to spend money before it hurts.

First, the events table. Every state change on every flow and task run produces events, and the event_persister writes them to the database. They are retained for seven days by default (PREFECT_SERVER_EVENTS_RETENTION_PERIOD, default P7D). The official guidance for deployments above roughly a thousand runs a day is to shorten that to a day or two. This table grows faster than any other, and it is the usual cause of a slow UI and a busy database long before you run out of API capacity.

Second, the connection pool. Each API replica holds a SQLAlchemy pool. The defaults are a pool size of 5 and a maximum overflow of 10 (PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE and PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW). Multiply the worst case, 15 connections, by the number of API replicas and service processes, and compare it with Postgres max_connections. Adding replicas without checking this sum is how a scale-out produces connection errors.

Third, task-run volume. Each task run records states and emits events, so a flow that maps a tiny task across a million items creates a million task runs. That is a design problem, not a capacity problem. The remedy is to batch: make the task process a chunk of a thousand items, or call plain Python functions for trivial steps and reserve @task for units where you want retries, caching, or visibility.

Fourth, table bloat. Postgres autovacuum must keep up with a high-churn schema. The official database maintenance guidance suggests lowering autovacuum_vacuum_scale_factor to the 0.05 to 0.1 range on high-churn tables, and using pg_repack rather than VACUUM FULL, because the latter takes a lock that stalls the API. The built-in db_vacuum service deletes old events by default (PREFECT_SERVER_SERVICES_DB_VACUUM_ENABLED); the flow-run vacuum is off unless you turn it on, so finished runs accumulate forever until you decide on a retention policy.

Fifth, worker throughput. Workers are cheap, and workers on one pool coordinate through the API, so you scale out by running more replicas. The levers are --limit on a worker, concurrency limits on the pool and queue, and the polling interval. Do not quote a polling default from memory, because the settings reference and the concept page disagree on PREFECT_WORKER_QUERY_SECONDS. Ask your own installation:

prefect config view --show-defaults | grep -i worker

There is a scheduling horizon too. The scheduler keeps at most 100 runs and 100 days ahead per deployment (PREFECT_SERVER_SERVICES_SCHEDULER_MAX_RUNS, ..._MAX_SCHEDULED_TIME) and aims for at least three runs and an hour ahead. A deployment with a per-minute schedule therefore has about 100 pending runs at any moment, and a large fleet of such deployments is a steady write load you can estimate in advance.

On Prefect Cloud the limits are imposed on you instead of discovered. Per minute, Hobby allows 625 API requests and 2,000 logs and events, Starter 1,250 and 2,800, Team 2,500 and 8,000, and Pro 5,000 and 40,000, with Enterprise custom. Retention of finished runs is 7 days on Hobby and Starter, 14 on Team, 30 on Pro, counted from the terminal state. A request body over 5 MiB is rejected with 413. When a user hits 429, the SDK already retries up to five times honouring Retry-After, so the fix is to reduce chatter, not to add a retry wrapper. Verify these numbers against the live rate limits page before you quote them in a capacity plan, because plans change.

A highly available self-hosted topology

The official scaling guide describes the shape to build, and it is worth following closely because each piece exists to prevent a specific failure.

The API replicas run the server without background services and without automatic migration:

export PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false
prefect server start --host 0.0.0.0 --port 4200 --no-services

The background services run as their own process, or several:

prefect server services start

Migrations run as a one-off job before a rollout, never on every replica at boot:

prefect server database upgrade -y

The reasoning is that if ten replicas each try to migrate a large database on startup, the migrations race each other and a slow one trips the readiness probe. A separate job makes the schema change a deliberate, observable step. For very large databases raise PREFECT_SERVER_DATABASE_MIGRATION_TIMEOUT (for example to 600) on that job. PostgreSQL must be version 14.9 or newer with the pg_trgm extension, and Prefect forces its sessions to READ COMMITTED from 3.8.0.

Redis is what lets the tiers cooperate. With the prefect-redis package installed, you point events messaging, the cache, causal ordering and concurrency lease storage at it:

PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_SERVER_EVENTS_CAUSAL_ORDERING=prefect_redis.ordering
PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage
PREFECT_REDIS_MESSAGING_URL=rediss://redis.internal:6379/0
PREFECT_SERVER_DOCKET_URL=redis://redis.internal:6379/1

The last setting deserves emphasis. The background-task backend, Docket, defaults to an in-memory store. If you run several services processes without pointing them at a shared Redis, each one schedules independently, and you get duplicate scheduled runs. The fix is PREFECT_SERVER_DOCKET_URL, and the symptom to recognise is two runs of the same deployment with the same scheduled time. Redis Sentinel is supported through a redis+sentinel:// URL.

The load balancer should use /api/health for liveness and /api/ready for readiness, since ready checks the database and returns 503 when Postgres is unreachable. It must pass WebSocket upgrade headers for live log streaming, and when the UI sits behind a proxy you set PREFECT_SERVER_UI_API_URL so the browser calls the right address. prefect server start --workers N is a shortcut for multiple processes on one host, but it requires Postgres and Redis, and SQLite can never back more than one server.

On Kubernetes, the official Helm charts package this. prefect-server bundles a PostgreSQL subchart, and you should set postgresql.auth.existingSecret so a chart upgrade never regenerates the database password. prefect-worker runs a worker, and prometheus-prefect-exporter exposes server metrics. The charts run as non-root with a read-only root filesystem by default. If you need a refresher on the machinery, the Kubernetes and Helm guides cover it, and Terraform is the natural place to codify the whole stack.

The security and trust model

Start from the honest statement: the open-source server has no users and no roles. Anyone who can reach the API can do anything the API can do, including creating deployments that run arbitrary code on your workers. That sentence drives the whole design.

What the OSS server gives you. Optional HTTP basic authentication: the server sets PREFECT_SERVER_API_AUTH_STRING to a user:password value and clients set PREFECT_API_AUTH_STRING to match. CSRF protection is available through PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED and is off by default, with tokens expiring after an hour. CORS is permissive by default (*) and worth narrowing through PREFECT_SERVER_API_CORS_ALLOWED_ORIGINS. The server binds to 127.0.0.1 unless you tell it otherwise. API keys are a Cloud concept and do not apply to self-hosted.

That is a shared password, not an identity system. For anything beyond a small trusted team you put the server behind an authenticating proxy, such as an SSO-aware reverse proxy, terminate TLS there, and treat network reachability as the real access control. Custom headers can be attached to client calls with PREFECT_CLIENT_CUSTOM_HEADERS, which is how a client presents a proxy token.

What Cloud adds. Account roles (Owner, Admin, Member) and workspace roles (Owner, Developer, Runner, Viewer and Worker, the last being the minimum scope for a worker). Service accounts with API keys are the right identity for CI and for workers, since a browser login key expires after 30 days and belongs to a person. Higher tiers add SSO, SCIM, audit logs and private connectivity. Packaging shifts, so check the pricing page for which plan includes custom roles before you promise RBAC to a stakeholder.

The trust question that matters most: who can write deployments? A deployment names code to fetch and infrastructure to run it on. Anyone allowed to create or edit one can make your worker execute their code with that worker's credentials. So the security boundary is not the UI, it is the work pool and its infrastructure identity. A worker running in a Kubernetes namespace with a service account that can read every secret in the cluster turns every deployment author into a cluster-secret reader. The defences are structural:

• Give each work pool its own infrastructure identity with the smallest permissions the workloads need.
• Keep production pools on a separate worker fleet, and ideally a separate account or namespace, from experimental ones.
• Restrict who can change a pool's base job template, since the template controls image, service account, volumes and environment.
• Require review for changes to prefect.yaml in the repository that feeds the pool.

Network posture. Workers make outbound calls to the API, to your git host or object store for code, and to the container registry. On Cloud, the allowlist includes app.prefect.cloud, api.prefect.cloud, auth.workos.com and api.github.com. Behind a corporate proxy, set HTTPS_PROXY and supply your certificate authority with SSL_CERT_FILE or PREFECT_API_SSL_CERT_FILE. The escape hatch PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true exists, but it is for development only. A corporate TLS-inspecting gateway that re-signs traffic is the usual reason people reach for it, and the right answer is to install the gateway certificate authority, not to disable verification.

Secrets and the supply chain

Prefect offers several places to put a secret, and they differ sharply. The rule to teach is short: secrets go in encrypted blocks or an external secret manager, and never in variables, parameters, git, or images.

Variables are unencrypted. A variable is a named JSON value meant for configuration, such as an environment name or a feature flag. It is visible to anyone with read access to the workspace. Secret blocks (prefect.blocks.system.Secret) and credential blocks such as AwsCredentials store fields as SecretStr, which is encrypted at rest and obfuscated in the UI. External managers are better still for regulated workloads: read from AWS Secrets Manager with prefect_aws.secrets_manager.AwsSecret, from GCP Secret Manager or Azure Key Vault through their integrations, or inject Kubernetes Secrets into the job through job_variables.env so the value never enters Prefect at all.

The second half of the rule concerns git. In prefect.yaml you reference secrets and blocks through templates, {{ prefect.blocks.secret.github-token }} or {{ $ENV_VAR }}, and they are resolved at deploy or run time. The file in git holds a pointer, never a value. A repository scan for credentials should therefore find nothing in any prefect.yaml, and you should run one in CI to keep it that way.

Parameters deserve a warning of their own. Flow parameters are stored in the database and shown in the UI, and on the OSS server the maximum size is 512 KB (PREFECT_SERVER_API_MAX_PARAMETER_SIZE). Do not pass credentials as parameters, and do not pass bulk data either: pass a URI and let the flow read the object.

The supply chain has three links. The first is the Prefect package itself. Pin it exactly in every image and lockfile, for example prefect==3.8.7, and upgrade deliberately, because the project does ship security-driven dependency bumps in patch releases. The second is the code a worker pulls. A git_clone pull step against a moving branch means production runs whatever was last merged, so pin to a tag or commit for production deployments, or bake the code into an image. The third is the dependencies installed at run time. Since 3.7.3 automatic dependency installation for pulled code is opt-in through PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES, which is good news for supply-chain discipline: if nothing installs at run time, nothing unreviewed can arrive at run time. Prefer images built in CI with locked dependencies, scanned and signed as described in the Docker guide, and use the pip_install_requirements step only for pools where convenience outweighs reproducibility.

A related hazard is the image tag. prefecthq/prefect:3-latest moves under you. Use an exact version tag, and verify the exact tag names on Docker Hub rather than trusting a blog post.

Multi-tenancy

How you share Prefect between teams is the decision that is hardest to reverse, so it is worth choosing deliberately.

On Prefect Cloud, a workspace is the tenancy unit. Blocks, variables, work pools, deployments and permissions are scoped to it, and the number of workspaces depends on your plan. A reasonable starting layout is one workspace per environment, such as dev, staging and prod, rather than one per team. Teams then share a workspace and are separated by convention: prefixes or tags, per-team work pools, and service accounts per team with a workspace role that grants only what they need.

On the open-source server, one server is one tenant. There are no workspaces and no per-team permissions, so stronger separation means separate servers, or at least separate databases. Teams on a shared server are separated only by convention, and a mistake by one team (deleting a block, pausing a pool) affects everyone. That is acceptable for a single platform team running its own pipelines, and a poor fit for distrustful tenants.

Inside a single tenant, work pools are the practical unit of isolation and control. A pool per team or per workload class gives you:

• a separate base job template, so each team gets an image, resources and service account suited to it,
• a separate concurrency limit, so one team's backfill cannot consume the whole cluster,
• a separate pause switch, which is a precise incident tool,
• and a separate worker fleet with its own credentials.

Queues inside a pool add priority, where 1 is the highest, and optional limits. Use them for urgency within a team, not for separation between teams.

Noisy neighbours are the operational face of multi-tenancy. Three controls help. Pool and queue concurrency limits cap how many runs execute at once. Global concurrency limits protect a shared downstream system, such as a warehouse that tolerates ten connections, through a named limit that any flow occupies with concurrency("warehouse", occupy=1). Tag-based task limits, which since 3.4.19 are backed by global concurrency limits named tag:<tag>, protect against a single task type flooding. Remember the odd rule that a tag limit of zero aborts runs instead of queueing them.

Naming and ownership conventions do more than tooling does. Require a team tag on every deployment, use a predictable pool name such as <team>-<env>-<type>, and record an owner in the deployment description. An unowned deployment at three in the morning is an incident with no first responder.

For teams that live in the orchestrator you might be replacing, see Airflow and Dagster for how they model the same problem differently.

Cost

Cost has three parts in Prefect, and only one of them is the orchestrator's price.

The control plane. Open source is free under Apache 2.0, and you pay for Postgres, Redis, the API replicas and the people who run them. Cloud is priced per plan. At the time of writing, Hobby is free with two users and five deployments, Starter is $100 a month, Team is $100 per user per month with four to eight users, and Pro and Enterprise are sold by conversation, with SSO, RBAC, SCIM, PrivateLink and an SLA on Enterprise. Pricing changes often, so quote the live pricing page and not this paragraph. The decision rule is honest and boring: Cloud trades a bill for an on-call rotation. If you cannot staff the rotation for a Postgres-backed service, Cloud is cheaper than it looks.

The execution plane. This is usually the larger number, and the architecture choice drives it. flow.serve() keeps a process running around the clock, so a hundred served deployments are a hundred always-on pods. Workers and work pools create infrastructure per run, so cost tracks actual work and idle cost approaches zero. On Kubernetes, set a finished-job TTL so completed jobs do not pile up, and size requests to real use. Managed and push pools bill in serverless minutes on Cloud.

The data you keep. Events, logs and run history take database space and, on Cloud, count against rate limits and retention. Set a retention policy on purpose.

Regional note: for employers in the Gulf and Egypt, data residency usually decides between Cloud and self-hosting. Prefect Cloud stores metadata, and parameters, logs and result references count as metadata, so review what your flows log and pass before you assume nothing sensitive leaves your region. Code and data stay on your infrastructure in any case, in whichever region you pick for workers.

Upgrades and migrations

Upgrading Prefect is safe when you respect a small number of rules, and painful when you do not.

The ordering rule. Within a major version, keep the client at or below the server. Upgrade the server first, then workers, then the images your flows run in. A newer client talking to an older server can receive 422 responses, because it sends fields the server does not know. The client also warns when the server is older than it is. Across a major version the mismatch is fatal, and you get Found incompatible versions: client: X, server: Y. Major versions must match.

The database rule. Back up Postgres before upgrading. Migrations are Alembic, run automatically on server start unless you disabled that, and in an HA setup you run them in a dedicated job with prefect server database upgrade -y. A downgrade is possible with prefect server database downgrade -y -r -1, but treat it as a last resort and treat the backup as the real rollback. Never run prefect server database reset -y outside a test system: it destroys data.

The release-notes rule. Read the notes for every minor version you cross, because breaking behaviour arrives in minors. The changes that have caught real teams recently are:

A safe upgrade procedure for a platform team looks like this:

1. Read the release notes for each minor between your version and the target, and list every item that touches your usage.
2. Take a Postgres backup and confirm you can restore it somewhere.
3. Upgrade a staging server, run the migration job, and run a representative set of flows, including one that retries, one that pauses and one that triggers an automation.
4. Upgrade production: migration job first, then API replicas, then the services process.
5. Upgrade workers, then publish new images. Keep the old worker image available for a rollback.
6. Watch Late and Crashed counts for a day.

Moving between servers or workspaces. prefect transfer, added in the 3.4 series, moves resources between servers or workspaces. Use it for a Cloud-to-self-hosted move, or consolidating workspaces, and rehearse it on a copy. Secrets do not travel the way configuration does, so plan to re-create credential blocks deliberately. For people coming from 2.x, the upgrade guide applies, and the facts to hold are that agents became workers, infrastructure blocks became typed work pools, Deployment.build_from_flow disappeared in favour of flow.deploy, flow.serve and prefect deploy, and infra_overrides became job_variables. The .publish_as_work_pool() helper converts an old infrastructure block into a work pool.

Governance: making the platform safe to share

Governance is what you build so that the secure, economical path is also the easy one. In Prefect it comes down to four mechanisms.

Infrastructure as code for the platform itself. Work pools, variables, blocks, automations and, on Cloud, workspaces and service accounts can be managed with the official Terraform provider prefecthq/prefect. Putting pools and job templates in Terraform means a change to a pool's service account or memory default arrives through a reviewed pull request with a plan, instead of through a click in the UI. Teams write flows and deployments, and the platform team owns pools. See the Terraform guide for the workflow, and use a GitOps tool such as Argo CD to keep the worker deployment itself reconciled.

Base job templates as the policy surface. A work pool's template defines what every run on it may do. Variables that a team may override, such as image and memory, are exposed in the template's variables schema. Anything you do not expose cannot be overridden through job_variables, because a variable only takes effect if the job configuration references it. That makes the template a policy document: expose image, cpu and memory with sensible maximums in the schema, and hard-code the service account, the namespace and the security context.

Deployment conventions enforced in CI. Because deployments are declared in prefect.yaml, you can lint them. A CI step can check that every deployment has a team tag, an owner in its description, a pinned pull step, no literal credentials, a concurrency limit on anything that hits a shared system, and a work pool from an approved list. This is cheaper and more reliable than a review meeting.

Alerting as a product. Since 3.4, notifications are automations. Offer teams a standard: a proactive trigger that fires when an expected run does not appear, which catches the failure a failure alert can never see, and a reactive trigger on Failed and Crashed. Both send to a channel the team owns. A scheduled job that silently stops scheduling produces no failed run at all, so an absence alert is the only thing that will tell you.

# Illustrative automation, created with: prefect automation create --from-file nightly-missing.yaml
name: nightly-etl-did-not-run
description: Fires if no completed run of the nightly ETL appears within 26 hours
trigger:
  type: event
  posture: Proactive
  expect:
    - prefect.flow-run.Completed
  match_related:
    prefect.resource.name: nightly-etl
  threshold: 1
  within: 93600
actions:
  - type: send-notification
    block_document_id: "<block-document-id>"
    subject: "nightly-etl has not completed"
    body: "No Completed run in the last 26 hours"

Treat that file as a sketch to adapt, and run prefect automation create --help and the automations reference on your version to confirm the field names before you standardise on it. Automation schemas are precisely the kind of detail that shifts between releases.

Incident playbooks

Each playbook starts from a symptom, gives the first check, and then the likely cause. Keep them in the runbook your on-call reads, not in your head.

Runs are piling up in Late.

1. Run prefect work-pool inspect <pool> and prefect work-pool preview <pool>. A paused pool prints a message saying so, and pausing is the commonest cause.
2. Check for a live worker: none means the worker died, the wrong type polls the pool, or credentials expired.
3. Check concurrency: the pool or queue limit may be full, or a global limit may be exhausted by a stuck holder.
4. If the workers are fine, check the late_runs and scheduler services: the API can be healthy while services are down.

Everything is Crashed at once. This is almost never Prefect. Look at the platform beneath it: a node pool scaled down, an image registry outage, a mis-rolled base image, an expired pull secret. The worker logs the non-zero exit code or the pull failure. Fix the infrastructure, then use prefect flow-run retry for the affected runs, or an automation that changes state at scale.

The UI is slow and the database is hot. Check the events table first. Shorten retention, confirm the db_vacuum service is running, check autovacuum health, and look for a flow producing a storm of tiny tasks. A recent deployment of a high-fanout flow is the usual trigger.

Duplicate scheduled runs. More than one services process is running without a shared Docket. Point PREFECT_SERVER_DOCKET_URL at Redis on all of them.

Workers cannot reach the API. You will see connection errors, or No Prefect API URL provided in a container that lost its environment. Confirm the URL ends with /api, that a proxy or certificate authority is configured, and that the account has not hit rate limiting on Cloud (429), where the answer is to reduce chatter.

A flow fails at startup while loading code. The messages include Unexpected exception encountered when trying to load flow and ScriptError. Suspect the entrypoint, pull-step credentials, or a missing dependency. If you upgraded across 3.7.3, suspect lost run-time dependency installation.

A deploy raised MissingResult. Result persistence is off and the state was fetched from the API. Enable persist_result with storage that every machine can reach, and never local disk when retries may land elsewhere.

Stop the bleeding. Three levers exist in increasing severity: pause a deployment's schedule, pause a work pool, and pause an automation. Use the smallest one that stops the damage, and record what you paused so someone resumes it.

Where Prefect stops

A platform owner's credibility comes from saying what a tool is not for. Prefect is an orchestrator for Python workloads, and its edges are worth stating plainly.

It is not a distributed compute engine. Prefect schedules and observes work. It does not shard a dataset across a cluster. For that, hand the heavy step to a compute framework and orchestrate it: the integrations with Dask and Ray exist for this, as do Spark and Databricks submissions. Running a large transformation inside a single flow container is fine until it is not, and the answer is a bigger engine called from Prefect, not a bigger Prefect.

It is not an event-streaming system. It reacts to events and can run small jobs in response to them, but it is not a replacement for a message broker or a stream processor. High-rate, low-latency event handling wants different infrastructure. Background tasks with .delay() and task workers cover a modest middle ground, and you should check the current documentation for their limits before relying on them.

It is not a model registry or an experiment tracker. Log runs and artifacts to a tracker such as MLflow and let Prefect trigger the training. Keep the two responsibilities apart.

It is not a CI system or a deployment system. It can run shell commands, but build pipelines and progressive delivery belong to dedicated tools. A Prefect flow that deploys your application is a sign of a missing CI pipeline.

Its governance on open source is thin. With no users, roles or audit log, a large organisation with compliance obligations has two honest options: Cloud tiers that provide them, or another orchestrator. If you need per-team RBAC on a self-hosted system, compare Airflow, which has its own role model, or Dagster, before you build a proxy layer that approximates it.

Pick something else when the pipeline is primarily SQL transformations in a warehouse with a mature tool already (use that tool's scheduler), when the work is a streaming job (use a streaming engine), when you need strict, audited, per-user access on a self-hosted system and cannot buy Cloud, or when the team has no Python skill and needs a visual builder. Choosing well here is a sign of seniority. Forcing Prefect into those shapes produces the incidents in the previous section.

Running Prefect as a platform for other teams

A platform is a product with users. The habits that separate a Prefect platform from a Prefect installation are mostly about reducing the work your users have to do and the variety you have to support.

Offer paved roads. Provide a template repository with a flow, a prefect.yaml, a Dockerfile, a CI pipeline that builds and pins the image and deploys, and a test. A new team should reach a scheduled, monitored deployment in an afternoon by changing names. Every deviation from the template is something you will later have to support by hand, so make the template genuinely good.

Standardise results and storage. Decide once where persisted results live: a shared bucket per environment, configured with PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK, with a lifecycle rule that expires old objects. Decide the serializer (pickle is the default, JSON is safer to read across versions) and document the trade. Flows that retry or run subflows on different machines need shared storage, and local disk silently breaks them.

Set quotas and defaults at the pool. Default resource requests, a maximum through the template schema, a pool concurrency limit and a default retry policy taught through the shared library. Quotas are a kindness: they turn one team's mistake into that team's slow run instead of everybody's outage.

Measure the platform. The numbers worth tracking are: the count of Late runs and how long runs stay Late, the count of Crashed runs by pool, worker heartbeat health, API latency, database size and events growth, time to first deployment for a new team, and the fraction of deployments that follow the template. The Prometheus exporter chart and the /api/health and /api/ready endpoints provide the raw signals. Emit your own events with emit_event for business milestones, and let automations act on them.

Document the escape hatch. Teams with a genuinely different need should have a documented way to run their own pool, with their own worker, without asking permission, inside guard rails. A platform that only says no pushes users to build shadow orchestrators.

For the surrounding stack, the MLflow guide covers tracking, KServe covers serving what your pipelines train, and Argo CD covers reconciling the workers and charts that run all of it.

The review checklist

Use this for design documents and for pull requests that touch platform configuration.

• Topology. Is the database managed, backed up and restore-tested? Are the API and services separate? Is Redis configured, including PREFECT_SERVER_DOCKET_URL?
• Capacity. Is events retention set deliberately? Does replicas times pool size fit under Postgres connections? Are tiny tasks batched?
• Identity. Does each pool have its own least-privilege infrastructure identity? Who can edit base job templates? Are CI and workers on service accounts?
• Secrets. Are secrets in blocks or a manager, never in variables, parameters, git or images?
• Reproducibility. Is prefect pinned exactly? Are images built in CI and pull steps pinned? Is run-time dependency installation deliberately chosen?
• Protection. Do flows that touch shared systems use a global concurrency limit? Do retries use backoff and jitter?
• Results. Is persisted storage shared, with a lifecycle rule and a serializer decision?
• Alerting. Is there an absence alert for every scheduled flow that matters, and an alert on both Failed and Crashed?
• Ownership. Does every deployment carry a team tag and an owner?
• Change. Is the upgrade ordered server first, tested in staging, preceded by a backup, and announced?

The complete picture

Prefect is a small set of ideas on top of Python: flows and tasks that record their state through an API, deployments that say where and when to run them, work pools and workers that turn a scheduled run into infrastructure, and automations that react to events. As a platform owner you look after the parts that are not Python: a stateless API tier, a services tier that must exist exactly as many times as your coordination backend allows, a Postgres database that is the whole of your durability, and Redis when you scale out. Your users look after their flows, and you look after the pools, templates, identities, limits and alerts that decide whether their flows are safe to run.

The failure modes are few and learnable. Runs go Late when nobody is polling. Runs go Crashed when infrastructure dies beneath them. The UI slows when events and task-run volume outgrow the database. Duplicate runs appear when services lack shared coordination. Upgrades hurt when ordering, backups and release notes are ignored. Each has a first command, a likely cause and a fix, and every one of them is cheaper to prevent in review than to repair in production.

Where the series leaves you

You began by writing a flow and watching it run. You can now decide how an organisation should run Prefect: Cloud or self-hosted, served or dispatched to pools, one tenant or several, pinned images or convenience installs. You can defend each choice on cost, risk and staffing, and you can say when the honest answer is a different tool.

Good next steps are to codify a pool and its template in Terraform, run the worker on Kubernetes through the Helm chart, and connect training runs to MLflow. Keep the release notes open, rehearse one upgrade, and run one game day. The platforms that stay healthy are the ones whose owners practise their failures before the failures arrive.

Sources

• Prefect documentation
• Prefect 3.8 release notes
• Prefect 3.7 release notes
• Prefect 3.6 release notes
• Prefect 3.5 release notes
• Prefect 3.4 release notes
• Self-hosted Prefect and scaling
• Database maintenance
• Security settings
• Prefect server concepts
• Work pools
• Workers
• Automations
• Global concurrency limits
• Settings and profiles
• Prefect Cloud rate limits
• Storing secrets
• Managing roles on Cloud
• Upgrading to Prefect 3
• Deploying with prefect.yaml
• Infrastructure as code
• Settings reference
• Prefect Helm charts
• Prefect pricing

Part three of three. This is the interview preparation for someone who owns Prefect as a platform. The questions are the ones a staff-level or platform-lead interviewer asks: design a system, respond to an incident, defend a trade-off to leadership. Read it once for the shape of a strong answer, then practise saying the answers aloud, because senior interviews are judged on how you reason more than on what you remember. The facts assume Prefect 3.8.

The thirty-second answer

Prefect is a Python orchestrator with a hybrid design. A control plane, either Prefect Cloud or a server you host, stores metadata and schedules. An execution plane of workers and containers that you own runs the code and only makes outbound calls. As a platform owner I look after the database, the services that schedule work, the work pools that turn a scheduled run into infrastructure, the identities those pools use, and the alerts that tell people when something did not run.

If you say nothing else, say this: the database is the one stateful component, the workers are disposable, and the security boundary is the infrastructure identity behind a work pool, not the UI.

The model you need in your head

Four sentences carry most senior answers. Workers poll, so nothing inbound is opened on the execution side. The API tier is stateless, and Postgres holds all durable state. Background services such as the scheduler must be coordinated when there is more than one. And Late means nobody claimed the run, whereas Crashed means infrastructure ended it, whereas Failed means your code raised.

System-design prompts

Design a Prefect platform for forty data and ML engineers across six teams

A strong answer begins with questions, not boxes. How many runs a day, and how many tasks per run? What are the availability and recovery expectations? What are the data-residency rules? Is there a platform team who can carry a pager? Then it commits to a decision.

My default answer: one environment per workspace or server for dev, staging and production. If the company has a compliance need for per-user audit and role-based access and no appetite to run a database, choose Prefect Cloud. If residency or cost rules out Cloud, run the self-hosted topology: several stateless API replicas started with --no-services, one or more prefect server services start processes, a one-off migration job, managed PostgreSQL 14.9 or newer with pg_trgm, Redis for messaging and leases, and a load balancer checking /api/health and /api/ready.

Then tenancy. One work pool per team per environment, each with a base job template that exposes image, cpu and memory with maximums, and hard-codes the service account and namespace. Each pool gets its own worker deployment on Kubernetes and its own least-privilege identity. Teams tag every deployment. Global concurrency limits guard shared systems such as the warehouse.

Then the paved road: a template repository, CI that builds and pins the image and deploys with prefect deploy, and lint rules on prefect.yaml. Then the operations: absence alerts, Failed and Crashed alerts, dashboards for Late counts and database growth, a documented upgrade calendar.

What the interviewer listens for: that you named the single point of failure, separated API from services, raised the Docket and Redis point, chose pools as the tenancy unit, and did not claim that queues are a security boundary.

A weak answer draws one box labelled Prefect server, says "we will scale it with Kubernetes", and never mentions the database.

Design a system that must survive the loss of the database

There are two distinct requirements. Keeping running work running, and getting back to a consistent state.

For the first, the honest answer is that in-flight flow runs usually keep computing, since the code does not depend on the API to do its arithmetic, but they cannot report state while the database is unreachable, and scheduling stops. So the goal is to shorten the outage: managed Postgres with automatic failover and a replica in another zone, connection retry behaviour understood, and readiness probes so that the load balancer stops sending traffic to replicas that cannot reach the database.

For the second, define the recovery point. Backups with point-in-time recovery, restore-tested, with the procedure written down. Explain what happens after a restore: schedules continue from the restored state, runs that started during the gap may be Crashed or stuck Running, and someone has to reconcile them. An automation that watches for long-running runs, and the flow run heartbeat, help detect those.

Finally be explicit about what you would not claim. You would not promise zero data loss without synchronous replication, and you would not promise that in-flight runs always recover.

A team wants to run five hundred small tasks in a mapped flow every minute

The instinct to say "Prefect scales" is wrong here. The accurate answer is that each task run records states and emits events, so this design creates around thirty thousand task runs an hour from one flow, which loads the events table and the API. The design fix is batching: a task processes fifty items at a time, or the trivial step becomes a plain function, and @task is reserved for units that need retries, caching or visibility. If the work is truly fine-grained and high-rate, say so, and consider a different engine for the inner loop with Prefect orchestrating at the batch level.

Follow with the platform response, not just the technical one: add a review question about task fan-out to the pull request checklist, and set a pool concurrency limit so a mistake costs that team only.

Design secret handling for a multi-team platform

Start with the rule: secrets never live in variables, parameters, git or images. Variables are unencrypted. Parameters are stored in the database and shown in the UI.

Then the layers. For secrets that Prefect must hold, use Secret blocks or credential blocks, whose secret fields are encrypted at rest. For regulated workloads, prefer an external manager, read at run time through the provider integration or injected into the job as environment variables from Kubernetes Secrets through job_variables.env. In prefect.yaml, reference secrets through templates so git holds pointers only, and scan the repository in CI to prove it.

Then identity. Each pool's worker identity can read only that team's secrets, which is enforced by the cloud provider's IAM, not by Prefect. That last point is the one that separates seniors from mid-level answers: Prefect is not the enforcement layer for who can read a secret on a shared worker, the infrastructure identity is.

Design the upgrade process for a platform with two hundred deployments

Order: read release notes, back up, upgrade staging, run the migration job, run representative flows, upgrade production server first, then workers, then images. Keep the client at or below the server. Announce breaking items with a concrete check per team, name the ones that matter for the versions crossed (for example that run-time dependency installation for pulled code became opt-in in 3.7.3, and that the V2 UI became the default in 3.8.0), and watch Late and Crashed for a day afterwards. Keep the old worker image available so rollback is a redeploy and not a rebuild.

Incident-response questions

Runs are stuck in Late across the whole platform

I would work from the outside in. First, is this one pool or all pools? All pools points at the control plane or at something shared, such as the scheduler, the database or a network path. One pool points at that pool: check whether it is paused, whether it has a worker, whether the worker's type matches the pool, and whether a concurrency limit is full.

Commands: prefect work-pool inspect <pool>, prefect work-pool preview <pool>, the worker logs, and /api/ready on the API. If workers are polling and the pool is healthy but nothing is claimed, suspect a concurrency limit held by a stuck run, which I would find through the concurrency views and clear deliberately.

A strong answer also states the communication plan: who is told, what they are told, and what you pause to stop the queue getting worse.

Every run that started in the last twenty minutes is Crashed

That pattern points beneath Prefect. I would look at the container platform: node pool scaling, an image registry problem, an expired image pull secret, a bad base image rolled out. The worker logs a non-zero exit code or the pull failure. I would check whether the crashes share a pool, an image or a node group, since the common factor is usually the cause.

Remediation: fix the infrastructure first, then retry the affected runs with prefect flow-run retry, or use an automation to change state in bulk. I would pause the affected pool while fixing it, so that new runs do not crash too. And I would add an alert on Crashed count if there was not one, because failure alerts alone would have been silent.

The UI takes thirty seconds to load and the database CPU is pinned

Check events first, since that table grows fastest. Confirm retention (PREFECT_SERVER_EVENTS_RETENTION_PERIOD, default seven days) and shorten it if you run thousands of runs a day. Confirm the vacuum service is running and that Postgres autovacuum is keeping up. Look for a recent deployment with high task fan-out. Check connection counts against max_connections, because a scale-out can exhaust them.

What I would not do is reset the database, or restart everything hoping it clears. Restarting hides the cause and, when services restart together, can produce duplicates.

Two copies of every scheduled run appeared after we scaled the services tier

The symptom is two runs of one deployment at the same scheduled time. Cause: several services processes using the default in-memory Docket backend, each scheduling independently. Fix: set PREFECT_SERVER_DOCKET_URL to a shared Redis across all of them, along with the Redis messaging settings for events. Then clean up the duplicates, by cancelling the extra scheduled runs, and add a check to your deployment template so the setting cannot be forgotten.

A contractor's deployment ran code that read another team's secrets

The design failure is that the worker identity was shared across teams. I would contain it first: pause the pool and revoke the affected credentials. Then the structural fix: per-team pools with their own least-privilege identities, job templates that hard-code the service account, and review on changes to templates. Then the process fix: CI checks on prefect.yaml, and a decision about whether untrusted contributors need their own server or workspace.

The mature point to make: on the open-source server there are no per-user permissions, so whoever can create a deployment can run code wherever the pool's worker can. The boundary has to be built from infrastructure.

A scheduled flow silently stopped running a week ago and nobody noticed

This is the classic failure that failure alerts cannot catch, since no run exists to fail. Likely causes: the schedule was paused or deleted by a redeploy, the deployment was replaced, or the scheduler service was down. The platform fix is an absence alert, a proactive automation that fires when an expected event does not occur within a window, for every flow that matters. I would add it to the paved-road template so that every new deployment inherits it.

How to argue a trade-off to leadership

Leadership wants a decision, the cost, the risk and what you need from them. Do not teach them orchestration.

Prefect Cloud or self-hosted

State it as money and people. Self-hosting is free in licence terms and costs a database, Redis, compute and an on-call rotation. Cloud costs a subscription and removes that rotation, and on higher tiers provides SSO, roles and audit logs the open-source server does not have. Then the residency question: what metadata leaves the region, and is that acceptable? A recommendation could be: Cloud if we cannot staff the pager or need per-user access control, self-hosted if residency or cost dominates and we already run Postgres well. Give a review date, because pricing and needs change.

Why not just keep the existing orchestrator

Be fair. A migration has a cost, and an existing system that works is an asset. Justify the change by named pain: authoring speed, dynamic workflows, local testing, event-driven triggers. Offer a pilot, with exit criteria agreed in advance, and run both for a period. Claim nothing you cannot measure, such as a fixed percentage improvement. Comparisons with Airflow and Dagster are fair game, and you should know one real strength of each.

Translate each item into an incident it prevents. Managed Postgres with failover prevents a database outage from becoming a platform outage. Restore testing turns a backup from a hope into a fact. Absence alerts prevent a week of missed pipelines. Give a figure for the cost of one missed day of a pipeline the business cares about, and compare.

Trade-off questions with model answers

Serve or work pools? serve is simple and fast to start: one long-lived process runs a flow's schedule, and runs only happen while it is alive. Work pools create infrastructure per run, scale to zero, and isolate runs, but need workers and a job template. For a handful of small flows, serve is fine. For a platform, pools, because of isolation, cost and the ability to pause a pool.

Bake code into the image or pull it at run time? Baking is reproducible, scannable and fast to start, at the cost of a build per change. Pull steps are flexible and need no rebuild, at the cost of a moving target and run-time dependency handling. Since 3.7.3 installing dependencies on pull is opt-in. For production, bake. For development pools, pull.

One big pool or many small ones? Many small ones give isolation, cost attribution and a precise pause switch, at the cost of more workers to run and more templates to maintain. Templates can be managed through Terraform to contain that cost.

Results on local disk or shared storage? Shared, whenever retries or subflows may land on another machine, since local paths vanish with the container. Persistence is off by default, and a cache policy turns it on for the tasks that use it.

Common interview questions

What happens to a running flow if the API goes away? The code keeps executing, since it does not need the API to compute. State updates and logs fail to send and the client retries with backoff. Whether the run ends cleanly depends on how long the outage lasts.

How do you detect a worker that died? The worker heartbeat, 30 seconds by default, and the worker being marked offline after missing three. Unclaimed runs go Late. For runs whose infrastructure died mid-run, the flow run heartbeat, 180 seconds by default since 3.6, helps the server notice.

How do you stop one team flooding a shared database? A global concurrency limit around the section of code that touches it, taken with the concurrency context manager. Combine with a pool limit for the team. If they hammer a rate-limited API instead, use rate_limit with a limit configured with a slot decay.

What changed in Prefect 3 that still trips teams? Agents became workers, infrastructure blocks became work pools, Deployment.build_from_flow is gone in favour of flow.deploy, flow.serve and prefect deploy, infra_overrides became job_variables, and task caching now defaults on inputs, source and flow run, so surprising Cached states appear.

What a strong answer covers that a weak one misses

• The single point of failure, named and protected.
• The difference between Late, Crashed and Failed, and what each implies.
• The services tier and its coordination requirement.
• Identity as the real security boundary, not queues.
• Absence alerting, not only failure alerting.
• Cost in both money and on-call time.
• Change management: ordering, backups, release notes, communication.
• A limit: what Prefect is not for, and when you would pick something else.

Sixty-second self-test

Answer aloud without looking.

1. Name the one stateful component of a self-hosted Prefect, and what happens when it is down.
2. Why must the API replicas and the services processes be separate in an HA setup?
3. What symptom tells you services lack a shared Docket, and what is the fix?
4. What does a Crashed run mean, and why might a Failed-only alert miss an incident?
5. Why are queues not a security boundary?
6. Name three things you would put in a pool's job template as hard-coded rather than overridable.
7. What upgrade order do you follow for server, workers and images?
8. What is an absence alert, and what failure does it catch that nothing else does?
9. Give one situation where you would recommend leaving Prefect.
10. How do you argue Cloud versus self-hosted to a finance lead?

If any answer took longer than a sentence or two, re-read that section of the detailed guide, then try again tomorrow. Related guides worth a pass before an interview are Kubernetes, Terraform, Docker and MLflow.

Part three of three. These are the traps that appear only when Prefect is shared and busy, the habits of people who run it well for other teams, and two review checklists you can paste into a pull request template. Everything assumes Prefect 3.8, tested on 3.8.7. Each trap gives the symptom, the cause and the fix, because at this level the symptom is usually far from the cause.

Production traps that only appear at scale

The events table eats the database

Symptom. The UI slows down over weeks, database storage climbs, and queries that were instant now take seconds.

Cause. Every state change on every flow and task run emits events, and events are retained for seven days by default. At a few thousand runs a day, with many tasks each, that is the largest table by a wide margin.

Fix. Set PREFECT_SERVER_EVENTS_RETENTION_PERIOD on purpose, one to two days for busy servers according to the official guidance, and confirm the db_vacuum service is enabled. Tune Postgres autovacuum on high-churn tables, and use pg_repack instead of VACUUM FULL, which takes a lock. Decide a retention policy for finished flow runs as well, because that vacuum is off by default and history accumulates forever.

Tiny tasks everywhere

Symptom. API request volume and database writes are far higher than run counts suggest. On Cloud, 429 responses appear.

Cause. A @task on a trivial function, mapped over a large collection, creates one task run per item, each with states and events.

Fix. Batch. Process chunks inside a task, and keep trivial steps as plain functions. Keep tasks for things you want to retry, cache or see. On Cloud the SDK already retries 429 with Retry-After, so a retry wrapper of your own adds nothing.

Connection exhaustion after scaling out

Symptom. After adding API replicas, errors about too many connections appear in the API logs, and the readiness probe flaps.

Cause. Each process holds a pool. The defaults, a pool size of 5 and an overflow of 10, allow 15 connections per process, and services processes count too. Replicas multiplied by 15 exceeded Postgres max_connections.

Fix. Do the sum before scaling, then either raise the database limit, lower the pool size through PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE, or place a connection pooler in front, after testing it with Prefect on a copy.

Duplicate runs after adding a second services process

Symptom. Two scheduled runs at the same time for the same deployment.

Cause. Several services processes with the default in-memory Docket backend.

Fix. Set PREFECT_SERVER_DOCKET_URL to a shared Redis for every services process, together with the Redis messaging, ordering and lease settings from the HA guide. Then cancel the duplicates.

The migration that races itself

Symptom. A rollout hangs, or some replicas crash at start, right after an upgrade. Readiness probes fail while the database is busy.

Cause. Every replica tried to run migrations on boot, and a large database makes migrations slow.

Fix. Set PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false on API replicas, run prefect server database upgrade -y as a separate job before the rollout, and raise PREFECT_SERVER_DATABASE_MIGRATION_TIMEOUT for that job if the database is big. Take a backup first.

A newer client than the server

Symptom. Deployments or runs fail with 422 responses after a worker image was rebuilt with the latest Prefect.

Cause. The client sent fields the older server did not understand.

Fix. Keep the client at or below the server. Upgrade the server first. Pin the version in every image, and add a build-time check that compares it to the server.

Runs that were deployed with run-time dependency installation

Symptom. After upgrading across 3.7.3, pulled-code deployments fail at startup with import errors.

Cause. Automatic dependency installation for pulled code became opt-in.

Fix. Prefer a baked image. Otherwise add an explicit pip_install_requirements pull step, or set PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES=true knowingly, accepting that code installs packages at run time.

Results that vanish

Symptom. MissingResult when reading a state, or a retry on another machine that cannot find the earlier result.

Cause. Result persistence is off by default, and local disk disappears with the container.

Fix. Turn persistence on where needed, point result_storage at storage every machine can reach, and set a lifecycle rule. Note that since 3.8.6 the default local path is $PREFECT_HOME/storage, which helps on one machine and does nothing for containers.

Surprise cache hits

Symptom. A task shows Cached and does not re-run, even though you expect fresh data.

Cause. The default cache policy combines inputs, task source and flow run ID, and a task whose inputs are unchanged inside one run is served from cache. A custom policy keyed only on inputs can cache across runs, which is what you may not want.

Fix. Choose the policy deliberately. cache_policy=NO_CACHE turns it off, refresh_cache=True forces recomputation, and cache_expiration bounds staleness. Teach teams that caching needs persistence and therefore storage.

Crashed hidden behind Failed-only alerts

Symptom. A cluster incident kills a hundred runs, and the alert channel is silent.

Cause. The alert fires on Failed. Infrastructure deaths are Crashed.

Fix. Alert on both, to different owners, and add an absence alert for the flows that matter.

Heartbeat and timeout confusion

Symptom. A zombie run sits in Running long after its pod disappeared, or a healthy long run is marked Crashed.

Cause. The flow run heartbeat defaults to 180 seconds (PREFECT_FLOWS_HEARTBEAT_FREQUENCY), and worker heartbeats are separate, 30 seconds by default. The two signals are easily mixed up, and they mean different things: one says the worker is alive, the other says the run is alive.

Fix. Learn which signal your alert depends on, test by killing a pod, and do not tune either number without a measurement from your own system.

Global concurrency slots held by a dead run

Symptom. Runs queue up waiting on a named limit although nothing is using it.

Cause. A lease was held by a run that died, and the slot is released when the lease expires, which takes time. The repossessor service reclaims expired leases.

Fix. Check that the services tier is running. Inspect the limit with prefect gcl inspect <name>. Use lease-aware concurrency(..., strict=True) where losing the lease must fail the run, and avoid very long holds.

Corporate proxies and TLS interception

Symptom. SSL: CERTIFICATE_VERIFY_FAILED from workers, which works on a laptop.

Cause. A gateway re-signs traffic with its own certificate authority, which the worker image does not trust.

Fix. Install the authority into the image or pass SSL_CERT_FILE, or PREFECT_API_SSL_CERT_FILE for the Prefect client, and set HTTPS_PROXY if needed. Do not use PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true in production.

The errors your users will bring you

A platform owner gets the same handful of tickets. Having the first response ready saves hours.

Put this table in your internal documentation with links to your own examples. Most tickets disappear once people can find it.

Platform-ownership habits

Treat the database like the product

The habit that matters most is to know, without looking, the size of your database, the growth rate of the events table, the last time you restored a backup and how long it took. Put these in a dashboard and review it monthly. A platform that depends on Postgres and never measures Postgres has not started operating.

Manage pools and templates as code

Work pools, variables, automations and their job templates belong in Terraform, reviewed like any infrastructure. The official provider covers them. Teams may write flows, and only the platform changes pools. A hand-edited template in the UI is configuration drift you cannot see in a pull request. The Terraform guide covers the workflow and the Argo CD guide covers reconciling the worker deployment.

Make the template the default and the exception visible

A paved-road repository gets most teams to a good state without a meeting. Measure how many deployments follow it, and treat every deviation as a conversation, not a prohibition. Some deviations will be right, and you should learn from those.

Put owners on everything

Every deployment needs a team tag and a named owner in its description. Every pool needs an owner. Every automation needs an owner. During an incident, ownership is the difference between a five-minute fix and an hour of asking around.

Alert on absence as a rule

For every scheduled flow the business cares about, create a proactive automation that fires when the expected completion does not arrive. Make it part of the template so it is not optional. Test it by pausing a schedule. Verify the automation fields against your version's reference before you standardise, since schemas shift between releases.

Rehearse, do not just document

Run a game day each quarter: stop a worker fleet, fail over the database in staging, expire a credential, upgrade a version. Time detection and recovery. The numbers will disagree with your assumptions, and the disagreement is the value.

Pin, then upgrade on purpose

Pin every image. Upgrade the server, then workers, then flow images, in that order, and never because a tool floated. Keep one minor version of headroom: do not sit three minors behind, since release notes compound and the jump is harder to reason about.

Guard shared systems centrally

Anything that is shared and finite, such as a warehouse, an API quota or a licence pool, gets a named global concurrency limit and a documented owner. Teams then take slots through the concurrency context manager, and you have one place to change the number. For rate-limited APIs, configure a slot decay and use rate_limit.

Decide your retry culture

Write down the standard: exponential backoff with jitter, a small retry count, no retry on non-idempotent work without a transaction or idempotency key. Provide it in the shared library so a retry decorator is a one-liner. Retries without a standard are the most common cause of self-inflicted load on upstream systems.

Keep the exit criteria

Know in advance what would make you recommend another tool, and write it down. Open-source Prefect has no per-user roles, so if a compliance requirement arrives that needs them, you will already have the options ready, either Cloud or another orchestrator. Comparing with Airflow and Dagster periodically keeps that knowledge fresh.

Review checklist for pull requests

Paste this into the pull request template for repositories that hold flows and prefect.yaml.

• Does every deployment have a team tag and an owner in the description?
• Are the work pool and its environment the approved ones for this team?
• Is the pull step pinned to a tag or commit, or is the code baked into a pinned image?
• Is prefect pinned exactly, and does it match the server version or sit below it?
• Are dependencies locked, and is run-time installation deliberate?
• Is there any secret in a variable, a parameter or the file itself? Are references used instead?
• Does any task run per item over a large collection? Could it be batched?
• Do retries use backoff and jitter, and are retried steps idempotent?
• Does the flow touch a shared system, and does it take a global concurrency limit?
• Is result persistence configured, with shared storage, where retries or subflows need it?
• Is caching intended? Is the cache policy explicit where the default would surprise?
• Is there an absence alert for this schedule, and alerts on Failed and Crashed?
• Is log_prints on, and is the log volume reasonable?
• Are parameters small, with data passed by reference?
• Has the change been tried on staging, including a failing run?

Review checklist for design documents

Use these questions on a proposal for a new platform feature, a new pool, or a migration.

• Failure. What happens when the database, Redis, the services tier, a worker, and the flow's infrastructure each fail? Is the answer written down?
• Blast radius. If this goes wrong, who is affected: one flow, one team, one pool, everyone?
• Identity. Which identity runs this, what can it read, and who can change what it runs?
• Capacity. What is the expected run and task count, and which of the first three bottlenecks does it approach?
• Cost. What is the monthly cost in compute, database and subscription, and how does it grow?
• Residency. What metadata, including parameters and logs, leaves the region, and is that permitted?
• Reversibility. How do we roll back, and what state must we keep to do it?
• Operations. Who is on call, what alerts exist, and which runbook applies?
• Upgrade. Which release-note items touch this design, and how do we test them?
• Alternatives. Why not another tool, or no tool? What would change our mind?

Operational recipes

See what a pool is doing

prefect work-pool inspect k8s-prod
prefect work-pool preview k8s-prod --hours 4
prefect flow-run ls --state-type FAILED
prefect flow-run ls --state Late

The preview shows what the pool expects to run soon, which is how you spot a pile-up before it becomes Late.

Stop the bleeding

prefect work-pool pause k8s-prod
prefect deployment schedule pause --help
prefect automation pause "nightly-etl-did-not-run"

Pause the pool to stop new runs from starting, pause a deployment schedule to stop a specific source, and pause an automation that is itself misfiring. Check the help output on your version for exact arguments, then resume each with its matching command, and record what you paused.

Give a team a guarded shared limit

prefect global-concurrency-limit create warehouse --limit 10
prefect gcl inspect warehouse

from prefect import flow, task
from prefect.concurrency.sync import concurrency

@task
def load_batch(rows):
    with concurrency("warehouse", occupy=1, timeout_seconds=300):
        ...  # write rows to the warehouse

@flow
def nightly_load():
    load_batch([1, 2, 3])

The limit exists in one place, the platform team owns the number, and teams cannot exceed it by accident.

Run the migration as its own step

export PREFECT_SERVER_DATABASE_MIGRATION_TIMEOUT=600
prefect server database upgrade -y

Run this as a job before rolling API replicas, with replicas configured with PREFECT_SERVER_DATABASE_MIGRATE_ON_START=false. Back up first.

Check readiness and health from outside

curl -fsS http://prefect.internal:4200/api/health
curl -fsS http://prefect.internal:4200/api/ready

Health says the process is up. Ready also checks the database and returns 503 if Postgres is unreachable. Use them for different probes.

Move resources between servers

prefect transfer --help

Read the help on your version, rehearse on a copy, and plan to recreate credential blocks deliberately, since secrets should not travel casually.

Habits to keep, in order of leverage

1. Protect and measure the database. Everything else depends on it.
2. Separate services from the API, and share the coordination backend. It prevents the subtlest outage.
3. One pool per owner, with a hard-coded identity. It is your security and cost boundary.
4. Absence alerts on every scheduled flow that matters. It catches what nothing else can.
5. Pin everything and upgrade on a calendar. It removes surprise.
6. Put platform configuration in code. It makes change reviewable.
7. Rehearse failure. It turns assumptions into numbers.
8. Know when to recommend another tool. It keeps your advice honest.

For the neighbouring skills, see Kubernetes for the workers' home, Docker for the images they run, and MLflow for the tracking that sits beside orchestration.