The Complete Metaflow Guide

This is part one of three. It covers everything you need to write, run, debug and inspect real Metaflow projects on your own laptop, written for someone who has never used the tool. By the end you can turn a messy notebook into a structured pipeline, run steps in parallel, pass parameters, survive failures without losing work, look back at any past run from Python, and build a small report that opens in your browser. Mid-level and Senior take the same topics further, into remote compute, scheduling and running Metaflow as a platform. Nothing here is thrown away.

This guide is written against Metaflow 2.19, the current release line. Metaflow keeps its promises about backwards compatibility, so nearly everything you learn will still work next year, but where a feature is recent the text says which version introduced it.

Each section ends with a Try it task. Do them as you go. Pipelines are something you understand by watching your own one break and recover, not by reading about it.

What Metaflow is, and the problem it solves

Machine learning work usually starts in a notebook. You load data, clean it, train a model, plot a score. It works, and then someone asks for it to run every night, or to train on a bigger machine, or to be reproduced by a colleague who is not you. At that point the notebook becomes the problem: its cells were run in an order nobody wrote down, its state lives in a kernel that dies when you close the laptop, and "the model I showed on Tuesday" is not stored anywhere.

Metaflow is a Python library that turns that work into a flow: a list of named steps that run in a defined order, with everything they produce saved automatically. It was created at Netflix to let data scientists build and deploy their own pipelines without becoming infrastructure engineers, and it is open source. You write ordinary Python. You do not write YAML, you do not learn a new language, and you do not need a server to start.

Three promises explain why people adopt it.

Your work is structured. A flow is a class, and each stage is a method. The order is explicit, so a colleague can read the file and understand the pipeline in a minute.

Your work is saved. Every value you assign to self inside a step is stored automatically and versioned per run. You never write code to save intermediate results, and you can open any old result later.

Your work can grow. The same file that runs on your laptop can later run its steps on a cluster, be scheduled by a production orchestrator, and react to events, without rewriting it. This guide only covers the laptop part, deliberately: the laptop is where you learn the model, and everything after that is the same model with more machinery behind it.

What came before? Teams either wrote one long script (fast to start, impossible to resume), or adopted a general orchestrator such as Airflow and wrote tasks as operators in that system's own vocabulary. Metaflow sits in between: it is designed around the data scientist's workflow (write code, run it, look at the results, iterate) and treats orchestration as something you switch on afterwards. If you have read the guides on MLflow or Prefect, it helps to know the division of labour. MLflow records experiments and models, Prefect and Airflow schedule work, and Metaflow is the authoring and tracking layer in which a pipeline is written, with its own record of every run.

The mental model: flows, steps, artifacts, runs

Metaflow has a small vocabulary. Learn these nouns well and every later topic is a variation.

A flow is a class that inherits from FlowSpec. It is the smallest thing you can run or schedule. You run it by executing the file as a script: python myflow.py run.

A step is a method marked with the @step decorator. Steps are the nodes of a graph, and the documentation calls a step the smallest resumable unit of computation. That phrase matters: if a step fails, you can fix the code and continue from that step without redoing the ones before it. A sensible step does one meaningful thing and, as a rule of thumb from the documentation, should take well under an hour.

A transition connects steps. The last line of every step is a call to self.next(...), naming the step that comes next. A flow must begin with a step named start and finish with a step named end. (Since version 2.19.23 you can instead mark the entry and exit steps with @step(start=True) and @step(end=True), but the plain names are what you will meet in almost every example, so this guide uses them.)

A run is one execution of a flow from start to end. Each run gets an ID, which for local runs is a simple counting number: 1, 2, 3. A task is one step executing inside a run. Usually a step produces exactly one task, but a step that fans out over a list produces one task per item, which is how Metaflow does parallel work. A task is addressed by a pathspec of the form FlowName/run_id/step_name/task_id, for example HelloFlow/3/start/1. You will see pathspecs in logs and use them to look things up, so get comfortable reading them.

An artifact is any value you store on self inside a step. If you write self.accuracy = 0.93, then accuracy is an artifact: Metaflow saves it when the task finishes, attaches it to that exact task in that exact run, and makes it available to every later step. Ordinary local variables, the ones without self., are not saved and vanish when the step ends. This is the single most important rule in the guide, and Section "Everyday commands" returns to it, because breaking it is the most common beginner mistake.

Two more nouns you will meet early. A parameter is a value you pass when starting a run, such as a learning rate, set once and then read-only. The datastore is where artifacts are kept. On your laptop that is a hidden folder called .metaflow in your working directory, and in a team setting it is a cloud bucket. Because the location is configuration rather than code, the flow file does not change when the storage does.

Installing Metaflow and checking the setup

Metaflow is a Python package and runs on macOS and Linux. Use a virtual environment so the install does not touch your system Python:

python -m venv .venv
source .venv/bin/activate
pip install metaflow

If you prefer uv, the documented route is:

uv init myflow
cd myflow
uv add pandas metaflow

The package depends on only two libraries, requests and boto3, so the install is quick. Check that it worked:

metaflow --version
python -c "import metaflow; print(metaflow.__version__)"

Either command should print a version number in the 2.19 line or later. To see which metadata provider your setup uses (more on that below), run metaflow status. On a fresh install it reports a local setup, which is exactly what you want for learning.

Windows is not supported natively. Metaflow's local runtime relies on a POSIX-only part of Python, so on a Windows machine you will see ImportError: No module named fcntl. Install WSL 2 with Ubuntu and do everything inside it, exactly as if you were on Linux. If you cannot install anything, the browser-based Metaflow Sandbox offered by Outerbounds, the company founded by Metaflow's creators, runs flows with no local setup. Either path teaches the same thing.

Metaflow can also pull in a set of tutorial flows that are useful for checking your install:

metaflow tutorials pull
metaflow tutorials list

The first command copies the tutorials into a folder called metaflow-tutorials in your current directory. Open 00-helloworld in it and you will recognise everything you are about to write yourself.

You do not need Docker, Kubernetes, a cloud account, or a database for anything in this guide. Everything runs locally and stores results under .metaflow. If your employer or university later gives you a configured cloud setup, a configuration file on your machine, not your code, will point Metaflow at it. That is covered near the end of the guide.

Your first flow, step by step

Create a file called hello_flow.py. Type it rather than pasting it; the shape matters.

from metaflow import FlowSpec, step


class HelloFlow(FlowSpec):
    """A flow that greets the world and keeps a message."""

    @step
    def start(self):
        print("HelloFlow is starting")
        self.next(self.hello)

    @step
    def hello(self):
        self.message = "Hello, Metaflow"
        print("Saying hello")
        self.next(self.end)

    @step
    def end(self):
        print("The message was:", self.message)


if __name__ == "__main__":
    HelloFlow()

Read it top to bottom. HelloFlow inherits from FlowSpec, which gives it all of Metaflow's behaviour. Each method has @step above it. Each step except the last ends with self.next(self.something), passing the next method itself, not a string. The end step has no self.next because nothing follows it. The last two lines create the flow when the file is executed as a script; without them, running the file would do nothing.

Notice self.message = "Hello, Metaflow" in the hello step and the read in end. The value crossed a step boundary because it is an artifact.

Now use the commands Metaflow gives you, in this order. First check the file for mistakes:

python hello_flow.py check

Then draw the graph:

python hello_flow.py show

show prints each step with its docstring and what it leads to, which is a fast way to confirm the flow is wired the way you think. Finally run it:

python hello_flow.py run

You will see output shaped like this, though the IDs and timestamps will differ:

Workflow starting (run-id 1):
[1/start/1 (pid 40211)] Task is starting.
[1/start/1 (pid 40211)] HelloFlow is starting
[1/start/1 (pid 40211)] Task finished successfully.
[1/hello/2 (pid 40217)] Task is starting.
[1/hello/2 (pid 40217)] Saying hello
[1/hello/2 (pid 40217)] Task finished successfully.
[1/end/3 (pid 40222)] Task is starting.
[1/end/3 (pid 40222)] The message was: Hello, Metaflow
[1/end/3 (pid 40222)] Task finished successfully.
Done!

Read the bracketed prefix. 1/start/1 is run 1, step start, task 1. The number in parentheses is the operating system process that executed it, because Metaflow runs each task as its own process. Every print you wrote appears after its task's prefix, so when a flow is noisy you can still tell which task said what.

Two things happened that you did not code. A folder named .metaflow appeared next to your file, holding run 1's artifacts. And every task ran in a separate process, which is why steps cannot share local variables, and why a crash in one step cannot corrupt another.

Run it a second time. You get run 2, stored alongside run 1. Nothing was overwritten. This is the behaviour that makes Metaflow useful for experiments: every attempt is preserved and comparable.

Artifacts: how data moves between steps

A flow is only useful if steps can hand data to each other, and the rule is simple: anything assigned to self is an artifact and is saved. Here is a flow that loads some numbers, computes statistics, and uses them later:

from metaflow import FlowSpec, step


class StatsFlow(FlowSpec):

    @step
    def start(self):
        self.numbers = [4, 8, 15, 16, 23, 42]
        self.next(self.compute)

    @step
    def compute(self):
        total = sum(self.numbers)
        self.mean = total / len(self.numbers)
        self.biggest = max(self.numbers)
        self.next(self.end)

    @step
    def end(self):
        print(f"mean={self.mean:.2f} biggest={self.biggest}")


if __name__ == "__main__":
    StatsFlow()

In compute, self.numbers was read, because it was saved by start. The variable total was a plain local, so it is gone after the step, while self.mean and self.biggest persist. By the end of the run, four artifacts exist: numbers, mean, biggest, and anything else you assigned.

There are three practical consequences worth knowing on day one.

Artifacts are saved when the task finishes, using Python's pickle. Most objects (numbers, strings, lists, dictionaries, pandas DataFrames, NumPy arrays, scikit-learn models) pickle fine. Open file handles, database connections and locks do not. If you store one, the step fails at the end with a pickling error, not where you created the object. The fix is to keep those objects as local variables inside a step and save only the results you need.

Artifacts are copied, not shared. A step that modifies self.numbers modifies its own task's version. Later steps see the new version; earlier steps are unchanged. That is a feature: old runs stay exactly as they were.

Size matters. Saving a five-gigabyte DataFrame as an artifact in five consecutive steps writes it several times. Metaflow deduplicates identical data within a flow, which helps, but you should still avoid storing large intermediate objects you will never read again. Keep them local if only one step needs them.

An everyday bug is worth naming now. If you write df = load_data() in one step and use df in the next, you will get AttributeError: 'StatsFlow' object has no attribute 'df', because the variable was never stored on self. The fix is always the same: assign it to self if a later step needs it.

Branches and joins: running work in parallel

Some work is naturally parallel, such as training two different models on the same data. Metaflow lets a step name more than one next step. Those steps, called branches, run at the same time, and you must bring them back together in a join step.

from metaflow import FlowSpec, step


class BranchFlow(FlowSpec):

    @step
    def start(self):
        self.data = list(range(1, 11))
        self.next(self.sum_it, self.max_it)

    @step
    def sum_it(self):
        self.result = sum(self.data)
        self.next(self.join)

    @step
    def max_it(self):
        self.result = max(self.data)
        self.next(self.join)

    @step
    def join(self, inputs):
        self.sum_result = inputs.sum_it.result
        self.max_result = inputs.max_it.result
        self.merge_artifacts(inputs, exclude=["result"])
        self.next(self.end)

    @step
    def end(self):
        print(self.sum_result, self.max_result, len(self.data))


if __name__ == "__main__":
    BranchFlow()

self.next(self.sum_it, self.max_it) with two arguments starts both branches. The join step is recognised by its second parameter, conventionally called inputs. It receives one entry per branch, and you can reach a branch by the name of its step: inputs.sum_it.result.

Both branches set an artifact called result, and they disagree. Metaflow cannot decide which is right, so it refuses to guess. Inside join you copy out the ones you want under distinct names (self.sum_result, self.max_result). Then self.merge_artifacts(inputs, exclude=["result"]) carries over the artifacts that are not in conflict, here data, and skips the one you excluded. If you forget to exclude or set the conflicting artifact, the run stops with "cannot merge the following artifacts due to them having conflicting values" and tells you to set them explicitly before calling merge_artifacts. Use either exclude or include on that call, never both.

The rules Metaflow enforces, and that check will report, are these. Every branch must be joined before end. A join step cannot itself start a new split: if it must, split it in two steps. And a step with an extra argument that is not preceded by a split is reported as a join with nothing to join.

Foreach: one task for every item in a list

Branches are fixed in the code. Often the number of parallel jobs depends on your data: one per file, one per customer, one per hyperparameter setting. That is what foreach is for.

from metaflow import FlowSpec, step


class ForeachFlow(FlowSpec):

    @step
    def start(self):
        self.learning_rates = [0.001, 0.01, 0.1]
        self.next(self.train, foreach="learning_rates")

    @step
    def train(self):
        lr = self.input
        self.lr = lr
        self.score = 1.0 - abs(0.01 - lr)
        self.next(self.join)

    @step
    def join(self, inputs):
        best = max(inputs, key=lambda i: i.score)
        self.best_lr = best.lr
        self.best_score = best.score
        self.next(self.end)

    @step
    def end(self):
        print("best learning rate:", self.best_lr)


if __name__ == "__main__":
    ForeachFlow()

The phrase foreach="learning_rates" names an artifact that must hold a list. Metaflow creates one train task per item, running them in parallel, up to a local limit. Inside each task, self.input is that task's item. Note the name of the argument: it is a string naming the variable, not the variable itself.

The join step receives all the tasks. You can loop over them with for i in inputs, or use max(inputs, ...) as above, and each one exposes the artifacts that task saved. Here each train task saved lr and score, so the join reads i.score. Saving self.lr = lr was deliberate: it makes the input part of the record so the join can tell which result belongs to which setting.

The limits are worth memorising. Locally at most 16 tasks run concurrently by default, controllable with --max-workers. A foreach that produces more than 100 tasks is rejected with a message telling you to raise --max-num-splits; that safety cap exists to stop an accidental list of a million items from starting a million tasks. An empty list also fails with a "zero splits" message, which usually means an earlier step forgot to fill it.

Since version 2.18, flows can also choose between branches based on a value (a conditional transition) or loop back to an earlier step (recursion). They are useful and they work locally, but they are not needed to get started, so they are left to the Mid-level guide.

Parameters: changing a run without editing code

A learning rate hard-coded inside a step means editing the file for every experiment, and the edit history becomes your only record of what you tried. A parameter moves that value to the command line. You declare it as a class attribute:

from metaflow import FlowSpec, Parameter, step


class ParamFlow(FlowSpec):

    alpha = Parameter("alpha", default=0.01, help="Learning rate")
    epochs = Parameter("epochs", default=5, help="Number of passes")
    label = Parameter("label", default="baseline", help="Name for this run")

    @step
    def start(self):
        print(f"alpha={self.alpha} epochs={self.epochs} label={self.label}")
        self.loss = self.alpha * 100 / self.epochs
        self.next(self.end)

    @step
    def end(self):
        print("loss:", self.loss)


if __name__ == "__main__":
    ParamFlow()

Pass values on the command line, after run:

python param_flow.py run --alpha 0.5 --epochs 10 --label bigger
python param_flow.py run --help

The second command lists every parameter with its help text and default, so a colleague can discover what your flow accepts without reading the code. Metaflow infers each parameter's type from its default: 0.01 makes a float, 5 makes an integer, and a string default makes a string. You can also set type=int explicitly, mark required=True so there is no default, or pass structured data with type=JSONType, where the command line supplies a JSON string.

Three facts keep you out of trouble. A parameter is read-only for the whole run, so you cannot reassign self.alpha in a step; copy it to another artifact if you need a variant. Parameters are stored as artifacts of the run, which means you can always look up later exactly what a run was given. And for that same reason, never pass a password or an API key as a parameter, because it would be saved in the run's record.

A fourth fact catches people the first time: the order of words matters. Options that belong to run, including your parameters, go after run. A small group of top-level options that change how Metaflow itself behaves, such as --environment, --with or --datastore, go before run:

python param_flow.py --with retry run --alpha 0.5

Write that shape as python flow.py [top-level options] command [command options] and you will rarely get it wrong. Metaflow also has a separate Config object, evaluated when a flow is deployed, used to set up decorators and production deployments. As a beginner you only need parameters; the Mid-level guide explains when to reach for Config instead.

You can also attach labels to a run with --tag, which you can repeat:

python param_flow.py run --tag experiment-7 --tag gpu-later

Tags help you find runs afterwards from the Client API, which a later section shows.

When things fail: resume, retry, catch and timeout

Real pipelines fail: a download times out, a file is missing, a cell of code has a typo. The question is how much work you lose. Metaflow's answer is that the work of every step that finished is saved, so a failure costs you only the failing step.

Here is a flow with a deliberate bug in its second step:

from metaflow import FlowSpec, step


class FragileFlow(FlowSpec):

    @step
    def start(self):
        print("Doing the slow, expensive part")
        self.data = list(range(1000))
        self.next(self.fragile)

    @step
    def fragile(self):
        self.ratio = len(self.data) / 0
        self.next(self.end)

    @step
    def end(self):
        print("ratio:", self.ratio)


if __name__ == "__main__":
    FragileFlow()

Running it, the fragile step crashes with ZeroDivisionError, and Metaflow stops the run and prints the failure with the task's prefix. Now fix the bug (change / 0 to / 4) and run:

python fragile_flow.py resume

resume finds the most recent run, clones the results of the steps that succeeded (it does not re-run start), and re-executes from the step that failed. The slow, expensive part is skipped. By default it resumes the last run; to choose another, use --origin-run-id 3, and to restart from a step of your choosing, name it: python fragile_flow.py resume start.

Resume uses the original values of the parameters, not new ones. If you want to change a parameter, that is a new run, not a resume. The new code is used, though, which is exactly why resume is the tool for the edit-and-retry loop.

Some failures are not bugs but bad luck: a network blip, a rate limit. For those Metaflow has three decorators you place above @step:

from metaflow import FlowSpec, step, retry, catch, timeout


class RobustFlow(FlowSpec):

    @retry(times=3, minutes_between_retries=1)
    @timeout(seconds=30)
    @step
    def start(self):
        import random
        if random.random() < 0.5:
            raise RuntimeError("flaky network")
        self.value = 42
        self.next(self.optional)

    @catch(var="failure", print_exception=True)
    @step
    def optional(self):
        raise ValueError("this part is nice to have")
        self.next(self.end)

    @step
    def end(self):
        print("value:", self.value)
        print("optional step failed:", self.failure is not None)


if __name__ == "__main__":
    RobustFlow()

@retry re-runs a failing task. Its defaults are three retries with two minutes between them, and the maximum you may ask for is times=4; asking for more gives "The maximum number of retries is @retry(times=4)". Each try is an attempt of the same task, and only the last attempt's artifacts and logs are kept. @timeout kills a step that runs too long; its seconds, minutes and hours arguments add up. @catch is different: it swallows the exception after all retries, stores it in the artifact whose name you give (failure), and lets the flow continue. Use it only when the flow can genuinely carry on without that step's output, and always check the artifact afterwards.

You can also add a decorator to every step at once from the command line, without touching the file, using --with, placed before run:

python robust_flow.py --with retry run

That is handy to make a whole flow tolerant of transient errors for one run.

Logs, dump and cards: looking inside a run

While a flow runs, its print output streams to your terminal. Afterwards, Metaflow keeps it, so you can read the output of a past task:

python hello_flow.py logs 1/hello/2

The argument is the task's run/step/task identifier from the log prefix. Add --stderr for error output or --timestamps to see when each line was written. To see what a task saved, use dump:

python stats_flow.py dump 1/compute/2

This prints the artifacts of that task and their values. Use --include mean,biggest to show only some, and --file out.pkl to write them to a file.

For anything more readable than console text, Metaflow has cards: small HTML reports attached to a task. You add @card above a step, and Metaflow generates a default card showing the task's artifacts, parameters and metadata. You can add your own content through current.card:

from metaflow import FlowSpec, step, card, current
from metaflow.cards import Markdown, Table


class CardFlow(FlowSpec):

    @step
    def start(self):
        self.scores = {"baseline": 0.81, "tuned": 0.87}
        self.next(self.report)

    @card(type="blank")
    @step
    def report(self):
        current.card.append(Markdown("# Model comparison"))
        rows = [[name, score] for name, score in self.scores.items()]
        current.card.append(Table(rows, headers=["model", "accuracy"]))
        self.next(self.end)

    @step
    def end(self):
        print("report built")


if __name__ == "__main__":
    CardFlow()

After a run, view the card in your browser:

python card_flow.py card view report

card view opens the newest card for that step. The blank type starts empty so you control everything, while the default type pre-fills artifacts for you. The components you will use most are Markdown for text, Table for rows, Image for pictures (including a matplotlib figure through Image.from_matplotlib) and Artifact for showing a stored value. Cards are the fastest way to share a result with a teammate who will not read your code. You can list a run's cards with card list, and card server starts a local viewer, by default on port 8324, that refreshes as the run progresses.

The Client API: reading past runs from Python

Everything Metaflow saved can be read back from ordinary Python, in a notebook or a script, without re-running anything. This is called the Client API, and it is where Metaflow's two halves meet: you author with FlowSpec, and you analyse with Flow, Run, Step and Task.

from metaflow import Flow, Run, Step, Task

flow = Flow("StatsFlow")
run = flow.latest_successful_run
print(run.id, run.successful, run.finished_at)
print(run.data.mean)

step = Step("StatsFlow/1/compute")
task = step.task
print(task.data.biggest)
print(task.stdout)

Flow("StatsFlow") is the collection of all runs of that flow. latest_run is the newest run, finished or not; latest_successful_run is the newest that completed. run.data exposes the artifacts of the end step of that run, so run.data.mean is the final value of mean. For artifacts from earlier steps, go through a Step or a Task and read .data there. Tasks also expose stdout, stderr, and exception if they failed.

You can iterate over history and filter it. A classic use is to find the best run of an experiment:

from metaflow import Flow

best = None
for run in Flow("ForeachFlow").runs():
    if run.successful and (best is None or run.data.best_score > best.data.best_score):
        best = run
print(best.id, best.data.best_lr, best.data.best_score)

Runs you tagged at the command line are easy to select with Flow("ParamFlow").runs("experiment-7").

One concept surprises nearly every beginner: namespaces. By default the Client API only shows runs that you created, filtered by a tag of the form user:<your username>. If you try to read someone else's run, or a run that was made by a scheduler, you get Object not in the current namespace. The fix is to widen the view:

from metaflow import namespace, Flow
namespace(None)
print(Flow("TeamFlow").latest_run)

namespace(None) selects the global namespace, showing every run. The documentation is clear that namespaces are a way to avoid confusion, not an access-control mechanism. They do not hide anything from anyone who can reach the datastore.

The other common error is Object not found with a message such as Flow('X') does not exist. On a local setup it almost always means you are in a different folder from the one where you ran the flow, because the local record lives in .metaflow in the working directory. Run metaflow status to see which metadata provider and working tree you are using, and change to the right folder.

What happens during a local run

It helps to know what Metaflow does when you type python flow.py run, because it explains behaviour that otherwise looks like magic. The file is executed as a normal Python script. Metaflow reads your class, builds the graph from the self.next calls, and runs the linter. If the linter is happy, the local runtime starts the start step as a separate operating system process, waits for it to finish, reads the transitions, and launches the next step the same way. Anything that can run at the same time, such as the branches of a split or the tasks of a foreach, runs in parallel, up to the worker limit.

That design has four consequences that are worth saying plainly.

Each task starts with a clean interpreter. Module-level code at the top of your file runs once per task, not once per run. If you load a large file at the top of the script, every task loads it again, and if you print something there it will print many times. Put expensive work inside steps, where it runs only when needed.

Tasks communicate only through saved artifacts. When a task finishes, its artifacts are written to the datastore. When the next task starts, it reads the ones it needs from the datastore. Nothing is passed in memory. This is slower than a function call, and it is exactly what makes a step resumable and later movable to another computer: the next step needs nothing except what was saved.

The local runtime is for development. The documentation is clear that the local scheduler is not a production-grade scheduler. It does not restart itself after a reboot and it does not run at night on its own. That job belongs to a production orchestrator, which is the subject of the Senior guide. The important point for a beginner is that the flow file does not change when you move to one.

A crash in a step leaves the datastore consistent. A task either finishes and saves its artifacts or it does not. That is why resume can trust the steps that completed.

You can see the concurrency limit in action. By default up to 16 tasks run at once, and you can lower it when you want deterministic, sequential output or when you are hunting a bug:

python foreach_flow.py run --max-workers 1

With one worker, foreach tasks are processed one after another, so their log lines do not interleave. Another option that stops you from flooding a laptop with output is --max-log-size, which limits how many megabytes of a task's log are stored. The default is ten.

Good habits from the first day

A few habits are cheap to adopt at the beginning and painful to add later.

Keep each flow in its own small folder, with its own requirements file. Metaflow stores records in a .metaflow folder under the directory you run from. If you run flows from different folders, the Client API will only find the ones in the folder you are in, and Flow does not exist will confuse you. One folder per project removes the problem.

Name steps after what they do, not their order. Names like load_data, clean, train and evaluate make the show output and the log prefixes readable six months later. Step names can only contain lowercase letters, digits and underscores.

Make steps small enough to be worth resuming, and big enough to be worth saving. A step that loads a file and a step that trains a model are separate because you will want to re-run the second without the first. A step that adds one to a variable is not worth a step, because every step pays a small cost to save its artifacts.

Store results, not machinery. Save accuracy numbers, trained models, prediction tables. Do not save database connections, open files or thread pools. If you do, the save fails with a pickling error at the end of the step, and the message will not point at the line where you created the object.

Set random seeds as parameters. A run whose seed is a parameter can be reproduced from its record alone. A run whose seed is hidden in the code cannot, unless you also remember the version of the code.

Tag runs you care about. Tags cost nothing and turn "which run was that?" into a one-line query later.

Run check before run. It is instant and it catches the structural mistakes that otherwise cost you a run.

Commit your flow files to Git. Metaflow remembers the results of each run, but the record of which code produced them is your repository. A run that used code nobody committed cannot be reproduced.

Dependencies: a short introduction

Your flow will soon need libraries beyond Metaflow, such as pandas or scikit-learn. On your laptop that is a simple matter: install them into your virtual environment and import them. Every step runs in a process started from that same environment, so they are all available.

Metaflow also has decorators, @pypi and @conda, that build a per-step environment with exact versions, which is essential when steps run on other machines. They only work if you launch with --environment=pypi (or conda), placed before run, and their packages must be imported inside the step. If you forget the flag, you get an "Incompatible environment" error that names the decorator and the flags it needs. The Mid-level guide covers this in depth. For now, know that the error means "tell Metaflow how to build environments", not "your code is wrong".

If you use uv, the documented pattern is uv run flow.py run for a flow inside a uv project. Keep a requirements.txt or lock file next to your flow; it is the easiest kind of reproducibility you can have, and it costs nothing.

Configuration and common errors

By default Metaflow stores everything locally. The three settings that matter are which datastore holds artifacts (local by default), which metadata provider records runs (local by default), and which environment builds dependencies (local by default). They live in a configuration file, config.json inside the folder ~/.metaflowconfig, and any setting can be overridden by an environment variable with the prefix METAFLOW_, such as METAFLOW_DEFAULT_DATASTORE. To use another profile, set METAFLOW_PROFILE=name, and Metaflow reads config_name.json instead.

You will rarely edit these by hand. If your organization has set up shared infrastructure, an administrator gives you a configuration, and the interactive metaflow configure commands (for example metaflow configure aws) write it for you; metaflow configure show prints the current values. As a learner you can ignore all of it, but knowing it exists explains two otherwise puzzling situations: flows that work on a colleague's machine because it points at the shared store, and flows that say Flow does not exist because your machine points somewhere else.

A note on where code lives. Metaflow packages the files of your working directory that end in .py, .R or .RDS when it needs to ship code to another machine. On a laptop that does not matter. It matters later, and it is one reason to keep each flow in a tidy folder of its own.

The errors you will actually meet, and how to read them, are below. Metaflow prints a short headline and then a message. Linter errors arrive under the headline "Validity checker found an issue", before anything runs.

• Step *X* is missing a self.next() transition to the next step. A step does not end with self.next(...). Add it as the last line.
• Step *X* specifies a self.next() transition to an unknown step, *Y*. A typo in the step name. Compare it with the method definitions.
• Step *X* is unreachable from the entry step *start*. A step no one leads to. Wire it in or delete it.
• There is a loop in your flow A cycle in the transitions. Loops of this kind are not allowed; restructure, or use the recursion support covered in the Mid-level guide.
• The terminal step *end* was reached before a split started at step(s) ... were joined. You split with a branch or foreach and never joined. Add a join step, which takes an inputs argument.
• Step *X* is both a join step ... and a split step ... One step cannot join and also fan out. Make them two steps.
• Your flow must have exactly one start step. Name one step start, or mark exactly one as the entry step.
• Step *X* has an invalid name. Step names may contain only lowercase letters, digits and underscores.
• Foreach variable *self.x* in step *s* does not exist. The artifact named in foreach= was not set. Assign the list to self first.
• Step *join* cannot merge the following artifacts due to them having conflicting values Set those artifacts explicitly in the join, or use exclude.
• AttributeError: 'MyFlow' object has no attribute 'x' The variable was local in an earlier step. Save it to self.
• @catch is defined for the step *s* but @catch is not supported in foreach split steps. Move @catch to the inner step, not the one that fans out.
• ImportError: No module named fcntl You are on native Windows. Use WSL 2.

check runs the same linter as run, so make it a reflex: if you change the shape of a flow, run python flow.py check first and save a failed run for real bugs.

Putting it all together

Here is one small project that uses almost everything above: train a few models in parallel with different settings, choose the best, report it on a card, and read the results afterward. It uses scikit-learn, so first pip install scikit-learn.

from metaflow import FlowSpec, Parameter, step, retry, card, current
from metaflow.cards import Markdown, Table


class TuneFlow(FlowSpec):

    seed = Parameter("seed", default=42, help="Random seed")
    depths = Parameter("depths", default="2,4,8", help="Tree depths, comma separated")

    @step
    def start(self):
        from sklearn.datasets import load_iris
        from sklearn.model_selection import train_test_split

        X, y = load_iris(return_X_y=True)
        self.split = train_test_split(X, y, test_size=0.3, random_state=self.seed)
        self.depth_list = [int(d) for d in self.depths.split(",")]
        self.next(self.train, foreach="depth_list")

    @retry(times=2, minutes_between_retries=0)
    @step
    def train(self):
        from sklearn.tree import DecisionTreeClassifier

        X_train, X_test, y_train, y_test = self.split
        self.depth = self.input
        model = DecisionTreeClassifier(max_depth=self.depth, random_state=self.seed)
        model.fit(X_train, y_train)
        self.accuracy = float(model.score(X_test, y_test))
        self.next(self.choose)

    @step
    def choose(self, inputs):
        results = [(i.depth, i.accuracy) for i in inputs]
        self.results = sorted(results)
        self.best_depth, self.best_accuracy = max(results, key=lambda r: r[1])
        self.next(self.report)

    @card(type="blank")
    @step
    def report(self):
        current.card.append(Markdown(f"# Best depth: {self.best_depth}"))
        current.card.append(Table([[d, a] for d, a in self.results],
                                  headers=["depth", "accuracy"]))
        self.next(self.end)

    @step
    def end(self):
        print(f"best depth {self.best_depth} with accuracy {self.best_accuracy:.3f}")


if __name__ == "__main__":
    TuneFlow()

Notice the decisions in it. The data split is stored on self because several later tasks need it. Imports happen inside steps, a habit that pays off when environments get involved. The parameter depths is a string split into a list, because a comma-separated string is the simplest way to accept a list on the command line. Each train task saves depth and accuracy so the join can pair them. The join is named choose rather than join: any name works, since a join is defined by its extra argument, not by its name.

Run and inspect it:

python tune_flow.py check
python tune_flow.py run --depths 1,3,5,9 --tag first-try
python tune_flow.py card view report

Then, from Python:

from metaflow import Flow

run = Flow("TuneFlow").latest_successful_run
print(run.data.best_depth, run.data.best_accuracy)
print(run.data.results)

If a train task fails, fix the code and python tune_flow.py resume; the data loading will not repeat. If you want to try other depths, start a new run, because resume keeps the original parameters.

What you can now do, and what comes next

You can now take a script and structure it as a flow with start, end and steps between them, and you know that only values on self cross a step boundary. You can branch and join, fan out over a list with foreach and collect the results, and merge artifacts sensibly. You can pass parameters and tags at run time, recover from failures with resume, and make flaky steps tolerant with retry, catch and timeout. You can look back at logs, dump artifacts, build cards, and read any past run from Python with the Client API. You can read Metaflow's linter errors and fix the structure, not just the symptom.

Mid-level takes every one of those topics further: conditional transitions and recursion, running individual steps quickly with spin, per-step dependencies with @pypi and @conda, sending steps to remote compute on Kubernetes or AWS Batch with @resources, Config objects, secrets, the Runner and Deployer APIs, and debugging flows in an IDE.

Senior then covers running Metaflow as a platform for a team: the datastore and metadata service, deploying flows to a production orchestrator such as Argo Workflows, scheduling and event triggers, @project branches, security and access, cost, upgrades, and when another tool is the better choice.

If you are deciding what to learn alongside it, MLflow is the usual companion for tracking experiments and models, DVC covers versioning data files, and Docker and Kubernetes are what you will meet when your steps leave the laptop.

Sources

• Metaflow documentation
• Installing Metaflow
• Metaflow tutorials
• What is Metaflow
• Metaflow basics
• Authoring flows
• Configuring flows
• Debugging flows
• Visualizing results with cards
• The Client API
• Dealing with failures
• Managing dependencies
• Metaflow release notes
• Metaflow on GitHub
• Metaflow on PyPI

This page prepares you to talk about Metaflow in an interview for a junior or entry-level machine learning or data role. It assumes you have worked through the beginner guide and run a few flows. Interviewers at this level do not expect production war stories. They want to see that you understand the model, can write a small flow from memory, and can reason about why something failed.

The thirty-second answer

Metaflow is an open-source Python library, created at Netflix, for building and running data science and machine learning pipelines. You write a flow as a Python class whose steps are methods, connect them with self.next(...), and Metaflow runs them in order, saves every value assigned to self as a versioned artifact, and lets you resume after a failure and read any past run from Python. The same flow file runs on a laptop and can later run on a cluster and be scheduled by a production orchestrator, without being rewritten.

If you can say those two sentences calmly, you have passed the first question in most interviews. Practise saying them aloud, because the most common weakness is not knowledge but rambling.

The model in one page

Keep these nouns ready, in this order, and be able to define each in a sentence.

• Flow: a class that subclasses FlowSpec. You run it by executing the file: python flow.py run.
• Step: a method decorated with @step. A step is a node of the graph and the smallest resumable unit of work. Every flow needs a start step and an end step.
• Transition: self.next(self.other_step), always the last line of a step. With two or more arguments it branches. With foreach= it fans out over a list.
• Join: a step with an extra argument (usually inputs) that collects the results of a branch or a foreach.
• Artifact: any value stored on self. It is saved when the task ends and versioned per run. A plain local variable is not saved.
• Run and task: a run is one execution of the flow; a task is one step executing in a run. A foreach makes one task per item. A task is addressed by a pathspec such as MyFlow/3/train/12.
• Parameter: a read-only value set when the run starts, declared with Parameter and passed as run --name value.
• Datastore and metadata: where artifacts and run records are kept. Local by default, in a .metaflow folder. Cloud storage and a shared metadata service are configuration, not code changes.

The mental picture that impresses interviewers is the one that explains behaviour: each task is a separate process, and tasks talk to each other only through saved artifacts. Everything else, including resume, parallelism and moving to a cluster, follows from that.

Commands to have ready

You should be able to type these without looking them up, and to say what each does.

pip install metaflow
python flow.py check
python flow.py show
python flow.py run
python flow.py run --alpha 0.5 --tag experiment-1
python flow.py resume
python flow.py resume --origin-run-id 3
python flow.py logs 3/train/12
python flow.py dump 3/train/12
python flow.py card view report
python flow.py --with retry run
python flow.py run --max-workers 1
python flow.py run --max-num-splits 1000
metaflow status
metaflow tutorials pull

Be ready to explain the word order: python flow.py [top-level options] command [command options]. Options such as --with, --environment and --datastore come before the command, while parameters come after run. Candidates who mix the two up stand out for the wrong reason.

And the Client API, which shows you have used the results and not just started runs:

from metaflow import Flow, namespace

run = Flow("TrainFlow").latest_successful_run
print(run.id, run.data.accuracy)
namespace(None)

The classic traps, and why they happen

Interviewers like to ask about mistakes because the answer shows whether you understand the design. Here are the ones that come up most, with the reasoning you should give.

"Why does my step say the attribute does not exist?" Because the variable was a local variable, not an artifact. Only values assigned to self are saved and passed on. The reason is that each step runs in its own process, so a local variable cannot survive. The fix is to assign to self, or to recompute it in the later step if it is cheap.

"Why does the run fail at the end of a step with a pickling error?" Artifacts are serialised with pickle when the task finishes. Objects that cannot be pickled, such as open files, database connections and locks, break at that moment, not where they were created. Keep such objects local and store only results.

"Why can I not change a parameter when I resume?" Because resume clones the finished steps of an earlier run and reuses the original parameter values so the results stay consistent. To try a new value, start a new run.

"Why does my join complain about conflicting artifacts?" Two branches set the same artifact name with different values, and Metaflow refuses to guess which is right. You set the artifact explicitly in the join, or call merge_artifacts with exclude for the conflicting names.

"Why does my foreach fail with more than 100 children?" There is a safety cap so that a mistaken list cannot launch a huge number of tasks. You can raise it with --max-num-splits, but the better question is whether each item deserves its own task.

"Why can I not see a run someone else made?" The Client API filters by namespace, by default to runs you created. namespace(None) widens the view. Namespaces are a convenience and not a security boundary.

"Why does my flow work here but say the flow does not exist there?" The local record lives in .metaflow under the folder where the flow was run, or on a different profile or service. Check the working directory and run metaflow status.

"Why is @pypi complaining about the environment?" Those decorators only work when you launch with --environment=pypi or --environment=conda, and the flag goes before run.

"Why does it not run on my Windows laptop?" Metaflow does not support native Windows, because its local runtime depends on a part of Python that only exists on POSIX systems. Use WSL 2.

Common interview questions

What problem does Metaflow solve, and why not just use a notebook or a script?

A notebook or script mixes exploration, execution order and state in one place. Its results are lost when the kernel dies, its run order is implicit, and nothing records which code and parameters produced a given number. Metaflow gives structure (a graph of named steps), automatic persistence (artifacts versioned per run), recovery (resume from the failed step) and a path to scale (the same file can run remotely and on a schedule). It does not replace notebooks for exploration. It is where a notebook's logic goes when it has to be repeatable.

How does data move between steps?

Through artifacts, meaning instance variables on self. When a task finishes, Metaflow pickles its artifacts and stores them in the datastore, tied to that run and task. The next step reads them from there. Local variables are not stored. Because nothing moves in memory, a step needs no knowledge of the process that ran the previous one.

Explain branch and foreach. When would you use each?

A branch is fixed in the code: self.next(self.a, self.b) runs two different steps in parallel, for example two different models. A foreach is driven by data: self.next(self.train, foreach="items") runs the same step once per item in a list and gives each task its item as self.input. Both must be joined by a step with an extra argument. I would use a branch when the parallel pieces are different kinds of work, and a foreach when they are the same work over different inputs, such as a hyperparameter sweep or one file per task.

What does a join step do and what goes wrong with it?

A join receives an object, usually called inputs, with one entry per branch or task. I read results from it by name for branches, such as inputs.model_a.score, or by iterating for a foreach. Artifacts set by several branches with conflicting values cannot be merged automatically, so I set them explicitly or call self.merge_artifacts(inputs, exclude=[...]). Also, a join cannot be a split in the same step, and every split must be joined before end.

What is the difference between run and resume?

run starts a new run from start with fresh parameters. resume takes an earlier run, by default the latest, reuses the results of the steps that succeeded, and re-executes from the step that failed, using the original parameters but the current code. I use it in the edit-fix-retry loop so that an expensive data step is not repeated.

What are retry, catch and timeout for?

@retry re-runs a failing task, up to four times, with a delay between attempts, which is right for transient failures like network errors. @timeout kills a step that exceeds a time limit. @catch catches the exception after retries, stores it in an artifact and lets the flow continue, which I would only use when the step's output is optional. I would add a caution that retried steps must be safe to repeat, because side effects happen again.

What are parameters, and how are they different from artifacts?

A parameter is an input supplied at the start of a run, set with a command-line option and read-only during the run. It is also stored with the run for reproducibility. An artifact is anything a step produces or derives and assigns to self. I never pass secrets as parameters, since parameters are persisted.

How would you look at the results of a flow after it finished?

For a quick look, logs shows a task's output and dump shows its artifacts. For analysis, I use the Client API: Flow("Name").latest_successful_run.data.metric reads an end-of-run artifact, and Step and Task give access to earlier artifacts. For sharing, I attach a @card with a table or a chart and open it with card view.

What is a card?

A card is an HTML report attached to a task. The default card shows the task's artifacts and metadata, and with current.card.append I can add Markdown, tables, images or charts. It is how I would share a model comparison with someone who will not read the code.

Where does Metaflow store data, and can that change?

By default in a .metaflow folder in the working directory. The datastore can be cloud object storage on AWS, Azure or Google Cloud, and run records can live in a shared metadata service. Those are configuration settings in a file under ~/.metaflowconfig or environment variables prefixed METAFLOW_, so the flow code stays the same. That separation is why a flow moves from a laptop to a team setup without rewriting.

Is Metaflow an orchestrator like Airflow?

Not at its core. It is first an authoring and tracking framework for data scientists: you write and run flows, and every run is recorded. The local runtime is for development. For production scheduling, Metaflow deploys flows to an orchestrator such as Argo Workflows, AWS Step Functions or Airflow, and the flow is translated to that system. So it complements an orchestrator. I would mention Airflow, Prefect and Kubeflow as alternatives for different ways of organising the work, and say the choice depends on whether the main need is authoring ergonomics for data scientists or general workflow orchestration.

How does it compare with MLflow?

They solve different problems. MLflow tracks experiments, parameters, metrics and models, and has a model registry. Metaflow defines and runs the pipeline and records its own artifacts and runs. Teams often use both: Metaflow for the pipeline, MLflow or a similar tool for experiment comparison and the model registry.

What do you do when a step is too slow?

First, measure: the logs have timestamps and the step's size is obvious. If the work is independent per item, turn it into a foreach so tasks run in parallel. If it is heavy, move only that step to bigger compute using resource requests, which is covered at the next level. If it runs for a long time, split it, because steps should be checkpoints and a huge step is expensive to redo.

Does Metaflow run on Windows?

Not natively. Use WSL 2 with Ubuntu, a Linux container, or a hosted sandbox.

How do you make a run reproducible?

Commit the flow code, pin library versions in a requirements or lock file, make seeds parameters, and tag runs. The record of each run holds the parameters and artifacts, and the Git history holds the code. Later-level practices add per-step environments with exact versions.

How do you test a flow?

At the beginner level: check for structure, a run with a small sample of data, and --max-workers 1 for readable output. Put the real logic in ordinary functions that can be unit-tested without Metaflow, and keep the steps thin. A step that only calls tested functions and saves their results is easy to trust.

A live-coding exercise to rehearse

A common task is: "Write a flow that loads a list of numbers, squares each in parallel, and returns the sum." Practise writing this from a blank file in about five minutes.

from metaflow import FlowSpec, step


class SquareFlow(FlowSpec):

    @step
    def start(self):
        self.numbers = [1, 2, 3, 4, 5]
        self.next(self.square, foreach="numbers")

    @step
    def square(self):
        self.value = self.input ** 2
        self.next(self.join)

    @step
    def join(self, inputs):
        self.total = sum(i.value for i in inputs)
        self.next(self.end)

    @step
    def end(self):
        print("total:", self.total)


if __name__ == "__main__":
    SquareFlow()

Talk while you type. Say why self.numbers is on self, what self.input is, why the join has two arguments, and what total will be (55). If you finish early, add a parameter for the list length and a @retry on square, and say why the retry is safe here (pure computation, no side effects).

Another common exercise is debugging. You are given a flow where a step uses df from a previous step. The answer is to store it on self, and to mention that you might instead store only what is needed if the DataFrame is large.

What strong answers have, and what weak ones miss

A strong beginner answer is concrete and honest. It names the mechanism (separate processes, artifacts in a datastore) and it says what the tool does not do: the local runtime is not a scheduler, namespaces are not security, and Windows needs WSL. It uses the right word for things (run, task, artifact, pathspec) and gives one example from a project the candidate actually wrote.

A weak answer is vague ("it makes pipelines easy"), confuses Metaflow with an experiment tracker, or claims features that the candidate has not used, such as running on a cluster. At this level it is entirely acceptable to say "I have only run flows locally, and I understand that remote execution uses the same flow with a decorator and a configured datastore." That answer is honest and shows you know where the boundary is.

If you are asked about something you have not met, such as conditional branches or spin, say so and reason from the model: "I have not used that, but given that every step is a process and tasks communicate through artifacts, I would expect it to..." Interviewers value reasoning from the model over memorised trivia.

Sixty-second self-test

Answer each out loud, without looking above. If you hesitate on more than two, reread the matching section of the beginner guide.

1. What is a flow, a step, a run and a task?
2. What is the last line of every step except end?
3. Which variables survive into the next step?
4. What is self.input, and when does it exist?
5. How does a join recognise a join, and what must you do about conflicting artifacts?
6. What is the difference between run and resume, and which parameters does resume use?
7. Where do top-level options such as --with go relative to run?
8. How do you pass a parameter, and why must you never pass a secret that way?
9. How do you read the result of the last successful run in Python?
10. Why does the Client API sometimes say the object is not in the current namespace?
11. What are the maximum values of @retry(times=...) and the default foreach split limit?
12. What do you do on a Windows laptop?

Answers in brief. A flow is the class, a step a method, a run one execution and a task one step instance. The last line is self.next(...). Only values on self. self.input is the item in a foreach task. A join has an extra argument, and conflicts are set explicitly or excluded in merge_artifacts. resume reuses finished steps and original parameters; run starts over. Top-level options go before run. Parameters use run --name value and are stored with the run. Flow("X").latest_successful_run.data. The namespace defaults to your own user. Retry times is at most 4 and the split limit is 100. Use WSL 2.

This page collects the practical lessons beginners learn the hard way with Metaflow. Each mistake below is described by what you see, why it happens, and what to do, because knowing the cause is what lets you recognise the next variation. It ends with habits worth adopting on day one and a checklist to run before you share a flow with anyone.

Mistakes beginners actually make

Using a local variable in the next step

Symptom. The run fails with AttributeError: 'MyFlow' object has no attribute 'df' in a step that worked fine in the notebook version.

Cause. The variable was created without self.. Each step runs as its own process, and only values assigned to self are saved and handed on. A local variable disappears when the step returns.

Fix. Assign self.df = df in the step that creates it. Then decide deliberately: if only one step needs a value, leave it local, and if several do, store it once. A good habit is to read every step and ask of each variable, "does a later step need this?"

Forgetting self.next or naming it wrongly

Symptom. Before anything runs, Metaflow prints "Validity checker found an issue" and either Step X is missing a self.next() transition, or specifies a self.next() transition to an unknown step.

Cause. Every step except end must finish with self.next(...), and the argument is the method itself, such as self.train, not a string and not a call. A typo in the name gives the "unknown step" message.

Fix. Put self.next(self.the_next_step) as the very last line of the step, and match the method name exactly. Run python flow.py check after every structural edit, because it finds these in a second.

Forgetting to join

Symptom. The terminal step end was reached before a split started at step(s) ... were joined. Add a join step before end.

Cause. You branched or used foreach and sent the branches straight to end. Metaflow requires that every split be closed by a join, a step that takes an extra inputs argument.

Fix. Add a step such as def join(self, inputs): between the parallel steps and end, collect what you need from inputs, and call self.next(self.end).

Conflicting artifacts in a join

Symptom. Step join cannot merge the following artifacts due to them having conflicting values: [result].

Cause. Several branches set an artifact with the same name and different values. Metaflow will not choose for you, which is correct, because choosing silently would hide a bug.

Fix. In the join, read the values you want from inputs into distinctly named attributes, then call self.merge_artifacts(inputs, exclude=["result"]) to carry over the rest. Do not use include and exclude together.

Putting @catch on the wrong step

Symptom. @catch is defined for the step s but @catch is not supported in foreach split steps.

Cause. The decorator was placed on the step that starts a foreach, not on the step that is repeated.

Fix. Move @catch to the inner step that does the work. If you want a failure in one item not to kill the others, this is how you get it, and the join must then check each task's failure artifact.

Wrong order of decorators and command-line options

Symptom. Metaflow says "no such option", or a flag seems to have no effect, for example python flow.py run --environment=pypi.

Cause. Options that configure Metaflow itself, such as --environment, --with, --datastore and --metadata, are top-level options and must come before the command. Options of the command, including your parameters, come after it.

Fix. Remember the shape: python flow.py [top-level options] run [run options]. The correct form is python flow.py --environment=pypi run. Decorators in code go directly above @step, with @step closest to the function.

Expecting resume to pick up a new parameter

Symptom. You change --alpha and use resume, and the result uses the old value.

Cause. By design, resume reuses the parameters of the run it continues, so that cloned steps and new steps are consistent.

Fix. Start a new run when you want different parameters. Use resume only for the edit-code-and-continue loop.

Raising the retry limit above four

Symptom. The maximum number of retries is @retry(times=4).

Cause. Metaflow caps retries at four additional attempts.

Fix. Use times=4 and raise minutes_between_retries to give a flaky service more time. If something fails more than five times in a row, it is not flaky, it is broken.

Reading a run and getting "Object not in the current namespace"

Symptom. Object not in namespace 'user:yourname' when you open a run in Python.

Cause. The Client API defaults to your own namespace. The run was made by someone else, by a scheduler, or under a different username.

Fix. Call namespace(None) before reading, or select a specific namespace. Remember this is a filter that keeps results tidy, and does not protect anything.

Running in the wrong folder

Symptom. Flow('X') does not exist, even though you ran it a minute ago.

Cause. With local metadata, the record of runs lives in .metaflow in the directory where you ran the flow. A notebook in another folder reads a different, empty record. A different METAFLOW_PROFILE has the same effect.

Fix. Change into the original folder. Run metaflow status to see which setup is active. Keep one folder per project so the question never comes up.

Storing something that cannot be pickled

Symptom. A step does all its work and fails at the very end with a pickling error.

Cause. Artifacts are serialised when the task finishes. File handles, database connections, locks and some client objects cannot be.

Fix. Keep such objects as local variables, use them inside the step, and store only results on self, such as a DataFrame or a number.

Storing huge intermediate data in every step

Symptom. Runs are slow and the .metaflow folder grows by gigabytes.

Cause. Every artifact is saved at the end of each task. Metaflow deduplicates identical values within a flow, but large objects that are modified each step are written again each time.

Fix. Keep bulky temporary data local to the step that uses it. Store the result, not every intermediate.

Importing libraries at the top of a flow that uses @pypi

Symptom. ModuleNotFoundError for a package you declared in @pypi.

Cause. The decorator builds the environment for the step. A module-level import runs outside it, before the step's packages exist.

Fix. Import inside the step, and declare the package in the decorator. The Mid-level guide covers this properly. For a beginner flow in your own virtual environment, top-level imports work fine.

Running Metaflow on native Windows

Symptom. ImportError: No module named fcntl.

Cause. The local runtime uses a POSIX-only module.

Fix. Install WSL 2 with Ubuntu, create a virtual environment inside it, and work there. Keep your project files inside the Linux filesystem, not under the Windows drive mounts, for speed.

Forgetting the __main__ block

Symptom. python flow.py run does nothing, or Python exits silently.

Cause. The file defines the class but never creates it.

Fix. End every flow file with if __name__ == "__main__": MyFlow().

Changing the name of a step and expecting resume to work

Symptom. After renaming or restructuring steps, resume fails or behaves strangely.

Cause. A resume matches the new graph against an old run. If the graph changed shape, the old run's records do not line up.

Fix. After a structural change, start a fresh run. Reserve resume for fixing the code inside steps.

Passing secrets as parameters

Symptom. None, until someone reads the run record and finds an API key in it.

Cause. Parameters are saved as artifacts with the run.

Fix. Read secrets from environment variables or a secrets manager inside the step. The Mid-level guide covers the @secrets decorator. Never commit keys to the flow file either.

Habits worth adopting on day one

Make check a reflex

python flow.py check takes a second and catches most structure errors. Run it whenever you edit a self.next, add a branch, or rename a step. Follow it with show once to confirm the graph looks like the diagram in your head.

Start with a tiny input

Before running over the full dataset, run the flow on a sample: a parameter such as --sample 1000 that limits rows. A flow that fails at step five after forty minutes of loading costs you a coffee, and one that fails after twenty seconds costs nothing.

Put the logic in plain functions

Write your data cleaning and modelling as ordinary Python functions in a separate module, and keep each step a few lines that call them and store the results. You can unit-test the functions without Metaflow, and the flow file remains a readable map of the pipeline. Mind one detail: if the flow imports from a neighbouring file, keep them in the same folder, because Metaflow packages the working directory's Python files.

Save what you would want to look at tomorrow

A good flow stores metrics, a trained model and a small summary as artifacts, not just a printed number. Ask "what would I want to compare across runs?" and store exactly those. Give the important ones short, stable names such as accuracy and model, because the Client API code you write later depends on them.

Make seeds and sizes parameters

Anything you might vary belongs in a Parameter: the seed, the test split, the number of epochs, the sample size. Then run --help documents your experiment space and every run's record shows its settings.

Tag runs with intent

--tag baseline, --tag after-feature-x. Tags turn a long history into something you can search, and they cost nothing. You can filter later with Flow("X").runs("baseline").

Keep the flow file in Git, and commit before you run something you care about

Metaflow remembers results, not the exact code in your editor at that moment. Committing first gives you a reliable link between a run and the code that made it.

Use resume instead of rerunning

When a late step fails, fix and resume. Rerunning from the start wastes time, and it also creates a new run that changes which one is "latest", which can confuse a notebook that reads latest_run.

Read the log prefix

[3/train/12 (pid 40217)] tells you run 3, step train, task 12. When a foreach has a hundred tasks and one fails, the prefix tells you which, and python flow.py logs 3/train/12 shows just that task's output.

Try a single worker when debugging

python flow.py run --max-workers 1 makes parallel work sequential and the output readable. Use it when you need to see one task at a time, including from an IDE debugger, which needs sub-process debugging enabled because every task is a child process.

Learn the two tutorial flows before inventing your own

metaflow tutorials pull downloads worked examples covering branches, foreach, parameters and cards. Reading a short, correct example is faster than debugging a clever, wrong one.

Pin your versions

Write your Metaflow and library versions into requirements.txt (pip freeze helps). Metaflow is careful about backwards compatibility, but your libraries may not be, and a flow that cannot be reproduced next month is not much of a record.

Mind your region and your data

If you work for an employer in the Gulf or Egypt, data residency rules often say where data may be stored. Local runs keep everything in .metaflow on your machine, which is fine for practice but may not be acceptable for real customer data. When a team moves to cloud storage, the region of the bucket is a real decision, so ask before copying production data to your laptop for a flow.

Short recipes

See the structure without running:

python flow.py check
python flow.py show

Run with extra tolerance for flakiness, without editing code:

python flow.py --with retry run

Run a small, readable trial:

python flow.py run --max-workers 1 --tag smoke

Continue after fixing a bug:

python flow.py resume

Find the best run later:

from metaflow import Flow

runs = [r for r in Flow("TuneFlow").runs() if r.successful]
best = max(runs, key=lambda r: r.data.best_accuracy)
print(best.pathspec, best.data.best_accuracy)

Read one task's artifact and log:

from metaflow import Task

task = Task("TuneFlow/5/train/12")
print(task.data.accuracy)
print(task.stdout)

List what a task saved, from the shell:

python flow.py dump 5/train/12

Share a result with a teammate:

python flow.py card view report

Reading an error the way Metaflow writes it

Metaflow errors have two parts: a short headline, such as "Validity checker found an issue", "Object not found" or "Incompatible environment", followed by a sentence that usually names the step and the cause. Read both before searching the internet, because the sentence often contains the fix, including the exact flag or decorator to use.

When the failure is inside your own code, you get a normal Python traceback with the task prefix in front of each line. Look for the line in your file, not in Metaflow's internals. If a task was retried, the log you see is the last attempt's. If a step failed and you used @catch, the exception is stored in the artifact you named, and the run may have ended "successfully" anyway, so check run.successful and the catch artifact, not just the final output.

Three quick diagnostic questions solve most problems. Did the structure pass check? Which task failed, and what does logs show for exactly that task? Is the variable I need stored on self in the step that made it?

A checklist before you share a flow

• The file ends with the __main__ block, and python flow.py check passes.
• Every step except end finishes with self.next(...), and every branch or foreach is joined.
• Step names say what the step does.
• Only results are stored on self; nothing unpicklable, nothing enormous and unused.
• Parameters cover the seed, sizes and anything a reader would vary, each with a help string.
• No secrets in parameters or in the file.
• Top-level options such as --with or --environment are before run in every command you document.
• Imports that need a special environment are inside steps.
• The flow was run once on a small sample and once in full, and resume was tried after a forced failure.
• Libraries and the Metaflow version are pinned in a requirements file.
• The flow file is committed, and the folder contains a short README with the exact command to run it.
• A @card or a stored summary makes the result readable without opening the code.
• If you are on Windows, the instructions say to use WSL 2.

Where to go when you are stuck

The official documentation at docs.metaflow.org has a page for each decorator and command, and its release notes are the place to check whether a feature exists in your version. The tutorials you pulled earlier are the best worked examples. If you are comparing Metaflow with tools such as Prefect, Dagster or Kedro, run the same small pipeline in each; the differences you feel while writing it are more informative than a feature table.

When you are comfortable, the Mid-level guide takes you to remote compute, per-step environments, Config and the Runner and Deployer APIs, and spin for fast step-by-step development.

Finally, keep a small notes file next to your flow listing the commands you ran and what you learned from each result. Metaflow records the runs, but only you can record why you ran them, and that short history is what makes an experiment understandable to a teammate, or to you a month from now.

This is part two of three. It picks up where Beginner ended: you can write a flow, run it locally, resume it and read results with the Client API. Now you run Metaflow at work, where flows execute on a cluster, depend on private packages, get scheduled, and fail at three in the morning. This level explains what Metaflow does under the hood precisely enough that you can predict its behaviour, then builds the production habits on top. The version used throughout is Metaflow 2.19 (2.19.39 at the time of writing).

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do against a real flow in a few minutes.

How a run executes

Beginner taught you that python flow.py run executes your steps in order. The mechanism matters the moment something misbehaves, so here it is in full.

When you run the file, Python loads your FlowSpec subclass and Metaflow builds the graph by reading your self.next(...) calls statically. A linter then checks that graph before anything runs, which is why typos in step names fail in milliseconds rather than after a two-hour job. Only then does the runtime start. The local runtime launches each task as a separate operating system process, running a hidden command roughly of the form python flow.py step <name> .... A parent process tracks the tasks, schedules the next ones as inputs become available, and caps concurrency at --max-workers (16 by default).

This process-per-task design has consequences worth internalising:

• Nothing is shared between steps in memory. Step B cannot see a stack variable, open file handle or database connection from step A. Anything B needs must be assigned to self so it is persisted as an artifact, or be recreated inside B.
• Every step is a checkpoint. Because each task ends by persisting its artifacts, a failed run can be resumed from the failed step without recomputing the earlier ones. The docs put the rule of thumb plainly: a step should not take more than an hour, preferably much less, because the step is the smallest resumable unit.
• Remote execution is the same thing with a different launcher. When you add @kubernetes or @batch, the task's command is wrapped in a container job instead of a local process. The container downloads your code package from the datastore, runs python flow.py step <name> ... inside it, reads its input artifacts from the datastore and writes its outputs back. Tasks never talk to each other directly.

That last point is the architectural key. Metaflow is a client-side library plus a few pluggable services:

Two settings choose the shared services. The datastore (local by default, which writes to a .metaflow/ directory in your working directory, or s3, azure, gs) holds artifacts, code packages and logs. The metadata provider (local by default, or service) records runs, tasks, attempts and tags. The Metaflow UI needs the metadata service. You can inspect which one you are pointing at with metaflow status, and most "my run disappeared" reports turn out to be one person reading local metadata while another wrote to the service.

The code package deserves its own paragraph because it explains a whole family of "works locally, fails remotely" bugs. Metaflow snapshots the files in your flow's directory whose suffixes match --package-suffixes (by default .py,.R,.RDS), adds the Metaflow library itself, and stores the snapshot in the datastore. Remote tasks unpack that snapshot. A JSON file, a YAML config or a model file next to your flow is therefore absent remotely unless you widen the suffixes or ship the file another way. You can look at exactly what would be shipped:

python flow.py package list
python flow.py --package-suffixes=.py,.json package list

Because Metaflow packages itself, remote tasks run the same Metaflow version as the client that launched them. You never rebuild images just to upgrade Metaflow. The flip side is that a deployed production workflow keeps the version snapshot it was created with, so re-running create after an upgrade is how you pick up fixes.

Artifacts, the datastore and moving bulk data

Beginner showed that anything assigned to self is available in later steps. At this level you need to know what that costs.

At the end of each task, Metaflow pickles every instance variable (by default; since 2.19.26 the serializer is pluggable) and writes it to the datastore. Artifacts are content-addressed: the blob is stored under a hash of its bytes, so identical values are stored once within a flow, and each task records a pointer to the blobs it produced. That is why assigning the same large DataFrame in five branches does not cost five copies, and why artifacts from any past run remain readable as long as the datastore keeps them.

The practical rules that follow:

1. Artifacts must be picklable. A database connection, a lock, a generator or an open file handle cannot be stored. Create those inside the step that uses them and keep them on the stack.
2. Stack variables are not persisted. A local name inside a step disappears when the step ends. Only self.something survives.
3. Every artifact is written and read through the datastore, even when the next step runs on the same machine. A step that assigns a 20 GB array to self pays a serialise, upload, download and deserialise cost at the boundary.
4. Only the last attempt's artifacts are visible. If a task retried, earlier attempts' values are not what the Client API returns.

For small and medium objects, artifacts are exactly right. For bulk data, Metaflow gives you a dedicated client, metaflow.S3, that parallelises transfers (64 workers by default, controlled by METAFLOW_S3_WORKER_COUNT) and handles retries:

from metaflow import FlowSpec, step, S3

class IngestFlow(FlowSpec):

    @step
    def start(self):
        with S3(s3root="s3://my-bucket/raw/2026-09/") as s3:
            keys = [obj.key for obj in s3.list_paths()]
            objs = s3.get_many(keys)
            self.paths = [o.path for o in objs]
            self.total = sum(1 for _ in self.paths)
        self.next(self.end)

    @step
    def end(self):
        print("downloaded", self.total, "files")

if __name__ == "__main__":
    IngestFlow()

Notice what is stored as an artifact here: a list of local paths and a count, not the file contents. The pattern to aim for is that artifacts carry small, meaningful state (a path, a model's metrics, a list of keys), and heavy data moves through S3, a warehouse, or a table format, with the flow passing references between steps. The objects returned by S3 point at temporary local files that exist only while the with block is open, so read or copy what you need inside the block. In the example above the paths would be dangling after the block exits, which is a good illustration of why you should copy data out or process it inside the context manager.

Two configuration keys matter once you use an S3-compatible store that is not AWS. METAFLOW_S3_ENDPOINT_URL points the client at MinIO or another compatible service, and METAFLOW_S3_VERIFY_CERTIFICATE controls TLS verification. If your organisation intercepts TLS, fix it by trusting the corporate certificate authority rather than disabling verification.

For data residency, the datastore bucket's region is where your artifacts live. Teams at Gulf and Egyptian employers often have to keep training data and derived artifacts in an in-country or approved region, and in Metaflow that is a bucket and IAM decision, not a code change. Because any artifact can contain customer data, anyone who can read the bucket can read every artifact, which is why sensitive flows get their own bucket or prefix.

Designing the graph: foreach, joins, conditionals and recursion

The shape of a flow is a production decision. It determines parallelism, how much work is lost on a failure, and what the orchestrator can run.

Foreach sizing and joins

A foreach creates one task per item in the list you name, and every task reads its item from self.input. It is the tool for hyperparameter sweeps, per-customer or per-country processing, and sharded data. Two limits guard it. Locally, --max-num-splits (default 100) makes the run fail if a foreach yields more items than that, with the message that it is more than the current maximum of 100 children. This is a safety rail against accidentally launching ten thousand cloud jobs. Raise it deliberately: python flow.py run --max-num-splits 1000. Concurrency is capped separately: --max-workers defaults to 16 for local runs and 100 on orchestrators such as Argo Workflows.

A foreach must be joined. A join is a step with a second argument:

@step
def join(self, inputs):
    self.scores = {i.country: i.auc for i in inputs}
    self.best = max(inputs, key=lambda i: i.auc).country
    self.merge_artifacts(inputs, exclude=["country", "auc"])
    self.next(self.end)

Inside a join, inputs iterates over the parents, and each element exposes that task's artifacts as attributes. self.merge_artifacts(inputs) copies across every artifact whose value is the same in all parents, which is how a value set before the split flows through untouched. If an artifact differs across parents, Metaflow refuses to guess and raises the "Unhandled artifacts in merge" error, telling you to set it explicitly before merging or to exclude it. include and exclude are mutually exclusive, and excluding the per-branch values, as above, is the usual fix.

The linter also rejects a step that is both a join and a split (add a separate step between them), and a foreach that ends in end without a join.

Conditional branches and recursion

Since Metaflow 2.18, a step can choose exactly one next step at runtime:

@step
def validate(self):
    self.outcome = "ok" if self.metrics["auc"] >= 0.80 else "retrain"
    self.next({"ok": self.publish, "retrain": self.tune}, condition="outcome")

The keys must be literals (strings, numbers, booleans or config expressions), never variables, and there must be a key for every value the condition can take. No join is needed because only one path runs. There is one restriction that surprises people: a conditional path cannot lead directly into a join. The linter tells you to insert an intermediate unconditional step on that path.

Recursion is a conditional where one option points back at the same step, which gives you loops such as "train until the loss converges" or "poll until the upstream table exists":

@step
def poll(self):
    self.ready = check_table_ready()
    self.next({True: self.load, False: self.poll}, condition="ready")

Each iteration is its own task, so each is individually visible in the UI and Client API, and resume continues from the last successful iteration rather than starting the loop again. Put a sensible bound in the logic, such as an attempt counter, because Metaflow will not stop an infinite loop for you.

Naming the entry and exit steps

By default the entry step must be named start and the exit step end. Since 2.19.23 you may mark them with @step(start=True) and @step(end=True) instead, which also allows single-step flows. The end step must not call self.next() and must not be a join.

Failure handling: retry, catch, timeout and resume

Production flows fail for boring reasons: a spot instance is reclaimed, an upstream API returns a 503, a node is out of memory. Metaflow has four tools, and their combination is the design.

@retry re-runs a failed task as a new attempt. The defaults are times=3 and minutes_between_retries=2, and the maximum is times=4; asking for more fails the linter with "The maximum number of retries is @retry(times=4)". If you need longer total patience, raise minutes_between_retries instead. Retry only steps that are safe to repeat. A step that appends rows to a table or sends an email needs to be idempotent first, meaning running it twice leaves the same end state as running it once, for example by writing to a partition keyed on the run ID.

@timeout kills a task that runs too long. The arguments seconds, minutes and hours add up. Two details: with @kubernetes the minimum is 60 seconds, and without any timeout a remote task is bounded only by METAFLOW_DEFAULT_RUNTIME_LIMIT, which is five days. A hung step that nobody notices will therefore hold a node for five days. Set an explicit, generous timeout on every remote step.

@catch turns an exception into data. When the step fails after its retries are exhausted, the flow continues and the exception is stored in the variable you name:

@catch(var="failed", print_exception=True)
@retry(times=2)
@step
def score_segment(self):
    self.result = risky_model_call(self.input)
    self.next(self.join)

Downstream code can check self.failed and decide what to do. Two rules: @catch is not allowed on the step that fans out (put it on the inner step), and @catch is not supported with @parallel on Kubernetes. Use it for partial-failure tolerance, such as "score the other 49 countries even if one breaks", not to hide failures. A flow that catches everything can finish green while producing nothing useful, so have the join step assert on the number of failures.

resume is the recovery tool, and its behaviour is worth stating exactly. python flow.py resume finds the last run, locates the step that failed, clones every successful task before that point (reusing their artifacts rather than recomputing) and re-executes from the failed step forward. You can choose another origin with --origin-run-id 12, and start from a chosen step with python flow.py resume train. Three consequences matter at work:

• Resume reuses the original parameter values and configs. Changing a parameter or config does nothing on resume by design; start a new run instead.
• Resume uses the current code for the steps it re-executes. That makes it a fast edit-debug loop: fix the bug in the failing step, resume, and only that step and its successors run.
• A resumed run gets a new run ID. The original failed run stays, untouched, in history.

Spot instances combine these tools. On AWS, cheap reclaimable capacity can disappear mid-task; pair @retry with code that can restart cheaply. For long training jobs the @checkpoint decorator from the separate metaflow-checkpoint extension saves progress inside a task. It is an extension, not core, and its APIs may change, so pin its version.

Remote compute with Kubernetes and AWS Batch

Local processes stop being enough when a step needs a GPU, 64 GB of RAM, or a hundred parallel workers. Metaflow's answer is that the same flow file runs remotely by attaching a decorator, with no rewrite.

There are two backends: @kubernetes (any cluster, including on-premises, EKS, AKS and GKE) and @batch (AWS Batch). You choose per step, or for every step from the command line:

python flow.py run --with kubernetes
python flow.py run --with batch

A step that carries its own @kubernetes(...) runs remotely even without --with, so you can mix: a cheap local step for data validation and one heavy remote step for training. A step cannot carry both decorators; the linter says it is "marked for execution both on AWS Batch and Kubernetes".

Requirements trip people up, so check them before the first remote run. Remote compute needs a cloud datastore (--datastore=s3, azure or gs), because the remote container has no access to your laptop's .metaflow/ directory. The error reads "The @kubernetes decorator requires --datastore=s3 or --datastore=azure or --datastore=gs at the moment." The client that launches the job also needs the Python kubernetes package (pip install kubernetes), or you get "Could not import module 'kubernetes'". The metaflow configure kubernetes and metaflow configure aws wizards write the settings into your configuration file, and every setting can be overridden with a METAFLOW_ environment variable.

Requesting resources

@resources states what a step needs in a backend-neutral way, and it has no effect locally:

@resources(cpu=4, memory=16000, gpu=1, disk=50000)
@step
def train(self):
    ...

Memory and disk are in megabytes. The @kubernetes defaults are 1 CPU, 4096 MB of memory and 10240 MB of disk; @batch defaults to 1 CPU and 4096 MB. If you put both @resources and @kubernetes on a step, Metaflow takes the larger of each request. For finer control, @kubernetes accepts the arguments you would expect from a pod spec: image, service_account, node_selector, tolerations, labels, annotations, secrets, namespace, gpu, gpu_vendor (default nvidia), shared_memory, persistent_volume_claims, use_tmpfs, qos and security_context. Since 2.19.34 it also accepts extended_resources for vendor-specific resources. The quality-of-service class defaults to burstable (from METAFLOW_KUBERNETES_QOS); use Guaranteed for steps that must not be evicted under node pressure, which requires requests to equal limits.

GPU steps need two things beyond gpu=1: a node pool with GPUs that your tolerations and node selector can reach, and an image with the matching CUDA libraries. Setting gpu=0 on a step forces it onto non-GPU nodes.

Reading a stuck or killed task

Two symptoms cover most remote incidents:

Metaflow's docs are candid that if a large enough instance is not available, the task simply will not start. Nothing in the flow fails; it waits. A dashboard or alert on Pending tasks older than some threshold is cheap insurance, and @timeout does not help here because the clock only starts once the task runs.

Finished Kubernetes jobs are kept for seven days by default (METAFLOW_KUBERNETES_JOB_TTL_SECONDS_AFTER_FINISHED), which is useful for debugging and wasteful in a busy cluster, so tune it. For how this sits inside your cluster operations, see the Kubernetes guide, and for provisioning the cluster and buckets as code, the Terraform guide.

Dependencies: @pypi, @conda, uv and images

"It works on my laptop" is a dependency problem, and remote execution makes it the most common one. There are three approaches, and you should choose deliberately.

Bake everything into a container image and point tasks at it with @kubernetes(image=...) or METAFLOW_KUBERNETES_CONTAINER_IMAGE. This is the most predictable choice for heavy stacks such as CUDA and compiled libraries, and it fits an existing registry and scanning process. See the Docker guide. The cost is an image pipeline and slower iteration.

Let Metaflow build per-step environments with @pypi or @conda, using the --environment flag:

from metaflow import FlowSpec, step, pypi

class TrainFlow(FlowSpec):

    @pypi(python="3.11.9", packages={"pandas": "2.2.3", "scikit-learn": "1.5.2"})
    @step
    def start(self):
        import pandas as pd
        from sklearn.linear_model import LogisticRegression
        self.model_type = LogisticRegression.__name__
        self.next(self.end)

    @step
    def end(self):
        print(self.model_type)

if __name__ == "__main__":
    TrainFlow()

python flow.py --environment=pypi run

Three details cause most of the confusion. First, the environment flag is required and goes before run; without it you get the "Incompatible environment" error saying @pypi requires --environment=conda or --environment=pypi (or fast-bakery). Second, import the package inside the step. The module-level imports at the top of the file run in your client environment, where the package may not exist, so move them inside steps that declare them. Third, pin versions. An unpinned package resolves at build time and quietly changes under you.

@conda works the same way, with packages= (the older libraries= is a backward-compatible alias) and channel syntax such as 'pytorch::pytorch': '2.1.2'. The default channel is conda-forge. For a dependency set shared by the whole flow use the flow-level forms @pypi_base and @conda_base. Built environments are cached in the datastore, so only the first resolve is slow. --force-rebuild-environments rebuilds them when you suspect a stale cache.

Use uv if your project is managed by it (supported since 2.15.8):

uv init myflow && cd myflow
uv add pandas metaflow
uv run flow.py --environment=uv run

Or set export METAFLOW_ENVIRONMENT=uv so you do not repeat the flag.

How to choose: images for GPU and system-library-heavy work and for organisations with an image supply chain; @pypi for pure-Python steps with differing needs, where each step can have its own versions; uv for a project that already has a lockfile. Whichever you pick, keep your own code importable remotely. If your flow imports a sibling package, it must be in the code package (package list shows what ships), and as of 2.19.39 the @package_sources flow mutator can add extra source directories to the package.

Configuration: Parameters, Configs, profiles and secrets

A flow accumulates settings: learning rates, table names, resource sizes, environments. Metaflow has three different mechanisms, and choosing the wrong one is a recurring source of confusion.

The docs give the decision rule, and it is worth memorising. Use a Config to alter a production deployment or to configure decorators. Use a Parameter to pass values from the CLI, a UI or an event payload. Use artifacts for everything else.

The difference is timing. A Parameter is set at the start of each run and is read-only during it. A Config, introduced in 2.13, is evaluated and persisted when the flow is deployed: for a local run that is just before execution, for Argo Workflows or Step Functions it is at create. Because it exists at deploy time, a Config can parametrise decorators, which a Parameter cannot:

from metaflow import FlowSpec, step, Config, config_expr, resources, timeout, pypi_base

@pypi_base(python="3.11.9", packages=config_expr("config.packages"))
class ForecastFlow(FlowSpec):
    config = Config("config", default="config.json")

    @timeout(seconds=config.limits.timeout_seconds)
    @resources(cpu=config.limits.cpu, memory=config.limits.memory_mb)
    @step
    def start(self):
        print("horizon", self.config.horizon)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    ForecastFlow()

{
  "horizon": 14,
  "limits": {"timeout_seconds": 3600, "cpu": 4, "memory_mb": 16000},
  "packages": {"pandas": "2.2.3"}
}

Inside a step the config is a read-only object you access with dot notation. Outside the class you use config_expr("config.packages"). To point a run at another file, put the option before the command: python flow.py --config config prod.json run, or provide values inline with --config-value config '{"horizon": 28}'. Those top-level options must come before run or create, like --environment, --with and --branch.

Older code uses deploy-time Parameters, where the default is a function. They still work, but the docs now recommend Config. If you inherit a flow that does this, leave it running and migrate when convenient.

Profiles for multiple environments

Metaflow's own settings (datastore roots, service URLs, cluster names) live in ~/.metaflowconfig/config.json. Set METAFLOW_PROFILE=staging and Metaflow reads config_staging.json from the same directory instead. Keep a profile per environment, and export one with metaflow configure export -o file.json to share with a colleague who imports it with metaflow configure import file.json. A missing profile produces "Unable to locate METAFLOW_PROFILE", which means the file name does not match the variable.

Secrets

Credentials do not belong in Parameters, Configs or code. Use @secrets:

from metaflow import FlowSpec, step, secrets
import os

class DbFlow(FlowSpec):

    @secrets(sources=["prod-db-credentials"])
    @step
    def start(self):
        user = os.environ["DB_USER"]
        self.rows = query_users(user, os.environ["DB_PASSWORD"])
        self.next(self.end)

    @step
    def end(self):
        print(len(self.rows))

if __name__ == "__main__":
    DbFlow()

The secret's key-value pairs arrive as environment variables in the task. The backend is set by METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE: aws-secrets-manager, az-key-vault or gcp-secret-manager. A role= argument lets the task assume a different IAM role. You can swap sources per environment at deploy time with --with 'secrets:sources=["staging-db"]'. The safest setup also restricts the secret with IAM so only the production orchestrator's role can read it. For a deeper treatment of secret stores, see the Vault guide.

Cards, the Client API, namespaces and tags

A pipeline earns trust when people can see what it did. Metaflow's answer is two-part: cards to report on a single task, and the Client API to query across runs.

Cards

A card is an HTML report attached to a task. Add @card and append components:

from metaflow import FlowSpec, step, card, current
from metaflow.cards import Markdown, Table, Image
import matplotlib.pyplot as plt

class ReportFlow(FlowSpec):

    @card(type="blank", id="metrics")
    @step
    def start(self):
        fig, ax = plt.subplots()
        ax.plot([0.61, 0.72, 0.78, 0.81])
        current.card["metrics"].append(Markdown("## Validation AUC"))
        current.card["metrics"].append(Image.from_matplotlib(fig))
        current.card["metrics"].append(Table([["auc", 0.81], ["f1", 0.74]], headers=["metric", "value"]))
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    ReportFlow()

View it with python flow.py card view start, or run python flow.py card server for a local viewer (port 8324 by default). Since 2.11 cards can update live while a task runs: call current.card.refresh() after appending, which is how you watch training curves on a long job. The Table.from_dataframe(df), VegaChart.from_altair_chart(chart) and ProgressBar helpers cover most needs. On remote runs the card is stored in the datastore, and the UI renders it. Experiment trackers, such as those covered in the MLflow guide and Weights & Biases guide, run happily inside steps, and a card is a good place to put the link to the tracked run.

The Client API in practice

The Client API reads what past runs stored, with no access to the running code:

from metaflow import Flow, Run, namespace

run = Flow("TrainFlow").latest_successful_run
print(run.id, run.finished_at, run.data.model_type)

for task in run["train"]:
    print(task.pathspec, task.successful, task.stdout[-200:])

The objects form a hierarchy: Flow, Run, Step, Task, with pathspecs like TrainFlow/12/train/345. run.data exposes the artifacts of the end step, which is why the final step should assign what consumers need. For model promotion or batch scoring that loads a previous run's model, prefer a pinned Run("TrainFlow/12") over latest_successful_run, so a new experimental run cannot silently change what production scores with.

Namespaces

By default, the Client API only sees runs from your user: the namespace is user:<username>. Reading a colleague's run, or a production run, raises "Object not in the current namespace". The fix is deliberate: namespace(None) switches to the global namespace (all runs), and namespace("production:...") targets a production deployment. The docs are explicit that namespaces are not about security or access control; they are a filter that keeps you from accidentally reading someone else's experiment. Real access control belongs to cloud IAM on the bucket.

Tags

Tags attach your own labels to runs, and they can be mutated after the fact, whereas system tags (user:, runtime:, metaflow_version:) cannot:

python flow.py run --tag candidate --tag q3-refresh
python flow.py tag add --run-id 12 approved

from metaflow import Flow
approved = Flow("TrainFlow").runs("approved")
best = next(iter(approved))

Tags are the lightweight promotion mechanism: tag a run approved after review and have downstream consumers query for it. Combined with namespaces, this gives you a workable model registry without extra infrastructure, though a dedicated registry such as MLflow is the better choice if you need stage transitions, lineage and access control on models as first-class objects.

Running flows programmatically: Runner, Deployer and spin

Everything so far used the command line. At work you also need to start flows from other code: a test, a notebook, a CI job, an internal API. Since 2.12 Metaflow has two supported APIs, and since 2.19 a third tool for fast iteration.

The Runner API

Runner launches a flow as a subprocess and gives you a handle:

from metaflow import Runner

with Runner("flow.py", show_output=False).run(alpha=0.5, tags=["nightly-test"]) as running:
    print(running.status)
    run = running.run
    print(run.successful, run.data.metric)

The keyword arguments to run are the flow's Parameters and run options such as max_workers. The result, an ExecutingRun, has .status, .stdout, .stderr and .run, the normal Client API Run. For async code use async_run() with await running.wait() and running.stream_log("stdout"). In notebooks, NBRunner takes a flow class defined in a cell. Use the context manager, or call runner.cleanup(), because the Runner creates temporary files. The Runner requires Python 3.7 or later.

The typical use is an integration test: run a small version of your flow against local or remote backends and assert on run.data. That test belongs in CI, which is where the local devstack comes in. metaflow-dev up needs Docker and starts a Minikube cluster with the metadata service, UI, Argo Workflows and storage; metaflow-dev shell opens a shell already configured to use it. Expect it to need several CPUs and around 8 GB of memory by default.

The Deployer API

Deployer does what argo-workflows create does from the command line, from Python:

from metaflow import Deployer

deployed = Deployer("flow.py").argo_workflows().create()
triggered = deployed.trigger(alpha=5)
triggered.wait_for_run(timeout=3600)
print(triggered.run.successful)

This is how a CI pipeline can deploy a flow, trigger it once as a smoke test and then delete the test deployment. DeployedFlow.from_deployment(...) and list_deployed_flows(...) for finding existing deployments are Argo-only, and the Deployer does not support Airflow. The Deployer is the recommended glue for CI/CD, where it replaces ad hoc shell scripts around the CLI.

spin: fast iteration on one step

Before 2.19, testing a change to the last step of a flow meant running the whole flow or using resume. spin runs one step using the inputs recorded by an earlier run:

python flow.py spin train
python flow.py spin 32455/train/355
python flow.py spin train --persist

The first form uses the latest run's inputs, the second a specific task. With --persist the outputs are kept in ./.metaflow_spin, which you can read back with inspect_spin(".") from metaflow. You can also override inputs with --artifacts-module artifacts.py, a module defining an ARTIFACTS dictionary. Know its limits: spin is local only, it ignores or disallows many decorators (the allowed ones include @conda, @pypi, @environment, @project, @timeout and @card, and @parallel is not allowed), and the docs warn it may not work properly on flows with conditionals. It may also require a newer metadata service. Treat it as an inner-loop tool, not a test strategy.

Debugging in an IDE

Because steps run in subprocesses, attaching a debugger needs two settings in VS Code's launch.json: "args": ["run"] and "subProcess": true. Add --max-workers 1 so you hit one breakpoint at a time.

Production orchestration: Argo Workflows, schedules and triggers

A flow run from your laptop is not a production system. python flow.py run depends on your machine staying awake, and the local runtime is explicitly not intended as a production-grade scheduler. Production means a workflow orchestrator runs your flow on a schedule or on events, with retries, alerting and history. Metaflow compiles your flow into the orchestrator's native format: Argo Workflows (on any Kubernetes), AWS Step Functions (AWS only), Apache Airflow, or Kubeflow Pipelines (via the metaflow-kubeflow extension). Each step still runs as a Metaflow task on Kubernetes or Batch; the orchestrator only decides when and in what order.

For most teams on Kubernetes the path is Argo Workflows. The feature set is also the widest: conditionals, recursion, nested foreach, event triggering and exit hooks all work there.

Deploying

python flow.py --environment=pypi argo-workflows create
python flow.py argo-workflows trigger --alpha 0.5
python flow.py argo-workflows list-workflow-templates

create compiles the flow into an Argo WorkflowTemplate, snapshots the code package and Config values, and uploads both. After that, the deployment runs the snapshot, not your working directory. A deployed flow does not run by itself: Argo only starts it when you trigger it or when it has @schedule or @trigger decorators.

Three operational details cause most first-week incidents:

1. Name length. Since 2.17, the full workflow name is limited to 45 characters because Kubernetes labels are capped at 63. Flows using @project get names truncated and hashed automatically. Without @project, a long name fails with a message telling you to rename or pass argo-workflows --name <name> create. Deploying the same flow with an older Metaflow than 2.17 after a long-name deployment creates two separate deployments, so keep all deployers on a current version.
2. Production tokens. The first create generates a production token associated with the flow and its creator. If another person later runs create for the same flow, they get "Incorrect production token" until they pass --authorize TOKEN. This protects against accidental overwrites by a colleague, and it is not a security boundary.
3. Snapshot semantics. Editing flow.py does nothing to an existing deployment. You must run create again, and you should do that from CI, not a laptop.

Isolating deployments with @project

Without structure, two engineers deploying ForecastFlow fight over one production deployment. @project solves this by namespacing deployments:

from metaflow import FlowSpec, step, project, schedule

@project(name="demand_forecast")
@schedule(cron="0 6 * * *", timezone="Asia/Riyadh")
class ForecastFlow(FlowSpec):
    ...

python flow.py argo-workflows create                        # your own user branch
python flow.py --branch feature_x argo-workflows create     # a test branch
python flow.py --production argo-workflows create           # production
python flow.py --production --branch v2 argo-workflows create

Each yields a separate deployment (for example demand_forecast.user.<name>.ForecastFlow and demand_forecast.prod.ForecastFlow), with its own namespace for results. Inside a run, current.project_name, current.branch_name and current.is_production let you route to a test database or the real one. A healthy pattern: developers deploy to their own user branch, CI deploys --branch per pull request for integration testing, and only the main-branch pipeline runs --production. Note that argo-workflows --name cannot be combined with @project; use --branch instead.

Schedules and events

@schedule accepts hourly=True, daily=True, weekly=True or cron="...", plus timezone (Argo only). Watch the defaults: a bare @schedule means daily. On Step Functions, cron uses the AWS six-field format, for example '0 10 * * ? *'. One compatibility note: by default Metaflow emits the legacy singular schedule field for Argo; setting METAFLOW_ARGO_WORKFLOWS_USE_SCHEDULES=true emits the list form, which needs Argo Workflows 3.6 or later.

For event-driven pipelines there are two flow decorators, Argo only because they use Argo Events:

from metaflow import FlowSpec, step, trigger, trigger_on_finish

@trigger_on_finish(flow="FeatureFlow")
class TrainFlow(FlowSpec):
    @step
    def start(self):
        upstream = current.trigger.run
        ...

@trigger(event="data_updated") reacts to an external event, which you can publish from any Python code with ArgoEvent(name="data_updated").publish(payload={...}) from metaflow.integrations. @trigger_on_finish chains flows, and with @project it matches within the same project and branch. You can test the trigger path locally by passing the upstream run: python flow.py --trigger FeatureFlow/233 run. Take care not to create a cycle of flows that trigger each other forever. Using events requires the Argo Events settings (METAFLOW_ARGO_EVENTS_EVENT, ..._EVENT_SOURCE, ..._SERVICE_ACCOUNT), and the error when they are missing tells you which one.

Alerting and exit hooks

On Argo, argo-workflows create accepts --notify-on-error and --notify-on-success with a Slack webhook (--notify-slack-webhook-url), PagerDuty or incident.io options. For custom behaviour use @exit_hook(on_error=[...], on_success=[...]) (2.16+), which runs your functions in separate pods after the flow ends. Exit hooks run locally and on Argo only, cannot create artifacts and cannot use @pypi or @conda. For pipelines that also run in Airflow, see the Airflow guide, and for wiring the deploy step into GitOps, the Argo CD guide.

Debugging and performance beyond the basics

Most production problems are one of four kinds: the code package is missing something, the environment is different, the resources are wrong, or the infrastructure is unreachable. A fixed order of checks finds them fast.

Reading the errors you will actually see

Performance levers

Metaflow rarely makes the computation slower; the overhead it adds is at task boundaries, so reduce them.

• Right-size steps. Many tiny steps pay per-task startup and serialisation cost (on Kubernetes, pod scheduling dominates). Steps under a minute usually belong merged. Steps over an hour are candidates for splitting or checkpointing.
• Move bulk data through S3 and keep artifacts small. Also consider use_tmpfs=True on @kubernetes for memory-backed scratch space when a step does heavy temporary I/O.
• Cap fan-out deliberately. --max-workers and --max-num-splits are cost controls as much as performance ones. A foreach of 5,000 items at 100 workers is fifty waves of work.
• Cache environments. @pypi and @conda resolves are cached in the datastore; changing a package version invalidates that step's environment.
• Use spot instances with care. They cut cost but need retries, and only steps that restart cheaply benefit.

For observability, Metaflow can export traces to an OpenTelemetry collector or Zipkin via METAFLOW_OTEL_ENDPOINT and METAFLOW_ZIPKIN_ENDPOINT, and the service name is configurable. That lets flows show up beside your other services, as described in the Prometheus guide for metrics scraped from the same cluster.

Putting it all together

Here is one flow that uses most of this level: a Config-driven nightly forecast with a per-country foreach, remote compute, pinned dependencies, secrets, retries, a card, and a production deployment under @project.

from metaflow import (FlowSpec, step, Config, config_expr, project, schedule,
                      kubernetes, resources, retry, catch, timeout, secrets,
                      pypi_base, card, current)
from metaflow.cards import Markdown, Table
import os

@project(name="demand_forecast")
@schedule(cron="0 3 * * *", timezone="Asia/Riyadh")
@pypi_base(python="3.11.9", packages=config_expr("cfg.packages"))
class ForecastFlow(FlowSpec):
    cfg = Config("cfg", default="config.json")

    @step
    def start(self):
        self.countries = list(self.cfg.countries)
        self.next(self.forecast, foreach="countries")

    @catch(var="failed", print_exception=True)
    @retry(times=2, minutes_between_retries=1)
    @timeout(minutes=45)
    @secrets(sources=["warehouse-readonly"])
    @resources(cpu=4, memory=16000)
    @kubernetes
    @step
    def forecast(self):
        import pandas as pd
        self.country = self.input
        df = pd.read_sql(
            "select * from sales where country = %s", os.environ["WAREHOUSE_URL"],
            params=[self.country],
        )
        self.mape = float(run_model(df, horizon=self.cfg.horizon))
        self.next(self.join)

    @card(type="blank")
    @step
    def join(self, inputs):
        ok = [i for i in inputs if not getattr(i, "failed", None)]
        self.bad = [i.country for i in inputs if getattr(i, "failed", None)]
        assert len(ok) >= len(self.cfg.countries) - self.cfg.max_failures, self.bad
        self.scores = {i.country: i.mape for i in ok}
        current.card.append(Markdown("## MAPE by country"))
        current.card.append(Table([[c, m] for c, m in self.scores.items()],
                                  headers=["country", "MAPE"]))
        self.next(self.end)

    @step
    def end(self):
        print("forecasted", len(self.scores), "countries; failed:", self.bad)

if __name__ == "__main__":
    ForecastFlow()

The deployment pipeline then does four things, all in CI and none from a laptop:

1. pip install -r requirements.txt with Metaflow pinned, plus pip install kubernetes.
2. python forecast_flow.py --environment=pypi run --max-num-splits 50 against a test profile, driven through Runner inside a pytest test that asserts run.successful and that every country produced a score.
3. On pull requests, python forecast_flow.py --environment=pypi --branch pr_$PR_NUMBER argo-workflows create, then a triggered smoke run and a delete.
4. On merge to main, python forecast_flow.py --environment=pypi --production argo-workflows create.

Notice the decisions in this design. The step list and decorators are visible in the file, so a reviewer can see resources, retries and secrets at a glance. Partial failure is tolerated but bounded by an assertion, which keeps green from meaning "did nothing". Country list, horizon and package versions are in config.json, so changing them means a reviewed commit and a re-deploy. Nothing sensitive is in a Parameter or artifact.

Where you are now

You can now explain what happens when a flow runs, why tasks communicate only through the datastore, and what travels in the code package. You design graphs with foreach, joins, conditions and recursion while knowing which orchestrators support them. You handle failure with retries, timeouts, catches and resume and know their limits. You run steps on Kubernetes or Batch with right-sized resources, manage dependencies three different ways, and choose between Parameters, Configs and secrets correctly. You can test flows with the Runner, iterate with spin, deploy with the Deployer, and run production on Argo Workflows with @project branches, schedules and triggers.

The next level treats Metaflow as a platform that other teams depend on: datastore and metadata-service architecture, scaling limits, security given that open-source Metaflow has no authentication of its own, multi-tenancy, upgrade strategy, and when a different orchestrator or tool is the better answer.

Sources

• Metaflow documentation
• Scaling flows
• Remote tasks on Kubernetes
• Remote tasks on AWS Batch
• Requesting resources
• Managing dependencies
• Dependencies with uv
• Dealing with failures
• Managing secrets
• Loading and storing data
• Configuring flows
• Visualizing results with cards
• Client API
• Tagging
• Runner API
• Deployer API
• spin: running a single step
• Debugging flows
• Scheduling with Argo Workflows
• Event triggering
• Coordinating larger projects
• Metaflow release notes
• Metaflow on GitHub

Part two of three. This is a cumulative review of Beginner and Mid-level Metaflow, organised by scenario rather than by level, in about thirty-five minutes. Interviewers at this level do not ask what a flow is. They describe a failing pipeline or a design decision and listen to how you reason. Read the fast review first, then work through the scenarios aloud. Senior reviews all three levels.

Where this picks up

Foundations in one screen

Have these five sentences ready, because almost every scenario question depends on one of them:

1. A step is the smallest resumable unit, and a task is one execution of a step. A foreach step creates many tasks.
2. Every task runs in its own process (or pod), so nothing is shared in memory. Data moves only as artifacts written to the datastore.
3. Remote tasks download a code package containing only files with the configured suffixes (.py,.R,.RDS by default) plus Metaflow itself.
4. Top-level options such as --environment, --with, --config, --branch and --production go before the command: python flow.py --environment=pypi run.
5. Metaflow's open-source namespaces and production tokens prevent accidents. They are not security. Cloud IAM is.

Scenario questions

"A step works locally and fails on Kubernetes with ModuleNotFoundError. What do you check?"

A strong answer separates two causes and checks the cheaper one first. The first cause is that the package is not installed in the remote image. If the flow uses @pypi or @conda, check that the run used --environment=pypi (or conda) and that the flag was placed before run; otherwise the decorator either errors or is ignored. Also check that the import is inside the step, because module-level imports execute in whatever environment loads the file, which may not have the package. The second cause is that it is your own code that is missing: a sibling module or a data file that was not in the code package. python flow.py package list shows exactly what ships, and --package-suffixes widens it. A weak answer says "rebuild the image" without distinguishing the two, or without mentioning the code package at all.

"The task is stuck and never starts. What is happening?"

The task is Pending on Kubernetes or RUNNABLE on AWS Batch. Nothing in the flow has failed, because the clock for @timeout has not started. The usual cause is that @resources asks for more CPU, memory or GPU than any node or compute environment can provide, or the node selector or tolerations match no node. You find out with kubectl describe pod (events show the unschedulable reason) or the Batch job's status reason. The fix is to right-size the request, add a node pool, or correct the selectors. A good answer adds prevention: alert on tasks Pending longer than some threshold, since Metaflow itself will wait indefinitely.

"A remote step dies with exit code 137. What does that tell you?"

Exit 137 is 128 plus signal 9, a kill, and on Kubernetes almost always OOMKilled: the container exceeded its memory limit. The answer is to raise memory= in @resources (in megabytes) after measuring actual usage, and then think about why usage was high: a DataFrame loaded fully when it could be chunked, artifacts carried through the step unnecessarily, or a foreach where each task loads the whole dataset. Adding @retry does not help a deterministic OOM, since every attempt fails the same way; it helps only for transient failures. Strong candidates also mention that a larger request can leave the task Pending, so the fix has to match what the cluster can offer.

"The pipeline is slow. What do you check first?"

Start by asking where the time goes, since the answer is usually not the model. Look at per-task timing in the UI or logs and sort into three buckets. Startup and scheduling: many tiny tasks, each paying pod scheduling and package download, are fixed by merging steps. Data movement: large artifacts serialised and uploaded between steps are fixed by passing references and using metaflow.S3 for bulk transfer, which is parallel. Compute: one step dominates, so give it more resources, parallelise inside it, or fan it out with a foreach. A weak answer jumps straight to "use a bigger machine". A strong one also mentions environment resolution on first run of a @pypi step, which is cached afterwards.

"A run failed at step 7 of 9 after four hours. What do you do?"

Read the failure first (python flow.py logs <pathspec>), fix the cause, and then run python flow.py resume. Resume clones every successful task before the failed step, reuses their artifacts and executes only from the failure onward, using your current code for those steps. Mention the two subtleties: resume reuses the original parameters and configs, so if the fix needs a different parameter you must start a new run; and the resumed run has a new run ID, with the failed run preserved. Then address the recurrence: should step 7 have @retry if the failure was transient, @timeout if it hung, more memory if it was killed? An answer that stops at "resume" misses the prevention half.

"You have a foreach over 2,000 items and it fails immediately. Why?"

The default --max-num-splits is 100, so a foreach with more items fails at the split with a message that it yielded more children than the current maximum. The limit is a guardrail against accidentally launching thousands of cloud jobs. If 2,000 tasks is intended, raise it explicitly and think about concurrency: --max-workers is 16 locally and 100 per foreach on orchestrators, so 2,000 items is about twenty waves. Also consider whether 2,000 tiny tasks is the right shape; batching items into chunks of fifty and looping inside each task often costs less.

"Two people deploy the same flow and the second gets an error. Explain."

The second person sees "Incorrect production token". The first argo-workflows create generated a production token tied to that flow name, and a different user must pass --authorize TOKEN to redeploy. The mechanism prevents accidental overwrites. Strong answers then give the real fix: nobody should be deploying production from a laptop. Use @project so each developer deploys to their own branch, let CI own --production, and treat the token as an accident guard rather than a credential, since the docs are clear it is not a security boundary.

"Your flow's name is long and deployment fails after upgrading Metaflow. Why?"

Since 2.17, Argo workflow names are limited to 45 characters because Kubernetes labels are capped at 63. With @project, long names are truncated and hashed automatically. Without it, create fails and asks you to shorten the class name or pass argo-workflows --name <name> create. The release notes add a trap: deploying the same flow with an older Metaflow can create two separate deployments. So the fix includes aligning every deployer and CI image on a current version. Note that --name cannot be combined with @project, where you use --branch.

"You changed a parameter and resumed, but the old value was used. Is that a bug?"

No, it is by design. Resume recreates the run from the original one, reusing its parameters and configs, because the cloned steps were computed with those values and mixing them would produce an inconsistent run. To use a new value, start a fresh run. This question checks whether you understand resume as "continue this run" rather than "rerun with my changes".

"The Client API says my flow does not exist, but I just ran it."

Two candidates, in order. First, the metadata provider: a local run writes to .metaflow/ in the directory where it ran, and a service-backed configuration reads from the metadata service. If your notebook uses a different profile or directory than the run, the flow is invisible. metaflow status and get_metadata() show which you are using, and METAFLOW_PROFILE selects a profile. Second, for "Object not in the current namespace": the Client API filters to your user by default, so a run from CI or a colleague is hidden until you call namespace(None). The two errors have different headlines, so read them: "Object not found" versus "Object not in the current namespace".

Trade-off questions

Parameter, Config or artifact?

Use the docs' rule. A Config changes how a deployment is built: decorator values, package lists, schedules, production settings. It is evaluated at deploy time and persisted with the deployment. A Parameter changes what a run does: a date, a learning rate, a customer ID, something the user, UI or event payload supplies at run start. Artifacts carry everything else. A Parameter cannot configure a decorator, because it is unknown when the decorator is applied. A common wrong answer is "Config is just a file Parameter". The better answer is about when the value is fixed. Bonus: never put credentials in either, because both are persisted.

@pypi, @conda, uv or a custom image?

Images are the most predictable for GPU and system-library-heavy work and fit an existing registry and vulnerability scanning process, at the cost of an image pipeline and slower iteration. @pypi gives each step its own pinned environment with no image build, cached in the datastore, which suits pure-Python steps with differing needs. @conda is for packages that need non-Python binaries from conda channels. uv suits a project already managed by it with a lockfile. A good answer states the criteria (reproducibility, GPU needs, iteration speed, security scanning) rather than a favourite. All of them need the discipline of pinning versions.

Kubernetes or AWS Batch?

If you already run Kubernetes, @kubernetes avoids a second system and works on any cloud or on-premises cluster. Batch suits AWS-only teams who want managed compute without operating a cluster; it bills per second of EC2 with no additional fee. The orchestrators matter more than the compute: Step Functions is AWS-only and lacks conditionals, recursion and event triggering, while Argo Workflows works on any Kubernetes and supports them. Also mention the regional angle where relevant: for Gulf or Egyptian employers with residency requirements, the datastore and cluster regions are the deciding factors, and both backends can meet them.

Argo Workflows, Step Functions, Airflow or Kubeflow as the orchestrator?

Argo Workflows has the widest Metaflow feature support, including events through Argo Events and exit hooks, and runs anywhere Kubernetes does. Step Functions is the natural choice on AWS without Kubernetes, with limitations on conditionals and recursion. Airflow fits an organisation that already standardises on it, but supports only @kubernetes, has no nested foreach or event triggering, and cannot be driven by the Deployer API. Kubeflow is a separate extension (metaflow-kubeflow). The senior-quality answer is "match the platform your organisation already operates, and check the feature matrix before designing the graph".

Catch failures or let them fail?

@catch turns a failing step into stored data, so the flow continues. It is right for partial-failure tolerance, like "forecast the other countries if one breaks", and wrong when it hides a failure that invalidates the result. A strong answer bounds it: assert on the number of failures in the join, and alert when any occur. It also notes where @catch cannot go, on the step that fans out or with @parallel on Kubernetes.

One large step or many small ones?

Steps are checkpoints with real overhead: serialisation, datastore round trips and, remotely, pod scheduling. Too many tiny steps waste time and money. Too few large ones lose more work on failure and make resume coarse. The docs' guideline is under an hour, preferably much less. The decision rule in practice: split where you want a restart point, where resources change (CPU preprocessing versus GPU training), or where you can fan out. Merge trivial glue steps.

Metaflow's tags and namespaces or a real model registry?

Tags give cheap promotion: tag a run approved and have consumers query for that tag. Namespaces keep experiments separate by user. Neither gives stage transitions, lineage or access control as first-class features, so a team with compliance needs or many models belongs on a registry such as MLflow. Metaflow and a tracker coexist well: the tracker runs inside steps.

What a strong answer covers that a weak one misses

Five habits separate the two:

• They localise before they fix. Linter error, ModuleNotFoundError, exit 137, Pending and a service error each point to a different layer.
• They know the defaults and limits: 16 local workers, 100 splits, 100 workers on orchestrators, @retry(times=4) as a ceiling, a five-day remote runtime limit, and a 45-character Argo name.
• They know which statements are guarantees. Steps are checkpoints; namespaces are not security; production tokens are not authentication.
• They keep the environment reproducible, with pinned dependencies and a pinned Metaflow version.
• They describe prevention as well as the fix: timeouts, alerts, idempotent steps and CI deployment.

Quick commands to have ready

python flow.py show                                   # print the DAG
python flow.py package list                           # what ships to remote tasks
python flow.py --environment=pypi run                 # top-level option before the command
python flow.py run --with kubernetes --max-workers 50
python flow.py resume                                 # continue from the failed step
python flow.py logs 12/train/345 --stderr --timestamps
python flow.py dump 12/train/345 --include metrics
python flow.py tag add --run-id 12 approved
python flow.py spin train                             # one step, inputs from the latest run
python flow.py --production argo-workflows create
python flow.py argo-workflows trigger --alpha 0.5
metaflow status                                       # which metadata provider am I using

Be ready to explain each line and what goes wrong if it is changed, such as moving --environment=pypi after run.

Design prompts at this level

"Design a nightly training pipeline for a model that uses a GPU for one hour."

A good answer has a shape. One step loads data (CPU, using S3 or a warehouse query, passing only a reference downstream), one step trains with @resources(gpu=1, ...) and a timeout, and one step evaluates and gates on a metric with a conditional or an assertion. Dependencies are pinned, either in an image or with @pypi. The job is deployed with @project and @schedule to Argo Workflows by CI, with failure notifications through --notify-on-error. The model artifact is tagged after evaluation, and batch scoring loads a pinned run rather than latest_successful_run. Mention spot instances and @checkpoint only as an optimisation, and mention the regional placement of buckets and cluster.

"How do you test a flow before it reaches production?"

Layers. The linter and show catch graph errors instantly. A Runner-driven pytest test runs a small data sample locally. spin accelerates iteration on one step. A pull-request branch deployment with --branch runs on the real cluster using a test profile and is deleted afterwards. Only the main-branch pipeline runs --production. The key claim: the same flow file runs in every layer, so the tests exercise the real code.

"Your team wants to run flows on a schedule. What do you need to set up?"

A cluster with Argo Workflows (and Argo Events for triggers), a cloud datastore, the metadata service if you want the UI, a service account for tasks, and a CI job that runs create. In the flow, @schedule with an explicit timezone, because a bare @schedule is daily. Explain that nothing runs on Argo without @schedule or @trigger, that the deployment is a snapshot, and that you need alerting for failed scheduled runs.

Sixty-second self-test

Answer each without looking. If you hesitate, reread that section.

1. What process model does a run use, and what is the consequence for sharing variables between steps?
2. Why does @pypi require --environment=pypi, and where does the flag go?
3. What does resume reuse, and what does it recompute?
4. What are the default values of --max-workers and --max-num-splits, and what happens when a foreach exceeds the second?
5. A task stays Pending. A task exits with 137. Name the cause and first action for each.
6. When is a Config evaluated, and why can it configure decorators when a Parameter cannot?
7. Which orchestrators support conditionals, and which support @trigger?
8. What does @project give you, and what is --production?
9. Why is a namespace not a security control?
10. What is the highest @retry(times=...), and what should you do if you need more patience?

Answers in one line each: separate processes, so only self artifacts survive; the decorator needs an environment builder, and it is a top-level option before run; it reuses the original parameters and successful tasks and re-executes from the failure with current code; 16 and 100 (a foreach over the second fails unless you raise it); Pending means the request cannot be scheduled so right-size it, 137 means out of memory so raise memory=; at deploy time, which is why decorators can use it; Argo Workflows and local runs for conditionals, Argo only for triggers; isolated user, branch and production deployments, and the production deployment; it filters results and is not access control; four, and raise minutes_between_retries.

Where to go next

If these answers felt natural, continue to the senior interview material on platform architecture, scaling limits and security. If any felt shaky, revisit the matching section of the Mid-level guide. Also pair this with the Kubernetes guide for scheduling and resource questions, and the Airflow guide if your organisation orchestrates there.

Part two of three. At this level your flows run and you have probably deployed one, so the problems change character. They are no longer "why is there a syntax error" but "why did this pass locally and fail on the cluster", "why is this run slow", and "why did production run old code". Start with the error table, then the practices ranked by leverage, the practice cards and the operational recipes.

Common errors at this level

Cumulative: Beginner's errors still apply. These appear once things work.

Three of these deserve more than a line, because they hide behind plausible explanations.

The "deployed but never runs" trap. A flow on Argo Workflows runs only when triggered, or when it carries @schedule, @trigger or @trigger_on_finish. Teams deploy a flow, see a WorkflowTemplate, and assume it is live. Nothing alerts you, because a workflow that never starts never fails. Add a freshness check on the output, such as "the forecast table was updated within 26 hours", and alert on that.

The "same flow, two deployments" trap. After the 2.17 naming change, deploying a long-named flow with an older Metaflow and then a newer one produces two separate deployments. Both may be scheduled. If you see duplicate runs, list deployments with argo-workflows list-workflow-templates and delete the stale one with the version that created it.

The practices that pay off most, in order of leverage

Expanding the first four, since they prevent most incidents:

Pin everything. For dependencies, write exact versions: @pypi(python="3.11.9", packages={"pandas": "2.2.3"}). For Metaflow, put the version in your lockfile. Because remote tasks run the Metaflow version of the machine that launched them, an unpinned CI image means tomorrow's run may use a different Metaflow than today's. Update deliberately, read the release notes on GitHub (the project states that it takes backward compatibility very seriously, and lists the rare exceptions there), and re-run create afterwards.

Deploy from CI. The Deployer API or the CLI in a pipeline gives you an audit trail and a clean environment. It also avoids the production-token conflict, since one identity deploys. Keep the pipeline short: install pinned requirements, run a Runner-driven test, deploy a branch for pull requests, deploy production on merge.

Isolate with @project. Without it, the second developer to run create either conflicts with the first or, worse, overwrites the first's deployment. With it, python flow.py argo-workflows create deploys to your user branch, --branch pr_482 to a test branch and --production to production. Inside the flow, current.is_production lets you choose the real table or a test one. Pair it with @trigger_on_finish, which matches within the same project and branch, so a test branch of the upstream flow only triggers the test branch of the downstream flow.

Time out and size every remote step. Explicit @timeout and @resources are documentation as much as protection: a reviewer reading the decorators sees what a step is expected to cost. Start from measured usage plus headroom, not guesses, and revisit after a month of runs.

Habits for remote work

• Run a cheap remote smoke test first. Before a large foreach, run the flow with a two-item list against the real cluster. It exercises the datastore, image pull, secrets and permissions for almost no cost.
• Cap fan-out before it caps you. Set --max-workers and --max-num-splits deliberately, and batch tiny items into chunks.
• Keep the node-selection story explicit. GPU steps need gpu= plus matching tolerations and node selectors. A step that sets gpu=0 is forced onto non-GPU nodes, which is a useful way to keep CPU work off expensive machines.
• Tag cost. Use labels and annotations on @kubernetes, or aws_batch_tags on @batch, so finance can attribute spend by flow and team.
• Tune job retention. Finished Kubernetes jobs are kept for seven days by default. Shorten METAFLOW_KUBERNETES_JOB_TTL_SECONDS_AFTER_FINISHED on busy clusters.
• Prefer a known-good image per environment. If you use custom images, set METAFLOW_KUBERNETES_CONTAINER_IMAGE in each profile, so dev and production do not drift.

For the cluster side of these habits, see the Kubernetes guide, and for provisioning buckets, clusters and IAM as code, the Terraform guide.

Practice cards

1. Find what shipsRun python flow.py package list on a real flow. Add a JSON file next to it and confirm it is missing. Then add --package-suffixes=.py,.json before package and confirm it appears.
2. Prove the boundary costAssign a large array to self in one step and time the next step's first line. Replace it with a path written to local disk or S3 and compare.
3. Break a join on purposeSet the same artifact to different values in two branches and call merge_artifacts(inputs). Read the error, then fix it with exclude.
4. Trigger the split limitMake a foreach over 150 items, read the error, then rerun with --max-num-splits 200.
5. Practise resumeMake the last step fail, fix it, and run resume. Confirm earlier steps are cloned. Change a parameter and resume again to see the original used.
6. Read a Pending taskRequest more memory than your node pool has and use kubectl describe pod to read why it never schedules.
7. Move a value into a ConfigTake a timeout or memory size out of a decorator literal into config.json and run with --config config other.json.
8. Write a Runner testUse Runner("flow.py").run() inside pytest and assert on running.run.successful and one artifact.
9. Deploy two branchesAdd @project, deploy twice with different --branch values, and compare names in list-workflow-templates.

Operational recipes

Recipe: run a flow on a cluster for the first time

1. Install the client dependencies: pip install metaflow kubernetes.
2. Run metaflow configure kubernetes and confirm the datastore is cloud storage with metaflow configure show.
3. Run a one-step flow: python flow.py run --with kubernetes.
4. Read the pod events if it does not start: kubectl describe pod <name>.
5. Only then add resources, dependencies and secrets, one at a time.

Recipe: triage a failed production run

Recipe: upgrade Metaflow safely

1. Read the GitHub release notes for every version between yours and the target, looking for breaking changes (the 2.17 naming change is the model example).
2. Upgrade in a branch: pip install --upgrade metaflow, and update the pin.
3. Run the Runner-based tests and a branch deployment on a test profile.
4. Upgrade CI and every person who deploys, so nobody deploys with an older version.
5. Re-run argo-workflows create for production flows, because existing deployments keep the snapshotted version.
6. If you use spin or newer UI features, check whether the metadata service needs upgrading too.

Recipe: rotate a secret without touching code

1. Update the secret's value in your secrets manager (AWS Secrets Manager, Azure Key Vault or GCP Secret Manager).
2. Because @secrets reads the value when the task starts, the next run picks it up. No redeployment is needed unless you changed the source name.
3. If you changed the name, update the sources list, and re-deploy.
4. Confirm IAM still restricts the secret to the orchestrator's role.

Recipe: promote a model by tag

python train_flow.py run --tag candidate
python train_flow.py tag add --run-id 57 approved

from metaflow import Flow, Run

approved = list(Flow("TrainFlow").runs("approved"))
model = Run(f"TrainFlow/{approved[0].id}").data.model

Keep the run ID in your scoring flow's config so promotion is an explicit, reviewed change, not whatever ran last.

Recipe: cap the blast radius of a big foreach

python flow.py run --max-workers 25 --max-num-splits 500

Combine with @timeout and @retry(times=2) on the inner step. Put @catch on the inner step only, and assert in the join that failures stay under your threshold.

Habits worth building

• Read the headline, then the message. Metaflow prints a headline such as "Incompatible environment" or "Object not in the current namespace", then details. The headline tells you the layer.
• Run python flow.py show and check after graph edits. They cost nothing and catch joins and transitions immediately.
• Import inside steps for anything declared by @pypi or @conda.
• Keep flows thin. Put business logic in a normal Python package with unit tests; the flow wires steps and decorators. Logic is then testable without running Metaflow, and the @package_sources mutator (2.19.39) can ship a sibling package.
• Write the end step for consumers. run.data exposes the end step's artifacts, so assign there what downstream readers need.
• Design for re-execution. Any step may run twice, so prefer overwrite-by-key to append.
• Watch the release notes. Metaflow ships frequent patch releases in the 2.19 line, and some, like the 2.19.23 entry and exit step naming, change what you can write.

Reading a flow review like a reviewer

When you review someone else's flow, read it in a fixed order, because the expensive mistakes cluster in a few places. Start with the decorators on each step, since they state the cost, the failure policy and the secrets in one glance: does every remote step have resources and a timeout, is every retried step safe to repeat, and does anything catch exceptions without a later check? Then read the transitions, looking for foreach sizes, joins that silently drop branches, and conditionals that assume an orchestrator supports them. Next read what is assigned to self, asking whether each artifact is small enough to live in a bucket and free of personal data or credentials. Finally read how the flow is deployed: which environment flag, which project and branch, and whether a schedule or trigger exists at all.

A useful question for every pull request is "what happens if this step runs twice, or not at all?" Retries, resume, and a missed schedule are all normal events in production, and a flow that only works on the first, uninterrupted execution is a flow that will eventually page someone. Ask it of the data writes first, then of the notifications, then of anything that calls an external API with side effects.

A short checklist

Before you merge a flow to main:

• Metaflow and all dependencies are pinned.
• Every remote step has @resources and @timeout.
• Retried steps are safe to run twice.
• No credentials in Parameters, Configs, artifacts or code; @secrets used instead.
• Imports for @pypi packages are inside the steps.
• python flow.py package list includes everything the flow needs.
• Foreach size and --max-num-splits are deliberate.
• @project is set; current.is_production guards real-data writes.
• @schedule or @trigger* exists if the flow should run by itself, with an explicit timezone.
• A Runner-based test passes in CI, and deploys come from CI.
• Someone is alerted when the scheduled output is stale.

Where to go next

The next level treats Metaflow as a platform: architecture and failure modes, scaling limits, the trust model given that open-source Metaflow has no authentication of its own, multi-tenancy, upgrades and when another tool is the better answer. If your flows feed a tracker or a model registry, read the MLflow guide next. If they publish features for serving, the Feast guide is the natural pair.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further and adding the work you own when Metaflow is not your tool but your team's platform, used by people who will never read its source and who will blame you when a Friday-night run disappears.

Up to this point the questions were about making flows work. From here they change: what happens when the metadata service is down, who can read another team's artifacts, which component saturates first when forty people launch foreaches at once, how you upgrade a library that is packaged into every running task, and when the honest answer is that Metaflow is the wrong tool for the job.

The version this guide is verified against is Metaflow 2.19.39, released on PyPI on 2026-09-02. The 2.19 line has had many patch releases and no 2.20 yet, so version numbers below matter: a few behaviours changed in 2.17, 2.18 and 2.19, and as the platform owner you are the person who has to know which.

Where this picks up

I am starting with architecture, because every operational decision in this track is a decision about one of its parts.

The architecture you operate

Metaflow is a client-side Python library plus a small set of pluggable services. That sentence contains the most important design fact for an owner: there is no central Metaflow server that executes your flows. The code that runs a step is the same library on the user's laptop, in a container on Kubernetes, or on an AWS Batch instance. What you operate is everything around the library.

The datastore is the source of truth for data. It holds artifacts (content-addressed and deduplicated by hash within a flow), the code package for every run, task logs and cards. The metadata service is the source of truth for structure: which runs exist, which tasks belong to them, which attempt is current, which tags are attached. It is a Python service backed by Postgres, and the UI and its backend service read from it. The compute layer is Kubernetes or AWS Batch. The orchestrator is what turns "run this flow every morning" into scheduled executions.

Follow one remote task to see how the parts meet. The orchestrator (or the local runtime) submits a job. The container starts, downloads the code package from the datastore, and runs python flow.py step <name> .... It reads its input artifacts from the datastore, executes your code, writes its outputs back to the datastore, and registers the task and its attempt with the metadata provider. Nothing flows directly from one task to the next. This is why steps can run on different machines, why resume can reuse the outputs of a past run, and also why every step pays a serialization and transfer cost.

Two consequences matter for ownership. First, the metadata provider and the datastore are independent settings. DEFAULT_METADATA can be local while the datastore is s3, which produces the confusing state where runs succeed and write to the bucket but nobody else can see them in the UI. Second, the user's Python environment and the platform's configuration meet in the client. A user with a wrong profile, a stale version or a missing kubernetes package can break a deployment that is otherwise healthy, so a large share of "platform incidents" are client configuration problems.

The infrastructure itself is deployed with Terraform, CloudFormation or Helm templates from the outerbounds/metaflow-tools repository, with guides in the Outerbounds engineering documentation. Those templates are a reasonable starting point, but they are templates: once you own the platform, the resulting Terraform state, the bucket policies and the Helm values are yours, and they should live in the same review process as the rest of your infrastructure (see the Terraform and Helm guides).

Failure modes, component by component

The most useful thing you can do as platform owner is to know in advance how each component fails and what the user sees. Metaflow's failure handling splits cleanly along the component boundaries.

The datastore is unavailable or throttled. Tasks fail while reading inputs or writing outputs. S3 requests are retried by Metaflow's client (S3_RETRY_COUNT defaults to 7 and S3_TRANSIENT_RETRY_COUNT to 20), so brief blips are absorbed, but a sustained outage or a throttling storm surfaces as failed tasks. The fix is usually outside Metaflow: a bucket policy change, an expired credential on the node role, a VPC endpoint that was removed. If you use an S3-compatible store, S3_ENDPOINT_URL and S3_VERIFY_CERTIFICATE are the first settings to check after a certificate rotation.

The metadata service is unavailable. This one is subtle because the service is not on the data path of a task's computation. The Client API and the UI stop working, and run registration can fail, which fails the run. SERVICE_RETRY_COUNT defaults to 5. Users see Metaflow service [url] unreachable. or a ServiceException. If the service sits behind an authentication proxy, an expired token gives the same symptom, and the headers it needs are configured through SERVICE_HEADERS or SERVICE_AUTH_KEY. Treat the service like any other internal API: health checks, autoscaling, a database with backups, and an alert that fires before users notice.

The compute layer cannot place a task. On Kubernetes a task stays Pending; on AWS Batch a job stays RUNNABLE. The docs state it plainly: if a large enough instance is not available, the task won't start. The typical causes are a @resources request bigger than the largest node, a missing GPU node pool, an exhausted quota or taints that no toleration matches. This is a capacity problem you own, not a bug in the flow. Exit code 137 or an OOMKilled status is the opposite failure: the task started and used more memory than it asked for.

The orchestrator loses or never starts a run. Flows deployed to Argo Workflows do not run by themselves unless they carry @schedule or a @trigger decorator. For event-triggered flows, Argo Events has to be configured, and the error An Argo Event name hasn't been configured for your deployment yet points at the missing METAFLOW_ARGO_EVENTS_* keys. A schedule that silently does not fire is almost always a deployment that was never re-created after an edit.

The user's client is wrong. Missing the kubernetes package, a stale profile (Unable to locate METAFLOW_PROFILE), no USER in a CI container (Unknown user, fixed by METAFLOW_USER), or a Metaflow older than 2.17 redeploying a flow whose name exceeds the Argo limit. These are the cheapest incidents to prevent and the most common ones to handle, which is why the platform section later in this guide puts so much weight on a pinned, shared client.

Scaling: which component breaks first

Metaflow scales by fan-out, and fan-out is exactly where limits appear. The useful exercise is to walk the path of a large foreach and note which ceiling it hits first.

Locally, the runtime starts each task as a separate process, limited by --max-workers (default 16), and refuses a foreach that yields more than --max-num-splits children (default 100). The docs say the local runtime is not intended as a production-grade scheduler, and nothing in this section changes that. On orchestrated deployments, --max-workers on create defaults to 100 and caps concurrent tasks per foreach. On Step Functions, --use-distributed-map exists for very large foreaches.

For a 5,000-way foreach on Argo, the order in which things break is usually this.

1. The split cap. The flow fails before it starts fanning out, with Foreach in step ... yielded N child steps which is more than the current maximum of 100 children. Raising --max-num-splits is a deliberate decision, not a fix, because it also raises your worst-case bill.
2. Cluster capacity. One hundred concurrent tasks each requesting 4 CPUs and 16 GB need a cluster that can place 400 CPUs at once. If the autoscaler is slow, tasks sit Pending and the run takes hours longer than the compute time suggests. Watch the time between task creation and start.
3. Datastore request rate. One hundred tasks starting simultaneously each download the code package and their input artifacts. Object storage is forgiving but not infinite, and small artifacts multiplied by thousands of tasks create more requests than bytes. Metaflow's S3 client is parallel (S3_WORKER_COUNT defaults to 64), which helps bulk transfers and can also produce bursts.
4. The metadata service and its database. Each task registers itself and its attempts, and the UI queries the same tables. A very wide foreach produces a surge of writes and heavier UI queries. This is the component most likely to be undersized, because it looked idle during the pilot.
5. The orchestrator. Argo Workflows stores workflow state in Kubernetes objects. Very large workflows stress the Kubernetes API server and etcd. Keep foreaches bounded, and prefer a few large tasks that each process a batch over thousands of tiny tasks, since every task has fixed overhead for container start, package download and artifact round trips.

Two more limits belong on your chart. A remote task has an implicit ceiling of five days (DEFAULT_RUNTIME_LIMIT), and @timeout overrides it per step; set a sensible platform default through DEFAULT_DECOSPECS or review rules rather than relying on the five-day ceiling to catch runaways. And on Kubernetes, finished Jobs are kept for KUBERNETES_JOB_TTL_SECONDS_AFTER_FINISHED (seven days by default), which is useful for debugging and expensive for the API server if you launch a very large number of tasks.

The security model, stated honestly

The most important sentence in this section is one the documentation itself supplies: open-source Metaflow has no authentication or role-based access control of its own. Namespaces, the feature that looks most like access control, are documented as not being about security or access control. They filter what the Client API returns by default, and any user can switch with namespace(None). Production tokens prevent accidental redeploys by the wrong person; they are not a security boundary.

So where does real access control come from? From the infrastructure under Metaflow:

• Cloud IAM on the bucket. Whoever can read the datastore bucket can read every artifact in it, including every model, every DataFrame and every Parameter value. If two teams must not see each other's data, they need separate buckets or at least separate prefixes with IAM conditions, which means separate DATASTORE_SYSROOT_* values per team or environment.
• Kubernetes RBAC and service accounts. What a task can reach is what its service account can reach. KUBERNETES_SERVICE_ACCOUNT or @kubernetes(service_account=...) ties a task to an identity, which with IRSA on AWS or Workload Identity on GCP becomes cloud permissions.
• The orchestrator's permissions. Whoever can create Argo Workflows templates in a namespace can run arbitrary containers there. Restrict who holds that permission, and think of CI as the only identity that deploys production flows.
• Network controls around the metadata service. It has no per-user authorization in the open-source version. It supports SERVICE_AUTH_KEY and custom SERVICE_HEADERS, so it can sit behind an authenticating proxy that your organisation already operates. The details of hardened deployments are in the Outerbounds engineering docs rather than the core docs.

Now the trust model. A Metaflow user can run arbitrary code, because a flow is arbitrary Python. The code executes in a container you configured, with whatever identity you attached. The real question is what that container can do, and the answer is set by the pod spec. @kubernetes(security_context={...}) supports keys such as privileged, allow_privilege_escalation and run_as_user, which means a user can ask for elevated settings unless admission policy stops them. Enforce the limits where they cannot be bypassed: a Kubernetes admission controller such as Kyverno or OPA Gatekeeper that refuses privileged pods in the Metaflow namespaces, network policies that block access to internal services the tasks have no reason to reach, and IAM roles with no more access than the team's data needs.

The supply chain is the second half. Metaflow packages itself into every code package, and @pypi/@conda resolve third-party dependencies at build time. Everything a flow imports therefore runs with the task's identity. Pin versions, prefer an internal package index or mirror where your compliance requires it, scan the base images that @kubernetes(image=...) points at, and keep Metaflow current: 2.19.31 replaced yaml.load with yaml.safe_load, which is exactly the kind of fix you do not want to lag behind. The Trivy and Docker guides cover image scanning and hardening in depth.

Secrets without leaks

The first rule has a Metaflow-specific edge: never pass credentials as Parameters. A Parameter is persisted as an artifact, stored in the datastore, visible to anyone who can read the run and printed in places you did not intend. The same is true of any self.x that holds a token, because artifacts are persisted at the end of every task.

The supported mechanism is @secrets(sources=[...]), which resolves named secrets from AWS Secrets Manager, Azure Key Vault or GCP Secret Manager and injects the key/value pairs as environment variables in the task. The default backend is set by DEFAULT_SECRETS_BACKEND_TYPE (aws-secrets-manager, az-key-vault, gcp-secret-manager or inline), and DEFAULT_SECRETS_ROLE or the decorator's role= lets a task assume a different IAM role when reading. Full ARNs or Key Vault URLs reach across regions or vaults. Azure Key Vault supports Secret objects only, and on GCP the identity needs the Secret Manager Secret Accessor role.

from metaflow import FlowSpec, step, secrets, kubernetes
import os

class PaymentsFlow(FlowSpec):

    @secrets(sources=["prod-warehouse-credentials"])
    @kubernetes(cpu=2, memory=8192)
    @step
    def start(self):
        # Values arrive as environment variables, never as artifacts.
        dsn = os.environ["WAREHOUSE_DSN"]
        self.rows_loaded = load(dsn)   # store the count, not the credential
        self.next(self.end)

    @step
    def end(self):
        print(self.rows_loaded)

if __name__ == "__main__":
    PaymentsFlow()

For the platform, the design question is how the source changes per environment without editing flows. The docs support setting sources at deploy time with --with 'secrets:sources=["prod-db-credentials"]', or through a Config, so that the development deployment reads one secret and production another. The enforcement, again, is IAM: only the production orchestrator's role can read production secrets, and a developer's laptop role cannot. If a flow named its production secret in code but the laptop role cannot read it, that is the system working.

Multi-tenancy: isolating teams on one platform

Multi-tenancy in Metaflow is layered, and each layer isolates something different. Confusing the layers is how platforms end up with friendly isolation of results and no isolation of data.

For most organisations the right starting point is one platform, one metadata service, per-team Kubernetes namespaces and service accounts, per-team datastore prefixes or buckets with IAM to match, and @project on every production flow. Move to separate metadata services only when a regulatory boundary requires that one team's run metadata, including flow names and parameter values, is invisible to another. For employers in the Gulf and Egypt who face data-residency expectations, the datastore's region is the first thing to settle: an S3 bucket, Azure Blob container or GCS bucket in a Middle East region, and compute placed in the same region so that artifacts do not cross a border on every step.

@project deserves care because it is the feature that makes shared deployment safe. With @project(name='forecasting'), a developer running python flow.py --branch feat argo-workflows create gets forecasting.test.feat.<Flow>, while CI running --production gets forecasting.prod.<Flow>. Nothing overwrites anything, each branch has its own namespace for results, and the flow knows where it is through current.project_name, current.branch_name and current.is_production. Two constraints to teach: argo-workflows --name cannot be combined with @project (the error says to use --branch), and @trigger_on_finish matches flows inside the same project and branch, so a test branch does not trigger production downstream flows.

Deployment as a pipeline, not a command

At Beginner level, deploying is argo-workflows create. At platform level, it is a pipeline with an identity, a version and an audit trail. The rule that holds it together is that production is deployed by CI, not by people.

Why this matters is written into Metaflow's own model. The first create of a flow generates a production token, and another user redeploying the same flow needs that token (--authorize TOKEN) or gets Incorrect production token. People respond to that error by sharing the token in chat, which defeats its purpose. With a CI identity as the only deployer, the token lives with the pipeline, and developers deploy only to their own branch through @project.

The Deployer API makes this scriptable from Python, which is useful when the pipeline needs to do more than one command, such as deploy, trigger a smoke run, wait for it and roll back on failure:

from metaflow import Deployer

def deploy_and_smoke_test(flow_file: str) -> None:
    deployed = Deployer(flow_file).argo_workflows().create()
    triggered = deployed.trigger(alpha=0.5)   # a parameter of the flow
    run = triggered.wait_for_run(timeout=1800)
    if not run.successful:
        deployed.delete()                      # or re-create the previous tag
        raise SystemExit(f"smoke run {run.pathspec} failed")

deploy_and_smoke_test("flows/forecast_flow.py")

The same pattern appears in CI as plain commands, with top-level options before the subcommand:

python flows/forecast_flow.py --environment=pypi --production argo-workflows create \
  --tag release:2026-10-01 --tag git:$GITHUB_SHA

The --tag options matter for governance: a run launched from that deployment carries the tags, so that a run ID can be traced to a commit. System tags such as user: and runtime: and metaflow_version: are immutable, while user tags can be changed later, so put provenance you want to trust in a place you control, like the deployment's own tags at create time and your CI's log.

Three things make a Metaflow deployment pipeline trustworthy.

One client version everywhere. The deploying job's Metaflow version is baked into the deployed template, and existing deployments keep the snapshotted version until you re-create them. The 2.17 Argo naming change is the cautionary tale: deploying the same flow with an older version after the change produces two separate deployments when the name exceeded the limit. Pin metaflow==2.19.39 (or whichever you run) in the CI image and in the shared platform package, and make developer machines warn when they drift.

Names that fit. Argo workflow names are limited to 45 characters in current source, because Kubernetes labels are limited to 63. With @project, long names are truncated and hashed automatically. Without @project, an oversize name is an error, with the fix argo-workflows --name <name> create or a shorter class name. Decide a naming scheme once, and lint it in CI before anyone deploys, because renaming a deployed flow creates a new deployment and orphans the old schedule.

A schedule is explicit. Flows on Argo do not run unless they have @schedule or @trigger*. The legacy singular schedule field is still emitted by default; setting METAFLOW_ARGO_WORKFLOWS_USE_SCHEDULES=true emits schedules and requires Argo Workflows 3.6 or later. Check your Argo version before you flip that setting, and note that timezone in @schedule works on Argo only, with Etc/UTC as the default, which is a frequent cause of "it ran an hour late" reports around daylight-saving changes in regions that observe them.

Choosing and operating the orchestrator

Metaflow is deliberately not an orchestrator. It authors flows and compiles them to one. The choice of target is the largest platform decision you make, and the fact sheet's feature matrix should sit on your design document's first page, because the orchestrators are not interchangeable.

Two observations. Argo Workflows is where the newest Metaflow features land first and the only target that supports event triggering, conditionals, recursion, exit hooks and custom notifications, so it is the natural default for a platform that is not already committed to something else. It is also the target that puts Kubernetes expertise on your team's critical path. Step Functions is a good fit when you are all-in on AWS and want a managed control plane, at the cost of losing events and the newer control-flow features. Airflow and Kubeflow are integration targets: choose them when your organisation already runs them and wants Metaflow authoring on top, and accept the feature gaps, which your authoring guidelines then need to encode so that users do not write a conditional flow that cannot deploy.

Operating Argo for Metaflow has a few recurring concerns. The Argo UI is where run logs and failures are visible to operators, and ARGO_WORKFLOWS_UI_URL points Metaflow's UI at it. Notifications are built in: --notify-on-error and --notify-on-success with Slack, PagerDuty or incident.io options, and @exit_hook(on_error=[...], on_success=[...]) for custom behaviour. Each exit hook runs as a separate pod, cannot create artifacts and cannot use @pypi or @conda, so they need an image that already contains what they import. suspend, unsuspend, status and terminate on the argo-workflows group give operators control of a live run by its argo-<name> run ID.

Finally, watch for trigger loops. @trigger_on_finish(flow='A') on flow B and the reverse on flow A is an infinite loop that costs real money before anybody notices. Treat the trigger graph as architecture, draw it, and review it. The Argo CD and Kubernetes guides cover the GitOps and cluster side of keeping it healthy.

Dependencies and packaging as shared infrastructure

For a solo user, @pypi is a convenience. For a platform, it is a shared build system with a cache, a trust boundary and a failure mode. You choose among three models, and the choice is the platform's dependency policy.

Custom container images. The team builds images and users reference them with @kubernetes(image=...) or METAFLOW_KUBERNETES_CONTAINER_IMAGE. This is the most predictable model: images are scanned, signed and pinned, and a task starts without resolving anything. The cost is a build-and-publish pipeline and some friction when a user needs one more library.

Resolved environments through --environment=pypi or --environment=conda with @pypi/@conda (and @pypi_base/@conda_base). Metaflow resolves packages, builds the environment and caches it in the datastore, so only the first resolve is slow. This is excellent for flexibility and needs discipline: declare python= and exact versions, import inside the step so that the module is not needed at flow-parse time, and note that --environment=pypi is internally an alias for the conda-based machinery. CONDA_DEPENDENCY_RESOLVER and the CONDA_PACKAGE_*ROOT keys configure where packages are cached. --force-rebuild-environments is your hammer after a bad resolve.

Most mature platforms combine them: a small number of curated base images for heavy, security-sensitive pieces (CUDA, system libraries), and @pypi for the pure-Python layer on top. Whichever you choose, write down which model a production flow is required to use. A production flow that resolves unpinned packages at launch is a flow that can break without a code change.

Cost: the levers and the guardrails

Metaflow itself costs nothing, so everything you spend is compute, storage, the metadata database and the operators' time. The aim is not minimal spend but predictable spend with someone accountable for each line. The levers sit in a clear order of impact, and the team and project labels are what make them enforceable: on Kubernetes @kubernetes(labels=..., annotations=...) and KUBERNETES_LABELS tag pods, and on AWS Batch @batch(aws_batch_tags=...) and BATCH_EMIT_TAGS do the same.

Compute right-sizing is first. @resources and the @kubernetes/@batch arguments are requests, and the defaults (one CPU, 4096 MB on both) are conservative guesses, so a user who copies a training flow's memory=65536 onto a data-validation step pays for it on every run. Teach a measure-then-set habit: run once, read the memory and CPU actually used from the cluster's metrics, then request that plus headroom and add @retry so that one OOM does not kill the run. The QoS class (KUBERNETES_QOS, default burstable) changes how the pod is scheduled and evicted, and Guaranteed is worth it for long jobs you cannot afford to have evicted.

Spot or preemptible capacity is second, and only safe with retries and checkpoints. The file at current.spot_termination_notice appears when the node is being reclaimed, and the @checkpoint extension saves in-task state so a retry resumes rather than restarts. Put long GPU work on spot only after a fault-injection test shows that it resumes.

Guardrails on fan-out are third: --max-num-splits, --max-workers, @timeout and the five-day DEFAULT_RUNTIME_LIMIT. The platform can apply decorators to every run through DEFAULT_DECOSPECS, such as a default timeout, so that absent a deliberate choice a runaway step is stopped.

Upgrades and migrations

Metaflow's maintainers say they take backwards compatibility very seriously, and the record since 2.12 supports them: the notable breaks, such as the 2.17 Argo naming rule and the 2.18 restriction on conditional paths leading directly into joins, are narrow and documented. The platform's upgrade practice should still be deliberate, because Metaflow's self-packaging changes who is exposed to an upgrade.

Metaflow packages itself into every code package. That means remote tasks run the same Metaflow version as the client that launched them. You do not rebuild task images when you upgrade, which is convenient, and it also means a deployment is a snapshot: existing deployments keep the version they were created with until you re-create them. A fix released in 2.19.39 does not reach a flow deployed under 2.19.30 until someone re-runs create. Plan a scheduled re-deploy of production flows through CI after each upgrade, for exactly this reason.

An upgrade runbook follows from the facts.

1. Read the GitHub release notes for every version between yours and the target. Look for notes on the Argo naming rules, conditionals, environment resolution and anything that mentions a required service version.
2. Check the server side. spin (2.19.0) may require a newer metadata service, and Client and UI features depend on the service version, governed by SERVICE_VERSION_CHECK. Upgrade the service before you tell users to use the new features.
3. Upgrade the CI image and the platform package first, then a canary team, then everyone. Keep the pin in requirements.txt or uv.lock so an upgrade is a reviewed diff.
4. After the upgrade, re-create production deployments through the pipeline in a window, run a smoke flow on each target, and keep the previous tag available for rollback.
5. Upgrade extensions separately. metaflow-checkpoint, metaflow-kubeflow and metaflow-netflixext are versioned apart from core, and the checkpoint extension's APIs may change while core's are guaranteed to stay compatible.

Observability and governance

A platform without observability is a support queue. Metaflow gives you several signals, and the owner's job is to join them into something operators can use without reading flow code.

The per-run signals are the ones users already know: the UI (which needs the metadata service), task logs through python flow.py logs or task.stdout and task.stderr in the Client API, cards (which can update live since 2.11) and the Argo UI. The platform-wide signals need setup. Tracing is available through METAFLOW_OTEL_ENDPOINT or ZIPKIN_ENDPOINT, with OTEL_SERVICE_NAME (default metaflow, honoured from an environment variable since 2.19.35) so your traces carry a name your collector can route on; see the OpenTelemetry guide for the collector side. The monitor and event-logger are pluggable sidecars (DEFAULT_MONITOR, DEFAULT_EVENT_LOGGER), a sidecar monitoring option was added in 2.19.36, and step and project context are sent to the monitor since 2.19.30. METAFLOW_RUN_ID is set in the task environment (2.19.36, and in Batch and Kubernetes tasks from 2.19.37), which makes it possible to correlate logs from your own code with the run.

What to alert on is a separate question from what is available. A short, useful set:

• Run failure on production deployments, routed to the owning team by --notify-on-error with PagerDuty, Slack or incident.io options, or an @exit_hook.
• Time to start, meaning the delay between a task being submitted and running. This is the capacity signal, and it moves before anything fails.
• Metadata service health, because its outage is invisible to tasks already running and loud to everyone else.
• Datastore errors and request throttling from the cloud provider's metrics.
• Stale schedules, meaning a production flow with no successful run inside its expected interval. This catches deployments that silently stopped, which no failure alert can.

Governance is the set of rules that keep this sustainable, and it works best when expressed as code in a shared library rather than a wiki. The rules that pay for themselves: every production flow uses @project and a team label; every production flow has @timeout on long steps, an @retry policy and a failure notification; no secrets in Parameters; production deploys come from CI; and a flow's Config is reviewed like code. Enforce what you can mechanically. A custom flow mutator can inspect the flow at deploy time and refuse one that lacks a team label, and a lint step in CI catches the rest before the pull request is merged. The spin command and the Runner API make it possible to test such rules cheaply.

Incident playbooks

Playbooks turn the failure modes above into first moves. Each has the same shape: how to recognise it, the one check that separates the causes, the mitigation, and the follow-up that prevents a repeat.

Playbook 1: nothing is starting. Tasks sit Pending on Kubernetes or RUNNABLE on Batch. Check the pod's events (kubectl describe pod) for the scheduling reason. If it says insufficient CPU, memory or GPU, the request exceeds what the node pools can offer, or the autoscaler has hit its ceiling. Mitigation: raise the node pool's maximum or temporarily lower concurrency with --max-workers at the next deploy; for one stuck run, terminate it (argo-workflows terminate <run_id>) rather than leaving it consuming a slot. Follow-up: a capacity alert on time-to-start and a review of the @resources request that could not be placed.

Playbook 2: runs succeed but nobody can see them. Runs show as successful in the user's terminal, but the UI and the Client API do not list them. The cause is nearly always a mismatch between DEFAULT_METADATA (local) and the datastore (S3), or the wrong profile. Check metaflow status and get_metadata() on the affected machine. The data is usually intact in the bucket; the fix is correct configuration, and the prevention is distributing a known-good profile through a bootstrap script rather than expecting users to run metaflow configure by hand.

Playbook 3: the metadata service is down. Users report Metaflow service [url] unreachable. New runs fail at registration, the UI is dead, and running tasks may finish or fail depending on when they next contact the service. Check the service's pods, its Postgres and any authentication proxy in front of it, including expired tokens, which present as the same error. Mitigation: restore the service; if the database is the cause, fail over to a replica. Follow-up: health checks, a database with backups you have tested restoring, and autoscaling for bursts from wide foreaches.

Playbook 4: an Argo Events trigger loops or storms. A @trigger_on_finish cycle, or an event publisher that emits in a tight loop, launches runs continuously. Mitigation is to stop the source first: delete or suspend the deployments involved, then terminate the runs. Do not start by terminating runs while the trigger keeps creating new ones. Follow-up: a review rule that every trigger graph is drawn and acyclic, and a concurrency guard in the flows that matter.

Playbook 5: a credential was exposed. A token appeared in a Parameter, a printed log or an artifact. Rotate first, as always: the datastore holds the value, cached copies may exist, and deleting the run does not undo exposure. Then find every run that holds it with the Client API (Flow('X').runs() in the global namespace, reading parameter values), remove or restrict access to those objects in the bucket, and add the review rule that would have caught it.

Where Metaflow stops

Knowing where a tool stops is part of owning it, and it is what lets you answer a leadership question honestly. Metaflow is at its best for data-science and ML workflows that a small team authors in Python and wants to scale without rewriting: flows that fan out across data, train models, and hand results to the next stage. It is weaker in several areas.

It is not a general-purpose orchestrator. Dependencies between unrelated systems, sensors on external data, cross-team DAGs with backfills and the wide library of integrations that Airflow or Dagster offer are not what it is designed for. If your organisation needs those, compile Metaflow flows into the orchestrator you already have (Airflow and Kubeflow targets exist for this) or use the other tool directly. The Airflow, Dagster and Prefect guides show what that world looks like.

It has no built-in access control, experiment comparison or model registry in the open-source version. Tracking is a record of runs and artifacts, not a comparison workbench; users who need parameter sweeps compared on a chart reach for MLflow or Weights & Biases inside a step, and the docs note that cards can link to them. Registry and approval workflows are built from tags and namespaces by convention, or supplied by another tool.

Some of the most useful features are orchestrator-specific. Events, conditionals, recursion and exit hooks work on Argo Workflows only (conditionals locally as well), so a platform on Step Functions or Airflow must accept a smaller feature set.

The decision rule that fits all of this: choose Metaflow when the people writing the pipelines are data scientists and ML engineers who want plain Python and a clean path to Kubernetes or AWS without learning an orchestrator's API; choose something else when the dominant need is cross-system scheduling, governance and lineage across a broad data estate, or when the hard part of the problem is serving rather than training. For a managed option with authentication, a hosted UI and more, Outerbounds is the commercial platform built by Metaflow's creators; present it to leadership as one option next to running your own, with the trade-off being operational burden against cost.

Running Metaflow as a platform for other teams

Everything so far has been about operating the infrastructure. The other half of platform ownership is shaping the authoring experience so that users fall into good practice. Metaflow offers three mechanisms, and using them is the difference between a platform and a shared cluster.

A golden path with a base flow. The BaseFlow pattern subclasses FlowSpec to share scaffolding: common Parameters, shared Config loading, standard decorators and helper methods. A team's flow inherits from your PlatformFlow and gets the project name, the labels, the notification hooks and the standard timeout without repeating them. This is the lightest mechanism and covers most needs.

Flow and step mutators (2.16) let you change a flow programmatically. A FlowMutator implements mutate(self, mutable_flow) and can inspect every step and attach decorators, which is how you implement rules like "every step without a @timeout gets one" or "every production step carries the team label". A StepMutator does the same at step level.

Custom step decorators. @user_step_decorator turns a generator function into a decorator whose code before yield runs before the step and whose code after runs afterwards. That is the right shape for cross-cutting concerns such as loading a company data client, recording a custom metric or cleaning a scratch directory.

from metaflow import FlowSpec, user_step_decorator

@user_step_decorator
def with_team_metrics(step_name, flow, inputs=None, attributes=None):
    started = start_metric_timer(step_name)   # your own helper
    yield
    stop_metric_timer(started)

class PlatformFlow(FlowSpec):
    """Base class every team flow inherits from."""
    pass

Good platform APIs share four traits. They are versioned: the platform package has a semantic version that users pin, and breaking changes come with a deprecation window. They are tested in isolation: the package has its own CI with a canary flow run through Runner, and integration tests against the metaflow-dev local stack (which runs Minikube, Tilt, the metadata service, the UI and Argo) catch real deployment problems without a cloud account. They are documented by example: a template repository with a working flow, a Config file and a CI pipeline is worth more than a long document. And they are escape-hatched: a team with a legitimate exception, say a custom image or a privileged step, can get it through a reviewed path, rather than forking the platform.

The review checklist

Use this list in design reviews and pull requests. Each item is a question with an answer someone can point to.

Architecture

• Which datastore, metadata provider, orchestrator and compute layer does this flow target, and is the feature set it uses supported on that orchestrator (conditionals, events, exit hooks)?
• Is the flow within step-size guidance, and do long steps have checkpoints or a split?
• Is every foreach bounded and batched so that task overhead is small compared with the work?

Security

• Are there credentials in Parameters, artifacts or logs? Are secrets read through @secrets with a source scoped by IAM?
• Which service account and namespace do the tasks use, and what can that identity read and write in the datastore?
• Is the image pinned and scanned, and are dependencies pinned?

Deployment

• Is production deployed from CI with @project and a --production flag, with the commit recorded in tags?
• Is the workflow name under the limit, and is the schedule explicit and in the intended timezone?
• Does the Config diff show exactly what changes between environments?

Operations

• What happens on failure: retries, @catch, notification, owner?
• What are the timeouts, and what is the expected cost per run?
• Are there trigger cycles? Are labels present for cost attribution?

Lifecycle

• What is the retention for artifacts and logs, and is it in the bucket's lifecycle policy?
• Who re-deploys this flow after a Metaflow upgrade, and when?

The complete picture

Put the whole series together and Metaflow looks like a thin, well-chosen layer. At Beginner, it is Python with decorators and a DAG that saves your variables. At Mid, it is a way to place steps on Kubernetes or Batch, pin their dependencies, deploy them to a scheduler and drive them from notebooks, tests and CI through Runner and Deployer. At Senior, it is the infrastructure those features sit on: a datastore that holds your data and therefore your security boundary, a metadata service that records structure and therefore your observability, a compute layer that defines your capacity and cost, and an orchestrator that decides which features you can use.

The through-line is that Metaflow moves complexity out of the flow code and into the platform. That is what makes it pleasant for data scientists and what makes it a real responsibility for you. The rules that hold the complexity together are few: one pinned client version everywhere, production deployed only by CI, real access control enforced by IAM and RBAC rather than namespaces, secrets kept out of artifacts, bounded and batched fan-out, explicit schedules and owners, and a playbook for each component's failure. Neighbouring guides complete the picture: Kubernetes for the compute layer, Docker for images, Terraform for the infrastructure, MLflow for tracking and registry, and Evidently for what happens to a model after a flow ships it.

Where the series leaves you

You can now reason about Metaflow from three positions: as a user writing a flow, as an engineer wiring it into delivery, and as the owner of the platform other people depend on. You can name the failure domain in an incident, predict which component saturates first under load, explain precisely why namespaces do not protect data, and defend an orchestrator choice with the feature matrix rather than a preference.

What to do next is practical. Stand up the metaflow-dev stack and break it on purpose: stop the metadata service mid-run, fill a node pool, deploy a long-named flow without @project. Write your own platform base class with one mutator. Draft the retention policy and the IAM design for your buckets. Then run a game day with a colleague who plays the user who says "my run vanished". The fastest way to become a trustworthy platform owner is to have seen each failure once before it happens for real.

Sources

• Metaflow documentation
• Technical overview
• Release notes and GitHub releases
• Metaflow 2.17.0 release
• Metaflow 2.18.0 release
• Metaflow 2.19.0 release
• Deploying infrastructure
• Local devstack
• Scheduling with Argo Workflows
• Scheduling with AWS Step Functions
• Scheduling with Airflow
• Scheduling with Kubeflow
• Event triggering
• Coordinating larger projects
• Deployer API
• Runner API
• Secrets
• Kubernetes remote tasks
• AWS Batch remote tasks
• Controlling parallelism
• Spot instances
• Dependencies
• Custom decorators
• Mutators
• BaseFlow
• Client API
• Configuration source (metaflow_config.py)
• Metaflow on PyPI

This is part three of three. At senior level, interviewers stop asking what a decorator does and start asking what you would build, what you would do at 3 a.m., and how you would convince a director to fund it. About forty minutes. System-design prompts first, then incident questions, then the leadership trade-offs, then a self-test.

The answers below are models, not scripts. What a strong answer shares is structure: it names the failure domain or the constraint first, states the trade-off explicitly, and ends with how you would know the decision was right.

The thirty-second answer

Metaflow is a Python library for writing ML and data workflows as flows of steps, where every step's outputs are saved automatically to an object store. The same code runs on a laptop, on Kubernetes or AWS Batch, and on a production scheduler such as Argo Workflows. As a platform owner I run what surrounds it: the datastore, the metadata service, the compute layer and the orchestrator, and I enforce access control through cloud IAM and Kubernetes RBAC, because Metaflow itself has none.

Say that last clause early. It shows you have read past the marketing and know where the security boundary really is.

The model to have ready

The four failure domains. Datastore, metadata service, compute, orchestrator. Every incident question is easier if you start by asking which of the four is involved.

The data path of a remote task. The orchestrator submits a container, which downloads the code package, runs python flow.py step <name>, reads input artifacts from the datastore, writes outputs back and registers the task with the metadata service. Nothing passes directly between tasks.

What is and is not a security control.

The version facts. The current release is 2.19.39 (September 2026). Conditionals and recursion arrived in 2.18 and work on Argo Workflows and locally, not on Step Functions, Airflow or Kubeflow. The Argo naming rule changed in 2.17, with a 45-character limit on names without @project. Custom decorators and mutators arrived in 2.16, spin in 2.19.0.

System-design prompts

1. Design a Metaflow platform for ten data-science teams

A strong answer starts with requirements before architecture: which cloud and region, how sensitive the data is, how many concurrent runs, whether the teams already have an orchestrator, and who is on call. Then it lays out the design in layers.

Datastore: one bucket per environment, with per-team prefixes or separate buckets where data must not be shared, in a region that satisfies data-residency expectations (for Gulf and Egyptian employers, a Middle East region for both bucket and compute so that artifacts do not cross borders on every step). Metadata: one service with a managed Postgres, backups, autoscaling and an authenticating proxy using SERVICE_HEADERS, since the open-source service has no authorization of its own. Compute: a Kubernetes cluster with a namespace and service account per team, quotas, separate node pools for CPU and GPU, and Workload Identity or IRSA so that a team's tasks get that team's IAM permissions. Orchestrator: Argo Workflows with Argo Events, because it supports the widest feature set, and deployment exclusively from CI. Dependencies: curated base images for heavy components and @pypi for the rest. Observability: notifications on failure, time-to-start and stale-schedule alerts, and OpenTelemetry tracing.

What makes it senior is the last third: how teams onboard (a template repository and a profile bootstrap), how you enforce rules (a base class, mutators and CI checks), how you charge back cost (labels on every pod), and what you would deliberately not build in the first quarter.

A weak answer draws boxes without asking a single question, says "we'll use namespaces for isolation", and never mentions who can read the bucket.

2. Two teams must not see each other's data. How do you isolate them on Metaflow?

Begin by stating that namespaces do not do it. The documentation says so directly. Real isolation is layered: separate datastore prefixes or buckets with IAM policies bound to each team's service account; separate Kubernetes namespaces with RBAC, quotas and network policies; separate secret sources scoped by IAM; and, if even the existence and names of the other team's flows must be hidden, separate metadata services and UIs. Profiles (METAFLOW_PROFILE) make the client side manageable, since each team's bootstrap points at its own infrastructure.

Then describe the cost of each layer, because isolation is a spectrum. Separate prefixes are cheap and sufficient for most organisations. Separate metadata services double operational work, and a regulatory requirement is the usual reason to pay it. A strong candidate finishes by saying how they would test it: from one team's identity, try to read the other's bucket and run namespace(None) against the shared service, and confirm what is and is not visible.

3. Design the deployment pipeline for production flows

Production is deployed by CI only. The pipeline lints the flow (step sizes, required decorators, name length), runs the flow's tests, builds or selects the image, then deploys with --production under @project, tagging the deployment with the commit. A smoke trigger through the Deployer API waits for a run to finish and rolls back on failure. The client version is pinned in the CI image. Configuration differences by environment are in a Config file that reviewers can read as a diff.

The follow-up questions test depth. Why CI only? Because the production token is a guard against accidents, and sharing it in chat defeats it; the CI identity holds the deploy permission and developers have only branch deployments. What if two different Metaflow versions deploy? The 2.17 naming change can leave you with two deployments of the same flow, and the snapshotted client version in each deployment is different. How do you roll back? Re-create the previous tagged commit through the same pipeline, since deployments are snapshots of code and Config.

4. A team wants to train a model on 64 GPUs across eight nodes. How does Metaflow handle it?

Metaflow supports gang-scheduled training through @parallel, with self.next(self.train, num_parallel=N), and framework-specific decorators such as @pytorch_parallel, which on Kubernetes use JobSets. Caveats worth naming: @catch is not supported with @parallel on Kubernetes, spin disallows @parallel, and a long training step is not resumable by design unless you use the @checkpoint extension, whose APIs are not covered by core's compatibility guarantee.

The design answer is about the cluster more than the flow: GPU node pools with a taint and matching toleration, a JobSet controller installed, quotas so that one team cannot take the whole pool, and a policy on spot capacity that depends on whether checkpointing is tested. A strong candidate also asks whether Metaflow is the right layer, since heavy distributed training sometimes belongs to a dedicated training platform that Metaflow merely launches.

5. Choose an orchestrator target

Give the feature matrix from memory. Argo Workflows supports events, conditionals, recursion, exit hooks and the full Deployer API, and runs anywhere Kubernetes does. Step Functions is AWS-only with a managed control plane, uses AWS Batch, and lacks events, conditionals and recursion. Airflow and Kubeflow are integration targets with only @kubernetes and the same gaps. The recommendation depends on context: an AWS-only organisation with no Kubernetes expertise can rationally choose Step Functions; everyone else usually chooses Argo. The senior move is to say what you will write into the authoring guidelines so that users do not use an unsupported feature on the chosen target.

Incident-response questions

6. "My run has been stuck for an hour and nothing is happening."

Check the state of the tasks first. If the flow is waiting on a task that is Pending (Kubernetes) or RUNNABLE (Batch), it is a capacity problem: read the pod events, compare the @resources request with the largest node, check the autoscaler's ceiling and the quota. If tasks are running but producing nothing, read logs with python flow.py logs <pathspec> and check for a hung external call with no timeout. If the run is not in the UI at all, suspect the metadata provider. A strong answer does not guess; it lists the discriminating checks in order of cost.

7. "Metaflow runs are failing with 'Metaflow service unreachable'."

The metadata service, the database behind it or a proxy in front is unhealthy, or a credential for the proxy expired. Verify with the health endpoint from inside the cluster and from a laptop, which separates a service outage from a network or authentication problem. Mitigate by restoring the service; tasks already running may finish, but new runs fail at registration. Afterwards, the prevention is health checks, a backup you have restored at least once, and autoscaling for foreach bursts. Mention that user-visible damage is lower than it looks, because artifacts live in the datastore rather than in the service.

8. "We deployed a flow and now there are two of it on Argo."

Suspect the 2.17 naming change: a flow name over the limit, deployed by a client older than 2.17 and then again by a newer one. List the templates with argo-workflows list-workflow-templates --all, decide which is authoritative, and delete the other with its own CLI, noting that the current client keeps backward-compatible name resolution for old deployments. Prevention: a single pinned deployer version and names within the limit.

9. "A secret was printed in a run's logs."

Rotate first, then investigate. Logs are stored in the datastore, the secret may have been read by anyone with bucket access, and deleting the run does not undo exposure. Then find the affected runs, restrict or remove the log objects, and add the review rule and the CI check that would have caught it. Mention that credentials in Parameters are the same class of problem because Parameters are persisted as artifacts.

10. "A scheduled flow has not run for three days and no alert fired."

Failure alerts only fire on failures. A flow that never started produces nothing. Check the deployment: was it re-created after an edit, does it have @schedule, is the timezone what the team thinks (it defaults to Etc/UTC and works on Argo only), and did anything delete the template during branch cleanup? The lesson for the platform is a stale-schedule alert that compares the last successful run with the expected interval.

11. "The cloud bill doubled last month."

Treat it as a forensic question. Group cost by the labels you attach to pods or Batch jobs to find the team and flow; then look for the usual causes: a forgotten scheduled branch, a foreach that was widened without batching, a GPU step holding nodes for minutes of work, storage growth from artifacts and a lack of lifecycle rules, or a retry storm after a change. Fix the cause, then add the guardrail: default timeouts through DEFAULT_DECOSPECS, a split cap, and a weekly review.

Arguing trade-offs to leadership

12. Run it ourselves or pay for a managed platform?

Put the trade-off in terms leadership cares about. Open-source Metaflow is free, runs on your cloud and keeps data in your accounts, but you carry the operational load: upgrades, the metadata service, authentication, the UI and on-call. A managed option such as Outerbounds, built by Metaflow's creators, adds authentication, a hosted UI and more, in exchange for cost and a vendor relationship. Present both with numbers: engineer-hours per month for the platform, the cost of an outage, and the compliance requirements that decide whether data may leave your account. Recommend, state what would change your mind, and propose a review date.

13. Why Metaflow instead of Airflow or Kubeflow?

Answer by audience. Data scientists write plain Python, get local-to-cluster parity and automatic versioning of every variable, and do not learn an orchestrator's DAG API. The cost is that Metaflow is not a general scheduler with sensors and broad integrations. The honest recommendation may be both: author in Metaflow and compile to Argo, or to Airflow if the organisation has already standardised on it. See the Airflow and Kubeflow guides.

14. How do you justify the platform team's headcount?

Measure outcomes the business sees: time from notebook to scheduled production flow, number of failed production runs caused by environment or deployment error, cost per run by team, and the share of teams on the golden path. Compare those to the state before, and be candid that some benefits are avoided incidents, which are visible only through playbooks and alert history.

15. A team wants an exception: a privileged container.

Do not say no by reflex. Ask what they need, because many privileged requests are a missing capability or a device mount. If the need is real, grant it through a reviewed path with a dedicated namespace, a restricted service account and an expiry date, and enforce with admission policy so the exception cannot spread. Document it. The point to make is that security_context is user-settable in the decorator, so the control must live in the cluster and not in the authoring guidelines.

What a strong answer covers that a weak one misses

Sixty-second self-test

Answer each aloud, then check.

1. Which four components can fail, and what does the user see for each?
2. Why do namespaces not protect data, and what does?
3. What happens to an existing Argo deployment when you upgrade Metaflow?
4. Which orchestrators support @trigger, and which support conditionals?
5. Why must production deployment come from CI?
6. What is the first thing to do when a credential appears in a Parameter?
7. What is the default ceiling on a remote task's runtime, and how is it overridden?
8. Name three guardrails on fan-out cost.
9. Which mechanism would you use to add a team label to every step automatically?
10. Give two situations where you would recommend something other than Metaflow.

Answers. 1: datastore (failed reads and writes), metadata service (service unreachable, UI and Client API down), compute (Pending or RUNNABLE, OOM), orchestrator (schedules and triggers silent). 2: they are default Client API filters, and IAM and RBAC protect data. 3: nothing; deployments keep the snapshotted version until re-created. 4: events on Argo only; conditionals on Argo and locally. 5: tokens are guards, not controls, and CI gives an auditable identity. 6: rotate. 7: five days, @timeout. 8: --max-num-splits, --max-workers, @timeout, right-sized @resources. 9: a flow mutator or base class, plus a default decorator config. 10: cross-system scheduling with sensors and backfills, low-latency serving, streaming.

Carry the habit out of the interview room: before proposing anything, say which component owns the problem and what it does when it fails.

This is part three of three. At this level the problems are not syntax errors. They are things that work perfectly for one team for six months and then fail when the platform has forty users, three environments and a bucket nobody has cleaned. Start with the traps, then the ownership habits, then the review checklists.

Production traps that only appear at scale

Each trap below lists the symptom, the real cause and the fix. Cumulative: everything from Beginner and Mid still applies.

The foreach that was fine at 100 and melts at 10,000

Symptom. A flow that ran comfortably in development takes hours longer than the compute time suggests in production, or fails before fan-out with Foreach in step ... yielded N child steps which is more than the current maximum of 100 children.

Cause. Every task has fixed overhead: a container start, a code package download, input artifact reads and output writes, and metadata registration. With 10,000 tiny tasks the overhead dwarfs the work, and the cluster autoscaler, datastore and metadata service are all pushed at once. The split cap exists as a safety net, not a design.

Fix. Batch the inputs so each task does meaningful work, then fan out over batches. Raise --max-num-splits only after doing the arithmetic on cost, and cap concurrency with --max-workers. Put a batching helper in the shared library so people do not write their own.

Namespaces treated as access control

Symptom. A security review asks how team A is prevented from reading team B's runs, and the answer is "namespaces".

Cause. The documentation says namespaces are not about security or access control. They are a default filter, and namespace(None) removes it. Anyone who can read the datastore bucket can read every artifact.

Fix. Enforce isolation with bucket IAM, Kubernetes RBAC and service accounts, network policies, and separate metadata services only where required. Write the real answer into the design document.

Credentials in Parameters or artifacts

Symptom. A token shows up in the UI, in run.data or in an audit of the bucket.

Cause. Parameters are persisted as artifacts, as is every self.x assigned in a step. Printing os.environ puts values in logs, which are also stored in the datastore.

Fix. Use @secrets(sources=[...]), which injects values as environment variables, and keep them out of self. Rotate anything that was exposed before cleaning up. Scope the secret to the production orchestrator's role so a laptop cannot read it.

Two deployments of the same flow

Symptom. After an upgrade or a colleague's deploy, the same flow exists twice on Argo, and both fire on schedule.

Cause. Metaflow 2.17 changed Argo workflow naming. Names over the limit (45 characters in current source) are truncated and hashed under @project, and are an error without it. The release notes warn that deploying the same flow with an older version leads to two separate deployments.

Fix. Pin one Metaflow version in CI and everywhere that deploys, use @project, keep names short, and find strays with argo-workflows list-workflow-templates --all.

Flows that work locally and fail remotely on imports

Symptom. ModuleNotFoundError in a remote task for a module that imports fine on the laptop.

Cause. Two separate causes that look identical. The code package contains only files with the configured suffixes (default .py,.R,.RDS) from the flow's directory, so sibling directories are missing. Or a package is imported at the top of the file under @pypi, where the environment does not exist yet.

Fix. Check python flow.py package list. Add sibling directories deliberately with @package_sources (2.19.39), remembering that its suffixes replace the default set. Import third-party packages inside the step and declare them in packages=.

Unpinned dependencies in production

Symptom. A production flow breaks on Monday with no code change.

Cause. @pypi resolved an unpinned package to a new release. Environments are cached in the datastore, so the break appears when the cache is rebuilt or a new environment is resolved.

Fix. Pin exact versions and the Python version in @pypi/@conda, or use curated images for production. Use --force-rebuild-environments deliberately, not as a reflex.

A long step that cannot survive a spot reclamation

Symptom. A fifteen-hour training step on spot capacity loses everything at hour fourteen.

Cause. Steps are the unit of resumption. Within a step nothing is saved unless you add checkpoints.

Fix. Split the work so a boundary falls where a restart is cheap, or use the @checkpoint extension with current.checkpoint.save(), noting it is an extension whose APIs may change. React to current.spot_termination_notice, and test the resume path before trusting it.

The schedule that silently stopped

Symptom. A nightly flow has not run for days and no alert fired.

Cause. Failure alerts need a failure. A deployment that was deleted, never re-created after an edit, or given the wrong timezone produces no events. On Argo a flow runs only if it has @schedule or @trigger.

Fix. Alert on staleness: no successful run within the expected interval plus slack. Review @schedule timezones, which work on Argo only and default to Etc/UTC.

Trigger loops

Symptom. Runs multiply without anyone launching them, and cost climbs.

Cause. @trigger_on_finish cycles between flows, or an event publisher in a loop.

Fix. Stop the source first, then terminate runs. Draw the trigger graph, keep it acyclic and review it like architecture.

Production redeploy fails with "Incorrect production token"

Symptom. A second engineer cannot redeploy a flow.

Cause. The first create generated a production token, and the flow is owned by whoever deployed it.

Fix. The fix is not to share the token. Deploy production from CI with a single identity and give humans branches under @project.

Upgrade surprises

Symptom. A bug fixed in the latest release persists in production; or a newer feature fails against the service.

Cause. Deployments are snapshots: Metaflow packages itself into each code package, so existing deployments keep the version they were created with. Some features, such as spin, may need a newer metadata service.

Fix. After each upgrade, re-create production deployments through CI. Upgrade the service before advertising a feature that needs it.

Tasks stuck Pending or killed with exit code 137

Symptom. A run sits for an hour with tasks Pending on Kubernetes (or RUNNABLE on AWS Batch), or a task dies with exit code 137 or OOMKilled.

Cause. These are opposite problems. Pending means the @resources request is larger than anything the node pools can offer, a GPU pool is missing, a quota is exhausted or a taint has no matching toleration; the docs say a task that cannot fit will not start. Exit 137 means the task started and used more memory than it requested.

Fix. For Pending, read the pod events, compare the request with the largest node and check the autoscaler ceiling. For 137, measure real memory use, raise memory= with headroom and keep @retry so one OOM does not end the run. If memory grows with data size, change the step to stream or chunk the data.

A client that is configured differently from the platform

Symptom. Runs succeed in a terminal and cannot be found in the UI, or Flow('X') does not exist appears for a flow you just ran.

Cause. The metadata provider is local while the datastore is in S3, or the wrong METAFLOW_PROFILE is active, or the command runs from a different working directory with local metadata.

Fix. Check metaflow status and get_metadata() first. The data is usually safe in the bucket. Prevent it by distributing a known-good profile rather than asking users to run the configure wizard.

Silent drift between environments

Symptom. A flow behaves differently in staging and production although the code is identical.

Cause. Differences live in places the code review never saw: a hand-edited Config, an environment variable on the orchestrator, a different image tag or a different secret source.

Fix. Keep per-environment Config files in version control, deploy them through CI and compare them as diffs. Avoid "deploy-time Parameters" that hide differences in a function default; since 2.13 the docs recommend Config instead.

Platform-ownership habits

Pin one client everywhere

Put metaflow==<version> in the CI image, the platform package and the developer template, and make drift visible. Most "works on my machine" platform incidents are version skew. Upgrade on a schedule with a canary team, reading the GitHub release notes first.

Distribute configuration instead of documenting it

A profile bootstrap script that writes ~/.metaflowconfig/config.json (or config_<name>.json with METAFLOW_PROFILE) removes a whole category of support tickets: local metadata with a cloud datastore, wrong service URL, missing user name in CI. Keep profile files in version control and generate them from the same Terraform outputs that create the infrastructure.

Turn rules into code

A wiki page that says "every production flow needs a timeout and a team label" is decoration. A base flow class, a flow mutator that adds the defaults and a CI lint that fails the pull request are a rule. Keep the platform library small, versioned with semantic versions, and tested with a canary flow that you run through Runner on every release.

Give every byte and every pod an owner

Attach labels through a default so that a cost report can name a team for every pod or Batch job. Put lifecycle rules on the bucket. Assign an owner to every production deployment and keep that mapping somewhere an alert can read it.

Practise the failures

Stand up the metaflow-dev stack and break it: stop the metadata service mid-run, exhaust a node pool, deploy with a long name, run with the wrong profile. Write what the user sees and what you check first. The playbooks in the detailed guide are only useful if the first reading is not during an incident.

Keep an honest list of where Metaflow stops

Record the requests you turned down and why: cross-system scheduling, serving, streaming, experiment comparison. When the same request appears three times, that is the signal to adopt a neighbouring tool such as MLflow, Airflow or BentoML, not to bend Metaflow.

Treat exceptions as data

When a team asks to bypass a rule, log it with an owner and an expiry. If the same exception is requested by three teams, the rule is wrong or the golden path is missing a capability. Fix the platform, not the requester. The exception log also becomes your roadmap, because it records in the users' own words where the golden path is too narrow.

Measure the platform like a product

Track time to first successful remote run for a new user, time-to-start for tasks, failure rate by cause, cost per team and the share of production flows on the golden path. Review them monthly. The numbers are also what you use when leadership asks what the platform team is for.

Review checklists

Pull request checklist for a flow

• Top-level CLI options are in the right place in every command in CI (before the subcommand).
• No credentials in Parameters, artifacts or printed output; secrets come from @secrets.
• Third-party imports are inside steps, and versions are pinned in @pypi or @conda.
• Each foreach is bounded and batched; its --max-workers is acceptable for the cluster.
• Long steps have @timeout, @retry and, where spot capacity is used, checkpoints.
• @resources values are measured, not copied.
• @catch is on the inner step of a foreach, not the step that fans out.
• Conditionals and recursion are used only if the target orchestrator supports them.
• @schedule has an explicit timezone where it matters, and @trigger* has been checked for cycles.
• The flow name fits the limit, and @project is present for deployed flows.

Design document checklist for a platform change

• Which component changes (datastore, metadata service, compute, orchestrator), and what does each do when it fails?
• What is the security boundary, stated in terms of IAM and RBAC rather than namespaces?
• How is the change rolled out and rolled back, and which deployments must be re-created?
• What does it cost per month and per run, and which label attributes it?
• Which versions are affected (client, service, extensions), and what do the release notes say?
• What is the retention policy for artifacts, logs and metadata?
• What new alert exists, and who receives it?
• What is the exit plan if the choice turns out wrong (a different orchestrator, a managed platform)?

Upgrade checklist

1. Read release notes for every version between yours and the target.
2. Confirm the metadata service version supports the client features you plan to use.
3. Upgrade the CI image and platform package, then a canary team, then everyone.
4. Re-create production deployments through CI and run a smoke flow on each target.
5. Keep the previous tag available for rollback, and record the change.
6. Upgrade extensions (metaflow-checkpoint, metaflow-kubeflow) separately.

Incident checklist

1. Name the failure domain: datastore, metadata service, compute or orchestrator.
2. Stop the source before cleaning up (triggers, schedules) and rotate before deleting (credentials).
3. Mitigate, then communicate, then find the cause.
4. Record what the user saw and what confirmed the diagnosis.
5. Add the alert, the review rule or the CI check that would have caught it.

Practice cards

Card 1, the cost audit. List every scheduled production flow with its owner, schedule, typical run time, largest @resources request and cost per run. Find the three most expensive and ask whether each needs its frequency, its hardware and its retention.

Card 2, the isolation test. From a developer role, try to read another team's bucket prefix, run namespace(None) and list their flows, and read their production secret. Write down what succeeded. Each success is a finding.

Card 3, the version audit. Collect the Metaflow version from CI, three developer machines and every production deployment. Decide the target, plan the re-deploy window, and find out which deployments are older than 2.17.

Card 4, the capacity drill. Launch a foreach at the largest width any team plans, in a test namespace. Measure time-to-start, datastore request rate and metadata service latency, and record which component saturated first.

Card 5, the exit-hook build. Write an @exit_hook that posts to your team channel with the run's pathspec on failure. Remember that it runs as a separate pod on Argo, cannot create artifacts and cannot use @pypi or @conda, so it needs an image with what it imports.

Operational recipes

# Is this the environment I think it is?
metaflow status
metaflow configure show

# What will be shipped to remote tasks?
python flow.py package list

# What is deployed, and what is stale?
python flow.py argo-workflows list-workflow-templates --all

# Operate a live run
python flow.py argo-workflows status argo-<workflow-execution-name>
python flow.py argo-workflows terminate argo-<workflow-execution-name>

# Deploy production from CI with provenance
python flow.py --environment=pypi --production argo-workflows create \
  --tag git:$GITHUB_SHA

from metaflow import Flow, namespace

namespace(None)  # global namespace: inspect every user's runs
for run in Flow("ForecastFlow").runs():
    if not run.successful and run.finished:
        print(run.pathspec, run.finished_at)

A short checklist to carry

• Name the failure domain before anything else.
• Namespaces filter; IAM and RBAC protect.
• One pinned client version, everywhere, including CI.
• Production deploys only from CI, under @project, tagged with the commit.
• No credentials in Parameters or artifacts; rotate before cleaning.
• Bound and batch every foreach; set timeouts and retries.
• Alert on staleness, not just on failure.
• Re-create deployments after upgrades.
• Put labels on every pod so every line of spend has an owner.
• Know where Metaflow stops, and say so.

The habits above repeat one idea: Metaflow makes it easy for users to do the right thing locally, and your job is to make the right thing the default in production.