The Complete MLflow Guide

This is part one of three. It covers everything you need to do real work with MLflow, not a teaser. By the end you can track an experiment without restructuring your code, log parameters, metrics, and artifacts on purpose, compare thirty runs in a table and a chart, package a model so someone else can load it without knowing your framework, register and promote it through stages, and serve it behind an HTTP endpoint. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and these ideas only stick once you have watched a metric appear in the MLflow UI while your script is still running.

What MLflow is, and the problem it solves

You trained a model last Tuesday. It scored 0.94 AUC. Today you cannot reproduce it. The notebook has moved on, the learning rate in the cell is not the one that ran, the feature list changed, and the only surviving record of 0.94 is a number in a Slack message.

Carelessness was never the problem. A system was missing: nothing was writing anything down. Every fact about that run lived in your terminal scrollback and your memory.

MLflow writes it down. You add a handful of lines, often just one, and it records the parameters, the metrics over time, the artifacts, the source file, the git commit, and the environment needed to load the model again, into a store you and your team can query.

MLflow's central idea is a framework-agnostic contract. A logged model is a directory with a declared flavour, a signature describing inputs and outputs, and a dependency list rather than a bare pickle from someone's laptop. Anything that understands that contract can load it: a batch job, a REST server, a Spark UDF, a Docker image. That single property is what turns a training script into something deployable.

That gives you four components, and they map exactly onto four problems:

The vocabulary is small, and getting it straight now prevents most later confusion:

You need little to follow along: Python, a script that trains something small, and a folder. A tiny dataset and a five-line model are the best place to start, because a broken run costs you nothing.

The four components, and the two stores behind them

MLflow is often described as one tool. It is four that share a store, and you can adopt them in order.

Two stores sit behind everything, and confusing them causes a whole category of errors:

# Local files only — no server, no database. Creates ./mlruns
python train.py

# A local server with a SQLite backend — needed for the Model Registry
mlflow server --backend-store-uri sqlite:///mlflow.db \
              --default-artifact-root ./mlartifacts \
              --host 127.0.0.1 --port 5000

# A team setup: Postgres for metadata, S3 for files
mlflow server --backend-store-uri postgresql://user:pass@host/mlflow \
              --default-artifact-root s3://my-bucket/mlflow \
              --host 0.0.0.0 --port 5000

Your code points at a tracking server with one line, or one environment variable:

import mlflow
mlflow.set_tracking_uri("http://127.0.0.1:5000")   # or "sqlite:///mlflow.db", or a path
mlflow.set_experiment("churn-baseline")

export MLFLOW_TRACKING_URI=http://127.0.0.1:5000

Your first tracked run, line by line

Start from a script with no MLflow in it:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

n_estimators, max_depth = 200, 5

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth, random_state=42)
model.fit(X_train, y_train)
print("auc", roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

Here it is tracked:

import mlflow
import mlflow.sklearn
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("churn-baseline")

params = {"n_estimators": 200, "max_depth": 5, "random_state": 42}

with mlflow.start_run(run_name="rf baseline") as run:
    mlflow.log_params(params)

    X, y = load_breast_cancer(return_X_y=True, as_frame=True)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    model = RandomForestClassifier(**params).fit(X_train, y_train)
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])

    mlflow.log_metric("auc", auc)
    mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))

    print("run id:", run.info.run_id, "auc:", auc)

Run it, then open the UI. Four things happened:

1. A run was createdWith a run id, an experiment, a status, a start and end time, and the user and source file that produced it.
2. Parameters were recordedImmutable strings. A parameter is set once and describes the configuration of the run.
3. A metric was recordedA float with a step and a timestamp, so a metric is a series rather than a single number.
4. A model was loggedNot a bare pickle, but a directory with an MLmodel file declaring its flavours, its signature, and its dependencies.

The with block matters more than it looks. It sets the run's status to FINISHED on success and FAILED on an exception, and it guarantees the run is closed. Without it, an interrupted script leaves a run stuck in RUNNING forever.

Params, metrics, tags, artifacts: choosing correctly

Four things you log, with four different meanings. Choosing correctly is most of what separates a readable experiment table from an unusable one.

# Parameters: configuration. Immutable, stored as strings.
mlflow.log_param("learning_rate", 3e-4)
mlflow.log_params({"batch_size": 64, "optimizer": "adamw", "features": "v3"})

# Metrics: numbers that can move. Float, with an optional step.
mlflow.log_metric("train_loss", 0.412, step=epoch)
mlflow.log_metric("val_loss", 0.503, step=epoch)
mlflow.log_metrics({"precision": 0.91, "recall": 0.88}, step=epoch)

# Tags: mutable, searchable labels for organisation.
mlflow.set_tag("stage", "baseline")
mlflow.set_tags({"team": "risk", "dataset": "v3", "owner": "amina"})

# Artifacts: files.
mlflow.log_artifact("outputs/confusion_matrix.png")
mlflow.log_artifact("outputs/report.html", artifact_path="reports")
mlflow.log_artifacts("outputs/plots", artifact_path="plots")   # a whole directory

# Convenience helpers that avoid touching disk yourself
mlflow.log_figure(fig, "plots/roc.png")
mlflow.log_table(predictions_df, "predictions.json")
mlflow.log_text(json.dumps(config, indent=2), "config.json")
mlflow.log_dict(config, "config.yaml")

Two rules resolve nearly every "which one?" question. If it is a number you might plot, it is a metric. If it is something you might change your mind about after the run, it is a tag.

Autologging: one line, and its boundaries

Most of the code in the previous section is unnecessary for common frameworks. One line replaces it.

import mlflow

mlflow.autolog()            # everything MLflow can detect

# or per-framework, which is more predictable
mlflow.sklearn.autolog()
mlflow.pytorch.autolog()
mlflow.tensorflow.autolog()
mlflow.xgboost.autolog()
mlflow.lightgbm.autolog()
mlflow.transformers.autolog()

With sklearn.autolog() active, calling .fit() logs the estimator's parameters, training metrics, the fitted model with a signature, and, for search objects like GridSearchCV, child runs for every candidate configuration:

import mlflow
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import GridSearchCV

mlflow.set_experiment("gb-sweep")
mlflow.sklearn.autolog(max_tuning_runs=10)

grid = GridSearchCV(
    GradientBoostingClassifier(random_state=42),
    {"max_depth": [2, 3, 4], "learning_rate": [0.05, 0.1]},
    scoring="roc_auc", cv=3,
)

with mlflow.start_run(run_name="gb grid search"):
    grid.fit(X_train, y_train)          # one parent run, plus child runs per candidate

What autologging does not do: log your own custom evaluation metrics, log artifacts you generated yourself, or know which of your variables were meaningful. So the realistic pattern is autolog plus a few deliberate calls:

mlflow.sklearn.autolog(log_models=True, log_datasets=True, silent=True)

with mlflow.start_run(run_name="rf + business metric"):
    model.fit(X_train, y_train)                     # autologged
    mlflow.log_metric("expected_loss_usd", cost)     # yours
    mlflow.set_tag("dataset", "v3")                  # yours
    mlflow.log_artifact("outputs/segment_report.html")

The MLflow Model format: flavours and signatures

This is the component that makes MLflow more than a logbook, and everything downstream depends on understanding it properly.

log_model does not save a pickle. It creates a directory:

model/
├── MLmodel                 # the manifest — flavours, signature, dependencies
├── model.pkl               # the serialised estimator
├── conda.yaml              # a conda environment that can load it
├── python_env.yaml         # a pip/venv equivalent
└── requirements.txt        # the pinned pip requirements

artifact_path: model
flavors:
  python_function:
    loader_module: mlflow.sklearn
    model_path: model.pkl
    python_version: 3.11.9
  sklearn:
    pickled_model: model.pkl
    sklearn_version: 1.5.1
    serialization_format: cloudpickle
mlflow_version: 2.16.0
model_uuid: 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
run_id: a1b2c3d4e5f64718b9c0d1e2f3a4b5c6
signature:
  inputs: '[{"name": "mean radius", "type": "double"}, {"name": "mean texture", "type": "double"}]'
  outputs: '[{"type": "long"}]'

Two ideas in that file carry the whole design:

Flavours are loaders. A model can declare several. The sklearn flavour lets you load it back as a scikit-learn object with all its methods. The python_function (pyfunc) flavour lets anything load it as a generic object with a predict method and no knowledge of scikit-learn at all. That second one is why the same artifact works in a REST server, a Spark job, and a Docker image.

import mlflow

# Native flavour: the real object, with its own API
sk_model = mlflow.sklearn.load_model("runs:/a1b2c3d4/model")
sk_model.feature_importances_

# Generic flavour: just predict(). Framework-agnostic.
pyfunc = mlflow.pyfunc.load_model("runs:/a1b2c3d4/model")
pyfunc.predict(X_test)

A signature is a contract. It declares the input columns and types and the output type, so a wrong-shaped or wrong-typed request is rejected at the boundary rather than producing nonsense predictions.

from mlflow.models import infer_signature

signature = infer_signature(X_train, model.predict(X_train))
mlflow.sklearn.log_model(
    model,
    name="model",
    signature=signature,
    input_example=X_train.head(3),      # also enables the UI's sample request
)

The URI schemes you will use constantly:

Comparing runs: the table, the chart, the query

Tracking pays off here, and it is largely a UI skill. Run your script three or four times with different parameters before reading on.

In the experiment table:

Learn the search syntax, because the Python API uses the same language:

metrics.auc > 0.9
params.max_depth = '5'
tags.dataset = 'v3' and metrics.auc > 0.88
attributes.status = 'FINISHED'
metrics.rmse < 1.5 and params.model_type != 'linear'

In code, which is how you build a report or a promotion gate:

import mlflow

runs = mlflow.search_runs(
    experiment_names=["churn-baseline"],
    filter_string="metrics.auc > 0.9 and tags.dataset = 'v3'",
    order_by=["metrics.auc DESC"],
    max_results=20,
)
print(runs[["run_id", "params.max_depth", "metrics.auc", "tags.mlflow.runName"]])

best = runs.iloc[0]
print("best run:", best.run_id, best["metrics.auc"])

search_runs returns a pandas DataFrame with flattened params.*, metrics.*, and tags.* columns, which makes "find the best run and do something with it" a three-line script rather than a project.

The Model Registry: versions and aliases

Tracking answers "what did we try". The registry answers "which one is live", and it is the boundary between experimentation and production.

Register a model in one of three ways:

# 1. At log time — the most common
mlflow.sklearn.log_model(model, name="model", registered_model_name="churn-classifier")

# 2. From an existing run afterwards
mlflow.register_model("runs:/a1b2c3d4e5f64718/model", "churn-classifier")

# 3. Through the client, with explicit control
from mlflow import MlflowClient
client = MlflowClient()
client.create_registered_model("churn-classifier")
client.create_model_version(
    name="churn-classifier",
    source="runs:/a1b2c3d4e5f64718/model",
    run_id="a1b2c3d4e5f64718",
)

Each registration creates a numbered version (version 1, 2, 3) which is immutable in content and permanently linked back to the run that produced it.

Then you point at a version by alias rather than by number, which is the part that makes deployment sane:

from mlflow import MlflowClient
client = MlflowClient()

client.set_registered_model_alias("churn-classifier", "champion", version=7)
client.set_registered_model_alias("churn-classifier", "challenger", version=8)

# Consumers never mention a version number
model = mlflow.pyfunc.load_model("models:/churn-classifier@champion")

Loading and serving: batch, REST, container

A registered model is only useful once something consumes it. Three ways, in increasing order of infrastructure.

In a batch job, which is the most common and the easiest:

import mlflow
import pandas as pd

model = mlflow.pyfunc.load_model("models:/churn-classifier@champion")
frame = pd.read_parquet("data/today.parquet")
frame["score"] = model.predict(frame[model.metadata.get_input_schema().input_names()])
frame.to_parquet("out/scored.parquet")

Behind a local REST endpoint, with one command:

mlflow models serve -m "models:/churn-classifier@champion" --host 127.0.0.1 --port 5001 --env-manager local

curl -X POST http://127.0.0.1:5001/invocations \
  -H 'Content-Type: application/json' \
  -d '{"dataframe_split": {"columns": ["mean radius", "mean texture"], "data": [[17.99, 10.38]]}}'

As a container, which is what you hand to a platform team:

mlflow models build-docker -m "models:/churn-classifier@champion" -n churn-scorer
docker run -p 5001:8080 churn-scorer

The --env-manager flag decides how the serving environment is built, and it is the setting people trip over:

Evaluating with mlflow.evaluate

Logging your own metrics is fine. For standard tasks there is a built-in evaluator that produces a consistent, comparable set, plus plots and an explainability summary, from one call.

import mlflow
from mlflow.models import infer_signature

with mlflow.start_run(run_name="rf + eval"):
    model.fit(X_train, y_train)
    signature = infer_signature(X_train, model.predict(X_train))
    info = mlflow.sklearn.log_model(model, name="model", signature=signature)

    eval_data = X_test.copy()
    eval_data["label"] = y_test

    result = mlflow.evaluate(
        model=info.model_uri,
        data=eval_data,
        targets="label",
        model_type="classifier",
        evaluators=["default"],
    )
    print(result.metrics["roc_auc"], result.metrics["f1_score"])

You can also add your own metric alongside the defaults, which is how a business number gets into the same comparable table:

from mlflow.models import make_metric

def _expected_cost(eval_df, _builtin_metrics):
    fp = ((eval_df["prediction"] == 1) & (eval_df["target"] == 0)).sum()
    fn = ((eval_df["prediction"] == 0) & (eval_df["target"] == 1)).sum()
    return 50 * fp + 500 * fn

expected_cost = make_metric(eval_fn=_expected_cost, greater_is_better=False, name="expected_cost")

result = mlflow.evaluate(
    model=info.model_uri, data=eval_data, targets="label",
    model_type="classifier", extra_metrics=[expected_cost],
)

Reproducibility: what a run records, and what it does not

A tracked run is not automatically a reproducible one. MLflow captures some of what you need and leaves the rest to you.

import mlflow.data
import pandas as pd

frame = pd.read_parquet("data/train_v3.parquet")
dataset = mlflow.data.from_pandas(frame, source="s3://bucket/data/train_v3.parquet", name="churn-train", targets="label")

with mlflow.start_run():
    mlflow.log_input(dataset, context="training")
    mlflow.log_params({"seed": 42, "feature_set": "v3"})
    ...

MLflow Projects: a run as one command

A Project is a directory with a declared entry point and environment, so a run becomes a command anyone can execute, including MLflow itself, straight from a git URL.

name: churn

python_env: python_env.yaml

entry_points:
  main:
    parameters:
      max_depth: {type: int, default: 5}
      n_estimators: {type: int, default: 200}
      data_version: {type: string, default: "v3"}
    command: >
      python train.py --max-depth {max_depth}
                      --n-estimators {n_estimators}
                      --data-version {data_version}

  evaluate:
    parameters:
      run_id: {type: string}
    command: "python evaluate.py --run-id {run_id}"

python: "3.11"
build_dependencies:
  - pip==24.2
dependencies:
  - -r requirements.txt

# Run locally, with overrides
mlflow run . -P max_depth=8 -P n_estimators=400

# Run a specific entry point
mlflow run . -e evaluate -P run_id=a1b2c3d4

# Run straight from git, at a pinned commit
mlflow run https://github.com/org/repo.git --version 3f9a1c2 -P max_depth=8

# Skip environment creation when you know the current env is right
mlflow run . --env-manager local

A run launched this way gets mlflow.source.type = PROJECT and records the entry point and its parameters, so the run itself tells you the command that produced it.

Reading a failed run, in order

Debugging MLflow has an order, and following it beats guessing.

1. Check the run's status and whether it exists at allA run stuck in RUNNING means the process died without closing it. No run at all means the tracking URI was wrong and you wrote to a local mlruns/ somewhere.
2. Confirm the tracking URI and experimentmlflow.get_tracking_uri() and mlflow.get_experiment_by_name(...). Runs landing in the wrong place is the most common "MLflow is broken".
3. Read the run's tagsSource file, git commit, and user. A surprising commit explains a lot.
4. Read the params, not your codeEspecially with autolog, the recorded parameters are what ran.
5. Open the model's MLmodel and requirements.txtServing and loading failures are nearly always environment mismatches, and this is where the truth is.
6. Reload the model in a clean environmentmlflow models predict -m ... --env-manager virtualenv reproduces the serving path without standing up a server.

Six failures cover most of what you will hit at this level:

import mlflow
from mlflow import MlflowClient

print(mlflow.get_tracking_uri())
client = MlflowClient()

run = client.get_run("a1b2c3d4e5f64718b9c0d1e2f3a4b5c6")
print(run.info.status, run.info.artifact_uri)
print(run.data.params)
print(run.data.metrics)
print({k: v for k, v in run.data.tags.items() if k.startswith("mlflow.")})
print([f.path for f in client.list_artifacts(run.info.run_id)])
client.set_terminated(run.info.run_id, status="FAILED")   # close a stuck run

Putting it all together

Everything above in one project. Nothing here is new. Read it as a whole and you should be able to justify every line.

.
├── MLproject                  # declared entry points and parameters
├── python_env.yaml            # the environment MLflow builds
├── requirements.txt           # pinned
├── data/
│   └── train_v3.parquet
├── src/
│   ├── train.py               # the tracked run
│   ├── evaluate.py            # mlflow.evaluate against a registered version
│   ├── promote.py             # moves the champion alias
│   └── batch_score.py         # loads models:/name@champion
└── README.md                  # the three commands to reproduce

import argparse
import mlflow
import mlflow.sklearn
import pandas as pd
from mlflow.models import infer_signature
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

ap = argparse.ArgumentParser()
ap.add_argument("--max-depth", type=int, default=5)
ap.add_argument("--n-estimators", type=int, default=200)
ap.add_argument("--data-version", default="v3")
ap.add_argument("--seed", type=int, default=42)
args = ap.parse_args()

mlflow.set_experiment("churn")                       # 1. named experiment
mlflow.sklearn.autolog(log_models=False, silent=True) # 2. autolog params/metrics only

frame = pd.read_parquet(f"data/train_{args.data_version}.parquet")
dataset = mlflow.data.from_pandas(                   # 3. the run names its data
    frame, source=f"data/train_{args.data_version}.parquet",
    name=f"churn-train-{args.data_version}", targets="label",
)

X = frame.drop(columns=["label"])
y = frame["label"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=args.seed, stratify=y,
)

run_name = f"rf d={args.max_depth} n={args.n_estimators} data={args.data_version}"
with mlflow.start_run(run_name=run_name) as run:      # 4. context manager, named
    mlflow.log_input(dataset, context="training")
    mlflow.log_params({"seed": args.seed, "data_version": args.data_version})
    mlflow.set_tags({"team": "risk", "purpose": "baseline"})   # 5. tags in code

    model = RandomForestClassifier(
        max_depth=args.max_depth, n_estimators=args.n_estimators,
        random_state=args.seed,
    ).fit(X_train, y_train)

    signature = infer_signature(X_train, model.predict(X_train))   # 6. a contract
    info = mlflow.sklearn.log_model(                               # 7. registered
        model, name="model", signature=signature,
        input_example=X_train.head(3),
        registered_model_name="churn-classifier",
    )

    eval_data = X_test.copy()
    eval_data["label"] = y_test
    result = mlflow.evaluate(                                      # 8. one evaluator
        model=info.model_uri, data=eval_data, targets="label",
        model_type="classifier",
    )
    print(run.info.run_id, result.metrics["roc_auc"])

import mlflow
from mlflow import MlflowClient

NAME, MARGIN = "churn-classifier", 0.002
client = MlflowClient()

def auc_of(version):
    return client.get_run(version.run_id).data.metrics.get("roc_auc", 0.0)

versions = client.search_model_versions(f"name = '{NAME}'")
best = max(versions, key=auc_of)

try:
    current = client.get_model_version_by_alias(NAME, "champion")
except mlflow.exceptions.MlflowException:
    current = None

if current is None or auc_of(best) > auc_of(current) + MARGIN:
    client.set_registered_model_alias(NAME, "champion", best.version)  # 9. alias move
    client.set_model_version_tag(NAME, best.version, "promoted_auc", f"{auc_of(best):.4f}")
    print("promoted version", best.version)
else:
    print("no promotion: improvement below margin")

# One-time setup
pip install mlflow
mlflow server --backend-store-uri sqlite:///mlflow.db \
              --default-artifact-root ./mlartifacts --port 5000
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000

# Everyday loop
mlflow run . -P max_depth=8 -P n_estimators=400     # 10. reproducible by command
python src/promote.py                                # threshold, then move the alias
python src/batch_score.py                            # consumes models:/…@champion

Ten decisions in there are the whole lesson of this page:

What you can now do, and what comes next

You can track runs with parameters, metrics, tags, and artifacts; autolog whole frameworks in one line; log models in a portable format with a signature and an environment; compare dozens of runs in the UI and in Python; register versions and move aliases so deployment is a pointer change; evaluate models consistently with built-in and custom metrics; serve a model locally, in a batch job, or as a container; declare a Project so a run is one reproducible command; and debug a failed run in a fixed order. That is a working practitioner's toolkit, enough to own the experiment-tracking and model-delivery story on a real project.

Mid-level takes every one of those topics further: nested runs and sweep structure, pyfunc custom models and wrappers, model dependencies and environment management in depth, dataset tracking and lineage, the registry as a promotion workflow with webhooks and CI, mlflow.evaluate with custom metrics and validation thresholds, MLflow Tracing and LLM evaluation, deployment targets and the deployments API, autologging internals, the client API and bulk operations, and the CI patterns that make a training run a pull-request check.

Senior then covers what you own when MLflow is your team's platform: the self-hosted deployment and its stores, authentication and multi-tenancy, artifact-store access control and credential handling, backend database scaling and what breaks first, backup and upgrade procedure, cost and retention policy for artifacts, lineage and audit for regulated work, promotion gates and approval trails, incident playbooks, and where MLflow stops and a feature store, an orchestrator, or a dedicated serving platform begins.

Part one of three. A fast review of everything in the Beginner Detailed track, in about twenty-five minutes. Fast review first, common questions at the end. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

MLflow is an open-source platform for the machine-learning lifecycle, in four parts: Tracking records runs with parameters, metrics, tags, and artifacts; Models packages a trained object in a framework-agnostic format with a declared signature and environment; the Model Registry gives that model a name, numbered versions, and moveable aliases; and Projects declares an entry point so a run is reproducible by command. The key idea is the model format: a logged model is a directory with a manifest, not a pickle from someone's laptop, so a batch job, a REST server, or a container can load it without knowing the framework.

Then add the sentence that shows you have used it: "the part people underestimate is aliases: consumers load models:/name@champion, so promoting a model is moving a pointer rather than editing and redeploying code."

The four components

The two stores

mlflow server --backend-store-uri postgresql://user:pass@host/mlflow \
              --default-artifact-root s3://bucket/mlflow --host 0.0.0.0 --port 5000

Vocabulary

The four things you log

mlflow.log_params({"max_depth": 5, "seed": 42})
mlflow.log_metric("val_auc", 0.94, step=epoch)
mlflow.set_tags({"team": "risk", "dataset": "v3"})
mlflow.log_artifact("outputs/roc.png")
mlflow.log_figure(fig, "plots/roc.png")
mlflow.log_table(preds_df, "predictions.json")

Two rules resolve nearly every question: a number you might plot is a metric; something you might change your mind about after the run is a tag.

The everyday code

import mlflow, mlflow.sklearn
from mlflow.models import infer_signature

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("churn")
mlflow.sklearn.autolog(log_models=False, silent=True)

with mlflow.start_run(run_name="rf d=5 feats=v3") as run:
    model.fit(X_train, y_train)                                # params/metrics autologged
    mlflow.log_metric("expected_cost", cost)                   # yours
    signature = infer_signature(X_train, model.predict(X_train))
    info = mlflow.sklearn.log_model(
        model, name="model", signature=signature,
        input_example=X_train.head(3),
        registered_model_name="churn-classifier",
    )
    mlflow.evaluate(model=info.model_uri, data=eval_df,
                    targets="label", model_type="classifier")

Autologging

mlflow.autolog()              # everything detectable
mlflow.sklearn.autolog(max_tuning_runs=10, log_models=True, log_datasets=True, silent=True)

Call autolog before the estimator is created. It patches the framework, so ordering matters, and a late call fails silently rather than loudly.

The MLflow Model format

├── MLmodel            # manifest: flavours, signature, python_version
├── model.pkl
├── conda.yaml
├── python_env.yaml
└── requirements.txt

flavors:
  python_function: {loader_module: mlflow.sklearn, model_path: model.pkl}
  sklearn: {pickled_model: model.pkl, sklearn_version: 1.5.1}
signature:
  inputs: '[{"name": "mean radius", "type": "double"}]'
  outputs: '[{"type": "long"}]'
run_id: a1b2c3d4e5f64718

sk = mlflow.sklearn.load_model("runs:/a1b2c3d4/model")     # the real object
pf = mlflow.pyfunc.load_model("models:/churn@champion")     # just predict()

The Model Registry

from mlflow import MlflowClient
client = MlflowClient()
client.set_registered_model_alias("churn-classifier", "champion", version=7)
model = mlflow.pyfunc.load_model("models:/churn-classifier@champion")

Stages are deprecated. None/Staging/Production/Archived with transition_model_version_stage appears in older code and tutorials; aliases replaced them because they are arbitrary, multiple, and not a fixed vocabulary. Recognise stages, write aliases.

Serving

mlflow models serve -m "models:/churn-classifier@champion" --port 5001 --env-manager virtualenv
mlflow models build-docker -m "models:/churn-classifier@champion" -n churn-scorer
mlflow models predict -m "runs:/abc/model" -i input.json          # no server needed

{"dataframe_split": {"columns": ["mean radius"], "data": [[17.99]]}}

Evaluation and comparison

mlflow.evaluate(model=info.model_uri, data=eval_df, targets="label", model_type="classifier")

runs = mlflow.search_runs(
    experiment_names=["churn"],
    filter_string="metrics.roc_auc > 0.9 and tags.dataset = 'v3'",
    order_by=["metrics.roc_auc DESC"],
)

metrics.auc > 0.9
params.max_depth = '5'
tags.dataset = 'v3' and metrics.auc > 0.88
attributes.status = 'FINISHED'

search_runs returns a DataFrame with flattened params.*, metrics.*, and tags.* columns, which is what makes "find the best run and act on it" three lines.

Reproducibility: what is and is not recorded

Projects

name: churn
python_env: python_env.yaml
entry_points:
  main:
    parameters:
      max_depth: {type: int, default: 5}
    command: "python train.py --max-depth {max_depth}"

mlflow run . -P max_depth=8
mlflow run https://github.com/org/repo.git --version 3f9a1c2 -P max_depth=8
mlflow run . --env-manager local

A Project run gets mlflow.source.type = PROJECT and records its entry point and parameters, so the run states the command that produced it.

The traps, and why they happen

Three ideas explain nearly every beginner failure: the tracking URI decides where runs go, params are immutable while tags are not, and a model's portability comes from its logged environment.

Common interview questions

1. What is MLflow and what problem does it solve?An open-source platform for the ML lifecycle with four components: Tracking for runs, Models for framework-agnostic packaging, a Model Registry for named versions and aliases, and Projects for reproducible entry points. The problem it solves is that a trained model is normally a file on somebody's laptop with no record of what produced it and no declared way to load it. MLflow turns it into an addressable artifact with parameters, metrics, a signature, and an environment.
2. What are the two stores, and why does the distinction matter?The backend store holds run metadata (params, metrics, tags, experiment and registry records) and the artifact store holds files, including model directories. They are configured separately, which is why you can have runs appearing in the UI with unreachable artifacts: the client could write metadata but not the files, or cannot read the artifact root.
3. Why does the Model Registry require a database backend?The default file-based store implements tracking only; the registry needs relational storage for models, versions, aliases, and their transitions. Practically it means --backend-store-uri pointing at SQLite for a single user or Postgres/MySQL for a team, and it is the first infrastructure decision MLflow forces on you.
4. Explain the difference between a parameter, a metric, and a tag.A parameter is immutable configuration stored as a string, and logging it twice with a different value raises. A metric is a float with a step, so it is a series you can plot. A tag is a mutable string for organisation and search. The rule of thumb: a number you might plot is a metric, something you might reclassify later is a tag.
5. What does log_model write?Not a pickle, but a directory containing an MLmodel manifest that declares the flavours it can be loaded as, the signature, and the Python version, plus the serialised model and three environment files. That manifest is what lets a REST server, a Spark UDF, or a Docker build load the model without knowing scikit-learn or PyTorch.
6. What is a flavour, and what does pyfunc give you?A flavour is a declared loader. A model usually has a native flavour, which returns the real framework object with all its methods, and the python_function flavour, which returns a generic object with just predict. pyfunc is the one deployment targets use, because it means the serving layer needs no framework-specific code.
7. Why log a signature?It declares the input columns and types and the output type, so a wrong-shaped or wrong-typed request fails at the boundary instead of producing confidently wrong predictions. Without it, a DataFrame with the columns in a different order will still get scored. infer_signature is one line, and input_example lets MLflow infer it for you.
8. Walk me through the model URI schemes.runs:/<run_id>/model for a model logged by a specific run, models:/<name>/<version> for a pinned registry version, models:/<name>@alias for whatever version currently holds that alias, and plain storage URIs for a directory. Production consumers should use the alias form, so promotion does not touch code.
9. Why did aliases replace stages?Stages were a fixed vocabulary (None, Staging, Production, Archived) and a version could hold exactly one. Aliases are arbitrary names, several can exist at once, and they map onto real patterns like champion/challenger or per-region pointers. You will still see transition_model_version_stage in older code; recognise it and write aliases in new work.
10. How does promotion become a deployment?Consumers load models:/name@champion, so moving the alias to a new version changes what they load with no code change and no redeploy. That makes the promotion decision itself the interesting part: it should be a script with a threshold and a log, not a person clicking, so there is a record of what was compared.
11. What does autologging cover, and what does it miss?It patches the framework to log constructor parameters, training metrics, the fitted model with an inferred signature, framework artifacts, and child runs for hyperparameter searches. It cannot know your business metrics, your own artifacts, or which of your variables mattered, so the realistic pattern is autolog plus a handful of deliberate calls. It must be enabled before the estimator is created, or it silently does nothing.
12. What is mlflow.evaluate for?It computes a consistent metric and plot set for a task type (classifier, regressor, or text) against a dataset, and logs them to the run. The value is comparability: hand-rolled metrics drift between runs, so a leaderboard built on them is untrustworthy. You can add custom metrics with make_metric, which is how a business number like expected cost gets into the same sortable table.
13. How would you find the best run programmatically?mlflow.search_runs with a filter string and order_by, which returns a pandas DataFrame with flattened params.*, metrics.*, and tags.* columns. The filter syntax is the same as the UI search box, so a query you built by clicking is directly reusable in a promotion gate.
14. What does MLflow not record?It does not store a diff of your working tree, only the commit, so a run from a dirty tree is not reproducible from that commit and nothing warns you. It does not know your data unless you call log_input or log a version. It does not capture system packages or CUDA, only pip. It does not set seeds for you.
15. How do you serve a model, and what are the options?mlflow models serve for a local REST endpoint, build-docker for a container to hand to a platform team, or pyfunc.load_model inside a batch job, which is the most common in practice. The --env-manager flag decides whether the environment is rebuilt from the logged requirements or reused from your current one; the default rebuild is what makes the model portable.
16. Serving fails with an import or version error. What is happening?The environment loading the model differs from the one that trained it. That is what the logged requirements.txt and python_env.yaml exist for, so the fix is to let the default virtualenv manager build from them rather than using --env-manager local. It is also the argument for pinning versions at training time rather than after a failure.
17. What is an MLflow Project?A directory with an MLproject file declaring named entry points, typed parameters with defaults, and an environment. mlflow run can then execute it locally or straight from a git URL at a pinned commit, building the environment first. The run records the entry point and parameters, so it states the command that produced it.
18. My runs are not showing up in the UI. Where do you look first?The tracking URI. With none set, MLflow writes to a local ./mlruns directory, so two people "using MLflow" can be writing to two different folders. Check mlflow.get_tracking_uri() and the experiment name before anything else. This is the most common false alarm.
19. How do you organise hundreds of runs?One experiment per problem, a descriptive run_name set in code, and tags for everything cross-cutting: team, dataset version, purpose, ticket. Then the searchable columns and the filter syntax do the work. The run you forget to name is always the one you need to find later, which is why it belongs in the script rather than in the UI.
20. How is MLflow different from a tool like ClearML or W&B?MLflow is deliberately narrow and unopinionated: it tracks, packages, registers, and serves, and it does not schedule or execute anything. There is no agent and no queue, so remote execution is your orchestrator's job. That is a strength when you already run Airflow or Kubernetes and want a standard model format, and a gap when you wanted tracking and execution in one system.

Sixty-second self-test

• Give the thirty-second answer, then the sentence that shows you have used it.
• Name the four components and what each one answers.
• Name the two stores and say which one a metric goes to versus a plot.
• Say why the Model Registry needs a database backend.
• Distinguish param, metric, and tag, including which is immutable.
• Say what log_model writes and name three files in it.
• Explain a flavour, and what pyfunc buys a deployment target.
• Say what a signature prevents, and how you produce one.
• List the four model URI schemes.
• Explain why aliases replaced stages, and how promotion reaches a consumer.
• Say what autologging misses, and the ordering rule it depends on.
• Name the three things MLflow does not record, starting with the diff.
• State the first thing to check when runs are missing from the UI.

Part one of three. Almost every beginner problem with MLflow comes from one of three things: the tracking URI pointing somewhere you did not expect, a model logged without a signature or a usable environment, or the difference between immutable params and mutable tags. Start with the error table, then work through the habits and practice cards underneath it.

Common errors at this level

The habits that pay off most

Practice cards

Short, self-contained exercises. Each one takes a few minutes and leaves you with a fact you will not forget.

1. Find the phantom mlrunsUnset the tracking URI, run a script, then find the local mlruns/ it created. This is the single most common MLflow confusion, and causing it once inoculates you.
2. Break param immutabilitylog_param("lr", 0.1) then log_param("lr", 0.2). Read the error, then do the same with set_tag and see it overwrite.
3. Watch a metric become a curveLog a metric once without a step, then in a loop with step=i. Compare the Metrics tab.
4. Read an MLmodel fileLog a model and open the manifest. Identify the flavours, the signature, and the Python version. This file is the design.
5. Prove what a signature doesLog one model with and one without, then feed both a DataFrame with reordered columns. One refuses, one lies.
6. Move an alias, change a deploymentRegister two versions, point champion at one, load by alias, then move the alias and rerun the identical code.
7. Serve without a servermlflow models predict -m runs:/…/model -i input.json --env-manager virtualenv. Watch it build the environment from the logged requirements.
8. Autolog a grid searchRun a six-candidate GridSearchCV under autolog and look at the parent's child runs.
9. Query your own historymlflow.search_runs with a filter on a metric and a tag. Params compare as quoted strings.
10. Reproduce a run from what MLflow recordedPick an old run and try to rebuild it from its tags and params alone. Write down what was missing.

Debugging order

Follow this rather than guessing, since the first two steps answer most problems.

1. Print the tracking URI and experimentmlflow.get_tracking_uri() and mlflow.get_experiment_by_name(...). If the run is not where you expected, nothing else matters.
2. Check the run's status and artifact URIclient.get_run(id).info. A RUNNING status means the process died; an unreachable artifact_uri explains missing files.
3. Read the params, not your codeUnder autolog especially, the recorded params are what ran.
4. Read the mlflow.* tagsSource file, git commit, user, run type. A surprising commit explains a surprising result.
5. Open the model's MLmodel and requirements.txtEvery load and serve failure is answered here. Compare against the environment doing the loading.
6. Reproduce the load path without a servermlflow models predict --env-manager virtualenv exercises what serving would do, with a traceback you can read.

import mlflow
from mlflow import MlflowClient

print("uri:", mlflow.get_tracking_uri())
client = MlflowClient()

run = client.get_run("a1b2c3d4e5f64718b9c0d1e2f3a4b5c6")
print(run.info.status, run.info.artifact_uri)
print("params:", run.data.params)
print("metrics:", run.data.metrics)
print("mlflow tags:", {k: v for k, v in run.data.tags.items() if k.startswith("mlflow.")})
print("artifacts:", [f.path for f in client.list_artifacts(run.info.run_id)])

# Full metric history, not just the last value
for point in client.get_metric_history(run.info.run_id, "val_auc"):
    print(point.step, point.value)

client.set_terminated(run.info.run_id, status="FAILED")   # close a stuck run

start_run options worth knowing on day one

Most defaults are right. These five are the ones you will reach for.

with mlflow.start_run(
    run_name="rf d=5 feats=v3",          # searchable, human-readable
    experiment_id=None,                   # or set_experiment() beforehand
    nested=False,                         # True for a child under an active run
    tags={"team": "risk", "purpose": "baseline"},
    description="Baseline for the Q3 review.",
) as run:
    ...

with mlflow.start_run(run_id="a1b2c3d4e5f64718") as run:
    mlflow.log_metric("holdout_auc", 0.918)
    mlflow.log_artifact("outputs/holdout_report.html")

Param, metric, tag, artifact: the decision

Half of all beginner MLflow confusion lives here, and two questions resolve it: is it a number I might plot? and might I change my mind about it after the run?

Tags are the third axis and the most underused: mutable, searchable labels that let you reclassify runs after the fact. broken, paper-v2, keep, dataset=v3. These are what make old runs findable.

Artifacts are everything that is a file. The rule: if a reviewer would want to look at it, log it as an artifact rather than describing it in a metric.

Set up so MLflow catches things for you

Do these four things once and an entire class of mistakes disappears.

# 1. One tracking URI for the whole machine — never per-script again
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
# and in .bashrc / .zshrc, or a project .env your shell loads

# 2. A server with a real backend, so the registry works from day one
mlflow server --backend-store-uri sqlite:///mlflow.db \
              --default-artifact-root ./mlartifacts \
              --host 127.0.0.1 --port 5000

# 3. Pin MLflow, so an upgrade never surprises a teammate mid-sprint
echo "mlflow==2.16.0" >> requirements.txt

# 4. Confirm what the client thinks it is talking to
python -c "import mlflow; print(mlflow.get_tracking_uri(), mlflow.__version__)"

mlruns/
mlartifacts/
mlflow.db
outputs/
*.pkl

Writing a tracked script that ages well

import argparse
import mlflow
import mlflow.sklearn
from mlflow.models import infer_signature

ap = argparse.ArgumentParser()                       # 1. every knob is a flag
ap.add_argument("--max-depth", type=int, default=5)
ap.add_argument("--data-version", default="v3")
ap.add_argument("--seed", type=int, default=42)
args = ap.parse_args()

mlflow.set_experiment("churn")                        # 2. never Default
mlflow.sklearn.autolog(log_models=False, silent=True) # 3. autolog before the estimator

run_name = f"rf d={args.max_depth} data={args.data_version}"
with mlflow.start_run(run_name=run_name,              # 4. named and tagged at creation
                      tags={"team": "risk", "dataset": args.data_version}) as run:
    mlflow.log_params({"seed": args.seed, "data_version": args.data_version})

    set_seed(args.seed)                               # 5. seeded, and the seed is logged
    model = train(args)

    signature = infer_signature(X_train, model.predict(X_train))   # 6. a contract
    info = mlflow.sklearn.log_model(
        model, name="model", signature=signature,
        input_example=X_train.head(3),
        registered_model_name="churn-classifier",     # 7. registered at log time
    )

    mlflow.evaluate(model=info.model_uri, data=eval_df,            # 8. one evaluator
                    targets="label", model_type="classifier")
    mlflow.log_artifact("outputs/segment_report.html")             # 9. show your work
    print("run:", run.info.run_id)

Reading the UI quickly

Learning to read these five places saves more time than any other single skill at this level.

mlflow.log_params({...})        → Parameters
mlflow.log_metric(..., step=i)  → Metrics  (a chart, because of the step)
mlflow.set_tags({...})          → Overview ▸ Tags, and the search index
mlflow.log_artifact("x.png")    → Artifacts
mlflow.log_figure(fig, "p.png") → Artifacts (no file written by you)
mlflow.log_input(dataset)       → Overview ▸ Datasets
mlflow.sklearn.log_model(...)   → Artifacts ▸ model/, and the registry if named
mlflow.evaluate(...)            → Metrics + Artifacts (plots and tables)
print(...)                      → your terminal, and nowhere sortable

Small things worth doing from day one

Name runs by what changed, not by number. rf d=8 feats=v3 beats run_47. The timestamp is already recorded; the intent is not.

One experiment per problem, not per person. churn with tagged runs beats churn-amina and churn-omar, because the whole point is comparing across people.

Log training time and model size as metrics. They become sortable columns, and they are the two numbers a reviewer asks for that nobody ever records.

Log the evaluation report as an artifact. A metric says 0.94; an artifact shows which segment it failed on.

Set and log the seed. Unexplained variance between two "identical" runs costs more time than any other problem in ML work.

Use log_figure and log_table instead of writing files. Fewer temp files, no cleanup, and the artifact lands where you meant it to.

Delete runs you know are junk, tag the rest. A table where every row is meaningful is a table people read.

mlflow.set_tags({"dataset": args.data_version, "purpose": "baseline", "ticket": "RISK-412"})
mlflow.log_metrics({"train_minutes": elapsed / 60, "model_mb": size_mb})

A starter setup worth keeping

Copy this into a new project and delete what you do not need. Every line is something from this page.

.
├── MLproject
├── python_env.yaml
├── requirements.txt          # includes a pinned mlflow==
├── .gitignore                # mlruns/, mlartifacts/, mlflow.db
├── src/
│   ├── train.py
│   ├── promote.py
│   └── batch_score.py
└── README.md                 # the three commands to reproduce

name: churn
python_env: python_env.yaml
entry_points:
  main:
    parameters:
      max_depth: {type: int, default: 5}
      data_version: {type: string, default: "v3"}
      seed: {type: int, default: 42}
    command: >
      python src/train.py --max-depth {max_depth}
                          --data-version {data_version}
                          --seed {seed}

import mlflow
from mlflow import MlflowClient

NAME, MARGIN, METRIC = "churn-classifier", 0.002, "roc_auc"
client = MlflowClient()

def score(version):
    return client.get_run(version.run_id).data.metrics.get(METRIC, 0.0)

versions = client.search_model_versions(f"name = '{NAME}'")
best = max(versions, key=score)

try:
    current = client.get_model_version_by_alias(NAME, "champion")
except mlflow.exceptions.MlflowException:
    current = None

if current is None or score(best) > score(current) + MARGIN:
    client.set_registered_model_alias(NAME, "champion", best.version)
    client.set_model_version_tag(NAME, best.version, "promoted_metric", f"{score(best):.4f}")
    print("promoted version", best.version)
else:
    print(f"no promotion: {score(best):.4f} vs {score(current):.4f}")

# One-time setup
pip install -r requirements.txt
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
mlflow server --backend-store-uri sqlite:///mlflow.db \
              --default-artifact-root ./mlartifacts --port 5000

# Everyday loop
mlflow run . -P max_depth=8 -P data_version=v3
python src/promote.py
mlflow models serve -m "models:/churn-classifier@champion" --port 5001

Eight details in there are the whole lesson of this page: one tracking URI in the environment, a named experiment, autolog enabled before the estimator with log_models=False, a run named and tagged at creation, the seed and data version as params, a signature on every logged model, promotion by moving an alias with an explicit margin, and an MLproject so the run is one typed command.

Mid-level tips go deeper on every one of these: nested runs and sweep hygiene, custom pyfunc models for preprocessing that has to travel with the model, dependency pinning that holds, dataset tracking and lineage, promotion gates in CI, mlflow.evaluate validation thresholds, tracing and LLM evaluation, deployment targets, and diagnosing "loads locally, fails in serving".

This is part two of three. It picks up exactly where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Nothing is dropped and nothing is repeated for its own sake. Where you already know the basics, we go straight to the depth.

Where this picks up

Each section starts with the problem it solves, and ends with a Try it you can do on a real project in a few minutes.

Runs, nested runs, and the client API

Beginner used one run per script. Real work needs structure, and the client API is what turns MLflow from a logger into something you can automate.

import mlflow

with mlflow.start_run(run_name="sweep: max_depth") as parent:
    mlflow.set_tag("sweep", "max_depth")
    for depth in (3, 5, 8, 12):
        with mlflow.start_run(run_name=f"depth={depth}", nested=True) as child:
            mlflow.log_param("max_depth", depth)
            model = fit(max_depth=depth)
            mlflow.log_metric("val_auc", score(model))
    # The parent records the summary; the children record the detail.
    mlflow.log_metric("best_val_auc", best)
    mlflow.set_tag("best_child_run_id", best_run_id)

The parent/child relationship is stored as a tag, mlflow.parentRunId, which means it is queryable:

children = mlflow.search_runs(
    experiment_names=["churn"],
    filter_string=f"tags.mlflow.parentRunId = '{parent.info.run_id}'",
    order_by=["metrics.val_auc DESC"],
)

The fluent API (mlflow.log_*) always writes to the active run, which is convenient until it is wrong: in a multi-threaded sweep, or when writing to a run you are not inside. The client API takes an explicit run id and has no global state:

from mlflow import MlflowClient
client = MlflowClient()

run = client.create_run(experiment_id=exp_id, tags={"team": "risk"}, run_name="explicit")
client.log_param(run.info.run_id, "max_depth", 5)
client.log_metric(run.info.run_id, "val_auc", 0.94, step=1)
client.set_terminated(run.info.run_id, status="FINISHED")

from mlflow.entities import Metric, Param, RunTag
import time

now = int(time.time() * 1000)
client.log_batch(
    run_id=run_id,
    metrics=[Metric("loss", v, now, step=i) for i, v in enumerate(losses)],
    params=[Param("optimizer", "adamw")],
    tags=[RunTag("phase", "warmup")],
)

Autologging internals

Beginner said "one line". At this level you have to control what that line does.

Each integration installs patches around specific framework functions:

The options that matter, and every integration accepts them:

mlflow.sklearn.autolog(
    log_input_examples=False,     # avoid embedding data samples in the model
    log_model_signatures=True,    # keep signatures on
    log_models=False,             # you log the model yourself, with registration
    log_datasets=True,            # dataset lineage where it can be inferred
    max_tuning_runs=10,           # cap child runs from a search
    disable=False,
    exclusive=False,              # True: skip autolog when a run is already active
    disable_for_unsupported_versions=False,
    silent=True,                  # suppress the warning noise
    registered_model_name=None,   # register autologged models
)

Three behaviours worth knowing precisely:

Autolog creates a run if none is active. Call fit() outside a with block and MLflow starts a run for you and ends it when fit returns. That is convenient in a notebook and surprising in a script where you expected your own run.

exclusive=True means "do not interfere". With it, autologging skips logging when a run is already active, which is how you keep a hand-instrumented run from being polluted by framework noise.

log_input_examples=True embeds data in the artifact. Useful for documentation, and a data-leak vector if the training data is sensitive, because the example ships inside the model directory that you might publish.

Custom pyfunc models

The single most useful advanced feature in MLflow. A pyfunc model can wrap anything (preprocessing, several models, business rules, a call to another service) behind one predict, and it travels as one artifact with one environment.

The problem it solves: your model needs a fitted scaler, a vocabulary, and a threshold. Log only the estimator and every consumer must reimplement the rest, correctly, forever.

import mlflow
import pandas as pd

class ChurnModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        """Runs once at load time. Read anything from context.artifacts."""
        import joblib, json
        self.scaler = joblib.load(context.artifacts["scaler"])
        self.clf = joblib.load(context.artifacts["classifier"])
        with open(context.artifacts["config"]) as handle:
            self.threshold = json.load(handle)["threshold"]

    def predict(self, context, model_input: pd.DataFrame, params=None) -> pd.DataFrame:
        threshold = (params or {}).get("threshold", self.threshold)
        features = self.scaler.transform(model_input[self.FEATURES])
        proba = self.clf.predict_proba(features)[:, 1]
        return pd.DataFrame({"proba": proba, "label": (proba >= threshold).astype(int)})

    FEATURES = ["tenure", "monthly_charges", "support_calls"]

import mlflow
from mlflow.models import infer_signature, ModelSignature
from mlflow.types.schema import Schema, ColSpec, ParamSchema, ParamSpec

signature = ModelSignature(
    inputs=Schema([ColSpec("double", "tenure"),
                   ColSpec("double", "monthly_charges"),
                   ColSpec("long", "support_calls")]),
    outputs=Schema([ColSpec("double", "proba"), ColSpec("long", "label")]),
    params=ParamSchema([ParamSpec("threshold", "double", 0.5)]),
)

with mlflow.start_run(run_name="churn pyfunc"):
    mlflow.pyfunc.log_model(
        name="model",
        python_model=ChurnModel(),
        artifacts={
            "scaler": "outputs/scaler.pkl",
            "classifier": "outputs/clf.pkl",
            "config": "outputs/config.json",
        },
        code_paths=["src/features.py"],          # your own modules travel with it
        signature=signature,
        input_example=X_test.head(3),
        pip_requirements=["scikit-learn==1.5.1", "pandas==2.2.2", "joblib==1.4.2"],
        registered_model_name="churn-classifier",
    )

model = mlflow.pyfunc.load_model("models:/churn-classifier@champion")
model.predict(frame)                                  # uses the logged threshold
model.predict(frame, params={"threshold": 0.7})       # overridden at call time

Dependencies and environments, precisely

Beginner noted that a model carries a requirements.txt. At this level you need to know how that list is produced, because it is where portability succeeds or fails.

MLflow infers requirements at log time by introspecting the loaded modules and resolving them to installed distributions, then pins them:

mlflow==2.16.0
scikit-learn==1.5.1
cloudpickle==3.0.0
numpy==1.26.4
psutil==6.0.0

Three ways to take control, in increasing order of explicitness:

# 1. Extra pins on top of inference — for a module inference cannot see
mlflow.sklearn.log_model(model, name="model", extra_pip_requirements=["prometheus-client==0.20.0"])

# 2. Replace inference entirely — for custom pyfunc models, do this
mlflow.pyfunc.log_model(name="model", python_model=ChurnModel(),
                        pip_requirements=["scikit-learn==1.5.1", "pandas==2.2.2"])

# 3. A file, so the pins are reviewable in the repo
mlflow.pyfunc.log_model(name="model", python_model=ChurnModel(),
                        pip_requirements="requirements-serving.txt")

Two ways to check the result before it becomes a production problem:

# Validate that the model loads and scores in a freshly built environment
mlflow models predict -m "runs:/abc123/model" -i input.json \
  --content-type json --env-manager virtualenv

# Print the environment MLflow would build
python -c "import mlflow; print(mlflow.pyfunc.get_model_dependencies('runs:/abc123/model'))"

from mlflow.models import validate_serving_input, convert_input_example_to_serving_input

payload = convert_input_example_to_serving_input(X_test.head(3))
validate_serving_input("runs:/abc123/model", payload)     # raises if the contract breaks

mlflow.evaluate with thresholds and baselines

Beginner used the default evaluator. Its real power at this level is validation: failing a job when a model is not good enough, which is what turns an evaluation into a gate.

import mlflow
from mlflow.models import MetricThreshold

thresholds = {
    "roc_auc": MetricThreshold(threshold=0.85, greater_is_better=True),
    "log_loss": MetricThreshold(threshold=0.40, greater_is_better=False),
    # Relative to a baseline model, not just an absolute floor
    "f1_score": MetricThreshold(
        threshold=0.80,
        min_absolute_change=0.01,
        min_relative_change=0.01,
        greater_is_better=True,
    ),
}

result = mlflow.evaluate(
    model=candidate_uri,
    data=eval_data,
    targets="label",
    model_type="classifier",
    evaluators=["default"],
    validation_thresholds=thresholds,
    baseline_model=baseline_uri,      # required for relative-change thresholds
)

If any threshold fails, mlflow.evaluate raises ModelValidationFailedException, so in CI the job fails, with the failing metric named.

You can also evaluate a static dataset with predictions already computed, which is how you evaluate something MLflow did not train:

static = eval_data.assign(prediction=external_predictions)
mlflow.evaluate(
    data=static,
    targets="label",
    predictions="prediction",
    model_type="classifier",
)

Add domain metrics that the default evaluator cannot know about:

from mlflow.models import make_metric

def _cost(eval_df, _builtin):
    fp = ((eval_df["prediction"] == 1) & (eval_df["target"] == 0)).sum()
    fn = ((eval_df["prediction"] == 0) & (eval_df["target"] == 1)).sum()
    return float(50 * fp + 500 * fn)

def _fairness_gap(eval_df, _builtin):
    grouped = eval_df.groupby(eval_df["segment"])["prediction"].mean()
    return float(grouped.max() - grouped.min())

result = mlflow.evaluate(
    model=candidate_uri, data=eval_data, targets="label", model_type="classifier",
    extra_metrics=[
        make_metric(eval_fn=_cost, greater_is_better=False, name="expected_cost"),
        make_metric(eval_fn=_fairness_gap, greater_is_better=False, name="fairness_gap"),
    ],
    validation_thresholds={
        "expected_cost": MetricThreshold(threshold=20000, greater_is_better=False),
        "fairness_gap": MetricThreshold(threshold=0.08, greater_is_better=False),
    },
)

The registry as a promotion workflow

Beginner registered versions and moved an alias. Most of the operational value lives in the workflow around it.

import mlflow
from mlflow import MlflowClient
from mlflow.models import MetricThreshold

NAME = "churn-classifier"
client = MlflowClient()

def metric(version, key="roc_auc"):
    return client.get_run(version.run_id).data.metrics.get(key, 0.0)

candidates = client.search_model_versions(
    f"name = '{NAME}' and tags.validated = 'true'"
)
best = max(candidates, key=metric)

try:
    champion = client.get_model_version_by_alias(NAME, "champion")
except mlflow.exceptions.MlflowException:
    champion = None

# Compare on a held-out dataset, not on the training run's own metric
result = mlflow.evaluate(
    model=f"models:/{NAME}/{best.version}",
    data=holdout, targets="label", model_type="classifier",
    baseline_model=(f"models:/{NAME}/{champion.version}" if champion else None),
    validation_thresholds={
        "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.01,
                                   greater_is_better=True),
    },
)   # raises ModelValidationFailedException on failure

client.set_model_version_tag(NAME, best.version, "holdout_auc", f"{result.metrics['roc_auc']:.4f}")
client.set_model_version_tag(NAME, best.version, "approved_by", "ci")
if champion:
    client.set_registered_model_alias(NAME, "previous", champion.version)
client.set_registered_model_alias(NAME, "champion", best.version)

Searching the registry is the same filter language, which is what makes governance scriptable:

client.search_registered_models(filter_string="name LIKE 'churn%'")
client.search_model_versions(f"name = '{NAME}' and tags.approved_by = 'ci'")

Dataset tracking and lineage

Beginner called log_input once. The call records more than it proves.

import mlflow.data
import pandas as pd

frame = pd.read_parquet("s3://bucket/data/train_v3.parquet")

dataset = mlflow.data.from_pandas(
    frame,
    source="s3://bucket/data/train_v3.parquet",
    name="churn-train",
    targets="label",
    digest=None,          # computed for you: a content hash of the frame
)

with mlflow.start_run():
    mlflow.log_input(dataset, context="training", tags={"version": "v3"})
    mlflow.log_input(eval_dataset, context="evaluation")

What a logged dataset records: a name, a source URI, a digest (a content hash of the profiled data), a schema, and an optional profile such as row and column counts. What that gives you is a run that names its input and can be compared against another run's input by digest.

What it does not give you: immutability. The digest proves the frame you profiled, not that the object at s3://bucket/data/train_v3.parquet is unchanged today. If someone overwrites that path, the run's recorded source now points at different bytes and nothing in MLflow notices.

The pattern that closes most of the gap without another tool:

# Dated, immutable prefixes plus the digest recorded on the run
SOURCE = "s3://bucket/data/churn/dt=2026-08-01/train.parquet"   # never overwritten
dataset = mlflow.data.from_pandas(frame, source=SOURCE, name="churn-train", targets="label")

with mlflow.start_run():
    mlflow.log_input(dataset, context="training")
    mlflow.log_params({"data_partition": "dt=2026-08-01"})   # also a searchable param

Deployment targets and the deployments API

Beginner served locally. At this level deployment is a target, and MLflow abstracts several behind one interface.

# Built-in local scoring server
mlflow models serve -m "models:/churn-classifier@champion" --port 5001 --workers 4

# A container to hand to a platform team
mlflow models build-docker -m "models:/churn-classifier@champion" \
  -n churn-scorer --enable-mlserver

# Managed targets via plugins
mlflow deployments create -t sagemaker --name churn -m "models:/churn-classifier@champion" \
  -C region_name=eu-west-1 -C instance_type=ml.m5.large
mlflow deployments list -t sagemaker
mlflow deployments update -t sagemaker --name churn -m "models:/churn-classifier@champion"
mlflow deployments delete -t sagemaker --name churn

from mlflow.deployments import get_deploy_client

client = get_deploy_client("sagemaker")
client.create_deployment(name="churn", model_uri="models:/churn-classifier@champion",
                         config={"region_name": "eu-west-1"})
client.predict(deployment_name="churn", inputs=payload)

Batch scoring in Spark deserves its own mention, because it is where the pyfunc contract pays off most:

from pyspark.sql.functions import struct
import mlflow.pyfunc

predict = mlflow.pyfunc.spark_udf(spark, "models:/churn-classifier@champion", result_type="double")
scored = df.withColumn("score", predict(struct(*df.columns)))

Two serving internals worth knowing:

Workers matter. The default scoring server runs a single worker; --workers N uses more. For real throughput, --enable-mlserver swaps in MLServer, which supports adaptive batching and is the better base for a container.

The /invocations contract is fixed. Four payload shapes (dataframe_split, dataframe_records, instances, inputs) plus params for signature params. Anything calling your endpoint is coding against that contract, so a signature change is a breaking API change.

MLflow Tracing and LLM evaluation

Newer MLflow versions add tracing and GenAI evaluation, and they are increasingly what interviews ask about.

Tracing records the internal steps of a call (a chain, a retrieval, a tool use, a model call) as a tree of spans with inputs, outputs, latency, and token counts.

import mlflow

mlflow.set_experiment("rag-assistant")
mlflow.langchain.autolog()          # traces every chain invocation
# also: mlflow.openai.autolog(), mlflow.llama_index.autolog(), and others

@mlflow.trace(span_type="RETRIEVER")
def retrieve(question: str, k: int = 4):
    return vector_store.similarity_search(question, k=k)

@mlflow.trace(span_type="LLM")
def answer(question: str):
    docs = retrieve(question)
    return llm.invoke(build_prompt(question, docs))

with mlflow.start_run():
    answer("What is our refund window?")

with mlflow.start_span(name="rerank", span_type="RERANKER") as span:
    span.set_inputs({"candidates": len(docs)})
    ranked = reranker.rank(docs)
    span.set_outputs({"kept": len(ranked)})
    span.set_attribute("model", "bge-reranker-v2")

LLM evaluation extends mlflow.evaluate with text metrics, including LLM-as-a-judge:

from mlflow.metrics.genai import answer_correctness, faithfulness, answer_relevance

eval_df = pd.DataFrame({
    "inputs": questions,
    "ground_truth": references,
    "context": retrieved_contexts,
})

result = mlflow.evaluate(
    model=my_qa_function,                 # any callable, or a pyfunc URI
    data=eval_df,
    targets="ground_truth",
    model_type="question-answering",
    extra_metrics=[
        answer_correctness(model="openai:/gpt-4o-mini"),
        faithfulness(model="openai:/gpt-4o-mini"),
        answer_relevance(model="openai:/gpt-4o-mini"),
    ],
    evaluator_config={"col_mapping": {"context": "context"}},
)

MLflow in CI

The pattern worth building: a pull request that changes training code produces a tracked run, evaluates it against thresholds and the current champion, and fails the check if it regresses. That makes model quality a reviewable property of the diff.

name: train
on:
  pull_request:
    paths: ['src/**', 'MLproject', 'requirements.txt', 'python_env.yaml']

permissions:
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: pip

      - run: pip install -r requirements.txt

      - name: Train and gate
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
          MLFLOW_TRACKING_USERNAME: ${{ secrets.MLFLOW_TRACKING_USERNAME }}
          MLFLOW_TRACKING_PASSWORD: ${{ secrets.MLFLOW_TRACKING_PASSWORD }}
          AWS_ROLE_ARN: ${{ secrets.AWS_ROLE_ARN }}
        run: python ci/train_and_gate.py --experiment "churn/ci"

import argparse, os, sys
import mlflow
from mlflow import MlflowClient
from mlflow.models import MetricThreshold

ap = argparse.ArgumentParser()
ap.add_argument("--experiment", required=True)
ap.add_argument("--name", default="churn-classifier")
args = ap.parse_args()

sha = os.environ.get("GITHUB_SHA", "local")[:8]
pr = os.environ.get("GITHUB_REF_NAME", "local")

mlflow.set_experiment(args.experiment)
with mlflow.start_run(run_name=f"pr {pr} @ {sha}",
                      tags={"ci": "true", "pr": pr, "commit": sha}) as run:
    model_info = train_and_log()          # your training, returning the ModelInfo

    client = MlflowClient()
    try:
        champion = client.get_model_version_by_alias(args.name, "champion")
        baseline = f"models:/{args.name}/{champion.version}"
    except mlflow.exceptions.MlflowException:
        baseline = None

    try:
        result = mlflow.evaluate(
            model=model_info.model_uri,
            data=load_holdout(), targets="label", model_type="classifier",
            baseline_model=baseline,
            validation_thresholds={
                "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.005,
                                           greater_is_better=True),
                "expected_cost": MetricThreshold(threshold=20000, greater_is_better=False),
            },
        )
    except mlflow.exceptions.MlflowException as exc:
        print(f"::error::validation failed — {exc}")
        sys.exit(1)

    url = f"{mlflow.get_tracking_uri()}/#/experiments/{run.info.experiment_id}/runs/{run.info.run_id}"
    print(f"::notice::roc_auc={result.metrics['roc_auc']:.4f} — {url}")

Debugging, one level deeper

Beginner had a tab order. When the UI runs out, work through these.

1. Separate the two stores in your headMetadata failures are backend-store problems (database, permissions, migrations). Missing files are artifact-store problems (credentials, URI, network). The symptoms are different.
2. Read the artifact URI on the run, not the configrun.info.artifact_uri is what that run used. A run created before a config change points at the old location forever.
3. Reproduce the serving path, not your notebookmlflow models predict --env-manager virtualenv exercises what a container would do, with a readable traceback.
4. Check the client and server versionsA newer client can send fields an older server rejects. mlflow.__version__ against the server's footer.
5. Turn up loggingMLFLOW_TRACKING_URI plus Python logging at DEBUG on mlflow shows every REST call, which is how you diagnose a hang or a 403.
6. Compare a good run against a bad one from the clientParams, tags, and the model's requirements, and the diff is usually one line long.

export MLFLOW_TRACKING_URI=http://mlflow.internal:5000
python - <<'PY'
import logging, mlflow
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("mlflow").setLevel(logging.DEBUG)
print(mlflow.get_tracking_uri(), mlflow.__version__)
print(mlflow.search_experiments()[:3])
PY

# Is the server even there, and what version?
curl -s "$MLFLOW_TRACKING_URI/health"
curl -s "$MLFLOW_TRACKING_URI/api/2.0/mlflow/experiments/search?max_results=1"

from mlflow import MlflowClient
client = MlflowClient()

good, bad = client.get_run("…good…"), client.get_run("…bad…")
for key in sorted(set(good.data.params) | set(bad.data.params)):
    if good.data.params.get(key) != bad.data.params.get(key):
        print(f"param {key}: {good.data.params.get(key)!r} → {bad.data.params.get(key)!r}")

print("commit:", good.data.tags.get("mlflow.source.git.commit"),
      "→", bad.data.tags.get("mlflow.source.git.commit"))
print("artifacts:", good.info.artifact_uri, "→", bad.info.artifact_uri)

Putting it all together

A complete project using everything on this page. Nothing here is new. Read it as a whole and you should be able to justify every line.

.
├── .github/workflows/train.yml     # PR gate: run, evaluate, threshold
├── MLproject                       # entry points: main, evaluate, promote
├── python_env.yaml
├── requirements.txt                # pinned, includes mlflow==
├── requirements-serving.txt        # the smaller explicit serving env
├── src/
│   ├── features.py                 # travels with the model via code_paths
│   ├── model.py                    # the custom pyfunc wrapper
│   ├── train.py                    # nested runs, autolog + explicit log_model
│   ├── evaluate.py                 # thresholds against the champion baseline
│   ├── promote.py                  # tags, previous, champion
│   └── batch_score.py              # spark_udf or pyfunc.load_model
└── ci/train_and_gate.py

import mlflow
import pandas as pd
from src.features import build_features        # copied in via code_paths

class ChurnModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        import joblib, json
        self.scaler = joblib.load(context.artifacts["scaler"])
        self.clf = joblib.load(context.artifacts["classifier"])
        with open(context.artifacts["config"]) as handle:
            self.threshold = json.load(handle)["threshold"]

    def predict(self, context, model_input: pd.DataFrame, params=None) -> pd.DataFrame:
        threshold = (params or {}).get("threshold", self.threshold)
        features = self.scaler.transform(build_features(model_input))
        proba = self.clf.predict_proba(features)[:, 1]
        return pd.DataFrame({"proba": proba, "label": (proba >= threshold).astype(int)})

import argparse
import mlflow
import mlflow.data
from mlflow.models import ModelSignature
from mlflow.types.schema import Schema, ColSpec, ParamSchema, ParamSpec
from src.model import ChurnModel

ap = argparse.ArgumentParser()
ap.add_argument("--depths", type=str, default="3,5,8,12")
ap.add_argument("--data-partition", default="dt=2026-08-01")
ap.add_argument("--seed", type=int, default=42)
args = ap.parse_args()

mlflow.set_experiment("churn")
mlflow.sklearn.autolog(log_models=False, log_input_examples=False, silent=True)  # 1

SOURCE = f"s3://bucket/data/churn/{args.data_partition}/train.parquet"           # 2
frame = read_parquet(SOURCE)
dataset = mlflow.data.from_pandas(frame, source=SOURCE, name="churn-train", targets="label")

signature = ModelSignature(                                                       # 3
    inputs=Schema([ColSpec("double", "tenure"),
                   ColSpec("double", "monthly_charges"),
                   ColSpec("long", "support_calls")]),
    outputs=Schema([ColSpec("double", "proba"), ColSpec("long", "label")]),
    params=ParamSchema([ParamSpec("threshold", "double", 0.5)]),
)

with mlflow.start_run(run_name=f"sweep {args.data_partition}",                    # 4
                      tags={"team": "risk", "sweep": "max_depth"}) as parent:
    mlflow.log_input(dataset, context="training")
    mlflow.log_params({"seed": args.seed, "data_partition": args.data_partition})

    best = None
    for depth in [int(d) for d in args.depths.split(",")]:
        with mlflow.start_run(run_name=f"depth={depth}", nested=True) as child:    # 5
            model, metrics = fit(frame, max_depth=depth, seed=args.seed)
            mlflow.log_metrics(metrics)                                            # 6 batch
            if best is None or metrics["val_auc"] > best[1]:
                best = (child.info.run_id, metrics["val_auc"], model)

    run_id, auc, model = best
    mlflow.log_metric("best_val_auc", auc)
    mlflow.set_tag("best_child_run_id", run_id)

    info = mlflow.pyfunc.log_model(                                               # 7
        name="model",
        python_model=ChurnModel(),
        artifacts={"scaler": "outputs/scaler.pkl",
                   "classifier": "outputs/clf.pkl",
                   "config": "outputs/config.json"},
        code_paths=["src/features.py", "src/model.py"],                            # 8
        signature=signature,
        input_example=frame.head(3).drop(columns=["label"]),
        pip_requirements="requirements-serving.txt",                               # 9
        registered_model_name="churn-classifier",
    )
    print(info.model_uri)

import mlflow
from mlflow import MlflowClient
from mlflow.models import MetricThreshold, make_metric

NAME = "churn-classifier"
client = MlflowClient()
version = client.get_latest_versions(NAME)[0]           # or a passed-in version

def _cost(eval_df, _builtin):
    fp = ((eval_df["prediction"] == 1) & (eval_df["target"] == 0)).sum()
    fn = ((eval_df["prediction"] == 0) & (eval_df["target"] == 1)).sum()
    return float(50 * fp + 500 * fn)

try:
    champion = client.get_model_version_by_alias(NAME, "champion")
    baseline = f"models:/{NAME}/{champion.version}"
except mlflow.exceptions.MlflowException:
    baseline = None

with mlflow.start_run(run_name=f"validate v{version.version}", tags={"purpose": "validation"}):
    result = mlflow.evaluate(                                                     # 10
        model=f"models:/{NAME}/{version.version}",
        data=load_holdout(), targets="label", model_type="classifier",
        baseline_model=baseline,
        extra_metrics=[make_metric(eval_fn=_cost, greater_is_better=False, name="expected_cost")],
        validation_thresholds={
            "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.005,
                                       greater_is_better=True),
            "expected_cost": MetricThreshold(threshold=20000, greater_is_better=False),
        },
    )

client.set_model_version_tag(NAME, version.version, "validated", "true")          # 11
client.set_model_version_tag(NAME, version.version, "holdout_auc", f"{result.metrics['roc_auc']:.4f}")

import mlflow
from mlflow import MlflowClient

NAME = "churn-classifier"
client = MlflowClient()

candidates = client.search_model_versions(f"name = '{NAME}' and tags.validated = 'true'")
best = max(candidates, key=lambda v: float(v.tags.get("holdout_auc", 0)))

try:
    champion = client.get_model_version_by_alias(NAME, "champion")
    client.set_registered_model_alias(NAME, "previous", champion.version)          # 12
except mlflow.exceptions.MlflowException:
    pass

client.set_registered_model_alias(NAME, "champion", best.version)
print("champion is now version", best.version)

# The everyday loop
mlflow run . -P depths=3,5,8,12 -P data_partition=dt=2026-08-01
mlflow run . -e evaluate
mlflow run . -e promote

# Validate the serving contract before anyone depends on it
mlflow models predict -m "models:/churn-classifier@champion" -i holdout.json \
  --content-type json --env-manager virtualenv

# Then serve, or hand over a container
mlflow models build-docker -m "models:/churn-classifier@champion" -n churn-scorer --enable-mlserver

Twelve decisions in there are the whole lesson of this page:

Where you are now

You can structure sweeps as nested runs and drive MLflow entirely from the client API, control what autologging does instead of accepting it, wrap arbitrary preprocessing and postprocessing into a portable custom pyfunc with its own code and dependencies, declare and enforce signatures including call-time params, pin and validate serving environments before they fail, gate promotions on thresholds relative to the current champion, run the registry as a workflow with validation tags and a rollback alias, record dataset lineage and state its limits, deploy to several targets including Spark and containers, trace and evaluate LLM applications, and gate a pull request on model quality.

Senior takes every one of those topics further: the self-hosted deployment and its backend and artifact stores, authentication and authorisation including proxied artifact access, multi-tenancy across teams, credential and access design, database scaling and what breaks first, backup and upgrade procedure, artifact cost and retention policy, lineage and audit that satisfies a reviewer, approval trails and separation of duties on promotion, incident playbooks, and where MLflow stops and a feature store, an orchestrator, or a dedicated serving platform begins.

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than by level, in about thirty-five minutes. Fast review first, common questions at the end. Senior reviews all three.

Foundations, in one screen

MLflow has four components: Tracking for runs, Models for framework-agnostic packaging, a Registry for named versions and moveable aliases, and Projects for reproducible entry points. The core idea is the model format (a directory with an MLmodel manifest declaring flavours, a signature, and an environment) which is why the same artifact loads in a batch job, a REST server, or a container.

Two stores: a backend store for metadata (--backend-store-uri) and an artifact store for files (--default-artifact-root). The registry needs a database backend. Params are immutable, metrics have steps, tags are mutable. Consume by alias, so promotion is a pointer move. MLflow records the commit, not a diff.

Runs, nesting, and the client API

with mlflow.start_run(run_name="sweep") as parent:
    for depth in (3, 5, 8):
        with mlflow.start_run(run_name=f"depth={depth}", nested=True):
            ...

Nesting is stored as the mlflow.parentRunId tag, so children are queryable: filter_string=f"tags.mlflow.parentRunId = '{parent_id}'".

Autologging internals

Integrations install patches around framework functions: sklearn on fit and search classes, Lightning on trainer hooks, xgboost/lightgbm on train, transformers on Trainer, langchain/openai on chain and client calls (which produces traces).

Three behaviours to state without prompting: autolog creates a run if none is active; it must be enabled before the estimator is created or it silently does nothing; and log_input_examples=True embeds training data in an artifact you might publish.

Custom pyfunc models

The most useful advanced feature: wrap preprocessing, several models, and business rules behind one predict, shipped as one artifact.

class ChurnModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        self.scaler = joblib.load(context.artifacts["scaler"])
        self.clf = joblib.load(context.artifacts["classifier"])

    def predict(self, context, model_input, params=None):
        t = (params or {}).get("threshold", 0.5)
        proba = self.clf.predict_proba(self.scaler.transform(model_input))[:, 1]
        return pd.DataFrame({"proba": proba, "label": (proba >= t).astype(int)})

mlflow.pyfunc.log_model(
    name="model", python_model=ChurnModel(),
    artifacts={"scaler": "outputs/scaler.pkl", "classifier": "outputs/clf.pkl"},
    code_paths=["src/features.py"],          # your modules travel with it
    signature=signature,                      # including a ParamSchema
    pip_requirements="requirements-serving.txt",
    registered_model_name="churn-classifier",
)

Dependencies and environments

Requirements are inferred at log time by introspecting loaded modules and pinning the resolved distributions.

Inference sees imports, not intent. A dynamic importlib call, or an import inside a branch that never ran, is invisible, and the model then fails to load elsewhere.

Evaluation with thresholds

mlflow.evaluate(
    model=candidate_uri, data=eval_df, targets="label", model_type="classifier",
    baseline_model=champion_uri,
    extra_metrics=[make_metric(eval_fn=_cost, greater_is_better=False, name="expected_cost")],
    validation_thresholds={
        "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.005, greater_is_better=True),
        "expected_cost": MetricThreshold(threshold=20000, greater_is_better=False),
    },
)   # raises ModelValidationFailedException on failure

The registry as a workflow

Stages (Staging/Production, transition_model_version_stage) are deprecated, so recognise them in old code and write aliases.

Dataset lineage

dataset = mlflow.data.from_pandas(frame, source=SOURCE, name="churn-train", targets="label")
mlflow.log_input(dataset, context="training")

Records a name, a source URI, a content digest, a schema, and a profile. Constructors exist for pandas, numpy, Spark/Delta, and HuggingFace.

Deployment targets

/invocations accepts dataframe_split, dataframe_records, instances, or inputs, plus params.

Tracing and LLM evaluation

mlflow.langchain.autolog()

@mlflow.trace(span_type="RETRIEVER")
def retrieve(q, k=4): ...

with mlflow.start_span(name="rerank", span_type="RERANKER") as span:
    span.set_inputs({"candidates": len(docs)})
    span.set_attribute("model", "bge-reranker-v2")

Two things to say without prompting: traces store prompts and responses verbatim, so they are data with retention and redaction obligations; and judged metrics cost money and vary, so pin the judge model and name it alongside the score.

CI

with mlflow.start_run(run_name=f"pr {pr} @ {sha}", tags={"ci": "true", "pr": pr}):
    info = train_and_log()
    mlflow.evaluate(model=info.model_uri, data=holdout, targets="label",
                    model_type="classifier", baseline_model=champion_uri,
                    validation_thresholds={...})     # raises → job fails

Separate */ci experiment, tags per pull request, credentials from secrets, a job timeout, and the run URL printed into the log.

Debugging

Common interview questions

1. When do you use the client API instead of the fluent API?When there is no single active run to write to: multi-threaded or async code, a job writing into a run it is not executing inside, bulk logging via log_batch, and anything in the registry. The fluent API is global state keyed on the active run, which is what breaks in those cases.
2. Someone says MLflow is slow. Where do you look first?The metric logging rate. A log_metric per training batch is one HTTP round trip per point, so a long run can make tens of thousands of calls and slow both the trainer and the server. The fix is to log per epoch or accumulate and use log_batch. After that, check backend database load and whether clients are hammering search endpoints.
3. What exactly does autologging patch, and how do you control it?It wraps framework functions: fit and search classes for scikit-learn, trainer hooks for Lightning, train for the boosting libraries, chain calls for LangChain. Control comes from its keyword arguments: log_models=False so you log the model with your own signature and registration, log_input_examples=False so training data is not embedded in the artifact, max_tuning_runs to cap child runs, and exclusive=True so it stays out of a hand-instrumented run.
4. Why would you write a custom pyfunc model?Because a model is rarely just an estimator. A fitted scaler, a vocabulary, a decision threshold, and postprocessing all have to travel with it, or every consumer reimplements them. A PythonModel with load_context and predict packages all of it as one artifact with one environment, and it loads through the same pyfunc interface as anything else, so serving needs no special casing.
5. Why does a custom pyfunc need code_paths?Because the model is pickled by reference to its class, so loading requires that module to be importable. On your machine the repo is on the path and it works; in a serving container it fails with ModuleNotFoundError. code_paths copies the modules into the artifact so the model is self-contained.
6. What are signature params, and why are they useful?A declared schema of call-time parameters (a threshold, a top-k, a temperature) with types and defaults. They are validated and documented like inputs, exposed by the scoring server, and they let a caller vary behaviour without a second endpoint or a redeploy. It is one of the more underused features.
7. How does MLflow decide a model's dependencies, and how does that fail?At log time it introspects loaded modules and pins the resolved distributions. It fails when an import is dynamic or inside a branch that never executed, so the dependency is invisible and the model will not load elsewhere. For anything custom, declare pip_requirements explicitly, and validate with mlflow models predict --env-manager virtualenv before shipping.
8. How do you make an evaluation into a gate?mlflow.evaluate with validation_thresholds raises ModelValidationFailedException when a metric fails, so a CI job exits non-zero with the failing metric named. Pass the current champion as baseline_model and use min_relative_change, so the gate means "better than production by a real margin" rather than "above a number we cleared last year".
9. Why do relative thresholds matter more than absolute ones?An absolute floor passes forever once you clear it, so it stops discriminating. A relative threshold against the current champion rejects improvements inside the noise band, which changes the conversation from "the number went up" to "is this difference real". It is also the only version of the gate that keeps working as the model improves.
10. Design a promotion workflow on the registry.Training registers a version tagged validated=false. A validation job evaluates it on a holdout set against the champion with thresholds, then tags validated=true and records the metric. A promotion job selects among validated versions, sets previous to the outgoing champion, moves champion, and tags who approved it. Consumers load models:/name@champion, so rollback is moving champion back to previous: one call, no redeploy.
11. How do you roll back a model?Move the champion alias to the previous alias's version. That works only if you set previous at promotion time; without it, rollback means hunting for the last good version number during an incident. It is a one-line habit that turns rollback into a rehearsable action.
12. What does dataset logging give you, and what does it not?It records a name, a source URI, a content digest, a schema, and a profile, so a run states what it read and two runs can be compared by digest. It does not give immutability: nothing stops someone overwriting the object at that URI, and MLflow will not notice. Real versioning comes from the storage side: versioned buckets, dated immutable prefixes, or a table format with time travel.
13. What are the deployment options, and which do you reach for?In practice, a batch job with pyfunc.load_model covers most cases and needs no infrastructure. Beyond that: mlflow models serve for a local endpoint, build-docker --enable-mlserver for a container to hand to a platform team, spark_udf for large-scale batch, and the deployments API for managed targets. The point of the pyfunc contract is that all of them load the same artifact.
14. Why is changing a model's signature risky?Because promotion moves an alias with no code change, so a renamed column or a tightened type can reach production without anyone reviewing an API diff, and every caller coding against /invocations breaks. Treat the signature as a public interface: additive changes only, or publish a new registered model name and migrate consumers deliberately.
15. What is MLflow Tracing for?Recording the internal structure of a call (retrieval, reranking, tool use, model calls) as a tree of spans with inputs, outputs, timing, and attributes. It is how you debug a RAG or agent application, where the failure is usually in retrieval rather than in the model. Autolog integrations trace popular frameworks; @mlflow.trace and start_span cover your own code.
16. What are the risks with tracing and LLM-as-a-judge evaluation?Traces store prompts and responses verbatim, so on production traffic they hold user data. That is a redaction and retention decision to make before enabling, not after. Judged metrics are API calls, so a large evaluation has a real bill and real latency, and judges are non-deterministic, so pin the judge model and report it alongside the score, or two evaluations are not comparable.
17. Sketch a CI gate for model quality.On pull request, install pinned requirements, train with a short budget into a dedicated */ci experiment tagged with the PR and commit, evaluate against the champion with thresholds so a regression raises, print the run URL into the log, and set a job timeout. Credentials come from secrets, and the pull-request credential can create runs but not move aliases.
18. Metrics appear in the UI but artifacts 404. What is happening?The client could reach the tracking server but not the artifact store. By default MLflow hands out an artifact URI and the client reads and writes object storage directly, so every user and CI job needs storage credentials. Either give them credentials or enable proxied artifact access so the tracking server brokers the transfer.
19. An old run's artifacts are unreachable after you changed the artifact root. Why?The artifact URI is stored per run at creation time, not resolved from current config. Changing --default-artifact-root affects new runs only. You either keep the old location readable or migrate the objects and rewrite the URIs, which is why the artifact root is a decision worth getting right early.
20. Give an honest reason not to use MLflow.It does not execute anything. There is no agent, no queue, no scheduler, so remote execution, retries, and dependency graphs are your orchestrator's problem. If you wanted tracking and execution in one system, something like ClearML fits better. MLflow's value is being a narrow, widely-supported standard for tracking and model packaging, and that narrowness is deliberate.

Final self-test

• Give the four components, the two stores, and why the registry needs a database.
• Say when the client API beats the fluent API, and name the batch-logging call.
• Explain the three autolog behaviours: run creation, ordering, and input examples.
• Say why you set log_models=False alongside an explicit log_model.
• Explain a custom pyfunc, and why code_paths is mandatory outside your repo.
• Say what signature params buy you, and where they are enforced.
• Describe how requirements are inferred and the one case that always fails.
• Give the command that reproduces the serving environment without a server.
• Explain validation_thresholds, baseline_model, and min_relative_change.
• Walk the five-step promotion workflow and name the rollback mechanism.
• Say what dataset logging proves and what it does not.
• State why a signature change is dangerous when promotion is an alias move.
• Name MLflow's most surprising architectural detail and the error it produces.

Part two of three. The problems at this level are no longer "why is nothing logging". They are a model that loads on your laptop and dies in the container, an evaluation gate that passes noise, an artifact store nobody can read from CI, and a tracking server that got slow because someone logged per batch. Start with the error table, then the practices, then the deeper material underneath.

Common errors at this level

Cumulative. Everything from Beginner still applies. These are the failures that cost hours rather than minutes.

The practices that pay off most

Practice cards

1. Fail on code_paths deliberatelyLog a custom pyfunc without it, then load it from a fresh shell in /tmp. Add code_paths and repeat. The error becomes instantly recognisable forever.
2. Measure the logging taxLog 5,000 metric points one call at a time, then with log_batch. The ratio is the argument.
3. Prove the artifact-store splitRemove your object-store credentials and run a training script. Metrics succeed, artifacts fail. That is the architecture in one experiment.
4. Trim an inferred environmentRead a model's inferred requirements.txt, then replace it with an explicit five-line serving file. Compare build times.
5. Reject a noise-level modelAdd baseline_model and min_relative_change=0.005, then try to promote something 0.0004 better.
6. Rehearse a rollbackServe @champion, promote a new version, then move champion back to previous while the endpoint is live.
7. Break a signature on purposeRename a feature, promote, and call the endpoint with the old payload. Then write your team's rule for signature changes.
8. Read your own traceTrace a RAG call with real input and look at what got stored. Decide what needs redacting.
9. Diff two runs from the clientParams, git commit, artifact URI. Practise finding the one line that differs.
10. Validate without a servervalidate_serving_input plus mlflow models predict, wired into CI as a check.

Making the serving path trustworthy

Most production surprises with MLflow are environment surprises. Four steps eliminate nearly all of them, and they belong in CI.

# 1. What would actually be installed?
python -c "import mlflow; print(mlflow.pyfunc.get_model_dependencies('models:/churn@champion'))"

# 2. Does the contract hold?
python - <<'PY'
from mlflow.models import validate_serving_input, convert_input_example_to_serving_input
import pandas as pd
payload = convert_input_example_to_serving_input(pd.read_json("holdout_sample.json"))
validate_serving_input("models:/churn@champion", payload)
print("signature contract OK")
PY

# 3. Does it load and score in a freshly built environment?
mlflow models predict -m "models:/churn@champion" -i holdout_sample.json \
  --content-type json --env-manager virtualenv

# 4. Does the container work?
mlflow models build-docker -m "models:/churn@champion" -n churn-scorer --enable-mlserver
docker run --rm -d -p 5001:8080 --name scorer churn-scorer
curl -sf -X POST localhost:5001/invocations -H 'Content-Type: application/json' \
  -d @holdout_sample.json > /dev/null && echo "container OK"
docker rm -f scorer

mlflow==2.16.0
scikit-learn==1.5.1
pandas==2.2.2
numpy==1.26.4
joblib==1.4.2

Sweep hygiene

Nested runs make sweeps readable. Four rules keep them useful rather than noisy.

1. The parent carries the summary, the children carry the detailLog the best metric and the winning child's run id on the parent, so the table is scannable without expanding anything.
2. Tag the sweepA sweep tag plus the parameter name makes every run from one campaign filterable and deletable as a group.
3. Do not log a model per child unless you need itForty children × a 200 MB model is 8 GB per sweep. Log models for the top candidates only.
4. Cap search autologgingmax_tuning_runs on a large grid, or your experiment table becomes unusable and your backend store grows for no benefit.

import mlflow

with mlflow.start_run(run_name=f"sweep depth {partition}",
                      tags={"sweep": "max_depth", "dataset": partition}) as parent:
    results = []
    for depth in (3, 5, 8, 12, 16):
        with mlflow.start_run(run_name=f"depth={depth}", nested=True) as child:
            mlflow.log_param("max_depth", depth)
            model, metrics = fit(max_depth=depth)
            mlflow.log_metrics(metrics)
            results.append((metrics["val_auc"], depth, child.info.run_id, model))

    auc, depth, run_id, model = max(results)
    mlflow.log_metrics({"best_val_auc": auc, "candidates": len(results)})
    mlflow.set_tags({"best_depth": str(depth), "best_child_run_id": run_id})
    # Only the winner gets a logged, registered model.
    log_registered_model(model)

from mlflow import MlflowClient
client = MlflowClient()

losers = mlflow.search_runs(
    experiment_names=["churn"],
    filter_string=f"tags.mlflow.parentRunId = '{parent_id}' and tags.keep != 'true'",
)
for run_id in losers["run_id"]:
    client.delete_run(run_id)     # soft delete; recoverable until GC

Gates that mean something

An evaluation that only logs numbers is a report. An evaluation with thresholds is a gate. Four properties separate a gate people trust from one they route around.

import mlflow
from mlflow.models import MetricThreshold, make_metric

HOLDOUT = "s3://bucket/data/churn/holdout/dt=2026-07-01/holdout.parquet"   # frozen

def _cost(eval_df, _builtin):
    fp = ((eval_df["prediction"] == 1) & (eval_df["target"] == 0)).sum()
    fn = ((eval_df["prediction"] == 0) & (eval_df["target"] == 1)).sum()
    return float(50 * fp + 500 * fn)

def _worst_segment_auc(eval_df, _builtin):
    from sklearn.metrics import roc_auc_score
    per = eval_df.groupby("segment").apply(
        lambda g: roc_auc_score(g["target"], g["prediction"]) if g["target"].nunique() > 1 else 1.0
    )
    return float(per.min())

result = mlflow.evaluate(
    model=candidate_uri,
    data=read_parquet(HOLDOUT), targets="label", model_type="classifier",
    baseline_model=champion_uri,
    extra_metrics=[
        make_metric(eval_fn=_cost, greater_is_better=False, name="expected_cost"),
        make_metric(eval_fn=_worst_segment_auc, greater_is_better=True, name="worst_segment_auc"),
    ],
    validation_thresholds={
        "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.005, greater_is_better=True),
        "expected_cost": MetricThreshold(threshold=20000, greater_is_better=False),
        "worst_segment_auc": MetricThreshold(threshold=0.75, greater_is_better=True),
    },
)

Keeping the tracking server healthy

At this level you may not own the server, but you will be blamed for its symptoms. Three client-side behaviours cause most of the load.

# What the server actually holds — ask before assuming
python - <<'PY'
import mlflow
for exp in mlflow.search_experiments():
    runs = mlflow.search_runs(experiment_ids=[exp.experiment_id], max_results=1,
                              output_format="pandas")
    print(exp.name, exp.experiment_id, "artifact:", exp.artifact_location)
PY

# Reclaim space from soft-deleted runs (server-side, coordinate with the owner)
mlflow gc --backend-store-uri postgresql://... --older-than 30d

"Loads locally, fails in serving"

The most common mid-level complaint, with exactly five causes. Work through them in order.

1. A module is missing from the artifactCustom pyfunc without code_paths. The error names your module.
2. A dependency was never inferredDynamic or conditional import. The error names a third-party package.
3. A version differsThe model loads but warns, or behaves differently. Compare the logged requirements.txt against the serving environment.
4. A system dependency is missinglibgomp, libgl, a locale. Only Python is managed. This is a base-image problem.
5. A path or credential only exists on your machineThe model reads a file or a secret at load time. Everything it needs must be an artifact or a declared environment variable.

# The fastest diagnosis: reproduce the load in a built environment, verbosely
mlflow models predict -m "models:/churn@champion" -i sample.json \
  --content-type json --env-manager virtualenv 2>&1 | tail -40

# Then compare the two environments explicitly
python -c "import mlflow; print(open(mlflow.artifacts.download_artifacts('models:/churn@champion/requirements.txt')).read())"
pip freeze | sort > /tmp/local.txt

Habits worth adopting now

Pin MLflow in requirements.txt and align it with the server. A client ahead of the server sends fields the server rejects, and the error message is unhelpful. One pinned version across the team removes a class of "works for me".

Keep a requirements-serving.txt separate from requirements.txt. Training needs notebooks, plotting, and test tooling; serving needs five packages. Smaller serving environments build faster and have less to go wrong.

Tag every run with its origin. ci, sweep, nightly, manual. Retention, cost attribution, and "what is filling the store" all become one filter.

Write the metric into the run name. rf d=8 auc=0.941 makes the experiment table self-documenting, and it makes a sweep's parent row readable at a glance.

Set previous at every promotion. One line, and rollback stops being an incident-time investigation.

Freeze the holdout set and version its path. A gate is only meaningful against data the model has never seen and that does not move underneath you.

Treat the signature as a published API. Additive changes only. If you must break it, publish a new registered model name and migrate consumers deliberately.

Validate in a built environment before every promotion. It is two minutes and it is the only test that exercises what production will do.

# A pre-promotion checklist worth aliasing
set -e
mlflow models predict -m "models:/$NAME/$VERSION" -i holdout_sample.json \
  --content-type json --env-manager virtualenv > /dev/null
python ci/validate_signature.py --model "models:/$NAME/$VERSION"
python ci/evaluate_gate.py --model "models:/$NAME/$VERSION" --baseline "models:/$NAME@champion"
echo "ready to promote $NAME v$VERSION"

Senior tips go deeper on every one of these: the hardening pass every self-hosted server needs, authentication and proxied artifact access, multi-tenant isolation and quota, backend database capacity and what breaks first, backup and restore drills, artifact retention that survives an audit, promotion approval and separation of duties, and incident playbooks for a lost server, an unreadable artifact store, or a model nobody can rebuild.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further (each one now has a security, scale, or ownership dimension) and adds the work you own when MLflow is your team's platform rather than your tool.

Up to this point the questions have been about making a model reproduce and deploy. From here they are different: who can move an alias into production, whether a model from eighteen months ago can still be rebuilt and proven, what the artifact store costs and who pays for it, which component breaks first when the team triples, what happens when the backend database fills up on a Friday night, and where MLflow stops being the right tool.

Where this picks up

I am starting with the trust model, because every other decision in this track depends on it.

The trust model: who can move an alias

The sentence that reframes MLflow security: moving an alias changes production, and by default nothing stops anyone with tracking credentials from doing it.

Most teams worry about the inverse and arrive asking "who can see the experiments?" The sharper question is "who can make a model live?"

Three properties of MLflow's design make this sharper than it looks:

Open-source MLflow has coarse authorisation. The basic auth plugin gives per-experiment and per-registered-model permissions, and that is useful, but it is not rich RBAC, and a plain unauthenticated server gives everyone everything. Say this plainly rather than discovering it in a security review.

Clients touch object storage directly. By default the tracking server returns an artifact URI and the client reads and writes it. That means every user and every CI job needs bucket credentials, which is a much wider credential distribution than people expect, and it is why proxied artifact access matters so much at this level.

A pickle is executable. Loading a model executes code: the unpickling itself, plus any module a custom pyfunc imports. A model artifact from an untrusted source is a code-execution vector in whatever runs your batch job.

The access shape that works:

Tracking server (behind TLS + auth)
  ├─ researchers      → create runs, log models, register versions
  ├─ ci-pr            → create runs in  churn/ci  only
  ├─ ci-main          → register + alias, from a gated workflow
  ├─ serving          → read registry + artifacts only
  └─ retention        → delete, scoped, audited

Artifact store  s3://ml-artifacts/
  ├─ prefix per team, write-restricted
  ├─ versioning ON
  └─ proxied through the tracking server, so clients hold no bucket keys

Authentication, authorisation, and proxied artifacts

Three changes turn a development server into something a team can share.

1. Terminate TLS and require auth at a proxy. MLflow ships basic auth; most teams put OIDC or an existing gateway in front instead.

# MLflow's built-in basic auth app
mlflow server \
  --backend-store-uri postgresql://mlflow:pass@db/mlflow \
  --artifacts-destination s3://ml-artifacts/mlflow \
  --serve-artifacts \
  --app-name basic-auth \
  --host 127.0.0.1 --port 5000

[mlflow]
default_permission = READ
database_uri = postgresql://mlflow:pass@db/mlflow_auth
admin_username = admin
admin_password = <from a secret store>
authorization_function = mlflow.server.auth:authenticate_request_basic_auth

from mlflow.server import get_app_client

auth = get_app_client("basic-auth", tracking_uri="https://mlflow.internal")
auth.create_user(username="amina", password="…")
auth.create_experiment_permission(experiment_id="12", username="amina", permission="EDIT")
auth.create_registered_model_permission(name="churn-classifier", username="amina", permission="READ")

2. Turn on proxied artifact access. --serve-artifacts with --artifacts-destination makes the tracking server broker every artifact transfer, so clients need no bucket credentials at all.

3. Restrict who can move an alias. Nothing in MLflow separates "register a version" from "promote it" beyond registered-model permissions, so the separation has to be operational: researchers get EDIT on the experiment and READ on the registered model; only a CI service account with EDIT on the model can move aliases, and it does so from a workflow that requires review.

Self-hosting: what you own

Beginner ran mlflow server on a laptop. A shared deployment has four moving parts and each has its own failure mode.

mlflow server \
  --backend-store-uri postgresql://mlflow:${PGPASS}@db.internal:5432/mlflow \
  --artifacts-destination s3://ml-artifacts/mlflow \
  --serve-artifacts \
  --app-name basic-auth \
  --host 127.0.0.1 --port 5000 \
  --workers 4 \
  --gunicorn-opts "--timeout 120 --graceful-timeout 30 --max-requests 2000"

services:
  db:
    image: postgres:16
    environment: { POSTGRES_DB: mlflow, POSTGRES_USER: mlflow }
    volumes: ['/opt/mlflow/pgdata:/var/lib/postgresql/data']
    command: ["postgres", "-c", "max_connections=200", "-c", "shared_buffers=1GB"]

  mlflow:
    image: ghcr.io/mlflow/mlflow:v2.16.0        # pinned, never :latest
    depends_on: [db]
    environment:
      MLFLOW_S3_ENDPOINT_URL: https://s3.eu-west-1.amazonaws.com
      AWS_ROLE_ARN: arn:aws:iam::123456789012:role/mlflow-artifacts
    command: >
      mlflow server --backend-store-uri postgresql://mlflow:pass@db/mlflow
                    --artifacts-destination s3://ml-artifacts/mlflow
                    --serve-artifacts --host 0.0.0.0 --port 5000 --workers 4

Non-negotiables for a server anyone else depends on:

Backup, restore, and upgrades

Having backups is easy. Having restored one is the part that counts.

A complete MLflow backup has two parts, and both are required for a usable system:

1. The backend databasepg_dump of the MLflow database, and the auth database too, if you use the basic-auth plugin. This is the system of record: runs, metrics, registry, aliases.
2. The artifact bucketWith versioning enabled, plus a replication or lifecycle policy. Without it, a restored database is a catalogue pointing at nothing.

# 1. Metadata
pg_dump -Fc -h db.internal -U mlflow mlflow > backup/mlflow-$(date +%F).dump
pg_dump -Fc -h db.internal -U mlflow mlflow_auth > backup/mlflow-auth-$(date +%F).dump

# 2. Artifacts — versioning plus cross-region replication is the real backup
aws s3api get-bucket-versioning --bucket ml-artifacts
aws s3api get-bucket-replication --bucket ml-artifacts

# 3. Config
cp -r /opt/mlflow/config backup/config

The restore drill, rehearsed on a separate host:

pg_restore -h scratch-db -U mlflow -d mlflow --clean backup/mlflow-2026-08-01.dump
mlflow server --backend-store-uri postgresql://mlflow:pass@scratch-db/mlflow \
              --artifacts-destination s3://ml-artifacts/mlflow --serve-artifacts &

# Then verify, in this order:
#   1. Can you log in?
#   2. Does a known run show its metrics?               ← the database restored
#   3. Does the registry list your aliases correctly?   ← promotion state survived
#   4. Can you download a known model artifact?         ← bucket + config are right
#   5. Can a client log a new run?                      ← writes work

Upgrades run schema migrations, so they follow the same discipline:

mlflow db upgrade postgresql://mlflow:pass@db.internal/mlflow

Multi-tenancy, quota, and organisation

One MLflow server shared by four teams degrades predictably: hundreds of experiments with no naming convention, a registry where nobody knows which models are live, and a backend database growing for reasons nobody can attribute.

What the open-source server gives you:

experiments:
  risk/churn/research        risk/churn/ci        risk/churn/validation
  fraud/scoring/research     fraud/scoring/ci
  platform/monitoring
registered models:
  risk-churn-classifier      fraud-scoring-xgb
artifacts:
  s3://ml-artifacts/risk/…     lifecycle: 90d → IA, 400d → delete (except tagged keep)
  s3://ml-artifacts/fraud/…

import os
import mlflow

def start_run(project: str, purpose: str, name: str, **kwargs):
    team = os.environ["ML_TEAM"]
    mlflow.set_experiment(f"{team}/{project}/{purpose}")
    tags = {
        "team": team,
        "cost_centre": os.environ["ML_COST_CENTRE"],
        "origin": os.environ.get("ML_ORIGIN", "manual"),
        **kwargs.pop("tags", {}),
    }
    return mlflow.start_run(run_name=name, tags=tags, **kwargs)

def log_model(model_or_pyfunc, *, project: str, **kwargs):
    team = os.environ["ML_TEAM"]
    kwargs.setdefault("registered_model_name", f"{team}-{project}")
    kwargs.setdefault("pip_requirements", "requirements-serving.txt")
    return mlflow.pyfunc.log_model(name="model", python_model=model_or_pyfunc, **kwargs)

Cost and retention

Three cost centres, and only one of them is usually noticed.

The artifact store is where the surprises live, because a logged model is much larger than it feels:

import mlflow
from collections import defaultdict

sizes = defaultdict(float)
for exp in mlflow.search_experiments():
    runs = mlflow.search_runs(experiment_ids=[exp.experiment_id],
                              max_results=50_000, output_format="pandas")
    sizes[exp.name] = len(runs)
for name, count in sorted(sizes.items(), key=lambda kv: -kv[1])[:10]:
    print(f"{count:>7}  {name}")
# Pair this with a bucket inventory report for real bytes per prefix.

s3://ml-artifacts/<team>/research/    30d → IA,  180d → Glacier, 400d → delete
s3://ml-artifacts/<team>/ci/          14d → delete
s3://ml-artifacts/<team>/registry/    no expiry — a registered model's artifact must live

Retention on the metadata side is mlflow gc, and it needs saying precisely: delete_run is a soft delete that only changes lifecycle stage. Nothing is reclaimed until garbage collection runs.

import os
from datetime import datetime, timedelta
import mlflow
from mlflow import MlflowClient

APPLY = os.environ.get("RETENTION_APPLY") == "1"
EXEMPT = {"keep", "paper", "release"}
RULES = [("risk/churn/ci", 30), ("risk/churn/research", 365)]

client = MlflowClient()
for experiment_name, days in RULES:
    exp = mlflow.get_experiment_by_name(experiment_name)
    cutoff = (datetime.utcnow() - timedelta(days=days)).timestamp() * 1000
    runs = mlflow.search_runs(experiment_ids=[exp.experiment_id],
                             output_format="list", max_results=50_000)
    for run in runs:
        if run.info.end_time and run.info.end_time > cutoff:
            continue
        if set(run.data.tags) & EXEMPT:
            continue
        if run.data.tags.get("registered_model_version"):     # never orphan the registry
            continue
        print(("DELETE " if APPLY else "would delete ") + run.info.run_id, run.info.run_name)
        if APPLY:
            client.delete_run(run.info.run_id)

# Only this actually reclaims space, and it is a server-side operation
mlflow gc --backend-store-uri postgresql://mlflow:pass@db/mlflow --older-than 30d

Scale: which component breaks first

Growth arrives in four dimensions, and each breaks something different. Knowing the order lets you fix the right thing.

Two client behaviours cause most server load, and both are code changes rather than infrastructure ones:

# The single worst pattern: a call per batch
for step, batch in enumerate(loader):
    mlflow.log_metric("loss", loss, step=step)          # thousands of HTTP round trips

# Better: per epoch
mlflow.log_metric("loss", epoch_loss, step=epoch)

# When you need resolution: accumulate and batch
from mlflow.entities import Metric
import time
points = [Metric("loss", v, int(time.time() * 1000), step=i) for i, v in enumerate(losses)]
client.log_batch(run_id=run_id, metrics=points)

# And: do not query in a loop
for name in model_names:
    runs = mlflow.search_runs(filter_string=f"tags.model = '{name}'")   # N expensive queries

# Query once, filter in pandas
frame = mlflow.search_runs(experiment_names=["risk/churn/research"], max_results=20_000)
grouped = frame.groupby("tags.model")

Reproducibility that survives eighteen months

MLflow records the commit, the params, and the model's pip dependencies. That is necessary and not sufficient, and the gaps are specific.

import mlflow

with mlflow.start_run(run_name=name, tags={"git_clean": str(is_clean())}) as run:
    if not is_clean():
        raise SystemExit("refusing to log a promotable run from a dirty tree")

    mlflow.log_params({"seed": SEED, "deterministic": True,
                       "data_partition": PARTITION, "base_image": IMAGE_DIGEST})
    mlflow.log_input(dataset, context="training")

    mlflow.pyfunc.log_model(
        name="model", python_model=ChurnModel(),
        code_paths=["src/features.py", "src/model.py"],   # feature logic travels
        pip_requirements="requirements-serving.txt",       # explicit, committed
        signature=signature, registered_model_name=NAME,
    )

The part that makes it a claim rather than a hope is a scheduled rebuild of the current champion:

import mlflow
from mlflow import MlflowClient

NAME, TOLERANCE = "risk-churn-classifier", 0.005
client = MlflowClient()
champion = client.get_model_version_by_alias(NAME, "champion")
original = client.get_run(champion.run_id)

# Rebuild from what was recorded: commit, params, data partition.
rebuilt_uri = retrain_from(
    commit=original.data.tags["mlflow.source.git.commit"],
    params=original.data.params,
)

with mlflow.start_run(run_name=f"rebuild check v{champion.version}",
                      tags={"purpose": "rebuild-check"}):
    result = mlflow.evaluate(model=rebuilt_uri, data=load_holdout(),
                             targets="label", model_type="classifier")
    original_auc = float(champion.tags.get("holdout_auc", original.data.metrics["roc_auc"]))
    delta = abs(result.metrics["roc_auc"] - original_auc)
    mlflow.log_metric("rebuild_delta", delta)
    assert delta <= TOLERANCE, f"rebuild drifted by {delta:.4f}"

Governed promotion: approval and separation of duties

Beginner moved an alias. Mid gated it on thresholds. At this level the question is who decided, and whether that decision is reviewable.

MLflow has no built-in approval workflow, so the trail has to be constructed. Three mechanisms, together, are enough:

1. PermissionsResearchers get EDIT on experiments and READ on registered models. Only a CI service account has EDIT on the model, so only CI can move an alias.
2. A gated workflowPromotion runs from a protected branch with required review, so the human approval is a pull-request approval with a named reviewer.
3. Version tags as the audit recordWho, when, on what data, against which baseline, with which thresholds, written onto the model version by the promotion job.

import os, json, datetime as dt
import mlflow
from mlflow import MlflowClient
from mlflow.models import MetricThreshold

NAME = "risk-churn-classifier"
client = MlflowClient()

candidate = client.get_model_version(NAME, os.environ["VERSION"])
champion = None
try:
    champion = client.get_model_version_by_alias(NAME, "champion")
except mlflow.exceptions.MlflowException:
    pass

run = client.get_run(candidate.run_id)
if run.data.tags.get("git_clean") != "True":
    raise SystemExit("refusing to promote a model trained from a dirty tree")

result = mlflow.evaluate(
    model=f"models:/{NAME}/{candidate.version}",
    data=load_frozen_holdout(), targets="label", model_type="classifier",
    baseline_model=(f"models:/{NAME}/{champion.version}" if champion else None),
    validation_thresholds={
        "roc_auc": MetricThreshold(threshold=0.85, min_relative_change=0.005,
                                   greater_is_better=True),
        "worst_segment_auc": MetricThreshold(threshold=0.75, greater_is_better=True),
    },
)   # raises on failure — the gate is the exception

audit = {
    "approved_by": os.environ["GITHUB_ACTOR"],
    "approved_at": dt.datetime.utcnow().isoformat() + "Z",
    "workflow_run": os.environ["GITHUB_RUN_ID"],
    "holdout": FROZEN_HOLDOUT_URI,
    "baseline_version": champion.version if champion else None,
    "metrics": {k: round(v, 5) for k, v in result.metrics.items()},
}
client.set_model_version_tag(NAME, candidate.version, "promotion_audit", json.dumps(audit))
if champion:
    client.set_registered_model_alias(NAME, "previous", champion.version)
client.set_registered_model_alias(NAME, "champion", candidate.version)

Lineage, audit, and erasure

The audit chain runs model → version → run → code, data, and configuration. It only exists if each link was recorded, and one script proves whether it was.

import json
from mlflow import MlflowClient

def lineage(model_name: str, alias: str = "champion") -> dict:
    client = MlflowClient()
    version = client.get_model_version_by_alias(model_name, alias)
    run = client.get_run(version.run_id)
    inputs = [
        {"name": d.dataset.name, "source": d.dataset.source,
         "digest": d.dataset.digest, "context": d.tags.get("mlflow.data.context")}
        for d in (run.inputs.dataset_inputs or [])
    ]
    return {
        "model": {"name": model_name, "version": version.version,
                  "alias": alias, "tags": version.tags},
        "run": {"id": run.info.run_id, "user": run.data.tags.get("mlflow.user"),
                "start": run.info.start_time, "status": run.info.status},
        "code": {"commit": run.data.tags.get("mlflow.source.git.commit"),
                 "source": run.data.tags.get("mlflow.source.name"),
                 "clean": run.data.tags.get("git_clean")},
        "data": inputs or "NO DATASET RECORDED",
        "config": run.data.params,
        "promotion": json.loads(version.tags.get("promotion_audit", "{}")),
    }

Erasure is the hard case, and MLflow's exposure is narrower than a data platform's but real:

Serving in production, and the supply chain

Beginner served locally. Two things change at this level: the endpoint is a service with the usual obligations, and the model artifact is a supply-chain component.

The service obligations are ordinary and frequently skipped:

import mlflow

# At startup: resolve the alias to a concrete version and log which one.
client = mlflow.MlflowClient()
version = client.get_model_version_by_alias(MODEL, "champion")
MODEL_URI = f"models:/{MODEL}/{version.version}"
MODEL = mlflow.pyfunc.load_model(MODEL_URI)
LOGGER.info("serving %s version %s", MODEL, version.version)   # in every log line's context

The supply-chain question is the one people miss. Loading a model executes code.

1. Unpickling runs codeA pickle is a program. A malicious or tampered artifact executes in whatever process loads it: your batch job, your serving pod.
2. `code_paths` ships arbitrary modulesWhatever was copied in runs at load time. That is the feature, and it is also the risk.
3. Inferred dependencies install from an indexA serving environment build resolves packages at deploy time unless pinned by hash, which is a dependency-confusion surface.
4. The artifact store is the trust boundaryAnyone who can write to it can replace the bytes behind a registered version.

Incident playbooks

Four incidents you should be able to work through without improvising.

The tracking server is gone.

1. Serving keeps working if it cached its modelConfirm that first. It decides whether this is an outage or an inconvenience.
2. Stand up the same pinned imageA different version runs schema migrations on top of an incident.
3. Point it at the existing databaseThe server is stateless; if the database survived, you are minutes away.
4. Verify in orderLogin → a known run's metrics → alias resolution → a model download → a new run logging.
5. ReconcileRuns interrupted mid-flight are stuck in RUNNING; close them and note the metric tail that was lost.

The backend database is full or corrupted. Everything stops: logging, the UI, and alias resolution. Free space or fail over to a replica, then fix the cause, usually per-batch metric writes or no retention. Prevention: a disk alert and mlflow gc on a schedule.

A model version's artifacts are unreadable. Check bucket versioning for a delete marker or an overwrite; check whether a lifecycle rule expired it; check whether the artifact root changed after the run was created. Roll champion back to previous while you investigate. That is what previous is for.

Someone promoted a model that should not be live. Move champion to previous immediately. Then read the version's promotion_audit tag to establish what happened, and fix the permission that allowed it. If a researcher could move the alias, this is an access-control finding rather than a mistake.

Where MLflow stops

Knowing the boundaries is a senior skill, and naming them is a strong interview signal.

Running MLflow as a platform

When MLflow is shared infrastructure, your job changes from using it to operating it. Five obligations.

The monitoring set that matters, in priority order and deliberately short:

1. Backend database disk > 80%When it fills, logging, the UI, and alias resolution all stop. Highest-value alert in the system.
2. Database connections near the limitPresents as intermittent client failures that look like flaky code.
3. Tracking server 5xx rateClients see hangs and failed logging; often a worker or timeout problem.
4. Artifact-store error rateCredential expiry and policy changes show up here first, as failed model uploads.
5. Rebuild check failingReproducibility has regressed and nothing else would have told you.
6. Artifact storage growth versus budgetWeekly, broken down by prefix, before finance notices.
7. Alias changesAny movement of a production alias, notified to a channel. Cheap, and it makes promotion visible.

import mlflow
from mlflow import MlflowClient

client = MlflowClient()
logger = mlflow.start_run(run_name="platform report", tags={"origin": "platform"})

runs_by_experiment = {}
for exp in mlflow.search_experiments():
    frame = mlflow.search_runs(experiment_ids=[exp.experiment_id], max_results=50_000)
    runs_by_experiment[exp.name] = len(frame)

for name, count in runs_by_experiment.items():
    mlflow.log_metric(f"runs::{name.replace('/', '::')}", count)

for model in client.search_registered_models():
    versions = client.search_model_versions(f"name = '{model.name}'")
    mlflow.log_metric(f"versions::{model.name}", len(versions))
    aliases = {a.alias: a.version for a in model.aliases} if model.aliases else {}
    mlflow.set_tag(f"aliases::{model.name}", str(aliases))

mlflow.end_run()

The review checklist

Run this against any MLflow setup, yours or someone else's. It finds something every time.

The complete picture

Everything from all three levels in one platform. Read it as a whole; you should be able to justify every line and name which level it came from.

.
├── .github/workflows/
│   ├── train-pr.yml            # PR: run in */ci, evaluate, threshold. No registry writes
│   ├── register-main.yml       # protected branch: register a version, tag validated
│   └── promote.yml             # gated, reviewed: evaluate, audit tag, move aliases
├── MLproject                   # main / evaluate / promote entry points
├── python_env.yaml
├── requirements.txt            # pinned, includes mlflow==
├── requirements-serving.txt    # small, explicit, hash-pinned
├── src/
│   ├── features.py             # ships with the model via code_paths
│   ├── model.py                # the custom pyfunc
│   ├── train.py                # dirty-tree guard, dataset input, signature
│   └── batch_score.py          # resolves the alias once, at startup
├── platform/
│   ├── mlflow_wrapper.py       # experiment path, tags, registered name, requirements
│   ├── promote.py              # thresholds, audit tag, previous + champion
│   ├── retention.py            # scheduled, dry-run gated, registry-aware
│   ├── rebuild_check.py        # weekly rebuild of the champion
│   ├── lineage.py              # code + data + config + approval, in one command
│   └── report.py               # runs and storage by team, alias inventory
└── runbooks/
    ├── server-lost.md
    ├── database-full.md
    ├── artifacts-unreadable.md
    └── wrong-model-promoted.md

import os, subprocess
import mlflow

IMAGE = "registry.internal/serving@sha256:9f2c4b19e0a7d3f1c6b8a2e4d7091f3b8a2e4d7091f3b"

def _git_clean() -> bool:
    return subprocess.run(["git", "diff", "--quiet"]).returncode == 0

def start_run(project: str, purpose: str, name: str, promotable: bool = False, **kwargs):
    if promotable and not _git_clean():
        raise SystemExit("refusing to create a promotable run from a dirty tree")
    team = os.environ["ML_TEAM"]
    mlflow.set_experiment(f"{team}/{project}/{purpose}")
    tags = {"team": team, "cost_centre": os.environ["ML_COST_CENTRE"],
            "origin": os.environ.get("ML_ORIGIN", "manual"),
            "git_clean": str(_git_clean()), "base_image": IMAGE,
            **kwargs.pop("tags", {})}
    return mlflow.start_run(run_name=name, tags=tags, **kwargs)

def log_model(pyfunc_model, *, project: str, signature, dataset_logged: bool, **kwargs):
    if not dataset_logged:
        raise SystemExit("refusing to log a model for a run with no dataset input")
    team = os.environ["ML_TEAM"]
    return mlflow.pyfunc.log_model(
        name="model", python_model=pyfunc_model, signature=signature,
        code_paths=["src/features.py", "src/model.py"],
        pip_requirements="requirements-serving.txt",
        registered_model_name=f"{team}-{project}", **kwargs,
    )

import mlflow, mlflow.data
from platform.mlflow_wrapper import start_run, log_model
from src.model import ChurnModel

SOURCE = f"s3://ml-data/churn/{PARTITION}/train.parquet"      # dated, immutable
frame = read_parquet(SOURCE)
dataset = mlflow.data.from_pandas(frame, source=SOURCE, name="churn-train", targets="label")

with start_run("churn", "research", f"rf {PARTITION}", promotable=True) as run:
    mlflow.log_input(dataset, context="training")
    mlflow.log_params({"seed": SEED, "deterministic": True, "data_partition": PARTITION})
    mlflow.sklearn.autolog(log_models=False, log_input_examples=False, silent=True)

    model, signature = fit(frame, seed=SEED)
    mlflow.log_metric("val_auc", model.val_auc)                # per epoch elsewhere
    info = log_model(ChurnModel(), project="churn", signature=signature, dataset_logged=True)
    print(info.model_uri)

python platform/retention.py        # daily, dry-run gated, registry-aware
python platform/rebuild_check.py    # weekly — reproducibility as a monitored property
python platform/report.py           # weekly — runs and storage by team, alias inventory
mlflow gc --backend-store-uri "$PG" --older-than 30d    # only this reclaims space

Eighteen decisions in there span the whole series:

Where the series leaves you

Across three levels you have gone from a handful of logging calls to owning a platform. You can track runs and structure sweeps; package arbitrary preprocessing into a portable model with a declared contract; register versions and promote by alias; gate promotions on thresholds relative to production; serve in batch, over HTTP, or in a container; and operate the whole thing with authentication, credential design, tested backups, monitored reproducibility, an audit chain, cost governance, and runbooks somebody else can follow.

The through-line of all three levels is a single idea: a model is only useful if something other than its author can load it, and only trustworthy if something other than its author's memory explains it. Everything else (signatures, aliases, pinned environments, audit tags, backups, rebuild checks) exists to keep that true as the number of models, people, and months grows.

Part three of three, and the one to read if you only read one. A cumulative review of the entire series (foundations, packaging and promotion machinery, and the governance and scale work a senior owns) organised by topic rather than by level. About fifty minutes. Fast review first, common questions at the end.

Part one: foundations

MLflow has four components: Tracking for runs, Models for framework-agnostic packaging, a Registry for named versions and moveable aliases, and Projects for reproducible entry points. The core idea is the model format (a directory with an MLmodel manifest declaring flavours, a signature, and an environment) which is why one artifact loads in a batch job, a REST server, or a container.

Two stores: backend for metadata, artifact store for files. The registry needs a database backend. Params immutable, metrics stepped, tags mutable. Consume by alias. MLflow records the commit, not a diff.

The four things you log

The model format and URIs

flavors:
  python_function: {loader_module: mlflow.sklearn, model_path: model.pkl}
  sklearn: {pickled_model: model.pkl, sklearn_version: 1.5.1}
signature: {inputs: '[…]', outputs: '[…]'}
run_id: a1b2c3d4e5f64718

Native flavour returns the real object; pyfunc returns a generic predict, which is what every deployment target uses. A signature is a contract; without it, misordered or mistyped input scores silently.

The rules that never change

Context manager always. Named experiment, named runs, tags in code. Signature on every model. mlflow.evaluate for the standard metric set. Aliases, not deprecated stages. Commit before a run you might defend.

The traps

Part two: packaging and promotion machinery

Runs, nesting, and the client API

Nesting is the mlflow.parentRunId tag, so children are queryable. Use the client API for threads, other runs, bulk log_batch, and all registry work, because the fluent API is global state keyed on the active run.

Autologging internals

Patches framework functions at import time. Three behaviours: it creates a run if none is active; it must be enabled before the estimator exists; log_input_examples=True embeds real training data in the artifact. Set log_models=False and log the model yourself, or you get two model directories with no statement of which is authoritative.

Custom pyfunc

class ChurnModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context): ...
    def predict(self, context, model_input, params=None): ...

mlflow.pyfunc.log_model(name="model", python_model=ChurnModel(),
    artifacts={"scaler": …, "classifier": …},
    code_paths=["src/features.py"], signature=signature,
    pip_requirements="requirements-serving.txt",
    registered_model_name="churn-classifier")

Missing code_paths is the top cause of "loads locally, fails in serving", because the class is pickled by reference, so the module must be importable.

Dependencies

Inferred by introspecting loaded modules. Inference sees imports, not intent, so dynamic or conditional imports are invisible. Controls: extra_pip_requirements, an explicit pip_requirements, get_model_dependencies, validate_serving_input, and mlflow models predict --env-manager virtualenv to reproduce the serving path.

Gates

mlflow.evaluate(model=candidate, data=holdout, targets="label", model_type="classifier",
    baseline_model=champion,
    extra_metrics=[cost_metric, worst_segment_metric],
    validation_thresholds={"roc_auc": MetricThreshold(threshold=0.85,
        min_relative_change=0.005, greater_is_better=True)})

Raises ModelValidationFailedException, so CI fails with the metric named. Relative to a baseline is the honest gate; an absolute floor stops discriminating once cleared. An aggregate metric hides a broken segment.

The registry as a workflow

Datasets, deployment, tracing

mlflow.data.from_pandas(...) plus log_input records name, source, digest, schema, and profile. It is lineage annotation, not versioning. Nothing stops the source URI being overwritten.

Targets: batch pyfunc.load_model, mlflow models serve --workers N --enable-mlserver, build-docker, spark_udf, and the deployments API. /invocations takes dataframe_split, dataframe_records, instances, inputs, plus params.

Tracing records spans (LLM, RETRIEVER, RERANKER, TOOL) with inputs, outputs, timing, attributes. GenAI metrics are LLM-as-a-judge: they cost money per row and vary, so pin and name the judge.

Part three: trust, scale, and ownership

The trust model

Moving an alias changes production, and by default nothing stops anyone with tracking credentials from doing it.

Three sharpening properties: authorisation is coarse (per-experiment and per-model permissions, not rich RBAC, and unauthenticated by default); clients touch object storage directly, so everyone needs bucket credentials unless you proxy; and a pickle is executable, so loading a model runs code.

Auth and proxied artifacts

mlflow server --backend-store-uri postgresql://… \
  --artifacts-destination s3://ml-artifacts/mlflow --serve-artifacts \
  --app-name basic-auth --workers 4

--serve-artifacts means only the server holds bucket credentials and artifact access follows MLflow's permissions, at the cost of making the server a throughput bottleneck and single point of failure for transfers.

Self-hosting

SQLite in a shared deployment is a time bomb: database is locked under concurrent writers. Postgres from the second user onwards. Pin the server image tag: latest means an unplanned schema migration on the next restart.

Backups and upgrades

Two parts: pg_dump of the MLflow (and auth) database, and the artifact bucket with versioning and replication. A restored database with an unreachable bucket is a catalogue of missing models. Verify: login → a run's metrics → alias resolution → a model download → a new run logging. Upgrades run migrations: back up, mlflow db upgrade deliberately, server before clients.

Multi-tenancy

Open-source MLflow is one server with coarse permissions. Isolation comes from experiment naming conventions, the basic-auth plugin, artifact prefixes with bucket policy, mandatory tags, and, for hard isolation, a separate server and database per tenant. A thirty-line wrapper turns every convention into a default.

Cost and retention

Scale, in order of what breaks

The backend database is the single point of failure, and it includes alias resolution, so serving should resolve the alias once at startup, not per request.

Reproducibility

MLflow will record an unreproducible run with no warning. The checks are yours: refuse to register from a dirty tree, require a dataset input, and run a weekly rebuild check against the champion with a tolerance.

Governed promotion, lineage, erasure

Three mechanisms make promotion auditable: permissions (only a CI account may move aliases), a gated workflow (protected branch with required review), and a promotion_audit version tag recording who, when, which holdout, which baseline, which thresholds, and the metrics.

Audit chain: model → version → run → code, data, configuration. A lineage script proves whether each link exists. Common gaps: a dirty tree, no dataset recorded, no approval trail.

Erasure exposure specific to MLflow: prediction artifacts with identifiers, input_example embedded in a model, and GenAI traces holding prompts verbatim. Weights are un-erasable in practice, and retraining is the remedy.

Serving and the supply chain

Authenticate at an ingress; the scoring server authenticates nobody. Resource limits, pinned base image by digest, resolve the alias at startup, and a rehearsed rollback.

Where MLflow stops

Common interview questions

1. Design MLflow for a team of thirty across three squads.Self-hosted behind TLS and auth, Postgres backend with disk and connection alerts, artifacts in object storage with a prefix per team, bucket versioning on, and proxied artifact access so clients hold no bucket keys. Experiments named team/project/purpose, mandatory team and cost_centre tags, and a shared wrapper making all of that default. Researchers get EDIT on experiments and READ on registered models; only a CI service account can move aliases, from a reviewed workflow that writes an audit tag. Scheduled retention plus mlflow gc, a weekly rebuild check, and a published platform report. Then say plainly that authorisation is coarse, so isolation is conventions, permissions, and buckets, not roles.
2. Why is alias permission the security question that matters?Because consumers load models:/name@champion, so moving that alias changes what production serves with no deploy and no code review. On a default install anyone with tracking credentials can do it, and nothing records who did. That makes alias write the production-change surface, and it should belong to one gated CI job rather than to every researcher.
3. What is MLflow's most surprising architectural detail?By default the tracking server returns an artifact URI and the client reads and writes object storage itself. So every user and every CI job needs bucket credentials, and artifact access is governed by bucket policy rather than by MLflow permissions. It is why "metrics appear but artifacts 404" is the most common support question, and --serve-artifacts with --artifacts-destination is the fix, at the cost of routing all transfers through the server.
4. Which component breaks first as you grow, and why?The backend database, on metric write rate. Per-batch logging from several concurrent runs produces enormous write volume, which slows logging and then the UI; after that it is run count making search slow. Both are fixed by client behaviour first, per-epoch logging and retention, before adding database resources. Because the database also serves alias resolution, a database problem is a production problem unless serving caches its model at startup.
5. What does a complete MLflow backup contain?A dump of the backend database, including the auth database if you use the basic-auth plugin, plus the artifact bucket with versioning and ideally replication. Metadata alone restores a complete-looking system where every model resolves and nothing downloads. That is why "can you download a known model" belongs in the restore verification, alongside login, a run's metrics, and alias resolution.
6. Why is SQLite unacceptable for a shared server?It locks under concurrent writers, so several people training at once produce database is locked exactly when the server is most needed. It also cannot be replicated or scaled, and migrating a live SQLite store to Postgres involves downtime and care. Postgres from the second user onwards.
7. How do you make a model from a year ago rebuildable?MLflow records the commit, params, and pip pins; everything else is yours. Refuse to register from a dirty tree, require a logged dataset with a dated immutable source and a digest, pin the base image by digest, log and set the seed, and keep an explicit committed serving requirements file. Then schedule a rebuild check that reruns the champion's recipe and compares the metric within a tolerance, because unpinned layers degrade silently and the first run of that check will fail.
8. Why does MLflow's lack of a stored diff matter?Because a run tagged with a commit but executed from a dirty tree looks as trustworthy as a clean one, and nothing warns you. Unlike tools that capture the working tree, MLflow cannot reconstruct what ran. The mitigation is procedural: a guard that refuses to create a promotable run from a dirty tree, and a git_clean tag that promotion checks.
9. How do you build an approval trail for promotion?Three things together. Permissions, so only a CI service account can move an alias. A gated workflow on a protected branch, so the human approval is a reviewed pull request by someone other than the author. An audit tag on the model version recording who, when, which frozen holdout, which baseline version, which thresholds, and the resulting metrics. The tag is the record; the permission is the control, because a tag written by the person who trained the model is a self-approval.
10. An auditor asks you to prove what produced the production model. What do you show them?One generated report: the alias, the version, the version tags, the producing run with its user and status, the git commit and whether the tree was clean, the dataset inputs with their sources and digests, the params as executed, and the promotion audit tag. If any of those is missing, usually the data chain or the approval, that is the finding, and it is better to know it before the audit than during it.
11. How would you handle an erasure request?Enumerate the MLflow-specific copies first: prediction artifacts containing identifiers, an input_example embedded in a model if autologging left it on, GenAI traces holding prompts verbatim, and database backups. Then be honest that weights retain the influence and are sometimes extractable, so retraining is the only real remedy for the model itself. The architectural fix is upstream: keep identifiers out of logged artifacts, turn off input examples, and decide trace redaction before enabling tracing on real traffic.
12. What is the supply-chain risk in an MLflow model?Loading a model executes code: unpickling is code execution, and code_paths ships arbitrary modules that run at load time. Worse, a registry version is immutable metadata pointing at mutable bytes: anyone with bucket write access can replace the object behind version 7 and every consumer loads different weights with no error. Write-restrict the artifact prefix, enable versioning, prefer proxied access, and treat an external model artifact like an unreviewed dependency.
13. Design a promotion gate a reviewer would trust.Evaluate on a frozen holdout the model has never seen, with the current champion as baseline_model and a min_relative_change so noise does not pass. Include a business-cost metric and a worst-segment metric with their own thresholds, because an aggregate improvement can hide a regression for one group. Fail on the exception rather than on a logged number, and write the audit tag. Then rehearse the rollback.
14. How do you roll back, and how fast?Move champion to the version held by previous. That is one call, and it only works because previous was set at promotion time. If serving caches its model at startup, the rollback also needs a restart or a cache expiry, which is a trade-off worth making explicit, since the alternative is resolving the alias per request and coupling production to the tracking database.
15. What is your retention policy, and what could it break?Runs in CI experiments deleted after thirty days, research runs after a year, with exemptions for tagged runs and anything a registered version references, then mlflow gc to reclaim space, because delete_run is only a soft delete. On the artifact side, lifecycle rules per prefix with the registry prefix exempt from expiry, because expiring the object behind a registered version produces a resolvable entry that fails to download.
16. How do you keep researchers productive without giving everyone everything?A wrapper: they call one function and get the right experiment path, mandatory tags, registered model name, serving requirements, and image digest. They can create runs and register versions; they cannot move an alias. Promotion is a reviewed workflow. The result is that the safe path is also the shortest path, which is the only governance that survives a deadline.
17. What monitoring does an MLflow platform need?Backend database disk above 80% first, because when it fills, logging, the UI, and alias resolution all stop. Then database connections, tracking-server 5xx rate, artifact-store error rate, the rebuild-check result, storage growth per team, and a notification on any production alias change. Deliberately not on that list: users' failed runs, which belong to their authors and will drown the real alerts.
18. You inherit an MLflow setup. What do you check first?Whether a researcher can move a production alias, and whether a restore has ever been performed. Those two are "anyone can change production" and "a bad day is permanent". Then: unauthenticated server, SQLite backend, artifact root on a server disk, unpinned server image, no bucket versioning, no retention, and whether promotion is a script with a threshold or a person clicking.
19. Where does MLflow stop?It executes nothing (no agent, no queue, no retries) so orchestration is Airflow, Dagster, or CI. Its dataset tracking is annotation, not versioning, so immutability comes from DVC, LakeFS, or a table format. It is not a feature store and not a serving mesh. Authorisation is coarse. It has no optimiser, so hyperparameter search is Optuna or Ray Tune logging into it. That narrowness is deliberate and is why it composes well with whatever you already run.
20. What is the single idea that connects the whole series?A model is only useful if something other than its author can load it, and only trustworthy if something other than its author's memory explains it. Signatures, flavours, pinned environments, aliases, audit tags, backups, and rebuild checks all exist to keep that true as the number of models, people, and months grows.

Final self-test

• Give the thirty-second answer, the four components, and the two stores.
• State the rules that never change: context manager, signature, alias, evaluate, commit.
• Explain what log_model writes and why pyfunc makes it portable.
• Say why per-batch metric logging is the classic scale mistake.
• Explain custom pyfunc, code_paths, and signature params.
• Describe how requirements are inferred and the one case that always fails.
• Give the command that reproduces the serving environment without a server.
• Explain baseline_model plus min_relative_change, and why an aggregate metric is not enough.
• Say why alias write is the production-change surface, and how you restrict it.
• Name MLflow's most surprising architectural detail and the error it causes.
• Say which component breaks first at scale and the fix order.
• List the two backup parts and the restore step that usually fails.
• Name every layer that must be pinned for an eighteen-month rebuild, and how you verify it.
• Explain why a registry version is immutable metadata over mutable bytes.
• Name three places MLflow is the wrong tool and what you would use instead.

Part three of three. The problems at this level are rarely about syntax. They are about a server thirty people depend on, an artifact bill nobody owns, a champion nobody can rebuild, and a promotion nobody can explain. Start with the error table, then the practices, verification, and playbooks underneath it.

Common errors at this level

Cumulative. Everything from Beginner and Mid still applies. These cause incidents rather than failed runs.

The practices that pay off most

Practice cards

1. Test the promotion boundaryWith a researcher credential, try to move a production alias. If it works, stop and fix that before anything else on this page.
2. Replace a version's bytesOn a test bucket, overwrite the object behind a registered version and load it. Nothing warns you. That is the demonstration.
3. Restore and time itDatabase and bucket onto a scratch host. Verify login, metrics, alias resolution, a model download, and a new run. Publish the number.
4. Fill the databaseOn a test instance, fill the backend disk and watch logging, the UI, and alias resolution fail together.
5. Prove the serving decouplingStop the tracking server while a batch job runs. If it dies, you resolve the alias per request.
6. Rebuild the championFrom only what MLflow recorded. Compare the metric, then list every layer that run does not pin.
7. Generate a lineage reportCode, data, config, and approval in one command. Note which of the four is missing.
8. Audit artifacts for identifiersSearch prediction files and embedded input examples. Decide what is in scope for erasure.
9. Dry-run retentionRead the list, the bytes it would free, and confirm the registry-reference exemption works.
10. Run the checklist on someone else's projectIt will find something, and it teaches you how much of this is convention rather than enforcement.

The hardening pass every deployment should get

Work down this list. Each step is independently valuable, and the order is by blast radius.

# 1. Alias permissions — the production-change surface
#    Researchers: EDIT on experiments, READ on registered models.
#    One CI service account: EDIT on the model, used only by a reviewed workflow.
python - <<'PY'
from mlflow.server import get_app_client
auth = get_app_client("basic-auth", tracking_uri="https://mlflow.internal")
auth.create_registered_model_permission(name="risk-churn-classifier",
                                        username="amina", permission="READ")
auth.create_registered_model_permission(name="risk-churn-classifier",
                                        username="ci-promote", permission="EDIT")
PY

# 2. Auth and TLS in front of the server
mlflow server --backend-store-uri "$PG" \
  --artifacts-destination s3://ml-artifacts/mlflow --serve-artifacts \
  --app-name basic-auth --host 127.0.0.1 --port 5000 --workers 4

# 3. Bucket safety — makes every other storage mistake survivable
aws s3api put-bucket-versioning --bucket ml-artifacts \
  --versioning-configuration Status=Enabled

# 4. Postgres, not SQLite; pinned server image, not latest
#    ghcr.io/mlflow/mlflow:v2.16.0  +  mlflow==2.16.0 clients

# 5. Alerts before anything else
#    backend database disk > 80%   → page
#    database connections > 80%    → page
#    server 5xx sustained          → page

s3://ml-artifacts/<team>/research/   30d → IA, 180d → Glacier, 400d → delete
s3://ml-artifacts/<team>/ci/         14d → delete
s3://ml-artifacts/<team>/registry/   NO EXPIRY — a registered version points here

APPLY = os.environ.get("RETENTION_APPLY") == "1"
EXEMPT_TAGS = {"keep", "paper", "release"}
# skip anything a registered version references; print totals; delete only when enabled

# 7. Only this reclaims space
mlflow gc --backend-store-uri "$PG" --older-than 30d

Verifying, not assuming

Every claim here is checkable in minutes. Assumed properties are the ones that fail.

import json
import mlflow
from mlflow import MlflowClient

NAME = "risk-churn-classifier"
client = MlflowClient()

champion = client.get_model_version_by_alias(NAME, "champion")
run = client.get_run(champion.run_id)

assert run.data.tags.get("git_clean") == "True", "champion trained from a dirty tree"
assert run.inputs.dataset_inputs, "champion has no recorded dataset"
assert "promotion_audit" in champion.tags, "no approval record on the champion"
assert client.get_model_version_by_alias(NAME, "previous"), "no rollback target"

audit = json.loads(champion.tags["promotion_audit"])
print("version:", champion.version)
print("commit:", run.data.tags.get("mlflow.source.git.commit"))
print("data:", [d.dataset.source for d in run.inputs.dataset_inputs])
print("approved by:", audit.get("approved_by"), "at", audit.get("approved_at"))
print("baseline:", audit.get("baseline_version"), "metrics:", audit.get("metrics"))

curl -sf "$MLFLOW_TRACKING_URI/health" && echo "server ok"
psql "$PG" -c "select pg_size_pretty(pg_database_size(current_database()));"
psql "$PG" -c "select count(*) from metrics;"
psql "$PG" -c "select count(*) from runs where lifecycle_stage='deleted';"
aws s3api get-bucket-versioning --bucket ml-artifacts
docker ps --format '{{.Names}}\t{{.Image}}' | grep mlflow      # is the tag pinned?

Cost governance

Cost becomes your problem the moment MLflow is shared, and it is a measurement problem before it is a technical one.

import mlflow
from collections import defaultdict

by_team = defaultdict(int)
by_origin = defaultdict(int)
models_logged = defaultdict(int)

for exp in mlflow.search_experiments():
    frame = mlflow.search_runs(experiment_ids=[exp.experiment_id],
                               max_results=50_000, output_format="pandas")
    if frame.empty:
        continue
    team = frame.get("tags.team", "untagged")
    for value in (team if hasattr(team, "__iter__") else [team]):
        by_team[value] += 1
    if "tags.origin" in frame:
        for value in frame["tags.origin"].fillna("manual"):
            by_origin[value] += 1

logger = mlflow.start_run(run_name="cost report", tags={"origin": "platform"})
for team, count in by_team.items():
    mlflow.log_metric(f"runs::{team}", count)
for origin, count in by_origin.items():
    mlflow.log_metric(f"runs_by_origin::{origin}", count)
mlflow.end_run()

Incident playbooks

Four incidents, each with a first action, a verification, and the prevention. Keep them in the repository.

The tracking server is gone.

1. Check whether serving survivedIf it cached its model at startup, this is an inconvenience rather than an outage. Establish that first.
2. Stand up the same pinned imageA different version runs schema migrations on top of an incident.
3. Point it at the existing databaseThe server is stateless. If the database is intact you are minutes away.
4. Verify in orderLogin → a known run's metrics → alias resolution → a model download → a new run logging.
5. ReconcileInterrupted runs are stuck in RUNNING; close them and record the metric tail that was lost.

The backend database is full. Everything stops together: logging, the UI, and alias resolution. Free space or fail over to a replica, then fix the cause, usually per-batch metric writes or no retention. Prevention: an 80% disk alert and mlflow gc on a schedule.

A model version's artifacts are unreadable. Roll champion back to previous first, then investigate: a bucket delete marker or overwrite, a lifecycle rule that expired the object, or an artifact root that changed after the run was created. Recover from bucket versioning if it is enabled, and if it is not, that is the finding.

The wrong model is live. Move champion to previous immediately. Then read the version's promotion_audit tag to establish what happened, and check whether the promoter should have been able to do it. If a researcher moved it, this is an access-control finding, not a mistake.

Machine-learning specifics

A few things are MLflow-and-ML rather than general platform work, and they come up in senior interviews.

A signature is a published API. Because promotion is an alias move with no deploy, a renamed column or a tightened type reaches production without an API diff being reviewed. Additive-only, or a new registered model name and a deliberate consumer migration.

Aggregate metrics hide segment regressions. A model can improve overall AUC while getting materially worse for one customer group. A worst-segment threshold in the gate costs ten lines and catches exactly the regression that damages trust.

Checkpoint policy is a cost decision. Best plus last, not every epoch. Every-epoch checkpoints of a large model, logged as artifacts, dominate an artifact bill within a quarter and nobody ever loads epoch 47.

Sweep children should not each log a model. Forty children × 200 MB is 8 GB per sweep for candidates you will never deploy. Log models for the winner, keep the children's metrics.

Loading a model executes code. Unpickling is code execution, and code_paths ships modules that run at load time. An external artifact is an unreviewed dependency running in your batch job.

Weights are the un-erasable artefact. Removing records from a dataset does not remove their influence, and for some model classes information is extractable. Retraining is the remedy, which is why the erasure conversation is architectural rather than a runbook.

Traces and input examples are the two data-exposure surfaces people miss. log_input_examples=True embeds real rows inside a shareable artifact; tracing stores prompts and responses verbatim. Both are near-default settings that create copies nobody remembers during a request.

The checklist to run before shipping

Nobody passes this on the first attempt. Two failures matter disproportionately: unrestricted alias permissions and an untested restore: one means anyone can change production with no record, the other means a bad day becomes a permanent loss. Rank by blast radius rather than by effort, fix those two first, and publish the exceptions you are choosing to live with along with the reason. A checklist with argued exceptions is a better artefact than a green one, because it shows the trade-offs were deliberate and it gives whoever inherits the platform a map instead of a mystery.