The Complete Kubeflow Guide

This is part one of three. It assumes you have never touched Kubeflow, and it is honest about one thing up front: Kubeflow is not a single program you install and open. It is a family of projects that share a home on Kubernetes, and half of being productive with it is knowing which member of the family you actually need. By the end of this guide you will know what each part is for, you will have written and run a real machine learning pipeline, you will know how to bring up the platform on a laptop or a cluster, and you will be able to read the errors that greet almost every newcomer. Mid-level and Senior take the same ground further; nothing here is thrown away.

Each section ends with a Try it task where it helps. Do them as you go. Kubeflow concepts only stick once you have watched your own pipeline run, fail, and run again.

What Kubeflow is, and the problem it solves

Training a model in a notebook is the easy part of machine learning. The hard part starts the day someone asks you to do it again: with new data, on a schedule, on a bigger machine, with five different learning rates, and with a record of exactly which code and data produced the model now running in production. A notebook cannot answer those questions. A folder of scripts held together by memory and shell history cannot either.

Kubeflow describes itself as the cloud native AI platform: a set of open-source projects that together make the machine learning lifecycle run on Kubernetes, the system most organisations already use to run containers across a cluster of machines. If Kubernetes is new to you, read the Kubernetes guide alongside this one, because almost everything Kubeflow does is expressed as Kubernetes objects. A pipeline step becomes a pod. A training job becomes a set of pods. A notebook server becomes a pod with a disk attached.

The design idea is that each stage of the machine learning lifecycle gets its own project, and each project can be used alone:

Before platforms like this, teams glued the stages together themselves. One engineer wrote a cron job that called a training script. Another kept a spreadsheet of hyperparameter runs. A third copied model files to a server by hand. Each of these worked until the person who built it left. Kubeflow's pitch is that these chores are the same in every company, so they should be shared, open-source infrastructure rather than private folklore.

There is a cost, and you should hear it now. Kubeflow runs on Kubernetes, so you inherit Kubernetes. The full platform is heavy: the reference bundle needs roughly 4.4 CPU cores and 12 GiB of memory just to sit idle, before you run anything. If you only want to track experiments from a laptop, a lighter tool such as MLflow is the better start. If you only want a schedule of data jobs, Airflow may fit better. Kubeflow earns its weight when you already have a cluster, several people sharing it, and a need for reproducible pipelines, distributed training and isolation between teams.

What people use it for:

How Kubeflow is versioned and packaged

Newcomers trip on this before they run a single command, so it comes early. Searching for "Kubeflow 1.12" leads nowhere, and old tutorials quote versions that no longer exist.

Kubeflow is a set of independent subprojects, each with its own repository, release schedule and version number. The ones you will meet in this guide are:

• Kubeflow Pipelines (KFP), for defining and running workflows.
• Kubeflow Trainer, for running training jobs, including distributed ones.
• Katib, for hyperparameter tuning.
• Kubeflow Notebooks, for notebook servers on the cluster.
• Kubeflow Dashboard, the web front door and the place where users and namespaces are managed.
• Kubeflow Hub, formerly called Model Registry, a catalogue of your models and their versions.
• Spark Operator, for running Apache Spark jobs on Kubernetes.

Serving models is handled by KServe. It is now classed as an ecosystem project rather than a Kubeflow subproject, but the standard bundle still ships it. See the KServe guide once you have a model to deploy.

On top of the subprojects sits a curated bundle called the Kubeflow Community Distribution, which lives in the repository kubeflow/community-distribution. You may see it called "the manifests", because it is a collection of Kustomize manifests (see the Kustomize guide) that install compatible versions of everything plus the supporting cast: Istio for networking, cert-manager for certificates, Dex and oauth2-proxy for login, and Knative for serverless serving. The repository used to be named kubeflow/manifests; GitHub redirects the old address.

The bundle moved to calendar versioning in 2026. A version now reads YY.MM, with patches as YY.MM.PATCH. The release this guide is written against is 26.03.1, published in June 2026, following the base 26.03 release in March. The last release under the old scheme was 1.11.0, in December 2025, and there is no 1.12. The project aims for two base releases a year, timed before the two KubeCon conferences, and the community supports each one on a best-effort basis for about six months.

Inside 26.03.1, the components have their own numbers: Pipelines 2.16.1, Trainer 2.2.0, Katib 0.19.0, Notebooks 1.11.0, Dashboard 2.0.0 and KServe 0.18.0. Newer standalone releases of several of these already exist. That is normal, and it is why you always check which version a tutorial used before copying its commands.

The mental model: the parts you will meet

Kubeflow has a lot of nouns. Learn them in groups and the platform stops feeling like a wall of logos.

The platform layer. The Central Dashboard is the web page you open in a browser. It is a shell that embeds the interface of each component, so you click "Notebooks" or "Pipelines" in a side menu and the page for that component appears inside the dashboard. A Profile is Kubeflow's unit of multi-user isolation. Each person or team gets a profile, and a profile creates a Kubernetes namespace of the same name, plus the permissions that make its owner an administrator of that namespace. Your work lives in your profile's namespace, and other users cannot see it unless you add them as contributors. Authentication happens before you reach any of this: a login proxy checks who you are and Istio attaches your email address to each request in a header called kubeflow-userid, which the web applications trust.

The pipeline layer. A component is one step, packaged as a container. A pipeline is a graph of components, where the output of one step feeds the input of the next. You write both in Python using the kfp library, then compile the pipeline into a YAML file, a portable description that the backend can execute. One execution of a pipeline is a run. Runs are grouped into experiments, which is just a folder for organisation. A run that repeats on a schedule is a recurring run.

The data passed between steps. Small values such as a number or a string are parameters. Files such as a dataset or a trained model are artifacts. Artifacts are stored in an object store (an S3-style bucket) and steps receive them as files. This distinction matters a great deal, and we will return to it in the first pipeline.

The training layer. A TrainJob is a request to train a model, possibly across several machines. It refers to a runtime, which is a blueprint written by an administrator that says how to launch, for example, PyTorch distributed training. You pick the runtime and supply your code. A Katib Experiment is a tuning job: a goal, a search space of settings, an algorithm to explore it, and a budget of trials. Each trial is one try.

The notebook layer. A Notebook in Kubeflow terms is a Jupyter or similar server running as a pod, with a persistent disk so your files survive restarts.

Hold on to this stack. When something breaks, the first question is always which layer: did the request get in (front door), did the component accept the work (components), or did Kubernetes fail to start a pod (bottom)? Most beginners look at the wrong layer for hours.

One more term you will see in official pages: KFP v2. Kubeflow Pipelines had an older SDK generation, v1, and it has been removed. Everything here uses v2, and if a tutorial mentions ContainerOp or "v2 compatible mode", it is out of date.

What you need before you install

Installing the whole platform is a cluster project, so decide what you want before you start. There are three realistic starting points, ordered from lightest to heaviest.

No cluster at all. The Pipelines SDK can run pipelines locally, on your own computer, using a local runner. You need Python and, optionally, Docker. This is where you should begin, and this guide's first project uses it. You learn the pipeline language without touching Kubernetes.

One component on a small cluster. You can install Kubeflow Pipelines on its own, without the dashboard, profiles or login. It needs a Kubernetes cluster and kubectl. A local cluster made with kind (Kubernetes in Docker) is enough. This is the right step after local runs, because you see real pods without the multi-user machinery.

The full distribution. The community distribution recommends 16 GB of memory and 8 CPU cores. You can trim the install file to fit into roughly 4 to 8 GB and 2 to 4 cores by removing components you do not need, but expect a slow, tight machine. The platform expects Kubernetes 1.34 or newer, kubectl, and Kustomize 5.8 or close to it. Kind needs Docker or Podman. You also need a cluster storage class that can create disks on demand, because Pipelines, Katib and Hub each request persistent volumes.

A few notes by operating system. The official documentation describes one path, kind on Linux, and does not have separate macOS or Windows pages. On a Mac, the practical route is to run Docker Desktop, Colima or Podman Machine, give that virtual machine at least 16 GB of memory and 8 CPUs, and run the same scripts. This is inferred from how kind works, not spelled out in the docs, so treat it as a best effort. On Apple Silicon, expect some images to fail with "no matching manifest for linux/arm64", because not every image is built for ARM. On Windows, use WSL2 with an Ubuntu shell and run everything inside it; raise the memory limit for WSL in your .wslconfig file. There is no native PowerShell install path.

On Linux, kind clusters that run many pods hit a low file-watcher limit. Raise it before installing:

sudo sysctl fs.inotify.max_user_instances=2280
sudo sysctl fs.inotify.max_user_watches=1255360

If you skip this, pods crash at random with errors mentioning "too many open files". It is one of the most common first-install failures, and the fix takes ten seconds.

If you work in the Gulf or Egypt, you may also need to think about where the cluster lives. A managed Kubernetes service in a Middle East cloud region keeps training data in-country, which some employers and regulators require. Kubeflow itself has no opinion about the region; it runs wherever the cluster runs. Only your object store and your data sources carry the residency question.

Installing the pieces

The SDK: start here

Install the Pipelines SDK into a virtual environment. Pin the version, because the SDK and the backend must agree:

python3 -m venv .venv
source .venv/bin/activate
pip install kfp==2.17.0
python -c "import kfp; print(kfp.__version__)"
kfp --version

The second-to-last command should print 2.17.0. The kfp package also installs a command-line tool, also named kfp. If you want to build your own component images later with kfp component build, install kfp[all], which adds the Docker client library. You can work through the local-run section below with just this.

Pipelines on its own, on a kind cluster

Create a cluster, then install the standalone Pipelines release. The official steps use a version variable and two applies, with a wait between them so the custom resource definitions are registered before anything uses them:

kind create cluster
export PIPELINE_VERSION=2.17.0
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"
kubectl get pods -n kubeflow
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

Give it a few minutes; the pods in the kubeflow namespace move from ContainerCreating to Running. When the port-forward is running, open http://localhost:8080 and you see the Pipelines interface. The dev overlay uses a PostgreSQL database that is meant for trying things out. A production install uses the platform-agnostic overlay with MySQL, which is a topic for the Mid-level guide.

The full distribution

The complete platform installs from the community distribution repository. Check out the tag, create a kind cluster with the helper script, and apply everything in a retry loop:

git clone https://github.com/kubeflow/community-distribution.git
cd community-distribution
git checkout 26.03.1
./tests/install_KinD_create_KinD_cluster_install_kustomize.sh
kind get kubeconfig --name kubeflow > /tmp/kubeflow-config
export KUBECONFIG=/tmp/kubeflow-config
while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

The loop is not a hack; it is the documented approach, and it is worth understanding. Kubernetes has two kinds of things in the install: definitions of new object types (CRDs), and objects of those new types. If an object arrives before its type is registered, kubectl fails with no matches for kind ... in version ... or ensure CRDs are installed first. The retry loop simply tries again after 20 seconds, by which time the definitions exist. Seeing those errors in the first few rounds is expected.

If Docker Hub rate limits are a concern, log in and create a pull secret named regcred as shown in the repository's README before you start the loop. The community distribution also documents installing component by component, which is useful when a single piece fails and you want to isolate it.

Checking that it works

Wait for the namespaces to fill with running pods, then look at each in turn:

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n auth
kubectl get pods -n oauth2-proxy
kubectl get pods -n knative-serving
kubectl get pods -n kubeflow
kubectl get pods -n kubeflow-system
kubectl get pods -n kubeflow-user-example-com

Every pod should show Running or Completed. Then open the front door with a port-forward:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Browse to http://localhost:8080. You land on the Dex login page. The development account is user@example.com with the password 12341234, and its profile namespace is kubeflow-user-example-com.

Also note that plain HTTP works on localhost only. On any other address, the web applications set secure cookies, so you need HTTPS in front of the gateway. Otherwise logins loop or fail with cookie errors.

Your first pipeline, built step by step

We will build a tiny pipeline that mimics a real one: make some data, train a trivial "model", and report a score. The content is deliberately silly because the point is the shape. Start on your laptop with no cluster.

Step one: a component

A component starts as an ordinary Python function with a decorator. The type hints are required, because Kubeflow uses them to work out what goes in and out:

from kfp import dsl, compiler

@dsl.component(base_image="python:3.11")
def make_numbers(count: int) -> list:
    return [float(i) for i in range(count)]

@dsl.component(base_image="python:3.11")
def average(numbers: list) -> float:
    return sum(numbers) / len(numbers)

Two things deserve a pause. First, @dsl.component turns the function into something that will run in its own container, using the base image you name. The function body is copied into that container and executed there, so the function must be self-contained: any import it needs goes inside the body, and it cannot see variables from the rest of your file. Second, the return type is the contract. int, float, str, bool, list and dict are parameters that pass by value between steps.

If a step needs a library that is not in the base image, list it, and Kubeflow installs it when the step starts:

@dsl.component(base_image="python:3.11", packages_to_install=["pandas"])
def summarise(path: str) -> str:
    import pandas as pd
    return str(pd.read_csv(path).describe())

Notice the import inside the function. That is not style; it is required.

Step two: a pipeline

A pipeline is a function too, decorated with @dsl.pipeline. Its body wires components together. This is where beginners are caught out, so read carefully: the body of a pipeline function does not run as ordinary Python. It runs once, at compile time, only to record which steps exist and which outputs feed which inputs. Do not put if statements on values, loops over data, or print calls there and expect them to run at training time. Everything real happens inside components.

@dsl.pipeline(name="average-pipeline", description="Make numbers and average them")
def average_pipeline(count: int = 10) -> float:
    numbers = make_numbers(count=count)
    result = average(numbers=numbers.output)
    return result.output

Calling a component inside the pipeline returns a task. The task's .output is a placeholder that means "whatever this step returns, once it has run". Passing numbers.output into average creates the dependency: average will not start until make_numbers has finished. Steps that do not depend on each other can run at the same time.

Step three: compile

Compiling turns the Python into a YAML file:

compiler.Compiler().compile(average_pipeline, package_path="average_pipeline.yaml")

Open the YAML. You will see the pipeline's parameters, each component's container image and command, and the graph of dependencies. This file is the real product. It is portable, it can be uploaded to any Kubeflow Pipelines backend, and you can commit it to git and diff it in review. The Python is only a convenient way to write it.

Step four: run it locally

Local execution needs no cluster. You initialise a local runner, then call the pipeline like a function:

from kfp import local

local.init(runner=local.SubprocessRunner())

task = average_pipeline(count=5)
print(task.output)

SubprocessRunner runs each step as a subprocess in a virtual environment on your machine, which is quick and needs no Docker. If you want each step in a container just as on a cluster, use local.DockerRunner() instead; it is slower but closer to the real thing. Expect the first run to pause while the runner creates environments. The printed output is 2.0, the average of 0 to 4.

Local mode is the fastest feedback loop you will have. Use it to debug logic before you pay for cluster time.

Step five: run it on a cluster

With standalone Pipelines port-forwarded to port 8080, submit the same pipeline to the cluster:

from kfp.client import Client
from first_pipeline import average_pipeline

client = Client(host="http://localhost:8080")
run = client.create_run_from_pipeline_func(
    average_pipeline,
    arguments={"count": 20},
    experiment_name="first-experiments",
)
client.wait_for_run_completion(run.run_id, timeout=3600)
print("done", run.run_id)

This compiles the pipeline, creates the experiment if needed, starts a run, and waits. In the browser, open the run and you see the graph with each step coloured by state. Click a step to see its logs, its inputs and outputs, and the pod that ran it.

On the full multi-user distribution, the client needs to prove who you are. The simplest place to do this is from a notebook inside the cluster, which is covered in the notebooks section.

Passing files between steps: artifacts

Parameters are fine for a number. A trained model or a table of a million rows is not something you pass by value. For files, Kubeflow uses artifacts, which are stored in the object store and handed to each step as a path.

The type hints change. A step that writes a file declares an Output, and a step that reads it declares an Input:

from kfp import dsl
from kfp.dsl import Dataset, Model, Input, Output, Metrics

@dsl.component(base_image="python:3.11")
def prepare(out_data: Output[Dataset]):
    with open(out_data.path, "w") as f:
        f.write("x,y\n1,2\n2,4\n3,6\n")

@dsl.component(base_image="python:3.11")
def train(data: Input[Dataset], model: Output[Model], metrics: Output[Metrics]):
    rows = open(data.path).read().strip().split("\n")[1:]
    slope = sum(float(r.split(",")[1]) / float(r.split(",")[0]) for r in rows) / len(rows)
    with open(model.path, "w") as f:
        f.write(str(slope))
    metrics.log_metric("slope", slope)

@dsl.pipeline(name="artifact-pipeline")
def artifact_pipeline():
    d = prepare()
    train(data=d.outputs["out_data"])

Read the function signatures closely. out_data: Output[Dataset] is not a value you return; it is a handle. You write your file to out_data.path, and Kubeflow uploads it to the object store afterwards. In the next step, data: Input[Dataset] gives you a local path to the downloaded copy. You never write an S3 client yourself.

The artifact types carry meaning: Dataset, Model and Metrics are the common ones, and there are others for classification metrics, HTML and Markdown. The interface uses the type to show them properly; Metrics values appear in the run's metrics view, and you can compare them across runs.

Notice that when a component has an Output artifact, the pipeline refers to it by name through d.outputs["out_data"]. When a component has a single plain return value, you use .output. That split causes many beginner errors: if Kubeflow complains that a task has more than one output, use the named form.

Where do the files go? The object store named in the pipeline root. On the standard install this is a store reached through a service still called minio-service, although the default store is now SeaweedFS, a replacement for MinIO that speaks the same S3 protocol. You will see minio://mlpipeline/v2/artifacts as the default root. On a cloud cluster you point it at an S3 or GCS bucket instead, which is a configuration topic for later.

Runs, caching and reading the interface

Once a pipeline is on a cluster, you spend most of your time in a few screens. Learn what each is telling you.

The pipelines list holds uploaded pipeline definitions. A pipeline can have several versions, each an uploaded YAML. The experiments list groups runs. The runs list shows every execution with its status, duration and the experiment it belongs to. Open a run and the graph view draws your steps as boxes, coloured for pending, running, succeeded, failed or cached. Clicking a box opens a side panel with tabs for input and output, visualisations, details, logs and the events from Kubernetes.

When a step fails, go to the logs tab first. If there are no logs, the step never started, so switch to the events view. A failure before logs exist is almost always a Kubernetes problem rather than a Python problem: the image could not be pulled, the pod could not be scheduled for lack of memory, or a volume could not mount. A failure with logs is usually your own code, and the traceback is right there.

Caching is on by default and is the feature that surprises people most. If a step's definition and inputs are identical to a previous successful run, Kubeflow skips it and reuses the earlier outputs, marking the step as cached. For expensive steps this saves hours. For a step that reads the current time or pulls fresh data from outside, it silently gives you stale answers. You have two controls. Per run, pass enable_caching=False to create_run_from_pipeline_func. Per step, set task.set_caching_options(False) inside the pipeline:

@dsl.pipeline(name="no-cache-demo")
def no_cache_demo():
    t = make_numbers(count=5)
    t.set_caching_options(False)

You can also set a cache-off default for a whole compile by passing --disable-execution-caching-by-default to kfp dsl compile.

A recurring run repeats a pipeline on a schedule. It is created from the interface, from client.create_recurring_run, or from kfp recurring-run create. Older tutorials call these "jobs"; the SDK methods with "job" in their names are deprecated in favour of the recurring-run names and still print a deprecation warning.

Under the hood, the backend converts your compiled pipeline into an Argo Workflow and runs it. You do not need Argo knowledge to be productive, but it explains two things you will see in pod lists: each step has a pod, and the pod names look like generated workflow names. When you search with kubectl get pods -n <your namespace> during a run, those are your steps.

Profiles, the dashboard and notebooks

If you installed the full distribution, you work through the Central Dashboard, and the ideas here apply.

After logging in, the top bar shows a namespace selector. That is your profile. Everything you create, such as notebooks, runs and experiments, belongs to the namespace selected. Switching it changes which resources you see. A frequent beginner complaint is "my run disappeared", which almost always means the selector is on a different namespace.

To create a profile for a new user, an administrator applies a small YAML to the cluster:

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: aya-team
spec:
  owner:
    kind: User
    name: aya@example.com

kubectl apply -f profile.yaml
kubectl get profiles

The Profile controller reacts by creating the namespace aya-team, a role binding that makes the owner an administrator there, and two service accounts named default-editor and default-viewer. Self-service profile creation through the dashboard is off by default and turned on with the setting CD_REGISTRATION_FLOW=true on the dashboard deployment.

Also know the limits of this isolation. The documentation states plainly that profiles give no stronger separation than Kubernetes namespaces do. They keep honest colleagues out of each other's way. They are not a hard security boundary between mutually distrustful tenants.

Notebooks

Open Notebooks in the dashboard and choose New Notebook. You pick a name, an image (Jupyter with various libraries is offered, including GPU builds), CPU and memory, and a workspace volume. The workspace volume is a persistent disk mounted into the pod, and it is the only place files survive a restart. Anything you save elsewhere vanishes when the notebook stops.

Behind the page, a Notebook object is created, and the notebook controller turns it into a pod named <name>-0. If a notebook never becomes ready, inspect it the Kubernetes way:

kubectl get notebooks -n aya-team
kubectl describe notebook my-notebook -n aya-team
kubectl logs my-notebook-0 -n aya-team

The usual reasons are an image that cannot be pulled, a disk that cannot be created, or a namespace that has hit its resource quota. Culling, meaning automatically stopping idle notebooks, is off by default. An administrator enables it with ENABLE_CULLING on the notebook controller, and the idle time defaults to 1440 minutes, one day.

Talking to Pipelines from a notebook

The easiest authenticated client is one created inside a notebook. The distribution defines a PodDefault named access-ml-pipeline: a rule that, for notebooks carrying a matching label, mounts a short-lived token for the Pipelines API. When you create the notebook, tick the configuration called "Allow access to Kubeflow Pipelines" in the form. Then from that notebook:

import kfp
client = kfp.Client()
print(client.list_experiments())

Without the PodDefault you get Unauthenticated: Request header error: there is no user identity header, which means the request reached the API without a login or a token.

Training jobs and tuning

Not every job is a pipeline step. When you need to train across several machines, or try a hundred learning rates, Kubeflow has purpose-built tools.

Kubeflow Trainer

Trainer is the current way to run training jobs on Kubernetes. It replaces an older project called the Training Operator, which defined objects like PyTorchJob and TFJob. Those old objects still appear in tutorials, and the old operator has been removed from the development branch of the distribution. Write new work for Trainer.

A TrainJob is the object you create. It names a runtime, which an administrator has already installed. A runtime is a template for a framework, such as torch-distributed, so you do not have to know how to wire processes together. The shipped runtimes cover PyTorch, DeepSpeed, JAX, MLX and XGBoost, plus ready-made fine-tuning recipes for a few small language models.

Install the Trainer and its runtimes on a cluster with Kubernetes 1.31 or newer using Helm (see the Helm guide):

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
  --namespace kubeflow-system --create-namespace \
  --version 2.3.0 --set runtimes.defaultEnabled=true

The full distribution already includes Trainer, in the kubeflow-system namespace. The easiest way to use it is the Kubeflow SDK, a separate Python package that wraps several components behind one client:

pip install -U kubeflow

from kubeflow.trainer import TrainerClient, CustomTrainer

def train_fn():
    import torch
    print("hello from", torch.__version__)

client = TrainerClient()
for runtime in client.list_runtimes():
    print(runtime.name)

job = client.train(
    trainer=CustomTrainer(func=train_fn, num_nodes=2,
                          resources_per_node={"cpu": 3, "memory": "16Gi"}),
    runtime=client.get_runtime("torch-distributed"),
)
client.wait_for_job_status(job)
for line in client.get_job_logs(job, follow=True):
    print(line)

You hand Trainer a plain Python function, the number of machines, and the resources each one gets. Trainer packages the function, starts the pods and sets up the environment variables PyTorch needs to find its peers. client.list_runtimes() is how you discover what your administrator installed. If it returns nothing, the runtimes are missing, not your code.

Under the hood, the Kubernetes objects are TrainJob and a JobSet, and you can inspect them:

kubectl get trainjobs -n aya-team
kubectl describe trainjob <name> -n aya-team
kubectl get jobsets -n aya-team

The same job can be written as YAML and applied with kubectl, which suits GitOps workflows. Its API group is trainer.kubeflow.org/v1alpha1. The v1alpha1 label tells you the API is still evolving, and Trainer 2.2 already contained breaking changes, so pin versions and read release notes.

Katib for hyperparameter tuning

A hyperparameter is a setting you choose before training, such as a learning rate, that the training does not learn by itself. Finding a good one by hand is slow. Katib automates it: you define an Experiment with an objective (maximise accuracy, say), a search space, an algorithm such as random search, grid search, Bayesian optimisation or Hyperband, and a limit on trials. Katib's suggestion service proposes settings, launches a trial for each, collects the result and feeds it back.

The key detail for newcomers is how Katib learns the result. By default it reads your program's standard output and looks for lines of the form name=value. If the objective metric is called accuracy, your code must print exactly accuracy=0.93. A trial that never prints in that format ends as MetricsUnavailable.

import kubeflow.katib as katib

def objective(parameters):
    result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2
    print(f"result={result}")

client = katib.KatibClient(namespace="aya-team")
client.tune(
    name="tune-experiment",
    objective=objective,
    parameters={
        "a": katib.search.int(min=10, max=20),
        "b": katib.search.double(min=0.1, max=0.2),
    },
    objective_metric_name="result",
    max_trial_count=12,
    resources_per_trial={"cpu": "2"},
)
client.wait_for_experiment_condition(name="tune-experiment")
print(client.get_optimal_hyperparameters("tune-experiment"))

The kubeflow-katib package provides KatibClient and installs with pip install -U kubeflow-katib. Katib's API group is kubeflow.org/v1beta1. Every trial runs as pods in your namespace, so your resource quota matters; twelve trials at two CPUs each can starve a small cluster if they run in parallel.

Everyday commands, grouped by what you want to do

Keep this section open in a tab. Every command below is covered by the official CLI or client.

Write and compile.

kfp dsl compile --py pipeline.py --output pipeline.yaml
kfp dsl compile --py pipeline.py --output pipeline.yaml --pipeline-parameters '{"count": 10}'

Upload and manage pipelines. These need --endpoint pointing at your API, or the environment set up for it:

kfp --endpoint http://localhost:8080 pipeline create -p my-pipeline pipeline.yaml
kfp --endpoint http://localhost:8080 pipeline list
kfp --endpoint http://localhost:8080 pipeline create-version -p PIPELINE_ID -v v2 pipeline.yaml
kfp --endpoint http://localhost:8080 pipeline delete PIPELINE_ID

Run and watch.

kfp --endpoint http://localhost:8080 experiment create first-experiments
kfp --endpoint http://localhost:8080 run create -e first-experiments -r run-one -f pipeline.yaml -w count=10
kfp --endpoint http://localhost:8080 run list -e EXPERIMENT_ID
kfp --endpoint http://localhost:8080 run get RUN_ID -w

The -w flag makes the command wait until the run finishes; key=value pairs at the end set the pipeline's parameters.

Schedule. kfp recurring-run create, list, get, enable, disable and delete manage scheduled runs.

Inspect Kubernetes objects.

kubectl get pods -n aya-team
kubectl describe pod POD_NAME -n aya-team
kubectl logs POD_NAME -n aya-team
kubectl get profiles
kubectl get notebooks -n aya-team
kubectl get trainjobs -n aya-team

Python client. The same operations exist on kfp.Client: create_run_from_pipeline_func, create_run_from_pipeline_package, list_runs, get_run, wait_for_run_completion, terminate_run, upload_pipeline, list_pipelines, create_experiment and list_experiments. Two renamings to remember: create_recurring_run replaces the old "job" functions, and GPU requests use set_accelerator_type and set_accelerator_limit rather than the deprecated set_gpu_limit.

Ask for resources per step. This is how a step gets memory and a GPU:

train_task = train(data=d.outputs["out_data"])
train_task.set_cpu_limit("4")
train_task.set_memory_limit("8G")
train_task.set_accelerator_type("nvidia.com/gpu")
train_task.set_accelerator_limit(1)

A step that asks for more than any node can give stays Pending forever, with no error in the logs; the Kubernetes events say Insufficient cpu or Insufficient memory.

Configuration and the most common errors

The few settings worth knowing now

You will not configure much at this level, but four settings explain a lot of behaviour.

The pipeline root says where artifacts are stored. Each profile namespace has a ConfigMap named kfp-launcher with a key defaultPipelineRoot. On the default install it points at the bundled object store. To use a cloud bucket, an administrator changes this value and supplies credentials. You can also override it for a single pipeline with pipeline_root when submitting.

The default base image for components is a recent Python 3.11 image. Naming base_image explicitly, as we did, makes your pipeline independent of that default.

Namespace for the client. On a multi-user install, runs belong to a namespace. Pass namespace="aya-team" when you create runs, or call client.set_user_namespace("aya-team") once; it is remembered in $HOME/.config/kfp/context.json.

Culling and registration for notebooks and profiles, as described above, are switched with environment variables on the respective deployments.

Reading the errors

Kubeflow errors usually come from one of the four layers in the stack diagram. Match the message to the layer and the fix becomes obvious.

Another message deserves its own paragraph: "output_metadata.json": proto: ... unknown field "custom_path". It appears when a pipeline compiled with SDK 2.15.0 or 2.15.1 runs against an older backend. The fix is to upgrade the SDK to 2.15.2 or later, or to upgrade the backend, which is why the SDK and backend versions should stay matched.

When you are lost, use a fixed order. First, is the step in the run graph failing with logs? Read the traceback. Without logs, use kubectl describe pod on the step's pod and read the events at the bottom. If the page itself will not load, check the ingress and login pods in istio-system, oauth2-proxy and auth. That three-step habit solves most problems before any search engine is needed.

Putting it all together

Here is one small, complete project that uses what you have learned. It is a pipeline that prepares data, trains a toy model with a configurable setting, evaluates it, and only reports success if the score clears a threshold. The project has three files.

from kfp import dsl, compiler
from kfp.dsl import Dataset, Model, Input, Output, Metrics

@dsl.component(base_image="python:3.11")
def prepare(rows: int, out_data: Output[Dataset]):
    with open(out_data.path, "w") as f:
        f.write("x,y\n")
        for i in range(1, rows + 1):
            f.write(f"{i},{2 * i}\n")

@dsl.component(base_image="python:3.11")
def train(data: Input[Dataset], scale: float, model: Output[Model]):
    lines = open(data.path).read().strip().split("\n")[1:]
    ratios = [float(l.split(",")[1]) / float(l.split(",")[0]) for l in lines]
    with open(model.path, "w") as f:
        f.write(str(scale * sum(ratios) / len(ratios)))

@dsl.component(base_image="python:3.11")
def evaluate(model: Input[Model], metrics: Output[Metrics]) -> float:
    slope = float(open(model.path).read())
    error = abs(slope - 2.0)
    metrics.log_metric("error", error)
    return error

@dsl.pipeline(name="toy-training", description="prepare, train, evaluate")
def toy_training(rows: int = 50, scale: float = 1.0):
    d = prepare(rows=rows)
    t = train(data=d.outputs["out_data"], scale=scale)
    evaluate(model=t.outputs["model"])

if __name__ == "__main__":
    compiler.Compiler().compile(toy_training, package_path="toy_training.yaml")

from kfp import local
from pipeline import toy_training

local.init(runner=local.SubprocessRunner())
toy_training(rows=20, scale=1.0)

from kfp.client import Client
from pipeline import toy_training

client = Client(host="http://localhost:8080")
for scale in [0.5, 1.0, 1.5]:
    client.create_run_from_pipeline_func(
        toy_training,
        arguments={"rows": 100, "scale": scale},
        experiment_name="toy-scale-sweep",
    )

The workflow is the one you would use on real work. You iterate with run_local.py until the logic is right, which costs nothing. You then compile and commit toy_training.yaml so the exact definition is reviewed in git. You submit three runs on the cluster, one per scale, all inside one experiment. Open that experiment in the interface, select the runs and compare the error metric: the run with scale=1.0 has the lowest error, which you can see at a glance because the metrics are logged against each run.

Two habits make this scale up. Every step names its base image, so it runs the same tomorrow. Every large output is an artifact, so nothing depends on memory between steps. Replace the toy steps with your real data loading and training code, keep the same skeleton, and you have a real pipeline.

When you want to tune scale automatically instead of by hand, that is Katib's job. When you want to run the training step across several GPUs, that is Trainer's. When the model is good enough to serve, continue to the KServe guide. If you want to log parameters and metrics across many experiments with a lighter tool alongside this, see the MLflow guide.

What you can now do, and what comes next

You can explain what Kubeflow is and why it is a family of projects rather than one program. You know that version numbers now follow the calendar, that the current bundle is 26.03.1, and that pipelines, Trainer, Katib, Notebooks, Hub and KServe each have a separate job. You can name the layers a request passes through and use that to locate a failure. You have installed the SDK, written components and a pipeline, compiled them to YAML, run them locally and on a cluster, passed files between steps as artifacts, used caching on purpose, and read a run in the interface. You can create a profile, start a notebook that survives restarts, launch a training job with the Kubeflow SDK and tune a function with Katib. And you can read the ten most common error messages.

What to practise next: write one pipeline for your own project rather than a toy, even if it has only three steps. Run it twice and watch caching. Break it deliberately and read the failure. Those three acts teach more than another hour of reading.

The Mid-level guide picks up with control flow (conditionals, loops, parallel fan-out and exit handlers), resource and secret management with kfp-kubernetes, how the backend works internally, production object stores and databases, and connecting your cluster to CI and GitOps. The Senior guide covers running Kubeflow as a platform for other teams: multi-tenancy limits, upgrades, security and capacity. Neighbouring guides in this catalogue worth reading alongside: Kubernetes for the substrate, Docker for the component images, KServe for serving, and MLflow for experiment tracking.

One last piece of advice for learning on your own. Keep a notebook of the failures you meet, with the message, the layer it came from and the fix. Kubeflow is a platform where the same dozen problems recur in different disguises: a missing permission, a pod that cannot start, a stale cache, a mismatched version. After a month, your own list will be a better reference than any guide, including this one, and it is the thing interviewers respond to most when you describe real debugging stories from your projects.

Sources

• Kubeflow documentation: https://www.kubeflow.org/docs/
• Kubeflow Community Distribution (manifests and install): https://github.com/kubeflow/community-distribution
• Kubeflow Pipelines documentation: https://www.kubeflow.org/docs/components/pipelines/
• Kubeflow Pipelines local execution: https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/execute-kfp-pipelines-locally/
• Kubeflow Trainer documentation: https://trainer.kubeflow.org/
• Kubeflow SDK documentation: https://sdk.kubeflow.org/
• Katib documentation: https://www.kubeflow.org/docs/components/katib/
• Kubeflow Notebooks documentation: https://www.kubeflow.org/docs/components/notebooks/
• Kubeflow Spark Operator: https://spark.kubeflow.org/

This is the interview companion to the beginner guide. Kubeflow interviews for junior and entry-level roles rarely ask you to recite YAML. They test whether you understand what the platform is for, whether you can explain its moving parts in plain language, and whether you can reason about a failure without panicking. Read the beginner guide first, then use this file to practise saying things out loud.

The thirty-second answer

Kubeflow is an open-source set of projects that runs the machine learning lifecycle on Kubernetes. Its best-known part, Kubeflow Pipelines, lets you write a workflow in Python, compile it to a YAML description, and run it as a graph of containers with tracked inputs, outputs and runs. Around it sit projects for notebooks, distributed training, hyperparameter tuning and model registration, and KServe is commonly paired with it for serving.

If you have one extra breath, add the honest trade-off: it is powerful when you already have a Kubernetes cluster and several people sharing it, and heavy when you do not. That sentence tells an interviewer you have judgement, not just vocabulary.

Practise saying it without the word "platform" twice. Interviewers often follow with "so what does it actually do for me?", and the answer is reproducibility: the same pipeline definition, the same container images, and a recorded history of every run.

The model you should be able to draw

Interviewers like a whiteboard. Be able to sketch four layers from memory: you, at the top, using a browser, the Python SDK or kubectl; the front door, made of the Istio gateway, a login proxy and the Central Dashboard; the components, meaning Pipelines, Trainer, Katib, Notebooks, Hub and KServe; and Kubernetes at the bottom, where everything becomes pods and volumes.

Then name the nouns in each group and stop. For pipelines, the nouns are component, pipeline, run, experiment, parameter and artifact. For multi-user work, they are profile, namespace and contributor. For training, they are TrainJob and runtime. For tuning, they are experiment, trial and suggestion. Candidates who can explain the difference between a pipeline and a run, or between a parameter and an artifact, clearly stand out from candidates who only know the logo.

A few facts to keep ready, because they show you are current. Kubeflow is not one program with one version. The bundle of everything is the Kubeflow Community Distribution, and since 2026 it uses calendar versions such as 26.03.1, with no 1.12. Kubeflow Pipelines is on its second-generation SDK, and the first-generation constructs have been removed. The Training Operator's old job types have been superseded by Kubeflow Trainer. The model registry was renamed Kubeflow Hub. If you mention these, say the detail that changed and nothing more; an interviewer can test anything you claim.

The commands to have ready

You will not be asked to type these, but you may be asked to talk through them, or to write them on a whiteboard. Know what each does and when you would reach for it.

pip install kfp==2.17.0
kfp dsl compile --py pipeline.py --output pipeline.yaml
kfp run create -e my-experiment -r run-one -f pipeline.yaml -w
kubectl get pods -n kubeflow
kubectl describe pod POD_NAME -n NAMESPACE
kubectl logs POD_NAME -n NAMESPACE
kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
kubectl get profiles

The first installs the SDK; you pin the version so it matches the backend. The second compiles without running Python yourself. The third starts a run inside a named experiment and waits for it. The last five are the everyday Kubernetes inspection tools: what is running, why is a pod unhappy, what did it print, how do I reach a web page on the cluster from my laptop, and which user profiles exist.

And a minimal piece of Python you should be able to write from memory:

from kfp import dsl, compiler

@dsl.component(base_image="python:3.11")
def add(a: float, b: float) -> float:
    return a + b

@dsl.pipeline(name="add-pipeline")
def my_pipeline(x: float = 1.0, y: float = 2.0) -> float:
    return add(a=x, b=y).output

compiler.Compiler().compile(my_pipeline, package_path="pipeline.yaml")

Be ready to point at each part: the decorator that turns a function into a containerised step, the type hints that define the inputs and outputs, the pipeline function that only wires steps together, and the compile call that produces the YAML.

The classic traps and why they happen

These come up in almost every Kubeflow interview, often disguised as "a colleague says their pipeline does something odd, what do you think is happening?".

The pipeline body is not normal Python. A pipeline function runs once, at compile time, to record the graph. An if on a task's output or a print of a value inside the pipeline function will not do what the author expects, because the value does not exist yet; it is a placeholder for something a container will produce later. The fix is to move the logic into a component, or use the conditional constructs provided for pipelines. If you can explain this in two sentences, you understand the core abstraction.

Components are self-contained. The body of a lightweight component is copied into a fresh container. It cannot see global variables or imports from the file that defines it. A NameError inside a step, for a name that clearly exists in the source file, is the telltale symptom. The fix is to import inside the function and to list extra libraries in packages_to_install.

Caching returns old answers. Caching is on by default, so a step whose definition and inputs are unchanged is skipped and its earlier output reused. A step that reads today's date or pulls fresh data gives stale results. The remedy is to turn caching off for that step, or for the run, and to make inputs explicit so that a change in data changes the cache key.

Big data as a parameter. Parameters are small values passed by value and stored in Kubernetes objects, which have size limits. Files and datasets belong in artifacts, which live in the object store. Weak candidates pass a huge list; strong ones reach for an Output[Dataset].

"My run vanished." On a multi-user installation, everything is per namespace. The run exists; you are looking at another profile. The namespace selector in the dashboard is the first thing to check.

A step is stuck in Pending. This is a Kubernetes scheduling problem, not a Python problem. The pod asked for more CPU, memory or GPU than any node has free. Logs do not exist because the container never started, so you read the pod's events with kubectl describe pod.

Login loops on a non-localhost address. The web applications set secure cookies, which browsers only accept over HTTPS. The proper fix is TLS in front of the gateway; turning the secure-cookie setting off is a last resort.

Old tutorials. A tutorial that mentions ContainerOp, v2-compatible mode or PyTorchJob as the way to train is out of date. A sign of a strong candidate is noticing this and saying which current construct replaces it.

Deleting the wrong thing. Deleting a Profile deletes its namespace and everything inside. Deleting the Profiles definition deletes every profile on the cluster. Mention this as something you would take care over.

Common interview questions

What problem does Kubeflow solve, and when would you not use it?

It solves the gap between a model that works once in a notebook and a process that can be repeated, scheduled, shared and audited. It packages that process as pipelines of containers on Kubernetes, and adds notebooks, distributed training and tuning around it. I would not use it if the team has no Kubernetes cluster and no one to run one, or if the need is only experiment tracking, where a lighter tool is a better fit. The full distribution needs roughly 4 CPU cores and 12 GiB of memory before any workload, so it has to be worth it.

What is the difference between a pipeline and a run?

A pipeline is the definition: the graph of steps and its parameters, compiled into YAML. A run is one execution of it with particular argument values. The same pipeline can have many runs, grouped into experiments. A recurring run is a run that repeats on a schedule.

What is a component?

One step of a pipeline, packaged as a container. The most common kind is a lightweight Python component: a Python function with a decorator, where the type hints declare the inputs and outputs. Its code is copied into a container built from a base image you name. There are also container components, where you supply an image and command directly.

What is the difference between a parameter and an artifact?

A parameter is a small value, such as a number, string, list or dictionary, passed between steps by value. An artifact is a file, such as a dataset or model, stored in the object store and handed to the next step as a path through Input and Output types. Artifacts keep large data out of Kubernetes objects, and their types let the interface display them properly.

What does compiling a pipeline produce, and why does it matter?

A YAML file describing the pipeline: its parameters, each step's image and command, and the dependency graph. It matters because it is portable and reviewable. You can commit it, diff it, upload it to any compatible backend, and know exactly what will run.

How would you debug a failed step?

I start with the layer. If the step has logs, I read them; the traceback is usually my own code. If it has no logs, the container never started, so I find the step's pod and run kubectl describe pod to read the events, looking for image pull errors, insufficient resources, or volume problems. If the whole interface is down, I look at the gateway and authentication pods. I also check whether caching or a stale image is involved before I change any code.

What is caching and how do you control it?

If a step's definition and inputs match an earlier successful execution, Kubeflow reuses the earlier output rather than rerunning. It is on by default. I can disable it per step with set_caching_options(False), per run with enable_caching=False, or for a compile with --disable-execution-caching-by-default.

Can you run a pipeline without a cluster?

Yes, for development. The SDK has a local mode: you call local.init with a SubprocessRunner or a DockerRunner and then call the pipeline like a function. It is good for checking logic quickly; it does not replace a cluster run, because there are no real pods, quotas or object store.

What is a Profile?

Kubeflow's unit of multi-user isolation. Creating one produces a namespace with the same name, makes the owner an administrator of it, and creates standard service accounts. Users see their own namespace's notebooks and runs, and the owner can add contributors. It is namespace-level separation, and the documentation is clear that it is not a hard security boundary between tenants who do not trust each other.

How does a user log in, and how does Kubeflow know who they are?

A login proxy, oauth2-proxy, optionally backed by Dex, authenticates the user. Istio then adds a header named kubeflow-userid carrying the user's email to each request, and the web applications and the Pipelines API trust that header. That is also why calling the API directly, bypassing the gateway, fails with "no user identity header".

What replaced the old Training Operator?

Kubeflow Trainer. Instead of separate job types per framework, there is a single TrainJob that refers to a runtime, which is a blueprint an administrator installs for a framework such as PyTorch. Users pick a runtime and supply code, number of nodes and resources. The old job types are legacy.

What does Katib do, and how does it learn a trial's result?

Katib automates hyperparameter tuning. An experiment defines an objective, a search space, an algorithm and a budget; Katib's suggestion service proposes settings and launches a trial for each. By default it reads the program's standard output for lines like accuracy=0.93, so the metric name and format have to match exactly, or the trial ends with no metrics.

What version of Kubeflow are you talking about?

The community distribution at 26.03.1, which uses calendar versioning. Its components have their own versions: Pipelines 2.16.1, Trainer 2.2.0, Katib 0.19.0 and so on. Newer standalone versions of some exist. I always say which version I mean, because features and commands differ between them.

How is Kubeflow different from MLflow?

MLflow is mostly about tracking experiments, packaging models and keeping a registry, and it runs happily on a laptop. Kubeflow is about running workloads on Kubernetes: pipelines, training jobs, notebooks and serving. They are complementary rather than rivals, and there is an experimental integration that lets them work together.

What would you change first on a fresh installation of the full distribution?

The default user and password. The default account is documented publicly, so on any cluster that other people can reach, I would replace it and connect a real identity provider, then make sure traffic goes over HTTPS.

How do you give a step a GPU?

Through the task configuration: set_accelerator_type("nvidia.com/gpu") and set_accelerator_limit(1). The older names set_gpu_limit and add_node_selector_constraint still work but print deprecation warnings. The cluster must have GPU nodes and the device plugin installed, or the step stays pending.

What a strong answer sounds like

A weak answer recites a feature list. A strong answer states the problem, gives the smallest accurate description of the mechanism, and mentions one cost or limit. For example, to "what is caching?", the weak answer is "it saves time". The strong answer is "it reuses a previous result when a step and its inputs are unchanged, which is great for slow steps, but it will return stale output for a step that depends on something outside its inputs, so I turn it off for those."

Another habit that helps: when you are unsure, say what you would check, not what you guess. "I would look at the pod's events" is a better answer than an invented cause.

If the interviewer asks about a feature you have not used, say so and relate it to one you have. "I have not used Spark Operator, but I understand it runs Spark jobs as Kubernetes objects, the way Trainer runs training jobs" shows that you understand the pattern. Bluffing costs far more than admitting a gap.

Finally, be careful with absolutes about versions. Kubeflow changes quickly; "as of the 26.03 release" is a safer way to start a sentence than "Kubeflow always".

Sixty-second self-test

Answer each out loud, without looking. If you hesitate on more than two, reread the matching section of the beginner guide.

1. In two sentences, what is Kubeflow, and what does it run on?
2. Name the four layers a request passes through, from the browser down.
3. What is the difference between the bundle and a subproject, and what does 26.03.1 mean?
4. Why does a pipeline function not run as ordinary Python?
5. Why must a lightweight component import its libraries inside the function?
6. When do you use an artifact instead of a parameter?
7. What does kfp dsl compile produce, and why commit it?
8. Your step is Pending with no logs. Where do you look?
9. What is a Profile, and what happens when you delete one?
10. How does Katib find out how well a trial did?
11. Which object replaced PyTorchJob, and what does a runtime contain?
12. You need to show results are not stale. How do you turn off caching for one step?

If you can answer all twelve in under a minute each, you are ready for a beginner-level Kubeflow interview. Move on to the Mid-level files for control flow, secrets, production storage and debugging beyond the basics, and pair this guide with the Kubernetes interview notes, because a large share of Kubeflow questions are really Kubernetes questions in disguise.

This file collects the mistakes beginners actually make with Kubeflow, each with the symptom you will see, the cause, and the fix, followed by habits worth adopting from your first day and a checklist. Most of these are not Kubeflow bugs. They come from the gap between how the tool looks (a Python library) and how it behaves (a system that turns your code into containers on a cluster).

Mistakes beginners actually make

Treating the pipeline function as ordinary Python

Symptom. You put a print, an if on a task's output, or a loop over results inside the @dsl.pipeline function. Nothing prints at run time, or compilation fails with an error about using an output in a condition.

Cause. The pipeline function runs once, at compile time, only to record which steps exist. A task's .output is a placeholder, not a value. The real values exist much later, inside containers.

Fix. Keep the pipeline function to wiring: call components, pass outputs to inputs. Put any computation in a component. When you need branching or repetition, use the pipeline constructs made for it, such as dsl.If and dsl.ParallelFor, which the Mid-level guide covers. Note that dsl.Condition is the deprecated spelling of dsl.If.

Forgetting that a component is isolated

Symptom. A step fails with NameError: name 'pd' is not defined or ModuleNotFoundError, even though the import sits at the top of your file and works locally.

Cause. The function's code is copied into a fresh container built from the base image. It does not carry your file's imports or global variables, and the image may not have your libraries.

Fix. Put imports inside the function body. List extra libraries in packages_to_install, for example @dsl.component(base_image="python:3.11", packages_to_install=["pandas"]). If the installation is slow or you need system libraries, build your own image and use it as base_image.

Missing or wrong type hints

Symptom. The compile step complains that a parameter has no type, or a run fails because a value arrives as a string when you expected a number.

Cause. Kubeflow reads the type hints to define the step's interface. Without them it cannot build the contract between steps.

Fix. Annotate every parameter and the return value. Use int, float, str, bool, list and dict for values, and Input[Dataset], Output[Model] and friends for files. Keep the annotations simple; the more exotic the type, the more likely it is unsupported.

Returning or passing big data as a parameter

Symptom. A run fails oddly when a dataset grows, with errors about size, or the interface becomes slow.

Cause. Parameters are stored inside Kubernetes objects, which have size limits. They are meant for small values.

Fix. Write large data to an Output[Dataset] path and read it with Input[Dataset]. Store only identifiers, counts and thresholds as parameters.

Mixing up .output and .outputs["name"]

Symptom. An error saying a task has multiple outputs, or that an output cannot be found.

Cause. A component with a single plain return value exposes it as .output. A component with named artifact outputs exposes them by name.

Fix. Use task.output for a single return value and task.outputs["name"] for artifacts, where the name is the parameter name in the function signature.

Being surprised by caching

Symptom. You changed the data or an external system, reran the pipeline, and got the same answer in seconds. The graph shows cached steps.

Cause. Caching is on by default. A step whose definition and inputs are unchanged is skipped. Changes outside those inputs, such as a table in a database or a file at a fixed URL, are invisible to the cache.

Fix. Turn caching off where freshness matters: task.set_caching_options(False) for a step, enable_caching=False for a run, or --disable-execution-caching-by-default at compile time. Better, make the changing thing an input, such as a date or a dataset version, so the cache key changes when it does.

Looking in the wrong namespace

Symptom. "My notebook, run or experiment disappeared" after logging back in.

Cause. On the multi-user distribution, everything belongs to a profile namespace, chosen in the selector at the top of the dashboard. Yours may have reset to another namespace.

Fix. Check the selector first. In code, pass namespace="your-profile" to the client or call set_user_namespace so it stays explicit. When using kubectl, add -n your-profile, and remember that forgetting it makes kubectl look in the default namespace.

Diagnosing the wrong layer

Symptom. You spend an hour reading Python code for a failure that happens before any code runs.

Cause. A step with no logs has not started. The problem is in Kubernetes: a missing image, a volume that cannot mount, or a pod that cannot be scheduled.

Fix. Follow a fixed order. Step has logs: read the traceback. Step has no logs: kubectl describe pod on its pod and read the events at the bottom. Interface will not load: check the gateway, login and Dashboard pods. The Kubernetes guide covers reading pod events in depth.

Asking for resources the cluster does not have

Symptom. A step stays Pending indefinitely. No logs, no error in the graph.

Cause. The pod requests more CPU, memory or GPUs than any node has free, or asks for a GPU type the cluster does not offer. The Kubernetes scheduler waits forever.

Fix. Run kubectl describe pod and look for events such as Insufficient cpu, Insufficient memory or Insufficient nvidia.com/gpu. Reduce the request, add nodes, or free capacity. Use set_cpu_limit, set_memory_limit, set_accelerator_type and set_accelerator_limit deliberately; the old set_gpu_limit only prints a deprecation warning.

Starting with the heaviest install

Symptom. The full distribution will not finish installing on your laptop. Pods restart, the machine freezes, or the retry loop never ends.

Cause. The reference bundle needs about 4.4 CPU cores and 12 GiB of memory before any workload, and the recommendation is 16 GB and 8 cores. A laptop with a container runtime that has only 4 GB allocated cannot hold it.

Fix. Begin with the SDK and local execution, which need no cluster. Then install standalone Pipelines on a kind cluster. Move to the full distribution when you need multi-user features, notebooks and Trainer together, and give the virtual machine 16 GB if you can. On Mac and Windows, remember the memory limit belongs to the virtual machine that runs Docker, or to WSL, not to the physical machine.

Forgetting the file-watch limits on Linux

Symptom. Random pod crashes during install on a Linux machine, with messages about "too many open files" or file watchers.

Cause. The default inotify limits are too low for a cluster running dozens of controllers inside kind.

Fix. Run sudo sysctl fs.inotify.max_user_instances=2280 and sudo sysctl fs.inotify.max_user_watches=1255360, then restart the cluster. To keep the values after a reboot, put them in a file under /etc/sysctl.d/.

Panicking at the first apply errors

Symptom. The install prints red errors such as no matches for kind or ensure CRDs are installed first and you assume it has failed.

Cause. The install applies custom resource definitions and objects of those types together. On the first pass some objects arrive before their types exist.

Fix. Do nothing; the documented one-line install retries every 20 seconds until it succeeds. Worry only if the same error persists after many rounds, or if a different error appears, such as an image pull failure.

Hitting Docker Hub limits

Symptom. Pods are in ImagePullBackOff and the event says toomanyrequests: You have reached your pull rate limit.

Cause. Docker Hub limits anonymous pulls, and an installation pulls many images from one address.

Fix. Log in to Docker Hub and create a pull secret named regcred from your Docker configuration file before installing, as the distribution's README shows. Or mirror the images to a registry you control.

Loading the dashboard over plain HTTP from another machine

Symptom. Logging in works on localhost, but from another host the pages loop back to login, or the Jupyter and Katib pages show cookie or CSRF errors.

Cause. The web applications set secure cookies, which browsers accept only over HTTPS, apart from localhost.

Fix. Put TLS in front of the Istio ingress gateway, through an ingress or a load balancer, and keep the gateway service internal. Turning secure cookies off with APP_SECURE_COOKIES=false exists but is not recommended.

Leaving the default password

Symptom. None, until someone else finds your cluster.

Cause. The default account user@example.com with the password 12341234 is published in the documentation.

Fix. Change it before exposing the cluster beyond your machine. Generate a bcrypt hash, store it in the Dex password secret, and restart the authentication pods, following the distribution's instructions. For a shared cluster, connect Dex to your organisation's identity provider.

Copying tutorials from older versions

Symptom. Imports fail, objects are rejected, or pages you are told to click do not exist.

Cause. Kubeflow changed a lot recently. Versions now use the calendar. ContainerOp and the first-generation pipeline SDK are gone. Training Operator job types such as PyTorchJob are legacy in favour of TrainJob. The model registry is now Kubeflow Hub, and MinIO has given way to SeaweedFS as the default object store, though the service keeps the old name.

Fix. Check the date and version of every tutorial. Prefer the official docs and the release you installed. When you see a legacy name, search the current docs for its replacement before writing any code.

Katib trials that never report a result

Symptom. A tuning experiment runs trials that finish but show MetricsUnavailable, and the experiment never finds a best value.

Cause. Katib's default collector reads standard output for name=value. If your code prints Accuracy: 0.93, or prints nothing, it finds nothing. In some namespaces the metrics collector is also not injected.

Fix. Print exactly the metric name you configured, an equals sign and a number, for example print("accuracy=0.93"). Make sure the namespace carries the label katib.kubeflow.org/metrics-collector-injection=enabled; the kubeflow namespace has it already.

Deleting too much

Symptom. A namespace, notebooks and months of runs vanish.

Cause. kubectl delete profile NAME removes the profile's namespace and everything in it. Deleting the Profiles definition removes every profile on the cluster.

Fix. Treat Profile deletion like dropping a database. Never delete the Profiles CRD. Before removing anything, list what lives in the namespace, and back up data you need.

Habits worth adopting on day one

Run locally first. The SDK's local runner, local.init(runner=local.SubprocessRunner()), lets you call a pipeline like a function on your laptop. Every bug you catch here costs seconds instead of cluster minutes. Use DockerRunner when you want the same container behaviour as a cluster.

Pin every version. Pin the SDK (kfp==2.17.0), the base image tag (python:3.11, not latest) and the tag of the distribution you install (26.03.1). Kubeflow changes quickly and mismatched SDK and backend versions produce errors such as unknown field "custom_path". Write the versions in a file you commit.

Commit the compiled YAML. The YAML is the real definition of what will run. Reviewing a diff of it in a pull request catches changes the Python hides, such as a base image that moved. Pair this with the ideas in the Kustomize guide when you deploy the platform itself.

Name things. Give pipelines, experiments and runs names that say what they are for. toy-scale-sweep tells your teammate what to expect; test3 does not. Use the experiment as a folder with a purpose.

Keep components small and single-purpose. One step that loads, cleans, trains and evaluates cannot be cached, retried or reused in pieces. Split along the places where an output is worth keeping.

Log metrics as metrics. Use Output[Metrics] and metrics.log_metric("accuracy", value), so runs can be compared in the interface instead of by scrolling logs.

Declare resources honestly. Give steps realistic CPU and memory limits from the start. Too low and they are killed; too high and they wait for capacity. Adjust based on what you observe, not on a guess.

Learn five kubectl commands. get pods -n NS, describe pod, logs, port-forward and get events -n NS. They answer most "why is it stuck" questions and they transfer to every Kubernetes tool, from Helm to serving with KServe.

Read release notes before upgrading. Several components have had breaking changes inside a single release line, including Dashboard 2.0 and Trainer 2.2. An upgrade is a planned task, never a casual git pull.

Write down what you ran. A short README with the exact install commands, versions and the URL pattern you used for port-forwarding saves a teammate, or future you, a day.

Stay within the tool's real limits. Profiles are namespace-level separation, not a security boundary, and pipeline definitions are shared across namespaces in the current release. Do not put secrets in a pipeline's parameters or in code. If a course project involves sensitive data from a regulated employer, ask which region the cluster and the object store run in before you start.

A short checklist

Before you start:

• I decided whether I need local runs, standalone Pipelines, or the full distribution.
• My container runtime or WSL has the memory I planned for.
• On Linux, I raised the two inotify limits.
• I know which distribution tag or SDK version I am following.

Before you run a pipeline:

• Every component has type hints on all inputs and outputs.
• Imports are inside the component functions, and extra libraries are listed.
• Large data travels as artifacts, and small values as parameters.
• I ran it locally first.
• I decided whether caching is right for each step.
• Resource requests fit the cluster.

When something fails:

• Does the step have logs? I read the traceback.
• No logs? I ran kubectl describe pod and read the events.
• Page will not load? I checked the gateway and login pods.
• I checked the namespace selector and the version of any tutorial I followed.

Before I share or expose anything:

• I changed the default password.
• HTTPS is in front of the gateway.
• I committed the compiled YAML and the pinned versions.
• I never placed a secret in a parameter or in source code.

Keep this list next to your first project. After a few weeks the items become reflexes, and you can move on to the Mid-level guide, where the same topics appear again with control flow, secrets, production storage and deeper debugging.

One closing habit

Pick one real problem from your own work and build it as a three-step pipeline within your first week. Write down what surprised you, then fix the surprises using the entries above. Beginners who learn Kubeflow through a toy example often understand the syntax but freeze on the first real dataset, because real data forces every decision in this file: what is a parameter and what is an artifact, which steps to cache, how much memory to ask for, and where to look when a pod never starts. Doing it once on something you care about turns those decisions from rules into instincts, and it gives you a concrete project to show in a portfolio or an interview.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. You already run pipelines and notebooks on a Kubeflow cluster. Here we explain why they behave the way they do, so you can predict failures before you hit them.

Where this picks up

Each section starts with the problem it solves and ends, for most of them, with a Try it you can do on a real cluster in a few minutes. Everything here is checked against the Kubeflow Community Distribution release 26.03.1 (Kubeflow Pipelines 2.16.1, Trainer v2.2.0, Katib v0.19.0). Where the standalone projects have moved on, such as KFP 2.17, the text says so.

Kubeflow is a bundle, and the bundle has a release process

The first thing to get straight at work is what "our Kubeflow version" means, because the answer decides how you upgrade and whom you ask for help. Kubeflow is not one binary. It is a set of independently released subprojects: Pipelines, Trainer, Katib, Notebooks, the Dashboard, the Hub model registry and the Spark Operator. The Kubeflow Community Distribution (KCD), in the repository kubeflow/community-distribution, is a curated set of Kustomize manifests that pins a compatible version of each one, plus the infrastructure they need: Istio, cert-manager, Dex, oauth2-proxy and Knative. Older material calls that repository kubeflow/manifests, and GitHub redirects the old name.

Versioning changed in 2026. The last semantic-versioned bundle was v1.11.0. Since then the distribution uses calendar versions in the form YY.MM[.PATCH]: a base release such as 26.03, and patches such as 26.03.1. If a colleague says "we are on 1.12", they are wrong or they are describing a vendor build. Two base releases per year are the target, and the community supports each one on a best-effort basis for about six months. That single number should shape your planning: you will upgrade about twice a year, so you want the procedure to be boring.

Two things follow from the bundle model. First, subproject versions move independently of the bundle. KCD 26.03.1 ships KFP 2.16.1, while the standalone KFP had already reached 2.17 by the time of writing. The SDK you install on your laptop and the backend running on the cluster are separate packages, and keeping them on the same minor version is the rule that prevents a whole class of confusing errors (we return to one in the debugging section). Second, KServe is an ecosystem project, not a Kubeflow subproject, although the distribution still ships it. Serving has its own guide in this catalogue, KServe, and in the next distribution its control plane moves from the kubeflow namespace to kserve.

A distribution from a vendor (Charmed Kubeflow, OpenShift AI, a cloud provider's build) is a different thing again. The community does not certify those, and their component versions, namespaces and auth stack may differ from everything in this guide. Always check which one you are on before copying a command.

What is actually running in the cluster

Before you can debug anything you need a map of the namespaces. The control planes live in two places, and your workloads live in a third.

The kubeflow namespace holds the Pipelines API server (ml-pipeline), its UI (ml-pipeline-ui), the persistence agent and scheduled-workflow controller, the cache server, MySQL, the object store, the Argo Workflows controller, Katib's controller and database manager, the notebook controller and the Central Dashboard. The kubeflow-system namespace holds Kubeflow Trainer's controller and the JobSet controller, because the newer projects have already moved there. Identity components sit in auth (Dex) and oauth2-proxy. Istio is in istio-system, Knative in knative-serving, cert-manager in cert-manager. Your own pods, including every pipeline step, every notebook and every training node, run in profile namespaces such as kubeflow-user-example-com.

A few details of the Pipelines stack matter repeatedly. The default artifact store is SeaweedFS. Older clusters used MinIO, and the swap happened with KFP 2.15. The Service is still named minio-service and pipeline roots still use the minio:// scheme, which is why the change is invisible to pipeline code but can confuse you when reading manifests. The metadata store (MLMD) is being phased out upstream, and MySQL is the supported production database. PostgreSQL is for development and evaluation only. In multi-user mode a pipelines-profile-controller syncs per-namespace resources, including the kfp-launcher ConfigMap, into every profile.

The full distribution has a steady-state footprint of about 4.4 CPU cores, 12.3 GiB of memory and 65 GB of persistent volumes before any user workload. Knative alone takes about 1.45 cores, and Pipelines about 35 GB of storage. If your team uses only Pipelines and notebooks, trimming example/kustomization.yaml to drop Knative, KServe, Spark and Katib is the cheapest cost saving available, and the distribution supports that explicitly.

Identity: the path a request takes and the header everything trusts

The most common mystery at this level is an authentication or authorization error that appears only from CI, only from a script, or only in one namespace. Understanding the identity path turns those from mysteries into a short checklist.

A browser request goes to istio-ingressgateway. Kubeflow's auth stack since 1.9.1 is oauth2-proxy with Dex (Dex is optional and can be replaced by pointing oauth2-proxy at your identity provider). Once a user is authenticated, Istio injects a request header, kubeflow-userid, containing the user's email. The web apps and the Pipelines API server trust that header. They do not re-verify the login themselves. That is an elegant design and also the source of a sharp edge: anything that reaches the API without going through the gateway has no identity at all.

That is exactly the error you see when a script talks straight to the service:

Unauthenticated: Request header error: there is no user identity header

The fix is never to add the header by hand. Go through the gateway with a cookie or token, or call from inside the cluster with a service-account token. Authorization is a second, separate step. After identity is established, the API checks that the user has a RoleBinding in the target namespace, which produces the other error you will meet:

PermissionDenied: User '<user>' is not authorized with reason: ...

That message means authentication worked. Either the user is not a contributor of the profile they are addressing, or the client passed the wrong namespace. Reading these two messages correctly saves an hour of chasing certificates.

For access from outside the cluster, such as a CI job, there are two supported routes. The first uses a Kubernetes service-account token:

TOKEN=$(kubectl -n kubeflow-user-example-com create token default-editor)

import kfp
client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=TOKEN)

This works only when the cluster was installed with a machine-to-machine oauth2-proxy overlay (m2m-dex-only for most clusters, m2m-dex-and-kind or m2m-dex-and-eks where tokens from outside the cluster must be accepted). If you see Jwks doesn't have key to match kid or alg from Jwt in a 401, the RequestAuthentication issuer and jwksUri do not match your cluster, and the documented fix is to use the matching overlay and patch those two values. The second route logs in to Dex and passes the resulting session cookie to the client, which is what the documented KFPClientManager helper does, and it suits a person more than a pipeline.

From a notebook the pattern is different and simpler. A PodDefault named access-ml-pipeline (selector label access-ml-pipeline: "true") projects a service-account token at /var/run/secrets/kubeflow/pipelines/token with audience pipelines.kubeflow.org and a two-hour expiry, and sets KF_PIPELINES_SA_TOKEN_PATH. You tick that configuration when creating the notebook, and kfp.Client() then needs no arguments. Explicitly, it is kfp.Client(host="http://ml-pipeline-ui.kubeflow", credentials=kfp.auth.ServiceAccountTokenVolumeCredentials(path=None)).

One operational rule belongs here. Outside localhost, you must serve Kubeflow over HTTPS, because the web apps set Secure cookies. On plain HTTP from another host, the Jupyter, Katib and Tensorboard apps loop or fail with CSRF and cookie errors. Keep the istio-ingressgateway Service as ClusterIP behind an ingress that terminates TLS. The documented escape hatch, APP_SECURE_COOKIES=false, is not recommended.

Profiles, contributors and the limits of multi-tenancy

A Profile is a cluster-scoped custom resource that wraps a Namespace of the same name and has exactly one owner, set in spec.owner. You can also attach a resourceQuotaSpec. When you apply a Profile, the Profile Controller creates the namespace, a RoleBinding named namespaceAdmin that gives the owner the kubeflow-admin ClusterRole, two service accounts (default-editor, bound to kubeflow-edit, and default-viewer, bound to kubeflow-view), and Istio AuthorizationPolicies that let the owner's traffic through. Because that is all ordinary Kubernetes objects, you can manage profiles as code:

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: team-forecasting
spec:
  owner:
    kind: User
    name: lead@example.com
  resourceQuotaSpec:
    hard:
      cpu: "32"
      memory: 128Gi
      requests.nvidia.com/gpu: "4"

Additional users are contributors, with edit or view access. The Dashboard's "manage contributors" page can add only edit. A view contributor has to be created manually with a RoleBinding named like user-{safe-email}-clusterrole-view and a matching AuthorizationPolicy, which is easy to forget and produces a user who can log in but sees an empty namespace. The API behind the page is KFAM, the Kubeflow Access Management service.

Now the limits, which the official documentation states bluntly: Kubeflow makes no hard security guarantees about profile isolation. A profile is a Kubernetes namespace with RBAC and Istio policies, nothing more. Three consequences matter at work. Notebook pods run as default-editor, which is bound to kubeflow-edit, so anyone with a shell in a notebook can use kubectl against everything in that namespace. Pipeline definitions are shared across all namespaces (runs and experiments are namespaced, definitions are not), so do not put secrets in a pipeline's parameters or defaults. And the trust in the kubeflow-userid header means the boundary is only as strong as your gateway configuration.

The one profile rule you must never forget: never delete the Profiles CRD. Deleting a CRD deletes every object of that kind, and deleting a Profile deletes its namespace and everything inside it, including notebooks, volumes and runs. kubectl delete profile NAME is the intended way to remove one team, and it is destructive on purpose.

How a pipeline really executes

At Beginner you wrote a function, decorated it, and got a run. The pipeline body looks like ordinary Python, which misleads people, so here is what happens in practice.

The @dsl.pipeline function is not executed at run time. When you call the compiler, KFP runs the function once, and each call to a component returns a PipelineTask object instead of running anything. The function body exists to record a directed acyclic graph: which task consumes which task's output. The result is serialized as the intermediate representation (IR), a PipelineSpec protobuf written as YAML. That is why you cannot put a print of a real value, a file read or an if on an output in the pipeline body and expect it to work. The values do not exist yet. Conditions on outputs need dsl.If, which the compiler turns into a branch in the graph.

The IR is hermetic and portable. The same YAML that runs on your cluster also runs on Vertex AI Pipelines, which is why teams on Google Cloud can share pipelines with teams on Kubeflow (see Vertex AI). It also means the compiled file is a build artifact you can store, diff and review.

On the cluster, the KFP v2 backend converts the IR into an Argo Workflow and Argo runs it. For every task, two extra pieces of KFP code surround your container:

1. The driver runs first. It resolves the task's inputs from upstream outputs, computes the cache key, and decides whether to execute at all or to reuse a previous result.
2. The launcher wraps your container. Before your code starts, it downloads input artifacts from the object store. After your code finishes, it uploads output artifacts and records metadata.

This explains several behaviours that otherwise look strange. A step that finishes instantly and shows a "cached" marker never started your container; the driver matched a key and skipped it. A step that dies with an object-store error died in the launcher, before or after your code, and your own logs will look clean. The KFP SDK and backend must agree on the IR schema, which is the reason the same-minor-version rule exists. Pipelines compiled with SDK 2.15.0 or 2.15.1 failed on older backends with this message:

"output_metadata.json": proto: (line 1:237): unknown field "custom_path"

The cure is SDK 2.15.2 or later, or a newer backend. The general lesson is to read that message as a version skew, not a bug in your code.

Since KFP 2.14 there is a second storage mode. Compiling with kubernetes_manifest_format=True produces Pipeline and PipelineVersion custom resources in the group pipelines.kubeflow.org/v2beta1, which you apply with kubectl instead of uploading through the API. That is the natural fit for GitOps, and we use it below. The v1 SDK surface (ContainerOp, kfp.v2, "v2-compatible mode") is gone. The v2 backend can still execute old v1 Argo-format pipelines, and KFP 2.17 adds admin switches to block v1 runs per namespace, which is worth enabling once you have migrated.

Components, artifacts and the exact rules of the cache

There are three ways to define a component, and choosing well is most of component design. Lightweight Python components use @dsl.component: the function's source is serialized into the container command, and packages_to_install runs pip install at start-up. They are fast to write and slow to start, because every run installs its dependencies from the network. Containerized Python components use @dsl.component(target_image=...): you build an image once with kfp component build src/ --component-filepattern my_component.py --push-image, and runs start immediately with the dependencies baked in. Container components use @dsl.container_component and return a dsl.ContainerSpec, for when the step is an existing image or a non-Python tool. The rule of thumb is to start lightweight, and move to containerized as soon as a component has more than a couple of dependencies or runs more than a few times a day. Pinning versions in packages_to_install matters too; an unpinned install is a silent change in a step's behaviour with no change in your code.

Data moves between components in two forms, and the difference is not cosmetic. Parameters are small values passed by value: str, int, float, bool, list and dict. Artifacts are files, typed with Input[T] and Output[T] where T is Dataset, Model, Metrics, ClassificationMetrics, HTML, Markdown or the generic Artifact. An artifact is written to the object store by the launcher, and your function sees a local path through .path. Large data must be an artifact. Putting a big list in a parameter pushes it into the workflow object and hits Kubernetes object-size limits, which is a real failure mode even though KFP 2.16 doubled the input size limit.

from kfp import dsl
from kfp.dsl import Dataset, Input, Metrics, Model, Output

@dsl.component(base_image="python:3.11", packages_to_install=["scikit-learn==1.5.2", "pandas==2.2.3"])
def train(data: Input[Dataset], model: Output[Model], metrics: Output[Metrics], C: float = 1.0):
    import pandas as pd, pickle
    from sklearn.linear_model import LogisticRegression
    df = pd.read_csv(data.path)
    clf = LogisticRegression(C=C).fit(df.drop(columns="y"), df["y"])
    metrics.log_metric("train_accuracy", float(clf.score(df.drop(columns="y"), df["y"])))
    with open(model.path, "wb") as f:
        pickle.dump(clf, f)
    model.metadata["C"] = C

Notice two things. Imports live inside the function, because only the function body is serialized into the container. And the model's metadata dictionary travels with the artifact, which the UI and lineage views read.

Caching is on by default, and it is the feature that most often surprises teams. The cache key is computed from the component's specification and its input values, and since KFP 2.15 it also covers mounted PVCs. If the key matches a previous successful execution, the driver reuses the old outputs and your code does not run. That is wonderful for an expensive feature-engineering step and dangerous for a step whose result depends on the outside world. A component that reads "today's rows" from a database with no input that changes will return yesterday's answer forever. You control this at three levels:

• Per task: task.set_caching_options(False).
• Per run: enable_caching=False when creating the run.
• Per compile: --disable-execution-caching-by-default, or the environment variable KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT=true, which must be set before you import your components, because the default is read at import time.

The design answer is better than the switch. Give a time-dependent step an explicit input, such as the date or a data snapshot identifier, so the cache key changes exactly when the answer should. Then caching stays on and is correct.

Control flow: conditions, fan-out, fan-in and cleanup

Pipelines become useful when they branch and repeat. All of these constructs record graph structure at compile time, so the conditions are expressions over task outputs, not live Python.

A condition uses dsl.If, with dsl.Elif and dsl.Else for further branches. The older dsl.Condition still works but emits a deprecation warning. A loop uses dsl.ParallelFor, which takes a list (or a task output that is a list), an optional parallelism bound (where 0 means unlimited), and yields an item you pass into the tasks inside. Gathering the results of a loop is the job of dsl.Collected:

from kfp import dsl

@dsl.component
def train_one(lr: float) -> float:
    return 1.0 / (1.0 + lr)

@dsl.component
def pick_best(scores: list) -> float:
    return max(scores)

@dsl.component
def notify(status: dsl.PipelineTaskFinalStatus):
    print(f"pipeline finished with state {status.state}")

@dsl.pipeline(name="sweep")
def sweep():
    exit_task = notify()
    with dsl.ExitHandler(exit_task=exit_task):
        with dsl.ParallelFor(items=[0.001, 0.01, 0.1], parallelism=2) as lr:
            t = train_one(lr=lr)
        pick_best(scores=dsl.Collected(t.output))

A few behaviours deserve attention. dsl.Collected is how you fan in: used as an input to a task outside the loop, it gathers every iteration's output into a list. dsl.OneOf does the opposite kind of merge, taking whichever branch of an If/Else actually ran, and it requires an Else branch so there is always a value. dsl.ExitHandler guarantees that its exit task runs however the pipeline ends, and an exit task can accept a PipelineTaskFinalStatus argument to learn whether the run succeeded. That is the right place for cleanup (deleting a temporary PVC) and for alerting. Two fields of that status object, pipelineJobResourceName and pipelineTaskName, are deprecated since 2.14, so avoid building on them.

Ordering without data dependencies uses task.after(other). By default a failure upstream stops downstream tasks; task.ignore_upstream_failure() lets a task run anyway, which suits a summary or notification step. Retries are configured per task with set_retry(num_retries, backoff_duration, backoff_factor, backoff_max_duration). Use retries for transient infrastructure failures such as a flaky network or a preempted node, and never as a substitute for fixing a deterministic error, because each retry is a full pod start and can re-run the expensive part.

A pattern worth adopting is a validation gate. A validate_data component outputs a boolean or a status string, and a dsl.If around the training branch proceeds only when validation passes. It costs one small task and turns "we trained on a broken dataset for three days" into a failed run that someone reads. Data validation tools in this catalogue, such as Great Expectations, slot into that task.

Configuring tasks: resources, GPUs, secrets and volumes

Pipelines on a shared cluster need to ask for what they use and only that. Resource requests are set on the task after you call the component:

t = train(data=prep.outputs["dataset"])
t.set_cpu_request("2").set_cpu_limit("4")
t.set_memory_request("8G").set_memory_limit("16G")
t.set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1)
t.set_retry(num_retries=2, backoff_duration="60s", backoff_factor=2.0)
t.set_env_variable(name="LOG_LEVEL", value="INFO")
t.set_display_name("train logistic regression")

If you are following older tutorials, note the renames. set_gpu_limit is deprecated in favour of set_accelerator_limit, and add_node_selector_constraint in favour of set_accelerator_type. They still work and emit a DeprecationWarning, but a cluster upgrade is a bad moment to discover a removal, so rename them now. KFP 2.17 adds set_debug_pause(), which holds a task so you can attach to its pod, a handy tool for a step that fails in ways you cannot reproduce.

Anything that touches Kubernetes specifics comes from the separate kfp-kubernetes package, imported as from kfp import kubernetes. Install it at the same version as kfp. The functions you will use most:

• Secrets: kubernetes.use_secret_as_env(task, secret_name=..., secret_key_to_env={...}) and use_secret_as_volume. The secret must exist in the profile namespace where the run executes. Never pass a credential as a pipeline parameter; parameters are visible in the UI and stored with the run.
• ConfigMaps: use_config_map_as_env and use_config_map_as_volume.
• Volumes: CreatePVC, mount_pvc(task, pvc_name=..., mount_path=..., sub_path=...) (the sub_path argument arrived in 2.16) and DeletePVC, plus add_ephemeral_volume and empty_dir_mount.
• Scheduling: add_node_selector, add_node_affinity and add_toleration, each with a _json variant, plus pod labels and annotations with add_pod_label and add_pod_annotation.
• Pod settings: set_image_pull_secrets, set_image_pull_policy, set_security_context and set_timeout.

A common production shape is a shared scratch volume for a multi-step job:

from kfp import dsl, kubernetes

@dsl.pipeline(name="shared-scratch")
def shared_scratch():
    pvc = kubernetes.CreatePVC(
        pvc_name_suffix="-scratch",
        access_modes=["ReadWriteOnce"],
        size="20Gi",
        storage_class_name="standard",
    )
    a = prepare()
    kubernetes.mount_pvc(a, pvc_name=pvc.outputs["name"], mount_path="/scratch")
    b = train_from_scratch().after(a)
    kubernetes.mount_pvc(b, pvc_name=pvc.outputs["name"], mount_path="/scratch")
    cleanup = kubernetes.DeletePVC(pvc_name=pvc.outputs["name"]).after(b)

Watch the access mode. A ReadWriteOnce volume mounts on one node at a time, so two parallel tasks that land on different nodes will stall in ContainerCreating. Either keep the tasks sequential, use a ReadWriteMany storage class, or use the newer pipeline workspace (PipelineConfig(workspace=WorkspaceConfig(size="10Gi")), available since 2.15), which gives every task in a run a shared path through dsl.WORKSPACE_PATH_PLACEHOLDER. Pairing a CreatePVC with a DeletePVC inside an exit handler is what stops abandoned scratch disks from accumulating in the quota.

Two recent security changes affect pod specs. Since 2.16 the platform enforces pod and container security contexts, and you can set runAsUser, runAsGroup and runAsNonRoot from the SDK or as admin-wide defaults. Since 2.17 user workloads default to hostUsers: false. An image that insists on root, or that writes into a root-owned directory, may fail on upgrade with permission errors. Fix the image rather than loosening the policy.

The artifact store and the kfp-launcher ConfigMap

Where do artifacts go, and whose credentials are used? The answer is configuration you can read, and it is worth reading once so failures stop being opaque.

The pipeline root is the object-store URI prefix under which a run writes its artifacts. The cluster default is minio://mlpipeline/v2/artifacts. You can override it per run with the pipeline_root argument, or per namespace through a ConfigMap named kfp-launcher in each run namespace, whose keys are defaultPipelineRoot and providers. In multi-user mode the profile controller creates this ConfigMap for you, and it is the first place to look when an artifact upload fails in one team's namespace but not another's.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kfp-launcher
  namespace: team-forecasting
data:
  defaultPipelineRoot: s3://acme-ml-artifacts/pipelines
  providers: |-
    s3:
      default:
        endpoint: s3.me-central-1.amazonaws.com
        region: me-central-1
        credentials:
          fromEnv: true

The providers key holds per-scheme settings for s3, gs and minio, each with a default and optional per-bucket overrides, and credentials from a secret reference or from the environment (for instance IAM Roles for Service Accounts). Prefer workload identity over long-lived keys wherever your cloud supports it. The region shown is a real consideration for Gulf and Egyptian employers: keep the artifact bucket in the same region as the cluster, both for data-residency rules and because cross-region egress on multi-gigabyte model artifacts is slow and billable.

On the API-server side, the object-store environment variables (OBJECTSTORECONFIG_HOST, _PORT, _REGION, _SECURE, _ACCESSKEY, _SECRETACCESSKEY) configure the platform's own access, and omitting the key variables tells it to use IRSA on the ml-pipeline service account. If your cluster sits behind a corporate proxy or a TLS-inspecting gateway, there are HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables and a custom CA bundle mechanism (CABUNDLE_SECRET_NAME, CABUNDLE_CONFIGMAP_NAME, CABUNDLE_KEY_NAME, mounted at /kfp/certs/ca.crt). A certificate signed by unknown authority from the launcher is almost always this.

Running pipelines from CI and treating them as code

A pipeline that only runs when someone clicks in a notebook is a prototype. The production shape has four parts: the pipeline source in git, a compile step in CI, a place where compiled pipelines are registered, and a scheduled or event-driven trigger.

Compile in CI with the CLI, so the build fails on a bad graph before anyone deploys it:

kfp dsl compile --py pipeline.py --output pipeline.yaml --function my_pipeline

Then register it as a new version of a named pipeline. Versions are how you roll back: each upload is an immutable revision, and a run records which version it came from.

kfp --endpoint http://localhost:8080/pipeline --namespace team-forecasting \
    pipeline create-version -p PIPELINE_ID -v "2026-10-01-abc123" pipeline.yaml

Runs belong to an experiment, a namespaced grouping that costs nothing and pays off when you search. A scheduled run is a recurring run, which older material calls a "job". The SDK renames reflect that: delete_job, disable_job and enable_job are deprecated in favour of delete_recurring_run, disable_recurring_run and enable_recurring_run, and the parameter job_id= is now recurring_run_id=. A nightly retraining run looks like this from the client:

client.create_recurring_run(
    experiment_id=exp.experiment_id,
    job_name="nightly-retrain",
    cron_expression="0 0 2 * * *",
    pipeline_package_path="pipeline.yaml",
)

Check the cron format in the Pipelines UI before relying on the expression, because a wrong field order schedules a job at an unexpected time. Pair scheduled runs with the TTL settings that arrived in 2.17 (ttl_seconds_after_finished, with success and failure variants, and active_deadline_seconds in PipelineConfig), or old workflows accumulate and slow the Argo controller and the database.

For a GitOps workflow, use the Kubernetes-native mode. Compile with kubernetes_manifest_format=True and commit the resulting Pipeline and PipelineVersion manifests, then let your continuous-delivery tool apply them. Argo CD and Kustomize fit this naturally, and the platform itself can be managed the same way: never edit upstream KCD manifests, and instead layer overlays on top of example/kustomization.yaml. That single discipline is what makes the next upgrade a diff review instead of an archaeology project.

Keep the SDK, the compiled YAML and the backend aligned. The documented compatibility position is that a 2.x backend runs both v2 IR and old v1 pipelines, and that you should keep the SDK and backend on the same minor version. In CI, pin kfp and kfp-kubernetes to the same exact version as each other and as the cluster's minor release, and change them in one commit.

Testing pipelines before they cost money

A cluster run is slow and expensive feedback. Kubeflow gives you three cheaper layers, and a mature team uses all of them.

Unit-test the function. A lightweight component is still a Python function. Call its python_func directly in a normal test with temporary files standing in for artifacts. This catches logic errors in milliseconds and needs no cluster.

Run locally. The SDK can execute a whole pipeline on your machine:

from kfp import local
local.init(runner=local.SubprocessRunner())   # or local.DockerRunner()

SubprocessRunner runs each component in a subprocess, by default in a virtual environment; DockerRunner runs each in a container and is closer to the real thing. Local execution verifies wiring, types and the data contract between steps, and it does not reproduce the cluster's resource limits, secrets, volumes, GPU scheduling or caching. It is for correctness, not for capacity. The newer Kubeflow SDK (pip install kubeflow) also has local backends for training jobs, through LocalProcessBackendConfig() and ContainerBackendConfig().

Smoke-test on the cluster. Keep a tiny dataset and run the full pipeline on it in CI against a shared test namespace, with short TTLs and a quota so a bad commit cannot starve the cluster. A smoke run catches what local execution cannot: missing secrets, wrong namespaces, image pull failures and RBAC gaps.

Distributed training with Kubeflow Trainer v2

Training on one node is a notebook problem. Training on several nodes with GPUs is a scheduling and configuration problem, and Kubeflow's answer changed substantially. The legacy Training Operator v1 defined one custom resource per framework (PyTorchJob, TFJob, MPIJob, XGBoostJob, PaddleJob, JAXJob) in the kubeflow.org/v1 group. Kubeflow Trainer v2 replaces it with one API group, trainer.kubeflow.org/v1alpha1, built on the Kubernetes JobSet API. Training Operator v1 still ships in 26.03.1 (version v1.9.2), but it has been removed from the distribution's master branch, so new work should target Trainer.

The mental model has two roles. An administrator writes a runtime: a ClusterTrainingRuntime (cluster-wide) or a TrainingRuntime (one namespace) that describes how a kind of job is launched, including the ML policy (torch, mpi, jax and others), an optional gang-scheduling policy (coscheduling or volcano), and a JobSet template. A data scientist writes a TrainJob that references a runtime and supplies only what varies: image, command, number of nodes and resources per node.

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: pytorch-simple
  namespace: kubeflow-user-example-com
spec:
  runtimeRef:
    name: torch-distributed
  trainer:
    numNodes: 2
    image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727
    command: ["python3", "/opt/pytorch-mnist/mnist.py", "--epochs=1"]
    resourcesPerNode:
      limits:
        nvidia.com/gpu: 1

The same job from Python uses the Kubeflow SDK, which is often what a notebook user wants:

from kubeflow.trainer import TrainerClient, CustomTrainer

def train_fn():
    import torch.distributed as dist
    dist.init_process_group("nccl")
    # training loop here

client = TrainerClient()
job = client.train(
    trainer=CustomTrainer(func=train_fn, num_nodes=2,
                          resources_per_node={"cpu": 3, "memory": "16Gi", "gpu": 1}),
    runtime=client.get_runtime("torch-distributed"),
)
client.wait_for_job_status(job)
for line in client.get_job_logs(job, follow=True):
    print(line)

The runtimes that ship with the install are torch-distributed, deepspeed-distributed, jax-distributed, mlx-distributed and xgboost-distributed, plus TorchTune runtimes for fine-tuning small language models. Initializers are a design worth appreciating. A dataset initializer and a model initializer run first, on CPU, to pull data from Hugging Face or S3 (credentials through secretRef), so expensive GPUs are not idle while gigabytes download. For configuration-driven LLM fine-tuning there is a BuiltinTrainer, and for your own code a CustomTrainer as above. The deep-learning side of these jobs is covered by guides such as DeepSpeed and PEFT and TRL.

Inspect a job with the usual tools:

kubectl get trainjobs -n kubeflow-user-example-com
kubectl describe trainjob pytorch-simple -n kubeflow-user-example-com
kubectl get jobsets -n kubeflow-user-example-com

A TrainJob that ends Failed with reason TrainingRuntimeNotSupported references a runtime without a supported ML policy. If the SDK's list_runtimes() does not show a runtime you created, it lacks the label trainer.kubeflow.org/framework. DeadlineExceeded means activeDeadlineSeconds elapsed.

Watch the breaking changes, because much online material predates them. In v2.2.0, PodTemplateOverrides was replaced by the runtimePatches API, numProcPerNode was removed from the Torch ML policy (it is derived from container resources, while trainJob.spec.trainer.numProcPerNode remains), and ElasticPolicy was removed. In v2.3.0, CRDs moved into the Helm chart templates, so you disable them with --set crds.enabled=false rather than --skip-crds. The distribution also pins the JobSet controller, and moving past 26.03 requires deleting the jobset-controller-manager Deployment in kubeflow-system once because a selector field is immutable.

For shared GPU clusters, integrate with a queueing system. A TrainJob has suspend and managedBy fields for that purpose, and Trainer supports Kueue, Volcano and coscheduling. The point of gang scheduling is to avoid a job holding three of four required GPUs forever while it waits for the fourth.

Hyperparameter tuning with Katib

Katib automates the search for good hyperparameters. An Experiment bundles an objective, a search space, an algorithm, a trial template and a budget. The algorithm's service (the Suggestion) proposes hyperparameters, each proposal becomes a Trial that runs a real Kubernetes workload, and a metrics collector reports the objective back. Katib's API group is still kubeflow.org/v1beta1.

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  name: random-search
  namespace: kubeflow-user-example-com
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: accuracy
  algorithm:
    algorithmName: random
  parallelTrialCount: 3
  maxTrialCount: 12
  maxFailedTrialCount: 3
  parameters:
    - name: lr
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.05"
  trialTemplate:
    primaryContainerName: training
    trialParameters:
      - name: learningRate
        reference: lr
    trialSpec:
      apiVersion: batch/v1
      kind: Job
      spec:
        template:
          spec:
            containers:
              - name: training
                image: docker.io/kubeflowkatib/pytorch-mnist:v1beta1-45c5727
                command: ["python3", "/opt/pytorch-mnist/mnist.py", "--lr=${trialParameters.learningRate}"]
            restartPolicy: Never

The choices that matter are these. Algorithm: random and grid are baselines; bayesianoptimization, tpe, multivariate-tpe, cmaes, sobol, hyperband and pbt are available, with enas and darts for architecture search. Start with random, which is hard to beat by much for few parameters, and move to a model-based algorithm when a single trial is expensive. Budget: maxTrialCount caps the total, parallelTrialCount caps concurrency (keep it at or below the number of GPUs you can actually spare), and maxFailedTrialCount stops a broken template from burning the whole budget. Early stopping with medianstop ends trials that fall behind the median of earlier ones. Resume policy (Never, LongRunning, FromVolume) decides what happens to a finished experiment's suggestion service; Never frees its resources.

The most common Katib failure is a missing metric, reported as MetricsUnavailable. By default the collector reads your container's standard output and looks for lines in the form name=value, so your code must print accuracy=0.93 exactly, with the same name as objectiveMetricName. For pull-based collectors, the namespace also needs the label katib.kubeflow.org/metrics-collector-injection=enabled (the kubeflow namespace already has it). Collector kinds include StdOut, File, TensorFlowEvent, PrometheusMetric, Custom, None and Push.

Admin-level configuration lives in the katib-config ConfigMap in the kubeflow namespace, edited with kubectl edit configMap katib-config -n kubeflow; it defines the suggestion and collector images. From Python, KatibClient(namespace=...).tune(...) builds an experiment from a function, and get_optimal_hyperparameters reads the result. The Kubeflow SDK also exposes OptimizerClient, and because a trial can run a TrainJob, Katib composes with Trainer for distributed trials. Katib v0.19.0 was validated up to Kubernetes 1.34, so check compatibility before pairing it with a newer cluster.

Notebooks and the cost of idle resources

A Notebook is a custom resource (kubeflow.org/v1) reconciled into a StatefulSet with one pod named NAME-0 and a Service. The pod runs as default-editor. Its storage is a PVC, and that matters for cost: a stopped notebook frees its CPU, memory and GPU, but the volume remains. The controller that manages the lifecycle supports culling, which stops idle notebooks. It is configured with environment variables on the notebook-controller Deployment, and the default is off:

A culled notebook gets the annotation kubeflow-resource-stopped, and the last-activity time is held in notebooks.kubeflow.org/last-activity. On a GPU cluster, enabling culling with a shorter idle time (a few hours) is usually the highest-value cost setting in the whole platform, together with a per-profile resourceQuotaSpec.

When a notebook will not start, use the same three commands every time: kubectl describe notebooks NAME -n NS (conditions and events), kubectl describe pod NAME-0 -n NS (scheduling and image pull), and kubectl logs NAME-0 -n NS. The usual causes are an image pull problem, a PVC that cannot bind, a quota breach or a conflicting PodDefault. PodDefaults (kubeflow.org/v1alpha1) are admission-webhook objects that inject environment variables, volumes and tolerations into any pod whose labels match their selector, and they are how you standardize things like credentials, proxies and shared data mounts for a whole team. Because the webhook mutates pods, a broken PodDefault breaks every notebook that selects it.

A debugging order that works

Kubeflow failures come from many layers, so the skill is isolating the layer quickly. Work from the outside in, and use the same order every time.

1. Is the platform healthy? kubectl get pods -A for anything not Running or Completed. Controllers that restart, or webhooks whose endpoints have no pods, cause symptoms far from their source.
2. Identity or authorization? Distinguish the two messages from the identity section. "No user identity header" means you bypassed the gateway. "PermissionDenied" means the user lacks a binding in that namespace. RBAC: access denied on the Dashboard points to an Istio AuthorizationPolicy mismatch, commonly when a network plugin such as Cilium is not configured for Istio.
3. Compile time or run time? A failure while compiling is your Python; nothing reached the cluster. A failure in the UI is a run problem.
4. Driver, launcher or your code? In the run's pod, look at which container failed. A driver failure is about inputs or cache; a launcher failure is about object storage and credentials; your container's log is your code. A run that succeeds instantly is probably cached.
5. Resources? OOMKilled, Pending pods and ContainerCreating stalls are scheduling, quota or volume issues. kubectl describe pod shows the event that explains it.
6. Version skew? Errors that mention an unknown field, or a deprecated call that suddenly stopped working, usually mean the SDK and the backend disagree.

Some install-time errors are expected and harmless. On first apply, no matches for kind "X" in version "Y" or ensure CRDs are installed first appears because custom resources were applied before their CRDs were established, which is why the official install is a retry loop around kustomize build example | kubectl apply --server-side --force-conflicts -f -. On local Kind clusters, "too many open files" points to low inotify limits (fs.inotify.max_user_instances=2280 and fs.inotify.max_user_watches=1255360), toomanyrequests to Docker Hub rate limits (create a pull secret or mirror images), and no matching manifest for linux/arm64 to images that are not built for Apple Silicon. An upgrade failing with spec.selector: Invalid value: field is immutable means a controller's selector changed between versions; delete that one Deployment and re-apply, accepting brief downtime.

Upgrades, versions and keeping the platform alive

With a six-month support window you will upgrade twice a year, and the work is easier if you have practised. The principles are these. Never edit upstream manifests; reference them from your own overlay and change only the overlay. Apply with --server-side. Label your resources so kubectl apply --prune can remove what a new release dropped, and always run a dry run first. Read the release notes for the specific hop, because upgrades have hop-specific steps:

• 1.10.2 to 1.11 or 26.03: the artifact store changes from MinIO to SeaweedFS. Delete MinIO and migrate data with S3 tools if you need to keep artifacts.
• 26.03 to 26.03.1: delete the jobset-controller-manager Deployment in kubeflow-system; delete the llmisvc-manager-rolebinding ClusterRoleBinding (KServe 0.16 to 0.17 and later); and apply the Dashboard v2 procedure.
• Dashboard v2.0.0 is a breaking, non-in-place change. The Central Dashboard, Profile Controller with KFAM and the PodDefaults webhook moved to a new repository and new image registry paths. You must delete the old Deployments, Services, ServiceAccounts and RBAC in kubeflow, and keep the CRDs, Namespaces and Profiles. Deleting the Profiles CRD would delete every team's namespace.
• Next release on master: KServe moves from kubeflow to kserve, and scale the old controllers to zero first because their leader-election leases differ. Monitoring that selects the pod label app: kserve must switch to control-plane.

Rehearse on a staging cluster built from the same overlay, and keep a restore plan for the database and the object store. The database (MySQL) holds run history and pipeline versions; the object store holds artifacts. Neither is recreated by reapplying manifests. Back both up before an upgrade, and test a restore at least once, because an untested backup is a hope.

Finally, think about what you monitor, since bundled dashboards moved. The KFP Prometheus and Grafana manifests were removed in 2.17, so bring your own stack (Prometheus and Grafana are the usual choice). The API server exposes gRPC request and latency metrics since 2.16, the notebook controller serves /metrics on port 8080, and Trainer reports progressPercentage, estimatedRemainingSeconds and metrics in TrainJob.status.trainerStatus. Alert on failed-run rate, run duration, pending-pod age and the health of the API server and the controllers.

Putting it all together

Here is the shape of a mid-level production setup, stitched from the sections above. It is deliberately small, and every piece is something you can verify.

A team gets a Profile per project, created from a manifest in git, with a resourceQuotaSpec so a runaway job cannot starve the cluster. Notebook culling is on, with an idle time of a few hours. A PodDefault gives notebooks access to Pipelines, and a second one injects the team's proxy settings.

The team's pipeline lives in a repository next to its tests. A pull request runs unit tests on component functions, compiles the pipeline with warnings treated as errors, and runs a local smoke test with SubprocessRunner. On merge, CI compiles the pipeline with kfp dsl compile, registers it as a new version named after the commit, and starts a run from that version in a shared test namespace on a small dataset. After a green run, a recurring run in the production profile is updated to the new version.

Inside the pipeline, a validation task gates training with dsl.If. Dates and data snapshot identifiers are explicit inputs, so caching stays on and stays correct. Credentials arrive through use_secret_as_env and workload identity, never as parameters. Scratch space is a pipeline workspace or a CreatePVC paired with a DeletePVC in an exit handler. Training for large models is a TrainJob against the admin-maintained torch-distributed runtime, queued through Kueue, and hyperparameter search is a Katib experiment whose trial runs that job. The model artifact is stored under a region-local pipeline root, and the model registry and serving are the next stop: see KServe and MLflow.

Where you are now

You can now read a Kubeflow cluster the way its operators do. You know that the version you run is a dated bundle of independent projects, that identity flows through one header, that a profile is a namespace and not a security boundary, and that a pipeline is a compiled graph that Argo executes through a driver and a launcher. You can predict the cache, configure secrets and volumes, test before you spend cluster time, run distributed training with Trainer v2, and search hyperparameters with Katib. You also have a debugging order and an upgrade routine.

Senior takes every one of these further, with a security, scale, or ownership dimension. Running the platform for other teams, sizing the control plane and finding which component breaks first, hardening the trust model, securing secrets and the supply chain, designing multi-tenancy and cost controls, and deciding when Kubeflow is the wrong tool altogether.

Sources

• Kubeflow documentation
• Kubeflow architecture
• Installing Kubeflow
• Kubeflow releases
• Kubeflow 26.03 release notes
• Kubeflow Community Distribution repository
• Kubeflow Community Distribution release 26.03.1
• Profiles and contributors
• Pipelines multi-user mode
• Pipelines server configuration
• Pipelines object store configuration
• Connect to the Pipelines API
• Compile a pipeline
• Pipelines CLI
• Caching in Pipelines
• Control flow in Pipelines
• Execute pipelines locally
• Pipelines version compatibility
• kfp-kubernetes reference
• KFP SDK reference
• Kubeflow Pipelines releases
• Kubeflow Trainer documentation
• Trainer runtime guide
• Trainer migration guide
• Trainer releases
• Kubeflow SDK documentation
• Katib configure an experiment
• Katib metrics collector
• Katib configuration
• Kubeflow Notebooks overview
• Notebook controller README
• Kubeflow Dashboard v2.0.0 release
• Kubeflow support policy

Part two of three. A cumulative review of Beginner and Mid-level material, organised by scenario and trade-off rather than by level. At this level an interviewer is not checking that you know what a pipeline is. They are checking whether you can predict what the platform will do and reason your way to a fix. Senior reviews all three.

Where this picks up

Foundations, in one screen

• Distribution versus project. The Kubeflow Community Distribution (kubeflow/community-distribution) pins compatible versions of Pipelines, Trainer, Katib, Notebooks, the Dashboard and friends, plus Istio, cert-manager, Dex and oauth2-proxy. Its current release as of this writing is 26.03.1, and versions are calendar-based (YY.MM[.PATCH]), not 1.x.
• Request path. Client, Istio ingress gateway, oauth2-proxy (with Dex), then Istio injects the kubeflow-userid header that the web apps and the Pipelines API trust.
• Pipeline life. @dsl.pipeline is compiled once into IR YAML. The backend turns that into an Argo Workflow. Each task is wrapped by a driver (inputs and cache) and a launcher (artifact upload and download).
• Artifacts and cache. Parameters are small values, artifacts are files in the object store. The cache key comes from the component spec and inputs, plus mounted PVCs since 2.15.
• Training. TrainJob references a ClusterTrainingRuntime or TrainingRuntime, all in trainer.kubeflow.org/v1alpha1, running on JobSet.
• Tuning. A Katib Experiment spawns Trials; the objective metric is read from name=value lines on standard output by default.

The commands to have ready:

kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get profiles
kubectl describe notebooks NAME -n NS
kubectl get trainjobs,jobsets -n NS
kubectl get trials -n NS
kfp dsl compile --py pipeline.py --output pipeline.yaml
kfp run list -e EXPERIMENT_ID

Scenario questions

"A pipeline step finished in one second and the output is wrong. What do you check first?"

A strong answer starts with the cache. A step that completes instantly with empty logs almost certainly never started its container: the driver matched a previous execution's key and reused its outputs. Open the run and look for the cached marker on that task. Then ask why the key matched. The key is built from the component's specification and its input values, so if the step reads something from outside, such as "today's rows" from a database, and no input changed, the cached answer is returned forever.

The fix is to make time or data identity an explicit input, for example a run_date parameter or a snapshot identifier, so the key changes when the answer should. Disabling the cache (set_caching_options(False) on the task, enable_caching=False for the run, or the compile flag --disable-execution-caching-by-default) is the blunt version and costs you the real benefit on expensive steps. Note the detail that the environment variable KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT=true must be set before your components are imported.

What a weak answer does: re-runs the pipeline hoping for a different result, or starts editing the component code without confirming it ever ran.

"Your CI job gets Unauthenticated: Request header error: there is no user identity header. Walk me through it."

This message means the call reached the Pipelines API without passing through the gateway, so Istio never injected kubeflow-userid. The API does not authenticate users itself; it trusts that header. So the answer is about the route, not the credentials. Point the client at the gateway path (/pipeline) and supply either a service-account token (created with kubectl -n NS create token default-editor, which needs the machine-to-machine oauth2-proxy overlay) or a Dex session cookie. From inside the cluster, use the projected service-account token provided by the access-ml-pipeline PodDefault.

A good candidate then separates the next error. If the call passes identity and gets PermissionDenied: User '...' is not authorized, authentication worked and the user or service account has no RoleBinding in that namespace, or the wrong namespace was passed. And if a token from outside the cluster fails with Jwks doesn't have key to match kid or alg from Jwt, the RequestAuthentication issuer and jwksUri do not match the cluster.

What separates strong from weak: naming the two distinct failures (identity versus authorization) and never suggesting that you add the header manually.

"A notebook user says they can see the namespace but it is empty, and they cannot create anything. Why?"

Likely they were added as a view contributor, or a contributor binding is incomplete. The Dashboard can only add edit contributors. A view contributor has to be created manually, with a RoleBinding named like user-{safe-email}-clusterrole-view and a matching Istio AuthorizationPolicy, and it is easy to create one without the other. The check is kubectl get rolebindings,authorizationpolicies -n NS. A user who is authenticated but has no binding sees nothing and gets PermissionDenied.

"Training jobs sit pending and GPUs are half-allocated across jobs. What is happening and how do you stop it?"

Two jobs each hold some of the GPUs they need and neither can start, the classic partial-allocation deadlock of distributed training. The answer is gang scheduling: the runtime's podGroupPolicy (coscheduling or volcano) or a queueing layer such as Kueue, through the TrainJob's suspend and managedBy fields, so a job starts only when all its nodes can. Add per-profile resourceQuotaSpec so one team cannot take every GPU, and use dataset and model initializers so the expensive GPUs are not idle while data downloads.

"A cluster upgrade from 26.03 to 26.03.1 fails with field is immutable. What do you do?"

The error, spec.selector: Invalid value: field is immutable, means a controller's selector changed between versions. For the JobSet controller the documented remedy is to delete its Deployment once (kubectl delete deployment jobset-controller-manager -n kubeflow-system --ignore-not-found) and apply again, accepting brief downtime for that controller. A strong answer also mentions the other hop-specific steps: deleting the llmisvc-manager-rolebinding ClusterRoleBinding for KServe, and the Dashboard v2 non-in-place upgrade, where you delete the old Dashboard, Profile Controller, KFAM and PodDefaults resources in kubeflow but never the Profiles CRD, because deleting it deletes every profile namespace.

"A Katib experiment runs but every trial shows MetricsUnavailable."

The collector could not find the objective. By default it reads standard output for lines like accuracy=0.93, so the training code must print the metric in exactly that form, and the name must match objectiveMetricName. For pull-based collectors the namespace also needs the label katib.kubeflow.org/metrics-collector-injection=enabled. A candidate who says "increase the timeout" has guessed.

"A pipeline that worked last month fails with unknown field "custom_path"."

That is version skew. Pipelines compiled with SDK 2.15.0 or 2.15.1 include a field that older backends do not know. Upgrade the SDK to 2.15.2 or later, or upgrade the backend. The general principle to state is that the SDK and backend should share a minor version, and CI should pin kfp and kfp-kubernetes together.

"Pods crash on a new Kind cluster with 'too many open files'. First thought?"

Low inotify limits on the host. The documented values are fs.inotify.max_user_instances=2280 and fs.inotify.max_user_watches=1255360. On macOS or Windows, Kind runs in a Linux VM, so the setting applies inside the VM, not on the desktop OS. Mention also that the full distribution recommends about 16 GB of memory and 8 cores, and that trimming example/kustomization.yaml lets it run in less.

Debugging questions

"Give me your general order for debugging a failed run."

A good candidate has a fixed order and says why it is outside-in:

1. Platform health: any pod not Running or Completed, any crash-looping controller, any webhook with no endpoints.
2. Identity versus authorization, using the two messages above, and RBAC: access denied for an Istio policy mismatch.
3. Compile time versus run time: a compile failure is Python, nothing reached the cluster.
4. Which container failed in the task pod: driver (inputs and cache), launcher (object store and credentials) or the user container.
5. Resources: OOMKilled, Pending, ContainerCreating and quota events via kubectl describe pod.
6. Version skew between SDK and backend.

Follow up with what they do not do: edit the generated Argo Workflow by hand. It is regenerated on the next run and the fix is lost.

"A run fails with a certificate error when the launcher uploads an artifact."

The launcher talks to object storage, so a certificate signed by unknown authority usually means a corporate proxy or TLS-inspecting gateway in the path. The documented mechanism is a custom CA bundle on the API server (CABUNDLE_SECRET_NAME, CABUNDLE_CONFIGMAP_NAME, CABUNDLE_KEY_NAME, mounted at /kfp/certs/ca.crt) and the proxy variables HTTP_PROXY, HTTPS_PROXY and NO_PROXY. Disabling verification is the wrong answer.

"Artifacts upload fine for one team and fail for another."

Compare the kfp-launcher ConfigMap in both run namespaces. It holds defaultPipelineRoot and the providers settings with credentials. In multi-user mode the profile controller creates it, so a namespace created outside that flow, or one with a stale secret reference, is the likely culprit.

Trade-off questions

"Lightweight component, containerized component, or container component?"

Lightweight components (@dsl.component) are fastest to write and serialize the function into the container command, but install packages_to_install from the network on every run, so they start slowly and drift if versions are unpinned. Containerized Python components (target_image, built with kfp component build) bake in dependencies, start fast and are reproducible, at the price of a build and a registry. Container components (@dsl.container_component) wrap an existing image or tool with no Python. A good answer gives a rule: start lightweight, move to containerized once dependencies grow or the step runs often, use container components for non-Python tools.

"Parameters or artifacts for passing data?"

Parameters are small values passed by value and stored with the run; artifacts are files in the object store, typed as Dataset, Model, Metrics and so on. Anything large must be an artifact, because large parameters land in the workflow object and hit Kubernetes object-size limits, even though KFP 2.16 doubled the input limit. Also, secrets belong in Kubernetes Secrets via use_secret_as_env, never in either.

"Kubeflow Pipelines, or another orchestrator?"

An honest answer compares on the axes that matter. Kubeflow Pipelines is Kubernetes-native, gives every task its own container and resources, has a typed artifact model and a UI, and its IR is portable to Vertex AI Pipelines. The cost is operating the platform: Istio, auth, a database and an object store. If a team has no Kubernetes expertise or needs general data orchestration with many non-ML jobs, a tool like Airflow, Prefect or Dagster may fit better. The mature answer is that they are not exclusive, and that the decision follows who will operate it.

"Full distribution or standalone Pipelines?"

The full distribution gives multi-user access, profiles, notebooks, Katib and Trainer, at about 4.4 cores, 12 GiB and 65 GB of volumes before workloads. Standalone Pipelines is lighter and suits a single team, but has no profile isolation and uses a simple install. Standalone also avoids the identity machinery, which can be a feature or a gap depending on your security requirements. Mention that you can trim the full distribution by removing Knative, KServe, Spark or Katib.

"Trainer v2 or the old Training Operator?"

Trainer v2 uses one TrainJob API with admin-owned runtimes, built on JobSet, with initializers, gang scheduling integrations and a Python SDK. The old PyTorchJob, TFJob and similar resources in kubeflow.org/v1 are legacy, and the operator is removed from the distribution's master. A strong answer names the v2.2.0 breaking changes (runtimePatches replacing PodTemplateOverrides, removal of numProcPerNode from the Torch policy, removal of ElasticPolicy) and says a migration is planned, not urgent on a stable cluster but required before the next bundle.

"Random search, Bayesian, or something else in Katib?"

Start with random, which is a strong baseline for a few parameters. Use bayesianoptimization, tpe or cmaes when trials are expensive and the space is smooth, hyperband when you can cut poor trials early, and medianstop early stopping to save compute. Always set maxFailedTrialCount so a broken template does not burn the budget, and keep parallelTrialCount within the GPUs you can actually spare.

"Where should model serving live?"

Kubeflow ships KServe, but KServe is an ecosystem project with its own release cadence. The mature answer is to treat serving as a separate concern with its own upgrade path; see KServe. Mention the next-release change, KServe moving to the kserve namespace.

What a strong answer covers that a weak one misses

Common interview questions, rapid fire

What is the default artifact store? SeaweedFS since KFP 2.15, reached under the old minio-service name; MinIO and any S3-compatible store still work.

Which database is supported in production? MySQL. PostgreSQL is development and evaluation only.

How do you make a pipeline run on a schedule? A recurring run (formerly a "job"), with the client methods create_recurring_run, disable_recurring_run and delete_recurring_run.

Why must imports be inside a lightweight component function? Only the function body is serialized into the container, so module-level imports do not exist there.

How do you give a step a credential? kubernetes.use_secret_as_env or use_secret_as_volume from kfp-kubernetes, with the Secret in the profile namespace.

What replaced dsl.Condition? dsl.If, dsl.Elif and dsl.Else.

What replaced set_gpu_limit? set_accelerator_limit, with set_accelerator_type replacing add_node_selector_constraint.

What do you run to test a pipeline without a cluster? local.init(runner=local.SubprocessRunner()) or local.DockerRunner(). It checks wiring, not capacity.

How do you stop idle GPU notebooks? Set ENABLE_CULLING=true and CULL_IDLE_TIME (minutes) on the notebook controller, and add a profile quota.

What happens if you delete the Profiles CRD? Every profile is deleted, and with it every namespace and its contents.

What is a PodDefault? A kubeflow.org/v1alpha1 object whose admission webhook injects environment, volumes and tolerations into pods matching its selector.

How do you run distributed training from Python? TrainerClient().train(trainer=CustomTrainer(...), runtime=client.get_runtime("torch-distributed")) from the kubeflow package.

Sixty-second self-test

Answer aloud, then check the section above.

1. Why is KFP caching surprising for a step that reads live data, and what is the design fix?
2. Which two messages separate an identity failure from an authorization failure?
3. What do the driver and launcher each do?
4. Why is a profile not a security boundary, and what must you never delete?
5. Name three breaking changes between Training Operator v1 and Trainer v2.
6. What two conditions make a Katib trial report MetricsUnavailable?
7. How do you stop two GPU jobs from deadlocking each other?
8. What is the documented procedure when an upgrade reports an immutable field?

Senior takes every one of these further, with a security, scale, or ownership dimension. System-design prompts such as "design a multi-tenant ML platform for forty teams", incident questions such as "the Pipelines API is down and runs are piling up", and how to argue a build-versus-buy or migration decision to leadership.

Part two of three. At this level your pipelines run and your notebooks start, so the problems change character. They are no longer "how do I submit a run" but "why did this step return yesterday's answer", "why does it work from my notebook and fail from CI", and "why did the upgrade break the Dashboard". Start with the error table, then read the practices in order of leverage, the practice cards and the recipes.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most, in order of leverage

1. Pin the SDK and backend to the same minor version

Most "it worked yesterday" stories at this level are version skew. The platform is a bundle of independent projects, and the compiled IR is a contract between the SDK and the backend. Pin kfp and kfp-kubernetes together, at the minor version your cluster runs, in one requirements file that CI and notebooks both use. Change them in one commit, together with the backend upgrade. A single pinned file saves more debugging time than any other habit here.

2. Make every time-dependent step take an explicit input

Caching is the feature that makes pipelines cheap, and the thing that makes them quietly wrong. Give steps that read live data a date or a snapshot identifier as an input, so the cache key changes exactly when the answer should. Then you never need to disable the cache globally. Reserve set_caching_options(False) for steps with real side effects, such as sending a notification.

3. Put credentials in Secrets, and Secrets in the right namespace

Use kubernetes.use_secret_as_env or use_secret_as_volume. Never pass a credential as a pipeline parameter, because parameters are visible in the UI and stored with the run, and pipeline definitions are shared across namespaces. Prefer workload identity (for example IAM Roles for Service Accounts) over long-lived object-store keys, and keep the artifact bucket in the same region as the cluster, both for data residency and for transfer cost.

4. Move to containerized components when a step matters

A lightweight component installs its packages from the network on every run. That is fine for a prototype and a liability for a nightly job: it is slow, it fails when a package index is down, and unpinned versions change behaviour with no change in your code. Build the image once with kfp component build and pin the versions you keep.

5. Test in three layers

Unit-test the component functions directly. Run the pipeline locally with local.init(runner=local.SubprocessRunner()) or local.DockerRunner() to check wiring and types. Smoke-test on a small dataset in a test namespace with a quota. Add a compile test that treats DeprecationWarning as an error, so a removed API fails a pull request instead of an upgrade.

6. Keep the platform manifests in git, as overlays

Never edit upstream manifests. Reference them from your own Kustomize overlay on example/kustomization.yaml, apply with --server-side, and label resources so kubectl apply --prune --dry-run can show you what a release dropped. Pair this with Kustomize and a GitOps tool such as Argo CD. The upgrade becomes a reviewed diff.

7. Set quotas and culling before the first GPU complaint

Give each Profile a resourceQuotaSpec, and turn on notebook culling (ENABLE_CULLING=true, CULL_IDLE_TIME in minutes). Idle notebooks holding GPUs are the largest avoidable cost on most clusters. A stopped notebook frees compute but keeps its volume, so review old volumes too.

8. Clean up what your pipelines create

Pair CreatePVC with DeletePVC in an exit handler. Use the TTL settings in KFP 2.17 (ttl_seconds_after_finished and its success and failure variants, plus active_deadline_seconds) so finished workflows do not pile up and slow the controller and the database. An exit handler also gives you a place to send an alert with the PipelineTaskFinalStatus.

9. Practise the upgrade, twice a year

With about six months of community support per release, upgrades are a routine. Rehearse on a staging cluster built from the same overlay, read the hop-specific notes, and back up MySQL and the object store first. Test one restore. You learn the real duration and the real surprises when nothing depends on it.

10. Bring your own monitoring

The bundled Pipelines Prometheus and Grafana manifests were removed in 2.17. Collect the API server's gRPC metrics (added in 2.16), the notebook controller's /metrics, and Kubernetes signals such as pending-pod age. Alert on failed-run rate and run duration. See Prometheus for the collection side.

Practice cards

Each card takes ten to twenty minutes on a test cluster. Do them in order.

Operational recipes

Find out what is actually installed

kubectl get deploy -n kubeflow -o custom-columns=NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image
kubectl get deploy -n kubeflow-system
python -c "import kfp; print(kfp.__version__)"

Compare against the release table in the distribution README for your tag.

Health sweep

kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -n kubeflow
kubectl get pods -n kubeflow-system
kubectl get pods -n auth
kubectl get pods -n istio-system

A crash-looping controller or a webhook with no ready pods explains a surprising number of symptoms in other components.

Reach the UI locally

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

Then browse to http://localhost:8080. Localhost is the one case where plain HTTP works. Anything else needs TLS.

Connect a client from outside the cluster

TOKEN=$(kubectl -n kubeflow-user-example-com create token default-editor)

import kfp
client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=TOKEN)
client.set_user_namespace("kubeflow-user-example-com")

This requires the machine-to-machine oauth2-proxy overlay. set_user_namespace persists to $HOME/.config/kfp/context.json.

Compile and register a pipeline in CI

kfp dsl compile --py pipeline.py --output pipeline.yaml --function my_pipeline
kfp pipeline create-version -p PIPELINE_ID -v "$(git rev-parse --short HEAD)" pipeline.yaml
kfp run create -e EXPERIMENT_NAME -r "ci-$(git rev-parse --short HEAD)" -f pipeline.yaml -w

The -w flag waits for the run. Check the flags with kfp run create --help on your installed version.

Inspect a run that failed

kfp run get RUN_ID -d
kubectl get pods -n NS --sort-by=.metadata.creationTimestamp | tail
kubectl describe pod POD -n NS
kubectl logs POD -n NS -c main

Decide from the failing container whether it was the driver, the launcher or your own code. A container named main is your code; if the failure is earlier, the problem is inputs, cache or object storage.

Debug a notebook that never becomes Ready

kubectl describe notebooks NAME -n NS
kubectl describe pod NAME-0 -n NS
kubectl logs NAME-0 -n NS

Look for image pulls, unbound PVCs, quota errors and PodDefault conflicts.

Debug a TrainJob

kubectl get trainjobs -n NS
kubectl describe trainjob NAME -n NS
kubectl get jobsets -n NS
kubectl get pods -n NS -l jobset.sigs.k8s.io/jobset-name=NAME

Read the conditions first. TrainingRuntimeNotSupported and DeadlineExceeded are the two documented failure reasons. The label selector on the last line is the standard JobSet label; confirm it with kubectl get pods --show-labels.

Turn on notebook culling

kubectl set env deployment/notebook-controller-deployment -n kubeflow \
  ENABLE_CULLING=true CULL_IDLE_TIME=240 IDLENESS_CHECK_PERIOD=1

Times are in minutes. Confirm the Deployment name on your cluster first with kubectl get deploy -n kubeflow | grep notebook, and set it in your overlay afterwards so a reapply does not remove it.

Change the default Dex password on a running cluster

kubectl delete secret dex-passwords -n auth
kubectl create secret generic dex-passwords --from-literal=DEX_USER_PASSWORD='BCRYPT_HASH' -n auth
kubectl delete pods --all -n auth

Generate the bcrypt hash first with the documented passlib one-liner, then connect a real identity provider as soon as you can.

Before any upgrade

1. Read the release notes for your exact hop and list its manual steps.
2. Back up MySQL and the object store, and record the current versions.
3. Rehearse on staging built from the same overlay.
4. Apply with --server-side, using a dry run and labels for pruning.
5. Delete immutable-field Deployments only where the notes say so.
6. Never delete the Profiles CRD.
7. Run a smoke pipeline, a notebook start and a small TrainJob afterwards.

Habits that separate calm teams from busy ones

Practices are one-off changes. Habits are what you do on an ordinary Tuesday, and on a platform with this many moving parts they decide whether incidents are short.

Read the error message to the layer it came from. The Kubeflow stack reports failures from Istio, oauth2-proxy, the Pipelines API, Argo, the driver, the launcher, Kubernetes and your own code, and each has its own voice. "No user identity header" is the gateway path. "PermissionDenied" is a RoleBinding. OOMKilled is a limit. unknown field is a version skew. Teams that can name the layer from the first line of the message spend minutes where others spend hours. When you resolve an incident, add its message and layer to the team's runbook in one line.

Change one thing at a time, and say which. When a run fails after a release, resist the urge to bump the SDK, rebuild the image and edit the pipeline in one commit. The platform has caching, retries and multiple version boundaries, so a bundled change leaves you unable to tell which part helped. Keep the pipeline version name tied to a commit so you can always answer "what was running".

Treat deprecation warnings as a to-do list with a date. The SDK renames you saw in this guide (dsl.Condition to dsl.If, set_gpu_limit to set_accelerator_limit, delete_job to delete_recurring_run) all still work today. They are free to fix now and expensive to discover during an upgrade. A compile test that fails on warnings makes this automatic.

Keep the smallest useful dataset next to the pipeline. A fifty-row sample that flows through every step lets a pull request prove the wiring in minutes, locally and on a test namespace. Teams without one test on full data, which is slow and expensive, and so test rarely.

Write down who owns what. Platform operators own the manifests, the overlay, the database and the object store. Data scientists own pipelines, components and images. The gray zones are runtimes (owned by the platform, used by everyone), the kfp-launcher ConfigMap per profile, and quotas. A one-page ownership note that names a person for each settles most "whose problem is this" arguments before they start.

Review the bill and the quota monthly. Look at which profiles hold GPUs, which notebooks have been stopped with large volumes left behind, and which namespaces hit their quota. A short monthly review, using the numbers from kubectl describe resourcequota -n NS, finds waste that no alert will.

Keep a region in mind. For employers in the Gulf and Egypt, where the cloud region holds the data matters to compliance and to latency. Put the cluster, the object-store bucket and the model registry in the same region, record that decision, and check it again when someone proposes a new bucket.

A short checklist

• SDK and backend share a minor version, pinned in one file
• Time-dependent steps take explicit inputs
• Secrets are Kubernetes Secrets in the profile namespace
• Hot components are containerized with pinned dependencies
• Unit, local and cluster smoke tests exist, plus a compile test
• Platform manifests are overlays in git
• Every profile has a quota, and notebook culling is on
• Pipelines clean up their volumes and set TTLs
• Monitoring and alerts do not depend on the removed bundled dashboards
• The default Dex password is gone and the gateway serves HTTPS
• An upgrade has been rehearsed on staging, with a tested restore

Senior takes every one of these further, with a security, scale, or ownership dimension. Production traps that only appear at scale, such as a control plane and database that degrade under thousands of runs, platform-ownership habits for supporting other teams, and review checklists for pull requests and design documents.

This is part three of three. The earlier levels taught you to build pipelines, train models and read the Central Dashboard. This level is about the other side of the glass: you are the person who runs Kubeflow so that thirty data scientists can use it without knowing how much machinery sits underneath, and who gets the page when it breaks.

The version under discussion is the Kubeflow Community Distribution 26.03.1 (released 2026-06-15), which ships Kubeflow Pipelines 2.16.1, Trainer v2.2.0, Katib v0.19.0, Notebooks v1.11.0, Dashboard v2.0.0, KServe v0.18.0 and Hub v0.3.9. Where a newer standalone release matters, the text says so.

Where this picks up

The first thing to settle is what you are actually operating, because "Kubeflow" is a word that hides a dozen deployments.

What you actually operate

Kubeflow is not one binary. It is a set of subprojects that release on their own schedules, plus a curated bundle of Kustomize manifests called the Kubeflow Community Distribution (KCD), which lives in the repository kubeflow/community-distribution. That repository used to be kubeflow/manifests, and older blog posts call the bundle "the manifests" or the "AI Reference Platform". Since the 26.03 release the bundle uses calendar versioning in the form YY.MM[.PATCH]; the last semantic version was 1.11.0 in December 2025. If a colleague says "we are on Kubeflow 1.12", they are wrong, and the conversation should start there, because nobody can tell you what they run.

Two base releases a year are planned, roughly one before each KubeCon, and community support is best-effort for about six months. That single fact shapes the whole platform job: you will perform two upgrades a year, or you will run an unsupported platform. Vendors sell longer support. Canonical Charmed Kubeflow, Red Hat OpenShift AI and others are packaged distributions; the project states that it does not endorse or certify any of them, so a vendor's claim of compatibility is a commercial statement, not a community one.

The subprojects have different maturity. Graduated: Hub, Katib, Notebooks, Pipelines, Spark Operator and Trainer. Incubating: Kale and the Kubeflow SDK. Experimental: the MLflow integration, the MCP server and the MCP Spark History server. KServe, Feast, Elyra and OGX are listed as ecosystem projects, meaning they are not Kubeflow subprojects even though KCD still ships KServe. You decide which of these your platform offers, and you should treat "ships in the bundle" and "we will support it" as different statements.

Read the diagram as two separate questions. Every human request enters through the top lane, so identity failures there look like "Kubeflow is down" to everyone at once. The bottom lane is where tenants live, and a tenant problem should never need a platform change. Most of the design work in this guide is about keeping those two lanes apart.

The trust model: who is this request?

Everything in multi-user Kubeflow hangs on one header. A client reaches the ingress gateway, oauth2-proxy authenticates the user (optionally through Dex, which itself can federate to OIDC, LDAP, SAML or GitHub), and Istio then injects the header kubeflow-userid containing the user's email. The Pipelines API server, the web apps and the Istio AuthorizationPolicies all trust that header. There is no second check.

That design is reasonable because the header can only be set after authentication, inside the mesh. It is also fragile in a specific way you must internalise: anything that can reach a backend Service without passing through the gateway can claim to be anyone, unless a policy stops it. This is why the Pipelines API error Unauthenticated: Request header error: there is no user identity header is a good sign when you see it from a stray client: the platform refused to guess. The fix for a legitimate caller is to go through the gateway path /pipeline with a cookie or token, or, from inside the cluster, to use a projected service-account token.

For machine clients, the oauth2-proxy overlays matter. The overlay m2m-dex-only suits most clusters, m2m-dex-and-kind permits in-cluster service-account tokens presented from outside the cluster on Kind, and m2m-dex-and-eks does the same for EKS. When those tokens are validated by Istio you can meet Jwks doesn't have key to match kid or alg from Jwt, which means the RequestAuthentication issuer or jwksUri does not match your cluster. You can read the issuer from the cluster's .well-known/openid-configuration endpoint using a service-account token, then patch the overlay.

Two defaults must never reach production. The first is the built-in user user@example.com with password 12341234. Change the bcrypt hash in dex-passwords, or better, connect an identity provider so that nobody has a static password at all. The second is plain HTTP: the web apps set Secure cookies, so any host other than localhost needs TLS at the gateway. The documented escape hatch APP_SECURE_COOKIES=false exists, and using it in production is a decision you would need to defend to your security team.

# rotate the default Dex password on a running cluster
kubectl delete secret dex-passwords -n auth
kubectl create secret generic dex-passwords --from-literal=DEX_USER_PASSWORD='<bcrypt-hash>' -n auth
kubectl delete pods --all -n auth

Point oauth2-proxy directly at your IdP if you already have one. Dex is optional since 1.9.1, and every extra hop is another component to patch. In a Gulf or Egyptian enterprise the IdP is usually Entra ID or a national SSO, and group claims from it are what you will map to profile ownership in the next section.

Profiles, tenancy and what isolation really is

A Profile is a cluster-scoped custom resource that wraps a Namespace of the same name. It has exactly one owner in spec.owner, an optional resourceQuotaSpec and optional plugins. When you create one, the Profile Controller builds the namespace, a namespaceAdmin RoleBinding to the kubeflow-admin ClusterRole for the owner, the service accounts default-editor and default-viewer, and the Istio AuthorizationPolicies that let the owner through. Contributors get RoleBindings named user-<email>-clusterrole-edit or -view, managed through KFAM, the access-management API.

Two details bite platform owners. First, the dashboard can only add edit contributors; view access has to be granted manually with a RoleBinding and a matching AuthorizationPolicy. Second, the AuthorizationPolicy conditions key on the kubeflow-userid header and allow the ingress gateway's and the pipelines UI's service accounts as principals, so a hand-written policy that forgets them locks people out while looking correct.

Now the sentence you should repeat to every stakeholder. The official docs state that Kubeflow makes no hard security guarantees about Profile isolation: a profile has no more isolation than a Kubernetes namespace provides. The notebook pod runs as default-editor, bound to kubeflow-edit, so anyone with a shell in a notebook can use kubectl against their own namespace. Pipeline definitions are shared across all namespaces as a current limitation, while experiments, runs and recurring runs are namespaced. For two teams inside one company who trust each other, that is fine. For two customers, two regulated data domains, or a team that holds regulated personal data, it is not, and the right answer is separate clusters, not cleverer policy.

Profiles as code is the practice that keeps this manageable. Keep one Profile manifest per team in Git, reconcile it with Argo CD or Flux (see Argo CD and Kustomize), and set the owner to a platform or team group identity rather than an individual who will leave.

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: team-ranking
spec:
  owner:
    kind: User
    name: ranking-leads@example.com
  resourceQuotaSpec:
    hard:
      cpu: "64"
      memory: 256Gi
      requests.nvidia.com/gpu: "8"
      persistentvolumeclaims: "20"

Never delete the Profiles CRD, and be careful with kubectl delete profile: deleting the Profile deletes the namespace and everything in it, including notebook volumes. Deleting the CRD does the same for every tenant at once. Put this in your runbook in bold, and consider an admission policy that blocks both.

The dashboard's self-service profile creation is controlled by CD_REGISTRATION_FLOW, disabled by default. Leave it off unless a quota is applied automatically, because a profile created without resourceQuotaSpec is an unlimited tenant.

Running the pipelines backend

For most teams Kubeflow Pipelines (KFP) is the product. Your job is the backend, not the pipelines. The pieces are the API server ml-pipeline, the UI ml-pipeline-ui, a persistence agent, the scheduled-workflow controller, a cache server that works as a webhook, the metadata stack (MLMD over gRPC with an Envoy proxy, which an enhancement proposal is removing), MySQL, an S3-compatible object store, and an Argo Workflows controller. In multi-user mode a pipelines-profile-controller built on metacontroller copies the per-namespace artifact-UI and kfp-launcher resources into every profile.

The compile-time picture explains the failure modes. The SDK compiles a pipeline to IR YAML. The backend turns that into an Argo Workflow. For each task a driver resolves inputs and checks the cache, and a launcher wraps the user container to move artifacts in and out of the object store. So a run depends on the API server, Argo, the database, the object store and the launcher configuration at once.

The database. MySQL is the supported production database. PostgreSQL is documented as development and evaluation only, it still depends on MLMD, and its overlay deliberately leaves sslmode unset, so the API server will not start until you set DBCONFIG_POSTGRESQLCONFIG_EXTRAPARAMS. If an application team asks for Postgres because "we already run it", the answer is that the platform documentation says not for production. Use a managed MySQL service with backups, and test the restore.

The object store. MinIO was replaced by SeaweedFS as the default artifact store from KFP 2.15, and the bundled MinIO manifests were removed in 2.16. The service is still reached through the name minio-service and the minio:// scheme, which surprises people. For production, prefer a real S3-compatible service and cloud identity over static keys. The API server reads OBJECTSTORECONFIG_HOST, _PORT, _REGION, _SECURE, _ACCESSKEY and _SECRETACCESSKEY; omit the keys to use IRSA on the ml-pipeline service account.

The launcher config. Each run namespace carries a ConfigMap kfp-launcher. Two keys matter: defaultPipelineRoot, the URI prefix where artifacts land, and providers, which sets per-scheme credentials for s3, gs and minio with a default and overrides. This is where you express "team A writes to its bucket, team B to its own", so bucket-level IAM becomes your real data boundary rather than the namespace.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kfp-launcher
  namespace: team-ranking
data:
  defaultPipelineRoot: s3://acme-ml-team-ranking/pipelines

Keep the providers block out of this sketch and add it from the object-store guide on kubeflow.org for your provider, since its shape depends on whether you use a secret reference or workload identity.

Garbage collection. Runs leave Argo Workflow objects, pods and database rows. Since KFP 2.17, PipelineConfig accepts ttl_seconds_after_finished, _success and _failure, plus active_deadline_seconds, so you can set a platform-wide default that nobody has to remember. Without it, the cluster's etcd is the thing that fills up; KFP 2.16 doubled the input size limit specifically because large inputs had hit etcd and custom-resource size limits.

Caching. It is on by default and the key covers the component spec and inputs (and mounted PVCs since 2.15). This is usually a saving, and occasionally a correctness bug, when a component reads a moving external source. You can switch the default off at compile time with --disable-execution-caching-by-default, or with the environment variable KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT=true set before the components are imported.

Kubernetes-native mode. Since 2.14, pipelines can be compiled to Pipeline and PipelineVersion custom resources under pipelines.kubeflow.org/v2beta1, and stored in Kubernetes rather than the database. For a platform team this is attractive because pipeline definitions can move through the same Git review and GitOps flow as everything else.

from kfp import compiler

compiler.Compiler().compile(
    my_pipeline,
    package_path="pipeline.yaml",
    kubernetes_manifest_format=True,
)

Then kubectl apply -f pipeline.yaml. Keep the SDK and backend on the same minor version: a pipeline compiled with SDK 2.15.0 or 2.15.1 fails on an older backend with unknown field "custom_path", fixed by moving to 2.15.2 or later or by upgrading the backend.

v1 pipelines. The 2.x backend still executes v1 Argo-YAML pipelines, but the SDK authoring path for v1 (ContainerOp, kfp.v2, v2-compatible mode) is gone. KFP 2.17 adds admin switches that block v1 runs and v1 recurring runs per namespace. If you inherited old pipelines, use the switch after migrating the last one, so nobody re-creates the old habit.

Running training as a service

Kubeflow Trainer v2 replaced the Training Operator v1. The old kinds (PyTorchJob, TFJob, MPIJob, XGBoostJob, PaddleJob, JAXJob in kubeflow.org/v1) are legacy, and the Training Operator has been removed from KCD master. The new API group is trainer.kubeflow.org/v1alpha1 with three kinds built on JobSet: TrainJob, TrainingRuntime and ClusterTrainingRuntime.

The split of ownership is the whole point, and it is a gift to a platform team. A data scientist writes a small TrainJob that names a runtime, a node count and resources. You, the admin, write the runtime: the blueprint that holds the multi-node topology, the framework policy, the pod template, and optionally a gang-scheduling policy. Cluster-wide runtimes are ClusterTrainingRuntime; a namespaced TrainingRuntime lives in the same namespace as its jobs and suits a team with special needs.

The shipped runtimes are torch-distributed, deepspeed-distributed, jax-distributed, mlx-distributed and xgboost-distributed, plus TorchTune-based runtimes for fine-tuning: torchtune-llama3.2-1b, torchtune-llama3.2-3b and torchtune-qwen2.5-1.5b. A runtime must carry the label trainer.kubeflow.org/framework (torch, deepspeed, mlx, jax, xgboost, mpi or torchtune), otherwise the SDK's list_runtimes() will not see it. That is the single most common "my runtime is invisible" cause.

apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: torch-gang
  labels:
    trainer.kubeflow.org/framework: torch
spec:
  mlPolicy:
    numNodes: 1
    torch: {}
  podGroupPolicy:
    coscheduling:
      scheduleTimeoutSeconds: 120
  # template: a JobSet spec, copied from torch-distributed and edited

Copy the shipped torch-distributed runtime and edit it rather than writing one from scratch. Note the stale documentation: the Runtime Guide still shows torch: numProcPerNode: auto, but the shipped runtime uses torch: {}, because v2.2.0 removed numProcPerNode from the Torch MLPolicy and derives it from container resources. ElasticPolicy was removed in the same release, and PodTemplateOverrides gave way to runtimePatches.

Per-job overrides use spec.runtimePatches, which can set nodeSelector, tolerations, affinity, volumes, container env, security contexts, serviceAccountName, imagePullSecrets, scheduling gates and, since v2.3.0, terminationGracePeriodSeconds. As owner you decide how much of this you want users to have. A TrainJob with patches can move itself onto nodes you reserved for another team, so constrain this with admission policy, not with trust.

Gang scheduling and queueing. A distributed job that gets 7 of 8 workers and waits for the eighth holds seven GPUs hostage. Runtimes support podGroupPolicy with coscheduling or volcano, and Trainer integrates with Kueue, Volcano, YuniKorn, KAI and Slurm-bridge. TrainJob exposes suspend and managedBy precisely so that a queue can admit it when quota exists. On a shared GPU pool, a queue is not optional.

Initializers pull datasets and models (Hugging Face, S3) on CPU pods before GPU pods start, so that expensive GPUs do not sit idle during a download. Secrets for them go through secretRef. Trainer also reports progressPercentage, estimatedRemainingSeconds and metrics in status.trainerStatus, which gives you something to alert on other than "it has been running a while".

Two failure reasons to recognise: TrainingRuntimeNotSupported means the referenced runtime has no supported mlPolicy or framework, and DeadlineExceeded means activeDeadlineSeconds fired. The Trainer v2.3.0 release also moved CRDs into the Helm chart's templates/, so --skip-crds becomes --set crds.enabled=false, and removed runtime finalizers.

Scaling: which component breaks first

The official documentation publishes no hard numeric scaling limits for Kubeflow, so any table of "supports N users" is invented. What you can do is know the order in which things tend to fail, and measure your own.

The idle footprint comes first. The full KCD reference install asks for about 4.4 CPU cores, 12.3 GiB of memory and 65 GB of persistent volumes before a single user arrives. Knative (about 1.45 cores), Pipelines (about 0.97 cores, 3.5 GiB and 35 GB), Istio, KServe and Hub are the big items. That matters on a small cluster, where the platform can be a third of the capacity.

After that, in rough order of how teams hit them:

1. The Kubernetes API server and etcd. Every pipeline task is a pod, every run an Argo Workflow object, every notebook a StatefulSet. Thousands of finished workflows hurt list calls and watch caches. The TTLs above are your first defence.
2. The Argo controller and MySQL. Pipelines throughput depends on the Argo controller, the database and the object store, not on the API server alone. Slow runs with idle nodes often mean a saturated controller.
3. The object store. Fan-out pipelines with dsl.ParallelFor create a burst of artifact writes and reads. Use real S3 with request-rate headroom, and set parallelism so a loop does not become a denial of service against your own store.
4. The GPU scheduler. Without a queue, large gangs starve small jobs or deadlock each other.
5. Image pulls. A hundred pods starting at once hit registry rate limits. The toomanyrequests: You have reached your pull rate limit error is a Docker Hub anonymous-limit symptom. Mirror images to your own registry.
6. The ingress path. One Istio gateway, oauth2-proxy and Dex serve every human request. Large uploads and notebook websockets share it.

Measure each of these directly rather than guessing. The Pipelines API server exposes gRPC request and latency metrics since 2.16; controllers expose /metrics; Kubernetes itself tells you about API server latency and etcd size.

Security and the supply chain

Start from what the platform does, not from a checklist. A Kubeflow user can run arbitrary code: a notebook is a shell, a pipeline component is a container of the user's choosing, and a TrainJob runs any image. Your security posture is therefore about containing user code and protecting shared components, not about preventing code execution.

Pod security. KCD applies Pod Security Standards at the restricted level to system namespaces; applying them to user namespaces in CI is planned, not shipped. Istio CNI removes the need for privileged init containers in every pod. From KFP 2.16, pod and container security contexts are enforced, and runAsUser, runAsGroup and runAsNonRoot can be set from the SDK and as admin-wide defaults. KFP 2.17 defaults user workloads to hostUsers: false, meaning Kubernetes user namespaces, with an override guard. Turn these defaults on at the platform level, because a default nobody has to ask for is the only one that reliably applies.

Network policy. Network policies ship for Pipelines, Hub, Knative eventing and others, but the namespace-per-team model leaves pod-to-pod traffic across profiles to your CNI policy. Write a default-deny ingress policy for every profile, created by the same automation that creates the Profile. Cilium users should note that the error RBAC: access denied on the dashboard is commonly a Cilium-and-Istio configuration problem; follow Cilium's Istio guide.

Keep Pipelines patched. KFP 2.17 fixed a server-side request forgery issue and a cross-namespace artifact IDOR (insecure direct object reference: fetching another tenant's artifact by guessing an identifier). That is a direct statement that artifact access across tenants was once possible, and is a reason to treat the six-month support window seriously. The UI also restricts which domains it will fetch artifacts from through ALLOWED_ARTIFACT_DOMAIN_REGEX, which defaults to ^.*$, allowing everything. Tighten it.

Supply chain. Your platform runs dozens of upstream images: Istio, Knative, cert-manager, Dex, oauth2-proxy, Argo and every subproject. The community's own CI moved from Trivy to OSV-Scanner for vulnerability scanning on master, and the SDK uses uv audit and OSV-Scanner. For your side, mirror every image into a registry you control, pin by digest, and scan on the way in. Air-gapped installs are supported: mirror the images the project's scan script lists, then rewrite image references with Kustomize or a Kyverno mutate policy. Dashboard v2 moved images to ghcr.io/kubeflow/dashboard/... and Notebooks to ghcr.io/kubeflow/notebooks/..., so a mirror list built for last year is silently incomplete.

For user images, decide whether tenants may pull arbitrary images. The safe platform answer is an allowlist enforced by admission policy (see Trivy for scanning and Kubernetes for policy), with a documented exception process. Pipelines make this easy to forget: base_image and packages_to_install in a lightweight component mean the component installs packages from PyPI at run time, on the cluster, every run. That is convenient for learning and a supply-chain hole for production. Steer production pipelines to containerised components built by your CI from pinned dependencies.

Secrets and credentials

Four kinds of secret exist on this platform, and each has a different owner.

Platform secrets: the Dex password secret dex-passwords in the auth namespace, OIDC client secrets (OIDC_CLIENT_ID, OIDC_CLIENT_SECRET), the database password, and the object-store credentials in the API server environment. Store these in your secret manager and inject them with an operator such as External Secrets rather than committing them to the overlay. See Vault if that is your store. Kustomize overlays in Git should reference secrets, never contain them.

Cloud identity for the control plane. Where you can, use workload identity: IRSA on AWS for the ml-pipeline service account, or the equivalent on your cloud, instead of long-lived access keys. The fewer keys exist, the fewer you rotate after an incident.

Tenant secrets. Pipeline tasks consume Kubernetes Secrets in their profile namespace through kubernetes.use_secret_as_env or use_secret_as_volume, from the kfp-kubernetes package. The consequence of namespace-level tenancy follows directly: anyone with edit rights in the profile can read those Secrets through the notebook's default-editor account. Give each team separate profiles for separate sensitivity levels, rather than one profile with everything.

from kfp import dsl, kubernetes

@dsl.pipeline(name="secret-demo")
def p():
    t = train()  # a component defined elsewhere
    kubernetes.use_secret_as_env(
        t,
        secret_name="warehouse-creds",
        secret_key_to_env={"password": "WAREHOUSE_PASSWORD"},
    )

Trainer secrets. Initializers take a secretRef for a Hugging Face token or S3 credentials, and PodDefaults can inject secrets into notebooks. A PodDefault that injects a credential into every notebook with a matching label is a convenient way to hand every matching user the same cloud permission, so review PodDefaults like RBAC.

The one secret pattern to ban outright is credentials in pipeline parameters. Parameters are stored in run metadata and visible in the UI to anyone who can see the run.

Observability you must bring yourself

Since KFP 2.17, the bundled Prometheus and Grafana third-party manifests are deleted, so Kubeflow no longer ships a monitoring stack. You bring one (see Prometheus), and you decide what to watch.

The signals the project gives you: the Pipelines API server's gRPC request and latency metrics from 2.16; controller /metrics endpoints (the notebook controller serves metrics on :8080 and probes on :8081); Kubernetes events; Trainer's status.trainerStatus and the SDK's get_job_events; and Argo Workflow status. The signals you must derive: queue wait time for GPU jobs, pipeline success rate per team, the age of the oldest unfinished run, and the count of Workflow objects.

A workable starting set of alerts follows from the failure modes in this guide:

• API server error rate and latency for ml-pipeline, since a slow API server looks like a flaky UI.
• MySQL connection failures and disk growth, and object-store error rates.
• Pending pods older than a threshold in profile namespaces, split by reason: quota, unschedulable, image pull.
• Certificate expiry for the webhooks, because cert-manager failures break admission for everything.
• Controller restarts in kubeflow and kubeflow-system.
• Idle GPU allocation, meaning GPUs reserved by pods whose utilisation is near zero for hours.

If KServe monitoring selects the pod label app: kserve, change it to control-plane after the namespace move described in the upgrade section, or your dashboards go blank after an upgrade that otherwise succeeded.

Audit matters too. Kubernetes audit logs record who created a Profile or deleted a RoleBinding; keep them. The platform's own "who ran this pipeline" record is in run metadata, which is shared only as far as the namespace.

There is also an experimental MLflow integration for Pipelines (2.17), which logs pipeline runs to MLflow. If your organisation already tracks experiments in MLflow, evaluate it, but treat it as experimental and pin the version.

Upgrades and migrations

Two upgrades a year is the cadence, and the work is in the gaps between releases, not the headline version. You will need the release notes for each step, because Kubeflow's upgrade path is a sequence of small manual actions.

The rule that protects you: never edit upstream manifests. Build your own Kustomize overlay that references a pinned KCD tag, changes only what you must, and keeps your changes in patches. Then an upgrade is a diff of tags. Label resources consistently so that kubectl apply --prune --dry-run can show what would be removed. Apply with --server-side, and expect the retry loop: the official install command loops because CRDs have to be established before the resources that use them, so the first no matches for kind error is expected, not a failure.

The known manual steps, in the order you will meet them:

1. 1.9 to 1.9.1: the move to oauth2-proxy; clean up istio-system once.
2. 1.9.1 to 1.10: delete the cluster role binding meta-controller-cluster-role-binding.
3. 1.10.2 to 1.11 / 26.03: MinIO to SeaweedFS. The release note says to delete MinIO and optionally migrate data to SeaweedFS using S3 tools. If your artifacts are precious, treat this as a data migration with a rollback plan, not a manifest change.
4. 26.03 to 26.03.1 and later:
   • Delete the jobset-controller-manager Deployment in kubeflow-system, because JobSet 0.10.1 to 0.11.0 changed an immutable selector (spec.selector: Invalid value: field is immutable).
   • Delete the cluster role binding llmisvc-manager-rolebinding for KServe 0.16 to 0.17 and later.
   • Dashboard v2.0.0 is a breaking, non-in-place upgrade. The dashboard, the Profile Controller with KFAM, and the PodDefaults webhook moved to the kubeflow/dashboard repository. Delete the old admission-webhook, dashboard and profile-controller Deployments, service accounts and RBAC in kubeflow, and keep the CRDs, the namespaces and the Profiles.
   • On master: delete the legacy KServe web app, then move KServe from kubeflow to kserve. Scale the old controllers to zero first, because their leader-election leases differ, then verify certificates and webhook endpoints in kserve.

# the JobSet selector change (26.03 -> 26.03.1)
kubectl delete deployment jobset-controller-manager -n kubeflow-system --ignore-not-found

The step that can destroy data is the Dashboard one. The Profile Controller owns Profile CRs, and deleting the Profiles CRD cascades to every profile namespace. Script the deletions by explicit resource name, review the script, and run it in a rehearsal cluster first.

Rehearse. Keep a staging cluster built from the same overlay, at least with representative Profiles, a pipeline run, a notebook with a volume and a TrainJob. An upgrade is verified when an old run is still visible, a new run completes, a notebook volume mounts, and a TrainJob schedules.

Not every subproject moves with the bundle. The newest KFP, Trainer and KServe releases run ahead of the distribution (for example Trainer v2.3.0 and KServe v0.21.0 are out while 26.03.1 ships v2.2.0 and v0.18.0). Taking one standalone is possible, but you then own the compatibility question, so do it only for a fix you need, and test it with the rest.

Cost

The bill has three parts: the idle platform, the notebooks people forget, and the GPUs that sit reserved but unused. Only the third is large, but the first two are where you look first, because they are fully under your control.

Idle platform. About 4.4 cores and 12 GiB of memory with 65 GB of volumes, before workloads. Remove what you do not offer. Knative, KServe, Spark and Katib can be dropped from example/kustomization.yaml; if your organisation serves models through another path, there is no reason to run Knative at all. Each removed component is also one less thing to upgrade and patch.

Notebooks. Culling is off by default. Set ENABLE_CULLING=true on the notebook-controller Deployment, choose CULL_IDLE_TIME (minutes; the default is 1440, one day) and IDLENESS_CHECK_PERIOD (default 1 minute). Culled notebooks stop by scaling to zero through the kubeflow-resource-stopped annotation; the volume remains, so storage cost continues. Decide how long stopped notebooks keep their volumes, and tell users before you delete anything.

GPUs. Use three levers together. Quotas per profile through resourceQuotaSpec stop a single team taking the pool. A queue (Kueue, Volcano or another supported integration) plus gang scheduling stops partial allocation. Initializers and the data cache keep GPU pods from waiting on downloads. Then report utilisation per team, so the conversation about cost is based on a graph.

Pipelines. Caching avoids re-running unchanged steps, and TTLs stop old runs accumulating. Also set memory and CPU requests on components: a component with no resource request is scheduled as a best-effort pod and evicted first, and one with a huge request blocks bin-packing. Platform defaults through a LimitRange in every profile are a simple guard.

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
  namespace: team-ranking
spec:
  limits:
    - type: Container
      defaultRequest: {cpu: 250m, memory: 512Mi}
      default: {cpu: "2", memory: 4Gi}

Regional cost matters in the Gulf and Egypt: GPU availability and price differ by cloud region, and data-residency rules may fix the region for you. Check that a region you must use actually has the accelerator you plan for before you build a platform around it, and see Terraform for provisioning node pools as code.

Governance

Governance is the set of decisions you made once so that teams do not make them repeatedly. On Kubeflow it comes down to five questions.

Who may create a profile, and with what quota? A request process, a template and a Git review. Self-service registration is off by default and should stay off until quotas are automatic.

Which images and base images are allowed? Publish a small set of blessed notebook and component base images, with owners and a patch schedule. Notebook images have to be kept alive by someone: example images live under ghcr.io/kubeflow/kubeflow/notebook-servers/..., and they are examples, not a supported fleet.

Where do artifacts and models live, and who can read them? Decide bucket layout and IAM per team. Hub (formerly Model Registry) registers models, versions and artifacts; in 26.03.1 it is enabled with its UI by default, the registry deploys into the user profile namespace, and the read-only Model Catalog is a cluster-wide singleton in kubeflow. Decide whether teams share a registry or each has its own, and what "promoted" means.

What is the deprecation policy? The SDK deprecates APIs with warnings: dsl.Condition to dsl.If, set_gpu_limit to set_accelerator_limit, add_node_selector_constraint to set_accelerator_type, and the *_job client methods to *_recurring_run. A platform that runs CI on user pipelines with warnings treated as errors finds these before the backend removes them. Tell teams which versions of kfp and kubeflow you support, and keep the SDK and backend on the same minor version.

Who is on call, and for what? Write down the boundary. You own the control planes, the gateway, identity, the database and object store, runtimes, quotas and upgrades. Teams own their pipelines, images, and code. A pipeline that fails because of user code is not an incident; a healthy-looking cluster where no run can start is.

Put these in a short document that each new team reads during onboarding, and link it from the Dashboard menu: the links live in the ConfigMap centraldashboard-config under the key links.

Incident playbooks

Each playbook starts from the symptom users report, not from a component, because that is what you receive.

"Nobody can log in" or "everything returns 403". Work from the edge inwards. Is the gateway reachable and is TLS valid? Are the pods healthy in istio-system, oauth2-proxy and auth? Did a cert-manager certificate expire? RBAC: access denied points at Istio AuthorizationPolicies: check the kubeflow-userid header arrives and that the profile's policies exist. Recent changes to Istio or Cilium are the prime suspect.

"Pipeline runs do not start." Check the API server and its logs first; PermissionDenied: User '<user>' is not authorized means the user lacks a RoleBinding in the namespace passed to the call (or the wrong namespace was sent), while Unauthenticated: Request header error: there is no user identity header means the request bypassed the gateway. If the API is healthy, look at Argo and the database. Then check the kfp-launcher ConfigMap, because a missing or wrong defaultPipelineRoot fails runs at the first task.

"Runs start but hang or fail at artifact steps." Suspect the object store, credentials, or the providers block. Check the object-store endpoint from inside a pod in the profile.

"A notebook never becomes Ready." kubectl describe notebook NAME -n NS, then kubectl describe pod NAME-0 -n NS and kubectl logs NAME-0 -n NS. The usual causes are an image pull problem, a PVC that cannot bind, a quota, or a PodDefault conflict.

"Our TrainJob is stuck." kubectl describe trainjob NAME -n NS and kubectl get jobsets -n NS. Suspended means a queue has not admitted it; TrainingRuntimeNotSupported means a bad runtime reference; DeadlineExceeded means the deadline fired. If pods are Pending with some workers running, you lack gang scheduling.

"A Katib experiment never finds its metric." The trial must print name=value to standard output, and the namespace needs the label katib.kubeflow.org/metrics-collector-injection=enabled for pull-based collectors. The error surfaces as MetricsUnavailable.

"Something deleted a team's namespace." Stop, preserve audit logs, and check whether a Profile or the Profiles CRD was deleted. Restore depends on your volume snapshots, which is why you take them before a deletion, not after.

# first-look triage for a platform incident
kubectl get pods -A | grep -Ev 'Running|Completed'
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get certificates -A

Where Kubeflow stops

Kubeflow is a good answer when you already run Kubernetes, want open-source control, and need several pieces (notebooks, pipelines, training, tuning and a registry) under one identity model. It is a poor answer in several situations, and saying so early saves your organisation a year.

You have no Kubernetes expertise. Kubeflow adds Istio, Knative, cert-manager, Dex, oauth2-proxy, Argo and a dozen controllers on top of Kubernetes. If nobody on the team can debug a webhook, use a managed service (SageMaker, Vertex AI or Azure ML). Notably, Vertex AI Pipelines runs the same KFP IR YAML, so pipeline code carries over.

You need hard tenant isolation. The project itself says namespaces provide no more isolation than Kubernetes does. Separate clusters, or another platform, is the answer.

You only need orchestration. If your need is scheduled data pipelines, a general orchestrator is lighter. Kubeflow Pipelines is container-per-step and tied to Kubernetes.

You only need experiment tracking or a model registry. MLflow is far simpler to run for that, and Hub is one option when you already have Kubeflow.

You only need model serving. KServe can run on its own; note that it is classed as an ecosystem project, not a Kubeflow subproject, and in the next distribution release it moves to its own kserve namespace.

You need Ray or a feature store. Ray sits under experimental/ in KCD, and Feast is an ecosystem project. See Ray and Feast, and treat them as separate platforms you integrate.

Your hardware is ARM. Some images lack arm64 builds, producing no matching manifest for linux/arm64. Check before you commit to Graviton or Apple Silicon nodes.

Honest platform ownership includes the exit plan: keep pipelines in KFP's portable IR and models in an open format, so that a move is a migration and not a rewrite.

Running it as a platform for other teams

A platform is a product with users. Your users are data scientists; their experience is the metric.

Offer a paved road. One documented way to start: a Profile template, a notebook image, a pipeline repository template with CI that compiles and tests the pipeline, and a runtime for training. Each team should reach a first successful run in a day. Pipelines can be tested without a cluster using local.init(runner=local.SubprocessRunner()), and the Kubeflow SDK offers local backends for Trainer that need no cluster, so your template should include a local smoke test.

Version everything you publish. Runtimes, notebook images and the SDK version belong in a changelog. A runtime renamed without notice breaks every job that references it.

Give a small number of supported configurations. Three runtimes you support beat twenty you do not. A team with special needs gets a namespaced TrainingRuntime and owns the consequences.

Provide a support channel and an SLO. "API availability of 99.5 percent, runs start within five minutes when quota exists" is an honest statement if you measure it. Do not promise what your monitoring cannot prove.

Make the cost visible. Show teams their GPU hours and idle notebooks monthly.

Deprecate loudly. Announce a removal two releases ahead, run a report of which teams still use the old API, and go to them.

The Kubeflow SDK (pip install kubeflow) is a good foundation for the paved road, because one package offers TrainerClient, OptimizerClient for Katib, ModelRegistryClient for Hub, SparkClient and, from 0.5.0, PipelinesClient. Pin its version in your template, and note that the older standalone kubeflow-training package serves the legacy Training Operator.

from kubeflow.trainer import TrainerClient, CustomTrainer

def train():
    print("training step")

client = TrainerClient()
job = client.train(
    trainer=CustomTrainer(func=train, num_nodes=2,
                          resources_per_node={"cpu": 2, "memory": "8Gi"}),
    runtime=client.get_runtime("torch-distributed"),
)
client.wait_for_job_status(job)

The review checklist

Use this for design documents and for changes to the platform overlay.

Identity and access

• No default user or password; TLS on the gateway; no path that bypasses the gateway.
• Profiles are in Git, owned by groups, each with a quota and default-deny network policy.
• Deletion of Profiles and the Profiles CRD is blocked by policy.

Pipelines backend

• MySQL, backed up and restore-tested; a real object store with cloud identity.
• kfp-launcher per profile with an explicit defaultPipelineRoot.
• TTLs and active deadlines set; ALLOWED_ARTIFACT_DOMAIN_REGEX tightened.
• SDK and backend on the same minor version.

Training and GPUs

• Runtimes carry the framework label; a queue and gang scheduling in front of GPUs.
• runtimePatches constrained by admission policy.

Supply chain and secrets

• Images mirrored, pinned and scanned; production pipelines use built components.
• No secrets in parameters; PodDefaults reviewed like RBAC.

Operations

• Overlay pins a KCD tag; upgrade runbook and staging rehearsal exist.
• Monitoring brought by you, with the alerts above; on-call boundaries written down.
• Culling on, unused components removed, GPU utilisation reported per team.

The complete picture

Put the pieces together as one sentence. A user authenticates through oauth2-proxy, the mesh stamps the identity, a Profile gives them a namespace with a quota, a notebook or a pipeline runs there under a service account, artifacts land in a bucket chosen by the launcher config, training jobs are admitted by a queue and scheduled as gangs from runtimes you own, and everything is reconciled from Git and upgraded twice a year by a person who read the release notes.

The recurring shape of every decision in this guide is the same. Kubeflow gives you a rich set of defaults designed for a demonstration: a shared password, unlimited profiles, no culling, no TTLs, PostgreSQL-friendly overlays, permissive artifact domains. The platform job is to replace each default with an explicit, reviewed, version-controlled choice. The project is honest about its limits, including the documented statement on isolation, so you can be honest with your users too.

Where the series leaves you

You began by running a pipeline, then learned to compose and operate them, and now you own the platform that lets others do so. The next steps are practical. Build the paved road for one team, and measure how long their first run takes. Put Profiles and the overlay under GitOps with Argo CD. Wire monitoring with Prometheus. If serving matters most to you, continue with KServe, and if experiment tracking does, with MLflow. Read each release's notes the week it appears, rehearse the upgrade, and keep the runbook current. That routine, more than any single feature, is what keeps a Kubeflow platform healthy.

Sources

• Kubeflow documentation
• Kubeflow architecture
• Installing Kubeflow
• Kubeflow releases
• Kubeflow 26.03 release notes
• Community distribution repository
• Community distribution release 26.03.1
• Profiles
• Pipelines server configuration
• Pipelines multi-user mode
• Pipelines object store
• Pipelines database
• Pipelines releases
• Trainer documentation
• Trainer runtime guide
• Trainer releases
• Kubeflow SDK
• Katib installation
• Notebooks overview
• Notebook controller README
• Dashboard v2.0.0 release
• Hub overview
• Support and versioning

Senior interviews about Kubeflow rarely ask you to recite a command. They hand you a platform problem and watch how you reason: what you ask first, which component you suspect, what you refuse to promise, and how you explain the cost of your choices to someone who does not care about Istio. This page gives you prompts in three groups, system design, incident response and arguing a trade-off, with the shape of a strong answer for each.

The facts below are as of the Kubeflow Community Distribution 26.03.1. If an interviewer quotes a different version scheme, politely correct it: the project moved to calendar versions in 2026, and the last semantic version was 1.11.0.

The thirty-second answer

Kubeflow is a set of independently released, Kubernetes-native projects for data and AI work: notebooks, pipelines, distributed training, hyperparameter tuning and a model registry, bundled with an identity and tenancy layer built on Istio, oauth2-proxy and Profiles. As a platform owner I run it as a product for other teams: I own identity, the pipelines backend, training runtimes, quotas, upgrades and cost, and I replace the demonstration defaults with explicit choices. I also know where it stops: namespaces are not hard isolation, and it is a poor fit for a team without Kubernetes skills.

Facts to have ready

• Versioning is CalVer, YY.MM[.PATCH]; 26.03.1 was released 2026-06-15; support is best-effort for about six months, so two upgrades a year.
• Request path: gateway, oauth2-proxy (with optional Dex), Istio injects kubeflow-userid, VirtualService routes. Backends trust that header.
• A Profile is a cluster-scoped CRD that wraps a namespace with one owner, an optional quota and generated RBAC and Istio policy.
• Pipelines: MySQL for production, S3-compatible store (SeaweedFS replaced MinIO as the default), kfp-launcher ConfigMap per namespace, Argo Workflows executing compiled IR.
• Trainer v2: TrainJob plus admin-owned ClusterTrainingRuntime on JobSet, replacing the Training Operator v1 kinds.
• The official position on isolation: no hard guarantees beyond Kubernetes namespaces.

System design: a shared ML platform for thirty teams

The prompt. "A bank in the Gulf wants a shared ML platform for thirty teams, with notebooks, scheduled pipelines and GPU training. Data must stay in-country. Design it around Kubeflow."

A strong answer starts with questions, not boxes. How sensitive is the data, and do any teams handle data that must never share a cluster with others? How many GPUs, of what type, in which region, and is the region's capacity confirmed? Who is the identity provider? Is there an existing Kubernetes platform team, and what is the support expectation? These decide the design, and naming them shows seniority.

Then the architecture. One cluster per environment, with the data-residency constraint fixing the region. Identity through the bank's IdP wired straight into oauth2-proxy, dropping Dex unless a connector is needed, and TLS at the gateway. Teams map to Profiles in Git, owned by groups, each with a resourceQuotaSpec, a LimitRange and a default-deny network policy created by the same automation. Regulated teams get their own profiles at minimum, and if the risk assessment says namespaces are not enough, their own cluster, because the project itself says profiles add no isolation beyond Kubernetes namespaces.

The pipelines backend gets MySQL on a managed service, with restore tests, an in-country S3-compatible store with cloud identity instead of static keys, and a platform-wide TTL and deadline. Each profile's kfp-launcher points at that team's bucket with bucket-level IAM. Training uses a few curated ClusterTrainingRuntime objects, Kueue in front of the GPU pool with gang scheduling, and initializers so GPUs are not waiting on downloads. Notebooks have culling enabled.

Operations: the platform overlay pins a KCD tag in Git, reconciled by GitOps, with a staging cluster that is upgraded first, because upgrades are twice yearly and have manual steps. Monitoring is brought in, since Kubeflow no longer ships it.

What a weak answer does. It draws the install diagram from the docs, names Kubeflow components, and never mentions quota, upgrade cost, isolation limits or who is on call.

Follow-up you should expect. "How many users can it handle?" The honest answer is that the documentation gives no numeric limits; you would load-test with realistic traffic, and you know the order in which things break: etcd and the API server from workflow objects, the Argo controller and database, the object store, the GPU scheduler, image pulls, the ingress path.

System design: pipelines as an internal product

The prompt. "Teams keep breaking each other's pipelines and nobody knows which SDK version runs where. Fix it."

A strong answer separates the contract from the implementation. The contract: a supported SDK version range (the SDK and the backend are kept on the same minor version), a pipeline repository template, and CI that compiles the pipeline to IR YAML, runs a local smoke test with local.init(runner=local.SubprocessRunner()), and treats deprecation warnings as failures so that dsl.Condition, set_gpu_limit and the like are found before removal. The implementation: pipelines compiled to Kubernetes-native Pipeline and PipelineVersion resources and applied through GitOps, so a change is a reviewed pull request and history is in Git. Note the current limitation that pipeline definitions are shared across namespaces, so naming conventions and ownership labels matter.

Then the runtime hygiene: production components are containerised and built by CI with pinned dependencies rather than installing packages at run time; caching policy is explicit; TTLs are a platform default. A known compatibility failure is worth citing: a pipeline compiled with SDK 2.15.0 or 2.15.1 failing on an older backend with unknown field "custom_path", which is exactly what a version-range policy prevents.

System design: a GPU training service

The prompt. "Teams fight over GPUs. Design fair, efficient access for distributed training."

Say first that the problem is scheduling, not training. Partial gang allocation wastes GPUs: seven of eight workers waiting for the eighth hold seven GPUs. The answer combines a queue (Kueue, using suspend and managedBy on TrainJob), a gang policy on the runtime (podGroupPolicy with coscheduling or volcano), quotas per team at queue level and namespace level, and preemption rules that you agree with the teams before an incident. Runtimes are admin-owned, so the platform decides topology, images and scheduling, and a team only supplies a function or image, node count and resources.

Mention runtimePatches: they let a job set node selectors, tolerations and affinity, so constrain them with admission policy or a job can land on another team's reserved nodes. Mention initializers and the data cache, which move I/O off the GPUs, and status.trainerStatus for progress. Finish with the report you would publish monthly: queue wait time, GPU-hours requested versus used, and idle reservations per team.

Common trap. Quoting the Training Operator v1 PyTorchJob for new work. It is legacy and removed from the distribution's master; the answer is TrainJob with a runtime.

System design: multi-tenancy for a regulated team

The prompt. "A team with personal data wants to use the shared platform. Is that acceptable?"

The best answer is conditional and does not flatter the tool. Namespaces give RBAC, quota and network policy, and Istio policy gives request-level access control, but the project states it provides no hard isolation beyond namespaces. Notebook pods run as default-editor bound to kubeflow-edit, so anyone with a notebook can read Secrets in the namespace. Pipeline definitions are visible across namespaces. The 2.17 release fixed a cross-namespace artifact access bug, which shows the boundary has had holes. So: if the regulator or the data classification demands isolation, use a dedicated cluster. If not, use a dedicated profile with its own bucket and IAM, separate credentials, default-deny network policy, audit logging, and a written risk acceptance. Say the decision belongs to security and compliance, and that your job is to lay out the options and costs.

Incident response: "everyone gets 403"

The prompt. "At 09:00 every user reports access denied on the dashboard. What do you do?"

A strong answer triages from the edge inwards and asks what changed. Is the gateway serving valid TLS? Are the pods in istio-system, oauth2-proxy and auth healthy? Did a certificate expire, which cert-manager would show? RBAC: access denied is an Istio authorisation result, so check that the kubeflow-userid header is reaching the service and the profile AuthorizationPolicies exist. The top suspect is whatever changed overnight: an Istio or CNI change, an upgrade, a policy rollout. Roll back the change before understanding it fully if impact is wide, then diagnose. Communicate early with a status message, and later write the blameless note naming the control that would have caught it.

What separates seniors: they mention the trust-model reason this is catastrophic, namely that one header is the whole identity system, and they keep mitigation and diagnosis separate.

Incident response: "runs will not start"

The prompt. "Users submit pipeline runs and they sit in Pending, or fail instantly."

Narrow by the error. Unauthenticated: Request header error: there is no user identity header means a client bypassed the gateway. PermissionDenied: User '<user>' is not authorized means a missing RoleBinding or the wrong namespace. If calls succeed but nothing runs, look at the API server, Argo controller, MySQL, then the namespace's kfp-launcher ConfigMap, because a wrong defaultPipelineRoot fails the first task. If tasks start but hang, suspect the object store and credentials. If pods are Pending, read their events for quota, scheduling and image-pull causes; toomanyrequests means a registry rate limit, fixed by mirroring images.

Also raise the slow-burn cause: thousands of finished Workflow objects and no TTL, which degrade the API server over weeks. The fix is platform-wide TTLs and an active deadline, both available in PipelineConfig since 2.17.

Incident response: a namespace disappeared

The prompt. "A team's profile namespace and notebook volumes are gone."

Stop further changes and preserve audit logs. Deleting a Profile deletes its namespace and everything in it, and deleting the Profiles CRD does this to every tenant. Find out who or what did it: a person, a GitOps prune, or an upgrade script that removed the CRD. Recovery depends on volume snapshots and object-store versioning; if neither exists, say so honestly. The prevention list is the valuable part: admission policy denying deletion of Profiles and the CRD, GitOps prune settings that exempt them, snapshots, and upgrade scripts that delete named resources instead of whole groups. The Dashboard v2 upgrade guidance makes the same point, since it tells you to delete old deployments but keep the CRDs, namespaces and Profiles.

Incident response: a failed upgrade

The prompt. "You upgraded 26.03 to 26.03.1 and the dashboard will not come up."

A senior answer knows this release has a breaking step. Dashboard v2.0.0 moved the dashboard, Profile Controller with KFAM and the PodDefaults webhook to a new repository and images, and it is not an in-place upgrade: the old Deployments, service accounts and RBAC in kubeflow must be deleted while CRDs, namespaces and Profiles are kept. There is also the JobSet selector change, where kubectl delete deployment jobset-controller-manager -n kubeflow-system --ignore-not-found is required, and the KServe role-binding cleanup. Explain that you would check the release notes against what was skipped, that a rehearsal in staging would have caught it, and that your overlay is pinned so rollback is a revert.

Incident response: a hung GPU job

The prompt. "A distributed training job has held eight GPUs for two days and shows no progress."

Check kubectl describe trainjob, events with the SDK's get_job_events, and status.trainerStatus for progress. Distinguish a stuck scheduling problem (some pods Pending, which points to missing gang scheduling) from a hung process (all pods Running and GPU utilisation near zero, which points to a collective-communication deadlock or a data stall). Terminate safely, and set activeDeadlineSeconds so this class of problem ends by itself. Finally, ask whether the job should have been admitted through a queue that enforces time limits.

Arguing a trade-off: Kubeflow or a managed service

The prompt. "The CTO asks why we should not simply use a cloud ML service."

Put the decision in the leader's terms: control, cost, skills and exit. Managed services remove the operations burden, and that is worth real money if you have few platform engineers. Kubeflow gives portability, open interfaces (KFP's IR YAML even runs on Vertex AI Pipelines), no per-service markup, and the ability to keep data and compute where regulation demands. Its costs are people: two upgrades a year, an on-call rotation, and the upkeep of identity, networking and observability. A fair recommendation: choose managed if you have fewer than a small platform team and no portability requirement; choose Kubeflow if you already run Kubernetes well, have sovereignty constraints, or serve many teams enough that the platform cost amortises. Offer to de-risk it: a pilot with two teams, a measured cost, a defined exit.

Arguing a trade-off: one cluster or many

Leadership often asks for "one platform for everyone" because it is cheaper. Explain what the saving hides. A shared cluster means a shared gateway, control plane and upgrade window, so a bad upgrade affects every team, and isolation is namespace-level. More clusters mean more upgrades and more idle overhead (about 4.4 cores and 12 GiB each for the full distribution). The pragmatic recommendation is a shared cluster for trusting teams, separate clusters for distinct trust or compliance domains, all built from the same overlay so the extra cost is automation rather than handwork.

Arguing a trade-off: the cost of staying current

A manager asks why you need engineer-weeks twice a year for upgrades. The honest answer is that support is about six months per release, the bundle has manual steps, and the security fixes matter: KFP 2.17 fixed a server-side request forgery issue and a cross-namespace artifact access bug. The cost of skipping is larger: upgrading two releases at once compounds the manual steps and removes your ability to bisect. Offer a plan: rehearsal in staging, scripted steps, a fixed window, and a reduction of scope by removing unused components.

Arguing a trade-off: building a paved road versus freedom

Data scientists will ask for any image, any version and any permission. Explain that freedom has a cost paid by the platform and, in the end, by the bank: unreviewed images, unpinned packages at run time, secrets in parameters. The compromise is a paved road that is easier than the alternative, with a documented exception path. Measure it by time to first successful run and by the share of pipelines on supported versions.

Questions to ask the interviewer

Good candidates interview back. Ask which Kubeflow release they run and when they last upgraded; whether identity goes through Dex or straight to their IdP; how they handle GPU queueing; who owns on-call; and whether any team's data needs stronger isolation than namespaces. The answers tell you whether the job is platform engineering or babysitting a demo install.

Sixty-second self-test

Answer aloud, without notes.

1. What header does the platform trust for identity, and what does that imply about bypass paths?
2. What does deleting a Profile do, and why is deleting the Profiles CRD worse?
3. Which database is supported for production Pipelines, and which is development only?
4. Who writes a ClusterTrainingRuntime, and who writes a TrainJob?
5. What label makes a runtime visible to the SDK?
6. Name three things that break first at scale, in order.
7. Why is the Dashboard v2 upgrade risky?
8. When would you recommend against Kubeflow?

If you hesitated on more than two, return to the detailed guide and reread the sections on the trust model, tenancy and upgrades before your next interview.

These are the traps that appear only once a Kubeflow platform has real users, real GPUs and a calendar of upgrades. Each one is written as symptom, cause and fix, because that is the order in which you meet them. The last sections turn the lessons into habits and review checklists for pull requests and design documents. Facts are as of the Kubeflow Community Distribution 26.03.1.

Production traps that appear at scale

Finished runs quietly fill the cluster

Symptom. Weeks after launch the pipelines UI becomes slow, list calls time out, and the Kubernetes API server shows rising latency with no change in user count.

Cause. Every run is an Argo Workflow object with pods and database rows, and nothing deletes them. The etcd database and watch caches grow. KFP 2.16 doubled the input size limit because large inputs had hit etcd and custom-resource size limits, which shows how close to the edge this is.

Fix. Set a platform default through PipelineConfig: ttl_seconds_after_finished (or the success and failure variants) and active_deadline_seconds, available since KFP 2.17. Until you upgrade, use Argo's own workflow TTL. Then count Workflow objects on a dashboard.

The default identity survived into production

Symptom. A security review finds user@example.com with the password from the quick-start.

Cause. The install works out of the box with a static user, and nobody changed it.

Fix. Replace the bcrypt hash in the dex-passwords secret in namespace auth, restart the pods there, and preferably remove static passwords by connecting your identity provider through a Dex connector or pointing oauth2-proxy straight at the IdP. Add a check to your overlay's CI that fails if the default hash is present.

HTTP works on localhost and loops on the real host

Symptom. Login appears to succeed but the Jupyter, Katib or Tensorboard web apps fail or redirect repeatedly.

Cause. The web apps set Secure cookies, which browsers reject over plain HTTP from any host except localhost.

Fix. Terminate TLS in front of istio-ingressgateway and keep that Service ClusterIP behind your ingress. The setting APP_SECURE_COOKIES=false exists as a last resort and is a production decision you should be prepared to justify.

A second route to the backend bypasses authentication

Symptom. Someone can call the Pipelines API without logging in, or claim another user's identity.

Cause. Backends trust the kubeflow-userid header that Istio sets after authentication. A NodePort, a long-lived port-forward, or a second Ingress that reaches ml-pipeline directly skips that step.

Fix. Audit every Service and Ingress that reaches the control planes, keep one front door, and apply AuthorizationPolicies that only admit the gateway and known service accounts. The error Unauthenticated: Request header error: there is no user identity header is the platform working, not a bug.

Partial gang allocation hoards GPUs

Symptom. GPUs show as allocated, utilisation is near zero, and multi-node jobs sit waiting while small jobs cannot start.

Cause. A job got some of its workers and waits for the rest. Without gang scheduling the scheduler happily places seven of eight pods.

Fix. Put a queue in front of the pool (Kueue through suspend and managedBy, or Volcano) and set podGroupPolicy on the runtime with coscheduling or volcano. Add activeDeadlineSeconds so no job can hold GPUs forever.

The runtime is invisible to the SDK

Symptom. client.list_runtimes() does not show the runtime you just applied.

Cause. The label trainer.kubeflow.org/framework is missing or wrong.

Fix. Add the label with one of torch, deepspeed, mlx, jax, xgboost, mpi or torchtune. Build new runtimes by copying a shipped one.

A stale document sent you in the wrong direction

Symptom. You follow the Runtime Guide and set torch: numProcPerNode: auto, and the job is rejected or behaves unexpectedly.

Cause. Trainer v2.2.0 removed numProcPerNode from the Torch MLPolicy and derives it from container resources; the shipped torch-distributed runtime uses torch: {}. Similar stale pages exist for the Trainer install (still showing VERSION=v2.1.0) and Katib (showing ref v0.17.0).

Fix. Trust the release notes and the shipped manifests over prose pages, and check a page's date against the version you run. Write your own runbook from what you verified.

Postgres was chosen because it was already there

Symptom. The API server will not start after an overlay change, or a team reports odd behaviour after a database migration.

Cause. PostgreSQL is documented as development and evaluation only, its overlay leaves sslmode unset by design, and it still depends on MLMD.

Fix. Use MySQL in production, on a managed service with restore-tested backups. If you must run Postgres for a pilot, set DBCONFIG_POSTGRESQLCONFIG_EXTRAPARAMS and the cache-server --db_extra_params, and label it as a pilot.

More traps from the moving parts

Parameters leaked a credential

Symptom. A token appears in the run details page.

Cause. Pipeline parameters are stored with the run and visible to anyone who can see it.

Fix. Pass a Secret name and use kubernetes.use_secret_as_env or use_secret_as_volume; rotate the leaked value.

The wrong namespace, the wrong bucket

Symptom. A run writes artifacts to a bucket belonging to another team, or fails with an object-store permission error.

Cause. The kfp-launcher ConfigMap is per namespace. A missing one falls back to defaults, such as minio://mlpipeline/v2/artifacts, and a copied one keeps the original team's bucket.

Fix. Generate kfp-launcher from the same automation that creates the Profile, with an explicit defaultPipelineRoot, and enforce bucket IAM as the real boundary.

Caching served a stale result

Symptom. A rerun finishes instantly with yesterday's output although the source data changed.

Cause. Caching is on by default, and the key covers the component spec and inputs, not the content behind an external URI.

Fix. Disable caching for components that read moving sources with set_caching_options(False), or make the changing identifier an input. Set the default off with --disable-execution-caching-by-default if your workload needs it.

The SDK and backend drifted apart

Symptom. A newly compiled pipeline fails with "output_metadata.json": proto: (line 1:237): unknown field "custom_path".

Cause. The pipeline was compiled with SDK 2.15.0 or 2.15.1 and ran on an older backend.

Fix. Use 2.15.2 or later, or upgrade the backend, and keep SDK and backend on the same minor version as a written policy.

Deprecation warnings turned into removals

Symptom. After an upgrade, a pipeline that worked fails on an API that no longer exists.

Cause. The warnings were ignored: dsl.Condition, set_gpu_limit, add_node_selector_constraint, delete_job and its siblings are already deprecated with replacements. Old authoring forms (ContainerOp, kfp.v2) are already gone.

Fix. Run pipeline CI with warnings as errors and keep a migration list per team.

Katib cannot find its objective

Symptom. Trials finish but the experiment reports MetricsUnavailable.

Cause. The training code did not print name=value on standard output, or the namespace lacks the injection label.

Fix. Print result=<value> for the exact metric name, and label the namespace katib.kubeflow.org/metrics-collector-injection=enabled.

Immutable fields break an upgrade

Symptom. The Deployment "..." is invalid: spec.selector: Invalid value: field is immutable.

Cause. Upstream changed a selector between versions, as happened with JobSet 0.10.1 to 0.11.0.

Fix. Delete the named Deployment and reapply: kubectl delete deployment jobset-controller-manager -n kubeflow-system --ignore-not-found. Expect a brief gap, and schedule it.

Monitoring went blank after an upgrade

Symptom. KServe dashboards show nothing after a release that otherwise succeeded.

Cause. KServe moves from kubeflow to the kserve namespace on the distribution's master, and monitoring selectors on the pod label app: kserve must become control-plane.

Fix. Read the upgrade notes for label and namespace changes, and keep dashboards and alerts in the same Git repository as the overlay.

Images fail on ARM nodes

Symptom. no matching manifest for linux/arm64.

Cause. Not every upstream image is published for arm64.

Fix. Use amd64 nodes for the platform, or test every image before committing to an ARM fleet.

Platform-ownership habits

Pin everything, and change one thing at a time. Your overlay references a KCD tag, images by digest, and the SDK version in every template. Upgrade components in a dedicated change, so that a failure has one suspect.

Never edit upstream manifests. Patches live in your overlay. When you need a change upstream does not support, patch it and record why, with a link to the issue, so the patch can be deleted later.

Rehearse upgrades in a staging cluster built from the same overlay. Include a Profile with a notebook volume, a finished run, a TrainJob and a KServe model if you serve. Verify four things: old runs are still visible, a new run completes, the notebook volume mounts, and a job schedules.

Script deletions by name. Upgrade steps that delete things (the JobSet Deployment, the Dashboard v1 resources, the KServe web app) are scripted with explicit names and a dry run, and read by a second person. Never use group-wide delete commands on anything that owns tenant data.

Put tenants in code. Profiles, quotas, LimitRanges, network policies and kfp-launcher ConfigMaps are generated from one team manifest. A team should exist in exactly one place in Git.

Bring your own monitoring and own the alerts. Kubeflow no longer ships Prometheus and Grafana manifests for Pipelines since 2.17. Alert on API error rate and latency, database and object-store health, old pending pods by reason, certificate expiry, controller restarts and idle GPU allocation.

Measure users, not pods. Track time to first successful run for a new team, the share of pipelines on supported SDK versions, queue wait time for GPUs, and support requests by category. A dashboard of green pods next to an unhappy user base is a warning.

Keep an exit route. Pipelines in IR YAML, models in open formats and data in buckets you own make Kubeflow replaceable. A platform that cannot be replaced cannot be negotiated with.

Read release notes the week they appear. The upgrade path is a sequence of manual steps, such as the MinIO to SeaweedFS migration and the Dashboard v2 replacement. The earlier you read them, the more time you have to plan.

Write down the boundary of responsibility. You own the control planes, identity, backend, runtimes, quotas and upgrades. Teams own pipeline code, images and data. Put it in the onboarding document and link it from the dashboard menu through the centraldashboard-config ConfigMap.

Cost habits

• Remove components you do not offer from example/kustomization.yaml; the full reference footprint is about 4.4 cores and 12.3 GiB of memory plus 65 GB of volumes at idle.
• Enable notebook culling (ENABLE_CULLING=true, with CULL_IDLE_TIME in minutes) and decide what happens to volumes of stopped notebooks.
• Require a quota on every Profile; keep self-service registration (CD_REGISTRATION_FLOW) off until the quota is automatic.
• Report GPU-hours requested versus used per team every month.
• Mirror images into a registry you control, which cuts pull failures and rate limits as well as risk.

Review checklist for pull requests to the platform overlay

Use this list on any change to the Kustomize overlay, runtimes, Profiles or policies.

1. Does the change reference a pinned tag or digest, rather than a branch?
2. Did any upstream manifest get edited directly instead of patched?
3. Does it add, rename or remove a CRD, and does the GitOps prune setting exempt Profiles?
4. If it touches a runtime, does it keep the trainer.kubeflow.org/framework label and a sensible podGroupPolicy?
5. If it touches Istio or AuthorizationPolicies, has someone tested login, /pipeline, and a service-account token from outside the cluster?
6. Are new secrets referenced from your secret manager rather than committed?
7. Does a new Profile include a quota, a LimitRange, default-deny network policy and a kfp-launcher ConfigMap?
8. Are new images mirrored, scanned and pinned?
9. Is there a rollback, and a note on whether any step is irreversible?
10. Did you rehearse it in staging, and is the result in the pull request?

Review checklist for design documents

Ask these questions of any proposal that adds a team, a workload type or a component.

• Isolation. What is the data classification, and does namespace-level isolation satisfy it? If not, where is the separate cluster in the design?
• Identity. How do users and machines authenticate, and what happens if the IdP is down?
• Capacity. Which component breaks first, and which measurement would tell us? Avoid any capacity claim that is not backed by a test.
• Failure. What is the user-visible symptom when each dependency (database, object store, gateway, queue) fails, and who is paged?
• Upgrade. Which manual steps does the next release add, and how is the change kept in step with the six-month support window?
• Cost. What is the idle cost, who sees the utilisation, and what stops a runaway job?
• Exit. Which artefacts (IR YAML, model formats, buckets) would survive a move to a managed service?
• Ownership. Who is on call for this, and where is that written down?

A short weekly routine

A platform is maintained by small repeated acts. Once a week, count finished Workflow objects and pending pods by reason, check certificate expiry, read the list of new subproject releases and security notes, review GPU utilisation per team, and skim the support channel for repeated questions. Once a month, publish the cost and utilisation report, and pick one recurring support question to remove by improving the paved road. Twice a year, run the upgrade you rehearsed.

If you take one idea from this page, take this: nearly every trap above is a default that suited a demonstration. Kubeflow gives you the parts; replacing each default with a deliberate, reviewed, version-controlled choice is the job, and it is what makes the platform trustworthy for everyone who builds on it.

Practice cards

Each card is a ten-minute exercise you can run in a staging cluster to turn a trap into a habit.

Card 1: the TTL. Compile a trivial pipeline with a PipelineConfig that sets ttl_seconds_after_finished to a few minutes, run it three times, and confirm the finished Workflow objects disappear. Then write down the default you would set for the whole platform and who would be allowed to override it.

Card 2: the bypass hunt. List every Service and Ingress that can reach ml-pipeline or the web apps without passing the gateway. For each one, write either "removed" or the reason it is safe. Then call the Pipelines API directly from a pod in a profile and read the error you receive.

Card 3: the partial gang. Create a runtime without podGroupPolicy, request more GPUs than exist, and observe how many pods start. Add a coscheduling policy and repeat. Keep the before and after outputs in your runbook.

Card 4: the invisible runtime. Apply a copy of a shipped runtime with the framework label removed and confirm it disappears from list_runtimes(). Restore the label and confirm it returns.

Card 5: the upgrade dry run. Take your overlay, bump the pinned tag in a branch, and run a server-side dry run against staging. List every conflict and every manual step in the release notes, and turn each into a scripted command.

Card 6: the deletion guard. Write an admission policy that denies deleting a Profile or the Profiles CRD without an explicit break-glass annotation. Test that a normal user is refused, and record how the break-glass path is approved.

Card 7: the leaked parameter. Submit a run with a fake token as a parameter, find it in the run details page, and then fix the pipeline to use a Secret reference. Seeing the value in the UI once is more persuasive than any policy document.

Card 8: the idle notebook. Enable culling on a staging notebook controller with a short idle time, leave a notebook alone, and confirm it scales to zero through the kubeflow-resource-stopped annotation while its volume remains. Decide how long the volume should live.

Questions that save an outage

Before any change reaches production, say these questions aloud in the review.

• What is the first user-visible symptom if this change is wrong?
• Can we roll it back in five minutes, and is that path tested?
• Does it delete anything, and is that thing scoped by name?
• Does it change an identity or network path, and who verified login and the Pipelines API afterward?
• Which alert would fire, and would we hear it?

The answers are usually short. The habit of asking them is what stops the long incidents.