The Complete Kedro Guide

This is part one of three. It covers everything you need to build and run a real Kedro project, not a teaser. By the end you can create a project from scratch, write small Python functions and wire them into a pipeline, describe your data in a YAML catalog instead of hard-coding file paths, pass parameters without editing code, run the whole pipeline or just a slice of it, look at it as a diagram, and read the error messages you will certainly meet along the way. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. Kedro's ideas look obvious on the page and only become yours after you have watched your own pipeline fail because of a typo in a dataset name and then fixed it.

This guide is written against Kedro 1.7, released at the end of September 2026. Kedro changed a lot between 0.19 and 1.0, and many tutorials on the internet still show the old commands. Where that matters, the text says so and shows the current form.

What Kedro is, and the problem it solves

Kedro is an open-source Python framework for writing data and machine-learning code so that it stays reproducible, modular and maintainable. It is an LF AI & Data project under the Apache 2.0 licence. The key word is framework: Kedro gives you a project layout, a way to describe steps and their dependencies, and a way to describe where data lives. You still write the actual logic (cleaning a table, training a model) in ordinary Python with pandas, scikit-learn, or whatever you like.

What Kedro is not is just as important, because beginners often expect the wrong thing. It is not a scheduler that runs your code every night. It is not a server, a dashboard, or a cloud service. It does not track experiments on its own. It is a way of authoring a pipeline well. When you later need scheduling, retries and monitoring, you hand the Kedro project to an orchestrator such as Airflow, and the project runs there unchanged. That separation is the design: write the logic once, cleanly, then choose where it runs.

The diagram is the whole idea. You write functions, you describe how they connect, you describe where the data comes from and goes to, and then Kedro runs them in the right order.

The problem: code that works once

Almost everyone starts a data project in a notebook. You load a CSV, clean it in one cell, engineer a feature in another, train a model in a third, and it works. The trouble starts a month later, or the moment a colleague tries to run it.

The notebook depends on the order you happened to run the cells in. A path like /Users/aya/Downloads/final_v3.csv is baked into cell four. A magic number such as 0.2 for the test split sits in the middle of a function, and nobody remembers why it is not 0.25. Your colleague runs the cells in a different order, gets a different answer, and neither of you can say which is right. Nothing is tested, because nothing is a function you can call on its own. When the project needs to run every night on a server, somebody must rewrite it, and the rewrite introduces new bugs.

These are not signs of a careless person. They are what you get when the structure of the project is an accident. Kedro's answer is to make the structure deliberate, and to make it the same in every project, so that a new team member opens any Kedro project and already knows where the data descriptions, the parameters, the functions and the wiring live.

What problems each part solves

Kedro addresses four separate pains, and it helps to keep them apart.

• Hard-coded paths and I/O inside functions. Kedro moves all reading and writing out of your functions and into a YAML file called the Data Catalog. Your function receives a DataFrame and returns a DataFrame. Whether that frame came from a local CSV, an S3 bucket or a database is a configuration detail.
• Hidden order of execution. Instead of remembering which cell to run first, you declare each step's inputs and outputs by name. Kedro works out the order from those names.
• Magic numbers and secrets in code. Parameters live in parameters.yml, credentials live in a git-ignored file, and code only refers to them by name.
• Every project laid out differently. kedro new creates the same folder structure each time, so habits transfer between projects and between teams.

There is a fifth, quieter benefit. Because each step is a small function with named inputs and outputs, you can test a step by calling it with a tiny DataFrame, without running anything else. That is the difference between code you can trust and code you can only hope about.

What came before: scripts, notebooks, and Makefiles

It is worth knowing what people did before, because Kedro only makes sense as a reaction to it.

The first approach is the notebook that grew. It is great for exploration and poor for anything that must run twice with the same result. The second is the folder of scripts numbered 01_clean.py, 02_features.py, 03_train.py, each reading a file the previous one wrote. This is better, because each script can be run alone, but the wiring is in your head. Nothing says that 03_train.py needs the output of 02_features.py, and the paths are repeated as strings in both files. Rename one and the other breaks silently.

The third approach is a build tool, such as a Makefile, that declares which file depends on which. It solves the ordering problem but knows nothing about Python functions, parameters or data formats, so you write a lot of glue by hand.

Kedro keeps the good part of each. From notebooks it keeps the ability to explore (it can open a notebook with your project's catalog already loaded, which we will do later). From scripts it keeps one-step-per-file modularity. From build tools it keeps the dependency graph, but the graph is drawn from the names of the datasets your functions consume and produce, so there is no separate file to keep in sync.

There are also neighbouring tools in this catalogue that people confuse with Kedro. Airflow and Prefect are orchestrators: they schedule and monitor tasks. DVC versions data and pipelines by tracking files. MLflow records experiments and models. Kedro sits at a different level, as the place where you author the code itself, and it cooperates with all of these. If you want to see how Kedro code is scheduled, the Airflow guide is the natural next read, and MLflow is where experiment tracking lives.

The mental model: four nouns

Kedro has a large surface area, but the core is four ideas. Learn these and everything else is a variation.

Node. A node wraps one Python function and gives names to its inputs and outputs. The function should be pure: given the same inputs, it returns the same outputs and does nothing else (no reading files, no printing results to disk). A node says: "call clean_orders with the dataset named orders, and call whatever it returns clean_orders."

Pipeline. A pipeline is a collection of nodes. The crucial fact is that the order of execution is worked out from the data dependencies, not from the order of the list. If node B consumes a dataset that node A produces, Kedro runs A first. You can list them in any order and get the same result. Pipelines can also be added together with +.

Data Catalog. The catalog is the registry of every dataset in the project, built from YAML files named catalog*.yml. A dataset entry says what kind of thing it is (a CSV, a Parquet file, a pickled model), where it lives and how to read and write it. Nodes never do their own I/O; Kedro loads inputs from the catalog before calling your function and saves the outputs after. If a name is used in the pipeline but is not in the catalog, Kedro quietly keeps that dataset in memory for the duration of the run.

Configuration. Everything that is not code: the catalog, the parameters, the credentials and the logging setup. It lives in a folder called conf/, split into environments such as base (shared, committed) and local (yours, git-ignored).

Two smaller nouns complete the picture. A runner decides how the nodes are executed, one after another by default. A session manages one run. You will rarely touch either as a beginner, but the words appear in error messages.

Installing Kedro and checking the setup

You need Python 3.10 or later and git. Git matters because kedro new downloads starter templates with it. Kedro 1.x no longer supports Python 3.9, so if python --version prints 3.9 or earlier, install a newer Python first. Kedro works on macOS, Linux and Windows, in cmd.exe and PowerShell.

Use one virtual environment per project. This is not a Kedro rule; it is how Python work stays sane. Two projects that need different versions of pandas cannot share one environment.

The official documentation recommends the tool uv. With it, you can try Kedro without installing anything permanently:

uvx kedro new --starter spaceflights-pandas --name spaceflights
cd spaceflights
uv run kedro run --pipelines __default__

uvx is an alias for uv tool run: it runs Kedro in a throwaway environment, just long enough to create the project. After that, uv sync and uv run use the project's own .venv.

If you prefer plain Python, create and activate an environment, then install:

python -m venv .venv
source .venv/bin/activate        # macOS and Linux
# .\.venv\Scripts\activate       # Windows cmd or PowerShell
pip install kedro

Other routes work too: uv pip install kedro, poetry add kedro, or conda install -c conda-forge kedro. On zsh, quote any package with brackets, for example pip install "kedro[server]", or the shell will try to expand the brackets.

Verify the install with:

kedro info

It prints an ASCII-art banner, the installed Kedro version, and a list of installed plugins, or the words "No plugins installed". You can also run kedro --version (or kedro -V), which prints only the version. python -m kedro works as well. If the command is not found, your environment is not activated; that is nearly always the cause.

Your first project: what kedro new creates

Everything in Kedro starts from kedro new. It asks a few questions and builds a project skeleton. You can answer the questions interactively, or pass them as flags so the command runs without asking. For this guide we will build a small project from an empty skeleton so that every file in it is one you wrote and understand:

kedro new --name orders_demo --tools=data --example=n

The flags mean: --name is the project name; --tools=data adds the numbered data/ folders (other tools are lint, test, log, docs and pyspark, and you can pass all or none); --example=n says do not include the example pipeline. Kedro refuses project names whose Python package name would shadow something built into Python, such as email, json or import, because that would break imports in confusing ways. Change into the new folder (run ls to see its name), create and activate a virtual environment there, and install the dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install "kedro-datasets[pandas-csvdataset]"

The second install adds the dataset classes we need. Kedro's core ships only a few in-memory datasets; everything that touches a real file format lives in a separate package called kedro-datasets, installed by groups. If you forget it, the first run fails with a message saying the class was not found, and that message is covered in the errors section.

The layout you now have looks like this:

orders-demo/
├── conf/
│   ├── base/          # catalog.yml, parameters.yml - committed to git
│   └── local/         # credentials.yml - git-ignored
├── data/              # 01_raw ... 08_reporting
├── notebooks/
├── src/orders_demo/
│   ├── __init__.py
│   ├── __main__.py
│   ├── pipeline_registry.py
│   ├── settings.py
│   └── pipelines/
├── tests/
├── pyproject.toml
├── requirements.txt
└── README.md

Read it as four zones.

conf/ is configuration. base holds settings shared by everyone and committed to git. local holds settings that are yours alone, above all credentials, and is ignored by git so passwords never end up in the repository.

data/ is where datasets live on your machine. The numbered folders are a convention, not a requirement: 01_raw for data exactly as received, 02_intermediate for lightly cleaned data, 03_primary for the tables you model from, 04_feature for features, 05_model_input for what goes into training, 06_models for trained models, 07_model_output for predictions, and 08_reporting for final tables and charts. The numbering tells a reader how far from the source a file is. Raw data is never edited in place.

src/orders_demo/ is your code. pipeline_registry.py lists the pipelines the project knows about. settings.py holds project-wide settings. Each pipeline gets its own folder under pipelines/, containing a nodes.py for the functions and a pipeline.py for the wiring.

pyproject.toml marks the project root and records its identity in a [tool.kedro] section: package_name, project_name and kedro_init_version. Kedro looks for this file to know it is inside a project, which is why you can run kedro run from any subfolder, and why running it from outside a project gives "Could not find the project configuration file 'pyproject.toml'".

Starters, if you want an example

If you would rather begin from a working example, kedro new --starter <alias> copies a finished project. The official aliases in 1.7 are astro-airflow-iris, spaceflights-pandas, spaceflights-pyspark, databricks-iris and support-agent-langgraph. The most widely used is spaceflights-pandas, a three-pipeline project that predicts shuttle prices, and it is the project the official tutorial walks through. Older articles mention a pandas-iris starter; it has not existed since version 0.19.

Your first pipeline, step by step

We will build a tiny but realistic pipeline. A CSV of shop orders comes in; the pipeline cleans it, then summarises revenue per city. Three datasets and two nodes, which is enough to show every moving part.

Step 1: put in some raw data

Create data/01_raw/orders.csv with this content. It has a few deliberate flaws: a missing city, a missing amount, and inconsistent capitalisation.

order_id,city,amount,created
1,cairo,120.5,2026-01-03
2,Riyadh,80,2026-01-04
3,  dubai ,200,2026-01-04
4,Cairo,,2026-01-05
5,,45,2026-01-06
6,riyadh,15.25,2026-01-07
7,Cairo,9,2026-01-08

Step 2: scaffold a pipeline

Kedro can create the folder structure for you:

kedro pipeline create orders

This makes src/orders_demo/pipelines/orders/ with nodes.py and pipeline.py, a conf/base/parameters_orders.yml for this pipeline's parameters, and a matching folder under tests/. The habit this builds is worth keeping: one pipeline, one folder, and its parameters and tests next to it.

Step 3: write the functions

Open src/orders_demo/pipelines/orders/nodes.py and replace its contents with two plain functions:

import pandas as pd


def clean_orders(orders: pd.DataFrame) -> pd.DataFrame:
    """Drop incomplete rows and tidy the city names."""
    cleaned = orders.dropna(subset=["city", "amount"]).copy()
    cleaned["city"] = cleaned["city"].str.strip().str.title()
    cleaned["amount"] = cleaned["amount"].astype(float)
    return cleaned


def summarise_by_city(orders: pd.DataFrame, min_amount: float) -> pd.DataFrame:
    """Count orders and total revenue per city, ignoring tiny orders."""
    large = orders[orders["amount"] >= min_amount]
    return large.groupby("city", as_index=False).agg(
        orders=("order_id", "count"),
        revenue=("amount", "sum"),
    )

Look at what is missing. There is no pd.read_csv, no file path, no to_csv, and the threshold is an argument rather than a constant. These functions know nothing about Kedro. You could import them into a test and call them with a three-row DataFrame. That is exactly the point: the logic is separate from the plumbing.

Step 4: wire them into a pipeline

Open pipeline.py in the same folder:

from kedro.pipeline import Node, Pipeline

from .nodes import clean_orders, summarise_by_city


def create_pipeline(**kwargs) -> Pipeline:
    return Pipeline(
        [
            Node(
                func=clean_orders,
                inputs="orders",
                outputs="clean_orders",
                name="clean_orders_node",
            ),
            Node(
                func=summarise_by_city,
                inputs=["clean_orders", "params:min_amount"],
                outputs="city_summary",
                name="summarise_by_city_node",
            ),
        ]
    )

Read this slowly, because it is the heart of Kedro. Each Node says: call this function, feed it these named inputs, and call its result by this name. The first node takes the dataset called orders and produces clean_orders. The second takes clean_orders (the very name the first produced, and this is the link that makes the order) plus a parameter called min_amount, and produces city_summary. The string "params:min_amount" is Kedro's syntax for "the parameter min_amount from the parameters file".

If a function takes several arguments, inputs is a list, matched to the function's arguments by position. If it returns several values, outputs is a list too. Everything after outputs in Node(...) is keyword-only, and name is optional but strongly recommended, because names appear in logs, in error messages and in --nodes on the command line.

Step 5: describe the data

Open conf/base/catalog.yml and add three entries:

orders:
  type: pandas.CSVDataset
  filepath: data/01_raw/orders.csv

clean_orders:
  type: pandas.CSVDataset
  filepath: data/02_intermediate/clean_orders.csv

city_summary:
  type: pandas.CSVDataset
  filepath: data/08_reporting/city_summary.csv

Each key is a dataset name, identical to the strings in pipeline.py. The type is the dataset class; pandas.CSVDataset is shorthand for kedro_datasets.pandas.CSVDataset. Note the spelling: Dataset with a lowercase s. Before kedro-datasets 2.0 it was DataSet, and old tutorials still show that. The old spelling produces a warning and then a "class not found" error.

Step 6: set the parameter

Open conf/base/parameters_orders.yml and set:

min_amount: 10

The pipeline's params:min_amount input now resolves to 10.

Running the pipeline and reading the output

Now run it:

kedro run

With no options, Kedro runs the pipeline named __default__. Where does that come from? Open pipeline_registry.py. The template calls find_pipelines(), which discovers every folder under pipelines/ that exposes a create_pipeline function, registers each under its folder name, and sets __default__ to the sum of all of them. So kedro run runs everything, and kedro run --pipelines orders runs just the orders pipeline.

The output is a stream of log lines. You will see Kedro loading the project configuration, then for each node a line announcing it is running, lines saying it is loading a dataset and saving a dataset, a line saying the node completed, and a final "Pipeline execution completed successfully." The exact wording and formatting differ slightly between versions and with Rich logging, so learn the shape rather than the text: load inputs, run function, save outputs, next node.

When it finishes, look at data/02_intermediate/clean_orders.csv and data/08_reporting/city_summary.csv. The cleaned file has five rows (the two incomplete ones were dropped), the cities are tidy ("Cairo", "Riyadh", "Dubai"), and the summary has one row per city. With min_amount: 10 the order from Cairo worth 9 is excluded, so Cairo shows one counted order rather than two. Change the parameter and run again to see the numbers move.

Notice what you did not have to do: write any code to load the CSV, or tell Kedro which node goes first.

Seeing the plan without running

Two commands show what Kedro intends to do, which is useful when a pipeline grows:

kedro registry list
kedro registry describe orders

The first lists the pipeline names. The second lists the nodes of one pipeline (or __default__ if you give no name).

The Data Catalog in depth

You have used three catalog entries. The catalog can do much more, and most of a beginner's day-to-day Kedro work is editing it.

Anatomy of an entry

Here is one entry with most of the options a beginner will meet:

companies:
  type: pandas.CSVDataset
  filepath: data/01_raw/companies.csv
  load_args:
    sep: ","
  save_args:
    index: false
  metadata:
    kedro-viz:
      layer: raw

• type names the dataset class. Kedro resolves a short name like pandas.CSVDataset first against kedro.io, then against kedro_datasets, and finally treats it as a full dotted path to a class in your own code.
• filepath is a path or URL. Local paths are relative to the project root. Because Kedro uses the fsspec library underneath, the same key accepts s3://, gcs://, abfs://, hdfs:// and http(s):// URLs. Moving a dataset from your laptop to cloud storage is therefore a one-line change, with no code change in the nodes.
• load_args are forwarded to the underlying reader (for the CSV dataset, pandas.read_csv), and save_args to the writer (DataFrame.to_csv). Here index: false stops pandas writing the row numbers as an extra column, a classic annoyance.
• credentials (not shown) names a block in a credentials file, covered below.
• metadata is free-form information that Kedro ignores but tools can use. The kedro-viz: layer: entry tells Kedro-Viz which layer to draw the dataset in. In old tutorials the layer: key sat at the top level of the entry; it has since moved under metadata.

Choosing a dataset type

The type depends on what you are storing. A few you will meet early:

Each group needs its own dependencies. Install them by naming the group in brackets, for example pip install "kedro-datasets[pandas-csvdataset,pandas-parquetdataset]". If you skip this you will see either No module named '...'. Please install the missing dependencies for kedro_datasets... or Class '...' not found, and both mean the same thing: install the right group.

Versioning a dataset

Add versioned: true to an entry and Kedro stops overwriting the file. Each save goes into a new timestamped folder, <filepath>/<timestamp>/<filename>, and loads read the latest one by default. This keeps a history of a model or a report without any code. You can load an older one with kedro run --load-versions=city_summary:2026-01-01T00.00.00.000Z. Versioning is not available for http(s):// paths, and asking for it there raises "Versioning is not supported for HTTP protocols."

What happens to datasets that are not in the catalog

A dataset that appears in the pipeline but has no catalog entry is held in memory while the run lasts and discarded afterwards. That is useful for throwaway intermediates and dangerous for anything you want to keep or inspect. The rule of thumb for beginners: if you might want to look at it afterwards, give it a catalog entry. Also remember that memory-only datasets cannot be passed between separate tasks when a pipeline is later split across an orchestrator, so in production every intermediate gets an entry.

Many similar datasets: factories

If you have twenty CSV files that follow a naming pattern, writing twenty entries is tedious. A dataset factory matches names with a pattern, like a reversed f-string:

"{name}_data":
  type: pandas.CSVDataset
  filepath: data/01_raw/{name}_data.csv

Now any dataset whose name ends in _data, such as customers_data or shipments_data, is served by this one entry, with {name} filled in. The quotes are mandatory; without them YAML reads the braces as its own syntax and you get a parser error. Explicit entries always beat patterns, and there may be at most one catch-all pattern of your own. You do not need factories on day one, but you should recognise them when a colleague's catalog seems to be missing entries that the pipeline uses.

To see how Kedro resolves names, use:

kedro catalog describe-datasets
kedro catalog list-patterns
kedro catalog resolve-patterns

describe-datasets groups the datasets into those defined explicitly, those produced by a factory pattern and those that will default to memory. list-patterns shows the patterns in priority order, and resolve-patterns prints the fully resolved configuration. These replaced the older kedro catalog list and kedro catalog rank in version 1.0.

Parameters and credentials

Parameters: numbers that are not code

A parameter is a value that changes the behaviour of a pipeline and that you may want to adjust without editing functions: a threshold, a split ratio, a random seed, a list of columns. Parameters live in YAML files whose names start with parameters, inside conf/base/ (or another environment). You already used one.

Nested values work too:

min_amount: 10
model_options:
  test_size: 0.2
  random_state: 3

In a node, reference a single value as "params:min_amount", a nested one with dots as "params:model_options.test_size", or the whole dictionary as "params:model_options". The special name "parameters" passes every parameter at once. The function receives ordinary Python values: a float, a dict.

You can override a parameter for one run, without editing any file, from the command line:

kedro run --params="min_amount=50"

The syntax is key=value, separated by commas for several (--params="min_amount=50,model_options.test_size=0.3"), with dots for nested keys. Kedro turns values into integers or floats where it can and leaves the rest as strings. A colon (key:value) was the syntax before 0.19 and some documentation examples still show it; use the equals sign. If a value contains spaces, quote the whole argument.

Credentials: secrets that stay out of git

Some datasets need a password, key or token, for example an S3 bucket or a database. Never put these in catalog.yml, because that file is committed. Put them in conf/local/credentials.yml, which git ignores:

dev_s3:
  client_kwargs:
    aws_access_key_id: your-key-id
    aws_secret_access_key: your-secret

and reference the block by name from the catalog:

orders:
  type: pandas.CSVDataset
  filepath: s3://my-bucket/orders.csv
  credentials: dev_s3

Anything in conf/local/ is ignored by git, and kedro package leaves those files out of the packaged configuration. On a server or in a CI job there is no conf/local/credentials.yml, so the usual approach is to read secrets from environment variables. Inside a credentials file only, you can write ${oc.env:AWS_ACCESS_KEY_ID} and Kedro substitutes the variable's value. For AWS specifically, the standard variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN are also picked up directly.

Configuration environments

The conf/ folder holds more than one environment, and the idea is simple but easy to misread. An environment is a subfolder. base is the foundation and is always loaded. On top of it Kedro loads a second environment, local by default, and where the same top-level key appears in both, the second one wins. By default this replacement is whole-key: if local defines model_options, it replaces base's entire model_options, not only the keys it mentions.

The effect is that one project can behave differently in different places with no code change. base holds the settings that are right for everyone. Your local folder holds your own tweaks, such as a smaller sample file to keep development fast. You can define further environments (dev, staging, prod) as folders and choose one at run time:

kedro run --env=staging

Or set the environment variable KEDRO_ENV=staging; if both are present the command-line flag wins.

Two warnings follow from how this works. First, within one environment, the same top-level key must not appear in two files, or Kedro raises ValueError: Duplicate keys found in ... naming both files. That protects you from silently losing a setting. Second, if you name an environment that does not exist, you get a MissingConfigException about a configuration path that does not exist or is not a directory. Create conf/<env>/ first.

Kedro also has a templating feature built on a library called OmegaConf, so a value in YAML can refer to another with ${...}. You do not need it yet; mid-level covers it.

The everyday commands, grouped by what you want to do

You do not need to memorise the full command list. Group it by intent.

I want to run things

kedro run                                  # the default pipeline
kedro run --pipelines orders               # one pipeline
kedro run --pipelines orders,training      # several at once
kedro run --env=staging                    # another configuration environment
kedro run --params="min_amount=50"         # override parameters

Use --pipelines (plural). Kedro 1.2 introduced it so several pipelines can run together, and it deprecated the singular --pipeline flag: the old flag still works but prints a deprecation warning, and using both at once is an error. Most online tutorials still show the old flag.

I want to run only part of a pipeline

While developing you rarely want the whole thing. Kedro offers several ways to pick a slice:

kedro run --nodes=clean_orders_node        # specific nodes, by name
kedro run --tags=cleaning                  # nodes that carry a tag
kedro run --from-nodes=summarise_by_city_node
kedro run --to-nodes=clean_orders_node
kedro run --from-inputs=clean_orders
kedro run --to-outputs=city_summary

--from-nodes runs that node and everything after it; --to-nodes runs everything up to and including it. --from-inputs and --to-outputs do the same by dataset name. Tags are labels you add to a node in code, Node(..., tags=["cleaning"]), and --tags selects nodes carrying any of the tags you list. Names and tags may contain only letters, digits, hyphens, underscores and full stops.

A very useful flag is --only-missing-outputs, which skips any node whose saved outputs already all exist. After a failure halfway through a long run, it lets you resume without recomputing the early steps.

If your filters match nothing, Kedro says "Pipeline contains no nodes after applying all provided filters." That message means a typo or a wrong name, not a broken pipeline.

I want to look around

kedro info                                 # version and plugins
kedro registry list                        # pipeline names
kedro registry describe orders             # nodes in a pipeline
kedro catalog describe-datasets            # how each dataset is defined
kedro starter list                         # available starters

I want to create or change things

kedro new                                  # new project
kedro pipeline create <name>               # new pipeline folder
kedro pipeline delete <name>               # remove one

I want to work interactively

kedro ipython                              # IPython with the project loaded
kedro jupyter lab                          # JupyterLab with a Kedro kernel
kedro jupyter notebook                     # classic notebook

The next sections cover the interactive and visual tools.

Exploring with notebooks and Kedro-Viz

Notebooks, the Kedro way

Notebooks are the best tool for looking at data, and Kedro does not ask you to give them up. It asks you to keep them as a window onto the project and not as the place where the logic lives. Start a notebook with your project already loaded:

kedro jupyter lab

This creates a kernel named after your package, shown as Kedro (orders_demo), and opens JupyterLab. In the kernel, these variables already exist: catalog, context, pipelines and session. Try:

catalog.keys()                      # all dataset names
df = catalog.load("clean_orders")   # read a dataset the way a node would
df.head()
catalog.load("parameters")          # every parameter as a dictionary
pipelines["__default__"]            # the pipeline object

catalog.load("name") is the important one. It returns a dataset exactly as your nodes would see it, so you can experiment on real data and, once a snippet works, paste it into nodes.py as a function. That is the healthy loop: explore in the notebook, then promote working code into a node.

If you launch Jupyter some other way, load the project with two magic commands: %load_ext kedro.ipython, then %reload_kedro <path to project root>. If you see "Line magic function '%reload_kedro' not found", the extension was not loaded. kedro ipython does the same for a terminal IPython session. If it complains that the module IPython is not found, install the project requirements first.

A warning for those who read older material: the catalog method catalog.list() no longer exists. Since 1.0 use catalog.keys() or catalog.filter("pattern").

Kedro-Viz: see the pipeline as a diagram

Kedro-Viz is a separate package that draws your pipeline as an interactive graph, with datasets and nodes as shapes and the dependencies as arrows. It is the best way to explain a pipeline to someone else, and also the quickest way to spot a mistake, such as a node with no inputs.

pip install kedro-viz
kedro viz run

It opens a browser tab on 127.0.0.1:4141. (The shorter kedro viz also works.) Click a node to see its code and parameters, click a dataset to see its type and file path, and use the filters on the side to show only some pipelines, tags or layers. The layers you set under metadata: kedro-viz: layer: in the catalog are drawn as horizontal bands, so a pipeline reads from raw data at one edge to reporting at the other. Use --no-browser if you are on a remote machine and want to open the address yourself.

Growing the project: several pipelines, namespaces and reuse

A project of one pipeline is a demo. Real projects have several: data preparation, training, evaluation. Kedro's answer is the modular pipeline: each one lives in its own folder with its own nodes.py and pipeline.py, and exposes a create_pipeline function. You already made one with kedro pipeline create orders. Making a second is the same command:

kedro pipeline create reporting

Because pipeline_registry.py uses find_pipelines(), the new pipeline is registered the moment its folder has a create_pipeline. Then kedro run runs both (the __default__ pipeline is their sum) and kedro run --pipelines reporting runs only the new one. Datasets are shared between pipelines by name: if reporting consumes city_summary, it finds it in the same catalog, produced by orders. Keeping one pipeline per folder means each can be understood, tested and run on its own.

Namespaces: using a pipeline twice

Sometimes the same pipeline should run twice on different inputs, for example one model trained on two regions. Wrapping it in a namespace prefixes its node names, dataset names and parameters with a label, so two copies do not collide:

from kedro.pipeline import Pipeline

base = create_pipeline()
riyadh = Pipeline(base, namespace="riyadh")
cairo = Pipeline(base, namespace="cairo")

Now the dataset clean_orders becomes riyadh.clean_orders in the one copy and cairo.clean_orders in the other. Namespaces are also how Kedro-Viz collapses groups of nodes into one expandable box. Run one with kedro run --namespaces=riyadh; the flag accepts several, comma-separated. The full dotted-name rules are covered at mid-level; for now remember that the dot character in a dataset name is reserved for namespaces, so avoid dots in your own dataset names.

If you reuse a pipeline without a namespace you will see "Pipeline nodes must have unique names". That message means the same node name appears twice, and a namespace is the fix.

Testing a node

The payoff of pure functions is that testing is ordinary Python. No Kedro needed:

import pandas as pd

from orders_demo.pipelines.orders.nodes import clean_orders, summarise_by_city


def test_clean_orders_drops_incomplete_rows():
    raw = pd.DataFrame(
        {"order_id": [1, 2], "city": ["cairo", None], "amount": [10.0, 5.0]}
    )
    result = clean_orders(raw)
    assert len(result) == 1
    assert result.iloc[0]["city"] == "Cairo"


def test_summary_ignores_small_orders():
    orders = pd.DataFrame(
        {"order_id": [1, 2], "city": ["Cairo", "Cairo"], "amount": [100.0, 1.0]}
    )
    summary = summarise_by_city(orders, min_amount=10)
    assert summary.iloc[0]["orders"] == 1

Run it with pytest from the project root. If you created the project with the test tool, pytest is already in the requirements; otherwise pip install pytest. The test builds a three-row DataFrame, calls the function and checks the answer. It does not read a file, does not need a catalog and runs in milliseconds. That is the benefit of keeping I/O out of functions.

Common errors and how to read them

Kedro errors are long because they include the chain of causes. The rule is: read the last line first, then the one above it. The final exception says what went wrong; the chained ones say where. Here are the ones you will meet in your first week.

Two habits make errors much cheaper. Always give nodes a name, because then the log line says which step failed in words you chose. And when a dataset fails to load, open a notebook and call catalog.load("that_name") directly. The error is then isolated from the pipeline, and you can try the fix interactively.

Finally, watch for two warnings. KedroDeprecationWarning means you used a flag or function scheduled for removal (the old --pipeline flag is the common one); it still works today, but fix it now. A KedroExperimentalWarning means you touched a feature whose design may still change.

Putting it all together

Here is the whole project in one pass, the way you would set it up for a colleague. Every command is one you have already used.

kedro new --name orders_demo --tools=data --example=n
cd orders-demo
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install "kedro-datasets[pandas-csvdataset]" pytest
git init && git add . && git commit -m "Kedro skeleton"
kedro pipeline create orders

Then edit five files:

1. data/01_raw/orders.csv, the raw data.
2. src/orders_demo/pipelines/orders/nodes.py, the two pure functions.
3. src/orders_demo/pipelines/orders/pipeline.py, the two nodes.
4. conf/base/catalog.yml, the three datasets.
5. conf/base/parameters_orders.yml, the min_amount parameter.

Run it, inspect it and test it:

kedro run
kedro registry describe orders
kedro run --params="min_amount=50"
pytest
kedro viz run

Step back and notice the properties of what you built. The functions contain no paths and no magic numbers, so they are reusable. The wiring is explicit and named, so a stranger can read it. The data locations are in YAML, so moving from a laptop to cloud storage means editing one file. The parameter can be changed from the command line, so experiments need no code edits. The code is tested without running the pipeline. And the project has the same shape as every other Kedro project in the world, so any Kedro user can find their way around it.

A final check for people working at companies in the Gulf or Egypt: where your data sits matters for regulation. Because locations are just filepath values, you can point a dataset at a storage bucket in a Middle East cloud region and keep the code identical, which is exactly the kind of change this design makes easy. Confirm the residency rules that apply to your employer, but know that Kedro will not stand in the way.

What you can now do, and what comes next

You can now create a Kedro project, explain its four nouns (node, pipeline, catalog, configuration), write pure functions and wire them into a pipeline, describe data in YAML, pass parameters and keep credentials out of git, choose a configuration environment, run all or part of a pipeline, explore it in a notebook, draw it with Kedro-Viz, split a project into modular pipelines, test a node, and read Kedro's common errors.

Here is what to check you can do without looking anything up:

Mid-level takes every one of those topics further: how the catalog, config loader and runners work internally, dataset factories and templating with OmegaConf, parameter and dataset validation, hooks, the choice of runner, packaging a project as a wheel, and deploying it with Docker and orchestrators.

Senior then covers what you own when Kedro is your team's platform: the security model, scaling limits, upgrade and migration planning, custom datasets and runners, and where Kedro stops and another tool should take over.

Natural next reads in this catalogue are Airflow for scheduling a Kedro project, MLflow for experiment tracking, DVC for versioning data, and Docker for packaging it all so it runs anywhere.

Sources

• Kedro documentation home
• Install Kedro
• Kedro concepts
• Commands reference
• Create a new project
• Project tools
• Starters
• Configuration basics
• Parameters and credentials
• The Data Catalog
• Dataset factories
• Nodes
• Modular pipelines
• Namespaces
• Run a pipeline
• Kedro and notebooks
• Telemetry
• Migration guide
• Kedro release notes
• Kedro-Viz documentation
• kedro-datasets documentation

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Kedro. Mid-level reviews this plus its own material; Senior reviews all three.

Interviewers rarely ask about Kedro's syntax. They ask whether you understand why it exists and whether you have actually used it. So this page is organised around explanations you can say out loud, the commands you should be able to type from memory, the traps that catch beginners, and model answers you can adapt to your own words. The version discussed throughout is Kedro 1.7; if an interviewer quotes a command from an older tutorial, you can politely show you know the current form.

The thirty-second answer

Kedro is an open-source Python framework for writing data and machine-learning pipelines that are reproducible and maintainable. You write plain functions, wrap each in a node that names its inputs and outputs, and Kedro works out the execution order from those names. Data locations live in a YAML Data Catalog and settings live in configuration files, so the functions contain no paths or magic numbers. It is for authoring pipelines, not for scheduling them.

Then the line that shows you have used it: "Kedro decides the order from data dependencies, not from the order I list the nodes, and a dataset is just a name that appears both in my pipeline and in catalog.yml."

The model

Keep the relationships in your head as a chain. Functions are wrapped in nodes; nodes form a pipeline; datasets named by the nodes are resolved by the catalog; the catalog reads its description from configuration; a runner executes it. If you can narrate that chain calmly, most follow-up questions become easy.

Commands to have ready

An interviewer may hand you a keyboard or ask you to write commands on a whiteboard. These should come without thought.

pip install kedro                               # install
kedro info                                      # version and plugins
kedro new --name demo --tools=data --example=n  # new project
kedro starter list                              # official starters
kedro pipeline create orders                    # new modular pipeline
kedro run                                       # run __default__
kedro run --pipelines orders                    # one pipeline
kedro run --params="min_amount=50"              # override a parameter
kedro run --env=staging                         # another environment
kedro run --tags=cleaning                       # nodes with a tag
kedro run --nodes=clean_orders_node             # named nodes
kedro run --from-nodes=x                        # x and everything after
kedro run --to-outputs=city_summary             # up to an output
kedro run --only-missing-outputs                # skip finished nodes
kedro registry list                             # pipeline names
kedro registry describe orders                  # nodes of one pipeline
kedro catalog describe-datasets                 # how datasets are defined
kedro jupyter lab                               # notebook with the project
kedro viz run                                   # the pipeline diagram

Know also what a Kedro catalog entry looks like, since "write a catalog entry for a CSV" is a favourite:

orders:
  type: pandas.CSVDataset
  filepath: data/01_raw/orders.csv
  load_args:
    sep: ","
  save_args:
    index: false

And a node:

from kedro.pipeline import Node, Pipeline

Pipeline([
    Node(func=clean_orders, inputs="orders", outputs="clean_orders",
         name="clean_orders_node"),
])

If you can write those three snippets from memory, you have covered the most common practical questions.

The classic traps, and why they happen

Thinking order comes from the list. A candidate says "the nodes run in the order I wrote them". The correct statement is that the order is resolved from data dependencies: a node runs once all its inputs are available. The list order is irrelevant. This follows from the design: because each node declares what it consumes and produces by name, Kedro builds a dependency graph and runs a valid ordering of it.

Doing I/O inside a function. Calling pd.read_csv inside a node works, and it is the single most common way to defeat Kedro's benefits. The function can no longer be tested with a small DataFrame, the path is hard-coded, and moving to cloud storage requires editing code. Kedro's rule is that nodes receive data and return data; the catalog does the reading and writing.

Unsaved intermediates. A dataset named in the pipeline but absent from the catalog is silently kept in memory and thrown away at the end of the run. That is convenient for scratch values and baffling when you go looking for a file that was never written. It also breaks when a pipeline is later split across separate machines, because memory datasets cannot cross task boundaries.

The old spelling. CSVDataSet with a capital S is from before kedro-datasets 2.0. The current name is CSVDataset. Old tutorials make this the top cause of "class not found".

Forgetting to install the dataset group. Core Kedro ships only in-memory datasets. File formats live in kedro-datasets, installed by group such as kedro-datasets[pandas-csvdataset]. A missing group gives a "No module named" or "Class not found" error.

Committing secrets. Credentials belong in conf/local/credentials.yml, which is git-ignored, and are referenced from the catalog by block name. Anything committed to conf/base is public to everyone with repository access.

Using outdated commands. --pipeline (singular) is deprecated since 1.2 in favour of --pipelines. --namespace became --namespaces in 1.0. kedro catalog list was replaced by describe-datasets, and catalog.list() was removed. --params uses key=value, not key:value. Quoting the right modern form is a cheap way to signal currency.

Misunderstanding what Kedro is. Candidates sometimes say Kedro "schedules" or "orchestrates" pipelines. It does not. There is no built-in scheduler, retry logic or run-history interface. Kedro is where you write the pipeline; Airflow, Prefect, Kubeflow, Databricks and similar tools are where you run it on a schedule.

Environment merging surprises. Overlay keys replace base keys wholesale by default. If local defines model_options, the entire block from base is replaced, not merged key by key.

Common interview questions

What is Kedro, and what problem does it solve? Kedro is a Python framework for building data and ML code in a consistent, modular, reproducible way. The problem is code that only works in the author's notebook: hard-coded paths, hidden execution order, magic numbers and no tests. Kedro separates logic (pure functions), wiring (the pipeline), data location (the catalog) and settings (configuration), and gives every project the same layout.

Is Kedro an orchestrator? No. It is a code-authoring framework. It has no scheduler, retries or run-history interface. You author the pipeline with Kedro and then run it on an orchestrator such as Airflow, or on a platform such as Databricks or Kubeflow, usually by packaging it in a container and running kedro run with node or tag selections as separate tasks.

What is a node? What makes a good one? A node wraps a Python function and names its inputs and outputs. A good node is a pure function: same inputs, same outputs, no hidden side effects, and no I/O. It is small enough to test with a three-row DataFrame.

How does Kedro decide what runs first? It builds a graph from dataset names. If one node's output name is another node's input name, the first runs before the second. Nodes with no dependency between them have no guaranteed order relative to each other. Listing order does not matter.

What is the Data Catalog, and why use it? It is the registry of datasets, described in YAML. Each entry names a dataset class, a path and optional arguments. Nodes never do I/O; Kedro loads inputs and saves outputs using the catalog. The benefits are that code is independent of storage, switching from local to S3 is a YAML change, and the same node works with any supported format.

What happens to a dataset that is not in the catalog? Kedro creates it as an in-memory dataset for the duration of the run. It is gone afterwards, so it cannot be inspected and cannot be passed between separate tasks. Anything you want to keep needs an entry.

How do you pass parameters, and why not hard-code them? Put them in conf/base/parameters.yml (or a parameters_<pipeline>.yml) and consume them in a node as params:name, or params:a.b for nested ones, or parameters for all of them. Hard-coded values hide decisions inside functions and force code edits to experiment. You can override a parameter for one run with kedro run --params="name=value".

Where do credentials go? In conf/local/credentials.yml, which is git-ignored. The catalog entry refers to a block by name with credentials: <block>. In CI or containers, read from environment variables instead: a credentials file can use ${oc.env:VAR}.

Explain configuration environments. conf/ has subfolders called environments. base is always loaded and committed. A second environment is layered on top, local by default, and its values override base. You select another with --env or the KEDRO_ENV variable, and the flag wins if both are set. It lets one codebase run with different data locations and settings in development, staging and production.

How do you run part of a pipeline? Use --nodes, --tags, --from-nodes, --to-nodes, --from-inputs, --to-outputs or --namespaces. To resume after a failure, --only-missing-outputs skips nodes whose saved outputs all exist. If no node matches, Kedro says the pipeline contains no nodes after applying the filters.

What is a modular pipeline? A pipeline that lives in its own folder under src/<package>/pipelines/<name>/, with nodes.py, pipeline.py and a create_pipeline function. kedro pipeline create <name> scaffolds it, including a parameters file and a test folder. The registry's find_pipelines() discovers it, and __default__ is the sum of all discovered pipelines.

What is a namespace? A prefix applied to node names and, by default, dataset and parameter names, so the same pipeline can be used more than once without collisions. It is also used to group nodes in Kedro-Viz. You run one with kedro run --namespaces=<name>.

How do you work in notebooks with Kedro? kedro jupyter lab starts a kernel with catalog, context, pipelines and session loaded. catalog.load("name") loads a dataset as a node would see it. Good code is then promoted out of the notebook into a node function, so the notebook stays a place to explore rather than the place the logic lives.

What is Kedro-Viz? A separate package, kedro-viz, that draws the pipeline as an interactive graph on port 4141 via kedro viz run. It is useful for explaining pipelines and spotting structural mistakes. It reads the project; it does not run it.

How do you test a Kedro pipeline? Because nodes are pure functions, unit-test them with pytest by calling the function with a small DataFrame. The project template creates a tests/ folder and kedro pipeline create adds one per pipeline. You can also test whole-pipeline behaviour by running a slice on a small dataset.

What changed between Kedro 0.19 and 1.0? Key points: KedroDataCatalog became DataCatalog; several catalog methods such as list() and add_feed_dict() were removed; kedro catalog list and friends became describe-datasets, list-patterns and resolve-patterns; --namespace became --namespaces; extra_params became runtime_params; micro-packaging was removed. Since 1.2, --pipelines replaces --pipeline. A project created with 0.19 fails with a version mismatch on 1.x until it is migrated.

How does Kedro fit with MLflow, Airflow, DVC? They do different jobs. MLflow tracks experiments and models (a community plugin, kedro-mlflow, connects them). Airflow schedules and monitors; the kedro-airflow plugin converts a Kedro pipeline into an Airflow DAG. DVC versions data files. Kedro authors the pipeline and works alongside all three.

Is Kedro free? What is the telemetry? It is Apache 2.0 licensed. A kedro-telemetry package collects anonymous usage statistics by default, never your data, and you opt out with KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK, or --telemetry=no at project creation.

Why would you use Kedro instead of a folder of scripts? Consistency and explicitness. Scripts hide dependencies and repeat file paths as strings. Kedro makes dependencies explicit, data access configurable, and the layout predictable, so a new team member is productive faster and the pipeline is testable and portable to an orchestrator.

When would you not use Kedro? For a one-off exploration that will never run twice, a notebook is fine. When the main need is scheduling, retries and monitoring, choose an orchestrator, possibly with Kedro for the authoring. When the workload is huge and distributed, the heavy lifting belongs to a data engine such as Spark, which Kedro can drive but does not replace.

How to answer the scenario question

Interviewers sometimes give you a scenario, such as "a colleague sends you a notebook that loads a CSV, trains a model and prints the score; how would you turn it into a Kedro project?" Here is a structure that works. Start by saying you would create the project with kedro new and commit the skeleton. Move the data to data/01_raw and describe it in the catalog, so the code stops hard-coding paths. Split the notebook into functions by responsibility (clean, split, train, evaluate) and make each pure. Wrap them in nodes with named inputs and outputs, and give every intermediate that matters a catalog entry. Move every constant, such as the split ratio and the random seed, into parameters.yml. Add a few unit tests for the functions. Run the whole thing with kedro run, and look at it in Kedro-Viz to check the shape matches what you intended. You have answered the question and demonstrated each concept in order.

A second scenario: "The pipeline fails with DatasetNotFoundError. What do you check?" Say that the error names a dataset that is used but defined nowhere, so you compare the string in the pipeline with the key in catalog.yml, looking for a typo or a missing entry. You would use kedro catalog describe-datasets and, if factory patterns are in play, kedro catalog resolve-patterns to see how names resolve.

A third: "It works on my machine, but a colleague gets a version mismatch." Say that Kedro compares the major version in pyproject.toml's kedro_init_version against the installed package, and pinning the Kedro version in requirements.txt fixes the drift.

Sixty-second self-test

Answer each without looking back. If you hesitate on more than two, reread the matching section of the detailed track.

If you can answer all fifteen in a minute, you are ready for a beginner-level Kedro conversation. The mid-level track turns to the questions that come next: how the catalog and runners work, how to validate parameters and data, and how to put a Kedro project into production.

Part one of three. A handful of mistakes produce most of the trouble beginners have with Kedro: a name that does not match between code and catalog, data read inside a function instead of through the catalog, a missing install, and advice copied from a tutorial written for an older version. Start with the table, then the longer explanations, the habits worth adopting on day one, and the checklist at the end. Everything here is checked against Kedro 1.7.

Common errors at this level

The mistakes beginners actually make

Each of these comes with a symptom, a cause and a fix, because knowing the cause is what stops you making the mistake twice.

Reading and writing files inside a node

Symptom. Your pipeline runs, but every function starts with pd.read_csv("data/01_raw/orders.csv"). Tests need real files, and changing the location means editing code.

Cause. Habit from notebooks, where loading data is the first line of every cell. In Kedro that job belongs to the catalog.

Fix. Make the function take a DataFrame argument and return a DataFrame. Add a catalog entry for the file. In the node, name that entry as the input. If you truly need the raw file for a one-off exploration, do it in a notebook with catalog.load("orders"), not inside a node.

The test of whether you have done this right is simple. Can you call the function from a test with a DataFrame you build in two lines, without a file and without Kedro? If yes, it is a good node.

A typo between code and catalog

Symptom. DatasetNotFoundError, or a Pipeline input(s) ... not found in the DataCatalog message, or an output file that never materialises.

Cause. The dataset name is a plain string in two places. "clean_orders" in pipeline.py and clean_order in catalog.yml look alike to a tired human and are different to Kedro. A name missing from the catalog does not always produce an error for outputs. Kedro treats it as a memory dataset, so the run succeeds and the file you expected never exists.

Fix. Use kedro catalog describe-datasets, which groups datasets into those defined explicitly, those created by patterns and those that default to memory. Anything in the defaults group that you meant to persist is a missing or misspelled entry. A second tool is kedro viz run, where an isolated dataset or an unexpected extra one jumps out visually.

Forgetting to install dataset dependencies

Symptom. Class 'pandas.CSVDataset' not found even though you spelled it correctly, or No module named 'pandas'.

Cause. kedro-datasets is split into groups so you install only what you use. The group for the CSV dataset is pandas-csvdataset, for Parquet pandas-parquetdataset, for Excel pandas-exceldataset.

Fix. pip install "kedro-datasets[pandas-csvdataset]", with the quotes if you use zsh. Put the line in requirements.txt so a colleague does not hit the same error.

Copying a tutorial written for 0.x

Symptom. You paste a command or a snippet and get an unknown option, a missing attribute or a deprecation warning.

Cause. Kedro changed substantially at 1.0 (July 2025) and added more in the releases after it. The internet still contains plenty of 0.18 and 0.19 material.

Fix. Learn the old-to-new mapping below, and check the publication date of any tutorial before trusting it.

Committing secrets

Symptom. A key or password appears in a conf/base file, or git status shows a credentials file ready to commit.

Cause. Putting a password straight into the catalog entry to make it work quickly.

Fix. Credentials go in conf/local/credentials.yml, which the generated .gitignore excludes. The catalog refers to the block name with credentials: <block_name>. If a real secret has ever been pushed, rotate it at the provider; removing it from the latest commit does not remove it from history. In containers and CI, use environment variables: inside a credentials file you can write ${oc.env:MY_SECRET}.

Losing intermediates and then blaming Kedro

Symptom. You run kedro run --from-nodes=second_node and it fails because the first node's output does not exist.

Cause. That output was never saved, because it is a memory dataset. Kedro cannot start from the middle of a pipeline if the middle's inputs were never written.

Fix. Give intermediates that you will want to restart from a catalog entry (put them in 02_intermediate or 03_primary). Then --from-nodes, --from-inputs and --only-missing-outputs all work as you expect.

Expecting a list order

Symptom. You reorder the nodes in the list to fix a problem and nothing changes, or you put a node "before" another and expect it to run first.

Cause. Order comes from data dependencies between names.

Fix. If a node must run after another, make its input the other's output. If two nodes have no data link, and one truly must precede the other, you have a hidden dependency, and the honest fix is to make it visible as a dataset.

Duplicated node names and outputs

Symptom. Pipeline nodes must have unique names or OutputNotUniqueError.

Cause. The same pipeline was reused twice without a namespace, or two nodes write to the same dataset name.

Fix. Give distinct name= values, rename one output, or wrap a reused pipeline in a namespace. Note also that node and tag names may contain only letters, digits, hyphens, underscores and full stops.

Overriding a whole block by accident

Symptom. A setting from base seems to disappear in your local run.

Cause. The overlay environment replaces whole top-level keys by default. If conf/local/parameters.yml defines model_options: {test_size: 0.3}, the random_state from base is gone, not retained.

Fix. Repeat every sub-key you need in the overlay, or keep per-environment differences in separate top-level keys. Mid-level covers the soft merge strategy that changes this behaviour.

Mixing up what Kedro does

Symptom. You wonder why Kedro will not run your pipeline at 6 a.m. or retry a failed step.

Cause. It is an authoring framework, not a scheduler.

Fix. Hand the project to an orchestrator. The guides on Airflow and Docker cover the usual path: package the project in a container and have the orchestrator run kedro run for chosen nodes or tags.

Habits worth adopting on day one

Commit the skeleton first. git init straight after kedro new, then commit. You can always see what you changed and return to a known-good state.

Use one virtual environment per project, and pin Kedro. A line such as kedro~=1.7 in requirements.txt stops a teammate quietly running a different major version. New projects are created with a pin of that kind already.

One pipeline, one folder. Use kedro pipeline create and keep the parameters and tests next to the pipeline they belong to. A pipeline that does two jobs should be two pipelines.

Name every node. name="clean_orders_node" costs ten characters and makes logs, errors and --nodes readable. Without it Kedro generates a name from the function name plus a hash, which is unwelcoming in a failure.

Keep functions pure and small. One job per function, arguments in, value out, no I/O, no global state. A function you can explain in one sentence is a function you can test.

Everything that might change goes in parameters. Thresholds, ratios, seeds, column lists. If you catch yourself typing a number inside a function, stop and ask whether it should be a parameter.

Give persistent intermediates a catalog entry and a layer folder. Follow the numbered data/ convention so the project reads from raw to reporting. Never edit the files in 01_raw.

Use the registry and catalog commands before guessing. kedro registry list, kedro registry describe, and kedro catalog describe-datasets are cheap and tell you what Kedro thinks, rather than what you believe.

Explore in a notebook, promote to a node. Start with kedro jupyter lab, use catalog.load(...), and move anything you want to keep into nodes.py. Never let the notebook become the only place a piece of logic lives.

Write one test per node. A three-row DataFrame and an assert is enough to start. It catches the regression you would otherwise discover three steps later.

Read the last line of a traceback first. Kedro chains exceptions. The bottom says what went wrong, and the layers above say where.

Fix deprecation warnings when you see them. The singular --pipeline flag and --async are scheduled for removal in Kedro 2.0. A warning today is an error later.

Decide on telemetry deliberately. It is on by default and never collects data contents. Set KEDRO_DISABLE_TELEMETRY if your employer or university asks you to.

Practice cards

Try each card cold, with no help, then check yourself against the answer.

Card 1: the missing output. You run the pipeline successfully, but the file data/08_reporting/city_summary.csv does not exist. Answer: city_summary has no catalog entry or the entry's name is misspelled, so it was a memory dataset. Check with kedro catalog describe-datasets.

Card 2: the colon. A tutorial says kedro run --params="min_amount:50". What do you write? Answer: kedro run --params="min_amount=50"; the equals sign has been the syntax since 0.19.

Card 3: the capital S. DatasetError: Class 'pandas.CSVDataSet' not found. Two possible causes? Answer: the old spelling (it is CSVDataset) or kedro-datasets is not installed.

Card 4: the override that overreaches. base has model_options with two keys; local defines it with one. How many keys does the run see? Answer: one, because the overlay replaces the whole block by default.

Card 5: restart from the middle. The second node failed after a long first node. How do you avoid redoing the first? Answer: make sure the first node's output is in the catalog and saved, then kedro run --from-nodes=<second> or --only-missing-outputs.

Card 6: secrets. Where does a database password go, and how does the catalog use it? Answer: conf/local/credentials.yml, referenced from the entry as credentials: <block>.

Card 7: which flag. You want to run two named pipelines in one command. Answer: kedro run --pipelines a,b.

Card 8: the environment. You want a staging configuration. Answer: create conf/staging/, run kedro run --env=staging or set KEDRO_ENV=staging.

Card 9: two outputs. A node is declared outputs=["train", "test"] but the function returns one DataFrame. Answer: Kedro stops with the "returned N output(s), whereas the node definition contains M" message; return two values.

Card 10: no project found. You open a terminal in your home folder and run kedro run. Answer: it cannot find pyproject.toml; change into the project folder.

Small recipes

Start over cleanly. Delete data/02_intermediate and onwards but keep 01_raw, then run kedro run. A correct pipeline rebuilds everything from raw data. If it cannot, you have a hidden dependency.

Find what a run would touch. kedro registry describe orders lists node names; kedro run --to-nodes=<name> runs a prefix of the pipeline.

Temporarily shrink the data. Put a small sample CSV in conf/local-style overlay by changing the filepath in conf/local/catalog.yml for your machine only. The base catalog, and your teammates' runs, stay untouched. Remember that a key in the overlay replaces the whole entry of the same name, so repeat the type as well as the changed path.

Inspect a dataset quickly. In any notebook or kedro ipython session: catalog.load("clean_orders").describe().

Silence a noisy catalog log. Python logging applies: logging.getLogger("kedro.io.data_catalog").setLevel(logging.WARNING) in a notebook reduces the chatter.

Check your project before sharing. Run git status, confirm no credentials are listed, run kedro run on a fresh clone, and pytest. If all three work, your colleague's first ten minutes will be pleasant.

The short checklist

Before you call a Kedro project done, tick every line.

• Virtual environment active; kedro info shows the version you expect.
• git init done and the skeleton committed.
• No file paths, thresholds or secrets inside function bodies.
• Every function is pure and takes and returns data.
• Every node has a name.
• Every dataset you want to keep has a catalog entry with the right type spelled ...Dataset.
• kedro-datasets groups installed and listed in requirements.txt.
• Parameters live in YAML and are read with params:.
• Credentials are in conf/local/credentials.yml and not committed.
• kedro run passes from a fresh clone.
• kedro registry describe and kedro viz run show the shape you meant.
• At least one test per node passes with pytest.
• No deprecation warnings: --pipelines, --namespaces, key=value parameters.

If every box is ticked, you have used Kedro the way it was designed to be used, and the mid-level track will build on a foundation that does not need to be redone.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. It is written against Kedro 1.7, and where older tutorials disagree with that version, this guide says so and shows the current form.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a real project in a few minutes.

What happens between pressing enter and the first node

A beginner sees kedro run print a list of nodes. A mid-level engineer needs to know what happened before that list, because almost every confusing failure is a failure in that preamble: a wrong environment, a pipeline that was not registered, a setting that was silently ignored.

Kedro is a framework for authoring pipelines, not an orchestrator. It has no scheduler, no retries and no run-history database. What it does is wire your functions, your YAML and a runner together in a fixed sequence. Knowing the sequence lets you predict what a change will affect.

Bootstrap. The CLI finds the project root by looking for pyproject.toml, which is why kedro run works from any subdirectory. It reads the [tool.kedro] table, which must contain package_name, project_name and kedro_init_version. It adds src to sys.path, then imports settings.py and the pipeline registry. In code you do this yourself with bootstrap_project(path). When a project is installed as a wheel rather than run from a checkout, configure_project(package_name) does the equivalent job, and the package's __main__.py calls it.

Session. A KedroSession represents exactly one run. It captures the environment, the runtime parameters and some metadata. A second session.run() on the same session raises KedroSessionError: A run has already been completed as part of the active KedroSession. This matters the moment you work in a notebook: %reload_kedro exists precisely to give you a new session.

Context. session.load_context() returns a KedroContext, which owns the loaded configuration, the parameters and the DataCatalog. The context is where settings.py choices take effect: CONFIG_LOADER_CLASS and CONFIG_LOADER_ARGS decide how YAML is read, DATA_CATALOG_CLASS decides which catalog is built, and HOOKS decides which hook implementations are registered.

Registry. pipeline_registry.py exposes register_pipelines(), which returns a dictionary of name to Pipeline. The template calls find_pipelines(raise_errors=True), which imports every pipelines/<name>/ package that exposes create_pipeline, and sets pipelines["__default__"] = sum(pipelines.values()). kedro run with no --pipelines runs __default__. Two details are worth knowing. First, raise_errors=True is the template default from Kedro 1.2, so a pipeline with an import error now fails loudly instead of disappearing from the registry. Second, since 1.5 a run with --pipelines x imports only the modules for x, but only if your registry uses find_pipelines(). A hand-written registry that imports everything at the top forfeits that saving.

Runner. The runner receives a pipeline and a catalog. Execution order comes from data dependencies, never from the order of the list you passed to Pipeline([...]). A node becomes runnable when all its inputs exist; the runner loads them from the catalog, calls your function, and saves the outputs back through the catalog.

The preamble also has a version check. Since 1.1.1 a mismatch between kedro_init_version and the installed Kedro is fatal only when the major version differs, so a 0.19 project on Kedro 1.x stops with Your Kedro project version X does not match Kedro package version Y you are running, while a 1.3 project on Kedro 1.7 runs.

Configuration that behaves: OmegaConfigLoader in full

Configuration problems are the most common source of "works on my machine" in a Kedro team, because the failure is silent: the wrong value is used rather than an error raised. The loader is deterministic, and once you know its rules you can reason about any value.

OmegaConfigLoader is the only built-in loader. ConfigLoader and TemplatedConfigLoader are gone, so any tutorial that mentions them is out of date. It reads .yml, .yaml and .json files selected by glob patterns. The defaults look like catalog*, catalog*/** and **/catalog* for each of catalog, parameters, credentials and logging, plus globals.yml for globals.

Environments and merging

An environment is a folder under conf/. base is always loaded, and one overlay is loaded on top, local by default. You choose the overlay with --env or the KEDRO_ENV variable, and the CLI flag wins when both are set. To change the default overlay for a project, set it in settings.py:

CONFIG_LOADER_ARGS = {
    "base_env": "base",
    "default_run_env": "local",
    "config_patterns": {
        "spark": ["spark*/"],
    },
    "merge_strategy": {"parameters": "soft", "catalog": "destructive"},
}

Merging across environments is destructive by default: a top-level key in the overlay replaces the same key in base wholesale. If base/parameters.yml defines model_options with three sub-keys and staging/parameters.yml defines model_options with one, the staging run sees a dictionary with one key. Setting "parameters": "soft" switches to a recursive merge, where the overlay changes only the sub-keys it names. The accepted values are soft and destructive; anything else raises ValueError: Merging strategy X not supported.

Within a single environment the rule is stricter. Two files that define the same top-level key raise ValueError: Duplicate keys found in <file1> and <file2>: <keys>. For parameters the check happens at the sub-key level. The override across environments is logged only at DEBUG, which is why a surprising value rarely announces itself.

Globals, runtime parameters and resolvers

Interpolation uses OmegaConf syntax. ${data.size} refers to another key in the same file. ${globals:key} reads from conf/base/globals.yml, with an optional default: ${globals:key, default}. ${runtime_params:key} reads a value passed at run time, also with an optional default, and it works in every config type except globals. You can nest them, so ${runtime_params:random, ${globals:my_global_value}} falls back to a global when no parameter is passed.

Runtime parameters are the mechanism for a single catalog that serves several runs:

_bucket: ${globals:bucket}

model_input_table:
  type: pandas.ParquetDataset
  filepath: s3://${globals:bucket}/${runtime_params:run_date, latest}/model_input.parquet
  credentials: dev_s3

kedro run --pipelines=training --params="run_date=2026-09-30"

The --params syntax is key=value, split on the first =, with nested keys written with dots. Values become integers or floats where possible and strings otherwise, and keys are always strings. Quote the whole argument if a value contains spaces. The old key:value form was removed in Kedro 0.19; the official parameter-validation page still shows a colon in one example, but = is the syntax to use.

A missing runtime parameter with no default raises InterpolationResolutionError: Runtime parameter 'x' not found and no default value provided. Treat that as a feature: it is a cheap way to make a required input impossible to forget.

Every OmegaConf built-in resolver is enabled except oc.env, which is permitted only in credentials files. You can register it yourself through custom_resolvers, but the docs discourage it, because environment variables then leak into catalog and parameter files where they can change behaviour invisibly. Custom resolvers of your own are registered through the same key:

from datetime import date

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "today": lambda: date.today().isoformat(),
        "add": lambda *x: sum(x),
    },
}

One trap when you build a loader by hand in a script: a manually constructed OmegaConfigLoader does not see --params from the CLI. Pass runtime_params explicitly, or go through a session, which does it for you.

The Data Catalog, precisely

A node never does I/O. It receives Python objects and returns Python objects, and the catalog does the loading and saving. Beginners learn that the catalog is a YAML file. At this level you need to know that it is a registry with a defined resolution order, because that order explains both the convenient behaviour (unlisted datasets just work) and the dangerous behaviour (a typo in a dataset name quietly creates an in-memory dataset).

Resolution order

When the runner asks for a dataset name, DataCatalog resolves it in this order:

1. An explicit entry in the catalog YAML.
2. A dataset factory pattern that matches the name.
3. One user catch-all pattern, such as "{default_dataset}". A second catch-all raises DatasetError.
4. The default runtime pattern, which produces a MemoryDataset.

The last step is why an unlisted intermediate dataset works: Kedro creates it in memory and discards it when the run ends. It is also why a misspelled name produces no error. The typo'd output is written to memory, the consumer reads a different, unwritten name, and you get a missing-input failure somewhere downstream rather than at the source. The command kedro catalog describe-datasets groups every dataset in a pipeline as datasets, factories or defaults, so the third group is the list of things you never declared.

Dataset factories

Factories remove repetition. A quoted key containing placeholders matches dataset names like a reversed f-string:

"{name}_raw":
  type: pandas.CSVDataset
  filepath: data/01_raw/{name}.csv

"{layer}.{name}":
  type: pandas.ParquetDataset
  filepath: data/{layer}/{name}.parquet

The quotes are mandatory. Without them YAML reads {name} as a flow mapping and you get ParserError: Invalid YAML or JSON file. Two commands make factories debuggable instead of mysterious. kedro catalog list-patterns lists the patterns in priority order, and kedro catalog resolve-patterns prints the fully resolved configuration for the datasets in a pipeline. Use the second whenever a dataset is written somewhere you did not expect.

Since Kedro 1.0 the . character in a dataset name is reserved for namespaces, which interacts with the second pattern above: a dataset called primary.companies looks like a dataset companies in namespace primary. Keep layer folders out of dataset names and put them in the filepath instead.

Transcoding

Some data needs to be read as different types by different nodes: a Parquet file written by pandas and read by Spark. Transcoding declares two catalog entries for one file, with names my_df@pandas and my_df@spark, each with its own dataset type. The pipeline treats them as one dataset for dependency ordering, so the Spark node runs after the pandas node; the catalog treats them as two, so each gets the right loader. One rule catches people: a node cannot have the same dataset as input and output even in transcoded form, so f(df@spark) -> df@pandas fails at construction with A node cannot have the same inputs and outputs even if they are transcoded. Give the output its own name.

Versioning

Setting versioned: true on an entry makes every save go to a timestamped folder, <filepath>/<YYYY-MM-DDThh.mm.ss.sssZ>/<filename>, and every load read the newest one. Three consequences follow. Loads of a versioned dataset that has never been saved fail with VersionNotFoundError: Did not find any versions for <dataset>. http(s):// paths cannot be versioned and raise Versioning is not supported for HTTP protocols. And a save that collides with an existing timestamp folder raises Save path '<p>' for <dataset> must not exist if versioning is enabled.

You pin a version at run time with --load-versions=companies:2026-01-01T00.00.00.000Z, exact format required. Since the 1.3 security fix, a version string must be a single path component; a string containing path separators is rejected, which closes a path-traversal hole. In code, list_versions(full_path=True|False) on a versioned dataset (added in 1.3) lists what exists.

Kedro's versioning is deliberately simple: it is a folder convention, not a lineage system. If you need dataset versions tied to code commits, reach for DVC, and if you need time travel on tables, a table format such as Delta Lake. Kedro also documents guides for using those alongside it.

The catalog in code

The 1.x DataCatalog is dict-like. You write catalog["x"], "x" in catalog, catalog.keys() and len(catalog), and assigning catalog["x"] = data wraps the value in a MemoryDataset. The methods load(name, version=None), save(name, data), exists(name), release(name) and confirm(name) act on one dataset. filter(name_regex=None, type_regex=None, by_type=None) replaces the old list(), which was removed in 1.0. Several older official examples still call catalog.list() or io.list(); they will fail on 1.x with an AttributeError, so use filter() or keys().

Saving None raises DatasetError: Saving 'None' to a 'Dataset' is not allowed. That message appears when a node returned nothing for a declared output, and the fix is in the node, not the catalog. Since 1.3, load and save errors include the dataset name, and since 1.6 credentials and signed-URL query strings are redacted from dataset representations and error messages, so pasting an error into a ticket is safer than it was.

Validation: make bad data and bad parameters fail early

A pipeline that runs for forty minutes and then fails on a malformed column has cost forty minutes. Kedro gives you two validation layers that fail earlier, one for parameters and one for data.

Parameter validation

From Kedro 1.3, a node argument fed from params: can be annotated with a Pydantic v2 BaseModel or a dataclass. Kedro validates the parameter and instantiates the model before any node runs, and a violation raises ParameterValidationError. You need the extra kedro[pydantic].

from pydantic import BaseModel, Field
import pandas as pd


class ModelOptions(BaseModel):
    test_size: float = Field(gt=0, lt=1)
    random_state: int = 3
    features: list[str]


def split_data(data: pd.DataFrame, options: ModelOptions):
    train = data.sample(frac=1 - options.test_size, random_state=options.random_state)
    return train, data.drop(train.index)

model_options:
  test_size: 1.5
  features: [engines, passenger_capacity]

With test_size: 1.5, the run stops immediately with Parameter validation failed and a line naming model_options and the constraint. Some rules to keep in mind: Optional[Model] and Model | None are unwrapped and validated; a union of several non-None types is skipped; Pydantic v1 is not supported. Validation is scoped to the pipelines named in --pipelines since 1.4, so an unrelated broken pipeline will not block you, although an older known-limitations note in the docs still says otherwise. If two pipelines annotate the same parameter key with different types, Kedro logs a warning and the last one wins, so keep one type per key.

Dataset validation

Kedro 1.6 added a validator: key to catalog entries. DataCatalog enforces it on every load and save, and failures raise DataValidationError, a subclass of DatasetError from kedro.validation. Pandera support installs through kedro[pandera-pandas] or kedro[pandera-polars].

companies:
  type: pandas.CSVDataset
  filepath: data/01_raw/companies.csv
  validator:
    class: myproject.schemas.CompaniesSchema
    on: [load, save]
    severity: error
    options:
      lazy: true

The short form is validator: myproject.schemas.CompaniesSchema. In the long form, on defaults to both directions, severity is error or warn, and options is passed to the Pandera validator (for example lazy to collect every failure rather than stopping at the first). A custom validator is a class with a validate(self, data) method that returns the data, or a function that returns None to mean the data passes. The limits at 1.7 are one validator per dataset, Pandera pandas and Polars backends supported, and the PySpark backend still experimental.

Where validation should sit is a design decision. Validating on load of raw datasets protects you from upstream changes you do not control. Validating on save of outputs protects downstream consumers from your own bugs. Validating everything on both sides is slow and noisy; a good default is load validation on the raw layer and save validation on the final layer.

Two operational switches exist for emergencies: DATASET_VALIDATION = False in settings.py and the environment variable KEDRO_DATASET_VALIDATION=0. The functions validate_dataset(catalog, name) and validate_catalog(catalog) return a ValidationResult instead of raising, which suits a scheduled data-quality job that reports rather than fails. For richer expectations and data docs, the Great Expectations guide covers the alternative, and the Kedro docs include an integration recipe for it.

Designing pipelines that can be reused and tested

The mid-level skill in pipeline design is deciding where boundaries go. Kedro offers three ways to select a subset of a pipeline, and they answer different questions.

Tags are flat labels, and --tags selects nodes with any of the listed tags. Tags are cheap but do not scale: with thirty tags nobody knows what they mean. Slicing flags are for the developer at the terminal, not for production entry points: --from-nodes, --to-nodes, --from-inputs and --to-outputs are exactly what you want while fixing the third node of a pipeline, and exactly what you do not want hardcoded in a scheduler.

Namespaces and reuse

A namespace lets you instantiate one pipeline more than once without name collisions. It prefixes node names, and by default dataset and parameter names, with ns.:

from kedro.pipeline import Node, Pipeline


def create_pipeline(**kwargs) -> Pipeline:
    base = Pipeline(
        [
            Node(split_data, ["model_input_table", "params:model_options"],
                 ["X_train", "X_test", "y_train", "y_test"], name="split"),
            Node(train_model, ["X_train", "y_train"], "regressor", name="train"),
        ]
    )
    return Pipeline(
        base,
        namespace="region_a",
        inputs={"model_input_table"},
        parameters={"params:model_options": "params:model_options_a"},
    ) + Pipeline(
        base,
        namespace="region_b",
        inputs={"model_input_table"},
        parameters={"params:model_options": "params:model_options_b"},
    )

Here inputs={"model_input_table"} keeps that dataset un-prefixed so both copies read the same shared table, and the parameters mapping points each copy at its own parameter group. The outputs become region_a.regressor and region_b.regressor. Without a namespace, reusing the pipeline fails with Pipeline nodes must have unique names; if two nodes write the same dataset you get OutputNotUniqueError. prefix_datasets_with_namespace=False prefixes node names only, for the case where you want shared datasets by default.

Run a namespace with kedro run --namespaces=region_a; since 1.0 you can pass several, comma-separated, although one official page still says you cannot. Kedro-Viz collapses namespaces into expandable groups, which makes a large pipeline readable. One known defect at 1.7.0: selecting a parent namespace together with its child, such as a,a.b, raises a duplicate-names ValueError. The fix is in the next release, so select the parent alone for now.

Write nodes as pure functions

The property that makes a Kedro project pleasant to maintain is that nodes are pure: given the same inputs they return the same outputs, with no reads from disk, no global state and no network calls hidden inside. Everything a node needs arrives as an argument. The payoff is testing, which is the next section's subject, and a second payoff that is easy to miss: a pure node can be moved between runners, between machines and between orchestrators without change.

Practical rules that follow from this:

• Return what you declared. The number of returned values must match outputs, otherwise the save fails with The node function returned N output(s), whereas the node definition contains M output(s). A node that returns a value while declared with outputs=None triggers a warning since 1.3.
• Name your nodes. Since 1.0 the default name is the function name plus a hash suffix, which is stable but unreadable in logs. Explicit names are also required by %load_node in notebooks and by --nodes. Names and tags may contain only letters, digits, -, _ and ..

Use the capital-N Node and Pipeline classes; the lowercase node() and pipeline() wrappers still work, but the classes are the preferred constructors and their keyword-only arguments catch mistakes.

Testing Kedro projects

Testing is where pure nodes pay for themselves. Because a node is a function, a unit test is a function call: build a small DataFrame, call the function, assert on the result. No catalog, no session, no files.

import pandas as pd
import pytest

from myproject.pipelines.training.nodes import split_data, ModelOptions


def test_split_respects_test_size():
    data = pd.DataFrame({"x": range(100), "y": range(100)})
    opts = ModelOptions(test_size=0.2, features=["x"])
    train, test = split_data(data, opts)
    assert len(test) == 20
    assert set(train.index).isdisjoint(test.index)


def test_test_size_out_of_range_is_rejected():
    with pytest.raises(ValueError):
        ModelOptions(test_size=1.5, features=["x"])

Three levels of test are worth having, in order of cost.

Unit tests of nodes. Fast, numerous, and the place where you test the logic: edge cases, empty inputs, missing columns. The kedro pipeline create command scaffolds tests/pipelines/<name>/ for you when the project was created with the test tool.

Pipeline wiring tests. These check the graph, not the functions. Build the pipeline, then assert on what it says about itself: pipeline.inputs() are the free inputs that must exist in the catalog, pipeline.outputs() are what it produces, pipeline.all_outputs() includes intermediates. A wiring test catches the classic refactoring failure, where renaming a dataset in one node leaves a consumer reading a name nothing produces, which would otherwise show up as Pipeline input(s) {'x'} not found in the DataCatalog at run time.

Integration tests with an in-memory catalog. Run the real pipeline with a runner against a catalog made of MemoryDatasets seeded with tiny fixtures. This exercises the wiring, the node signatures and the outputs in seconds:

import pandas as pd
from kedro.io import DataCatalog, MemoryDataset
from kedro.runner import SequentialRunner

from myproject.pipelines.training.pipeline import create_pipeline


def test_training_pipeline_runs():
    catalog = DataCatalog(
        {
            "model_input_table": MemoryDataset(pd.DataFrame({"x": range(50), "y": range(50)})),
            "params:model_options": MemoryDataset({"test_size": 0.2, "features": ["x"]}),
        }
    )
    outputs = SequentialRunner().run(pipeline=create_pipeline(), catalog=catalog)
    assert "regressor" in outputs

Since 1.0, runner.run() returns all pipeline outputs as a dictionary of name to dataset object, so call .load() on each one you want to assert on. Pass SequentialRunner() as an instance; passing the class raises KedroSessionError: KedroSession expect an instance of Runner instead of a class in the session API.

Runners, performance and resumable runs

Kedro parallelism is per machine. There are three built-in runners, and choosing correctly is mostly about what kind of work your nodes do.

Select one with --runner=ThreadRunner, and pass constructor arguments through --runner-params=max_workers=4. Kedro 1.6 added --runner-params and deprecated --async, which is removed in 2.0, so the modern way to enable async loads and saves is --runner-params=is_async=True.

Why ParallelRunner surprises people

ParallelRunner uses a ProcessPoolExecutor, and needs a SharedMemoryDataCatalog when you build one by hand. Everything crossing a process boundary must be picklable, so three situations fail:

• Datasets holding lambdas, closures or open connections fail with AttributeError: The following datasets cannot be used with multiprocessing: [...].
• Cloud-backed fsspec datasets frequently cannot be pickled. Since 1.6 ParallelRunner warns you and suggests ThreadRunner or SequentialRunner.
• SparkDataset does not work with it at all. Spark already parallelises inside the engine, so use ThreadRunner if nodes are independent.

Node and dataset hooks do not run inside workers under ParallelRunner. A timing hook you rely on in development will silently report nothing in a parallel run, which is a common source of missing metrics. The process start method comes from the KEDRO_MP_CONTEXT variable (fork, spawn or forkserver); the defaults are fork on Linux up to Python 3.13, forkserver on Linux with 3.14 or later, and spawn on macOS and Windows. With spawn, Kedro re-runs configure_project in each worker, which is one reason a project that runs in parallel on Linux can fail on a colleague's laptop.

Not recomputing work

Three features avoid redundant computation, and they compose.

• Persist expensive intermediates by giving them a catalog entry, so a rerun can start from them.
• --only-missing-outputs skips any node whose persisted outputs all exist. It replaced the removed AbstractRunner.run_only_missing().
• Slicing with --from-nodes or --from-inputs reruns only the downstream part.

The caution with --only-missing-outputs is that "exists" means the file exists, not that it is current. After changing a node's code you must delete its outputs or run without the flag, otherwise the stale output is reused. It suits resumable runs after a crash far better than it suits incremental development.

Hooks: extend the run without editing the nodes

Hooks are pluggy-based extension points. They exist so that cross-cutting behaviour, such as timing, alerting, metrics, loading credentials or starting a Spark session, lives in one class instead of being copied into every node. A hook implementation is a class whose methods carry the @hook_impl decorator, registered in settings.py.

import logging
import time

from kedro.framework.hooks import hook_impl

logger = logging.getLogger(__name__)


class TimingHooks:
    def __init__(self):
        self._start = {}

    @hook_impl
    def before_node_run(self, node):
        self._start[node.name] = time.perf_counter()

    @hook_impl
    def after_node_run(self, node):
        elapsed = time.perf_counter() - self._start.pop(node.name)
        logger.info("node %s took %.2fs", node.name, elapsed)

    @hook_impl
    def on_node_error(self, error, node):
        logger.error("node %s failed: %s", node.name, error)

from myproject.hooks import TimingHooks

HOOKS = (TimingHooks(),)

The available specifications are:

There are also before_command_run and after_command_run for CLI commands.

The rules that catch people

Declare only the arguments you need. Pluggy lets an implementation accept any subset of a spec's arguments, which is why before_node_run(self, node) works above. Do not give those parameters default values: pluggy then passes your default rather than the value Kedro supplies, and the hook appears to run with nonsense.

Order is last-in, first-out. Plugin hooks that are auto-discovered through the kedro.hooks entry point run first, then those in settings.py. Use @hook_impl(tryfirst=True) or trylast=True when order matters. A plugin you did not write can therefore run code in your pipeline, so when you want one off, use DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",) and audit installed entry points as part of dependency review.

Hooks are not free. At DEBUG log level, pluggy traces every hook call, which is noisy and slow. Expensive work inside after_dataset_saved runs once per dataset per node.

Parallel workers skip them. As covered above, ParallelRunner does not run node or dataset hooks in its workers.

Good uses include creating a Spark session in after_context_created, loading credentials from a vault (see the Vault guide), alerting from on_pipeline_error, and recording lineage events after dataset saves. Data-changing logic belongs in a node, not a hook.

Logging and observability

Kedro logs through Python's standard logging. It looks for configuration in a fixed order: the file named by the KEDRO_LOGGING_CONFIG environment variable first, then conf/logging.yml (or .yaml) relative to the working directory, then the framework's own rich_logging.yml when Rich is installed, otherwise default_logging.yml. The automatic use of conf/logging.yml dates from 0.19.6, and although the docs still phrase it as if you must set the variable, the file is picked up by itself. Changing CONF_SOURCE does not move the logging lookup; set KEDRO_LOGGING_CONFIG instead.

The kedro new template, when created with the log tool, gives you conf/base/logging.yml with a kedro.logging.RichHandler for the console, which takes rich_tracebacks: True, and an info_file_handler writing info.log through a RotatingFileHandler. File handlers fail on read-only filesystems, such as Databricks Repos, so remove the file handler there and log to the console.

The 1.7 logging restriction

Kedro 1.7 tightened this as a security fix, and it is a breaking change for projects with custom logging classes. A logging.yml may reference only handler, filter and formatter classes from logging, logging.handlers and kedro.logging. Your own class needs its module listed in the environment variable KEDRO_LOGGING_MODULE_ALLOWLIST, a comma-separated list of module prefixes:

export KEDRO_LOGGING_MODULE_ALLOWLIST=myproject.logging
kedro run

Without it you get Module '<m>' specified as a logging class is not permitted. Only 'logging', 'kedro.logging', and modules listed in the KEDRO_LOGGING_MODULE_ALLOWLIST environment variable are allowed. The standard () factory key from dictConfig is rejected outright with The '()' key is not allowed in logging configuration as it poses a security risk; rewrite it with class:. If a JSON log formatter from a third-party library stopped working after the upgrade, this is the first place to look.

Logs, metrics and lineage

For a production pipeline you want three things, each with a natural home.

Metrics come from hooks, as above. Emit node durations and row counts to Prometheus through a push gateway or to whatever your platform uses, from after_node_run.

Experiment tracking is deliberately not built in. Kedro-Viz's own experiment tracking was removed in earlier versions, so use MLflow, through either the community kedro-mlflow plugin, which auto-registers hooks and reads conf/local/mlflow.yml, or MLflow datasets from kedro-datasets.

Lineage and visual inspection come from Kedro-Viz. kedro viz run serves on 127.0.0.1:4141, kedro viz build produces a static site and kedro viz deploy publishes it to S3, GCP or Azure. For programmatic access without running anything, kedro.inspection.get_project_snapshot(project_path, env=None, conf_source=None, runtime_params=None) returns a ProjectSnapshot that lists metadata, pipelines, datasets and parameter keys, and each node records its function name and source location. It is the right starting point for custom documentation or lineage export.

Credentials and secrets

Kedro separates configuration that is safe to commit from configuration that is not, using the environment folders. Files matching credentials* are git-ignored, and conf/local/ is ignored entirely. A catalog entry refers to a credentials block by name with credentials: dev_s3, and the block's keys are passed to the dataset.

That arrangement works on a laptop and fails in CI, where conf/local/credentials.yml does not exist. Three patterns cover the real cases.

Environment variables in the credentials file. The oc.env resolver is permitted in credentials files, and only there:

dev_s3:
  client_kwargs:
    aws_access_key_id: ${oc.env:AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${oc.env:AWS_SECRET_ACCESS_KEY}

You can generate this file in a CI step from secrets the CI system holds, or mount it into the container.

Cloud-native credential chains. AWS datasets read AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN from the environment without any credentials block. On a cloud runner with an attached role, or with gcloud auth application-default login, you need no secret files at all, and this is the better option when the platform supports it.

A vault, loaded by a hook. An after_context_created or after_catalog_created hook can fetch credentials from a secret manager at start-up and insert them into the config. The official docs include a recipe for this.

Some limits to keep in view. --conf-source pointing at remote storage authenticates only through the environment or the cloud's own mechanisms, because a credentials file cannot bootstrap its own download. And kedro package excludes conf/local/*.yml from its configuration archive, so packaged artefacts do not carry your laptop's secrets. Never pass secrets in GET /snapshot?params= on the HTTP server: query strings end up in logs.

Packaging and handing off to an orchestrator

Kedro does not schedule. The documented route to production has a consistent shape: containerise the project, push the image to a registry, and translate nodes (or groups of nodes) into tasks on a platform that does scheduling, retries and history. Each task runs kedro run with a selector, against the same image.

Package first

kedro package runs python -m build --wheel into dist/ and writes dist/conf-<package>.tar.gz, which excludes local/*.yml. On the target machine, pip install the wheel (Kedro itself need not be preinstalled, because it is a dependency of the wheel), then run python -m <package> or the console script named after the repository, from a directory that contains conf/, and data/ if needed. Alternatively run kedro run --conf-source dist/conf-<pkg>.tar.gz without extracting. The archive lets you ship the same wheel everywhere and vary only the configuration, which fits the twelve-factor pattern and the Docker image promotion you already know: one artefact, configuration chosen at run time.

Choosing the task granularity

An orchestrator's DAG comes from Pipeline.node_dependencies, and each task typically runs kedro run --nodes=<name>, --tags=<tag> or --namespaces=<ns>. The key constraint is that every dataset that crosses a task boundary must be persisted to shared storage, because a MemoryDataset exists only inside one process. Project-wide, that means all catalog entries are remote (S3, GCS, Azure Blob and so on) before you deploy. The rule for what happens to an unlisted intermediate is the same as in the catalog section: it becomes a MemoryDataset, and in a task-per-node deployment the next task then finds nothing.

How finely to split is a trade-off. One task per node gives the best retry granularity and the most container start-up overhead; one task for the whole pipeline gives no overhead and no partial retry. The middle path is grouping. Pipeline.group_nodes_by(group_by) returns a list of GroupedNodes, with strategies such as one group per node or grouping by namespace, and the kedro-airflow plugin offers --group-by memory, which merges nodes that are connected only by MemoryDatasets into one task, since those cannot be split anyway:

pip install kedro-airflow
kedro airflow create --target-dir=dags/ --env=airflow --group-by memory

That command generates an Airflow DAG file from your pipeline. The Airflow guide covers running the result, and the Kedro docs have a worked Astronomer example. Other documented targets include Kubeflow Pipelines, Prefect, Dagster (through the community kedro-dagster), Vertex AI, Amazon SageMaker, AWS Step Functions, Amazon EMR Serverless, Azure ML and Databricks. The Argo Workflows and AWS Batch guides are labelled legacy and untested, so do not start a new project from them. kedro-docker still exists, but its last release was in 2024, so most teams write their own Dockerfile and run it on Kubernetes.

Serving and inspection over HTTP

Since 1.5, kedro server start (with kedro[server] installed) runs a FastAPI application with GET /health, GET /snapshot and POST /run. Swagger UI is at /docs. The POST /run body is strict JSON where unknown fields are rejected, with fields including pipeline_names, params, tags, node_names, namespaces, runner, is_async and only_missing_outputs, and the response carries status, run_id and duration_ms. env and conf_source are fixed when the server starts.

Security and supply chain

Kedro's own security model is stated plainly in its docs: it is a code-authoring framework, not a web application, and the boundary that matters is between code and data. Project code, custom datasets, runners, hooks, plugins and resolvers are trusted, and running an untrusted Kedro project is equivalent to running untrusted Python. YAML, parameters, runtime parameters and dataset contents must remain data. Any path from data to code execution is treated as a vulnerability.

For you, the practical implications are these.

• Audit plugins. Hooks from installed plugins are auto-discovered. A dependency that registers a kedro.hooks entry point runs code in every pipeline in that environment. Review what kedro info lists, and disable what you do not want.
• Pickle is code. Pickle-based datasets execute code when loading untrusted files. kedro-datasets 9.5 and 9.6 made weights_only=True the default for PyTorch and safe_mode=True the default for TensorFlow, and added deserialisation warnings, but a pickle file from an unknown source is still a hazard.
• Pin and review. Pin kedro~=1.7 and your kedro-datasets extras. Since kedro-telemetry is a hard dependency and on by default, decide your policy: set KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK to any value to opt out, which overrides everything else, or create projects with kedro new --telemetry=no. The telemetry collects counts and types, never data contents, and the .telemetry file must not be committed.

Debugging beyond the basics

Most Kedro failures fall into a short list of shapes, and each has a first command that narrows it down.

Beyond the table, three techniques earn their keep.

Use kedro ipython. It opens IPython with catalog, context, pipelines and session already loaded. From there catalog.load("companies") gives you the actual object a node would receive, pipelines["__default__"].describe() prints the graph, and catalog.get("x") shows which dataset a name resolves to. Reproducing a node failure with the real loaded data takes a minute this way. In Jupyter, %load_node <node_name> loads a node's code into cells (the node needs a name and persisted inputs), and it remains experimental.

Read the chained exception. A DatasetError: Failed while loading data from dataset ... wraps the underlying error: a missing file, a permission problem, a missing library. The top message names the dataset; the cause is underneath it. Since 1.6 credentials are redacted from these messages.

Reproduce with the smallest slice. Run kedro run --to-nodes=<failing-node> or --from-inputs=<dataset> rather than the whole pipeline, and use --load-versions to pin a dataset to the exact version that failed.

Upgrading and staying current

Kedro follows semantic versioning, and since 1.0 only major releases are supposed to break things, although the security fixes in 1.6 and 1.7 did change behaviour. Upgrades are therefore routine, with a checklist.

1. Read RELEASE.md and the migration guide for every minor version you skip.
2. Upgrade the package, then set kedro_init_version in pyproject.toml to the new version. Only a major-version mismatch is fatal.
3. Pin kedro~=1.7 and the kedro-datasets extras you use, in requirements.txt or pyproject.toml.
4. Run the tests with deprecation warnings visible, and fix every KedroDeprecationWarning.
5. After 1.6 or 1.7, rename RUNNER_MODULES_WHITELIST to RUNNER_MODULE_ALLOWLIST and set KEDRO_LOGGING_MODULE_ALLOWLIST if you use custom logging classes.

The deprecations to hunt for are kedro run --pipeline (use --pipelines), session.run(pipeline_name=...) (use pipeline_names=[...]) and kedro run --async (use --runner-params=is_async=True). All three still work but are removed in Kedro 2.0, and passing both --pipeline and --pipelines is an error.

Putting it all together

Here is how the pieces combine in one realistic scenario: a team trains a demand model for two regions nightly, on shared data, with one deployment image.

The configuration has base for shared defaults and staging and prod overlays selected by KEDRO_ENV. Parameters use the soft merge strategy so an overlay changes one value, not a whole block. The bucket name lives in globals.yml, run-specific paths use ${runtime_params:run_date, latest}, and credentials come from the cloud role in production and ${oc.env:...} in CI.

The catalog declares every persisted dataset, uses factory patterns for the raw layer, and puts a Pandera validator: on the raw input on load and on the final forecast on save. Nothing that crosses a task boundary is left to default to memory, which kedro catalog describe-datasets proves in CI.

Tests are three layers: fast unit tests of each node, a wiring test asserting inputs() and outputs(), and one in-memory integration test per pipeline. CI also runs kedro registry list and kedro catalog resolve-patterns.

Nothing in that scenario required a clever trick. Each choice is a consequence of one fact from the sections above: execution order comes from data, the catalog resolves in a fixed order, merges are destructive by default, and only persisted datasets survive a process boundary.

Where you are now

You can predict what happens before the first node runs, and you know why a value came from the file it did. You can design a catalog with factories, versions and validators, read the resolution order when a dataset lands in the wrong place, and choose a runner with a reason. You write nodes that are pure enough to test in milliseconds, reuse pipelines with namespaces, add behaviour through hooks without touching the graph, and hand the project to an orchestrator with every cross-task dataset persisted. You know which Kedro-specific security changes to check on upgrade and which deprecations to remove before 2.0.

What Senior adds is the view from above a single project: running Kedro as a platform for several teams, where custom starters, shared dataset libraries, per-tenant configuration and upgrade policy matter; the failure modes of the HTTP server and sessions at scale; and the point where Kedro should hand a problem to something else entirely.

Sources

• Kedro documentation
• Kedro concepts
• Commands reference
• Configuration basics
• Parameters and credentials
• Templating
• The Data Catalog
• Dataset factories
• Dataset validation
• Partitioned and incremental datasets
• Nodes
• Modular pipelines
• Namespaces
• Run a pipeline
• Hooks
• Sessions
• Serving
• Inspecting a project
• Logging
• Deployment overview
• Packaging a project
• Grouping nodes for deployment
• Deploying with Airflow
• Kedro and MLflow
• Migration guide
• Security model
• Telemetry
• Experimental features
• Release notes
• Kedro on PyPI
• kedro-datasets documentation
• Kedro-Viz documentation

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than level, in about thirty-five minutes. Fast review first, scenario and trade-off questions after. Senior reviews all three. Everything here is written against Kedro 1.7.

Where this picks up

Foundations, in one screen

Kedro is an open-source Python framework for maintainable data and machine-learning code. You write ordinary functions, wrap each in a Node that names its inputs and outputs, collect nodes in a Pipeline, and declare every dataset in a YAML catalog. The runner works out the order from data dependencies and executes it.

The sentence that separates a mid-level answer from a beginner one: Kedro authors pipelines, and something else schedules them. It has no scheduler, no retries, no run-history UI and no backfill. Those come from an orchestrator such as Airflow or Kubeflow, which is why so many interview questions are really about the hand-off.

Scenario questions

"A teammate's run uses a different value than yours for the same parameter. What do you check?"

A strong answer goes in a fixed order. First, which environment each person ran: --env on the command line beats KEDRO_ENV, and with neither, the default overlay is local. Second, the merge strategy. The default is destructive, so a top-level key in an overlay replaces the same key in base entirely. If the overlay defines model_options with one sub-key, the run sees only that one, and the other sub-keys are gone rather than inherited. The fix is either to set "parameters": "soft" in CONFIG_LOADER_ARGS or to repeat the whole block. Third, runtime parameters: --params key=value overrides values that use ${runtime_params:key}. Fourth, leftovers in conf/local, which is git-ignored and so differs per machine.

What a weak answer misses: the override across environments is logged only at DEBUG, so nothing warns you, and a duplicate of the same key in two files of one environment is a hard error, not a silent override.

"The pipeline ran, but the output is not where I expected. Why?"

The catalog has a resolution order: an explicit entry first, then a dataset factory pattern, then at most one user catch-all, then the default runtime pattern, which produces a MemoryDataset. So the answer is usually that a factory pattern matched before you thought it would, or the name matched a catch-all. The diagnostic is kedro catalog resolve-patterns, which prints the resolved configuration, and kedro catalog list-patterns, which shows pattern priority. kedro catalog describe-datasets groups every dataset as datasets, factories or defaults, and anything in defaults was never declared.

A strong answer also raises the opposite failure: a typo in an output name does not error, because Kedro quietly creates an in-memory dataset. The producer writes to memory, the consumer reads a different name, and the failure shows up as a missing input downstream.

"Everything works locally but the second task in our Airflow DAG cannot find its input."

Datasets that cross a task boundary must be persisted to storage both tasks can reach. A MemoryDataset lives inside one process, and an undeclared intermediate becomes one. Locally there is one process, so it works. The fix is to add a catalog entry with a remote path, for example S3 or Blob storage, for every dataset that one task writes and another reads. Run kedro catalog describe-datasets and look at the defaults group to find them.

Extend the answer with granularity. One task per node gives fine retries and high container overhead. Grouping nodes reduces overhead; kedro airflow create --group-by memory merges nodes that are connected only by memory datasets into a single task, because those cannot be separated anyway.

"--runner=ParallelRunner fails, but SequentialRunner works. What is going on?"

Processes need to pickle what they exchange. Typical culprits are datasets holding lambdas or open connections, cloud-backed fsspec datasets, and Spark datasets, which produce AttributeError: The following datasets cannot be used with multiprocessing. Since 1.6, Kedro warns about unpicklable cloud datasets and suggests ThreadRunner or SequentialRunner. On macOS and Windows the start method is spawn (set with KEDRO_MP_CONTEXT), which demands that every dataset and node function be picklable and re-runs configure_project in each worker, so a project that works on Linux with fork can fail on a laptop.

The best answer then chooses the runner by workload rather than trial and error: ThreadRunner for I/O-bound work, Spark and cloud datasets; ParallelRunner for CPU-bound picklable work; SequentialRunner when the graph is mostly linear. And it mentions a side effect: node and dataset hooks do not run in ParallelRunner workers, so timing hooks silently report nothing.

"After upgrading Kedro, kedro run fails with a logging error."

This is the 1.7 security change. A logging.yml may only reference handler, filter and formatter classes from logging, logging.handlers and kedro.logging. A custom class needs its module listed in the environment variable KEDRO_LOGGING_MODULE_ALLOWLIST, a comma-separated list of module prefixes. The dictConfig () factory key is rejected entirely with a message saying it poses a security risk, and must be rewritten with class:. A related silent failure from 1.6: RUNNER_MODULES_WHITELIST was renamed RUNNER_MODULE_ALLOWLIST, and the old name is ignored without any message.

"The pipeline takes forty minutes and fails on a bad column at the end. How do you prevent that?"

Validate early, at two layers. For parameters, annotate the node argument with a Pydantic v2 model or a dataclass (install kedro[pydantic]), and Kedro validates and instantiates it before any node runs, raising ParameterValidationError. For data, add a validator: to the catalog entry (1.6 onwards), backed by Pandera through kedro[pandera-pandas] or kedro[pandera-polars], and failures raise DataValidationError on load or save. Choose where to validate: load validation on raw datasets guards against upstream changes; save validation on final outputs guards consumers against your own bugs. Mention the emergency switches (DATASET_VALIDATION = False, KEDRO_DATASET_VALIDATION=0) and severity: warn for a soft rollout.

"How would you test a Kedro project?"

Three layers. Unit tests call node functions directly with small DataFrames, since pure functions need no catalog or session. Wiring tests assert on pipeline.inputs(), pipeline.outputs() and all_outputs(), catching renamed datasets. Integration tests run the pipeline with SequentialRunner().run(pipeline=..., catalog=...) against a catalog of MemoryDatasets seeded with tiny fixtures; in 1.x the return value is a dictionary of every pipeline output as dataset objects, on which you call .load(). Add one full-session smoke test in CI, and cheap gates: kedro registry list and kedro catalog resolve-patterns.

Trade-off questions

Tags, namespaces or slicing?

They answer different questions. Tags are flat labels on nodes, good for concerns such as preprocessing, and --tags selects nodes with any listed tag; they become unmanageable in numbers. Namespaces make several independent copies of one pipeline, prefixing node names and, by default, dataset and parameter names; they suit the same model trained for several regions, and Kedro-Viz collapses them into groups. Slicing (--from-nodes, --to-nodes, --from-inputs, --to-outputs) is for a developer at a terminal, re-running part of a graph while debugging. A good answer says: slicing belongs in a developer's shell, not in a scheduler; namespaces are for reuse; tags are for loose grouping.

Soft or destructive config merging?

Destructive, the default, is predictable and explicit: an overlay replaces a whole block, so you can read one file to know the value. The cost is that a small override needs the whole block repeated, and it is easy to lose sub-keys. Soft merging lets an overlay change one sub-key, which is convenient for parameters but means the effective value is now spread over two files. A sensible position is soft for parameters and destructive for catalog, where a half-merged dataset definition is a bug waiting to happen.

Factories or explicit entries?

Factories cut repetition dramatically in projects with dozens of similarly shaped datasets, and they make adding a dataset a zero-YAML change. They also hide where a dataset is defined and can match a name you did not intend. The middle path is to use factories for genuinely uniform groups such as a raw layer, keep important datasets explicit, never use a catch-all in production unless you want undeclared datasets persisted, and use kedro catalog resolve-patterns in CI to check the result.

Kedro's versioning or DVC or a table format?

Kedro's versioned: true is a timestamped-folder convention: simple, automatic and useful for rolling back a dataset. It is not tied to code commits, and it does not work on HTTP paths. DVC links data versions to git commits and is the choice when reproducing "the exact data for that commit" matters. A table format such as Delta Lake gives time travel within a table. A good answer names the requirement first (rollback, reproducibility, audit) and then picks.

One task per node or one task per pipeline?

Per node: finest retries, best parallelism in the orchestrator, highest start-up overhead, and every dataset must be persisted. Per pipeline: no overhead, no partial retry, long tasks. Grouped tasks, by namespace or by memory-connected nodes, sit in between. The deciding facts are how expensive a failure is, how long container start takes, and how much of the graph is genuinely parallel.

Should we use the built-in HTTP server to trigger runs?

Only as an internal trigger. kedro server start (needs kedro[server]) offers GET /health, GET /snapshot and POST /run, but has no authentication, no queue, no run history, and no per-request isolation, because concurrent runs share one session in a thread pool. KedroServiceSession is flagged as in active development. For anything that needs queuing, auth or history, put an orchestrator or an API gateway in front, or use the orchestrator's own trigger.

Fast-review table

Short questions with model answers

What is the difference between KedroSession and KedroContext? The session manages the lifecycle of one run and stores its metadata; the context holds the loaded config, parameters and catalog. You get the context from session.load_context().

What does bootstrap_project do? It reads pyproject.toml, puts the project on sys.path and calls configure_project. You use it whenever you drive a project from a script rather than the CLI.

Why does kedro run work from a subdirectory? The CLI searches upward for pyproject.toml, which marks the project root.

How do you pin a dataset version for a run? --load-versions=companies:2026-01-01T00.00.00.000Z, in exactly the timestamp format; other shapes raise an error, and version strings with path separators are rejected since the 1.3 security fix.

What is transcoding? Two catalog entries (x@pandas, x@spark) on one file, with different dataset types. For ordering they are one dataset; for loading they are two. A node cannot take and return the same dataset even transcoded.

What does --only-missing-outputs do, and what is its catch? It skips nodes whose persisted outputs all exist. It cannot tell a stale output from a current one, so after changing code you must delete outputs or drop the flag.

How do you keep secrets out of the repository? conf/local/ and credentials* files are git-ignored; in CI use ${oc.env:VAR} inside the credentials file or the cloud's own credential chain; for vaults, load secrets in an after_context_created hook. kedro package excludes local/*.yml.

What are the two ways a hook can go wrong silently? Giving hook parameters default values, which makes pluggy pass the default instead of Kedro's value, and relying on hooks under ParallelRunner, where node and dataset hooks do not run in workers.

How do you opt out of telemetry? Set KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK to any value, which overrides everything else, or create a project with kedro new --telemetry=no.

What is the LLM context node? A node type, stable since 1.7, that composes an LLMContext of an LLM object, prompt datasets and tools. It does not call the model itself.

What a strong answer covers that a weak one misses

• It names the mechanism. "The catalog falls back to a MemoryDataset" rather than "Kedro handles it".
• It separates authoring from running. It says what Kedro does and what the orchestrator does.
• It gives the failure mode. Every feature has a way to go wrong silently: a destructive merge, a factory match, a typo'd output, a missing hook in a parallel worker, a renamed setting that is ignored.
• It uses the current form. --pipelines, DataCatalog, filter(), --runner-params, RUNNER_MODULE_ALLOWLIST. An answer built on KedroDataCatalog or ConfigLoader dates itself to 0.18.
• It picks by requirement. Runner, merge strategy, versioning and task granularity all get a "depends on" followed by the thing it depends on.
• It mentions the neighbours. MLflow for tracking, Airflow for scheduling, Docker for packaging, Great Expectations as a validation alternative.

Mock scenario to practise aloud

You join a team whose Kedro project runs nightly in Airflow. Runs take ninety minutes, fail about once a week, and nobody knows which node is slow. Talk through what you do in the first two days.

A good outline: day one, read the project without running it, using kedro registry describe, kedro catalog describe-datasets and Kedro-Viz to see the graph's shape, and check kedro info for plugins. Look for undeclared intermediates and for one giant task. Add a timing hook on before_node_run and after_node_run and ship the durations to your metrics system, so the next night's run tells you where the time goes. Day two, act on the data: if one node dominates, profile it; if the graph is wide and I/O-bound, try ThreadRunner; if the weekly failure is always a bad upstream file, add a validator: on load of that raw dataset so the failure is immediate and named instead of forty minutes in. Persist the slow intermediate and use --from-nodes for a developer to iterate. Finally, read the upgrade notes against the pinned version, and list deprecation warnings.

The habit an interviewer is listening for is measure first, then change one thing.

Sixty-second self-test

Answer aloud without looking up.

1. Name the four steps of the catalog's resolution order.
2. What does destructive merging do, and what is the soft alternative?
3. Why does a typo in an output name not raise an error at the producer?
4. What must be true of every dataset that crosses an orchestrator task boundary?
5. Which runner for cloud datasets, and why does ParallelRunner fail on them?
6. What does a KedroSession allow, and what happens on the second run()?
7. Name three deprecated things in 1.7 and their replacements.
8. What changed for logging.yml in 1.7?
9. Where does oc.env work by default?
10. Which commands tell you, without running anything, which pipelines exist and how a dataset name resolves?

Senior takes every one of these further, with an ownership dimension. Running Kedro as a platform for several teams with shared starters and dataset libraries. The trust model behind the code-versus-data boundary, plugin auditing and pickle risk. Multi-tenancy through environments and per-tenant configuration sources, and why the HTTP server is not a tenant boundary. Scaling limits and when the work belongs to Spark or an orchestrator's fan-out. Upgrade and migration policy across many projects, and when to pick something other than Kedro altogether.

Part two of three. At this level your pipelines run, so the problems change character. They are no longer "why won't this start" but "why did staging use the wrong value", "why does this only fail in the orchestrator", and "why did the upgrade break logging". Start with the error table, then the practices ranked by leverage, the practice cards and the recipes. Everything here is written against Kedro 1.7.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most, in order of leverage

1. Persist every dataset that crosses a process boundary

An undeclared intermediate is a MemoryDataset. Locally that is convenient; in a task-per-node deployment it is the most common first failure. Before any deployment, run kedro catalog describe-datasets and move the defaults group that matters into the catalog with a remote path. It also buys resumable runs and lets a teammate debug from the dataset instead of rerunning everything.

2. Make bad input fail before the expensive part

Validation is the cheapest insurance in the project. Type the parameters with Pydantic models (kedro[pydantic]), so a bad value stops the run before node one, and add a validator: to raw datasets on load (kedro[pandera-pandas]), so a schema change upstream fails immediately with the dataset's name. One validator per dataset, severity: warn while rolling out.

3. Keep nodes pure, named and small

Pure functions test in milliseconds and move between runners and orchestrators unchanged. Give every node a name=, because the default is the function name plus a hash suffix, which is unreadable in logs and breaks --nodes habits. Return exactly what you declared; a mismatch fails at save time.

4. Test at three levels, and gate CI on the cheap ones

Unit-test node functions; assert the graph's inputs() and outputs(); run an in-memory integration test with SequentialRunner and MemoryDataset fixtures. In CI also run kedro registry list and kedro catalog resolve-patterns. All of it runs in seconds, and it catches the failures that otherwise appear at 3 a.m.

5. Choose the merge strategy on purpose

Decide once, in settings.py, and write it down: soft for parameters, destructive for catalog is a sound default. Whichever you pick, a new teammate should be able to predict an overlay's effect from one sentence.

6. Put configuration variation in environments and runtime parameters, not in code

Environment folders for things that differ by deployment (buckets, endpoints); ${runtime_params:...} for things that differ by run (a date, a region); globals.yml for values used in several places. Branches on an environment name inside a node are a smell.

7. Use hooks for cross-cutting behaviour, and keep them boring

Timing, alerting, metrics, credentials from a vault, Spark session creation. Do not use hooks to change data. Declare only the arguments you use and never give them defaults.

8. Pick the runner by workload

ThreadRunner for I/O-bound work, Spark and cloud datasets; ParallelRunner for CPU-bound picklable work; SequentialRunner otherwise. Look at the graph in Kedro-Viz first. A linear chain has nothing to parallelise.

9. Read deprecation warnings as a to-do list

--pipeline, session.run(pipeline_name=) and --async are removed in 2.0 and still work today. Fixing them now costs minutes. Pin kedro~=1.7, and read RELEASE.md for every minor version you skip.

10. Decide your plugin and telemetry policy

Plugin hooks are auto-discovered and run in your pipeline. Review kedro info, and use DISABLE_HOOKS_FOR_PLUGINS for any you do not want. Telemetry is on by default; set KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK in CI and images if policy requires it.

Good versus bad

Practice cards

Operational recipes

Recipe: a fast CI gate

kedro registry list
kedro catalog resolve-patterns > /dev/null
kedro run --pipelines=__default__ --env=test

The first checks that pipelines are discovered, the second that every pattern resolves, the third is a smoke run against fixtures in conf/test. Keep fixtures tiny so the run takes seconds.

Recipe: re-run part of a pipeline

Recipe: run from a config file

A run: file keeps long commands reviewable and repeatable:

run:
  pipelines: [training]
  runner: ThreadRunner
  runner_params:
    max_workers: 4
  tags: []

kedro run --config conf/run_nightly.yml

Keys use underscores, and any flag you pass on the command line overrides the file.

Recipe: credentials in CI without committing them

dev_s3:
  client_kwargs:
    aws_access_key_id: ${oc.env:AWS_ACCESS_KEY_ID}
    aws_secret_access_key: ${oc.env:AWS_SECRET_ACCESS_KEY}

Generate this file in the CI step from the secret store, export the two variables, and run. On a cloud runner with an attached role, skip the block entirely: AWS datasets read AWS_* variables and the role's chain without one. Confirm with git status --ignored that nothing under conf/local/ is tracked.

Recipe: upgrade from 1.6 to 1.7 safely

1. Bump the pin to kedro~=1.7 and set kedro_init_version to match.
2. grep -rn "RUNNER_MODULES_WHITELIST" src/ and rename to RUNNER_MODULE_ALLOWLIST.
3. Look in conf/**/logging.yml for classes outside logging, logging.handlers and kedro.logging, and for () keys. Rewrite () with class: and export KEDRO_LOGGING_MODULE_ALLOWLIST=<your.module> where needed, in every environment including CI and the container image.
4. Run the tests and read every warning. llm_context_node users: the experimental warning is gone, so remove any filter you added for it.
5. Run the smoke pipeline in staging before production.

Recipe: package and run without Kedro pre-installed

kedro package
pip install dist/*.whl
python -m my_project --pipelines=__default__ --conf-source dist/conf-my_project.tar.gz

The wheel pulls Kedro in as a dependency. The tarball carries base and other non-local configuration, leaving local/*.yml out, so the target machine supplies its own secrets.

Recipe: hand off to Airflow

pip install kedro-airflow
kedro airflow create --target-dir=dags/ --env=airflow --group-by memory

Before deploying, check that every dataset crossing a task boundary is persisted and that the image contains the same package version the DAG expects. The Airflow guide covers running the generated file, and the Docker guide covers building the image it runs.

Recipe: quiet the noisy logs

In conf/base/logging.yml, keep the console handler at INFO and add a logger entry for kedro.io.data_catalog at WARNING. In code, logging.getLogger("kedro.io.data_catalog").setLevel(logging.WARNING) has the same effect for one script. On read-only filesystems such as Databricks Repos, remove the file handler.

Reading a failing run quickly

When a run fails, resist opening the code first. Read the last line of the traceback for the exception type, then the dataset or node name in the message, since 1.3 load and save errors include the dataset name. A DatasetError that says Failed while loading data from dataset wraps the real cause, so scroll to the chained exception: a missing file, a permission problem or a missing library. A ValueError naming duplicate keys or a missing pipeline is a configuration or registry problem, and kedro registry list or kedro catalog resolve-patterns will show it in seconds. Keep a note of the exact message, because the wording is stable across versions and searchable in the release notes. Only when the failure is inside your own function is it time to reproduce it: open kedro ipython, load the node's inputs with catalog.load(...), and call the function by hand. To go further, run kedro run --to-nodes=<name> so the smallest slice reproduces the failure, and pin inputs with --load-versions when the data changed between runs.

Habits to adopt this month

• Look at the resolved state before theorising. resolve-patterns, describe-datasets, registry describe, and kedro ipython with catalog.load(...) answer most questions faster than reading code.
• Keep one effective source of truth per value. If you cannot say which file a parameter came from, simplify the environments.
• Name things. Nodes, pipelines and namespaces get explicit names; dataset names follow one convention.
• Write the failure down. When a run fails for a reason that cost you an hour, add a wiring test, a validator, or a CI command that would have caught it.
• Read the release notes. Five minutes against RELEASE.md at each minor version beats an afternoon of debugging an ignored setting.
• Connect the neighbours. Track experiments in MLflow, check data with validators or Great Expectations, and ship with Docker and GitHub Actions.

Checklist before you ship a pipeline

• Every dataset that crosses a task boundary has a catalog entry with a remote path.
• kedro catalog describe-datasets shows no unintended defaults.
• Raw datasets have validators on load; parameters are typed.
• Every node has a name=, and returns exactly what it declares.
• Unit, wiring and integration tests pass; kedro registry list is clean.
• The merge strategy is set deliberately and documented.
• No secrets in conf/base; credentials come from ${oc.env:...} or a cloud role.
• The runner matches the workload, and hooks you rely on do not depend on ParallelRunner.
• kedro~=1.7 is pinned, kedro_init_version matches, and no deprecation warnings remain.
• The 1.6 and 1.7 renames are done: RUNNER_MODULE_ALLOWLIST and KEDRO_LOGGING_MODULE_ALLOWLIST.
• Telemetry and plugin policy are decided and applied in CI and the image.

Senior builds on all of this with platform-level traps: starters and shared dataset libraries for several teams, tenancy through environments and per-tenant configuration sources, the limits of the HTTP server and sessions under load, plugin and pickle supply-chain risk, and upgrade policy across many projects at once.

This is part three of three. The earlier levels taught you to write Kedro projects and to run them well. This one is about owning Kedro as a platform: the template, the settings, the plugins and the deployment path that twenty other people's pipelines inherit. The questions change. Not "why does my node fail" but "what does a compromised config file reach", "which component breaks when we have four hundred pipelines", and "how do we upgrade without stopping every team".

Everything here is checked against Kedro 1.7.0, released on 28 September 2026. Kedro moves quickly for a framework that promises SemVer, and two of its last three minor releases changed behaviour for security reasons, so version discipline is a recurring theme.

Where this picks up

I am starting with the architecture, because every incident in this guide is easier to reason about once you can name the objects involved and who creates them.

Architecture and failure modes

Kedro is a code-authoring framework. It is not a scheduler, not a distributed executor and not a service. That sentence decides most platform questions, so hold it firmly: there is no built-in scheduling, retry policy, SLA tracking, run history or backfill. When a team asks the platform for those, the answer is always an orchestrator.

What Kedro does provide is a set of objects with a clear life cycle. At start-up, bootstrap_project reads pyproject.toml, puts the project on sys.path and calls configure_project, which loads settings.py and the pipeline registry. A KedroSession then manages one run: it builds a KedroContext, which holds the loaded config, the catalog, the parameters and the project path. The session runs a pipeline through a runner, the runner calls nodes, and nodes never touch I/O because the catalog does it for them. Hooks fire at every boundary.

Knowing this chain lets you predict the failure modes. They fall into three groups.

Silent failures are the expensive ones. A renamed setting that is ignored, a misspelt dataset name that quietly becomes a MemoryDataset, a hook that never fires in a ParallelRunner worker. Kedro treats any dataset name missing from the catalog as an in-memory dataset, which is wonderful for intermediate values and treacherous for outputs: a typo in an output name means the result is computed, held in memory, discarded at the end of the run, and reported as success. Only the orchestrator-level symptom, "the downstream task cannot find its input", reveals it. The fix is a platform rule, not a developer habit: in deployed runs, fail if a pipeline's declared outputs resolve to a default pattern. You can implement that with an after_catalog_created hook that inspects catalog.describe_datasets style information, or more simply by checking in CI that kedro catalog describe-datasets puts no terminal output under defaults.

Loud failures are cheap by comparison: a missing catalog entry raises ValueError: Pipeline input(s) {'x'} not found in the DataCatalog before any node executes, and duplicate config keys raise ValueError: Duplicate keys found in <f1> and <f2>. These stop the run early, which is what you want.

Lifecycle failures come from the one-run-per-session rule. A second session.run() on the same session raises KedroSessionError: A run has already been completed as part of the active KedroSession. This is fine for a batch job and awkward for a service, which is why KedroServiceSession exists; it is covered in the serving section.

One more architectural fact matters for platform work: the pipeline is resolved from data dependencies, not from list order. Two nodes have an edge between them only because one's output name is the other's input name. That makes the DAG easy to derive (Pipeline.node_dependencies) and therefore easy to translate to an orchestrator, but it also means a renamed dataset silently rewires the graph. Treat dataset names as a public interface between nodes, reviewed like an API.

The next question is scale: once the architecture is clear, which part gives out first?

Scaling: which component breaks first

Kedro's parallelism lives inside one machine. ParallelRunner uses processes, ThreadRunner uses threads, and SequentialRunner is the default. There is no cluster mode. Anything bigger than a machine is delegated either to a data engine (Spark, Dask, Ibis, Polars, all reachable through kedro-datasets) or to an orchestrator that fans tasks out across machines. A platform owner therefore answers scale questions by asking which layer is growing.

Project size breaks first, long before data size. A monorepo with hundreds of pipelines pays at start-up: every kedro run imports the registry. Kedro 1.5 added selective pipeline loading, where --pipelines x imports only the modules for x, but it works only if register_pipelines() calls find_pipelines(). A hand-written registry that imports every pipeline eagerly defeats it without any warning. Keep the template's find_pipelines(raise_errors=True) call; the raise_errors=True part matters because it turns an import error in one pipeline into a visible failure rather than a pipeline that quietly vanishes from the registry.

ParallelRunner is the next limit, and it is a sharp one. Its restrictions:

• Datasets and node functions must be picklable. Lambdas, closures, open clients and many cloud fsspec datasets are not. Since 1.6 the runner warns when a cloud-backed dataset cannot be pickled and suggests ThreadRunner or SequentialRunner.
• Node and dataset hooks do not run in its workers. If your platform's audit or metrics hooks use before_node_run, they go dark under ParallelRunner. Pipeline-level hooks still run in the parent process.
• SparkDataset does not work with it. Use ThreadRunner for Spark and for I/O-bound cloud work.
• The multiprocessing start method comes from KEDRO_MP_CONTEXT. The defaults are fork on Linux up to Python 3.13, forkserver on Linux from Python 3.14, and spawn on macOS and Windows. A pipeline tested on a Linux CI runner with fork can fail on a developer's Mac with spawn, where Kedro re-runs configure_project in each worker and everything must pickle.
• It needs a SharedMemoryDataCatalog when you build a catalog by hand.

The honest platform guidance is that ParallelRunner is a convenience for CPU-bound, picklable work on one big machine, and that anything you would want to run on a schedule should be parallelised by the orchestrator instead.

Resumability is how you avoid paying twice. kedro run --only-missing-outputs skips nodes whose persisted outputs all exist, and --from-nodes restarts from a chosen point. Both depend on intermediates being persisted in the catalog, which is another reason the platform template should persist by default rather than rely on MemoryDataset.

Scaling questions are really questions about a boundary. The next section is about the boundary that matters most to a platform owner: what Kedro trusts.

The trust model

Kedro's official security model, added to the documentation in 1.4, opens with a sentence worth memorising: Kedro is a code authoring framework, not a web application. The boundary it draws is between code and data, and its golden rule is that code is responsibility while data becoming code is a vulnerability.

In practice this splits everything you run into two piles.

Trusted, and therefore your responsibility: project source, custom datasets, custom runners, hooks, plugins, custom config resolvers, and anything installed into the environment. Running an untrusted Kedro project is equivalent to running untrusted Python, because that is what it is. A platform that lets a stranger submit a project repository to run is a remote-code-execution service by design, and the isolation has to come from containers, network policy and IAM, not from Kedro.

Data, which must never execute: YAML configuration, parameter values, runtime parameters, and dataset contents. A path from any of these to code execution is a Kedro vulnerability and is reported through the project's SECURITY.md. The recent hardening history shows where those paths were found:

• 1.3 fixed path traversal through version strings and remote code execution through the logging configuration.
• 1.6 stopped a catalog type: from being chosen through ${runtime_params:...} over the HTTP server's POST /run (the error is InterpolationResolutionError, naming the dataset and the disallowed type), redacted credentials and signed-URL query strings from dataset representations, errors and HTTP responses, and renamed the runner allowlist setting.
• 1.7 restricted conf/logging.yml to handler, filter and formatter classes from logging, logging.handlers and kedro.logging, with extra module prefixes admitted through the environment variable KEDRO_LOGGING_MODULE_ALLOWLIST, and rejected the () factory key outright.

The pattern is consistent: Kedro closes routes by which a string in a file chooses which code runs. As the platform owner you should apply the same test to your own extensions. If you write a custom resolver or a hook that imports a class named in config, you have rebuilt the vulnerability the maintainers just removed.

Pickle-based datasets are the standing exception. Loading a pickle executes code. The kedro-datasets 9.5 and 9.6 releases made safer defaults (weights_only=True for PyTorch, safe_mode=True for TensorFlow) and added deserialisation warnings, but a pickle file from an untrusted source remains executable. The rule for the platform: datasets in a shared bucket that several teams write to must not be pickle-backed if another team reads them. Use Parquet, CSV or another data format at team boundaries.

Plugin hooks auto-register. Any installed package that declares a kedro.hooks entry point has its hooks added to every project in that environment. This is deliberate and convenient, and it is also a supply-chain channel: a compromised plugin runs inside every pipeline. Two controls exist. DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",) in settings.py turns off auto-registration for named plugins, and kedro info lists installed plugins and their entry points so you can audit them. The platform should run kedro info in CI and compare its output to an approved list.

The trust model tells you what to protect. The next section covers the thing most worth protecting, credentials, and the chain that delivers your code.

Secrets and the supply chain

Kedro separates credentials from configuration by convention: files matching credentials* live in conf/local/, which is git-ignored, and a catalog entry refers to a block by name with credentials: dev_s3. That convention protects a laptop. It does not protect a fleet, because in containers and CI there is no conf/local/.

The supported patterns for deployed runs are:

1. Environment variables through the credentials file. The oc.env resolver is enabled only in credentials files, so conf/local/credentials.yml can contain ${oc.env:AWS_ACCESS_KEY_ID}. You can register oc.env for other config types through custom_resolvers, but the docs discourage it, because it lets any config file read any environment variable.
2. Cloud-native credential chains. AWS datasets can rely on AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN; Google and Azure have equivalents such as application default credentials and managed identity. Prefer this wherever the platform runs on a cloud with workload identity, because no secret exists to leak. On Kubernetes that means a service account mapped to a cloud role, which is where Kubernetes and Terraform meet this guide.
3. A vault, loaded by a hook. The documentation has a recipe for using after_context_created or after_catalog_created to fetch external credentials and inject them. This is the route for Vault or a cloud secret manager, and it keeps secrets out of files entirely.

Two boundaries deserve a sentence each. Remote --conf-source can only authenticate through the environment or cloud mechanisms, because a credentials.yml cannot bootstrap its own download. And do not pass secrets through GET /snapshot?params=: query strings end up in access logs.

Leakage has a second path, which is error output. Before 1.6, a failed load could print a dataset representation containing a credential. Since 1.6, credentials and signed-URL query strings are redacted in dataset __repr__, DatasetError messages and HTTP responses. That is a fix worth forcing through: any fleet below 1.6 should be treated as capable of leaking credentials into its log aggregator, and the logs searched accordingly.

The supply chain has four links in a Kedro deployment, and each has a control.

kedro package excludes local/*.yml from the conf archive, which is the right default. The .telemetry file must also not be committed, and telemetry itself is on by default. It sends a random UUID, the masked CLI command, a hashed project ID, versions, the operating system, counts of datasets, nodes and pipelines, and dataset types, never data contents. For a regulated Gulf or Egyptian employer that restricts outbound connections, set KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK in the base image so no project has to remember, since that environment variable overrides every other setting. kedro new --telemetry=no is the per-project equivalent.

Secrets and code are one half of the estate. The other half is configuration, which at platform scale becomes both your tenancy mechanism and your largest source of drift.

Configuration at scale and multi-tenancy

Kedro has no concept of a tenant. Nothing in the framework distinguishes team A's project from team B's, or one customer's data from another's. Tenancy is something you build from four pieces Kedro does provide, and being precise about what each gives you prevents false confidence.

Configuration environments are folders under conf/. base is the foundation, local is the default overlay, and --env or KEDRO_ENV picks another (the CLI flag wins). You could create conf/tenant_a/ and conf/tenant_b/. By default the overlay is merged destructively: a top-level key in the overlay replaces the whole key from base. Setting merge_strategy to soft for a config type merges at the sub-key level instead, which suits parameters and hurts catalogs, where you rarely want half of a dataset definition inherited.

Remote config with --conf-source. The option accepts a directory, a .tar.gz or .zip archive, or an s3://, gs://, abfs:// or https:// location. Per-tenant config in s3://bucket/tenantA/conf/ is an approach the official documentation itself mentions. Compressed archives cannot be read directly from cloud storage, so pick a directory layout there. This separates config deployment from code deployment, which is usually what a platform wants, but it also creates the drift problem: the same image can now behave differently depending on which config it is pointed at. Log the conf_source and the image digest together on every run.

Namespaces let you instantiate one pipeline several times, prefixing node names, dataset names and parameters with ds_2. and so on. A namespaced pipeline can keep some inputs un-prefixed and remap parameters. It is how you run "the same pipeline per region" without copying code. Run one with kedro run --namespaces=ds_2. One 1.7.0-specific trap: selecting a parent namespace together with its child, for example a,a.b, raised a duplicate-names ValueError, and the fix is in the next release. Until you are on it, select one level at a time.

Runtime parameters with ${runtime_params:key} template values into config at run time, and they work in every config type except globals. They are the right tool for a tenant identifier in a path. They are the wrong tool for anything that selects code, and the 1.6 change that rejects type: via runtime params over HTTP says so explicitly.

What Kedro does not give you is isolation. A conf/tenant_a/ folder does not stop a run with --env=tenant_b from reading tenant B's credentials if both are present in the same container. Real isolation is the deployment manager's job in the official model: separate buckets, separate IAM roles, separate containers or namespaces, separate service accounts. The safe design is one deployment per tenant, each with only its own credentials mounted and its own cloud role, with Kedro's environments used only for non-sensitive variation.

The duplicate-key rule is the quiet governance tool here. Within one environment, the same top-level key in two files raises ValueError: Duplicate keys found in <file1> and <file2>: <keys>, and for parameters the check goes to sub-key level. Keys beginning with _ are hidden: never reported as duplicates, never returned. That is why YAML anchors and catalog template values must start with an underscore. A platform convention worth adopting is that shared templates in conf/base use _-prefixed keys, so teams can reuse them without colliding.

Configuration decides what a run does. The next layer decides whether the data going through it is correct, which is where the platform can enforce contracts for other teams.

Data contracts: validation, parameters and versioning

A platform owner does not write the pipelines, so the owner needs ways to enforce quality from outside. Kedro offers three, at three points in the data's journey.

Parameter validation (since 1.3) checks configuration. A node argument fed from params:x and annotated with a Pydantic v2 model or a dataclass is validated and instantiated before any node runs. A failure raises ParameterValidationError naming the parameter and the model. It needs kedro[pydantic] (Pydantic v1 is not supported). Two details matter for a fleet. Since 1.4 validation is scoped to the pipelines named in --pipelines, so the official limitation text about all pipelines is out of date. And if two pipelines declare conflicting types for the same parameter key, Kedro logs a warning and the last one wins. A platform lint that greps for one parameter key with two annotations is cheap insurance.

Dataset validation (since 1.6) checks data. A catalog entry takes validator:, and DataCatalog enforces it on every load and save. The short form names a class; the long form adds on, severity, enabled, skip_load_after_save and options:

companies:
  type: pandas.ParquetDataset
  filepath: data/02_intermediate/companies.parquet
  validator:
    class: my_project.schemas.CompaniesSchema
    on: [load, save]
    severity: error
    options: {lazy: true}

Failure raises DataValidationError, a subclass of DatasetError, from kedro.validation. Pandera support comes from kedro[pandera-pandas] or kedro[pandera-polars]; the PySpark backend is experimental. There is one validator per dataset today. A custom validator is a class with validate(self, data) that must return the data, or a function returning None to mean pass. validate_dataset(catalog, name) and validate_catalog(catalog) return a ValidationResult instead of raising, which suits a scheduled contract check that reports rather than halts.

The governance lever is severity: warn. Roll a new contract out as a warning for a sprint, read the warnings, then flip it to error. The emergency switch, for a production incident caused by an over-strict schema, is DATASET_VALIDATION = False in settings.py or KEDRO_DATASET_VALIDATION=0 in the environment. Decide in advance who may pull it, because it disables every contract at once.

Versioning (versioned: true) protects reproducibility. A versioned save writes to <filepath>/<timestamp>/<filename>, and --load-versions=ds:2026-01-01T00.00.00.000Z pins a load. The timestamp format must be exact; a bad value is refused by a check added in the 1.3 security fix, which requires a version string to be a single path component. Versioning is not supported for HTTP(S) paths. Versions pile up, so a retention rule belongs on the bucket, not in Kedro, and list_versions() (1.3) lets a clean-up job enumerate them. For table-level versioning consider Delta Lake, and for file versioning across repositories DVC.

How do these relate to other tools? Kedro's dataset validation is the in-process check. Great Expectations is the heavier, report-oriented option, and the Kedro docs include an integration recipe. For drift on production data after the run, look at Evidently; Kedro's validators judge shape, not distribution.

Contracts are enforced from outside the node code. The next section covers the mechanisms by which a platform extends Kedro itself, and where those extensions hurt.

Extending Kedro: hooks, datasets, runners and plugins

Kedro's extension points are the platform's policy layer. Used well, they let you enforce standards without asking twenty teams to write the same boilerplate. Used carelessly, they become the place where a platform's bugs hide.

Hooks are pluggy-based. You write a class with @hook_impl methods and register it in settings.py with HOOKS = (ProjectHooks(),). The specs cover the whole life cycle: after_context_created, after_catalog_created, before_pipeline_run, after_pipeline_run, on_pipeline_error, before_node_run, after_node_run, on_node_error, and the four dataset hooks for load and save. Command-line hooks, before_command_run and after_command_run, exist for plugins. An implementation declares only the arguments it needs, which is pluggy's opt-in style.

Four sharp edges, each of which has cost someone a day:

1. Do not give hook parameters default values. Pluggy then passes the default instead of Kedro's value, and the hook appears to run with empty data.
2. Order is LIFO. Auto-discovered plugin hooks run first, then those in settings.py. Use @hook_impl(tryfirst=True) or trylast=True when order matters, for example when an audit hook must see the final state.
3. ParallelRunner does not run node or dataset hooks in workers. A metrics hook that works in development is silent in a parallel production run.
4. At DEBUG level pluggy traces every hook call, which is noisy and slow. Do not leave DEBUG on for a platform that has many hooks.

What belongs in a platform hook? Timing and memory metrics around nodes, alerts from on_node_error and on_pipeline_error, lineage or data-quality events from the dataset hooks, credential loading from a vault, and the catalog policy check described earlier. What does not belong there is business logic, because hooks are invisible to the people reading the pipeline, and invisible behaviour is the hardest kind to debug.

Custom datasets subclass AbstractDataset[InputType, OutputType] and implement load, save and _describe, with _exists and _release optional. For versioning, extend AbstractVersionedDataset and use _get_load_path() and _get_save_path(). A catalog entry references the class by dotted path. Since 1.6, dataset representations are redacted, but only for what Kedro knows is sensitive: make sure your _describe does not return a raw credential, because that is the string that ends up in error messages.

Custom runners subclass kedro.runner.AbstractRunner, implement _get_executor(max_workers) and _run(pipeline, catalog, hook_manager=None, run_id=None), and are selected with --runner=<package>.runner.MyRunner. Constructor arguments come from --runner-params. Over the HTTP server a runner is admitted only if it sits in kedro.runner, in the project package or in a module listed in RUNNER_MODULE_ALLOWLIST.

Plugins are packages with entry points: kedro.project_commands, kedro.global_commands, kedro.hooks, kedro.init, kedro.line_magic and kedro.starters. A plugin command overrides a built-in command with the same name, which is powerful and dangerous in equal measure: a plugin can change what kedro run does. A kedro.starters entry point, using KedroStarterSpec(alias, template_path, directory=None), is how a company distributes its own project template. That is usually the best platform investment of all, because the starter carries your settings.py, your hooks, your logging configuration and your CI files into every new project.

Extensions shape how one project behaves. Getting many projects from a laptop to production is a different problem: deployment.

From laptop to orchestrator

The documented path from a working project to a production schedule has five steps, and a platform owner should turn each into a paved road.

1. Containerise the project. A Dockerfile that installs the wheel or the project requirements and sets the entry point. The kedro-docker plugin can scaffold one, but its dormant status means many teams write their own. The Docker guide covers the hardening.
2. Push the image to a registry, tagged with an immutable identifier.
3. Translate nodes into orchestrator tasks. The DAG comes from Pipeline.node_dependencies. Each task runs kedro run --nodes=<name>, --tags=<tag> or --namespaces=<ns>, always from the same image.
4. Make every catalog entry remote. A MemoryDataset cannot cross a task boundary, because each task is a separate process, often on a separate machine. This is the most common first-deployment failure: a pipeline that works on a laptop because intermediates lived in memory and fails in Airflow because the second task cannot find them.
5. Optionally publish a custom starter so the scaffolding is reused.

The official guides cover Airflow through kedro-airflow and Astronomer, Kubeflow, Prefect, Dagster through the community kedro-dagster, Databricks with Spark Connect, Unity Catalog and Databricks Asset Bundles, SageMaker, Vertex AI, Azure ML, AWS Step Functions, EMR Serverless and Dask. The Argo Workflows and AWS Batch guides are labelled legacy and untested, so do not build a platform on them without your own testing. Single-machine options remain: a container, a wheel, or a git clone plus kedro run.

Node grouping is the cost and latency lever. One task per node is simple and expensive: each task pays container start-up and image pull. Pipeline.group_nodes_by(group_by) returns a list of GroupedNodes, with strategies including one group per node and grouping by namespace. kedro airflow create --target-dir=dags/ --env=airflow --group-by memory merges nodes connected only by MemoryDatasets into one task, which is both faster and correct, since memory-linked nodes must share a process anyway. The official page that says you cannot run more than one namespace at once is out of date: since 1.0, --namespaces=a,b works.

The packaging choice deserves a decision. kedro package writes a wheel into dist/ plus dist/conf-<package>.tar.gz. On the target, pip install the wheel, then either run python -m <package> from a directory holding conf/, or keep the config archive intact and run kedro run --conf-source dist/conf-<package>.tar.gz. Kedro itself need not be preinstalled separately from the wheel. For an orchestrator the container wins, for a single server the wheel is lighter, and for a notebook-heavy team a git clone is honestly fine.

Orchestrators run jobs. Some platforms also want Kedro to answer requests, and that is what the next section examines.

Serving: sessions and the HTTP server

Kedro 1.4 and 1.5 added a way to run pipelines on request rather than on a schedule. Two pieces matter: a session type for repeated runs, and a built-in server.

KedroSession manages one run, and a second run() raises KedroSessionError. KedroServiceSession (1.4) is long-lived and supports many runs, each with its own runtime_params:

from pathlib import Path
from kedro.framework.session import KedroServiceSession
from kedro.framework.startup import bootstrap_project
from kedro.utils import find_kedro_project

root = find_kedro_project(Path.cwd())
bootstrap_project(Path(root))
svc = KedroServiceSession.create(project_path=root, env="prod", serving_mode=True)
svc.run(runtime_params={"k": 2})
svc.close()

Passing serving_mode=True (1.6) preloads all pipelines, which makes concurrent runs thread-safe. The class is not decorated as experimental, but its documentation says it is in active development, so breaking changes are possible and you should isolate it behind your own wrapper. You can make it the default with the SESSION_CLASS setting.

The built-in server, installed with kedro[server] and started with kedro server start (defaults --host 127.0.0.1, --port 8000), is a FastAPI and uvicorn application with three endpoints: GET /health, GET /snapshot and POST /run. POST /run takes a strict JSON body, and unknown fields are rejected. The accepted fields are pipeline_names, params, tags, node_names, from_nodes, to_nodes, from_inputs, to_outputs, namespaces, load_versions, runner, is_async and only_missing_outputs. It returns status, run_id and duration_ms, plus an error object on failure. The environment and config source are fixed when the server starts. GET /snapshot always returns HTTP 200, so a failure appears as "status": "failure" in the body, and a health probe that only checks the status code will not notice a broken project. Swagger lives at /docs.

Now the part a platform owner must say out loud. The server is minimal. It has no authentication, no queue, no asynchronous jobs, no run history and no per-request isolation. Concurrent /run calls share one session in a thread pool. Do not expose it publicly, do not use it as a multi-tenant service, and put it behind your own gateway that handles identity, rate limits and audit. If you need to extend it, create_http_server(project_path=..., env=...) returns a FastAPI app you can add routes and middleware to; the environment variables KEDRO_PROJECT_PATH, KEDRO_SERVER_ENV and KEDRO_SERVER_CONF_SOURCE configure it. For high-volume model serving this is the wrong tool: use FastAPI directly or a serving framework, and use Kedro to produce the artifact.

For inspection without running anything, kedro.inspection.get_project_snapshot(project_path, env=None, conf_source=None, runtime_params=None) returns a ProjectSnapshot with metadata, pipelines, datasets and parameter keys, and each node carries its function name and source location. From 1.7, GET /snapshot?params=k=v,k2=v2 resolves ${runtime_params:...}. This is an excellent foundation for a platform catalogue that lists every team's pipelines, but remember the secrets rule: query strings are logged.

Serving and scheduling both produce runs. The next section is about changing Kedro under a fleet that is already running.

Upgrades and migrations

Since 1.0, Kedro promises that only major releases break compatibility. That promise is true for APIs and less true for security fixes, which is why 1.6 and 1.7 each changed behaviour in a minor release. A platform owner should treat "minor" as "read the release notes", not "safe".

The version check. pyproject.toml carries kedro_init_version in [tool.kedro]. Since 1.1.1, a mismatch is fatal only when the major version differs, producing Your Kedro project version X does not match Kedro package version Y you are running. A 0.19 project on Kedro 1.x hits it, and the fix is the migration, not editing the number. For 1.x to 1.x upgrades the number is informational, so your fleet tooling should compare it to the installed version itself.

The 0.19 to 1.0 migration is the big one, and old tutorials still teach 0.19 idioms. The changes a platform must handle:

• KedroDataCatalog became DataCatalog; delete any DATA_CATALOG_CLASS = KedroDataCatalog line from settings.py.
• Removed catalog methods: _get_dataset() (use get()), add_all(), add_feed_dict() (assign with catalog["x"] = data), list() (use filter() or keys()), shallow_copy().
• CLI: kedro catalog create was removed; list, rank and resolve became describe-datasets, list-patterns and resolve-patterns; --namespace became --namespaces; kedro micropkg and micro-packaging were removed.
• Renames: extra_params to runtime_params, session_id to run_id in runners and hooks, ModularPipelineError to PipelineError, and the modular_pipeline module folded into kedro.pipeline.
• Behaviour: default node names now include a hash suffix, and run() always returns all pipeline outputs as a dictionary of dataset objects.

Custom hooks that still use session_id or extra_params fail on the first run, and the platform's hook library is the first place to fix.

The 1.x deprecation calendar has three items flagged for removal in 2.0: kedro run --async, kedro run --pipeline (singular) and session.run(pipeline_name=...). Each emits KedroDeprecationWarning today. A CI job that runs the test suite with warnings promoted to errors finds every use in the fleet in one pass. After 1.6 or 1.7, also rename RUNNER_MODULES_WHITELIST and set KEDRO_LOGGING_MODULE_ALLOWLIST if you use custom logging classes.

A fleet upgrade procedure that works:

1. Upgrade the platform starter and base image first, in a branch, and run the starter's own tests.
2. Pick three pilot projects of different shapes (a Spark one, a parallel one, one with custom datasets) and upgrade them.
3. Run each with -W error::DeprecationWarning style strictness in CI to surface warnings early.
4. Publish a short migration note that lists only the changes that affected the pilots.
5. Roll the rest with an automated pull request, and track the fleet-wide version in one place.

Upgrades are planned events. Incidents are not, so the next section prepares for them.

Governance and incident playbooks

Governance for a Kedro platform is mostly a set of things a pull request must not do and a few things that must be true of every deployed run. Keep it short enough that people follow it.

Standing rules worth enforcing in CI:

• No credentials* content outside conf/local/, and none in conf/base.
• No pickle-backed datasets at team boundaries.
• No MemoryDataset intermediates in orchestrated pipelines: kedro catalog describe-datasets shows nothing under defaults for deployed outputs.
• No use of deprecated flags or of --async.
• find_pipelines(raise_errors=True) remains in the registry.
• Every dataset that crosses a team boundary has a validator.

Playbook: a run succeeded but downstream data is missing. Suspect the silent MemoryDataset. Run kedro catalog describe-datasets -p <pipeline> and kedro catalog resolve-patterns, and look for the expected output under defaults or a factory pattern that did not match. Check for a typo in the dataset name, a factory key missing its quotes (which becomes a YAML ParserError), or a second catch-all pattern (which raises DatasetError).

Playbook: everything failed after a minor upgrade. Read RELEASE.md for that version first. On 1.7 look for The '()' key is not allowed in logging configuration or Module '<m>' specified as a logging class is not permitted, then set KEDRO_LOGGING_MODULE_ALLOWLIST or remove the factory key. On 1.6 look for a custom runner rejected over HTTP because the old allowlist name was ignored. Roll back by pinning the previous minor in the image, which is why images are immutable and tagged.

Playbook: the parallel run hangs or errors in production only. Check KEDRO_MP_CONTEXT and the platform's operating system: AttributeError: The following datasets cannot be used with multiprocessing means unpicklable datasets. Switch to ThreadRunner or the orchestrator's fan-out. Confirm that missing hook output is the known worker gap, not a lost log line.

The blameless-review question for every incident is the same: which of the standing rules would have caught this, and is it automated yet?

Governance covers what you forbid. A good platform owner also knows when the right answer is to use something else.

Where Kedro stops

Choosing Kedro should be a decision you can defend, including the cases where you would not choose it.

It is not an orchestrator. If the core need is scheduling, retries, backfills, SLAs and a run-history interface, use Airflow, Dagster or Prefect, and use Kedro inside them for the structure of the code. These are complements: the official deployment guides exist because Kedro expects to be handed to one.

It is not a distributed compute engine. Parallelism is one machine. Data that does not fit needs Spark, Dask or a warehouse, with Kedro describing the flow. For Spark-heavy work note the special cases: SparkDataset does not work with ParallelRunner, and kedro-datasets 9.6 raised the minimum PySpark to 3.3 for the Spark extras.

It is not a model registry or experiment tracker. Use MLflow or a similar tool, and Feast if feature serving is the problem. Kedro organises the code that calls them.

It is not a low-latency serving layer. The HTTP server has no authentication, queue or isolation. For online inference use a serving framework.

With the boundaries clear, the remaining question is how to run all of this as a service to other teams.

Running Kedro as a platform

A platform is what other teams can use without talking to you. For Kedro, that means four deliverables and a few habits.

The starter. Publish a company starter through the kedro.starters entry point or a Git repository that kedro new --starter can consume. It carries the settings.py with validated loader arguments, the hook library, the logging configuration with only allowed classes, the Dockerfile, the CI definition (GitHub Actions or your CI of choice), the pre-commit rules and the catalog conventions. Pin its kedro_version to the supported minor. Kedro itself refuses project names that shadow the standard library or a keyword, such as email or json, so your naming guidance can lean on that.

The base image. One image that holds Python, the pinned Kedro and dataset extras, the telemetry opt-out and the approved plugins, rebuilt on a schedule. Teams add their wheel on top. This gives you a single place to apply a security fix.

The deployment templates. For each supported orchestrator, a template that turns a pipeline into tasks using the same image, with node grouping chosen deliberately. Your platform promises that a project built from the starter deploys with a documented command, and that is the real product.

The contract library. Shared Pandera schemas, Pydantic parameter models and custom datasets for your organisation's common shapes. These are where platform ownership pays off most, because a fix in one package reaches every team on its next upgrade.

Cost deserves a line. Kedro is Apache 2.0 and free; the money goes on compute and storage. Persist intermediates and use --only-missing-outputs so reruns skip finished work, use tags and --from-nodes to rerun only a slice, and put a lifecycle rule on buckets that hold versioned datasets, since versions never expire by themselves.

Habits that make it sustainable:

• A support boundary. Write down what the platform guarantees (the starter, the image, the supported orchestrators, security fixes within a stated time) and what teams own (their pipeline code and data).
• A deprecation policy. Announce removals a version ahead, and use the warnings Kedro itself emits as the trigger.
• Golden-path metrics. Track how many projects are on the supported minor, how many have validators, how many use find_pipelines, and how long an upgrade takes.
• An escape hatch. Teams will need a custom runner or dataset. Give them a documented path and a review, rather than letting them fork the template.
• Region awareness. For employers in the Gulf or Egypt with data-residency rules, bind each deployment to buckets and compute in the approved region, and make the region part of the deployment template rather than a per-team decision. Config per tenant in a region-specific bucket is the clean way to do it.
• AI assistants. The kedro-skills package and the documentation's pages on agent skills and MCP give coding assistants project conventions. If your teams use assistants, encode your starter's rules there, then review what they generate against the checklist below.

The test of the platform is boring: a new team can go from kedro new --starter <yours> to a scheduled, monitored pipeline without a meeting.

The review checklist

Use this for pull requests to the platform and for design documents from teams.

Project shape

• Does register_pipelines() call find_pipelines(raise_errors=True)?
• Is each pipeline in its own folder with create_pipeline, and are node names unique or namespaced?
• Are dataset names treated as an interface, with no dots invented outside namespaces?

Data and config

• Is every output, and every intermediate that crosses a task boundary, in the catalog?
• Does kedro catalog describe-datasets show nothing under defaults for deployed outputs?
• Do datasets that cross a team boundary have a validator, introduced at severity: warn first?
• Are parameters typed with Pydantic models or dataclasses?
• Are there duplicate keys or anchors not prefixed with _?

Security

• No secrets under conf/base; credentials come from environment, workload identity or a vault hook.
• No pickle datasets shared across teams; no untrusted files loaded through pickle.
• No custom resolver, hook or dataset that imports a class named in configuration.
• Plugin list matches the approved baseline from kedro info.
• Telemetry opt-out is set in the image.
• The HTTP server, if used, sits behind a gateway with authentication.

Execution

• Runner chosen by workload, with the ParallelRunner restrictions checked.
• No --async, --pipeline or pipeline_name= left.
• Orchestrator tasks use the same image and grouped nodes where sensible.

Upgrade and operations

• kedro~=1.7 and dataset extras pinned; kedro_init_version updated.
• RUNNER_MODULE_ALLOWLIST spelled correctly; logging classes allowlisted.
• Logs go to stdout in a structured format; run start and end events carry the image digest and config source.
• There is a written rollback path.

Where the series leaves you

You started by writing a node and a catalog entry. You now know how the session and context are built, which runner to choose and why, what Kedro trusts and what it refuses to, how to keep secrets and plugins from becoming your supply chain, how to enforce contracts on other teams' data, how to get from a laptop to an orchestrator, how to upgrade a fleet, and when to choose another tool.

Three habits carry furthest. Read the release notes before every upgrade, including minor ones. Make silent failures loud in CI rather than in production. And keep the platform small: a starter, an image, deployment templates and a contract library, each owned, tested and documented.

The next step is practical. Pick the checklist above, run it against your two most important pipelines this week, and turn every failed item into an automated check.

Sources

• Kedro documentation
• Security model
• Telemetry
• Migration guide
• Experimental APIs
• Configuration basics
• Parameters and credentials
• Templating
• Data Catalog
• Dataset validation
• Namespaces
• Run a pipeline
• Hooks
• Sessions
• Serving
• Inspecting a project
• Logging
• Deployment overview
• Distributed deployment
• Nodes grouping
• Package a project
• Airflow deployment
• Kedro release notes
• Kedro security policy
• kedro on PyPI
• kedro-datasets documentation
• Kedro-Viz documentation

Part three of three. A senior Kedro interview is rarely about syntax. It tests whether you can design a platform around a framework that deliberately does little, defend the boundaries you draw, and respond calmly when a run that reported success produced nothing. About forty minutes of reading. System-design prompts first, then incidents, then the leadership conversation.

The version throughout is Kedro 1.7, released on 28 September 2026. Saying the version out loud early in an answer signals that you know the behaviour changed recently.

The thirty-second answer

Kedro is a Python framework for writing data and ML pipelines as pure functions wired by data dependencies, with all I/O and configuration pulled out into a catalog and layered YAML. It is a code-authoring framework, not an orchestrator: scheduling, retries and run history come from Airflow, Dagster, Prefect or a cloud service, and heavy compute comes from a data engine. As a platform owner, I provide the starter, the base image, the deployment templates and the data contracts, and I treat the boundary between code and data as the security model.

If you can say that cleanly, every follow-up is an elaboration.

System design: a Kedro platform for twenty teams

The prompt. "Your company has twenty data science teams. Design a Kedro-based platform so they ship pipelines to production without each inventing their own structure."

A strong answer, in the order you would say it:

1. Clarify first. How many pipelines, what data sizes, which orchestrator exists, which cloud and region, any data-residency rules, and who is on call. A platform designed before these are known is a guess.
2. State the division of labour. Kedro describes the work. The orchestrator schedules it. A data engine does heavy computation. A tracker records experiments. I own the conventions that join them.
3. Deliverables. A company starter distributed through a kedro.starters entry point or a Git repository. A base image with pinned Kedro and dataset extras. Deployment templates per orchestrator. A library of shared contracts: Pandera schemas, Pydantic parameter models, and custom datasets.
4. Execution model. One image per project. Each orchestrator task runs kedro run with --nodes, --tags or --namespaces from that image. Every catalog entry that crosses a task boundary is remote, because a MemoryDataset cannot. Nodes are grouped so memory-linked nodes share a task.
5. Configuration and isolation. Base and environment folders for non-sensitive variation, remote --conf-source for per-tenant config, and separate deployments with separate cloud roles for real isolation. Kedro does not provide tenancy.
6. Guardrails in CI. No secrets in conf/base, no pickle datasets across teams, kedro catalog describe-datasets showing no deployed outputs under defaults, plugin list matching an approved baseline from kedro info, no deprecated flags.
7. Operations. Structured logs to stdout, run start and end events carrying image digest and config source, alerts from on_pipeline_error, and a nightly Kedro-Viz build so teams can see the graph.
8. Upgrades. Pin kedro~=1.7, upgrade the starter and image first, pilot three projects, then roll the rest with automated pull requests.

What separates strong from weak. A weak answer describes Kedro's features in the order of the docs. A strong answer starts with what Kedro refuses to do, names the platform deliverables, and says where isolation comes from. Saying "the ParallelRunner does not run node hooks in its workers, so my audit hooks must not rely on it" shows you have operated the thing.

System design: turning a pipeline into orchestrated tasks

The prompt. "A team has a Kedro pipeline that works on a laptop. Get it onto Airflow."

A strong answer.

• Containerise the project and push an immutably tagged image. The Docker guide covers hardening.
• Derive the DAG from Pipeline.node_dependencies or use kedro airflow create --target-dir=dags/ --env=airflow, which generates the DAG definition.
• Decide the grain. One task per node is simple and pays container start-up every time. --group-by memory merges nodes connected only by memory datasets into one task, which is both cheaper and necessary, since such nodes must share a process.
• Move every intermediate that crosses a task boundary into the catalog with a remote path. The first failure almost everyone hits is that the second task cannot find an in-memory dataset produced by the first.
• Put credentials in the orchestrator's secret mechanism or workload identity, never in the image.
• Add the airflow configuration environment for orchestrator-specific paths and parameters.
• Decide retry behaviour at the orchestrator, and make nodes idempotent so a retry is safe. --only-missing-outputs helps a rerun skip finished work.

Where candidates go wrong. They forget that kedro run --nodes=x in a new container has no memory of the previous task, or they choose one task per node on a pipeline with two hundred small nodes and then complain about scheduling overhead.

If the orchestrator is not Airflow. The same shape applies to Kubeflow, Prefect, Dagster and the cloud services. The official deployment guides exist for each, and note that the Argo Workflows and AWS Batch guides are labelled legacy and untested. Say so rather than recommend them.

System design: pipelines on request

The prompt. "Product wants to trigger a Kedro pipeline from an API call with per-request parameters. How?"

A strong answer. First ask whether it is a batch job or a latency-sensitive request. If the work takes minutes, this is a job submission, so use the orchestrator's API to trigger a run with parameters, and Kedro's --params or runtime_params carry the values. If a synchronous call is really needed, there is the built-in server: kedro server start with the kedro[server] extra exposes POST /run, GET /health and GET /snapshot, and KedroServiceSession supports multiple runs with their own runtime_params.

Then say what the server is not. It has no authentication, no queue, no asynchronous jobs, no run history and no per-request isolation, because concurrent runs share one session in a thread pool. So it goes behind a gateway that supplies identity, rate limits and audit, it is not exposed publicly, and it is not used as a multi-tenant service. KedroServiceSession is documented as under active development, so wrap it. For low-latency model inference, the answer is a serving layer such as FastAPI or a dedicated model server, with Kedro producing the model artifact.

A detail that impresses. The server rejects a catalog type: chosen through ${runtime_params:...}, because letting a request pick a dataset class would turn a request parameter into code execution. Mentioning it shows you understand the code-versus-data boundary.

System design: multi-tenancy

The prompt. "Design a Kedro deployment serving several customers whose data must never mix."

A strong answer. Start with honesty: Kedro has no tenants. Then build isolation in layers that Kedro does not own.

1. Deployment per tenant, each with its own cloud role, its own buckets and its own secrets mounted, so a run cannot read another tenant's data even with the wrong --env.
2. Configuration per tenant in a remote --conf-source, such as a per-tenant bucket prefix, versioned separately from code. Log the conf source and image digest on every run.
3. Namespaces if one codebase runs the same pipeline for several regions or business units inside one trust boundary.
4. Runtime parameters for non-sensitive values like a tenant identifier in a path.
5. Audit: a hook that records which tenant, which image and which configuration produced each output.

Then the trap question: "Could you just use conf/tenant_a and conf/tenant_b in one image?" The answer is that it is configuration convenience, not isolation: anyone who can run the image with another --env reads the other tenant's config. For regulated customers in Gulf states or Egypt, add data residency: the deployment, buckets and compute for a tenant live in the approved region.

Design scenario: choosing a runner

The prompt. "A pipeline is slow. The team wants to switch to ParallelRunner. Do you agree?"

A reasoned answer. Find out what is slow first. If one node dominates, parallelism across nodes will not help. If the work is CPU-bound across independent branches and everything pickles, ParallelRunner is a reasonable choice on one large machine. If the work is I/O-bound, or it uses Spark or cloud filesystem datasets, ThreadRunner is the right fit, and SparkDataset does not work with ParallelRunner at all.

Then the restrictions: datasets and functions must be picklable, node and dataset hooks do not run in the workers, the start method depends on the operating system and Python version through KEDRO_MP_CONTEXT (so a Linux fork success can fail on a Mac with spawn), and cloud fsspec datasets often cannot be pickled. Since 1.6 Kedro warns when that is the case. Finally, past one machine the answer changes entirely: fan out in the orchestrator or push computation into a data engine. Runner flags are passed as --runner=ParallelRunner --runner-params=max_workers=4, and the old --async flag is deprecated in favour of --runner-params=is_async=True.

Incident: the run succeeded but the data is missing

Symptom. A pipeline reports success. The downstream task finds no input.

How to reason. Kedro treats any dataset name absent from the catalog as an in-memory dataset. A typo in an output name means the result was computed, held in memory, and discarded at the end of the run. No error is raised.

Steps.

1. Run kedro catalog describe-datasets -p <pipeline> and look for the expected output under defaults.
2. Run kedro catalog resolve-patterns to see whether a factory pattern matched or failed to.
3. Check for a factory key missing its quotes, which shows up as a YAML ParserError, and for a second catch-all pattern, which raises DatasetError.
4. Fix the name, rerun with --only-missing-outputs if earlier work is persisted.

The prevention an interviewer wants to hear. A CI check that fails when a deployed pipeline's outputs resolve to defaults. "We trained people not to make typos" is the weak answer.

Incident: a minor upgrade broke every run

Symptom. After moving to 1.7, runs fail at start-up with The '()' key is not allowed in logging configuration as it poses a security risk, or Module '<m>' specified as a logging class is not permitted.

Reasoning. This is a deliberate security fix, not a regression. In 1.7, conf/logging.yml may only reference classes from logging, logging.handlers and kedro.logging. The () factory key is rejected outright. Custom classes need their module prefix in KEDRO_LOGGING_MODULE_ALLOWLIST.

Steps. Roll back by pinning the previous image if production is down. Then fix forward: replace () with class: and set the environment variable in the image. Check the 1.6 rename as well: RUNNER_MODULES_WHITELIST became RUNNER_MODULE_ALLOWLIST, and the old name is silently ignored, so a custom runner used over HTTP starts failing without any hint.

The lesson to state. "Minor" does not mean "safe to bump blindly". Kedro promises only major releases break APIs, but security fixes can change behaviour in a minor release, so the platform reads the release notes and pilots upgrades.

Incident: works in CI, fails on a colleague's laptop

Symptom. ParallelRunner passes in Linux CI and fails on macOS with AttributeError: The following datasets cannot be used with multiprocessing, or a pipeline errors with Package name not found. Make sure you have configured the project using 'bootstrap_project'.

Reasoning. macOS and Windows default to spawn, which requires everything to pickle and re-runs configure_project in each worker. Linux CI up to Python 3.13 defaults to fork, which shares memory and hides the problem. Set KEDRO_MP_CONTEXT=spawn in CI to reproduce it on Linux, or avoid ParallelRunner for that workload. The bootstrap error is the same cause from another angle: a worker or a script that has not called bootstrap_project or configure_project.

Incident: a credential appeared in logs

Response order. Rotate the credential first, because logs are copied and indexed. Then establish the Kedro version. Before 1.6, credentials and signed-URL query strings were not redacted from dataset representations and error messages; since 1.6 they are. Search the log store for the pattern, upgrade, and confirm no custom dataset's _describe returns a raw secret. Finally, check how the secret got into the dataset configuration, because the durable fix is to move it to an environment variable in the credentials file, workload identity or a vault hook.

Incident: a contract blocks production

Symptom. DataValidationError: Validation failed for dataset '<ds>' on load after an upstream team changed a column.

Reasoning. The validator worked. Fix the data first. If business impact is severe and the data is acceptable, lower that dataset's severity to warn by config change with an incident record. Do not reach for KEDRO_DATASET_VALIDATION=0, which disables every contract at once. Afterwards, agree a schema-change process with the upstream team, and roll out new contracts as warnings first.

Trade-off questions

Kedro or a general orchestrator's own pipeline definitions? Kedro gives code structure, a catalog and configuration that survive a change of orchestrator. The orchestrator gives scheduling and run history. They are complements. The trade-off is one more framework to learn against portability and testability.

Kedro or ZenML or Metaflow? ZenML and Metaflow bundle more of the execution and tracking story. Kedro is more neutral about where pipelines run, and its catalog and config system are its distinctive strengths. Choose by where your team's pain is: structure and portability points to Kedro, integrated execution points elsewhere.

Persist everything or keep intermediates in memory? Persisting costs storage and I/O time but enables --only-missing-outputs, reruns from a node, debugging and orchestrator hand-off. Memory is fast and cannot cross tasks. The platform default should be persisted, with memory as a deliberate optimisation.

One large project or many small ones? A monorepo shares conventions and makes refactoring easy, and pays at start-up time unless find_pipelines and selective loading are used. Many repositories isolate teams and multiply upgrade work. Pick by ownership boundaries, not by taste.

Pandera validators or Great Expectations? The built-in validator: runs in-process on every load and save and suits schema contracts. Great Expectations suits richer, report-oriented checks. They can coexist.

Hooks or node code for cross-cutting behaviour? Hooks keep policy out of every node, at the price of invisibility and the ParallelRunner gap. Use hooks for metrics, audit and credentials, and never for business logic.

Arguing it to leadership

Leadership hears cost, risk and time. Translate each.

The pitch for a Kedro platform. "We have twenty teams each building pipelines differently, so every hand-over and every incident starts from scratch. A standard starter, image and set of data contracts means a new pipeline reaches production with a documented command, an audit trail and a tested upgrade path. The framework is free under Apache 2.0; the cost is a small platform team and the discipline to keep it small."

The risk conversation. Be direct about three things. Kedro does not schedule or retry, so we are buying an orchestrator regardless. Its behaviour can change in minor releases for security reasons, so we budget upgrade time each quarter. And it provides no tenant isolation, so sensitive workloads get separate deployments.

The "why not just notebooks" objection. Notebooks are excellent for exploration and poor for repeatable production: no catalog, no tests by default, and hidden state. Kedro does not forbid notebooks; kedro jupyter lab loads the catalog and context into them, so exploration and production share definitions.

The "why not buy a platform" objection. Managed platforms can be the right answer, and Kedro deploys onto several of them, including Databricks, SageMaker, Vertex AI and Azure ML. The question is which layer you want to own. Owning the code conventions while renting the execution is a common, defensible split.

Numbers to offer. Time from kedro new to first scheduled run. Share of projects on the supported minor. Share of cross-team datasets with validators. Number of incidents traced to silent in-memory outputs, which should fall to zero after the CI check. Do not invent figures you cannot measure.

Sixty-second self-test

Close any answer by naming the control that would have caught the problem earlier. That habit is what separates someone who operates a platform from someone who has read about one.

Part three of three. At this level the problems are operational: a fleet of pipelines on three different Kedro versions, a hook that goes silent under one runner, a config file that chose which code ran. Start with the error table, then the production traps, the habits, and finally checklists you can paste into pull request and design templates.

Everything is checked against Kedro 1.7.0. Where the official documentation and the shipped behaviour disagree, this page follows the behaviour.

Common errors at this level

Cumulative: everything from Beginner and Mid still applies. These are the errors that show up at scale or after an upgrade, and each one has a cause that is not where the message points.

The first two rows cause more platform incidents than the rest combined, and both are silent until something downstream breaks.

Production traps that only appear at scale

The registry that imports everything. A hand-written register_pipelines() that imports each pipeline eagerly defeats selective pipeline loading from 1.5. With ten pipelines you will not notice. With four hundred, every kedro run --pipelines x pays the import cost of all of them. Keep the template's find_pipelines(raise_errors=True). The raise_errors=True matters: without it an import error in one pipeline makes that pipeline vanish from the registry rather than fail loudly.

The plugin nobody audited. Hooks from installed plugins are auto-discovered and registered in every project in the environment. A plugin pulled in transitively by another dependency runs on every node. Run kedro info in CI, diff it against an approved list, and use DISABLE_HOOKS_FOR_PLUGINS when you need an immediate mitigation.

The dormant dependency. kedro-docker was last released in November 2024. Community plugins such as kedro-mlflow and kedro-dagster move at their own pace. Before you standardise on one, read its release history against Kedro's, and check what happens to it on the next minor version.

Pickle at team boundaries. Loading a pickle runs code. The 9.5 and 9.6 dataset releases made PyTorch and TensorFlow loading safer by default, but a generic pickle dataset from a bucket another team writes to is still an execution path. Use Parquet, CSV or another data format wherever one team reads what another wrote.

Versions that never expire. versioned: true writes a new timestamped folder on every save, and Kedro never deletes old ones. A nightly pipeline on a large dataset fills a bucket quietly. Put a lifecycle rule on the bucket, and use list_versions() (1.3) in a clean-up job if you need logic.

The HTTP server as a product. kedro server start has no authentication, no queue, no run history and no per-request isolation, and concurrent runs share one session. It is fine behind a gateway for internal triggers. It is a liability as a public or multi-tenant endpoint. Also, GET /snapshot returns HTTP 200 even when it fails, so a health check that looks only at the status code will pass on a broken project.

Secrets in query strings. GET /snapshot?params= accepts runtime_params values. Query strings are logged by proxies and servers. Never pass a secret through it.

Timestamps that do not parse. --load-versions=ds:2026-01-01T00.00.00.000Z requires that exact format, with dots in the time. Since the 1.3 security fix, a version string must be a single path component, so anything with a slash is rejected. Scripts that build version strings from date output fail here, so centralise that formatting.

Start-method drift. KEDRO_MP_CONTEXT decides fork, spawn or forkserver. Defaults are fork on Linux up to Python 3.13, forkserver on Linux with Python 3.14 or later, and spawn on macOS and Windows. An upgrade of the base image's Python from 3.13 to 3.14 therefore changes multiprocessing behaviour without touching your code. Pin the variable in the image if you use ParallelRunner.

Experimental APIs in a platform. Anything marked with @experimental raises KedroExperimentalWarning on first use and can change in a minor release. As of 1.7, Node(preview_fn=...) with the preview classes, and %load_node, are still experimental, while llm_context_node and its companions graduated in 1.7. KedroServiceSession is not decorated but is documented as in active development. Do not build a platform contract on any of them without a wrapper that you control.

Dataset renames in the extras package. kedro-datasets is versioned separately. 9.4 renamed experimental Langfuse, Opik and LangChain classes, for example langfuse.LangfusePromptDataset to langfuse.PromptDataset, and 9.0 removed MatplotlibWriter. A catalog that worked last quarter can fail on a dataset upgrade even when kedro did not change. Pin both and bump them together.

Stale tutorials. Searches still return 0.19-era advice: DataSet with a capital S (now Dataset, with a warning), kedro catalog list, --namespace, extra_params, pandas-iris, ConfigLoader and TemplatedConfigLoader, kedro micropkg. Teams copy these. Put a short "do not use" list in the starter's README, and let the linter catch what it can.

Platform-ownership habits

These are ordered by leverage: what pays off most comes first.

1. Own the starter, not the pipelines. A starter reaches every future project. Publish it through the kedro.starters entry point or a Git repository, and keep its settings.py, hooks, logging, Dockerfile and CI in one place. Every hour spent here is repaid for each team.
2. Turn silent failures into CI failures. The most valuable check is that deployed outputs never resolve to defaults. The second is that find_pipelines(raise_errors=True) is still present. The third is that the plugin list equals the approved baseline.
3. Pin and upgrade together. kedro~=1.7 with kedro-datasets extras pinned in the same file, upgraded in the same pull request, with kedro_init_version updated to match. Maintain a table of which project runs which version.
4. Read release notes as a security practice. 1.3, 1.6 and 1.7 each carried security-relevant behaviour changes. Someone on the platform team reads RELEASE.md on the day a release lands.
5. Roll contracts out as warnings. New validator: entries start at severity: warn, with the count exported as a metric. Flip to error after a quiet week. Keep KEDRO_DATASET_VALIDATION=0 as a documented emergency switch with a named owner.
6. Make dataset names an interface. Because the DAG is derived from names, renaming a dataset rewires the pipeline. Review renames like API changes, and never invent dotted names, since . is reserved for namespaces.
7. Keep hooks boring. Metrics, audit, alerts and credential loading are good hook work. Business logic is not, because hooks are invisible to the people reading a pipeline. Write tests for hooks, run them under the same runners your teams use, and keep them out of ParallelRunner assumptions.
8. Bake the telemetry opt-out into the image. KEDRO_DISABLE_TELEMETRY or DO_NOT_TRACK in the base image overrides every other setting. Telemetry is on by default, and it is easy to forget in a regulated environment.
9. Emit a run contract. Every orchestrated run logs the image digest, conf_source, environment, pipelines, run identifier and, on failure, the node and error type. The first hour of every incident is spent finding these, so have them ready.
10. Keep a written support boundary. What the platform guarantees and what teams own, including how fast security fixes reach the base image.
11. Use deprecation warnings as your calendar. --async, --pipeline and session.run(pipeline_name=) are flagged for removal in Kedro 2.0. Run tests with warnings promoted to errors in a scheduled job so that you find stragglers before 2.0 does.
12. Publish a graph. A nightly kedro viz build for each project, published internally, shows other teams how their data and yours connect, and costs almost nothing.

Verification recipes

These are commands you can run to prove a claim rather than assume it.

# What is actually installed, including plugins and their entry points?
kedro info

# Which datasets fall back to in-memory defaults for this pipeline?
kedro catalog describe-datasets -p __default__

# How do factory patterns resolve, and in which order?
kedro catalog list-patterns
kedro catalog resolve-patterns -p __default__

# Does the registry list every pipeline you expect?
kedro registry list

# Rerun only what is missing, to check persistence is real
kedro run --pipelines=__default__ --only-missing-outputs

For hooks under different runners, add a one-line log in before_node_run, run once with SequentialRunner and once with ParallelRunner, and count the lines. For the logging change, set KEDRO_LOGGING_MODULE_ALLOWLIST and try a handler from your own package. For secrets, run kedro package and list the configuration archive with tar -tzf, then confirm that no local/ file is inside.

Checklist for a pull request

• Does any new dataset name collide with a namespace or use a dot?
• Is every output in the catalog, and is each one remote if the pipeline is orchestrated?
• Does kedro catalog describe-datasets show nothing new under defaults?
• Do any new datasets that cross a team boundary have a validator: at severity: warn or error?
• Are new parameters typed with a Pydantic model or dataclass, and is the same key not annotated two ways?
• Did anything add --async, --pipeline or pipeline_name=?
• Does any hook parameter have a default value?
• Do any hooks, resolvers or datasets import a class named in configuration?
• Are there new pickle-backed datasets, and who reads them?
• Are any new dependencies plugins with entry points, and were they approved?
• Are all secrets out of conf/base, and are credentials read from the environment, workload identity or a vault hook?
• Do the tests run under the runner production uses, including spawn if ParallelRunner is involved?

Checklist for a design document

• Scope. Which orchestrator, which data engine, which cloud and region? Does data residency constrain buckets or compute?
• Execution. Runner and why; node grouping; what happens on retry; is every node idempotent?
• Data. Which datasets cross task or team boundaries, where are they stored, in what format, with which validators?
• Configuration. Which environments, whether --conf-source is remote, and how parameters are typed. Who can change production configuration?
• Isolation. What keeps one tenant or team from reading another's data? If the answer is an environment folder, send it back.
• Security. Where do credentials come from? Which plugins and hooks are installed? Does anything let data choose code?
• Observability. Where do logs go, what run contract is emitted, who is alerted from on_pipeline_error?
• Serving. If the HTTP server or KedroServiceSession is used, what sits in front of it?
• Upgrades. Pinned versions, deprecation exposure, pilot plan, rollback path.
• Cost. Persistence versus recomputation, storage lifecycle for versioned datasets, task granularity.
• Exit. What would make us choose something other than Kedro for this workload?

Practice cards

Card one: the silent typo. Take a scratch project, misspell one output, and run it. Find the failure with only kedro catalog describe-datasets. Then write the CI step that would have caught it.

Card two: the quiet hook. Write before_node_run logging a line. Run under three runners. Write down which ones fired and explain why.

Card three: the upgrade drill. Take a 1.6-style project with a custom logging handler. Upgrade it to 1.7, read the error, and fix it with the allowlist variable. Then search the repository for the old runner allowlist name.

Card four: the contract rollout. Add a Pandera schema with severity: warn, break the data, read the warning, then flip to error and read DataValidationError. Finally run once with the kill switch and decide who should be allowed to do that.

Card five: the orchestrator hand-off. Generate an Airflow DAG with kedro airflow create --target-dir=dags/ --env=airflow, then again with --group-by memory. Count tasks. Break one intermediate by leaving it in memory and watch where the hand-off fails.

Card six: the isolation argument. Write three sentences explaining to a product manager why separate environment folders do not protect one customer's data from another, and what does.

Finish each card by writing the control that would have prevented the problem. Platforms improve when incidents become checks, and these six cards cover most of what you will meet in the first year.

One last habit ties the rest together: hold a short review after every upgrade, not only after every incident. Ask which warnings appeared, which settings changed meaning, and which team found the problem first. If a team found it before your checks did, the next check to write is obvious, and over a year those small additions are what turn a collection of projects into a platform that people trust.