The Complete ClearML Guide

This is part one of three. It covers everything you need to do real work with ClearML, not a teaser. By the end you can track an experiment without changing your training loop, log metrics and artifacts on purpose, compare twenty runs in a table, version a dataset, clone a run and re-launch it on a GPU box you never SSH into, and answer "what exactly produced this model?" with a task id. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and these ideas only stick once you have watched a scalar appear in the web UI while your script is still running.

What ClearML is, and the problem it solves

You trained a model on Tuesday. It scored 0.94. On Thursday you cannot reproduce it. The notebook has moved on, the learning rate in the cell is not the one you ran, the dataset folder has been "cleaned up", and the only record of 0.94 is a screenshot in Slack.

Discipline was never the problem. The tooling was: nothing was watching. Every fact about that run lived in your head, your terminal scrollback, and a filename.

ClearML watches. You add two lines to your script and it records the code, the git commit, the uncommitted diff, the installed packages, the command-line arguments, every metric you print to TensorBoard, the model file you saved, and the machine it ran on, all into a server you and your team can query.

ClearML's central trick is that a recorded run is executable. Other tools log an experiment so you can look at it. ClearML logs enough (repository, commit, diff, packages, arguments, environment) that a worker can rebuild the environment and run it again on a different machine. That single property is what turns a log into a workflow.

That gives you four things, and they are the reason ClearML exists rather than a spreadsheet and a naming convention:

The vocabulary is small, and getting it straight now saves confusion later:

You need little to follow along: Python, a script that trains something small, and a free account on the hosted server. A tiny model on a tiny dataset is the best place to start, because a broken run costs you thirty seconds.

The architecture: four parts, three data stores

ClearML has more moving pieces than a single library, and knowing which piece does what makes every error message readable.

Behind the API server sit three data stores, and knowing which is which explains later performance advice: MongoDB holds task metadata, Elasticsearch holds metrics and console logs, and Redis holds ephemeral state. You do not touch them directly at this level, but "my scalars are slow" and "my task list is slow" are two different problems for that reason.

You have two options for the server, and for learning there is only one sensible answer:

Install and connect: credentials and config

Two commands, and the second one is interactive.

pip install clearml
clearml-init

clearml-init asks you to paste a credentials block, which you copy from the web UI under Settings → Workspace → Create new credentials. It looks like this, and it is the only setup step in the whole guide:

api {
  web_server: https://app.clear.ml
  api_server: https://api.clear.ml
  files_server: https://files.clear.ml
  credentials {
    "access_key" = "ABC123..."
    "secret_key" = "xyz789..."
  }
}

That gets written to ~/clearml.conf, a large, heavily commented file worth skimming once. The four lines that matter today:

Every one of those can be overridden by an environment variable, which is how you configure CI and containers where there is no config file to edit:

export CLEARML_API_HOST=https://api.clear.ml
export CLEARML_WEB_HOST=https://app.clear.ml
export CLEARML_FILES_HOST=https://files.clear.ml
export CLEARML_API_ACCESS_KEY=ABC123...
export CLEARML_API_SECRET_KEY=xyz789...

Confirm it works before writing any code:

clearml-init          # re-run any time to reconfigure
python -c "from clearml import Task; print(Task.get_projects()[:3])"

Your first tracked task, in two lines

This is the whole beginner story in two lines. Start from a script with no ClearML in it:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

n_estimators = 100
max_depth = 4

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)
model.fit(X_train, y_train)
print("accuracy", model.score(X_test, y_test))

Here it is tracked:

from clearml import Task

task = Task.init(project_name="iris-demo", task_name="random forest baseline")

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

params = task.connect({"n_estimators": 100, "max_depth": 4})

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=params["n_estimators"], max_depth=params["max_depth"])
model.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
task.get_logger().report_single_value("accuracy", accuracy)
print("accuracy", accuracy)

Run it. The first thing it prints is the link that matters:

ClearML Task: created new task id=8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
ClearML results page: https://app.clear.ml/projects/.../experiments/8f2c4b19.../output/log

Open that link while the script is still running and you are watching the console output stream into the UI. Four things happened without you asking:

1. A task was createdWith an id, a project, a name, a status of running, and a start time.
2. Your environment was recordedGit remote, branch, commit, the diff of your uncommitted changes, and the installed packages the script imported.
3. Your stdout and stderr were capturedStreamed to the server, so the console tab is a live log you can read from your phone.
4. Your parameters were registeredBecause you passed that dict through task.connect, they are now editable in the UI, which is what makes the run re-launchable.

What gets captured automatically, and what does not

The word ClearML uses for this is "automagic", and knowing its exact boundaries stops you from either duplicating work or expecting magic that is not there.

Captured with no code from you, as long as Task.init ran first:

That means an existing TensorBoard-instrumented project needs no logging changes at all:

from clearml import Task
task = Task.init(project_name="vision", task_name="resnet baseline")

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter()

for epoch in range(epochs):
    loss = train_one_epoch()
    writer.add_scalar("loss/train", loss, epoch)      # appears in ClearML Scalars
    torch.save(model.state_dict(), "checkpoint.pt")   # appears in ClearML Models

What is not captured, and this list matters:

You can turn parts of the magic off when it gets in the way:

task = Task.init(
    project_name="vision",
    task_name="resnet baseline",
    auto_connect_frameworks={"matplotlib": False, "pytorch": True},
    auto_connect_arg_parser=False,      # do not hoover up argparse
    output_uri="s3://my-bucket/clearml",  # send models and artifacts to S3
)

Logging on purpose: the Logger

Automatic capture gets you a long way. The Logger is for everything you want recorded deliberately, and its API is small enough to memorise.

logger = task.get_logger()

# A number that changes over training → a curve
logger.report_scalar(title="loss", series="train", value=0.412, iteration=epoch)
logger.report_scalar(title="loss", series="val", value=0.503, iteration=epoch)

# A number that happens once → a table row, comparable across tasks
logger.report_single_value(name="test_accuracy", value=0.941)

# A table
logger.report_table(title="confusion", series="test", iteration=0, table_plot=df)

# An image
logger.report_image(title="samples", series="epoch-3", iteration=3, local_path="grid.png")

# Free text into the task log
logger.report_text("Trained on dataset version 3fa1b2c")

# Any matplotlib figure
logger.report_matplotlib_figure(title="roc", series="test", iteration=0, figure=fig)

The title / series distinction is the one people get wrong, and it decides what your Scalars tab looks like:

So title="loss" with series="train" and series="val" gives you one chart with two lines, which is what you want. Using title="train_loss" and title="val_loss" gives you two separate charts you cannot visually compare, which is not.

A realistic training loop, fully instrumented:

from clearml import Task

task = Task.init(project_name="vision", task_name="resnet18 baseline")
logger = task.get_logger()

params = task.connect({"lr": 3e-4, "batch_size": 64, "epochs": 20, "seed": 42})
set_seed(params["seed"])

for epoch in range(params["epochs"]):
    train_loss = train_one_epoch(model, train_loader, lr=params["lr"])
    val_loss, val_acc = evaluate(model, val_loader)

    logger.report_scalar("loss", "train", train_loss, epoch)
    logger.report_scalar("loss", "val", val_loss, epoch)
    logger.report_scalar("accuracy", "val", val_acc, epoch)
    logger.report_scalar("lr", "current", scheduler.get_last_lr()[0], epoch)

test_acc = evaluate(model, test_loader)[1]
logger.report_single_value("test_accuracy", test_acc)
logger.report_single_value("params_millions", count_params(model) / 1e6)

Hyperparameters: the editable surface of a task

Parameters are not just a record. In ClearML they are the editable surface of a task, the thing you change when you clone a run and re-launch it. That is why how you register them matters more than it looks.

Three mechanisms, for three shapes of configuration:

# 1. A dict of scalars — the common case
params = task.connect({"lr": 3e-4, "batch_size": 64, "epochs": 20})
print(params["lr"])                     # read from the returned dict, always

# 2. argparse — captured automatically, no call needed
parser = argparse.ArgumentParser()
parser.add_argument("--lr", type=float, default=3e-4)
args = parser.parse_args()              # already registered under "Args"

# 3. A nested config object or file — too big for a parameter list
config = task.connect_configuration(
    configuration="configs/model.yaml",
    name="model config",
)

The critical detail is that connect is bidirectional. When your code runs normally, connect uploads the dict to the server. When an agent runs a clone of that task, connect reads the values back from the server and overwrites your defaults. Same line of code, opposite direction.

# Locally: lr is 3e-4, uploaded to the server.
# In the UI you clone the task, edit lr to 1e-3, and enqueue.
# On the agent: connect() returns {"lr": 1e-3} — your 3e-4 default is ignored.
params = task.connect({"lr": 3e-4})
train(lr=params["lr"])          # runs with 1e-3, no code change

Artifacts: files and objects on a task

An artifact is any file or Python object you attach to a task. Use them for the things that are neither metrics nor the final model: a preprocessed dataframe, an evaluation report, a vocabulary, a set of predictions.

# A pandas DataFrame — stored as a compressed CSV, previewable in the UI
task.upload_artifact("test predictions", artifact_object=predictions_df)

# A local file or folder
task.upload_artifact("report", artifact_object="outputs/report.html")
task.upload_artifact("plots", artifact_object="outputs/plots/")   # zipped

# Any picklable object
task.upload_artifact("vocab", artifact_object=vocab_dict)

# A numpy array
task.upload_artifact("embeddings", artifact_object=embeddings)

Reading them back from another script is the part that makes them useful:

from clearml import Task

producer = Task.get_task(project_name="vision", task_name="resnet18 baseline")
df = producer.artifacts["test predictions"].get()          # deserialised object
path = producer.artifacts["report"].get_local_copy()       # downloaded file path

task = Task.init(project_name="vision", task_name="run", output_uri="s3://my-bucket/clearml")
# or, for every task on this machine, in clearml.conf:
#   sdk.development.default_output_uri: "s3://my-bucket/clearml"

Models and the registry: weights with provenance

A model is an artifact with privileges: its own registry entry, its own tags, and a recorded link back to the task that produced it. Most of the time you get one for free.

# Automatic: ClearML intercepts the framework's save call
torch.save(model.state_dict(), "model.pt")
joblib.dump(sklearn_model, "model.pkl")
# → both appear in the task's Models tab and in the project's Models list

When you want control, register explicitly:

from clearml import OutputModel

output_model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
output_model.update_weights(weights_filename="model.pt")
output_model.update_design(config_dict={"arch": "resnet18", "classes": 10})
output_model.tags = ["baseline", "candidate"]

Loading one back, which is the half that makes a registry a registry:

from clearml import InputModel

model = InputModel(model_id="a1b2c3d4e5f6")                    # by id
model = InputModel(project="vision", name="resnet18-cls",      # or by query
                   tags=["production"], only_published=True)

local_weights = model.get_local_copy()      # downloads and caches

The lifecycle is deliberately simple at this level:

Datasets: immutable, incremental versions

A tracked experiment that reads /data/train.csv is only half-reproducible. ClearML Dataset closes that gap: an immutable, versioned, content-addressed collection of files that is itself a task, so it has an id, a project, tags, and lineage.

From the command line:

clearml-data create --project datasets --name iris --version 1.0.0
clearml-data add --files ./data
clearml-data close                     # uploads and makes the version immutable

Or from Python, which is what you will use inside a pipeline:

from clearml import Dataset

ds = Dataset.create(dataset_project="datasets", dataset_name="iris", dataset_version="1.0.0")
ds.add_files("./data")
ds.upload()
ds.finalize()
print(ds.id)

Consuming it is one line, and this is the line that makes an experiment reproducible:

from clearml import Dataset

path = Dataset.get(dataset_project="datasets", dataset_name="iris", alias="training data").get_local_copy()
df = pd.read_csv(f"{path}/train.csv")

get_local_copy() downloads once and caches. Call it again on the same machine and it returns the cached path immediately. The alias argument is important: it records the dataset id into the consuming task's parameters, so the experiment's Configuration tab names the exact data version it used.

Versions are incremental rather than full copies, which is what makes them cheap:

child = Dataset.create(
    dataset_project="datasets",
    dataset_name="iris",
    dataset_version="1.1.0",
    parent_datasets=[ds.id],       # inherit everything from 1.0.0
)
child.add_files("./new-batch")     # only the delta is uploaded
child.finalize()

Comparing experiments: table, chart, and code

This is where the tracking pays off, and it is almost entirely a UI skill. Run your script three times with different learning rates before reading on, so you have something to compare.

In the experiments table:

The compare view has three tabs, and each answers a different question:

The same queries work from Python, which is how you build a report or a promotion gate:

from clearml import Task

tasks = Task.get_tasks(
    project_name="vision",
    task_filter={"status": ["completed"], "order_by": ["-last_update"]},
)

for t in tasks[:10]:
    metrics = t.get_last_scalar_metrics()
    print(t.name, metrics.get("accuracy", {}).get("val", {}).get("last"))

best = max(tasks, key=lambda t: t.get_last_scalar_metrics()
           .get("accuracy", {}).get("val", {}).get("last", 0))
print("best:", best.id, best.name)

Agents and queues: running a task elsewhere

Everything so far worked with no worker anywhere. The second half of ClearML takes a recorded run and executes it somewhere else.

An agent is a process that watches a queue. When a task is enqueued, the agent claims it, rebuilds its environment from the recorded packages, clones the recorded git commit, applies the recorded diff, and runs the recorded entry point.

pip install clearml-agent
clearml-agent init                        # same credentials flow as clearml-init

# Run a worker that watches the "default" queue
clearml-agent daemon --queue default

# On a GPU box: one worker per GPU
clearml-agent daemon --queue gpu --gpus 0
clearml-agent daemon --queue gpu --gpus 1

# In Docker mode — the environment is a container, not a virtualenv
clearml-agent daemon --queue gpu --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04

The everyday loop, and it is this short:

1. Right-click a completed task → CloneYou get an identical task in draft status. The original is untouched; a completed task is read-only.
2. Edit whatever you wantHyperparameters, the docker image, the git branch, even the entry-point arguments. All editable while in draft.
3. Enqueue it, choosing a queueThe task moves to queued. If no agent watches that queue it waits, which is a normal state and not an error.
4. The agent picks it upCreates a virtualenv (or container), installs the recorded packages, clones the commit, applies the diff, runs the script.
5. Watch it in the Console tabIdentical experience to a local run, on hardware you never logged into.

The mechanism that makes this smooth in development is execute_remotely:

from clearml import Task

task = Task.init(project_name="vision", task_name="resnet18")
params = task.connect({"lr": 3e-4, "epochs": 50})

# Everything above this line runs locally: the task is created and configured.
# Then the local process exits and the task is enqueued for an agent.
task.execute_remotely(queue_name="gpu")

# Nothing below here runs locally — only on the agent.
train(**params)

Run that on your laptop and it takes two seconds: it registers the task, uploads the config, enqueues it, and quits. The fifty-epoch training happens on the GPU box. Comment out one line and the identical script runs locally for debugging. That is the pattern to learn.

Your first pipeline: tasks that create tasks

Once runs are re-executable, chaining them is a small step. A pipeline is a task that creates and monitors other tasks.

The decorator style is the clearest starting point:

from clearml import PipelineDecorator

@PipelineDecorator.component(return_values=["data_path"], cache=True, execution_queue="cpu")
def prepare(dataset_name: str):
    from clearml import Dataset
    return Dataset.get(dataset_project="datasets", dataset_name=dataset_name).get_local_copy()

@PipelineDecorator.component(return_values=["model_path", "accuracy"], execution_queue="gpu")
def train(data_path: str, lr: float, epochs: int):
    # ordinary training code
    return "model.pt", 0.941

@PipelineDecorator.component(return_values=["report"], execution_queue="cpu")
def evaluate(model_path: str, data_path: str):
    return {"auc": 0.97}

@PipelineDecorator.pipeline(name="train-and-eval", project="vision", version="1.0.0")
def main(dataset_name="iris", lr=3e-4, epochs=20):
    data_path = prepare(dataset_name)
    model_path, accuracy = train(data_path, lr, epochs)
    if accuracy > 0.9:
        report = evaluate(model_path, data_path)
        print(report)

if __name__ == "__main__":
    PipelineDecorator.run_locally()      # remove this line to run on agents
    main()

Three things in there matter. Each component becomes its own task with its own console log, metrics, and artifacts. execution_queue is per component, so the CPU preprocessing and the GPU training run on different hardware without you orchestrating anything. The graph is inferred from the data flow: train depends on prepare because it consumes its return value, not because you declared an edge.

The last one is the one that shows what ClearML is doing:

from clearml import PipelineController

pipe = PipelineController(name="nightly", project="vision", version="1.0.0")
pipe.add_parameter("lr", 3e-4)

pipe.add_step(name="prepare", base_task_project="vision", base_task_name="prepare data")
pipe.add_step(
    name="train",
    parents=["prepare"],
    base_task_project="vision",
    base_task_name="resnet18 baseline",
    parameter_override={"General/lr": "${pipeline.lr}"},
)
pipe.start(queue="services")

A pipeline step here is literally "clone this existing task, override these parameters, run it". Nothing new is invented. It is the clone-and-enqueue loop from the previous section, expressed as a graph.

Reading a failed task, in order

Debugging ClearML has an order, and following it beats guessing.

1. Read the Console tab firstYour script's stdout and stderr are there in full. Most failures are ordinary Python tracebacks and you can stop here.
2. Then the Execution tabConfirm the git commit, the branch, the entry point, and the working directory are what you expected. A wrong commit explains a lot.
3. Then the installed packages listThe agent installs exactly this list. A missing system library or a package your script imports lazily shows up as an ImportError here.
4. Then ConfigurationAn agent-run clone uses the server's parameter values, not your defaults. A surprising value here is usually the whole bug.
5. Check the workerWorkers & Queues shows whether an agent is even watching that queue. A task sitting in queued forever is almost always no agent, or the wrong queue name.
6. Reproduce locally with the same configTask.get_task(task_id=...), read its parameters, and run the script with them by hand.

Five failures cover most of what you will hit at this level:

from clearml import Task

t = Task.get_task(task_id="8f2c4b19e0a7d3f1c6b8a2e4d7091f3b")
print(t.status, t.get_last_iteration())
print(t.get_parameters())                  # what it actually ran with
print(t.get_last_scalar_metrics())         # what it produced
print(t.data.script.diff[:500])            # the uncommitted patch it applied
t.mark_stopped()                           # abort a stuck run

Putting it all together

Everything above in one project. Nothing here is new. Read it as a whole and you should be able to justify every line.

.
├── .gitignore              # includes clearml.conf
├── requirements.txt
├── configs/
│   └── model.yaml          # connected as a configuration object
├── src/
│   ├── make_dataset.py     # creates a ClearML Dataset version
│   ├── train.py            # the tracked experiment
│   ├── evaluate.py         # loads a model by tag, reports metrics
│   └── pipeline.py         # chains the three
└── README.md               # the three commands to reproduce

from clearml import Dataset

ds = Dataset.create(
    dataset_project="datasets",
    dataset_name="iris",
    dataset_version="1.0.0",
)
ds.add_files("./data")
ds.upload()
ds.finalize()
print("dataset id:", ds.id)

from clearml import Task                       # 1. import first

task = Task.init(                              # 2. init before heavy imports
    project_name="vision",
    task_name="resnet18 baseline",
    output_uri="s3://my-bucket/clearml",       # 3. artifacts to object storage
)
task.add_tags(["baseline"])                    # 4. taggable from day one

import torch
from clearml import Dataset, OutputModel

params = task.connect({                        # 5. editable on a clone
    "lr": 3e-4,
    "batch_size": 64,
    "epochs": 20,
    "seed": 42,
})
cfg = task.connect_configuration("configs/model.yaml", name="model config")

# 6. remote execution is one line; comment it out to debug locally
task.execute_remotely(queue_name="gpu")

set_seed(params["seed"])
data = Dataset.get(                            # 7. data version recorded via alias
    dataset_project="datasets",
    dataset_name="iris",
    alias="training data",
).get_local_copy()

logger = task.get_logger()
for epoch in range(params["epochs"]):
    train_loss = train_one_epoch(data, lr=params["lr"])
    val_loss, val_acc = validate(data)
    logger.report_scalar("loss", "train", train_loss, epoch)     # 8. curves
    logger.report_scalar("loss", "val", val_loss, epoch)
    logger.report_scalar("accuracy", "val", val_acc, epoch)

test_acc = test(data)
logger.report_single_value("test_accuracy", test_acc)            # 9. comparable
task.upload_artifact("predictions", predictions_df)              # 10. artifact

model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
model.update_weights(weights_filename="model.pt")                # 11. registry
model.tags = ["candidate"]

from clearml import PipelineController

pipe = PipelineController(name="nightly", project="vision", version="1.0.0")
pipe.add_parameter("lr", 3e-4)

pipe.add_step(name="prepare", base_task_project="vision", base_task_name="make dataset")
pipe.add_step(
    name="train",
    parents=["prepare"],
    base_task_project="vision",
    base_task_name="resnet18 baseline",
    parameter_override={"General/lr": "${pipeline.lr}"},
)
pipe.add_step(name="evaluate", parents=["train"],
              base_task_project="vision", base_task_name="evaluate")

pipe.start(queue="services")            # start_locally() while developing

# One-time setup
pip install clearml clearml-agent
clearml-init

# Everyday loop
python src/train.py                       # tracked; enqueued to gpu by execute_remotely
# → compare in the UI, clone the winner, edit lr, enqueue again

# A worker, wherever the hardware is
clearml-agent daemon --queue gpu --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04

Eleven decisions in there are the whole lesson of this page:

What you can now do, and what comes next

You can track an experiment with two lines, log metrics and artifacts deliberately, register and query models, version datasets incrementally, compare dozens of runs in a table and in code, clone a run and re-launch it on other hardware without SSH, chain runs into a cached pipeline, and diagnose a failed task in a fixed order. That is a working practitioner's toolkit, enough to own the experiment-tracking and reproducibility story on a real project.

Mid-level takes every one of those topics further: hyperparameter optimisation with HyperParameterOptimizer and Optuna, task types and the services queue, agent modes and caching in depth, docker mode and clearml-task for launching unmodified code, remote debugging with clearml-session, dataset internals and squashing, the model registry as a promotion workflow, ClearML Serving, autoscalers, reports, and the CI patterns that make a training run a pull-request check.

Senior then covers what you own when ClearML is your team's platform: the self-hosted deployment and its three data stores, the access and credential model, multi-tenancy across teams, storage cost and retention, Elasticsearch and MongoDB scaling, upgrade and backup procedure, audit and lineage for regulated work, GPU fleet economics, incident playbooks, and where ClearML stops and a feature store, a data warehouse, or a dedicated serving stack begins.

Part one of three. A fast review of everything in the Beginner Detailed track, in about twenty-five minutes. Fast review first, common questions at the end. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

ClearML is an open-source MLOps platform that tracks experiments, versions datasets, and re-executes runs on remote hardware. You add Task.init to a script and it records the code, git commit, uncommitted diff, installed packages, arguments, metrics, and models into a server. Because it records the environment and not just the results, any past run can be cloned, reconfigured in the UI, and re-launched on an agent, so a logged experiment is executable, not just readable.

Then add the sentence that shows you have used it: "the part people underestimate is that task.connect is bidirectional: locally it uploads your parameters, but when an agent runs a clone it reads them back and overrides your defaults. That is what makes clone-edit-enqueue work without touching the code."

The architecture

MongoDB holds task metadata, Elasticsearch holds metrics and console logs, Redis holds ephemeral state. Worth naming. It explains why a slow task list and slow scalars are two different problems.

Vocabulary

Why ClearML rather than the alternatives

Know the follow-up: "how is this different from MLflow?" MLflow tracks and packages; ClearML tracks and executes. MLflow has no agent, no queue, and no built-in dataset versioning, so an MLflow setup usually means MLflow plus Airflow plus DVC. Whether that is better is a real architectural argument, do not pretend it is not, but the one-sentence difference is that ClearML records enough to rerun the experiment, not just enough to describe it.

The everyday code

from clearml import Task

task = Task.init(project_name="vision", task_name="resnet18")   # first, before heavy imports
task.add_tags(["baseline"])

params = task.connect({"lr": 3e-4, "epochs": 20})               # editable on a clone
cfg = task.connect_configuration("configs/model.yaml")          # a whole config file

task.execute_remotely(queue_name="gpu")                          # everything below runs on an agent

logger = task.get_logger()
logger.report_scalar("loss", "train", value, iteration)          # a curve
logger.report_single_value("test_accuracy", 0.941)               # a comparison column
task.upload_artifact("predictions", df)                          # a file or object

clearml-init                                     # write ~/clearml.conf
clearml-agent daemon --queue gpu --gpus 0        # a worker
clearml-agent daemon --queue gpu --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04
clearml-data create --project datasets --name iris --version 1.0.0
clearml-data add --files ./data && clearml-data close

What Task.init captures

Being able to list this quickly is the core question at this level:

1. CodeGit remote, branch, commit, the uncommitted diff, entry point, working directory.
2. EnvironmentPython version, and the packages your script imported with exact versions.
3. ArgumentsEvery argparse flag, plus click, fire, and hydra.
4. MetricsAnything written to TensorBoard, TensorBoardX, or matplotlib.
5. ModelsEvery checkpoint saved by PyTorch, TF/Keras, joblib, XGBoost, LightGBM.
6. Console and machinestdout/stderr streamed live, plus CPU, GPU, and memory utilisation.

What is not captured: values you only print(), data read from an unversioned path, conda/apt/system libraries, environment variables (deliberately, because they hold secrets), and seeds you never set.

Logging: title, series, and single values

logger.report_scalar(title="loss", series="train", value=v, iteration=epoch)
logger.report_scalar(title="loss", series="val",   value=v, iteration=epoch)
logger.report_single_value(name="test_accuracy", value=0.941)

One title="loss" with two series gives one comparable chart. Two titles gives two charts you cannot overlay. Report both kinds: curves to diagnose, single values to rank.

Parameters are the editable surface

params = task.connect({"lr": 3e-4})
train(lr=params["lr"])          # read from the return value, always

Three mechanisms: task.connect(dict) for hyperparameters, argparse captured automatically, task.connect_configuration(path) for YAML/JSON and anything nested.

Artifacts and models

task.upload_artifact("predictions", df)               # DataFrame → CSV with a UI preview
task.upload_artifact("report", "outputs/report.html") # a file
other = Task.get_task(task_id="...")
df = other.artifacts["predictions"].get()             # object back
p  = other.artifacts["report"].get_local_copy()       # path back

from clearml import OutputModel, InputModel
OutputModel(task=task, name="resnet18-cls").update_weights("model.pt")
model = InputModel(project="vision", name="resnet18-cls", tags=["production"])
weights = model.get_local_copy()

Datasets

ds = Dataset.create(dataset_project="datasets", dataset_name="iris", dataset_version="1.1.0",
                    parent_datasets=[parent_id])       # only the delta is uploaded
ds.add_files("./new-batch"); ds.upload(); ds.finalize()  # finalize = immutable

path = Dataset.get(dataset_project="datasets", dataset_name="iris",
                   alias="training data").get_local_copy()

Agents and queues

An agent claims a queued task, then: creates a virtualenv or container → clones the recorded git commit → applies the recorded diff → installs the recorded packages → runs the recorded entry point.

task.execute_remotely(queue_name="gpu")   # register + enqueue locally, then exit

Pipelines

@PipelineDecorator.component(return_values=["path"], cache=True, execution_queue="cpu")
def prepare(name): ...

@PipelineDecorator.component(return_values=["model", "acc"], execution_queue="gpu")
def train(path, lr): ...

@PipelineDecorator.pipeline(name="train-and-eval", project="vision", version="1.0.0")
def main(lr=3e-4):
    path = prepare("iris")
    model, acc = train(path, lr)

The traps, and why they happen

Three ideas explain nearly every beginner failure: Task.init must be first, the agent rebuilds from git plus a package list, and parameters flow back from the server on a remote run.

Common interview questions

1. What is ClearML and what problem does it solve?An open-source MLOps platform covering experiment tracking, data versioning, and orchestration. Two lines in a script record the code, commit, diff, packages, arguments, metrics, and models to a server. Because the environment is captured too, any run can be cloned, reconfigured, and re-executed on remote hardware, so the log is executable rather than descriptive.
2. What are the components of a ClearML deployment?The SDK inside your process; an API server for metadata backed by MongoDB, Elasticsearch, and Redis; a web server for the UI; a file server for artifacts when there is no object storage; and agents that pull tasks off named queues and execute them.
3. What exactly does Task.init do?Creates or reuses a task, then hooks into the process: it records the git state including the uncommitted diff, the imported packages, the argument parser, stdout and stderr, machine metrics, and it patches supported frameworks so TensorBoard scalars and saved checkpoints are captured without extra code.
4. Why must Task.init come before the framework imports?Automatic capture is implemented by patching those libraries at import time. If they are already imported, the patch never gets applied and metrics or checkpoints silently go missing. It is a silent failure, not an error, which is what makes it worth knowing.
5. What is not captured automatically?Values you only print, data read from an unversioned path, system and conda packages, environment variables, excluded on purpose since they hold secrets, and any randomness you never seeded. Each has a deliberate replacement: report it, version it, use docker mode, or connect it as a parameter.
6. Explain title versus series in report_scalar.Title is the chart, series is a line within it. One title with `train` and `val` series gives a single comparable chart; two titles gives two charts you cannot overlay. A curve is not a comparison column, so final numbers go through report_single_value and can be sorted in the experiments table.
7. Why is task.connect described as bidirectional?Running locally it uploads your dict to the server. When an agent executes a clone of that task, the same call reads the server's values and returns them, overriding your defaults. That is what makes clone-edit-enqueue work with no code change, and why you must read from the value connect returns rather than from your original literal.
8. How would you rerun a past experiment with a different learning rate?Clone the task in the UI, which produces an editable draft; change the parameter; enqueue it to a queue an agent is watching. The agent rebuilds the environment from the recorded packages, checks out the recorded commit, applies the recorded diff, and runs it. No code change and no SSH.
9. What does execute_remotely do?Everything above the call runs locally: the task is created, parameters uploaded, configuration registered. Then the local process enqueues the task and exits, and the agent runs everything below the line. Commenting out that one line gives you an identical local run for debugging.
10. What does an agent do with a queued task?Claims it, creates an isolated environment (virtualenv, or a container in docker mode), clones the repository at the recorded commit, applies the stored diff, installs the recorded package list, then executes the recorded entry point with the server's current parameter values.
11. Why did my agent task fail with a git error?Almost always because the commit exists only locally. The diff travels through the server, but the base commit has to be fetchable from the remote, so the branch must be pushed before enqueueing.
12. Artifact versus model: when do you use each?An artifact is any attached file or object: a dataframe, a report, a vocabulary. A model is a first-class registry entry with tags, a publish state, and a link back to the task that produced it. Serving code should query the registry by project, name, and tag, so promotion is a tag move rather than a code change.
13. Where do artifacts go, and when is that wrong?To the ClearML file server by default. That is fine for small reports and dataframes and wrong for checkpoints, so set output_uri to S3 or another bucket and large objects go to real object storage with its own lifecycle rules.
14. How does ClearML Dataset versioning work?A dataset is itself a task holding a content-addressed file set. Versions are incremental: a child declares parents and uploads only its delta. finalize() makes a version immutable, which is what gives "trained on 1.1.0" any meaning. Consumers call get_local_copy(), which flattens the parent chain and caches locally.
15. Why pass alias= to Dataset.get?It records the resolved dataset id into the consuming task's configuration, so the experiment itself names the exact data version it read. Without it the run is reproducible only if nobody has to guess which dataset "the training data" meant.
16. How do you compare twenty experiments?In the UI: add hyperparameters and single-value metrics as columns, sort, then select rows and hit Compare with "hide identical values" on, which reduces the runs to the fields that differ. In code: Task.get_tasks with a filter plus get_last_scalar_metrics, which is also how you would write a promotion gate.
17. What is a ClearML pipeline, mechanically?A task that creates and monitors other tasks. Each step is an independent task with its own logs, metrics, and target queue, so different steps can run on different hardware. With the controller API a step is literally "clone this base task and override these parameters", the same clone-and-enqueue loop expressed as a graph.
18. How do you organise hundreds of runs?Projects for structure, with / for nesting, and **tags** for everything cross-cutting: baseline, ablation, candidate, broken. Add tags in code with task.add_tags so they are never forgotten, since a table you cannot filter is a table nobody reads.
19. How is this different from MLflow?MLflow tracks and packages; it has no agent, no queue, and no built-in dataset versioning, so a full setup is usually MLflow plus a scheduler plus a data-versioning tool. ClearML puts tracking, data versioning, and execution in one system, at the cost of running a heavier server. Which trade is right depends on whether you already have an orchestrator.
20. How do you debug a failed remote task?Console tab for the traceback, Execution tab to verify the commit, entry point, and package list, Configuration to check the values it ran with, and Workers & Queues to confirm an agent was even watching. Then reproduce locally with the same parameters via Task.get_task(...).get_parameters().
21. Where do credentials live and what can they do?An access/secret pair in ~/clearml.conf, or the CLEARML_API_* environment variables in CI. That pair can read every experiment in the workspace and enqueue tasks that execute code on your agents, so it is a full-access credential and must never be committed.

Sixty-second self-test

• Give the thirty-second answer, then the sentence that shows you have used it.
• Name the four parts of a ClearML deployment and the three data stores behind the API server.
• List the six categories Task.init captures automatically.
• Say why Task.init must precede the framework imports, and what fails if it does not.
• Explain title versus series, and when to use report_single_value instead.
• Explain why task.connect is bidirectional, and the bug that follows from ignoring it.
• Walk the clone → edit → enqueue → agent loop, naming the status at each step.
• Say what execute_remotely runs locally and what it does not.
• Give the difference between an artifact and a registered model.
• Say where artifacts go by default and when that is the wrong choice.
• Explain incremental dataset versions, finalize(), and what alias= records.
• Name the first two UI tabs to open when a remote task fails.

Part one of three. Almost every beginner problem with ClearML comes from one of three things: Task.init running too late, the agent rebuilding an environment you never fully declared, or parameters flowing back from the server on a remote run. Start with the error table, then work through the habits and practice cards underneath it.

Common errors at this level

The habits that pay off most

Practice cards

Short, self-contained exercises. Each one takes a few minutes and leaves you with a fact you will not forget.

1. Break the init order on purposeMove Task.init below import torch, run, and watch the scalars go missing with no error. Move it back. This one experiment prevents the most common ClearML support question.
2. Watch the diff travelMake an uncommitted edit, run a task, then open Execution → Uncommitted Changes in the UI and read your own patch. Now you know why a dirty tree is still reproducible.
3. Do the title/series experimentLog train and val loss with one title and two series, look at the chart, then split into two titles and look again. Two minutes, permanently fixes your plotting.
4. Run the agent on your own laptopclearml-agent daemon --queue default, then clone a task in the UI and enqueue it. Watching both halves on one machine is what makes agents click.
5. Prove the parameter overrideRead from your literal instead of connect's return, clone the task, change the value, enqueue. Watch it run with the old value. Then fix it and repeat.
6. Fail on an unpushed commitCommit locally, do not push, enqueue. Read the agent's git error. Push and re-enqueue the same task.
7. Cause the lazy-import failureMove an import inside a function, run locally (fine), then on the agent (ImportError). This is the failure mode that only exists remotely.
8. Version and consume a datasetCreate 1.0.0, add a child 1.1.0 with one extra file, then consume it with alias= and find the recorded id in the training task's config.
9. Compare three runs properlyThree learning rates, then Compare with "hide identical values" on. If more than one field differs, something in your setup is non-deterministic, so find out what.
10. Recover a task id from a modelOpen a registered model in the UI and trace it back to the exact commit that produced it. Time yourself; it should be under ten seconds.

Debugging order

Follow this rather than guessing. The first two steps answer most problems.

1. Console tabYour full stdout and stderr. Most failures are ordinary tracebacks and you can stop here.
2. Execution tabCommit, branch, entry point, working directory, and the installed package list the agent used. A wrong commit or a missing package explains most of the rest.
3. Configuration tabThe values it ran with. On an agent-run clone these come from the server, not your defaults, and a surprising value here is usually the whole bug.
4. Workers & QueuesIs an agent watching that queue at all? A task stuck in queued is nearly always this.
5. Machine metricsThe CPU/GPU/memory curves. A run that stops mid-training with no traceback was almost certainly killed for memory, and the curve shows it.
6. Reproduce locally with the task's own configTask.get_task(task_id=…).get_parameters(), then run the script by hand with those values.

from clearml import Task

t = Task.get_task(task_id="8f2c4b19e0a7d3f1c6b8a2e4d7091f3b")
print(t.status, t.get_last_iteration())
print(t.get_parameters())                 # what it ran with
print(t.get_last_scalar_metrics())        # what it produced
print(t.data.script.repository, t.data.script.version_num)
print(t.data.script.diff[:400])           # the patch that was applied
print(t.get_requirements() if hasattr(t, "get_requirements") else t.data.script.requirements)
t.mark_stopped()                          # abort a stuck run

Task.init options worth knowing on day one

Most defaults are right. These five are the ones you will reach for.

task = Task.init(
    project_name="vision/experiments",       # "/" nests projects
    task_name="resnet18 lr sweep",
    task_type=Task.TaskTypes.training,       # shows up as a filter in the UI
    output_uri="s3://my-bucket/clearml",     # artifacts and models to object storage
    reuse_last_task_id=False,                # always a fresh task, never overwrite
    auto_connect_frameworks={"matplotlib": False},
)

Clone, do not rerun

The mental shift that makes ClearML click: a completed task is a record, not a workspace.

Set up so ClearML catches things for you

Do these four things once and an entire class of mistakes disappears.

# 1. Never commit credentials again
echo "clearml.conf" >> ~/.gitignore_global
git config --global core.excludesfile ~/.gitignore_global

sdk {
  development {
    # Every task on this machine sends artifacts and models to object storage
    default_output_uri: "s3://my-bucket/clearml"
  }
  storage {
    cache {
      # A big disk, so dataset caches survive and are shared between runs
      default_base_dir: "/mnt/data/clearml-cache"
    }
  }
}

# 2. Pin the SDK, so an upgrade never surprises a teammate mid-sprint
echo "clearml==1.16.4" >> requirements.txt

# 3. A local agent, for the clone-and-enqueue loop
clearml-agent daemon --queue default --detached

# 4. Confirm what the SDK thinks it is talking to
python -c "from clearml.config import get_config_object; print(get_config_object('api.api_server'))"

Writing a tracked script that ages well

from clearml import Task                          # 1. ClearML import first

task = Task.init(                                 # 2. before heavy imports
    project_name="vision/experiments",
    task_name="resnet18 baseline",
    output_uri="s3://my-bucket/clearml",
    reuse_last_task_id=False,
)
task.add_tags(["baseline", "resnet18"])           # 3. tags in code

import torch                                      # 4. frameworks after init
from clearml import Dataset, OutputModel

params = task.connect({                           # 5. one flat dict of knobs
    "lr": 3e-4,
    "batch_size": 64,
    "epochs": 20,
    "seed": 42,
    "dataset_version": "1.1.0",
})

task.execute_remotely(queue_name="gpu")           # 6. comment out to debug locally

set_seed(params["seed"])                          # 7. seeded and recorded
data = Dataset.get(                               # 8. data version recorded
    dataset_project="datasets",
    dataset_name="iris",
    dataset_version=params["dataset_version"],
    alias="training data",
).get_local_copy()

Reading the UI quickly

Learning to read these five tabs saves more time than any other single skill at this level.

task.connect({...})               → Configuration ▸ Hyperparameters ▸ General
argparse                          → Configuration ▸ Hyperparameters ▸ Args
task.connect_configuration(...)   → Configuration ▸ Configuration Objects
Dataset.get(..., alias="x")       → Configuration ▸ Hyperparameters ▸ Datasets
report_scalar(...)                → Scalars ▸ (one chart per title)
report_single_value(...)          → Scalars ▸ single values table
report_table / matplotlib figure  → Plots
report_image(...)                 → Debug samples
upload_artifact(...)              → Artifacts
torch.save / OutputModel          → Artifacts ▸ Output models, and the project's Models list
print(...)                        → Console  (and nowhere sortable — that is the point)

Small things worth doing from day one

Name tasks by what changed, not by date. resnet18 lr=1e-3 aug=heavy beats run_47. The date is already recorded; the intent is not.

Nest projects with /. research/nlp/ner gives you a tree instead of four hundred siblings, and it costs nothing to adopt later.

Report a single value for training time and model size. They become columns you can sort, and they are the two numbers a reviewer asks for that nobody ever logs.

report_text the things a future reader needs. One line naming the dataset version and the reason for the run turns the task into a note to your future self.

Set and connect the seed. Two runs that differ for unknown reasons cost more time than any other single problem in ML work.

Archive rather than delete. Archived tasks leave the default views but stay fully queryable, so a wrong archive is recoverable and a wrong delete is not.

Check the machine metrics on any run that ended strangely. A missing traceback plus a memory curve that hits the ceiling is an OOM kill, and it looks like nothing else.

task.add_tags([f"lr-{params['lr']}", f"bs-{params['batch_size']}"])
task.get_logger().report_text(f"data={params['dataset_version']} reason=baseline for Q3 report")

A starter setup worth keeping

Copy this into a new project and delete what you do not need. Every line is something from this page.

clearml.conf
.clearml/
outputs/
*.pt
*.pkl

from clearml import Task

task = Task.init(
    project_name="vision/experiments",
    task_name="resnet18 baseline",
    task_type=Task.TaskTypes.training,
    output_uri="s3://my-bucket/clearml",
    reuse_last_task_id=False,
)
task.add_tags(["baseline"])

import torch
from clearml import Dataset, OutputModel

params = task.connect({
    "lr": 3e-4, "batch_size": 64, "epochs": 20,
    "seed": 42, "dataset_version": "1.1.0",
})

task.execute_remotely(queue_name="gpu")     # comment out for a local debug run

set_seed(params["seed"])
data = Dataset.get(dataset_project="datasets", dataset_name="iris",
                   dataset_version=params["dataset_version"],
                   alias="training data").get_local_copy()

logger = task.get_logger()
for epoch in range(params["epochs"]):
    tr, va, acc = train_one_epoch(data, params)
    logger.report_scalar("loss", "train", tr, epoch)
    logger.report_scalar("loss", "val", va, epoch)
    logger.report_scalar("accuracy", "val", acc, epoch)

logger.report_single_value("test_accuracy", test(data))
logger.report_single_value("train_minutes", elapsed_minutes())
task.upload_artifact("predictions", predictions_df)

model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
model.update_weights(weights_filename="model.pt")
model.tags = ["candidate"]

# One-time setup
pip install clearml clearml-agent
clearml-init
echo "clearml.conf" >> ~/.gitignore_global

# Everyday loop
python src/train.py                      # registers, enqueues to gpu, exits
# → compare in the UI, clone the winner, edit one value, enqueue again

# A worker wherever the hardware is
clearml-agent daemon --queue gpu --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 --detached

Eight details in there are the whole lesson of this page: ClearML imported and initialised before any framework, output_uri pointed at object storage, reuse_last_task_id=False so no run overwrites another, tags added in code, every knob in one connected dict read back from the return value, the seed and the dataset version as parameters, execute_remotely as a one-line local/remote switch, and single values for the two numbers a reviewer will ask about.

Mid-level tips go deeper on every one of these: hyperparameter optimisation without wasting GPU hours, agent caching and why the second run is fast, docker mode and clearml-task for code you cannot modify, clearml-session for remote debugging, dataset squashing and cache strategy on ephemeral runners, the services queue, model promotion as a workflow, and diagnosing "runs locally, fails on the agent" properly.

This is part two of three. It picks up exactly where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Nothing is dropped and nothing is repeated for its own sake. Where you already know the basics, we go straight to the depth.

Where this picks up

Each section starts with the problem it solves, and ends with a Try it you can do on a real project in a few minutes.

Task creation, reuse, and lifecycle

Beginner used one entry point: Task.init. At this level you need three, and knowing which is which removes a whole category of confusion.

# 1. init — "this process IS the task". Patches frameworks, captures the environment.
task = Task.init(project_name="vision", task_name="train")

# 2. create — "build a task object for someone ELSE to run". No patching, no capture.
child = Task.create(
    project_name="vision",
    task_name="train on the new split",
    script="src/train.py",
    repo="https://github.com/org/repo",
    branch="main",
    requirements_file="requirements.txt",
    docker="nvidia/cuda:12.1.0-runtime-ubuntu22.04",
)
Task.enqueue(child, queue_name="gpu")

# 3. clone — "copy an existing task as an editable draft"
clone = Task.clone(source_task="8f2c4b19…", name="lr=1e-3")
clone.set_parameter("General/lr", 1e-3)
Task.enqueue(clone, queue_name="gpu")

Two lifecycle behaviours surprise people, and both are documented but easy to miss:

Reuse. Task.init with default settings may reuse the last task created by the same script in the same project if that task has no recorded iterations or artifacts. It exists to stop a debugging session producing forty empty tasks. During a sweep it looks like your runs are vanishing, so pass reuse_last_task_id=False.

Continuation. continue_last_task=True (or =task_id) resumes an existing task, appending scalars from the last reported iteration instead of restarting at zero. That is the right tool for a job that got OOM-killed at epoch 40 of 100, and the wrong tool for a fresh experiment, because the scalar history will be a splice of two runs.

task = Task.init(
    project_name="vision",
    task_name="resnet18",
    continue_last_task=True,        # resume, appending iterations
    reuse_last_task_id=False,       # never silently reuse a different run
)

Task types are more than decoration. They drive UI filtering and, for pipelines and services, actual behaviour:

import os
from clearml import Task

rank = int(os.environ.get("RANK", 0))
task = Task.init(project_name="vision", task_name="ddp") if rank == 0 else None

# Later, anywhere in the codebase:
current = Task.current_task()          # the task on rank 0, None elsewhere
if current:
    current.get_logger().report_scalar("loss", "train", loss, step)

Automatic capture: the mechanism, and its edges

Beginner said "automagic". At this level you have to fix it when it goes wrong, so you need the mechanism.

On Task.init, the SDK installs binding patches. Each supported framework has a binding module that wraps specific functions:

Requirements detection is separate and works by inspecting sys.modules at the end of the run, mapping imported modules to installed distributions. That yields three failure modes worth knowing:

1. A lazily imported package is missedImported inside a function that never ran, so it is not in sys.modules. The agent then fails with ImportError. Import at module level or declare it.
2. The wrong index is usedA detected torch==2.3.0 installs the CPU build from PyPI, not the CUDA build from the PyTorch index. Docker mode with a CUDA base image is the real fix.
3. A local or private package is unresolvableDetected as a name the agent cannot install. Vendor it into the repo, or make the private index available to the agent.

You can take control:

# Declare the environment explicitly rather than relying on detection
task.set_packages("requirements.txt")            # a file
task.add_requirements("torch", "2.3.0")          # a single pin
task.add_requirements("-r requirements-gpu.txt") # pass-through to pip

# Or force docker mode from the code
task.set_base_docker(
    docker_image="nvidia/cuda:12.1.0-runtime-ubuntu22.04",
    docker_arguments="--shm-size=8g --ipc=host",
)

# On the agent side, ignore detection entirely and trust the container
clearml-agent daemon --queue gpu --docker my-registry/train:2.3.0 \
  --force-current-version

agent {
  package_manager {
    type: pip
    system_site_packages: true       # trust what the container already has
    force_upgrade: false
    extra_index_url: ["https://download.pytorch.org/whl/cu121"]
  }
}

Configuration: sections, objects, and references

Beginner connected one flat dict. Real projects have layered config, and ClearML has three answers.

# Sections keep unrelated knobs apart in the UI
task.connect({"lr": 3e-4, "epochs": 20}, name="optimizer")
task.connect({"layers": 18, "width": 64}, name="architecture")
task.connect({"num_workers": 8, "prefetch": 2}, name="dataloader")

Each name becomes its own collapsible section, so a clone's Configuration tab is navigable instead of being a wall of forty keys.

For anything nested, connect_configuration is the right tool, and it round-trips both directions:

# Local run: reads the file, uploads its contents, returns the local path
# Agent run: downloads the server's (possibly edited) content to a temp file,
#            returns THAT path — so a UI edit changes behaviour with no code change
config_path = task.connect_configuration("configs/model.yaml", name="model")
cfg = yaml.safe_load(open(config_path))

# A dict works too, and comes back as a dict
cfg = task.connect_configuration({"arch": "resnet18", "head": {"dropout": 0.1}}, name="model")

Parameter references are what make pipelines and HPO able to rewire a task without editing code:

# In a pipeline controller
pipe.add_parameter("lr", 3e-4)
pipe.add_step(
    name="train",
    base_task_name="resnet18 baseline",
    parameter_override={
        "General/lr": "${pipeline.lr}",                 # from the pipeline's own params
        "General/data": "${prepare.artifacts.dataset.url}",  # from a previous step
        "Args/epochs": "${pipeline.epochs}",
    },
)

Agents in depth: modes, caching, and pinning

The agent is the piece with the most operational surface, and understanding its three phases explains every "why is setup slow" question.

Two modes, and the choice matters more than any other agent decision:

The caches are what make the second run fast. There are four, and they are separate:

agent {
  vcs_cache { enabled: true, path: "/mnt/cache/vcs" }
  pip_download_cache { enabled: true, path: "/mnt/cache/pip" }
  venvs_cache { max_entries: 20, free_space_threshold_gb: 20, path: "/mnt/cache/venvs" }
  docker_apply_cuda_version_runtime: true
  package_manager {
    type: pip
    system_site_packages: true
    extra_index_url: ["https://download.pytorch.org/whl/cu121"]
  }
}
sdk { storage { cache { default_base_dir: "/mnt/cache/clearml" } } }

Running workers, in the four shapes you will need:

# One worker per GPU, both on the same queue
clearml-agent daemon --queue gpu --gpus 0 --detached
clearml-agent daemon --queue gpu --gpus 1 --detached

# Multi-GPU jobs: pin two GPUs to one worker
clearml-agent daemon --queue gpu-big --gpus 0,1 --detached

# Queue priority: drain "urgent" before "gpu"
clearml-agent daemon --queue urgent gpu --detached

# The services queue: many light, long-lived tasks on one CPU box
clearml-agent daemon --queue services --services-mode --cpu-only --detached

--services-mode is the one worth explaining. Normally an agent runs one task at a time. In services mode it launches each task as a background container and immediately goes back to polling, so a single small machine can host dozens of pipeline controllers, HPO optimisers, and monitoring tasks concurrently. Controllers spend their lives waiting, so they must never occupy a GPU worker.

clearml-task: launching code with no SDK in it

You will often need to run a repository you cannot or should not modify: someone else's training script, an open-source benchmark, a colleague's branch. clearml-task builds a task from the outside.

clearml-task \
  --project vision \
  --name "external repo baseline" \
  --repo https://github.com/some/repo.git \
  --branch main \
  --script train.py \
  --args lr=0.001 epochs=30 \
  --requirements requirements.txt \
  --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 \
  --queue gpu

It creates a task with the repository, commit, entry point, arguments, and environment recorded, then enqueues it. The script itself never imported ClearML, yet the run is now tracked, comparable, cloneable, and re-launchable. Automatic capture of TensorBoard scalars still works, because the agent injects the SDK before execution.

The related trick is offline mode, for a machine with no network access to the server:

from clearml import Task
Task.set_offline(offline_mode=True)
task = Task.init(project_name="vision", task_name="air-gapped run")
# ... trains normally, writing everything to a local zip

# Later, from a connected machine
clearml-task --import-offline /path/to/offline-<id>.zip

Hyperparameter optimisation

Beginner ran three learning rates by hand. HyperParameterOptimizer does that properly: it clones a base task N times, sets parameters from a search strategy, enqueues them, watches the objective, and kills the runs that are losing.

from clearml import Task
from clearml.automation import (
    HyperParameterOptimizer, DiscreteParameterRange,
    UniformParameterRange, UniformIntegerParameterRange,
)
from clearml.automation.optuna import OptimizerOptuna

task = Task.init(
    project_name="vision",
    task_name="resnet18 HPO",
    task_type=Task.TaskTypes.optimizer,
    reuse_last_task_id=False,
)

optimizer = HyperParameterOptimizer(
    base_task_id="8f2c4b19e0a7d3f1c6b8a2e4d7091f3b",   # a completed, working run
    hyper_parameters=[
        UniformParameterRange("General/lr", min_value=1e-5, max_value=1e-2, log_scale=True),
        DiscreteParameterRange("General/batch_size", values=[32, 64, 128]),
        UniformIntegerParameterRange("General/layers", min_value=2, max_value=8),
    ],
    objective_metric_title="accuracy",
    objective_metric_series="val",
    objective_metric_sign="max",
    optimizer_class=OptimizerOptuna,
    execution_queue="gpu",
    max_number_of_concurrent_tasks=4,
    total_max_jobs=40,
    max_iteration_per_job=30,
    min_iteration_per_job=5,
    time_limit_per_job=45.0,          # minutes
)

optimizer.set_report_period(2.0)      # minutes between progress reports
optimizer.start()                      # or start_locally() to debug
optimizer.wait()
print(optimizer.get_top_experiments(top_k=5))
optimizer.stop()

The parameters that decide whether this saves GPU hours or wastes them:

Datasets at scale

Beginner created a version and consumed it. The machinery matters now, because dataset decisions are where storage bills and job startup times are decided.

A dataset is a task whose artifacts are chunked zip archives plus a file-list state. A child version stores only its own chunks and points at its parents:

iris 1.0.0   ── chunk_000.zip (10k rows)
   └─ 1.1.0  ── chunk_000.zip (+2k rows)          parents: [1.0.0]
        └─ 1.2.0 ── chunk_000.zip (relabelled 500)  parents: [1.1.0]

get_local_copy() walks that chain, downloads every needed chunk, and materialises a flattened view. That is cheap for three versions and expensive for three hundred, which is what squash is for:

from clearml import Dataset

# Flatten a long chain into one self-contained version
Dataset.squash(
    dataset_name="iris-flat",
    dataset_project="datasets",
    dataset_ids=["id-1.0.0", "id-1.1.0", "id-1.2.0"],
)

The operations that matter beyond create/add/finalize:

ds = Dataset.create(dataset_project="datasets", dataset_name="images",
                    parent_datasets=[parent_id])

ds.add_files("./new", wildcard="*.jpg", recursive=True)
ds.remove_files("old/*.jpg")                       # tombstoned in this child
ds.add_external_files(source_url="s3://bucket/raw/", recursive=True)  # link, do not copy
ds.upload(chunk_size=512, max_workers=8, compression="ZIP_DEFLATED")
ds.finalize()

ds.get_logger().report_table("class balance", "train", 0, table_plot=balance_df)
ds.set_metadata({"rows": 12000, "labelled_by": "vendor-a"})
print(ds.list_files(), ds.file_count, ds.total_size)

Two operational habits that matter here:

Chunk size is a real tuning knob. Default chunking produces many medium zips. Many small files benefit from larger chunks (fewer requests); a few huge files benefit from smaller ones (better parallelism on download).

Cache placement decides your job startup time. On ephemeral runners the SDK cache is empty every time, so every job re-downloads the dataset. Mount a persistent volume at sdk.storage.cache.default_base_dir and the second job on that node starts immediately.

Pipelines properly

Once pipelines become the thing you ship, Beginner's three-component example stops being enough.

The controller API, with everything you need for production:

from clearml import PipelineController

def pre_execute(pipeline, node, params):
    """Runs before a step is enqueued. Return False to skip it."""
    if node.name == "evaluate" and params.get("General/skip_eval"):
        return False
    return True

def post_execute(pipeline, node):
    """Runs after a step finishes — good place for a promotion decision."""
    acc = node.job.task.get_last_scalar_metrics() \
        .get("accuracy", {}).get("val", {}).get("last", 0)
    pipeline.get_logger().report_single_value(f"{node.name}_accuracy", acc)

pipe = PipelineController(
    name="nightly-train",
    project="vision",
    version="1.2.0",
    add_pipeline_tags=True,          # tag every step with the pipeline id
    docker="python:3.11-slim",
    repo="https://github.com/org/repo",
)

pipe.add_parameter("lr", 3e-4)
pipe.add_parameter("dataset_version", "1.1.0")

pipe.add_step(
    name="prepare",
    base_task_project="vision",
    base_task_name="make dataset",
    parameter_override={"General/version": "${pipeline.dataset_version}"},
    cache_executed_step=True,
    execution_queue="cpu",
)
pipe.add_step(
    name="train",
    parents=["prepare"],
    base_task_project="vision",
    base_task_name="resnet18 baseline",
    parameter_override={
        "General/lr": "${pipeline.lr}",
        "General/data_id": "${prepare.artifacts.dataset_id.url}",
    },
    execution_queue="gpu",
    retry_on_failure=2,
    pre_execute_callback=pre_execute,
    post_execute_callback=post_execute,
    monitor_metrics=[("accuracy", "val")],
    monitor_models=["resnet18-cls"],
)
pipe.add_step(
    name="evaluate",
    parents=["train"],
    base_task_project="vision",
    base_task_name="evaluate",
    parameter_override={"General/model_url": "${train.models.output.-1.url}"},
    execution_queue="cpu",
)

pipe.start(queue="services")        # start_locally(run_pipeline_steps_locally=True) to debug

Caching deserves a precise statement, because people expect it to be smarter than it is: a step is reused when its base task, its parameters, and its code are unchanged. Change a parameter and it reruns. Change nothing and it is skipped, and the previous run's artifacts are wired into the downstream step. It does not hash your input files, so a step reading a mutable path will reuse a stale result, which is a good argument for versioned datasets rather than paths.

Scheduling turns a pipeline into a product:

from clearml.automation import TaskScheduler

scheduler = TaskScheduler(sync_frequency_minutes=5)
scheduler.add_task(
    schedule_task_id="pipeline-controller-task-id",
    queue="services",
    minute=0, hour=2,               # 02:00 daily
)
scheduler.start()                   # itself a task, on the services queue

There is also a trigger scheduler, which is the more interesting one: it fires when a dataset or model changes rather than on a clock.

from clearml.automation import TriggerScheduler

trigger = TriggerScheduler()
trigger.add_dataset_trigger(
    schedule_task_id="pipeline-controller-task-id",
    queue="services",
    trigger_project="datasets",
    trigger_name="iris",
)
trigger.add_model_trigger(
    schedule_task_id="eval-task-id",
    queue="services",
    trigger_project="vision",
    trigger_on_tags=["candidate"],
)
trigger.start()

The model registry as a promotion workflow

Beginner registered a model and tagged it. At this level the registry is a workflow, and the tag is the contract between training and serving.

from clearml import Model, Task

candidates = Model.query_models(
    project_name="vision",
    model_name="resnet18-cls",
    tags=["candidate"],
    only_published=False,
    max_results=20,
)

def score(model):
    return model.get_metadata().get("val_accuracy", 0)

best = max(candidates, key=score)
current = Model.query_models(project_name="vision", model_name="resnet18-cls",
                             tags=["production"], max_results=1)

if not current or score(best) > score(current[0]) + 0.002:
    if current:
        current[0].tags = [t for t in current[0].tags if t != "production"] + ["archived"]
    best.tags = [t for t in best.tags if t != "candidate"] + ["production"]
    best.publish()                        # freeze it
    print("promoted", best.id)
else:
    print("no promotion: improvement below threshold")

Metadata is the part people skip, and it is what makes the gate writable at all:

from clearml import OutputModel

model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
model.update_weights(weights_filename="model.pt", auto_delete_file=False)
model.update_design(config_dict={"arch": "resnet18", "classes": 10, "input": [3, 224, 224]})
model.set_metadata("val_accuracy", 0.941, v_type="float")
model.set_metadata("dataset_version", "1.1.0")
model.set_metadata("commit", task.data.script.version_num)
model.tags = ["candidate"]

ClearML Serving

The registry answers "which weights"; Serving answers "how do requests reach them". It is a separate package that runs as a set of containers and reads the registry.

pip install clearml-serving

# 1. A serving service — itself a ClearML task on the services queue
clearml-serving create --name "vision-inference"
# → prints a service id; export it for the compose stack

# 2. Register an endpoint backed by a registry model
clearml-serving --id <service-id> model add \
  --engine triton \
  --endpoint "resnet18" \
  --model-id <model-id> \
  --preprocess "preprocess.py" \
  --input-size 1 3 224 224 --input-type float32 --input-name "input" \
  --output-size 1 10 --output-type float32 --output-name "output"

# 3. Or track a tag instead of an id — new production model, no redeploy
clearml-serving --id <service-id> model auto-update \
  --engine triton --endpoint "resnet18" \
  --project vision --name "resnet18-cls" --tags production --max-versions 2

# 4. Canary between two versions
clearml-serving --id <service-id> model canary \
  --endpoint "resnet18" --weights 0.9 0.1 --input-endpoints resnet18/1 resnet18/2

class Preprocess:
    def preprocess(self, body, state, collect_custom_statistics_fn=None):
        import numpy as np
        arr = np.array(body["image"], dtype=np.float32) / 255.0
        return arr.reshape(1, 3, 224, 224)

    def postprocess(self, data, state, collect_custom_statistics_fn=None):
        probs = data[0].tolist()
        if collect_custom_statistics_fn:
            collect_custom_statistics_fn({"max_prob": max(probs)})
        return {"class": int(max(range(len(probs)), key=probs.__getitem__)), "probs": probs}

Remote debugging with clearml-session

A task that fails only on the agent is the most annoying failure mode there is. clearml-session gives you an interactive shell inside the agent's own environment.

# A JupyterLab + VS Code server session on a GPU worker
clearml-session --queue gpu \
  --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 \
  --packages "torch==2.3.0 torchvision" \
  --git-credentials

# Attach to the environment of a specific failed task
clearml-session --queue gpu --base-task-id 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b

# Reconnect to a session you left running
clearml-session --shutdown            # when you are done — it holds a worker

It prints a local URL tunnelled to the remote machine, with Jupyter, a browser VS Code, and SSH. The environment is the same one the agent builds for a task, which is the whole point: you are debugging the actual failure, not an approximation of it.

ClearML in CI

The pattern worth building: a pull request that changes training code launches a short training run, and the check passes only if the metric clears a threshold. That makes model quality a reviewable property of the diff.

name: train
on:
  pull_request:
    paths: ['src/**', 'configs/**', 'requirements.txt']

permissions:
  contents: read

jobs:
  smoke-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0            # the agent needs a real commit to clone

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: pip

      - run: pip install -r requirements.txt

      - name: Launch a tracked run on the GPU queue
        env:
          CLEARML_API_HOST: ${{ secrets.CLEARML_API_HOST }}
          CLEARML_WEB_HOST: ${{ secrets.CLEARML_WEB_HOST }}
          CLEARML_FILES_HOST: ${{ secrets.CLEARML_FILES_HOST }}
          CLEARML_API_ACCESS_KEY: ${{ secrets.CLEARML_API_ACCESS_KEY }}
          CLEARML_API_SECRET_KEY: ${{ secrets.CLEARML_API_SECRET_KEY }}
        run: python ci/launch_and_gate.py --epochs 3 --min-accuracy 0.85

import argparse, os, sys, time
from clearml import Task

ap = argparse.ArgumentParser()
ap.add_argument("--epochs", type=int, default=3)
ap.add_argument("--min-accuracy", type=float, default=0.85)
args = ap.parse_args()

task = Task.create(
    project_name="vision/ci",
    task_name=f"pr smoke {os.environ.get('GITHUB_SHA', 'local')[:8]}",
    script="src/train.py",
    repo=os.environ["GITHUB_SERVER_URL"] + "/" + os.environ["GITHUB_REPOSITORY"],
    commit=os.environ["GITHUB_SHA"],
    requirements_file="requirements.txt",
    docker="nvidia/cuda:12.1.0-runtime-ubuntu22.04",
)
task.set_parameters({"General/epochs": args.epochs})
task.add_tags(["ci", f"pr-{os.environ.get('GITHUB_REF_NAME', 'local')}"])
Task.enqueue(task, queue_name="gpu")

while task.status in ("queued", "in_progress"):
    time.sleep(20)
    task.reload()

if task.status != "completed":
    sys.exit(f"training task {task.status}: {task.get_output_log_web_page()}")

acc = task.get_last_scalar_metrics().get("accuracy", {}).get("val", {}).get("last", 0)
print(f"::notice::val accuracy {acc:.4f} — {task.get_output_log_web_page()}")
if acc < args.min_accuracy:
    sys.exit(f"accuracy {acc:.4f} below threshold {args.min_accuracy}")

Reports and sharing

A tracked experiment is only useful if someone reads it. ClearML Reports are markdown documents that embed live task resources, so a plot in a report updates when the task does.

The pieces you need in code to make a report worth writing:

# A plot the report can embed
logger.report_table("per-class F1", "test", 0, table_plot=f1_df)

# A one-line summary a reader sees without opening anything
logger.report_text(
    f"Baseline 0.921 → 0.940 on iris 1.1.0. "
    f"Change: depth 4 → 8. Cost: +6 min/epoch."
)

# The comparison link, built from ids
print("compare:", f"{task.get_output_log_web_page().split('/output')[0]}")

Debugging, one level deeper

Beginner had a tab order. When the tabs run out, work through these.

1. Turn up SDK loggingCLEARML_LOG_LEVEL=DEBUG shows every API call the SDK makes, which is how you diagnose a hang: you can see which request never returned.
2. Read the agent's own log, not the task'sSetup failures (pip resolution, docker pull, git fetch) happen before your code runs and appear in the agent's stdout, not in the task console.
3. Reproduce the environment with clearml-session --base-task-idInteractive access to the exact environment the task got.
4. Compare a working and a broken task in the UIDetails view with "hide identical values" narrows a mystery to a package version or a single parameter in seconds.
5. Check the diff, not just the commitTwo tasks on the same commit can differ by the stored uncommitted patch. task.data.script.diff shows it.
6. Verify the server is not the problemcurl $CLEARML_API_HOST/debug.ping. Elasticsearch under pressure looks like the SDK "hanging" on metric reports.

export CLEARML_LOG_LEVEL=DEBUG          # verbose SDK
export CLEARML_AGENT_LOG_LEVEL=DEBUG    # verbose agent
clearml-agent daemon --queue gpu --foreground   # do not detach; watch it live
curl -s "$CLEARML_API_HOST/debug.ping"

from clearml import Task

good = Task.get_task(task_id="…good…")
bad  = Task.get_task(task_id="…bad…")

gp, bp = good.get_parameters(), bad.get_parameters()
for key in sorted(set(gp) | set(bp)):
    if gp.get(key) != bp.get(key):
        print(f"{key}: {gp.get(key)!r} → {bp.get(key)!r}")

print("commit:", good.data.script.version_num, "→", bad.data.script.version_num)

Putting it all together

A complete project using everything on this page. Nothing here is new. Read it as a whole and you should be able to justify every line.

.
├── .github/workflows/train.yml     # PR gate: launch, wait, threshold
├── requirements.txt                # pinned, and the agent's source of truth
├── configs/
│   ├── model.yaml                  # connect_configuration
│   └── features.yaml
├── src/
│   ├── make_dataset.py             # data_processing task → Dataset version
│   ├── train.py                    # training task, execute_remotely
│   └── evaluate.py                 # qc task, writes model metadata
├── ops/
│   ├── pipeline.py                 # PipelineController, services queue
│   ├── hpo.py                      # HyperParameterOptimizer
│   ├── promote.py                  # tag-moving gate
│   └── triggers.py                 # TriggerScheduler on the dataset
└── ci/launch_and_gate.py

import os
from clearml import Task

task = Task.init(
    project_name="vision/experiments",
    task_name="resnet18",
    task_type=Task.TaskTypes.training,
    output_uri="s3://ml-artifacts/clearml",
    reuse_last_task_id=False,
)
task.add_tags(["resnet18"])
task.set_packages("requirements.txt")                 # explicit, not detected
task.set_base_docker("nvidia/cuda:12.1.0-runtime-ubuntu22.04",
                     docker_arguments="--shm-size=8g")

import torch
from clearml import Dataset, OutputModel

opt  = task.connect({"lr": 3e-4, "epochs": 20, "seed": 42}, name="optimizer")
arch = task.connect({"layers": 18, "width": 64}, name="architecture")
data = task.connect({"dataset_version": "1.1.0"}, name="data")
model_cfg_path = task.connect_configuration("configs/model.yaml", name="model")

task.execute_remotely(queue_name="gpu")               # comment out to debug locally

set_seed(opt["seed"])
root = Dataset.get(
    dataset_project="datasets", dataset_name="iris",
    dataset_version=data["dataset_version"], alias="training data",
).get_local_copy()

logger = task.get_logger()
for epoch in range(opt["epochs"]):
    tr, va, acc = train_one_epoch(root, opt, arch)
    logger.report_scalar("loss", "train", tr, epoch)
    logger.report_scalar("loss", "val", va, epoch)
    logger.report_scalar("accuracy", "val", acc, epoch)   # the HPO objective

test_acc = test(root)
logger.report_single_value("test_accuracy", test_acc)
task.upload_artifact("predictions", predictions_df)

model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
model.update_weights(weights_filename="model.pt")
model.update_design(config_dict={"arch": "resnet18", "input": [3, 224, 224]})
model.set_metadata("val_accuracy", test_acc, v_type="float")
model.set_metadata("dataset_version", data["dataset_version"])
model.tags = ["candidate"]

from clearml import PipelineController

pipe = PipelineController(name="nightly", project="vision", version="1.2.0",
                          add_pipeline_tags=True, docker="python:3.11-slim")
pipe.add_parameter("lr", 3e-4)
pipe.add_parameter("dataset_version", "1.1.0")

pipe.add_step(name="prepare", execution_queue="cpu", cache_executed_step=True,
              base_task_project="vision", base_task_name="make dataset",
              parameter_override={"General/version": "${pipeline.dataset_version}"})

pipe.add_step(name="train", parents=["prepare"], execution_queue="gpu",
              base_task_project="vision/experiments", base_task_name="resnet18",
              parameter_override={"optimizer/lr": "${pipeline.lr}",
                                  "data/dataset_version": "${pipeline.dataset_version}"},
              retry_on_failure=2,
              monitor_metrics=[("accuracy", "val")],
              monitor_models=["resnet18-cls"])

pipe.add_step(name="evaluate", parents=["train"], execution_queue="cpu",
              base_task_project="vision", base_task_name="evaluate",
              parameter_override={"General/model_url": "${train.models.output.-1.url}"})

pipe.add_step(name="promote", parents=["evaluate"], execution_queue="services",
              base_task_project="vision", base_task_name="promote")

pipe.start(queue="services")

# The fleet
clearml-agent daemon --queue services --services-mode --cpu-only --detached
clearml-agent daemon --queue cpu --cpu-only --detached
clearml-agent daemon --queue gpu --gpus 0 --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 --detached
clearml-agent daemon --queue gpu --gpus 1 --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 --detached

# Search, then ship
python ops/hpo.py                       # 40 trials, 4 at a time, pruned
python ops/triggers.py                  # new dataset version → pipeline runs
clearml-serving --id $SVC model auto-update --endpoint resnet18 \
  --project vision --name resnet18-cls --tags production --max-versions 2

Twelve decisions in there are the whole lesson of this page:

Where you are now

You can control task creation and reuse precisely, fix requirement detection instead of fighting it, structure configuration so a clone is fully reconfigurable, tune agent modes and caches so setup is seconds rather than minutes, launch code that has no ClearML in it, run a real HPO with a correct objective and a budget, manage datasets at a scale where chunking and squashing matter, build pipelines with caching, retries, callbacks, and triggers, run the registry as a promotion workflow, serve a model that follows a tag, debug remotely inside the agent's own environment, and gate a pull request on a metric.

Senior takes every one of those topics further: the self-hosted deployment and its three data stores, the credential and access model across workspaces, multi-tenancy and quota, storage cost and retention policy, Elasticsearch and MongoDB scaling and what breaks first, backup and upgrade procedure, audit and lineage for regulated work, GPU fleet economics and autoscaling, incident playbooks for a lost server or a corrupted index, and where ClearML stops and a feature store, a warehouse, or a dedicated serving stack begins.

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than by level, in about thirty-five minutes. Fast review first, common questions at the end. Senior reviews all three.

Foundations, in one screen

ClearML tracks experiments, versions data, and re-executes runs. Task.init records the code, commit, uncommitted diff, packages, arguments, metrics, and models. Because the environment is captured, any run can be cloned, reconfigured in the UI, and re-launched on an agent. The log is executable, not just readable.

Task.init goes before every framework import, because capture works by patching at import time. task.connect is bidirectional: locally it uploads, on an agent it downloads and overrides, which is why you read from its return value. One title, several series for a comparable chart; report_single_value for anything you want as a sortable column. output_uri to object storage for anything large. finalize() is what makes a dataset version mean something.

Task creation: three entry points

task = Task.init(project_name="p", task_name="t", reuse_last_task_id=False)
child = Task.create(project_name="p", task_name="t2", script="src/train.py",
                    repo="…", branch="main", requirements_file="requirements.txt")
Task.enqueue(child, queue_name="gpu")

Two lifecycle behaviours to state without prompting. Reuse: Task.init may reuse the last empty task from the same script, which is anti-clutter behaviour that looks like a vanished run during a sweep, so pass reuse_last_task_id=False. Continuation: continue_last_task=True appends scalars from the last iteration, correct for resuming an OOM-killed job and wrong for a fresh experiment.

Task types drive UI filtering and, for controller / service / optimizer, real behaviour: training, data_processing, testing, qc, inference, controller, service, optimizer.

Automatic capture: the mechanism

Bindings patch specific functions at import time:

Requirements detection inspects sys.modules at the end of the run, mapping imported modules to distributions. Three failure modes follow:

task.set_packages("requirements.txt")
task.add_requirements("torch", "2.3.0")
task.set_base_docker("nvidia/cuda:12.1.0-runtime-ubuntu22.04",
                     docker_arguments="--shm-size=8g")

Configuration: sections, objects, references

task.connect({"lr": 3e-4}, name="optimizer")            # a named UI section
cfg_path = task.connect_configuration("configs/m.yaml", name="model")  # returns a PATH

Reference syntax, used by pipelines and HPO to rewire a task with no code change:

Agents: modes and caches

clearml-agent daemon --queue gpu --gpus 0 --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04
clearml-agent daemon --queue gpu-big --gpus 0,1          # multi-GPU job
clearml-agent daemon --queue urgent gpu                  # priority order
clearml-agent daemon --queue services --services-mode --cpu-only

system_site_packages: true in docker mode is the biggest single speed win: trust the image instead of rebuilding a venv. --services-mode launches each task as a background container and keeps polling, so one small box hosts dozens of controllers.

clearml-task and offline mode

clearml-task --project vision --name "external baseline" \
  --repo https://github.com/some/repo.git --branch main --script train.py \
  --args lr=0.001 epochs=30 --requirements requirements.txt \
  --docker nvidia/cuda:12.1.0-runtime-ubuntu22.04 --queue gpu

Builds a task from the outside for a script with no SDK in it. The agent injects the SDK, so TensorBoard capture still works. Offline mode (Task.set_offline(True)) writes a zip on an air-gapped machine, imported later with clearml-task --import-offline.

Hyperparameter optimisation

optimizer = HyperParameterOptimizer(
    base_task_id="…a completed, working run…",
    hyper_parameters=[
        UniformParameterRange("General/lr", 1e-5, 1e-2, log_scale=True),
        DiscreteParameterRange("General/batch_size", values=[32, 64, 128]),
    ],
    objective_metric_title="accuracy", objective_metric_series="val",
    objective_metric_sign="max",
    optimizer_class=OptimizerOptuna,
    execution_queue="gpu",
    max_number_of_concurrent_tasks=4, total_max_jobs=40,
    min_iteration_per_job=5, max_iteration_per_job=30, time_limit_per_job=45.0,
)
optimizer.start(); optimizer.wait(); optimizer.get_top_experiments(top_k=5)

Datasets at scale

A dataset is a task holding chunked zips plus a file list; a child stores only its delta and points at parents. get_local_copy() walks the chain and flattens it.

Two operational facts: chunk size is a real knob (many small files want larger chunks, few huge files want smaller), and cache placement decides startup time, because an ephemeral runner with an empty cache re-downloads everything every job.

Pipelines properly

pipe.add_step(
    name="train", parents=["prepare"], execution_queue="gpu",
    base_task_project="vision", base_task_name="resnet18",
    parameter_override={"optimizer/lr": "${pipeline.lr}",
                        "General/data": "${prepare.artifacts.dataset.url}"},
    cache_executed_step=True, retry_on_failure=2,
    pre_execute_callback=skip_if, post_execute_callback=record_metric,
    monitor_metrics=[("accuracy", "val")], monitor_models=["resnet18-cls"],
)

Caching keys on the base task, the parameters, and the code, not on input file contents. A step reading a mutable path will reuse a stale result, which is the strongest argument for versioned datasets over paths.

Scheduling turns a pipeline into a product: TaskScheduler for cron-style, TriggerScheduler for dataset and model triggers. A new finalized dataset version, or a model gaining the candidate tag, starts the pipeline.

The registry as a promotion workflow

model.update_design(config_dict={"arch": "resnet18", "input": [3, 224, 224]})
model.set_metadata("val_accuracy", 0.941, v_type="float")   # what the gate reads
model.tags = ["candidate"]

ClearML Serving

Remote debugging and CI

clearml-session --queue gpu --base-task-id <id> --git-credentials   # a shell in that env
clearml-session --shutdown                                          # it holds a worker

CI pattern: Task.create + Task.enqueue from the runner, poll to completion, read get_last_scalar_metrics, exit non-zero below threshold. fetch-depth: 0 so the agent can clone the commit. Credentials as CLEARML_API_* secrets, not a config file.

Debugging, one level deeper

export CLEARML_LOG_LEVEL=DEBUG
export CLEARML_AGENT_LOG_LEVEL=DEBUG
clearml-agent daemon --queue gpu --foreground

Common interview questions

1. When would you use Task.create instead of Task.init?When the process launching the task is not the process that will run it: a CI job, a controller, or launching a repository you do not modify. init means "this process is the task" and does the patching and capture; create builds a task object with the repo, commit, entry point, and environment declared explicitly, which you then enqueue.
2. Why did my run seem to disappear when I ran the script twice?Task.init reuses the last task created by the same script in the same project if that task recorded nothing: no iterations, no artifacts. It stops a debugging session producing forty empty tasks. During a sweep it is wrong, so pass reuse_last_task_id=False.
3. How does requirements detection work, and how does it fail?At the end of the run the SDK inspects sys.modules and maps imported modules to installed distributions. So a package imported inside a function that never ran is missed and the agent fails with ImportError; a detected pin carries no index URL so CUDA wheels resolve to CPU builds; and private packages resolve to names the agent cannot install. The fixes are module-level imports, set_packages, and docker mode.
4. What does system_site_packages: true do and why does it matter?It tells the agent to use the container's existing packages rather than building a fresh virtualenv and reinstalling everything. In docker mode with a prepared image, setup drops from minutes to seconds and you stop fighting CUDA wheel resolution, because the image already resolved it.
5. Explain --services-mode.Normally an agent runs one task at a time. In services mode it launches each task as a background process or container and immediately resumes polling, so one small CPU box can host dozens of pipeline controllers, HPO optimisers, and schedulers concurrently. Those tasks spend their lives waiting, so they must never occupy a GPU worker.
6. What happens if you run a pipeline controller on the GPU queue?It holds a GPU for the pipeline's entire duration while doing almost nothing, and its own GPU steps queue behind it. With a single worker that is a deadlock. Controllers, optimisers, and monitors belong on a services queue.
7. What exactly does ClearML's pipeline caching key on?The base task, the step's parameters, and the code. Not the contents of input files. So a step that reads a mutable path can be skipped even though the data changed, which is the practical argument for consuming versioned datasets rather than paths, since a version change is a parameter change.
8. How do you pass data between pipeline steps?Through artifacts and references. A step uploads an artifact; a downstream step's parameter is set to ${step.artifacts.name.url}, or in the decorator API you return a value and consume it, which is also how the DAG edge is inferred. Models use ${step.models.output.-1.url}.
9. Why is an HPO objective metric a common source of wasted compute?Because a mismatch is silent. The title and series must match a report_scalar call in the base task exactly; if they do not, the optimiser sees no objective, cannot rank trials, never prunes, and behaves like random search with no early stopping. You find out when the budget is gone. The other half is the base task itself. HPO clones it, so a broken base means every trial fails identically.
10. Which HPO knobs control cost?total_max_jobs is the budget, max_iteration_per_job is the per-trial cap and the main lever, time_limit_per_job catches hangs, and max_number_of_concurrent_tasks has to match your worker count or trials just queue. min_iteration_per_job protects slow starters from being pruned too early.
11. When would you use add_external_files?When the data is large and already in your bucket. It records paths and hashes without copying bytes, so a multi-terabyte prefix becomes a versioned dataset for almost no storage. The trade is that immutability now depends on the bucket, so it needs versioning enabled and a write-restricted prefix. Otherwise an overwrite silently changes a "finalized" version.
12. What does Dataset.squash solve?Long parent chains. get_local_copy() walks every ancestor and downloads all needed chunks, which is fine at three versions and slow at three hundred. Squashing flattens a chain into one self-contained version, trading storage for fetch speed.
13. How do you make a promotion decision reproducible?Write the numbers into the model's metadata during evaluation, then have a promotion task query candidates, compare against the current production model with an explicit margin, move the tags, and publish the winner. The decision is then a task with a log, not a person clicking in the UI.
14. How does a promotion become a deployment?The serving endpoint is registered with model auto-update against a project, name, and tag rather than a model id. Moving the production tag is what deploys, so there is no redeploy step and no code change, which also means the tag vocabulary has to be agreed and enforced.
15. What is clearml-session for?Starting an interactive Jupyter, VS Code, and SSH session inside the agent's own environment, optionally reconstructed from a specific task id. It turns the worst class of bug, "only fails on the agent", into an ordinary debugging session. It is a task, so it holds a worker until you --shutdown.
16. Design a CI check that gates a pull request on model quality.On PR, the runner does a full-depth checkout, builds a task with Task.create pinned to the commit with a short epoch budget, enqueues it to a GPU queue, polls until completion, reads the objective with get_last_scalar_metrics, prints the run URL into the log, and exits non-zero below a threshold. Credentials come from secrets as env vars, and fork builds are isolated.
17. Why can fork pull requests not be given ClearML credentials?Because enqueueing a task means executing arbitrary code on your agents. A fork PR with the workspace secret is remote code execution on your GPU fleet, plus read access to every experiment. Isolate fork builds on ephemeral agents with no other credentials, or require review before they run.
18. Where do you look when setup fails but the task console is empty?The agent's own log. Docker pulls, pip resolution, and git fetches all happen before your code starts, so those failures never reach the task console. Running the agent with --foreground and CLEARML_AGENT_LOG_LEVEL=DEBUG is how you watch it.
19. How would you handle a distributed multi-GPU training run?One task, not one per rank: initialise on rank zero and use Task.current_task() elsewhere, so scalars and artifacts land in a single readable task. On the agent side, pin multiple GPUs to one worker with --gpus 0,1 rather than running one worker per GPU.
20. Give a reason not to use ClearML.It is a stateful platform: a self-hosted server means owning MongoDB, Elasticsearch, and Redis, with backups and upgrades. If you already run Airflow and a tracking server and your data versioning is settled, adopting ClearML means replacing working components rather than filling a gap. It earns its keep when you need tracking, data versioning, and execution together, and it is a real cost when you do not.

Final self-test

• Distinguish Task.init, Task.create, and Task.clone in one sentence each.
• Explain task reuse and task continuation, and when each is correct.
• Say how requirements detection works and name its three failure modes.
• Name the agent's four caches and which one usually saves the most time.
• Explain system_site_packages: true and --services-mode.
• Say what happens when a controller runs on a GPU queue.
• State what pipeline caching keys on, and what it does not.
• Give the reference syntax for a step's parameter, artifact, and last output model.
• Say why a mismatched HPO objective is expensive, and how you would verify it.
• Explain the trade-off in add_external_files and what squash is for.
• Walk the five-stage promotion workflow and say how the tag reaches serving.
• Name the log to read when setup fails, and why the task console is empty.
• Say why fork pull requests must not hold ClearML credentials.

Part two of three. The problems at this level are no longer "why is nothing logging". They are a GPU queue full of idle sessions, an HPO that burned four hundred GPU-hours optimising nothing, a pipeline that reused a stale step, and a run that works on your laptop and dies on every agent. Start with the error table, then the practices, then the deeper material underneath.

Common errors at this level

Cumulative. Everything from Beginner still applies. These are the failures that cost hours rather than minutes.

The practices that pay off most

Practice cards

1. Measure the setup phase three waysTime the agent's setup in virtualenv mode cold, virtualenv with venv cache warm, and docker with system_site_packages. Three numbers, one obvious conclusion.
2. Break the HPO objective on purposeRun eight trials with a misspelled series. Watch the optimiser complete with no ranking and no pruning. This is the failure you must be able to recognise in ten seconds.
3. Prove pipeline caching is not input-awareA step that reads a file from disk, cached. Change the file, rerun the pipeline, and watch the step be skipped. Then switch to a versioned dataset and repeat.
4. Deadlock yourself deliberatelyOne GPU worker, controller on the GPU queue, one GPU step. Watch it hang. Move the controller to services and watch it work.
5. Land a silent override failureUse parameter_override={"lr": …} without the section prefix. Confirm no error, then find the unread parameter in the step's config.
6. Reproduce an agent-only failure in a sessionCause a lazy-import failure, then clearml-session --base-task-id that task and fix it interactively.
7. Squash a chain and measureFive-deep dataset chain, timed fetch, squash, timed fetch again.
8. Make promotion deploy itselfRegister an endpoint with auto-update on a tag, then move the tag with a script. Confirm the endpoint changed with no deploy step.
9. Launch something you did not writeclearml-task against a public repo, then clone it in the UI and change an argument.
10. Time a full-cold job on an ephemeral runnerEmpty caches, then mount a persistent cache volume and repeat. That delta is your per-job tax.

Making the agent fast

Setup time is the tax on every single run, and four things account for nearly all of it.

agent {
  vcs_cache { enabled: true, path: "/mnt/cache/vcs" }
  pip_download_cache { enabled: true, path: "/mnt/cache/pip" }
  venvs_cache { max_entries: 20, free_space_threshold_gb: 20, path: "/mnt/cache/venvs" }
  docker_apply_cuda_version_runtime: true
  package_manager {
    type: pip
    system_site_packages: true
    force_upgrade: false
    extra_index_url: ["https://download.pytorch.org/whl/cu121"]
  }
}
sdk { storage { cache { default_base_dir: "/mnt/cache/clearml" } } }

FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y --no-install-recommends \
      python3.11 python3-pip git && rm -rf /var/lib/apt/lists/*
COPY requirements.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements.txt
# The agent injects clearml at run time; system_site_packages reuses everything above.

Structuring HPO so it does not waste money

The optimiser will spend your entire GPU budget learning nothing. Four rules prevent that.

1. Validate the base task on an agent firstHPO clones it. A base task that fails on the agent produces N identical failures and a bill.
2. Copy the objective strings from the UIOpen the base task's Scalars tab and copy title and series literally. A typo is silent and total.
3. Set a budget before a strategytotal_max_jobs and max_iteration_per_job are the cost. Decide them from the hours you are willing to spend, then pick the strategy.
4. Protect slow startersmin_iteration_per_job must be past the point where a good configuration looks bad. Set it too low and pruning kills the winner.

base = Task.get_task(task_id=BASE_ID)
assert base.status == "completed", f"base task is {base.status}"

metrics = base.get_last_scalar_metrics()
assert "accuracy" in metrics and "val" in metrics["accuracy"], \
    f"objective not reported by the base task; got {list(metrics)}"

optimizer = HyperParameterOptimizer(
    base_task_id=BASE_ID,
    hyper_parameters=[...],
    objective_metric_title="accuracy",
    objective_metric_series="val",
    objective_metric_sign="max",
    optimizer_class=OptimizerOptuna,
    execution_queue="gpu",
    max_number_of_concurrent_tasks=WORKERS,     # not more than you have
    total_max_jobs=40,
    min_iteration_per_job=5,
    max_iteration_per_job=30,
    time_limit_per_job=45.0,
    save_top_k_tasks_only=10,                   # stop the sweep filling storage
)

Those two assertions at the top are worth more than any tuning advice on this page. They cost three lines and catch the two failures that account for nearly every wasted HPO budget.

Pipelines that are safe to schedule

An unattended pipeline needs four properties a hand-run one does not.

def post_execute(pipeline, node):
    task = node.job.task
    acc = task.get_last_scalar_metrics().get("accuracy", {}).get("val", {}).get("last")
    pipeline.get_logger().report_single_value(f"{node.name}_accuracy", acc or -1)
    if node.name == "train" and (acc or 0) < 0.85:
        pipeline.get_logger().report_text(f"REGRESSION: {acc} — halting promotion")
        return False        # stop the pipeline before it promotes a bad model

# Instead of: prepare() reads /data/latest
# Do this: the version is a parameter, so a data change is a cache miss
pipe.add_parameter("dataset_version", "1.1.0")
pipe.add_step(name="prepare", cache_executed_step=True,
              parameter_override={"General/version": "${pipeline.dataset_version}"})

Queue design for a small fleet

Queue layout is the cheapest scheduling tool you have, and three or four queues cover most teams.

clearml-agent daemon --queue services --services-mode --cpu-only --detached
clearml-agent daemon --queue cpu --cpu-only --detached
clearml-agent daemon --queue urgent gpu --gpus 0 --detached   # drains urgent first
clearml-agent daemon --queue gpu --gpus 1 --detached
clearml-agent daemon --queue gpu-big --gpus 2,3 --detached

Two design rules that keep this from degrading:

A worker listing several queues drains them in order. --queue urgent gpu means "always take urgent work first, otherwise take normal work", which is a priority system with no extra machinery.

Never let two things compete for one GPU. One worker per GPU, or one worker pinned to a group. Two workers claiming the same device produces out-of-memory failures that look like a model problem.

"Runs locally, fails on the agent"

The most common mid-level complaint, and it has exactly five causes. Work through them in this order.

1. The commit is not pushedThe diff travels through the server; the base commit must be fetchable. Check Execution ▸ commit against your remote.
2. A package was missedCompare Execution ▸ Installed Packages against your local pip freeze. A lazy import is the usual culprit.
3. The wrong wheel was installedCPU torch on a GPU box. The console shows CUDA not available. Docker mode with a CUDA image.
4. A system dependency is missingffmpeg, libgl, a locale, a font. Only Python is managed in virtualenv mode. This is a docker-image problem.
5. A path or environment variable only exists on your machine~/data, a mounted share, an exported token. Environment variables are deliberately not captured. Pass real config as parameters; mount real data as a dataset.

# The fastest possible diagnosis
clearml-session --queue gpu --base-task-id <failed-task-id>
# then, inside the session:
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
pip list | sort > /tmp/agent.txt          # compare against your laptop
ls -la ~/data 2>&1 | head                # does the path even exist here?

Cost hygiene at this level

Storage and GPU-hours both grow silently, and both are noticed by someone else first.

from datetime import datetime, timedelta
from clearml import Task

cutoff = (datetime.utcnow() - timedelta(days=30)).timestamp()
for t in Task.get_tasks(project_name="vision/ci"):
    if t.data.last_update.timestamp() < cutoff and t.status in ("failed", "aborted"):
        t.delete(delete_artifacts_and_models=True)

Habits worth adopting now

Pin the SDK and the agent version together. An SDK newer than the agent can record fields the agent does not understand. One pinned pair across the team removes a class of "works for me".

Put the image tag on the task, not only in the agent config. task.set_base_docker(...) records the environment on the run, so a task from three months ago still names the image it needs.

Give every launched task a tag naming its source. ci, hpo, nightly, manual. Cleanup, cost attribution, and "what is filling my bucket" all become one filter.

Report per epoch, not per batch. Per-batch reporting on a long run is thousands of Elasticsearch writes for a curve nobody zooms into that far, and it is the usual cause of lagging scalars.

Write the objective metric into the task name. resnet18 lr=1e-3 acc=0.940. The experiments table becomes self-documenting, and an HPO result list becomes readable without opening anything.

Keep a scripts/env_report.py in the repo. One task per worker pool tells you what that hardware has, and it is the first thing you run when a queue starts behaving oddly.

Treat services as production. The controllers, schedulers, and triggers living there are what make everything else automatic. A dead services agent is an outage that presents as "the nightly did not run".

# A pre-flight worth aliasing before you launch a sweep or a pipeline
python -c "
from clearml import Task
t = Task.get_task(task_id='$BASE_ID')
print(t.status)
print(t.get_last_scalar_metrics().keys())
print(t.data.script.version_num, t.data.script.repository)
"

Senior tips go deeper on every one of these: the hardening pass every self-hosted server needs, credential scoping and service accounts, multi-tenant quota and isolation, Elasticsearch and MongoDB capacity and what breaks first, backup and restore drills, retention policy that survives an audit, GPU fleet economics and autoscaling, and incident playbooks for a lost server, a corrupted index, or a model nobody can rebuild.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further (each one now has a security, scale, or ownership dimension) and adds the work you own when ClearML is your team's platform rather than your tool.

Up to this point the questions have been about making a run reproduce. From here they are different: who can enqueue code onto your GPU fleet, whether a model from eighteen months ago can still be rebuilt and proven, what the platform costs and who pays for it, which component breaks first when the team triples, what happens when Elasticsearch fills up on a Friday night, and where ClearML stops being the right tool.

Where this picks up

I am starting with the trust model, because every other decision in this track depends on it.

The trust model: enqueue is code execution

The single most important sentence about ClearML security: enqueueing a task is arbitrary code execution on your agents. A task carries a repository, a commit, a diff, a package list, and an entry point. An agent claims it and runs it. Anyone who can enqueue can run code on your GPU fleet, with whatever credentials that fleet holds.

That reframes the whole access question around "who can execute" instead of "who can see the experiments".

Three properties of ClearML's model make this sharper than it first looks:

An access/secret pair is workspace-wide by default. It is not scoped to a project. A leaked researcher key reads every experiment, every dataset pointer, and every model in the workspace, and can enqueue.

The uncommitted diff is stored on the server. That is the feature that makes a dirty working tree reproducible. It also means anything sitting in your working tree at run time (a .env you forgot to gitignore, a notebook with a pasted token) is now in the task record, readable by everyone with workspace access, and captured in your backups.

Agents hold ambient credentials. A GPU worker typically has cloud credentials so it can read datasets and write artifacts. Any task that lands on it inherits them. That is why fork pull requests are dangerous and why queue-level isolation is a security control, not tidiness.

The access shape that works:

services     ← enqueue: platform team + schedulers only.  Creds: registry read/write
gpu          ← enqueue: researchers.                      Creds: data read, artifact write
cpu          ← enqueue: researchers.                      Creds: data read, artifact write
ci           ← enqueue: CI on protected branches.         Creds: artifact write only
ci-untrusted ← enqueue: fork PRs after approval.          Creds: NONE. Ephemeral hosts
serving      ← enqueue: platform team only.               Creds: registry read

Credentials, service accounts, and rotation

At Mid the goal was "not committed". At this level the goal is short-lived, scoped, and attributable.

# Per-purpose credentials, never shared. Each is separately revocable.
# Settings → Workspace → App Credentials, one per:
#   - each CI pipeline
#   - each agent pool
#   - each serving service
#   - each human (personal, never in automation)

# systemd unit or Kubernetes secret — nothing on disk in a home directory
Environment=CLEARML_API_HOST=https://clearml.internal/api
Environment=CLEARML_WEB_HOST=https://clearml.internal
Environment=CLEARML_FILES_HOST=https://clearml.internal/files
Environment=CLEARML_API_ACCESS_KEY=...      # from the secret store
Environment=CLEARML_API_SECRET_KEY=...
Environment=CLEARML_WORKER_ID=gpu-a100-01   # attributable in Workers & Queues

For the cloud credentials the agent needs, the goal is that there is nothing to store at all:

Self-hosting: what you own

The hosted server hides five components. Self-hosting means owning all of them, plus their failure modes.

services:
  elasticsearch:
    image: elasticsearch:8.x
    environment:
      ES_JAVA_OPTS: "-Xms4g -Xmx4g"        # ≤ 50% of container RAM, and never > 31g
      bootstrap.memory_lock: "true"
      cluster.routing.allocation.disk.watermark.low: "90%"
      cluster.routing.allocation.disk.watermark.high: "95%"
    volumes: ['/opt/clearml/data/elastic_7:/usr/share/elasticsearch/data']

  mongo:
    image: mongo:6.x
    volumes: ['/opt/clearml/data/mongo_4/db:/data/db']

  apiserver:
    image: allegroai/clearml:latest
    environment:
      CLEARML__services__async_urls_delete__enabled: "true"   # actually free storage
    depends_on: [mongo, elasticsearch, redis]

# The three health checks worth having on a dashboard
curl -s localhost:8008/debug.ping
curl -s 'localhost:9200/_cluster/health?pretty' | grep -E 'status|unassigned'
curl -s 'localhost:9200/_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc' | head

# Release a read-only block after clearing disk space
curl -X PUT 'localhost:9200/_all/_settings' -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

Non-negotiables for a server anyone else depends on:

Backup, restore, and upgrades

You have backups. Whether you have ever restored one is the open question.

A ClearML backup has three parts, and taking only the first is the common mistake:

1. MongoDBmongodump of the backend and auth databases. This is the system of record: tasks, projects, models, users, queues.
2. ElasticsearchA snapshot to a repository. Without it a restore gives you tasks with no scalars, no plots, and no console logs.
3. Artifact storageYour bucket, with versioning enabled. If you still use the file server, its volume.

# 1. Mongo
docker exec clearml-mongo mongodump --archive=/tmp/mongo.gz --gzip \
  --db backend --db auth
docker cp clearml-mongo:/tmp/mongo.gz ./backup/

# 2. Elasticsearch — register a repository once, then snapshot
curl -X PUT 'localhost:9200/_snapshot/backups' -H 'Content-Type: application/json' -d '
{"type":"fs","settings":{"location":"/usr/share/elasticsearch/snapshots"}}'
curl -X PUT "localhost:9200/_snapshot/backups/snap-$(date +%F)?wait_for_completion=true"

# 3. Config
cp -r /opt/clearml/config ./backup/config

The restore drill is the part that matters, and it must be rehearsed on a separate host:

docker compose down
docker exec -i clearml-mongo mongorestore --archive --gzip < ./backup/mongo.gz
curl -X POST "localhost:9200/_snapshot/backups/snap-2026-08-01/_restore"
docker compose up -d
# Then verify, in this order:
#   1. Can you log in?
#   2. Does a known task still show its scalars?          ← proves Elastic restored
#   3. Can you download a known artifact?                 ← proves storage config
#   4. Can an agent claim a queued task?                  ← proves the fleet reconnects

Upgrades follow the same discipline, because a server upgrade runs data migrations:

Multi-tenancy, RBAC, and quota

One workspace shared by four teams degrades predictably: a flat project list nobody can search, a GPU queue one team monopolises, and no way to tell who spent the storage.

The open-source server has a single workspace and no role-based access control: every user with credentials can see and do everything. That is a real limitation and you should state it plainly rather than discover it during a security review. RBAC, multiple workspaces, and per-user permissions are paid-tier features.

What you can do with the open-source server:

projects:
  platform/…            ← pipelines, schedulers, serving control
  team-vision/…         ← their experiments
  team-nlp/…
  datasets/…            ← shared, write-restricted by process
queues:
  vision-gpu (2 workers) · nlp-gpu (2 workers) · shared-gpu (2, spot) · services (1)
storage:
  s3://ml-artifacts/team-vision/…   lifecycle: 90d → IA, 365d → delete
  s3://ml-artifacts/team-nlp/…
  s3://ml-datasets/…                versioning on, write-restricted

# In a shared internal wrapper every team imports:
def init_task(project: str, name: str, **kwargs):
    team = os.environ["ML_TEAM"]                       # from the environment
    task = Task.init(project_name=f"{team}/{project}", task_name=name,
                     output_uri=f"s3://ml-artifacts/{team}/clearml", **kwargs)
    task.add_tags([f"team:{team}", f"cost-centre:{os.environ['ML_COST_CENTRE']}"])
    return task

Cost: what the platform costs

Four cost centres, and only one of them is usually noticed.

Idle GPU time is nearly always the largest single line, and it is invisible unless you measure it:

from datetime import datetime, timedelta
from clearml import Task

week = datetime.utcnow() - timedelta(days=7)
tasks = Task.get_tasks(task_filter={"status": ["completed", "failed", "aborted"]})

gpu_seconds, wasted = 0.0, 0.0
for t in tasks:
    if t.data.last_update < week or "gpu" not in (t.data.execution.queue or ""):
        continue
    runtime = (t.data.completed - t.data.started).total_seconds()
    gpu_seconds += runtime
    if t.status in ("failed", "aborted"):
        wasted += runtime

print(f"GPU hours: {gpu_seconds/3600:.1f}  wasted on failed/aborted: {wasted/3600:.1f}")

from clearml.automation.auto_scaler import AutoScaler, CloudDriver

AutoScaler(
    hyper_params=AutoScaler.Settings(
        max_idle_time_min=10,          # terminate an idle instance after 10 minutes
        polling_interval_time_min=2,
        max_spin_up_time_min=25,
        workers_prefix="dyn",
    ),
    driver=CloudDriver(...),           # AWS/GCP driver with instance types per queue
).start()

Scale: which component breaks first

Growth arrives in four dimensions, and each one breaks something different. Knowing the order lets you fix the right thing.

Reporting discipline is the cheapest and most effective lever, and it is a code change rather than an infrastructure one:

# Bad: 500k Elasticsearch writes for a curve nobody zooms into that far
for step, batch in enumerate(loader):
    logger.report_scalar("loss", "train", loss, step)

# Good: per epoch, plus a moving average within the epoch
for epoch in range(epochs):
    epoch_loss = train_one_epoch()
    logger.report_scalar("loss", "train", epoch_loss, epoch)

# When you genuinely need intra-epoch resolution, sample it
if step % 100 == 0:
    logger.report_scalar("loss", "train_batches", loss, step)

Retention is what keeps Mongo and Elastic bounded. It has to be a scheduled task, because nobody does it by hand:

from datetime import datetime, timedelta
from clearml import Task

RULES = [
    ("vision/ci",          30,  ("failed", "aborted", "completed")),
    ("vision/hpo",         60,  ("failed", "aborted")),
    ("vision/experiments", 365, ("failed", "aborted")),
]

for project, days, statuses in RULES:
    cutoff = datetime.utcnow() - timedelta(days=days)
    for t in Task.get_tasks(project_name=project):
        if t.data.last_update >= cutoff or t.status not in statuses:
            continue
        if set(t.get_tags()) & {"keep", "published", "paper"}:
            continue                       # an explicit exemption, always needed
        print(f"delete {t.id} {t.name}")
        # t.delete(delete_artifacts_and_models=True)   ← enable after a dry run

Reproducibility that survives eighteen months

ClearML records the commit, the diff, and the Python packages. That is necessary and not sufficient. Everything below the Python layer is unpinned unless you pin it.

task.set_base_docker(
    # A digest, not a tag. Tags move; digests do not.
    "myregistry/train@sha256:9f2c4b19e0a7d3f1c6b8a2e4d7091f3b8a2e4d7091f3b8a2e4d7091f3b8a2e",
    docker_arguments="--shm-size=8g",
)
task.set_packages("requirements.lock")           # hash-pinned, committed

env = task.connect({"seed": 42, "deterministic": True}, name="reproducibility")
torch.manual_seed(env["seed"]); np.random.seed(env["seed"]); random.seed(env["seed"])
if env["deterministic"]:
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False

data = Dataset.get(dataset_project="datasets", dataset_name="iris",
                   dataset_version=params["dataset_version"],
                   alias="training data").get_local_copy()

The part that makes it a claim rather than a hope is a scheduled rebuild of a past release:

from clearml import Task

RELEASE_TASK_ID = "…the task behind the current production model…"
TOLERANCE = 0.002

original = Task.get_task(task_id=RELEASE_TASK_ID)
clone = Task.clone(source_task=original, name=f"rebuild check {original.name}")
Task.enqueue(clone, queue_name="gpu")
clone.wait_for_status(status=("completed", "failed", "aborted"))

def acc(t):
    return t.get_last_scalar_metrics().get("accuracy", {}).get("val", {}).get("last", 0)

delta = abs(acc(clone) - acc(original))
Task.current_task().get_logger().report_single_value("rebuild_delta", delta)
assert clone.status == "completed", f"rebuild failed: {clone.get_output_log_web_page()}"
assert delta <= TOLERANCE, f"rebuild drifted by {delta:.4f}"

Lineage, provenance, and audit

For regulated work, or any post-incident review, you need to answer four questions about a deployed model without a conversation:

1. What code produced it?The model links to its task; the task records the repository, commit, and diff.
2. What data produced it?The task's configuration records the dataset id via alias; the dataset records its parents.
3. What configuration produced it?The task's hyperparameters and configuration objects, as executed.
4. Who approved it and when?The promotion task: who ran it, what it compared, what threshold it applied, when the tag moved.

The chain only exists if you build it, and a script is what proves it does:

from clearml import Model, Task

def lineage(model_id: str) -> dict:
    m = Model(model_id=model_id)
    t = Task.get_task(task_id=m.task)
    params = t.get_parameters()
    datasets = {k: v for k, v in params.items() if "dataset" in k.lower()}
    return {
        "model": {"id": m.id, "name": m.name, "tags": m.tags,
                  "published": m.published, "metadata": m.get_metadata()},
        "task": {"id": t.id, "name": t.name, "user": t.data.user,
                 "started": str(t.data.started), "status": t.status},
        "code": {"repo": t.data.script.repository,
                 "commit": t.data.script.version_num,
                 "entry": t.data.script.entry_point,
                 "dirty": bool(t.data.script.diff)},
        "data": datasets,
        "container": t.data.container,
    }

Deleting data from a content-addressed, immutable store

An erasure request under GDPR or a similar regime meets ClearML's design head-on. Datasets are immutable by intent, versions have children, and tasks reference them. This is the hardest problem in this track and it is primarily architectural, not operational.

What deletion has to touch:

The architecture that makes erasure tractable:

1. Version derived data, not raw personal dataPseudonymise or aggregate before it enters a ClearML Dataset. The versioned artefact then holds no directly identifying records.
2. Keep the identity mapping outside ClearMLIn a system built for deletion: a database with row-level deletes and an audit trail.
3. Crypto-shredding where raw data must be versionedEncrypt per subject and destroy the key. The bytes remain, the content is unrecoverable, and the deletion is provable.
4. Never print records to the consoleConsole output goes to Elasticsearch and into your backups, and it is the most commonly forgotten copy.
5. Bound your backup retention deliberatelyAn indefinite backup is an indefinite copy. Erasure obligations and retention policy have to be reconciled on paper before they collide.

from clearml import Dataset, Task

def find_references(dataset_id: str):
    """Every task whose parameters mention this dataset id."""
    hits = []
    for t in Task.get_tasks():
        if dataset_id in str(t.get_parameters()):
            hits.append((t.id, t.name, t.status))
    return hits

# 1. Enumerate the blast radius BEFORE deleting anything
refs = find_references("…dataset-id…")
print(f"{len(refs)} tasks reference this dataset")

# 2. Enumerate child dataset versions that inherit from it
children = Dataset.list_datasets(dataset_project="datasets", partial_name="iris")

# 3. Only then decide: delete, crypto-shred, or retrain

Incident playbooks

Four incidents you should be able to work through without improvising.

The server is gone.

1. Agents keep runningTasks in progress continue and buffer their reports. You have some time.
2. Stand up the server from the pinned imageSame version, not latest, because a different version means an unplanned migration on top of an incident.
3. Restore Mongo, then ElasticsearchMongo first: it is the system of record. Elastic second, for metrics and logs.
4. Verify in orderLogin → a known task's scalars → an artifact download → an agent claiming a queued task.
5. ReconcileTasks that finished during the outage may show as running. Mark them appropriately and note the gap.

Elasticsearch went read-only. Free disk (delete old indices or expand the volume), release the read-only block, then fix the cause: reporting rate, retention, or capacity. Symptom to recognise: metrics stop while training continues.

A model cannot be rebuilt. Run the lineage script; find the unpinned layer. It is nearly always the base image tag or an unpinned wheel. If the artifacts are gone, check whether a retention job deleted the task, then check bucket versioning. Record what you could not recover.

Someone enqueued something they should not have. Abort the task, revoke the credential that enqueued it, audit what that queue's agents could reach with their ambient credentials, and rotate anything exposed. Then fix the queue-level isolation that allowed it. Treat this as a credential compromise, not a mistake.

Where ClearML stops

Knowing the boundaries is a senior skill, and being able to name them is a strong interview signal.

Running ClearML as a platform

When ClearML is shared infrastructure, your job changes from using it to operating it. Five obligations, and they are what people mean by "platform ownership".

The monitoring set that matters, in priority order:

from clearml import Task
from clearml.backend_api.session.client import APIClient

client = APIClient()
report = Task.current_task().get_logger()

for q in client.queues.get_all():
    depth = len(q.entries or [])
    report.report_single_value(f"queue_depth_{q.name}", depth)

workers = client.workers.get_all()
report.report_single_value("workers_online", len(workers))
report.report_single_value(
    "workers_idle",
    sum(1 for w in workers if not getattr(w, "task", None)),
)

The review checklist

Run this against any ClearML setup, yours or someone else's. It finds something every time.

The complete picture

Everything from all three levels in one platform. Read it as a whole; you should be able to justify every line and name which level it came from.

.
├── .github/workflows/
│   ├── train-pr.yml            # PR gate: create, enqueue to ci, threshold
│   └── image.yml               # builds the training image, tags by requirements hash
├── requirements.lock           # hash-pinned
├── Dockerfile.train            # CUDA + system libs + wheels; digest recorded on tasks
├── src/{make_dataset,train,evaluate}.py
├── platform/
│   ├── clearml_wrapper.py      # project prefix, output_uri, tags, image — the governance
│   ├── retention.py            # scheduled, dry-run-first, exempts published models
│   ├── rebuild_check.py        # weekly rebuild of the production release
│   ├── platform_report.py      # utilisation, queue depth, storage by team
│   ├── lineage.py              # code + data + config + approval for a model id
│   └── promote.py              # threshold, tag move, publish — a task, not a click
├── ops/{pipeline,hpo,triggers}.py
└── runbooks/
    ├── server-lost.md
    ├── elastic-readonly.md
    ├── model-unrebuildable.md
    └── unauthorised-enqueue.md

import os
from clearml import Task

IMAGE = "myregistry/train@sha256:9f2c4b19e0a7d3f1c6b8a2e4d7091f3b8a2e4d7091f3b8a2e4d70"

def init_task(project: str, name: str, queue: str | None = None, **kwargs) -> Task:
    team = os.environ["ML_TEAM"]
    task = Task.init(
        project_name=f"{team}/{project}",
        task_name=name,
        output_uri=f"s3://ml-artifacts/{team}/clearml",   # never the file server
        reuse_last_task_id=False,                         # one task per run
        **kwargs,
    )
    task.add_tags([f"team:{team}", f"cost-centre:{os.environ['ML_COST_CENTRE']}"])
    task.set_packages("requirements.lock")                # pinned, not detected
    task.set_base_docker(IMAGE, docker_arguments="--shm-size=8g")   # by digest
    if queue:
        task.execute_remotely(queue_name=queue)
    return task

from platform.clearml_wrapper import init_task

task = init_task("experiments", "resnet18", queue="gpu")

import torch
from clearml import Dataset, OutputModel

opt  = task.connect({"lr": 3e-4, "epochs": 20}, name="optimizer")
repro = task.connect({"seed": 42, "deterministic": True}, name="reproducibility")
data = task.connect({"dataset_version": "1.1.0"}, name="data")

set_seed(repro["seed"], deterministic=repro["deterministic"])
root = Dataset.get(dataset_project="datasets", dataset_name="iris",
                   dataset_version=data["dataset_version"],
                   alias="training data").get_local_copy()

logger = task.get_logger()
for epoch in range(opt["epochs"]):
    tr, va, acc = train_one_epoch(root, opt)
    logger.report_scalar("loss", "train", tr, epoch)      # per epoch, not per batch
    logger.report_scalar("accuracy", "val", acc, epoch)   # the HPO objective

test_acc = test(root)
logger.report_single_value("test_accuracy", test_acc)

model = OutputModel(task=task, name="resnet18-cls", framework="PyTorch")
model.update_weights(weights_filename="model.pt")
model.set_metadata("val_accuracy", test_acc, v_type="float")
model.set_metadata("dataset_version", data["dataset_version"])
model.tags = ["candidate"]                                 # the gate promotes, not you

# services — production. Everything automatic lives here. Liveness-alerted.
clearml-agent daemon --queue services --services-mode --cpu-only --detached

# cpu / gpu — researcher work. Data read + artifact write, no delete.
clearml-agent daemon --queue cpu --cpu-only --detached
clearml-agent daemon --queue gpu --gpus 0 --docker $IMAGE --detached
clearml-agent daemon --queue gpu --gpus 1 --docker $IMAGE --detached

# ci — protected branches only. Artifact write, nothing else.
clearml-agent daemon --queue ci --docker $IMAGE --detached

# ci-untrusted — fork PRs, after approval. Ephemeral hosts, NO credentials.
clearml-agent daemon --queue ci-untrusted --docker $IMAGE --detached

# On the services queue, all of them tasks with logs:
python platform/retention.py         # daily, dry-run gated
python platform/rebuild_check.py     # weekly — reproducibility as a monitored property
python platform/platform_report.py   # weekly — utilisation, depth, storage by team
python ops/triggers.py               # dataset version → pipeline

Eighteen decisions in there span the whole series:

Where the series leaves you

Across three levels you have gone from two lines in a script to owning a platform. You can track an experiment automatically and log deliberately; version data incrementally and consume it reproducibly; clone and re-launch any run on remote hardware; run HPO with a verified objective and a budget; build cached, retried, triggered pipelines; drive a registry as a promotion workflow that deploys itself; and operate the whole thing with a trust model, a cost model, tested backups, monitored reproducibility, an audit chain, and runbooks somebody else can follow.

The through-line of all three levels is a single idea: a recorded run that can be executed again is worth more than a log of one that cannot. Everything else (the agents, the queues, the registry, the pinning, the backups) exists to keep that property true as the number of runs, people, and months grows.

Part three of three, and the one to read if you only read one. A cumulative review of the entire series (foundations, platform machinery, and the governance and scale work a senior owns) organised by topic rather than by level. About fifty minutes. Fast review first, common questions at the end.

Part one: foundations

ClearML tracks experiments, versions datasets, and re-executes runs. Task.init records the code, commit, uncommitted diff, packages, arguments, metrics, and models. Because the environment is captured, any run can be cloned, reconfigured in the UI, and re-launched on an agent. The log is executable, not just readable.

Five components: SDK in your process; API server (MongoDB for metadata, Elasticsearch for metrics and logs, Redis for ephemeral state); web server; file server; agents pulling named queues.

The rules that never change

Task.init before every framework import, because capture is import-time patching. Read parameters from connect's return value. It is bidirectional and overrides on an agent. One title, several series for one comparable chart; report_single_value for sortable columns. output_uri to object storage for anything large. finalize() is what makes a dataset version mean anything. Clone, do not rerun, since a completed task is a record.

Statuses and the loop

Clone → edit parameters → enqueue → agent rebuilds the env, clones the commit, applies the diff, runs the entry point. execute_remotely does the register-and-enqueue part from inside the script, then exits locally.

The traps

Part two: platform machinery

Three entry points

Reuse: init may reuse the last empty task from the same script: anti-clutter, wrong during a sweep, so reuse_last_task_id=False. Continuation: continue_last_task=True appends from the last iteration; right for resuming an OOM kill, wrong for a fresh run. Distributed: one task, init on rank 0, Task.current_task() elsewhere.

Capture and requirements

Bindings patch TensorBoard add_*, torch.save, Model.save, joblib.dump, XGBoost/LightGBM saves, matplotlib, argparse, and hydra. Requirements come from inspecting sys.modules at the end of the run, which produces three failures: lazy imports missed, CUDA wheels resolved as CPU builds, private packages unresolvable. Fixes: set_packages, add_requirements, and docker mode.

task.set_packages("requirements.lock")
task.set_base_docker("registry/train@sha256:…", docker_arguments="--shm-size=8g")

Agents, caches, queues

system_site_packages: true in docker mode is the biggest speed win: trust the image. --services-mode runs many light tasks concurrently. Controllers, optimisers, and schedulers never go on a GPU queue: a controller mostly sleeps and would hold the GPU while its own steps queue behind it. A worker listing --queue urgent gpu drains in order, which is a priority system for free.

HPO

HyperParameterOptimizer(
    base_task_id=BASE, hyper_parameters=[...],
    objective_metric_title="accuracy", objective_metric_series="val",
    objective_metric_sign="max", optimizer_class=OptimizerOptuna,
    execution_queue="gpu", max_number_of_concurrent_tasks=WORKERS,
    total_max_jobs=40, min_iteration_per_job=5, max_iteration_per_job=30,
    time_limit_per_job=45.0, save_top_k_tasks_only=10,
)

A mismatched objective is silent and total: no ranking, no pruning, random search with a full budget spent. The base task must itself be a working completed run, or all N trials fail identically. save_top_k_tasks_only is what stops a sweep becoming a storage incident.

Datasets and pipelines

Registry to serving

Tag vocabulary must be agreed and set from code only: prod vs production means serving silently keeps the old model. clearml-session --base-task-id gives a shell in the exact failing environment, and holds a worker until --shutdown.

Part three: trust, scale, and ownership

The trust model

Enqueueing a task is arbitrary code execution on your agents. That reframes access from "who can see" to "who can execute".

Three sharpening properties: credentials are workspace-wide by default (a leaked researcher key reads everything and can enqueue); the uncommitted diff is stored on the server (so an untracked .env in your working tree is now in the task record and in backups); and agents hold ambient cloud credentials that any task landing on them inherits.

Queues as a security boundary

services     ← platform team + schedulers.   Creds: registry read/write
gpu / cpu    ← researchers.                  Creds: data read, artifact write
ci           ← protected branches.           Creds: artifact write only
ci-untrusted ← fork PRs after approval.      Creds: NONE. Ephemeral hosts
serving      ← platform team only.           Creds: registry read

Credentials: one per purpose, so revocation is surgical and the audit log names a caller. CLEARML_WORKER_ID set explicitly. Workload identity (IRSA, GKE workload identity, instance profiles) for cloud access, so there is nothing on disk to rotate or leak.

Self-hosting

Backups and upgrades

Three parts: mongodump of backend + auth; an Elasticsearch snapshot; and the artifact bucket with versioning. A Mongo-only backup restores a complete-looking system with empty Scalars tabs. That is the mistake. Verify a restore in order: login → a known task's scalars → an artifact download → an agent claiming a task. Upgrades: read notes for every skipped version, back up all three, pin the tag, server before agents, pair SDK and agent versions.

Multi-tenancy

The open-source server is one workspace with no RBAC. Everyone with credentials can do everything. Say that plainly. What you can do: nested projects by convention, separate queues with fixed workers as the quota, separate agent pools with separate credentials as the real isolation, bucket prefixes per team with their own lifecycle, mandatory team and cost-centre tags. Hard isolation means a server per tenant. A twenty-line internal wrapper turns every one of those conventions into a default.

Cost

Idle GPU is nearly always the largest line: abandoned sessions, runs that fail late, sweeps with no pruning, workers idle between bursts. Spot instances need retry_on_failure plus checkpointing and continue_last_task, or you have bought cheaper failures.

Scale, in order of what breaks

Retention must be a scheduled task, dry-run first, exempting published models, keep-tagged tasks, and anything a registered model still references. Deleting a task can delete a production model's weights.

Reproducibility

"We are reproducible" is a claim until a weekly rebuild check clones a past release, reruns it, and compares the metric within a tolerance. The first run will fail; that is the point.

Lineage and erasure

Four audit questions: what code (task's repo, commit, diff), what data (dataset id via alias, and its parents), what configuration (parameters as executed), and who approved it (the promotion task's user, timestamp, comparison, and threshold). Promotion must be a task, never a manual tag move, or there is no approval record.

Erasure against an immutable content-addressed store touches dataset chunks, child versions, task artifacts, debug samples, console logs in Elasticsearch, backups, and weights, which you cannot meaningfully erase. The architectural fix is upstream: version derived or pseudonymised data, keep the identity mapping in a system built for deletes, and crypto-shred where raw data must be versioned.

Where ClearML stops

ClearML is strongest as the experiment-to-model spine; weakest as a general orchestrator, a feature store, and a serving mesh.

Common interview questions

1. Design ClearML for a team of thirty across three squads.Self-hosted with TLS at a proxy, artifacts in S3 with per-team prefixes and lifecycle rules, separate volumes for Mongo and Elastic with disk alerts at 80%. Queues split by trust and hardware: services for controllers and schedulers, per-team GPU queues with fixed worker counts as quota, ci for protected branches, ci-untrusted on ephemeral credential-free hosts. One credential per purpose, agents on workload identity. A shared wrapper enforcing project prefix, output_uri, tags, and image digest. Scheduled retention, a weekly rebuild check, and a published platform report. Then say plainly that the open-source server has no RBAC, so isolation is queues, credentials, and buckets rather than permissions.
2. Why is enqueue permission a security decision?Because a task carries a repository, commit, diff, package list, and entry point, and an agent runs it. Enqueue is remote code execution on your fleet, inheriting whatever ambient cloud credentials that agent holds. So queues become the security boundary, each pool gets its own credentials, and fork pull requests get an isolated ephemeral pool with nothing attached.
3. What is the risk in ClearML storing your uncommitted diff?It is a good feature, since a dirty working tree is still reproducible. It also means anything present but untracked at run time, like a .env or a service-account JSON, is captured into the task record, visible to everyone with workspace access, and persisted in backups. Remediation is deleting the task and rotating the credential anyway, so prevention is gitignore discipline plus a pre-run check.
4. Which ClearML component breaks first as you grow, and why?Elasticsearch, on reporting rate. Metrics and console logs grow monotonically and never shrink; crossing the high disk watermark marks indices read-only, and the symptom is that metric reporting silently stops while training carries on. Second is MongoDB on task count, which shows up as a slow experiments table. The fixes are per-epoch reporting, retention, and capacity, in that order, because the first is free.
5. What does a complete ClearML backup contain?Three parts: a mongodump of the backend and auth databases, an Elasticsearch snapshot, and the artifact bucket with versioning enabled. Mongo alone restores every task, model, and parameter with empty Scalars tabs, because scalars, plots, and console output live in Elasticsearch. The backup is only real once you have restored it and verified login, a known task's scalars, an artifact download, and an agent claiming a task.
6. How do you make a model from a year ago rebuildable?ClearML records code and Python packages; everything below that is unpinned unless you pin it. Base image by digest rather than tag, a hash-pinned lock file, the seed as a connected parameter, determinism flags recorded, and the dataset consumed by version with an alias. Then a scheduled job that clones a past release, reruns it, and compares the metric within a tolerance, because otherwise reproducibility degrades silently and you find out during an incident.
7. A customer asks you to delete their data. Walk me through it.First enumerate: dataset chunks, every child version inheriting them, task artifacts, debug samples, console logs in Elasticsearch, and backups. Then the honest part: trained weights retain the influence and for some model classes information is extractable, so if the regime treats a model as personal data the only real remedy is retraining. Which is why the architecture matters more than the runbook: version derived or pseudonymised data, keep the identity mapping in a system built for row deletes, and crypto-shred if raw data must be versioned.
8. How do you control GPU cost?Measure first: GPU-hours consumed versus hours that produced a completed run. The waste is usually abandoned interactive sessions, runs that fail late, and sweeps that never prune. So: session timeouts and a weekly report naming owners, fail-fast validation of config and data access in the first minute, verified HPO objectives with iteration caps and `save_top_k_tasks_only`, and an autoscaler with a scale-to-zero floor. Spot only if the job checkpoints and the pipeline retries.
9. How do you handle multi-tenancy on the open-source server?By being honest that it has one workspace and no RBAC. Isolation comes from queues with per-pool credentials, bucket prefixes enforced by cloud IAM, nested projects and mandatory tags by convention, and worker counts as quota. A shared wrapper makes those defaults rather than documentation. If genuine hard isolation is required (different legal entities, different data classifications) it is the paid tier or a server per tenant, and pretending otherwise fails a security review.
10. Why must a pipeline controller not run on a GPU queue?A controller mostly sleeps while its steps execute. On a GPU worker it occupies that GPU for the pipeline's whole duration doing nothing, while its own GPU steps queue behind it, which with one worker is a deadlock. Controllers, optimisers, schedulers, and the serving control plane all belong on a services-mode agent, which launches tasks in the background and keeps polling.
11. What is the failure mode of pipeline caching?It keys on the base task, the parameters, and the code, not on the contents of input files. A step reading a mutable path has identical parameters when the data changes, so it is skipped, downstream steps consume a stale artifact, and the pipeline is green. The fix is to consume a versioned dataset, which turns a data change into a parameter change and therefore a cache miss.
12. How does a promotion become a deployment without a deploy step?The serving endpoint is registered with model auto-update against a project, name, and tag rather than a model id, so moving the production tag is the deploy. That makes the tag vocabulary load-bearing: prod versus production means the query finds nothing and the old model silently stays. Fix four tags, set them only from the promotion task, and publish the winner so it cannot be swapped.
13. What monitoring does a ClearML platform need, in priority order?Elasticsearch disk above 80%, because read-only is silent. Services-agent liveness, because everything automatic lives there and a dead agent presents as "the nightly did not run" with no error. Queue depth sustained over an hour, meaning a dead worker or a runaway sweep. Then Mongo disk, API 5xx rate, the rebuild-check result, and weekly GPU utilisation. Task failures are deliberately not on that list. Those belong to their authors.
14. You inherit a ClearML setup. What do you check first?Whether fork pull requests can enqueue anywhere, and whether backups have ever been restored. Those two are an RCE path and an unrecoverable data loss. Then artifacts on the file server instead of S3, an unpinned server image, missing Elastic disk alerts, no retention job, base images by tag rather than digest, and whether promotion is a task or a click.
15. How would you audit a deployed model?One command, not a conversation: from the model id, resolve the producing task, then report repository, commit, whether the tree was dirty, the container digest, the parameters as executed, the dataset ids from the parameters, the model metadata and tags, and the promotion task's user, timestamp, comparison, and threshold. Generated, not narrated, and the usual gaps are a missing data chain and no approval record.
16. ClearML versus MLflow plus Airflow plus DVC?ClearML puts tracking, data versioning, and execution in one system, so a recorded run is re-executable and a promotion can deploy itself. The cost is a stateful platform: MongoDB, Elasticsearch, Redis, backups, and upgrades that run migrations. If you already run Airflow and a tracking server and your data versioning is settled, adopting ClearML replaces working components rather than filling a gap. It earns its place when you need the spine, and it is real operational cost when you do not.
17. How do you keep researchers productive without giving everyone everything?A thin wrapper: they call one function and get the right project prefix, output_uri, tags, package pinning, and image digest. They enqueue to their team's queue, which is credentialed for data read and artifact write only. Promotion and serving live on queues only the platform team can reach. The result is that the safe path is also the shortest one, which is the only governance that survives contact with a deadline.
18. What is your incident playbook for a lost server?Agents keep running and buffer, so there is some slack. Stand up the same pinned image, not latest, because a version change means a migration on top of an incident. Restore Mongo, then Elasticsearch. Verify login, a known task's scalars, an artifact download, and an agent claiming a queued task, in that order. Then reconcile tasks that finished during the outage and still show as running. The number that matters is the RTO you measured last quarter, not the one you assume.
19. Where would you not use ClearML?Point-in-time feature serving with online/offline parity. That is a feature store. SQL analytics over versioned tables belongs in a table format or a warehouse. Company-wide orchestration beyond ML belongs in Airflow, Dagster, or Temporal. High-scale low-latency serving belongs in KServe or Triton directly. Fine-grained RBAC on the open-source tier. Stretching ClearML across those costs more than the second tool would.
20. What is the single idea that connects everything?A recorded run that can be executed again is worth more than a log of one that cannot. Agents, queues, the registry, image digests, lock files, and backups all exist to keep that property true as the number of runs, people, and months grows, and the weekly rebuild check is the only thing that proves it still holds.

Final self-test

• Give the thirty-second answer, then the five components and the three data stores.
• State the rules that never change: init order, connect's direction, title/series, output_uri, finalize.
• Distinguish init, create, and clone, and explain reuse versus continuation.
• Explain why enqueue permission is code execution, and describe the fork-PR danger precisely.
• Name the three properties that make ClearML's access model sharper than it looks.
• Say which component breaks first at scale, what the symptom is, and the fix order.
• List the three parts of a backup and what a Mongo-only restore looks like.
• Name every layer that must be pinned for an eighteen-month rebuild, and how you verify it.
• State what pipeline caching keys on, and the stale-result failure it causes.
• Walk the promotion chain and explain how a tag move becomes a deployment.
• Enumerate everything an erasure request touches, and the one thing you cannot erase.
• Give the seven platform alerts in priority order, and say why task failures are not among them.
• Name three places ClearML is the wrong tool, and what you would use instead.

Part three of three. The problems at this level are rarely about syntax. They are about a server thirty people depend on, a GPU bill nobody owns, a model that cannot be rebuilt, and an erasure request against an immutable store. Start with the error table, then the practices, verification, and playbooks underneath it.

Common errors at this level

Cumulative. Everything from Beginner and Mid still applies. These cause incidents rather than failed runs.

The practices that pay off most

Practice cards

1. Prove the fork path is closedOpen a pull request from a fork on a public repo and confirm it cannot enqueue anywhere with credentials. If it can, stop reading and fix that.
2. Find a secret in a diffLeave an untracked secrets.txt in a working tree, run a task, and read it back from Uncommitted Changes. Thirty seconds, and it changes behaviour permanently.
3. Break Elasticsearch on purposeFill the volume on a test instance past the high watermark, watch a running task stop reporting, then release the read-only block.
4. Restore and time itAll three backups onto a scratch host. Verify login, scalars, artifact download, agent claim. Publish the number.
5. Kill the services agentStop it and see how long before anyone notices. That gap is your detection time, and it is usually days.
6. Rebuild your production modelClone the task behind it, run it today, compare the metric. Then list every layer it does not pin.
7. Generate a lineage reportFrom a model id, produce code, data, config, and approval in one command. Note which of the four is missing.
8. Scope an erasure requestPick a dataset, count the referencing tasks and child versions, and list every other copy including backups. Write down what you would do.
9. Measure real utilisationGPU-hours consumed versus hours that produced a completed run, for one week. Then list the running sessions and their owners.
10. Run the review checklist on someone else's projectIt will find something, and the exercise teaches you how much of this is convention rather than enforcement.

The hardening pass every deployment should get

Work down this list. Each step is independently valuable, and the order is by blast radius.

# 1. Fork trust — the only RCE path in the system
#    Audit every public repo's workflows for ClearML secrets on pull_request.
#    Fork builds → ci-untrusted queue, ephemeral hosts, NO other credentials.

# 2. Bucket versioning — makes every other storage mistake survivable
aws s3api put-bucket-versioning --bucket ml-artifacts \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket ml-datasets \
  --versioning-configuration Status=Enabled

# 3. TLS at a reverse proxy — no plaintext credential path
#    Terminate at nginx/traefik; never expose 8008/8080/8081 directly.

# 4. Credentials: one per purpose, then delete the shared one
#    Settings → Workspace → App Credentials. Set CLEARML_WORKER_ID per host.

# 5. Artifacts off the file server
#    sdk.development.default_output_uri: "s3://ml-artifacts/<team>/clearml"

# 6. Pin the server image, and record the pair you support
#    allegroai/clearml:1.16.2  +  clearml==1.16.4  +  clearml-agent==1.9.2

# 7. Disk alerts before anything else
#    Elasticsearch volume > 80%  → page
#    MongoDB volume > 80%        → page

s3://ml-artifacts/<team>/clearml/    30d → IA,  180d → Glacier, 365d → delete
s3://ml-artifacts/<team>/ci/          14d → delete
s3://ml-datasets/                     versioning on; no expiry; write-restricted

DRY_RUN = os.environ.get("RETENTION_APPLY") != "1"      # safe by default
EXEMPT_TAGS = {"keep", "published", "paper", "release"}
# ...enumerate, print totals, and only delete when explicitly enabled

Verifying, not assuming

Every claim in this section is checkable in a few minutes. Assumed properties are the ones that fail.

curl -s localhost:8008/debug.ping
curl -s 'localhost:9200/_cluster/health?pretty' | grep -E 'status|unassigned'
curl -s 'localhost:9200/_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc' | head
df -h /opt/clearml/data/elastic_7 /opt/clearml/data/mongo_4
docker ps --format '{{.Names}}\t{{.Image}}' | grep clearml     # is the tag pinned?

from clearml import Model, Task

serving = Model.query_models(project_name="vision", model_name="resnet18-cls",
                             tags=["production"], max_results=5)
print("production models:", [(m.id, m.published) for m in serving])
assert len(serving) == 1, "exactly one production model expected"
assert serving[0].published, "production model is not published — it can be swapped"

t = Task.get_task(task_id=serving[0].task)
assert not t.data.script.diff, "production model was trained from a dirty tree"
assert t.data.container, "no container recorded — the environment is not pinned"
print("commit:", t.data.script.version_num, "image:", t.data.container)

Cost governance

Cost becomes your problem the moment ClearML is shared, and it is entirely a measurement problem before it is a technical one.

from datetime import datetime, timedelta
from collections import defaultdict
from clearml import Task

week = datetime.utcnow() - timedelta(days=7)
by_queue, wasted, by_team = defaultdict(float), defaultdict(float), defaultdict(float)

for t in Task.get_tasks():
    if not t.data.completed or t.data.completed < week:
        continue
    runtime = (t.data.completed - t.data.started).total_seconds() / 3600
    queue = t.data.execution.queue or "unknown"
    by_queue[queue] += runtime
    if t.status in ("failed", "aborted"):
        wasted[queue] += runtime
    team = next((x.split(":", 1)[1] for x in t.get_tags() if x.startswith("team:")), "untagged")
    by_team[team] += runtime

logger = Task.current_task().get_logger()
for q, hours in sorted(by_queue.items(), key=lambda kv: -kv[1]):
    logger.report_single_value(f"hours_{q}", round(hours, 1))
    logger.report_single_value(f"wasted_{q}", round(wasted[q], 1))
for team, hours in by_team.items():
    logger.report_single_value(f"hours_team_{team}", round(hours, 1))

Incident playbooks

Four incidents, each with a first action, a verification, and the prevention. Keep these in the repository.

The server is gone.

1. Do not rush the versionStand up the same pinned image. A different version runs migrations on top of an incident.
2. Mongo first, then ElasticsearchMongo is the system of record; Elastic restores metrics and logs.
3. Verify in orderLogin → a known task's scalars → an artifact download → an agent claiming a queued task.
4. ReconcileTasks that finished during the outage may show as running. Mark them and record the gap.
5. Publish the actual RTOCompare it against your rehearsed number and fix whichever step was slower than expected.

Elasticsearch went read-only. Symptom: metrics stop, training continues. Free disk (delete old indices or expand the volume), release the block with a _settings PUT, then fix the cause: reporting rate, retention, or capacity. Prevention: an 80% disk alert.

A model cannot be rebuilt. Run the lineage report and find the unpinned layer; it is nearly always the base image tag or an unpinned wheel. If artifacts are gone, check whether retention deleted the task, then check bucket versioning. Record what could not be recovered. That list is your next sprint.

Unauthorised enqueue. Treat it as a credential compromise, not a mistake: abort the task, revoke the credential that enqueued it, audit everything that queue's agents could reach with their ambient credentials, rotate anything exposed, then fix the queue isolation that permitted it.

Running ClearML as a platform

Five obligations. This is what "platform ownership" means day to day.

The alert list, in priority order, and it is deliberately short:

1. Elasticsearch disk > 80%Read-only stops all metric reporting silently. Highest-value alert in the system.
2. Services agent livenessNo heartbeat for ten minutes. Pipelines, schedulers, and triggers all stop with no error anywhere.
3. Queue depth sustained over an hourA dead worker, a stuck session, or a runaway sweep.
4. MongoDB disk > 80%The system of record.
5. API server 5xx rateAgents fail to claim; users experience it as hangs.
6. Rebuild check failingReproducibility has regressed and nothing else would have told you.
7. Weekly GPU utilisation < 40%You are paying for idle hardware.

Machine-learning specifics

A few things are ClearML-and-ML rather than general platform work, and they come up in senior interviews.

Checkpoint policy is a cost decision, not a training decision. Best plus last, not every epoch. Every-epoch checkpointing of a 2 GB model over a 100-epoch run is 200 GB per experiment, and nobody ever loads epoch 47.

Sweeps need a storage plan before they need a search strategy. Forty trials × 2 GB is 80 GB per sweep. save_top_k_tasks_only is the difference between a weekly sweep and a quarterly storage escalation.

Determinism is a spectrum, and you should record where you sit. Seeding gets you most of the way; torch.use_deterministic_algorithms(True) costs speed; some kernels are non-deterministic regardless. Record the flags and set a metric tolerance in the rebuild check rather than pretending bit-exactness.

Debug samples are data. Images and sample outputs land in Elasticsearch and object storage, and if the inputs are personal data those samples are copies nobody remembers during an erasure request. Report them sparsely and include them in your deletion scope.

Model weights are the un-erasable artefact. For some model classes, training data is extractable. Which is why the erasure conversation is architectural, keep personal data out of the versioned store, rather than a runbook.

A published model is the only thing that cannot be silently swapped. An unpublished "production" model can have its weights replaced by anyone with workspace access, and serving would pick up the change with no record. Publishing is a cheap, real control.

The checklist to run before shipping

Nobody passes this on the first attempt. Two failures matter disproportionately: an untested restore and a fork pull request with credentials. One is unrecoverable data loss, the other is remote code execution. Rank by blast radius rather than by effort, fix those two first, and publish the exceptions you are choosing to live with along with the reason. A checklist with argued exceptions is a better artefact than a green one, because it shows the trade-offs were made deliberately and it gives whoever inherits the platform a map instead of a mystery.