The Complete Comet Guide

This is part one of three. It covers everything you need to do real work with Comet's experiment tracking, not a teaser. By the end you can install the Python SDK, log in, record a training run with its settings, its metrics and its results, open the dashboard and read it, compare runs, version a dataset, register a model, and recover when a run fails to upload. Mid-level and Senior take the same topics further. Nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and tracking only makes sense once you have watched your own numbers appear in a browser tab and then compared them with a second run.

One naming note before we start. Comet is the company, and it sells two families of products on one platform. The first is the MLOps family: experiment management, artifacts, the model registry, the hyperparameter optimizer, and production monitoring. The second is Opik, an open-source product for tracing and evaluating LLM applications. This guide is about the first family and the Python package comet_ml. If you want to trace prompts and agents, go to the Opik guide after this one.

What Comet is, and what you can do with it

Comet is a place to keep a record of every machine-learning run you ever do. You add a few lines to your training script. Each time the script runs, the SDK streams the run's hyperparameters, metrics, code, environment, and any files you choose to a server. You then open a web dashboard and see the run as a page: loss curves drawn live, a table of settings, the exact code and git state that produced it, the Python packages installed, even CPU and GPU usage.

The value is not the charts. The value is that the record exists. Three weeks from now, when someone asks which learning rate produced the model in production, you do not search your terminal history. You open the run and read it.

Here is what a beginner can do by the end of this guide:

• Run a training script and see its metrics update live in a browser.
• Record the settings of each run so that two runs can be compared in a table.
• Attach a confusion matrix, a chart, a dataset and a trained model to a run.
• Give runs readable names and tags so you can find them later.
• Promote a good model into the model registry with a version number.
• Fetch your runs back from Python to analyse them in a notebook.
• Keep working when the network drops, by logging offline and uploading later.

The diagram is the whole idea. Everything else in this guide is detail about the four boxes.

The problem Comet solves, and what came before

Machine learning has a problem that ordinary software mostly does not. In ordinary software, the code is the thing. If two developers have the same commit, they have the same program. In machine learning, the same code can produce a different result each time, depending on the data version, the random seed, the hyperparameters, the library versions, the GPU, and the amount of patience you had when you pressed Ctrl+C. The result you care about is a function of everything, and most of that everything is not in git.

Before experiment trackers, people coped in a few ways, and you have probably used at least one of them.

The notebook with a graveyard of cells keeps results in cell outputs. It works until you restart the kernel, and then the numbers are gone. Nobody can tell which cell ran in which order.

The spreadsheet holds one row per run: learning rate, batch size, accuracy. It works until you forget to fill a row, mistype a number, or need the loss curve rather than the final value. The spreadsheet is also never linked to the code that produced the row.

The folder naming convention gives you directories like run_final_v2_lr0.001_REALLY_final. It works for a week.

The TensorBoard log directory was a real step forward, because it records curves automatically. But logs live on the machine that trained, they are awkward to share, they do not capture the code and environment, and comparing forty runs from three laptops and a cluster means copying files around.

A hosted experiment tracker replaces these with one habit: every run logs itself to a shared server, and the server stores the settings, the curves, the code, and the files together. Comet is one such tracker. Others are covered elsewhere in this series, for example MLflow, Weights and Biases and ClearML. They differ in details, but the core idea is the same, and what you learn here transfers.

What Comet adds on top of plain logging is a set of neighbours around the experiment. Artifacts version your datasets. A model registry holds the models you promoted, with version numbers and approval states. An optimizer runs hyperparameter searches and logs each trial as a run. Panels and reports let you turn charts into shared documents. A beginner needs the first two or three. The rest wait for later levels.

The mental model: four nouns and a hierarchy

Comet's documentation describes a hierarchy of containers. From the outside in:

1. An Organization is your company or school. There is normally one.
2. A Workspace sits inside it, typically one per team. Every user also has a personal default workspace.
3. A Project sits inside a workspace, typically one per machine-learning problem, such as "churn-prediction" or "arabic-sentiment".
4. An Experiment sits inside a project. It is one execution of your code.

As a beginner you will live almost entirely at the bottom two levels. You will pick a project name, and every run you start becomes an experiment in it. If you do not pick a project name, runs land in a project called Uncategorized. That works but fills up with clutter, so name your projects from the start.

Now the nouns you need to be precise about.

An experiment is the unit of measurement. The docs define it as a single execution of code with some associated data. One training run with one set of hyperparameters is one experiment. Each experiment has an experiment key, a unique identifier. By default Comet generates a random one. If you supply your own, it must be alphanumeric and between 32 and 50 characters long. You will rarely need to.

A parameter is a setting of the run: learning rate, batch size, number of trees, the name of the dataset. A parameter is set once, or rarely changes during the run. You log parameters so that runs can be compared by their settings.

A metric is a measured value that usually changes over time: training loss at each step, validation accuracy at each epoch. A metric is a time series. Each value carries a step (how many batches so far) and optionally an epoch, plus a timestamp. Comet draws metrics as line charts. One detail to know early: beyond 15,000 values for a single metric in a single experiment, Comet down-samples what it displays, so a very dense curve may look slightly thinned. That is almost never a problem for beginners.

An asset is a file attached to one experiment: an image, a chart, a plot, a saved model, a piece of source code. Assets belong to the run that logged them and are not versioned.

An artifact is different, and the difference trips beginners. An artifact is a versioned object that lives at the workspace level, not inside one run. Each version is an immutable snapshot of files. The typical use is a dataset: train-data version 1, then version 2 after you clean it. Comet records which experiment produced each artifact version and which experiments used it, so you can trace a model back to the exact dataset it learned from. If an asset is "a file stuck to this run", an artifact is "a named, versioned thing many runs share".

Two more terms appear in this guide. A model logged with log_model() is a set of files attached to an experiment, shown in the Assets and Artifacts tab. A registered model is what you get after promoting one into the model registry, a workspace-level catalogue where every version has a number, tags, and a status. We come to that later.

Installing the SDK and checking the setup

The SDK is a pure Python package named comet_ml. The steps are the same on Linux, macOS and Windows. It needs Python 3.8 or newer. Create a virtual environment first, so that this course does not disturb other projects:

python -m venv .venv
source .venv/bin/activate        # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install comet_ml

You can also write pip install comet-ml; PyPI treats the two spellings as the same package. In a notebook such as Jupyter or Colab, use %pip install comet_ml in a cell. If you prefer conda, the docs give conda install -c anaconda -c conda-forge -c comet_ml comet_ml.

The install adds one command-line tool named comet. Check that it works and see which SDK version you got:

comet --version

At the time of writing, the current release is 3.58.7, so the command prints 3.58.7. If your shell says comet: command not found, the scripts directory of your environment is not on your PATH. On Windows this is common. The fix is to activate your virtual environment again or call the module through Python; the simplest check is pip show comet_ml.

NumPy is optional for the SDK. If it is missing you will see a line saying numpy not installed; some functionality will be unavailable. Since nearly every machine-learning script installs NumPy anyway, that message usually never appears.

Now the most useful diagnostic you will meet in this guide:

comet check

This command tests whether your machine can reach the three Comet services the SDK talks to: the server that receives logged data, the REST API, and the optimizer service. It also prints your Python version, any proxy settings, whether TLS certificate checking is on, and the path of the certificate bundle in use. It ends with a summary line such as Server connectivity True. Run it once right now, before you have any problem, so you know what healthy output looks like. Add --debug when something is wrong, to see the raw HTTP exchange.

If you have no API key yet, comet check can still show whether the network path is open. A corporate proxy or a security gateway that inspects encrypted traffic will show up here as a failed connection or a certificate complaint. Section thirteen covers what to do then.

Logging in: the API key and where it lives

Comet needs to know who you are. It identifies you with an API key, a long secret string tied to your account. To find it, open the Comet website, click your avatar, choose Account settings, open the API keys tab, and copy the key.

Treat the key like a password. Anyone who has it can log runs into your account and read your projects. Never paste it into code that you commit. This is more important with Comet than with many tools, because Comet automatically records your source code, so a key typed inside the script would be uploaded along with it.

The safe and simple way to store it is the login command:

comet login

It prompts for the key and writes it into a configuration file named .comet.config in your home directory. That is ~/.comet.config on Linux and macOS, and %USERPROFILE%\.comet.config on Windows. From then on, every script on this machine finds the key automatically.

From Python, the equivalent is:

import comet_ml

comet_ml.login()

This call asks for the key only if none is configured already. If a key exists, it does nothing visible. There are two variants worth knowing. comet_ml.login(anonymous=True) logs to a public, claimable project without an account; it is fine for trying the tool, but never for real or sensitive data, because anyone can see what you log. And comet_ml.login(project_name="my-project") saves extra settings such as the project name into the config file as well.

Older tutorials tell you to call comet_ml.init() or to run comet init --api-key. Both are deprecated. Using them prints a warning such as comet_ml.init() is deprecated and will be removed soon. Please use comet_ml.login(). Use login everywhere and ignore any tutorial that uses init.

You can also provide the key through an environment variable, which is the standard approach on servers and in automated jobs:

export COMET_API_KEY="paste-your-key-here"      # macOS and Linux

$env:COMET_API_KEY="paste-your-key-here"        # Windows PowerShell

Which source wins when several are present? Comet reads its settings in a fixed order of priority. Arguments written directly in code come first. Then operating-system environment variables. Then a .env file in the current folder. Then a file named by COMET_CONFIG. Then a .comet.config in the current folder, and finally the one in your home directory. The practical rule is that the more specific and closer to the run a setting is, the more power it has. A value in code beats a value in the environment, which beats a value in a file.

The config file uses the INI format. Open yours after logging in and you will see something like this:

[comet]
api_key = your-key-here
workspace = my-team
project_name = my-project

You can edit it by hand to set a default workspace and project. Each line has an environment-variable equivalent: COMET_API_KEY, COMET_WORKSPACE, COMET_PROJECT_NAME. The pattern is COMET_ plus the setting name in capitals.

Your first experiment, step by step

Let us log a run that does no real machine learning, so that nothing distracts from the Comet part. Create a file called hello_comet.py:

import comet_ml

comet_ml.login()

exp = comet_ml.start(project_name="first-steps")

exp.log_parameters({"batch_size": 32, "learning_rate": 1e-4})

for step in range(100):
    exp.log_metrics({"loss": 1 / (step + 1)}, step=step)

exp.end()

Run it with python hello_comet.py. Read the console output as it goes. Near the top you will see a line like COMET INFO: Experiment is live on comet.com followed by a web address. At the end you will see a summary of what was logged. The summary level is one by default and you can ask for more detail later.

Now walk through the script line by line, because each line teaches a concept.

import comet_ml brings in the SDK. Import order matters, and we will return to it in a moment.

comet_ml.login() makes sure the key is available. If you already ran comet login, this line is harmless.

comet_ml.start(project_name="first-steps") is the heart of it. It creates a new experiment in the project named first-steps, creating the project if it does not exist. It returns an object, here called exp, which you use for all logging. Since version 3.47.1, start() is the recommended way to begin a run. You will see older code that writes comet_ml.Experiment(...). That still works, but the current docs recommend start, and this guide uses it throughout.

exp.log_parameters({...}) records settings. You pass a dictionary. Nested dictionaries are supported. You can call log_parameter("name", value) for a single one.

exp.log_metrics({...}, step=step) records a measured value at a given step. Here the loss falls as one over the step number, which produces a nice decaying curve. You can log several metrics in one call by putting several keys in the dictionary. Calling it once with a dictionary is cheaper than calling log_metric many times, which matters when you hit rate limits later.

exp.end() tells Comet the run is finished. It waits until all queued data has been sent and then closes the experiment. Always call it, especially in notebooks. In a notebook, end() also records the cells you executed.

Open the web address printed in the console. You will land on the experiment page. Look at these tabs, which are the ones you will use daily:

• Charts or Panels: your loss curve. Hover over it to read values.
• Parameters: a table with batch_size and learning_rate.
• Metrics: the final, minimum and maximum of each metric.
• Code: the exact script that ran, stored automatically.
• System metrics: CPU, memory and, if present, GPU usage over time.
• Installed packages: every package and version in your environment.
• Output: what the script printed to the console.

You wrote three lines of logging and got an archive of code, environment and results. That trade is the reason people use the tool.

Now the import-order rule. If your script uses PyTorch, TensorFlow, Keras, fastai or similar, put import comet_ml before importing them. Comet hooks into those libraries to log metrics automatically, and it can only do so if it is loaded first. If you get the order wrong, the current SDK prints a warning: To get all data logged automatically, import comet_ml before the following modules. Older documentation describes this as an ImportError, but the behaviour today is a warning and partial logging. There are two escape hatches: run your script with comet python script.py, which prepares the SDK before your code starts, or set COMET_DISABLE_AUTO_LOGGING=1 to turn the automatic logging off and log everything by hand.

A real training run with scikit-learn

The toy script showed the mechanics. A real run has data, a model, a split, training, and evaluation. We will use scikit-learn because it is installed on most machines and needs no GPU. Install it with pip install scikit-learn if you need to. The example trains a random forest on the built-in breast-cancer dataset and logs what a beginner should always log.

import comet_ml
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
import joblib

comet_ml.login()

params = {
    "n_estimators": 200,
    "max_depth": 6,
    "random_state": 42,
    "test_size": 0.2,
}

exp = comet_ml.start(project_name="breast-cancer")
exp.set_name("rf-200-trees-depth6")
exp.add_tags(["baseline", "random-forest"])
exp.log_parameters(params)

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=params["test_size"], random_state=params["random_state"]
)
exp.log_parameter("n_train_rows", len(X_train))

model = RandomForestClassifier(
    n_estimators=params["n_estimators"],
    max_depth=params["max_depth"],
    random_state=params["random_state"],
)
model.fit(X_train, y_train)

with exp.train():
    exp.log_metric("accuracy", accuracy_score(y_train, model.predict(X_train)))

with exp.test():
    preds = model.predict(X_test)
    exp.log_metric("accuracy", accuracy_score(y_test, preds))
    exp.log_metric("f1", f1_score(y_test, preds))
    exp.log_confusion_matrix(y_true=y_test, y_predicted=preds)

joblib.dump(model, "rf_model.joblib")
exp.log_model("breast-cancer-rf", "rf_model.joblib")

exp.end()

Run it and open the experiment. Here is what each new idea does.

exp.set_name(...) gives the run a readable name. Without it, Comet assigns a random name. Names are for humans. A good name states what is different about this run, such as rf-200-trees-depth6. You can change it later in the interface.

exp.add_tags([...]) attaches short labels. Tags are for filtering: show me everything tagged baseline. You can add a single tag with add_tag.

The with exp.train(): and with exp.test(): blocks are contexts. Inside a context, Comet prefixes the metric names, so the same call log_metric("accuracy", ...) becomes train_accuracy in one block and test_accuracy in the other. This is how you keep training and evaluation numbers apart without writing different names by hand. There is also exp.validate() for a validation context.

exp.log_confusion_matrix(y_true=..., y_predicted=...) takes the true labels and the predicted labels and draws a confusion matrix in the dashboard. A confusion matrix is a table that shows, for each true class, how many examples were predicted as each class. It tells you not just how often the model is wrong but how it is wrong, which a single accuracy number hides.

exp.log_model("breast-cancer-rf", "rf_model.joblib") uploads the model file and attaches it to the run under the name you chose. The first argument is the name, the second is the path of a file or folder. The docs recommend logging models with log_model on the experiment, rather than as artifacts, because logged models can later be promoted to the registry.

Look at the Assets and Artifacts tab after the run. The model file is there, downloadable.

A word about what you did not have to write. Comet also recorded the script text, the git commit if you are in a repository, the operating system, the Python version, the installed packages, and the console output. These happen automatically. They are controlled by settings you can turn off, which Mid-level covers. If your code or data is sensitive, consider these defaults. For example, if your employer has data-residency rules, be aware that the source code and console output are uploaded to Comet's servers, or to your own servers if your organisation self-hosts.

Reading and using the dashboard

You now have several runs in a project. Click the project name to open the project view. This is a table with one row per experiment, plus a set of charts above or beside it. The table is the most useful part for a beginner.

The columns show run names, status, creation time, the parameters you logged, and the last value of each metric. Click a column header to sort. Click the column chooser to show or hide columns. If you logged max_depth as a parameter, it is now a column you can sort by. That is the payoff for logging parameters properly: the question "which setting gave the best F1?" is answered by sorting.

You can filter the table. Type in the search box or build a filter on a parameter, a metric, or a tag. Filtering by the baseline tag shows only runs you tagged that way. This is why tags deserve the thirty seconds it takes to add them.

Select two or more rows and choose Compare. The compare view overlays the curves and shows parameter differences side by side, highlighting what changed. In the current cloud release the compare view handles up to ten experiments at once, while the project view can chart many more. Comparing is the thing you do a hundred times, so get comfortable with it early.

Panels are the charts. A project view and an experiment view are made of panels, and you can add built-in ones such as line charts, scatter plots, bar charts and parallel-coordinates plots. A scatter plot of learning rate against accuracy, or parallel coordinates across several hyperparameters, shows patterns a table cannot. You can also write custom Python panels, but that belongs to later levels.

If you work in a notebook, you can even embed the dashboard inside a cell:

exp.display(tab="panels")

The tab argument accepts names such as code, parameters, metrics, output, system-metrics, installed-packages, assets and artifacts. This is handy in Colab or Jupyter when you do not want to switch windows.

Logging more: images, figures, text, code and notes

Metrics and parameters cover most of what you log. A few more methods earn their place, and they all follow the same pattern: call a method on exp, and something appears in the dashboard.

To attach a chart you made with Matplotlib, use log_figure:

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.bar(["a", "b", "c"], [3, 7, 5])
exp.log_figure(figure_name="example-bar", figure=fig)

The figure appears in the graphics tab. From version 3.51.0, the format argument defaults to automatic selection, so you rarely set it.

To attach an image from a file or an array, use log_image:

exp.log_image("confusion.png", name="confusion-plot")

To attach any file, such as a CSV of predictions or a text report, use log_asset:

exp.log_asset("predictions.csv", file_name="predictions.csv")

To store free-form key-value data that does not fit parameters or metrics, such as the name of the GPU node or the ticket number, use log_other("ticket", "ML-142").

To record the source code explicitly, for example when the automatic capture cannot find your script, call exp.log_code(file_name="train_rf.py"). You will see this needed when your entry point is wrapped by a launcher. If the console prints COMET ERROR: Failed to set run source code, this is the fix.

Steps and epochs deserve a word, because people confuse them. A step is usually one batch or one update. An epoch is one full pass through the training data. log_metric(name, value, step=..., epoch=...) accepts both. If you always log at the end of an epoch, pass epoch. If you log per batch, pass step. You can set defaults with exp.set_step(n) and exp.set_epoch(n), and later calls use them. Parameters accept step only.

On rates: Comet limits how fast the cloud accepts data, per data type. Metrics are accepted at 12,000 per minute, parameters at 10,000 per minute, and console output at 12,000 per minute. If you exceed them you see Experiment has been throttled. Some data (like experiment metrics) might be missing. The cure is to log per epoch rather than per batch, or to send several metrics in one log_metrics call. Artifacts are never throttled.

Organising runs: projects, names, tags and contexts

After twenty runs, a project without structure becomes hard to read. Four habits keep it usable.

One project per problem. Do not put every experiment of the course into Uncategorized. Name projects after the question they answer: breast-cancer, arabic-sentiment. Set the name in start(project_name=...) or in the config file, and the project is created automatically. Choose the name deliberately, because a typo creates a new project silently. Once it exists, you can move or rename runs in the interface.

Names that describe the difference. run-7 tells you nothing. rf-200-trees-depth6 tells you what the run is. Setting a name costs one line. If you forget, you can rename in the dashboard.

Tags for categories that cut across runs. Use a short, consistent vocabulary: baseline, tuned, bug, final, ablation. Agree on it with your team, because Baseline and baseline are two tags.

Contexts for train, validate and test. As shown earlier, the with exp.train(): style keeps metric names consistent.

Datasets as artifacts

Models are the output of a run. Data is the input, and data changes. If you clean your dataset on Tuesday and retrain on Wednesday, a score change might come from the cleaning rather than the model. Artifacts exist to make that visible.

An artifact is created locally, given files, and logged from an experiment:

from comet_ml import Artifact, start

exp = start(project_name="breast-cancer")

art = Artifact(name="cancer-train-data", artifact_type="dataset")
art.add("data/train.csv")
exp.log_artifact(art)

exp.end()

The constructor takes a name and an optional type, such as dataset or model. add attaches a local file or folder. log_artifact uploads it and creates version 1.0.0. If you log an artifact with the same name again and the content differs, Comet creates a new version. Nothing is overwritten; each version is an immutable snapshot. Artifact names are limited to 100 characters.

Later, in another run or another script, you use the dataset by name:

exp = start(project_name="breast-cancer")
logged = exp.get_artifact("cancer-train-data")
logged.download("./data")

The signature is get_artifact(artifact_name, workspace=None, version_or_alias=None). Without a version it fetches the latest. Because you fetched it through the experiment, Comet records that this run consumed that artifact version. The artifact page then shows its lineage: which run produced it and which runs used it. When a result looks suspicious, lineage tells you which dataset version was in play.

If your data is large or already lives in cloud storage, you do not have to upload it. A remote artifact records a reference instead:

art = Artifact(name="big-images", artifact_type="dataset")
art.add_remote("s3://my-bucket/images/")
exp.log_artifact(art)

For S3 or Google Cloud Storage addresses, the SDK lists the objects under the prefix and records their addresses and checksums, and their version identifiers if bucket versioning is on. Your machine needs credentials configured the provider's normal way. Without them, only the address string is recorded. This suits Gulf and Egyptian teams who keep data in a regional bucket for residency reasons: the data stays in your region, and Comet stores only a pointer and metadata.

If you already know DVC, think of artifacts as a hosted, UI-visible alternative for small and medium datasets. They serve a similar goal with a different workflow.

The model registry: from a good run to a named model

Logging a model on a run keeps it attached to that run. The model registry is the catalogue where you keep the models you actually intend to use. Each registered model has a name, and each version has a number, so the question "which model is in production?" has an answer.

The flow has two steps. First log the model on the experiment, then register it:

exp.log_model("Breast Cancer RF", "rf_model.joblib")
exp.register_model("Breast Cancer RF")

Two details. The registry name becomes lowercase with hyphens: "Breast Cancer RF" appears as breast-cancer-rf. And the first version gets the default version string 1.0.0. Versions follow semantic versioning, which means three numbers separated by dots, such as 1.0.0 or 2.1.5. Register again from a better run and the new version is created under the same model name, which you can also set explicitly with the version argument.

Open the registry in the dashboard: look for the model registry in the left navigation, click your model, and you see its versions, each with the run it came from, tags, notes, and a history of changes.

Older tutorials talk about model stages: Staging, Production. Stages are deprecated. Their data became tags, and the official lifecycle mechanism is now status: a model version has a status such as Development, Staging or Production, which you set per version. In many setups, a regular member changes the status by sending an approval request that a workspace owner approves. This is deliberate: it puts a human between "a run finished" and "a model is labelled production". Using the stage-based API methods prints deprecation warnings, so skip them.

From code, after a run has ended, you can manage versions through the API object:

api = comet_ml.API()
model = api.get_model(workspace="my-team", model_name="breast-cancer-rf")

model.add_tag(version="1.0.0", tag="reviewed")
model.set_status(version="1.0.0", status="Staging")
model.download("1.0.0", output_folder="./downloaded-model")

get_model takes the workspace and the registry name. add_tag and set_status take the version first. download fetches the files of one version into a folder you name. From the command line there is a matching helper:

comet models list -w my-team
comet models download -w my-team --model-name breast-cancer-rf --model-version 1.0.0 --output ./model

The built-in help text of the command contains example flags that do not exist, written with underscores. The real flags are --model-name and --model-version, with hyphens, as above. If you paste an example from the help and it is rejected, that is why.

Registered models are also the point where serving tools and deployment pipelines connect: a deployment job asks the registry for "version 1.0.0 of this model" instead of copying a file from someone's laptop.

Getting your data back: the API and querying

The SDK has two halves. comet_ml.start() gives you an object for logging during a run. comet_ml.API is a thin client over Comet's REST interface and is for reading afterwards: listing experiments, searching, downloading. You use it in an analysis notebook after your runs are done.

import comet_ml

api = comet_ml.API()

experiments = api.get_experiments("my-team", project_name="breast-cancer")
for e in experiments:
    print(e)

The call takes a workspace and a project name, and returns a list of experiment objects. You can also fetch one by its path, api.get("my-team/breast-cancer/<experiment-key>"), or by key alone with api.get_experiment_by_key(key). The old get_experiment_by_id is deprecated.

To filter by content, use the query helpers:

from comet_ml.query import Tag, Metric, Parameter

baseline = api.query("my-team", "breast-cancer", Tag() == "baseline")

You can build conditions on tags, metrics and parameters. From version 3.54.0 there is also api.search(workspace, project_name, query), which supports combining conditions with AND and OR and handles paging through large result sets.

To bring metric curves into a pandas table for your own plotting:

df = api.get_metrics_df(
    experiment_keys=[e.id for e in experiments],
    metrics=["test_accuracy"],
)
print(df.head())

The function returns a data frame with one row per logged value. This is how you build your own comparison plot, or a table for a report, without using the web interface.

You can also append to a finished run using APIExperiment, for example to attach a late-arriving evaluation score. comet_ml.APIExperiment(previous_experiment="<key>") gives you a handle on an existing run for one-off reads and writes through the REST API. To continue a training run that is still going, rather than patch one afterwards, use comet_ml.start(mode="get", experiment_key="<key>").

Working offline and when the network fails

Not every training job has a good internet connection. You might train on a cluster without outbound access, on a train, or behind a firewall that blocks Comet. Comet handles this with offline experiments, which write everything to a local archive that you upload later.

exp = comet_ml.start(online=False, project_name="breast-cancer")

Log as normal. When you call exp.end(), the SDK writes a .zip file into a folder named .cometml-runs by default, and prints a message telling you the exact command to upload it:

comet upload .cometml-runs/*.zip

You can point comet upload at individual archives, and pass --workspace and --project-name to override where they go. Offline experiments have no rate limits. The only limits are the hard limits on counts, such as the number of images.

You do not need to choose offline in advance. Comet has a fallback to offline: if the connection drops during a run and the SDK cannot re-establish it after several retries, it switches to logging locally. If the connection returns, it uploads what it held. If it does not, you get a zip. The console tells you so: Could not send live data to Comet during experiment runtime. An offline experiment will be available for upload, followed by a command. The printed command includes --force-reupload. That flag is deprecated; use --force-upload instead. If comet upload complains experiment was not found, you can upload it as a new experiment by using the --force-upload flag, it is telling you the same thing.

What matters for a beginner is to not panic. A failed upload is not a lost run. Look for a zip file, read the last lines of the console, and run the command.

Finally, sometimes you want a script to run with Comet code in it but send nothing, for example during a dry run or a unit test. Set COMET_AUTO_LOG_DISABLE=1. This disables all network communication. Since version 3.37.2 it does not even need an API key.

Common errors and how to read them

Comet's messages start with a prefix: COMET INFO, COMET WARNING or COMET ERROR. Read the whole line; it usually names the cause. Here are the ones you will meet.

The given API key '...' is invalid ... Your experiment will not be logged. The key is wrong, or it belongs to a different installation, for example a key from the cloud used against a self-hosted server. Copy it again from Account settings, then API keys. Then run comet check.

API key is not set, or a ValueError from start(). No key was found anywhere. Run comet login or set COMET_API_KEY. To run code without logging, set COMET_AUTO_LOG_DISABLE=1.

Run will not be logged. The first handshake with the server failed. This points to the network, a proxy, a certificate problem, or an outage. Run comet check --debug. You can also test the server directly with curl -i https://www.comet.com/clientlib/isAlive/ping, which should return a short JSON body containing Healthy Server.

Failed to create Comet experiment, reason: .... Something went wrong inside start(), and the reason after the colon says what. Authentication, workspace and mode mistakes are the usual ones.

Workspace ... doesn't exist. A typo in the workspace name, or you are not a member. Check COMET_WORKSPACE.

To get all data logged automatically, import comet_ml before the following modules. You imported Comet after a machine-learning library. Move the import to the top of the file, or use comet python script.py.

Experiment has been throttled. You log too fast. Log per epoch, batch your metrics, or go offline.

Maximum number of experiments reached for project. A project holds at most 25,000 experiments. Archive old ones or use a new project. You will not meet this in the first months.

The SSL certificate message. There's seem to be an issue with your system's SSL certificate bundle. The message is Comet's wording. It means Python cannot verify the server's certificate. Three causes are common: a Python installed from python.org on macOS without its certificates installed, a company or security gateway that inspects encrypted traffic with its own certificate, or a self-signed certificate on a self-hosted server. The fix is to repair the certificate store, or to point the REQUESTS_CA_BUNDLE environment variable at your organisation's certificate file. Turning verification off with COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 exists, but it removes the protection that keeps your key safe. Use it only as a last resort.

Comet failed to send all the data back. Shown at the end of a run, this means end() ran out of patience waiting for the upload, or the process was killed. Always call exp.end(), and check your network. The waiting time is controlled by COMET_TIMEOUT_CLEANING.

A deprecation warning about init or COMET_INI. Not an error. Replace init() with login(), and COMET_INI with COMET_CONFIG.

One more class of problem is not an error message at all: the dashboard looks wrong. If curves look thinned, you probably logged more than 15,000 values of one metric. If a metric is missing, check the name and whether a context added a prefix. If a run is stuck on "running", your script crashed before end(); it will time out, and you can archive it.

Configuration you will actually touch

Most beginners need only a handful of settings, and the simplest place for them is the [comet] section of .comet.config or the matching environment variables.

Options that control what is logged automatically go into comet_ml.ExperimentConfig, which you pass to start:

config = comet_ml.ExperimentConfig(
    name="rf-baseline",
    tags=["baseline"],
    log_code=True,
    auto_output_logging="simple",
)
exp = comet_ml.start(project_name="breast-cancer", experiment_config=config)

ExperimentConfig accepts the name and tags, as shown, plus switches for logging code, the git state, environment details, and framework parameters and metrics. You can set disabled=True to switch Comet off entirely, which is useful for unit tests.

Putting it all together

Here is one small end-to-end project that uses everything so far. The goal: train three random forests with different depths, log them properly, register the best, and read the results back.

import comet_ml
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
import joblib

comet_ml.login()

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

best_f1, best_key = 0.0, None

for depth in [2, 4, 8]:
    params = {"n_estimators": 200, "max_depth": depth, "random_state": 42}
    exp = comet_ml.start(project_name="breast-cancer-final")
    exp.set_name(f"rf-depth-{depth}")
    exp.add_tags(["final-comparison"])
    exp.log_parameters(params)

    model = RandomForestClassifier(**params).fit(X_train, y_train)
    preds = model.predict(X_test)
    score = f1_score(y_test, preds)

    with exp.test():
        exp.log_metric("f1", score)
        exp.log_confusion_matrix(y_true=y_test, y_predicted=preds)

    joblib.dump(model, f"rf_depth_{depth}.joblib")
    exp.log_model(f"rf-depth-{depth}", f"rf_depth_{depth}.joblib")

    if score > best_f1:
        best_f1, best_key = score, depth
    exp.end()

print("best f1:", best_f1, "at depth", best_key)

Run it. The three runs appear in breast-cancer-final. In the project view, filter by the tag and sort by test_f1. Open Compare on all three and read the parameter difference next to the curves. Open the best run, go to its Assets tab, and register its model. Then in a notebook, use comet_ml.API() to list the runs and pull test_f1 into a table.

Notice what happened without any extra effort: each run has its code, its environment, its settings and its confusion matrix stored. If a teammate asks next month why depth eight was chosen, you send a link.

Also notice what this small project does not do. It does not version the dataset; adding an artifact would. It does not automate the search; Comet's Optimizer would. It does not run on a schedule or in a pipeline. Those are the next steps. They belong to later levels, and to neighbouring tools such as Kubernetes for running jobs, and Evidently for watching a model after deployment.

What you can now do, and what comes next

You can install the SDK, verify it with comet --version and comet check, and log in without putting a secret in code. You can start an experiment with comet_ml.start(), log parameters and metrics, use contexts to separate training from testing, attach figures, confusion matrices, files and models, and end the run cleanly. You can read the dashboard: sort, filter, compare, and chart. You can version a dataset as an artifact and trace which run used it. You can register a model, tag it, and set its status. You can fetch runs back with the API, work offline, and read the common error messages.

What to practise next, in order of value:

1. Use Comet for every training run in your next project, even tiny ones. The habit matters more than any feature.
2. Always log the seed, the dataset name and the data size.
3. Keep a tag vocabulary and use it.
4. Move to the Mid-level guide, which explains how the SDK works under the hood: what it captures automatically, how configuration resolves, how to resume runs, how artifacts and the registry behave in a team, the hyperparameter Optimizer, and testing your logging code.

For the LLM side, where you trace prompts, tool calls and agents instead of training curves, go to the Opik guide. For a different tracker with the same core idea, compare MLflow. When you are ready to put a registered model behind an API, see BentoML.

Sources

• Comet documentation home
• Quickstart
• Python SDK overview
• Start an experiment
• comet_ml.start reference
• ExperimentConfig reference
• Experiment reference
• API reference
• Running offline
• Fallback to offline
• Command line overview
• Configure the SDK
• Log metrics and parameters
• Log models
• Using artifacts
• Remote artifacts
• Troubleshooting and FAQ
• Limits and performance
• Changelog

This page prepares you for the questions a junior machine-learning or MLOps interview asks about Comet and experiment tracking in general. Read it after the detailed guide, and say the answers out loud. Interviewers rarely want a definition recited. They want to hear that you have used the tool, that you know why it exists, and that you know where it bites.

The thirty-second answer

Comet is a hosted platform for tracking machine-learning experiments. You add a few lines of the comet_ml Python SDK to a training script, and every run streams its hyperparameters, metrics, code, environment and output files to a server, where you can view, compare and share them in a web dashboard. Around that core it offers versioned datasets called artifacts, a model registry for promoted models, and a hyperparameter optimizer.

If you get thirty more seconds, add the reason: machine-learning results depend on data, seeds, settings and library versions that git does not capture, so a tracker keeps the record that makes a result reproducible.

The model you should be able to draw

An interviewer may hand you a marker. Draw the hierarchy and name each box.

An organisation contains workspaces, a workspace contains projects, and a project contains experiments. An experiment is one execution of your code and has a unique experiment key. Inside it you store four kinds of things. Parameters are the settings you chose. Metrics are measured values over steps or epochs. Assets are files attached to that one run, such as images, plots and a saved model. Artifacts are different: they are versioned objects at workspace level, typically datasets, shared across runs, with lineage that records which run produced or consumed each version.

Add the registry if asked about deployment. A registered model is a named entry in the workspace catalogue, and each version has a semantic version number, tags and a status. The status replaced the older idea of stages.

The single distinction interviewers like most is asset versus artifact, because candidates who have only read the front page confuse them. An asset belongs to one run and is not versioned. An artifact is named, versioned and shared.

The commands and calls to have ready

Know these cold. If you can type them from memory, you sound like someone who has done it.

pip install comet_ml       # the package; import name is comet_ml
comet login                # prompts for the API key, writes ~/.comet.config
comet --version            # print the SDK version
comet check                # test connectivity to server, REST API and optimizer
comet upload run.zip       # upload an offline experiment archive

import comet_ml                      # before torch, tensorflow, keras

comet_ml.login()                     # never login(api_key="...") with a literal key
exp = comet_ml.start(project_name="churn")
exp.log_parameters({"lr": 1e-3, "batch_size": 32})
exp.log_metrics({"loss": 0.41}, step=10)
exp.log_model("churn-model", "model.joblib")
exp.end()

Be ready to explain three details in those lines. First, start is the current entry point, and older code writes Experiment(). Second, the import goes first so that auto-logging can hook into the ML libraries. Third, end() flushes queued data, which matters most in notebooks.

For the registry, know exp.register_model("name"), and that the registered name is lowercased. For reading data back, know that comet_ml.API() is the client for queries after the run, while start() gives the object for logging during the run.

The classic traps and why they happen

Interviewers love trap questions because they reveal experience. These are the ones that come up most, with the mechanism behind each.

The key in the script. Comet uploads your source code automatically. If you write the API key into the script, the secret travels with the code, and then probably into a git repository. The safe alternatives are comet login, the COMET_API_KEY environment variable, or a secrets store. The general principle is that a tool which records your code turns every literal in that code into something stored.

Wrong import order. Comet patches ML libraries to capture metrics automatically. If you import PyTorch or Keras first, the patch cannot be applied, and you lose the automatic logging. The current SDK warns instead of failing. The fixes are to import Comet first, to use comet python script.py, or to log by hand.

Forgetting end(). A script can finish while data is still queued. end() blocks until the queue is sent. In a notebook the kernel stays alive, so without end() the run stays open, and executed cells are not recorded. If you then call start() again with the same settings, you get the already-running experiment back rather than a fresh one, which explains a common "why are my two runs merged" confusion.

The metric name mismatch. Contexts such as with exp.test(): prefix metric names, so accuracy is stored as test_accuracy. If a hyperparameter search is told to optimise accuracy, it finds no values, prints Experiment ignored in sweep, and behaves strangely. The cause is always a name that is not exactly the logged one.

Logging too fast. The cloud throttles data per type, for instance metrics at 12,000 per minute. A loop that logs every batch of a large dataset gets throttled, and some metrics go missing, with a message saying so. The fixes are to log per epoch, to batch metrics into one log_metrics call, or to use offline mode, which has no rate limits. Artifacts are never throttled.

Down-sampling. Beyond 15,000 values per metric per experiment, Comet down-samples. A curve that looks smoothed may be thinned, not noisy. Total-fidelity storage is an organisation setting for teams that need every point.

The deprecated APIs. comet_ml.init(), model stages, comet upload --force-reupload and COMET_INI all still appear in old tutorials. The current replacements are login(), status and tags, --force-upload and COMET_CONFIG. Knowing this signals that you work from the current documentation.

Offline does not mean lost. When the connection fails mid-run, Comet falls back to writing a local zip and tells you the upload command. Candidates who say "the run is lost" are wrong. The right answer is to find the zip and run comet upload.

Common interview questions

What problem does an experiment tracker solve?

Machine-learning results depend on more than code: the data version, the random seed, hyperparameters, library versions and hardware. Git captures the code only. Without a tracker, teams rely on notebooks, spreadsheets and folder names, which lose information and cannot be compared at scale. A tracker records each run automatically, so any result can be reproduced, compared and explained later.

A strong answer adds a concrete example: "Someone asks which learning rate produced the model in production. With a tracker I open the run and read it. Without one I guess."

What is the difference between a parameter and a metric?

A parameter is a setting you chose for the run and it typically does not change during it: learning rate, batch size, number of trees. A metric is a measured value that normally changes over the run, such as loss per step or accuracy per epoch, stored as a time series. Parameters answer "what did I configure", metrics answer "what happened". Parameters accept a step but not an epoch.

What is the difference between an asset and an artifact?

An asset is a file attached to a single experiment and is not versioned. An artifact is a named, versioned, immutable snapshot of files that lives at the workspace level and can be used by many experiments. Comet tracks which experiment produced each artifact version and which experiments consumed it, which gives you lineage. Typical artifact: a dataset. Typical asset: a plot or a model file from one run.

Why would you log a model with log_model rather than as an artifact?

The documentation recommends log_model on the experiment for models, because logged models can be promoted to the model registry with register_model. Artifacts suit datasets and other inputs that many runs reuse.

How do you reproduce a run someone else did?

Open the experiment and read what Comet captured. The Parameters tab gives the settings, the Code tab gives the script, the Installed packages tab gives the environment, and if the work was in a git repository the commit is recorded. The artifact lineage shows which dataset version it used. Then recreate the environment, fetch the artifact and run the code with the same seed. The caveat you should add: anything that was not logged, such as a seed left to chance or data read from an untracked path, cannot be recovered. A tracker records what you give it, plus what it captures automatically.

How do you keep the API key safe?

Never put it in code, because Comet uploads the code. Use comet login, which writes it to ~/.comet.config, or set COMET_API_KEY in the environment, or read it from a secrets manager, for example Colab's secrets. In a team, each person uses their own key. In automated jobs, inject it as a secret from the scheduler or the platform's secret store, and never echo it in a log. If a key leaks, regenerate it in account settings.

What do you do if a run does not show up in the dashboard?

Work from the outside in. First, read the console for a COMET ERROR line, such as an invalid key or Run will not be logged. Second, run comet check, which tests the server, the REST API and the optimizer, and reports proxy and certificate information. Third, check the workspace and project names. Fourth, check whether the run fell back to offline and left a zip that needs comet upload. A good answer is an ordered checklist, not a guess.

What does comet check tell you?

It tests connectivity to the three services the SDK uses, prints the Python and request library versions, proxy settings, whether TLS verification is enabled and the certificate bundle path, and ends with a summary line. It is the first thing to run when a connection misbehaves, and adding --debug shows the raw HTTP exchange.

What changed in how you start an experiment?

Since version 3.47.1, comet_ml.start() is the general-availability entry point. It replaces creating Experiment, ExistingExperiment and OfflineExperiment directly: start() covers new runs, resuming with an experiment_key, and offline mode with online=False. The older classes still work, but new code should use start. The project argument is project_name; the old project spelling is deprecated.

How does the model registry work, and what replaced stages?

A model logged on an experiment can be registered, which creates a named entry in the workspace's registry with a semantic version such as 1.0.0. Each version has tags, notes, a history and a status. Stages such as Staging and Production were deprecated; the stage data became tags and the lifecycle moved to the status field, where changes by regular members go through an approval request that an owner approves. The registered name is lowercase, so "MNIST CNN" becomes mnist-cnn.

How would you compare forty runs?

In the project view, use the table. Sort by the metric you care about, filter by tags or parameters, then select a few rows and open Compare. For more than the compare view handles, about ten experiments in the current cloud release, use a scatter or parallel-coordinates panel on the whole project, or pull the data into pandas with the API and plot it yourself. The answer that impresses is the combination: dashboard for exploring, API for analysis you need to repeat.

What happens if the network drops during training?

The SDK retries a few times with backoff. If the connection does not come back, it falls back to offline mode and writes a local archive, printing the command to upload it. If the connection returns during the run, it uploads what it held. Either way the training itself is not interrupted, and no data is lost unless the process is killed before it can write.

When would you run offline deliberately?

On clusters with no outbound internet, in restricted environments, when you want to avoid rate limits for a very chatty job, or when data must not leave the machine until reviewed. You start with comet_ml.start(online=False) and upload with comet upload later.

How would you use Comet in a team?

Agree on a workspace and on project naming, one project per problem. Agree on a tag vocabulary such as baseline and final. Require that every run logs its seed, dataset name and version. Use artifacts for datasets so lineage exists. Use the registry status for promotion, with an owner approving. And keep keys per person or per service, never shared in chat.

What are the alternatives, and how do you choose?

MLflow, Weights and Biases, ClearML and Neptune solve the same core problem. MLflow is open source and commonly self-hosted. Weights and Biases and Comet are hosted, with polished comparison interfaces. ClearML bundles orchestration. There is no universal winner. Choose by where the data may live, what the team already uses, and integration with the rest of the stack. This guide series covers MLflow, Weights and Biases and ClearML, so you can compare them concretely.

How is Comet different from Opik?

They are two product families on one platform. The MLOps family tracks training experiments, artifacts and models. Opik is an open-source product for tracing and evaluating LLM applications: prompts, tool calls and agent steps. The old comet_llm package is legacy and superseded by Opik. If an interviewer is asking about LLM applications, pivot to Opik.

Is experiment tracking enough for reproducibility?

No. It is necessary, not sufficient. It records settings, code, environment and results, but you still need versioned data, pinned dependencies and fixed seeds. Comet's artifacts help with data versioning for moderate sizes; tools such as DVC approach the same problem from the storage side. Reproducibility also depends on hardware nondeterminism that no logger can remove.

How would you handle sensitive data or data-residency rules?

Be precise. Comet uploads your source code, console output and whatever you log. For regulated data, log aggregates and metadata rather than raw records, avoid logging files that contain personal data, and use remote artifacts so the data stays in a bucket in your region while only a pointer and metadata go to Comet. Some organisations self-host Comet so all of it stays inside their network. Raise these points before an interviewer has to ask, especially in the Gulf and Egypt where residency questions are common.

What a strong answer covers that a weak one misses

A weak answer recites features: "it logs metrics and has a dashboard". A strong answer is shaped like a story: the problem, the model of the tool, one trap you hit and how you fixed it, and one limit you respect.

Strong answers name the current API. Saying comet_ml.start() and login() rather than Experiment() and init() shows recency. Saying "status, not stages" does the same for the registry.

Strong answers separate concepts that beginners blur: parameter against metric, asset against artifact, logging object against API object, offline against failed.

Strong answers admit what they do not know. If asked about the internals of the optimizer, say you used it in a simple search and that the Bayesian option uses earlier trial results, and stop. Inventing details is worse than an honest edge.

Sixty-second self-test

Answer each in one sentence without looking. If you stall, reread the matching section of the detailed guide.

1. What are the four levels of the hierarchy, from the outside in?
2. What is the difference between a parameter and a metric?
3. What is the difference between an asset and an artifact?
4. Which command stores your API key, and in which file?
5. Why should import comet_ml come before import torch?
6. What does exp.end() do, and why does it matter in a notebook?
7. Why does with exp.test(): change the metric name?
8. What does comet check test?
9. A run lost its connection halfway. What do you do?
10. What replaced model stages in the registry?
11. Why is comet_ml.login(api_key="abc") a bad idea?
12. What is the current way to start an experiment, and what did it replace?

Answers in brief: workspace, project, experiment, with the organisation above; settings versus measurements over time; run-bound unversioned file versus shared versioned snapshot; comet login, in ~/.comet.config; so auto-logging can hook the library; flush the queue and close the run, and record cells; it prefixes with the context name; server, REST API and optimizer connectivity; find the zip and run comet upload; status and tags; the key would be uploaded with the code; comet_ml.start(), replacing direct Experiment() creation.

If you answered ten or more without hesitating, you are ready for a beginner-level question on Comet. If fewer, reread the sections on the mental model and on common errors, then run the small end-to-end project from the detailed guide again. Interviews reward people who have done the thing.

This page collects the mistakes beginners actually make with Comet, each with the symptom you will see, the cause underneath it, and the fix. After the mistakes come the habits worth adopting on day one, and a checklist to run before you trust a run. Skim the symptoms first. When something goes wrong, you will recognise yours.

Mistakes beginners make

Mistake 1: nothing shows up in the dashboard

Symptom. The script ran and printed no obvious error, but the project is empty or the run is missing.

Cause. There are four usual suspects. The API key is missing or wrong. You logged into a different workspace or project than the one you are looking at. The script never called start() because it crashed first. Or the connection failed and the run fell back to offline, leaving a zip file on disk.

Fix. Read the console from the top. A line beginning COMET ERROR or Run will not be logged names the cause. Then run comet check. Then look at the URL Comet printed at the start of the run, which says exactly which workspace and project it used, and open that address directly. Finally, look for a .cometml-runs folder next to your script. If it holds a zip, run comet upload on it.

Mistake 2: the API key ends up in the code

Symptom. You wrote comet_ml.login(api_key="...") or set the key as a string in the script, and later find the key visible in the dashboard's Code tab or in a git repository.

Cause. Comet automatically records your source code so that a run can be reproduced. A secret typed into the script is part of that source.

Fix. Run comet login once, which writes the key into your home directory's .comet.config, or use the COMET_API_KEY environment variable. In Colab, read it from the secrets panel. If a key has already leaked, regenerate it in Account settings under API keys and treat the old one as burned. Also add .comet.config to your .gitignore if you ever create a project-level one.

Mistake 3: importing Comet after PyTorch or TensorFlow

Symptom. The run appears, but automatic metrics and parameters from your framework are missing. The console says To get all data logged automatically, import comet_ml before the following modules.

Cause. Comet hooks into ML libraries when they load. If they are already loaded, the hook cannot be applied.

Fix. Make import comet_ml the first import in the file. If your structure makes that hard, run the script as comet python train.py, which sets things up before your code starts. Or disable the automatic part with COMET_DISABLE_AUTO_LOGGING=1 and log everything by hand. Older documentation describes this as an ImportError. Today it is a warning and partial data, which is easier to miss.

Mistake 4: forgetting exp.end()

Symptom. In a notebook, the run shows as still running long after you finished. Or at the end of a script you see Comet failed to send all the data back. Some of your last metrics are missing.

Cause. Comet sends data from a background queue. end() waits until the queue is empty and closes the experiment. Without it, the process may exit first, and in a notebook the kernel keeps the run open indefinitely.

Fix. Call exp.end() at the end of every run. In a script, put it in a finally block if training can raise. In a notebook, call it at the end of the cell that trains, not in a cell you might forget to run. If upload needs longer, raise COMET_TIMEOUT_CLEANING.

Mistake 5: starting a second run and getting the first one

Symptom. You rerun a notebook cell that calls comet_ml.start(), expecting a new run, and find your new metrics appended to the old one.

Cause. If an experiment is already running in the same process and you call start() with matching settings, Comet returns that experiment instead of creating another.

Fix. End the previous run first with exp.end(), then start the next. When you deliberately run many trials in a loop, end each one at the bottom of the loop, as in the detailed guide's end-to-end project.

Mistake 6: using the old API from an old tutorial

Symptom. Deprecation warnings such as comet_ml.init() is deprecated and will be removed soon. Please use comet_ml.login(). Or code that calls Experiment(...), OfflineExperiment(...) and comet init.

Cause. Much of the tutorial material online was written before 2024. Comet changed its recommended entry points, and the old ones survive for compatibility.

Fix. Use the current forms: comet_ml.login(), comet_ml.start(project_name=...), comet_ml.start(online=False) and comet login. Model stages are deprecated too; use status and tags. The upload flag --force-reupload is replaced by --force-upload, and COMET_INI by COMET_CONFIG. When you meet an unfamiliar call, check it against the official documentation before copying it.

Mistake 7: logging every batch

Symptom. The console prints Experiment has been throttled. Some data (like experiment metrics) might be missing. Charts have gaps.

Cause. Comet limits how fast each data type can be sent, for example 12,000 metrics per minute, per type. A loop that logs several metrics on every batch for a large dataset exceeds it.

Fix. Log per epoch, or every hundred steps. Put several metrics into one log_metrics call. Or train offline with comet_ml.start(online=False), which has no rate limits. Artifacts are never throttled, so this only concerns metrics, parameters and output.

Mistake 8: the metric name does not match

Symptom. You look for accuracy in the table and find train_accuracy and test_accuracy. Or a hyperparameter search ignores every trial and prints Optimizer metrics is 'accuracy' but no logged values found.

Cause. Contexts such as with exp.train(): and with exp.test(): prefix the metric names. The name you pass is not the name that is stored.

Fix. Decide on names once and write them in one place, for example a constant. If you use a context, refer to the prefixed name when you configure anything that reads it back. When in doubt, look at the Metrics tab of a finished run, which shows the stored names.

Mistake 9: the project name typo

Symptom. A new empty project appears with a name like breast-cancr, and your run is in it. Your real project is missing the latest run.

Cause. Any project name you pass is created automatically. Comet does not complain about a misspelling, because from its side, it is a valid new project.

Fix. Put the project name in one place, the config file or an environment variable, rather than typing it in every script. If a typo happens, move the run into the right project in the interface and archive the stray one. If you never set a name, runs land in Uncategorized, so set one from the first day.

Mistake 10: a run nobody can understand

Symptom. A month later you open the project and see twenty runs with random names, no tags, and the same three parameters logged. You cannot tell which is the baseline.

Cause. Names and tags take effort at the moment of the run and pay off later. Most beginners skip them.

Fix. Use exp.set_name(...) with a name that states the difference, such as rf-depth-8. Use exp.add_tags([...]) with a small shared vocabulary. Log the seed, the dataset name and size, and anything you changed on purpose. If you cannot describe a run in one sentence, you are not logging enough about it.

Mistake 11: storing a model as an artifact and then hunting for the registry

Symptom. You logged a trained model as an artifact, and now you cannot register it or promote it cleanly.

Cause. The documentation recommends logging models with log_model on the experiment. Registering works from logged models. Artifacts are meant for datasets and other shared inputs.

Fix. Use exp.log_model("name", "path") and then exp.register_model("name"). Keep datasets in artifacts. If you already logged the model as an artifact, log it again properly from a run.

Mistake 12: a surprise in the registry name

Symptom. You registered "Breast Cancer RF" and cannot find it by that name. api.get_model(...) fails.

Cause. Registry names are lowercase, so the stored name is breast-cancer-rf. The first version is 1.0.0, and versions must follow semantic versioning.

Fix. Use the lowercase name with hyphens when you fetch the model. Look at the registry page to confirm the exact name and version.

Mistake 13: pasting an example flag that does not exist

Symptom. comet models download rejects the flags you copied from the help text.

Cause. The built-in help example uses names with underscores that the command does not accept.

Fix. The real flags are --model-name and --model-version, with hyphens, plus -w for the workspace and --output for the folder.

Mistake 14: fighting an SSL error instead of reading it

Symptom. COMET ERROR: There's seem to be an issue with your system's SSL certificate bundle, typically on a laptop that uses a corporate security gateway, or on a macOS Python installed from python.org.

Cause. Python cannot verify the server's certificate. A gateway that inspects encrypted traffic presents its own certificate, which Python's bundle does not know. Or the certificates were never installed for that Python.

Fix. Give Python the right certificates. Set the environment variable REQUESTS_CA_BUNDLE to your organisation's certificate file. On macOS, run the certificate installer that comes with the python.org build. The setting COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 disables verification, but that removes the protection around your key, so keep it as a last resort and never in production.

Habits worth adopting on day one

Make tracking automatic, not optional. Put the comet_ml.start(...) call in a small function your training scripts all use, so that logging the seed, dataset and name is a default, not a chore. The runs you do not log are the ones you will want later.

Log the settings before the training starts. If your script crashes at epoch five, the parameters are already in the dashboard, and you can see what configuration crashed. Log parameters immediately after starting the experiment, not after training.

Name runs after the difference, not the date. Timestamps are already recorded. A name like lr-1e-3-no-augmentation is worth far more than run-2026-10-01.

Keep one project per problem and one tag vocabulary per team. Write the vocabulary in the project's README. Four or five tags are plenty: baseline, tuned, ablation, final, broken.

Always log the things that make a run reproducible. The random seed, the train and test split ratio, the dataset name and row count, the model architecture name, and the git branch if your code is not in git by default. Comet captures code and environment for you, but it cannot guess your seed.

Compare before you believe. When a score improves, open Compare on the new run and the previous best, and read the parameter differences. A surprising improvement often comes from a changed split or dataset, not the change you intended.

Use artifacts for data the moment it changes. Even for a small CSV, versioning it costs three lines and saves an argument later. Lineage tells you which dataset version trained which model.

Keep secrets out of logged values. Anything you pass to log_parameter, log_other or log_asset, and anything your script prints, is stored. Check that you are not logging passwords, tokens, or personal records. If data must stay in your region, log aggregates and pointers, and use remote artifacts for the data itself.

Check the SDK version when behaviour surprises you. Run comet --version and compare it with what the documentation describes. Pin the version in your project's requirements file so that a teammate's run behaves like yours. At the time of writing, the current release is 3.58.7.

Clean up. Archive failed and experimental runs so the project table stays meaningful. Archived runs are hidden but not destroyed, and you can restore them.

Do a dry run without the network. Setting COMET_AUTO_LOG_DISABLE=1 runs your script with Comet calls switched off. Use it when you test code changes or run unit tests, so that you do not fill the project with junk.

Practice cards

Try these short exercises when you have ten minutes. Each one teaches one habit.

Card one: reproduce from the dashboard. Open an old run. Without looking at your code, write down the settings and the dataset from the Comet page alone. Then rerun it from that information. If the score differs, find out what was not logged, and add it to your template.

Card two: break the key. Set COMET_API_KEY to a wrong value, run your script, and read the message. Then fix it. Practise reading errors while the stakes are low.

Card three: compare three runs. Train the same model with three values of one setting. Name them by the value. Select all three and open Compare. Write one sentence about what changed.

Card four: offline round trip. Start a run with online=False, finish it, find the zip, and upload it with comet upload. Confirm it appears.

Card five: version a dataset. Log a CSV as an artifact, edit one row, log it again, and open the artifact page. Confirm two versions.

Card six: register and find. Register a model from a run. Fetch it by its lowercase name with comet_ml.API(). Download version 1.0.0 into a folder.

Small decisions that save you hours

A few judgement calls are worth making once, early, because they are painful to change after forty runs.

Decide what goes in a parameter and what goes in a tag. Numbers and settings you may want to sort by belong in parameters. Labels for groups belong in tags. "Dropout is 0.3" is a parameter. "This was the first attempt after the bug fix" is a tag or a note. Mixing them makes filters clumsy.

Decide what a project is. A project should hold runs you might reasonably compare with each other. If two sets of runs use different metrics or different data, they probably belong in different projects. A project that mixes everything produces charts that overlay things that do not belong together.

Decide when to register. Not every good run becomes a registered model. Register when a model is a candidate for use by someone else, such as a teammate's evaluation or a deployment. The registry is a catalogue for things people will look up. If you register everything, it becomes another cluttered folder.

Decide how long to keep runs. Comet stores a lot, and projects have limits, for example up to 25,000 experiments. You will not reach that in a course. In a team, agree when to archive, and keep the runs behind any model that shipped.

Decide who holds the key. On a shared server or a scheduled job, use a key made for that job, stored as a secret, rather than a person's personal key. Personal keys leave when people do.

A short checklist

Run through this before you call a run "done" and before you share a link.

• The run has a readable name and at least one tag.
• The project is the right one, and its name is spelled as intended.
• The parameters include the seed, the split, the dataset name and its size.
• Training and test metrics are kept apart with contexts or clear names.
• A confusion matrix or other diagnostic is attached for classification.
• The model is logged with log_model, and registered only if it is a candidate.
• The dataset is an artifact if it changed, or if others will reuse it.
• exp.end() ran, and the dashboard shows the run as finished.
• No secrets are in the code, the parameters, or the console output.
• If the run fell back to offline, the zip has been uploaded.
• The SDK version is noted, and pinned in the requirements file.

If every line is ticked, anyone on your team, including you in three months, can open the run and understand what you did. That is the goal. The tool is only the way to get there, and the habits above matter more than any single feature. When you are comfortable with all of this, move on to the Mid-level guide, and when you want to trace LLM applications rather than training runs, pick up Opik.

Where this picks up

You already log runs to Comet. You can call comet_ml.start(), push a few parameters and metrics, and open the experiment page to see a loss curve. This level is for the person who uses Comet at work, where the runs are not a notebook you watch but a hundred jobs launched by a scheduler overnight, where the training node sits behind a proxy, where a teammate needs to reproduce a number you reported last month, and where a model has to move from "it trained" to "it is serving traffic" with someone's approval in between.

The version this guide is written against is comet_ml 3.58.7, uploaded to PyPI on 24 September 2026. One honest caveat before anything else: the official release notes stop at 3.56.0, so there are no public notes for 3.57.x and 3.58.x. Everything below was checked against the shipped code and the current documentation rather than against a changelog, and where the documentation and the code disagree this guide says so and follows the code.

By the end you should be able to predict what the SDK does when the network drops mid-run, choose where a setting belongs among the eight places Comet looks for one, design artifacts and registry entries so that lineage survives a team of ten, run a hyperparameter search in parallel without corrupting it, and wire registry events into your delivery pipeline. If you need the basics again, the beginner guide covers them. If you are choosing between trackers, the neighbouring guides on MLflow, Weights & Biases and Opik are the natural comparisons.

How the SDK works under the hood

Most production surprises with Comet come from not knowing that log_metric does not talk to the server. When you call it, the SDK puts a message on an in-process queue and returns. A background thread drains that queue, batches messages, optionally compresses them, and sends them to the server's /clientlib/ endpoint. Files such as images, models and assets are uploaded separately, directly to object storage, with multipart upload for large files. The REST API at /api/rest/v2/ and the optimizer service at /optimizer/ are distinct endpoints that the SDK uses for other jobs.

That design explains three behaviours you will meet. First, your training loop is almost never slowed by logging, because it only pays for the enqueue. Second, a run can finish its Python code while the uploader is still busy, which is why exp.end() exists: it waits for the queue and the uploads to drain, and it waits at most comet.timeout.cleaning, which defaults to 3,600 seconds. If the process is killed before that flush completes, the server never receives the tail of your data, and the SDK's next run will not fix it. Third, because the uploader is a thread, what you log is not guaranteed to be visible the instant you call it. Do not write code that logs a value and immediately reads it back through the API.

The handshake matters too. comet_ml.start() first contacts the server to validate the key and create or fetch the experiment. If that fails, you see Run will not be logged and the script carries on without tracking. This is deliberate: a tracking outage should not kill a training job. It is also why a silent wrong API key is a real operational risk, and why the section on testing later shows how to turn these soft failures into hard ones.

Three kinds of limit shape what reaches the server. Throttling is per data type: metrics at 12,000 per minute, parameters at 10,000 per minute, captured output at 12,000 per minute, and everything else at 8,000 per minute. When you exceed one you get Experiment has been throttled. Some data (like experiment metrics) might be missing, and the message adds that mission-critical data such as artifacts is never throttled. Sampling is separate: a single metric keeps at most 15,000 values per experiment by reservoir sampling, so a curve logged every step for a very long run is a sampled picture, not the full record. Hard caps are the third kind: 25,000 experiments per project and, per experiment, 50,000 unique metrics, 50,000 unique parameters, 100,000 images, 1,000 curves, 100 models, 500 text samples and 5,000 videos.

The practical consequence is that a training loop should log in proportion to what you will read. A metric every step is useful for a short debugging run and wasteful for a three-day job, where a value every 50 steps plus an epoch summary gives you the same conclusions with a fraction of the throttle budget. Batch related values with log_metrics rather than calling log_metric per name, because one message carrying a dictionary is cheaper than ten.

Configuration as a system

Comet reads settings from eight places, and knowing the order is the difference between a five-minute fix and an afternoon of guessing. From highest priority to lowest, the current code looks at: explicit arguments in code, such as comet_ml.start(api_key=..., ...) or an ExperimentConfig; operating-system environment variables named COMET_<SECTION>_<NAME>; a .env file in the working directory; the file named by COMET_CONFIG, which may be a file or a directory; a .comet.config in the working directory; a .comet.config at /content/drive/MyDrive/ for Colab users; ~/.comet.config; and finally overrides provided by the backend.

Two consequences follow. An environment variable beats any config file, so a scheduler that sets COMET_PROJECT_NAME will override the project your repository's .comet.config names, which is exactly what you want in CI and exactly what confuses people running the same code locally. And a .comet.config committed to a repository sits above the developer's personal ~/.comet.config, so the project-level file should hold project-level facts (workspace, project name, logging preferences) and never a key. The older variable COMET_INI still works but warns COMET_INI is deprecated; use COMET_CONFIG, even though the configuration page of the docs still lists COMET_INI as current. Use COMET_CONFIG.

The file itself is INI with sections:

[comet]
workspace = vision-team
project_name = defect-detector

[comet_logging]
file = /tmp/comet-defect.log
file_level = debug

[comet_auto_log]
git_patch = True
env_cpu_per_core = False

Each key has an environment form: [comet] workspace is COMET_WORKSPACE, [comet_auto_log] code is COMET_AUTO_LOG_CODE, and so on. COMET_AUTO_LOG_DISABLE switches off all network communication, which is the disabled mode. Since 3.37.2 that mode does not need an API key, so it is the right setting for unit tests and dry runs. COMET_DISABLE_AUTO_LOGGING=1 is a different switch: it turns off framework auto-logging only.

For per-run behaviour, pass an ExperimentConfig. It is the place for the flags that used to be constructor arguments on the old Experiment class.

import comet_ml

config = comet_ml.ExperimentConfig(
    name="resnet50-lr3e-4",
    tags=["baseline", "nightly"],
    log_code=True,
    log_git_patch=True,
    auto_metric_step_rate=10,
    auto_output_logging="simple",
    display_summary_level=1,
)
exp = comet_ml.start(project_name="defect-detector", experiment_config=config)

Several defaults deserve attention. auto_output_logging changed from "native" to "simple" in 3.47.3, so anything you read online about how stdout is captured may describe the old behaviour. auto_metric_step_rate is 10, meaning framework auto-logging records a metric every tenth step rather than every one. Histogram, gradient, activation and weight logging are all off by default because they are expensive. auto_log_co2 is on by default, which records carbon estimates through codecarbon when it is available, so turn it off if you do not want that overhead or data.

System metrics (CPU, GPU, memory, network, disk) sample every 30 seconds. The configuration docs still say 60, but release 3.39.0 lowered the default and the code says 30. The minimum is one second, which is rarely a good idea on a busy node.

Experiment identity, modes and resuming

An experiment is identified by its key, and the mode argument to start() controls how the key is used. The default, get_or_create, reuses an experiment if the key exists and makes a new one otherwise. get demands the experiment exist: it raises ExperimentNotFound if it does not, and InvalidExperimentMode if you forgot to supply a key. create always creates, and raises if the supplied key is already taken. That last one is the right choice inside hyperparameter loops and retry wrappers, where silently attaching a second attempt to the first attempt's experiment would merge two different runs into one confusing chart.

Keys you supply must be alphanumeric and between 32 and 50 characters, otherwise you see COMET_EXPERIMENT_KEY is invalid: '%s' must be alphanumeric and between 32 and 50 characters. Two ways to produce one are uuid.uuid4().hex, which is exactly 32 characters, and comet_ml.get_experiment_key(...), which derives a stable key from input you provide. The derived form is what you want when the same logical job, say the training of a named model on a named date, must land on the same experiment every time it is retried.

import comet_ml

key = comet_ml.get_experiment_key("defect-detector", "2026-10-01", "seed-7")
exp = comet_ml.start(experiment_key=key, project_name="defect-detector")

Resuming is where people get hurt. When you resume an existing experiment, these defaults are switched off so that you do not overwrite what the first attempt recorded: log_code, log_graph, parse_args, log_env_details, log_git_metadata, log_git_patch, log_env_gpu, log_env_cpu and log_env_host. That is correct behaviour, but it means a resumed run on a different machine will not update its hardware description. If it matters, log it yourself with log_other.

There are three ways to reach a running or finished experiment, and they serve different purposes. comet_ml.start(mode="get", experiment_key=...) continues live logging during training, for instance after a preemption. comet_ml.APIExperiment(previous_experiment=...) is a REST wrapper for reading or appending after the fact, for instance attaching an evaluation result to last week's run. comet_ml.get_running_experiment() returns the experiment already active in this process, or None, which is how library code can log into the caller's experiment without being handed it.

Another thing start() does quietly: if an experiment is already running in the process, it returns that one when the key settings match (api_key, disabled, distributed_node_identifier, experiment_key, offline_directory, online, project and workspace), and otherwise ends the running one first. In a test suite this can make two tests share an experiment, so either end the experiment in teardown or use mode="create".

Logging that stays useful

The logging methods you already know have options that matter at scale. log_metric(name, value, step=None, epoch=None, include_context=True) and log_metrics(dic, prefix=None, step=None, epoch=None, include_context=True) record time series. log_metrics flattens nested dictionaries since 3.43.2, so a dict of dicts becomes dotted names. Parameters are different in kind: log_parameter(name, value, step=None) and log_parameters(parameters, prefix=None, step=None, nested_support=True) record configuration, they accept step but not epoch, and they handle nested dictionaries with nested_support=True. log_other(key, value) is for everything that is neither, such as the dataset hash or the commit of a dependency. The optimizer writes its own values there under the prefix optimizer_.

Context managers change names without you touching every call. Inside with exp.train(): metrics are recorded with a train prefix, exp.validate() and exp.test() do the same for their phases, and exp.context_manager("name") makes your own. set_step and set_epoch set defaults so individual calls can omit them. Used consistently, this gives you charts where train and validation curves share an x-axis without any string formatting in your loop.

for epoch in range(epochs):
    exp.set_epoch(epoch)
    with exp.train():
        for step, batch in enumerate(train_loader):
            loss = train_step(batch)
            if step % 50 == 0:
                exp.log_metrics({"loss": loss, "lr": scheduler.get_last_lr()[0]},
                                step=global_step)
    with exp.validate():
        exp.log_metrics({"loss": val_loss, "f1": val_f1}, step=global_step)
    exp.log_confusion_matrix(y_true=y_true, y_predicted=y_pred, labels=labels,
                             title=f"epoch-{epoch}", step=global_step)

Pay attention to log_parameter versus get_parameter. Reading a parameter by a short name when you logged it with a prefix or nested dictionary triggers Getting a parameter without its full name is deprecated, and the warning lists the candidate full names. Always read by the full name.

For figures and images, log_figure(figure_name=None, figure=None, overwrite=False, step=None, metadata=None, format="auto") has had format="auto" since 3.51.0, so a Matplotlib figure does not need a manual format choice. log_image and log_figure count against the 100,000-image cap, so logging one per step for a week-long run is a way to hit a hard limit; log a fixed grid per epoch instead. For files, log_asset(file_data, file_name=None, overwrite=False, copy_to_tmp=True, step=None, metadata=None) attaches an unversioned file to one experiment. The copy_to_tmp argument is a deprecated no-op on this and on the other log methods, so leave it out. When a file is larger than the upload limit you get Asset %r size %d is greater than the upload limit %d (an AssetIsTooBig), and the fix is a remote reference instead of an upload, covered in the next section.

In notebooks, call exp.end() explicitly, because it also logs the executed cells. exp.display(tab="panels") embeds the UI in the cell, with tabs such as code, metrics, parameters, artifacts and system-metrics.

The import order rule still applies in a service codebase: import comet_ml before torch, keras, tensorflow, fastai and similar. Today the SDK only warns, with To get all data logged automatically, import comet_ml before the following modules: %s, and the troubleshooting docs describe an older ImportError version of the same problem. The warning matters because auto-logging silently degrades. If you cannot control import order, run comet python script.py, which bootstraps the SDK first.

Artifacts and lineage

An asset is a file attached to one experiment. An artifact is a versioned, workspace-level object: each version is an immutable snapshot of files and assets arranged in a folder-like structure, with metadata, tags and aliases, and Comet records which experiment produced each version and which experiments consumed it. That last part is lineage, and it is the reason to use artifacts for datasets instead of dumping CSVs into assets.

from comet_ml import Artifact, start

exp = start(project_name="defect-detector")

art = Artifact(name="defects-train", artifact_type="dataset",
               metadata={"rows": 48211, "source": "line-3-camera"})
art.add("data/train.parquet", logical_path="train.parquet")
art.add_remote("s3://plant-data/labels/2026-09/")
exp.log_artifact(art)
exp.end()

Consumption is the other half, and it is what builds the graph:

exp = start(project_name="defect-detector")
logged = exp.get_artifact("defects-train", version_or_alias="latest")
logged.download("./data", overwrite_strategy=False)

Because the consuming experiment called get_artifact, the lineage graph now shows the dataset version feeding this run. If instead you read the file from a path on a shared drive, the link does not exist and, a month later, nobody can say which data trained the model.

Some details decide whether this works for a team. Names are limited to 100 characters, and since 3.39.0 a longer one raises an error rather than truncating. Versions are immutable; you change the data and log again to get the next version, and you use aliases such as latest or prod-candidate for moving pointers. Artifacts are workspace-level, so one dataset version can be consumed from several projects. Artifacts are also never throttled, which makes them a safe place for data you cannot afford to lose.

Remote artifact assets are references rather than copies. For s3:// and gs:// URIs, add_remote lists the matching objects and records the URI, a checksum and, if bucket versioning is on, the object version ID. That last point is the important one: a remote reference to an unversioned bucket prefix records a location, not content, so someone overwriting the object changes what your "immutable" artifact means. Turn on bucket versioning for any prefix you register. Also know that remote sync needs credentials configured the provider's normal way on the machine doing the logging; without them only the URI string is recorded. sync_mode=True is the default and max_synced_objects defaults to 10,000. Version 3.56.0 fixed connection leaks with add_remote(sync_mode=True) and the handling of / in artifact names, so stay current if you used either before.

The documentation recommends logging models with log_model on an experiment rather than as artifacts, so that they can be promoted to the registry. Keep artifacts for data and derived files, and models in the model path.

Comet also has a Hugging Face bridge: Experiment.get_hf_dataset(), added in 3.52.0, is worth knowing if your datasets already live on the Hub. For versioning data in a git-centred way instead, DVC is the neighbouring tool, and the two can coexist with the DVC revision recorded as artifact metadata.

The model registry as a release process

Logging a model is log_model(name, file_or_folder, file_name=None, overwrite=False, metadata=None, copy_to_tmp=True, prepend_folder_name=True) on an experiment. Registering it is a second, deliberate step that promotes it into a workspace-level catalogue where versions, tags, status and history live.

exp.log_model("defect-detector", "./output/model")
exp.register_model("defect-detector", version="1.2.0",
                   description="Adds night-shift images",
                   tags=["candidate"], status="Development")

Registry names are lowercase, so a model logged as MNIST CNN registers as mnist-cnn, and versions are semantic versions: 1.0.0 by default, with 2.1.5 and 1.0.0-alpha also valid. The call is register_model(model_name, version=None, workspace=None, registry_name=None, public=None, description=None, comment=None, tags=None, stages=None, status=None, sync=False, status_configuration=None). Two of those arguments are dead ends. workspace is deprecated and ignored; the message is that to register into another workspace you first log the model to the registry and then use the move option in the UI. stages is deprecated with the whole stage concept.

That last point is the most common source of outdated code at work. Older tutorials promote models by moving them between stages. Stages are deprecated: their data became tags, and the lifecycle mechanism is now status, with an approval workflow. Every stage-based API method emits a deprecation warning. The replacement path goes through api.get_model(...):

import comet_ml

api = comet_ml.API()
model = api.get_model(workspace="vision-team", model_name="defect-detector")
model.find_versions(status="Staging")
model.set_status(version="1.2.0", status="Production")
model.add_tag(version="1.2.0", tag="customer-a")
model.download("1.2.0", output_folder="./serving-model")

Status changes by ordinary members do not apply directly. They create approval requests that a workspace owner approves, unless a member has been granted approval rights, which has been possible at workspace level since July 2025. This is the feature to lean on for governance: it means "Production" in the registry reflects a decision by someone accountable, not a keystroke by whoever trained the model. Since June 2025, model versions also carry a status configuration, key-value data per target environment. The docs note that this changed the structure of the webhook payload, so any receiver written before then deserves a check.

Webhooks turn registry activity into automation. A webhook on a registered model fires on MODEL_VERSION_CREATED, MODEL_VERSION_DELETED, MODEL_TAG_CHANGED or MODEL_STATUS_CHANGED (plus a TEST event for verifying your endpoint). An optional secret is sent as the Authorization header, and your receiver must validate it, because otherwise anyone who discovers the URL can trigger your deployment. Custom headers are allowed and there is a payload format for GitHub Actions, which is the quickest route to a pipeline that starts when status moves to Production. See GitHub Actions for the receiving side.

From the command line, comet models list -w WORKSPACE lists models and comet models download -w WORKSPACE --model-name NAME --model-version 1.0.0 --output DIR fetches one, with the output directory defaulting to model. The built-in help example shows --registry_name and --model_version, but those are not real flags; the real ones use hyphens as above. Avoid --model-stage, which relates to the deprecated stages.

Mind the download limits. Registry downloads return HTTP 429 beyond 5,000 per day per account and per model version on paid plans, 1,000 on free plans, and 20 per second. A fleet of pods that each download the model on start, during a rollout, can hit that. Cache the model in your own storage or bake it into the image at build time.

Hyperparameter search with the Optimizer

Comet's Optimizer is a client-server service. You describe a search, the server hands each process a set of parameters, and every trial becomes its own experiment. Algorithms are grid, random and bayes; Bayes uses sequential model-based optimisation with a Tree Parzen Estimator.

import comet_ml

config = {
    "algorithm": "bayes",
    "spec": {"maxCombo": 20, "metric": "val_f1", "objective": "maximize",
             "retryLimit": 20, "retryAssignLimit": 0},
    "parameters": {
        "learning_rate": {"type": "float", "scalingType": "loguniform",
                          "min": 1e-5, "max": 1e-3},
        "batch_size": {"type": "discrete", "values": [32, 64, 128]},
    },
    "name": "defect-detector-hpo",
    "trials": 1,
}

opt = comet_ml.Optimizer(config=config)
for exp in opt.get_experiments(project_name="defect-hpo"):
    lr = exp.get_parameter("learning_rate")
    bs = exp.get_parameter("batch_size")
    score = train_and_eval(lr, bs)
    exp.log_metric("val_f1", score)
    exp.end()

The details that decide whether a search is meaningful sit in spec. metric defaults to "loss", so if you log val_f1 and forget to name it, the server will optimise a metric you never logged. The console then says Optimizer metrics is '<metric_name>' but no logged values found. Experiment ignored in sweep, and for Bayes that is critical, because ignored trials teach it nothing. objective is minimize by default and only drives Bayes. maxCombo is how many combinations to try (by default all), and total runs equal maxCombo times trials. gridSize defaults to 10 and minSampleSize to 100, both for grid and random. retryLimit defaults to 20 and retryAssignLimit to 0.

Parameter types are integer, float, double, categorical and discrete. scalingType is linear by default, or uniform, normal, loguniform, lognormal. Use min and max with linear, uniform and loguniform, and mu and sigma with normal and lognormal. Some documentation examples write log_uniform; the reference says loguniform, so use that. For a learning rate, log scaling is nearly always the right choice, because a linear range from 1e-5 to 1e-3 spends almost all of its samples above 1e-4.

To parallelise, use the CLI rather than threads:

export COMET_OPTIMIZER_ID=$(comet optimize opt.json)
comet optimize -j 4 -t 1 train.py opt.json

comet optimize [-j PARALLEL] [-t TRIALS] [-e EXECUTABLE] [-d DUMP] PYTHON_SCRIPT [OPTIMIZER] takes a JSON file or an existing optimizer ID, and your script reads it from sys.argv[1]. Each process runs (maxCombo * trials) / j of the work, and sets COMET_OPTIMIZER_ID, COMET_OPTIMIZER_PROCESS_ID and COMET_OPTIMIZER_PROCESS_JOBS in its environment. The first command above creates an optimizer without running it, which is useful when several scheduler jobs should join one search by sharing the ID. Setting COMET_OPTIMIZER_ID also resumes a search.

Do not parallelise Bayes fully. It chooses the next point from earlier results, so ten workers starting simultaneously each choose from almost no information, and you have paid for a random search. Use a few workers for Bayes and many for grid or random. If the optimizer runs out of memory, the search space is too big: remove parameters or lower gridSize and minSampleSize.

A deprecation trap: passing an Experiment to the Optimizer constructor logs a warning telling you to pass it to get_experiments or next instead. The warning is safe to ignore but it means you are using an old pattern. Since 3.55.0 there is also Optimizer.update(), which lets you send metadata and status back.

Querying what you logged

The API object is the read side. comet_ml.API(api_key=None, cache=True, version="v2") caches responses by default, which is convenient and occasionally confusing: if you write through one path and read through an API instance, call clear_cache(). The old update_cache() is deprecated in favour of clear_cache().

The basics are api.get("workspace/project/experiment"), api.get_experiments(workspace, project_name=None, pattern=None, page=None, page_size=None, sort_by=None, sort_order=None) and api.get_experiment_by_key(key). get_experiment_by_id is deprecated in favour of the by-key form. For filtering, there are two tools. api.query(workspace, project_name, query, archived=False) takes a query object built from Tag(), Metric(...) and Parameter(...) in comet_ml.query and combines with &. api.search(workspace, project_name, query, archived=False, page=None, page_size=None) was added in 3.54.0 and supports both AND and OR with pagination, which query does not, so prefer it for anything large.

import comet_ml
from comet_ml.query import Tag, Metric

api = comet_ml.API()
runs = api.query("vision-team", "defect-detector",
                 (Tag() == "baseline") & (Metric("val_f1") > 0.9))
df = api.get_metrics_df([r.key for r in runs], ["val_f1"], x_axis="step")

get_metrics_df(experiment_keys, metrics, x_axis="step", interpolate=True) hands you a DataFrame for analysis outside the UI. Two cautions. It returns sampled metrics, so if a number in a review has to match the stored record exactly, enable total fidelity (the organisation setting "Store metrics without sampling") and read with APIExperiment.get_metric_total_df(), added in the 3.49 line. And the REST API behind all of this limits you to 15,000 submissions per hour per API key. A dashboard script that polls every few seconds with a shared key can exhaust that for everyone using the same key, which is why the docs recommend distinct service accounts for distinct purposes.

For outside-the-SDK access, the REST API lives at https://www.comet.com/api/rest/v2/ with the API key in the Authorization header, and at https://<host>/api/rest/v2/ on a self-hosted install. Endpoints divide into read, write and monitoring groups.

The community cometx package (not an official Comet project) adds CLI commands for bulk work: list, download, copy, log, delete-assets, reproduce, update, config, smoke-test and an admin chargeback-report. Its most valuable use at work is migration: cometx download --from comet|wandb|neptune supports moving in from other trackers, and copying between installations is a two-step download then copy.

When the network fails: offline, fallback and error mode

A tracking client has to decide what to do when the server is unreachable, and Comet gives you three levers. The first is explicit offline mode: comet_ml.start(online=False) writes the experiment to a local zip, by default in .cometml-runs, with no rate limits at all and only the hard limits. You upload later with comet upload.

exp = comet_ml.start(
    online=False,
    experiment_config=comet_ml.ExperimentConfig(offline_directory="/scratch/comet"),
)

At the end the SDK prints the exact command, comet upload /scratch/<key>.zip. comet upload [--force-upload] [--workspace W] [--project-name P] archives... accepts globs, and comet offline inspects a zip without uploading. --force-reupload is deprecated; use --force-upload. If comet is not on the PATH, which happens on Windows when the Scripts directory is missing from it, python -m comet_ml.scripts.comet_upload does the upload. For automation, exp.set_offline_zip_uploader(callback) lets you ship the zip to S3 or another store yourself, which suits clusters with no route to Comet from the compute nodes but a route to storage.

The second lever is fallback to offline, which needs backend 3.3.11 or later. If connectivity is lost mid-run, the SDK retries (comet.http_session.retry_total defaults to 3, with backoff factor 2 so waits are factor * 2**(attempt-1)), then switches to local logging. If the connection comes back during the run it re-uploads; otherwise it leaves a zip and prints Could not send live data to Comet during experiment runtime. An offline experiment will be available for upload, followed by an upload command that still shows the deprecated flag. Use --force-upload. Setting comet.fallback_streamer.keep_offline_zip to true keeps a local zip for every run, a belt-and-braces option for jobs that cannot be repeated.

Timeouts: comet.timeout.http and comet.timeout.api default to 10 seconds, the optimizer timeout to 30, and comet.timeout.file_download to 600. If end() hits timeout.cleaning or the process dies you see Comet failed to send all the data back (%d messages and %d uploads), experiment ID: %s, which is the signal to raise COMET_TIMEOUT_CLEANING or fix the network.

The third lever is error mode, comet_ml.error_mode(). By default invalid, discarded or truncated data produces a warning and the run continues. In error mode those cases raise. It applies for the whole process, it covers log_artifact, log_model and log_metric(s) among others, and it does not make asynchronous upload errors raise, since they happen on another thread. Use it in CI and integration tests, where you want a bad call to fail the build, and leave it off for production training where tracking must not kill a job.

Distributed training and framework integrations

With multiple processes, create one Experiment object per process, not a shared one. There are two patterns. In the single-experiment pattern all ranks share a pre-set experiment_key and you pass ExperimentConfig(distributed_node_identifier=...) so that Comet can tell the nodes apart. A race in multi-node creation was fixed in 3.49.2 and retries were lengthened in 3.49.12, so use a recent SDK on clusters. In the per-worker pattern each rank makes its own experiment and you group them with a shared run ID that you log as a parameter or tag.

The single-experiment pattern is tidier for a large training job, where you want one chart; the per-worker pattern is safer for jobs that are really independent workers, such as data-parallel evaluation shards. In either case, compute the key once in the launcher and pass it in, rather than letting each rank generate its own.

Comet auto-logs a long list of frameworks: TensorFlow, Keras, PyTorch, PyTorch Lightning, Hugging Face Transformers and Accelerate, XGBoost, LightGBM, scikit-learn, fastai, Ray, DeepSpeed, Composer, Catalyst, Ludwig, MLflow, Prophet, PyCaret, PySpark, SHAP, Spark NLP, Gymnasium and Matplotlib. Auto-logging is convenient but not free of judgment, because it decides what to name and how often to record. Use the comet_auto_log keys to trim what you do not want rather than disabling everything, and keep your own log_parameters for the values that define the experiment, since auto-captured parameters depend on what the framework exposes.

Helpers live in comet_ml.integration.*: pytorch.log_model, load_model and watch; sklearn.load_model; kubeflow.initialize_comet_logger (which needs kfp); metaflow with @comet_flow and @comet_step; ray; sagemaker; vertex with CometVertexPipelineLogger; snowflake; spacy; and gymnasium. For PyTorch Lightning, Kubeflow, Metaflow, Ray, SageMaker and Vertex AI, the neighbouring guides cover the orchestration side and this section's helpers cover the tracking side. The Keras callback comes from exp.get_callback('keras'); the older get_keras_callback() is deprecated.

Secrets, access and platform controls

The API key is a bearer credential with the user's rights. In practice you need a way to get it to jobs without putting it in code, images or logs. On Kubernetes, put it in a Secret and inject it as COMET_API_KEY. On AWS or GCP, the SDK ships a secrets-manager integration: from comet_ml.secrets.aws import get_api_key_from_secret_manager and the GCP equivalent, with store_api_key_in_secret_manager to write one. The AWS path needs boto3 and the GCP path needs google-cloud-secret-manager 2.0.0 or later. The result is an "API Key Secret" string you pass as api_key. The prose of the docs shows comet_ml.secret.aws with a singular secret; the package actually ships comet_ml/secrets/aws, plural, so use that. See Vault if your organisation keeps secrets there. If your identity provider issues JWTs, COMET_API_KEY or api_key= can hold a JWT when the organisation enables it, which removes long-lived keys from the picture.

hide_api_key defaults to true and redacts keys in Comet's logs, and since 3.49.1 the Everett configuration logger is forced to warning level so that keys do not leak through it. Do not rely on redaction as a substitute for not printing the key yourself.

Roles: organisation members are Opik-only, platform members (experiment management plus Opik in invited workspaces), admins, or view-only platform members, who cannot log and get ViewOnlyAccessException if they try. In a workspace, members and owners differ mainly in who approves model status changes, manages webhooks, makes projects public and manages users. Service accounts, which are Enterprise only, have names prefixed sa-, come with an API key, and by default carry organisation-admin-level permissions. Treat their keys as very privileged, restrict the workspaces they can reach in the Admin dashboard, and use different service accounts for different jobs so one runaway script does not consume the REST quota of another. If a service account has no default workspace, you must pass workspace explicitly or get an error, and Service account doesn't have access to workspace %s means it has not been authorised for that workspace.

Projects can be public or private, and a private experiment can be symlinked into a public project without being copied. Multi-team setups usually map a workspace per team; artifacts and the registry are workspace-scoped, which is a boundary to design around, not discover.

Running Comet in your own infrastructure

Many employers in the Gulf and Egypt need tracking data to stay inside a region or a network, which is why the self-hosted and single-tenant options matter. The documentation describes new self-hosted deployments as Kubernetes-only on EKS, GKE or AKS, plus the AWS SageMaker Partner AI App; the older single-VM "cometctl" install still appears in pages on SSL and backups but is legacy. A Helm install is helm upgrade --install --namespace comet-ns --create-namespace -f override-values.yaml comet-ml ., and a healthy cluster shows pods for the Python backend, the React backend, the optimizer, post-processing, an nginx frontend, MySQL, Redis, MinIO and a Python panels orchestrator, along with an Opik stack that uses ClickHouse. In production you point these at managed MySQL, Redis and object storage instead of the in-cluster MinIO. See Kubernetes and Helm for the operational skills this assumes.

From the client side, a self-hosted server needs COMET_URL_OVERRIDE (INI key url_override) pointing at https://<host>/clientlib/, unless smart keys are enabled on the deployment, in which case the SDK derives the URL from the API key (since 3.40.0). Verify a deployment with curl http://<host>/api/isAlive/ping, kubectl -n <ns> get pods and cometx smoke-test WORKSPACE. TLS terminates at a layer-7 load balancer (an AWS ALB rather than a classic ELB, a GCP HTTPS load balancer, or an Azure Application Gateway). If the certificate is self-signed or your company inspects TLS, SDK hosts must trust the CA, through the OS store, REQUESTS_CA_BUNDLE or comet.include.custom_ca_bundle. The setting COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 disables verification and is a last resort.

Uploads are where on-premises installs most often break. Files larger than about 8 MB use multipart upload, so the bucket policy must allow s3:PutObject, s3:PutObjectAcl, s3:InitiateMultipartUpload, s3:UploadPart, s3:CompleteMultipartUpload, s3:AbortMultipartUpload and s3:ListMultipartUploadParts. A missing one shows up as a 500 with failed to save asset or an AmazonS3Exception: Access Denied. Keep SDK and backend within reach of each other: some features require minimum backend versions, and an old server can raise BackendVersionTooOld while an old SDK gets Obsolete SDK version. Please update.

Self-hosted installs send usage reports to stats.comet.com on TCP 443, including usernames, emails, the licence key and versions. If your network is air-gapped or restricted, raise this with Comet early, because the default expects outbound access.

Testing and debugging

Tracking code deserves tests because its failure mode is silence. A test suite should at minimum run with COMET_AUTO_LOG_DISABLE=1 so no test needs a network or key. A second, thinner layer runs a smoke job against a real workspace with comet_ml.error_mode() on, to prove the credentials, project and permissions work. The community cometx smoke-test WORKSPACE[/PROJECT] exercises optimizer, panels, experiments, metrics, images, assets, dataset info, confusion matrices and embeddings in one go, which is a fast check after a server upgrade.

For debugging, console logging at DEBUG is forbidden since 3.45.0; setting COMET_LOGGING_CONSOLE=debug prints Switching the console logger level from DEBUG to INFO, as DEBUG is now deprecated. Use a file log instead, set before comet_ml is imported:

COMET_LOGGING_FILE=/tmp/comet-{project}.log \
COMET_LOGGING_FILE_LEVEL=debug \
python train.py

The path may contain patterns like {project}. Read the file for the handshake result, the retry behaviour and the upload queue. comet check --debug prints the raw HTTP exchange against the server, REST API and optimizer, along with the Python and requests versions, proxy settings and CA bundle path. It ends with a summary line such as Server connectivity True. The docs' sample also shows a websocket check, but 3.58.7 does not run one, so do not treat its absence as a fault.

A short table of symptoms worth memorising: The given API key '%s' is invalid on '%s' means a wrong key or a key from another installation; Run will not be logged means the handshake failed (proxy, TLS, outage); Workspace %s doesn't exist. is a typo or missing membership; Unsupported experiment start mode means mode is not one of the three; and Maximum number of experiments reached for project means 25,000 is exhausted and you should archive or move to a new project. An SSL certificate bundle error is usually a system problem, not Comet's: the macOS python.org build, corporate TLS inspection or a self-signed server. Metrics that look sparse or smoothed usually mean the 15,000-value sample, not a bug.

Putting it all together

Take a realistic week. A team trains a defect detector on images from a factory line. The dataset is versioned as an artifact defects-train, registered from a remote, versioned S3 prefix, with row counts and the source camera in metadata. A nightly job starts with comet python so import order never matters, reads its project from .comet.config and its key from a Kubernetes secret, calls get_artifact("defects-train", version_or_alias="latest") so the lineage records the data, and names the run from a derived key so a retry lands on the same experiment.

Training logs a metric every 50 steps with log_metrics, parameters once through log_parameters, and a confusion matrix per epoch, well under the throttle and image caps. Compute nodes cannot always reach Comet, so the job runs offline and uploads its zip from a sidecar, with fallback_streamer.keep_offline_zip on as insurance. A weekly search uses comet optimize -j 4 with Bayes against val_f1, objective set to maximize, and the exact metric name logged. The best run's model is logged with log_model and registered as a new semver version with status Development.

Promotion is a human step: a workspace owner approves the status change to Staging, a webhook with a validated secret starts a deploy job, and that job downloads the model once from the registry with comet models download, stores it in internal storage, and deploys from there so a rollout never meets the download limit. The CI suite runs with auto-logging disabled, and a scheduled smoke test runs with error mode on, so a silent tracking regression fails loudly within a day. When the team later compares this setup with MLflow or Weights & Biases, they know exactly which of these guarantees they are getting and which they would have to build.

Where you are now

You now know that logging is asynchronous and what that costs you at shutdown, that Comet reads settings from eight places in a fixed order, and that experiment identity is a key you can derive, resume and protect with modes. You can design artifacts so that lineage is real, run the registry as an approval-gated release process with stages replaced by status, run Bayes and grid searches in parallel without lying to yourself about the metric, and choose between offline, fallback and error mode based on what a failure should do to your job.

What remains for the senior level is the platform view: running Comet for other teams, scaling and cost limits, single-tenant and JWT setups, upgrade and migration planning, and deciding when another tool is the better answer. Before that, the practice that pays most is unglamorous: pick one real project at work and make its runs reproducible from a cold start, which is the test that exposes every gap this guide mentioned.

Sources

• Comet documentation home
• Python SDK releases
• Starting an experiment
• ExperimentConfig reference
• Experiment reference
• API reference
• Configure the SDK
• Resume an experiment
• Limits and performance
• Total fidelity metrics
• Distributed training
• Troubleshooting and FAQ
• Running offline
• Fallback to offline
• Error mode
• API key secret
• Using artifacts
• Remote artifacts
• Artifact lineage
• Using the model registry
• Model approval
• Model registry webhooks
• Optimizer configuration
• Running the optimizer in parallel
• Command-line reference
• REST API overview
• Admin dashboard and service accounts
• Deploying Comet
• Smoke testing a deployment
• Integrations overview
• comet_ml on PyPI

Part two of three for the mid level. Interviews for someone who uses Comet at work are rarely about definitions. They give you a situation, such as runs that vanish, a search that never improves or a model that reached production without review, and watch whether you reason from how the tool actually behaves. Each question below has a reasoned answer, then what a strong answer adds and what a weak one misses. The version discussed is comet_ml 3.58.7.

How to use this file

Read the scenario, answer it aloud before you read on, and compare. The goal is not to recite the answers but to practise the move every good answer makes: name the mechanism, then name the check that would confirm it. Interviewers at this level are listening for that second half. "It is probably throttled" is a guess; "it is probably throttled, and I would look for the throttle warning in the console and the warning icon on the experiment, then check the logging rate per step" is an engineer.

Scenario questions

A training job finished, but the last hundred steps of loss are missing in Comet

Start from the architecture. Logging calls enqueue messages, and a background thread sends them, so a job can finish executing while data is still in the queue. exp.end() waits for the queue and uploads to drain, for at most comet.timeout.cleaning, default 3,600 seconds. Missing tail data usually means the process exited or was killed before the flush: a scheduler with a short termination grace period, a kill -9, an out-of-memory kill, or a script that never called end() and was torn down abruptly. I would look in the job's console output for Comet failed to send all the data back (%d messages and %d uploads), experiment ID: %s, which is the SDK admitting the flush did not finish, then check whether the log mentioned throttling, since throttled data is dropped rather than queued.

The fixes follow from the cause. Call exp.end() in a finally block, give the container a grace period longer than the flush, raise COMET_TIMEOUT_CLEANING if uploads are genuinely slow, and log less often if the cause was throttling.

A strong answer separates "never sent" from "sent and sampled", because a metric with more than 15,000 values is down-sampled by reservoir sampling and looks different even when nothing was lost. A weak answer blames the UI, or says to "rerun with a bigger buffer", as if there were one.

Your team says Comet is slowing training. How do you find out whether that is true?

Logging is asynchronous, so the enqueue itself should cost microseconds. The usual culprits are elsewhere: logging a figure or image on every step (Matplotlib rendering is expensive and images are uploaded), histogram or gradient logging switched on (off by default because it is costly), a metric per step at high frequency that hits the throttle and retries, or system metrics set to a one-second interval on a busy node. I would profile with logging disabled using COMET_AUTO_LOG_DISABLE=1, compare step times, then enable categories one at a time. The default auto_metric_step_rate of 10 and the 30-second system-metrics interval are sensible, so I would first check whether someone overrode them.

A strong answer quantifies: measure the delta, not the feeling, and report which category costs what. It also recommends log_metrics with a dictionary in place of many single calls, and logging a sample of steps. A weak answer says to switch Comet off, or assumes the SDK blocks the loop.

A colleague's experiment page shows Experiment has been throttled. What do you tell them?

Throttling is per data type: metrics 12,000 per minute, parameters 10,000 per minute, captured output 12,000 per minute, and other data 8,000 per minute. The message also says that mission-critical data such as artifacts is never throttled, which is useful to know because it means the model and dataset are safe even if some metric points are missing. The typical cause is a per-step metric in a tight loop, or a training script printing a progress bar that is captured as output. Remedies are logging per epoch or every N steps, batching through log_metrics, reducing captured output, or using offline mode, which has no rate limits (only hard limits).

A strong answer mentions the limit that is different from throttling: the 15,000-value sampling per metric, and total fidelity (an organisation setting) for when exact values matter, read back with get_metric_total_df(). A weak answer treats throttling and sampling as the same thing.

comet_ml.start() runs without an error, but nothing appears in the workspace

The SDK is designed to keep training when tracking fails, so "no error" does not mean "logged". Candidates: Run will not be logged, meaning the handshake failed (proxy, TLS, outage); an invalid key, which prints The given API key '%s' is invalid on '%s' and continues untracked; disabled mode through COMET_AUTO_LOG_DISABLE; logging into a different workspace or project than the one being looked at, since project_name defaults to Uncategorized and the workspace defaults to the user's own; or a key belonging to a different installation, for instance a cloud key used against a self-hosted url_override. I would read the console output first, then run comet check --debug on the same host, then confirm the effective configuration by working down the precedence order: explicit arguments, environment variables, .env, COMET_CONFIG, working-directory .comet.config, home-directory .comet.config.

A strong answer adds an error-mode smoke test, comet_ml.error_mode(), so that this class of problem fails a CI job instead of hiding. A weak answer retries with a new key without looking at the output.

The same script logs to the right project on your laptop but the wrong one in CI. Why?

Precedence. An environment variable outranks any configuration file, so a CI system that sets COMET_PROJECT_NAME overrides the project_name in the repository's .comet.config. The reverse can also happen: a developer's personal ~/.comet.config is the lowest-priority file, so a repository .comet.config beats it, which is the behaviour you want. I would print the effective configuration in the job, check which environment variables the pipeline defines, and decide where each setting belongs. Project-level facts go in the repository file, secrets go in environment variables from the secret store, and CI overrides go in pipeline variables.

Strong answers also know COMET_INI is deprecated in favour of COMET_CONFIG, even though an older doc page still lists the former, and that explicit arguments in code beat everything, which is another reason not to hardcode project names in library code.

A hyperparameter search has run 40 trials and the best score barely moved. What do you check?

First that the search is optimising what I think it is. spec.metric defaults to "loss"; if the code logs val_f1 and the spec never names it, trials are ignored and the console says Optimizer metrics is '<metric_name>' but no logged values found. Experiment ignored in sweep. Next, objective: it defaults to minimize, and it drives Bayes, so maximising an F1 with the default would steer the search the wrong way. Then the space: a learning rate on a linear scale from 1e-5 to 1e-3 wastes samples, and loguniform is the usual choice. Then parallelism: a Bayes search run with many simultaneous workers chooses from little information and behaves like a random search. Finally, the metric itself may simply be noisy; with a noisy validation score, trials greater than 1 averages over repeats at the cost of total runs, which equal maxCombo * trials.

A strong answer notes the loguniform spelling versus the log_uniform that appears in some docs, and that comet optimize -j gives each process (maxCombo*trials)/j of the work. A weak answer blames the algorithm.

How would you promote a model to production, and how does that differ from what older tutorials show?

Log the model on the experiment with log_model, register it with register_model under a lowercase registry name and semantic version, and move it through statuses. Older tutorials move models between stages. Stages are deprecated: their data became tags, and status, with an approval workflow, is the lifecycle mechanism. A regular member's status change becomes an approval request that a workspace owner (or someone granted approval rights, available at workspace level since July 2025) approves, so Production means an accountable person agreed. Then a webhook on MODEL_STATUS_CHANGED can start the deploy job, and the job downloads with comet models download -w WORKSPACE --model-name NAME --model-version 1.0.0.

A strong answer mentions the secret on the webhook and the payload change when status configuration arrived in June 2025, plus download limits and the 429 response at scale. A weak answer describes moving a tag by hand with no approval.

Pods restart during a rollout and some fail to pull the model from the registry. Diagnose it.

HTTP 429 on model downloads. The registry limits downloads to 5,000 per day per account and per model version on paid plans, 1,000 on free plans, plus 20 downloads per second. A fleet that downloads on every start can hit the per-second or daily cap during a rollout, and a crash loop makes it worse. The fix is architectural: download once in the deploy job, put the model in your own object store or bake it into the image, and let pods read from there. I would also check that nothing polls the registry for "latest" on a timer.

A strong answer links this to the general principle of not making production startup depend on a third-party API, and to the REST quota of 15,000 submissions per hour per API key. A weak answer asks for higher limits as the first step.

A dataset changed and nobody can say which version trained last month's model. How do you prevent a repeat?

Use artifacts for data and consume them with get_artifact, so the lineage graph records the dataset version that fed each experiment. Names are limited to 100 characters, versions are immutable and aliases give moving pointers such as latest. If the data lives in object storage, use add_remote for S3 or GCS, and enable bucket versioning so the recorded URI, checksum and version ID mean something; a remote reference to an unversioned prefix records only a location. Also log a content hash in metadata and record the git state, which Comet captures by default through log_git_metadata.

A strong answer notes that reading a file from a path bypasses lineage, and that resumed experiments do not refresh code or environment logging by default. A weak answer suggests a naming convention in a spreadsheet.

Compute nodes have no route to Comet, only to an internal object store. Design the logging.

Run offline: comet_ml.start(online=False) with an offline_directory on storage the job can reach. The SDK prints the upload command at the end, and exp.set_offline_zip_uploader(callback) can push the zip to the store; a host that can reach Comet then runs comet upload on the archives. Offline mode has no rate limits, so it also helps with very high-volume jobs. I would add comet offline to inspect a zip if an upload is rejected, and use --force-upload (not the deprecated --force-reupload) if the original experiment is gone, which is what Upload failed for %r; experiment was not found suggests.

A strong answer adds that live monitoring is lost, so this suits batch training and not jobs a person watches, and that an air-gapped host also needs a wheelhouse installed with pip install --no-index --find-links. A weak answer proposes a proxy without asking whether one is allowed.

Trade-off questions

Comet versus MLflow for a mid-size team

MLflow is open source and self-managed by default, with a simple tracking server and a model registry that many platforms already integrate. Comet is a hosted-first product with a richer UI (panels, reports, compare views), an Optimizer service, artifact lineage built in, registry approval workflows and production monitoring, at a per-user price on the Pro plan and custom on Enterprise. The honest trade-off is operations against features: self-hosting MLflow costs engineering time but gives control; Comet costs licence money and, if self-hosted, a Kubernetes deployment with MySQL, Redis and object storage. Data residency may decide it: Gulf and Egyptian employers with strict rules may need the self-hosted or single-tenant option, which is Enterprise. Compare with MLflow and Weights & Biases before answering, and say which constraint you would check first.

What separates answers: naming the decision criteria (residency, team size, registry governance needs, existing platform integrations) and not claiming a winner in the abstract.

start() or the old Experiment class

The classes still work, but comet_ml.start() has been the canonical entry point since it became generally available in 3.47.1. It unifies online and offline behaviour, takes a mode (get_or_create, get, create), returns the running experiment when one exists with matching settings, and is where the modern options live. The cost of migrating is small, and the risk of not doing it is accidentally following tutorials that use deprecated calls such as comet_ml.init(), which was replaced by login() in 3.44.0.

Strong answers mention mode="create" for loops and mode="get" for strict resume.

One experiment for all ranks, or one per worker

A single experiment with a shared key and distributed_node_identifier gives one coherent chart and is natural for one big training job. One experiment per worker suits independent workers and isolates failures, but you need a shared run ID to group them. The hazard in the single-experiment pattern is key generation: compute it once in the launcher, since a race in multi-node creation was fixed only in 3.49.2 and retries lengthened in 3.49.12. A strong answer mentions that, and that the choice should follow what you will look at afterwards.

Auto-logging versus explicit logging

Auto-logging captures a lot cheaply and covers many frameworks, but it decides names and frequency, and framework parameters depend on what the library exposes. Explicit logging is deliberate and stable across framework upgrades. The practical position is to keep auto-logging for system metrics and framework curves, trim what is noisy through COMET_AUTO_LOG_*, and log the defining configuration and the evaluation results yourself.

Webhooks versus polling the registry

Webhooks are event-driven: MODEL_VERSION_CREATED, MODEL_VERSION_DELETED, MODEL_TAG_CHANGED and MODEL_STATUS_CHANGED fire to your endpoint, and a GitHub Actions payload format exists. Polling spends REST quota and adds latency. The price of webhooks is that you run and secure a receiver, validating the secret sent in the Authorization header. Choose webhooks for deploy triggers and keep a slow reconciliation poll as a safety net if an event could be missed.

What strong answers have in common

A strong answer names the mechanism. It says the queue, the throttle type, the precedence level, the registry status. It then proposes the check that would confirm it, such as comet check --debug, a file log at debug level, error_mode, or reading the console for a specific message, and only then the fix. It notices version drift: init() versus login(), stages versus status, Experiment() versus start(), --force-reupload versus --force-upload, COMET_INI versus COMET_CONFIG, and the comet_ml.secret versus comet_ml.secrets import path in the documentation prose. It respects hard numbers: 15,000 sampled values, 25,000 experiments per project, 100 models per experiment, 15,000 REST submissions per hour.

Weak answers have their own signature. They treat the tracker as magic and blame it, or treat it as unimportant and switch it off. They describe the feature set from marketing pages. They forget that failure is silent by design. They promote models by convention and call it governance. And they propose a bigger limit before proposing a smaller workload.

Short questions that come up

What is the difference between an asset and an artifact? An asset is an unversioned file attached to one experiment; an artifact is a versioned, workspace-level, immutable snapshot with lineage.

What happens to a metric logged more than 15,000 times? It is down-sampled by reservoir sampling unless total fidelity is enabled.

What does comet check test? The server, the REST API and the optimizer, plus environment details such as proxies and the CA bundle.

Why import comet_ml first? Auto-logging hooks frameworks as they are imported; importing later degrades what is captured, and the SDK warns about it.

Name two ways to keep the API key out of source. A Kubernetes secret into COMET_API_KEY, or the AWS or GCP secrets-manager helpers in comet_ml.secrets.

A closing exercise

Pick the scenario you found hardest and write a four-line answer in this shape: the symptom, the mechanism you suspect, the check that confirms it, and the fix plus how you would stop it recurring. If you can do that for the throttling, precedence, optimizer and registry questions without looking at the answers, you are answering at the level this interview expects.

Part three of three for the mid level. By now your runs log and your dashboards render, so the problems have changed character. They are no longer "why is nothing showing up" but "why did a third of last night's runs miss their tail", "why does CI log to the wrong project" and "why can't we say which data trained this model". Start with the error table, then the practices in order of leverage, then the practice cards and recipes. Everything is checked against comet_ml 3.58.7.

Common errors at this level

Cumulative: the beginner errors (wrong key, import order, forgotten end()) still apply. These are the ones that appear once things mostly work. Message text is quoted from the SDK; console lines are prefixed COMET ERROR:, COMET WARNING: or COMET INFO:.

Three of these deserve a closer read because they look alike and mean different things. Run will not be logged is about the handshake: the SDK could not start the experiment at all. Experiment has been throttled is about rate: the experiment exists, but some messages were dropped. Comet failed to send all the data back is about shutdown: the experiment exists and most data arrived, but the tail did not. Naming which of the three you are looking at tells you whether to read proxy settings, logging frequency or process lifecycle.

Practices, in order of leverage

These are ordered by how much trouble each prevents, not by how interesting it is. The first four are cheap and cover most of the incidents above.

1. Make shutdown deterministic

Put exp.end() in a finally block, or wrap the training function so that every exit path calls it. Give the container or scheduler a termination grace period longer than your typical flush. This single habit removes the commonest cause of "the last part of my run is missing". In a notebook, end() also logs the executed cells, so it serves two purposes at once. Raise COMET_TIMEOUT_CLEANING only if you have evidence the uploads are slow; the default is an hour.

2. Decide where each setting lives

Put project facts in a repository .comet.config (workspace, project name, logging preferences), secrets in COMET_API_KEY from your secret store, and CI overrides in pipeline variables. Never put a key in code, because code is auto-logged. Remember the order: code arguments, environment variables, .env, COMET_CONFIG, working-directory file, then home-directory file. When a run logs somewhere surprising, walk that list before anything else.

3. Log in proportion to what you will read

A metric every 50 steps plus an epoch summary tells the same story as every step for a fraction of the throttle budget. Use log_metrics with a dictionary instead of many single calls. Log images and figures as a fixed grid per epoch. Leave histogram, gradient, activation and weight logging off unless you are debugging, since they are off by default for a reason. Keep auto_metric_step_rate and the 30-second system-metrics interval unless you have measured a need.

4. Name runs for the query you will write

Use ExperimentConfig(name=..., tags=[...]) with a team convention: model family and data version in the name, purpose in tags. Derive experiment keys with comet_ml.get_experiment_key(...) from the logical identity of the job, so a retry lands on the same experiment instead of spawning a twin. Use mode="create" in loops so two attempts never merge.

5. Use artifacts for data, and consume them through the API

Log datasets as artifacts, read them with get_artifact, and the lineage graph fills itself in. Reading a CSV from a shared path gives you no lineage, and a month later nobody can say what the model saw. For remote data, turn on bucket versioning so the recorded version ID means something. Keep names under 100 characters. Put models in log_model, not artifacts, so that they can be registered.

6. Run the registry as a release process

Register each candidate with a semver version, move it by status rather than stage, and let an owner approve. Attach a webhook with a secret and validate the secret in the receiver. Download the model once in the deploy job and cache it, because the registry has daily and per-second download limits and a rollout can reach them. Use tags for facts like a customer or a region, and status for lifecycle.

7. Protect the search from its own defaults

Name spec.metric exactly, set objective explicitly, use loguniform for rates, run Bayes with a few workers, and use comet optimize -j N instead of hand-rolled threads. Put the optimizer ID in your job definitions so a search can be resumed or shared.

8. Test the tracking path

Run unit tests with COMET_AUTO_LOG_DISABLE=1, which needs no key or network. Run a scheduled smoke job against a real workspace with error mode on. After a server upgrade, cometx smoke-test WORKSPACE[/PROJECT] exercises the whole surface quickly. Run comet check on the compute host, not your laptop.

9. Separate identities

Use different service accounts for different jobs so that one runaway script cannot exhaust the REST limit of 15,000 submissions per hour per key for another. Remember that service accounts are Enterprise only and start with organisation-admin-level rights, so restrict their workspaces and rotate their keys.

10. Pin and upgrade on purpose

Pin comet_ml==3.58.7 in production and upgrade with pip install -U comet_ml in a test environment first. The public release notes stop at 3.56.0, so for later versions, read your own smoke test results rather than a changelog. Some features need a minimum backend version, so keep an eye on the server side too when you self-host.

Practice cards

Each card is a twenty-minute exercise. Do them in a scratch project so that nothing you break matters.

Operational recipes

Recipe: a training wrapper that cannot lose its tail

The wrapper below ends the experiment on every exit path, names the run from its identity, and reads the project from configuration instead of code.

import comet_ml

def run(job_name: str, date: str, seed: int, train_fn):
    key = comet_ml.get_experiment_key(job_name, date, f"seed-{seed}")
    config = comet_ml.ExperimentConfig(
        name=f"{job_name}-{date}-s{seed}",
        tags=["nightly", job_name],
    )
    exp = comet_ml.start(experiment_key=key, experiment_config=config)
    try:
        exp.log_parameters({"seed": seed, "date": date})
        return train_fn(exp)
    finally:
        exp.end()

Note what it leaves out. There is no api_key, no project_name and no workspace, so the same code runs locally, in CI and in production with whatever each environment configures.

Recipe: a health check for a new compute host

Run this on the host that will run the job, as the user that will run it.

comet --version
comet check
python - <<'EOF'
import comet_ml
comet_ml.error_mode()
exp = comet_ml.start(project_name="host-smoke")
exp.log_parameter("host", "check")
exp.log_metric("ok", 1, step=0)
exp.end()
print("logged")
EOF

comet --version should print the SDK version, for instance 3.58.7. comet check tests the server, the REST API and the optimizer and ends with a line such as Server connectivity True. With error mode on, a bad call raises, so the script printing logged means the path works. If comet check reports a certificate problem, set REQUESTS_CA_BUNDLE to the company bundle before anything else.

Recipe: debug logging without breaking the console rule

Console debug is forbidden and is downgraded to INFO with a warning. Write a debug file instead, set before comet_ml is imported:

COMET_LOGGING_FILE=/tmp/comet-{project}.log COMET_LOGGING_FILE_LEVEL=debug python train.py
tail -n 50 /tmp/comet-defect-detector.log

Read for the handshake outcome, retry messages and the upload queue at exit.

Recipe: a parallel search

export COMET_OPTIMIZER_ID=$(comet optimize opt.json)
comet optimize -j 4 train.py "$COMET_OPTIMIZER_ID"

The first line creates the optimizer without running it. The second runs four processes against it, each doing (maxCombo * trials) / 4 of the work, and train.py reads the optimizer from sys.argv[1]. To let a second job join the same search later, give it the same ID.

Recipe: upload offline runs from a host that can reach Comet

ls /scratch/comet/*.zip
comet offline /scratch/comet/<key>.zip
comet upload /scratch/comet/*.zip
comet upload --force-upload /scratch/comet/<key>.zip   # original experiment missing

Use the last form only for the experiment was not found case, since it creates a new experiment.

Recipe: a deploy job that downloads once

comet models download -w vision-team --model-name defect-detector \
  --model-version 1.2.0 --output ./serving-model

Then copy ./serving-model to your own object store and have pods read from there. The command downloads into model by default when --output is omitted. Do not use --model-stage; stages are deprecated.

Recipe: read back what you logged

import comet_ml
from comet_ml.query import Tag

api = comet_ml.API()
runs = api.query("vision-team", "defect-detector", Tag() == "nightly")
for r in runs[:5]:
    print(r.key, r.get_name() if hasattr(r, "get_name") else "")

Call api.clear_cache() if you read back something you just wrote through another path, since the API object caches by default. For larger result sets use api.search(...), which supports OR and pagination.

Recipe: a webhook receiver that checks its secret

The receiver should compare the Authorization header with the secret you configured, reject mismatches with a 401, acknowledge quickly, and do the deploy work asynchronously. Register the webhook for MODEL_STATUS_CHANGED and handle TEST so you can verify the path. Because the June 2025 status configuration changed the payload structure, log the first real payload and code against it, not against an old example.

Habits that pay off late

Keep a one-page "how we use Comet" note in the repository with the naming convention, the project map, where secrets come from and who approves Production. New joiners will otherwise invent their own, and the workspace will slowly turn into a pile. Archive experiments that are finished and uninteresting before a busy project approaches 25,000. Review who holds service-account keys quarterly. And before a major server or SDK upgrade, run the smoke test on a copy of your real workflow, not a hello world, because the failures that matter are in the integrations.

If you only do three things from this file, make them these: end experiments in finally, keep keys and project names out of code, and consume datasets through get_artifact. They prevent the largest share of the pain described above, and they cost almost nothing.

For the wider picture of tools that sit around Comet, see the guides on MLflow, Weights & Biases, DVC and Docker, which cover the neighbouring decisions about tracking, data versioning and packaging.

This is part three of three. It takes the experiment tracking, artifacts, registry and optimizer you learned at Beginner and Mid and asks what changes when forty people, several teams and a release process depend on them. The questions are no longer "how do I log this run" but "what breaks first when everyone logs at once", "who is allowed to promote a model", "what happens when the network drops mid-training", and "how do we leave if we ever have to".

The facts here are checked against the comet_ml 3.58.7 SDK (released 24 September 2026) and the official documentation. Where the two disagree I say so, because a platform owner is the person who discovers that drift first.

Where this picks up

I start with how the pieces fit together, because every failure mode later in this guide is easier to predict once you can picture where the bytes go.

Architecture and failure modes

Comet is a client-server system. Your training process imports comet_ml, and comet_ml.start() opens a session with the server, then returns an experiment object. After that, nothing you log is sent synchronously. Metrics, parameters, captured standard output and system metrics are batched in a background thread and streamed to the /clientlib/ endpoint, optionally compressed. Files such as images, models and assets go a different way: since SDK 3.33.3 they upload directly to object storage, using multipart upload for large files. The REST API lives at /api/rest/v2/, the hyperparameter Optimizer is a separate service at /optimizer/, and there is a heartbeat so the server knows a run is alive.

This split matters for operations. The streaming path carries many small messages and is what rate limits protect. The file path carries few large objects and is what your storage bucket permissions protect. A platform owner debugging "my experiment is missing data" must first decide which path the missing data travelled, because the causes and the fixes are unrelated.

The self-hosted topology

Comet Cloud hides all of this. If you self-host, the current documentation describes Kubernetes only (EKS, GKE or AKS), installed with Helm, plus an AWS SageMaker Partner AI App where AWS deploys and manages the service for you. A healthy Helm install shows these pods: comet-ml-backend-python, backend-react, backend-optimizer, backend-postprocess, frontend-nginx, mysql, redis-master, minio with a minio-provisioning job, and python-panels-orchestrator. Because the Opik LLM product ships alongside, you also see an Altinity ClickHouse operator, a ClickHouse cluster, and Opik backend and frontend pods.

Read that list as three data layers and some stateless services. MySQL holds metadata (Postgres appears only when Model Production Monitoring is enabled). Redis is the cache. Object storage holds every asset, model and artifact file. In production you replace the in-cluster MinIO with S3 or GCS and supply your own credentials, as the "external dependencies" documentation requires. The older single-VM installation driven by cometctl still appears in the docs for SSL and backups, but new deployments are not meant to use it. Treat it as legacy and plan a migration if you inherited one.

helm upgrade --install --namespace comet-ns --create-namespace \
  -f override-values.yaml comet-ml .
kubectl -n comet-ns get pods
curl -i http://<host>/api/isAlive/ping

The ping endpoint returns {"msg":"Healthy Server","code":200,"data":null,"sdk_error_code":0} when the server is healthy. On Comet Cloud the equivalent is https://www.comet.com/clientlib/isAlive/ping. Build your synthetic monitor on it, but remember it tells you the web tier is alive, not that uploads to your bucket work.

How the client fails, and what it does about it

The SDK is designed so that a Comet outage does not kill a training job. That design has four behaviours you must understand, because they decide what your users lose.

First, if the initial handshake fails, the SDK prints Run will not be logged and your code continues with no tracking. A wrong API key behaves the same way: The given API key '...' is invalid on '...'... Your experiment will not be logged. The training script does not crash, which is kind to researchers and dangerous to a platform owner, because a whole fleet can run untracked for days before anyone notices. Alert on the absence of expected experiments, not only on errors.

Second, if connectivity drops during a run, the SDK retries (by default three times with an exponential backoff factor of 2) and then switches to fallback-to-offline. It keeps writing to a local zip, and if the connection returns during the run it re-uploads. If not, it leaves an archive and prints the exact command comet upload that will send it. This needs backend version 3.3.11 or later.

Third, end() waits up to comet.timeout.cleaning, which is 3,600 seconds, for queued data to flush. If the process is killed first, or the timeout passes, you see Comet failed to send all the data back (%d messages and %d uploads), experiment ID: %s. On preemptible or spot machines, a kill signal during that flush is the usual cause, so handle termination signals by calling exp.end().

Fourth, errors are quiet by default. comet_ml.error_mode() changes that, making the SDK raise exceptions for invalid, discarded or truncated data. It applies to the whole process and does not make asynchronous upload errors raise. I recommend it in CI and in any pipeline where a silently missing metric would let a bad model through.

The failure-mode table

Scaling: which component breaks first

For a cloud or self-hosted Comet, the thing that breaks first is almost never compute. It is a limit: a rate limit on a busy project, a REST quota on a shared key, or a hard cap on an experiment. Knowing the numbers in advance turns an incident into a design parameter.

The streaming path is rate limited per data type, and each type is throttled independently. The published cloud limits are 12,000 metric submissions per minute, 10,000 parameters per minute, 12,000 output lines per minute, and 8,000 per minute for everything else. When you exceed one, the experiment prints Experiment has been throttled. Some data (like experiment metrics) might be missing. Mission-critical data, like Artifacts, are never throttled. and the UI shows a warning icon on the run. Self-hosted deployments can raise some of these. Offline experiments have no rate limits at all, only the hard limits.

The REST API has a separate budget: 15,000 submissions per hour per API key. This is the one platform teams trip over, because a dashboard, a CI system and a cron job all sharing one service-account key draw from the same hour. The documentation recommends distinct service accounts for different purposes, and I follow that without exception: one per pipeline or per automation, never one per platform.

Then there are the hard limits. A project holds up to 25,000 experiments (the message is Maximum number of experiments reached for project...; archive experiments or use a new project). Each experiment holds up to 50,000 unique metrics, 50,000 unique parameters, 100,000 images, 1,000 curves, 100 models, 500 text samples and 5,000 videos. Metrics are down-sampled above 15,000 values per metric by reservoir sampling. Registry downloads have their own ceiling: on paid plans 5,000 per day per account and 5,000 per day per model version (free plans 1,000 each), and 20 downloads per second. A fleet of inference pods that each pull the model from the registry at start-up will hit HTTP 429 during a rollout.

The sampling trap

Down-sampling is invisible. A curve with 40,000 loss values is silently reduced to 15,000 representative values. For exploratory work this is harmless. For anyone doing forensic analysis of a divergence, or computing a statistic from downloaded metrics, it is a wrong answer delivered confidently. On a self-hosted deployment you can ask your deployment engineer to raise the sampling size; the SDK's comet.offline_sampling_size default is also 15,000. Decide per project, and document it.

Security and the trust model

Who can do what in Comet is split across two layers, and a platform owner needs to be able to draw both from memory.

At the organisation level there are four roles. An Opik member can use only the Opik product. A platform member can use experiment management and Opik in the workspaces they are invited to. An admin has the Admin dashboard, billing, user and service-account management, and access to all workspaces. A view-only platform member cannot log through the SDK or API; the SDK raises ViewOnlyAccessException. At the workspace level there are members and owners. Owners approve model status changes, manage webhooks, make projects public, and manage workspace users. Since July 2025 those rights can be granted to individual members per workspace, which is how you avoid making every release engineer an owner.

Two defaults deserve attention. On a self-hosted install the first person to sign up becomes the admin. If you lose that account, recover with ADMIN_USER_LIST:['username'] in /etc/default/comet-ml or in Helm additionalConfigurations:, then restart the backend pods. And there is an organisation setting to make only admins able to invite users; it defaults to off, so turn it on if invitations are part of your access-control story. Deleting a user does not delete their data, and an admin can reassign its ownership, which is what you want for offboarding.

Service accounts carry admin power

A service account is a non-human identity, its name is prefixed sa-, and it has an API key. It is an Enterprise feature. The detail that matters is that a service account has organisation-admin-level permissions by default. Restrict the workspaces it may use in the Admin dashboard. If you do not, a leaked CI key is a leaked organisation. A service account with no default workspace forces every run to name one, and a run against a workspace it is not authorised for fails with Service account doesn't have access to workspace %s. You can regenerate the key without deleting the account, and REST endpoints exist to create accounts and rotate keys, so rotation can be automated.

JWT authentication

If you have an identity provider, the cleaner design is to stop handing out long-lived keys and let the IdP issue short-lived tokens. Comet supports JWT authentication on Enterprise Cloud and on all single-tenant deployments. You configure it in Admin dashboard, Organization settings, JWT Authentication. The key source is either a JWKS URI (recommended) or a static PEM public key (single-tenant only), never both. subjectMappingType is EMAIL or USER_NAME, subjectClaimName defaults to sub, and allowedIssuers and allowedAudiences are optional but you should set them. The JWKS cache refreshes every hour and the fetch timeout is 60 seconds by default. The token travels as Authorization: Bearer <jwt>, and Comet then acts with that user's API key. Both COMET_API_KEY and api_key= may hold the JWT. On Cloud the token header needs a kid matching a stored key.

The effect is that a pipeline's permissions come from the identity it runs as, and they expire. This pairs well with Kubernetes workload identity and with a secrets store such as Vault.

Transport, telemetry and compliance

For self-hosted installs, terminate TLS at a layer-7 load balancer (an AWS ALB, not a classic ELB; a GCP HTTPS load balancer; or an Azure Application Gateway). If the certificate is self-signed or issued by a private CA, every SDK host must trust it, through the OS store or REQUESTS_CA_BUNDLE, or through comet.include.custom_ca_bundle. The SDK verifies TLS by default. COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 disables verification, and I treat its appearance in a repository as a finding, not a workaround. Corporate TLS inspection is the most common reason for There's seem to be an issue with your system's SSL certificate bundle; the right fix is to add the inspection CA to the trust bundle.

Two behaviours surprise security reviewers. The SDK sends error telemetry to Sentry by default. A key named comet.error_tracking.enable exists, but its exact semantics are not documented, so ask Comet before you tell a compliance team it is switched off. And a self-hosted server sends usage reports and business-intelligence events to stats.comet.com over TCP 443, including usernames, emails, the licence key and versions. For an air-gapped deployment, whitelist that destination or agree an alternative with Comet in advance. On compliance, the pricing page claims SOC 2, ISO 27001, ISO 9001, HIPAA and GDPR; ask for the current reports rather than quoting a web page.

Secrets and API key distribution

Comet's docs warn against comet_ml.login(api_key="...") and hard-coded keys for a concrete reason: the SDK logs your source code by default, so a key in a script ends up in the experiment's code tab, visible to everyone with access to the project. The key is also redacted in logs by default (hide_api_key is true) and the Everett configuration library is forced to WARNING level since 3.49.1, but neither helps if the key is in the file itself.

There are three acceptable patterns, in order of how much I prefer them.

1. Short-lived JWT from your IdP, as above. No long-lived secret exists.
2. A secret manager. The SDK ships helpers for AWS and Google Cloud. They store the API key in the manager and return an "API Key Secret" string that you pass as api_key, so the plaintext never appears in code.
3. Platform-injected environment variable. A Kubernetes secret or scheduler secret mapped to COMET_API_KEY.

import comet_ml
from comet_ml.secrets.aws import get_api_key_from_secret_manager

# The secret string is stored once by an administrator using
# store_api_key_in_secret_manager(...), then referenced here.
api_key = get_api_key_from_secret_manager("<API Key Secret string>")
exp = comet_ml.start(api_key=api_key, project_name="fraud-detection")

Note the import path: comet_ml.secrets.aws, plural. The documentation prose shows comet_ml.secret.aws, singular, which does not exist in 3.58.7; the package ships comet_ml/secrets/aws and comet_ml/secrets/gcp, each exporting get_api_key_from_secret_manager and store_api_key_in_secret_manager. AWS needs boto3 and GCP needs google-cloud-secret-manager>=2.0.0. On-premises deployments can additionally use AWS KMS or Google Cloud KMS, arranged with Comet. Cloud credentials for remote artifacts are a separate matter: the SDK uses whatever credentials the provider's own tooling finds, so the job's identity decides whether a remote artifact records only a URI or also a checksum and version.

Keeping secrets out of the experiment record

The SDK captures more than the source file. By default it records the command line arguments, environment details, git metadata and optionally the git patch, plus standard output. Any of these can carry a secret: a token passed on the command line, an environment dump printed at start-up, a patch containing a config file. Turn off what you do not need with ExperimentConfig flags (log_git_patch, log_env_details, parse_args) or their COMET_AUTO_LOG_* environment variables, and add names to the parameters_ignore and others_ignore lists under [comet_logging]. Set the platform default in a shared .comet.config baked into your base image, and let teams override deliberately.

Multi-tenancy, governance and chargeback

The hierarchy is organisation, workspace, project, experiment. The isolation boundary for teams is the workspace: artifacts and the Model Registry are workspace-scoped, and membership decides who sees what. The usual mapping is one workspace per team or per business unit, and one project per ML workflow. Projects can be public or private, and a private experiment can be symlinked into a public project, which shows it in a second place without copying it. That is the mechanism for publishing a curated set of runs to stakeholders while the raw work stays private.

Create workspaces deliberately. The Admin dashboard's Workspaces tab creates and deletes them and sets the default. A workspace per individual fragments artifacts and makes sharing hard; one workspace for the whole company makes permissions a fiction. My rule is that a workspace is a group of people who would all be comfortable reading each other's failed experiments and who share a release process.

Chargeback

Shared platforms need someone to pay. The community tool cometx includes cometx admin chargeback-report, which produces per-month usage by team. It is community-supported rather than an official Comet product, so test it against your installation before you build a finance process on it. Pair it with the per-workspace mapping above and you can report storage and usage by team without any custom accounting.

Governance through tags and lineage

Two features do the governance work without extra tooling. Tags on experiments, artifacts and model versions are searchable and scriptable. Lineage records which experiment produced an artifact and which consumed it. For an audit question such as "which dataset version trained the model that was in production on the third of March", the lineage view and api.search() or api.query() give you an answer from the platform itself, provided your teams logged datasets as artifacts. Make that a platform requirement for regulated projects, and you turn an archaeology project into a query.

import comet_ml
from comet_ml.query import Tag

api = comet_ml.API()
runs = api.query("fraud-team", "fraud-detection", Tag() == "release-candidate")
for exp in runs:
    print(exp.id, exp.name)

API.search(), added in 3.54.0, supports AND and OR filtering and pagination, so prefer it for anything that may return many experiments.

The registry as a release gate

At Mid level the registry was a place to put models. At this level it is a control point, and getting its design right prevents the most expensive class of mistake: the wrong model in production.

A model is logged on an experiment with log_model, then promoted with register_model. The docs recommend logging models as experiment models rather than as artifacts, exactly so they can be promoted. Each registered version has a semantic version string (default 1.0.0), tags, a status, a status configuration, notes and history. Stages are deprecated: their data became tags, and status, with an approval workflow, is the official lifecycle mechanism. Every stage-based API method now emits a deprecation warning. If you are still using register_model(stages=...) or comet models download --model-stage, migrate to model.set_status, add_tag and find_versions.

Status changes by regular members go through approval requests that a workspace owner, or someone granted the right, approves. That is your separation of duties. A data scientist can propose "Production"; a release manager approves. June 2025 added a Status Configuration, a key-value pair per target environment, and the release notes warn that this changes the structure of the webhook payload. If you wrote a webhook consumer before that date, test it against a current payload.

Webhooks into CI/CD

A registered model can call an HTTP endpoint on MODEL_VERSION_CREATED, MODEL_VERSION_DELETED, MODEL_TAG_CHANGED and MODEL_STATUS_CHANGED, plus TEST for wiring checks. There is a GitHub Actions payload format, which makes the simplest pipeline one where a status change to Staging starts a workflow on GitHub Actions that downloads the model, runs an evaluation and posts the result back as a tag. Keep the webhook receiver small: validate the Authorization secret, check the event type, and trigger a pipeline rather than doing the work in the handler.

comet models download -w fraud-team --model-name fraud-classifier \
  --model-version 2.1.0 --output ./model

The real flags are --model-name and --model-version. The built-in help example shows --registry_name and --model_version, which are not valid, so do not copy it into a runbook. The default output folder is model.

Serving itself is outside Comet. A model leaves the registry for BentoML or KServe, and its production behaviour is watched by a monitoring tool such as Evidently. Comet sells its own Model Production Monitoring for drift, feature distributions and custom metrics, but it is sales-gated, so evaluate it against the open-source route before you commit.

Distributed training and the Optimizer at scale

Logging from one process is simple. Logging from sixty-four GPUs across eight nodes is where platform conventions start to matter, because each process is a separate Python interpreter with its own experiment object. The rule is one experiment object per process, and two patterns for what that means.

In the single-experiment pattern, every rank shares a pre-set experiment_key, and ExperimentConfig(distributed_node_identifier=...) tells the nodes apart. The key must be alphanumeric and between 32 and 50 characters, or you see COMET_EXPERIMENT_KEY is invalid. Generate it once, in the launcher, with uuid.uuid4().hex (32 characters) or comet_ml.get_experiment_key(), and pass it to every rank through the environment. A race in multi-node experiment creation was fixed in 3.49.2 and the retries lengthened in 3.49.12, so anything older than that in a training image is a reason to upgrade. In the second pattern, each worker creates its own experiment and you group them by a run identifier in a tag. It costs more experiments against the 25,000 cap but isolates failures, and I prefer it when ranks have very different roles, for example parameter servers versus workers.

import os, comet_ml

key = os.environ["COMET_EXPERIMENT_KEY"]        # 32 to 50 alphanumeric chars
rank = int(os.environ.get("RANK", "0"))
cfg = comet_ml.ExperimentConfig(distributed_node_identifier=f"rank-{rank}")
exp = comet_ml.start(project_name="llm-pretrain", experiment_key=key,
                     experiment_config=cfg)

Log scalar metrics only from rank zero unless you need per-rank data. Sixty-four ranks each logging loss every step is sixty-four times the throttle budget and usually sixty-three copies of the same number.

When a run is resumed after a failure, use comet_ml.start(mode="get", experiment_key=...). A resumed run disables code, graph, argument, environment and git logging by default so it does not overwrite the original record. The mode values are get_or_create (the default), get and create. Use create inside hyperparameter loops so a repeated key raises instead of silently appending to an old run, and get when a missing run should be an error, which raises ExperimentNotFound.

The Optimizer is a service you run capacity for

The Optimizer is a client-server search. The client asks the service for the next parameter set, runs a trial as a new experiment, and reports the metric. Algorithms are grid, random and bayes; bayes uses sequential model-based optimisation with a Tree Parzen Estimator. For a platform owner the operational facts are these.

The search space has a memory cost. A large grid makes the optimizer service run out of memory, and the documented remedies are to remove parameters, lower gridSize or minSampleSize, or add memory. On a self-hosted installation, this is a pod-limit question on backend-optimizer. Total runs equal maxCombo multiplied by trials, so a user who sets maxCombo to unlimited on a grid launches every combination. Put the maximum in a review checklist.

Bayes cannot be fully parallelised because each suggestion depends on earlier results. comet optimize -j 8 script.py opt.json splits the work across eight processes, and each runs its share, but for bayes the practical parallelism is modest. The spec metric must be logged under exactly the configured name, otherwise you see Optimizer metrics is '<metric_name>' but no logged values found. Experiment ignored in sweep. and the trial is ignored. That single mistake wastes more GPU hours than any algorithm choice.

Use loguniform as the scaling type in examples; some documentation pages write log_uniform, but the reference list says loguniform. Passing an Experiment to the Optimizer constructor is deprecated; pass it to Optimizer.get_experiments or Optimizer.next. Since 3.55.0 Optimizer.update() can send metadata and status back. Each trial is an experiment, so a two-thousand-trial search consumes two thousand of your 25,000 slots in that project. Put searches in a project of their own.

Network, offline and resilience tuning

Every platform with remote logging eventually gets a question from a team on a slow or restricted network. The SDK has a few knobs, and knowing their defaults lets you answer in minutes.

Set these as environment variables with the COMET_ prefix or in .comet.config. The precedence, highest first, is explicit arguments in code, operating-system environment variables, a .env file in the working directory, the file named by COMET_CONFIG, then ./.comet.config, the Colab Drive path, ~/.comet.config, and finally overrides that the backend supplies. COMET_INI still works but warns that it is deprecated; the configuration page still lists it, so expect users to find it there. A platform owner's practical use of this ordering: bake organisation defaults into /etc-style files or an image-level .comet.config, let a repository's .env override them, and accept that an explicit argument in code wins over everything.

Offline mode as a first-class workflow

Some workloads should not stream at all. A secured training cluster with no route to the internet, or a high-frequency job that would exceed the throttle, runs with comet_ml.start(online=False). The experiment writes a zip into .cometml-runs by default, or the directory set with offline_directory, and it prints the comet upload command at the end. There are no rate limits, only hard limits. comet upload --force-upload handles a resumed offline run whose original was deleted; the older --force-reupload still appears in some printed messages but is deprecated. comet offline inspects archives, and exp.set_offline_zip_uploader(callback) lets the job push the zip to an object store so that a separate uploader with network access can ingest it.

For an air-gapped site, install the SDK without internet by running pip download comet_ml into a wheelhouse/ directory on a connected machine that has the same Python version and ABI, then pip install --no-index --find-links /PATH/TO/wheelhouse comet_ml on the offline host. Combined with a self-hosted server inside the enclave, nothing needs to leave.

Defaults worth knowing: three retries with a backoff factor of 2, a 10-second HTTP timeout, a 3,600-second end() flush, and system metrics every 30 seconds (the configuration page says 60, but 3.39.0 lowered it and the code uses 30).

Upgrades and migrations

There are three things to upgrade and they move at different speeds: the SDK in your training images, the self-hosted backend, and your own integrations.

The SDK

pip install -U comet_ml upgrades the package, and --upgrade-strategy eager also upgrades its dependencies. In production, pin the version, for example comet_ml==3.58.7, and upgrade through a base image on a schedule. The release notes documented by Comet stop at 3.56.0 in February 2026. Versions 3.57.x and 3.58.x have no public release notes, so for those, your test suite is the changelog. Run a smoke test after each bump: start an experiment, log a metric, an image, a model and an artifact, register a model, and download it.

Watch the changes that altered behaviour, not just the features. 3.47.3 changed the default auto_output_logging from "native" to "simple", which changes how standard output is captured. 3.45.0 made DEBUG console logging forbidden, downgrading it to INFO with a warning. Use a debug-level file log instead: set COMET_LOGGING_FILE and COMET_LOGGING_FILE_LEVEL=debug before importing comet_ml. 3.44.0 deprecated comet_ml.init() in favour of comet_ml.login(), and Python 3.7 support ended in 3.53.0 (3.6 in 3.48.0). The SDK requires Python 3.8 or newer. Grep your fleet for deprecated calls with the warning strings, which are stable and searchable.

SDK and backend compatibility

The SDK checks features against the backend version. Fallback to offline needs backend 3.3.11, the novel model registry API 3.5.42, deleting tags 3.12.26, deleting parameters 3.29.724, set_project_owner 4.35.0, and raw throttling messages 3.62.294. An old self-hosted backend with a new SDK can raise BackendVersionTooOld, and the reverse raises SDKVersionIsTooOldException, with the message "Obsolete SDK version. Please update." The latest documented self-hosted release is backend 4.7.420 with frontend 5.109.14, optimizer 2.1.8 and Python panels 1.1.89, from the October 2025 changelog. Some recent cloud features, such as the ten-experiment compare view and improved lineage graphs, are not yet in self-hosted builds, so set expectations with users before they assume parity.

Upgrading a self-hosted server

Use the same Helm command as the install and verify in three steps: kubectl -n <ns> get pods for readiness, /api/isAlive/ping for health, and cometx smoke-test WORKSPACE for function. The smoke test exercises optimizer, mpm, panel, opik, experiment, metric, image, asset, dataset-info, confusion-matrix and embedding, with include and exclude lists. Take a backup first. On cloud Kubernetes that means volume snapshots; on bare metal it means backing up the whole disk with rsync, Restic or Borg. Backing up individual data directories alone means reconfiguring everything on restore, which is the failure people discover only during a restore drill.

Leaving, or moving between installations

The exit path is worth testing before you need it. cometx download exports from Comet, and cometx copy writes into another installation, so moving a project between two Comet servers is two commands, one per side, each with its own API key. cometx download --from comet|wandb|neptune also supports importing from Weights & Biases and Neptune, which makes it the tool for a migration from Weights & Biases in. Moving to MLflow or ClearML is not covered by these tools, so you would script it with the REST API. Because cometx is community-supported and not an official project, rehearse it on a copy of real data first.

Cost and capacity

The pricing page, read on 30 September 2026, lists three MLOps tiers. Free gives core tracking, 100 GB of storage and community support. Pro is 19 dollars per user per month, up to ten users, 1,500 training hours and 500 GB of storage. Enterprise is custom and adds unlimited usage, advanced monitoring, flexible deployment (self-hosted or in your VPC), SSO, service accounts and view-only users. Academics get Pro free. Opik, the LLM product, is priced separately by spans, with a free open-source edition. Prices change often, so send people to the page instead of quoting it in a proposal.

The feature split is what a platform owner actually decides on. Service accounts, SSO enforcement and view-only users are all Enterprise features. A platform with shared pipelines needs service accounts, so in practice "Comet for a company" means Enterprise, and the honest comparison with an open-source option is Enterprise price against engineering time.

Incident playbooks

These are the incidents I have seen or would expect, each with a first move. Keep them in the on-call runbook keyed to the message text, because people paste the message into chat.

Nothing is being logged from anywhere. Run comet check --debug from an affected host. Test curl -i .../isAlive/ping. If those pass, check whether an API key was revoked or rotated; a mass failure after a rotation is the most common cause. Then check proxy settings and TLS: a new inspection appliance or an expired certificate on your load balancer produces Run will not be logged or the SSL-bundle error across the fleet at once.

One team's runs are throttled. Identify the project, find the experiments with the warning icon, and look at logging frequency. The fix belongs to the team: log per epoch, batch with log_metrics, or run offline. Do not raise the limit first.

Uploads fail with 500 or 403 on a self-hosted server. The message looks like Uploading file failed (500) with max retries: 4 or AmazonS3Exception: Access Denied Status Code: 403. Files over about 8 MB use multipart upload, and the bucket policy lacks multipart permissions. Grant s3:PutObject, s3:PutObjectAcl, s3:InitiateMultipartUpload, s3:UploadPart, s3:CompleteMultipartUpload, s3:AbortMultipartUpload and s3:ListMultipartUploadParts. As a workaround for one team, use remote assets.

Inference pods fail to start with HTTP 429. A rollout pulled the registry model in every pod. Cache the model in the image or a shared volume and stagger the rollout. The limit is 20 downloads per second and daily caps per account and per version.

An experiment reached its limits. Experiment has reached the maximum number of {asset_type} allowed. Hard limits are fixed per experiment, so the answer is to log fewer of that type, for example a sample of images, or to contact support.

A key leaked. Revoke it, regenerate the service-account key with the menu action (the account and its data survive), search the logs and code for where it appeared, and check whether it was in an experiment's code tab. For a user key, remove the key in account settings. Then find out why it was in a place a human could see it.

Where Comet stops

A senior engineer knows when not to use the tool, and Comet has clear edges.

Comet tracks and governs experiments and models; it does not orchestrate work. It logs a pipeline's steps through integrations such as the Kubeflow logger, @comet_flow and @comet_step for Metaflow, the Ray and SageMaker helpers and CometVertexPipelineLogger, but scheduling, retries and dependencies belong to Kubeflow, Metaflow, Airflow or Prefect. It does not serve models, so pair it with BentoML or KServe. Its data versioning is artifacts with remote references, which is not a substitute for a data-versioning system such as DVC when the question is "reproduce this exact dataset from git".

Its LLM story is now a separate product. LLM tracking moved to Opik, the comet_llm package is legacy with no releases since September 2024, and logging an ML experiment into an LLM project raises an error that tells you to use an Experiment Tracking project. If your users build agents, send them to Opik and keep comet_ml for classical and deep-learning experiments.

There are three cases where I would choose something else. If you need an open-source, self-managed tracker with no licence conversation, MLflow is the default. If your team is deeply invested in one cloud's managed ML suite, such as SageMaker or Vertex AI, its built-in tracking may remove a system. And if what you most need is pipeline-first orchestration with tracking as a by-product, choose ClearML or an orchestrator first and add a tracker second. Comet's strengths are the comparison interface, the artifact lineage, the approval-based registry and the packaged enterprise features, so choose it when those are what your organisation will actually use.

Running Comet as a platform

A platform is a product with users, so run it like one. Here is the operating model I use.

Publish a golden path. One page, one template repository, one base image. The image pins comet_ml, sets organisation defaults in a .comet.config, installs the CA bundle if you use an inspection proxy, and ships a helper that builds ExperimentConfig with the standard name and tags. A team starting a project copies the template and gets correct behaviour without reading this guide.

Own the identity. Decide how keys reach jobs, and make the decision once: a JWT from the IdP, a secret manager, or injected secrets. Forbid literals. Issue one service account per pipeline and restrict its workspaces.

Watch the platform. Monitor the ping endpoint, pod health, bucket errors and database capacity on self-hosted installs, using Prometheus and Grafana. Add business signals the platform does not give you: experiments created per day per workspace, count of crashed runs (experiment state running, finished or crashed, read with APIExperiment.get_state()), and the age of the oldest unsynchronised offline zip. The third is a good early warning of a network problem.

Run a smoke test on a schedule. cometx smoke-test WORKSPACE after every upgrade and nightly against a canary workspace. It costs little and catches an unhealthy optimizer or panel service before users do.

Automate the boring admin. Key rotation, service-account creation and webhook configuration all have REST endpoints. Put them in Terraform or a script under review rather than clicking in the dashboard. See the Terraform and Helm guides for the infrastructure side of a self-hosted install.

The review checklist

Use this for design documents and for onboarding a new team onto the platform.

1. Identity. Is the API key in code? Is each pipeline using its own service account, restricted to the workspaces it needs? Is there a rotation story?
2. Data egress. What does auto-logging send (code, git patch, stdout, environment)? Has the owner of the data agreed that it leaves the machine?
3. Logging volume. What is the step rate and the number of metrics? Does it fit the 12,000 per minute metric budget, including all ranks?
4. Failure visibility. If logging fails, how does anyone find out? Is error_mode() on in CI?
5. Resilience. Is offline mode or fallback configured for restricted networks? Who uploads the zips?
6. Sampling. Does the analysis depend on more than 15,000 values per metric? If so, is total fidelity on for that project?
7. Large files. Are datasets and large models remote references, not uploads?
8. Lineage. Are datasets logged as artifacts with a consumer relationship, so the question of what trained a model has an answer?
9. Registry. Is deployment pinned to a version? Who approves status changes? Are webhook secrets validated?
10. Search spaces. Is maxCombo bounded? Is the spec metric logged under exactly that name? Are searches in their own project?
11. Versions. Are comet_ml and the backend pinned and compatible? Is there an upgrade test?
12. TLS. Is verification on, and is the right CA trusted?
13. Exit. Could we export this project with cometx download and could we restore a backup?

The complete picture

Here is the whole thing in one paragraph. A team copies the golden-path template. Its job gets a short-lived identity and starts an experiment, which logs metrics at a sensible rate, records datasets as artifacts so lineage exists, and registers a model at the end. A release manager approves a status change, which fires a webhook into the pipeline; the pipeline evaluates and deploys an explicit version to the serving layer. The platform team watches ping endpoints, throttle warnings and the absence of expected runs, upgrades the SDK through a base image with a smoke test, and upgrades the server with a backup behind it. The finance side reads a chargeback report. Everything in this paragraph is a choice you can defend, and each one maps to a limit, a message or a role in the preceding sections.

Next, read the neighbouring trackers, MLflow and Weights & Biases, so you can answer "why this one" with specifics.

Where the series leaves you

At Beginner you learned to start an experiment and read a chart. At Mid you learned artifacts, the registry, offline runs and the optimizer. At this level you learned to own the platform: to predict which limit a team will hit, to choose how identity reaches a job, to put approval between a model and production, to upgrade without surprises, and to say when Comet is the wrong answer.

The habits that carry over to any tracking platform are the durable part. Make failure loud. Put the record of what trained what in the platform rather than in someone's memory. Give every automation its own identity. Pin versions and test upgrades. And keep the exit path rehearsed, because a platform you cannot leave is a platform you do not really control.

Sources

• Comet documentation home
• Quickstart
• Changelog
• Python SDK releases
• Start an experiment
• ExperimentConfig reference
• API reference
• Running offline
• Fallback to offline
• Error mode
• API key secret managers
• Offline installation
• Command line reference
• REST API overview
• MCP server
• Comet extensions (cometx)
• Configure the SDK
• Distributed training
• Limits and performance
• Total fidelity metrics
• Troubleshooting and FAQ
• Remote artifacts
• Artifact lineage
• Using the Model Registry
• Model approval
• Webhooks
• Configure the Optimizer
• Run the Optimizer in parallel
• Admin dashboard
• Service accounts
• JWT token authentication
• Deploying Comet
• Smoke testing
• Amazon SageMaker deployment
• Backups
• Data collection
• Integrations
• Opik documentation
• Pricing
• comet-ml on PyPI

How senior interviews differ

At this level nobody asks what log_metric does. The interviewer wants to know whether you can own a tracking platform for other teams: whether you predict failures before they happen, whether you can reason about trade-offs with numbers, and whether you know the tool's edges. The questions below are grouped as system-design prompts, incident questions and leadership arguments. Every answer is written the way I would say it aloud, with the specifics a strong candidate would include, and a note on what separates a strong answer from a weak one.

A few facts to have ready, because senior answers lean on numbers. The current SDK is comet_ml 3.58.7. The hierarchy is organisation, workspace, project, experiment. Streaming limits on Comet Cloud are 12,000 metric submissions per minute, 10,000 parameters, 12,000 output lines and 8,000 for everything else, per data type. The REST API allows 15,000 submissions per hour per API key. A project holds 25,000 experiments. Metrics are down-sampled above 15,000 values. Registry downloads are capped at 20 per second and, on paid plans, 5,000 per day per account and per model version. Service accounts are Enterprise-only and carry organisation-admin permissions by default.

System-design prompts

Design experiment tracking for 40 data scientists in five teams

A strong answer starts with boundaries, not features. "I would map the organisation to one Comet organisation, each team to a workspace, and each ML workflow to a project. The workspace is the isolation boundary, and since artifacts and the registry are workspace-scoped, it decides who can see a dataset or promote a model. I would not give every person a workspace, because artifacts fragment, and I would not use one workspace for everyone, because permissions become meaningless."

Then identity. "Nobody gets a long-lived key in code. Jobs authenticate through the identity provider using JWT if we are on Enterprise or single-tenant, otherwise through a secret manager, with the SDK's secret-manager helpers or a Kubernetes secret mapped to COMET_API_KEY. Each pipeline gets its own service account restricted to the workspaces it needs, because service accounts have admin-level permissions by default and share a REST budget of 15,000 submissions an hour per key."

Then the golden path. "A template repository and a base image with comet_ml pinned, organisation defaults in .comet.config, the CA bundle installed, and a helper that applies standard names and tags through ExperimentConfig. I want teams to get correct behaviour without reading documentation."

Then operations. "Health checks on the ping endpoint, an alert when a scheduled pipeline produces no experiment, because the SDK fails quietly, and a nightly cometx smoke-test against a canary workspace if we self-host."

What a weak answer misses: it lists features (charts, artifacts, registry) and never mentions identity, limits or how failure is noticed. It also says "one workspace per person" or "one big workspace" without a reason.

Where would you host it: Comet Cloud or self-hosted?

Good answers turn this into a set of questions rather than a preference. "What data leaves the machine? Auto-logging sends source code, git metadata and optionally a git patch, standard output and environment details. For a bank or a health provider in the Gulf or Egypt with residency rules, that is a legal question before it is a technical one. If the answer is that code and data must stay in-country, I self-host in a region we control."

Then the cost of self-hosting. "The current documentation describes Kubernetes only, on EKS, GKE or AKS, installed with Helm, plus the AWS SageMaker Partner AI App where AWS runs it. That means MySQL, Redis and object storage, a cluster, backups, upgrades and an on-call rotation. The older single-VM install is legacy. I would also agree data-collection terms: a self-hosted server sends usage reports to stats.comet.com including usernames, emails and the licence key, so an air-gapped site needs an arrangement. And some recent cloud features, such as the ten-experiment compare view and improved lineage graphs, are not yet in self-hosted builds."

The decision: "Cloud if residency allows, because it removes an operational burden. Self-hosted if residency or network isolation requires it, priced as a service we run."

Strong signals: naming the data-egress path, the Kubernetes-only direction, the telemetry, and the feature lag. Weak signal: "self-hosted is more secure" with no reasoning.

A training cluster has no internet access. How do you track experiments?

"There are two options and I might use both. If there is a self-hosted Comet inside the enclave, jobs log online to it. If not, jobs run with comet_ml.start(online=False), which writes a zip to .cometml-runs or the directory set by offline_directory, with no rate limits. The SDK prints the comet upload command at the end. To automate, I would use exp.set_offline_zip_uploader(callback) to copy the zip to an object store, and a separate uploader host with network access that runs comet upload against new archives. For installation, I would run pip download comet_ml into a wheelhouse on a connected machine with the same Python version and ABI, and install on the isolated host with pip install --no-index --find-links."

I would add the failure mode: "A zip is a file, so it can be lost. I would alert on the age of the oldest un-uploaded archive." That specific metric is what a platform owner adds.

Design a model release process on the registry

"Models are logged on the experiment with log_model, not as artifacts, so they can be promoted. Registering creates a version with a semantic version and, since stages are deprecated, a status. A data scientist proposes a status change. A regular member's change goes through an approval request, and a workspace owner or someone we have granted the right approves it. That gives separation of duties without making every release engineer an owner, which has been possible per workspace since July 2025."

"Promotion fires a webhook on MODEL_STATUS_CHANGED. The receiver validates the secret sent in the Authorization header, checks the event type and triggers a pipeline, for example in GitHub Actions, for which there is a payload format. The pipeline downloads the model with comet models download, runs an evaluation and deploys an explicit version to the serving layer. Serving is outside Comet, so it is BentoML or KServe."

"One rule: deployment pins a version in git, never 'whatever is Production'. Otherwise a status change alters a running fleet the next time a pod restarts."

A strong candidate also mentions the June 2025 status configuration that changed the webhook payload structure, and that stage-based methods warn as deprecated.

Design hyperparameter search for a team running 500 trials

"The Optimizer creates one experiment per trial, so 500 trials is 500 of the 25,000 slots in a project. I would put searches in their own project. Total runs equal maxCombo times trials, so I would bound maxCombo. I would pick bayes for expensive trials, knowing it cannot be fully parallelised because each suggestion depends on earlier results, and random or grid when I want massive parallelism through comet optimize -j. I would check that the training code logs the metric under exactly the name in spec.metric, because otherwise the trial is ignored with the message 'Optimizer metrics is ... but no logged values found'. And I would watch the optimizer service's memory, since a large space makes it run out of memory; the remedies are fewer parameters, lower gridSize or minSampleSize, or more memory."

Audit question: which data trained the model in production in March?

"If teams followed the platform rule, datasets are artifacts and each experiment consumed them, so lineage records it. I find the registered model version, open the experiment that produced it, and follow the lineage to the artifact version, which is an immutable snapshot with metadata, tags and aliases. To do it at scale I use API.search() or API.query() with tag filters. If datasets were remote artifacts on S3 or GCS with bucket versioning enabled, the SDK recorded the URI, checksum and version ID; without versioning, only the URI and checksum. So the honest answer includes a condition: the trail is only as good as the logging rules we enforced."

Incident-response questions

Every job shows a green status but no experiments appear

"I suspect silent failure first. The SDK does not crash a job when it cannot log; it prints Run will not be logged or The given API key ... is invalid and carries on. I would run comet check --debug from an affected host, and curl -i the ping endpoint. If the server is healthy, the likely cause is a revoked or rotated key, or a proxy or TLS change. After fixing, I would add the alert that should have existed: scheduled pipelines must produce an experiment within a window."

Strong signal: naming the silent-failure design and proposing the missing alert. Weak signal: restarting the server.

A team says their metrics are missing after step 40,000

"Two candidates. One is throttling: if the warning icon is on the experiment and the log shows Experiment has been throttled, they exceeded a per-minute budget for that data type. The fix is to log per epoch, batch with log_metrics, or run offline. The other is sampling: above 15,000 values per metric the platform down-samples by reservoir sampling, so curves look sparse but are not wrong. If they need every value, enable total fidelity for the project and read with get_metric_total_df(). I distinguish them by checking the warning and counting what they logged."

Self-hosted uploads fail with 500 or 403

"The message is Uploading file failed (500) with max retries: 4 or an AmazonS3Exception: Access Denied Status Code: 403. Files above about 8 MB use multipart upload, and the bucket policy likely lacks multipart permissions. I would grant s3:PutObject, s3:PutObjectAcl, s3:InitiateMultipartUpload, s3:UploadPart, s3:CompleteMultipartUpload, s3:AbortMultipartUpload and s3:ListMultipartUploadParts. As a stopgap, teams can use remote assets so the bytes never pass through the server."

A rollout of inference pods fails with HTTP 429

"Registry downloads are limited to 20 per second, and daily to 5,000 per account and per model version on paid plans. A rollout where every pod downloads at start-up can trip either. I would bake the model into the image or cache it on a shared volume, stagger the rollout, and have the deployment pipeline download once. If we genuinely need more, we contact support."

An API key was committed to a repository

"Revoke and regenerate immediately. For a service account, regenerate the key from the menu; the account and its data stay. Check where else the key appeared: the SDK logs source code by default, so a key in a script could be in an experiment's code tab. Then look at what the key could reach, because a service account has organisation-admin permissions unless restricted to specific workspaces. Finally fix the cause: move keys to a secret manager or JWT, and add a scanner in CI."

After a server upgrade, SDKs fail with version errors

"The SDK gates features on backend versions. A backend that is too old raises BackendVersionTooOld, and an SDK that is too old raises SDKVersionIsTooOldException with 'Obsolete SDK version. Please update.' I would check the version matrix: fallback to offline needs backend 3.3.11, the current registry API needs 3.5.42, and set_project_owner needs 4.35.0. The fix is to align versions, and the lesson is to pin SDK versions in base images and to run a smoke test after every upgrade."

Trade-off and leadership arguments

Convince leadership to fund Enterprise

"I would not argue features; I would argue risk and work avoided. Service accounts, SSO enforcement and view-only users are Enterprise features, and a shared platform needs service accounts so that pipelines do not share one person's key. The alternative is engineering time building key rotation and access control around a cheaper plan. I would compare a per-seat price for the number of users we actually have against the cost of a platform engineer's time and the audit exposure of shared keys. I would also be honest that Pro is 19 dollars per user per month for up to ten users, which may be right for a small team, and that prices change so the pricing page is the source."

Argue for or against moving to MLflow

"The honest frame is what we use. Comet's strengths are the comparison interface, artifact lineage, the approval-based registry and packaged enterprise features. MLflow is open source, which suits teams that want no licence conversation and are willing to run it. I would measure which Comet features teams actually use. If it is only metric logging and a table, the migration cost is lower and the saving is real. If teams rely on lineage, approvals and panels, we would be rebuilding them. Migration tooling exists into Comet from Weights & Biases and Neptune through cometx; out of Comet I would script against the REST API and I would pilot on one project first."

Justify self-hosting despite the operational burden

"I would state the driver, which is data residency or network isolation, and then price the burden honestly: cluster, MySQL, Redis, object storage, backups, upgrades, on-call, and a feature lag behind cloud. I would propose a rehearsed exit path as a condition, meaning a tested restore and an export with cometx download. Leadership usually accepts a burden they can see a number for."

Say no to a team that wants to log everything

"I would explain in their terms. Per-step logging of many metrics from many ranks consumes the 12,000 per minute metric budget and produces charts no one can read. I would offer a better design: log scalars from rank zero, log per epoch or every N steps, batch with log_metrics, and keep the big files behind remote references. If they have a real need for high frequency, offline mode has no rate limits. The answer is a path to what they need, not a refusal."

A team wants LLM tracing in Comet

"LLM tracking moved to Opik. The old comet_llm package is legacy with no releases since 2024, and logging an ML experiment into an LLM project raises an error telling you to use an Experiment Tracking project. So I would point the team to Opik, which ships on the same platform with its own pricing by spans, and keep comet_ml for classical and deep-learning experiments."

Fast-recall round

How do you start an experiment today? comet_ml.start(...), generally available since 3.47.1. The classes Experiment, ExistingExperiment and OfflineExperiment still work but are not preferred. comet_ml.init() is deprecated in favour of comet_ml.login().

What are the three start modes? get_or_create (default), get (must exist, raises ExperimentNotFound) and create (raises if the key exists, good for loops).

What does error_mode() do? It makes the SDK raise on invalid, discarded or truncated data. It lasts for the whole process, and asynchronous upload errors still do not raise.

What replaced registry stages? Tags, and a status with an approval workflow.

How do you debug without DEBUG console logging? DEBUG console logging is forbidden since 3.45.0 and is downgraded to INFO. Use COMET_LOGGING_FILE and COMET_LOGGING_FILE_LEVEL=debug, set before importing comet_ml.

What is the config precedence? Code arguments, environment variables, .env, the file named by COMET_CONFIG, ./.comet.config, the Colab Drive file, ~/.comet.config, then backend overrides.

Which import path for the AWS secret-manager helper? comet_ml.secrets.aws, plural, although some documentation prose shows the singular.

What interviewers listen for

Across these questions the same signals separate senior answers from mid-level ones. The first is that you start from a constraint or a limit rather than from a feature. The second is that you know how things fail quietly, and you propose the alert that would have caught it. The third is that you attach a number: 15,000, 25,000, 12,000 per minute, 20 per second. The fourth is that you know when Comet is the wrong answer, and you say so without being asked. The fifth is that every design you describe has an exit: a backup you have restored, an export you have tried, a version you can roll back to.

Sixty-second self-test

Answer each without looking back.

1. Name the four levels of the hierarchy and say which one is the isolation boundary for teams.
2. Which limit do shared service-account keys hit, and what is its value?
3. What does the SDK do when the server is unreachable during a run?
4. Why is a service-account key more dangerous than a personal key?
5. What is the safe way to deploy a model from the registry?
6. What happens to a trial when the optimizer's spec.metric is never logged?
7. Name two things to check before you tell a compliance officer that telemetry is off.
8. When would you not use Comet?

If you can answer all eight with specifics, you are ready for the senior loop. If any answer was "it depends" without saying on what, go back to the section that covers it and add the number or the message.

How to use this page

These are the things that only hurt at scale: the trap that is invisible with three users and expensive with thirty, the habit that keeps a platform calm, and the checklists to run against pull requests and design documents. Each trap gives the symptom, the cause and the fix, using the exact messages where the SDK prints one. The numbers come from the official documentation and the 3.58.7 SDK; when a number is a cloud default, the self-hosted value may be different, so confirm it with your deployment.

Production traps that only appear at scale

Trap 1: The fleet that logs nothing and reports success

Symptom: a week of training jobs finished with exit code zero, and the project has no new experiments. Cause: the SDK is designed not to crash your job. A wrong or revoked key prints The given API key '...' is invalid on '...'... Your experiment will not be logged, and a failed handshake prints Run will not be logged. Training continues. Fix: alert on absence. Every scheduled pipeline should produce an experiment within a window, and a platform job should compare expected with actual. In CI, call comet_ml.error_mode() so invalid, discarded or truncated data raises instead of logging a warning. Remember that asynchronous upload errors still do not raise, so error mode is not a substitute for the absence alert.

Trap 2: One service-account key shared by everything

Symptom: dashboards, CI and a nightly report slow down or fail in the same hour with rate-limit errors. Cause: the REST API allows 15,000 submissions per hour per API key. A shared key is one budget. A second, worse issue: a service account has organisation-admin-level permissions by default, so the shared key is also the most powerful credential you own. Fix: one service account per purpose, restricted to the workspaces it needs in the Admin dashboard, with a rotation routine. Service accounts are Enterprise-only, so if you are not on Enterprise, this is a reason to have the plan conversation.

Trap 3: Every rank logs every step

Symptom: a sixty-four-GPU job is throttled, the UI shows a warning icon, and the log says Experiment has been throttled. Some data (like experiment metrics) might be missing. Cause: each process has its own experiment object, and each submits metrics. The cloud budget is 12,000 metric submissions per minute per data type. Sixty-four ranks at a few steps a second exhaust it quickly. Fix: log scalars from rank zero, log per epoch or every N steps, and use log_metrics with a dictionary. If every rank must log, share one experiment_key and set distributed_node_identifier per node, or use offline mode, which has no rate limits. Upgrade anything older than 3.49.2, which fixed a race in multi-node experiment creation.

Trap 4: Silent down-sampling

Symptom: a chart looks jagged or sparse and a downloaded metric has fewer points than the loop ran. Cause: above 15,000 values per metric per experiment, Comet down-samples with reservoir sampling. Nothing warns you. Fix: decide per project. For regulated or forensic work, enable the organisation setting "Store metrics without sampling" and read values with APIExperiment.get_metric_total_df(). For everything else, teach people that curves are a summary.

Trap 5: Searches that eat the project

Symptom: Maximum number of experiments reached for project '...' of workspace '...'. Archive experiments or use a different project. Data will not be logged. Cause: a project holds 25,000 experiments, and a hyperparameter search creates one experiment per trial. Two large searches fill a project, and data stops being logged. Fix: searches live in their own project, old experiments are archived on a schedule, and maxCombo is bounded, because total runs equal maxCombo times trials. The error is nasty because it arrives as the project fills, mid-run, for everyone using it.

Trap 6: The search whose metric never arrives

Symptom: the optimizer runs trials but the sweep learns nothing, with Optimizer metrics is '<metric_name>' but no logged values found. Experiment ignored in sweep. Cause: the spec.metric name does not match the logged name exactly, for example val_acc versus val/acc, or a prefix added by with exp.validate() context. Fix: log the exact name, and have a one-trial dry run in the review checklist. Note that objective only drives bayes and defaults to minimize; a maximise-by-default assumption turns accuracy search into a hunt for the worst model.

Trap 7: Everyone pulls the model at once

Symptom: a rollout half-fails with HTTP 429 on model download. Cause: registry downloads are limited to 20 per second and, on paid plans, 5,000 per day per account and per model version; free plans allow 1,000. A deployment where each pod downloads on start exceeds them. Fix: download once in the pipeline, bake the model into the image or a shared volume, and stagger restarts. Do not let autoscaling turn into a download storm.

Trap 8: Deploying "whatever is Production"

Symptom: a pod restart in the middle of the night serves a different model than the one the release notes describe. Cause: the deploy resolved a status or tag at start-up, and someone changed it. Fix: deploy an explicit version, recorded in git. Use status for governance, not for resolution. A rollback should be a git revert, not a registry edit.

Trap 9: Stage-based code and status-based platform

Symptom: deprecation warnings everywhere, and eventually a failure when a stage method is removed. Cause: registry stages are deprecated. Their data became tags, and status with an approval workflow is the official lifecycle. register_model(stages=...), comet models download --model-stage and all stage methods warn. Fix: migrate to model.set_status, add_tag, find_versions and download. Also remember the June 2025 status configuration changed the structure of the webhook payload, so test consumers that predate it.

Trap 10: The webhook that trusts everyone

Symptom: an unexpected pipeline run, or no validation of where a call came from. Cause: Comet can send a secret in the Authorization header, but your receiver must check it. Fix: validate the secret, verify the event type (MODEL_VERSION_CREATED, MODEL_VERSION_DELETED, MODEL_TAG_CHANGED, MODEL_STATUS_CHANGED or TEST), and have the handler trigger a pipeline instead of doing work in-line. An unauthenticated endpoint that can start a deployment is a remote-execution path.

Trap 11: Secrets in the code tab

Symptom: a key or password visible in an experiment. Cause: the SDK logs source code by default, and also records command-line arguments, environment details, git metadata and output. A literal in a script, a token passed as an argument, or a printed environment ends up in a place many people can read. Fix: never write comet_ml.login(api_key="...") with a literal. Use JWT, a secret manager or injected environment variables, turn off the auto-logging you do not need through ExperimentConfig, and add sensitive names to parameters_ignore and others_ignore. The key itself is redacted in logs by default, but that does not help with a key in the file.

Trap 12: Disabling TLS verification to get unblocked

Symptom: There's seem to be an issue with your system's SSL certificate bundle. after a network change. Cause: a corporate inspection proxy, a broken CA store or a self-signed server certificate. Fix: add the CA to the trust store, or point REQUESTS_CA_BUNDLE or comet.include.custom_ca_bundle at it. COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 works and is a finding in a review. It is reasonable to make this a blocked pattern in repository scanning.

Trap 13: Assuming the docs match the code

Symptom: copied code raises ModuleNotFoundError, or a flag is refused. Cause: documented drift. Real examples: the secret-manager import is comet_ml.secrets.aws, plural, but one documentation page shows comet_ml.secret.aws; the comet models download help text shows --registry_name and --model_version but the real flags are --model-name and --model-version; the configuration page lists COMET_INI although it is deprecated in favour of COMET_CONFIG; and the system-metrics interval is 30 seconds, not the 60 the page says. Fix: verify in a scratch environment against the pinned version, and record the result in your internal runbook. The public release notes stop at 3.56.0, so for 3.57 and 3.58 the source and your tests are the reference.

Trap 14: Server and SDK drift apart

Symptom: BackendVersionTooOld or SDKVersionIsTooOldException with "Obsolete SDK version. Please update." Cause: features depend on backend versions, for example fallback to offline at 3.3.11, the registry API at 3.5.42 and set_project_owner at 4.35.0. Fix: pin both sides, upgrade in a staging workspace first, and run cometx smoke-test WORKSPACE.

Trap 15: Bucket permissions that only break big files

Symptom: small uploads work, models fail with Uploading file failed (500) with max retries: 4 or an S3 403. Cause: files above about 8 MB use multipart upload and the bucket policy lacks multipart actions. Fix: grant s3:PutObject, s3:PutObjectAcl, s3:InitiateMultipartUpload, s3:UploadPart, s3:CompleteMultipartUpload, s3:AbortMultipartUpload and s3:ListMultipartUploadParts. Test with a file larger than 8 MB as part of the install checklist, because every small test passes.

Platform-ownership habits

Keep the golden path thinner than the documentation

A platform that needs a wiki page to be used correctly will be used incorrectly. Put the decisions in a template: a base image with comet_ml pinned, a .comet.config with organisation defaults, the CA bundle, and a small helper that creates ExperimentConfig(name=..., tags=[...]) with the standard tags (pipeline, git commit, owner). A team should be able to copy the repository and get correct, loud behaviour.

Decide identity once

Pick one key distribution mechanism and forbid the others in review: a JWT from the identity provider, a secret manager with get_api_key_from_secret_manager, or an injected COMET_API_KEY. Every automation gets its own service account. Rotate on a schedule through the REST endpoints for service-account rotation, not through the dashboard by hand, and make sure the rotation path is tested before a leak forces it.

Measure the platform, not just the pods

Pod health and the ping endpoint tell you the web tier is alive. They do not tell you that training is being recorded. Track four numbers: experiments created per day per workspace, the ratio of runs whose state is crashed (read with APIExperiment.get_state()), the age of the oldest un-uploaded offline zip, and the count of throttled experiments. Put them on a Grafana board, fed from Prometheus or a script that reads the API.

Rehearse restore and exit

A backup you have not restored is a hope. On cloud Kubernetes use volume snapshots; on bare metal back up the whole disk with rsync, Restic or Borg, because backing up individual data directories means reconfiguring everything on restore. Twice a year restore into a scratch environment and time it. Also rehearse the exit: cometx download a project and cometx copy it into a second installation with its own API key. cometx is a community-supported tool and not an official Comet project, so test it on real data before you depend on it.

Upgrade on a schedule, through an image

Upgrade the SDK through the base image, not through individual requirements files, with a smoke test that starts an experiment, logs a metric, an image, a model and an artifact, registers a model and downloads it. Read what changed, but remember the public notes stop at 3.56.0 and your tests are the changelog for 3.57 and later. Know the behaviour changes that have bitten others: auto_output_logging default moved from "native" to "simple" in 3.47.3, DEBUG console logging was forbidden in 3.45.0, and Python 3.7 support ended in 3.53.0.

Teach the four messages

Most support volume is four messages. Run will not be logged (handshake: check network, proxy, TLS with comet check --debug). Throttling (log less often, batch, or go offline). The import-order warning, To get all data logged automatically, import comet_ml before the following modules (import comet_ml first or run comet python script.py). And 429 on download (cache the model). Put these in a short FAQ with the exact text so people find it by pasting.

Treat assistants as key holders

The comet-mcp server (1.4.2) lets an assistant list projects, read experiment code and fetch metric data using a COMET_API_KEY. Whatever key you give it is what it can read, so give it a narrowly scoped identity and never an admin-equivalent service account.

Know where the platform stops

Say no, politely and early, to orchestration inside Comet (use Kubeflow, Metaflow or Airflow), serving (use BentoML or KServe), and LLM tracing (use Opik, because LLM tracking moved there and comet_llm has had no release since 2024). A platform that says yes to everything turns into three half-platforms.

Review checklists

For a pull request that adds tracking to a training job

• Does the code import comet_ml before any ML framework, or run under comet python?
• Is the entry point comet_ml.start(...), not init() or a bare Experiment(...), and does it use project_name, not the deprecated project?
• Is the API key absent from the source, and absent from command-line arguments?
• Is end() called, including on termination signals? In a notebook, is it called explicitly?
• Is the logging rate sensible (per epoch or every N steps), with log_metrics batching, and rank-zero gating under distributed training?
• Are datasets logged as artifacts, and are large files remote references?
• Are name and tags set through ExperimentConfig?
• Does any argument, environment dump or patch contain something sensitive that auto-logging would capture?
• Is the model logged with log_model so it can be registered, rather than as an artifact?
• Does COMET_INTERNAL_CHECK_TLS_CERTIFICATE=0 appear anywhere? It should not.

For a pull request that touches a hyperparameter search

• Is spec.metric logged under exactly that name?
• Is objective set explicitly, knowing the default is minimize?
• Is maxCombo bounded, and is the total (maxCombo times trials) understood?
• Is the scaling type spelled loguniform, not log_uniform?
• Is the search in its own project?
• Is bayes run with modest parallelism?

For a design document that adds a team to the platform

• Which workspace, which projects, and who are the owners?
• How do jobs obtain credentials, and is each automation on its own service account, limited to specific workspaces?
• What leaves the machine through auto-logging, and has the data owner accepted it? If residency matters, is the server in the right region?
• What happens if Comet is unreachable for an hour, and who uploads the offline zips?
• What are the expected volumes against the limits (12,000 metrics per minute, 15,000 REST submissions per hour, 25,000 experiments per project, 100 models per experiment)?
• How is a model promoted, who approves, how is the webhook secured, and does deployment pin a version?
• Which SDK version, and which backend version if self-hosted? Is there an upgrade test?
• What is the exit path?

For a self-hosted change request

• Is TLS terminated at a layer-7 load balancer, with a certificate the SDK hosts trust?
• Does the bucket policy include the multipart actions, and was a file above 8 MB tested?
• Is there a backup that has been restored, and is the restore time known?
• Is the usage reporting to stats.comet.com allowed or agreed with Comet?
• Did kubectl get pods, the ping endpoint and cometx smoke-test pass after the change?
• Is a first-admin recovery path documented (ADMIN_USER_LIST in /etc/default/comet-ml or Helm additionalConfigurations:)?

Practice cards

Card one. Write the alert that detects an unlogged fleet. Decide the window, the source of expected runs and who is paged. Test it by revoking a key in a staging workspace.

Card two. Take one real training script and estimate its metric submissions per minute, multiplied by the number of ranks. Compare with 12,000. Change the code until it is under half of the budget, and verify there is no throttle warning.

Card three. Create a service account, restrict it to one workspace, and confirm that it fails with Service account doesn't have access to workspace %s against another. Rotate its key and update the secret store without dropping a run.

Card four. Register a model, trigger the webhook TEST event, and save the payload. After your next platform upgrade, compare. Then build the GitHub Actions workflow that downloads a pinned version.

Card five. Take a deployment that resolves a model by status and change it to an explicit version in git. Practice a rollback by reverting the commit.

Closing advice

The habits above reduce to a short list. Make failure loud. Give every automation its own identity and keep secrets out of code. Put limits in front of users before they find them. Pin versions, test upgrades and read the source where the docs drift. Enforce lineage where audits matter. Keep the exit rehearsed. A platform owner who does those six things will spend their time on improvements, not on the same four support messages every week.