The Complete Weights & Biases Guide

This is part one of three. It covers everything you need to do real work with Weights & Biases, not a teaser. By the end you can log a training run from a script, read the charts it produces, compare it against other runs, version a dataset and a model as artifacts, launch a hyperparameter search, and recover when something breaks. Mid-level and Senior take the same topics further, and nothing you learn here is thrown away.

This guide is written against the Python SDK 0.30.0, released in September 2026. The SDK is still a 0.x library, which means the minor version can remove things, so if you copy code from an older blog post and it fails with an AttributeError, the reason is often a removal and not your mistake. We point out the important removals as we meet them.

Each section ends with a Try it task. Do them as you go. Experiment tracking only makes sense once you have watched your own numbers appear on a chart.

What Weights & Biases is, and what you can do by the end

Weights & Biases, usually shortened to W&B and spoken as "wand-B", is a service and a Python library for recording what happens while you train machine learning models. Your training script calls a few functions. Those functions send the numbers (loss, accuracy, learning rate), the settings you used, the machine's GPU usage and even the code version to a web dashboard. You then open the dashboard, see your runs drawn as charts, and compare them side by side.

That sounds modest, and it is also the thing that separates a person who is guessing from a person who is learning. Consider what happens without it. You train a model on Monday and it reaches 91 percent accuracy. On Wednesday you change the learning rate, train again and get 93 percent. On Friday your manager asks which settings produced the 93 and whether it is better than last week's model. If the answer lives in terminal scrollback and file names like model_final_v2_REAL.pt, you cannot give it.

By the end of this guide you will be able to:

• Install the library, create an account, and authenticate safely.
• Write a training script that records its settings and metrics in a run.
• Read the dashboard: line charts, the run table, the system metrics and the overview page.
• Store a dataset or a model as a versioned artifact and fetch it again from another script.
• Describe a hyperparameter search in a YAML file and let W&B drive it.
• Work offline, sync later, and read the most common error messages.

The diagram is the whole product in one line. Everything else in this guide is a detail of one of those four boxes.

A note on where the service runs. The free, hosted version is called the Multi-tenant Cloud. The documentation has recently begun branding it CoreWeave Forge, after the company that now operates W&B, and some pages still say "W&B Multi-tenant Cloud" and link to wandb.ai. They are the same service for your purposes. The library still talks to https://api.wandb.ai by default, so you do not need to change any URL to use it. Companies with strict data rules can also run a Dedicated Cloud in a region they choose, or host the server themselves. That matters for employers in the Gulf and Egypt who must keep data in a particular country, and Mid-level and Senior explain the options. For this guide, use the hosted service.

The problem it solves, and what came before

Training a model is an experiment, and experiments need a lab notebook. Before tools like W&B, people used three approaches, each with a clear weakness.

The first was print statements and terminal scrollback. You watch the loss scroll by. When the run ends, the numbers are gone unless you redirected the output to a file, and even then you cannot draw a chart without writing more code.

The second was spreadsheets and naming conventions. You copy final numbers into a sheet by hand and encode settings into folder names such as run_lr0.01_bs64. This works for three runs and collapses at thirty, because people forget to fill in the row, misspell a setting, or never record the code version.

The third was TensorBoard, a viewer that reads log files written to your disk. It draws good charts, but the files live on whichever machine trained the model. Sharing them with a teammate means copying directories around, and comparing a run on your laptop with one on a cloud GPU means moving files first.

W&B's approach is to make the record automatic and central. The library sends data to a server while your code runs, so the run exists in one place the moment it starts, visible to anyone on your team. It also records things you did not ask for: the Python version, the operating system, the command that started the script, the GPU, and the git commit. When a teammate asks "how did you get that number?", the run page answers.

Two products in the same family are worth a name, so you are not confused later. Weave is a separate W&B product for tracing and evaluating applications built on large language models, and it is outside this guide. Registry, which we touch on in the artifacts section, is the organisation-wide shelf where finished models and datasets are published. Everything in this guide concerns the experiment-tracking product, which W&B calls Models.

Other tools solve the same problem. MLflow does experiment tracking and you can self-host it for free, and our MLflow guide is a good companion. Comet and ClearML are commercial and open alternatives. DVC focuses on versioning data and pipelines, and there is a DVC guide as well. W&B's strength is the dashboard, the speed of getting started and the depth of its team features. Its cost is that the hosted service is a third-party dependency, which you should weigh if your employer has data-residency rules.

The core mental model: entity, project, run, config, history, summary

W&B has a small vocabulary. Learn these words now, because every error message and every doc page uses them.

A run is one unit of computation that you log. In practice, it is one execution of your training script. You create a run by calling wandb.init(), and it ends when the script finishes or you call finish. Everything else hangs off a run. Each run has a unique ID (eight characters, such as 3x7k9a2q, generated for you) and a human-friendly name, which W&B invents as two random words unless you choose one.

A project is a folder of runs that belong together. You might have a project called churn-model holding every run you trained for that task. Project names can be up to 128 characters and cannot contain /, \, #, ?, % or :. If you forget to name one, W&B guesses from your git folder or script name, and if it cannot guess it uses uncategorized. Give projects names on purpose, because a pile of runs in uncategorized is hard to use.

An entity is the owner of a project. It is either your personal username or a team. A team is a group of people who share projects, and it is what you use when several colleagues need to see the same runs. The entity must exist already, and if you leave it out, your default entity is used. A detail that will matter later: artifacts logged under a personal entity cannot be linked to the organisation Registry, so work you intend to share or promote should live under a team.

Config is the set of inputs to a run, meaning the hyperparameters and settings that define the experiment: learning rate, batch size, number of epochs, model name, dataset version. You pass it as a dictionary, and it is stored in run.config. Config answers the question "what did I choose?".

History is the time series of everything you log during the run. Each call to run.log() adds a row of numbers, and by default each call moves the run forward by one step. History answers "what happened, and when?", and it is what draws the line charts.

Summary holds one final value per metric. By default it is the last value you logged for each name, so a run that ended with val_acc of 0.93 has a summary of 0.93. You can override it, for instance to keep the best value instead of the last. Summary answers "how did it end?", and it is what the run table sorts on.

An artifact is a versioned bundle of files that a run consumes or produces: a dataset, a model checkpoint, an evaluation set. Artifacts are the answer to "which exact data and which exact weights?", and we cover them in their own section.

A sweep is a hyperparameter search. You describe the space of settings to explore, and W&B hands different combinations to one or more worker processes called agents.

There is one more piece of machinery to know about, because it explains behaviour that would otherwise seem magical. When you call wandb.init(), the library starts a helper process called wandb-core, written in Go. Your Python code hands data to it, and it does the uploading, the streaming of metrics and the monitoring of CPU, memory and GPU. It also writes everything to a local file named run-<id>.wandb inside a folder called wandb/ in your working directory. This has a comforting consequence: if the network drops or the helper crashes, your training keeps going, and the local file lets you upload the data later with wandb sync.

Installing and checking your setup

W&B needs Python 3.10 or newer. Older releases of Python are no longer supported by the current library. On macOS you need version 13 (Ventura) or newer, and on Windows you need a 64-bit Python, because 32-bit wheels are no longer built. Linux works on both x86-64 and ARM machines.

Create a virtual environment so the library does not collide with other projects, then install it:

python -m venv .venv
source .venv/bin/activate        # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install --upgrade wandb

Inside a notebook, use !pip install wandb in a cell. Confirm the version from both the command line and Python:

wandb --version
python -c "import wandb; print(wandb.__version__)"

Both should print 0.30.0 or newer. If they disagree, you have two Python installations and the wandb command belongs to a different one than the python you are using. Activating the virtual environment normally fixes that.

Next, create an account at the W&B website and generate an API key. The key is the password your scripts use. Open your profile menu, choose User settings, then API keys, then New key. Copy it straight away: W&B shows the full key only once, and afterwards the page shows only a short prefix. Newer keys are around 86 characters long. Very old library versions (before 0.22.3) reject them with API key must be 40 characters long, yours was 86, and the cure is to upgrade the library.

Now log in:

wandb login

It prompts you to paste the key. The library then stores it in a file called .netrc in your home directory, so you only do this once per machine. You can also pass the key directly, as in wandb login <KEY>, or set an environment variable, which is the normal approach on servers and in automation:

export WANDB_API_KEY=your-key-here        # Linux or macOS

$env:WANDB_API_KEY="your-key-here"        # Windows PowerShell

Verify the whole setup with two small checks:

wandb login --verify
wandb status

The first confirms that the stored credentials work and shows which source they came from and your default team. The second prints your current settings. As of 0.30.0 it no longer prints the API key itself, which is sensible because people paste this output into support chats. You can also test from Python, which is handy in scripts and returns a boolean without prompting:

import wandb
print(wandb.login(prompt=False))   # True if credentials are configured

Finally, a smoke test. This tiny script should print a line ending in View run at ... and produce a run marked Finished in your dashboard:

import wandb

with wandb.init(project="smoke-test", config={"lr": 0.01}) as run:
    run.log({"loss": 0.1})

Your first real run, built step by step

The smoke test logged one number. Now build something that looks like real training. To keep the focus on W&B and not on a machine-learning framework, we will fit a straight line to noisy data with gradient descent in plain Python. The structure is identical to a neural network: a loop, a loss that shrinks, and numbers worth recording.

Start with the data and the settings. Save this as train.py:

import random
import wandb

config = {
    "learning_rate": 0.05,
    "epochs": 30,
    "noise": 0.3,
    "seed": 42,
}

random.seed(config["seed"])
xs = [i / 10 for i in range(50)]
ys = [2.0 * x + 1.0 + random.gauss(0, config["noise"]) for x in xs]

The true relationship is y = 2x + 1, with noise added. Our model has two unknowns, a slope w and an intercept b, which training should recover. Now add the training loop inside a run:

with wandb.init(project="line-fit", name="baseline", config=config,
                tags=["baseline"], notes="First attempt, plain gradient descent") as run:
    w, b = 0.0, 0.0
    lr = run.config["learning_rate"]

    for epoch in range(run.config["epochs"]):
        grad_w = grad_b = loss = 0.0
        for x, y in zip(xs, ys):
            err = (w * x + b) - y
            loss += err * err
            grad_w += 2 * err * x
            grad_b += 2 * err
        n = len(xs)
        w -= lr * grad_w / n
        b -= lr * grad_b / n
        run.log({"train/loss": loss / n, "weights/w": w, "weights/b": b})

    run.summary["final_w"] = w
    run.summary["final_b"] = b

Run it with python train.py. The terminal prints a link to the run. Open it, and you will see three line charts appear: the loss falling toward the noise level, and the two weights climbing toward 2 and 1. You wrote no chart code at all.

Walk through what each part did, because the details are where beginners go wrong.

The with block. Using wandb.init(...) as a context manager means the run is finished for you when the block ends. If your code raises an exception, the run is marked Failed, which is honest. Without the with block, a crash often leaves the run marked Crashed, which W&B decides only after it stops receiving heartbeats, and that is slower and less informative. Prefer the context manager in every script.

config=config. The dictionary becomes run.config. We then read learning_rate back from run.config and not from our original dictionary. That habit matters because in a sweep, W&B overwrites the config with the values it chose, and reading from run.config picks them up with no code change.

name, tags and notes. The name labels the run in the table. Tags are short words you can filter by, and notes are free text. All three are optional, and all three save you later.

run.log({...}). Each call records one step: a row of numbers. Put everything you want to see at the same step into one dictionary and one call. Calling run.log separately for each metric advances the step each time, so your charts end up with misaligned x-axes and far more steps than you intended.

Metric names with a slash. Names such as train/loss and weights/w make the dashboard group charts into collapsible sections called train and weights. That is the standard convention in the official examples. The documentation recommends names built from letters, digits and underscores, with / used for grouping. Avoid spaces, commas and hyphens in metric names, because they can cause trouble when you query the data later.

run.summary. We set two extra values by hand. Anything you assign there appears as a column in the run table.

Now run the script a second time with a different learning rate. Edit config to use "learning_rate": 0.01, change the name to "slow-lr", and run it again. Go to the project page. You now have two runs in the same project, and this is the moment the tool begins to earn its keep. The charts overlay both runs, and the run table has a column for learning_rate, so you can see immediately that the lower rate converges more slowly.

Two habits are worth forming from the start. First, log inside the with block and never afterwards, because logging to a finished run produces the warning Run (<id>) is finished. The call to log will be ignored. Second, prefer run.log(...) over the module-level wandb.log(...). The module-level version works on a global variable that can point at the wrong run if you ever create more than one, and the documentation steers learners toward the method on the run object.

Logging in depth: metrics, config, summary and media

The first project used plain numbers. Real projects log more kinds of data, and knowing the options saves you from reinventing them.

Scalars are single numbers like loss and accuracy. You have seen them. Log training metrics every step or every few steps, and validation metrics once per epoch. There is no need to log every step of a long run. The documentation suggests staying below roughly 1,000 log calls per minute, and logging every batch of a fast loop can exceed that.

Config values are set when the run starts. You may add to them later with run.config.update({"dataset_size": 5000}). One rule catches beginners: by default, changing a config key that is already set raises an error such as Attempted to change value of key "lr" from 0.01 to 0.02. The protection exists because config is meant to be a fixed record. If you really need to change it, pass allow_val_change=True. Better still, if a value changes during training, it was never a config value. Log it as a metric. Config keys should also avoid a dot in their names, and the whole config should stay under 10 MB, which is generous.

Summary defaults to the last value of each metric, and that is often wrong. For a validation loss, the last epoch is usually not the best one. You can tell W&B how to summarise a metric with define_metric:

run.define_metric("val/loss", summary="min")
run.define_metric("val/acc", summary="max")

Call these once, right after init. The accepted summaries include min, max, mean and last. Two older spellings are gone: summary="best" and the goal= argument were removed in 0.29.0, so if you find them in an old tutorial, use min or max instead.

define_metric has a second job, which is choosing the x-axis. By default the x-axis is the step. If you would rather plot against epochs, log an epoch number and declare it:

run.define_metric("epoch")
run.define_metric("val/*", step_metric="epoch")

The star matches every metric that starts with val/, so one line covers them all.

Images are logged with wandb.Image. This is essential for computer-vision work, because a wrong label is easier to spot in a picture than in a number. Pass a PIL image, a NumPy array or a path:

run.log({"examples": wandb.Image(img_array, caption="input 17")})

There is a change to know about. Since 0.29.0, wandb.Image no longer rescales pixel values for you, and its normalize argument was removed. An array must already hold values in the range 0 to 255. If you pass floats between 0 and 1, the image appears almost black, and the fix is to multiply by 255 and convert to an unsigned 8-bit integer first.

Tables are the tool for examining predictions. A wandb.Table holds rows and columns that you can sort and filter in the browser, and it can contain images too:

table = wandb.Table(columns=["id", "text", "label", "prediction"],
                    data=[[1, "great film", "pos", "pos"],
                          [2, "dull plot", "neg", "pos"]])
run.log({"val_predictions": table})

Open the run and you can sort by the rows where label differs from prediction, which is often the fastest way to find out what your model misunderstands. Other media types exist, including audio, video, HTML, histograms and 3D objects. Note that wandb.Video built from an array now requires a format= argument.

System metrics need no code at all. While a run is active, W&B samples the machine every 15 seconds and records CPU, memory, disk and network use, plus GPU utilisation, memory and temperature for NVIDIA cards. It also supports AMD GPUs, Apple Silicon and TPUs. These charts live in the run's System tab, and they answer a common question: is my GPU actually busy, or is my data loader the bottleneck? A GPU hovering at 20 percent usually points at the input pipeline and not the model.

Code and git. W&B records the git commit and any uncommitted difference for runs started inside a git repository, so the run page can tell you which version of your code produced it. If you also want a copy of the script itself, pass save_code=True to wandb.init. To switch git and code capture off, there are the settings disable_git and disable_code, which you would use if the repository contains something you may not upload.

Reading the dashboard

Logging is half the skill. The other half is reading what you logged, and the dashboard rewards a few minutes of orientation.

The project page has a workspace, which is a grid of charts across all the project's runs, and a runs table along the side or in its own tab. The table is the place to sort, filter and group. Click a column header such as a summary value to sort the runs by it. Use the filter to show only runs with a given tag, and the group option to collapse runs that share a setting into one averaged line. Colours are assigned per run, and clicking the eye icon beside a run hides or reveals it on every chart.

A workspace can be automatic, where W&B makes a panel for every metric you log, or manual, where you choose the panels. Automatic is perfect while you are learning. On a project with hundreds of metric names, switch to manual and add only the panels you care about, because automatic workspaces become slow.

Charts have useful controls. Hover to see exact values for every run at a step. Drag across a region to zoom. Change the x-axis from step to another logged metric, or to wall-clock time. Smoothing is on by default to make noisy curves readable, and it can hide a real spike, so lower it when you are hunting for instability.

A run page shows one run in detail, and its tabs are worth knowing:

Learn the run states, because they explain what you see in the table. Running means data is still arriving. Finished means the script ended normally. Failed means the script ended with a non-zero exit code or an exception. Crashed means W&B stopped receiving heartbeats from the process, typically because the machine ran out of memory, the job was killed, or the network was lost for a long time. Killed means someone stopped the run from outside. A sweep run can also be Pending while it waits in a queue.

When a run is Crashed and you know the process actually kept going, the data is not lost. It sits in the local wandb/ folder, and wandb sync wandb/run-<timestamp>-<id> uploads it.

Beyond the table and charts, a report is a document that mixes text with live panels. It is how teams write up an experiment, with a chart, a paragraph of interpretation and a link back to the runs. Creating one is a matter of clicking Create report from a project, and it stays connected to the data underneath.

Artifacts: versioning data and models

Runs record numbers, and artifacts record files. An artifact is a versioned bundle of files, and it solves a problem every team meets: a model is the product of its code, its settings and its data, and you cannot reproduce it without all three. Artifacts give the data and the trained weights a permanent, versioned identity.

The best way to see the value is through an example. Imagine a training script that reads data/train.csv. Someone fixes a labelling bug in that file. A week later, nobody remembers which version a given model saw. If the CSV had been an artifact, each version would have its own name, train-data:v0, train-data:v1, and each run would list exactly which version it used.

Here is the producing side. One script creates a dataset and logs it:

import wandb

with wandb.init(project="line-fit", job_type="prepare-data") as run:
    art = wandb.Artifact(name="line-data", type="dataset",
                         description="Noisy points around y = 2x + 1",
                         metadata={"noise": 0.3, "rows": 50})
    art.add_file("data.csv")
    run.log_artifact(art)

Three nouns appear here. The name identifies the artifact within the project. The type is a free-form label, conventionally dataset or model, used to group things in the UI. The metadata is a dictionary of facts about the contents. The job_type on the run is a label that describes what the run does, such as prepare-data or train, and helps the lineage graph read well.

W&B gives each new version a number starting at zero: v0, v1, v2. It also keeps movable labels called aliases. The alias latest is added automatically and always points at the newest version. You may add your own, such as best or production, when logging:

run.log_artifact(art, aliases=["best"])

If you log identical contents twice, W&B notices that nothing changed and does not create a new version. Newer library versions print a warning when this happens, which surprises people the first time.

Now the consuming side. A training script fetches the data by name and alias:

import wandb

with wandb.init(project="line-fit", job_type="train") as run:
    data_art = run.use_artifact("line-data:latest")
    folder = data_art.download()          # downloads into ./artifacts/...
    print("data is in", folder)

use_artifact records that this run used that version. download copies the files to a local folder, by default under ./artifacts. Because both the log and the use are recorded, W&B can draw a lineage graph: a picture showing dataset v1 feeding run baseline, which produced model v0. The graph is in the Artifacts tab and is one of the most persuasive features for explaining a result to a colleague.

Saving a trained model follows the same pattern. After training, write the weights to a file, then log it:

model_art = wandb.Artifact(name="line-model", type="model",
                           metadata={"final_loss": 0.09})
model_art.add_file("model.json")
run.log_artifact(model_art, aliases=["best"])

Two more ideas complete the picture. A reference artifact records a pointer to files that live elsewhere, such as an s3:// or gs:// location, using add_reference, without copying the bytes into W&B. Choose it when the data is large or must stay in your own cloud storage, which is a common requirement for regional data rules. And the Registry is an organisation-wide catalogue. When a model is good enough to share, you link a version from a project into a registry collection. Linking creates a pointer and not a copy. Remember that artifacts logged under a personal entity cannot be linked there, which is why teams log to a team.

You can also reach artifacts from the command line. wandb artifact ls <project> lists them, wandb artifact get <path> downloads one, and wandb artifact put <path> uploads a file or folder. These are useful in shell scripts and CI jobs where writing Python is overkill.

Hyperparameter sweeps

Once you can track runs, the next question is how to choose settings. Trying learning rates by editing the file and re-running gets slow. A sweep automates it: you describe the space of values to explore, and W&B hands out combinations.

A sweep has two parts. The controller lives on the W&B server and decides which configuration to try next. Agents are processes on your machines that ask the controller for a configuration, run your script with it, and report back. You can start several agents on different machines and they share the work.

Describe the search in a YAML file:

program: train.py
method: random
metric:
  name: val/loss
  goal: minimize
parameters:
  learning_rate:
    distribution: log_uniform_values
    min: 0.001
    max: 0.5
  epochs:
    values: [20, 40, 60]

Read it line by line. program is the script to run. method is the search strategy, and the choices are grid (every combination), random (random draws) and bayes (Bayesian optimisation, which uses the results so far to choose promising values). metric tells W&B what to optimise, and it must match a name your script logs exactly, so val/loss here. Under parameters, a list of values is a choice, and a distribution with min and max is a range. The distribution log_uniform_values draws numbers evenly on a logarithmic scale between the two values, which is what you want for learning rates, since 0.001 and 0.01 are as different as 0.1 and 1.

Your script needs to read from run.config, which we recommended earlier. Change the hard-coded lr line to read run.config["learning_rate"] and keep epochs likewise. Give config a default for each key so the script still works when run alone.

Create the sweep and start an agent from the terminal:

wandb sweep --project line-fit sweep.yaml

This prints a sweep ID and a ready-made command of the form wandb agent <entity>/line-fit/<sweep_id>. Run that command to start an agent, and cap it so it does not run forever:

wandb agent --count 10 your-entity/line-fit/SWEEP_ID

The --count 10 flag makes this agent run ten configurations and stop. Open the sweep page in the dashboard and you will see a parallel coordinates chart and a parameter importance panel. The first draws one line per run through each parameter, so you can spot which ranges produce low loss. The second estimates which parameters mattered most.

Two cautions. Grid search over a continuous range never ends, so use random or bayes with a --count or a run_cap line in the YAML. And a sweep can be controlled while it runs: wandb sweep --pause, --resume, --stop (let active runs finish) and --cancel (kill them) each take the sweep ID.

You can start a sweep from Python as well, which is convenient in notebooks:

sweep_id = wandb.sweep(sweep=sweep_config, project="line-fit")
wandb.agent(sweep_id, function=train, count=10)

Here sweep_config is the same structure as a dictionary, and train is a function that calls wandb.init and logs as usual.

Offline mode, environment variables and configuration

Not every machine has a network, and not every script should call the cloud. W&B handles both through modes and environment variables.

The mode of a run is online by default. Set it to offline and the run writes only to your disk, then you upload later. Set it to disabled and every W&B call becomes a harmless no-op, which is ideal for unit tests and quick debugging where you do not want clutter on the dashboard.

export WANDB_MODE=offline
python train.py
wandb sync wandb/offline-run-20260930_101500-abc123xy

The offline run lives in a folder beginning offline-run-. Running wandb sync with that path, or with no argument to sync everything pending, uploads it. The command has a --dry-run flag for checking first, and --yes for scripts that must not ask for confirmation. Since 0.30.0, several older wandb sync options were removed. In particular --sync-all now errors, so plain wandb sync is the current form. Once runs are safely uploaded, the new command wandb clean deletes local data for runs that have already been synced.

The command line also has toggles. wandb offline and wandb online switch the mode for the current directory, and wandb disabled and wandb enabled turn W&B off and on.

Environment variables let you configure without touching code, which is how you keep a script identical between your laptop and a server. The ones a beginner uses are:

A script can therefore call wandb.init() with no arguments, and the environment decides where the run goes. In CI or on a cluster node, set WANDB_API_KEY, WANDB_ENTITY and WANDB_PROJECT and you are done.

W&B keeps a few settings files as well. The system-wide one lives at ~/.config/wandb/settings, and a per-directory one lives at ./wandb/settings, which wandb init writes. One quirk from recent versions: if any key in a settings file is invalid, the library ignores all settings files and prints an error. Fix the typo, or reset with wandb init --reset.

Other places W&B writes are worth knowing so your disk does not surprise you. Run files go in ./wandb, downloaded artifacts in ./artifacts, and a cache in ~/.cache/wandb. The cache can grow. The command wandb artifact cache cleanup and the newer wandb purge-cache tidy it.

Add wandb/ to your .gitignore, because the run folders are local working data and should not be committed.

Working in notebooks and with your framework

Many beginners meet W&B in a Jupyter or Colab notebook. The rules are the same, with a few differences worth knowing.

Install with !pip install wandb in a cell, then call wandb.login(). In a notebook the login prompts you inline to paste the key. In init, the notebook default differs from scripts in one respect: changing a config value that is already set is allowed, which is why you may see no error in a notebook that would fail in a script. Use the with wandb.init(...) as run: form here too, so the run is closed when the cell finishes. If you forget, re-running a cell can start a second run while the first is still open, and the console warns you about it.

Most frameworks have a hook. You will meet these in real projects:

• PyTorch Lightning and Hugging Face Transformers can log to W&B with a single setting, such as choosing W&B as the logger or reporting destination. Both send the loss and learning-rate curves for you.
• Keras has callbacks. The old WandbCallback was removed in 0.27.1. The current ones are WandbMetricsLogger, WandbModelCheckpoint and WandbEvalCallback, imported from wandb.integration.keras.
• scikit-learn, XGBoost and LightGBM work simply: train as usual, then call run.log with the scores.

When a framework integration logs for you, you still call wandb.init first, and you still choose the project and config. The integration is only a convenience for the run.log calls.

If you use a framework that watches gradients, run.watch(model) logs gradient and parameter histograms during training for a PyTorch model. It is useful for diagnosing vanishing gradients, and it also adds overhead, so switch it off once the model behaves.

Reading errors, and fixing the common ones

Most first-week problems fall into a handful of messages. Learn to read them by their first line.

UsageError: No API key configured. Use wandb login to log in. The script has no credentials and no terminal to ask for them, as happens on servers, in cron jobs and in containers. Set WANDB_API_KEY, or run wandb login beforehand. A quick test is wandb.login(prompt=False), which returns False when nothing is configured.

Invalid API key or a key-length message. Usually the key was truncated when pasted, or the variable includes quote marks or trailing whitespace. Generate a fresh one. If you see API key must be 40 characters long, yours was 86, your library is older than 0.22.3. Run pip install -U wandb, then wandb login --relogin.

AuthenticationError: Both WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE are set. These two are alternatives. Unset one.

Failed to verify credentials. Your key is fine but the destination is wrong, for example a key from the hosted service used against a company's own server, or the reverse. Run wandb status and check which host is configured.

Timed out initializing run followed by advice about init_timeout. The script could not reach W&B within 90 seconds, often behind a firewall, a proxy or a bad connection. Fix the network, or run with mode="offline" and sync later. Raising the timeout is a last resort:

wandb.init(settings=wandb.Settings(init_timeout=180))

ConfigError: Attempted to change value of key "lr". You assigned a config key that was already set. Decide whether it is really config. If it changes, log it as a metric. If you truly mean it, pass allow_val_change=True.

Run (<id>) is finished. The call to log will be ignored. You logged after the run ended. Move the run.log call inside the with block.

Invalid project name. The name contained a forbidden character, such as a slash, or exceeded 128 characters.

429 Rate limit exceeded. You are logging faster than the service allows. Metric uploads are retried with backoff and do not stop your training, but they can delay the end of the run. Log less often, put several metrics in one log call, and stagger many simultaneous job starts.

402 Payment Required. You have hit a plan or storage limit. Since 0.30.0 this is reported at once and not retried. Delete old artifacts or raise the plan.

A run shows Crashed. Open the Logs tab and the files debug.log and debug-internal.log in the Files tab. Out-of-memory kills are the usual cause. If training kept running, recover with wandb sync.

AttributeError after an upgrade. Something was removed. Commonly hit examples are wandb.PublicApi, run.get_url(), run.project_name() and the length attribute on the results of api.runs. The replacements are wandb.Api(), the properties run.url and run.project, and len(runs).

The last habit is to read the debug logs when the error text is not enough. Each run folder contains logs/debug.log and logs/debug-internal.log, and support staff will ask for them.

Putting it all together

Here is one small end-to-end project that uses every idea above. It prepares a dataset as an artifact, trains from it with config and metrics, logs the trained model as an artifact, and can be swept. Save it as project.py.

import json
import random
import wandb

DEFAULTS = {"learning_rate": 0.05, "epochs": 30, "noise": 0.3, "seed": 42}


def make_points(noise, seed):
    random.seed(seed)
    xs = [i / 10 for i in range(50)]
    ys = [2.0 * x + 1.0 + random.gauss(0, noise) for x in xs]
    return xs, ys


def prepare():
    with wandb.init(project="line-fit", job_type="prepare-data", config=DEFAULTS) as run:
        xs, ys = make_points(run.config["noise"], run.config["seed"])
        with open("points.json", "w") as f:
            json.dump({"xs": xs, "ys": ys}, f)
        art = wandb.Artifact("line-data", type="dataset", metadata={"rows": len(xs)})
        art.add_file("points.json")
        run.log_artifact(art)


def train():
    with wandb.init(project="line-fit", job_type="train", config=DEFAULTS) as run:
        run.define_metric("train/loss", summary="min")
        folder = run.use_artifact("line-data:latest").download()
        with open(f"{folder}/points.json") as f:
            data = json.load(f)
        xs, ys = data["xs"], data["ys"]

        w = b = 0.0
        lr = run.config["learning_rate"]
        for epoch in range(run.config["epochs"]):
            loss = gw = gb = 0.0
            for x, y in zip(xs, ys):
                err = (w * x + b) - y
                loss += err * err
                gw += 2 * err * x
                gb += 2 * err
            n = len(xs)
            w -= lr * gw / n
            b -= lr * gb / n
            run.log({"epoch": epoch, "train/loss": loss / n,
                     "weights/w": w, "weights/b": b})

        with open("model.json", "w") as f:
            json.dump({"w": w, "b": b}, f)
        model = wandb.Artifact("line-model", type="model",
                               metadata={"final_loss": loss / n})
        model.add_file("model.json")
        run.log_artifact(model, aliases=["best"])


if __name__ == "__main__":
    prepare()
    train()

Run python project.py. Afterwards, in the dashboard you should find one prepare-data run and one train run, a dataset artifact, a model artifact, and a lineage graph connecting them in order: data, to training run, to model. Open the train run and confirm that learning_rate and epochs appear in its config and that the summary shows the minimum loss. For a sweep, point program in sweep.yaml at a small wrapper that calls only train(), since the data needs preparing once.

Notice what this gives a teammate. They can open the model artifact, follow its lineage to the exact data version and the exact run, read the settings, and reproduce the result.

What you can now do, and what comes next

You can install W&B, authenticate without leaking a key, write a training script that records its settings and metrics, and read the resulting dashboard. You can version a dataset and a model as artifacts and follow their lineage, describe a sweep in YAML, work offline and sync later, and read the common errors by their first line.

Here is the quick reference for the everyday tasks.

Mid-level takes each topic further. It covers how the client and the wandb-core process actually move data, resuming and forking runs that were interrupted, artifact workflows with aliases and the Registry, sweeps with early termination, the public API for pulling runs back into pandas, grouping runs for cross-validation, and using W&B from CI. It also connects W&B to its neighbours, including MLflow and Docker, so the tracking fits the rest of your pipeline.

Senior covers what you own when W&B is a platform for a team: deployment choices between the hosted service, Dedicated Cloud and self-managed Kubernetes, scaling limits and what slows down first, security through single sign-on, service accounts and identity federation, bringing your own storage bucket for data-residency needs, rate limits, upgrades, automations, and when to choose a different tool.

The best next step is a real one. Take a model you already train, add five lines of W&B to it, run it three times with different settings, and answer the question you wrote in the first section.

Sources

• W&B quickstart
• Create an experiment
• Configure experiments
• Log data from experiments
• Environment variables
• Limits and performance guidance
• Run states
• Command-line reference
• wandb login
• wandb sync
• wandb sweep
• wandb agent
• wandb artifact
• Python reference: init
• Python reference: Run
• Python reference: Artifact
• SDK release notes
• SDK changelog at v0.30.0
• Breaking-change policy
• wandb on PyPI

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Weights & Biases. Mid-level reviews this plus its own material, and Senior reviews all three. The version here is the Python SDK 0.30.0.

The thirty-second answer

Weights & Biases is a service and Python library for tracking machine learning experiments. A training script calls wandb.init() to start a run, records its hyperparameters in the run's config and its metrics with run.log(), and a dashboard draws the results so runs can be compared. It also versions datasets and models as artifacts, and it can drive hyperparameter searches called sweeps.

Then the line that shows you have used it: "a run is one execution, config is what I chose, history is what happened over time, and summary is how it ended."

The model

Keep one more fact in your pocket: if the network drops or the helper crashes, training continues, and the local run-<id>.wandb file lets you upload the data later with wandb sync.

Commands to have ready

And the five lines of Python every interviewer expects you to be able to write from memory:

import wandb

with wandb.init(project="demo", config={"lr": 1e-3, "epochs": 10}) as run:
    for epoch in range(run.config["epochs"]):
        run.log({"train/loss": 1.0 / (epoch + 1)})
    run.summary["best_loss"] = 0.1

The classic traps and why they happen

Logging each metric in a separate call. Each run.log call advances the step by default. Logging loss in one call and accuracy in the next gives two steps per batch, so the charts have misaligned x-axes. Put everything for one moment into a single dictionary.

Forgetting to finish the run. Without the with block or an explicit run.finish(), an exception leaves the run in the Crashed state after heartbeats stop, which tells you much less than Failed. The context manager marks an exception as Failed and ends the run cleanly.

Changing config after it is set. The default is to raise ConfigError: Attempted to change value of key. The reason is that config is meant to be an honest record of inputs. If a value changes during training, it is a metric. allow_val_change=True is an override, and in notebooks it is on by default, which is why the same code can pass in a notebook and fail in a script.

Using the wrong summary. The summary is the last value by default, so a run that overfits at the end shows a poor validation loss even though its best epoch was excellent. Use run.define_metric("val/loss", summary="min").

Leaking the API key. A key committed to git can be used by anyone who reads the repository. Use wandb login or the WANDB_API_KEY environment variable, and rotate any key that has leaked.

Expecting a run to use the terminal in automation. With no credentials and no terminal, wandb.init() raises UsageError: No API key configured. In CI, set WANDB_API_KEY, WANDB_ENTITY and WANDB_PROJECT.

Linking a personal artifact to the Registry. Artifacts logged under a personal entity cannot be linked to the Registry, so production work should log to a team entity.

Copying old tutorials. The SDK is still 0.x, and minor versions remove things. WandbCallback for Keras, define_metric(summary="best"), wandb.Image(normalize=...) and runs.length have all gone.

Common interview questions

What problem does experiment tracking solve? Training is an experiment with many inputs: code, data, settings and hardware. Without a record, nobody can say which settings produced a result, or reproduce it. A tracking tool records these automatically in a central place, so runs can be compared and a result can be traced back to its inputs.

What is the difference between config, history and summary? Config holds the inputs I chose, stored once per run. History is the time series of values logged during the run, one row per run.log call. Summary holds one final value per metric, by default the last one logged, and it is what the runs table sorts on. I can change the summary with define_metric, for example to keep the minimum validation loss.

What happens when I call wandb.init()? It creates a run on the server, starts a helper process called wandb-core, creates a local folder wandb/run-<timestamp>-<id>, and returns a run object. The helper does the uploading and samples system metrics. Everything is also written to a local .wandb file, which is why offline mode and crash recovery work.

Why use with wandb.init(...) as run? The context manager calls finish for me. If the code raises an exception, the run is marked Failed and not Crashed. It also makes it obvious where the run ends, so I do not log to a finished run by accident.

How do I log a metric, and what are the naming rules? run.log({"train/loss": loss}). Names should use letters, digits and underscores, with a slash to group charts into sections. I avoid spaces, commas and hyphens, which can cause trouble in queries.

How is a step counted? Each run.log call commits one step by default. I can only write to the current or next step, never a past one. To plot against something else, such as epochs, I log that value and declare it with run.define_metric("epoch") and step_metric=.

What is an artifact, and how does it differ from just saving a file? An artifact is a versioned, content-addressed bundle of files with a name, type, metadata and aliases. Logging the same contents twice does not create a new version. Runs record which artifact versions they used and produced, so W&B can draw a lineage graph from data to model. A plain file has none of that history.

What are aliases? Movable labels on a version. latest is added automatically and always points at the newest version. I can add my own, such as best or production, and a consuming script can ask for model:production without knowing the version number.

How do I use a dataset artifact in a training script? run.use_artifact("line-data:latest") records the dependency, and .download() copies the files to a local folder, by default under ./artifacts.

What is a reference artifact? An artifact that stores a pointer, such as an s3:// or gs:// location added with add_reference, without copying the bytes into W&B. I would choose it for large data or data that must stay in my own cloud storage.

How does a sweep work? I write a YAML file with the program, the method (grid, random or bayes), the metric to optimise and the parameter space. wandb sweep creates the sweep on the server, which acts as the controller. wandb agent starts a worker that asks for a configuration, runs my script with it, and reports back. I can run several agents in parallel, and I must give the agent a --count or the sweep a run_cap, because random and Bayesian searches run until stopped.

Why must my training script read values from run.config? In a sweep, W&B overwrites the config with the values it chose. A script that reads from run.config picks them up with no change, and what it trained with is exactly what was recorded.

What is the difference between grid, random and Bayesian search? Grid tries every combination, which is exhaustive but explodes with more parameters and never ends on a continuous range. Random draws values from distributions, and is a strong baseline. Bayesian uses earlier results to choose promising values, which can need fewer runs when each run is expensive.

What do I do if my machine has no internet? Set WANDB_MODE=offline. Runs are written to a folder starting with offline-run-, and I upload them later with wandb sync. To switch W&B off entirely, for example in unit tests, I use WANDB_MODE=disabled.

How do I authenticate on a server or in CI? I set WANDB_API_KEY, ideally from a secret store, together with WANDB_ENTITY and WANDB_PROJECT. I never put the key in the code or the repository.

A run is marked Crashed. What do I check? I read the run's Logs tab and the debug.log and debug-internal.log files. The usual causes are out-of-memory kills, an uncaught exception without the context manager, a preempted cluster job, or a long network loss. If the process kept running, wandb sync wandb/run-<timestamp>-<id> recovers the data.

What do the run states mean? Running is live. Finished ended normally. Failed ended with a non-zero exit code or an exception. Crashed means heartbeats stopped. Killed means it was stopped from outside. Pending means it is queued, for instance in a sweep.

How is W&B different from MLflow? Both track experiments, parameters, metrics and artifacts. MLflow is open source and is commonly self-hosted, so you control where data lives. W&B is primarily a hosted service with a stronger dashboard, collaboration features and built-in sweeps, and it can be self-managed on a licence. Data-residency rules often decide which one an employer picks. See the MLflow guide for the other side.

Why might a dashboard be slow? Usually because of too many distinct metric names, for example names that embed a step number, too many runs in one project, too many files per run, or very large logged values. The documented soft limits are 10,000 runs per project, under 1,000 files per run and under 1 MB per value. Switching to a manual workspace helps.

What is wandb-core, and what happens if it crashes? It is a Go helper process that wandb.init() starts. It uploads metrics and files and records system statistics. If it crashes, the training script keeps running, and the transaction log on disk lets me upload the data later.

What is the Registry? An organisation-level catalogue of curated artifact versions, starting with Models and Datasets registries. Linking a version from a project creates a pointer to it and not a copy. Linking sets the version's time-to-live to none.

Where should API keys never appear? In source code, notebooks that get shared, and git history. They belong in stored credentials or environment variables, and a leaked key is revoked on the API keys page.

What changed recently that a new user should know? The library needs Python 3.10 or newer. In 0.29.0, wandb.Image stopped normalising pixel values, summary="best" was removed, and several run methods became properties. In 0.30.0, wandb.login(prompt=False) was added, several old wandb sync options were removed, and wandb clean took over from wandb sync --clean.

What a strong answer covers that a weak one misses

A weak answer describes W&B as "a place to see charts". A strong answer names the four nouns, explains that config, history and summary are different things, mentions that training continues when the network fails, and knows how to work offline. On artifacts, a weak answer says "it saves my model"; a strong one talks about versions, aliases, lineage and reference artifacts. On sweeps, a weak answer stops at "it tries settings"; a strong one says who the controller and the agents are, and why a count or a run cap is needed. Mention the region when relevant: for employers in the Gulf or Egypt with data-residency rules, reference artifacts and a Dedicated or self-managed deployment are the relevant answers.

How to answer a practical question live

Sometimes the interviewer hands you a laptop or a whiteboard and says "instrument this training loop". A reliable order keeps you calm. First, decide what belongs in config: every choice that could change the result, including the seed. Second, open the run with a with block and a named project. Third, log one dictionary per step, using slash-prefixed names. Fourth, say out loud which summary you want for each metric, and add define_metric where the last value is not the best one. Fifth, mention what you would version as artifacts, which is usually the dataset and the final weights. Finally, say how it runs in automation: the key comes from an environment variable, and tests use disabled mode. Narrating these choices matters as much as the code, because it shows you understand why each line exists and not only which lines to type.

If you are asked what you would check after the first run, answer with the dashboard: the state is Finished, the loss curve behaves, the System tab shows a busy GPU, and the config matches what you intended.

Sixty-second self-test

Answer out loud, without looking back.

1. What are the three things a run records, and which one is the x-axis counter?
2. Why should the wandb.init call be in a with block?
3. What does run.log do to the step, and what goes wrong if you call it twice per batch?
4. Why is summary often misleading, and how do you fix it?
5. What is the difference between an artifact version and an alias?
6. What must a training script read its hyperparameters from so that a sweep can control them?
7. Which environment variable makes a run offline, and which command uploads it afterwards?
8. You see UsageError: No API key configured in CI. What do you set?
9. You see ConfigError: Attempted to change value of key. What are the two reasonable fixes?
10. Which state does an out-of-memory kill usually produce, and how do you recover the data?

If you can answer eight of ten, you are ready for the beginner round. The weak spots are usually items 3, 4 and 9, which are where people have not yet been burned.

Part one of three. Most beginner problems with Weights & Biases come from four misunderstandings: a run is not finished until you finish it, each run.log call is a step, config is not a metric, and the dashboard shows only what you chose to send. Start with the table, then the habits, then the checklist. The version here is the Python SDK 0.30.0.

Common errors at this level

Read these from the top of each row: the symptom is what you see, the cause is what to suspect, and the fix is the smallest change. The deeper explanation of each lives in the detailed track.

The mistakes beginners actually make, in detail

Mistake 1. Logging each metric separately

Symptom. The x-axis of the loss chart reads 0 to 2,000 after 1,000 batches, the accuracy line lags behind the loss line, and hovering shows values that do not line up.

Cause. Each call to run.log commits one step by default. Two calls per batch means two steps per batch, and each call carries only one of the metrics.

Fix. Collect the values first, then log once:

run.log({"train/loss": loss, "train/acc": acc, "lr": scheduler_lr})

If you genuinely need several calls at the same step, pass commit=False on every call except the last. Beginners rarely need that, and the one-dictionary habit avoids the question.

Mistake 2. Not finishing the run

Symptom. The runs table shows runs stuck in Running, or marked Crashed, even though the script ended hours ago. A notebook creates a new run every time you re-run a cell and the old ones never close.

Cause. The run was closed only implicitly, or not at all, and an exception prevented the normal shutdown.

Fix. Use the context manager everywhere:

with wandb.init(project="churn", config=cfg) as run:
    train(run)

The context manager calls finish for you. If the body raises an exception, the run becomes Failed, which is an honest and useful state. In a notebook, this is the most reliable way to keep one cell execution to one run.

Mistake 3. Reading hyperparameters from a global instead of run.config

Symptom. A sweep launches ten runs, but every run has identical results and the sweep's charts are empty of variation.

Cause. The script reads LEARNING_RATE = 0.05 from a constant, so the values chosen by the sweep are recorded in the config but never used.

Fix. Read every hyperparameter from the run object, after init:

with wandb.init(config={"learning_rate": 0.05, "epochs": 30}) as run:
    lr = run.config["learning_rate"]
    epochs = run.config["epochs"]

Give every key a default in the dictionary so the script still works when you run it by hand.

Mistake 4. Treating config as a place for values that change

Symptom. ConfigError: Attempted to change value of key "lr" from 0.01 to 0.02.

Cause. The code updates a config key from inside the training loop, typically a learning rate that a scheduler lowers. Config is meant to be a fixed record of what you chose at the start.

Fix. Log anything that changes as a metric: run.log({"lr": current_lr}). Keep config for the starting value. If you truly need to overwrite a config key, use run.config.update({...}, allow_val_change=True), but first ask whether it is really config.

Mistake 5. Trusting the default summary

Symptom. You sort the runs table by val/loss and the run you remember as best is near the bottom.

Cause. The summary is the last logged value by default, and the model overfit near the end.

Fix. Declare what the summary means, once, right after init:

run.define_metric("val/loss", summary="min")
run.define_metric("val/acc", summary="max")

If an old tutorial shows summary="best", that spelling was removed in 0.29.0. Use min or max.

Mistake 6. Putting the API key in the code

Symptom. Nothing visible, until a colleague or a scanner tells you the key is in the repository history.

Cause. A quick test script had wandb.login(key="...") or an os.environ assignment with the literal key, and it was committed.

Fix. Delete the key on the API keys page in your user settings, create a new one, and keep it out of files. Use wandb login on your own machine, which stores it in .netrc in your home directory, and an environment variable from a secret manager in automation. Add .env files to .gitignore.

Mistake 7. Committing the wandb/ folder

Symptom. A pull request with thousands of changed files, or a repository that swells by hundreds of megabytes.

Cause. The wandb/ directory holds local run data, including the transaction log of every run. It is working storage, not source code.

Fix. Add wandb/ and artifacts/ to .gitignore. If they are already committed, remove them from the index with git rm -r --cached wandb artifacts, then commit. Clean up local data for already uploaded runs with wandb clean.

Mistake 8. Logging too much, too often

Symptom. Training slows down, the terminal fills with rate-limit warnings, or the end of the run hangs while uploads catch up.

Cause. Logging every batch of a very fast loop, logging large tensors or many images each step, or starting dozens of jobs at once that all create runs at the same moment.

Fix. Log scalars every N steps, images once per epoch, and keep each value under 1 MB. The documentation suggests staying under about 1,000 log calls per minute. Batch metrics into one call, and stagger job starts on a cluster.

Mistake 9. One-off metric names

Symptom. The workspace takes many seconds to load and has hundreds of tiny charts.

Cause. Names that contain a changing part, such as loss_epoch_7 or layer_12_grad_norm for every layer and every epoch, create a new metric per value. The number of distinct names in a project is called its cardinality, and it is the most common reason a workspace is slow.

Fix. Use a stable name and let the step carry the changing part. Log train/loss at every epoch, not loss_epoch_7 once. For per-layer values, log a histogram or a table rather than a metric per layer.

Mistake 10. Forgetting that images need pixel values from 0 to 255

Symptom. An image panel shows a black square, even though the array looks fine when plotted with another library.

Cause. The array holds floats between 0 and 1. Since 0.29.0, wandb.Image does not rescale, and the old normalize argument was removed.

Fix. Convert first: arr = (arr * 255).astype("uint8"). Use a NumPy or torch tensor on the CPU, detached from the graph, and check the shape matches what wandb.Image expects.

Mistake 11. Assuming a failed upload means a failed run

Symptom. A scary Timed out or 429 message appears and you kill the training.

Cause. The uploader is a separate process. Network trouble delays the dashboard, not your loop, and metric uploads are retried with backoff.

Fix. Let the training finish. The data is on disk in the run folder. Afterwards, wandb sync wandb/run-<timestamp>-<id> uploads it. If the network is the permanent problem, set WANDB_MODE=offline from the start.

Mistake 12. Copying tutorials written for another SDK version

Symptom. AttributeError, TypeError or a deprecation message on code from a blog post.

Cause. The SDK is 0.x. Breaking changes happen when the minor version increases, after a deprecation warning. Since 0.24 the wandb.beta.workflows helpers are gone, in 0.27.1 the Keras WandbCallback was removed, in 0.29.0 several run methods became properties, and in 0.30.0 wandb.PublicApi was removed.

Fix. Check the date and SDK version of the tutorial, read the current reference, and keep the library pinned in requirements.txt so an upgrade is a deliberate act.

Habits worth adopting on day one

Name things on purpose. Pass project, and give runs a name that describes the idea, such as baseline or bigger-batch. Random two-word names are fine for sweeps and poor for experiments you will refer to in a meeting.

Tag runs, and write notes. A tag like baseline or augmented lets you filter the table in one click. A line of notes such as "first run after fixing the label bug" is the difference between a useful history and a mystery six weeks later.

Log the configuration completely. Put everything that could affect the result into config: learning rate, batch size, model name, optimizer, dataset version, random seed, and the preprocessing choices. If a teammate cannot rebuild the experiment from the config and the git commit, the config is incomplete.

Seed your randomness, and record the seed. Two runs with different results are only comparable if you know whether the settings or the luck changed them. Put seed into the config and use it.

Use slash-prefixed metric names. train/loss, val/loss, val/acc. The dashboard groups them into sections, and they stay tidy as the project grows. Avoid spaces, commas and hyphens.

Use the context manager. It costs one line and buys correct run states.

Version data and models as artifacts from the first project. It is easy to start and painful to retrofit. Even a small CSV benefits from having a v0, because the day it changes you will want the old one.

Work under a team entity when others will use the results. If the work will ever be shared or promoted, log it under the team from the start, since personal-entity artifacts cannot be linked to the Registry.

Keep credentials out of code. wandb login locally, and WANDB_API_KEY from a secret store in automation.

Use disabled mode in tests. WANDB_MODE=disabled means unit tests run fast, need no network, and leave no clutter on the dashboard.

Ignore the local folders in git. wandb/ and artifacts/ belong in .gitignore.

Look at the System tab once per project. A GPU that idles at 20 percent tells you the data loader is the bottleneck. You will spend hours tuning a model that is not the slow part, unless you look.

Compare before you conclude. Select two or more runs in the table, overlay their charts, and check that the configs differ only in the setting you think matters. A difference in the random seed can look like a difference in method.

Write a short report for results that matter. A report with two charts and three sentences of interpretation is what you will share with your team, and it stays linked to the runs.

Mind where data lives. If your employer is in the Gulf or Egypt and has data-residency rules, ask before logging real customer data, images of people or text from production. The hosted service is run by a third party, and a reference artifact (which stores only a pointer to your own bucket) or a Dedicated Cloud in a chosen region may be the required answer. Log synthetic or anonymised data while you are learning.

Practice cards

Card 1: the smoke test. Run the five-line script from the detailed track, open the printed link, and find the run's ID, name, state and config. Goal: prove that login and upload work.

Card 2: two learning rates. Train the same toy model with learning rates 0.1 and 0.01. Give the runs good names. In the project page, overlay the two loss curves and write one sentence about what you see.

Card 3: the best epoch. Log val/loss each epoch, add define_metric("val/loss", summary="min"), and sort the table. Confirm that the summary differs from the last value when the model overfits.

Card 4: break it on purpose. Log after the with block, change a config key twice, and run with no credentials in a fresh shell. Read each message and match it to the table above.

Card 5: offline round trip. Set WANDB_MODE=offline, run a script, list the offline-run-... folder, and upload it with wandb sync. Confirm the run appears.

Card 6: a dataset artifact. Log a small CSV as an artifact, log it again unchanged, then change a row and log it a third time. Note which uploads created a new version.

Card 7: lineage. Write a training script that uses the dataset artifact and logs a model artifact with the alias best. Open the lineage graph and describe it aloud.

Card 8: a tiny sweep. Write a sweep.yaml with method: random, one metric, and two parameters. Run an agent with --count 6 and read the parallel coordinates chart.

A short checklist

Before you start a run:

• I am in a virtual environment and wandb --version shows 0.30.0 or newer.
• wandb login --verify succeeds, and the key is not in any file in the repository.
• The project has a deliberate name, and I know which entity the run will land in.
• The config holds every setting that could change the result, including the seed.
• My hyperparameters are read from run.config.
• wandb/ and artifacts/ are in .gitignore.

While writing the logging code:

• wandb.init is inside a with block, and all logging is inside it.
• Every step logs one dictionary, with slash-prefixed names.
• Metrics that change over time are metrics, not config.
• define_metric sets the summary for each metric where the last value is not the best.
• Images are scaled to 0 to 255.
• Values are small, and the names are stable.

After the run:

• The state is Finished, and I have checked the System tab for a starved GPU.
• Data and models that matter are logged as artifacts with aliases.
• I compared against at least one other run and noted what actually differed.
• Any failed or offline runs were recovered with wandb sync.
• I wrote a line of notes or a tag so that future me understands the run.

Worked through honestly, these cards and checklists take an afternoon, and they prevent nearly every problem in the error table. When they feel routine, open the Mid-level track, which starts where this one stops.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. The version it was checked against is the Python SDK 0.30.0 (released 2026-09-09) and W&B Server v0.85.0.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a real project in a few minutes. One housekeeping note first. The documentation has moved: docs.wandb.ai now redirects to docs.coreweave.com/products/wandb/, and the hosted multi-tenant service is being rebranded as CoreWeave Forge. The SDK still defaults to https://api.wandb.ai, so on the hosted service you change nothing. Only on Dedicated Cloud or Self-Managed do you set WANDB_BASE_URL or pass --host.

What wandb.init() actually does

At beginner level wandb.init() is a line you paste at the top of a script. At work you will debug it, and that needs a picture of what it starts. The call does three things in order: it resolves settings, it launches a helper process called wandb-core, and it asks the server to create a run.

wandb-core is a Go binary shipped inside the Python wheel. Your Python process never talks to the network for logging. It hands messages to the core service, and the core does the uploading, the streaming of history and console output, the file uploads through pre-signed URLs to object storage, and the sampling of system metrics. The practical consequences are worth memorising:

• Logging is asynchronous. run.log() returns as soon as the message is queued. Your training loop does not wait for the network, which is why W&B adds little overhead, and also why a run can finish its Python code long before finish() returns. The remaining time is the core draining its queue.
• If the core crashes, training continues. The Python side keeps running and you lose live sync, not your job.
• Everything is written to a local transaction log first. That is the file run-<id>.wandb in the run directory, and it is the reason offline mode and crash recovery exist.

The run directory is created under ./wandb/ (or wherever dir= or WANDB_DIR points) and named run-YYYYMMDD_HHMMSS-<RUN_ID>, or offline-run-... in offline mode. Open it once and learn its layout:

wandb/run-20260930_101500-a1b2c3d4/
├── run-a1b2c3d4.wandb        # the transaction log: the source of truth for sync and recovery
├── files/
│   ├── output.log            # captured stdout and stderr
│   ├── config.yaml           # the run's config
│   ├── wandb-summary.json    # final summary values
│   └── requirements.txt      # installed packages
└── logs/
    ├── debug.log             # SDK side
    └── debug-internal.log    # core side

When a run misbehaves, debug.log and debug-internal.log are the first files to read, and they are also visible in the run's Files tab once uploaded. System metrics are sampled every 15 seconds by default, covering CPU, memory, disk and network, NVIDIA GPUs through NVML, AMD and Apple Silicon, and TPUs. Inside containers, CPU and memory percentages use cgroup v2 limits since 0.27.0, so a pod limited to 4 GiB now reports against 4 GiB rather than the host's memory.

The four modes change where that data goes:

There is one subtle trap with disabled. Since SDK 0.19.2, wandb.init(mode="disabled") only affects that one run. To turn W&B off for everything in the process, use wandb.setup(settings=wandb.Settings(mode="disabled")), or set WANDB_MODE=disabled in the environment.

Settings themselves resolve from several layers, and knowing the order explains most "why did it log to the wrong project" questions. Settings come from code arguments, environment variables, the workspace file ./wandb/settings (written by wandb init) and the system file ~/.config/wandb/settings, with the more specific source generally winning. wandb status prints what the SDK sees. Since 0.24.1 there is a sharp edge: one invalid key in any settings file makes the SDK ignore all settings files and print an error. If your entity and project suddenly stop being applied, this is a likely cause. Fix the file or reset it with wandb init --reset.

Steps, commits and the history model

The beginner rule was "call run.log with a dict". The precise rule is that each call commits one step unless you say otherwise. The history is a time series of dicts, one row per step, and the step counter is what the x-axis uses by default. Two behaviours follow from that.

First, you can only write to the current or the next step, never the past. Second, several metrics logged in separate calls land on separate steps, which makes line charts look staggered:

# Bad: three steps per training iteration
run.log({"train/loss": loss})
run.log({"train/acc": acc})
run.log({"lr": lr})

# Good: one step, one call, one request's worth of overhead
run.log({"train/loss": loss, "train/acc": acc, "lr": lr})

Batching keys into one call is also the simplest defence against rate limits, because the soft guidance is fewer than 1,000 run.log calls per minute per project. If you must compose a step over several calls, pass commit=False on all but the last, and the step is finalised when a call without commit=False arrives.

For anything other than training steps, do not fight the default x-axis. Declare your own with define_metric:

with wandb.init(project="vision", config=cfg) as run:
    run.define_metric("epoch")
    run.define_metric("val/*", step_metric="epoch")
    run.define_metric("val/loss", summary="min")
    run.define_metric("val/acc", summary="max")
    run.define_metric("debug/*", hidden=True)

    for epoch in range(cfg["epochs"]):
        train_one_epoch()
        run.log({"epoch": epoch, "val/loss": vl, "val/acc": va})

The step_metric argument accepts globs, so one line re-keys every validation panel to the epoch axis. The summary argument decides what the run's summary holds for that key. The default summary is the last logged value, which is exactly wrong for a loss that bottoms out and rises again. The accepted aggregations include "min", "max", "mean" and "last". Two older forms are gone: summary="best" and the goal= argument were removed in 0.29.0, and summary="copy" is deprecated and planned for removal. If an old script raises on those, the replacement is summary="min" or summary="max".

Config deserves the same precision. run.config is a dict-like object holding the inputs to the run, and the docs ask you to keep keys free of dots and the total under 10 MB. By default a value that is already set cannot be changed: you get ConfigError: Attempted to change value of key "lr" from 0.01 to 0.02. The default of allow_val_change is False in scripts and True in notebooks, which is why the same code can pass in a notebook and fail in a pipeline. The fix is not always to pass allow_val_change=True. If a value changes over time it is not configuration, it is a metric, and it belongs in run.log. Use run.config.update(d, allow_val_change=True) only for genuine late corrections, such as a batch size that the framework adjusted after a memory probe.

Also pass a path, not just a dict, when you have a YAML file. wandb.init(config="config.yaml") loads it, and the loaded values are what a sweep later overrides.

A word about the lifecycle. Always use the context manager, with wandb.init(...) as run:. It calls run.finish() for you, and when an exception escapes the block the run is marked Failed instead of Crashed. That difference matters operationally: Failed means your code raised, Crashed means the heartbeats stopped, and only the second requires investigating the machine. Use run.log rather than module-level wandb.log. The module-level functions act on the single global run and stop working once you use reinit="create_new".

Metric names, cardinality and workspace performance

The most common complaint from a team that has used W&B for six months is that the project page is slow. The cause is almost never the number of runs. It is metric cardinality: the number of distinct metric keys across the project, where nested dicts are flattened into dot-joined names. A loop that logs f"layer_{i}/grad_norm" for 200 layers across 50 experiments with different layer counts creates thousands of keys, and each one is a column the workspace may try to draw.

The docs publish soft limits for the multi-tenant service. Performance degrades beyond them rather than failing:

Those numbers translate into habits. Log scalars to W&B and large objects to artifacts. Do not log one image per step for a 500,000-step run. Keep a small, stable vocabulary of metric names. On naming: the docs' strict pattern is letters, digits and underscores because names pass through GraphQL, but slash-prefixed names such as train/loss are used throughout the official examples and they create collapsible sections in the workspace. Use / for grouping and avoid spaces, commas and hyphens.

When a workspace is slow, there are three fixes in increasing order of effort. Switch the workspace from automatic to manual so W&B stops creating a panel per key, then add the panels you need with Quick add. Archive old runs into another project so the active project stays under the run guidance. And fix the logging code so that one-off keys stop being created. If you need to report a slow page to support, appending &PERF_LOGGING to the URL produces the extra timing they ask for.

Resuming, forking and recovering runs

Training jobs get preempted, nodes get drained, and someone will press Ctrl-C on the wrong terminal. The question is what W&B does when the same job starts again. The answer is controlled by resume and an explicit id, and the table is worth learning by heart:

For a preemptible job the recommended pattern is resume="allow" with an ID you control, because the same command works on the first start and on every restart:

import os, wandb

run_id = os.environ.get("RUN_ID", "resnet50-2026-09-30-a")   # stable across restarts

with wandb.init(project="vision", entity="my-team", id=run_id, resume="allow") as run:
    start_epoch = load_checkpoint_if_any() or 0
    for epoch in range(start_epoch, run.config.get("epochs", 30)):
        ...
        run.log({"epoch": epoch, "val/loss": vl})

Note that resuming the W&B run does not resume your model. You still need a checkpoint, and it is common to store it as an artifact so the restart can fetch it (covered below). The environment-variable equivalents are WANDB_RESUME and WANDB_RUN_ID, and the ID is limited to 64 characters in that variable. Boolean forms of resume are deprecated, and in offline mode resume is ignored with a warning.

The error messages name the mistake directly. The value 'must' is not a valid option for resuming the run (<id>) that has not been initialized means you asked to resume something that does not exist, so switch to "allow". The value 'never' is not a valid option for resuming a run (<id>) that already exists means the reverse.

Forking starts a new run whose history begins at a step of an existing one, using fork_from="RUNID?_step=200". It is the tool for "branch off at step 200 and try a lower learning rate" without copying the first 200 steps. The fork needs a different ID from the source, requires monotonic steps, and it is in preview on Multi-tenant and Dedicated Cloud and not available on Self-Managed. resume_from (rewind) uses the same format but the docs recommend forking because of rewind's performance limits. The three options fork_from, resume and resume_from are mutually exclusive, and passing two raises a ValueError saying so.

Several runs in one process is a third pattern. The reinit argument controls what a second wandb.init() does. The values are "default", "return_previous", "finish_previous" and "create_new", and booleans are deprecated. The "default" behaviour is "finish_previous" in notebooks and "return_previous" elsewhere, which is the reverse of what one documentation page states. Do not depend on it. If you want concurrent runs, say reinit="create_new" and keep each returned run object; if you want sequential runs, use the context manager and let each one finish.

Finally, recovery. If a run shows Crashed but the process was actually fine, or if you ran offline, the transaction log still has your data:

# Preview what would be uploaded, then upload everything unsynced under ./wandb
wandb sync --dry-run
wandb sync --yes

# Upload one specific run directory
wandb sync wandb/offline-run-20260930_101500-a1b2c3d4

# After syncing, reclaim local disk for synced runs
wandb clean

If you have old scripts, note that wandb sync was cleaned up in 0.30.0. --sync-all now errors (use plain wandb sync, or --yes in scripts), --job_type became --job-type, and the --clean family moved to the separate wandb clean command. The options --sync-tensorboard, --include-globs, --exclude-globs, --include-offline, --mark-synced, --skip-console and --legacy raise errors. For TensorBoard logs, use wandb.init(sync_tensorboard=True) in the training script instead.

Artifacts in depth

An artifact is a versioned, content-addressed bundle of files or references, used as the input or output of a run. The value at mid level is not storing files; it is the lineage graph that use_artifact and log_artifact build. Six months from now, "which dataset trained this model" is an answer the graph gives you, and a folder of files does not.

Versions are numbered from zero (v0, v1), and aliases are mutable pointers to versions. latest is maintained automatically. You add your own, such as best or production. Because the artifact is content-addressed, logging identical content does not create a new version. Since 0.25.1 the SDK prints a warning when that happens, which saves you from wondering why v3 never appeared.

import datetime, wandb

# Producer: a data-prep job
with wandb.init(project="vision", entity="my-team", job_type="prepare-data") as run:
    ds = wandb.Artifact("mnist-clean", type="dataset",
                        metadata={"rows": 60000, "source": "s3-export-2026-09"})
    ds.add_dir("data/clean")
    run.log_artifact(ds, aliases=["candidate"])

# Consumer: a training job
with wandb.init(project="vision", entity="my-team", job_type="train") as run:
    data = run.use_artifact("my-team/vision/mnist-clean:candidate")
    data_dir = data.download()                 # cached; root= chooses the folder
    model = train(data_dir)

    mdl = wandb.Artifact("mnist-cnn", type="model", metadata={"val_acc": 0.981})
    mdl.add_file("model.pt")
    mdl.ttl = datetime.timedelta(days=30)      # expire if unreferenced by policy
    run.log_artifact(mdl, aliases=["best"])

Use the job_type argument deliberately. It is what makes the lineage view readable: a graph whose runs are all labelled train tells you nothing, while prepare-data, train, evaluate and export read like a pipeline. Pass the entity explicitly too. Artifacts logged to a personal entity cannot be linked to a Registry, which is the single most common reason a promotion step fails.

Reference artifacts record a pointer, not the bytes. add_reference("s3://bucket/prefix") (and gs://, file://) stores checksums and locations, so a multi-terabyte dataset that already lives in your bucket is versioned without being copied. The trade-off is that W&B cannot protect the data from being modified or deleted in place, so the lineage is only as trustworthy as your bucket policy. For Gulf and Egyptian teams with data-residency rules, this is also the tool that lets the data stay in a regional bucket while W&B holds only metadata.

Storage control is three levers. A TTL deletes an artifact version after a period, and it applies to user-generated artifacts; team admins decide who may set it, and a team default TTL exists. Deleting versions needs garbage collection on the server, and Self-Managed installs must enable it with GORILLA_ARTIFACT_GC_ENABLED. References avoid the copy entirely. One trap: linking a version into a Registry sets its TTL to None, and unlinking does not restore it, so a version you linked once is effectively permanent until someone removes it.

Downloads are cached. The default cache is ~/.cache/wandb (set WANDB_CACHE_DIR), downloaded files go to ./artifacts (set WANDB_ARTIFACT_DIR), and staging uses the platform data directory (set WANDB_DATA_DIR). On CI runners with small disks, point all three at a volume you control and clean up with wandb artifact cache cleanup or the newer wandb purge-cache (added in 0.25.1). The CLI also has wandb artifact put, wandb artifact get and wandb artifact ls, which are useful in shell-based pipelines that do not want a Python step.

A final detail that matters for trust. Artifact digests use MD5 by default. SDK 0.29.0 added digest_algorithm="XXH128" as an option, and BREAKING.md says XXH128 will become the default a few releases later. If you compare digests across systems, pin the algorithm explicitly now so the upgrade does not change them under you.

The Registry: from experiment to something people rely on

An artifact in a project is a working file. An artifact in the Registry is a statement: "this version is the one others should use." Every organization starts with a Models and a Datasets registry, and each registry is divided into collections, where a collection represents one task or use case (for example mnist-classifier).

Linking creates a pointer, not a copy. The linked version keeps its identity and lineage, and the registry shows it under a path of the form wandb-registry-{REGISTRY}/{COLLECTION}, with the full name of a version written wandb-registry-{REGISTRY}/{COLLECTION}:v{N}. The two ways to link from code:

with wandb.init(project="vision", entity="my-team", job_type="train") as run:
    art = wandb.Artifact("mnist-cnn", type="model")
    art.add_file("model.pt")
    run.log_artifact(art, aliases=["best"])
    run.link_artifact(art, target_path="wandb-registry-model/mnist-classifier")

# Later, from a deploy job with no training run involved
api = wandb.Api()
model = api.artifact("wandb-registry-model/mnist-classifier:production")
model.download(root="./serving-model")

Promotion by alias is the pattern worth adopting. The consumer never hard-codes v17. It asks for production, and a human or an automation moves the alias when a model is approved. That separates "who trains" from "who deploys", and it gives you one place to point an audit at. Registry visibility is set per registry, and it is on by default on Self-Managed Server v0.71.0 and later, while on Dedicated Cloud your account team must enable it.

The Registry is also where automations attach. An automation is an event plus an action: when the alias production is added to a version in a collection, post to Slack or call a webhook that starts your deployment pipeline. A webhook automation can reference team secrets written as ${SECRET_NAME}, which are stored server side, cannot be revealed after creation, and are created only by admins. The clean separation is that W&B emits the event, and your CI (see GitHub Actions) performs the deployment with its own credentials. Keep the webhook payload minimal: the artifact name, version and alias are enough for the receiver to call wandb.Api() and fetch the rest.

If your team already uses MLflow for its model registry, the two can coexist, but avoid keeping two sources of truth for "what is in production". Pick one registry and have the other system read from it.

Sweeps: hyperparameter search as a service

A sweep has a controller that decides the next set of hyperparameters and agents that pull a config and run your program with it. By default the controller lives on the W&B server, so agents on many machines all draw from one queue. That is why scaling a sweep is nothing more than starting more agents.

The configuration is a YAML file or a Python dict. The required keys are method, parameters, and program (unless command overrides it):

program: train.py
method: bayes
metric:
  name: val/loss
  goal: minimize
parameters:
  lr:
    distribution: log_uniform_values
    min: 0.00001
    max: 0.1
  batch_size:
    values: [32, 64, 128]
  optimizer:
    value: adam
early_terminate:
  type: hyperband
  min_iter: 3
  eta: 3
run_cap: 50

Three points here prevent the usual mistakes. First, the metric.name must be a key that your script actually logs, spelled identically, or the Bayesian method has nothing to optimise. Second, log_uniform takes exponents as bounds while log_uniform_values takes the values themselves, so min: 0.00001 belongs with the second. The docs list many more distributions (int_uniform, q_uniform, normal, log_normal and their variants), and a continuous parameter under grid runs forever, so grid search only makes sense over lists of values. Third, random and bayes run until you stop them, so set run_cap or stop the sweep deliberately.

Your script must read its hyperparameters from run.config, not from its own argparse defaults:

def train():
    with wandb.init() as run:          # config is injected by the agent
        cfg = run.config
        model = build(cfg.lr, cfg.batch_size, cfg.optimizer)
        for epoch in range(10):
            run.log({"epoch": epoch, "val/loss": evaluate(model)})

Launching is three commands, and the sweep ID printed by the first is what the second uses:

wandb sweep --project vision sweep.yaml
# prints: wandb agent my-team/vision/abc123xy
wandb agent --count 20 my-team/vision/abc123xy    # run on each GPU machine
wandb sweep --pause my-team/vision/abc123xy       # also --resume, --stop, --cancel

The four lifecycle controls differ: pause stops handing out configs and resume restarts that, stop lets active runs finish and hand out nothing more, and cancel kills the active runs. wandb sweep --update only works while the sweep is pending, before any run has started, so you cannot retune the search space midway. The command key can reshape how the program is called, with macros such as ${env}, ${interpreter}, ${program} and ${args}. The variants ${args_no_hyphens}, ${args_json} and ${args_json_file} are for programs that do not use --key=value style arguments. The Python equivalent is wandb.sweep(sweep=cfg, project="vision") followed by wandb.agent(sweep_id, function=train, count=10).

Early termination with Hyperband stops unpromising runs at defined iterations. min_iter sets the first checkpoint and eta (default 3) the reduction factor. It needs your script to log the metric repeatedly (per epoch, say), because a script that logs the metric once at the end gives Hyperband nothing to compare. On preemptible hardware, runs that are killed by the scheduler can be marked with run.mark_preempting() in a signal handler; the run then becomes PREEMPTED and is re-queued, and the next agent must call wandb.init() within 60 minutes. Do not call it unconditionally, and remember that a SIGKILL simply ends in Crashed after about five minutes.

The public API: querying what you logged

You will want numbers out of W&B eventually: a table for a report, a check in CI that a candidate beat the baseline, a cleanup script. That is wandb.Api(). It authenticates the same way as the SDK, and since 0.27.1 its GraphQL calls go through wandb-core, so failures raise WandbApiFailedError rather than requests exceptions. In 0.30.0 the older aliases wandb.PublicApi and wandb.InternalApi were removed, so wandb.Api() is the only spelling.

import wandb

api = wandb.Api()

# Filter with MongoDB-style operators; lazy and paged 50 at a time
runs = api.runs("my-team/vision",
                filters={"config.lr": {"$gt": 1e-3}, "state": "finished"},
                order="-created_at", per_page=50)
print(len(runs))                     # not runs.length, which was removed in 0.29.0

best = min(runs, key=lambda r: r.summary.get("val/loss", float("inf")))
print(best.name, best.summary["val/loss"], best.url)

# Sampled history (default 500 points): fast, good for plots
df = best.history(keys=["val/loss"])

# Full, unsampled history: use for exact numbers
for row in best.scan_history(keys=["val/loss"]):
    pass

The distinction between history() and scan_history() is where people lose afternoons. history() returns a sample, 500 points by default, so a min computed from it can miss the true minimum. scan_history() iterates every logged row and, since 0.27.1, reads parquet exports when they exist, which makes it faster. Use the first for charts and the second for any number you will quote.

Two habits keep API scripts healthy. Respect the rate limits: on the multi-tenant service wait at least a second between bulk requests and prefer one filtered api.runs call over many api.run calls. And do not assume attributes that were removed in 0.29.0 still exist: use the properties run.project, run.url, run.project_url and run.sweep_url instead of get_url()-style methods. Newer additions you may find handy are run.console_logs(last=100) (reading from the beginning needs Server 0.77 or newer) and api.run("entity/project/id").stop().

A CI gate is a good first use. It downloads the candidate's metric and the current production model's metric and fails the job if the candidate is worse:

import sys, wandb

api = wandb.Api()
cand = api.artifact("wandb-registry-model/mnist-classifier:candidate")
prod = api.artifact("wandb-registry-model/mnist-classifier:production")
c, p = cand.metadata["val_acc"], prod.metadata["val_acc"]
print(f"candidate={c:.4f} production={p:.4f}")
sys.exit(0 if c >= p else 1)

That relies on you writing val_acc into artifact metadata at training time, as in the earlier example. Metadata is cheap and queryable, so make it a convention.

Running W&B in CI, containers and Kubernetes

Everything so far assumed a human at a terminal. In CI, cron and cluster jobs there is no TTY, so the first error you meet is UsageError: No API key configured. Use wandb login to log in. The SDK looks for credentials in a fixed order: an identity token file (WANDB_IDENTITY_TOKEN_FILE), then an explicit key argument, then WANDB_API_KEY, then api_key in a settings file, then the netrc file, and only last an interactive prompt, and only when there is a TTY. In automation you want the environment variable or the token file, never the prompt. Setting both WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE is an error, so pick one.

To check without triggering a prompt, 0.30.0 added wandb.login(prompt=False). It returns False when there are no credentials, without prompting and without silently switching to offline mode. That is the right call for a preflight step:

import sys, wandb
if not wandb.login(prompt=False):
    sys.exit("W&B credentials are not configured for this job")

Use a service account, not a person's key. A service account is a non-human identity scoped to the organization or to a team. It has its own keys, does not consume a paid seat, and can attribute runs to a human through WANDB_USERNAME or WANDB_USER_EMAIL (the user must be a member of the team or the attribution is ignored). Prefer team-scoped accounts for least privilege. They must be added explicitly to restricted projects. The key is shown once at creation, so put it straight into your secret manager.

A Kubernetes training job then needs three environment variables and nothing else from you:

apiVersion: batch/v1
kind: Job
metadata:
  name: train-resnet
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: registry.example.com/vision/train:1.4.2
          env:
            - name: WANDB_API_KEY
              valueFrom:
                secretKeyRef: { name: wandb-credentials, key: api-key }
            - name: WANDB_ENTITY
              value: my-team
            - name: WANDB_PROJECT
              value: vision
            - name: WANDB_DIR
              value: /tmp/wandb
          resources:
            limits: { memory: 16Gi }

Notice WANDB_DIR=/tmp/wandb. In a container with a read-only root filesystem, the default ./wandb directory may not be writable, and the SDK's failure looks like a mysterious init error. Point it at a writable volume. The secret itself should come from your secret manager or from a sealed or external secret, as with any credential (see Kubernetes for the mechanics). Build the image with the SDK pinned (wandb==0.30.*) so a rebuild next month cannot cross a minor version that removes an API your code uses, and see Docker for layer-cache habits on the install step.

For network-restricted clusters, the SDK must reach api.wandb.ai (or your own host) for GraphQL and file streaming, and it uploads files by pre-signed URL directly to object storage, so your egress rules must allow both. Symptoms of a blocked egress are the init timeouts: Timed out initializing run: ... followed by the suggestion to raise init_timeout. The default is 90 seconds, and since 0.28.2 the message includes the underlying error, which tells you whether it was DNS, TLS or a proxy. The fix is connectivity. Raise the timeout only when the network is merely slow, and use mode="offline" with a later wandb sync when the cluster genuinely cannot reach the service during the job.

W&B Server does not support self-signed certificates or custom CAs on its endpoints, so TLS must terminate at your ingress or load balancer with a certificate that clients already trust. insecure_disable_ssl exists in the SDK but it is not a production fix.

Identity federation (preview, Enterprise) removes long-lived keys altogether. An organization admin registers your JWT issuer, your job obtains a short-lived token from its platform identity (a Kubernetes service account token or a CI OIDC token, for example), and you point WANDB_IDENTITY_TOKEN_FILE at the file. W&B checks that iss matches the registered issuer, sub matches the user's email or the configured subject for an external service account, aud is the organization name, and the token has not expired. The token file is re-read on each exchange since 0.28.2, so a rotated token works without restarting the job.

Distributed training and framework integrations

Multi-GPU training raises a question the single-process tutorials skip: which process owns the run? There are three valid answers, and the choice changes what you see in the UI.

1. Log from rank 0 only. One run, simple, and it hides per-rank behaviour. Initialise W&B only when the rank is zero and let other ranks skip every call.
2. One run per rank, grouped. Every rank calls wandb.init with the same group= and a job_type such as worker. The UI groups them, and you can see one slow or failing rank. You get N runs per experiment, which raises your run count and cardinality.
3. One shared run. Every node initialises with settings=wandb.Settings(mode="shared", x_label=f"rank_{r}", x_primary=(r == 0), x_update_finish_state=(r == 0)) and the same WANDB_RUN_ID. It is experimental, but it gives one run with per-node system metrics. Only the primary should finish the run.

import os, wandb

rank = int(os.environ.get("RANK", "0"))
run = wandb.init(
    project="vision", entity="my-team", group="resnet50-8gpu", job_type="worker",
    name=f"rank-{rank}",
) if rank == 0 or os.environ.get("LOG_ALL_RANKS") else None

Start with pattern one. Move to two when you need to diagnose stragglers. Treat three as something to pilot on a non-critical job.

For frameworks, use the official hooks rather than reimplementing them. PyTorch Lightning uses WandbLogger, Hugging Face Transformers uses report_to="wandb" in its training arguments, and plain PyTorch can call run.watch(model, log="gradients", log_freq=...) to record gradient and parameter histograms. Do not leave watch on at high frequency for a big model, since histograms are large values. For Keras, the old WandbCallback was removed in 0.27.1. Use WandbMetricsLogger, WandbModelCheckpoint and WandbEvalCallback from wandb.integration.keras. Several other legacy integrations (the stale OpenAI, Cohere and LangChain autologs, and the YOLOv8 callback) were also removed in 0.27.1, so an old tutorial that imports them will fail on current versions.

Orchestrators integrate through service accounts more than through special code. Airflow (Airflow), Prefect and Dagster tasks simply call wandb.init with credentials injected as environment variables. Kubeflow Pipelines v2 has a @wandb_log decorator since 0.26.0, described on the Kubeflow side in Kubeflow. The habit that matters in any orchestrator is to pass the orchestrator's run ID as group or in tags, so a W&B run can be traced back to the pipeline execution that produced it, and to put the W&B run URL (run.url) in the pipeline's output.

If you version large datasets with DVC, record the DVC revision in config or artifact metadata, and let W&B hold the experiment record. The two answer different questions, and the commit hash is the glue between them.

Alerts, automations and keeping people informed

Most teams check a dashboard only when something is already wrong. Two features invert that. Alerts are sent from code with run.alert(title=..., text=..., level=...), and they notify you on Slack or email when a condition you define is met, such as a loss that became NaN. Automations react to events in the platform: an artifact alias added, a collection tag, or (on the multi-tenant service) a run metric crossing a threshold. The scopes are project, registry, collection, team and organization, and at team and organization scope only artifact and collection events are offered. Run-metric automations are SaaS-only.

The useful alert for long jobs is the "run crashed after N minutes" team alert, because a Crashed state is otherwise silent. Combine it with a code-level alert for the failure you can detect yourself:

import math

if not math.isfinite(loss):
    run.alert(title="Loss is not finite",
              text=f"step {step}: loss={loss}. Stopping.")
    raise RuntimeError("non-finite loss")   # context manager marks the run Failed

Raising inside the with block is deliberate. The run becomes Failed with an exception visible in output.log, and the alert has already told a human. Without the raise, the job would continue to burn GPU hours on garbage.

For the Registry, the webhook automation is what connects W&B to deployment. Keep the receiver idempotent, since events can be retried, and verify the payload before acting on it. Webhook secrets live in the team secret store and are referenced as ${SECRET_NAME}; never paste a token directly into the payload template.

Debugging beyond the basics

When something goes wrong, go through the same short sequence every time, from the cheapest check to the most expensive.

1. Is W&B even logged in and pointed at the right place? Run wandb login --verify (verification is the default in 0.30.0) and wandb status. The first shows the credential source and default team. For a Dedicated or Self-Managed server, also run wandb verify --host https://wandb.example.com. It checks login, signed-URL upload, large payloads through your proxy, and requests to the base URL, and it exits with status 1 if a critical check fails. Do not run it against the hosted service.

2. Read the error text literally. The SDK's messages are specific, and the table below covers the ones you will meet at work:

3. A 429 is not always fatal. Filestream 429s are retried with backoff and do not crash training, though they can delay finish(). Run-creation 429s do block training, so a fleet of jobs that all start in the same second can hit them. Stagger starts, and cap end-of-run upload time on expensive GPU nodes with the finish_timeout setting. Its default of 0.0 means "wait until the upload finishes"; a positive number gives up, marks the run Crashed or Failed, and leaves the data on disk for wandb sync.

4. Why is a run Crashed? The heartbeats stopped. The typical triggers are an out-of-memory kill (Killed, or CUDA out of memory), an uncaught exception without the context manager, scheduler preemption, kill -9, or a long network loss. Read output.log and debug-internal.log from the Files tab. If the process kept running, recover with wandb sync wandb/run-<TIMESTAMP>-<ID>.

5. After an upgrade, expect removed APIs. The symptoms are an AttributeError on wandb.PublicApi, run.get_url(), run.project_name() or runs.length, a failure in define_metric(summary="best"), wandb.finish(quiet=True) or wandb.Image(normalize=...), and a wandb.Video(array) without format=. The changelog lists each replacement. Two behaviours changed silently in ways that matter for correctness: wandb.Image no longer normalises pixel values, so float arrays must already be in the 0 to 255 range or your images will look black, and wandb.Video now requires format= when built from a numpy array or an io object.

For the terminal-minded, wandb leet opens a terminal UI over local .wandb files, remote runs and (with wandb leet symon) live system metrics. It was wandb beta leet in 0.27.x and became a top-level command in 0.28.x. It is handy on a compute node that has no browser.

Upgrades, pinning and the support window

The SDK is still 0.x, and its policy in BREAKING.md is explicit: breaking changes land only when the minor version is bumped (0.29 to 0.30), and only after a deprecation warning has already shipped. Patch releases (0.30.1) do not break. That gives you a workable discipline.

• Pin to a minor line in every environment: wandb==0.30.*. Patch fixes arrive, breaking changes do not.
• Promote minor upgrades deliberately. Before you bump, read the "Notable Changes", "Removed" and "Deprecated" sections of the changelog, and run your test suite with deprecation warnings turned into errors (python -W error::DeprecationWarning) so you find what is about to break.
• Match the client to the server. Some features need a minimum server version, and since 0.28.1 the artifact APIs (Artifact.files(), Artifact.download(), Api().artifact()) fail against end-of-life servers because the legacy fallbacks were removed.
• Know the platform floor. The SDK needs Python 3.10 or newer, pydantic 2.6 or newer, protobuf 5 or newer but below 8, and macOS 13 or newer; there are no 32-bit Windows wheels. GitPython is no longer a dependency, since the SDK calls the git executable directly and still honours GIT_PYTHON_GIT_EXECUTABLE.

The server side has its own rhythm. Major versions ship monthly and each is supported for 12 months. On Dedicated Cloud upgrades happen for you. On Self-Managed you upgrade, and W&B recommends doing so at least once a quarter. The supported install is the W&B Kubernetes Operator, and the legacy single-pod Helm chart is deprecated and unsupported. If you run it, the requirements as of v0.85 include Kubernetes 1.34 or newer, MySQL 8.4 (8.0 reached end of life in April 2026), Redis 7, and ClickHouse only when you use Weave. If you are only a client of a hosted or managed server, your job is to know the version you are talking to and ask the platform team when it changes; if you run it, continue with the senior level of this guide.

Several deprecations have notices live now, and scripts written today should avoid them: boolean reinit, config_exclude_keys and config_include_keys, the anonymous setting, summary="copy", and wandb.api with wandb.ensure_configured(). Replacing them costs minutes now and avoids a forced migration later.

Putting it all together

Here is one small pipeline that uses most of this level. It is a data-prep job, a training job that resumes, a registry link, and a gate. Each piece is a script you already have the parts for, and the point is how they connect.

prepare_data.py   --logs-->  dataset artifact  mnist-clean:candidate
        |                               |
        v                               v (use_artifact)
train.py (resume="allow", id=$RUN_ID) --logs--> model artifact + val/* metrics
        |                               |
        v                               v (link_artifact)
gate.py  <--reads metadata--  wandb-registry-model/mnist-classifier:candidate
        |
        v  (only if better than :production)
move alias "production"  -->  webhook automation  -->  CI deploy job

Walk through it the way you would explain it in a design review. The data job runs with a service account, logs the dataset to a team entity with metadata that records the row count and the source export, and tags the version candidate. The training job is started by your orchestrator with a stable RUN_ID, calls use_artifact so the lineage records the dataset, logs validation metrics with define_metric so the summary shows the minimum loss, saves a checkpoint each epoch (as a file or artifact), and on success logs the model with val_acc in its metadata and links it into the registry. The gate script compares that metadata with the production version and exits non-zero if the candidate is worse. When a human or the gate moves the production alias, a webhook automation posts to your CI, which deploys the model by downloading :production through wandb.Api().

Notice what is not in the pipeline: nobody copies a file by hand, nobody asks which dataset was used, and the deployment job never contains a version number. Everything is keyed by alias and recorded in lineage. This is also where W&B sits beside the other tools in this catalogue: MLflow solves similar tracking and registry problems with a different hosting model, and Evidently picks up after deployment to monitor drift in the data your model sees.

Before you call it finished, run through a short review: credentials come from a secret manager and not from the image, WANDB_ENTITY and WANDB_PROJECT are set, the SDK is pinned to a minor line, metric names are stable, artifacts have metadata and a TTL policy, and the preflight check exists.

Where you are now

You can now predict what wandb.init() does and where its data lives, log steps without staggering, declare summaries and custom axes, keep a project's cardinality under control, and resume, fork and recover runs. You can version data and models with lineage, promote by alias through the Registry, run sweeps with sensible search spaces, query results with the right history call, and run all of it from CI and Kubernetes with service accounts instead of personal keys. You also know how the SDK's versioning policy protects you and how to read its error messages.

The next level treats W&B as a platform you operate for other teams: the server architecture, sizing and failure modes, SSO and SCIM, bring-your-own-bucket storage, audit logs, multi-tenancy, cost control and the governance questions that come with them.

Sources

• W&B quickstart
• Python SDK release notes
• Server release notes
• Release policies
• SDK CHANGELOG at v0.30.0
• SDK BREAKING.md at v0.30.0
• CLI reference
• wandb sync reference
• wandb sweep reference
• wandb agent reference
• wandb verify reference
• wandb.init reference
• Run reference
• Artifact reference
• Settings reference
• Configure experiments
• Log data
• Log distributed training
• Environment variables
• Limits and performance
• Public API guide
• Resuming runs
• Forking runs
• Run states
• Sweeps
• Sweep configuration keys
• Artifacts
• Artifact TTL
• Track external files
• Registry
• Link a version to a registry
• Automations
• Service accounts
• Identity federation
• Why is my run showing as crashed
• Rate limit exceeded on metric logging

This is the mid-level interview file. Interviewers at this level stop asking what a run is and start asking what you would do when something is slow, broken or ambiguous. Every answer below is written the way a strong candidate would say it aloud: the conclusion first, then the reasoning, then the caveat. The version assumed is the Python SDK 0.30.0.

The thirty-second answer

Weights & Biases is an experiment tracking and model lifecycle platform. A training script calls wandb.init(), which starts a helper process called wandb-core. The helper writes everything to a local transaction log and streams it to the server, so logging never blocks training and a crashed job can be recovered. On top of runs sit artifacts (versioned, content-addressed files with lineage), the Registry (curated, aliased model and dataset versions), sweeps (hyperparameter search) and automations (events to Slack or webhooks). At mid level the interesting questions are about behaviour: how steps work, what resumes, what scales and what fails.

How a strong answer is built

Interviewers mark mid-level answers on four things, and it helps to check yourself against them as you practise.

• You name the mechanism, not just the command. "Use resume="allow"" is a weak answer. "Use resume="allow" with a stable ID so the same command works on first start and on every restart, and remember it resumes the W&B run, not the model weights" is a strong one.
• You separate the fix from the prevention. The fix stops tonight's problem. The prevention is the pin, the preflight check or the naming convention that stops it recurring.
• You know the current version's traps. Several APIs were removed in 0.29 and 0.30. Saying so shows you work with the tool now, not from a two-year-old tutorial.
• You say what you would check first, and why that check is cheap.

Scenario questions

The project page takes a minute to load. What do you check?

I would not start with the number of runs. I would start with metric cardinality, the number of distinct metric keys in the project, because the docs name it as the most common cause of slow workspaces. Nested dicts are flattened into dot-joined names, so a loop that logs a key per layer or per class can create thousands of keys quickly. I would check a single run's summary for its key count and look for one-off keys with generated names.

Next I would check the soft limits: around 10,000 runs per project, 500,000 steps per run, fewer than 1,000 run.log calls per minute, and values under 1 MB with a whole log call under 25 MB. A team that logs an image every step, or logs a table every step, blows through the size limits first.

The fixes, in order of effort: switch the workspace from automatic to manual so it stops drawing a panel per key, archive old runs into another project, and then fix the logging code so the keys stop multiplying. If I need support to look at it, I append &PERF_LOGGING to the URL to get timing output.

What a weak answer misses: it blames the server or the network, or says "delete old runs" without finding the cause, so the project is slow again in a month.

A training job on Kubernetes fails immediately with "No API key configured". Why, and what do you change?

That error, UsageError: No API key configured. Use wandb login to log in, appears when there are no credentials and no TTY to prompt on. A pod has no TTY, so the prompt path is unavailable. The SDK looks for credentials in a fixed order: an identity token file, an explicit key, WANDB_API_KEY, a settings file, the netrc file, and only last a prompt. So the fix is to inject WANDB_API_KEY from a Kubernetes secret, ideally sourced from the organisation's secret manager, together with WANDB_ENTITY and WANDB_PROJECT.

I would also add a preflight check using wandb.login(prompt=False), which was added in 0.30.0 and returns False without prompting and without switching to offline mode. The job then fails in seconds with my message instead of reaching the first init.

The follow-up I expect is about whose key it is. The answer is a service account, not a developer's personal key: it does not consume a seat, it survives people leaving, and it can attribute runs to a human through WANDB_USERNAME when the user is a member of the team. Team-scoped service accounts give least privilege.

A run shows "Crashed" but the job finished fine. What happened and can you get the data?

"Crashed" means the heartbeats stopped, and it says nothing about your exit code. A job that raised an exception inside the with wandb.init() block would show Failed instead. So I look for what stopped the heartbeats: a long network loss, a node drained by the scheduler, an out-of-memory kill, or kill -9. The evidence is in output.log, debug.log and debug-internal.log in the run's Files tab and in the local run directory.

The data is recoverable because everything was written to the local transaction log first. If the process kept running, I run wandb sync wandb/run-<TIMESTAMP>-<ID> from the project directory, preceded by wandb sync --dry-run to see what would upload. Prevention is the context manager, which turns an uncaught exception into Failed, and handling preemption properly.

What a strong answer adds: old tutorials use wandb sync --sync-all and --clean. Both were removed in 0.30.0. Current usage is plain wandb sync (or --yes in scripts) and the separate wandb clean command.

Your spot instances get preempted. How do you make training and tracking survive that?

There are two layers, and candidates often only describe one. The tracking layer: give the run a stable id and call wandb.init(id=run_id, resume="allow"). With "allow", an existing ID resumes from its last step and a missing one creates a run, so the same command works on first start and every restart. The training layer: W&B resuming a run does not restore weights. The job must write checkpoints, and on restart load the latest one and continue from that epoch. Storing checkpoints as artifacts means the restarted job can fetch them by alias, and lineage shows which checkpoint each segment started from.

I would also mention resume="must" fails if the run does not exist, "never" fails if it does, and boolean values are deprecated. If the runs are part of a sweep, a signal handler can call run.mark_preempting() and exit non-zero, which puts the run in PREEMPTED and re-queues it, and the next agent must call wandb.init() within 60 minutes. I would warn against calling it unconditionally, and note that a SIGKILL just ends as Crashed after about five minutes.

A teammate says the best run's validation loss in the API does not match the UI. Where do you look?

I would look at how the number was retrieved. run.history() returns a sample, 500 points by default, so a minimum taken from it can miss the true minimum. run.scan_history() iterates every row, and since 0.27.1 it reads parquet exports when they are available. Any number you intend to quote should come from scan_history, or better, from a summary that was declared to hold the minimum.

The second possibility is the summary itself. The default summary for a key is the last logged value. If the loss bottomed out at epoch 12 and rose afterwards, the runs table shows the last value. run.define_metric("val/loss", summary="min") fixes that at the source. The removed form summary="best" will error on current versions.

After upgrading the SDK a nightly job fails with an AttributeError. Walk me through it.

First I identify what the job called, and I compare the SDK versions before and after. The SDK is 0.x, and its policy is that breaking changes arrive only when the minor version is bumped, after a deprecation warning has already shipped. So I read the "Removed" section for each minor version crossed. The usual suspects in the last few releases: wandb.PublicApi and wandb.InternalApi (removed in 0.30.0, use wandb.Api()), runs.length (use len(runs)), run.get_url() and run.project_name() (use the url and project properties), define_metric(summary="best"), the quiet argument of finish(), and the Keras WandbCallback (removed in 0.27.1).

Then the prevention: pin wandb==0.30.* in the image, bump minors deliberately, and run the test suite with python -W error::DeprecationWarning before the bump so the deprecations fail loudly while they are still harmless.

Images logged from a new version of the training code all look black. Why?

Since 0.29.0 wandb.Image no longer normalises pixel values, and the normalize argument was removed. Arrays and tensors must already be in the range 0 to 255. A float image in 0 to 1 is now nearly black. The fix is to scale and convert to uint8 before logging. A related change: wandb.Video built from a numpy array or an io object now requires format=. This is a good question for checking whether a candidate reads changelogs.

link_artifact does not offer the registry collection, or fails. Causes?

The most common cause is that the artifact was logged to a personal entity. Artifacts in a personal entity cannot be linked to a Registry, so production work should log to a team entity. The second is the target path: it must be in the form wandb-registry-{REGISTRY}/{COLLECTION}, and the easiest way to get it exactly right is to open an empty collection in the UI. The third is availability: Registry is on by default on Self-Managed Server v0.71.0 and later, but on Dedicated Cloud the account team must enable it.

A sweep is running but nothing improves. What are the usual causes?

Four things, checked in this order. The metric.name must be a key the script actually logs, with identical spelling, otherwise the search has no signal. The script must read hyperparameters from run.config, not from argparse defaults, otherwise every run is the same run. The distribution bounds must match the distribution: log_uniform takes exponents while log_uniform_values takes values, and confusing them gives a search over a range nobody intended. And for Hyperband early termination the script must log the metric repeatedly, because logging once at the end leaves nothing to compare. A fifth check for grid search: a continuous parameter under grid does not terminate.

Trade-off questions

Log from rank 0 only, one run per rank, or one shared run?

Rank 0 only is simplest, cheapest in run count and cardinality, and hides per-rank behaviour. One run per rank with a shared group and a job_type lets me see a straggler or a failing rank, at the cost of N runs per experiment. A single shared run (mode="shared" with x_label, x_primary and x_update_finish_state, plus the same WANDB_RUN_ID on every node) gives one run with per-node system metrics, but it is experimental. I default to rank 0, move to grouped runs when I am diagnosing distributed behaviour, and pilot shared mode on non-critical jobs.

Store data in artifacts, or use reference artifacts?

Regular artifacts copy the bytes into W&B-managed or bring-your-own storage, and give integrity and easy download. Reference artifacts (add_reference("s3://...")) record checksums and locations only. Use references when the data is large and already lives in a bucket you control, and when data-residency rules, common for Gulf and Egyptian employers, mean it should stay in a regional bucket. The cost is that W&B cannot prevent in-place modification or deletion, so the lineage is only as trustworthy as the bucket's policy.

Sampled history or full history for an analysis script?

Sampled is fast and right for plots. Full is slower and right for any number someone will quote or a gate will act on. I would put a comment in the script saying which one it is, because the mistake is invisible in the output.

W&B or MLflow for tracking?

I would not answer with a brand. I would ask who runs it and who needs to see it. W&B's strengths are the hosted collaboration experience, sweeps, rich panels and the Registry with automations; MLflow's are open-source self-hosting with no seat model and broad framework support. The MLflow guide covers it. A strong answer adds the real constraint: do not run two sources of truth for "what is in production", and pick the one your deployment pipeline can read from.

Alias promotion or hard-coded versions?

Aliases. A consumer that asks for production never changes when a new model ships, and the act of moving the alias is the auditable decision. Hard-coded versions scatter that decision across repositories. The one case for pinning a version is a reproducibility study, where I want the exact bytes whatever the alias says.

Long-lived API keys or identity federation?

Federation removes a secret that can leak: the job presents a short-lived JWT through WANDB_IDENTITY_TOKEN_FILE, and W&B checks the issuer, subject, audience and expiry. It is a preview feature for Enterprise plans, and it needs an admin to register the issuer. Where it is unavailable, use team-scoped service accounts, store keys in a secret manager, and rotate by creating a new key and deleting the old one, remembering that the key is shown only once.

Questions that test depth

What does wandb-core do, and why should I care?

It is a Go service process spawned by wandb.init() that handles uploads, file streaming and system monitoring. I care for three reasons. Logging is asynchronous, so finish() can take a long time after my code has finished because the queue is draining. If the core crashes, training continues, so a missing run is a tracking incident and not a training one. And the transaction log it writes is what makes wandb sync and offline mode work. Since 0.26 a detached shared core can be started with wandb beta core start and reached through WANDB_SERVICE, and it must be restarted after an SDK upgrade.

Why do my metrics look staggered on the chart?

Because each run.log call commits a step by default. Three calls per iteration make three steps. Combine keys in one call, or use commit=False on all but the last call. For non-step x-axes, declare them with define_metric(..., step_metric=...).

Why does the same code pass in a notebook and fail in a pipeline with a ConfigError?

Because allow_val_change defaults to True in notebooks and False in scripts. A config key that is set twice with different values raises in the script. The better fix is usually to log the changing value as a metric, since a value that changes over time is not configuration.

What is the difference between Failed, Crashed and Killed?

Failed: the process exited non-zero, for example an exception under the context manager. Crashed: heartbeats stopped without a clean exit. Killed: the run was force-stopped. Sweep runs can also be Pending when queued, and PREEMPTED when re-queued after mark_preempting.

Red flags in candidate answers

Interviewers also listen for answers that reveal a lack of production experience. Some patterns to avoid:

• "I just put the API key in the script." Any answer that puts a key in code, an image or a committed wandb/settings file fails immediately. The secret belongs in a secret manager.
• "We delete old runs when it gets slow." It treats the symptom and ignores cardinality.
• "Resume restores the model." It restores the tracking run only.
• "We always use the latest SDK." Without pinning, a minor bump can remove an API overnight.
• "We log everything, every step." It ignores the rate and size guidance and produces unreadable workspaces.
• "Set insecure_disable_ssl." It papers over a certificate problem. W&B Server does not support self-signed certificates or custom CAs, so terminate TLS properly at the ingress.

Practice prompts

Try answering these aloud in two minutes each, then compare against the sections above.

1. A colleague's run was resumed and the loss curve restarts from zero. What happened?
2. Design the tracking for a nightly retraining pipeline that promotes a model only if it beats production.
3. Your CI fails with HTTP 429 during wandb.init. What do you change?
4. You inherit a project with 400,000 distinct metric keys. What is your plan?
5. A sweep agent on a GPU node hangs at finish() for ten minutes after training ends. Why, and what setting caps it?

For the last one, the answer is that finish() waits for the upload queue to drain, and the finish_timeout setting bounds that wait. Its default of 0.0 waits until upload finishes. A positive value gives up, marks the run Crashed or Failed and leaves the data on disk for wandb sync.

Sixty-second self-test

Answer without looking back.

1. What does each run.log call do by default, and what is the consequence for three separate calls?
2. Which resume value works on both first start and restarts?
3. What is the difference between history() and scan_history()?
4. Why can an artifact not be linked to the Registry?
5. Which credential source wins: WANDB_API_KEY or the netrc file?
6. Name three APIs removed in 0.29.0 or 0.30.0.
7. What does Hyperband need from your script?
8. Why is a service account better than a personal key in CI?

If you can answer seven of the eight, you are ready for the tips file and then the senior level. For more tooling context, read the Kubernetes guide for the secrets and job patterns, and the Evidently guide for what happens to a model after the Registry hands it over.

This is the mid-level tips file. You can already log runs and use artifacts. What separates a comfortable team from a frustrated one is a handful of habits, and most of the pain at this level comes from the same dozen mistakes. Each entry below gives the symptom, the cause and the fix, and is checked against SDK 0.30.0.

Errors you will meet at this level

"No API key configured" in CI or a cluster job

Symptom. UsageError: No API key configured. Use wandb login to log in. appears the moment wandb.init() runs, in a pipeline, a cron job or a pod.

Cause. No credentials were found and there is no TTY to prompt on. The SDK checks the identity token file, then an explicit key, then WANDB_API_KEY, then settings files, then netrc, and prompts only at the end and only if it can.

Fix. Inject WANDB_API_KEY from your secret manager, and set WANDB_ENTITY and WANDB_PROJECT so runs land in the right place. Add a preflight check so the failure is fast and readable:

import sys, wandb

if not wandb.login(prompt=False):
    sys.exit("W&B credentials missing: set WANDB_API_KEY")

"API key must be 40 characters long, yours was 86"

Symptom. A fresh key is rejected on an older machine or image.

Cause. New keys are about 86 characters long, and SDKs older than 0.22.3 only accept 40. A stale base image is the usual culprit.

Fix. pip install -U wandb, then wandb login --relogin. Pin the new version in the image so the problem does not return.

Both WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE set

Symptom. AuthenticationError: Both WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE are set, which is not allowed.

Cause. A base image or a CI template set one, and a job added the other.

Fix. Unset one. If you are moving to identity federation, remove the key from the environment completely.

ConfigError when a value changes

Symptom. ConfigError: Attempted to change value of key "lr" from 0.01 to 0.02. It works in a notebook and fails as a script.

Cause. allow_val_change is False in scripts and True in notebooks.

Fix. Ask whether the value is really configuration. A value that changes over time is a metric, so log it. If it is a genuine late correction, use run.config.update({...}, allow_val_change=True).

Logging after the run is finished

Symptom. Run (<id>) is finished. The call to log will be ignored.

Cause. A callback or thread logs after the with block closed, or module-level wandb.log is used across a reinit.

Fix. Keep the run object and log through run.log. Close background threads before leaving the with block.

Staggered, sawtooth charts

Symptom. Two metrics that should align are offset, and the step count is a multiple of what you expect.

Cause. Each run.log call commits a step. Three calls per iteration are three steps.

Fix. One call with all keys, or commit=False on every call but the last.

Summary shows the last value instead of the best

Symptom. The runs table says val/loss is 0.41 while the chart's minimum was 0.22.

Cause. The default summary is the last logged value.

Fix. run.define_metric("val/loss", summary="min"). The old summary="best" and goal= forms were removed in 0.29.0.

Black images after an upgrade

Symptom. wandb.Image of a float array in 0 to 1 renders nearly black.

Cause. Since 0.29.0 pixel values are no longer normalised, and the normalize argument was removed.

Fix. Scale to 0 to 255 and cast to uint8 before logging. For video, pass format= when you build wandb.Video from an array.

AttributeError after bumping the SDK

Symptom. AttributeError on wandb.PublicApi, run.get_url(), runs.length or the Keras WandbCallback.

Cause. They were removed in 0.30.0, 0.29.0 and 0.27.1.

Fix. Use wandb.Api(), the run.url and run.project properties, len(runs), and WandbMetricsLogger with WandbModelCheckpoint from wandb.integration.keras.

Init timeout

Symptom. Timed out initializing run: ... followed by a suggestion to raise init_timeout.

Cause. The backend is unreachable or slow, often because egress is blocked by a firewall or proxy. The default is 90 seconds, and since 0.28.2 the message includes the underlying error.

Fix. Read the underlying error first. Repair DNS, TLS or proxy settings. Raise the timeout only if the network is truly slow. If the cluster cannot reach W&B during the job at all, use mode="offline" and run wandb sync afterwards.

429 rate limit

Symptom. Rate limit exceeded in the logs, slow finish(), or jobs that fail to start together.

Cause. Too many log calls or too much data per project. Filestream 429s are retried with backoff. Run-creation 429s block training.

Fix. Batch keys into one call, log less often, stagger the start of job fleets, and sync offline data later if needed.

Linking to the Registry fails

Symptom. The registry collection is not offered, or link_artifact raises.

Cause. The artifact was logged to a personal entity, or the target path is wrong.

Fix. Log with entity="<team>". Copy the target path from an empty collection in the UI. The form is wandb-registry-<registry>/<collection>.

The practices that pay off most, in order of leverage

1. Pin the SDK to a minor line

The SDK's policy is that breaking changes arrive only with a minor bump, after a deprecation warning. So wandb==0.30.* gives you patch fixes and protects you from removals. Promote a minor upgrade on purpose: read the changelog, run tests with python -W error::DeprecationWarning, then bump.

2. Use a service account in every automated job

Personal keys leave when people do. A team-scoped service account has its own key, does not consume a seat, can attribute runs to a real person with WANDB_USERNAME, and limits the blast radius of a leak. Keep the key in a secret manager. The full key is shown only once at creation, so store it immediately. Never commit wandb/settings or netrc files that contain a key.

3. Use the context manager and run.log

with wandb.init(...) as run: finishes the run for you, and turns exceptions into Failed instead of Crashed. That single difference saves hours of guessing whether a machine or your code is at fault.

4. Give every run a stable identity and every job a job_type

Set entity and project explicitly or by environment variable. Give pipeline jobs a job_type such as prepare-data, train and evaluate, and put the orchestrator's execution ID into group or tags. The lineage graph and the runs table become readable, and a run can always be traced back to the pipeline execution that produced it.

5. Keep the metric vocabulary small and stable

Use / for grouping, letters, digits and underscores for names, and avoid spaces, commas and hyphens. Never generate metric names from data values. If the project workspace slows down, cardinality is the first suspect.

6. Declare summaries and axes with define_metric

Five lines at the top of the script decide whether your runs table is useful. Declare summary="min" or "max" for the metrics you rank by, and step_metric for anything that is not per-step.

7. Attach metadata to artifacts, and promote by alias

Put the numbers a gate will need (val_acc, row count, source export) in artifact metadata. Consumers ask for production or candidate, never v17. A CI gate then needs only wandb.Api() and a comparison.

8. Make checkpoints resumable

A stable id with resume="allow", plus a checkpoint saved at least every epoch, makes preemption boring. Resuming the W&B run does not restore weights, so the checkpoint is the other half of the contract.

9. Put a TTL policy on anything you can regenerate

Intermediate checkpoints and scratch datasets accumulate quickly. A TTL deletes user-generated artifact versions after a period, and a team default is available. Remember that linking into a Registry clears the TTL, so link only what you mean to keep.

10. Use scan_history for numbers you quote

history() samples 500 points by default. Use it for plots. Use scan_history() for any figure that ends up in a report or a gate.

Practice cards

Good versus bad

Operational recipes

Recipe 1. Check that a machine is ready to log

wandb --version                 # expect 0.30.0 or a 0.30.x patch
wandb login --verify            # shows the credential source and default team
wandb status                    # shows which settings file each value comes from
python -c "import wandb; print(wandb.login(prompt=False))"

For a Dedicated or Self-Managed server, add wandb verify --host https://wandb.example.com. It checks login, signed-URL upload, large payloads through your proxy and requests to the base URL. Do not run it against the hosted service.

Recipe 2. A Kubernetes job that can log

Give the pod WANDB_API_KEY from a secret, plus WANDB_ENTITY, WANDB_PROJECT and a writable WANDB_DIR such as /tmp/wandb when the root filesystem is read-only. Pin the SDK in the image. Set memory limits, because an out-of-memory kill appears in W&B as a Crashed run and not as an error. The Kubernetes guide covers the secret and job mechanics, and the Docker guide covers keeping the install layer cached.

Recipe 3. Recover a crashed or offline run

cd /path/to/project
wandb sync --dry-run                        # what would be uploaded
wandb sync wandb/run-20260930_101500-a1b2c3d4
wandb clean                                 # remove local data for synced runs

If wandb sync complains about options, you are using pre-0.30 flags. --sync-all, --clean, --sync-tensorboard, --include-globs and several others were removed, and --job_type became --job-type.

Recipe 4. Run a sweep across machines

wandb sweep --project vision sweep.yaml      # prints the sweep ID
wandb agent --count 20 my-team/vision/<SWEEP_ID>      # run on every GPU node
wandb sweep --pause  my-team/vision/<SWEEP_ID>
wandb sweep --resume my-team/vision/<SWEEP_ID>
wandb sweep --stop   my-team/vision/<SWEEP_ID>        # active runs finish
wandb sweep --cancel my-team/vision/<SWEEP_ID>        # active runs are killed

Recipe 5. Reduce slow uploads at the end of training

If finish() holds an expensive GPU node for minutes after training has ended, bound it with a setting:

run = wandb.init(
    project="vision",
    settings=wandb.Settings(finish_timeout=120),   # give up after 120 s
)

The default of 0.0 waits until upload finishes. With a positive value the run is marked Crashed or Failed and the data stays on disk for a later wandb sync.

Recipe 6. Keep caches from filling a CI runner

Point WANDB_CACHE_DIR, WANDB_ARTIFACT_DIR and WANDB_DATA_DIR at a volume you control. Clean up with wandb artifact cache cleanup or wandb purge-cache (added in 0.25.1) at the end of the job. A dataset downloaded from an artifact is cached, so a runner that reuses its cache between jobs is fast and a runner with a small disk is not.

Recipe 7. Find out why a run crashed

Open the run's Files tab and read output.log, then debug.log and debug-internal.log. Look for Killed, CUDA out of memory, a traceback, or nothing at all, which points to a node loss. If the machine still has the run directory, wandb sync it. For terminal access on a compute node, wandb leet opens a terminal UI over local .wandb files, and wandb leet symon shows live system metrics.

Recipe 8. Promote a model with a webhook

Create an automation on the registry collection with the event "alias added" for production. The action is a webhook to your CI. Keep the token in a team secret and reference it as ${SECRET_NAME} in the payload template. The receiver should be idempotent, and it should download the model with wandb.Api().artifact("wandb-registry-model/<collection>:production"). The GitHub Actions guide shows how a workflow can be triggered from outside.

Why these habits work

None of the practices above is clever. Each one removes a decision that a tired engineer would otherwise make at the wrong moment: which key to use, which version is live, whether a run is resumable, what a metric is called. The pattern is to push the decision into something that can be reviewed, such as a pinned requirements file, a service account, a preflight script or an alias, and then let the tool enforce it. When you review a colleague's training code, ask which of these decisions are still being made by hand, and move the most expensive one into configuration first.

A short weekly routine

Fifteen minutes a week prevents most platform pain. Check the changelog for the SDK and read the deprecation notices that affect you. Skim the organisation's storage usage and delete or expire what you can regenerate. Open your busiest project and look at the workspace load time, because a slowdown is easier to fix when it begins. Confirm that a service-account key is not older than your rotation policy. And re-run your preflight script from a clean container to prove the whole path still works.

Checklist

• SDK pinned to wandb==0.30.* in every image and environment
• Automated jobs use a team-scoped service account, with the key in a secret manager
• Preflight check with wandb.login(prompt=False) runs first
• WANDB_ENTITY and WANDB_PROJECT are set explicitly
• with wandb.init(...) as run: and run.log everywhere
• One run.log call per step, with keys combined
• define_metric declares summaries and custom axes
• Metric names are stable, use / for grouping, and are never generated from data
• Resumable jobs use a stable id, resume="allow" and real checkpoints
• Artifacts carry metadata, and a TTL policy covers regenerable ones
• Consumers use aliases such as production, never version numbers
• Artifacts are logged to a team entity so they can be linked to the Registry
• Numbers used in reports or gates come from scan_history or declared summaries
• Tests run with deprecation warnings treated as errors before each minor upgrade
• Workspaces for busy projects are in manual mode
• Nobody has committed a wandb/settings file or a netrc file with a key

This is part three of three. It takes every topic from Beginner and Mid one level further, and adds the work you own when Weights & Biases is a platform that dozens of people depend on rather than a library you call from one script. This guide is written against SDK 0.30.0 (released 2026-09-09) and W&B Server v0.85.0 (2026-09-23).

Until now the questions were about logging well: what to put in config, how to name metrics, how to version a dataset. From here they change. What happens to two hundred training jobs when the tracking backend is down? Which component degrades first as projects grow? Who can read a model that was logged by a service account? How do you upgrade a server that holds the only record of a year of experiments? And when is W&B the wrong answer?

Where this picks up

I am starting with architecture, because every failure mode in this guide follows from where the components sit and who talks to whom.

Architecture and failure modes

A W&B deployment has two halves that fail differently. The client half is a Python process that imports wandb. The server half is either a multi-tenant service run by the vendor or a set of services you run yourself.

When a training script calls wandb.init(), the SDK spawns wandb-core, a Go binary that runs as a separate process. The Python side only enqueues records. wandb-core owns everything that touches the network: GraphQL calls, the file stream that carries history, summary and console output, uploads of files and artifacts straight to object storage through pre-signed URLs, and system-metric sampling every 15 seconds (CPU, memory, disk, network, NVIDIA through NVML, AMD, Apple Silicon, TPU through libtpu, and cgroup v2 limits in containers). It also writes a local transaction log, run-<id>.wandb, inside the run directory ./wandb/run-YYYYMMDD_HHMMSS-<RUN_ID>/.

That separation is the most useful fact a platform owner has, because it explains the failure behaviour. If wandb-core crashes, training continues. If the network drops, records accumulate in the transaction log and upload resumes when it returns. If the process is killed or the job ends before the upload finishes, the log is still on disk and wandb sync wandb/run-<TIMESTAMP>-<ID> replays it. Tracking is designed not to be in the critical path of training, and your job is to keep it that way.

On the server side, the operator deploys the services in the diagram behind a single ingress. An optional wandb-weave-trace service on /traces (port 8722) exists for Weave, the separate LLM tracing product, and ClickHouse is required only if you use it. The state layer is where your operational work concentrates. MySQL holds metadata. The application database supports a primary with a hot standby, and multi-primary and read replicas are not supported. Redis is a single node used for queues and caching; the bundled Redis is for proofs of concept only. Object storage holds the bytes and must support pre-signed URLs and CORS: S3, GCS, Azure Blob, CoreWeave AI Object Storage, or an enterprise S3-compatible system. The open-source MinIO is in maintenance mode and not recommended.

Here is the failure taxonomy worth memorising, in the order you are likely to meet it.

Two SDK settings encode that decision. finish_timeout defaults to 0.0, meaning wait until the upload finishes. On an expensive GPU node that is a cost leak: a slow server can keep eight GPUs allocated while a process waits to flush metrics. A positive value makes the SDK give up, mark the run Crashed or Failed, and leave the data on disk for wandb sync. finish_timeout_raises (default False) decides whether the timeout raises. stop_on_fatal_error (default False) and stop_fn control whether the run stops the script when the backend reports a fatal error. These are platform policy and belong in a shared wrapper, not in each researcher's script.

Scaling: which component breaks first

Performance problems in W&B are rarely the server's raw capacity. They are usually a dimension of your data that crossed a soft limit. The Multi-tenant guidance gives the numbers, and they are useful for Self-Managed too because they describe where the query patterns degrade.

Of all these, metric cardinality is the one that surprises people, because nothing in a single script looks wrong. Cardinality is the number of distinct metric keys in a project, and nested dictionaries count as flattened, dot-joined names. The docs name it as the most common cause of slow workspaces. The pattern that creates it is a key that embeds a variable: loss_layer_{i}_{step}, a metric per sample id, or a metric per hyperparameter combination. One such script, run five hundred times by a sweep, produces millions of distinct keys in one project. The fix is upstream: log the variable as a value in a table or histogram, and keep key names from a fixed set.

The second scaling axis is write rate. On Multi-tenant Cloud, metric logging is limited per project by request rate and bytes, GraphQL by user, and some SDK requests by database query time. On Self-Managed the rate limits are unset by default, and that is a decision to make on purpose rather than leave to chance. The recommended defaults are:

The per-run limit of two requests per second matters more than it looks. It is why the guidance is to batch keys into one run.log call instead of logging each metric separately, and why a training loop that logs at every step of a fast model should log every N steps. A sweep of a thousand tiny jobs respects the per-run limit and still exhausts the per-project run-creation limit.

Choosing a deployment model

The first platform decision is where the server runs. There are three options, and the choice is mostly about who carries the pager and where the data may live.

The Multi-tenant service is now branded CoreWeave Forge in parts of the documentation, while other pages still say "W&B Multi-tenant Cloud" and the SDK still defaults to https://api.wandb.ai. Do not tell users to change WANDB_BASE_URL for the SaaS. For a Gulf or Egyptian employer with data-residency obligations the Multi-tenant option, hosted in North America, is frequently a non-starter for anything that contains regulated data, which pushes the decision toward Dedicated Cloud in a region that satisfies the requirement, or Self-Managed on infrastructure inside the jurisdiction. Verify the regions your account team actually offers before you design around them; the catalogue of Dedicated regions changes.

Self-Managed buys control and sells you operations. You own the MySQL primary and its backup, the Redis node, the bucket policy and CORS, the ingress and its TLS certificate, the upgrade schedule, and the service-level objective. The documented requirements as of server 0.85 are Kubernetes 1.34 or newer, Helm 3, MySQL 8.4 (8.0 reached end of life in April 2026), Redis 7.x, and x86-64 nodes only. There is no ARM support. The legacy single-pod wandb Helm chart is deprecated and unsupported. The supported install is the W&B Kubernetes Operator:

helm repo add wandb https://charts.wandb.ai
helm upgrade --install operator wandb/operator -n wandb-cr --create-namespace

You then apply a WeightsAndBiases custom resource, and the operator reconciles the rest.

apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  name: wandb
  namespace: wandb-cr
spec:
  values:
    global:
      host: https://wandb.example.com
      license: <from your secret manager>
      bucket:
        provider: s3
        name: my-wandb-bucket
        region: me-central-1
      mysql:
        host: wandb-db.internal
        database: wandb_local
        user: wandb
        password: <from your secret manager>

Treat the field names above as a shape, not a contract: check the operator page for the exact keys of your chart version, and never commit the licence or database password. The precedence rule to remember is that Release Channel values are overridden by values entered in the System Console, which are overridden by values in the custom resource. When a setting "won't stick", the cause is almost always a value in the custom resource that wins over the console. Infrastructure is usually created through the Terraform modules (wandb/wandb/aws, .../google, .../azurerm, plus wandb/wandb/helm), which fits naturally next to your Terraform estate and Helm practice, and the cluster itself is ordinary Kubernetes.

One constraint trips up enterprise networks: W&B Server does not support self-signed certificates or custom certificate authorities. TLS must terminate at the ingress or load balancer with a certificate that clients already trust. The SDK has an insecure_disable_ssl setting, default False, and it is not a production fix. If your organisation runs a TLS-inspecting proxy, plan for an exemption on the W&B host, because clients that cannot validate the chain fail with Failed to verify credentials or init timeouts, and the workaround of disabling verification converts a networking problem into a security one.

The trust model: authentication and authorization

W&B separates who you are (authentication) from what you may touch (authorization), and most platform mistakes come from conflating them.

Authentication for people. Multi-tenant Cloud uses Auth0, with SSO over OIDC or SAML on Enterprise plans. Dedicated and Self-Managed use OIDC (implicit flow with form post, or authorization code with PKCE) or LDAP, configured with variables such as OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_AUTH_METHOD and GORILLA_OIDC_SECRET. A SCIM API provisions and deprovisions users, and domain capture can auto-provision people whose email matches your domain. The operational point is that identity lifecycle belongs to your identity provider. If offboarding means "someone remembers to remove them from W&B", it will eventually not happen, so wire SCIM or group-based provisioning and audit it.

Authorization. The hierarchy is organization, then teams, then projects. Organization roles are Admin, Member, Viewer (Enterprise), Billing-only (Multi-tenant) and Custom roles (Enterprise). Team roles are Admin, Member, View-Only (Enterprise) and Custom, and there are project-level roles too. Project visibility is Open, Public, Team or Restricted. Registries have their own visibility settings.

Three design rules follow.

1. Make the team the unit of access, and log to a team entity. Artifacts logged to a personal entity cannot be linked to a Registry. A researcher who logs a model to their personal namespace has created something that cannot be promoted, and that shows up late, at the moment someone tries to ship. Set GORILLA_DISABLE_PERSONAL_ENTITY=true on Dedicated and Self-Managed to remove the trap. Also note GORILLA_ALLOW_USER_TEAM_CREATION is false by default, so team creation stays with admins.
2. Use Restricted projects for sensitive work, and remember service accounts. A service account must be added to a restricted project explicitly. "The pipeline suddenly gets Unauthorized" after a project is restricted is the predictable symptom.
3. Harden the API surface. On Self-Managed and Dedicated, GORILLA_GRAPHQL_DISABLE_INTROSPECTION=true stops anonymous schema discovery, and GRAPHQL_REJECT_UNAUTHED_REQUESTS=true (from v0.80.0) rejects unauthenticated GraphQL requests. Both are cheap and there is rarely a reason to leave them off.

Everything above assumes the server is reachable only by the people and machines that should reach it. Put it behind your normal network controls. Dedicated Cloud supports IP allowlisting and private connectivity, and on Self-Managed the ingress and security groups do the same job.

Machine identity and secrets

Humans are the smaller part of the attack surface. Most of your traffic comes from pipelines, schedulers and training jobs, and they authenticate with keys.

An API key is shown once at creation; afterwards the UI shows only the key ID prefix. New keys are about 86 characters long, and SDKs older than 0.22.3 reject them with ValueError: API key must be 40 characters long, yours was 86. Fleets with pinned old images hit this when a rotation issues a new-format key, so inventory SDK versions before you rotate. There are three key types: personal, service-account, and organization. Service accounts are the non-human identities, scoped to an organization or a team, holding their own keys. They do not consume a seat, which removes the usual incentive to share one human's key across a pipeline. A service account can attribute runs to a person through WANDB_USERNAME or WANDB_USER_EMAIL, provided that person is a member of the team; if attribution is silently ignored, that membership is what is missing.

Prefer team-scoped service accounts. An organization-scoped account can access every non-restricted project and needs a default team, which is a wide grant. A team-scoped account gives least privilege and is the right default for a pipeline that serves one team. Since Server v0.83.0 an admin can disable creation of team-scoped service accounts, so check the current variable name in the server environment-variables page before relying on it, and do not disable them unless you have a replacement.

The mature alternative to long-lived keys is identity federation, in preview on Enterprise. An organization admin registers a JWT issuer. W&B fetches ${ISSUER}/.well-known/openid-configuration and the JWKS, and clients point WANDB_IDENTITY_TOKEN_FILE at a file containing a token. The checks are strict and worth knowing when it fails: iss must equal the registered issuer, sub must equal the user's email (or the configured Subject for an external service account), aud must equal the organization name (overridable on Dedicated and Self-Managed with FEDERATED_AUTH_AUDIENCES), and exp must not have passed. Since 0.28.2 the SDK re-reads the token file on each exchange, so a sidecar that refreshes the token in place works. Set WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE together and the SDK refuses with AuthenticationError: Both WANDB_API_KEY and WANDB_IDENTITY_TOKEN_FILE are set, which is not allowed. The credential lookup order the SDK follows is: identity token file, explicit key argument, WANDB_API_KEY, settings-file api_key, netrc, then an interactive prompt if a TTY exists.

For the keys you do keep, the rules are the usual ones, and they are worth enforcing mechanically.

• Store keys in a secret manager such as Vault, AWS Secrets Manager or Azure Key Vault, and inject them as WANDB_API_KEY at job start. Never bake them into images.
• Never commit ./wandb/settings or netrc files that contain api_key. Add both to your secret-scanning rules. Since 0.30.0, wandb status --settings no longer prints the key, which reduces but does not remove the leak risk from shared terminal logs.
• Rotate by creating a new key, rolling it out, then deleting the old one. Because the full key is shown only once, the new key must go straight into the secret manager.
• Keep the pipeline from logging into a personal account. A CI job using an engineer's key stops working the day they leave.

Team secrets are a separate server-side feature: sensitive strings scoped to a team, referenced as ${SECRET_NAME} in webhook payloads and used by Weave Playground and sandboxes. Only admins can create, rotate or delete them, names are letters, digits and underscores, and the value cannot be revealed after creation. On self-hosted deployments enable them with ENABLE_SECRET_STORE and choose the backend with GORILLA_CUSTOMER_SECRET_STORE_SOURCE; the default is k8s-secretmanager://wandb-secret, and W&B recommends a cloud secret manager over Kubernetes secrets.

The data plane: storage, pre-signed URLs and residency

Where bytes live decides your compliance position. The split is consistent across deployments. In the database live metadata, reports, experiment logs, system metrics and console logs. In the bucket live files, artifacts, media, run files and parquet history exports. Nothing in the metrics path needs to touch the bucket, which is why a bucket outage and a database outage feel different.

The upload mechanism is pre-signed URLs. The client asks the API for a URL, then sends the file directly to object storage. Read URLs expire after one hour and write URLs after 24 hours. They are scoped by team access and carry X-User=<username> (or scid on Azure), which means bucket audit logs can be correlated back to a person. The trust consequence is worth stating plainly: the bucket must be reachable from every client network that uploads or downloads, and a pre-signed URL works for anyone holding it until it expires. Restrict which networks may use them through bucket IAM policies, and do not paste artifact URLs into tickets as if they were access-controlled links.

Bring your own bucket (BYOB), also called the Secure Storage Connector, lets you point storage at infrastructure you control. Multi-tenant Cloud supports team-level BYOB, but not Azure Blob. Dedicated and Self-Managed support both instance-level and team-level BYOB, with an optional sub-path. Team-level buckets are the multi-tenancy lever for residency: a team with regulated data gets its own bucket, even in a different region or cloud, while the rest share a default. Buckets can also be shared across teams through sub-paths. Server v0.85 added AWS-to-AWS S3 role assumption for BYOB. The metadata in MySQL, however, still lives in the deployment's region, so BYOB alone does not make a deployment "resident"; state precisely which data classes stay where when you answer a compliance questionnaire.

Reference artifacts avoid the copy entirely. add_reference() with s3://, gs://, file:// or an Azure location records the location and checksums, but W&B does not store the bytes. They are the right tool for multi-terabyte datasets, and the trap is that the referenced object can be overwritten or deleted behind W&B's back. Enable object versioning on the source bucket, or the lineage will describe data that no longer exists.

Artifacts are content-addressed, so identical content does not create a new version, and 0.25.1 and later warn when that happens. The digest algorithm defaults to MD5; 0.29.0 added digest_algorithm="MD5"|"XXH128", and BREAKING.md plans to make XXH128 the default a few releases later. If your tooling compares digests, do not hard-code the algorithm. A security fix in 0.29.0 also rejects artifact file-path traversal on download, which is a good reason to keep clients current.

Multi-tenancy: structuring the organization

The hierarchy has a cost of getting it wrong, and the cost is mostly paid in permissions and in reports nobody can find. A workable mapping is one team per product or research group, one project per model family or problem, and runs grouped with group= and job_type= inside the project. Resist one project per experiment: the workspace, reports and registry links all work better with many runs in one project than with hundreds of single-run projects.

Project names are at most 128 characters and cannot contain / \ # ? % :. A run's ID cannot contain those characters either and cannot be reused after the run is deleted, which matters for resume logic: a deleted run's ID is gone. WANDB_RUN_ID is limited to 64 characters; auto-generated IDs are 8. If you derive IDs from a job name, keep them short and deterministic.

Naming is part of the platform. Agree on a convention for projects (<problem>-<model-family>), for tags, and for metric prefixes (train/, val/, system/), and publish a thin wrapper library that applies them. A common pattern is an internal package, company_tracking.init_run(...), that sets the entity from the team, applies organisation-wide Settings (timeouts, git capture, ignore globs), reads credentials from the environment, and returns a context-managed run. This is the single most effective lever you have, because every safe default lives in one place and upgrades are one pull request.

import os
import wandb


def init_run(project: str, *, config: dict, job_type: str, group: str | None = None):
    """Start a tracked run with the platform's defaults applied."""
    settings = wandb.Settings(
        init_timeout=120,
        finish_timeout=300,          # never hold a GPU node hostage on upload
        finish_timeout_raises=False, # data stays on disk for `wandb sync`
        disable_git=os.environ.get("TRACKING_DISABLE_GIT") == "1",
    )
    return wandb.init(
        entity=os.environ["WANDB_ENTITY"],   # always a team, never a person
        project=project,
        config=config,
        job_type=job_type,
        group=group,
        settings=settings,
    )

Use it as with init_run(...) as run:, so the context manager calls finish() and an exception marks the run Failed rather than Crashed. Sweeps are the tenant behaviour that most needs a policy, because they are the largest consumer of shared compute and of run-creation rate. A sweep's controller runs on the W&B server by default, and each agent is a process you paid for. wandb agent --count N bounds work per agent, and run_cap in the sweep YAML bounds the whole sweep. Random and Bayesian sweeps run until stopped and a grid over a continuous space never ends, so require a run_cap or an owner who checks. Sweep states matter for cost too: stop lets active runs finish, cancel kills them. For preemptible instances, wandb agent --forward-signals (added in 0.24.0) passes termination signals to the training process, and --term-timeout (0.28.2) sets how long the agent waits after terminating. In the training script, handle the signal by calling run.mark_preempting(), checkpointing, and exiting non-zero (for example 128 + signum). The sweep run becomes PREEMPTED and is re-queued, and the next agent must call wandb.init() within 60 minutes. Never call mark_preempting() unconditionally, and remember that a SIGKILL leads to Crashed after roughly five minutes, not a requeue.

For distributed training, pick one of three patterns and document it. Log from rank 0 only, which is simplest. Give every rank its own run, joined by group= and job_type=, which gives per-rank system metrics. Or use one shared run, where each node initialises in shared mode:

import os
import wandb

rank = int(os.environ["RANK"])
run = wandb.init(
    settings=wandb.Settings(
        mode="shared",
        x_label=f"rank_{rank}",
        x_primary=(rank == 0),
        x_update_finish_state=(rank == 0),
    ),
)

Every node must use the same WANDB_RUN_ID. Shared mode is marked experimental, so it is not a default to standardise on without testing on your own cluster. If many processes on one node each start their own wandb-core, wandb beta core start runs a single detached service reused through WANDB_SERVICE; after upgrading the SDK, restart it, because 0.29.0 changed the message format between processes.

Registry, automations and promotion as policy

Experiment tracking is reversible: a bad run can be deleted. Promotion is not, because other systems will act on it. The Registry is where W&B stops being a notebook and becomes part of your release process.

A Registry is the organization-level catalogue of artifact versions. Every organization starts with the Models and Datasets registries. Linking an artifact version to a registry collection creates a pointer, not a copy. The target path is wandb-registry-{REGISTRY_NAME}/{COLLECTION_NAME}, and the full name of a linked version is wandb-registry-{REGISTRY}/{COLLECTION}:v{N}. Consumers load by alias, most often production, staging or latest:

import wandb

api = wandb.Api()
art = api.artifact("wandb-registry-model/mnist-classifier:production")
path = art.download()

Three facts about Registry behave like policy, and every owner must know them.

1. Linking sets the version's TTL to None, and unlinking does not restore the TTL. Linked versions therefore never expire, and an unlinked one becomes permanent storage you pay for. Include this in the cleanup runbook, and delete artifacts explicitly when a model is retired.
2. Aliases are mutable. production points at a version today and another tomorrow. Anything that needs a reproducible deployment must record the immutable version (:v7) or the digest, not the alias. Lineage, the graph built from use_artifact and log_artifact, is what lets you answer "which data trained the model in production" months later, but only if pipelines log through artifacts rather than writing files to a path.
3. Registry availability varies. It is on by default on Self-Managed Server v0.71.0 and later. On Dedicated Cloud it must be enabled by your account team.

Automations turn registry events into actions. An automation is an event plus an action: a Slack notification, a webhook, or (as of 0.30.0) SendPromptToAria. Scopes include project, registry, collection, team and organization. At team and organization scope the App offers only artifact and collection events, and run-metric automations are SaaS-only. The Python API for this is Api.create_automation() and Api.update_automation(); both raise UnsupportedError when the server lacks the feature, so a script that works on SaaS can fail on an older Self-Managed server. Events include artifact tags, collection tags and unlink events (0.27.1).

The standard production pattern is event-driven. A webhook automation fires when the production alias moves, calls your CI/CD system with the artifact's name and version, and the pipeline deploys. The webhook's credentials live in a team secret referenced as ${SECRET_NAME}, never in the payload template. This connects directly to the delivery tooling in this catalogue: the receiver might be a GitHub Actions workflow or a step that updates a manifest watched by Argo CD, which then rolls out a KServe InferenceService or a BentoML service. The promotion gate is then a role: who is permitted to add the production alias, enforced through registry roles rather than by convention.

name: deploy-model
on:
  repository_dispatch:
    types: [model-promoted]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Show what was promoted
        run: |
          echo "artifact: ${{ github.event.client_payload.artifact }}"
          echo "version:  ${{ github.event.client_payload.version }}"

Cost, retention and storage governance

Two bills matter. Seats for Models are billed per seat, with Full, Viewer (Enterprise) and No-access seat types, plus usage. Weave has no seat cost and is usage-based. Service accounts do not consume seats. Storage is the one that grows unnoticed: artifacts accumulate with every checkpoint, and shared storage on Free and Pro plans can hit limits, which admins manage in the Billing tab. A plan or quota limit shows up to clients as HTTP 402, which since 0.30.0 is reported immediately rather than retried, so a storage-full condition now fails fast and loudly in every pipeline at once.

The levers, in the order I would reach for them:

1. Checkpoint less, reference more. The largest costs are checkpoints logged as artifacts every epoch. Log the best and the last, and keep the rest in your own bucket as reference artifacts.
2. Artifact TTL. art.ttl = datetime.timedelta(days=30) schedules deletion. TTL applies only to user-generated artifacts, team admins decide who may set it, and a team default TTL is available. Combine a short default with explicit exemptions for things that must be kept. Remember the Registry interaction above.
3. Delete and garbage-collect. Deleting an artifact version does not free bytes unless garbage collection runs. On Self-Managed, GORILLA_ARTIFACT_GC_ENABLED is required for that; without it, deletions remove records while storage stays full, which is among the most common "we deleted terabytes and the bill did not move" findings.
4. Archive cold projects. Move old runs into an archive project to keep active workspaces inside the 10,000-run guidance.
5. Cap end-of-run waits with finish_timeout so a slow backend does not keep GPUs allocated.

Deleted data is retained for seven days by default, and support can restore it within that window. On Dedicated, change the period with GORILLA_DATA_RETENTION_PERIOD (for example "336h") or the Helm value env.dataRetentionPeriod. That window is a safety net for mistakes, and it is also the reason a deletion requested for privacy reasons is not immediate; tell data-protection colleagues the real timing.

Upgrades and migrations

Upgrades are two separate programmes that people conflate: the SDK, which ships to every training image and notebook, and the server, which you or the vendor run.

SDK policy. The SDK is still 0.x. Per BREAKING.md, breaking changes land only when the minor version is bumped (for example 0.29 to 0.30) and only after a deprecation warning has shipped first. Patch releases do not break. So pin wandb==0.30.* in requirements, and treat minor bumps as a change to review. Before each one, read the changelog sections "Notable Changes", "Removed" and "Deprecated", and run your suite with DeprecationWarning escalated. The recent removals show why:

• 0.30.0 removed wandb.InternalApi and the wandb.PublicApi aliases (use wandb.Api()), deprecated wandb.api and wandb.ensure_configured(), and added wandb.login(prompt=False) to test for credentials without prompting. It also removed legacy wandb sync options: --sync-all now errors, and --clean became the separate wandb clean command.
• 0.29.0 removed summary="best" and goal= in run.define_metric(), the quiet argument of finish(), and .length on paginators (use len(api.runs(...))). It stopped normalising wandb.Image pixel values, so arrays must already be in the range 0 to 255, and made wandb.Video() require format= for arrays.
• 0.27.1 removed the Keras WandbCallback in favour of WandbMetricsLogger, WandbModelCheckpoint and WandbEvalCallback.
• 0.24.0 removed wandb.beta.workflows; use Run.log_artifact, Run.use_artifact and Run.link_artifact.

A sensible SDK rollout: build a canary image with the new minor, run one real but low-stakes pipeline against your staging server for a week, read debug.log for warnings, then roll the pin across base images. Restart any wandb beta core service. Note the coupling to the server: since 0.28.1 artifact APIs (Artifact.files(), Artifact.download(), Api().artifact()) fail against end-of-life servers, and some features need a minimum server version, for example reading console_logs from the beginning needs Server 0.77.

Server policy. Major releases (such as 0.85.0) ship monthly, with patches as needed, and each major is supported for 12 months. Dedicated Cloud upgrades automatically. On Self-Managed you upgrade, and W&B recommends doing so at least once a quarter. Falling behind is how you end up with an unsupported version: v0.73 reached end of life at 0.85.0. With the operator, the application updates itself, and the chart upgrade is:

helm upgrade operator wandb/operator -n wandb-cr --reuse-values

Licences can be rotated in the System Console at /console/settings/license. A Terraform-managed install upgrades through wandb_version and license. If you are still on the legacy Helm chart, updates go through image.tag and license, but that chart is deprecated, so the migration to the operator is itself a project to schedule.

The upgrades that carry real risk are the dependencies, not W&B. MySQL 8.0 reached end of life in April 2026, so moving to 8.4 is now a prerequisite: run the MySQL Shell upgrade checker, take and test a backup, upgrade, then run wandb verify. If you use Weave, check the ClickHouse version first: 25.8 to 25.12 and 26.3 or newer are supported, and 26.1 and 26.2 are not. Write the upgrade runbook with a rollback column, and rehearse it on a staging copy restored from a real backup, because a rehearsal on an empty database proves nothing about migration time.

Observability and incident playbooks

You cannot run a platform you cannot see. Run-level visibility comes from system metrics, console logs (with capture_loggers and run.write_logs()), and the two logs in every run directory, debug.log and debug-internal.log. Teams get alerts from run.alert(), from team alerts for a run finished or crashed after N minutes, and from automations to Slack or webhooks. Platform-level visibility comes from the Organization Dashboard (usage), audit logs, LOGGING_ENABLED=true (streams access logs to stdout, or tail /var/log/gorilla.log), and the operator's Prometheus stack. Feed server metrics into your own Prometheus and dashboards, and define an SLO that users can feel: for instance, the fraction of run creations that succeed within ten seconds, and the delay between a run.log and a chart update.

The playbooks below are the ones that recur.

Playbook 1: ingestion is slow or failing for many jobs at once.

1. Confirm scope: one team or everyone? One project suggests a cardinality or rate-limit problem; everyone suggests the platform.
2. Check the client symptom. Timed out initializing run and Timed out while retrying point at connectivity or the API; 429 points at rate limits (the SaaS sends RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset); 402 is a plan or quota limit, not an outage.
3. On Self-Managed, check MySQL connections and replication health, Redis, then API pod saturation, in that order.
4. Protect training: switch the affected jobs to mode="offline" with a config change if you have a wrapper, and sync later.
5. After recovery, run wandb sync for runs that finished without uploading, and tell owners which runs to check.

Playbook 3: a key has leaked.

1. Delete the key in the UI immediately. Deleting ends its access; do not wait to find where it is used.
2. Identify the key's identity and what it could reach. Pull audit logs filtered by the actor and the window.
3. Check bucket access logs for pre-signed URL use correlated by the user marker.
4. Issue a replacement from the secret manager path, roll it out, and confirm the old key returns an authentication error.
5. Search code history and CI logs for the string, and add a scanning rule so the same file type cannot be committed again.

Where W&B stops

A platform owner's credibility comes from recommending against the tool when it is wrong. W&B is excellent at experiment tracking, visualisation, collaboration and lightweight artifact lineage. It is not everything around it.

• It is not a data versioning system for huge mutable datasets. Artifacts version files and references, but if your core need is Git-like branching of data with reproducible pipelines, look at DVC or lakeFS. W&B then records which data version a run used.
• It is not a pipeline orchestrator. Sweeps and Launch queue jobs, but dependency graphs, retries and scheduling belong in Airflow, Prefect or Dagster. W&B integrates with them through service accounts. Launch is still documented, with Docker, Kubernetes, SageMaker and Vertex AI as targets, but for a serious platform I would put orchestration in a dedicated tool rather than build on an experimental scheduler.
• It is not a model server. The Registry stores and promotes models; it does not serve them. Serving belongs in KServe, BentoML or your cloud's endpoint service.
• It is not production monitoring. It tracks training and evaluation. Drift and data-quality monitoring of live traffic is the job of tools like Evidently.
• It is not an open-source-only path. The SDK is open source, but the server requires a licence and is not a drop-in to self-host for free. If a hard requirement is a fully open-source tracking server, MLflow is the usual alternative, and it makes different trade-offs: simpler to self-host, with a thinner collaboration and visualisation layer.
• For LLM applications, use the right product. Weave is the W&B product for LLM and agent tracing, and it is separate from Models, with its own storage requirement (ClickHouse) and billing. Tools such as Langfuse occupy the same space, so choose on data-residency, hosting and team workflow rather than on brand.

The honest summary for leadership is that W&B is the system of record for experiments and model lineage. The moment someone asks it to do orchestration, serving or live monitoring, you are extending it beyond its strengths and should integrate a specialised tool instead.

Running it as a platform for other teams

Everything above is technical. What makes a platform succeed is the product work around it. You are offering a service with users, an onboarding path, a support channel and a roadmap, and the failure mode is a capable system that nobody uses consistently.

A golden path. One documented way to start: install the pinned SDK from your internal base image, authenticate through the secret manager, call the wrapper, log to the team entity, and link to the registry. New users should be productive in under an hour. Provide a template repository with the wrapper, a sample training script, a CI job using a team service account, and a manual workspace template.

Onboarding and offboarding as code. Create a team, its service account, its registry permissions and its bucket sub-path from one definition, through Terraform or the admin API, so two teams never differ by accident. Offboarding is the same in reverse, with a checklist for the keys and project ownership the departing person held. Identity provider groups drive membership, so access follows employment.

Measure adoption and health. Track the fraction of training jobs that log to W&B, the share of production models with a registry entry, run-creation success rate, time to first run for new teams, storage per team and cost per team. These are the numbers you take to leadership, and they tell you where the platform is failing before users complain.

The review checklist

Use this for design documents and pull requests that touch tracking. Each item maps to a section above.

Architecture and resilience

• Does training survive the tracking backend being down, and is that tested?
• Are init_timeout, finish_timeout and the offline fallback set deliberately by the wrapper?
• Is run creation staggered for large fleets and sweeps?

Scale

• Are metric keys drawn from a fixed set, with no ids, steps or hyperparameters embedded in names?
• Are keys batched into one run.log call per step, and is the log frequency under the guidance?
• Is the project inside 10,000 runs, and the workspace manual if it has many metrics?
• Do bulk API readers sleep between requests and avoid scan_history unless they need full data?

Identity and security

• Is every non-human workload on a team-scoped service account, with the key in a secret manager?
• Is federation used where available, and is it never combined with WANDB_API_KEY?
• Are wandb/settings and netrc files covered by secret scanning?
• Are sensitive projects Restricted, and are service accounts added to them on purpose?
• Are GraphQL introspection disabled and unauthenticated GraphQL rejected on self-hosted?

Data

• Do runs log to a team entity, and is the personal entity disabled?
• Does any artifact reference mutable external data without bucket versioning?
• Is BYOB configured for any team with residency requirements, and do you know which metadata still lives in the deployment region?
• Is bucket access logging on?

Promotion and cost

• Does deployment resolve aliases to a version or digest at deploy time?
• Is the receiver of a promotion webhook idempotent and does it verify the digest?
• Does every checkpoint policy have a TTL or a reference, and is the linking-clears-TTL effect accounted for?
• Is garbage collection enabled on Self-Managed?

Operations

• Is the SDK pinned to a minor, and are deprecation warnings escalated in CI?
• Is the server within its 12-month support window, with MySQL on 8.4 and, if Weave is used, a supported ClickHouse?
• Has the restore of a MySQL backup been tested, and is there a rollback step in the upgrade runbook?
• Do you have an SLO, dashboards, audit-log export, and a game-day record?

The complete picture

Training jobs run a pinned SDK through a thin wrapper, authenticate as team-scoped service accounts or with federation, and stream to a server whose state (MySQL, Redis, bucket) you or the vendor protect. Teams are the unit of access and cost, promotion goes through the Registry with versions resolved at deploy time, and upgrades follow a pinned, canaried, rehearsed cadence. The platform succeeds through defaults, limits, runbooks and clear ownership, and by not being asked to do what other tools do better.

Where the series leaves you

At Beginner you could track one experiment. At Mid-level you could version data and models, run sweeps, and use the API well. At this level you can design a deployment, reason about which component fails first, choose a trust model, own the upgrade programme, and argue for or against the tool with evidence. The next steps are outward. Deepen the delivery side with the Kubernetes, Terraform, Helm and Argo CD guides, since a Self-Managed W&B is one more workload in that estate. Compare approaches by reading the MLflow guide, and follow the model downstream through KServe, BentoML and Evidently.

The skill that does not change between versions is asking the same questions of any platform you inherit: what happens when it is down, what breaks first as it grows, who can reach what, how it is upgraded, and when you would choose something else.

Sources

• W&B quickstart
• SDK release notes
• Server release notes
• Release policies
• wandb SDK changelog at v0.30.0
• Python reference: init
• Python reference: Artifact
• Track limits and performance
• Run states
• Artifact TTL
• Track external files
• Link a version
• Secrets
• Self-managed requirements
• Kubernetes operator
• Server upgrade process
• Secure storage connector (BYOB)
• SSO
• Service accounts
• Restricted projects
• Organization dashboard
• Support: why is my run showing as crashed

Senior interviews for a platform role are not quizzes about function signatures. The interviewer wants to see whether you can reason about a system you would be on call for: where it fails, what it costs, who can touch what, and how you would persuade people to use it. This file gives you system-design prompts, incident questions, and the trade-off arguments you will need to make to leadership. Each answer is a model, not a script. Adapt the numbers to your own experience, and say so when you are unsure. Versions referenced here are SDK 0.30.0 and W&B Server v0.85.0.

The thirty-second answer

If you are asked what you would own as the W&B platform owner, a good answer runs like this: "W&B is the system of record for experiments, datasets and model lineage. I would own the deployment model, identity and machine credentials, the storage and cost policy, the Registry promotion path, and the upgrade programme. The design principle is that tracking must never sit in the critical path of training, so the client is built to survive an outage, and my job is to keep it that way while keeping the platform cheap, secure and boring."

Then stop and let them pick a thread. The worst senior answers recite features. The best ones show an ordering of concerns.

The model you must be able to draw

Be ready to sketch this on a whiteboard in under two minutes.

• Client: the Python SDK enqueues records. It spawns wandb-core, a Go process that streams history, summary and console output to the API, uploads files straight to object storage through pre-signed URLs, samples system metrics every 15 seconds, and writes a local transaction log, run-<id>.wandb.
• Server: an ingress in front of wandb-app, wandb-api and wandb-console, with MySQL for metadata, Redis for queues and cache, and an object store for bytes. ClickHouse only appears for Weave.
• Hierarchy: organization, teams, projects, runs. Artifacts are versioned, content-addressed bundles. The Registry is the organization-level catalogue that holds pointers to artifact versions.
• Deployment options: Multi-tenant Cloud (vendor-run, North America), Dedicated Cloud (single tenant in your chosen cloud and region, auto-upgraded), and Self-Managed (you run it on the operator, with a licence).

Facts to keep at your fingertips: if wandb-core crashes, training continues; data recovers with wandb sync; run creation is the path that can block a job; MySQL 8.4, Redis 7.x, Kubernetes 1.34 or newer, x86-64 only for Self-Managed; server majors are supported for 12 months.

System-design prompts

Prompt 1: Design experiment tracking for 150 ML engineers across four countries

What the interviewer is testing: whether you start from requirements instead of a product.

A strong answer. Start by asking questions, and say why each matters. What data classes will be logged, and do any have residency rules? That decides the deployment model. How many concurrent jobs, how many runs per day, how many sweeps? That sizes rate limits. What identity provider exists? What does the release process look like for models? Who carries the pager?

Then give a design with reasoning. "Assuming regulated data in at least one country, I would not use Multi-tenant Cloud, which is hosted in North America. I would choose Dedicated Cloud in a region that satisfies the requirement if the vendor offers one, because it removes the database and upgrade burden, and Self-Managed on in-country infrastructure if not. I would use team-level buckets for teams with residency constraints and say clearly that metadata still lives in the deployment's region."

Continue with structure. "One team per product or research group. Projects per problem, with runs grouped by group and job_type. Logging to team entities only, with the personal entity disabled on self-hosted. Pipelines authenticate as team-scoped service accounts with keys in our secret manager, moving to identity federation once we confirm it fits our issuer. SSO through OIDC, SCIM for lifecycle so offboarding follows HR."

Then the delivery mechanism, which is the part weak candidates omit. "A thin internal wrapper around wandb.init applies organisation settings: init_timeout, a finish_timeout so a slow backend cannot hold GPUs, the entity from the team, and the context manager. A template repository and a manual workspace template make the right path the easy path."

Finish with operations. "SLOs on run-creation success and ingestion delay. Audit logs exported to storage we control. A quarterly upgrade cadence, MySQL backups restored on a schedule, and a game day."

What a weak answer misses: residency, who owns the pager, machine identity, and any mention of how teams actually adopt it.

Prompt 2: Training jobs must not fail because the tracking server is down

What the interviewer is testing: knowledge of client failure behaviour, and judgement about when to fail closed.

A strong answer. "By design, wandb-core is a separate process and the Python side only enqueues, so a network or server failure does not stop training. Records accumulate in the on-disk transaction log, and wandb sync replays it. The exceptions are at the edges. At startup, wandb.init() waits for the backend and raises after init_timeout, 90 seconds by default. At the end, finish() waits for the upload unless finish_timeout is set."

Then the design choices. "For the start, I decide per workload whether a job should fail or fall back. For a nightly batch job I would fall back to mode="offline" and sync later, because losing the training run costs more than missing live charts. For a job whose whole purpose is an audited run, I might fail closed. That is a wrapper-level policy, not something each script chooses. For the end, I set finish_timeout so eight GPUs are not idle waiting on a flush, and leave finish_timeout_raises false so data stays on disk. I monitor for runs that finished without uploading, and have a sweeper job that runs wandb sync on leftover directories."

Mention two traps. "A 429 on run creation does block training, whereas filestream 429s are retried with backoff. And since 0.30.0, an HTTP 402 for plan or quota is reported immediately instead of retried, so an exhausted storage quota now fails fast in every pipeline at the same time."

Follow-up you may get: "What if the disk fills with transaction logs?" Answer: they are proportional to what was logged, so a cleanup step that runs wandb sync and then wandb clean (0.28.2 and later, for synced runs) on the node belongs in the job epilogue, and ephemeral containers discard the directory entirely, so the offline fallback needs a persistent volume or a sync step before exit.

Prompt 3: Design the promotion path from experiment to production

What the interviewer is testing: whether you understand that aliases are mutable, and that a Registry is a policy surface.

A strong answer. "Models are logged as artifacts to a team entity, because an artifact in a personal entity cannot be linked to a Registry. A candidate is linked to a collection at wandb-registry-model/<collection>. An approver with the right registry role adds the staging alias, then production. A webhook automation on the alias change calls CI/CD, with its credential stored as a team secret and referenced as ${SECRET_NAME}."

"The receiver must do three things. Resolve the alias to an immutable version or digest and write it into the deployment manifest, so the manifest is the record, not the alias. Verify the digest of what it downloads. And be idempotent, because webhooks can repeat and aliases can move backwards. Then a GitOps tool rolls out the serving workload, for example a KServe InferenceService."

Close with lineage and governance. "Because the training run used an artifact input, lineage lets me answer which dataset version trained the production model. I would require a report or registry description with evaluation runs and an approver. I would also document the TTL behaviour: linking sets a version's TTL to None, and unlinking does not restore it, so retired models need explicit deletion."

Incident-response questions

Interviewers ask these to see whether your first move is sensible. A good pattern for any answer is: scope it, protect training and data, find the cause, fix, then prevent.

"Every job in the company is failing to start tracking."

"Scope first: everyone at once points at the platform, one team at one project points at limits or permissions. Read the error a client shows. Timed out initializing run and Timed out while retrying mean connectivity or an unhealthy API; Failed to verify credentials means host, key or a TLS problem; a 429 means rate limits; a 402 means plan or quota. On Self-Managed I check the ingress and TLS, then API pod health, then MySQL connections and Redis, because the API is stateless and the databases are the usual suspects."

"While investigating, I protect training. If the wrapper supports it, I flip affected workloads to mode="offline" by configuration so jobs start, and sync later. I communicate early: what is affected, what is safe, what to do. After recovery I replay leftover run directories with wandb sync, tell owners which runs to check, and write the postmortem."

"An engineer pushed an API key to a public repository."

"Delete the key immediately in the UI, before investigating. Then establish what it could do: a personal key has that person's access, a service-account key has the account's scope. Pull audit logs for that actor and the exposure window, looking at artifact downloads and project access. Correlate with bucket access logs, since pre-signed URLs carry a user marker. Issue a replacement from the secret manager, roll it out, and confirm the old key fails."

"Then prevention: secret scanning on repositories for wandb/settings and netrc files and on the key format, moving the workload to a team-scoped service account or identity federation, and making rotation routine so it is not an emergency skill. I would also remember new keys are about 86 characters long, so a scanner rule built for the old 40-character format would miss them."

"Storage doubled this month and we deleted terabytes, but the bill did not move."

"Two likely causes. On Self-Managed, deleting artifacts removes records but does not free bytes unless garbage collection is enabled with GORILLA_ARTIFACT_GC_ENABLED. On any deployment, versions linked to a Registry have their TTL set to None, and unlinking does not restore it, so a TTL policy quietly skips them. I would confirm both, find the team and project that dominate storage from the Billing tab and Organization Dashboard, and look for checkpoints logged every epoch. The fix is a checkpoint policy (best and last, the rest as reference artifacts), TTL defaults, explicit deletion, and only then quotas. Also note deleted data is retained for seven days by default, so some of the lag is the retention window."

"After an SDK upgrade, several pipelines are throwing errors."

"Separate SDK from server. AttributeError on an API such as wandb.PublicApi, run.get_url(), runs.length, WandbCallback, or errors about summary="best", finish(quiet=True), wandb.Image(normalize=...) or wandb.Video without format= all point at removals in 0.27 through 0.30. Roll the pin back for immediate relief, then fix forward with the documented replacements. I would also check whether the failing images are on an old Python, since current SDKs need 3.10 or newer."

"The lesson to draw is process: pin a minor, run CI with deprecation warnings as errors, and canary new minors for a week against staging. The SDK's policy helps, since breaking changes land only with a minor bump and only after a warning."

Arguing trade-offs to leadership

The hardest senior skill is making a case to people who do not care about the technology. Use their language: risk, cost, time, and options. Below are four common arguments and how to structure them.

Build on W&B or standardise on MLflow?

"Both work. The question is what we value. W&B gives a richer collaboration and visualisation layer, a hosted option, sweeps, a Registry with automations, and a vendor carrying much of the operational load if we choose Dedicated Cloud. MLflow is open source, simpler to self-host, and does not require a licence, but we would build more of the collaboration layer and carry more operations ourselves. I would compare them on total cost including engineer time, data-residency fit, and how many teams already know each tool. If we cannot justify the licence cost per seat, or need a fully open-source path, MLflow is the right answer. If researcher productivity is the bottleneck, the licence is cheap relative to engineer salaries. Either way, I would keep our tracking wrapper thin so a later switch is a change in one library."

Note the honest caveat: you should be able to state that the SDK is open source but the server requires a licence. Do not say it is free to self-host.

Self-managed or Dedicated Cloud?

"Self-Managed gives maximum control and costs an on-call rotation. We would own MySQL backups, Redis, the bucket policy, TLS at the ingress, the upgrade cadence, and the SLO. Dedicated Cloud gives single-tenant isolation in a region we choose, with automatic upgrades and rare downtime. Unless we have a hard constraint Dedicated cannot meet, such as an on-premises requirement or a region the vendor does not offer, the operating cost of Self-Managed usually exceeds the licence difference. I would make the case with a number: the engineer-weeks per quarter to run it, including upgrades, which W&B recommends at least quarterly."

Should we enforce standards or let teams choose?

"Enforce the things that cause incidents and cost, and leave the rest to teams. Non-negotiables: logging to team entities, service accounts for automation, keys in a secret manager, TTL defaults, and a registry entry for production models. Flexible: how they structure experiments, which panels they build, which frameworks they use. I enforce through tooling, not documents: a wrapper with safe defaults, pre-commit checks for committed keys, and a scheduled report that flags high-cardinality projects. People follow the path of least resistance, so the goal is to make the compliant path the easy one."

What a strong answer covers that a weak one misses

• Failure modes before features. Strong candidates explain what happens when each component is down. Weak ones describe the happy path.
• Identity beyond people. Strong answers discuss service accounts, scoping, rotation and federation. Weak answers stop at "SSO".
• Mutable versus immutable. Strong answers treat aliases as pointers and deploy by version or digest.
• Limits with numbers. Strong answers cite cardinality, run counts, rate limits and the per-run two-requests-per-second filestream limit, and explain what to do about them.
• Operations and cadence. Strong answers include upgrades, backups that are restore-tested, and end-of-life dates, not only architecture.
• Honesty about fit. Strong answers say where W&B should not be used, for example orchestration, serving and live drift monitoring, and name the tool that should be.
• Measurement. Strong answers propose metrics for the platform itself.
• Region awareness. For employers in the Gulf and Egypt, strong answers raise data residency unprompted and are precise about what BYOB does and does not move.

Sixty-second self-test

Answer each aloud, then check against the sections above.

1. What keeps training alive when the tracking server is down, and where is the one place a job can still be blocked?
2. Name three dimensions with soft limits and what you do about each.
3. Why can't a model logged to a personal entity be promoted to the Registry?
4. What happens to a version's TTL when it is linked to a Registry?
5. Why must a deploy step resolve production to a version?
6. Which identity is better for a pipeline, a human's key or a team-scoped service account, and why?
7. What do you check first when a project's workspace is slow?
8. Which SDK changes in 0.29.0 and 0.30.0 would break a script that worked on 0.28?
9. Which MySQL version does Self-Managed need now, and why?
10. Name two jobs you would not give to W&B and the tool you would use instead.

If you can answer eight of ten without hesitating, you are ready to talk about this tool at a senior level.

This file collects what tends to go wrong when Weights & Biases stops being one person's tool and becomes something a whole organisation depends on, and the habits that prevent it. It is organised as production traps that appear only at scale, habits for owning a platform, and checklists for pull requests and design documents. It is written for SDK 0.30.0 and W&B Server v0.85.0. Where a trap has a symptom, a cause and a fix, they appear in that order.

Production traps that only appear at scale

Trap 1: Metric cardinality grows silently

Symptom. A project that was fast for months becomes slow. Panels take a long time to render, the automatic workspace freezes, and the URL with &PERF_LOGGING shows heavy query time.

Cause. Cardinality is the number of distinct metric keys in a project, and nested dictionaries count as flattened dot-joined names. One script that puts a sample id, a layer index combined with a step, or a hyperparameter value into a key name creates a new key per run. Run it in a sweep and the project has millions of keys. Nothing in a single script looks wrong, so code review does not catch it.

Fix. Keep key names from a fixed set and move the variable into a value: a wandb.Table, a histogram, or a config entry. Switch the project to a manual workspace. Add a scheduled job that counts keys per project and posts the worst offenders to their owners. The soft limit to plan around is 100,000 distinct metrics per project.

Trap 2: Logging every metric in its own call

Symptom. 429 errors in client logs, delayed charts, and a slow finish(), even though the training job is small.

Cause. Each run.log() commits a step. Ten metrics logged in ten separate calls make ten steps and ten times the request load, and the recommended limit on Self-Managed is two filestream requests per second per run. Filestream 429s are retried with backoff and will not crash the job, which is why people ignore them until finish() takes twenty minutes.

Fix. Batch all metrics for a step into one dict and call run.log once. Log every N steps for fast loops. Keep under about 1,000 run.log calls a minute and 100,000 values a minute per project.

Trap 3: A fleet starts at the same minute

Symptom. Jobs fail in wandb.init() with Timed out initializing run, all at the top of the hour.

Cause. Scheduled jobs and sweep agents start together, and run creation is rate-limited per project (80 per second in the recommended Self-Managed defaults). Unlike filestream throttling, a run-creation 429 blocks training.

Fix. Stagger starts with jitter. Raise init_timeout in the wrapper for large fleets. Decide whether a job should fall back to mode="offline" or fail, and implement that decision once in the shared wrapper.

Trap 4: A GPU node waits on an upload

Symptom. Training finished an hour ago, but the node is still allocated and the job is still running.

Cause. finish_timeout defaults to 0.0, which means wait until the upload finishes. A slow or unreachable server holds the process open.

Fix. Set finish_timeout to a bounded value in the wrapper. A positive value makes the SDK give up and leave the data on disk, and wandb sync replays it later. Leave finish_timeout_raises false unless you want the job itself to fail.

Trap 5: Personal entities make models unpromotable

Symptom. link_artifact fails or the target collection is unavailable for a model that a researcher trained weeks ago.

Cause. Artifacts logged to a personal entity cannot be linked to a Registry. The mistake is invisible until the moment someone tries to ship.

Fix. Log to a team entity from the first run. On Dedicated and Self-Managed, set GORILLA_DISABLE_PERSONAL_ENTITY=true. Make the wrapper read WANDB_ENTITY from the environment and refuse to start without it.

Trap 6: Linking a version makes it immortal

Symptom. A storage clean-up with TTL does not shrink the bill, and old models stay forever.

Cause. Linking an artifact version to a Registry sets its TTL to None, and unlinking does not restore the TTL. The version is now permanent storage.

Fix. Treat the registry as a curated shelf, not a dump. Delete retired versions explicitly. Document the behaviour in the cleanup runbook, and list linked versions in your storage audit.

Trap 7: Deletion does not free bytes

Symptom. Terabytes of artifacts were deleted and storage usage did not fall.

Cause. On Self-Managed, deleting an artifact removes records, while freeing the bucket needs garbage collection, enabled with GORILLA_ARTIFACT_GC_ENABLED. Separately, deleted data is retained for seven days by default and can be restored by support within that window.

Fix. Enable garbage collection, then confirm bucket usage over the following days. On Dedicated, tune the window with GORILLA_DATA_RETENTION_PERIOD or the Helm value env.dataRetentionPeriod if your policy requires it.

Trap 8: Aliases used as release records

Symptom. A service behaves differently after a restart and nobody deployed anything.

Cause. The service loads :production at startup. Someone moved the alias, and the next pod loaded a different model. Aliases are mutable pointers.

Fix. Resolve the alias to a version (:v7) or digest at deploy time, and write that into the deployment manifest. The manifest becomes the record and the alias becomes a convenience.

Trap 14: Buckets that reject pre-signed uploads

Symptom. Metrics appear but artifacts and media fail to upload, and wandb verify fails its signed-URL check.

Cause. Uploads go directly from the client to the bucket through pre-signed URLs. The bucket needs CORS configured and must be reachable from every client network, and a proxy that strips or limits large payloads breaks uploads.

Fix. Run wandb verify --host https://wandb.example.com from a training subnet and a laptop. It checks login, signed-URL upload, large payloads through the proxy and requests to the base URL. It is only for self-hosted and Dedicated deployments, so never run it against api.wandb.ai.

Trap 15: An old server with new clients

Symptom. Artifact.files(), Artifact.download() or Api().artifact() fail for everyone after an SDK bump, or create_automation() raises UnsupportedError.

Cause. Since 0.28.1 the SDK removed legacy fallbacks for end-of-life servers, and some features require a minimum server version. For example, reading console_logs from the beginning needs Server 0.77.

Fix. Keep the server inside its 12-month support window and upgrade at least quarterly. Publish a support matrix that pairs SDK ranges with server versions.

Trap 18: Removed APIs discovered in production

Symptom. AttributeError or a thrown error right after an upgrade.

Cause. Breaking changes land with minor bumps. Recent examples: wandb.PublicApi and wandb.InternalApi (0.30.0), summary="best", finish(quiet=True), .length on paginators and run.get_url() (0.29.0), the Keras WandbCallback (0.27.1), and wandb.beta.workflows (0.24.0). wandb.Image no longer normalises pixel values, so arrays must already be 0 to 255, and wandb.Video() needs format=.

Fix. Pin wandb==0.30.*, run CI with python -W error::DeprecationWarning, and canary new minors. Grep your code for the removed names before you bump.

Platform-ownership habits

Habit 1: Own the wrapper, not the scripts

The highest-leverage artefact you can ship is a small internal library that calls wandb.init with safe defaults. It sets the entity from the environment, the timeouts, git and code capture policy, and returns a context-managed run. Upgrading the SDK, changing a timeout, or adding a new organisation-wide setting then becomes one pull request instead of two hundred. Keep it thin, and never hide wandb.init arguments users legitimately need.

Habit 2: Write down the limits and publish them

Keep a one-page contract for new teams: soft limits (runs per project, steps per run, cardinality, log frequency, value sizes), the batching rule, naming rules, and the retention policy. Put it in the onboarding doc, link it from the wrapper's README, and cite it in every review comment. A limit that only lives in your head is a postmortem waiting to happen.

Habit 3: Prefer machine identity and short-lived credentials

Every automated workload should use a team-scoped service account. Service accounts do not consume a seat, so there is no saving in sharing a person's key. Move towards identity federation, with a token file from your issuer, when your plan and version support it. Rotate on a schedule, not only after incidents, so the skill is practised. Keep keys in Vault or your cloud's secret manager, and inject at job start.

Habit 4: Make the compliant path the easy path

People follow the path of least resistance. A template repository with the wrapper, a sample script, a CI job using a service account, and a manual workspace template will do more for consistency than a policy document. Enforce by tooling where you can: pre-commit hooks for committed keys, a scheduled job that flags high-cardinality projects, a check that production deploys reference a registry version.

Habit 5: Automate onboarding and offboarding

Define teams, service accounts, registry permissions and bucket sub-paths in code, through Terraform or the admin API, so teams cannot differ by accident. Drive membership from identity provider groups, with SCIM where available, so access follows employment. Write the offboarding checklist: keys, project ownership, scheduled jobs and automations created by the departing person.

Habit 6: Treat promotion as a controlled change

Define who may add the production alias. Require a registry description or report with evaluation runs, the data version and the approver. Resolve aliases to versions at deploy time, verify digests, and make the receiving pipeline idempotent. Keep a record of what was promoted, by whom, and when, using audit logs exported to storage you control.

Habit 7: Measure the platform itself

Pick a few numbers and track them monthly: run-creation success rate, ingestion delay, share of training jobs that log to W&B, share of production models with a registry entry and lineage, storage and cost per team, SDK version distribution, and time to first run for a new team. They tell you where the platform is failing before users complain, and they are what you take to leadership.

Habit 8: Rehearse restores and upgrades

Restore last night's MySQL backup to a staging instance on a schedule and time it. Rehearse the server upgrade on that copy, with a rollback column in the runbook. Run a game day where the server is unreachable for ten minutes during a real job, and record what you learn. A backup that has never been restored is a hope.

Practice cards

Each card is a short drill. Do them on a scratch team or a test server, not on production.

Card 1: cardinality. Log 2,000 distinct keys from one run in an automatic workspace, then the same data as one table. Compare load times. Write the rule you will publish.

Card 2: outage. Block egress for a minute during a run. Watch debug-internal.log, then restore the network and let the run finish. Delete the online run and replay the directory with wandb sync. Note what you learned about states.

Card 3: finish timeout. Start a run against a blocked network with finish_timeout=30 and then without it. Measure how long each waits and what state the run ends in.

Card 4: personal versus team. Log a model to a personal entity and try to link it. Repeat with a team entity. Then turn on the personal-entity restriction on a test server.

Card 5: alias drift. Load :production in a script, move the alias, load again. Then write the deploy step that records the resolved version.

Card 6: TTL and linking. Give an artifact a 30-day TTL, link it to a registry, and read the TTL. Unlink and read again.

Operational recipes

Recipe: check who a machine is

wandb status
wandb login --verify
python -c "import wandb; print(wandb.login(prompt=False))"

The first shows settings and which file is in use. The second shows the credential source and default team. The third prints whether credentials are configured, without prompting, in 0.30.0 and later.

Recipe: recover a run that did not finish uploading

wandb sync wandb/run-20260930_101500-abc123xy
wandb sync --dry-run
wandb clean

The first replays one run directory. --dry-run lists what would sync. After a successful sync, wandb clean (0.28.2 and later) removes local data for synced runs. Do not use the removed --sync-all or --clean options; in scripts, plain wandb sync with --yes replaces them.

Recipe: upgrade the operator

helm repo update
helm upgrade operator wandb/operator -n wandb-cr --reuse-values

With the operator, the application updates itself. Take a MySQL backup first, read the release notes for removed settings and end-of-life notices, and check dependencies (MySQL 8.4, Redis 7.x, ClickHouse versions if you use Weave) before you start. Licences can be rotated in the System Console at /console/settings/license.

Review checklist for pull requests

Use this when a change touches tracking code or the platform wrapper.

• Does it log to a team entity and read WANDB_ENTITY from the environment?
• Are metric keys from a fixed set, with no ids, steps or hyperparameters embedded in names?
• Are metrics batched into one run.log call per step, at a reasonable frequency?
• Does it use with wandb.init(...) as run: so failures are marked Failed and not Crashed?
• Does it call run.log rather than module-level wandb.log, particularly if it uses reinit="create_new"?
• Is there any removed or deprecated API, and does CI run with deprecation warnings as errors?
• Are credentials read from the environment, and are there no keys in code, images, or committed settings files?
• For sweeps: is there a run_cap, a stop plan, and preemption handling that calls run.mark_preempting() only on real preemption?
• For artifacts: is there a TTL or a reason for none, and are checkpoints limited to best and last?
• For reference artifacts: is source-bucket versioning enabled?
• For registry links: does the target path have the form wandb-registry-<name>/<collection>, and is the deploy reading a version rather than an alias?
• For API readers: does the script sleep between bulk requests and use history() when a sample is enough?

Review checklist for design documents

Use this when a team proposes a new W&B deployment, a new integration, or a change to how models are promoted.

Requirements and fit

• What are the data classes, and which residency rules apply? Is the deployment model justified against them?
• Who carries the pager, and what is the target SLO?
• Is W&B being asked to do something outside tracking, lineage and promotion? If so, which tool should own it?

Architecture

• What happens to training when the tracking backend is unreachable at start, mid-run and at the end?
• What is the init_timeout and finish_timeout policy, and is the offline fallback decided?
• Which components are external (MySQL with hot standby, Redis 7.x, bucket with CORS and pre-signed URL support), and how are they backed up and restored?
• Is TLS terminated at the ingress with a certificate clients trust?

Scale and cost

• Expected runs per project, steps per run, metrics per project, and log frequency, compared with the soft limits.
• Rate limits set on Self-Managed, and a plan for run-creation bursts.
• A storage plan: checkpoint policy, TTL defaults, garbage collection, archive policy, and a per-team cost report.

Security

• Human authentication (OIDC or SAML, SCIM) and machine authentication (team-scoped service accounts or federation, keys in a secret manager).
• Restricted projects for sensitive work, with service accounts added deliberately.
• GraphQL introspection disabled, unauthenticated GraphQL rejected, and personal entities disabled.
• Audit logs exported, bucket access logs enabled, and a leaked-key playbook.

Change management

• SDK pin and canary process, Python and dependency floors, support matrix.
• Server upgrade cadence, MySQL and ClickHouse compatibility, rehearsed restore and rollback.

Adoption

• The golden path: wrapper, template repository, onboarding and offboarding as code.
• The measures you will report: run-creation success, production-model coverage, storage per team, time to first run.

A short checklist to carry away

• Tracking is never in the critical path of training: test it.
• Keys come from a fixed set, batched per step.
• Everything automated uses a team-scoped service account.
• Log to team entities, deploy by version or digest.
• Linking clears TTL, and garbage collection must be on.
• Pin a minor, warn as errors, canary, then roll.
• Restore a backup before you need one.
• Export audit and bucket logs.
• Publish the limits and the support matrix.
• Integrate other tools rather than stretching this one.