The Complete PyTorch Lightning Guide

This is part one of three. It covers everything you need to train real models with PyTorch Lightning, starting from zero Lightning knowledge. By the end you can turn a plain PyTorch model into a Lightning one, train and validate it, log metrics, save and resume checkpoints, switch from CPU to GPU by changing one argument, and read the errors you are most likely to meet. Mid-level and Senior take the same topics further (multi-GPU, configuration files, production); nothing here is thrown away.

You should already know the basics of PyTorch: what a tensor is, what an nn.Module is, and roughly what a training loop does. You do not need to be good at it. Lightning exists precisely because the training loop is where most people's PyTorch code gets long, repetitive and fragile.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas only stick once you have watched your own model train, stop, and resume.

The guide was checked against Lightning 2.6.6, released on 10 September 2026. Where an older tutorial shows something different, a later section tells you which form is current.

What Lightning is, and the problem it solves

PyTorch Lightning is a Python library that sits on top of PyTorch and takes over the engineering part of training a model, so that your code only contains the research part. The research part is the model, the loss and the optimizer. The engineering part is everything around them: looping over epochs and batches, moving tensors to the right device, turning gradients on and off, calling backward, saving checkpoints, writing logs, using several GPUs, and using mixed precision.

The diagram is the whole idea. You write the recipe once. The Trainer cooks it on whatever kitchen you give it.

Why does this matter? Because the engineering code is the part that breaks. A hand-written loop that works on your laptop CPU typically needs rewriting when you move to a GPU, rewriting again for two GPUs, and again when you add mixed precision. Each rewrite is a chance to forget optimizer.zero_grad(), to leave the model in training mode during validation, or to log from every process at once. Lightning's promise is that the same LightningModule runs unchanged on a CPU, one GPU, many GPUs or many machines. Changing hardware means changing Trainer arguments, not model code.

Two things Lightning is not are worth stating early, because beginners often assume otherwise.

It is not a different framework from PyTorch. A LightningModule is a subclass of torch.nn.Module. Your layers, losses, optimizers and tensors are all plain PyTorch. If you know how to write a PyTorch model, you already know how to write the interesting half of a Lightning one.

It is not a service. There is no server, daemon or dashboard to run. Lightning is a library you import. You run a Python script, the script uses the Trainer, and the Trainer writes files (logs and checkpoints) to disk. That simplicity is why it fits so easily into the rest of an MLOps stack: a training job is just a script that a scheduler or CI system can launch.

What people use it for:

What came before: the hand-written loop

To see what Lightning removes, look at what it replaces. Here is a typical plain-PyTorch training loop for a classifier, with a validation pass and a device move:

import torch
import torch.nn.functional as F

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(5):
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        loss = F.cross_entropy(model(x), y)
        loss.backward()
        optimizer.step()

    model.eval()
    val_loss = 0.0
    with torch.no_grad():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            val_loss += F.cross_entropy(model(x), y).item()
    print(f"epoch {epoch} val_loss {val_loss / len(val_loader):.4f}")

torch.save(model.state_dict(), "model.pt")

This is fine for a first experiment. Now list what it does not do. It has no progress bar. It does not log to a file. It does not save the best checkpoint, only the last. It cannot resume, because the optimizer state is not saved. It does not stop early. It uses one device. And every feature you add (mixed precision, gradient clipping, several GPUs) adds lines to the middle of the loop, tangled with your research code.

Before Lightning, people handled this in one of two ways. Some copied a personal "training utilities" file from project to project, which slowly became an undocumented in-house framework. Others adopted a heavyweight framework that hid the loop completely and made customisation painful. Lightning takes a middle path: it owns the loop, but it exposes hooks, which are methods you can override at specific moments, so you keep control where it matters.

The same task in Lightning has no loop in your code at all. You write what happens for one batch, and the Trainer repeats it. The rest of this guide shows how.

The mental model: four nouns

Lightning has a large API, but you can navigate nearly all of it with four nouns. Learn these and every page of the documentation becomes easier to place.

LightningModule. This is your model plus its training recipe, in one class. It holds the layers (in __init__), the forward pass (forward), what to do with one training batch (training_step), what to do with one validation batch (validation_step), and which optimizer to use (configure_optimizers). It is the research code.

Trainer. This is the engine. You create it with arguments that describe how to run (how many epochs, which hardware, which precision) and then call trainer.fit(model, ...). It owns the loops, the device placement, the checkpointing and the logging. It is the engineering code.

DataModule or DataLoaders. This is where the data lives. The simplest option is to pass ordinary PyTorch DataLoader objects straight to the Trainer. When your data handling grows (downloading, splitting, transforms), you package it into a LightningDataModule, a class that bundles the data steps so they can be reused and shared.

Callback. This is a small object with hook methods like on_train_start or on_train_batch_end, passed to the Trainer in a list. It holds reusable, non-essential behaviour: saving the best model, stopping early, monitoring the learning rate. Callbacks keep that behaviour out of your module.

The direction of control is the thing to internalise. In plain PyTorch you call the model. In Lightning the Trainer calls your methods, at the right moments, in the right order. This is sometimes called inversion of control, and it explains a common beginner confusion: you never write for batch in loader and you never call training_step yourself.

A few more words you will meet in the first hour:

• Epoch: one full pass over the training data.
• Step or batch: one iteration of the loop, over one batch of data.
• Hook: a method with a fixed name that the Trainer calls at a fixed moment. training_step is the most important hook.
• Checkpoint: a .ckpt file holding everything needed to resume training (the weights, the optimizer state, the epoch counter and more).
• Logger: the adapter that sends metrics somewhere you can look at them.
• Accelerator: the hardware type (CPU, GPU and so on).

Installing and checking the setup

Lightning needs Python 3.10 or newer. Python 3.9 support was removed in release 2.6.1. It also needs PyTorch, which pip installs for you if it is missing. Use a virtual environment so the project's packages stay separate from the rest of your machine.

On Linux or macOS:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install lightning

On Windows (PowerShell):

py -3.12 -m venv .venv
.venv\Scripts\activate
pip install lightning

There are two package names on PyPI. lightning is the recommended one; you import it as import lightning as L. pytorch-lightning is the same Trainer code under the older pytorch_lightning namespace. Both are at 2.6.6, and either works, but pick one per project.

If you need a GPU build of PyTorch, install PyTorch first using the selector at pytorch.org/get-started, which gives you the right command for your CUDA version, and then install Lightning. If you install Lightning first, pip picks a default PyTorch wheel for your platform, and on some systems that is a CPU-only build. You would then see the GPU "not available" error described in the troubleshooting section.

Apple Silicon Macs use the GPU through the mps accelerator. accelerator="auto" finds it for you. Multi-device training is not available on MPS, which is fine for learning. Some PyTorch operations are not implemented for MPS yet, and PyTorch has an environment variable, PYTORCH_ENABLE_MPS_FALLBACK=1, that makes those operations run on the CPU instead of failing.

Windows works, with two differences. Distributed runs use the gloo backend instead of NCCL, and you must wrap the entry point of your script in if __name__ == "__main__":. That guard is good practice everywhere, and it is mandatory on Windows and macOS whenever DataLoader workers start new processes. WSL2 is a practical option if you want a Linux-like experience.

For the examples in this guide, also install torchvision, which provides the MNIST dataset:

pip install torchvision

An optional extras bundle adds the pretty progress bar, command-line tooling and plotting helpers:

pip install "lightning[extra]"

Now verify the installation. This one-liner prints the Lightning version, the PyTorch version, and whether a CUDA GPU is visible:

python -c "import lightning as L, torch; print(L.__version__, torch.__version__, torch.cuda.is_available())"

You should see something like 2.6.6 2.10.0 False on a CPU-only laptop. False is fine for this guide. On a Mac, check MPS instead:

python -c "import torch; print(torch.backends.mps.is_available())"

Then run a smoke test with a model that Lightning ships for exactly this purpose:

import lightning as L
from lightning.pytorch.demos.boring_classes import BoringModel

L.Trainer(fast_dev_run=True).fit(BoringModel())

fast_dev_run=True runs one training batch and one validation batch and then stops, with logging and checkpointing disabled. If this finishes without an error, your installation works end to end. You will use fast_dev_run constantly; it is the fastest way to find out whether a change broke your code.

Your first project, step by step

We will train a small classifier on MNIST, the dataset of 28 by 28 handwritten digits. The model is deliberately tiny, because the point is the Lightning structure, not the accuracy. We build it in four steps: the module, the data, the Trainer, and then a run.

Step 1: write the LightningModule

import torch
import torch.nn.functional as F
import lightning as L


class LitClassifier(L.LightningModule):
    def __init__(self, lr: float = 1e-3):
        super().__init__()
        self.save_hyperparameters()
        self.net = torch.nn.Sequential(
            torch.nn.Flatten(),
            torch.nn.Linear(28 * 28, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        self.log("train_loss", loss, prog_bar=True)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = F.cross_entropy(logits, y)
        acc = (logits.argmax(dim=1) == y).float().mean()
        self.log("val_loss", loss, prog_bar=True)
        self.log("val_acc", acc, prog_bar=True)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)

Read it top to bottom, because every line earns its place.

The class inherits from L.LightningModule, which is itself an nn.Module. That is why self.parameters() works and why you build layers in __init__ exactly as you would in PyTorch. Always call super().__init__() first.

self.save_hyperparameters() records the arguments of __init__ (here, lr) into self.hparams, and also into every checkpoint the Trainer writes. This matters later: it lets you rebuild the model from a checkpoint without remembering how you constructed it. It is a one-line habit with a large payoff.

forward defines how the model turns an input into predictions. By convention it is what you call at inference time. Here it just runs the Sequential stack. Lightning does not require forward, but writing it keeps inference clean.

training_step(batch, batch_idx) receives one batch from the training loader and the batch's index. You compute the loss and return it. Returning the loss is what tells Lightning what to differentiate. Notice what is absent: no optimizer.zero_grad(), no loss.backward(), no optimizer.step(), no .to(device). With the default automatic optimization, the Trainer does all of that.

self.log("train_loss", loss, prog_bar=True) records a metric. It sends the value to your logger and, because of prog_bar=True, shows it on the progress bar. One call, two destinations.

validation_step is the same idea for validation data. The Trainer turns gradients off and puts the model in evaluation mode for you before calling it, and switches back afterwards. You do not write model.eval() or torch.no_grad().

configure_optimizers returns the optimizer. The Trainer calls it once at the start of training. self.hparams.lr reads the learning rate that save_hyperparameters stored.

Step 2: prepare the data

For now we use ordinary PyTorch data loaders. The Trainer accepts them directly.

from torch.utils.data import DataLoader, random_split
from torchvision import transforms
from torchvision.datasets import MNIST


def make_loaders(batch_size: int = 64, data_dir: str = "data"):
    tfm = transforms.ToTensor()
    full = MNIST(data_dir, train=True, download=True, transform=tfm)
    train_set, val_set = random_split(full, [55000, 5000])
    train_dl = DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)
    val_dl = DataLoader(val_set, batch_size=batch_size, num_workers=2)
    return train_dl, val_dl

The first call downloads MNIST into a data/ folder. We split the 60,000 training images into 55,000 for training and 5,000 for validation. The split matters: validation data is data the model never trains on, so its metrics tell you whether the model generalises rather than memorises. We shuffle the training loader and not the validation one.

Step 3: create the Trainer and run

import lightning as L

from model import LitClassifier
from data import make_loaders


def main():
    L.seed_everything(42, workers=True)
    train_dl, val_dl = make_loaders()
    model = LitClassifier(lr=1e-3)
    trainer = L.Trainer(max_epochs=3, accelerator="auto", devices="auto")
    trainer.fit(model, train_dataloaders=train_dl, val_dataloaders=val_dl)


if __name__ == "__main__":
    main()

L.seed_everything(42, workers=True) seeds Python, NumPy and PyTorch, and with workers=True also the DataLoader workers, so reruns start from the same random state. It does not make everything bit-for-bit identical on a GPU (the Trainer's deterministic=True argument pushes further), but it removes the biggest source of run-to-run variation.

The Trainer has three arguments here. max_epochs=3 stops after three passes over the data. accelerator="auto" picks a GPU if one is available and the CPU otherwise. devices="auto" uses all the devices of that type. This pair is why the same script runs on a laptop and on a GPU server unchanged.

trainer.fit(...) starts training. The if __name__ == "__main__": guard keeps the script safe for worker processes.

Step 4: run it

python train.py

On first run you will see MNIST download, then Lightning's messages, then a progress bar per epoch. Training three epochs on a CPU takes a minute or so; on a GPU it is faster. When it finishes, look at the folder: a new lightning_logs/version_0/ directory exists next to your script. That is where the default logger wrote the metrics and where the default checkpoint was saved. Run the script again and you get version_1, then version_2: each run gets its own folder, so you never overwrite a previous run.

Reading what the Trainer prints

Beginners often scroll past the Trainer's output. Reading it is a skill, because it tells you what Lightning decided on your behalf. A typical start looks like this (details vary by version and machine):

GPU available: False, used: False
TPU available: False, using: 0 TPU cores
  | Name | Type       | Params | Mode
---------------------------------------------
0 | net  | Sequential | 101 K  | train
---------------------------------------------
101 K     Trainable params
0         Non-trainable params
101 K     Total params
0.407     Total estimated model params size (MB)
Sanity Checking: |          | 0/? [00:00<?, ?it/s]
Epoch 0: 100%|██████████| 860/860 [00:08<00:00, 100.2it/s, v_num=0, train_loss=0.21, val_loss=0.18, val_acc=0.95]
`Trainer.fit` stopped: `max_epochs=3` reached.

Take it line by line.

The hardware lines tell you which accelerators Lightning found and which it is using. If you expected a GPU and see GPU available: False, stop and fix that now, before waiting hours on a CPU run.

The model summary table lists each top-level submodule with its type and parameter count, and then the totals. It is the quickest sanity check that your model has the size you intended. A model with two million parameters when you expected two thousand is a bug you can spot here.

Sanity Checking is a short validation run on two batches before training begins. Its purpose is to catch bugs in validation_step immediately, instead of at the end of your first epoch. If your validation code is broken, you find out in seconds. If the sanity check bothers you, num_sanity_val_steps=0 disables it, but leave it on while you learn.

The epoch line shows the batch count (860 is 55,000 divided by 64, rounded up), the speed, and the values you marked prog_bar=True. v_num=0 is the run's version number, matching lightning_logs/version_0.

The final line states why training stopped. Here it is max_epochs=3 reached. If you later set up early stopping, this line will say so.

The default epoch cap is worth knowing. If you set neither max_epochs nor max_steps, the Trainer defaults to 1000 epochs, which is almost never what you want while experimenting. Always set one explicitly.

The LightningModule in depth

You have seen the four methods that matter most. Here is how each behaves in more detail, so you can predict what the Trainer does.

What is required

To call trainer.fit, a module needs training_step and configure_optimizers, plus a training dataloader (from the module, a DataModule, or passed to fit). Everything else is optional. If you misspell one, you get a MisconfigurationException that says No training_step() method defined, which is covered in the errors section.

training_step return values

You can return the loss tensor directly, or a dictionary that contains a "loss" key. Under automatic optimization (the default), the Trainer then calls zero_grad, backward and the optimizer step. If you return None, that batch is skipped, which is occasionally useful.

forward versus training_step

forward is the model's behaviour; training_step is the training recipe. A habit worth building: put the prediction logic in forward and call self(x) from the step methods, as we did. Then inference code reuses exactly the same path as training, and you avoid subtle train-versus-serve differences.

configure_optimizers return shapes

The simplest return is a single optimizer. For a learning-rate scheduler you return a dictionary:

def configure_optimizers(self):
    optimizer = torch.optim.Adam(self.parameters(), lr=self.hparams.lr)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.9)
    return {
        "optimizer": optimizer,
        "lr_scheduler": {"scheduler": scheduler, "interval": "epoch", "frequency": 1},
    }

interval says whether the scheduler steps each "epoch" or each "step". For ReduceLROnPlateau you must also add "monitor": "val_loss", naming the logged metric the scheduler should watch. Returning several optimizers, as a GAN needs, requires switching to manual optimization, which is an advanced topic covered at the next level.

Logging metrics with self.log

Everything you want to track goes through self.log(name, value). It is the single most-used call after training_step, so it is worth understanding its defaults.

self.log("train_loss", loss, prog_bar=True)
self.log("val_acc", acc, prog_bar=True)
self.log_dict({"val_loss": val_loss, "val_acc": acc})

What self.log does with the value. It sends it to the logger (so it appears in TensorBoard or the CSV file) and records it in trainer.callback_metrics, a dictionary of the latest logged values. That dictionary is what callbacks such as ModelCheckpoint and EarlyStopping read when you tell them to monitor a name. This is why the name string matters so much: it must match exactly everywhere.

Step versus epoch. The defaults depend on where you call it. In training_step, the value is logged every step (on_step=True) and not aggregated per epoch. In validation_step and test_step, it is logged once per epoch (on_epoch=True), averaged over the batches. You can override either: self.log("train_loss", loss, on_step=False, on_epoch=True) gives you a smooth per-epoch curve for training too.

How often it writes. By default the Trainer writes training-step metrics every 50 steps (log_every_n_steps=50). If your dataset is tiny and you have fewer than 50 batches per epoch, you will see a warning that the interval is larger than the number of batches, and few points on your curves. Lower log_every_n_steps in the Trainer.

Logging several values. Use self.log_dict({...}) for a dictionary of metrics. Passing a dictionary to plain self.log is not supported.

Values must be single numbers. self.log expects a scalar. If you pass a tensor with more than one element you get an error suggesting value.mean().

Seeing the logs

The default log location is lightning_logs/version_N/. What is inside depends on what is installed. If TensorBoard is installed, the default logger is a TensorBoardLogger, and you can view curves with:

pip install tensorboard
tensorboard --logdir lightning_logs/

Open the address it prints (usually http://localhost:6006) and you get live loss and accuracy curves for every version side by side. If neither TensorBoard nor tensorboardX is installed, Lightning falls back to a CSVLogger and prints a warning saying so. The CSV file, metrics.csv, is perfectly good for a first look and loads straight into pandas.

You can choose a logger explicitly and use several at once:

from lightning.pytorch.loggers import CSVLogger, TensorBoardLogger

trainer = L.Trainer(
    max_epochs=3,
    logger=[TensorBoardLogger("logs/", name="mnist"), CSVLogger("logs/", name="mnist_csv")],
)

Other built-in loggers include MLFlowLogger, WandbLogger and CometLogger, which send the same self.log values to those platforms; see the MLflow and Weights & Biases guides in this series (MLflow, Weights & Biases). To turn logging off completely, pass logger=False.

Organising data: DataLoaders and the LightningDataModule

Passing two DataLoader objects to fit is fine for a single script. As soon as you want the same data setup in training, testing and a teammate's project, the setup code needs a home. That home is the LightningDataModule: a class that packages download, splitting, transforms and loaders so that data handling is as reusable as the model.

import lightning as L
from torch.utils.data import DataLoader, random_split
from torchvision import transforms
from torchvision.datasets import MNIST


class MNISTDataModule(L.LightningDataModule):
    def __init__(self, data_dir: str = "data", batch_size: int = 64):
        super().__init__()
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.tfm = transforms.ToTensor()

    def prepare_data(self):
        MNIST(self.data_dir, train=True, download=True)
        MNIST(self.data_dir, train=False, download=True)

    def setup(self, stage: str):
        if stage == "fit":
            full = MNIST(self.data_dir, train=True, transform=self.tfm)
            self.train_set, self.val_set = random_split(full, [55000, 5000])
        if stage == "test":
            self.test_set = MNIST(self.data_dir, train=False, transform=self.tfm)

    def train_dataloader(self):
        return DataLoader(self.train_set, batch_size=self.batch_size, shuffle=True, num_workers=2)

    def val_dataloader(self):
        return DataLoader(self.val_set, batch_size=self.batch_size, num_workers=2)

    def test_dataloader(self):
        return DataLoader(self.test_set, batch_size=self.batch_size, num_workers=2)

You then train with trainer.fit(model, datamodule=dm). The Trainer calls the DataModule's methods in a fixed order, and the split between the first two is the most important thing to understand.

prepare_data() is for work that must happen once: downloading files, writing tokenised data to disk. When you later train on many GPUs, the Trainer runs prepare_data on only one process per machine, so several processes do not download the same file into the same place at once. Because it runs on one process only, do not assign state here (self.something = ...). Other processes would never see it.

setup(stage) is for work that every process needs: building the datasets, splitting them, applying transforms. It receives a stage string, one of "fit", "validate", "test" or "predict", so you can build only what the current phase needs. Assign self.train_set and similar here.

The *_dataloader() methods return the loaders. Each is called when its phase needs it.

A few DataLoader settings affect speed more than people expect. num_workers is how many background processes prepare batches; with zero, the main process loads data and the GPU waits. Lightning even warns you when it looks too low: a PossibleUserWarning saying the dataloader does not have many workers which may be a bottleneck. A sensible start is 2 to 4 workers, tuned later. pin_memory=True speeds up host-to-GPU copies, and persistent_workers=True keeps workers alive between epochs so you do not pay their start-up cost each time. If you set persistent_workers=True, num_workers must be above zero.

On Windows and macOS, workers start as new processes, so your script must be guarded by if __name__ == "__main__":. Without it you get an error about starting a new process before the current one has finished its bootstrap.

Validating, testing and predicting

The Trainer has four entry points, one per phase of a model's life.

The distinction between validation and test is about discipline, not mechanics. You look at validation metrics many times while you tune the model, so you gradually overfit your choices to that set. The test set is the one you touch rarely, ideally once, to get an honest estimate of how the final model performs. That is why it has its own method and its own hook.

To add test support, write a test_step that mirrors the validation step:

def test_step(self, batch, batch_idx):
    x, y = batch
    logits = self(x)
    self.log("test_loss", F.cross_entropy(logits, y))
    self.log("test_acc", (logits.argmax(dim=1) == y).float().mean())

Then evaluate:

trainer.test(model, datamodule=dm)

The Trainer prints a table of the logged test metrics and also returns them as a list of dictionaries. For prediction, define predict_step:

def predict_step(self, batch, batch_idx):
    x, _ = batch
    return self(x).argmax(dim=1)

and call preds = trainer.predict(model, dataloaders=some_loader). You get a list with one result per batch. Predict hooks cannot log.

A habit worth adopting: after training, evaluate with the best checkpoint rather than the last weights. Pass ckpt_path="best" to test, which works when you have a ModelCheckpoint that monitors a metric, as the next section sets up. If you call validate or test with no model and no checkpoint, the Trainer defaults to "best" and warns you.

Checkpoints, early stopping and resuming

A long training run you cannot resume is a liability. Lightning writes a checkpoint by default, but the default only keeps the most recent state. Almost every real project configures two callbacks: ModelCheckpoint to save good models, and EarlyStopping to stop when progress stalls.

from lightning.pytorch.callbacks import EarlyStopping, ModelCheckpoint

ckpt = ModelCheckpoint(
    dirpath="checkpoints/",
    filename="{epoch}-{val_loss:.3f}",
    monitor="val_loss",
    mode="min",
    save_top_k=3,
    save_last=True,
)
early = EarlyStopping(monitor="val_loss", patience=3, mode="min")

trainer = L.Trainer(max_epochs=30, callbacks=[ckpt, early])
trainer.fit(model, datamodule=dm)
print(ckpt.best_model_path)

What a checkpoint holds. A .ckpt file is not just weights. It contains the model's state_dict, the optimizer and scheduler states, the epoch and global step counters, the state of callbacks, the hyperparameters recorded by save_hyperparameters, and the Lightning version. That completeness is what makes true resuming possible.

The ModelCheckpoint arguments. monitor="val_loss" names a logged metric to rank checkpoints by. mode="min" means lower is better (use "max" for accuracy). save_top_k=3 keeps the three best and deletes the rest (-1 keeps all). save_last=True also writes a last.ckpt, the most recent state, which is what you resume from after a crash. The filename template can include logged metric names in braces, so files are named like epoch=4-val_loss=0.083.ckpt.

EarlyStopping. It watches a metric and stops training when it has not improved for patience checks. The word checks matters: patience counts validation runs, not epochs. With default settings validation runs once per epoch, so the two coincide, but they differ if you validate more or less often.

Loading a model back

model = LitClassifier.load_from_checkpoint(ckpt.best_model_path)
model.eval()

Two points trip people up. First, load_from_checkpoint is a class method that returns a new model. Calling it on an existing instance, model.load_from_checkpoint(path), builds a new model and discards it, leaving your instance unchanged. Second, you must call model.eval() yourself before inference, since you are outside the Trainer's control. Because we used save_hyperparameters(), you did not have to pass lr; it was restored from the file. You can override saved values by passing keyword arguments.

Resuming training

To continue an interrupted run with the optimizer state and counters intact, pass the checkpoint to fit:

trainer.fit(model, datamodule=dm, ckpt_path="last")

"last" and "best" are special values for checkpoints created earlier in the same session; you can also pass a file path. The old Trainer argument resume_from_checkpoint no longer exists; ckpt_path on fit replaced it. To save a checkpoint by hand at any moment, use trainer.save_checkpoint("manual.ckpt").

A security point you should know on day one

Checkpoint files are built on Python's pickle format, and loading an untrusted pickle can run arbitrary code. Treat a .ckpt file from a stranger like an executable. Since PyTorch 2.6, torch.load defaults to weights_only=True, which blocks most of that. Lightning 2.6.6, released on 10 September 2026, closed a further hole in load_from_checkpoint that could be exploited through crafted checkpoint metadata, and 2.6.5 was still affected. So: run 2.6.6 or newer, and never load checkpoints you do not trust.

You may see UnpicklingError: Weights only load failed when loading your own checkpoints. It happens when your saved hyperparameters contain custom Python objects. The cleanest fix is to keep hyperparameters to plain numbers, strings and lists. For files you fully trust, you can pass weights_only=False to load_from_checkpoint or to fit.

Hardware and precision: the Trainer flags you will change most

Because the Trainer owns the hardware, switching machines is a matter of arguments. You have already used accelerator="auto" and devices="auto". Here are the values behind them.

The accelerator is the type of hardware: "cpu", "gpu", "cuda", "mps", "tpu" or "auto". "gpu" resolves to CUDA on NVIDIA machines and MPS on Apple Silicon. devices is how many: an integer, a list of indices like [0, 1], "auto", or -1 for all.

L.Trainer(accelerator="cpu")                    # force the CPU
L.Trainer(accelerator="gpu", devices=1)         # one GPU
L.Trainer(accelerator="gpu", devices=[1])       # specifically GPU number 1
L.Trainer(accelerator="auto", devices="auto")   # whatever is available

Using more than one GPU switches on a strategy, which is how the work is distributed. The default strategy="auto" picks sensibly: ordinary data-parallel training (DDP) when you ask for several devices. Multi-GPU work has extra rules (the script runs once per process, and metrics need synchronising), and it is covered properly at the mid level. For now, know that the old arguments gpus=2, tpu_cores and num_processes were removed in version 2.0. The current spelling is accelerator="gpu", devices=2.

Precision controls the number format. The default is full 32-bit floats. Mixed precision runs most of the maths in 16 bits to save memory and time:

L.Trainer(precision="bf16-mixed")   # recent NVIDIA GPUs (Ampere or newer)
L.Trainer(precision="16-mixed")     # older GPUs; uses a gradient scaler

Teach yourself the string forms. Older material writes precision=16; the integer still maps to "16-mixed", but the strings say what is happening. Mixed precision rarely changes accuracy noticeably and often lets you use a larger batch or finish sooner. Start with the default while you debug, and add mixed precision once the model trains correctly.

A note for readers in the Gulf and Egypt who rent GPUs from a cloud provider: GPU instance availability and pricing differ by region, and some employers require data to stay in-country. The nice property of Lightning here is that none of this touches your model code. You develop on a CPU or a Mac, run on whatever region's GPU instance you are allowed to use, and change only accelerator, devices and precision.

Debugging flags that save hours

Training runs are slow, and a bug found at epoch 40 is expensive. The Trainer has flags that shrink a run so you can find bugs quickly. They are the most underused feature among beginners.

fast_dev_run=True runs one batch of training and one of validation, and stops. It disables loggers and checkpointing, so it leaves no files behind. Pass an integer, fast_dev_run=5, for five batches. Run it every time you change the module, before launching a real run.

limit_train_batches and limit_val_batches use only a fraction (a float such as 0.1) or a fixed number (an int such as 20) of the data. This is a full epoch loop on a tenth of the data, a good middle ground between a smoke test and a real run.

overfit_batches=10 trains repeatedly on just ten batches. A healthy model should drive the training loss close to zero on ten batches. If it cannot, something is wrong with the model, the loss or the labels, and no amount of extra data will help. This is one of the best sanity checks in deep learning.

detect_anomaly=True makes PyTorch report the operation that produced a NaN or infinite gradient. It slows training a lot, so use it only while hunting such a problem.

profiler="simple" prints a table of how long each hook took, which tells you whether time goes to data loading or computation.

# a quick, cheap run while developing
trainer = L.Trainer(max_epochs=2, limit_train_batches=0.1, limit_val_batches=0.1)

# can the model learn at all?
trainer = L.Trainer(max_epochs=50, overfit_batches=10)

Configuration: the Trainer arguments worth knowing

The Trainer has many arguments, and all of them are keyword-only. You will use a dozen in your first month. Here they are, grouped by what you are trying to do, with the defaults that matter.

Run length. max_epochs (set it; the default is 1000 if neither it nor max_steps is set), max_steps (stop after a number of optimizer steps), and max_time (a limit such as "00:02:00:00", meaning two hours, in days:hours:minutes:seconds form). Setting max_epochs=-1 means train until something else stops you, such as early stopping.

A realistic beginner configuration pulls several of these together:

trainer = L.Trainer(
    max_epochs=20,
    accelerator="auto",
    devices="auto",
    precision="32-true",
    gradient_clip_val=1.0,
    log_every_n_steps=10,
    default_root_dir="runs/mnist",
    callbacks=[ckpt, early],
)

You can also let a configuration file and command line drive these values using LightningCLI, which turns your module and DataModule into a command-line program with YAML configs. That is the subject of the Mid-level guide, so skip it for now.

Common errors and how to read them

Lightning's error messages are unusually informative. They usually name both the problem and the fix. The skill is reading them calmly. Here are the ones beginners meet most.

"No training_step() method defined." The Trainer found no training_step on your class. The cause is almost always a misspelling (training_stap) or indentation that put the method outside the class. The same message appears for a missing configure_optimizers. Fix the name.

"ModelCheckpoint(monitor='val_loss') could not find the monitored key in the returned metrics." The checkpoint callback wants a metric you never logged under that exact name. Check three things: that you call self.log("val_loss", ...), that the spelling matches character for character, and that validation actually ran (you passed a validation loader and did not set limit_val_batches=0). EarlyStopping raises a closely related error with the words "conditioned on metric which is not available", with the same causes.

"CUDAAccelerator can not run on your system since the accelerator is not available." You requested a GPU that PyTorch cannot see. Run python -c "import torch; print(torch.cuda.is_available())". If it prints False, you probably have a CPU-only PyTorch build. Reinstall PyTorch from pytorch.org with the right CUDA version, or use accelerator="auto".

"self.log(name, value) was called, but the tensor must have a single element." You logged a tensor with several values. Reduce it with .mean() or log elements separately.

"You are trying to self.log() but it is not managed by the Trainer control flow." You logged outside a Trainer-driven hook, for example in __init__. Move the call into a step or epoch hook.

"Trainer(strategy='ddp') is not compatible with an interactive environment." You tried multi-GPU in a Jupyter or Colab notebook. Either run the code as a script, or use strategy="ddp_notebook", which is notebook-compatible (and not available on Windows).

"strategies from the DDP family are not supported on the MPS accelerator." On a Mac you asked for several devices. Use one device on MPS.

"Weights only load failed" (UnpicklingError). Covered in the checkpoint section: your checkpoint contains custom Python objects. Keep hyperparameters simple, or for trusted files pass weights_only=False.

"The class ... requested by the checkpoint does not resolve to an already imported subclass." This is a safety check added in 2.6.6. Import the module that defines your model class before calling load_from_checkpoint.

CUDA out of memory. The batch does not fit on the GPU. Lower the batch size first. If you still need the larger effective batch, use accumulate_grad_batches, then precision="bf16-mixed".

Old tutorials: what changed, and what to write instead

Most Lightning content online was written for version 1.x. Copying from it is the most common source of beginner errors. Here is a translation table for the things you will meet. Lightning 2.0 removed them, and the current spelling is on the right.

Two more things changed more recently. Release 2.6.1 deprecated LightningModule.to_torchscript, because TorchScript itself is being phased out of PyTorch; for exporting a model, use torch.export.export() or to_onnx instead. And Python 3.9, which older machines may still run, is no longer supported.

When a snippet from the internet fails, check its date and its imports before anything else. If it imports pytorch_lightning and passes gpus=, it predates version 2.0, and the fix is the table above.

Putting it all together

Here is one small project that uses everything from this guide: a DataModule, a module with metrics, checkpointing, early stopping, logging, resuming and a final test. It is the template to copy for your next experiment. Save it as mnist_full.py and put MNISTDataModule from earlier in datamodule.py.

import torch
import torch.nn.functional as F
import torchmetrics
import lightning as L
from lightning.pytorch.callbacks import EarlyStopping, ModelCheckpoint
from lightning.pytorch.loggers import CSVLogger

from datamodule import MNISTDataModule


class LitMNIST(L.LightningModule):
    def __init__(self, lr: float = 1e-3, hidden: int = 128):
        super().__init__()
        self.save_hyperparameters()
        self.net = torch.nn.Sequential(
            torch.nn.Flatten(),
            torch.nn.Linear(28 * 28, hidden),
            torch.nn.ReLU(),
            torch.nn.Linear(hidden, 10),
        )
        self.val_acc = torchmetrics.classification.MulticlassAccuracy(num_classes=10)
        self.test_acc = torchmetrics.classification.MulticlassAccuracy(num_classes=10)

    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        self.log("train_loss", loss, on_step=False, on_epoch=True, prog_bar=True)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        self.log("val_loss", F.cross_entropy(logits, y), prog_bar=True)
        self.val_acc(logits, y)
        self.log("val_acc", self.val_acc, prog_bar=True)

    def test_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        self.log("test_loss", F.cross_entropy(logits, y))
        self.test_acc(logits, y)
        self.log("test_acc", self.test_acc)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)


def main():
    L.seed_everything(42, workers=True)
    dm = MNISTDataModule(batch_size=64)
    model = LitMNIST(lr=1e-3, hidden=128)

    ckpt = ModelCheckpoint(
        dirpath="checkpoints/",
        filename="{epoch}-{val_loss:.3f}",
        monitor="val_loss",
        mode="min",
        save_top_k=2,
        save_last=True,
    )
    early = EarlyStopping(monitor="val_loss", patience=3, mode="min")

    trainer = L.Trainer(
        max_epochs=20,
        accelerator="auto",
        devices="auto",
        logger=CSVLogger("logs/", name="mnist"),
        callbacks=[ckpt, early],
    )
    trainer.fit(model, datamodule=dm)
    trainer.test(model, datamodule=dm, ckpt_path="best")
    print("best checkpoint:", ckpt.best_model_path)


if __name__ == "__main__":
    main()

Notice the new piece: torchmetrics. Lightning installs it as a dependency. You create a metric object as an attribute of the module (self.val_acc), call it with predictions and targets, and pass the object to self.log. Lightning then handles accumulating across batches, averaging at epoch end, resetting between epochs, and (later, on many GPUs) combining across processes. That removes a lot of hand-written bookkeeping and a classic bug where accuracy is averaged wrongly across batches of different sizes.

Run it:

python mnist_full.py

Then walk through what happened, as a checklist for any Lightning run:

1. The sanity check ran two validation batches.
2. Each epoch trained, validated, logged val_loss and val_acc, and saved a checkpoint if val_loss was in the top two.
3. Training either reached 20 epochs or EarlyStopping ended it, and the last line of Trainer output says which.
4. trainer.test(..., ckpt_path="best") loaded the best checkpoint and printed test metrics.
5. logs/mnist/version_0/metrics.csv holds the numbers and checkpoints/ holds the models.

To reuse the trained model later without the Trainer:

from mnist_full import LitMNIST

model = LitMNIST.load_from_checkpoint("checkpoints/last.ckpt")
model.eval()
with torch.no_grad():
    logits = model(batch_of_images)

Importing the class first is not optional in 2.6.6: the loader checks that the checkpoint's class is already imported, as the error section explained.

What you can now do, and what comes next

You started this guide knowing PyTorch but not Lightning. You can now write a module, feed it data, log, checkpoint, resume, switch hardware, debug cheaply and read the common errors.

What comes next. The Mid-level guide builds on this with multi-GPU training with DDP, mixed precision in depth, LightningCLI and YAML configuration, more loggers and callbacks, the Tuner, and wiring Lightning into CI and containers. The Senior guide covers large-model strategies such as FSDP and DeepSpeed, fault tolerance on schedulers, security of the checkpoint supply chain, upgrades, and where Lightning stops being the right tool.

Neighbouring guides in this series help you around Lightning. For experiment tracking beyond a CSV file, read MLflow or Weights & Biases. To package a training job so it runs the same everywhere, read Docker. For scaling out to a cluster, see Kubernetes and Ray. For large-model training libraries that work with PyTorch, see DeepSpeed, and for fine-tuning language models, see PEFT and TRL.

The best next step is not more reading. Take a small dataset of your own, write a module and DataModule for it using the template above, and make it run with fast_dev_run=True first. When you can do that without looking at this guide, you are ready for the Mid-level one.

Sources

• PyTorch Lightning documentation (stable)
• Installation
• Versioning policy
• Trainer
• LightningModule
• LightningDataModule
• Basic checkpointing
• Early stopping
• Logging extensions
• Basic debugging
• Precision
• MPS (Apple Silicon) basics
• Migration guide to 2.x
• Release notes on GitHub
• PyPI: lightning

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about PyTorch Lightning. Mid-level reviews this plus its own material; Senior reviews all three. Everything here was checked against Lightning 2.6.6.

The thirty-second answer

PyTorch Lightning is a library on top of PyTorch that separates the research code (the model, the loss, the optimizer) from the engineering code (the training loop, devices, precision, checkpointing, logging). You put the research code in a LightningModule, hand it to a Trainer, and the Trainer runs the loops. The same module then runs on a CPU, one GPU or many GPUs by changing Trainer arguments, not model code.

Then the line that shows you have used it: "I never write the loop or call backward; I write what happens for one batch in training_step, return the loss, and the Trainer does the rest."

The model

The thing to be able to say aloud is the direction of control. In plain PyTorch you call the model in your own loop. In Lightning the Trainer calls your methods. That is why you never write for batch in loader and never call training_step yourself.

Commands and calls to have ready

You are not expected to memorise the API, but an interviewer will often ask you to write a small example on a whiteboard or shared editor. These are the pieces to be able to produce from memory.

pip install lightning                      # recommended package
pip install "lightning[extra]"             # rich, CLI tooling and plotting helpers
python -c "import lightning as L, torch; print(L.__version__, torch.__version__, torch.cuda.is_available())"
tensorboard --logdir lightning_logs/       # view curves

import lightning as L

class LitModel(L.LightningModule):
    def __init__(self, lr=1e-3):
        super().__init__()
        self.save_hyperparameters()
        self.net = ...                       # your layers

    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = loss_fn(self(x), y)
        self.log("train_loss", loss)
        return loss

    def validation_step(self, batch, batch_idx):
        ...
        self.log("val_loss", val_loss)

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=self.hparams.lr)

trainer = L.Trainer(max_epochs=5, accelerator="auto", devices="auto")
trainer.fit(model, datamodule=dm)            # or train_dataloaders=..., val_dataloaders=...
trainer.test(model, datamodule=dm, ckpt_path="best")

And the debug flags, which interviewers like because they show practical habit: fast_dev_run=True, limit_train_batches=0.1, overfit_batches=10, detect_anomaly=True, profiler="simple".

The classic traps, and why they happen

Calling .cuda() inside the module. The Trainer already moves the model and the batches. Hard-coded device calls break CPU runs and conflict with multi-GPU. Use self.device for new tensors.

Mixing pytorch_lightning and lightning. They are two separate namespaces with separate classes. A module from one is rejected by the Trainer of the other. Pick import lightning as L and stay there.

Copying a 1.x tutorial. gpus=2, resume_from_checkpoint=, training_epoch_end(outputs) and precision=16 meanings changed or were removed in 2.0. The current forms are accelerator="gpu", devices=2, fit(..., ckpt_path=...), on_train_epoch_end() and precision="16-mixed".

A monitored name that was never logged. ModelCheckpoint(monitor="val_loss") reads trainer.callback_metrics, which is filled by self.log. If the string differs, or validation never runs, you get a MisconfigurationException about the monitored key.

Assuming load_from_checkpoint loads in place. It is a class method that returns a new instance. Calling it on an existing model and ignoring the return value does nothing. You also have to call model.eval() afterwards.

Setting state in prepare_data. It runs on one process per machine, so state assigned there is missing elsewhere. Downloads belong in prepare_data; splits and datasets belong in setup.

Forgetting to set max_epochs. With neither max_epochs nor max_steps set, the Trainer defaults to 1000 epochs.

Treating a checkpoint as harmless data. A .ckpt file is pickle-based. Loading an untrusted one can execute code. Release 2.6.6 fixed a vulnerability in load_from_checkpoint, and 2.6.5 was still affected.

Common interview questions

1. What problem does PyTorch Lightning solve?

Training a PyTorch model needs a lot of code that is not about the model: epoch and batch loops, zero_grad, backward, step, moving data to devices, switching between train and eval mode, saving and loading checkpoints, logging, mixed precision, distributed training. That code is repetitive and easy to get subtly wrong, and every change of hardware means editing it. Lightning moves it into a tested Trainer so that your code contains only the model, the loss, the optimizer and the data. The benefits I would name are less boilerplate, fewer loop bugs, the same code on any hardware, and a common structure that makes projects readable to other people.

2. Is Lightning a different framework from PyTorch?

No. A LightningModule is a subclass of torch.nn.Module, and everything inside it is ordinary PyTorch: layers, losses, optimizers, tensors. Lightning organises the training code and runs it; it does not replace PyTorch. If I stop using the Trainer, the same module can still be called as a normal nn.Module.

3. What are the minimum methods a LightningModule needs for fit?

training_step(batch, batch_idx) and configure_optimizers(), plus a training dataloader from somewhere (the module, a DataModule, or an argument to fit). validation_step, test_step, predict_step and forward are optional. Missing a required method gives a MisconfigurationException naming it, usually because of a typo.

4. What should training_step return?

The loss tensor, or a dictionary with a "loss" key. With automatic optimization, Lightning then zeroes gradients, calls backward and steps the optimizer. Returning None skips that batch. If I need several optimizers, as with a GAN, I switch to manual optimization by setting self.automatic_optimization = False and calling self.manual_backward myself.

5. How does self.log work, and what are its defaults?

It sends a value to the logger and also stores it in trainer.callback_metrics, which callbacks like ModelCheckpoint and EarlyStopping read. The defaults depend on the hook: in training_step it logs every step and does not aggregate per epoch; in validation and test steps it logs once per epoch, averaged. I can override with on_step and on_epoch. prog_bar=True also shows it on the progress bar. For several values I use self.log_dict, because passing a dictionary to self.log is not supported. Under multi-GPU I would add sync_dist=True for epoch-level validation metrics so they are reduced across processes.

6. What is the difference between prepare_data and setup in a DataModule?

prepare_data runs once per machine (on one process) and is for downloads and writing prepared data to disk. I must not assign state there. setup(stage) runs on every process and builds the datasets and splits; stage is "fit", "validate", "test" or "predict". The reason for the split is multi-process training: you do not want eight processes downloading the same file, but you do want every process to have its own dataset objects.

7. How do you run the same code on CPU, one GPU and several GPUs?

Only by changing Trainer arguments. accelerator="auto" and devices="auto" use whatever is available. accelerator="gpu", devices=1 forces one GPU, and devices=4 uses four, with the default strategy choosing DDP. Model code never mentions devices. The old gpus= argument was removed in 2.0. For several GPUs the script must be guarded by if __name__ == "__main__": because processes re-run it, and in a notebook I would use strategy="ddp_notebook".

8. How do you save the best model and stop training when it stops improving?

Two callbacks. ModelCheckpoint(monitor="val_loss", mode="min", save_top_k=3, save_last=True) keeps the best three checkpoints and a last.ckpt. EarlyStopping(monitor="val_loss", patience=3, mode="min") stops when the metric has not improved for three validation checks. I would point out that patience counts validation checks, not epochs, and that both need val_loss to be logged with exactly that name.

9. What is inside a checkpoint, and how do you resume from one?

The model's state_dict, optimizer and scheduler states, epoch and global step, callback states, the hyperparameters from save_hyperparameters, and the Lightning version. To resume with all of that restored, I call trainer.fit(model, datamodule=dm, ckpt_path="last"). That is different from loading just the weights with load_from_checkpoint, which is for inference or fine-tuning and does not restore the optimizer or the epoch counter. The old resume_from_checkpoint Trainer argument is gone.

10. What does save_hyperparameters() do and why use it?

It stores the arguments of __init__ in self.hparams and writes them into every checkpoint and logger. As a result Model.load_from_checkpoint(path) can rebuild the model without me passing the constructor arguments again. For arguments that are whole nn.Module objects I would use ignore=[...]. I would also avoid passing secrets as constructor arguments, because they would be written into checkpoints and logs.

11. How do you debug a training script quickly?

I use the cheapest tool first. fast_dev_run=True runs one batch of training and validation and tells me within seconds whether the code works. overfit_batches=10 checks the model can learn at all: if it cannot drive the loss down on ten batches, there is a bug in the model, loss or labels. limit_train_batches=0.1 runs a real epoch on a slice. detect_anomaly=True finds the operation behind a NaN, and profiler="simple" shows where time goes. Only after those do I launch the full run.

12. Why do validation and test exist as separate phases?

Validation metrics are looked at many times while tuning, so the choices overfit to that set. The test set is held back and used rarely, ideally once, to estimate final performance. In Lightning they are different methods (validate and test) with different hooks (validation_step and test_step) so the separation is explicit. For the final evaluation I use ckpt_path="best" so the test uses the best checkpoint rather than the last weights.

13. What does the Trainer do before training starts, and why?

It runs a sanity check: two validation batches by default (num_sanity_val_steps=2). The purpose is to expose a bug in validation_step immediately instead of at the end of the first epoch.

14. What is mixed precision and how do you turn it on?

Mixed precision does most arithmetic in 16-bit floats to save memory and time while keeping some operations in 32-bit for stability. In Lightning it is one argument: precision="bf16-mixed" on recent NVIDIA GPUs, or "16-mixed" on older ones, which uses a gradient scaler. The default is "32-true". I would turn it on after the model trains correctly in full precision, not while debugging.

15. A teammate's tutorial code fails with an unexpected-keyword error on Trainer(gpus=1). What is wrong?

That code predates Lightning 2.0, where gpus, tpu_cores and num_processes were removed. The fix is Trainer(accelerator="gpu", devices=1). I would also check the rest of the snippet for other 1.x leftovers: pytorch_lightning imports, *_epoch_end(outputs) hooks, resume_from_checkpoint, and precision=16.

16. How is Lightning different from Fabric?

The Trainer owns the loop and you fill in hooks. Fabric is a lower-level tool from the same package where you keep your own training loop and let Fabric handle devices, precision and distribution (fabric.setup(model, optimizer), fabric.backward(loss)). I choose the Trainer for standard training and Fabric when I need a loop the Trainer cannot express. Custom loop objects inside the Trainer are no longer supported.

17. How would you track experiments?

self.log is the single entry point, and the logger decides where it goes. The default is TensorBoard if installed, otherwise a CSV logger. I can pass logger=[...] with several, or use MLFlowLogger, WandbLogger or CometLogger. The Neptune logger was removed in 2.6.4, so older tutorials using it no longer work.

18. What would you check if a metric curve has very few points?

log_every_n_steps defaults to 50, so a small dataset with fewer than 50 batches per epoch logs rarely, and Lightning warns about it. Lowering log_every_n_steps fixes it. I would also check whether I am logging per step or per epoch.

What a strong answer covers that a weak one misses

Two things also raise a candidate's standing. One is honesty about limits: Lightning speeds up your development and not your model. It does not make a slow DataLoader fast. The other is knowing the neighbours: how a Lightning job gets packaged with Docker, tracked with MLflow or Weights & Biases, and scaled with Kubernetes.

A whiteboard exercise to rehearse

Interviewers sometimes give a small task: "Here is a PyTorch loop. Convert it to Lightning." The conversion is mechanical, and being able to narrate it is worth practising. Take the loop and move things in this order.

The model's layers go into __init__, after super().__init__(). The forward pass goes into forward. The body of the inner loop becomes training_step: keep the forward pass and the loss, return the loss, and delete the .to(device), zero_grad, backward and step lines. The validation loop's body becomes validation_step, with the eval() and no_grad lines deleted and self.log calls replacing the manual averaging and printing. The optimizer construction goes into configure_optimizers. The loaders stay as they are, or become a DataModule. The two outer for statements are replaced by a single trainer.fit(...), and torch.save is replaced by a ModelCheckpoint callback.

A strong candidate also says what they would test after converting: run fast_dev_run=True to check wiring, then compare the first epoch's loss against the original loop to make sure nothing changed numerically, and then check the saved checkpoint loads with load_from_checkpoint. Mentioning the comparison with the original loop is the detail that signals engineering maturity, because a refactor that silently changes the loss is worse than no refactor.

If asked what you would not do, a good answer is "I would not hide a bug by disabling the sanity check or the warnings." The sanity check and the PossibleUserWarning messages exist to surface exactly the mistakes beginners make, such as too few DataLoader workers or modules left in the wrong mode.

Questions to ask the interviewer

Candidates who ask a few sensible questions of their own stand out. For a role that uses Lightning, good ones are: which Lightning and PyTorch versions the team pins and how upgrades are handled; whether they train from LightningCLI configs or hand-written scripts; where checkpoints and logs are stored; and how they treat checkpoints from outside the team. These show that you think of training as part of a system, not only as a notebook.

Sixty-second self-test

Cover the answers and say them aloud.

1. In one sentence, what is the split between a LightningModule and a Trainer?
2. Which two methods must a module define for fit?
3. What does training_step return, and what does Lightning do with it?
4. Why do prepare_data and setup exist separately?
5. Which Trainer arguments replace gpus=2?
6. What are two callbacks you would add to nearly every real project, and what does each need to be logged?
7. Name three debug flags, and say what each tells you.
8. Is load_from_checkpoint an instance method or a class method, and what must you call afterwards?
9. How do you resume a run with the optimizer state intact?
10. Why is a .ckpt file from a stranger risky, and which Lightning version is the minimum safe one today?

Part one of three. Most beginner problems with PyTorch Lightning come from three habits: copying code from a pre-2.0 tutorial, forgetting that the Trainer, not you, calls your methods, and trusting a run you have not first shrunk. Start with the table, then the detailed fixes, the habits and the checklist. Everything was checked against Lightning 2.6.6.

Common errors at this level

Each one is explained below, with what you see, why it happens and what to do.

Mistake 1: copying a pre-2.0 tutorial

Symptom. You paste a snippet and get an unexpected keyword argument on Trainer, or a hook is silently never called.

Cause. Most Lightning material online targets version 1.x. Lightning 2.0 removed gpus=, tpu_cores=, num_processes=, resume_from_checkpoint=, the *_epoch_end(outputs) hooks and trainer.tune(). The silent case is the dangerous one: a training_epoch_end method you copy over is simply never called anymore, so your epoch-level logic quietly does nothing.

Fix. Check the date and the imports of any snippet first. If it says import pytorch_lightning as pl and Trainer(gpus=1), translate it: accelerator="gpu", devices=1; fit(..., ckpt_path=...); on_train_epoch_end(self) without the outputs argument; precision="16-mixed" instead of precision=16. Run the result with fast_dev_run=True.

Mistake 2: moving tensors by hand

Symptom. Your code works on a laptop and crashes on a GPU machine, or the reverse. Or you get "expected all tensors to be on the same device".

Cause. You wrote .cuda() or .to("cuda") inside the module or the data pipeline. The Trainer already places the model and every batch on the right device, so hard-coded calls either fail on a CPU or fight the Trainer on several GPUs.

Fix. Delete the hard-coded device calls. When you must create a tensor inside the module, use self.device: torch.zeros(3, device=self.device) or mask.to(self.device).

Mistake 3: a monitored name that does not exist

Symptom. A MisconfigurationException at the end of the first epoch saying ModelCheckpoint(monitor='val_loss') could not find the monitored key in the returned metrics. EarlyStopping raises a close relative saying it is conditioned on a metric that is not available.

Cause. Callbacks read trainer.callback_metrics, which self.log fills. Three things empty that dictionary: a spelling difference (val_loss versus val_losss), logging under a different name (valid_loss), or validation not running at all (no validation loader, or limit_val_batches=0).

Fix. Read the error: it lists the metric names that exist. Match your monitor to one of them. Then confirm validation runs by looking for the validation part of the progress bar.

Mistake 4: expecting load_from_checkpoint to load in place

Symptom. You load a checkpoint, run inference, and get the predictions of an untrained model.

Cause. load_from_checkpoint is a class method that returns a new object. The line model.load_from_checkpoint("best.ckpt") creates a loaded model and throws it away, leaving model as it was.

Fix. Assign the result: model = LitModel.load_from_checkpoint("best.ckpt"). Then call model.eval() yourself, because outside the Trainer nothing switches the model to evaluation mode for you. If you only want to continue training, do not use this at all; pass ckpt_path="last" to fit.

Mistake 5: the Python object problem in checkpoints

Symptom. Loading a checkpoint fails with UnpicklingError: Weights only load failed ... Unsupported global.

Cause. From PyTorch 2.6, torch.load defaults to weights_only=True, which refuses to unpickle arbitrary Python objects. Your checkpoint contains one, usually because a hyperparameter passed to __init__ was a custom class, a dataclass or a whole nn.Module.

Fix. Keep hyperparameters to numbers, strings, booleans, lists and dictionaries of those. Exclude module arguments with self.save_hyperparameters(ignore=["backbone"]). For a file you produced yourself and fully trust, weights_only=False is available on load_from_checkpoint and on fit. Never use it on a file from someone else.

Mistake 6: forgetting to import the model class before loading

Symptom. On 2.6.6, loading a checkpoint fails with a ValueError saying the class requested by the checkpoint does not resolve to an already imported subclass, and was blocked to prevent arbitrary code execution.

Cause. A security change in 2.6.6. Loading now only accepts a class that you have already imported, so a crafted checkpoint cannot make Lightning import and run arbitrary code.

Fix. Import the module that defines your model before calling load_from_checkpoint, for example from my_project.model import LitModel.

Mistake 7: not setting an epoch limit

Symptom. A "quick test" is still running an hour later.

Cause. If neither max_epochs nor max_steps is set, the Trainer defaults to 1000 epochs.

Fix. Always set max_epochs explicitly, and for experiments add max_time="00:00:30:00" (days, hours, minutes, seconds) as a safety net. max_epochs=-1 means unlimited, and you should only use it together with early stopping.

Mistake 8: trusting the full run before the small one

Symptom. A bug in validation_step or the loss shows up an hour in. Or the loss never falls and you cannot tell why.

Cause. You launched the full-size run first.

Fix. Use the debugging ladder: fast_dev_run=True, then overfit_batches=10, then limit_train_batches=0.1, then the full run. If the model cannot reach near-zero training loss on ten batches, the problem is in the code (the loss, the labels, the learning rate), not the amount of data. The next section turns this into a habit.

Mistake 9: a slow DataLoader that looks like a slow model

Symptom. Epochs take far longer than the model size suggests, and the GPU utilisation is low. Lightning may print a PossibleUserWarning that the dataloader does not have many workers.

Cause. num_workers=0 makes the main process load and transform every batch, and the GPU waits. Heavy transforms on the CPU make it worse.

Fix. Set num_workers to a small positive number such as 2 or 4 and measure. Add persistent_workers=True (it requires num_workers above zero) and pin_memory=True for GPU runs. Use profiler="simple" to see whether time is spent in the data hooks or in the step. On Windows and macOS, wrap the entry point in if __name__ == "__main__":, or workers will fail to start.

Mistake 10: logging at the wrong granularity

Symptom. Training loss is a jagged mess, or validation shows one point per epoch while training shows hundreds, and you cannot compare them.

Cause. The on_step and on_epoch defaults differ by hook: training logs each step, validation logs once per epoch, averaged.

Fix. Choose deliberately. For a smooth training curve comparable to validation, self.log("train_loss", loss, on_step=False, on_epoch=True). For debugging, keep per-step logging and lower log_every_n_steps if your epochs have fewer than 50 batches.

Mistake 11: confusing validation and test

Symptom. The test accuracy looks great, and then the model disappoints in the real world.

Cause. You tuned hyperparameters against the test set, so it stopped being an honest estimate.

Fix. Tune on the validation set. Touch the test set rarely, ideally once, with trainer.test(model, datamodule=dm, ckpt_path="best").

Mistake 12: loading a checkpoint you did not make

Symptom. None, until something goes badly wrong.

Cause. A .ckpt file is built on Python's pickle format. Loading an untrusted one can execute attacker-chosen code. A vulnerability in the Lightning loader itself was fixed in 2.6.6, and 2.6.5 was still affected.

Fix. Run Lightning 2.6.6 or newer. Treat checkpoints from outside your team like executables: do not load them on a machine that holds credentials. Prefer receiving plain weights (a state dict) from others, and see the Senior track for the full supply-chain picture.

Habits worth adopting on day one

These cost almost nothing and prevent most of the problems above.

Start every change with fast_dev_run=True

Make it a reflex. After editing the module, the DataModule or a callback, run L.Trainer(fast_dev_run=True).fit(model, datamodule=dm). It runs one training and one validation batch, writes nothing, and fails fast. Teams that do this in CI avoid burning GPU hours on typos.

# a fast smoke test you can run in CI on a CPU machine
trainer = L.Trainer(fast_dev_run=True, accelerator="cpu", logger=False)
trainer.fit(model, datamodule=dm)

Seed, and say so

Call L.seed_everything(42, workers=True) at the top of main. It seeds Python, NumPy, PyTorch and the DataLoader workers. Add deterministic=True to the Trainer when you need runs to repeat as closely as possible, knowing it may be slower. Write the seed in your experiment notes; "I got 94 percent once" is not a result.

Call save_hyperparameters() first

Put it right after super().__init__(). Everything passed to the constructor is then in the checkpoint and the logs, and load_from_checkpoint needs no arguments. Use ignore=[...] for objects that are not plain values, and never pass secrets as constructor arguments, because they would be written to every checkpoint.

Keep forward for inference

Put prediction logic in forward and call self(x) from your step methods. Then the training path and the serving path are literally the same code, and you remove a whole family of train-versus-serve discrepancies.

Use torchmetrics for metrics that are not simple averages

Accuracy computed per batch and then averaged is wrong when batches differ in size. Use a metric object, self.val_acc = torchmetrics.classification.MulticlassAccuracy(num_classes=10), call it in the step, and log the object: self.log("val_acc", self.val_acc). Lightning handles accumulation, resetting and, on several GPUs, synchronisation. It installs torchmetrics for you.

Always configure checkpointing and early stopping

The default checkpoint saves only the latest state. Set ModelCheckpoint(monitor=..., mode=..., save_top_k=..., save_last=True) so that you keep the best model and can resume after a crash. Add EarlyStopping so a run that has stopped improving does not use up your time or your budget.

Pin versions together

Lightning does not follow semantic versioning: minor releases can remove deprecated features. In requirements.txt pin lightning, torch and torchmetrics together, for example lightning==2.6.6. Upgrade deliberately, and run your tests with deprecation warnings visible, since LightningDeprecationWarning messages are the advance notice that something is about to be removed.

Look at the whole log, not just the bar

Read the lines the Trainer prints at the start: which hardware it found, the parameter count, and the reason training stopped at the end. A model with ten times the parameters you expected, or GPU available: False when you wanted a GPU, are both visible there in the first seconds.

Think about where your data may live

If your employer is in the Gulf or Egypt and your data must stay in-country, check where your cloud GPU instances and your storage live before you start. Lightning itself is only a library and sends no data anywhere; what matters is the region of the machine running the script and of any hosted logger you choose. A local CSVLogger or a self-hosted tracker keeps everything on your side.

Practice cards

Work through these in order. Each takes ten to twenty minutes and builds on the one before.

Reading output without panic

Beginners often treat a wall of text as a single failure. It rarely is. A few patterns make a Lightning log quick to triage.

Warnings are not errors. Lightning prints many warnings, and most are advice. A PossibleUserWarning about few DataLoader workers, a note that TensorBoard is missing so a CSV logger is used instead, and a hint about matrix multiplication precision on newer NVIDIA GPUs are all optional improvements. Read them once, decide, and move on. But do not suppress them wholesale: they exist to catch the mistakes this guide lists.

Errors come with the fix in the message. Lightning's MisconfigurationException and ValueError messages normally include a hint such as "Did you call log('val_loss', value) in the LightningModule?" or "use ddp_notebook". Read the final lines of the traceback before searching the web.

The last line explains the stop. After training, the Trainer says why it stopped, such as max_epochs=3 reached. If it stopped earlier than expected, early stopping or a max_time limit fired, and the log says which.

CUDA out of memory. The reliable order is: reduce the batch size, then use precision="bf16-mixed", then accumulate_grad_batches to recover the effective batch size. Larger changes (sharding the model across GPUs) belong to the mid and senior levels.

Checklist

Before you launch a real run, check each box.

• I import lightning as L and nothing from pytorch_lightning.
• lightning, torch and torchmetrics versions are pinned, and Lightning is 2.6.6 or newer.
• The module calls super().__init__() and then self.save_hyperparameters().
• There are no .cuda() or .to(device) calls inside the module.
• training_step returns the loss, and validation_step logs val_loss under a name I reuse exactly.
• max_epochs (or max_steps) is set, with max_time as a safety net for long jobs.
• ModelCheckpoint monitors a logged metric with the right mode, and save_last=True is set.
• EarlyStopping is configured, and I know that patience counts validation checks.
• I ran fast_dev_run=True after my last code change.
• I checked that overfit_batches=10 drives the training loss close to zero.
• The entry point is under if __name__ == "__main__":.
• num_workers is above zero, and I know my data loading is not the bottleneck.
• The Trainer's hardware line shows the accelerator I intended.
• I tune on the validation set and use the test set once, with ckpt_path="best".
• Any checkpoint I load was produced by me or by someone I trust.

When you can tick all fifteen without looking anything up, you have the Beginner level. Continue to the Mid-level track for multi-GPU training, configuration files and production patterns, and meanwhile read Docker and MLflow to see how a training script like this one is packaged and tracked in a team setting.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. It targets Lightning 2.6.6, the current release, on Python 3.10 or newer. Where you already know the basics, we go straight to the depth.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a real project in a few minutes.

What the Trainer actually does

Beginner treated trainer.fit() as a black box that trains. At work you will need to predict its behaviour, so here is the skeleton. Lightning is a library, not a service: there is no daemon and no server. Your script starts, builds a Trainer, and the Trainer runs a set of nested loops in your process. When you use several GPUs, every process runs the same script and the same loops; this style is called SPMD, single program multiple data.

The fit loop runs in this order. First the strategy sets up the environment and wraps your model. Then setup hooks fire on the datamodule and the module, configure_optimizers is called, and a sanity check runs num_sanity_val_steps validation batches (default 2) so that a broken validation step fails in seconds rather than after an hour of training. Training then proceeds as epochs of batches, with validation inserted at the cadence you set through check_val_every_n_epoch and val_check_interval. Callbacks and the module share hook names, and the module's hook runs after the callbacks' for most pairs, so a callback sees the state just before your module's code.

Three numbers are easy to confuse. A batch index counts batches within an epoch. The epoch counts passes over the data. global_step counts optimizer steps, which is not the same as batches: with accumulate_grad_batches=4, four batches produce one optimizer step and global_step advances by one. This matters whenever a setting is expressed in steps, such as max_steps, every_n_train_steps for checkpoints, or a scheduler with interval="step". It also matters under multi-GPU, because each optimizer step consumes a batch from every device.

A useful habit is to learn the hook order by printing it rather than memorising it. A tiny callback that prints each hook name tells you more than a table, and the official hooks page lists the full order for every loop.

import lightning as L
from lightning.pytorch.callbacks import LambdaCallback

trace = LambdaCallback(
    on_train_epoch_start=lambda trainer, pl_module: print("train epoch start"),
    on_train_batch_end=lambda trainer, pl_module, outputs, batch, batch_idx:
        print("batch", batch_idx, "global_step", trainer.global_step),
    on_validation_epoch_end=lambda trainer, pl_module: print("val epoch end"),
)
L.Trainer(max_epochs=1, limit_train_batches=4, limit_val_batches=2,
          callbacks=[trace], logger=False, enable_checkpointing=False).fit(model, datamodule=dm)

The limit_*_batches flags are your friend while exploring: they take a float fraction or an integer count, and they let you see a whole epoch's lifecycle in a few seconds. One mid-level subtlety: since 2.6, val_check_interval also accepts a time value such as "00:01:00:00" for one hour, a timedelta, or a dict. When it is not a float fraction, you must also set check_val_every_n_epoch=None. That is the right tool for long jobs where an epoch takes many hours and you want validation, and therefore checkpoint decisions, at a predictable cadence.

Logging done correctly

Beginner showed self.log("val_loss", loss) and moved on. Most of the strange numbers on a dashboard come from not knowing what that call does between your code and the logger.

The signature has many arguments, but four decide behaviour: on_step, on_epoch, reduce_fx and sync_dist. The defaults depend on the hook. Inside training_step the default is on_step=True, on_epoch=False, so you get one value per logging interval. Inside validation_step and test_step the default is on_step=False, on_epoch=True, so Lightning accumulates values across batches and reduces them at the end of the epoch. The default reduction is the mean, and it is a batch-size-weighted mean when Lightning can infer the batch size. If your batches are not simple tensors, for example a dict or a custom object, Lightning guesses and may warn; pass batch_size= explicitly to be safe.

Logged values land in two places. They are sent to every logger, subject to log_every_n_steps (default 50) for step-level values, and they are stored in trainer.callback_metrics. The second place matters more than it looks: ModelCheckpoint(monitor="val_loss") and EarlyStopping(monitor="val_loss") read from callback_metrics. If the name you monitor was never logged, you get the error that every Lightning user meets once: MisconfigurationException: ModelCheckpoint(monitor='val_loss') could not find the monitored key in the returned metrics. The causes are a typo, logging under a different name, or validation never running because limit_val_batches=0 or no validation loader exists.

sync_dist is the setting that separates correct multi-GPU numbers from plausible-looking wrong ones. Under DDP each process sees only its own shard of the validation set. A value logged with on_epoch=True is averaged over that process's batches only, and then the logger shows rank 0's number. With sync_dist=True Lightning reduces across processes before logging. Use it for epoch-level validation and test metrics. Do not turn it on for every per-step training value; it triggers a collective operation on each call, which costs a communication round per step.

def validation_step(self, batch, batch_idx):
    x, y = batch
    logits = self(x)
    loss = F.cross_entropy(logits, y)
    # reduced across all GPUs, available to ModelCheckpoint as "val_loss"
    self.log("val_loss", loss, prog_bar=True, sync_dist=True, batch_size=x.size(0))
    self.val_acc(logits, y)              # a torchmetrics object
    self.log("val_acc", self.val_acc)    # log the object, not a computed tensor

For accuracy, F1 and similar, use torchmetrics instead of averaging per-batch values. A mean of per-batch accuracies is wrong when the last batch is smaller, and a mean of per-batch F1 scores is wrong in general. Create the metric as an attribute in __init__, for example self.val_acc = torchmetrics.classification.MulticlassAccuracy(num_classes=10), call it in the step and log the metric object itself. Lightning then handles accumulation, cross-process synchronisation and reset at the right time. Because the metric is an nn.Module attribute, it also moves to the right device with the model, so you never call .to(device) on it.

A few failure modes recur. Logging a tensor with more than one element raises an error telling you to log value.mean(). Logging a nested dictionary is rejected, so use self.log_dict with a flat dict. Calling self.log outside a Trainer-driven hook, such as in __init__ or when you call model(x) by hand, raises "You are trying to self.log() but it is not managed by the Trainer control flow". Logging inside a predict hook is not supported at all. And never call .item() or .cpu() on a logged tensor inside the step: it forces a GPU synchronisation on every batch and slows training for no benefit, because Lightning detaches and moves values for you.

When you need to accumulate outputs across a whole epoch, for example to compute a confusion matrix, the pattern since Lightning 2.0 is explicit. The old validation_epoch_end(outputs) hooks were removed. Append to a list attribute in the step, process it in on_validation_epoch_end, then clear it.

def on_validation_epoch_start(self):
    self.val_preds = []

def validation_step(self, batch, batch_idx):
    x, y = batch
    self.val_preds.append(self(x).argmax(dim=1))

def on_validation_epoch_end(self):
    preds = torch.cat(self.val_preds)
    self.log("val_pred_mean", preds.float().mean(), sync_dist=True)
    self.val_preds.clear()   # forgetting this leaks memory across epochs

Under DDP, remember that the list holds only this process's predictions. If you need global statistics, gather with self.all_gather(preds) or prefer a torchmetrics metric that does the synchronisation for you.

Data pipelines that keep the GPU fed

The most common reason a training job is slow is not the model. The GPU is idle while the CPU prepares the next batch. Lightning cannot fix a slow DataLoader, but it will tell you about one: the PossibleUserWarning "The 'train_dataloader' does not have many workers which may be a bottleneck" appears when num_workers is low, and a second warning suggests persistent_workers=True. Treat both as real advice.

The DataLoader settings that matter are num_workers, pin_memory=True for CUDA, persistent_workers=True when num_workers > 0 so workers survive between epochs, and prefetch_factor. Start with num_workers equal to the CPU cores per GPU that you have available, measure, and stop increasing when throughput stops improving. On a shared node, every worker also multiplies your memory use, because each worker holds its own copy of anything the dataset keeps in Python objects.

The datamodule is where you encode which code runs where, and the rule has real consequences in a multi-process job. prepare_data() runs on one process per node (local rank 0; or global rank 0 if you set prepare_data_per_node=False). Use it for work with side effects on disk: downloading, extracting, tokenising to files. Never assign state there, such as self.train_ds = ..., because the other processes will not see it. setup(stage) runs on every process, with stage equal to "fit", "validate", "test" or "predict". Build datasets and splits there. Make the split deterministic: use a seeded generator such as torch.Generator().manual_seed(42) in random_split, because every rank must compute the same split or the ranks will train and validate on overlapping data.

class TextDataModule(L.LightningDataModule):
    def __init__(self, data_dir: str, batch_size: int = 64, num_workers: int = 8):
        super().__init__()
        self.save_hyperparameters()

    def prepare_data(self):
        # one process per node: heavy, idempotent, side effects only
        tokenize_to_disk(self.hparams.data_dir)

    def setup(self, stage: str):
        # every process: build datasets, deterministic split
        full = load_tokenized(self.hparams.data_dir)
        gen = torch.Generator().manual_seed(42)
        self.train_ds, self.val_ds = torch.utils.data.random_split(full, [0.9, 0.1], generator=gen)

    def train_dataloader(self):
        return torch.utils.data.DataLoader(
            self.train_ds, batch_size=self.hparams.batch_size, shuffle=True,
            num_workers=self.hparams.num_workers, pin_memory=True, persistent_workers=True)

    def val_dataloader(self):
        return torch.utils.data.DataLoader(
            self.val_ds, batch_size=self.hparams.batch_size, num_workers=self.hparams.num_workers,
            pin_memory=True, persistent_workers=True)

Under multi-GPU training, Lightning replaces your sampler with a DistributedSampler automatically because use_distributed_sampler=True by default. Each process then sees a different slice of the data. Two consequences follow. If you pass your own sampler, for example a weighted sampler for class imbalance, the automatic replacement can discard your logic, so you must read the documentation for the case or set use_distributed_sampler=False and write a distributed-aware sampler yourself. And a DistributedSampler pads the dataset so every rank gets the same number of samples, which means a few samples are seen twice in evaluation; for exact final test numbers, evaluate on a single device.

Seeding deserves one sentence of precision. L.seed_everything(42, workers=True) seeds Python, NumPy and torch and sets environment variables so DataLoader workers are seeded as well. Together with Trainer(deterministic=True) it gives repeatable runs on the same hardware, at a speed cost for some operations. Different GPU models, or different numbers of devices, will still give different numbers, because the order of floating point reductions changes.

Finally, if your data are too large for local disks, the Lightning ecosystem has a streaming dataset library called LitData, which sits in lightning.data territory. Check its documentation before relying on it, as it moves independently of the Trainer.

Optimization: automatic, manual, and everything between

Under automatic optimization, the default, your training_step returns a loss and Lightning does the rest: it calls zero_grad, backward, gradient clipping, optimizer.step and the scheduler step. Three Trainer flags shape this: accumulate_grad_batches, gradient_clip_val and gradient_clip_algorithm ("norm" by default, or "value").

Accumulation trades memory for batch size. If a per-device batch of 32 is all that fits and you want an effective 256, the effective batch is per-device batch × devices × num_nodes × accumulate_grad_batches. When you change any of these, rescale the learning rate or you have silently changed the experiment. Remember that a scheduler with interval="step" counts optimizer steps, so a warmup length of 1,000 means 1,000 optimizer steps, which is 1,000 times the accumulation factor in batches.

configure_optimizers can return several shapes. The one you will use most at work is the dictionary form, which attaches a scheduler with explicit behaviour.

def configure_optimizers(self):
    opt = torch.optim.AdamW(self.parameters(), lr=self.hparams.lr, weight_decay=self.hparams.wd)
    sched = torch.optim.lr_scheduler.OneCycleLR(
        opt, max_lr=self.hparams.lr,
        total_steps=self.trainer.estimated_stepping_batches,
    )
    return {"optimizer": opt,
            "lr_scheduler": {"scheduler": sched, "interval": "step", "frequency": 1}}

self.trainer.estimated_stepping_batches is the right way to size a step-based schedule: it accounts for max_epochs, accumulation, the number of devices and max_steps, so you do not hand-compute it. A ReduceLROnPlateau scheduler needs a "monitor" key naming a logged metric, and, as with checkpointing, that metric must exist. Add the LearningRateMonitor(logging_interval="step") callback to see the actual learning rate in your logger; it is the quickest way to catch a schedule that is not doing what you drew on paper.

Manual optimization is for training procedures that do not fit the one-loss-one-step model: GANs, multiple optimizers, some reinforcement learning and self-supervised setups. You switch it on with self.automatic_optimization = False in __init__, and you take over four calls.

class LitGAN(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.automatic_optimization = False
        self.generator, self.discriminator = make_g(), make_d()

    def training_step(self, batch, batch_idx):
        opt_g, opt_d = self.optimizers()
        real = batch[0]
        z = torch.randn(real.size(0), 128, device=self.device)   # self.device, never .cuda()

        # discriminator step
        d_loss = self.d_loss(real, self.generator(z).detach())
        opt_d.zero_grad()
        self.manual_backward(d_loss)
        opt_d.step()

        # generator step
        g_loss = self.g_loss(self.generator(z))
        opt_g.zero_grad()
        self.manual_backward(g_loss)
        opt_g.step()
        self.log_dict({"d_loss": d_loss, "g_loss": g_loss}, prog_bar=True)

    def configure_optimizers(self):
        return [torch.optim.Adam(self.generator.parameters(), 2e-4),
                torch.optim.Adam(self.discriminator.parameters(), 2e-4)]

Three rules apply. Always call self.manual_backward(loss) rather than loss.backward(), because the former lets the precision plugin and strategy do their work, such as loss scaling. The Trainer flags gradient_clip_val and accumulate_grad_batches are not allowed with manual optimization and raise a configuration error; clip yourself with self.clip_gradients(opt, gradient_clip_val=1.0), and accumulate by stepping every N batches inside your own code. And you step the schedulers yourself, via self.lr_schedulers(). Since 2.5.2 there is also a self.toggled_optimizer(opt) context manager that freezes the parameters owned by other optimizers while you update one, which is the clean way to write the GAN example above without requires_grad bookkeeping.

If you only need to inspect gradients, you do not need manual optimization. The on_before_optimizer_step(optimizer) hook runs after backward and before the step. The old track_grad_norm flag was removed, so log norms yourself:

from lightning.pytorch.utilities import grad_norm

def on_before_optimizer_step(self, optimizer):
    self.log_dict(grad_norm(self, norm_type=2))

Precision, compile and finding the real bottleneck

Mixed precision is the cheapest speed and memory win available, and in Lightning it is one flag. The values to know are "32-true" (the default), "16-mixed", "bf16-mixed", "16-true" and "bf16-true". Teach and write the strings; the older integer 16 still maps to "16-mixed" but the string states what you mean.

The choice is mostly decided by hardware. On Ampere GPUs and newer, and on TPUs, use "bf16-mixed". It has the same exponent range as fp32, so it needs no loss scaling and rarely produces overflow. On older GPUs without bf16, use "16-mixed", which uses a gradient scaler to keep small gradients from underflowing. The -true variants keep the weights themselves in half precision: they save the most memory but make training less stable, and they are a deliberate choice for large models rather than a default. Since 2.6.0, MPS on Apple Silicon also supports mixed precision.

Separately, on Ampere and newer GPUs, torch.set_float32_matmul_precision("high") lets fp32 matrix multiplications use TF32 hardware. Lightning prints a hint about this when it detects a capable GPU, and ignoring that hint leaves speed on the table for fp32 sections of your model.

torch.compile works with Lightning by compiling the module and passing the result to the Trainer: model = torch.compile(model); trainer.fit(model, datamodule=dm). Expect a slow first step as compilation happens, a benefit that appears over thousands of steps, and breakage if your model has data-dependent control flow. Measure before committing: compile a copy, run 200 steps, and compare steady-state throughput, excluding the first few steps.

"Measure" needs a tool. Lightning ships several, in rising order of cost:

• Trainer(profiler="simple") reports the time spent in each hook and loop section, which is enough to tell you whether the time goes to the data loader, the training step or the callbacks.
• profiler="advanced" uses cProfile for a function-level Python view.
• PyTorchProfiler(filename=..., export_to_chrome=True) records a GPU and CPU trace you can open in a trace viewer, for kernel-level questions.
• The ThroughputMonitor callback logs samples per second and, when you give it a way to measure compute, model FLOPs utilisation, which shows how close you are to the hardware's limit.
• DeviceStatsMonitor logs GPU memory and utilisation to your logger.

from lightning.pytorch.callbacks import ThroughputMonitor, DeviceStatsMonitor

trainer = L.Trainer(
    accelerator="gpu", devices=1, precision="bf16-mixed",
    profiler="simple",
    callbacks=[ThroughputMonitor(batch_size_fn=lambda b: b[0].size(0)), DeviceStatsMonitor()],
    max_steps=300,
)

Read the simple profiler's output top down. If [_TrainingEpochLoop].train_dataloader_next dominates, you have a data problem and the fix is in the previous section. If the training step dominates and GPU utilisation is high, you are compute-bound and precision, compilation or a smaller model will help. If utilisation is low but the loader is not the issue, look for host-to-device copies, .item() calls and Python-side logging in the step.

Multi-GPU with DDP, without the surprises

When one GPU is not enough, Trainer(accelerator="gpu", devices=4, strategy="ddp") runs DistributedDataParallel: one process per GPU, each holding a full copy of the model, each processing its own shard of the batch, with gradients averaged by an all-reduce during the backward pass. Setting strategy="auto" with more than one device picks DDP for you. devices can be an integer, a list of indices, -1 for all, or "auto".

How Lightning starts those processes explains the first class of bugs. The default ddp strategy launches the extra processes by re-running your script as subprocesses. This is why every piece of training code must sit under if __name__ == "__main__":: any code at module level runs once per GPU, so a stray download, an experiment-tracker init call or a print will appear four times or, worse, race against itself. The alternatives are ddp_spawn, which uses multiprocessing spawn and brings pickling constraints and slower startup, and ddp_fork and ddp_notebook, which fork. In Jupyter or Colab, strategy="ddp" raises a configuration error telling you to use ddp_notebook or run as a script. Forking is unavailable on Windows, so ddp_fork fails there with a message recommending ddp_spawn, and on Windows the distributed backend is gloo because NCCL is Linux-only. On Apple Silicon, DDP-family strategies are not supported on MPS at all.

import lightning as L

def main():
    L.seed_everything(42, workers=True)
    model = LitClassifier(lr=3e-4)
    dm = TextDataModule("data/")
    trainer = L.Trainer(accelerator="gpu", devices=4, strategy="ddp",
                        precision="bf16-mixed", max_epochs=20)
    trainer.fit(model, datamodule=dm)

if __name__ == "__main__":
    main()

Each rank has three identifiers worth knowing. global_rank is the index across all processes, local_rank is the index within a node, and world_size is devices × num_nodes. By default only rank 0 writes logs and checkpoints. Use trainer.is_global_zero or the @rank_zero_only decorator from lightning.pytorch.utilities for pure side effects, and for printing use self.print, which prints only on rank 0.

Effective batch size, as above, scales with the number of devices, so moving from one to four GPUs without touching the learning rate is a real change. The common starting point is to scale the learning rate with the effective batch, then validate it with a short run, remembering that very large batches may need warmup.

There are three failure patterns to recognise by sight:

1. "Expected to have finished reduction in the prior iteration before starting a new one." Some parameters did not receive gradients, for example a branch of the model that is not used in a given forward pass. DDP waits for gradients that never come. The fix is strategy="ddp_find_unused_parameters_true", which adds overhead, so first check whether the unused parameters can simply be removed or frozen.
2. Hangs and NCCL timeouts. Typical causes are a collective call made on only some ranks (the warning above), num_nodes and devices that do not match the scheduler's allocation, or a firewall blocking the rendezvous port. Set NCCL_DEBUG=INFO to see the communication log and PYTHONFAULTHANDLER=1 to get Python tracebacks if the process is killed.
3. Different numbers of batches per rank. If you wrote a custom sampler or an iterable dataset that gives ranks unequal lengths, one rank finishes early and the others wait forever. Make shard lengths equal.

For nodes with several GPUs of which you want a subset, CUDA_VISIBLE_DEVICES is respected, and devices=[0, 2] selects specific indices. sync_batchnorm=True converts BatchNorm layers to synchronised versions, which helps when the per-device batch is very small.

When the model does not fit: FSDP and DeepSpeed

DDP keeps a full copy of weights, gradients and optimizer state on every GPU. When that total exceeds one device's memory, you need sharding. Two strategies cover most work, and a third is experimental. The Senior level goes deeper; this section is the entry point so that you can recognise when to reach for them. If you already use DeepSpeed directly, see the DeepSpeed guide for what the library does underneath.

FSDP, fully sharded data parallel, is PyTorch's native approach: parameters, gradients and optimizer state are sharded across ranks and gathered layer by layer as needed. In Lightning you select it with strategy="fsdp", or configure it through FSDPStrategy, whose most important options are auto_wrap_policy (which layers become shard units, for example the transformer block class), activation_checkpointing_policy, sharding_strategy, state_dict_type ("sharded" or "full") and cpu_offload. FSDP requires a GPU accelerator.

DeepSpeed is selected with names such as strategy="deepspeed_stage_2" or "deepspeed_stage_3_offload", or with a DeepSpeedStrategy object when you need a configuration file. It is Linux only, and the strategies extra installs it.

Whichever you pick, three rules keep you out of trouble. First, build large layers in configure_model() instead of __init__: Lightning calls it after the strategy is ready, so layers can be created directly sharded, on the right device and in the right dtype, rather than being fully materialised on one device first. This is the replacement for the deprecated configure_sharded_model; overriding both raises a RuntimeError. Second, configure_optimizers must use self.parameters(), not a reference to the wrapped model. Third, remember the checkpoint format changes: FSDP with sharded state writes a directory of shards, not a single file, and you consolidate it with python -m lightning.pytorch.utilities.consolidate_checkpoint <ckpt_dir>. DeepSpeed ZeRO checkpoints convert with convert_zero_checkpoint_to_fp32_state_dict from lightning.pytorch.utilities.deepspeed.

from lightning.pytorch.strategies import FSDPStrategy
import torch.nn as nn

class LitLM(L.LightningModule):
    def configure_model(self):
        if hasattr(self, "model"):
            return                      # configure_model can be called more than once
        self.model = build_transformer(self.hparams.cfg)

    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=self.hparams.lr)

strategy = FSDPStrategy(auto_wrap_policy={nn.TransformerEncoderLayer},
                        state_dict_type="sharded")
trainer = L.Trainer(accelerator="gpu", devices=8, strategy=strategy, precision="bf16-mixed")

An important fixed point from 2.6.4: an FSDP bug that initialised parameters in half precision under bf16-mixed and 16-mixed was fixed, so if you saw odd FSDP behaviour on older 2.6 releases, upgrade before debugging further.

There is also ModelParallelStrategy, which combines tensor parallelism and FSDP2 on a device mesh, applied inside configure_model through self.device_mesh. It is marked experimental, so do not build a team's pipeline on it without budgeting for change.

Checkpoints and resuming, precisely

A checkpoint is more than weights. It is one dictionary holding the state_dict, the optimizer and scheduler states, epoch and global_step, the loop state (including your position inside an epoch), the state of every stateful callback, the hyperparameters from save_hyperparameters(), the datamodule state if you defined one, and the pytorch-lightning_version that wrote it. That completeness is why there are two different ways to load one, and why mixing them up is the commonest checkpoint mistake at work.

Model.load_from_checkpoint(path) is a classmethod. It builds a new instance from the saved hyperparameters and loads the weights, and nothing else: no optimizer, no step counter. It returns a model you should put in eval() mode yourself before inference. trainer.fit(model, ckpt_path=path) restores the whole training state, so the run continues as if it had never stopped. Use the first for evaluation and serving, the second for resuming.

# inference: weights and hparams only
model = LitClassifier.load_from_checkpoint("checkpoints/best.ckpt").eval()

# resume: everything, including optimizer, schedulers and loop position
trainer.fit(model, datamodule=dm, ckpt_path="last")

The special values "best" and "last" refer to the previous fit call on the same trainer, so they only work in a process that has just trained. A fresh process has to pass an actual path. If you call validate or test with no model, "best" is used and you get a warning. ckpt_path="best" also requires a ModelCheckpoint with a monitor, otherwise you get .test(ckpt_path="best") is set but ModelCheckpoint is not configured to save the best model.

Choosing when to save

By default there is one ModelCheckpoint that keeps the last epoch. For production training you usually want three things on top: the best few by a metric, a rolling last for resume, and time- or step-based saves so that a long epoch cannot lose hours of work.

from lightning.pytorch.callbacks import ModelCheckpoint

best = ModelCheckpoint(dirpath="ckpt/", filename="{epoch}-{val_loss:.3f}",
                       monitor="val_loss", mode="min", save_top_k=3, save_last=True)
rolling = ModelCheckpoint(dirpath="ckpt/rolling/", every_n_train_steps=2000,
                          save_top_k=1, save_on_exception=True)
trainer = L.Trainer(callbacks=[best, rolling])

save_top_k greater than one needs a monitor; -1 keeps everything and will fill a disk. save_on_exception=True, added in 2.5.3, writes a checkpoint when the run dies with an exception, and train_time_interval accepts a timedelta for wall-clock saves. Since 2.5.5, step- and time-triggered saves are deferred until validation metrics exist, which prevents a monitored-key error early in a run. Resuming with ckpt_path="last" is only as good as your reproducibility: if the datamodule's setup shuffles differently each time, the mid-epoch position is restored but the data order is not.

Checkpoint safety and the 2.6.6 security fix

A .ckpt file is built on Python pickle, so loading one can execute code. This used to be a theoretical worry; at this level it is an operational rule. Two things changed recently, and both affect what you run.

First, from PyTorch 2.6, torch.load defaults to weights_only=True. Lightning 2.6.0 added a weights_only argument to fit, validate, test and predict, defaulting to None, which lets torch decide. The symptom of the stricter default is an UnpicklingError: Weights only load failed ... Unsupported global. It appears when a checkpoint contains arbitrary Python objects, most often hyperparameters that hold a custom class or dataclass. The right fix is to keep hyperparameters primitive: numbers, strings, lists, dicts. If you must load a file you trust completely, pass weights_only=False, or allowlist specific classes with torch.serialization.add_safe_globals.

Second, Lightning 2.6.6 (September 2026) fixed an arbitrary-code-execution bug, CVE-2026-58659, in load_from_checkpoint. A crafted checkpoint could name an _instantiator or _class_path in its hyperparameters and have Lightning run it, bypassing weights_only=True. The fix restricts the instantiator to a short allowlist and requires that a requested class resolve to an already imported subclass of the class being loaded. Version 2.6.5 was still vulnerable, so 2.6.6 is the minimum safe version.

The new checks produce two errors you may meet in legitimate use. The instantiator '...' from the checkpoint is not in the allowlist of trusted instantiators means the checkpoint was written with a custom instantiator. The class '...' requested by the checkpoint does not resolve to an already imported subclass means the module defining your model has not been imported in the loading process. The second is the common one, and the fix is simply to import your model module before calling load_from_checkpoint.

For a team, the practical policy is short. Pin lightning>=2.6.6. Never load checkpoints from a source you do not control without weights_only=True. Distribute models to other teams as state dicts or safetensors rather than full .ckpt files. Remember that PyTorch has its own advisories on untrusted checkpoints, so keep torch patched too. The MLflow guide covers a model registry, which is the right place to control which artifacts are allowed to reach production.

LightningCLI and configuration management

Hard-coded Trainer arguments in a script work until you want to run the same code with ten configurations on a cluster. LightningCLI solves that by generating a command line and a YAML schema from the signatures of your classes, using jsonargparse. It needs pip install "lightning[pytorch-extra]".

from lightning.pytorch.cli import LightningCLI

def cli_main():
    LightningCLI(MyModel, MyDataModule)

if __name__ == "__main__":
    cli_main()

That one file gives you the subcommands fit, validate, test and predict. The workflow that makes it pay off has three steps. Run python main.py fit --print_config > config.yaml to dump every option with its default. Edit the file down to what you care about. Run python main.py fit --config config.yaml. Anything after --config on the command line overrides the file, so --trainer.max_epochs 100 or --model.learning_rate 0.1 adjusts a single value without a new file. You can pass several --config files, and later ones override earlier ones, which gives you a base file plus a small per-environment overlay.

seed_everything: 42
trainer:
  accelerator: gpu
  devices: 4
  precision: bf16-mixed
  max_epochs: 30
model:
  learning_rate: 0.001
data:
  batch_size: 64
optimizer:
  class_path: torch.optim.AdamW
  init_args:
    lr: 0.001

The CLI saves the resolved configuration into the log directory as config.yaml through SaveConfigCallback, so every run records exactly what produced it. If the file already exists it refuses to overwrite it, and you pass save_config_kwargs={"overwrite": True} when that is what you want. Optimizers and schedulers can be chosen from the command line (--optimizer AdamW --lr_scheduler CosineAnnealingLR), and a different model class can be selected with --model Model1 when you use subclass mode. python main.py fit --model.help Model1 prints the arguments of a class. When parsing goes wrong, JSONARGPARSE_DEBUG=true shows the full traceback.

Two cautions. A config can instantiate arbitrary classes through class_path, and --config accepts remote URLs such as s3://bucket/config.yaml, so treat configuration as code and control who can write it. Secondly, the CLI writes every argument of your module to config.yaml, and save_hyperparameters() writes them into every checkpoint and logger. Never pass credentials as constructor arguments; read them from environment variables at the point of use.

Testing Lightning code and keeping CI cheap

GPU hours spent discovering a typo are the most avoidable cost in training. A small layered test strategy catches most bugs in seconds on a CPU runner.

The first layer is plain unit tests. A LightningModule is an nn.Module, so call forward with a random tensor and assert on shape, and call training_step with a fabricated batch to check for a scalar loss.

The second layer is a smoke test through the real Trainer. fast_dev_run=True runs one training batch, one validation batch and one test batch, and disables loggers and checkpointing, so it is quick and leaves no files behind. It exercises hook wiring, self.log calls, optimizer construction and the dataloaders, which is where most beginner-to-mid bugs live.

import lightning as L
from my_project.model import LitClassifier
from my_project.data import TinyDataModule

def test_fit_smoke():
    trainer = L.Trainer(fast_dev_run=True, accelerator="cpu", devices=1)
    trainer.fit(LitClassifier(), datamodule=TinyDataModule())

def test_overfits_one_batch():
    trainer = L.Trainer(overfit_batches=2, max_epochs=30, accelerator="cpu",
                        logger=False, enable_checkpointing=False,
                        enable_progress_bar=False)
    model = LitClassifier()
    trainer.fit(model, datamodule=TinyDataModule())
    assert trainer.callback_metrics["train_loss"] < 0.1

The third layer is the overfit test, shown above. overfit_batches makes the Trainer reuse a few training batches for training and validation. A model that cannot drive the loss close to zero on two batches has a bug in the loss, the labels, the learning rate or the optimizer wiring, and no amount of data will fix it. It is the single most informative test you can add.

A few practical notes for CI. Set accelerator="cpu" in tests so that they behave the same on a laptop and a runner. Use logger=False and enable_checkpointing=False unless a test is about those features, so tests do not write lightning_logs/ into the working tree. Use L.seed_everything(0, workers=True) for determinism. Run the multi-GPU path separately and only when a GPU runner is available, since DDP tests are slow and flaky on shared hosts. Lightning itself is tested against torch 2.1 through 2.10 on Python 3.10 to 3.13, so a matrix of your own should pin the combination you deploy, not the whole range. A workflow in GitHub Actions with a cached pip directory runs this layer in a minute or two.

Debugging beyond the basics

Most training bugs fall into four families: the wrong thing is computed, the right thing is computed too slowly, the run hangs, or it runs out of memory. A fixed order of operations is faster than guessing.

Wrong results. Start by shrinking the problem. limit_train_batches=0.1 makes iteration fast, overfit_batches=10 checks learning, and num_sanity_val_steps=0 skips the initial validation if you suspect it. Turn on detect_anomaly=True when you see nan or inf; it makes autograd report the operation that produced the first bad value, at a large speed cost, so switch it off afterwards. If a loss turns non-finite, EarlyStopping(check_finite=True) is on by default and will stop the run, and the SpikeDetection callback can flag sudden loss spikes. Log gradient norms to see instability coming, using on_before_optimizer_step with grad_norm(self, norm_type=2) and self.log_dict.

The self.log family of errors. Several look alike. You are trying to self.log() but it is not managed by the Trainer control flow means you called it outside a Trainer-driven hook, such as in __init__ or after calling the model by hand. the tensor must have a single element means you logged a batch-sized tensor; reduce it with .mean(). Nested dictionaries cannot be logged, and logging from predict hooks is not supported.

A monitored metric is missing. ModelCheckpoint(monitor='val_loss') could not find the monitored key and the matching EarlyStopping message both mean the metric name was never logged at the time the callback checked. Common causes are a typo, no validation loop (a missing val_dataloader or limit_val_batches=0), or logging inside a branch that did not run. Print trainer.callback_metrics.keys() in on_validation_end to see what exists.

Hangs. In distributed runs, a collective operation that only some ranks execute will block the rest forever. The usual culprit is a self.log(..., sync_dist=True) or a self.all_gather inside an if self.global_rank == 0: block. The other causes are a num_nodes or devices value that does not match what the scheduler allocated, and a firewall that blocks the MASTER_PORT. Set NCCL_DEBUG=INFO and PYTHONFAULTHANDLER=1 and rerun to see which rank is stuck where. If DDP raises Expected to have finished reduction in the prior iteration, some parameters received no gradient; either remove them or use strategy="ddp_find_unused_parameters_true", which costs some speed.

Out of memory. In order of effort: lower the per-device batch and use accumulate_grad_batches to keep the effective size, switch to bf16-mixed, use activation checkpointing, then move to FSDP or DeepSpeed. Tuner(trainer).scale_batch_size(model, mode="power") finds the largest batch that fits, provided your module or datamodule exposes a batch_size attribute.

For performance, use profiler="simple" first, which reports time per hook, then PyTorchProfiler(export_to_chrome=True) for a kernel-level trace. ThroughputMonitor reports samples per second and, with the right arguments, model FLOPs utilisation, which tells you how far you are from the hardware limit. DeviceStatsMonitor logs GPU memory and utilisation to your logger.

Running on clusters, containers and schedulers

Lightning is a library, not a service. There is no server to deploy: a training job is a process that your scheduler starts, with checkpoints and logs written to storage that outlives the process. Where you run it decides which launcher you use, and the single most important rule is that what you tell the Trainer must match what the scheduler allocated.

Single node, container. Package the training script in an image built from a CUDA-enabled base, install torch first for the right CUDA build, then lightning with a pinned version. Run it with --gpus or the runtime's equivalent and set Trainer(accelerator="gpu", devices="auto"). In a container, DataLoader workers and NCCL both use shared memory, so a default small /dev/shm produces confusing worker crashes and bus errors; give the container a larger shared memory size. The Docker guide covers the mechanics of building and running these images.

Multi-node with torchrun. Lightning detects TorchElastic, so you launch with torchrun --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=... --master_port=... train.py on each node and set Trainer(devices=8, num_nodes=2). The manual alternative is to set MASTER_ADDR, MASTER_PORT, WORLD_SIZE and NODE_RANK yourself.

SLURM. The script header must agree with the Trainer.

#!/bin/bash
#SBATCH --nodes=4              # must equal Trainer(num_nodes=4)
#SBATCH --ntasks-per-node=8    # must equal Trainer(devices=8)
#SBATCH --gres=gpu:8
#SBATCH --mem=0
#SBATCH --signal=SIGUSR1@90

srun python train.py

With --signal=SIGUSR1@90, SLURM warns the job 90 seconds before its time limit, and Lightning's SLURMEnvironment plugin saves a checkpoint and requeues the job; on restart it resumes automatically through ckpt_path="hpc". You can change the signal with SLURMEnvironment(requeue_signal=...) or disable the behaviour with auto_requeue=False. A common source of silent failure is launching with plain python instead of srun, which starts one process while the scheduler reserved many.

Kubernetes. Lightning recognises Kubeflow environments through KubeflowEnvironment, so a PyTorchJob that provides the standard environment variables works without extra code. See the Kubernetes guide and the Kubeflow guide for the job objects themselves. On spot or pre-emptible nodes, the pattern is the same as SLURM: frequent checkpoints to remote storage, plus resume on restart. Since 2.5.2, SIGTERM handling is synchronised across ranks in DDP, which avoids a deadlock when only some ranks receive the signal at pre-emption.

Remote storage. default_root_dir, ModelCheckpoint(dirpath=...), logger save directories and ckpt_path all accept fsspec URLs such as s3://, gs:// or abfs://, provided s3fs, gcsfs or adlfs is installed. Credentials come from the usual cloud mechanisms such as instance roles, not from Lightning. For teams in the Gulf region with data-residency rules, point checkpoints and datasets at buckets in an in-region location and check that logger backends you use do not ship artifacts elsewhere. To avoid stalls while a large checkpoint uploads, use AsyncCheckpointIO, which writes in a background thread.

Experiment tracking, export and serving

Lightning gives you one self.log call and routes it to any number of loggers. Built-in options are TensorBoardLogger, CSVLogger, MLFlowLogger, WandbLogger, CometLogger and, since 2.6.1, LitLogger. Pass a list to log to several at once. If neither TensorBoard nor tensorboardX is installed, the default falls back to CSVLogger with a warning, which is why a fresh environment sometimes seems to log "somewhere else". The Neptune logger was removed in 2.6.4 after the service was sunset, and instantiating it raises a RuntimeError; migrate to another logger. Connect to MLflow or Weights & Biases by passing the logger object, and reach the backend client through self.logger.experiment.

from lightning.pytorch.loggers import MLFlowLogger, CSVLogger

trainer = L.Trainer(logger=[
    MLFlowLogger(experiment_name="churn", tracking_uri="http://mlflow:5000"),
    CSVLogger("logs/"),
])

Tracking credentials stay in each tool's own environment variables, such as WANDB_API_KEY or MLFLOW_TRACKING_URI. Give every run its own default_root_dir or logger version so concurrent runs do not collide on checkpoint directories.

For export, the current guidance moved away from TorchScript. LightningModule.to_torchscript has been deprecated since 2.6.1 because TorchScript itself is deprecated in PyTorch. Use torch.export.export(model, (example,)) for a graph you can hand to other runtimes, or model.to_onnx("model.onnx", example, export_params=True); passing dynamo=True uses the newer exporter and returns an ONNXProgram. Version 2.6 also added to_tensorrt for a Torch-TensorRT export. Always call model.eval() and export on representative input shapes, and check numerical agreement between the exported and original model on a held-out batch before shipping.

Lightning does not serve models. The documented serving helpers, ServableModule and the ServableModuleValidator callback, are experimental, and the ecosystem points to LitServe, a separate package. In practice many teams export to ONNX or a state dict and serve with something purpose-built: see Triton or BentoML for the options. The important design decision is to keep inference code free of Trainer dependencies, so the serving image does not need training libraries.

Putting it all together

A repository holds the module, the datamodule, a main.py using LightningCLI, a base config.yaml and one small overlay per environment. A pull request runs the unit tests, the fast_dev_run smoke test and the overfit test on CPU. A merge builds a container with pinned lightning, torch and torchmetrics, tagged with the commit hash. A scheduled job starts that image on GPU nodes through the scheduler, with devices and num_nodes matching the allocation, precision="bf16-mixed", torch.set_float32_matmul_precision("high"), and a ModelCheckpoint pair writing best and rolling checkpoints to remote storage. Metrics go to the tracking server, along with the resolved config. If the node is pre-empted, the requeue resumes from the last checkpoint. After training, a separate evaluation step loads the best checkpoint with load_from_checkpoint, runs trainer.test, exports the model with torch.export or ONNX, and registers it only if the metrics beat the current model.

import torch
import lightning as L
from lightning.pytorch.callbacks import ModelCheckpoint, LearningRateMonitor

def main():
    torch.set_float32_matmul_precision("high")
    L.seed_everything(42, workers=True)
    trainer = L.Trainer(
        accelerator="gpu", devices=4, strategy="ddp", precision="bf16-mixed",
        max_epochs=30, gradient_clip_val=1.0,
        callbacks=[
            ModelCheckpoint(monitor="val_loss", save_top_k=3, save_last=True),
            LearningRateMonitor(logging_interval="step"),
        ],
    )
    trainer.fit(LitClassifier(), datamodule=MyDataModule())
    trainer.test(ckpt_path="best", datamodule=MyDataModule())

if __name__ == "__main__":
    main()

The shape matters more than the details. Each concern sits where Lightning expects it: research code in the module, data in the datamodule, hardware and distribution in Trainer flags, reusable behaviour in callbacks, and run-specific values in configuration rather than in code.

Where you are now

You can now predict what the Trainer does at each stage, log metrics correctly under distribution, keep a GPU fed, choose between automatic and manual optimization, pick a precision and measure its effect, run DDP and recognise its failure modes, and know when sharding is needed. You can checkpoint and resume safely, avoid the checkpoint security trap by staying on 2.6.6 or newer, drive experiments through LightningCLI, test cheaply, debug in a fixed order, launch on SLURM and in containers, and export a model for serving.

The Senior level takes the next step: owning this as a platform for several teams, with FSDP and DeepSpeed in depth, multi-node scaling limits, upgrade policy, and incident playbooks. If distributed training is your next problem, read the DeepSpeed guide in parallel.

Sources

• Lightning documentation
• Trainer
• LightningModule
• Hooks
• LightningDataModule
• LightningCLI
• Basic checkpointing
• Intermediate checkpointing
• Early stopping
• Precision
• Remote filesystems
• Multi-GPU training
• FSDP
• DeepSpeed
• Cluster training
• Logging
• Profiler
• Debugging
• Speed
• Production
• Release 2.6.6
• Changelog at 2.6.6

This is the mid-level interview file. Beginner questions ask what a thing is; these ask what you would do and why. Interviewers at this level are listening for reasoning: what you check first, what you rule out, and what trade-off you accept. The answers below are written the way a strong candidate would say them aloud, and each ends with what separates a strong answer from a weak one. Facts here are for Lightning 2.6.6.

How to answer scenario questions

Use the same four beats every time. State what you would observe first, give the most likely causes in order, say how you would confirm each cheaply, and name the fix and its cost. Candidates who jump straight to a fix ("add more workers") sound like they are guessing. Candidates who start with a measurement ("I would run the simple profiler and look at where the time goes") sound like they have done it.

Also be ready to say what you do not know. Lightning is a fast-moving library, so a candidate who says "I would check the changelog for that version" is more credible than one who recites a flag from memory that was removed in 2.0.

Scenario questions

Training is slow and the GPU shows low utilisation. What do you check first?

I would first confirm the symptom with numbers: GPU utilisation from nvidia-smi or DeviceStatsMonitor, and samples per second from ThroughputMonitor. If the GPU is idle in bursts, the model is waiting for data. I would run profiler="simple" and look at the time spent in the training dataloader hook compared with the training step.

Most often the cause is the DataLoader: num_workers left at zero, no pin_memory, no persistent_workers, or heavy per-item preprocessing that belongs in a cached or precomputed step. Lightning even emits a PossibleUserWarning about too few workers. The second cause is Python overhead from too much logging, for example calling self.log with on_step=True on many metrics every step; log_every_n_steps defaults to 50 for the logger, but computing and syncing metrics still costs. The third is precision and compile: running in 32-true on hardware that supports bf16 leaves speed on the table, and torch.set_float32_matmul_precision("high") enables TF32 on Ampere and newer.

A strong answer measures first, names the dataloader as the usual suspect, and changes one thing at a time. A weak answer lists every optimisation at once or buys a bigger GPU.

Validation accuracy is different on one GPU and on four. Why?

There are several possibilities, and I would separate them. Effective batch size changes: it is per-device batch times devices times nodes times gradient accumulation, so four GPUs with the same per-device batch is a four times larger batch, and the learning rate usually needs rescaling. Second, metric aggregation: if I log a validation metric without sync_dist=True, each rank logs its own shard and the logged value comes from one rank, not the whole validation set. The correct fix is to use torchmetrics objects, which synchronise themselves, or pass sync_dist=True for epoch-level means. Third, the distributed sampler pads the dataset so every rank sees the same number of samples, which can double-count a few examples when the dataset size is not divisible by the world size. That is small but visible on tiny validation sets.

A strong answer mentions batch size, sync_dist and sampler padding. A weak answer says "distributed training is non-deterministic" and stops.

The job hangs at the start of epoch two on a multi-node cluster. What do you do?

A hang in distributed training almost always means ranks have diverged: some are waiting in a collective operation that others never entered. I would set NCCL_DEBUG=INFO and PYTHONFAULTHANDLER=1, and send a signal to dump tracebacks, so I can see where each rank is stuck. Typical causes are a metric logged with sync_dist=True or an all_gather inside an if self.global_rank == 0: branch, a different number of batches on different ranks, a mismatch between num_nodes and devices and what SLURM or the Job allocated, or a blocked MASTER_PORT. The fact that it starts in epoch two points toward something that only happens after the first validation or checkpoint: code that runs on rank zero only, such as a custom callback saving files and then calling a collective.

A strong answer reasons from "ranks diverged" and names concrete evidence to collect. A weak answer restarts the job and hopes.

A teammate says "the checkpoint loads but training does not continue where it stopped." What happened?

The likely mistake is load_from_checkpoint. It rebuilds the model with its weights and hyperparameters, but it does not restore the optimizer, scheduler, epoch counter or loop position. To continue a run you call trainer.fit(model, ckpt_path=path), which restores everything. If they did use ckpt_path, I would check whether the topology changed, since optimizer state is tied to parameter layout, and whether the data order is reproducible, since Lightning restores the position but not a shuffle that is seeded differently.

ModelCheckpoint raises "could not find the monitored key". How do you debug it?

The callback looks up the name in trainer.callback_metrics, so I would print that dictionary at the end of validation. The usual causes are a name mismatch (val_loss versus val/loss), a metric logged only in a code path that did not run, or no validation loop at all because there is no validation dataloader or limit_val_batches=0. It can also be a timing problem: if the checkpoint is configured on a step interval before validation has ever run, the metric does not exist yet. Since 2.5.5 those saves wait until validation metrics exist, so on an older version that is another reason to upgrade.

You need to train a model that does not fit on one GPU. Walk me through the options.

I would go in order of cost and complexity. First reduce memory in place: bf16-mixed, a smaller per-device batch with accumulate_grad_batches to preserve the effective batch, and activation checkpointing. If the weights plus optimizer state alone do not fit, DDP cannot work because it replicates everything, so I need sharding. FSDP is PyTorch-native and integrates through strategy="fsdp" or FSDPStrategy with an auto-wrap policy for the transformer block. DeepSpeed ZeRO stages 2 or 3, with optional CPU or NVMe offload, is the alternative and is Linux only. Whichever I choose, I would create the layers in configure_model() so they are not fully materialised on one device, and plan for sharded checkpoints, which are directories that need consolidation.

The trade-off is that sharding adds communication cost and makes checkpoints and debugging more complex. I would also mention ModelParallelStrategy for tensor parallelism, and that it is experimental.

How do you make training safe on spot or pre-emptible instances?

Make progress cheap to recover. Save checkpoints on a step or time interval to remote storage, keep save_last=True, and enable save_on_exception. Make the job resume automatically: on SLURM, the SLURMEnvironment plugin requeues on the signal and resumes through ckpt_path="hpc"; on Kubernetes, the restart policy plus a wrapper that passes the last checkpoint path does the same. Use AsyncCheckpointIO so large uploads do not stall training. Handle SIGTERM: since 2.5.2 Lightning synchronises it across ranks, which avoids a deadlock when only some processes get the signal. Finally, test the recovery path on purpose by killing a job, because untested resume logic fails when you need it.

A new hire downloaded a checkpoint from a public hub and ran load_from_checkpoint. Is that a problem?

Potentially, yes. Checkpoints are pickle-based, so loading is code execution. PyTorch 2.6 made weights_only=True the default to reduce that risk, but a bug in Lightning's load_from_checkpoint allowed a crafted _instantiator or _class_path in the hyperparameters to run code anyway, tracked as CVE-2026-58659 and fixed in 2.6.6. Version 2.6.5 was still affected. So I would check the Lightning and torch versions, require 2.6.6 or newer, load untrusted files only with weights_only=True, and prefer safetensors or state dicts for third-party weights. I would also make the point that a model file is an artifact from a supply chain, and should come from a registry we control.

Your CI takes forty minutes and uses GPUs for every pull request. How do you improve it?

Move most checks to CPU. Unit tests for forward and training_step on small tensors, a fast_dev_run=True smoke test through the real Trainer, and an overfit_batches test that proves the model can learn. Run the multi-GPU and long-training checks on a schedule or on demand, not per pull request. Keep the dependency matrix to the versions you deploy, and cache the pip directory. The result is a tiny cost per pull request and early detection of the most common mistakes.

Trade-off questions

DDP, FSDP or DeepSpeed?

DDP is the default and the simplest: if the model, gradients and optimizer state fit on one GPU, use it, because it has the least communication and the least to go wrong. FSDP and DeepSpeed exist for when they do not fit. Between those two, I would pick FSDP when I want to stay close to native PyTorch and share code with Fabric or plain torch, and DeepSpeed when I need ZeRO-style offload to CPU or NVMe, or my team already has DeepSpeed configurations. The DeepSpeed guide covers what happens under the hood. The honest answer is to benchmark a real model on the hardware you have.

Automatic or manual optimization?

Automatic by default, because Lightning handles zero_grad, backward, step, scheduler stepping, gradient clipping and accumulation correctly. Manual optimization is for cases the automatic loop cannot express: GANs, multiple optimizers with different update schedules, and some research algorithms. The cost is that gradient_clip_val and accumulate_grad_batches on the Trainer are not allowed, so I must call self.clip_gradients and handle accumulation myself, and I own the correctness.

bf16-mixed, 16-mixed or 32-true?

On Ampere or newer GPUs, bf16-mixed is the first choice: the range matches fp32, so there is no loss scaling and fewer numerical surprises. On older GPUs 16-mixed works with a gradient scaler but is more fragile. 32-true is the baseline and the right choice while debugging numerics. The "true" half-precision modes, bf16-true and 16-true, save more memory by holding weights in half precision but can hurt stability, so I would only use them with evidence.

LightningCLI or plain scripts with argparse?

CLI when the team runs many configurations and wants every run to record a resolved, reproducible config, because the schema comes from type hints and the file is saved with each run. Plain scripts when the project is small or the config system is already Hydra. The risk with the CLI is treating configuration as harmless: class_path can instantiate arbitrary classes, so config files need the same review as code.

Lightning or Fabric or plain PyTorch?

Lightning when the standard loop fits and you want structure, logging, checkpointing and distribution handled. Fabric when you need a custom loop but still want device, precision and distribution handling. Plain PyTorch when the extra abstraction costs more than it gives, such as a small research script. Custom Loop objects were removed in 2.0, so "I will subclass the loop" is not an answer on current versions.

One logger or several?

Several is easy since the Trainer accepts a list. A common pattern is CSVLogger as a local fallback plus the team's tracking server, such as MLflow or Weights & Biases. The cost is duplicated hyperparameters and artifacts, so decide which system is the source of truth.

Knowledge-check questions with model answers

What does global_step count? Optimizer steps, not batches. With accumulate_grad_batches=4, four batches make one step, so schedulers and max_steps count in steps.

Where do you put dataset downloads and where do you put splits? Downloads in prepare_data(), which runs on one process per node, with no state assigned. Splits and tensors in setup(stage), which runs on every process.

What are the defaults of on_step and on_epoch in self.log? In training_step, on_step=True and on_epoch=False. In validation and test steps, on_step=False and on_epoch=True.

What replaced training_epoch_end? on_train_epoch_end() with no outputs argument; you accumulate values yourself in an attribute and clear it at the end of the epoch.

What replaced Trainer(gpus=2)? Trainer(accelerator="gpu", devices=2).

How do you find a good learning rate? Tuner(trainer).lr_find(model), which needs the module to expose lr or learning_rate. The old auto_lr_find flag was removed in 2.0.

Why can DDP fail in a Jupyter notebook? DDP re-launches the script in subprocesses, which cannot work from an interactive session. Use strategy="ddp_notebook", which is fork-based and unavailable on Windows, or run a script.

What is the effective batch size? Per-device batch times devices times number of nodes times accumulation steps.

What is deterministic=True for, and what does it cost? It asks PyTorch to use deterministic algorithms, which makes runs reproducible but can be slower or raise errors for operations with no deterministic implementation.

What a strong answer covers that a weak one misses

A strong answer states a measurement before a fix. It names the version-specific behaviour (for example weights_only, the 2.6.6 fix, or the removal of old flags) rather than describing the library as it was three years ago. It mentions the cost of each choice, such as the communication cost of sharding or the loss of Trainer features under manual optimization. It separates "works on my laptop" from "works under DDP with a scheduler". It treats checkpoints and configs as code from a security viewpoint. And it ends with how it would verify the fix, because "I changed it and the loss looks fine" is not verification.

A weak answer recites flags without mechanism, confuses load_from_checkpoint with resuming, gives answers from removed APIs, proposes the largest hardware as the first fix, or cannot say what it would look at when a distributed job hangs.

Questions to ask the interviewer

Good candidates ask too. Reasonable questions include how the team handles checkpoint storage and retention, how they test training code without GPUs, which precision they train in and why, how pre-emption is handled, and which Lightning and torch versions they pin and how they upgrade. These show that you think about running training as an operation, not only about getting a loss curve.

Sixty-second self-test

Answer these aloud without notes. What are the three things you check when a GPU is underused? Why does sync_dist matter? What is the difference between loading a checkpoint for inference and resuming? Which Lightning version is the minimum safe one today and why? What must match between a SLURM script and the Trainer? What is the order of memory-saving steps before you reach for FSDP? If you hesitate on more than two, reread the matching section of the detailed file.

This file collects what goes wrong at the mid level, what pays off most, and recipes you can paste into a terminal. It targets Lightning 2.6.6 on Python 3.10 or newer. Many tutorials online were written for 1.x, so a good share of the errors below come from copying code that has since been removed.

Errors you will meet, and how to read them

"Weights only load failed ... Unsupported global"

Symptom. Loading a checkpoint raises _pickle.UnpicklingError: Weights only load failed, naming a global such as a custom class.

Cause. PyTorch 2.6 made weights_only=True the default for torch.load. Your checkpoint contains a Python object that is not on the safe list, usually a hyperparameter that is a dataclass, an enum or a custom config class.

Fix. The best fix is to keep hyperparameters to primitives and rebuild objects in __init__. For files you fully trust, pass weights_only=False to load_from_checkpoint or to trainer.fit(..., ckpt_path=..., weights_only=False). Alternatively allowlist specific classes with torch.serialization.add_safe_globals. Never use weights_only=False on a file from someone else.

"The instantiator ... is not in the allowlist" and "does not resolve to an already imported subclass"

Symptom. After upgrading to 2.6.6, load_from_checkpoint raises a ValueError mentioning blocking arbitrary code execution.

Cause. The security fix for CVE-2026-58659. Lightning now only accepts known instantiators, and only resolves a checkpoint's class if it is already imported and is a subclass of the class you are loading.

Fix. Import the module that defines your model before loading. If the file really uses a custom instantiator and you trust it, the error message tells you where the allowlist lives. Do not downgrade to avoid the error: 2.6.5 and earlier are vulnerable.

"ModelCheckpoint could not find the monitored key"

Symptom. MisconfigurationException listing the available metrics.

Cause. The name you monitor was never logged, or was logged under another name, or no validation ran.

Fix. Log exactly the name you monitor, in a hook that runs, and compare against the list printed in the message. If you intend to monitor a training metric, say so with save_on_train_epoch_end=True.

"Early stopping conditioned on metric which is not available"

Same family as above. It is a RuntimeError unless strict=False. Remember that patience counts validation checks, not epochs, so with val_check_interval=0.25 patience of 3 means three quarters of an epoch.

Logging errors

• the tensor must have a single element: reduce with .mean() before logging.
• nested dictionaries cannot be logged: flatten, and use self.log_dict for several values. Passing a dict to self.log was removed.
• not managed by the Trainer control flow: you logged from __init__ or outside a Trainer-driven hook.
• Epoch metrics look wrong on multiple GPUs: add sync_dist=True, or use a torchmetrics object.

"Both configure_model and configure_sharded_model are overridden"

You copied older sharding code. configure_sharded_model is deprecated; keep only configure_model.

Strategy and accelerator errors

• Trainer(strategy='ddp') is not compatible with an interactive environment: you are in a notebook. Use ddp_notebook or run a script.
• strategies from the DDP family are not supported on the MPS accelerator: on a Mac use one device, or set accelerator="cpu".
• process forking is not supported on this platform: Windows has no ddp_fork; use ddp_spawn, or WSL2.
• CUDAAccelerator can not run on your system: usually a CPU-only torch wheel got installed. Reinstall torch from pytorch.org for your CUDA version first, then Lightning.
• The strategy fsdp requires a GPU accelerator: FSDP needs CUDA.
• HPU is currently not supported: the Habana integration was removed in 2.5.6.
• NeptuneLogger is no longer supported: the service was sunset; removed in 2.6.4. Switch loggers.

DDP says "Expected to have finished reduction in the prior iteration"

Some parameters did not receive a gradient in the forward pass, often because of conditional branches or frozen layers that are not set to requires_grad=False. Fix the model, or use strategy="ddp_find_unused_parameters_true", accepting a small slowdown.

Warnings worth acting on

• The train_dataloader does not have many workers: raise num_workers.
• Consider setting persistent_workers=True: do so when num_workers > 0.
• A warning that modules are in eval mode when training starts: something called .eval() on part of the model before fit. That is sometimes intended, for frozen batch norm, but usually a bug.
• LightningDeprecationWarning: read it now; the next minor release may remove the API.

Practices that pay off most, in order of leverage

1. Pin versions together and keep them current. Pin lightning, torch and torchmetrics as a set. Stay on 2.6.6 or newer for the checkpoint fix. Lightning does not follow semantic versioning: minor releases may remove deprecated APIs and a patch once removed a logger, so run your tests with warnings visible before every upgrade.
2. Overfit one batch first. overfit_batches=2 and thirty epochs tells you in a minute whether the model, loss and optimizer are wired correctly.
3. Smoke-test with fast_dev_run=True in CI on CPU. It costs seconds and catches hook and logging mistakes before GPUs are involved.
4. Fix the input pipeline before tuning the model. num_workers, pin_memory, persistent_workers, and caching preprocessing give more speed than most model changes.
5. Use torchmetrics objects for metrics. They handle accumulation, reset and distributed synchronisation, which removes a whole class of silent errors.
6. Treat checkpoints and configs as code. Load untrusted files with weights_only=True, keep hyperparameters primitive, and control who can write YAML that your jobs read.
7. Write a resume path and test it. Frequent checkpoints are useless if resuming has never been run. Kill a job on purpose and bring it back.
8. Use LightningCLI and save the resolved config with every run. Reproducibility is mostly knowing exactly what you ran.
9. Measure before optimising. profiler="simple", then ThroughputMonitor, then a PyTorch trace if needed.
10. Pick the smallest distribution strategy that works. DDP before sharding, bf16 and accumulation before DDP-to-FSDP.

Practice cards

Each card is a short exercise. Do them on a small model so they take minutes.

Card 1: Predict the hook order. Create a callback and a module whose every hook prints its name, run fast_dev_run=True, and write down the order before looking at the output. Then compare. Note where setup, on_fit_start, the sanity check, on_train_epoch_start and on_validation_epoch_end appear.

Card 2: Break and fix sync_dist. Train on two CPU processes with strategy="ddp_spawn" and devices=2. Log a validation value computed from different data on each rank, once without sync_dist and once with. Compare the numbers.

Card 3: Effective batch size. Compute the effective batch size for per-device 32, 4 devices, 2 nodes and accumulation 4. Then work out what global_step will be after one epoch on 100,000 examples.

Card 4: Resume correctly. Train two epochs, stop, resume to four epochs with ckpt_path, and verify the learning rate curve is continuous. Repeat with load_from_checkpoint plus a new fit, and see the optimizer restart.

Card 5: Precision study. Run the same short training under 32-true, 16-mixed and bf16-mixed and record time per epoch, peak memory and final validation loss. Use torch.cuda.max_memory_allocated() or DeviceStatsMonitor.

Card 6: Metric hunting. Remove the self.log("val_loss", ...) call and read the monitored-key error. Put it back under a different name and read the new error. Learn the message shape.

Card 7: Manual optimization. Convert a module to automatic_optimization = False, add gradient clipping with self.clip_gradients, and try setting gradient_clip_val on the Trainer to see the error.

Card 8: Export check. Export with to_onnx, run in ONNX Runtime, compare outputs to PyTorch with a tolerance, and write the tolerance down.

Misconceptions that cost real time

"More workers is always faster." Each DataLoader worker is a process with its own copy of the dataset object and its own memory. Past the point where the GPU is fed, more workers only add memory pressure and start-up time. Raise num_workers until the training-dataloader time in the simple profiler stops shrinking, then stop. On Windows and macOS, remember the if __name__ == "__main__": guard, because workers start by spawning.

"The progress bar loss is the epoch loss." With prog_bar=True on a step-level metric, the bar shows a running value that is smoothed over recent steps. The logged epoch value is a separate reduction, controlled by on_epoch=True and reduce_fx. If two numbers disagree, check which one you are reading before suspecting a bug. Since 2.6.0 the bar is rendered by Rich when it is installed and by tqdm otherwise, so screenshots from older tutorials will look different.

"seed_everything makes runs identical." It seeds the random number generators, and with workers=True it seeds DataLoader workers too. Identical results also need deterministic=True, the same hardware, the same library versions and the same device count. Different GPU counts change the sharding of data and the order of floating-point reductions, so do not promise bit-for-bit equality across topologies.

"Early stopping with patience 3 waits three epochs." It waits three validation checks. With a validation interval shorter than an epoch, three checks may be a fraction of an epoch. With check_val_every_n_epoch=5, three checks is fifteen epochs.

"global_step is the number of batches." It counts optimizer steps. With accumulate_grad_batches=4, it advances once every four batches, and with two optimizers under manual optimization the counting rules need reading carefully. Schedulers configured with "interval": "step" follow this counter, so a warm-up of 1000 steps means 4000 batches under accumulation of four.

"The checkpoint is the model." The checkpoint is training state. The model you ship should be a smaller artifact: weights only, with its input and output contract documented, and ideally exported to a format that does not need your training code. Shipping the full .ckpt carries optimizer state that doubles or triples the file size and drags in pickle risks.

Reading a Lightning run directory

When something looks wrong after the fact, the run directory is your evidence. By default, logs go to lightning_logs/version_N/, where N increments for each run in the same directory. Inside you will find the event files or metrics.csv depending on the logger, hparams.yaml with the saved hyperparameters, and checkpoints/ unless you set dirpath. With LightningCLI there is also the resolved config.yaml.

Two habits make this directory useful. Give each experiment its own default_root_dir or logger name so unrelated runs do not share a numbering sequence, and record the git commit and container tag as a hyperparameter or logger tag so a result can be traced to code. When two concurrent jobs share a directory, checkpoint filenames can collide; ModelCheckpoint adds -v1 suffixes through enable_version_counter=True, which hides the collision instead of preventing it, so the better fix is separate directories.

If the metrics file is empty or the curves look sparse, remember log_every_n_steps defaults to 50 and a short run with few steps per epoch may log almost nothing at step level. Lower it for small experiments, and leave it alone for large ones, where frequent logging slows training.

One more habit belongs here: read the summary table the Trainer prints before the first step. It lists every module with its parameter count, so a layer you meant to freeze but that still shows as trainable, or a model far larger than you expected, is visible before any GPU time is spent. ModelSummary(max_depth=2) expands it by one level when the default is too coarse. Compare the trainable and non-trainable totals with what you expect from the architecture, and treat any mismatch as a bug to explain before the run continues.

Operational recipes

Verify an environment

python -c "import lightning as L, torch; print(L.__version__, torch.__version__, torch.cuda.is_available())"
python -m pip show lightning torch torchmetrics | grep -E "Name|Version"

If you see 2.6.5 or older, schedule an upgrade. If torch.cuda.is_available() is false on a GPU machine, the torch wheel is probably CPU-only.

Quick smoke test of any module

import lightning as L

L.Trainer(fast_dev_run=True, accelerator="cpu", logger=False).fit(MyModel(), datamodule=MyData())

Overfit test

L.Trainer(overfit_batches=2, max_epochs=30, accelerator="cpu",
          enable_checkpointing=False, logger=False).fit(MyModel(), datamodule=MyData())

Find the learning rate and batch size

from lightning.pytorch.tuner import Tuner

trainer = L.Trainer(accelerator="gpu", devices=1)
tuner = Tuner(trainer)
tuner.scale_batch_size(model, mode="power")   # model needs a batch_size attribute
lr_finder = tuner.lr_find(model)              # model needs lr or learning_rate
print(lr_finder.suggestion())

Run these on a single device, then set the results in your config. They are tools for choosing starting values, not final answers.

Debug a hanging distributed job

export NCCL_DEBUG=INFO
export PYTHONFAULTHANDLER=1
torchrun --nproc_per_node=4 train.py

Look for the rank that stops printing first, and check the code path it was in. Search for collective calls inside rank-conditional blocks.

Consolidate sharded checkpoints

python -m lightning.pytorch.utilities.consolidate_checkpoint path/to/checkpoint_dir

For a DeepSpeed ZeRO checkpoint use convert_zero_checkpoint_to_fp32_state_dict from lightning.pytorch.utilities.deepspeed.

Upgrade an old checkpoint

python -m lightning.pytorch.utilities.upgrade_checkpoint path/to/old.ckpt

Lightning also migrates old checkpoints automatically on load, so only run this when you want the file itself updated. Take a copy first.

View logs

tensorboard --logdir lightning_logs/

If nothing appears, check whether TensorBoard was installed. Without it, the default logger falls back to CSV files under the same directory.

Run on SLURM

sbatch train.sbatch
squeue -u "$USER"
scontrol show job <jobid>

Check that --nodes equals num_nodes and --ntasks-per-node equals devices, and that the batch script launches with srun python.

Run in a container

docker run --gpus all --shm-size=16g my-train:2026-10-02 python train.py fit --config config.yaml

The larger shared memory prevents DataLoader worker crashes, and the image tag should carry a date or a commit, not latest. See the Docker guide for the details.

A weekly checklist for a training codebase

• Are all environments on Lightning 2.6.6 or newer, with torch and torchmetrics pinned alongside?
• Does CI run the smoke test and the overfit test on CPU?
• Is every logged metric name used by a callback spelled the same everywhere?
• Do hyperparameters contain only primitives, and no secrets?
• Can a killed job resume without manual steps, and has that been tried this month?
• Are checkpoints and logs written to storage that outlives the job, in the region your data rules require?
• Is the resolved configuration saved with every run?
• Are deprecation warnings read and tracked?
• Are the profiler and throughput numbers recorded for the main training run, so a regression is noticed?

Keep the list short enough to actually use. A team that checks five things every week does better than one that wrote twenty and checks none.

Final reminders

Most mid-level problems are not exotic. They come from a mismatched name, a metric that is not synchronised, a loader that starves the GPU, a resume path nobody tested, or a checkpoint trusted too easily. Put the cheap guards in place early, measure before changing anything, and read the message the library gives you: Lightning's errors usually name the exact setting that is wrong. For related reading, the MLflow guide covers experiment tracking and registries, and the Kubernetes guide covers scheduling the jobs.

This is part three of three. The first two levels taught you to write a training run that works and then to make it fast and repeatable. This level is about the thing that sits one floor above: you own PyTorch Lightning as a platform that other teams build their training jobs on. Your problems are no longer "why does my loss go to NaN" but "why did forty jobs hang at the same moment", "who is allowed to load this checkpoint", and "what breaks for everyone when we move to the next minor release".

A framing point before anything else. Lightning is a library. It has no server, no daemon and no control plane. Everything you would normally call "the platform" (scheduling, storage, identity, quotas, tracking) lives in other systems that Lightning talks to. Your platform is therefore a set of conventions, a shared base package, and the integration points between a Python library and those systems. That shapes every section below.

Where this picks up

The version this guide is written against is Lightning 2.6.6, released on 10 September 2026. Where an older tutorial disagrees with it, the older tutorial is wrong, and I will say so.

The architecture you actually own

Start with the shape of a Lightning job in production, because most platform decisions follow from it.

Three consequences are worth stating plainly.

First, the strategy owns more than distribution. It starts the processes, wraps the model, and decides how checkpoints are saved and loaded. Switching from DDP to FSDP therefore changes your checkpoint format, not just your memory use. A platform that hides the strategy behind a flag without also hiding the checkpoint format will break the first consumer who loads a file in a notebook.

Second, the scheduler and Lightning must agree about the world. Trainer(num_nodes=N, devices=G) has to match what the scheduler allocated. Lightning detects several environments automatically: SLURM, TorchElastic, Kubeflow, LSF and MPI, through the ClusterEnvironment plugins. When the numbers disagree, you do not get a clean error. You get processes waiting for peers that were never started.

Third, the artifacts are the only durable state. The checkpoints and the logs are what survive a crashed node. Everything else is reconstructed from them. That is why the checkpoint section below is longer than the distribution one.

Failure modes of a distributed run

A single-process program fails loudly. A distributed one fails by going quiet, and a platform owner needs a catalogue of the quiet failures, because your users will report all of them as "training is stuck".

The rank-0-only collective. The most common cause of a hang that I see in code review is a collective operation placed on a subset of ranks. A collective is any call that needs every process to participate: an all-reduce, a barrier, or self.log(..., sync_dist=True). If a user writes this:

def validation_step(self, batch, batch_idx):
    loss = self.compute_loss(batch)
    if self.global_rank == 0:
        self.log("val_loss", loss, sync_dist=True)   # wrong: other ranks never arrive

then rank 0 waits in the reduction for peers that skipped the call, and nothing ever fails. The rule to teach is that logging with sync_dist=True must run on every rank. If you only want one rank to write something, use rank_zero_only=True on the log call or the @rank_zero_only decorator for side effects such as writing files. Keep those for I/O, never for anything that synchronises.

Parameter mismatch in DDP. When some parameters receive no gradient in a step, DDP raises a message beginning "Expected to have finished reduction in the prior iteration". The fix named in the docs is strategy="ddp_find_unused_parameters_true". Understand the cost: that flag makes DDP traverse the graph every step to find unused parameters, which is slower. It is the right answer for models with genuinely conditional branches, and the wrong answer for a model where a layer was simply forgotten.

Allocation mismatch. If num_nodes or devices disagrees with the scheduler, some ranks never start. The job sits in rendezvous. A cheap guard is to compute the expected world size in your launcher wrapper and assert it equals devices × num_nodes before constructing the Trainer.

Scaling: which component breaks first

When a team says "we need to scale", the useful question is which resource is the limit. Lightning gives you levers for each, and they are not interchangeable.

Stage one: the model fits on a device, and you want more throughput. This is DDP. Every GPU holds a full copy of the model and its optimizer state, processes its own slice of the data, and averages gradients. The limit is memory: the parameters, gradients and optimizer state must fit on one device. For many production models that is the right place to stay, because DDP is the simplest strategy and the best supported. ddp (subprocess launch) is the recommended default. ddp_spawn and ddp_notebook are slower and impose pickling constraints, so keep them for interactive work.

Stage two: the data pipeline breaks first. Before reaching for a more exotic strategy, check whether the GPUs are actually busy. DataLoader throughput and storage I/O are the usual bottleneck at this stage. The warning "The 'train_dataloader' does not have many workers which may be a bottleneck" is a PossibleUserWarning, and on a platform you should treat it as a defect worth failing a review over. The standard remedies are num_workers, pin_memory=True and persistent_workers=True. For measurement, use the ThroughputMonitor callback, which reports samples per second and, when you give it the FLOPs, model FLOPs utilisation (MFU, the fraction of the hardware's peak that you actually use). A low MFU with idle GPUs points at data. A low MFU with busy GPUs points at communication or at small kernels.

Stage three: the model no longer fits. Now you must shard. The options form a ladder, and a platform should offer one default rung rather than all of them.

The last row deserves a caution. ModelParallelStrategy(data_parallel_size="auto", tensor_parallel_size="auto") uses PyTorch-native DTensor tensor parallelism and FSDP2, applied by you inside configure_model() through self.device_mesh. It is the most capable option and the least settled, and it only supports the precisions 32-true, bf16-mixed, bf16-true and 16-true. Offering an experimental strategy as a platform default means you inherit the risk of its API changing. Offer it as an opt-in track with an owner.

Stage four: the interconnect. Across nodes, NCCL needs a fast fabric. Adding nodes to a job whose step time is already dominated by all-reduce can leave you paying for more GPUs without a matching speed-up, even though the nominal batch is larger. Effective batch size is the per-device batch times devices times nodes times accumulate_grad_batches, and the learning rate has to be reconsidered whenever it changes. A platform that lets users resize a job without recomputing the learning rate will ship worse models and not notice.

Two scaling traps that are easy to miss. First, global_step counts optimizer steps, not batches. With accumulate_grad_batches=4, four batches make one step, so any schedule or checkpoint interval you express in steps changes meaning when someone changes accumulation. Second, on MPS everything is single-device, so a user who develops on a Mac cannot reproduce a multi-GPU bug locally. Provide a small shared GPU environment for that.

Checkpoints as the platform's contract

A checkpoint is the one artifact that crosses team boundaries. The training team writes it, the evaluation team reads it, the serving team converts it, and the next training run resumes from it. If you do not define what a checkpoint is on your platform, each consumer will invent a definition, and they will disagree.

Recall what a .ckpt contains: the state_dict, optimizer and LR-scheduler states, epoch and global_step, loop state, callback states, hyper_parameters from save_hyperparameters(), datamodule state, and the pytorch-lightning_version that wrote it. That is a training artifact. It is large, because it carries optimizer state, and it is not what you serve. A good platform separates two products:

• Resume checkpoints, full fidelity, used only to continue training, short-lived, kept for the retention window you set.
• Release artifacts, weights only (a state dict, ideally in a format that cannot execute code), versioned, immutable, and promoted through your registry.

ModelCheckpoint(save_weights_only=True) produces the second kind of file in Lightning's own format, but for distribution to other teams I would go one step further and export to an explicit weights format, covered under security below.

Sharded checkpoints. Under FSDP with state_dict_type="sharded", a checkpoint is a directory of per-rank shards, not a single file. This is efficient to write, because every rank writes its own piece, and it resumes cleanly on the same topology. It is useless to a consumer who wants one file. To turn it into one:

python -m lightning.pytorch.utilities.consolidate_checkpoint path/to/checkpoint_dir

For checkpoints written through Fabric the equivalent is fabric consolidate <folder> --output_file out.ckpt. DeepSpeed ZeRO checkpoints have their own converter, convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, "out.pt") from lightning.pytorch.utilities.deepspeed. Make consolidation a pipeline stage that runs once after training, not a favour users ask each other for. Note also that a sharded resume on a different number of GPUs is a topology change, so test it before you promise it.

Where checkpoints live. Lightning writes through fsspec, so s3://, gs://, hdfs:// and abfs:// URLs work for default_root_dir, ModelCheckpoint(dirpath=...), logger save_dir, ckpt_path and load_from_checkpoint, provided the matching package (s3fs, gcsfs, adlfs) is installed. Two operational points follow. Checkpoint uploads to S3 and GCS are written atomically, which was improved in 2.6.5, so a job killed mid-upload does not leave a truncated file that a later resume trusts. And if you are subject to data-residency rules in the Gulf or in Egypt, the bucket region is part of your training design: pick it deliberately, and enforce it with bucket policy rather than with a convention in a README.

Not blocking the GPUs. Synchronous checkpointing stops training while the file is written. For large models the AsyncCheckpointIO plugin saves in a background thread, which you pass through plugins=[...]. The trade-off is that a checkpoint is no longer durable the instant the hook returns, so the failure you now have to reason about is "the job died while a save was in flight". Keep save_last=True plus a periodic cadence so that an earlier complete checkpoint always exists.

Cadence. Pick the checkpoint interval from the cost of lost work, not from habit. ModelCheckpoint(every_n_train_steps=...) or train_time_interval=... bound the lost work to a known amount of GPU time. Since 2.5.5, step- and time-triggered saves are deferred until validation metrics exist, so a monitored checkpoint does not fail on a missing key early in the run. Since 2.5.3, save_on_exception=True writes a checkpoint when the job crashes, and OnExceptionCheckpoint is the callback form.

Resuming. trainer.fit(model, ckpt_path="last") restores the optimizer, schedulers, loop position (including the place in the epoch) and callback state. Two traps matter at platform level. The first is save_last="link", which only works for local paths, so it is not an option on object storage. The second is that ckpt_path="best" requires a ModelCheckpoint with a monitor, otherwise you get the error beginning ".test(ckpt_path="best") is set but ModelCheckpoint is not configured to save the best model".

Fault tolerance, preemption and requeue

Large jobs run for days on hardware that is shared, preempted or simply unreliable. Fault tolerance in Lightning is not a separate subsystem. It is the combination of periodic checkpoints, a deterministic resume, and a scheduler that restarts the job. Your platform's job is to make the combination reliable.

On SLURM the integration is the most complete. In the batch script, #SBATCH --nodes must equal num_nodes, #SBATCH --ntasks-per-node must equal devices, and you launch with srun python train.py. Asking for --signal=SIGUSR1@90 tells SLURM to warn the job ninety seconds before the time limit. Lightning's SLURMEnvironment plugin catches that signal, saves a checkpoint, and requeues the job. Auto-requeue is on by default. You can change the signal with plugins=[SLURMEnvironment(requeue_signal=signal.SIGHUP)] or turn it off with SLURMEnvironment(auto_requeue=False). On requeue the resume path is ckpt_path="hpc", which Lightning selects automatically. If you let users disable auto-requeue, make that a visible, reviewed decision, since the failure mode is a long job that silently ends at the wall-clock limit.

On Kubernetes there is no built-in requeue, because Lightning does not run a control plane. The pattern is to run the job as a PyTorchJob (Kubeflow's training operator) or another gang-scheduled workload, to rely on KubeflowEnvironment detection for ranks, and to implement "resume" in your launcher: before trainer.fit, look in the checkpoint location for a last.ckpt and pass it as ckpt_path if present. That small wrapper is the Kubernetes equivalent of hpc. If you operate Kubeflow already, see Kubeflow for the operator side, and Kubernetes for the scheduling primitives (priority classes, node affinity and pod disruption budgets) that decide whether a training pod survives a node drain.

import os
import lightning as L

def resume_path(ckpt_dir: str) -> str | None:
    candidate = os.path.join(ckpt_dir, "last.ckpt")
    return candidate if os.path.exists(candidate) else None

trainer = L.Trainer(max_time="03:00:00:00")   # DD:HH:MM:SS
trainer.fit(model, datamodule=dm, ckpt_path=resume_path("/mnt/ckpts/run-42"))

Passing ckpt_path=None on a fresh run starts from scratch, so the same launcher works for the first attempt and every restart. For object storage, replace the os.path.exists check with an fsspec call.

The trust model: checkpoints are code

This is the section that changed most in the last year, and it is the one most likely to cost you a real incident.

A .ckpt file is a pickle, and loading a pickle can execute arbitrary code. That fact is old. What Lightning's recent history shows is that the defences have been repeatedly bypassed, so a platform cannot rely on a single flag.

The timeline matters:

• CVE-2026-31221 covered insecure deserialization in versions up to and including 2.6.0.
• PyTorch 2.6 changed the default of torch.load to weights_only=True, and Lightning 2.6.0 added a weights_only argument to Trainer.fit, validate, test and predict, defaulting to None, which lets torch decide.
• CVE-2026-58659 (GHSA-qqmf-gpg7-g8gw) was an arbitrary-code-execution flaw in load_from_checkpoint: a checkpoint could name its own _instantiator or _class_path in the stored hyperparameters and have Lightning import and call it, bypassing weights_only=True. Lightning 2.6.6 fixes it. The issue tracker confirms that 2.6.5 was still vulnerable, so the minimum safe version is 2.6.6.

The 2.6.6 fix does two things. It restricts the checkpoint's _instantiator to an allowlist, lightning.pytorch.core.saving._ALLOWED_INSTANTIATORS, which contains lightning.pytorch.cli.instantiate_module and pytorch_lightning.cli.instantiate_module. And it refuses a _class_path that does not resolve to an already imported subclass of the class you are loading. You will recognise both when they trigger, because the messages say the instantiator "is not in the allowlist of trusted instantiators" or that the class "does not resolve to an already imported subclass". The remedy for the second is benign: import the module that defines your model before calling load_from_checkpoint. Do not respond to the first by editing the allowlist unless you have verified the checkpoint's origin.

From this follows a set of platform rules.

1. Pin the floor. Your shared base package requires lightning>=2.6.6, and CI fails any project resolving to something older.
2. Keep torch patched too. The loader is only as safe as the PyTorch underneath it. PyTorch has its own advisory for remote code execution through untrusted checkpoint files (GHSA-63cw-57p8-fm3p, CVE-2026-24747). Track it in the same dependency policy.
3. Classify checkpoints by origin. Files your own jobs wrote into your own bucket are trusted. Files from a download, a partner, or a public hub are untrusted. For untrusted files use weights_only=True, and do not pass weights_only=False to make an error go away. The Unsupported global error from torch is a signal that the file contains arbitrary Python objects; the choices are to allowlist specific classes with torch.serialization.add_safe_globals([...]), or to keep hyperparameters primitive so the problem never arises.
4. Distribute weights, not training state. For anything crossing a trust boundary, prefer safetensors or a plain state-dict artifact over a full .ckpt. A safetensors file cannot carry code, which removes the whole class of problem.
5. Treat LightningCLI configs as code. A class_path: entry in YAML instantiates a class named by the file. A config read from --config s3://bucket/config.yaml means whoever controls that bucket controls your job. Lock down write access to config locations as you would to a deployment repo.

For where the secrets themselves should live, see Vault. Lightning reads no secrets of its own: logger credentials come from each tool's own variables (WANDB_API_KEY, COMET_API_KEY, the MLflow tracking variables), and storage credentials come from the fsspec backend, which means AWS environment variables or instance roles, or GCP application default credentials. The platform decision is to prefer short-lived workload identity over long-lived keys wherever your cloud supports it.

Observability: the standard signal set every job emits

When forty teams run Lightning jobs, the worst outcome is forty private definitions of "healthy". Your platform should publish a small, mandatory set of signals and make them the default in the shared base package, so an on-call engineer can read any job without learning its author's habits.

Lightning gives you the building blocks. Loggers adapt an experiment tracker: TensorBoardLogger, CSVLogger, MLFlowLogger, WandbLogger, CometLogger and, since 2.6.1, LitLogger. You can pass several at once, for example Trainer(logger=[TensorBoardLogger("logs/"), CSVLogger("logs/")]). Only rank 0 logs by default, which is what you want for metrics but means a problem on rank 5 is invisible unless you arrange otherwise. Callbacks add the rest: LearningRateMonitor(logging_interval="step") for schedule bugs, DeviceStatsMonitor for GPU memory and utilisation, and ThroughputMonitor(batch_size_fn=...) for samples per second, TFLOPs and MFU. The ModelSummary callback records the parameter count at the start.

A reasonable platform contract has four tiers.

Provenance is the one teams skip. Log it as hyperparameters or as tags in the tracker, but not as secrets. If your tracker is MLflow, MLflow covers the server side, and mlflow.pytorch.autolog() supports Lightning. If it is Weights & Biases, see Weights & Biases. One honest warning about trackers: NeptuneLogger no longer works. Neptune was sunset, version 2.6.4 removed the logger from lightning.pytorch.loggers, and the stub that remains raises RuntimeError: NeptuneLogger is no longer supported. Neptune has been sunset. If a team still has it in a template, that is a migration you own, and it is a good example of a breaking change arriving in a patch release.

Alerting on stalls, not on errors. Because the characteristic failure is a hang, alert on the absence of progress. The signal is simple: the step counter has not advanced for N minutes while the job is still marked running. Export step and throughput to your metrics system and alert on the derivative. The surrounding stack is covered in Prometheus and Grafana, and if you already run tracing, OpenTelemetry is the natural place for a launcher-level span around the whole job. Lightning itself does not emit Prometheus metrics, so the bridge is a small callback that you own.

import time
import lightning as L
from lightning.pytorch.callbacks import Callback
from lightning.pytorch.utilities import rank_zero_only

class HeartbeatCallback(Callback):
    """Writes a heartbeat file that a sidecar or the scheduler can watch."""

    def __init__(self, path: str, every_n_steps: int = 50):
        self.path = path
        self.every_n_steps = every_n_steps

    @rank_zero_only
    def on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx):
        if trainer.global_step % self.every_n_steps == 0:
            with open(self.path, "w") as f:
                f.write(f"{trainer.global_step} {time.time()}\n")

The decorator keeps the file write on rank 0, and nothing in the hook is a collective, so it cannot cause the hang this guide warned about earlier. A liveness probe or a sidecar can then compare the timestamp with the clock.

Multi-tenancy and isolation

Lightning has no multi-tenancy of its own. There are no accounts, no quotas and no namespaces inside the library. Isolation is therefore entirely a property of the systems around it, and your design work is deciding where each boundary sits.

Compute. On Kubernetes, use namespaces with resource quotas and priority classes per team, and see Kubernetes for the primitives. On SLURM, use accounts, partitions and fair-share. Both let you cap GPU hours per team and preempt low-priority work for high-priority work. Lightning's requeue behaviour is what makes preemption affordable, so the two policies reinforce each other.

Storage. Give each run a unique default_root_dir and logger version. Concurrent runs that share a lightning_logs/ directory get automatic version_N increments, but checkpoint directories can collide, and enable_version_counter=True in ModelCheckpoint only appends -v1 style suffixes to avoid overwriting; it does not give you isolation. The launcher should build the path from team, project and run identifier, and the bucket policy should only allow each team's identity to write under its own prefix.

Tracking. MLflow experiments, W&B projects and teams, and Lightning.ai teamspaces are the isolation units there. Decide whether tracking is shared read-only across the company or private per team, and apply the same rule to checkpoints, because a tracker that is private while the bucket is public leaks the same information.

Cost and capacity governance

A training platform spends money in large, visible units, and most of the savings are available through a handful of Lightning settings plus scheduling policy. The levers, roughly in order of payoff:

1. Do not burn GPU hours on bugs. fast_dev_run=True runs one train, one validation and one test batch with loggers and checkpointing disabled, and limit_train_batches and limit_val_batches shrink a run for CI. A pipeline that runs a smoke job on CPU or one small GPU before queuing the expensive job catches most mistakes cheaply.
2. Use the right precision. precision="bf16-mixed" on Ampere-class and newer GPUs or TPU roughly halves activation memory and speeds up matrix multiplications, and "16-mixed" on older GPUs uses a gradient scaler. "bf16-true" and "16-true" keep weights in half precision and save more memory, at a stability cost that you should test per model. Also call torch.set_float32_matmul_precision("high") where TF32 is acceptable, which Lightning hints at on Ampere.
3. Compile where it pays. torch.compile(model) can be passed to trainer.fit. The benefit depends on the model, and compile time is a cost on short jobs, so measure rather than mandate.
4. Right-size the batch. Tuner(trainer).scale_batch_size(model, mode="power") finds the largest batch that fits, and it needs a batch_size attribute on the model or datamodule. Remember the effective-batch and learning-rate relationship before accepting the result.
5. Stop early and stop on time. EarlyStopping with sensible patience, remembering patience counts validation checks and not epochs, and max_time as a hard budget.
6. Use spot capacity. Safe because of periodic checkpoints and resume, as described earlier.
7. Watch MFU. A job running at low MFU on expensive GPUs is the most common silent waste. ThroughputMonitor is the cheapest way to find it.

The defaults that cost money. If you set neither max_epochs nor max_steps, the Trainer defaults to 1000 epochs. A user who forgets this and has a fast epoch loop can run for a very long time. Set max_epochs=-1 only on purpose, and make the platform require an explicit budget: either epochs, steps or max_time. Likewise, Trainer(accelerator="auto", devices="auto") will use every device it can see, which on a shared node means a small job quietly takes the whole machine unless the scheduler restricts visible devices.

Upgrades, migrations and the stale-tutorial problem

Lightning does not follow semantic versioning, and your upgrade policy has to be built around that. The documented rules are: a patch release contains bug fixes only; a minor release may contain backwards-incompatible changes, with deprecations first; a major release may break without deprecations. Everything is stable unless explicitly marked experimental. The project aims to support the latest five PyTorch minor releases.

There is a caveat, and this guide has already shown it. Version 2.6.4 removed the Neptune logger inside what looks like a patch bump, because the service behind it was sunset. The rule "patch releases are safe" is a good default, not a law, so read release notes for every bump and keep a canary.

A pinning policy. Pin lightning, torch and torchmetrics together as one unit in your base image, and resolve the lock file in the build image. Python must be 3.10 or newer, since 3.9 support ended in 2.6.1. The CI matrix at 2.6.6 covers PyTorch 2.1 through 2.10 and Python 3.10 through 3.13, but note that the compatibility table on the official versioning page stops at the 2.5 row and has no 2.6 row. For 2.6, treat the package metadata on PyPI as the authority, and verify on your own hardware.

A canary process.

Deprecations to retire now. Several APIs still work in 2.6.6 but warn. LightningModule.configure_sharded_model is replaced by configure_model, and overriding both raises a RuntimeError. LightningModule.to_torchscript is deprecated because TorchScript is deprecated in PyTorch. The warning text says it was deprecated in v2.7 and will be removed in v2.8, and that wording is what the source contains, even though the warning ships from 2.6.1, so treat the removal as coming soon. The replacements are torch.export.export(model, (x,)) or model.to_onnx(..., dynamo=True), which returns an ONNXProgram. DDPStrategy.is_distributed and the LightningDoublePrecisionModule wrapper are also deprecated.

Governance and incident playbooks

Governance for a library platform is mostly about three things: who may change the shared base, what a job must declare before it runs, and how you respond when something goes wrong. Keep the first two light and the third rehearsed.

What a job declares. Before a job is queued, the launcher should be able to read: team, project, run identifier, expected duration or max_time, strategy, number of nodes and devices, checkpoint location and retention class. Everything else is the team's business. This declaration is the basis for chargeback, quota checks and the pre-flight allocation assertion discussed earlier.

Three playbooks follow. Each states symptom, first check and fix, because during an incident you read, you do not reason from first principles.

Playbook one: the job is running but nothing is logged.

1. Confirm the scheduler says the job is alive and look at GPU utilisation. If utilisation is zero on all ranks, the job is likely blocked in a collective or in rendezvous.
2. Check allocation against arguments: does devices × num_nodes equal the number of started ranks?
3. Re-run a short reproduction with NCCL_DEBUG=INFO and PYTHONFAULTHANDLER=1, and look for ranks stopping at different places.
4. Search the code for collectives inside rank conditionals, with sync_dist=True inside if self.global_rank == 0: as the first suspect.
5. Check whether MASTER_PORT is reachable between nodes.

Playbook two: a checkpoint will not load.

Playbook three: a vulnerable version is found. For a disclosure like CVE-2026-58659: inventory every image and lock file with lightning below 2.6.6; block new builds on those versions; patch the base and roll it out through your canary ring; then audit where untrusted checkpoints were loaded in the exposure window. Do not wait to be certain nobody loaded a hostile file. The inventory step is only quick if the first question in the upgrade section, "which images run which version", already has an answer.

Where Lightning stops

A senior owner knows when not to use the tool. Lightning is excellent at one job, which is removing engineering boilerplate from single-program-multiple-data PyTorch training. It is not the right answer to everything near that job.

It is not a scheduler. Lightning has no queue, no gang scheduling, no quotas and no retry logic outside SLURM requeue. If you find yourself writing a scheduler in a callback, stop and use Kubeflow, SLURM, or a cloud batch service. For distributed Python workloads beyond training, Ray is a different execution model, and Ray Train has its own Lightning integration.

It is not a serving stack. lightning.pytorch.serve.ServableModule and the ServableModuleValidator callback exist but are experimental. The Lightning ecosystem points to LitServe, a separate package, for production serving. For model serving as a platform, look at BentoML, KServe or Triton, and export through torch.export.export(), to_onnx(..., dynamo=True) or to_tensorrt(...), which was added in 2.6.0.

It is not a custom-loop framework any more. Custom Loop objects and trainer.fit_loop.replace were removed. If a research team's algorithm does not fit the fit, validate, test and predict loops, or needs unusual control of the step, the supported answer is Fabric. fabric = Fabric(accelerator=..., devices=..., strategy=..., precision=...) handles devices, precision and distribution while you keep your own loop. Consider it for reinforcement learning, unusual multi-model schedules, or anything where you spend more time fighting the Trainer than using it. Manual optimization inside a LightningModule covers GANs and multiple optimizers, and it is the first thing to try before leaving the Trainer.

Running it as a platform for other teams

Everything above converges on one product: a golden path, a way of running Lightning jobs that is the easiest option and is also the safe one. The golden path has four parts.

The base package. A small internal library, versioned and released like any product, that depends on lightning>=2.6.6 and pins its companions. It provides a build_trainer(...) function that fills in the platform's defaults: the standard callback set, loggers, checkpoint location, max_time, heartbeat, and the SLURM or Kubernetes environment settings. Teams call it instead of constructing L.Trainer directly, and they can override arguments explicitly and visibly.

import lightning as L
from lightning.pytorch.callbacks import ModelCheckpoint, LearningRateMonitor, DeviceStatsMonitor

def build_trainer(run_dir: str, max_time: str, **overrides) -> L.Trainer:
    """Platform defaults. Overrides are allowed, silence is not."""
    callbacks = [
        ModelCheckpoint(
            dirpath=f"{run_dir}/checkpoints",
            save_last=True,
            every_n_train_steps=1000,
            save_on_exception=True,
        ),
        LearningRateMonitor(logging_interval="step"),
        DeviceStatsMonitor(),
    ]
    defaults = dict(
        max_time=max_time,
        default_root_dir=run_dir,
        callbacks=callbacks,
        log_every_n_steps=50,
    )
    defaults.update(overrides)
    return L.Trainer(**defaults)

This is a sketch, not a drop-in. Notice what it does not do: it does not hide the strategy, which stays a visible argument, and it does not catch exceptions. The goal is defaults and guardrails, not a second framework on top of the first.

The launcher. A command or service that takes a declared job, validates it, injects identity and paths, selects the image, submits to the scheduler, and implements the resume wrapper. Keep it boring. Its most valuable feature is the pre-flight check: allocation versus arguments, checkpoint location writable, version floor satisfied.

The templates. A repository of working examples that pass your own lint rules: one DDP, one FSDP, one with LightningCLI, one resuming from a checkpoint. A template that runs in CI on every release of the base package is documentation that cannot go stale, which cannot be said of a wiki page. The delivery pipeline is ordinary CI/CD, and GitHub Actions and Argo CD are common homes for it. Infrastructure around the platform is usually described with Terraform and Helm.

Region and residency. For employers in the Gulf and in Egypt, residency obligations commonly decide where data, checkpoints and logs may live. Because Lightning reads and writes through fsspec URLs, residency is enforceable at the storage layer: region-pinned buckets, policy that denies cross-region writes, and trackers hosted in the same jurisdiction. The Trainer will happily write a checkpoint wherever you point it, so the control must exist outside the library. If you use any hosted Lightning.ai service, check where it stores data before you adopt it.

The review checklist

Use this for design documents and pull requests that touch your platform. Each item names the failure it prevents.

Version and supply chain

• Does the image resolve lightning to 2.6.6 or newer, with torch and torchmetrics pinned together? Prevents the checkpoint-loading vulnerabilities.
• Is Python 3.10 or newer? Prevents resolving an unsupported combination.
• Is the code using one namespace, either lightning or pytorch_lightning, not both? Prevents type failures.

Distributed correctness

• Does every sync_dist=True log, barrier and collective run on all ranks? Prevents silent hangs.
• Do devices and num_nodes match the scheduler allocation, and is there a pre-flight assertion? Prevents ranks waiting forever.
• Is the effective batch size recorded, and was the learning rate reconsidered when it changed? Prevents quietly worse models.
• Is the entry point under if __name__ == "__main__":? Prevents spawn recursion.

Checkpoints

• Is there a resume checkpoint with save_last=True and a cadence bounded by lost-work cost?
• Is a release artifact produced separately, weights only and ideally safetensors?
• Are hyperparameters primitive, with ignore=[...] for modules and no secrets?
• For sharded strategies, is consolidation a pipeline stage?
• Is the checkpoint location region-correct and policy-protected?

Security

• Are untrusted checkpoints loaded with weights_only=True, and is weights_only=False justified by origin in a comment?
• Who can write to the config locations? Configs are code.
• Do credentials come from workload identity or environment, never from constructor arguments?

Cost and operations

• Is there an explicit budget: max_epochs, max_steps or max_time? Prevents the 1000-epoch default.
• Does a smoke test with fast_dev_run=True run before the expensive job?
• Are throughput, MFU, GPU stats and a heartbeat emitted?
• Is the job's team, project and run identifier labelled for chargeback?

Lifecycle

• Are deprecation warnings visible, and are removed APIs absent from the code?
• Is there a canary result for the version in use?
• Is there a runbook link in the job metadata?

The complete picture

Put the whole series together and Lightning looks like this. At the lowest level it is a LightningModule that holds a recipe, a Trainer that runs the loops, and a strategy that decides how many processes cooperate and how a checkpoint is written. Around that sit callbacks and loggers, the extension points that you standardise into a signal set.

At the platform level the picture inverts. The library is the small part. The large part is the surrounding system: a scheduler that allocates what the job asks for and restarts it when it dies; storage that holds checkpoints with the right region, retention and access; a tracker that shows what happened; an identity layer that says who may read what; and an upgrade process that keeps all of it current. Lightning connects these through a few narrow interfaces, namely cluster environments, fsspec paths, loggers and plugins, and your craft is keeping those interfaces consistent.

Three ideas carry most of the weight. First, hangs, not errors, are the characteristic failure, so you build detection for the absence of progress. Second, a checkpoint is both your recovery mechanism and your largest attack surface, so you separate resume checkpoints from release artifacts and treat the loader as a trust boundary. Third, a library without a control plane means every guarantee comes from conventions that the golden path enforces, so the leverage is in defaults rather than in documentation.

Where the series leaves you

You started this series by running a model through a Trainer. You now own the infrastructure that lets other teams do the same safely: a base package, a launcher, a checkpoint contract, a trust model, an observability set, an upgrade process and a set of rehearsed responses. That is the actual scope of the role, and almost none of it is about writing a better training_step.

What to do next depends on where your platform is weakest. If you cannot yet answer "which version does each image run", fix that first, because it gates every security response. If hangs are your biggest pain, build the heartbeat and the pre-flight check. If cost is, instrument MFU and add the smoke stage. If teams keep fighting the Trainer, introduce Fabric as a supported lane.

Sources

• PyTorch Lightning documentation (stable)
• Installation
• Versioning policy
• Migration guide
• Trainer
• LightningModule
• LightningCLI
• Checkpointing
• Checkpoint migration
• Remote filesystems
• Precision
• FSDP
• DeepSpeed
• Cluster training
• Loggers
• Callbacks
• Profiling
• Production deployment
• Release 2.6.6
• Release 2.6.4
• Issue 21913
• Changelog at 2.6.6
• lightning on PyPI

Senior interviews for a Lightning platform role rarely ask what configure_optimizers returns. They hand you a situation and watch how you reason: what you measure first, what you refuse to guess, which trade-offs you name, and whether you can explain the decision to someone who pays the bill. This file is organised the same way. It starts with system-design prompts, moves to incident questions, then to the conversations with leadership, and ends with a self-test.

One habit runs through every answer below. Say the version you are assuming. This guide is written against Lightning 2.6.6, and an interviewer who hears "I would pin lightning>=2.6.6 because 2.6.5 was still vulnerable" learns in one sentence that you follow the project's security history, not just its tutorials.

The thirty-second answer

If you are asked what you would own as the platform owner for Lightning, say this.

"Lightning is a library, not a service, so there is no control plane to run. What I own is everything around it: a base package that fills in safe defaults, a launcher that validates jobs and injects identity and paths, a checkpoint contract that separates resume files from release artifacts, a trust model that treats checkpoints and configs as code, a standard set of signals, and an upgrade process built around the fact that the project does not follow SemVer. Failures in distributed training are usually silent hangs, so I invest in detecting missing progress rather than catching exceptions."

That answer shows you know the shape of the problem. Everything else in the interview is you proving it.

System-design prompts

Prompt one: "Design a training platform for twenty teams on a shared GPU cluster."

A weak answer lists strategies and precision flags. A strong answer begins with the layers Lightning does not provide and asks clarifying questions before drawing anything: which scheduler exists already, how large the biggest model is, what the preemption policy is, where data must reside, and who consumes the checkpoints.

Then structure the design in four parts.

Scheduling and identity. Lightning has no queue, so the scheduler is SLURM or Kubernetes with a gang-scheduling operator such as Kubeflow's PyTorchJob. Each team gets a namespace or account with quotas and a priority class. A job's identity is the identity of the pod or SLURM user, and that identity is what grants write access to the team's own storage prefix. State plainly that Lightning reads no credentials, so isolation comes from the systems around it.

The golden path. A base package that depends on lightning>=2.6.6, pins torch and torchmetrics with it, and exposes a function that builds the Trainer with platform defaults: a standard callback set, periodic checkpoints with save_last=True, an explicit time budget through max_time, and logger configuration. A launcher validates a declared job, rejects mismatches between scheduler allocation and devices × num_nodes, and implements resume by looking for last.ckpt and passing it as ckpt_path.

Checkpoints and artifacts. Resume checkpoints are full fidelity and short lived. Release artifacts are weights only, immutable, and recorded in a registry with the Lightning version that wrote them. Sharded strategies get a consolidation stage after training.

Signals and governance. Progress, throughput, resource and provenance signals for every job, an alert on a stalled step counter, chargeback labels, and a canary upgrade process.

What the interviewer is listening for: do you put the strategy choice behind a decision rule, such as "stay on DDP until memory forces you off, because DDP is the simplest and best supported"? Do you mention that switching the strategy changes the checkpoint format? Do you keep the abstraction thin, so that a team can see and override what you set?

Prompt two: "A team wants to train a model that no longer fits on one GPU. Walk me through the decision."

Start by refusing to jump. First check whether the problem is really memory. If GPUs show low utilisation, sharding will not help, because it only fixes memory, and the real bottleneck may be the data pipeline. Measure with ThroughputMonitor and DeviceStatsMonitor.

If memory is truly the limit, walk the ladder in order of invasiveness. Cheap changes first: precision="bf16-mixed" on hardware that supports it, a smaller per-device batch with accumulate_grad_batches to keep the effective batch, and activation checkpointing. If that is not enough, shard. Describe the options honestly.

• FSDP (strategy="fsdp") shards parameters, gradients and optimizer state across ranks, is PyTorch-native, and needs layers created in configure_model() so the full model is never materialised on one device. Its checkpoints can be sharded directories, which means a consolidation step for consumers.
• DeepSpeed ZeRO (deepspeed_stage_2, deepspeed_stage_3, with optional offload) is Linux-only, and offload trades memory for step time.
• ModelParallelStrategy combines tensor parallelism with FSDP2 on a device mesh, and is marked experimental.

Give a recommendation, not a menu. Something like: "I would default the platform to FSDP, offer DeepSpeed for teams that already have configurations and expertise, and make ModelParallelStrategy an opt-in track with a named owner, because an experimental API is a risk I should take deliberately."

Then add the second-order consequences, which is what separates senior from mid. The checkpoint format changes. Effective batch size changes if the number of devices changes, so the learning rate must be revisited. A sharded resume on a different number of GPUs is a topology change that should be tested before it is promised. And mixed-precision under FSDP had a bug fixed in 2.6.4, where half-precision modes initialised parameters in half precision, so version matters.

Prompt three: "Design the checkpoint and model-release flow."

Lead with the distinction. A .ckpt is a training artifact: it holds the state dict, optimizer and scheduler state, loop position, callback state, hyperparameters, datamodule state and the writer's Lightning version. It is large and it is a pickle. A release artifact is something else, and conflating the two is the root of most checkpoint trouble.

The flow: training writes resume checkpoints to the team's prefix, with a retention window. On success, a pipeline stage consolidates if the strategy was sharded, extracts weights, writes them in a format that cannot execute code (safetensors, or a plain state dict), records metadata including the Lightning version, config hash and git commit, and registers an immutable version. Downstream systems consume the registry entry only.

Discuss the trust boundary explicitly. Your own jobs writing to your own bucket are trusted. Anything arriving from outside is untrusted and is loaded with weights_only=True, never with weights_only=False to silence an error. Mention the 2.6.6 loader hardening: the checkpoint's _instantiator is restricted to an allowlist and _class_path must resolve to an already imported subclass. That shows you understand the loader is part of the attack surface.

A strong close: "Serving does not load Lightning checkpoints at all. It receives an exported artifact: torch.export.export, ONNX through to_onnx(..., dynamo=True), or TensorRT through to_tensorrt. I avoid to_torchscript, since it is deprecated."

Prompt four: "How would you run Lightning jobs on Kubernetes when preemption is common?"

The honest starting point is that Lightning has no control plane, so there is no requeue on Kubernetes the way there is on SLURM. SLURM integration catches a warning signal and requeues, and on requeue resumes with ckpt_path="hpc". On Kubernetes you build the equivalent.

Run the job as a gang-scheduled workload. Rely on the Kubeflow environment detection for ranks. Write a launcher that, before fit, checks the checkpoint location for last.ckpt and passes it as ckpt_path. The same entry point then serves a fresh start and every restart. Checkpoint cadence is derived from the cost of lost work using every_n_train_steps or train_time_interval, and save_on_exception=True adds a save when the job crashes.

Raise the design constraint that surprises candidates: the preemption notice has to be longer than the checkpoint write time, or the notice is useless. So you measure checkpoint write time for the largest model and, if it is long, use AsyncCheckpointIO, and then describe the new failure it introduces: a save may be in flight when the job dies, which is why an earlier complete checkpoint must always exist.

Close with the cross-cutting pieces: scheduler priority classes and pod disruption budgets decide whether a training pod survives a node drain, and a job-level wall-clock timeout catches the hangs that Lightning cannot.

Incident-response questions

Interviewers use incidents to test method. The pattern for any answer is the same: stabilise, observe, form a hypothesis from evidence, change one thing, and write down what you learned.

"Forty jobs hung at the same time. What do you do?"

Forty simultaneous hangs suggest a shared cause, not forty bugs. Say so, then check what the jobs have in common: the same node pool, the same image or Lightning version, the same storage system, the same network fabric, the same time. A rolling image update, a storage outage or a switch failure produce this pattern. Individual code bugs do not.

First actions: confirm whether GPUs show zero utilisation, which points at blocked collectives or rendezvous. Check recent changes to the base image, the launcher and the cluster. Look at the storage backend for the checkpoint location, because a hung filesystem call inside checkpointing looks identical to a distributed hang. On a sample job, run with NCCL_DEBUG=INFO and PYTHONFAULTHANDLER=1 on a short reproduction and see where ranks stop.

Then mitigate: kill hung jobs so they stop consuming quota, and resume them from last.ckpt, which is the reason that cadence matters. The follow-up is prevention: a stalled-step alert that would have fired in minutes, and a job-level timeout.

"A single job hangs only at validation. Why?"

Suspect a collective on a subset of ranks. The classic cause is self.log(..., sync_dist=True) inside an if self.global_rank == 0: block, so rank 0 waits in a reduction for peers who skipped the call. Another cause is uneven validation data across ranks, where one rank runs out of batches and the others wait. Explain that the rule is every rank must reach every collective, and that if only one rank should write something, rank_zero_only=True or the @rank_zero_only decorator is for side effects such as files, not for synchronising calls.

"A researcher says the loss became NaN after you upgraded the base image."

Do not blame the upgrade first and do not dismiss it. Establish what changed besides Lightning: torch version, CUDA build, precision, the data. Reproduce on a short run with detect_anomaly=True to find the first operation that produces non-finite values, accepting that it is slow. Check precision: instability that appears in 16-mixed and not in bf16-mixed or 32-true suggests a range problem and gradient scaling. Log gradient norms through on_before_optimizer_step and grad_norm. Bisect between the two image versions on the same data and seed, using L.seed_everything(42, workers=True), and remember that exact repeatability needs Trainer(deterministic=True).

If FSDP with half-precision mixed modes is involved, mention that a bug where parameters were initialised in half precision instead of fp32 was fixed in 2.6.4, so an old version could be the culprit in the other direction.

"We found a vulnerable Lightning version in production images. Respond."

This tests security reflexes. State the facts: the arbitrary code execution flaw in load_from_checkpoint is CVE-2026-58659, fixed in 2.6.6, and the issue tracker confirms 2.6.5 was still affected. An earlier deserialization issue, CVE-2026-31221, affected versions up to 2.6.0.

Sequence: inventory every image and lock file by installed lightning version; block new builds below the floor; patch the base and roll out through the canary ring; then investigate exposure. The exposure question is "where did we load checkpoints we did not produce, and from whom", because the risk is a hostile file, not the mere presence of the version. Finally, keep torch patched too, since PyTorch has its own advisory for untrusted checkpoint files.

A senior answer admits the uncomfortable part: the inventory is only fast if you could already answer "which image runs which version" before the incident.

"A job fails with Weights only load failed ... Unsupported global. A teammate wants weights_only=False."

This is a judgement test. Explain the cause: torch 2.6 changed the torch.load default to weights_only=True, and the checkpoint contains arbitrary Python objects, commonly custom classes saved as hyperparameters. Then say that weights_only=False is acceptable only for a file whose origin you have verified, and that the durable fix is to keep hyperparameters primitive and use save_hyperparameters(ignore=[...]) for modules. For a known trusted class, torch.serialization.add_safe_globals([...]) is the narrower option. Ask where the file came from before anything else.

Trade-off questions

"DDP or FSDP as the platform default?"

Answer with a rule, not a preference. DDP replicates the full model and optimizer state on each device, so it is the simplest and best supported, and it is the right default when the model fits. FSDP is the default for models that do not fit. Name the cost of each: DDP's cost is memory, FSDP's cost is operational complexity, mainly sharded checkpoints and topology-dependent resumes. The platform default should be the one most teams need, and the move to FSDP should be triggered by a measured condition, not a feeling.

"LightningCLI with YAML, or plain Python entry points?"

LightningCLI gives you reproducibility and a resolved configuration that you can review, store and diff, with --print_config producing the full file. Its cost is that configs can instantiate arbitrary classes through class_path, so they are code, and a config loaded from a remote location is controlled by whoever controls that location. Plain Python is easier to debug and has a smaller surface, but it makes standard behaviour harder to enforce. A balanced answer: use LightningCLI for the golden path, review resolved configs like code, and restrict write access to config locations.

"When would you tell a team to leave the Trainer?"

When they are fighting it. Custom loops were removed from Lightning, so unusual algorithms belong in Fabric, which gives device, precision and distribution handling with your own loop. Manual optimization covers GANs and multiple optimizers inside the module, and that is the first step before leaving. For fine-tuning large language models, purpose-built tooling may encode recipes you would otherwise write.

Arguing a trade-off to leadership

Leadership does not want strategies. They want risk, cost and time in terms they own. Three scripts follow.

Asking for time to upgrade. "Lightning does not follow semantic versioning, minor releases can remove deprecated APIs, and one patch release removed a logger outright. Our current version is below the one that fixes a remote code execution flaw in checkpoint loading. Two engineer-weeks now buys a canary process, so every future upgrade is a routine afternoon. The alternative is upgrading in a hurry during the next disclosure."

Asking for budget for a smoke-test stage. "A GPU job that fails an hour in costs us the hour on every GPU. A CPU smoke test with fast_dev_run=True catches the same mistake in a minute. Last quarter the cost of failed jobs was X, and I expect the stage to remove most of it." Use your own measured numbers, never invented ones, and say where they come from.

What a strong answer covers that a weak one misses

• Version awareness. Strong answers cite 2.6.6 and the reasons. Weak answers recite old APIs such as Trainer(gpus=...), resume_from_checkpoint or training_epoch_end, which were removed in 2.0.
• Hangs over errors. Strong answers design detection for missing progress. Weak ones add try/except.
• Strategy and checkpoint coupling. Strong answers know that changing strategy changes the artifact.
• Trust. Strong answers treat checkpoints and configs as code. Weak ones treat weights_only=False as a fix.
• Boundaries. Strong answers say what Lightning is not: scheduler, serving stack, custom-loop framework.
• Measurement. Strong answers start with data: MFU, utilisation, checkpoint write time.
• People. Strong answers include onboarding, support lines and ownership.

Sixty-second self-test

Answer these aloud without notes. Then check against the sections above.

1. What does Lightning own and what does it not, and which does that make you responsible for?
2. Why is a hang the characteristic failure, and what is the first alert you add?
3. Name three differences between a resume checkpoint and a release artifact.
4. What did 2.6.6 change in load_from_checkpoint, and why is 2.6.5 not safe?
5. How do you resume a job on Kubernetes without SLURM's requeue?
6. When do you move from DDP to FSDP, and what changes besides memory?
7. What is the effective batch size formula, and why does it matter when a job is resized?
8. Why can self.log(..., sync_dist=True) inside a rank-zero condition hang a job?
9. Which two APIs did 2.6.x deprecate or remove that old templates still use?
10. Give one thing Lightning should not be asked to do, and what to use instead.

For the neighbouring tools, Kubeflow, Kubernetes and MLflow are the three interviewers most often ask about next.

These are the traps that do not show up in a single-GPU notebook. They appear when many teams, many ranks and many weeks are involved, and each one is paired with the symptom you will see, the cause, and the fix. After the traps come the habits of someone who owns the platform, then review checklists you can paste into pull requests and design documents. Everything here assumes Lightning 2.6.6.

Production traps that only appear at scale

Trap one: the silent hang from a partial collective

Symptom. A job starts, trains for a while or reaches validation, and then stops. No error, no traceback, GPUs at zero utilisation.

Cause. One rank reached a call that needs every rank, and the others did not. The textbook case is self.log("val_loss", loss, sync_dist=True) inside if self.global_rank == 0:. Another cause is a validation set that gives some ranks fewer batches than others.

Fix. Any call that synchronises, including a log with sync_dist=True, must run on every rank. Use rank_zero_only=True or the @rank_zero_only decorator for side effects such as writing a file. Add a job-level timeout, and use NCCL_DEBUG=INFO plus PYTHONFAULTHANDLER=1 to see where ranks stopped.

# wrong: only rank 0 reaches the reduction
if self.global_rank == 0:
    self.log("val_loss", loss, sync_dist=True)

# right: every rank logs, the framework decides who writes
self.log("val_loss", loss, sync_dist=True)

Trap two: the scheduler and the Trainer disagree

Symptom. A multi-node job sits in initialisation forever, or starts with fewer ranks than expected.

Cause. Trainer(num_nodes=N, devices=G) does not match the allocation. On SLURM, --nodes must equal num_nodes and --ntasks-per-node must equal devices. Lightning does not reconcile them for you.

Fix. Make the launcher assert the expected world size before it constructs the Trainer, and fail fast with a clear message. A two-line check saves hours of expensive idle GPUs.

Trap three: the 1000-epoch default

Symptom. A job that was meant to run for a day is still running a week later.

Cause. If neither max_epochs nor max_steps is set, the Trainer uses 1000 epochs. A fast epoch loop makes that a very long time.

Fix. Require an explicit budget in the platform: epochs, steps or max_time. Use max_epochs=-1 only when infinite training is intentional.

Trap four: devices="auto" on a shared node

Symptom. One user's small experiment occupies every GPU on a node, and another team's job cannot start.

Cause. accelerator="auto" and devices="auto" use everything visible to the process.

Fix. Let the scheduler restrict visible devices, with cgroup or CUDA_VISIBLE_DEVICES controls, and have the launcher set devices explicitly rather than relying on auto.

Trap five: effective batch size changes quietly

Symptom. After a team doubled their GPU count to go faster, the model is slightly worse.

Cause. The effective batch size is the per-device batch times devices times nodes times accumulate_grad_batches. Doubling devices doubles it, and the learning rate no longer fits.

Fix. Log the effective batch size at the start of every run, and require the learning rate to be revisited whenever it changes. Also remember global_step counts optimizer steps, not batches, so step-based schedules and checkpoint intervals change meaning with accumulation.

Trap six: sharded checkpoints surprise their consumers

Symptom. An evaluation script fails to load a checkpoint, saying the path is a directory.

Cause. With FSDP and state_dict_type="sharded", a checkpoint is a directory of per-rank shards.

Fix. Make consolidation a pipeline stage:

python -m lightning.pytorch.utilities.consolidate_checkpoint path/to/checkpoint_dir

For DeepSpeed ZeRO checkpoints use convert_zero_checkpoint_to_fp32_state_dict(ckpt_dir, "out.pt") from lightning.pytorch.utilities.deepspeed. Test resuming a sharded checkpoint on a different number of GPUs before you promise it works.

Trap seven: secrets inside the checkpoint

Symptom. A security scan finds an API key in a checkpoint, in the tracker's hyperparameters and in config.yaml in the log directory.

Cause. save_hyperparameters() writes constructor arguments into every checkpoint and into every logger's record, and LightningCLI writes the resolved configuration to the log directory. A credential passed as an __init__ argument lands in all of them.

Fix. Credentials come from environment variables, workload identity or a secret manager. Use save_hyperparameters(ignore=[...]) as a backstop. See Vault for the secret store.

Trap eight: weights_only=False as a reflex

Symptom. After the PyTorch 2.6 upgrade, loading fails with Weights only load failed ... Unsupported global. Someone adds weights_only=False everywhere.

Cause. Torch 2.6 changed torch.load to default to weights_only=True, and the checkpoint holds arbitrary Python objects, usually custom classes among the hyperparameters.

Fix. Keep hyperparameters primitive. For a known trusted class, allowlist it with torch.serialization.add_safe_globals([...]). Reserve weights_only=False for files whose origin you have verified, and write that verification in a comment next to the call. This matters more since the 2.6.6 loader hardening, because a checkpoint that names its own instantiator or class is exactly what an attacker uses.

Trap nine: relying on a patch release being harmless

Symptom. A template that worked last month raises RuntimeError: NeptuneLogger is no longer supported.

Cause. Version 2.6.4 removed the Neptune logger in a patch release because the service was sunset. Lightning does not follow SemVer: minor releases may break with deprecations, and exceptions happen in patches.

Fix. Read release notes for every bump, run a canary suite, and keep your tracker behind a small wrapper so a logger can be replaced in one place.

Trap ten: an old tutorial in your template repository

Symptom. A new service uses Trainer(gpus=2), resume_from_checkpoint= or training_epoch_end(self, outputs) and fails, or worse, someone adds a compatibility shim.

Cause. These were removed in 2.0 and still circulate in blog posts and old internal code.

Fix. Use accelerator="gpu", devices=2, trainer.fit(..., ckpt_path=...), and on_train_epoch_end with your own accumulation. Add a lint rule that rejects the removed names so they fail in CI. Also note the other deprecated APIs still present: configure_sharded_model (use configure_model) and to_torchscript (use torch.export.export() or to_onnx(..., dynamo=True)).

Trap eleven: mixed namespaces

Symptom. The Trainer rejects a model with a message that it must be a LightningModule or a compiled module, although the class clearly subclasses LightningModule.

Cause. The module subclasses pytorch_lightning.LightningModule while the Trainer comes from lightning, or the reverse. They are the same code under two namespaces and are not interchangeable.

Fix. Standardise on import lightning as L and from lightning.pytorch import ... across the platform, and check imports in CI.

Trap twelve: the experiment that cannot be reproduced

Symptom. A rerun with the same configuration gives a slightly different result, and the team wants to know which one is right.

Cause. Seeds were not set for data-loader workers, or nondeterministic kernels were allowed.

Fix. L.seed_everything(42, workers=True) and Trainer(deterministic=True) where exactness matters, accepting the speed cost. Log the Lightning, torch and torchmetrics versions with every run, since version differences are the other common cause.

Platform-ownership habits

Know which version every image runs. This one habit decides how fast you can answer a security disclosure. Keep an inventory that maps each image tag to its installed lightning, torch and torchmetrics. A one-line check in each image is enough to start:

python -c "import lightning as L, torch; print(L.__version__, torch.__version__)"

Keep the abstraction thin. A base package with sensible defaults is valuable. One that hides the strategy, swallows exceptions or invents its own configuration language becomes a second framework that your team maintains forever. Defaults and guardrails, with explicit overrides, are the target.

Treat configs as code. LightningCLI configs can instantiate arbitrary classes through class_path, and --config accepts remote locations. Review the output of --print_config, not a partial file, and restrict who can write to config locations.

Separate resume checkpoints from release artifacts. Resume files carry optimizer and loop state, are large, and are short lived. Release artifacts are weights only, preferably safetensors, immutable and registered with metadata, including the writer's Lightning version.

Alert on absence. The failure you will see most is a job that stops without crashing. Alert on a step counter that has not advanced, not only on error logs.

Measure before you spend. Most platform arguments about scaling end when someone measures MFU and GPU utilisation with ThroughputMonitor and DeviceStatsMonitor. Sharding fixes memory, not data starvation.

Rehearse. A runbook that nobody has used is a guess. Run a game day each quarter: inject a rank-zero collective bug, a checkpoint that fails to load, a vulnerable version in an image. Fix the runbook with what you learn.

Measure onboarding. Time how long a new engineer takes to get a successful two-GPU job. It is the most honest metric of platform quality, because every sharp edge turns into lost hours.

Say what you do not support. Custom training loops were removed from Lightning, so the supported lane for unusual algorithms is Fabric. Experimental features such as ModelParallelStrategy and the ServableModule serving API are opt-in with a named owner. Declaring the boundary prevents resentment later.

Mind residency. For employers in the Gulf and Egypt, data and checkpoint location can be a legal question. Lightning writes through fsspec to whichever bucket you name, so the control has to be region-pinned storage and bucket policy, not a note in a README.

Close the loop on deprecations. Keep warnings visible, collect LightningDeprecationWarning centrally, and schedule removal before the version that ends them. The to_torchscript warning, for example, already says removal is planned.

Review checklists

Pull request checklist for training code

• Imports use one namespace, lightning, throughout.
• No removed APIs: gpus, resume_from_checkpoint, training_epoch_end, trainer.tune(), LightningLite.
• Every sync_dist=True log and every collective runs on all ranks.
• The entry point is under if __name__ == "__main__":.
• save_hyperparameters ignores modules, and no argument is a secret.
• An explicit budget exists: max_epochs, max_steps or max_time.
• ModelCheckpoint has save_last=True and a cadence justified by lost-work cost, and a monitor if ckpt_path="best" is used.
• Metrics are logged under the exact names that ModelCheckpoint and EarlyStopping monitor.
• Validation metrics logged at epoch level use sync_dist=True.
• DataLoaders set num_workers, pin_memory=True and persistent_workers=True where appropriate.
• Checkpoint loading states the origin of the file, and weights_only=False carries a justification.

Design document checklist for a new training workload

• The model size, memory estimate and strategy are stated, with the measured evidence for choosing it.
• Effective batch size and learning rate are written down for each planned scale.
• The scheduler allocation matches num_nodes and devices, and a pre-flight check exists.
• Checkpoint format, size, location, region and retention are defined.
• For sharded strategies, consolidation and cross-topology resume are covered.
• Preemption behaviour is defined: notice period, checkpoint write time, resume path.
• The export path to serving is named: torch.export, ONNX or TensorRT. No new to_torchscript dependency.
• Observability: progress, throughput, resources, provenance, stall alert.
• Cost: smoke-test stage, expected GPU hours, chargeback labels.
• Security: checkpoint trust classes, config write access, credential source.
• Rollback: which image and Lightning version to return to, and whether new checkpoints load on it.

Upgrade checklist

• Read the release notes for every version between the current and target one.
• Confirm Python is 3.10 or newer, and that torch and torchmetrics are pinned with lightning.
• Run the canary suite: DDP, FSDP, resume, LightningCLI, export.
• Load last release's checkpoints on the candidate, and test loading new ones on the old image.
• Collect LightningDeprecationWarning output and open tickets for each.
• Roll out by ring and keep the previous image available for one cycle.

Practice cards

Card one: reproduce a hang. On a two-process CPU run with Trainer(accelerator="cpu", devices=2, strategy="ddp"), write the rank-zero collective bug, run it with a short timeout, then fix it. The goal is to recognise the symptom in thirty seconds next time.

Card two: rehearse the loader. Take one checkpoint and trigger the three loader messages on purpose: the Unsupported global error, the instantiator allowlist message and the class-resolution message. Read each one until the cause is obvious without looking it up.

Card three: time a checkpoint. Measure how long your largest model takes to write a checkpoint, synchronously and with AsyncCheckpointIO. Compare it with the preemption notice.

Card four: draw your own architecture. On one page, draw the scheduler, storage, tracker and identity layer your platform uses, and mark who owns each. Any box with no owner is the next thing to fix.

Card five: write the runbook entry. Choose one incident from this file and write it as symptom, first check, fix. Hand it to a colleague who has not seen Lightning's internals and see whether they can follow it.

Operational recipes

A pre-flight check in the launcher. Before the Trainer exists, compare what the scheduler gave you with what the job asks for. The point is to fail in a second, with a message that names both numbers, instead of hanging in rendezvous for an hour on reserved GPUs.

import os

def preflight(devices: int, num_nodes: int) -> None:
    expected = devices * num_nodes
    world = int(os.environ.get("WORLD_SIZE", expected))
    if world != expected:
        raise SystemExit(
            f"allocation mismatch: scheduler WORLD_SIZE={world}, "
            f"job asks for devices*num_nodes={expected}"
        )

This is deliberately simple. It reads WORLD_SIZE, which torchrun sets, and on a scheduler that does not set it the check passes trivially, so adapt it to the variable your scheduler exports.

A resume-or-start wrapper. One entry point should serve a fresh run and every restart. Look for last.ckpt in the run directory and pass it as ckpt_path when it exists, or None when it does not. Because restarts then need no special flags, a preempted job and a first attempt are the same code path, and that is far easier to test.

A weekly version report. Once a week, list every image tag in use with its lightning, torch and torchmetrics versions, and flag anything below your floor. The report costs almost nothing and turns a future disclosure from an investigation into a query.

A checkpoint size budget. Multiply checkpoint size by save_top_k by the number of active runs, and compare it with your storage budget each month. Optimizer state makes a resume checkpoint much larger than the weights alone, so the figure surprises people who estimate from parameter count.

A canary after every base-image change. Run five small jobs on real GPU hardware: a DDP job, an FSDP job, a resume, a LightningCLI job and an export. Keep the outputs and the exact versions used. When something breaks six months later, the last green canary tells you where to bisect.

A closing note

Platform work on Lightning is mostly the work of preventing quiet failures. Hangs are quiet, version drift is quiet, a secret inside a checkpoint is quiet, and a learning rate that no longer fits the batch is quiet. The common remedy is making things visible: a heartbeat, an inventory, a resolved config, a logged effective batch size. If you remember one rule from this file, make it that one.

For the tools around it, see Kubeflow for running jobs on Kubernetes, MLflow for tracking and registry, and DeepSpeed for ZeRO on its own terms.