The Complete DeepSpeed Guide

This is part one of three. It takes you from never having used DeepSpeed to running real multi-GPU training with it, and explaining what it did and why. By the end you can install it and check the install, write a training script that uses it, read and write its JSON configuration, choose a ZeRO stage and a precision mode, offload to CPU when memory runs out, launch on one machine, save and recover checkpoints, plug it into Hugging Face Trainer, and read its most common error messages without panic. Mid-level and Senior take the same ground further; nothing here is thrown away.

This guide was checked against DeepSpeed 0.19.7, released on 16 September 2026. DeepSpeed is still a 0.x project, which means minor and even patch releases can remove features, and 0.19.7 did exactly that. Where something changed recently, the guide says so and shows the current form.

Each section ends with a Try it task. You do not need a cluster for most of them. A single NVIDIA GPU is enough to run every example, and where an example needs two GPUs the text says so.

What DeepSpeed is, and the problem it solves

Training a neural network on one GPU is easy to picture. The model's weights sit in GPU memory, a batch of data goes in, the loss comes out, gradients flow backwards, and an optimizer updates the weights. The trouble starts when the model grows. A model with several billion parameters does not fit in the memory of one GPU once you count everything training needs, and even when it fits, one GPU is slow. So you want many GPUs working together, and you want the work split so that adding GPUs lets you train models that were impossible before, not merely the same model a bit faster.

DeepSpeed is a PyTorch library, plus a command-line launcher, that handles this. You hand it your ordinary torch.nn.Module and a JSON configuration file. It hands back an object called the engine that wraps your model. The engine takes over the distributed setup, mixed-precision arithmetic, splitting of memory across GPUs, offloading to CPU memory or disk, gradient accumulation, gradient clipping, checkpointing and the optimizer step. Your training loop shrinks to three lines: run the model forward, call backward, call step.

The diagram is the whole idea. Everything else in this guide is detail about one of those four boxes.

It helps to know what came before. The standard PyTorch answer to multi-GPU training is DistributedDataParallel, usually shortened to DDP. DDP starts one process per GPU, and every process holds a complete copy of the model, a complete copy of the gradients and a complete copy of the optimizer state. Each process trains on different data, and the gradients are averaged across processes after every backward pass. DDP is simple and fast, but it is wasteful in one specific way: if you have eight GPUs, you hold eight identical copies of everything. Adding GPUs gives you speed, not room. The biggest model you can train is still the biggest model that fits on a single GPU.

DeepSpeed's headline contribution, called ZeRO (Zero Redundancy Optimizer), removes that redundancy. Instead of every GPU holding every copy, ZeRO splits the training state across the GPUs, so with eight GPUs each one holds roughly an eighth of it. Suddenly adding GPUs does buy you room. The same project added offload to CPU memory and to NVMe disks, ways to split a model by layers or by weight matrices, long-sequence support, and tools for inference. The beginner path needs only a small slice: the engine, the config, the launcher, ZeRO stages 1 to 3, mixed precision and checkpoints.

Where the memory goes

Before touching any configuration, you need a feel for why training uses so much more memory than the model's file size suggests. Without that, ZeRO's stages look like arbitrary numbers. With it, they are obvious.

When you train with the Adam optimizer in mixed precision, four kinds of data live in GPU memory for every parameter of the model.

The first is the parameters themselves, the weights, stored in a 16-bit format such as bf16 or fp16 for the forward and backward passes. That costs 2 bytes per parameter. The second is the gradients, one number per parameter, also in 16-bit, another 2 bytes. The third and fourth are the optimizer's own state. Adam keeps two running averages per parameter (the momentum and the variance), and mixed-precision training also keeps a full-precision 32-bit master copy of the weights, because tiny updates would vanish if they were added to 16-bit numbers. Those three 32-bit values cost 12 bytes per parameter. The DeepSpeed team's own figure for fp32 Adam is about 12 bytes per parameter on top of the weights, which matches. Add it up: 2 plus 2 plus 12 is roughly 16 bytes per parameter. That is the number from the previous Try it task. Treat it as an estimate, because activations and buffers are extra.

A model with 7 billion parameters therefore needs about 112 billion bytes, roughly 112 GB, for model state alone, before a single activation is stored. The weights file for that model in 16-bit is only 14 GB, which is why people are surprised. Most of the cost is not the model but the optimizer's bookkeeping.

Now the three ZeRO stages make sense, because each one decides how much of the three big items to split across the GPUs. Stage 1 splits the optimizer state, the 12-byte part. Stage 2 splits the optimizer state and the gradients. Stage 3 splits all three, including the parameters themselves. The more you split, the less each GPU holds, and the more the GPUs have to talk to each other. That trade, memory against communication, is the central trade of the whole library, and later sections return to it.

The mental model: four nouns

You need only a handful of terms to read DeepSpeed code and documentation. Learn these and the rest follows.

The engine. The object returned by deepspeed.initialize is a DeepSpeedEngine. It wraps your model, and you call it just like the model: model_engine(batch) runs the forward pass. It also has the methods that replace the usual training boilerplate, notably backward, step, save_checkpoint and load_checkpoint. If you remember one fact, remember that after initialization you stop using your original model object for training and use the engine instead.

The config. A JSON file, or a Python dictionary, describing everything the engine should do: batch sizes, the optimizer and its learning rate, the learning-rate scheduler, the precision mode, the ZeRO stage, offload, clipping and monitoring. It is parsed when the engine is created, and mistakes are reported then, which is good: you find out in the first seconds, not after an hour.

Ranks. Distributed training starts several copies of your script, one process per GPU. Each process is called a rank and has an integer ID. The world size is the total number of ranks. The local rank is a process's index on its own machine, which is how it picks its GPU. The launcher sets the environment variables RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT for you, so your script does not compute any of them.

ZeRO. The memory-splitting scheme described above, with stages 0 to 3. Stage 0 means no splitting, which behaves like ordinary data parallel training. You select the stage in the config with "zero_optimization": {"stage": 2}, and nothing else in your training code changes. That is the quiet superpower: moving from stage 1 to stage 3 is a one-number edit.

There is a fifth idea worth carrying even though it is not a noun: data parallelism. Every rank runs the same model on a different slice of the data, and the gradients are averaged. Even with ZeRO stage 3, DeepSpeed is doing data parallelism. It merely stores the state in pieces and gathers what it needs just in time.

One piece of arithmetic appears in almost every beginner error, so learn it now. DeepSpeed relates three batch numbers:

train_batch_size = train_micro_batch_size_per_gpu x gradient_accumulation_steps x number of GPUs

The micro-batch is how many samples one GPU processes in a single forward and backward pass; it is what has to fit in memory. Gradient accumulation means running several micro-batches and summing their gradients before updating the weights, which lets you simulate a big batch on a small GPU. The train batch size is the effective batch per weight update across all GPUs. You set any two of the three and DeepSpeed infers the third. If you set all three and they disagree, you get the first error message in the troubleshooting section.

Installing DeepSpeed and checking the setup

The golden rule: install PyTorch first. DeepSpeed's setup.py imports torch while building, so installing DeepSpeed into an environment with no PyTorch fails. DeepSpeed requires PyTorch 2.0 or newer, and the documentation recommends a current stable PyTorch 2.x. Pick the PyTorch build that matches your GPU driver, using the selector on pytorch.org.

On Linux, the primary platform, use a virtual environment:

python -m venv .venv && source .venv/bin/activate
pip install torch            # pick the CUDA or ROCm build matching your driver
pip install deepspeed        # source distribution; ops compile on first use
ds_report                    # or: python -m deepspeed.env_report

Python 3.10 to 3.12 is a safe choice. PyPI publishes DeepSpeed only as a source distribution, with no pre-built wheels; that is normal, and it is why the next idea matters.

Ops and JIT compilation. DeepSpeed includes fast C++ and CUDA extensions, called ops: a fused Adam optimizer, a CPU Adam for offload, an asynchronous file-I/O library for NVMe and others. By default these are compiled just in time the first time something needs them. The first run of a job that uses a fused optimizer therefore pauses for a minute or two while ninja and your compiler build the extension. The result is cached, by default in ~/.cache/torch_extensions, so later runs start fast. To compile these ops at install time instead, which is smart for containers and clusters where every node would otherwise compile the same thing, use build flags:

DS_BUILD_OPS=1 pip install deepspeed                       # all compatible ops
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 pip install deepspeed  # selected ops only

Compiling CUDA ops needs a CUDA compiler (nvcc) whose version matches the CUDA version your PyTorch was built with, plus ninja and a C++ compiler. A pure-Python install without nvcc works for many things, but any op that needs compiling then fails when first used, with MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s). For NVMe offload you also need the system library libaio, installed with apt install libaio-dev on Debian and Ubuntu.

On Apple Silicon, DeepSpeed 0.19.6 added support for the mps accelerator, covering ZeRO stages 0 to 3. The documented route is:

pip install torch
DS_ACCELERATOR=mps pip install --no-build-isolation deepspeed
ds_report

That is a useful way to learn the API on a laptop, with one important limit: a Mac has a single MPS device, so you cannot do multi-GPU data parallelism on it. Treat it as a place to test that your script and config are valid, not to train seriously.

Checking the install. The tool for this is ds_report. It prints two things. First, a table of every op with an installed column and a compatible column. [YES] under installed means the op was pre-built. [OKAY] under compatible means it can be compiled on demand. [NO] under compatible means your machine cannot build it, for example because libaio is missing. Second, general environment information: the torch version and location, the DeepSpeed version and location, the CUDA version torch was built with, the nvcc version and the shared-memory size. Read this output before debugging anything else, because many mysterious failures turn out to be version mismatches visible right there.

Your first project, step by step

We will train a deliberately tiny model on random data. The point is not the model; it is seeing every moving part of DeepSpeed work end to end with the least possible code. Create a folder with three files: train.py, ds_config.json and, later, nothing else.

Start with the config. It is the contract between you and the engine:

{
  "train_micro_batch_size_per_gpu": 8,
  "gradient_accumulation_steps": 2,
  "optimizer": {
    "type": "AdamW",
    "params": { "lr": 0.0003, "weight_decay": 0.01 }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": { "warmup_min_lr": 0, "warmup_max_lr": 0.0003, "warmup_num_steps": 10 }
  },
  "bf16": { "enabled": true },
  "zero_optimization": { "stage": 2 },
  "gradient_clipping": 1.0,
  "steps_per_print": 10
}

Read it as a list of decisions. We say each GPU processes 8 samples per micro-step, and we accumulate 2 micro-steps per update. We do not give train_batch_size, because DeepSpeed computes it: 8 x 2 x the number of GPUs. We choose AdamW and a warmup schedule that ramps the learning rate from 0 over 10 steps. We turn on bf16, which needs a GPU of the Ampere generation or newer; on older cards use fp16 instead, covered below. We choose ZeRO stage 2, and we state clipping at 1.0 explicitly even though that is now the default.

Now the script:

import argparse

import deepspeed
import torch
import torch.nn as nn
from torch.utils.data import Dataset


class RandomData(Dataset):
    """1,024 random regression examples: 64 features in, 1 number out."""

    def __init__(self, n=1024):
        self.x = torch.randn(n, 64)
        self.y = self.x.sum(dim=1, keepdim=True)

    def __len__(self):
        return len(self.x)

    def __getitem__(self, i):
        return self.x[i], self.y[i]


class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 1))
        self.loss_fn = nn.MSELoss()

    def forward(self, x, y):
        return self.loss_fn(self.body(x), y)


parser = argparse.ArgumentParser()
parser.add_argument("--local_rank", type=int, default=-1)  # the launcher passes it
parser = deepspeed.add_config_arguments(parser)             # adds --deepspeed, --deepspeed_config
args = parser.parse_args()

model = TinyNet()
model_engine, optimizer, train_loader, lr_scheduler = deepspeed.initialize(
    args=args,
    model=model,
    model_parameters=model.parameters(),
    training_data=RandomData(),
)

for epoch in range(3):
    for step, (x, y) in enumerate(train_loader):
        x = x.to(model_engine.device, dtype=torch.bfloat16)
        y = y.to(model_engine.device, dtype=torch.bfloat16)
        loss = model_engine(x, y)
        model_engine.backward(loss)
        model_engine.step()
    print(f"epoch {epoch} done, last loss {loss.item():.4f}")

Walk through it from the top, because every line has a reason.

deepspeed.add_config_arguments(parser) adds two command-line flags to your argument parser: --deepspeed and --deepspeed_config. That is how the path of your JSON file reaches the script. Alternatively you can skip the flags and pass config="ds_config.json" to initialize, but you must not do both; if you do, DeepSpeed stops with a message that it was given configs in two places. The --local_rank argument is there because the launcher passes it to your script, and an argument parser that does not declare it would reject the unknown flag.

deepspeed.initialize is the heart of the program. It takes your model, the list of parameters to optimize, and optionally a dataset. It returns four things: the engine, the optimizer, a data loader and a learning-rate scheduler. The last three can be None if you did not supply or configure them. Here, because we passed training_data and put the optimizer and scheduler in the JSON, we get all four. The data loader it builds is a DeepSpeedDataLoader, sized to the micro-batch in the config and sharded across ranks so each GPU sees different samples. Note that initialize also sets up the distributed process group for you; you do not call init_process_group yourself.

Inside the loop, we move each batch to model_engine.device, which is the right GPU for this rank. Because bf16 is enabled, the engine converts the model to bf16, so we cast the inputs to match. Then comes the three-line core: model_engine(x, y) runs forward and returns our loss, model_engine.backward(loss) runs backpropagation, and model_engine.step() does everything that follows. That step call is more than an optimizer step. It applies gradient clipping, runs the optimizer, advances the learning-rate scheduler and zeroes the gradients, but only at the gradient accumulation boundary. With gradient_accumulation_steps at 2, the first call to step accumulates and the second updates the weights. You never write if step % 2 == 0, and you never call optimizer.zero_grad(). If you do those by hand you will fight the engine.

Launch it with the DeepSpeed launcher:

deepspeed --num_gpus 2 train.py --deepspeed --deepspeed_config ds_config.json

The launcher starts two processes, one per GPU. Rank 0 prints a banner that includes DeepSpeed info: version=0.19.7, git-hash=..., git-branch=..., then configuration details, and then your epoch lines. With one GPU, use --num_gpus 1. On a machine with no --num_gpus flag, the launcher uses every GPU it finds. Because the script prints from every rank, you will see each message twice; real scripts guard prints with a rank check, which later sections show.

The configuration file, key by key

A DeepSpeed config looks long until you realise it is a set of independent blocks. You add a block only when you want that feature. This section covers the ones a beginner needs.

Batch size. Covered above: train_batch_size, train_micro_batch_size_per_gpu and gradient_accumulation_steps. Provide at least train_batch_size or train_micro_batch_size_per_gpu, because there are no numeric defaults. If you omit both you get Either train_batch_size or train_micro_batch_size_per_gpu needs to be provided.

Optimizer. The optimizer block has a type and a params dictionary. DeepSpeed natively understands Adam, AdamW, Lamb, Lion, Adagrad and Muon, matched case-insensitively. Any other name is looked up in torch.optim, so "SGD" works. For Adam and AdamW, DeepSpeed uses its fused implementation by default, or its CPU implementation when offloading, which is faster than the stock PyTorch loop. One important note on removed features: the 1-bit optimizers OneBitAdam, OneBitLamb and ZeroOneAdam were deleted in 0.19.7. Older tutorials mention them; do not use them.

If you pass an optimizer object to deepspeed.initialize in code, it overrides the optimizer block in the JSON. Use one or the other so you know which is in charge. And if you use a custom, non-DeepSpeed optimizer with ZeRO, DeepSpeed will refuse unless you set "zero_allow_untested_optimizer": true.

Scheduler. The scheduler block takes LRRangeTest, OneCycle, WarmupLR, WarmupDecayLR or WarmupCosineLR, each with parameters. They step once per engine.step() call, which means once per weight update, not once per micro-batch. Remember that when you reason about warmup_num_steps.

Precision. This is where beginners spend the most time, so slow down. You choose how numbers are stored and computed:

• "bf16": {"enabled": true} uses bfloat16, a 16-bit format with the same range as 32-bit floats but fewer digits of precision. It is stable and needs no loss scaling. It requires hardware that supports it, meaning NVIDIA Ampere and newer, or recent AMD and Intel accelerators. This is the default recommendation for new work. Since 0.19.3, bf16 also checks gradients for infinities and NaNs and skips such steps, matching fp16.
• "fp16": {"enabled": true} uses half precision. It has a narrower range, so it needs loss scaling: the loss is multiplied by a large number before backward so small gradients do not underflow to zero, and the scale adjusts dynamically. Setting "loss_scale": 0 means dynamic scaling. You will see log lines such as OVERFLOW! Rank 0 Skipping step when the scale is too big; DeepSpeed skips that step and lowers the scale. A few of these at the start are normal. A constant stream means instability.
• "torch_autocast" drives PyTorch's native autocast from the config.
• The amp block (Apex) is deprecated. Do not start anything new with it.

You cannot enable fp16 and bf16 together; DeepSpeed stops with bfloat16 and fp16 modes cannot be simultaneously enabled. If neither is enabled, training runs in fp32, which is the safest and slowest choice and a decent way to debug.

Gradient clipping. "gradient_clipping": 1.0 rescales gradients so their overall norm never exceeds 1.0, which protects against sudden spikes. Pay attention to the history here, because old tutorials contradict current behaviour: until 0.19.3 the default was 0.0 (disabled). Since 0.19.3 the default is 1.0. A config that omits the key now clips. If you really want no clipping, write "gradient_clipping": 0.0.

ZeRO stages in practice

The previous sections introduced ZeRO as a memory-splitting idea. This section turns it into a decision you can make.

Stage 0 disables ZeRO. The engine behaves like plain data parallelism. Use it when the model and its full training state fit comfortably, or to compare behaviour.

Stage 1 partitions the optimizer state, the 12-byte part, across the ranks. It needs almost no extra communication, so it is nearly free. With eight GPUs, the optimizer cost per GPU drops to about an eighth. Because the optimizer state is the largest of the three pieces, stage 1 alone recovers a large share of the memory.

Stage 2 also partitions gradients. Each rank keeps only the gradients for the slice of parameters whose optimizer state it owns, and the averaging step becomes a reduce-scatter, which sends each slice to its owner. Communication volume stays similar to plain data parallelism, which is why stage 2 is the most common choice for fine-tuning when the 16-bit weights themselves fit on each GPU. If you are unsure where to start, start here.

Stage 3 also partitions the parameters. No GPU holds the full model at rest. During the forward pass, right before a layer runs, its weights are gathered from all ranks; right after, the gathered copy is dropped. The same happens in backward. This is what lets you train models larger than a single GPU's memory. The price is communication: the volume is about 1.5 times that of plain data parallelism, so a fast interconnect (NVLink inside a machine, InfiniBand or RoCE between machines) matters. On slow networks stage 3 can crawl.

A beginner-friendly rule of thumb. Use stage 2 first. Move to stage 3 only when the model weights alone, or stage 2 with offload, still do not fit. Move down to stage 1 or 0 if you have memory to spare and want maximum speed.

A realistic stage 3 block looks like this:

{
  "train_micro_batch_size_per_gpu": 4,
  "gradient_accumulation_steps": 8,
  "bf16": { "enabled": true },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "contiguous_gradients": true,
    "stage3_gather_16bit_weights_on_model_save": true
  }
}

The keys beyond stage are tuning knobs. overlap_comm runs communication alongside computation to hide latency, and in the current code it resolves to true on its own at stage 3, though writing it is harmless. contiguous_gradients copies gradients into a contiguous buffer to reduce fragmentation and is true by default. stage3_gather_16bit_weights_on_model_save matters when you want to export a normal 16-bit weights file from a stage 3 run, as the checkpoint section explains. The other bucket-size and prefetch settings have sane defaults. Leave them alone until Mid-level.

What changes in your code at stage 3? For most scripts nothing. But two things surprise people. First, parameters are partitioned, so if you inspect param.shape on a stage 3 model during or after initialization you may see torch.Size([0]), because that rank holds only a shard. Second, if the model is so large that even building it on one GPU runs out of memory, you must construct it inside a deepspeed.zero.Init context, which partitions the parameters as they are created:

with deepspeed.zero.Init(config_dict_or_path="ds_config.json"):
    model = BigModel()

Hugging Face's from_pretrained does this for you when Trainer is configured for stage 3, as the integration section explains.

Offloading to CPU and NVMe

Sometimes even stage 3 across all your GPUs is not enough, or you only have one GPU and a large model. DeepSpeed can push parts of the training state out of GPU memory into the much larger CPU RAM, or even onto NVMe disk, and bring it back when needed. This is called offload.

The cheapest and most popular form is ZeRO-Offload: keep the optimizer state in CPU memory and run the optimizer update on the CPU. The 12-byte-per-parameter piece that dominated GPU memory moves off the GPU entirely. It works with stages 1, 2 and 3:

{
  "train_micro_batch_size_per_gpu": 2,
  "gradient_accumulation_steps": 16,
  "bf16": { "enabled": true },
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": { "device": "cpu" }
  }
}

With this setting DeepSpeed uses its DeepSpeedCPUAdam implementation, a vectorized CPU version of Adam. The reward is a model that trains on a GPU that could not otherwise hold it. The cost is speed: every step moves gradients over the PCIe bus to the CPU and the updated weights back, and the CPU does the optimizer maths. Expect a slowdown, and expect it to depend heavily on how many CPU cores and how much memory bandwidth you have.

For stage 3 you can additionally offload the parameters with "offload_param": {"device": "cpu"}. And you can offload to NVMe by setting "device": "nvme" together with an nvme_path; that combination is ZeRO-Infinity, which is stage 3 only and requires libaio and an NVMe drive fast enough to be worth it. Offloading parameters is only valid at stage 3. A beginner should try optimizer-to-CPU first, then parameters to CPU, and only consider NVMe if the model really demands it.

The pinned memory change. Pinned (page-locked) CPU memory makes transfers between CPU and GPU much faster. Since 0.19.5, pin_memory inside the offload blocks defaults to true; before, it was false. That helps speed but pinned memory counts against the operating system's locked-memory limit. If you upgrade and suddenly see out-of-memory errors in offload runs, particularly on hosts or containers with a low ulimit -l, either raise the limit (in Docker, --ulimit memlock=-1) or set "pin_memory": false explicitly in both offload blocks.

The launcher: running on one machine and beyond

The deepspeed command is more than a convenience wrapper; it is how ranks, environment variables and, across machines, SSH connections get set up.

On one machine the common forms are:

deepspeed train.py --deepspeed --deepspeed_config ds_config.json         # all local GPUs
deepspeed --num_gpus 2 train.py --deepspeed --deepspeed_config ds_config.json
deepspeed --include localhost:0,1 train.py --deepspeed --deepspeed_config ds_config.json
CUDA_VISIBLE_DEVICES=0,1 deepspeed train.py --deepspeed --deepspeed_config ds_config.json

The first uses every GPU on the machine. The second limits it to a number. The third chooses specific GPU indices; here GPUs 0 and 1. The fourth uses the familiar CUDA environment variable and works too. Everything after the script name goes to your script untouched, which is how --deepspeed and --deepspeed_config reach add_config_arguments.

The launcher uses port 29500 by default for the rendezvous where ranks find each other. If you run two jobs on the same machine at once, give the second a different one with --master_port, or the second fails with an address-already-in-use error. You may also see a warning that says Unable to find hostfile, will proceed with training with local resources only. On a single machine this is expected and harmless; it only means you did not provide a multi-machine host list.

You do not have to use the deepspeed command at all. Because DeepSpeed reads the standard PyTorch distributed environment variables, torchrun --nproc_per_node 2 train.py --deepspeed_config ds_config.json works, and so does accelerate launch. Hugging Face Trainer works under all of these.

Multiple machines. The default method is pdsh, which logs in to every machine over SSH without a password and starts the job. You describe the machines in a hostfile, an MPI-style text file:

worker-1 slots=8
worker-2 slots=8

Each line names a machine and how many GPUs (slots) it offers. Then:

deepspeed --hostfile=hostfile train.py --deepspeed --deepspeed_config ds_config.json

Multi-node runs need passwordless SSH between machines and pdsh installed, otherwise you meet Cannot find pdsh, please install via 'apt-get install -y pdsh'. Alternatives exist: --launcher openmpi, slurm and others, and on Kubernetes or any system without SSH you start the launcher on each node yourself with --no_ssh --node_rank=<n> --master_addr=<ip>. All of those belong to Mid-level and Senior. As a beginner, get everything right on one machine first. The vast majority of fine-tuning jobs people do never need more than a single node of 8 GPUs, and multi-node adds failure modes you do not want while learning.

Saving, loading and exporting checkpoints

Training jobs die: hardware fails, a scheduler preempts you, you press the wrong key. Checkpoints let you resume. With ZeRO, they work differently from a plain torch.save(model.state_dict()), and the difference trips up nearly everyone once.

The reason is sharding. Under ZeRO each rank holds only a slice of the optimizer state, and at stage 3 only a slice of the parameters. There is no single place that has the whole thing. So DeepSpeed saves in pieces: every rank writes its own shard.

model_engine.save_checkpoint("checkpoints", tag=None, client_state={"epoch": epoch})

The critical rule: every rank must call save_checkpoint, not just rank 0. If you wrap it in if rank == 0, the other ranks never write their shards, and the program may hang waiting on a collective operation that rank 0 started alone. With tag=None, the tag becomes global_step<N>, a folder name, and a small text file called latest records which tag is newest. The client_state dictionary is for anything of your own that you want stored, such as the epoch number. Do not put secrets in it.

The result looks like this on disk: a checkpoints/ directory holding a folder per tag, with a model-states file named something like mp_rank_00_model_states.pt, optimizer-states files per rank for ZeRO, the latest file, and a copy of a script called zero_to_fp32.py.

To resume:

load_path, client_state = model_engine.load_checkpoint("checkpoints")
if load_path is not None:
    start_epoch = client_state["epoch"] + 1

load_checkpoint returns the path it loaded from, or None if nothing was found, plus your client_state. By default it restores the module weights, the optimizer state and the scheduler state. Call it after deepspeed.initialize, on every rank. One documented caveat for stage 3: you cannot call load_checkpoint right after save_checkpoint within the same engine; re-create the engine first. Normal resume, which starts a fresh process, is unaffected.

Getting a normal weights file out. Sooner or later you want a plain file you can load with from_pretrained or torch.load outside DeepSpeed, and the sharded folder is not that. There are two routes.

The first is for ZeRO-3 only, and has to be planned: set stage3_gather_16bit_weights_on_model_save: true in the config, and then call model_engine.save_16bit_model("export_dir", save_filename="pytorch_model.bin") on all ranks. The engine gathers the full 16-bit weights and rank 0 writes the file. This is why that flag appeared in the stage 3 example.

The second route needs no GPU and works after the fact. The checkpoint folder contains the script zero_to_fp32.py, which merges the shards into a full-precision file:

python checkpoints/zero_to_fp32.py checkpoints exported_fp32 --safe_serialization

In 0.19.x the second argument is an output directory. Older tutorials, including some in the Hugging Face documentation, show the form python zero_to_fp32.py . pytorch_model.bin, which does not match the current behaviour. The optional --safe_serialization flag writes safetensors instead of a pickle file, and -t global_step1000 selects a specific tag. Expect it to use roughly twice the checkpoint size in CPU RAM.

A safety point for later: DeepSpeed checkpoints use PyTorch's pickle-based format, so loading a checkpoint from someone you do not trust can run arbitrary code. Load only checkpoints you or your team produced, and prefer safetensors when sharing.

Using DeepSpeed through Hugging Face Trainer

Many people never write deepspeed.initialize themselves. They fine-tune with the Hugging Face Trainer, which has built-in DeepSpeed support, and hand it the same JSON config. This is the most common way DeepSpeed is used in practice, so learn it.

Install the integration extras with pip install transformers[deepspeed], then pass the config file to the training arguments:

from transformers import AutoModelForCausalLM, Trainer, TrainingArguments

args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    bf16=True,
    deepspeed="ds_config.json",
)
model = AutoModelForCausalLM.from_pretrained("your-model-id")
trainer = Trainer(model=model, args=args, train_dataset=my_dataset)
trainer.train()

Two details. First, create TrainingArguments before loading the model. For ZeRO-3, the arguments object is what lets from_pretrained load the model through the partitioned zero.Init path, so a model too large for one GPU never fully materializes on it. If you reverse the order at stage 3, loading can run out of memory. Second, launch this script with deepspeed train_hf.py, with torchrun, or with accelerate launch; a bare python starts only one process.

The slick part is the "auto" value. In the JSON, instead of repeating numbers that already live in TrainingArguments, write "auto" and Trainer fills them in:

{
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "train_batch_size": "auto",
  "gradient_clipping": "auto",
  "bf16": { "enabled": "auto" },
  "optimizer": {
    "type": "AdamW",
    "params": { "lr": "auto", "weight_decay": "auto" }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": { "warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto" }
  },
  "zero_optimization": { "stage": 2 }
}

Now per_device_train_batch_size flows into the micro-batch, learning_rate into the optimizer, and bf16=True into the bf16 block. One source of truth means no mismatch between the two systems. This also removes the batch-identity assertion for most Trainer users, because Trainer computes a consistent set.

If you use Hugging Face Accelerate instead, you run accelerate config, choose DeepSpeed, and either answer questions about ZeRO stage and offload or point it at your JSON file with deepspeed_config_file. Do not do both: if the file is given, Accelerate warns that its own variables will be ignored, and an inconsistent mix produces a ValueError listing them. The Hugging Face guide and the PEFT and TRL guide show these training libraries in full.

Common errors and how to read them

DeepSpeed's errors are mostly assertions raised while the config is parsed or the engine is built. They look alarming and are usually one-line fixes. Here are the ones you will meet, with the cause and the repair.

Check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size 32 != 4 * 1 * 4. The three batch keys disagree with the real GPU count. In the example, you asked for a batch of 32 but 4 x 1 x 4 GPUs gives 16. This typically appears after you change the number of GPUs. Give only two of the three keys, or use "auto" in Hugging Face.

DeepSpeed requires --deepspeed_config to specify configuration file. No config reached initialize. Either you forgot deepspeed.add_config_arguments, or you did not pass --deepspeed_config path on the command line, or you did not give config=. The sibling message, Not sure how to proceed, we were given deepspeed configs in the deepspeed arguments and deepspeed.initialize() function call, means you passed it both ways.

bfloat16 and fp16 modes cannot be simultaneously enabled. Enable only one precision block.

DeepSpeedConfigError: DeepSpeed Sparse Attention has been removed..., or the same for Nebula, the compression library, MoQ and MiCS. You are using an old config on 0.19.7 or later. Delete the named block. For MiCS the replacement is ZeRO-3 with ZeRO++ hierarchical partitioning (zero_hpz_partition_size). The message ends with the link to the tracking issue that explains why.

You are using an untested ZeRO Optimizer. Please add "zero_allow_untested_optimizer": true. You passed an optimizer DeepSpeed has not validated with ZeRO. Either set that flag or use a native optimizer through the config.

MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s). An op needs compiling and there is no CUDA toolkit. Common inside runtime-only containers. Use a -devel CUDA image, set CUDA_HOME, or pre-build ops at install time with DS_BUILD_OPS=1.

CUDAMismatchException: Installed CUDA version 12.4 does not match the version torch was compiled with 12.8. The system nvcc and PyTorch's CUDA disagree. Install the matching toolkit, point CUDA_HOME at the right one, or match the torch build to the toolkit. DS_SKIP_CUDA_CHECK=1 forces acceptance, but treat that as a last resort.

Watchdog timeout: Watchdog caught collective operation timeout ... Timeout(ms)=600000. One rank stopped participating in a collective operation: a slow data loader, a crashed rank, an uneven forward pass or a very slow checkpoint write. Since 0.19.6 the default timeout is 10 minutes, down from 30. Find the stuck rank with NCCL_DEBUG=INFO before raising the limit with DEEPSPEED_TIMEOUT=30, which fits only genuinely long operations.

OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768. Not an error. It is fp16 loss scaling doing its job. A few at the start are normal. Constant skipping with a loss that never moves means instability: switch to bf16 or lower the learning rate. NaN loss from the very first step means a data or learning-rate problem, not DeepSpeed.

Out-of-memory after upgrading to 0.19.5 or later with offload. pin_memory now defaults to true and counts against the locked-memory limit. Raise ulimit -l or set "pin_memory": false.

Putting it all together

Here is one small end-to-end project that uses most of what you learned. We fine-tune a small causal language model with Trainer, ZeRO stage 2, bf16, optional CPU offload, and a clean export at the end. It runs on a single GPU or several.

The config, with everything driven by Trainer through auto:

{
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "train_batch_size": "auto",
  "gradient_clipping": "auto",
  "bf16": { "enabled": "auto" },
  "optimizer": { "type": "AdamW", "params": { "lr": "auto", "weight_decay": "auto" } },
  "scheduler": {
    "type": "WarmupLR",
    "params": { "warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto" }
  },
  "zero_optimization": { "stage": 2, "overlap_comm": true }
}

The script:

from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM, AutoTokenizer, DataCollatorForLanguageModeling,
    Trainer, TrainingArguments,
)

MODEL = "your-small-model-id"          # a small causal LM from the Hugging Face Hub

tok = AutoTokenizer.from_pretrained(MODEL)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

raw = load_dataset("your-text-dataset", split="train[:2000]")
data = raw.map(lambda b: tok(b["text"], truncation=True, max_length=256),
               batched=True, remove_columns=raw.column_names)

args = TrainingArguments(                  # created BEFORE the model
    output_dir="out",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    num_train_epochs=1,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    deepspeed="ds_config.json",
)

model = AutoModelForCausalLM.from_pretrained(MODEL)
trainer = Trainer(
    model=model, args=args, train_dataset=data,
    data_collator=DataCollatorForLanguageModeling(tok, mlm=False),
)
trainer.train()
trainer.save_model("final")

Launch it:

deepspeed --num_gpus 2 train_hf.py

The launcher starts two ranks, and Trainer calls deepspeed.initialize for you, filling the auto values from the training arguments: 4 per GPU, 4 accumulation steps, so 32 samples per update on two GPUs. ZeRO stage 2 splits the optimizer state and gradients between the GPUs. If memory is still tight, add "offload_optimizer": {"device": "cpu"} inside zero_optimization. For stage 3, change the stage to 3 and add "stage3_gather_16bit_weights_on_model_save": true; nothing else changes.

What you can now do, and what comes next

You can explain what DeepSpeed solves and why training needs about 16 bytes per parameter. You can name the engine, the config, ranks and ZeRO. You can install DeepSpeed in the right order and verify it with ds_report, write a training loop around deepspeed.initialize, backward and step, and keep the three batch keys consistent. You can choose bf16 or fp16, pick a ZeRO stage, turn on CPU offload, launch on one machine, save and resume with every rank participating, export a plain weights file, drive all of it from Hugging Face Trainer, and read the common errors.

Mid-level takes each topic further. It explains how ZeRO works under the hood precisely enough to predict its behaviour, tunes the bucket and prefetch settings, introduces ZeRO++ and NVMe offload in depth, covers multi-node launching with hostfiles, SLURM and Kubernetes, and builds production patterns around checkpoints, monitoring and CI. Senior covers parallelism composition such as tensor and pipeline parallelism, security and the trust model of the launcher, cost, upgrades under a 0.x release cadence, and where DeepSpeed stops and PyTorch FSDP or a serving engine such as vLLM takes over. For the cluster side, the Kubernetes guide is the natural next stop, and MLflow is a good home for the runs you create.

Sources

• DeepSpeed home and tutorials
• Getting started
• Advanced install
• Configuration JSON reference
• ZeRO tutorial
• ZeRO-Offload tutorial
• Monitoring tutorial
• API reference: training
• API reference: model checkpointing
• DeepSpeed on GitHub
• Release notes
• Feature removal tracking issue
• DeepSpeed on PyPI
• Hugging Face Transformers DeepSpeed integration
• Hugging Face Accelerate DeepSpeed guide

This is part one of three. A twenty-minute review of the Beginner Detailed track, then the questions interviewers ask about DeepSpeed. Mid-level reviews this plus its own material; Senior reviews all three. Everything here is checked against DeepSpeed 0.19.7.

The thirty-second answer

DeepSpeed is a PyTorch library and launcher for training large models across GPUs. You wrap your model with deepspeed.initialize and a JSON config, and the returned engine handles distributed setup, mixed precision, gradient accumulation, clipping, checkpointing and the optimizer step. Its key idea, ZeRO, splits the optimizer state, gradients and parameters across GPUs instead of copying them on every GPU, so adding GPUs adds memory as well as speed.

Then the line that shows you have used it: "the config is the control surface: moving from ZeRO stage 1 to stage 3, or turning on CPU offload, is a config edit and not a code change."

The model

The commands and calls to have ready

pip install torch && pip install deepspeed       # torch first
ds_report                                         # ops table and environment
deepspeed --num_gpus 2 train.py --deepspeed --deepspeed_config ds_config.json
deepspeed --include localhost:0,1 train.py ...    # choose GPUs
deepspeed --hostfile=hostfile train.py ...        # several machines
python ckpt/zero_to_fp32.py ckpt out_dir --safe_serialization

model_engine, optimizer, loader, scheduler = deepspeed.initialize(
    args=args, model=model, model_parameters=model.parameters(), training_data=ds)
loss = model_engine(x, y)
model_engine.backward(loss)
model_engine.step()
model_engine.save_checkpoint("ckpt")          # every rank
model_engine.load_checkpoint("ckpt")

Know why each line exists. step() is not just the optimizer: it clips, runs the optimizer, advances the scheduler and zeroes gradients, but only at the gradient accumulation boundary.

The classic traps and why they happen

The batch identity. train_batch_size = micro-batch x accumulation steps x GPUs. Set two and the third is inferred. Set all three inconsistently, or change the GPU count after tuning, and you get the assertion train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size.

Saving on rank 0 only. Each rank writes its own shard, so save_checkpoint must run on every rank. Guard prints with a rank check, never the save.

Stage 3 parameters look empty. They are partitioned, so a rank sees torch.Size([0]) for parameters it does not own. Read them inside GatheredParameters or export properly.

fp16 and bf16 together. Not allowed. bf16 needs hardware support and no loss scaling. fp16 needs dynamic loss scaling and prints OVERFLOW! ... Skipping step while it settles.

Old configs on a new version. 0.19.7 removed Nebula, the compression library, 1-bit Adam, 1-bit LAMB, 0/1 Adam, MiCS and Sparse Attention. Leftover blocks now raise DeepSpeedConfigError.

Default flips. gradient_clipping defaults to 1.0 since 0.19.3. pin_memory in offload blocks defaults to true since 0.19.5. The process-group timeout is 10 minutes since 0.19.6.

JIT surprises. The first run compiles ops and pauses. A runtime-only container with no CUDA toolkit fails with MissingCUDAException.

Common interview questions

What problem does DeepSpeed solve? Training state, not just weights, is what fills GPU memory. With mixed-precision Adam you pay roughly 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients and about 12 for optimizer state with a fp32 master copy. Plain data parallelism repeats all of it on every GPU. DeepSpeed shards it so more GPUs means a larger trainable model.

How is DeepSpeed different from DistributedDataParallel? DDP replicates the model, gradients and optimizer on every rank and averages gradients. DeepSpeed with ZeRO partitions those pieces. Stage 0 behaves like DDP. DeepSpeed also adds offload, its own launcher and fused ops. Both are data parallel in spirit.

Explain ZeRO stages 1, 2 and 3. Stage 1 partitions the optimizer state. Stage 2 adds gradients, using a reduce-scatter so each rank keeps only the gradients it owns. Stage 3 adds the parameters: weights are gathered just before a layer runs and released after. Each stage saves more memory and adds more communication, with stage 3 around 1.5 times the volume of plain data parallelism.

Which stage would you start with, and why? Stage 2 for fine-tuning: big savings with little extra communication. Move to stage 3 only when the 16-bit weights themselves, or stage 2 with offload, still do not fit. Strong answers add that it depends on the interconnect.

What does deepspeed.initialize return and what do you do with it? A tuple of the engine, the optimizer, a training data loader and an LR scheduler. The last three may be None. After that, train with the engine, not with the original model.

What does model_engine.step() do? Clips gradients, steps the optimizer, steps the scheduler and zeroes gradients, at the accumulation boundary. Calling optimizer.zero_grad() yourself is unnecessary.

What is ZeRO-Offload, and what does it cost? It moves optimizer state and the optimizer update to CPU memory using DeepSpeedCPUAdam, working with stages 1 to 3. It lets a larger model fit, at the price of PCIe transfers and CPU speed. ZeRO-Infinity adds NVMe and is stage 3 only.

Why did my offload job start failing after an upgrade? Since 0.19.5, pin_memory defaults to true and pinned memory counts against ulimit -l. Raise the limit or set "pin_memory": false.

bf16 versus fp16? bf16 has the range of fp32 and needs no loss scaling, so it is the default recommendation on hardware that supports it. fp16 is narrower and needs dynamic loss scaling. They cannot be enabled together.

How do you save a model from a ZeRO-3 run for normal use? Either set stage3_gather_16bit_weights_on_model_save and call save_16bit_model on all ranks, or run the zero_to_fp32.py script from the checkpoint folder offline. In 0.19.x its second argument is an output directory.

Why must every rank call save_checkpoint? Each rank owns a shard of the optimizer state, and at stage 3 of the parameters. Without all shards the checkpoint is incomplete, and collectives can hang.

How do you use DeepSpeed with Hugging Face? Pass deepspeed="ds_config.json" in TrainingArguments, use "auto" values so Trainer fills them, create the arguments before loading the model for stage 3, and launch with deepspeed, torchrun or accelerate launch.

What does ds_report tell you? Which ops are pre-built or can be compiled, plus the torch, DeepSpeed, CUDA and nvcc versions. It is the first command to run when something compiles or imports wrongly.

A job hangs with a watchdog timeout. What do you do? It means one rank stopped joining a collective. Find which one with NCCL_DEBUG=INFO and per-rank logs. Raise DEEPSPEED_TIMEOUT only for legitimately long operations.

Is DeepSpeed safe to upgrade casually? No. It is a 0.x project, and 0.19.7 removed features in a patch-level release. Pin versions and read release notes first. The MLflow guide shows how to record the versions per run.

Can I run DeepSpeed on Windows or a Mac? Windows has no PyPI wheel since 0.16.5, so you build from source and are scoped to single GPU without NVMe offload; WSL2 or Linux is the practical advice. Apple Silicon gained ZeRO 0 to 3 support in 0.19.6, but with one device, so it is for testing.

Scenario questions

Your teammate says "DeepSpeed made my training slower." What do you check first? Ask which stage and whether offload is on. On a model that already fits, stage 3 and CPU offload add communication and PCIe traffic for no gain. Then check the interconnect, the micro-batch size (small micro-batches make communication a larger share), and whether the first run included JIT compilation. Turn on wall_clock_breakdown to see forward, backward and step times separately. A strong answer measures before changing anything.

A job ran on four GPUs yesterday and fails on two today with a batch-size assertion. Why, and how do you fix it? The three batch keys were all set, and the identity no longer holds with the new world size. Remove train_batch_size and give the micro-batch and accumulation steps, or recompute accumulation to preserve the effective batch. A strong answer also notes that the effective batch affects training behaviour, so you decide deliberately whether to preserve it.

Training resumes from a checkpoint but the optimizer seems reset. What might be wrong? Check that load_checkpoint was called after initialize with the right directory and tag, that load_optimizer_states and load_lr_scheduler_states were left at their default of true, and that load_module_only was not set. Also check that the tag you loaded is the latest one, using the latest file.

You copied a config from a two-year-old tutorial and the job fails on start. What happened? Probably a removed feature. DeepSpeed 0.19.7 removed Nebula checkpointing, the compression library, MoQ, MiCS and Sparse Attention, and leftover blocks raise DeepSpeedConfigError. Delete the block. A strong answer adds that the failure is intentional, because ignoring those blocks silently would have produced checkpoints or quantization that quietly did not happen.

Your fine-tune of a 7-billion-parameter model runs out of memory on a single 40 GB GPU. Walk through your options. Estimate: about 16 bytes per parameter is 112 GB of model state, so it cannot fit unsplit. With a single GPU there is nothing to shard across, so the lever is offload: ZeRO stage 2 or 3 with optimizer state offloaded to CPU, then parameters if needed, accepting slower steps. Also reduce the micro-batch and raise accumulation, and consider a parameter-efficient method that trains far fewer weights, as in the PEFT and TRL guide. A strong answer does the arithmetic out loud.

Trade-off questions

Stage 2 or stage 3? Stage 2 costs little extra communication and shards the largest pieces, so it is the default for fine-tuning. Stage 3 is for models whose weights do not fit, at about 1.5 times the communication of plain data parallelism, and so wants NVLink or InfiniBand-class networking. The weak answer says "stage 3 is always better because it saves more memory".

DeepSpeed or plain DDP? If the model, gradients and optimizer state fit on one GPU with room to spare, DDP is simpler and equally fast. DeepSpeed earns its place when memory is the limit, or when you want offload and the config-driven workflow.

Native DeepSpeed loop or Hugging Face Trainer? Trainer if you are fine-tuning a Hugging Face model: it handles initialization, auto values and checkpoints. The native engine when you own a custom loop and want direct control. The strong answer notes both use the same JSON.

CPU offload or more GPUs? Offload trades time for memory and works when GPUs are scarce. More GPUs with ZeRO trade money for speed. If you have only one GPU, offload is the only route to a large model; if you have several, shard first and offload last.

Behavioural and practical prompts

Tell me about a time you debugged a distributed training failure. Use a concrete story: what the symptom was, how you reduced it to one GPU, how you read the error source (config, build or communication), and what you changed in your process afterwards, such as a smoke test or a pinned version.

How would you teach a teammate to start using DeepSpeed? Begin with the memory arithmetic, show the minimal config and three-line loop, run it on one GPU, then change only the stage. Teaching by changing one number is a good answer.

What would you put in a code review for a DeepSpeed training script? Batch keys that do not hard-code GPU counts, a single precision mode, every rank saving checkpoints, no secrets in config, pinned versions, and a tested resume.

What a strong answer covers that a weak one misses

A weak answer lists the stages from memory. A strong answer ties each stage to the three memory items and says what communication it costs. A weak answer says "DeepSpeed makes training faster". A strong one says it mainly lets you fit more, and that some settings trade speed for memory. A weak answer forgets that every rank must save. A strong one names the symptom when it is forgotten. A weak answer quotes old defaults; a strong one knows gradient_clipping, pin_memory and the timeout changed in 0.19.x, and that features were removed in 0.19.7. A weak answer ignores the environment; a strong one reaches for ds_report first.

Vocabulary that sounds right in an interview

Say "optimizer state" rather than "optimizer memory", and name its contents: the two Adam moments plus the fp32 master weights. Say "shard" or "partition" for what ZeRO does, and "gather" for reassembling a stage 3 layer's weights just in time. Say "reduce-scatter" when describing how stage 2 averages gradients while leaving each rank only its own slice. Say "accumulation boundary" for the step on which weights really update. Use "just-in-time compilation" for ops, and mention that ds_report shows which are pre-built. Precise terms signal that you have run the tool, not only read about it, and interviewers notice the difference within a minute.

Questions to ask back

• What GPUs and interconnect do we train on, and how big are the models?
• Do we launch with the deepspeed command, torchrun or accelerate, and where do configs live in the repository?
• How do we pin DeepSpeed and PyTorch versions, and how do we know ops match them?
• Where do checkpoints go, and who has tested a resume?

Sixty-second self-test

Answer aloud without looking back.

1. Why does training use about 16 bytes per parameter with mixed-precision Adam?
2. What does each ZeRO stage partition?
3. What is the batch identity?
4. What does step() do beyond the optimizer update?
5. Who calls save_checkpoint?
6. Why do stage 3 parameters sometimes have shape [0]?
7. Which two precision blocks cannot be on together?
8. What default changed for gradient_clipping in 0.19.3?
9. What does pin_memory default to since 0.19.5?
10. Name three features removed in 0.19.7.
11. How do you export a normal weights file from stage 3?
12. What is the first command you run when an op will not build?

If you stumble on more than three, reread the matching section of the Beginner Detailed track. See also the sibling guides on Hugging Face and PyTorch Lightning for where DeepSpeed plugs in.

This is part one of three. Most DeepSpeed beginner pain comes from four sources: batch arithmetic that no longer adds up, precision settings that clash, memory trade-offs chosen blindly, and checkpoint habits borrowed from single-GPU PyTorch. Start with the table, then the habits and the checklist.

Common errors at this level

Mistake 1: Changing the GPU count and forgetting the batch keys

Symptom. A config that worked on four GPUs fails on two with the batch assertion, quoting numbers such as 32 != 4 * 1 * 4.

Cause. The identity train_batch_size = micro-batch x accumulation x GPUs is checked against the real world size. Hard-coding all three keys pins the GPU count into the file.

Fix. Specify the micro-batch per GPU and the accumulation steps, and omit train_batch_size. DeepSpeed derives the effective batch from the GPUs it finds. If you must keep a target effective batch, recompute the accumulation steps for the new GPU count, and write the target in a comment in your launch script.

{
  "train_micro_batch_size_per_gpu": 4,
  "gradient_accumulation_steps": 8
}

Be aware that an effective batch that changes with GPU count also changes training behaviour. Keep the effective batch constant when comparing runs.

Mistake 2: Writing your own loop around the engine's job

Symptom. Gradients are wrong, weights update at the wrong moments, or training is mysteriously slow.

Cause. Habits from plain PyTorch: calling optimizer.zero_grad(), dividing the loss by accumulation steps, writing if step % accum == 0, or calling optimizer.step() directly. The engine already does all of this.

Fix. The loop is three calls: forward through the engine, model_engine.backward(loss), model_engine.step(). Do not divide the loss yourself, and do not zero gradients. If you need to know whether this micro-step is the update step, ask model_engine.is_gradient_accumulation_boundary().

Mistake 3: Saving checkpoints on rank 0 only

Symptom. The job hangs when saving, or the checkpoint folder exists but cannot be loaded.

Cause. Each rank writes its own optimizer shard, and at stage 3 its parameter shard. A rank guard around save_checkpoint means the other shards never exist, and collective operations can wait forever.

Fix. Call save_checkpoint on every rank. Guard only the things that truly belong to one rank, such as printing or writing a custom file.

model_engine.save_checkpoint("checkpoints", client_state={"epoch": epoch})   # all ranks

import torch.distributed as dist

if dist.get_rank() == 0:
    print("saved epoch", epoch)

Related habit: after saving, check that the folder contains a tag directory and a latest file, and test a resume in a fresh process. A checkpoint you have never loaded is a guess.

Mistake 4: Trying to use state_dict() on a ZeRO-3 model

Symptom. An empty state dictionary, tensors with shape torch.Size([0]), or a saved file that is missing weights.

Cause. At stage 3 each rank holds a slice of every parameter. The full tensor exists nowhere at rest.

Fix. For a normal weights file, enable stage3_gather_16bit_weights_on_model_save and call save_16bit_model on all ranks. Or convert later, with no GPU, using the script placed inside the checkpoint folder:

python checkpoints/zero_to_fp32.py checkpoints exported_fp32 --safe_serialization

The second argument is an output directory in 0.19.x. The older zero_to_fp32.py . pytorch_model.bin form shown in some tutorials is outdated. Also remember the file needs about twice the checkpoint size in CPU RAM.

Mistake 5: Mixing up precision settings

Symptom. An assertion about bf16 and fp16, a NaN loss, or a Trainer run where the JSON says one thing and the arguments another.

Cause. Two precision blocks enabled, or fp16 chosen on a model that overflows, or bf16 requested on hardware without support, or the same value set in two systems.

Fix. Pick bf16 where the hardware supports it. If you need fp16, expect some OVERFLOW! ... Skipping step lines at the start while the loss scale settles, and treat a continuous stream as instability. With Hugging Face, write "auto" in the JSON so the training arguments decide.

Fix for NaN from step one. Test with precision off for a few steps. If fp32 is fine, the problem is precision; if fp32 also gives NaN, the problem is data or learning rate, not DeepSpeed.

Mistake 6: Copying an old config from a blog post

Symptom. DeepSpeedConfigError naming Nebula, compression, MoQ, MiCS or Sparse Attention, or an optimizer type that no longer exists.

Cause. DeepSpeed 0.19.7 removed those features and now fails at parse time instead of quietly ignoring them. The 1-bit optimizers OneBitAdam, OneBitLamb and ZeroOneAdam are also gone.

Fix. Delete the block, and use Adam or AdamW. Treat any configuration older than a year with suspicion, and check the keys against the current config reference. A few more features are still accepted but scheduled for removal, notably Apex amp, progressive_layer_drop and sparse_gradients; do not start new work on them.

Mistake 7: Assuming defaults are what the tutorial says

Symptom. Training behaves differently from the tutorial you followed, with no error.

Cause. Defaults moved. Three to know: gradient_clipping is 1.0 (it was 0.0 before 0.19.3); pin_memory in offload blocks is true (it was false before 0.19.5); the process-group timeout is 10 minutes (it was 30 before 0.19.6).

Fix. Write the values you care about explicitly, and note the DeepSpeed version in your run notes. For an experiment where clipping must be off, write "gradient_clipping": 0.0.

Mistake 8: Starting at stage 3 and offload for everything

Symptom. A small model trains slower than before and nobody knows why.

Cause. Stage 3 adds communication, and offload adds PCIe and CPU time. On a model that fits, they are pure overhead.

Fix. Climb the ladder only when needed: stage 0 or 1 if everything fits, stage 2 for most fine-tuning, stage 3 when weights do not fit, CPU offload when stage 3 still does not, NVMe last. Measure step time at each rung. A few minutes of testing saves hours of slow training.

Mistake 9: Ignoring the environment until something breaks

Symptom. Compile failures, mismatch errors, or "works on my machine" between nodes.

Cause. DeepSpeed compiles ops against a specific torch, CUDA and compiler. Anything that changes underneath, such as a new torch, a different CUDA toolkit or a runtime-only container, breaks the match.

Fix. Run ds_report when you set up and after every upgrade, and save its output with your run. Pin versions. In containers, use a CUDA -devel base image or pre-build ops with DS_BUILD_OPS=1; see the Docker guide for building such images. Install torch before DeepSpeed.

Mistake 10: Letting JIT compile on every node and every run

Symptom. Every job start has a multi-minute pause, or jobs on shared storage stall on Waiting for a lock....

Cause. Ops compile on first use, and caches in a shared home directory can collide or hold stale locks from killed builds.

Fix. Pre-build the ops you need at image build time, for example DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 pip install deepspeed. Otherwise give each job its own cache with TORCH_EXTENSIONS_DIR, and delete a stale op folder when a build hangs.

Habits worth adopting on day one

Run a single-GPU smoke test first. deepspeed --num_gpus 1 train.py ... for a few steps catches config, import and compile errors with one clean traceback instead of eight interleaved ones.

Keep the config in version control, next to the code. Name it for what it does, such as ds_zero2_bf16.json, and keep one file per experiment. The config is as much part of the experiment as the learning rate.

Change one thing at a time. The stage, the precision, offload and batch keys all interact. When a run misbehaves, revert to the last config that worked and change one key.

Print only from rank 0. Otherwise your logs repeat once per GPU. Use --enable_each_rank_log <dir> when you want per-rank files for debugging.

Estimate memory before you rent GPUs. DeepSpeed provides helper functions such as estimate_zero3_model_states_mem_needs_all_live, taking the model and the GPUs per node, which print what each configuration needs. Use them for a first sanity check, and treat the answer as the model state only, with activations on top.

Record versions. Write the DeepSpeed, PyTorch and CUDA versions into every run's metadata. A tracker such as MLflow or Weights & Biases is the natural place, and the DeepSpeed monitor blocks can send loss and learning-rate series to TensorBoard, W&B and CSV.

Test resume early. Kill the job after a checkpoint, restart, and confirm the loss continues from where it stopped. Do this on day one, not on day nine.

Never put secrets in the config, client_state or .deepspeed_env. The launcher copies environment values to every worker, and checkpoints are pickled. Use your scheduler's secret mechanism for tokens.

Load only trusted checkpoints. Checkpoint files are pickle-based, so a file from a stranger can run code when loaded. Prefer safetensors when sharing, using --safe_serialization on export.

Pin exact versions in anything that matters. DeepSpeed is 0.x, and 0.19.7 removed features in a patch release. Read the release notes before every upgrade and run a config-parse smoke test afterwards.

Prefer the engine for training and the Trainer for convenience. If you are fine-tuning a Hugging Face model, using Trainer with an all-auto config removes most of the mistakes above. The Hugging Face guide and the PEFT and TRL guide show the surrounding workflow.

Mistake 11: Trusting loss curves without checking the effective batch

Symptom. Two runs of the same model differ in loss curve, and nobody can say why. One used four GPUs and one used eight.

Cause. With the micro-batch and accumulation fixed, doubling the GPUs doubles the effective batch, which changes how many weight updates one epoch contains and how the learning-rate schedule plays out. The scheduler steps once per weight update, so warmup_num_steps covers a different amount of data.

Fix. Write the effective batch size into your run notes and treat it as a hyperparameter. When you scale GPUs, adjust accumulation steps to keep it constant, or deliberately change the learning rate and warmup with it. Compare runs only when this number matches.

Mistake 12: Debugging on eight GPUs

Symptom. A failure that produces eight interleaved tracebacks, each slightly different, and a log nobody can read.

Cause. Every rank runs your script and every rank reports the problem. Most errors a beginner meets, such as config assertions, import errors and compile failures, are not about multi-GPU at all.

Fix. Reproduce with --num_gpus 1. If the error vanishes, it is a communication or sharding problem; if it stays, fix it on one GPU where the output is readable. For hangs, use --enable_each_rank_log to write a file per rank and compare where each one stopped.

Practice cards

A note on asking for help

When you post a DeepSpeed problem to a teammate or the project's issue tracker, include the output of ds_report, the exact DeepSpeed and PyTorch versions, your config file, the launch command, and the first traceback from a single-GPU run if you can reproduce it. Most replies begin by asking for exactly these, and giving them up front turns a day of back and forth into one answer. Remove any tokens or private paths before pasting, and say which hardware you ran on, since many problems are specific to one GPU generation or one interconnect.

Checklist before you launch

• PyTorch installed first, DeepSpeed second, ds_report clean.
• DeepSpeed and PyTorch versions pinned and recorded.
• Config has no removed blocks, and only one precision mode on.
• Batch keys: two of three given, effective batch known.
• ZeRO stage chosen deliberately, with offload only if needed.
• Optimizer defined in the JSON when offloading.
• gradient_clipping stated explicitly.
• Single-GPU smoke test passed.
• Every rank calls save_checkpoint, and a resume was tested.
• Prints and custom file writes guarded by a rank check.
• No secrets in config, client_state or .deepspeed_env.
• Port 29500 free, or --master_port set.

Print this list and tick it for the first few jobs; after that it becomes muscle memory, and you will notice a missing item before the scheduler does. A good team also keeps it beside the launch script, so the next person to run the job inherits the same discipline.

When the checklist is green, you have avoided the large majority of beginner failures. The next track, Mid-level, takes ZeRO apart so you can predict its behaviour, tune it and run it across machines.

This is part two of three. It assumes you have already trained something with deepspeed.initialize, written a small JSON config, and launched with the deepspeed command. The version covered is DeepSpeed 0.19.7, released on 16 September 2026. The project is still 0.x, so releases can remove features, and this one did. The goal here is to make you the person on your team who can predict what a config will do before the job starts, and who can explain why a run is slow, out of memory, or hung.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run in a few minutes.

What the engine does in one step

Beginner gave you three calls: forward, backward, step. At this level you need to know what hides behind them, because every performance and memory problem is a statement about one of these phases.

deepspeed.initialize parses your config into a DeepSpeedConfig, initializes the process group (it calls deepspeed.init_distributed() for you), wraps your module in a DeepSpeedEngine, builds the optimizer from the optimizer block unless you passed one in code, and builds the scheduler. If you pass an optimizer in code, it overrides the JSON block. Everything after that is the engine reacting to your three calls.

During forward under ZeRO-3, each submodule's parameters are all-gathered just before it runs and released straight after. During backward the same gather happens again, and gradients are reduce-scattered into per-rank shards as they are produced. The engine counts micro-steps. step() is cheap on every micro-step except the last one in an accumulation window, where it performs the real work: unscale and clip the gradients, check for overflow, run the optimizer on each rank's shard, step the scheduler, and zero the gradients. You can ask the engine where it is with engine.is_gradient_accumulation_boundary().

Since 0.18.3 you can call loss.backward() directly on a tensor produced by an engine, instead of engine.backward(loss). The direct form is not supported at ZeRO stage 0, and on PyTorch builds that lack the internal APIs it raises Direct calls to tensor.backward() are not supported in this configuration. The portable habit is to keep engine.backward(loss), and treat the direct form as a convenience when you are composing with code you do not own.

ZeRO stages as memory arithmetic

You already know the stages shard optimizer state, then gradients, then parameters. The skill at this level is turning that into numbers, so that you can size a job on a napkin.

With mixed precision and Adam, each parameter costs roughly 2 bytes for the 16-bit weight, 2 bytes for the 16-bit gradient, and 12 bytes of optimizer state (a 4-byte fp32 master copy, plus two 4-byte Adam moments). That is 16 bytes per parameter before any activations. A 7-billion-parameter model therefore needs about 112 GB of model state, which does not fit on an 80 GB GPU under plain data parallelism whatever the batch size.

On 8 GPUs, a 7B model needs about 63 GB per GPU of model state at stage 1, about 31 GB at stage 2, and about 14 GB at stage 3. Activations come on top of that, which is why activation checkpointing remains relevant at every stage.

Memory is only half the story. Communication is the other half. Stage 1 and stage 2 communicate about as much as plain data parallelism, because reduce-scatter plus all-gather of the updated parameters moves the same volume as an all-reduce. Stage 3 adds a parameter all-gather in forward and again in backward, so its traffic is about 1.5 times that of data parallelism. That is the price of its memory savings, and it is why stage 3 on a slow network can be dramatically slower than stage 2 while using less memory.

This gives you the first decision rule. Choose the lowest stage that fits. If stage 2 fits, stay on it, because it has the best throughput of the sharded options. Move to stage 3 when the parameters themselves no longer fit, or when you need the extra headroom to grow the micro-batch enough to pay for the extra communication.

You do not have to guess whether a model fits. DeepSpeed ships estimators:

from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live
from deepspeed.runtime.zero.stage_1_and_2 import estimate_zero2_model_states_mem_needs_all_live

# model is an instantiated nn.Module (use the meta device or a small proxy if it is huge)
estimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=8, num_nodes=1)
estimate_zero2_model_states_mem_needs_all_live(model, num_gpus_per_node=8, num_nodes=1)

Each prints a table of per-GPU and per-CPU memory for offload and no-offload variants. There are matching ..._all_cold functions that take a parameter count instead of a model, which is useful before you have downloaded anything.

Reading the ZeRO config like an owner

The ZeRO block has many keys, and most tutorials copy a block with big round numbers and no explanation. Knowing the real defaults lets you remove noise from your configs and focus on the three or four keys that matter. These defaults come from the 0.19.7 code, and where the web documentation disagrees with the code, the code is right.

Two traps hide in this table. The first is overlap_comm. Its default is not a plain boolean: it is unset and resolves to true at stage 3 and false at stages 1 and 2. If you are on stage 2 and want communication overlapped with backward compute, you must set it true yourself. The second is stage3_prefetch_bucket_size. The documentation table lists 5e8, but the code default is 5e7. If you tuned your config from the web page you may have changed it tenfold without realising.

How do you use the stage 3 knobs? Think of stage3_max_live_parameters and stage3_max_reuse_distance as a memory budget for gathered parameters. Lowering them reduces peak memory and increases how often parameters are re-gathered. stage3_param_persistence_threshold is the opposite lever: small tensors such as layer norms and biases are cheap to keep and expensive to gather one at a time, so keeping them resident saves many tiny collectives. Raise it when a profile shows a flood of small all-gathers. reduce_bucket_size trades latency for memory: smaller buckets start communicating sooner, larger buckets use bandwidth more efficiently. The Hugging Face integration sets these from the model's hidden size when you write "auto": reduce_bucket_size becomes hidden size squared, the prefetch bucket becomes 0.9 times hidden size squared, and the persistence threshold becomes 10 times hidden size.

A sensible stage 3 starting point is therefore short:

{
  "train_micro_batch_size_per_gpu": 2,
  "gradient_accumulation_steps": 8,
  "bf16": { "enabled": true },
  "gradient_clipping": 1.0,
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "steps_per_print": 50
}

Everything else stays at its default until a measurement tells you otherwise. That is the habit to build: a short config that you understand beats a long one you copied.

Precision, loss scaling and the 0.19 default changes

Mixed precision is not only about speed. It is also about which copy of the numbers the optimizer updates, and what happens when a gradient is not finite. Three modes are current: fp16, bf16, and torch_autocast. The legacy amp block, which wraps Apex, is deprecated and scheduled for removal, so do not start anything new on it. You cannot enable fp16 and bf16 together, and the config raises bfloat16 and fp16 modes cannot be simultaneously enabled if you try.

fp16 has a narrow numeric range, so it needs loss scaling. The engine multiplies the loss by a scale (2^initial_scale_power, which is 65536 by default with the power at 16), runs backward, and checks the gradients for inf or NaN. If any rank finds one, the step is skipped, the scale is reduced, and the log prints a line starting with OVERFLOW! Rank 0 Skipping step. After loss_scale_window clean steps (1000 by default) the scale grows again. Setting loss_scale to 0 means dynamic scaling, which is what you want. A few skips at the start of training are normal and healthy. A steady stream of skips that drives the scale down to min_loss_scale means real instability, and the fix is a lower learning rate, clipping, or moving to bf16.

bf16 has the range of fp32, so it needs no loss scaling, and it is the default choice on any hardware that supports it. Since 0.19.3, bf16.check_grad_overflow defaults to true, matching fp16: a bf16 step with infinite or NaN gradients is now skipped instead of poisoning the weights. If your bf16 job upgraded across 0.19.3 and its step count and optimizer state no longer line up with an old run, that is why.

The other 0.19.3 flip is bigger. gradient_clipping used to default to 0.0, meaning disabled. It now defaults to 1.0, chosen to match PyTorch's FSDP2 reference. A config that omits the key is now clipped at norm 1.0. This is usually good for stability, but it silently changes the training dynamics of an old recipe. If you want no clipping, write "gradient_clipping": 0.0 explicitly.

The torch_autocast mode drives native torch.autocast from the config, with a dtype and a list of module types to run in lower precision (lower_precision_safe_modules, defaulting to linear and convolution layers). The data_types.grad_accum_dtype key sets the dtype used to accumulate gradients across micro-steps, which is worth keeping at fp32 when you accumulate many micro-batches.

For memory pressure, there are low-precision master states. bf16.bf16_master_weights_and_grads keeps the master weights and gradients in bf16, and bf16.bf16_optimizer_states also keeps the Adam moments in bf16. The second requires the first, and both require ZeRO stage 1, 2 or 3. Using master weights in bf16 without bf16 optimizer states needs CPU offload with DeepSpeedCPUAdam. The official example measured allocated memory falling from 25.74 GB to 16.17 GB, with step time rising from 0.60 s to 0.64 s. That is a large memory saving for a small throughput cost, but you are updating weights in a coarser format, so validate the loss curve against an fp32-master baseline before you adopt it.

Batch sizes and the accumulation boundary

Beginner taught the identity train_batch_size = train_micro_batch_size_per_gpu × gradient_accumulation_steps × data-parallel world size. At work, the identity bites in a specific way: you tuned a config on 8 GPUs, then moved to 16, and the job dies with

AssertionError: Check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size 32 != 4 * 1 * 4

The message prints the three terms as it sees them, so you can read exactly which one is stale. The robust habit is to specify only two of the three keys and let DeepSpeed infer the third. For most recipes the two to specify are the micro-batch per GPU, which is a memory decision, and the accumulation steps, which is an effective-batch decision. In Hugging Face you can write "auto" for all three and let TrainingArguments supply them.

Why does this matter beyond the assertion? The global batch size is a hyperparameter. If you double the GPUs and keep the micro-batch and accumulation steps fixed, you have quietly doubled the global batch, and the learning rate that worked before may now be wrong. Decide whether you are holding the global batch constant (so accumulation steps halve) or scaling it, and change the learning rate deliberately.

Gradient accumulation has a performance cost that is easy to miss. At stage 2 and stage 3, each backward reduces gradients into shards, so with accumulation you pay reduction communication on every micro-step, not only at the boundary. Two tools exist for this. engine.no_sync() is a context manager that skips synchronization, and since 0.19.2 engine.coalesce_grad_reduction() skips the per-backward reduce-scatters when you call backward many times per step, then reduces once. The coalescing manager is not supported under pipeline parallelism and cannot be nested inside no_sync.

Since 0.19.4 there is also managed_gradient_accumulation, which defaults to true. With it false, you own the boundary: you call engine.backward(loss, scale_wrt_gas=False) to accumulate only, and decide yourself when to call step(). This exists for training loops, such as reinforcement learning, where the number of backward passes per update is not fixed. It is incompatible with pipeline parallelism, DeepCompile and Apex AMP. Most teams should leave it at the default.

A related key is gradient_allreduce_op, added in 0.19.6. It defaults to "mean", which is what you expect from data parallelism. Setting "sum" is rejected with ZeRO-3, ZenFlow or DeepCompile, and the error says so: gradient_allreduce_op='sum' is not supported with ZeRO stage 3.

CPU and NVMe offload without surprises

Offload exists for one situation: the state does not fit on the GPU, and you accept a slower step to make it fit. Understanding its mechanics lets you predict how much slower.

ZeRO-Offload moves the optimizer state and the optimizer computation to the CPU, using DeepSpeedCPUAdam, which is a vectorized C++ implementation (AVX on x86, with ARM SVE and NEON kernels added in 0.19.7). It works at stages 1, 2 and 3 and is enabled with:

"zero_optimization": {
  "stage": 2,
  "offload_optimizer": { "device": "cpu", "pin_memory": true }
}

ZeRO-Infinity goes further, at stage 3 only: offload_param moves the sharded parameters to CPU or NVMe, and offload_optimizer can target NVMe too. NVMe offload needs nvme_path and the async_io op, which needs libaio-dev installed on the host. Check ds_report: if async_io shows [NO] under compatible, install the package before expecting NVMe offload to work. offload_param is a stage 3 feature, and so is partial optimizer offload through ratio (a value below 1.0 offloads only that fraction). Using ratio at another stage gives Partial offloading only supported for ZeRO Stage 3.

Where does the time go? The CPU optimizer step takes time proportional to the parameters each rank owns, divided by how many CPU cores that rank can use. Gradients travel to the CPU and updated parameters travel back over PCIe. So the step cost scales with PCIe bandwidth and CPU core count, not with GPU speed. On a machine with eight GPUs and a modest number of cores, each GPU's CPU Adam competes for the same cores, which is a classic reason an offloaded run is much slower than the arithmetic suggested. Setting thread counts (OpenMP) deliberately and binding cores with the launcher's --bind_cores_to_rank helps on multi-socket machines.

The 0.19.5 release changed a default that causes real incidents. pin_memory now defaults to true for both offload blocks, and ZeRO-3 gradient offload became asynchronous. Pinned (page-locked) host memory is faster to transfer but counts against the process's locked-memory limit, ulimit -l. On a host or container with a low limit, the job can fail with an out-of-memory error that has nothing to do with the GPU. The documentation's advice is to set "pin_memory": false explicitly if you see out-of-memory errors after upgrading, and in containers you can instead raise the limit with --ulimit memlock=-1.

The aio block tunes NVMe I/O, with defaults of a 1 MiB block_size, queue_depth 8, thread_count 1, single_submit false and overlap_events true. You do not have to guess: the ds_nvme_tune --nvme_dir /local_nvme command benchmarks the drive and suggests a configuration, and ds_io is a microbenchmark for the same I/O path. Local NVMe is dramatically better than a network filesystem for this purpose.

Two safety flags matter with offload. zero_force_ds_cpu_optimizer defaults to true, and it makes offload insist on DeepSpeed's CPU optimizers. If you give it a client optimizer you get a ZeRORuntimeException explaining that this will yield poor performance. The fix is to define the optimizer in the JSON, or set the flag false if you accept the cost. Separately, zero_allow_untested_optimizer must be true to use a non-DeepSpeed optimizer with ZeRO at all, otherwise you get You are using an untested ZeRO Optimizer.

Building models too big for one GPU

ZeRO-3 shards parameters during training, but there is a chicken-and-egg problem at construction time: if you build the model normally, every rank first materializes the whole thing, which is exactly what did not fit. The context manager deepspeed.zero.Init solves this by partitioning each parameter as it is created.

import torch, deepspeed

with deepspeed.zero.Init(config_dict_or_path=ds_config, dtype=torch.bfloat16):
    model = BigModel()

Inside that block, every parameter is sharded across the data-parallel group the moment its module is constructed, so no rank ever holds the full model. It accepts further keyword arguments, including remote_device to place shards on CPU and pin_memory. In 0.19.4, zero.Init began to warn when it silently falls back to a single rank, which is the signal that your process group was not initialized as you thought.

Once parameters are sharded, reading them is a trap. A parameter that belongs to a partitioned module has shape=torch.Size([0]) outside a gather, and state_dict() returns empty tensors. The tool for legitimate access is deepspeed.zero.GatheredParameters:

with deepspeed.zero.GatheredParameters(model.embed.weight, modifier_rank=0):
    if torch.distributed.get_rank() == 0:
        model.embed.weight.data.normal_()

On entry it all-gathers the named parameters, and on exit it releases them. With modifier_rank=0, whatever rank 0 changed inside the block is broadcast back to the others. Without a modifier rank the access is read-only. Use it for custom weight initialization, tying weights, and inspecting values while debugging.

With Hugging Face models the pattern differs. Outside the Trainer, create an HfDeepSpeedConfig object from your config before calling from_pretrained, and keep that object alive for as long as you need ZeRO-3 loading to apply:

from transformers import AutoModelForCausalLM
from transformers.integrations import HfDeepSpeedConfig

dschf = HfDeepSpeedConfig(ds_config)   # must exist before from_pretrained and must stay referenced
model = AutoModelForCausalLM.from_pretrained("my-org/my-model")

That object registers the ZeRO-3 setting globally so that from_pretrained loads through zero.Init. Inside the Trainer, the equivalent rule is to create TrainingArguments before you load the model. If you reverse the order, the model loads fully on every rank, and you find out when the first GPU runs out of memory.

The Hugging Face contract: Trainer and Accelerate

Most teams reach DeepSpeed through Hugging Face rather than writing the loop themselves. See the Hugging Face guide for the libraries themselves. Here the point is how the config and the arguments divide responsibility, because duplicated settings are the most common source of confusing errors.

With the Trainer, you pass TrainingArguments(deepspeed="ds_config.json"). In the JSON you write the literal string "auto" wherever the value should come from the training arguments. The filled keys include the micro-batch per GPU (from per_device_train_batch_size), the accumulation steps, the optimizer learning rate and weight decay, the scheduler warmup, gradient_clipping, and the fp16 or bf16 enable flags. A good config therefore looks like this:

{
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "bf16": { "enabled": "auto" },
  "optimizer": { "type": "AdamW",
    "params": { "lr": "auto", "weight_decay": "auto", "betas": "auto", "eps": "auto" } },
  "scheduler": { "type": "WarmupDecayLR",
    "params": { "total_num_steps": "auto", "warmup_min_lr": "auto",
                "warmup_max_lr": "auto", "warmup_num_steps": "auto" } },
  "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_gather_16bit_weights_on_model_save": true
  }
}

The rule behind "auto" is that one source of truth exists for each value. If you hard-code a value in the JSON that also appears in TrainingArguments and they disagree, the Trainer raises an error naming the mismatch rather than guessing. Install the extra with pip install transformers[deepspeed] and launch with deepspeed, torchrun or accelerate launch; all three work because DeepSpeed reads the standard distributed environment variables.

With Accelerate, you run accelerate config and choose DeepSpeed, which writes distributed_type: DEEPSPEED into the YAML. You can then either set the plugin keys there (zero_stage, offload_optimizer_device, offload_param_device, zero3_init_flag, zero3_save_16bit_model, gradient_clipping, and deepspeed_multinode_launcher), or point at a full file with deepspeed_config_file. Do not mix the two. If you provide a config file and also leave plugin keys in the YAML, Accelerate raises When using deepspeed_config_file, the following accelerate config variables will be ignored: [...]. Remove the duplicates, or put "auto" in the JSON.

When the optimizer and scheduler are defined in the JSON, your Accelerate code must hand Accelerate placeholder objects, DummyOptim and DummyScheduler, so that it does not try to build real ones. Accelerate also does not support DeepSpeed pipeline parallelism, a custom mpu, or training more than one model through the engine. If your design needs any of those, you are writing against deepspeed.initialize directly.

Checkpoints that survive a real job

A training job that runs for days will be interrupted, resumed on a different cluster, or evaluated by someone who has no GPUs. Your checkpoint strategy determines whether any of that is painful.

engine.save_checkpoint(save_dir, tag=None, client_state={}) must be called on every rank, not only rank 0, because each rank writes its own shard. With no tag, the tag becomes global_step<N>, and a latest file records which tag is newest. The layout under save_dir/<tag>/ contains mp_rank_00_model_states.pt plus per-rank optimizer shards, and DeepSpeed copies a zero_to_fp32.py script into save_dir for you. Putting save_checkpoint inside an if rank == 0 block is a classic hang, because the other ranks are not there for the collective writes.

Three common questions have three different answers.

"I want to resume training." Call engine.load_checkpoint(load_dir, tag=None). It returns the load path and the client_state you saved, which is a good place for a data-loader position and a random seed. Be aware that client_state is pickled, so do not store secrets in it. One documented restriction: under ZeRO-3 you cannot call load_checkpoint right after save_checkpoint in the same engine. Re-initialize the engine first.

"I want weights to evaluate or publish." There are two routes. In-job, set stage3_gather_16bit_weights_on_model_save true and call engine.save_16bit_model(save_dir, save_filename="pytorch_model.bin") on all ranks. Offline, with no GPU, run the consolidation script:

python save_dir/zero_to_fp32.py save_dir output_dir --safe_serialization --max_shard_size 5GB

In 0.19.x the second positional argument is an output directory. Many blog posts and the older tutorials show the form ./zero_to_fp32.py . pytorch_model.bin, which is out of date. The script needs about twice the checkpoint size in CPU RAM, so run it on a large-memory machine. You can select a tag with -t global_step1000 and skip frozen parameters with --exclude_frozen_parameters. From Python, get_fp32_state_dict_from_zero_checkpoint returns a state dictionary, and load_state_dict_from_zero_checkpoint loads it into your model. After that second function the model sits on the CPU and must be re-initialized with DeepSpeed before you train further.

"I want to resume on a different number of GPUs." This is where ordinary ZeRO checkpoints fall short, because shards are tied to the world size that wrote them. The answer is Universal Checkpointing. Convert, then load with a flag:

python -m deepspeed.checkpoint.ds_to_universal \
  --input_folder ckpt/global_step1000 \
  --output_folder ckpt_universal/global_step1000

{ "checkpoint": { "load_universal": true } }

Then call engine.load_checkpoint(ckpt_universal_dir, tag=...) with an explicit tag. Without a tag, the loader looks for a latest_universal file and raises Invalid for universal checkpoint: <dir>/latest_universal does not exist when it is absent. Universal Checkpointing is also the replacement for ZeRO-3 elastic checkpoints, which were deprecated in 0.19.3 and log ZeRO-3 elastic checkpointing is deprecated and no longer supported. Stage 1 and 2 elastic checkpoints are unchanged.

Size your storage honestly. With fp32 Adam, optimizer state costs about 12 bytes per parameter on top of the weights, so a 7B-parameter checkpoint with optimizer state is on the order of 100 GB. Keeping every checkpoint for a long run adds up quickly, so decide a retention policy before the job starts.

Launching across machines

On a single node, deepspeed train.py is enough. Past one node, you need to decide how processes get started on the other machines, and the launcher offers several answers. All of them end the same way: one process per accelerator, with RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT set.

The default is pdsh over SSH. You provide a hostfile in MPI style:

worker-1 slots=8
worker-2 slots=8

and launch from one machine:

deepspeed --hostfile=hostfile --num_nodes 2 --num_gpus 8 \
  train.py --deepspeed --deepspeed_config ds.json

The default hostfile path is /job/hostfile. If there is none, the launcher logs Unable to find hostfile, will proceed with training with local resources only, which is a harmless warning on one node and a worrying one on two. You can restrict hosts and slots with --include or --exclude (mutually exclusive), for example -i "worker-1@worker-2:0,1". This path needs passwordless SSH from the launching machine to every worker, and pdsh installed, otherwise you see Cannot find pdsh, please install via 'apt-get install -y pdsh'.

In Kubernetes, SSH between pods is usually unwelcome, and you have two cleaner routes. The first is to skip SSH entirely: run the same command on every pod with --no_ssh --node_rank=<n> --master_addr=<ip> --master_port=29500, and let your orchestrator supply the rank and address. A headless Service gives you a stable name for the rank-0 pod. The second is torchrun: DeepSpeed reads the torch.distributed environment, so torchrun --nproc_per_node N train.py --deepspeed_config ds.json works, and the Kubeflow Training Operator's PyTorchJob and MPIJob resources are built around exactly this model. See the Kubernetes guide and the Kubeflow guide for the platform side.

Environment variables are the part people forget. The launcher exports every NCCL_* variable and the prefixes MLFLOW, PYTHON, MV2 and UCX to the workers. For anything else, put KEY=VALUE lines in a .deepspeed_env file in the working directory or your home directory (change the filename with DS_ENV_FILE). Because those values are passed through the pdsh or MPI command line, they can show up in process listings on every node. That means a .deepspeed_env file is a poor place for API keys. Inject tokens through your scheduler's secret mechanism instead.

When two jobs share nodes, give each a different --master_port, otherwise the second fails with an address-already-in-use error. Use --include or CUDA_VISIBLE_DEVICES to keep them on different GPUs. --enable_each_rank_log <dir> writes one log file per rank, which turns a 64-way interleaved console into something you can read.

Ops, builds and containers

DeepSpeed's speed comes partly from compiled extensions: fused Adam, CPU Adam, async I/O and others. PyPI ships only a source distribution, so by default these ops are compiled just in time, with ninja, the first time something needs them. That is convenient on a laptop and a hazard in production, and understanding the two failure families will save you hours.

The first family is a missing toolchain. A runtime-only CUDA container has no nvcc, so the first op that needs compiling fails with MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s). The fix is either to build on a -devel image or to pre-build the ops. The second is a version mismatch: CUDAMismatchException when the system nvcc differs from the CUDA version PyTorch was built with, and RuntimeError: PyTorch version mismatch! DeepSpeed ops were compiled and installed with a different version than what is being used at runtime when you upgraded torch after pre-building. Both mean the same thing: DeepSpeed's ops must be built against the torch and CUDA that will run them.

The production answer is to pre-build in a controlled image stage:

FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS build
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3-pip python3-venv libaio-dev && rm -rf /var/lib/apt/lists/*
RUN python3 -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
RUN pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
ENV TORCH_CUDA_ARCH_LIST="8.0;9.0" DS_BUILD_OPS=1
RUN pip install deepspeed==0.19.7 --no-build-isolation
RUN ds_report

The image tag, torch version and architectures above are placeholders; match them to your own stack, and keep the CUDA toolkit used for the build consistent with the torch build. DS_BUILD_OPS=1 builds every op compatible with the machine, and you can build selectively with flags such as DS_BUILD_CPU_ADAM=1 and DS_BUILD_AIO=1. TORCH_CUDA_ARCH_LIST is the lever that matters when a job fails with CUDA error: no kernel image is available for execution on the device: the ops were built for different GPU architectures than the ones you are running on. Rebuild with an architecture list that covers your hardware. Read the Docker guide for multi-stage patterns, and note that the build stage needs the toolkit while the final stage only needs the compiled result and the runtime.

If you do rely on JIT, control where it writes. TORCH_EXTENSIONS_DIR overrides the default cache of ~/.cache/torch_extensions. Give each job its own directory on shared filesystems, because a stale lock file from a killed build produces a hang at Loading extension module with a message about waiting for a lock. Deleting that op's directory in the cache clears it.

ds_report is the single most useful diagnostic. Its operator table has an installed column ([YES] means pre-built) and a compatible column ([NO] means this machine cannot build the op, usually from a missing system library or an unsupported architecture). Run it in the image at build time and again at job start, and save the output with the run's artifacts. Diffing two ds_report outputs is often enough to explain why "the same job" behaves differently on two clusters.

Measuring: monitors, profilers and communication

Tuning without measurement is folklore. DeepSpeed has built-in instruments, each with a different cost, and knowing which to reach for first is half the skill.

Step timing and progress. steps_per_print controls the progress line. Set wall_clock_breakdown to true and the engine logs forward, backward, backward allreduce and step times, which quickly tells you whether time goes to compute, communication or the optimizer. These timers synchronize, so they perturb the thing they measure slightly, and you should turn them off for production runs.

Metrics backends. The tensorboard, wandb, comet and csv_monitor blocks log to your tracker. The fields include Train/Samples/train_loss, Train/Samples/lr and, for fp16, Train/Samples/loss_scale. The elapsed-time metrics only appear when wall_clock_breakdown or the flops profiler is on. Keep tokens out of the config: the documentation itself advises against saving the Comet API key in code, and the same applies to W&B. Take them from the environment. For experiment tracking at the team level, the MLflow guide and the Weights & Biases guide cover what to do with the numbers.

The FLOPs profiler. Enable flops_profiler with a profile_step and it reports parameters, FLOPs and latency per module for that one step, which finds the layers that dominate. Pick a step after warmup, since the first steps include compilation and allocator noise.

Communication logging. The comms_logger block records every collective, and deepspeed.comm.log_summary() prints a table by operation and message size. Be careful: comms logging synchronizes each collective, so it changes performance. Use it on a short diagnostic run, not a long one. For raw network health independent of your model, the ds_bench command runs all-reduce, all-gather, all-to-all and point-to-point benchmarks across your nodes, and the numbers tell you whether the fabric delivers what you paid for.

A practical diagnosis order for a slow job follows from these tools:

If you use ZeRO++ on stage 3 to reduce cross-node traffic, three keys apply: zero_quantized_weights quantizes weights for the all-gather, zero_hpz_partition_size keeps a secondary shard within each node (set it to the number of GPUs per node), and zero_quantized_gradients quantizes gradients. These help when the inter-node network is the bottleneck, which is common on cloud instances with modest networking. The hierarchical partition trades GPU memory for less cross-node traffic, so check that you have headroom.

Debugging beyond the basics

Most distributed failures come in a handful of shapes. Learn to recognise the shape from the first message and you skip most of the guesswork.

Config errors at startup. These are cheap and loud. The batch identity assertion, the missing --deepspeed_config message (DeepSpeed requires --deepspeed_config to specify configuration file), and the pair Not sure how to proceed, we were given deepspeed configs in the deepspeed arguments and deepspeed.initialize() function call are all about where the config comes from. Pass it one way: either config= or the command-line flag. A config that works with an old release and fails with 0.19.7 usually contains a removed block, and the DeepSpeedConfigError text names it.

Hangs. A hang means ranks disagree about which collective comes next. The NCCL watchdog message, Watchdog caught collective operation timeout ... Timeout(ms)=600000, is the end of the story, not the beginning. Work backwards. Which rank skipped a step or stalled? Typical causes are uneven data (one rank got an empty batch), a rank-0-only branch that calls a collective, a very slow checkpoint write, or a dataloader worker that died. Set NCCL_DEBUG=INFO for the network picture, and use the PyTorch flight recorder through TORCH_NCCL_TRACE_BUFFER_SIZE to see the last collectives each rank issued. Raise the timeout with DEEPSPEED_TIMEOUT only when an operation is legitimately long, never to hide a desynchronization.

Memory errors. Separate the three kinds. A CUDA out-of-memory error during construction means you needed zero.Init. One at the first forward means activations or gathered parameters, so look at the micro-batch, activation checkpointing and the stage 3 live-parameter limits. One after an upgrade, on a host with offload, is probably the pinned-memory default described earlier. see_memory_usage and the memory_breakdown option help attribute the numbers.

Wrong or empty weights. If you read a parameter or call state_dict() under ZeRO-3 and see zero-sized tensors, the parameters are partitioned and you must use GatheredParameters, save_16bit_model or zero_to_fp32.py.

Upgrades, deprecations and what 0.19.7 removed

Because DeepSpeed is pre-1.0, an upgrade is a change to your training recipe, not only to a dependency. The recent history shows why.

Release 0.19.3 changed two defaults: gradient_clipping from 0.0 to 1.0, and bf16.check_grad_overflow from false to true. Release 0.19.5 made pin_memory default to true in offload blocks. Release 0.19.6 lowered the default process-group timeout from 30 minutes to 10. None of those is a bug, and each can change a run's behaviour with no change to your config. That is why you read the release notes before every bump.

Release 0.19.7 went further and removed features outright. Gone are the Nebula checkpoint engine, the deepspeed.compression library and Mixture-of-Quantization, the 1-bit optimizers (OneBitAdam, OneBitLamb and ZeroOneAdam), MiCS sharding, and Sparse Attention. Leftover config blocks for them now fail at parse time with a DeepSpeedConfigError, rather than being silently ignored, which is the safer behaviour: a leftover compression_training block used to mean the model trained unquantized without telling you. Migrations are straightforward:

Other features are still present but scheduled for removal under the project's tracking issue: sparse_gradients, LoCo-ZeRO++, Progressive Layer Dropping and the Apex amp block. Treat them as deprecated today. For mixed precision, use fp16, bf16 or torch_autocast.

A sound upgrade routine has five steps. Read the release notes between your version and the target. Parse your configs with the new version in CI. Run ds_report and diff it against the old one. Run a short job on the new version and compare the first few hundred steps of loss against a stored baseline. Rebuild any pre-compiled ops, since a new torch or CUDA invalidates them. Resume from checkpoints through Universal Checkpointing if you also change the parallelism layout.

Putting it all together

Here is how the pieces combine for a realistic task: fine-tuning a 7B model on one node of 8 GPUs, from a repository you will hand to a colleague.

1. Size it. Run the stage 2 and stage 3 estimators for your GPU count. At 16 bytes per parameter the model state is about 112 GB, so stage 2 on 8 GPUs needs roughly 31 GB of state per GPU, which fits an 80 GB card with room for activations. Start with stage 2, because it avoids stage 3's extra communication.
2. Write the short config. Specify the micro-batch and accumulation steps, not the global batch. Use bf16, set gradient_clipping explicitly, and set overlap_comm true for stage 2.
3. Build the image. Use a CUDA devel stage with DS_BUILD_OPS=1 and an explicit TORCH_CUDA_ARCH_LIST, pin DeepSpeed to 0.19.7, run ds_report at build time, and ship only the result.
4. Launch. Use deepspeed --num_gpus 8 on one node, or torchrun under Kubernetes, with --enable_each_rank_log for the first runs. Pass secrets through the scheduler.
5. Measure. Turn on wall_clock_breakdown for 50 steps, and send metrics to TensorBoard or W&B. Record the step time, peak memory and the number of skipped steps.
6. Checkpoint. Save from every rank on a schedule that matches your risk tolerance. At the end, consolidate with zero_to_fp32.py --safe_serialization on a CPU machine, and load the result for evaluation.
7. Escalate only when measured. If memory is short, move to stage 3, then to optimizer offload, explicitly deciding on pin_memory for the host.

Where you are now

You can now predict a configuration instead of hoping. You can turn a model size into per-GPU memory for each ZeRO stage and pick the lowest stage that fits. You know the real defaults behind the ZeRO keys, and where the documentation and code disagree. You understand the 0.19 default flips on clipping, bf16 overflow checks, pinned memory and timeouts, and you read a release note as a change to your recipe. You can build models with zero.Init, inspect them with GatheredParameters, checkpoint them in three different ways, and move a run to a different world size with a Universal Checkpoint. You can launch across machines with pdsh, MPI, SLURM, torchrun or no SSH at all, and you know why a hostfile or a .deepspeed_env is the wrong place for a secret.

The Senior guide takes the same material to the platform level: parallelism composition with tensor, pipeline and sequence parallelism, failure modes at scale, security and cost, and when to choose something other than DeepSpeed.

Sources

• DeepSpeed home
• Getting started
• Advanced install
• Configuration JSON reference
• ZeRO tutorial
• ZeRO-Offload tutorial
• ZeRO++ tutorial
• Universal Checkpointing
• Monitor
• FLOPs profiler
• Communication logging
• API reference: training
• API reference: ZeRO-3
• API reference: model checkpointing
• DeepSpeed on GitHub
• Release notes
• Feature removal tracking issue
• DeepSpeed on PyPI
• Hugging Face Transformers DeepSpeed integration
• Hugging Face Accelerate DeepSpeed guide

Part two of three. A cumulative review of Beginner and Mid-level material for DeepSpeed 0.19.7, organised by topic rather than level, in about thirty-five minutes. At this level interviewers stop asking what ZeRO stands for and start asking what you would check, what you would trade, and why. Senior reviews all three levels.

Where this picks up

The model in one screen

DeepSpeed wraps a PyTorch model in an engine that owns distributed setup, mixed precision, memory partitioning, accumulation, clipping and checkpointing, all driven by one config. ZeRO removes the redundancy in data parallelism by sharding optimizer state, then gradients, then parameters, so total memory per GPU falls roughly in proportion to the GPU count.

The numbers to keep in your head: with mixed precision and Adam, a parameter costs about 16 bytes (2 weights, 2 gradients, 12 optimizer state). Stage 1 leaves 4 + 12/N, stage 2 leaves 2 + 14/N, and stage 3 leaves 16/N. Stages 1 and 2 communicate about as much as plain data parallelism, and stage 3 communicates about 1.5 times as much.

Commands and files to have ready

deepspeed --num_gpus 8 train.py --deepspeed --deepspeed_config ds_config.json
deepspeed --hostfile=hostfile --num_nodes 2 --num_gpus 8 train.py --deepspeed --deepspeed_config ds.json
torchrun --nproc_per_node 8 train.py --deepspeed_config ds.json
ds_report                                   # op status, versions, environment
python save_dir/zero_to_fp32.py save_dir out_dir --safe_serialization
python -m deepspeed.checkpoint.ds_to_universal --input_folder IN --output_folder OUT
ds_bench                                    # collective benchmarks
ds_nvme_tune --nvme_dir /local_nvme         # tune NVMe offload

Scenario questions

"The job runs out of memory. Walk me through it."

A strong answer separates when the failure happens. During model construction: the full model is being materialized on every rank, so you need deepspeed.zero.Init (or HfDeepSpeedConfig kept alive before from_pretrained, or TrainingArguments created before loading in the Trainer). At the first forward or backward: the model states fit, but activations or gathered parameters do not, so check the micro-batch, activation checkpointing, and the stage 3 live-parameter limits. After an upgrade, on a host with offload: since 0.19.5 pin_memory defaults to true and pinned memory counts against ulimit -l, so either raise the limit or set "pin_memory": false. At steady state: run the memory estimators to find which stage should have fit, then move up the ladder in order: stage 2, stage 3, optimizer offload, parameter offload, NVMe.

"The job is slower than expected. What do you check first?"

Start with whether the GPUs are busy. Idle GPUs mean the problem is the input pipeline or communication, not compute. Then turn on wall_clock_breakdown for a short run and split the step into forward, backward, backward allreduce and optimizer step. If collectives dominate, run ds_bench to see whether the fabric delivers its rated bandwidth, and consider whether you are on stage 3 across a slow inter-node link when stage 2 would fit. ZeRO++ (zero_quantized_weights, zero_hpz_partition_size, zero_quantized_gradients) is the next lever for network-bound stage 3 jobs. If the optimizer step dominates and you offload, the cost is CPU Adam and PCIe, so check core binding with --bind_cores_to_rank and pinned memory. Only after this should you touch bucket sizes, prefetch and persistence thresholds, one at a time.

A good candidate also mentions the cost of the measurement itself: the wall-clock timers and comms_logger synchronize, so they perturb performance and belong on short diagnostic runs.

"The job hangs for ten minutes and then dies with a watchdog timeout."

A hang means ranks disagree about which collective comes next. The timeout message is the end of the story. Ask which rank was different: one that got an empty batch, one that skipped a forward on a data-dependent branch, one that called a collective inside an if rank == 0 block (a common one is save_checkpoint), a dataloader worker that died, or a slow checkpoint write. Use NCCL_DEBUG=INFO and the PyTorch flight recorder (TORCH_NCCL_TRACE_BUFFER_SIZE) to see what each rank last issued, and per-rank logs with --enable_each_rank_log. The default timeout fell from 30 to 10 minutes in 0.19.6, so a job that "never timed out before" may have been hiding a slow step. Raise DEEPSPEED_TIMEOUT only for a legitimately long operation.

"You upgraded DeepSpeed and the loss curve changed. Why?"

Defaults moved. In 0.19.3, gradient_clipping changed from 0.0 (off) to 1.0, and bf16.check_grad_overflow changed from false to true. In 0.19.5, offload pin_memory became true by default. In 0.19.6, the process-group timeout fell to 10 minutes. A config that never mentioned clipping is now clipped. The fix is to write explicit values for anything you rely on, and to diff release notes before comparing runs. In 0.19.7, features were removed outright (Nebula, compression, 1-bit optimizers, MiCS, Sparse Attention), and a leftover config block now fails at parse time with a DeepSpeedConfigError rather than being ignored.

"Training fails with a batch size assertion after we doubled the GPUs."

The identity train_batch_size = micro-batch × accumulation × world size no longer holds, and the message prints the three terms so you can see which is stale. The fix is to specify only two of the three. The deeper point, which separates mid from beginner, is that the global batch is a hyperparameter. Doubling GPUs with the same micro-batch and accumulation doubles it, so you must decide whether to hold it constant (halve accumulation) or scale it (and adjust the learning rate).

"We need to evaluate the model on a laptop without GPUs."

Use the offline consolidation script, which needs no GPU: python save_dir/zero_to_fp32.py save_dir out_dir --safe_serialization. It needs about twice the checkpoint size in CPU RAM. In 0.19.x the second argument is an output directory, and older tutorials that show a single output filename are stale. For in-job export under ZeRO-3, set stage3_gather_16bit_weights_on_model_save and call save_16bit_model on all ranks. Mention that safetensors output avoids pickle.

"Resume the run on a different number of GPUs."

Ordinary ZeRO checkpoints are tied to the world size that wrote them. Convert with ds_to_universal, set "checkpoint": {"load_universal": true}, and load with an explicit tag. A missing tag fails with latest_universal does not exist. Universal Checkpointing is also the supported replacement for ZeRO-3 elastic checkpoints, which have been deprecated since 0.19.3.

Trade-off questions

ZeRO-2 or ZeRO-3?

Stage 2 has the better throughput because its communication matches plain data parallelism, and it fits any model whose weights and activations fit on one GPU. Stage 3 saves the most memory but adds a parameter gather in forward and backward, about 1.5 times the traffic. Pick the lowest stage that fits, and move to stage 3 when parameters do not fit or when the freed memory buys a larger micro-batch that repays the communication. A strong answer adds that network quality decides this: stage 3 across slow inter-node links can lose badly.

Offload or more GPUs?

Offload is cheaper per job and slower per step. It makes sense when GPUs are scarce, the job is short, or you are fine-tuning on one node. It is bounded by PCIe bandwidth and CPU Adam throughput, and NVMe offload by drive bandwidth. More GPUs cost more but keep the step fast. The answer should mention measuring the step time at each rung of the ladder rather than assuming.

fp16 or bf16?

bf16 on any hardware that supports it: it has fp32's range, needs no loss scaling, and removes a class of overflow problems. fp16 is for older hardware, and needs dynamic loss scaling, where a few early OVERFLOW skips are normal and a steady stream is not. Note that bf16 now skips non-finite steps too, since 0.19.3.

DeepSpeed or Hugging Face Accelerate alone?

They are not opposed. Accelerate can drive DeepSpeed, with distributed_type: DEEPSPEED. The limits are that Accelerate does not support DeepSpeed pipeline parallelism, a custom mpu, or multiple models through the engine. If you need those, you write against deepspeed.initialize directly. If you do not, Accelerate or the Trainer gives you the same engine with less code.

Config file or code arguments?

A config file is reviewable, diffable, and can be tested in CI. For Trainer-driven jobs, use "auto" so there is one source of truth per value. Hard-coded values that disagree with TrainingArguments raise errors. Passing the config both through config= and the command-line flag also fails, so choose one.

Pre-built ops or JIT?

JIT is convenient on one machine. In production, pre-build in a CUDA devel image with DS_BUILD_OPS=1 and a stated TORCH_CUDA_ARCH_LIST, so every node starts with the same binaries and no compile at job start. The cost is that ops must be rebuilt whenever torch or CUDA changes, otherwise you see PyTorch version mismatch!.

Pipeline, tensor or sharded data parallelism?

At the mid level the expected answer is to prefer ZeRO and data parallelism first, because they need no model changes. Tensor and pipeline parallelism change the model structure, bring their own limits (pipeline models use train_batch, not forward, backward and step, and Accelerate does not support them), and are for when ZeRO alone no longer scales. Senior goes deeper.

What a strong answer covers that a weak one misses

Other signals of experience: knowing that the documentation and the code sometimes differ (for example the stage3_prefetch_bucket_size default is 5e7 in code, though a docs table says 5e8), that overlap_comm resolves by stage rather than being a plain boolean, and that engine.backward(loss) is the portable form while direct loss.backward() is unsupported at stage 0.

Gotchas interviewers love

Common interview questions

Why does ZeRO-3 communicate more than stage 2? Parameters are partitioned, so each module's weights are all-gathered before forward and again before backward, then released. Stage 2 only reduce-scatters gradients and all-gathers updated parameters, which matches the volume of an all-reduce.

What is the difference between zero.Init and GatheredParameters? zero.Init partitions parameters as the model is constructed, so the full model never exists on one GPU. GatheredParameters temporarily all-gathers already partitioned parameters so you can read or modify them, with modifier_rank broadcasting rank 0's changes back.

What does a skipped step mean? In fp16, or bf16 with the overflow check on, a step whose gradients contain inf or NaN is skipped and the scale reduced. A few at the start are normal.

How do you launch without SSH on Kubernetes? Use --no_ssh --node_rank=<n> --master_addr=<ip> --master_port=29500 on every pod, or torchrun, with a headless Service for the rank-0 address. See the Kubernetes guide.

What does ds_report tell you? For each op, whether it is pre-built (installed) and whether this machine can build it (compatible), plus torch, CUDA and DeepSpeed versions. It is the first command to run on a strange environment.

How do you keep memory down when offloading to NVMe? NVMe offload is stage 3 only, needs libaio-dev for the async_io op, and a local drive. ds_nvme_tune suggests the aio settings.

How does the Trainer integration avoid duplicate settings? With "auto" values in the JSON, so each value has one source of truth in TrainingArguments. For Accelerate, either use plugin keys or a config file, not both.

What changed in 0.19.7? Nebula, the compression library, MoQ, 1-bit Adam, 1-bit LAMB, 0/1 Adam, MiCS and Sparse Attention were removed, and leftover blocks fail with DeepSpeedConfigError.

When would you not use DeepSpeed? When the model fits comfortably with plain data parallelism, when your stack already standardizes on PyTorch FSDP2, or when you only need serving, where vLLM is the usual choice.

How do you track runs? The built-in tensorboard, wandb, comet and csv_monitor blocks, or the MLflow and Weights & Biases integrations, with tokens from the environment rather than the config.

Sixty-second self-test

Answer aloud, then check against the sections above.

1. How many bytes per parameter does mixed-precision Adam need, and what does stage 3 leave per GPU?
2. Which ZeRO stages communicate more than plain data parallelism, and by how much?
3. Name three defaults that changed between 0.19.3 and 0.19.6.
4. Why must every rank call save_checkpoint?
5. What is the difference between zero_to_fp32.py and a Universal Checkpoint?
6. A job hangs and then reports a watchdog timeout. What is your first question?
7. Why does overlap_comm need setting by hand at stage 2?
8. What two things make the Trainer's ZeRO-3 load go wrong?
9. Which blocks does a 0.19.7 config fail on?
10. Where would you not put an API key?

If you stumble on more than three, reread the matching sections of the Mid-level detailed guide before moving on.

Part two of three. This file is the working companion to the Mid-level guide for DeepSpeed 0.19.7: the mistakes people make once the first job runs, the habits that pay off most, practice cards, and recipes you can paste into a runbook. It assumes you know the basics, and each item says what you will see, why it happens, and what to do.

Mistakes at this level

Beginner mistakes are about getting a job to start. These are about jobs that start, run, and then cost you time or money.

The batch identity broke when the GPU count changed

Symptom. AssertionError: Check batch related parameters. train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size 32 != 4 * 1 * 4.

Cause. You hard-coded all three batch keys for one world size. The message prints the three terms it computed, so the stale one is visible.

Fix. Specify only two of the three, normally the micro-batch and accumulation steps. In Hugging Face, use "auto". Then decide on purpose whether the global batch should stay constant or scale, and adjust the learning rate if it scales.

A config worked on 0.18 and fails on 0.19.7

Symptom. DeepSpeedConfigError: DeepSpeed Sparse Attention has been removed; ..., or the equivalent for Nebula, compression, MoQ or MiCS.

Cause. 0.19.7 removed those features, and leftover blocks now fail at parse time instead of being ignored.

Fix. Delete the block. For MiCS use ZeRO-3 with zero_hpz_partition_size. For 1-bit optimizers use AdamW, or ZeRO++ quantized communication. For compression, use an external quantization toolkit.

Old clipping behaviour assumed

Symptom. A recipe that trained one way on 0.18 behaves differently after the upgrade, with no config change.

Cause. gradient_clipping used to default to 0.0 (off). Since 0.19.3 it defaults to 1.0. In the same release bf16.check_grad_overflow flipped to true.

Fix. Write the clipping value explicitly, including 0.0 if you want it off. Log the number of skipped steps.

Out of memory after an upgrade, on an offload job

Symptom. Host or CUDA out-of-memory errors on a config that worked, only with offload enabled.

Cause. Since 0.19.5, pin_memory defaults to true in both offload blocks, and pinned memory counts against ulimit -l.

Fix. Raise the limit (--ulimit memlock=-1 in a container) or set "pin_memory": false.

Timeout that used to be 30 minutes is now 10

Symptom. Watchdog caught collective operation timeout ... Timeout(ms)=600000 on a job whose slowest phase, such as a big checkpoint write, takes twelve minutes.

Cause. 0.19.6 lowered the default process-group timeout from 30 to 10 minutes.

Fix. If the operation is legitimately slow, set DEEPSPEED_TIMEOUT=30 (minutes). If you do not know why it is slow, investigate first. Raising the timeout on a desynchronized job only makes it fail later.

save_checkpoint inside if rank == 0

Symptom. The job hangs at the first checkpoint, then dies with a timeout.

Cause. Each rank writes its own shard, so every rank must call it.

Fix. Call engine.save_checkpoint(...) on all ranks, and guard only the extra work you do around it, such as uploading to a bucket.

Reading ZeRO-3 weights directly

Symptom. Parameters with shape=torch.Size([0]), or a state_dict() full of empty tensors.

Cause. Parameters are partitioned across ranks.

Fix. Use deepspeed.zero.GatheredParameters for reads and edits, engine.save_16bit_model in the job, or zero_to_fp32.py offline.

Following an outdated zero_to_fp32.py example

Symptom. ./zero_to_fp32.py . pytorch_model.bin fails or does something unexpected.

Cause. In 0.19.x the second positional argument is an output directory, not a filename.

Fix. python save_dir/zero_to_fp32.py save_dir output_dir --safe_serialization. Allow about twice the checkpoint size in CPU RAM.

Pre-built ops against the wrong torch

Symptom. RuntimeError: PyTorch version mismatch! DeepSpeed ops were compiled and installed with a different version than what is being used at runtime, or CUDAMismatchException.

Cause. Torch or CUDA changed after the ops were built, or the system nvcc differs from the one torch was built with.

Fix. Rebuild DeepSpeed against the current torch in the same image. Never carry a built DeepSpeed across torch versions.

Everything works on your laptop, nothing compiles in the container

Symptom. MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s).

Cause. A runtime-only CUDA image has no nvcc, and the op needs JIT compilation.

Fix. Pre-build in a -devel stage with DS_BUILD_OPS=1, and ship the result. Check with ds_report.

Mixing Accelerate config keys with a DeepSpeed file

Symptom. ValueError: When using deepspeed_config_file, the following accelerate config variables will be ignored: [...].

Cause. You gave Accelerate both plugin keys and a config file.

Fix. Use one. If you keep the file, remove duplicate keys from the YAML, or set the file's values to "auto".

Practices that pay off most, in order of leverage

1. Pin the version and read the release notesDeepSpeed is 0.x, and 0.19.3 to 0.19.7 changed defaults and removed features. Pin the exact version in the lockfile, the image and CI, and read notes before each bump.
2. Do the memory arithmetic before you launch16 bytes per parameter for mixed-precision Adam, then divide by the stage. Run the estimators. Most failed first runs are sizing errors that arithmetic would have caught.
3. Choose the lowest stage that fitsStage 2 keeps the best throughput. Stage 3 costs about 1.5 times the communication, so earn it with a measurement.
4. Keep the config short and explicit where it mattersDelete keys that equal the default. Write clipping, precision and stage explicitly so that a default flip cannot change your run.
5. Pre-build ops in the imageA devel build stage, DS_BUILD_OPS=1, an explicit TORCH_CUDA_ARCH_LIST, then ds_report at build time. No node compiles at job start.
6. Checkpoint on a schedule and test the restoreAn untested checkpoint is a hope. Restore on a different GPU count through a Universal Checkpoint at least once before you need it.
7. Measure before tuningTurn on wall_clock_breakdown for a short run, then touch one knob at a time.
8. Per-rank logs from day one--enable_each_rank_log turns a hang from a mystery into a diff of two files.
9. Reproduce at the smallest scaleA two-GPU run on a tiny model finds ordering and config errors in seconds.

Habits for teams

Individual skill matters less than shared habits, because distributed jobs fail in ways that only the person who was there can remember. Four team habits earn their keep quickly.

The first is a run record. For every serious job, save the DeepSpeed version, the torch and CUDA versions, the ds_report output, the exact config, the launch command and the git commit next to the checkpoints. When a result cannot be reproduced six weeks later, this record is the difference between a ten-minute diff and a week of guessing. Because several defaults changed between 0.19.3 and 0.19.6, the version line alone often explains a discrepancy.

The second is a config review rule. A change to a DeepSpeed config is reviewed like code, and the reviewer asks one question per non-default key: what measurement justified it? Large round numbers copied from a blog post are the usual offenders. Deleting a key that equals its default is a welcome change, not a risky one.

The third is a restore drill. Once per project, restore from a checkpoint onto a different GPU count and a different machine, and time it. People discover during an outage that nobody has ever tried it. Use a Universal Checkpoint for the drill, since it is the path that survives a change of layout.

The fourth is a pinned upgrade lane. One person bumps the version on a branch, parses every config in CI, runs a short job against a stored baseline loss curve, and writes down which defaults moved. Everyone else stays pinned until that note exists. With a 0.x library that removes features in patch releases, this lane is cheap insurance. If your jobs run under Kubeflow or a similar operator, bake the pinned version into the image so that the lane and production cannot drift apart.

Practice cards

Reading the startup log

Most of what you need to know about a run is printed in its first minute, and reading it deliberately saves a lot of later confusion. On rank 0, look for the line DeepSpeed info: version=0.19.7, git-hash=..., git-branch=..., and confirm the version is the one you pinned. A different version means the image or environment is not what you think.

Next, find the resolved batch sizes. DeepSpeed prints the final train_batch_size, micro-batch and accumulation steps it settled on, and this is where you notice that a doubled GPU count doubled your global batch. Then look for warnings. Deprecation warnings such as the one for config_params, --deepscale_config or a ZeRO-3 elastic checkpoint are an early notice of what the next release will remove, so fix them while they are warnings. A warning that zero.Init fell back to a single rank means your process group was not what you expected.

Finally, check the optimizer line. If you see a CPU optimizer when you did not intend offload, or a client optimizer being used with ZeRO, the config and the code disagree about who owns the optimizer. Remember that an optimizer passed in code overrides the JSON block.

A short checklist

Before a long job:

• DeepSpeed, torch and CUDA versions are pinned and match the build.
• ds_report output was saved, and the ops you need show [YES] under installed.
• The config parses under the pinned version, with no leftover removed blocks.
• Batch keys specify two of three, and the global batch was decided on purpose.
• gradient_clipping and precision are written explicitly.
• If offload is on: ulimit -l was checked and pin_memory was chosen deliberately.
• Every rank calls save_checkpoint, and a restore was tested.
• Per-rank logs are enabled, and tokens come from the scheduler.
• --master_port is unique if the nodes are shared.

Operational recipes

Check an install

ds_report
python -c "import deepspeed; print(deepspeed.__version__); from deepspeed.accelerator import get_accelerator; print(get_accelerator()._name)"

Read the operator table: [YES] under installed means pre-built, [OKAY] means it can be JIT-compiled, and [NO] under compatible means this machine cannot build it, usually from a missing system library such as libaio-dev.

Pre-build ops for specific GPUs

TORCH_CUDA_ARCH_LIST="8.0;9.0" DS_BUILD_OPS=1 pip install deepspeed==0.19.7 --no-build-isolation

Install PyTorch first, because setup.py imports it. Use DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 instead if you only need a few ops.

Clear a stuck JIT build

Symptom: the job stops at Loading extension module ..., waiting for a lock. A killed build left a stale lock file.

export TORCH_EXTENSIONS_DIR=/scratch/$USER/torch-ext-$SLURM_JOB_ID   # per-job cache
rm -rf ~/.cache/torch_extensions/*/<op_name>                        # or clear the stale op directory

On shared filesystems, give every job its own extensions directory.

Estimate memory before provisioning

from deepspeed.runtime.zero.stage3 import estimate_zero3_model_states_mem_needs_all_live
estimate_zero3_model_states_mem_needs_all_live(model, num_gpus_per_node=8, num_nodes=2)

The output covers model states only, so treat it as a floor.

Launch on two nodes

worker-1 slots=8
worker-2 slots=8

deepspeed --hostfile=hostfile --num_nodes 2 --num_gpus 8 \
  --enable_each_rank_log logs/ \
  train.py --deepspeed --deepspeed_config ds.json

This needs passwordless SSH and pdsh. Without them, use the next recipe.

Launch without SSH (Kubernetes or a scheduler)

deepspeed --hostfile=hostfile --no_ssh --node_rank=$NODE_RANK \
  --master_addr=$MASTER_ADDR --master_port=29500 \
  train.py --deepspeed --deepspeed_config ds.json

Run the same command on every node, with the rank and the rank-0 address supplied by your orchestrator. torchrun --nproc_per_node 8 train.py --deepspeed_config ds.json is an equivalent route. See the Kubernetes guide.

Consolidate a checkpoint on a CPU machine

python save_dir/zero_to_fp32.py save_dir output_dir --safe_serialization --max_shard_size 5GB
# a specific tag:  -t global_step1000

Move a run to a different GPU count

python -m deepspeed.checkpoint.ds_to_universal \
  --input_folder ckpt/global_step1000 \
  --output_folder ckpt_universal/global_step1000

{ "checkpoint": { "load_universal": true } }

Then call engine.load_checkpoint(ckpt_universal_dir, tag="global_step1000"), with the tag explicit.

Profile a slow step

{
  "wall_clock_breakdown": true,
  "flops_profiler": { "enabled": true, "profile_step": 20, "module_depth": -1, "top_modules": 1, "detailed": true }
}

Run about 50 steps, read the forward, backward, allreduce and step times, then turn both off. Both perturb performance. For the network alone, run ds_bench.

Debug a hang

1. Open the per-rank logs and find the rank whose last line differs.
2. Set NCCL_DEBUG=INFO and, for the last collectives each rank issued, TORCH_NCCL_TRACE_BUFFER_SIZE.
3. Look for rank-conditional code that reaches a collective: save_checkpoint, a gather, or an evaluation loop that only some ranks run.
4. Check for uneven data, and a dataloader worker that died.
5. Only then consider DEEPSPEED_TIMEOUT.

Guard against removed features in CI

import glob, deepspeed
from deepspeed.runtime.config import DeepSpeedConfig

for path in glob.glob("configs/*.json"):
    DeepSpeedConfig(path)          # raises DeepSpeedConfigError on removed blocks
    print("ok", path)

Run this under the exact version you deploy. Treat it as a smoke test rather than a full validation, and run it with the same environment variables your launcher sets.

Practice

1. Size a real jobRun the stage 2 and stage 3 estimators for your model and GPU count, then launch both and compare the estimate with the peak memory you observe.
2. Remove the defaultsDelete every key from your config that equals its default, and confirm 100 steps of loss are unchanged.
3. Flip the clipping defaultRun the same short job with "gradient_clipping": 0.0 and 1.0, and compare the gradient norms you log.
4. Force the pinned-memory failureLower ulimit -l in a shell, enable optimizer offload, and read the error. Then set "pin_memory": false.
5. Round-trip a checkpointSave on 2 GPUs, consolidate on a CPU machine, then convert to a Universal Checkpoint and resume on 4 GPUs.
6. Pre-build a runtime imageBuild ops in a devel stage, copy the result to a runtime stage, and prove with ds_report that no op needs to compile at start. See the Docker guide.
7. Induce and diagnose a hangMake one rank skip a forward with a 2-minute timeout, and identify the culprit from per-rank logs alone.

When something is still wrong

Once these habits are automatic, the Senior guide takes the same tool to the platform level: composing tensor, pipeline and sequence parallelism, failure modes at scale, security, and when to choose something else.

This is part three of three. It takes the DeepSpeed you already use, a config file, a launcher and an engine, and treats it as something you own for other teams: the thing a dozen researchers and engineers depend on, whose upgrades you schedule and whose failures wake you up. The version used throughout is DeepSpeed 0.19.7, published on 2026-09-16.

Up to this point the questions were about making a training job fit and run. From here they change. What breaks first when the cluster doubles? What does a bad upgrade cost, and how do you notice it before your users do? Who is allowed to put what in a config? What does a checkpoint promise to the person who loads it a year from now?

Where this picks up

I am starting with the architecture, because every failure mode in this track is a property of how the pieces are connected.

Architecture and failure modes

Hold the whole system in your head as four layers, because an incident is almost always one layer lying to the layer above it. The first layer is the launcher: a single controller process (the deepspeed command, also installed as ds) that reads a hostfile, applies --include and --exclude, and turns the result into a description of the world. It then starts a per-node launcher on each machine, over pdsh and SSH by default or through MPI or SLURM, and that per-node launcher forks one process per accelerator. The second layer is rendezvous and communication: those processes find each other through MASTER_ADDR and MASTER_PORT (29500 by default), then talk through deepspeed.comm on top of the PyTorch backends, which means NCCL on NVIDIA, RCCL on AMD, oneCCL or gloo on CPU, HCCL on Ascend, and gloo on Apple's MPS. The third layer is the engine, the DeepSpeedEngine wrapping your module, which owns precision, gradient accumulation, clipping, ZeRO partitioning and the optimizer step. The fourth layer is the ops: C++, CUDA, HIP or Metal extensions such as fused Adam, CPU Adam and async I/O, built just in time unless you pre-built them.

Each layer has a characteristic way of failing, and learning those signatures is what lets you page the right person.

There is one more architectural fact that shapes everything you will decide: DeepSpeed has no scheduler, no tenancy model and no control plane. One job owns its ranks. There is no daemon you can query for "what is running", no admission control, no queue. Everything a platform needs beyond the library (placement, quotas, retries, identity) comes from the system you run it on, whether that is SLURM or Kubernetes. A senior owner's first job is to be honest about this: you are not operating DeepSpeed so much as operating a set of conventions around it.

Scaling: which component breaks first

"It scales" is not an answer a platform owner can give. The honest answer is a ranked list of what saturates first as you add GPUs, and the ranking changes with model size, stage and hardware.

Memory breaks first on a single node, and the stage is the lever. ZeRO stage 1 partitions optimizer states, stage 2 adds gradients, stage 3 adds parameters. Each step trades memory for communication. Under stage 3, parameters are all-gathered just in time for each module and released afterwards, so the communication volume is roughly 1.5 times that of plain data parallelism. That figure is the price of fitting a model that does not otherwise fit, and it is the number to keep in mind when someone asks why their stage 3 job is slower than a stage 2 job on the same hardware. If a model fits under stage 2, stage 3 is a loss.

Interconnect breaks first across nodes. Inside a node, NVLink or its equivalent absorbs the all-gathers. Between nodes you are on InfiniBand or RoCE, or on whatever your cloud provides, and ZeRO-3's gather traffic crosses it on every layer of every step. The practical guidance from the project is to keep tensor parallelism and sequence parallelism within a node and let ZeRO and data parallelism (and pipeline stages) span nodes. If your nodes are connected by ordinary cloud networking rather than a dedicated fabric, that single fact may decide your architecture, and it is worth measuring with the bundled ds_bench suite (all_reduce, all_gather, all_to_all and point-to-point) before you promise anyone a scaling curve.

ZeRO++ is the interconnect relief valve. For stage 3 it offers three independent features: quantized weight all-gather (zero_quantized_weights, called qwZ), a secondary per-node shard (zero_hpz_partition_size, called hpZ, which you set to the number of GPUs per node), and quantized gradient communication (zero_quantized_gradients, called qgZ). hpZ is the instructive one: it keeps a second copy of the parameters sharded within each node so that most gathers never leave it. That is a deliberate trade of GPU memory for inter-node bandwidth, which is exactly the sort of trade you want to make consciously on a platform rather than discover.

Host resources break first when you offload. CPU offload is bound by PCIe bandwidth and by the throughput of CPU Adam. NVMe offload is bound by the drive. Both draw on host-wide pools, pinned memory and disk, that every job on the node shares. Since 0.19.5, pin_memory defaults to true for both offload blocks, and pinned memory counts against the locked-memory limit (ulimit -l). A node that ran offload jobs happily on 0.19.4 can produce out-of-memory errors on 0.19.5 purely because of that default; the fix is a higher memlock limit (in containers, --ulimit memlock=-1) or an explicit "pin_memory": false. Tune NVMe offload with ds_nvme_tune --nvme_dir <path> and benchmark raw I/O with ds_io before you give anyone a number.

Composing parallelism without surprises

Large jobs rarely use ZeRO alone. They compose it with other forms of parallelism, and each composition has constraints that a platform owner needs to know before a user asks. The vocabulary: data parallelism replicates the model and splits the batch; tensor parallelism (TP) shards individual weight matrices across ranks; pipeline parallelism (PP) splits the layers into stages; sequence parallelism (SP) splits the sequence dimension; expert parallelism (EP) distributes the experts of a mixture-of-experts model.

AutoTP gives you tensor parallelism for training without rewriting the model. You set tensor_parallel.autotp_size, and DeepSpeed shards the weights of known architectures, using built-in presets (llama, bloom, chatglm, mixtral, deepseek_v2, qwen2, phi3) or a Hugging Face base_model_tp_plan when the model provides one. For models that need custom treatment you supply partition_config with layer specs, each naming regex patterns and a partition_type of row, column (the default) or skip. It composes with ZeRO stages 0 to 3. For a platform, the point is that TP is now a config decision rather than a model-code decision, which means you can set policy on it: for example, require that autotp_size divides the GPUs per node.

Pipeline parallelism is the odd one out, and the one that surprises people in review. You build the model as a PipelineModule of LayerSpec entries, you pass mpu=None to initialize, and you drive training with engine.train_batch(data_iter) rather than the forward, backward and step triple. Several newer features are explicitly incompatible with it: managed_gradient_accumulation: false and engine.coalesce_grad_reduction() do not work under pipeline parallelism, and Hugging Face Accelerate does not support DeepSpeed pipeline parallelism at all. If your platform's golden path is Accelerate or the Hugging Face Trainer, pipeline parallelism is off that path, and you should say so in your documentation rather than let users find out at launch time.

Mixture of experts has a hard constraint you will hit quickly: native deepspeed.moe.layer.MoE is not supported with ZeRO stage 3, and raises Native DeepSpeed MoE is not supported with ZeRO Stage 3.... The routes are ZeRO-2 with native MoE, or AutoEP (arrived in 0.19.2), which detects Hugging Face MoE layers automatically and supports ZeRO 0, 1 and 2 plus a constrained stage 3. An opt-in DeepEP transport for AutoEP arrived in 0.19.7.

The platform lesson is that the compatibility matrix is large and mostly implicit, enforced by assertions at runtime. A senior owner turns it into an explicit table, and ideally into a validator that runs before a job is admitted.

import json
import sys

def check(cfg: dict) -> list[str]:
    problems = []
    z = cfg.get("zero_optimization", {})
    stage = z.get("stage", 0)

    if cfg.get("fp16", {}).get("enabled") and cfg.get("bf16", {}).get("enabled"):
        problems.append("fp16 and bf16 are both enabled")
    if "amp" in cfg:
        problems.append("amp (Apex) is deprecated; use fp16, bf16 or torch_autocast")
    if cfg.get("gradient_allreduce_op") == "sum" and stage == 3:
        problems.append("gradient_allreduce_op='sum' is not supported with ZeRO stage 3")
    if "offload_param" in z and stage != 3:
        problems.append("offload_param requires ZeRO stage 3")
    off = z.get("offload_optimizer", {})
    if off.get("ratio", 1.0) < 1.0 and stage != 3:
        problems.append("partial optimizer offload (ratio < 1) requires ZeRO stage 3")
    if off.get("device") == "nvme" and "nvme_path" not in off:
        problems.append("nvme offload needs nvme_path")
    if "gradient_clipping" not in cfg:
        problems.append("gradient_clipping not set explicitly (default changed in 0.19.3)")
    for removed in ("nebula", "compression_training", "sparse_attention", "quantize_training"):
        if removed in cfg:
            problems.append(f"{removed} was removed in 0.19.7")
    if "mics_shard_size" in z:
        problems.append("MiCS was removed in 0.19.7; use ZeRO++ hpZ")
    return problems

if __name__ == "__main__":
    bad = check(json.load(open(sys.argv[1])))
    for p in bad:
        print("POLICY:", p)
    sys.exit(1 if bad else 0)

Checkpoints as a platform contract

A checkpoint is a promise: it says that whoever loads it later will get a usable model. A platform owner has to decide what that promise covers, because the default behaviour covers less than people assume.

By default, save_checkpoint writes one set of shards per rank, laid out as save_dir/<tag>/mp_rank_00_model_states.pt plus optimizer shards (zero_pp_rank_* under ZeRO), with a latest file naming the tag and a copy of zero_to_fp32.py beside them. Every rank must call it, not only rank 0. Those shards are tied to the parallel layout that wrote them: the data-parallel size, and any TP or PP degree. Resume on a different world size and you are off the happy path. The older answer was ZeRO elastic checkpointing, but for stage 3 it is deprecated as of 0.19.3 and logs ZeRO-3 elastic checkpointing is deprecated and no longer supported. Use Universal Checkpointing instead.

Universal Checkpointing (UCP) is the replacement, and it is what makes your fleet schedulable. You convert a ZeRO checkpoint into a parallelism-agnostic form, then load it on a different data-parallel, tensor-parallel, pipeline or world size:

python -m deepspeed.checkpoint.ds_to_universal \
  --input_folder ckpt/global_step1000 \
  --output_folder ckpt_universal/global_step1000

{ "checkpoint": { "load_universal": true } }

The load then goes through engine.load_checkpoint(ckpt_universal_dir, tag=...). Always pass the tag explicitly: with tag=None the loader looks for a latest_universal file, and if the conversion did not write one you get ValueError: Invalid for universal checkpoint: <dir>/latest_universal does not exist. The documentation spells the converter as a script path, python deepspeed/checkpoint/ds_to_universal.py; the module form above is what runs from an installed package. UCP support covers AutoTP as of 0.18.9. Verify the exact feature coverage for your parallel layout in the release notes of the version you pin before you promise conversion to anyone.

Three platform decisions follow.

Decide what is the archival format. The per-rank shards are a working format for resuming a job. The format you keep for a year should be one you can load without DeepSpeed's parallel layout. For ZeRO checkpoints, that is the consolidated 16-bit or fp32 weights, produced either on the live job with engine.save_16bit_model(...) (stage 3 needs stage3_gather_16bit_weights_on_model_save: true, and every rank calls it) or offline, with no GPU, using the script copied into the checkpoint directory:

python ckpt/zero_to_fp32.py ckpt out_dir --safe_serialization --max_shard_size 5GB

Note two details that old tutorials get wrong. The second positional argument is an output directory in 0.19.x, not a file name, and the script needs roughly twice the checkpoint size in CPU RAM, so run it on a machine sized for that, never on a login node.

Decide your retention and your storage tier. With fp32 Adam, optimizer state costs about 12 bytes per parameter on top of the weights, so a resumable checkpoint is several times the size of the model. Keep the last few resumable checkpoints hot, convert milestones to the archival format, and delete the rest on a schedule. When there is no shared filesystem, use_node_local_storage: true writes shards to each node's local disk, which is cheaper but ties recovery to those nodes surviving.

Decide what loading means for trust. This deserves its own section, and it is next.

Security and the trust model

DeepSpeed's security posture is that of a research library running inside a trusted cluster. A platform owner has to know where that assumption is load-bearing, because your users' laptops, notebooks and untrusted data are not inside it.

The launcher assumes a trusted network. The default pdsh launcher needs passwordless SSH from the controller to every node, which is a root-equivalent trust relationship between hosts. Rendezvous on MASTER_PORT (29500) is unauthenticated, and so are the NCCL ports. The rule is simple and absolute: the training network is not reachable from outside the job, and 29500 and the NCCL ports are never exposed publicly. On Kubernetes, that means a NetworkPolicy that allows traffic only between the pods of one job. On SLURM, it means the compute fabric is not routed to user networks.

Checkpoints are code. Checkpoints are pickle-based torch.save files, and unpickling untrusted data executes arbitrary code. Loading a checkpoint someone sent you is the same act as running a script someone sent you. The platform responses are to load only checkpoints from storage your jobs wrote, to prefer safetensors output (zero_to_fp32.py --safe_serialization) for anything that crosses a team boundary, and to treat a model hub download as untrusted input. The client_state you pass to save_checkpoint is pickled as well, so never put secrets in it.

Secrets travel further than you expect. The launcher exports every NCCL_* variable and the prefixes MLFLOW, PYTHON, MV2 and UCX (plus accelerator-specific ones) to the workers, and it also reads KEY=VALUE lines from a .deepspeed_env file in the current directory or the home directory (the filename is overridable with DS_ENV_FILE). Because the pdsh and MPI launchers pass these on the command line of remote processes, they can show up in process listings and logs on every node. Do not put API keys in .deepspeed_env. Put tokens for Weights & Biases, Comet or Hugging Face into the environment through your scheduler's secret mechanism, with Vault or Kubernetes Secrets behind it, and keep them out of the DeepSpeed config. The config documentation itself warns against saving the Comet API key in code.

The supply chain is mostly your build. PyPI ships DeepSpeed as a source distribution only, with no wheels, and its install requirements include ninja, pydantic>=2.0.0, torch>=2.0.0 and nvidia-ml-py on CUDA. A default install therefore compiles things on first use, on whichever machine runs the job, with whatever compiler is there. Move that into a controlled image build instead: install torch first, pre-build ops with DS_BUILD_OPS=1 against a pinned TORCH_CUDA_ARCH_LIST, pin the exact DeepSpeed version, and scan and sign the image like any other. The Docker guide covers the image side, and the Trivy guide covers scanning it. Recent project hardening is a useful signal (buffer-length checks in shm.cpp, subprocess replacing os.system, shell=True removed from ROCm detection, CI running pull-request code in a sandbox), but the project defers its threat model to PyTorch's security policy, and reports go privately through GitHub Security Advisories. Subscribe to those advisories for the repository, which now lives at github.com/deepspeedai/DeepSpeed.

Multi-tenancy on shared clusters

Because DeepSpeed has no tenancy features, multi-tenancy is something you assemble from the scheduler and from conventions. The failure modes are specific, so it helps to list what jobs on the same machine can collide on.

Ports. Two jobs on the same node with the default --master_port 29500 produce an "address already in use" failure. Give each job a distinct port, derived from something unique such as the scheduler's job ID, or give each job its own network namespace (which is what one pod per job does naturally).

GPUs. --include localhost:0,1 or CUDA_VISIBLE_DEVICES pins a job to specific devices. On Kubernetes, the GPU device plugin does that for you; on SLURM, the GPU allocation does. Never rely on users to pin devices by hand.

Compile caches. The JIT cache defaults to ~/.cache/torch_extensions. Shared home directories plus concurrent jobs plus killed builds equals stale lock files and hangs. A per-job TORCH_EXTENSIONS_DIR removes the problem, and pre-built ops remove the need.

Pinned memory and swap. Pinned memory is a host-wide, limited resource, and since 0.19.5 offload uses it by default. Two offload jobs on one node can exhaust it between them even though each fits alone. Decide a per-job memlock allocation as part of the job's resource request, not as something the library works out.

At the scheduler level, the established options are SLURM partitions and QOS, Kubernetes namespaces with resource quotas and the GPU device plugin, and a queueing layer such as Kueue for fair sharing. For Kubernetes the common launch patterns are the Kubeflow Training Operator (Kubeflow guide), using an MPIJob or PyTorchJob, or JobSet with --no_ssh --node_rank, torchrun rendezvous, or a launcher pod with SSH keys in the MPI-operator style. A headless Service provides a stable MASTER_ADDR. Whichever you choose, the no-SSH route is the better default for a platform, because it removes the SSH trust relationship from the previous section:

deepspeed --hostfile=hostfile --no_ssh --node_rank=$NODE_RANK \
  --master_addr=$MASTER_ADDR --master_port=29500 \
  train.py --deepspeed --deepspeed_config ds_config.json

Here each pod runs the same command with its own rank. torchrun --nproc_per_node N train.py --deepspeed_config ds.json also works, because DeepSpeed reads the torch.distributed environment variables, and so do accelerate launch and the Hugging Face Trainer. That flexibility is useful: your platform can standardize on one launcher without forbidding the others.

Cost: where the money actually goes

Training cost is GPU-hours multiplied by how well you used them. DeepSpeed offers several levers, each of which moves cost by trading one resource for another, and a platform owner should know the price of each trade.

Offload trades throughput for GPU memory. ZeRO-Offload and ZeRO-Infinity let a model fine-tune on fewer or smaller GPUs than it would otherwise need. That is cheaper per job when the GPUs are the scarce resource, but the step time rises, and a job that takes three times as long on a third of the GPUs is not cheaper once you count people waiting. Always compare cost per trained token, not cost per hour.

Low-precision master state trades a little speed for a lot of memory. With bf16.bf16_master_weights_and_grads and bf16.bf16_optimizer_states (the second requires the first, both require ZeRO stage 1, 2 or 3), plus torch_autocast, the project's own example measured allocated memory falling from 25.74 GB to 16.17 GB, about 35 to 40 percent, while the step time moved from 0.60 s to 0.64 s, roughly 7 percent slower. That is an excellent trade for memory-bound jobs, but it changes numerics, so it belongs behind an evaluation gate rather than a default.

Communication relief trades memory or accuracy for bandwidth. ZeRO++ matters most on cloud instances with modest inter-node networking. hpZ spends GPU memory, and the quantized options spend some numerical precision; validate both against your quality metrics.

Batch size and stage selection are the cheapest levers. The flops profiler (flops_profiler in the config) and the autotuner (deepspeed --autotuning tune|run train.py --deepspeed ds.json, installed with pip install deepspeed[autotuning]) help pick micro-batch size and stage. The autotuner runs real experiments, so budget for the GPU time it consumes and cache its results per model family.

Idle time is the largest hidden cost. The hangs described earlier, jobs that sit in a queue holding allocations, and nodes that spend ten minutes JIT-compiling at start all consume paid GPUs doing nothing. Measure the ratio of "allocated GPU-hours" to "GPU-hours with utilization above a threshold" per team, and put it on a dashboard in Grafana. It is usually the first number that changes behaviour.

Upgrades and migrations for a 0.x project

DeepSpeed is still a 0.x project. That is not a criticism; it is a policy you have to plan around, because under 0.x conventions minor and even patch releases can remove features, and 0.19.7 did exactly that. The release cadence is also brisk: 0.19.0 on 2026-05-06, then roughly one release every two to five weeks through 0.19.7 on 2026-09-16. A platform that follows every release will spend its time upgrading, and a platform that never upgrades will fall off supported PyTorch and CUDA versions.

Pin exactly, upgrade deliberately. Pin deepspeed==0.19.7 (or whichever version you validate), together with the torch and CUDA build, in your base images. Treat the triple as one unit.

Know the defaults that flipped, because they change behaviour without changing any config file:

The right response to a flipped default is to set the value explicitly in every config, which is exactly what the policy validator earlier does for gradient_clipping. An explicit value is immune to the next flip. The fact sheet notes one more wrinkle worth carrying: a constant in runtime/constants.py still says the bf16 overflow check is off, while the effective default comes from the pydantic model and is on, so reading the constants file will mislead you.

Read what is scheduled to go next. The tracking issue for feature removal (issue 8489 in the repository) lists what remains and is slated to leave: sparse_gradients, the LoCo-ZeRO++ parameter zeropp_loco_param, the 1-bit communication backends, the Mixture-of-Quantization and eigenvalue leftovers, Progressive Layer Dropping, and the Apex amp block, for which the issue says to use fp16, bf16 or torch_autocast instead. Search your fleet for these now and migrate on your schedule, not the project's. Teach users that the 1-bit optimizers OneBitAdam, OneBitLamb and ZeroOneAdam no longer exist, and that there is no in-tree replacement; use Adam or AdamW, or ZeRO++ quantized communication.

Run an upgrade as a procedure.

1. Read the release notes for every version between yours and the target, not only the last one.
2. Build the new image and diff ds_report output against the old one.
3. Run a config-parse smoke test: construct the engine from every production config on a small model, so removed keys fail fast and flipped defaults show up in logs.
4. Run a short convergence canary on a representative job and compare the loss curve and step time against the baseline.
5. Roll out by team, not by fleet, with the previous image kept for rollback.

Plan checkpoint migrations separately. Moving between versions, world sizes or parallel layouts is safest through Universal Checkpointing, so keep converted copies of any checkpoint you expect to outlive a version. Rebuild pre-compiled ops whenever torch or CUDA changes, or you will see the version-mismatch errors again.

Governance: policy for configs, images and exceptions

Governance sounds bureaucratic until the first time a config change costs a week of fleet time. The goal is a small number of rules, enforced by machines, with a cheap path to an exception.

Treat the ds_config as a reviewed artifact. It controls memory, precision, communication and checkpointing, which are the four things most likely to cost money or ruin a run. Keep configs in version control beside the code, review them like code, and generate team configs from a small set of vetted templates (a stage 2 default, a stage 3 default, an offload variant) with documented overrides. Hugging Face users can lean on "auto" values so the TrainingArguments stay the single source of truth for batch size, learning rate and precision, while the JSON holds only the DeepSpeed-specific decisions. That removes a whole class of the batch-size assertion error, train_batch_size is not equal to micro_batch_per_gpu * gradient_acc_step * world_size, which appears whenever someone changes the GPU count and forgets one of three numbers.

Make the golden path narrow and well-lit. A reasonable platform default is: bf16, ZeRO stage 2 unless the model does not fit, stage 3 with ZeRO++ only when measured to help, explicit gradient_clipping, an explicit process-group timeout set for the job's longest legitimate collective, TensorBoard or Weights & Biases monitoring on, and Universal Checkpointing for anything meant to outlive its job. Everything off that path is allowed but requires a design note.

Automate the checks. The validator above is the shape of it: a script that reads a config and returns violations, run in CI and again at admission. Keep the rule list short, tie each rule to an incident or a documented constraint, and print the reason with the failure so that nobody has to ask.

Define an exception process that does not rely on goodwill. Exceptions are normal, since research legitimately does odd things. Make them cheap (a one-paragraph note in the pull request), time-boxed (an expiry date), and visible (a list you review monthly). An exception list that only grows is a sign that your rules are wrong, not that your users are.

Incident playbooks

An incident playbook is a decision order, written when you are calm. The principle is to find the layer that is lying before touching anything. For each of the four situations below, the order of questions matters more than the answers.

The job hangs

1. Is every GPU idle, or are some busy? A mix means one rank is slow or stuck while others wait.
2. Which rank is late? Turn on per-rank logs with --enable_each_rank_log <dir>, set NCCL_DEBUG=INFO, and use the PyTorch flight recorder through TORCH_NCCL_TRACE_BUFFER_SIZE to find the collective that did not match. The rank named by the watchdog is often not the culprit.
3. What changed? An empty batch on one rank, a dataloader worker killed by the host OOM killer, a slow checkpoint write, a node with a degraded link.
4. Only after you understand the cause, decide whether the timeout is wrong. Raise it for operations that are legitimately long, with DEEPSPEED_TIMEOUT=30 or deepspeed.init_distributed(timeout=timedelta(minutes=30)). Raising it to hide a hang just makes the hang cost more.

Loss goes to NaN, or steps are skipped

Under fp16 the line OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, reducing to 32768 is dynamic loss scaling working as designed. A few at the start are normal. Skips that continue down to min_loss_scale are real instability. The ordered options are: switch to bf16, which needs no loss scaling; lower the learning rate; confirm gradient_clipping is what you think it is, which since 0.19.3 is 1.0 unless set; and only then adjust initial_scale_power or hysteresis. Since 0.19.3 the bf16 overflow check also skips steps with infinite or NaN gradients, so a bf16 job can now show skipped steps that it never showed before an upgrade, which is information, not a regression.

Separate host from device. A device OOM is a ZeRO stage, micro-batch or activation-checkpointing question. A host OOM after moving to 0.19.5 or later with offload is almost always pinned memory against ulimit -l, and the fix is a higher memlock limit or "pin_memory": false. Do not fix a host problem by shrinking the micro-batch.

The job cannot start

Read the first error, not the last. The common ones form a short list: a removed config block (DeepSpeedConfigError); a CUDA toolkit mismatch (CUDAMismatchException); no toolkit in the image (MissingCUDAException); an op that is incompatible on this host (ds_report shows [NO] under compatible, and async_io without libaio-dev is the classic case); a stale JIT lock; or a torch upgrade after ops were built. ds_report (also dsr, or python -m deepspeed.env_report) is the first command in every one of these, because it shows the installed and compatible state of each op, the ninja status, and the torch, CUDA and DeepSpeed versions in one screen.

Resuming from a checkpoint fails

If the world size or parallel layout changed, the checkpoint needs conversion to the universal format and a load with load_universal: true and an explicit tag. If you saved and loaded in the same process under ZeRO-3, reinitialize the engine. If state-dict tensors have shape=torch.Size([0]), the parameters are partitioned: gather them with GatheredParameters, save with save_16bit_model, or consolidate offline.

For every incident, finish with the same three steps: write the timeline, add the check that would have caught it to the validator or the smoke test, and update this playbook. An incident that does not leave behind a new automated check will come back.

Observability for a fleet

The engine gives you rich per-job signals and no fleet view. Your task is to join the two.

Per job, the built-in tools are: steps_per_print progress lines; wall_clock_breakdown for forward, backward, all-reduce and step timers; monitor backends for TensorBoard, Weights & Biases, Comet and CSV, which record fields such as Train/Samples/train_loss, Train/Samples/lr and, under fp16, Train/Samples/loss_scale; the flops_profiler; and the comms_logger, summarized with deepspeed.comm.log_summary(). The communications logger synchronizes every collective, so it perturbs the thing it measures; use it on a diagnostic run, not on the production fleet. NVTX ranges, including a DeepSpeed-specific domain since 0.19.0, let you read the run in Nsight. Verbosity is controlled by --log_level and -q on the launcher, and by the top-level log_level config key added in 0.19.3, which you should treat as optional because its default leaves the logger level unchanged.

There is no native Prometheus exporter. The fleet view therefore has three sources: GPU metrics from DCGM, scraped by Prometheus; training metrics pushed through Weights & Biases, TensorBoard or MLflow (the MLFLOW* environment variables are propagated to workers for exactly this reason); and scheduler metrics such as queue time and allocation. A small set of signals is enough to start:

• GPU utilization per job, with an alert when a Running job sits near zero (the hang detector).
• Step time per job against its own baseline, with an alert on a sustained increase (a degraded link or a noisy neighbour).
• Skipped steps and loss scale for fp16 jobs (instability).
• Checkpoint duration against interval (storage saturation).
• Allocated versus utilized GPU-hours per team (the cost signal).

Where DeepSpeed stops

A platform owner earns trust by saying when not to use the tool. DeepSpeed is strongest when a model does not fit in memory and you need to train or fine-tune it on PyTorch with the least code change. It is not the answer to everything.

When the model fits comfortably, plain data parallelism or PyTorch's own FSDP2 is simpler. FSDP2 is the closest native alternative to ZeRO-3, and the DeepSpeed project itself chose the 1.0 gradient-clipping default to match the FSDP2 reference. If your whole stack is PyTorch-native and you want fewer dependencies, FSDP2 is a legitimate choice, and the migration decision should come from a benchmark on your models, not from a preference.

When you need maximum throughput on very large models with heavy 3D parallelism, specialized stacks such as Megatron-Core or TorchTitan are built for it. AutoEP borrows TorchTitan grouped-GEMM kernels, which shows how interlinked the ecosystem is. Megatron-DeepSpeed, which combines the two, still exists. ColossalAI is another alternative with similar goals. I have not benchmarked these against each other for this guide, and any ranking would be marketing; measure on your hardware.

When you need a managed experience, a cloud training service, whether SageMaker, Vertex AI or Azure ML, removes the cluster operations. These services can run DeepSpeed inside their jobs, so the choice is not exclusive; it is a choice about who owns the nodes.

When the task is serving, not training, DeepSpeed's inference path is legacy. DeepSpeed-Inference v1 kernel injection (init_inference with replace_with_kernel_inject) is legacy, and the project directs serving to DeepSpeed-MII and FastGen. Most teams today choose a dedicated server such as vLLM for production LLM serving; DeepSpeed remains useful for the training side and for loading checkpoints that came from it.

When you need a feature that was removed. If a team's method depends on 1-bit Adam, the compression library or Sparse Attention, DeepSpeed 0.19.7 and later is not the tool for it, and an older version is a dead end you should avoid pinning your platform to.

Running it as a platform for other teams

Everything above converges here. A platform is a product with users, and its quality shows in how quickly a new team becomes productive and how rarely an old team is surprised.

Offer a paved road with three tiers of support. The paved road is the supported image, the vetted config templates, the supported launch method and the monitoring defaults. A second tier is "supported with a design note": a custom parallelism layout, offload to NVMe, pipeline parallelism. A third tier is "you own it": experimental features, forks, anything off the matrix. Say plainly which is which, so a team that picks tier three knows that nobody will page for it.

Ship an intake checklist that a team completes before it gets cluster time: model size and the memory estimate; the intended stage, precision and parallelism; the checkpoint plan (format, interval, retention, whether it must outlive the version); the resume plan (will the world size change); data location and any residency constraint; and the person who owns the job. Ten questions answered up front are cheaper than one failed week.

Provide a smoke test as a service. A short job that every new image and every new config must pass: it constructs the engine from the config, runs a few steps on two GPUs, saves a checkpoint, converts it, and loads it. When an upgrade breaks something, this is where you want to find out.

Own the base images. Publish them for each supported (DeepSpeed, torch, CUDA) triple, with ops pre-built for your GPU architectures, libaio-dev present when NVMe offload is supported, and scanned and signed. Rebuild them on a schedule and when a security advisory lands. Teams that bring their own images get no support for ops problems, which keeps the matrix finite.

Run an on-call that matches the failure modes. The playbooks above should be the on-call's first reference, and the first questions should be the ones that find the lying layer. Give on-call the authority to roll back an image, because rolling back is the fastest remedy for upgrade-related incidents.

Communicate changes the way a product does. Keep a changelog for the platform, announce deprecations with dates, and give each team a migration owner on your side. When 0.19.7 removed five features, the right behaviour was a notice weeks earlier, a list of affected configs per team, and an offer of help with each migration. The Terraform guide and Argo CD guide cover the infrastructure-as-code and delivery machinery that makes this repeatable, and the Helm guide covers packaging the job templates.

The review checklist

Use this for pull requests that touch a training config, a launcher script or a platform image, and for design documents that propose a new training setup. Every line is a question with a reason behind it.

Configuration

• Is gradient_clipping set explicitly? Its default changed in 0.19.3.
• Is the precision mode explicit, with only one of fp16, bf16 or torch_autocast, and no amp block?
• Are two of the three batch keys set, or are they "auto" under Hugging Face?
• Is bf16.check_grad_overflow understood, rather than inherited?
• If offload is used, is pin_memory set deliberately, and does the host's memlock limit match?
• Does the config contain any key on the removal list: nebula, compression_training, sparse_attention, mics_shard_size, sparse_gradients, progressive_layer_drop, amp, ZeRO-3 elastic_checkpoint?
• Does the validator pass, with exceptions recorded?

Parallelism and memory

• Is the stage justified by an estimate, not by habit? If stage 3, is the communication cost acceptable on this interconnect?
• Are TP and SP degrees within a node, and does each divide the GPU count?
• If MoE: is it ZeRO-2 with native MoE or AutoEP, not native MoE with ZeRO-3?
• If pipeline parallelism: is the launch path compatible (not Accelerate), and are mpu and the newer accumulation features avoided?

Checkpoints

• What is the format that outlives the job, and where is it written?
• Is there a tested conversion path (UCP, or zero_to_fp32.py with --safe_serialization) and has it been run on this model?
• Does the resume plan survive a different world size?
• Does client_state contain only plain data and no secrets?
• Is the checkpoint source trusted, since loading is code execution?

Launch and environment

• Is the image pinned to an exact (DeepSpeed, torch, CUDA) triple with ops pre-built, and scanned?
• Does the job set its own port, extensions directory and NVMe path?
• Is the process-group timeout set consciously?
• Are secrets absent from .deepspeed_env, the config and client_state?
• Is the training network isolated, with 29500 and NCCL ports unreachable from outside?

Operations

• Are the monitor backends on, and is the job identifier present in every system?
• Does the job have an owner and a rollback plan?
• Is there a smoke test, and did it pass on this exact image?

The complete picture

Put the pieces together and the shape of the senior job is visible. DeepSpeed is a library that wraps a model in an engine, driven by one config, started by a launcher that trusts its network. It is powerful because ZeRO, offload, ZeRO++ and the parallelism features let one codebase span a laptop and a cluster, and it is demanding because the same power creates a large space of combinations, each with a cost and a constraint, enforced mostly by assertions at runtime.

As an owner you convert that space into a product. You narrow it with a paved road and vetted templates. You make it safe with an isolated network, no-SSH launching where you can, secrets kept out of environment files, and checkpoints treated as code. You make it cheap with the right stage, measured offload, and relentless attention to idle GPUs. You make it durable with Universal Checkpointing and an archival format. You make it survivable with an upgrade procedure for a 0.x project, explicit values wherever a default can flip, and a smoke test every release must pass. You make it understandable with playbooks that start by finding the lying layer, and with a fleet view that joins GPU metrics, training metrics and scheduler data. And you stay honest about where it stops, in favour of FSDP2, a specialized stack, a managed service, or a dedicated server for inference.

The details will change: this guide describes 0.19.7, and the next release will move something. The habits will not. Pin, validate, canary, roll back, and write down what you learn.

Where the series leaves you

You started this series able to run a training script and you finish able to argue for how an organization should train models. The beginner level gave you the vocabulary and a first working job. The mid level gave you the machinery well enough to predict its behaviour. This level gave you the responsibilities that come with owning it.

Where to go next depends on what you now carry. If you will run this on Kubernetes, continue with the Kubernetes guide, then Kubeflow for job operators. If you work with the Hugging Face stack, the Hugging Face guide and PEFT and TRL cover the layers above DeepSpeed. If your concern is moving from training to production, the vLLM guide covers serving, and the Ray guide covers an orchestration layer that integrates with DeepSpeed through its actor model. If experiment tracking is your gap, start with MLflow or Weights & Biases.

Finally, a request. The project moves quickly and its documentation lags its code; this guide had to resolve places where the two disagree, such as the ZeRO prefetch bucket default, which the docs table gives as 5e8 while the code says 5e7. When your own measurements disagree with a page, trust the code at your pinned version, and when you find the discrepancy, report it. That small habit, repeated across a community, is how a fast-moving tool stays teachable.

Sources

• DeepSpeed documentation
• Getting started
• Advanced install
• Configuration JSON reference
• ZeRO tutorial
• ZeRO-Offload tutorial
• ZeRO++ tutorial
• Universal Checkpointing tutorial
• AutoTP training tutorial
• Pipeline parallelism tutorial
• Monitor tutorial
• FLOPS profiler tutorial
• Communication logging tutorial
• Autotuning tutorial
• Inference tutorial
• API reference (Read the Docs)
• Model checkpointing API
• DeepSpeed repository
• Release notes
• Release v0.19.7
• Feature removal tracking issue 8489
• Security policy
• DeepSpeed on PyPI
• Hugging Face Transformers DeepSpeed integration
• Hugging Face Accelerate DeepSpeed guide

This is the senior interview file for DeepSpeed 0.19.7. At this level nobody asks you what ZeRO stage 2 partitions. They give you a messy situation and watch how you reason: what you ask first, what you measure, what you refuse to promise, and how you explain a trade-off to someone who does not care about optimizer states. Every answer below is written the way a strong candidate would say it aloud, with the reasoning visible.

The thirty-second answer

"DeepSpeed is a PyTorch library and launcher that wraps a model in an engine driven by one JSON config. Its core idea, ZeRO, shards optimizer state, gradients and eventually parameters across data-parallel ranks, so a model that does not fit on one GPU can still train with ordinary data-parallel code. Around that it offers offload to CPU and NVMe, communication optimizations, tensor, pipeline and sequence parallelism, and a checkpoint format that survives changes in world size. As a platform owner I think of it as a set of conventions, not a service: it has no scheduler, no tenancy and no control plane, so everything operational comes from the system underneath it, and everything risky comes from its defaults changing between releases."

That answer shows scope, mechanism and the ownership stance in about a breath. Practise saying it without hedging.

How interviewers score senior answers

Strong answers share four habits, and weak answers lack them.

• They ask before they answer. "How big is the model, what is the interconnect, is this training or serving, and who loads the checkpoint?" Four questions change the design, and a candidate who skips them is guessing.
• They name the number that decides. Not "ZeRO-3 is slower" but "stage 3 moves roughly 1.5 times the communication volume of plain data parallelism, so it only wins when the model does not fit under stage 2."
• They separate what they know from what they would measure. "I would not promise a scaling curve; I would run ds_bench within and across nodes first."
• They finish with an operational plan. Alerts, rollback, ownership, a smoke test. A design with no on-call story is half a design.

System design: a shared fine-tuning platform

Prompt: "Design a platform where fifty researchers fine-tune open models from 7B to 70B parameters on a shared GPU cluster. You own DeepSpeed."

Start by asking the questions above, then state assumptions out loud: say, nodes with eight GPUs each, a fast fabric between nodes, a shared filesystem, and Kubernetes or SLURM underneath. Then walk through layers.

Placement and launch. DeepSpeed does not schedule, so the scheduler must. On Kubernetes I would use a job operator or JobSet with one pod per node, and launch with --no_ssh --node_rank rather than the default pdsh, because that removes the passwordless-SSH trust relationship between hosts. Each job gets its own namespace-level quota, a unique --master_port or its own pod network, and a headless Service for MASTER_ADDR.

Defaults, the paved road. bf16, ZeRO stage 2 unless the model does not fit, stage 3 for the 70B class, with hpZ from ZeRO++ only after measuring that it helps on this fabric. For a 7B model on eight 80 GB GPUs, stage 2 is usually the right start; for 70B, stage 3 with offload is a conversation about memory arithmetic, which I would settle with the estimator functions in deepspeed.runtime.zero.stage3 before provisioning anything. I would let researchers use Hugging Face Trainer or Accelerate with "auto" values, so the training arguments stay the single source of truth.

Config governance. Vetted templates, a validator in CI and at admission that rejects incompatible combinations and removed keys, and explicit values for anything whose default has flipped, notably gradient_clipping and the pin_memory offload keys.

Images. One base image per supported (DeepSpeed, torch, CUDA) triple, ops pre-built for our GPU architectures with DS_BUILD_OPS=1 and TORCH_CUDA_ARCH_LIST, scanned and signed. No JIT compilation in production.

Checkpoints. Resumable checkpoints on the shared filesystem with a retention schedule, plus conversion of milestones to a portable format (zero_to_fp32.py --safe_serialization), plus Universal Checkpointing for anything that must resume on a different world size.

Observability. GPU metrics from DCGM into Prometheus, training metrics into Weights & Biases or MLflow, and a hang alert: a Running job with GPU utilization near zero for ten minutes.

Failure story. A smoke test that every new image passes, a rollback to the previous image, and playbooks for hangs, NaNs, host OOM and resume failures.

What a weak answer misses: security and checkpoint trust, the support matrix, and anything about who gets paged.

Follow-up you should expect: "Why not let each team bring its own image?" Because the compatibility matrix between DeepSpeed, torch, CUDA and prebuilt ops is where most startup failures live, and every extra triple multiplies your support surface. I would allow it as an unsupported tier with no on-call, and make the paved road faster than the alternative.

System design: a 70B model that does not fit

Prompt: "A team's 70B fine-tune runs out of GPU memory on one node of eight 80 GB GPUs. Walk me through your options in order."

Work through the levers from cheapest to most expensive, and say what each costs.

1. Confirm what is using memory. Is it states, activations or fragmentation? see_memory_usage and the memory breakdown separate them. Activation memory is solved by activation checkpointing, not by a different stage.
2. Move to stage 3. Parameters, gradients and optimizer state are all partitioned. Build the model under deepspeed.zero.Init so it never fully materializes on one GPU, or, with Hugging Face, create the HfDeepSpeedConfig or TrainingArguments before from_pretrained.
3. Reduce state size. Low-precision master state with bf16_master_weights_and_grads and bf16_optimizer_states plus torch_autocast. The project's own measurement was about 35 to 40 percent less memory for about 7 percent slower steps. I would gate it behind an evaluation, because it changes numerics.
4. Offload. Optimizer state to CPU, then parameters, then NVMe. Each step down costs throughput, and since 0.19.5 pinned memory is the default, so I would check the memlock limit up front.
5. Add a node. Often cheaper in wall-clock than offload, but it makes the interconnect the new bottleneck, which is where ZeRO++ enters.
6. Reconsider the method. A parameter-efficient method such as LoRA may remove the problem. That is a product decision, not a DeepSpeed setting.

What a strong answer adds: "At each step I would record step time and peak memory, and I would compare cost per trained token, not cost per hour. Offload that makes the job three times slower on a third of the GPUs is not a saving."

System design: moving a team off a removed feature

Prompt: "A team's pipeline uses 1-bit Adam and the compression library. You are upgrading the fleet to 0.19.7. What do you do?"

First, establish facts. In 0.19.7 the 1-bit optimizers (OneBitAdam, OneBitLamb, ZeroOneAdam), the compression library, Nebula checkpointing, Sparse Attention and MiCS were removed, and leftover config blocks now raise DeepSpeedConfigError at parse time. So the team's job will refuse to start after the upgrade; it will not silently degrade. That is good for safety and bad for schedule.

Second, say what the replacements are, and be honest where there are none. 1-bit Adam has no in-tree replacement: use Adam or AdamW, or ZeRO++ quantized communication if the goal was reducing bandwidth. The compression library and Mixture-of-Quantization have no direct replacement; an external quantization toolkit is the route. For MiCS, ZeRO-3 with zero_hpz_partition_size is the nearest option. These change behaviour, so each needs an evaluation run, not a find-and-replace.

Third, address history. Before 0.19.7 those blocks were ignored in some cases, so past results may not have used the features the config suggested. I would ask the team to confirm which results they care about, and whether they were produced with the feature active.

Fourth, the schedule. Do not upgrade the fleet around this team. Keep their image pinned to the last compatible version, with a dated plan to migrate, and tell them the old line will not be supported forever. Meanwhile upgrade everyone else.

What a weak answer misses: the silent-ignore history, and the difference between "feature removed" and "feature deprecated but still present", which is the next question.

Incident questions

"A 64-GPU job has shown no log output for twelve minutes and GPU utilization is near zero. What do you do?"

Say the order. A hang, not a crash. First, look at which ranks are alive: per-rank logs with --enable_each_rank_log, and NCCL_DEBUG=INFO. Second, find the rank that is not in the same collective as the others; the watchdog's named rank is usually a victim. Third, ask what changed: a dataloader worker killed by the host OOM killer, an empty batch on one rank, a slow checkpoint write, a degraded link on one node. Fourth, only then consider the timeout. The default has been 10 minutes since 0.19.6, and DEEPSPEED_TIMEOUT or init_distributed(timeout=...) raises it, but raising it to hide a hang just makes the hang cost more. Finally, kill the job, replace the suspect node, and add the missing alert so the next one is caught in ten minutes, not twelve minutes plus however long it took someone to look.

"After a routine upgrade from 0.19.4 to 0.19.7, three teams report out-of-memory errors on offload jobs. Cause?"

Pinned memory. Since 0.19.5, pin_memory defaults to true in both offload blocks, and pinned memory counts against the locked-memory limit (ulimit -l). Hosts or containers with a low limit now fail where they used to work. The fix is a higher limit (in containers, --ulimit memlock=-1) or an explicit "pin_memory": false. The better answer adds the lesson: the upgrade procedure skipped reading the intermediate release notes, and the smoke test did not include an offload config on a production-like host. I would add both.

"Training loss is fine but the team says the final model is worse than last quarter's."

I would look for silent default changes. Between releases, gradient_clipping went from off to 1.0 in 0.19.3, and the bf16 gradient-overflow check went from off to on. Either could change a run's trajectory without any config edit. Compare effective configs from the two runs, check the version of each, and rerun the old config with explicit values to isolate the cause. The principle I would state to the room: explicit beats inherited, and I would put explicit values in the template.

"A researcher shares a checkpoint from an external collaborator and asks to load it on the cluster."

Loading a pickle-based checkpoint is arbitrary code execution, so I would not load it on the cluster directly. Options: ask for safetensors weights, load in an isolated sandbox with no credentials and no network, or inspect the file with a restricted unpickler. I would also remind them that client_state is pickled, and that our own checkpoints should never contain secrets, since they will eventually travel.

"Two jobs on the same node both fail at start-up with 'address already in use'."

Both used the default --master_port of 29500. Fix: a unique port per job, derived from the scheduler's job identifier, or one network namespace per job. A related check is the JIT extension cache and any NVMe offload path, since the same collision logic applies to them.

"Resume on a different number of GPUs fails."

The shards are tied to the layout that wrote them. Convert the checkpoint with python -m deepspeed.checkpoint.ds_to_universal --input_folder ... --output_folder ..., set "checkpoint": {"load_universal": true}, and pass the tag explicitly to load_checkpoint, since with no tag the loader needs a latest_universal file. For ZeRO-3, do not rely on elastic checkpoints, which are deprecated.

Trade-off questions

"ZeRO-3 or FSDP2?"

It depends on the stack, and I would not claim a winner without a benchmark. FSDP2 is PyTorch-native and has fewer dependencies; DeepSpeed offers a broader feature set around ZeRO, including offload to NVMe, ZeRO++, Universal Checkpointing and AutoTP. If the model fits comfortably and the team is PyTorch-native, FSDP2 may be simpler. If we need NVMe offload or the checkpoint portability, DeepSpeed earns its place. Either way, the decision should be on measured throughput and operational fit, not preference.

"Stage 2 with offload, or stage 3 without?"

Stage 3 moves more bytes over the interconnect; stage 2 with optimizer offload moves bytes over PCIe and spends CPU time in CPU Adam. On a fast fabric with limited host memory, stage 3 tends to be attractive; on a slow fabric with a strong host, offload may win. I would measure both for the specific model and record cost per token.

"Is it safe to run on the newest release?"

DeepSpeed is 0.x, and 0.19.7 removed features in a patch release, so "newest" is not a safety property. I pin an exact version and upgrade deliberately: read all intermediate notes, build the image, diff ds_report, run a parse smoke test and a convergence canary, then roll out team by team with a rollback.

"Why not just use a managed training service?"

Managed services remove cluster operations, and they can run DeepSpeed inside their jobs, so it is not either-or. The question is who owns node health, images and networking. In the Gulf and Egypt, data-residency rules and regional capacity may decide it before technology does.

Arguing a trade-off to leadership

Leadership wants a decision, a cost and a risk, in that order, in their language. Here are three worked examples.

Spending engineering time on a config validator. "Last quarter we lost roughly N GPU-hours to jobs that failed in the first ten minutes from configuration errors, and N more to an upgrade that flipped a default. A validator that costs two weeks to build removes most of both, and it is the control that lets us upgrade faster. Without it, upgrades are a risk we take on trust." Fill the numbers from your own dashboards; an unmeasured claim is the first thing that gets challenged.

Freezing on a version. "Pinning is the right default, but a frozen version is a debt, because 0.19.7 removed features and more are scheduled. I propose a quarterly upgrade window with a canary, so the debt never grows beyond one quarter."

Declining a feature request. A team asks for a feature that is on the removal list. "We can support it for one more quarter, with a dated migration plan. After that, the maintenance cost lands on us, not on the project, and we would be carrying it alone."

The pattern: lead with the decision, quantify the cost and the risk, offer the cheaper alternative, and name the date.

What a strong answer covers that a weak one misses

• Strong: names the interconnect and the measurement. Weak: says "it scales."
• Strong: separates host memory from device memory. Weak: shrinks the batch for every OOM.
• Strong: treats the launcher and checkpoints as security surfaces. Weak: assumes a private cluster is a safe cluster.
• Strong: pins, canaries and keeps a rollback. Weak: upgrades to latest.
• Strong: says when not to use DeepSpeed. Weak: treats it as the answer to everything.
• Strong: ends with who is paged and how they know. Weak: ends with the config.

Sixty-second self-test

Answer aloud, then check against the guide.

1. What is the default process-group timeout in 0.19.7, and which environment variable changes it?
2. Which two defaults flipped in 0.19.3, and what do they affect?
3. Why can an offload job that worked on 0.19.4 fail on 0.19.5?
4. Name three things removed in 0.19.7 and the failure a leftover config now produces.
5. Why is native MoE incompatible with ZeRO-3, and what are the two ways round it?
6. What two details of zero_to_fp32.py do old tutorials get wrong?
7. Name two reasons never to put credentials in .deepspeed_env.
8. What is the first command you run on a job that will not start?
9. Which parallelism forms should stay within a node, and why?
10. When would you tell a team not to use DeepSpeed?

If you hesitated on more than three, reread the guide's sections on upgrades, checkpoints and incidents before you interview.

This is the senior tips file for DeepSpeed 0.19.7. The earlier levels listed mistakes a person makes with the tool. The mistakes here are different: they only appear at scale, across many jobs and many releases, and they are usually the platform's mistakes rather than a user's. Each trap gives the symptom, the cause and the fix, then the habits that prevent it, then checklists to paste into your review templates.

Production traps that only appear at scale

The silent default flip

Symptom. After a routine upgrade, runs behave differently: loss curves shift, some steps are skipped, a final metric drifts. No config file changed.

Cause. DeepSpeed is a 0.x project and changes defaults between releases. In 0.19.3 gradient_clipping moved from 0.0 (disabled) to 1.0, and bf16.check_grad_overflow moved from false to true, so bf16 steps with infinite or NaN gradients are now skipped. In 0.19.5 pin_memory became true for both offload blocks. In 0.19.6 the process-group timeout fell from 30 to 10 minutes.

Fix. Set every one of these explicitly in your config templates, so the next flip cannot reach you. A trap inside the trap: a constant in the source tree still says the bf16 overflow check is off, while the effective default is on, so reading constants.py will mislead you. Trust the behaviour you observe, at your pinned version.

The hang that bills like a run

Symptom. A job shows Running, logs nothing for many minutes, and the GPUs sit idle until a watchdog error appears.

Cause. One rank is out of step with the others, and the rest wait inside a collective. Common sources: a dataloader worker killed by the host, a batch that is empty on one rank, a slow checkpoint write, one node with a degraded link.

Fix. Alert on near-zero GPU utilization for a Running job, since a hung job often logs nothing. Use --enable_each_rank_log <dir> and NCCL_DEBUG=INFO to locate the late rank. Do not simply raise DEEPSPEED_TIMEOUT; raise it only for operations that are legitimately long.

The offload job that dies on a different host

Symptom. An offload config that ran on one cluster runs out of host memory on another, or after an upgrade.

Cause. Pinned memory counts against the locked-memory limit, ulimit -l, and the limit differs by host and by container runtime. Since 0.19.5 pinning is on by default.

Fix. Raise the limit (in containers, --ulimit memlock=-1) or set "pin_memory": false knowingly. Put the limit in the job's resource request so the scheduler knows about it, and never respond to a host OOM by shrinking the micro-batch.

JIT compilation in production

Symptom. First jobs on new nodes take many minutes to start. Sometimes a job hangs at Loading extension module ... or Waiting for a lock.... Sometimes a job fails with a toolkit mismatch.

Cause. PyPI ships DeepSpeed as a source distribution only, and ops are compiled just in time on first use. A hundred nodes sharing a cache directory will collide on stale locks, and a runtime-only container has no nvcc at all (MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s)).

Fix. Build ops into your image: install torch first, then DS_BUILD_OPS=1 pip install deepspeed with TORCH_CUDA_ARCH_LIST set to the architectures you own, using a -devel CUDA image for the build stage. Give each job its own TORCH_EXTENSIONS_DIR as a belt-and-braces measure. After any torch or CUDA bump, rebuild, or you will meet PyTorch version mismatch! DeepSpeed ops were compiled and installed with a different version than what is being used at runtime.

Wrong architecture in the pre-built ops

Symptom. CUDA error: no kernel image is available for execution on the device.

Cause. The ops were built for different GPU architectures than the node has, typically because the image was built on a build host with a different GPU, or on one with none.

Fix. Set TORCH_CUDA_ARCH_LIST explicitly (for example "8.0;9.0") when building, and verify with ds_report on a real node of each type in the fleet. If you build on a machine without a GPU, DS_IGNORE_CUDA_DETECTION skips local GPU detection for cross-building.

The save that stalls the world

Symptom. Training pauses at every checkpoint interval and sometimes dies with a watchdog timeout during a save.

Cause. Every rank writes its own shards. At large world sizes that is a write burst the filesystem may not absorb, and a rank stuck in a slow write is a rank missing from the next collective.

Fix. Measure save time against the interval and the timeout. Reduce frequency, use node-local storage where there is no shared filesystem (checkpoint.use_node_local_storage: true), and consider asynchronous checkpointing. Keep the timeout longer than your worst save, and never shorten the interval without checking the storage.

Checkpoints that cannot be resumed on a different size

Symptom. A job is preempted, comes back with fewer GPUs, and cannot load its state.

Cause. Per-rank shards are tied to the parallel layout that wrote them, and ZeRO-3 elastic checkpoints are deprecated as of 0.19.3.

Fix. Convert with python -m deepspeed.checkpoint.ds_to_universal --input_folder <in> --output_folder <out>, set "checkpoint": {"load_universal": true}, and always pass the tag to load_checkpoint; without it the loader wants a latest_universal file. Rehearse the conversion on every model family before you need it in anger.

Loading right after saving under ZeRO-3

Symptom. A test that saves and immediately reloads works at stage 2 and fails, or loads wrong weights, at stage 3.

Cause. The load_checkpoint docstring is explicit: under ZeRO-3 you cannot load right after saving in the same engine. Reinitialize first.

Fix. Make your smoke test cover stage 3 and reinitialize the engine between save and load.

Credentials in the launch environment

Symptom. An API token shows up in a process listing or a log on a worker node.

Cause. The launcher exports NCCL_* variables and the prefixes MLFLOW, PYTHON, MV2 and UCX, and reads KEY=VALUE lines from .deepspeed_env in the current or home directory. Those can appear on remote command lines.

Fix. Keep secrets out of .deepspeed_env and out of the config, and inject them through your scheduler's secret mechanism. Never place secrets in client_state, which is pickled into the checkpoint.

Trusting a checkpoint

Symptom. Nothing, until it is too late.

Cause. Checkpoints are pickle-based torch.save files. Loading one runs whatever code it contains.

Fix. Load only checkpoints your own jobs wrote. For anything that crosses a team or organization boundary, use safetensors output (zero_to_fp32.py --safe_serialization) and treat third-party files as untrusted input, to be opened in an isolated environment.

Port and path collisions between tenants

Symptom. address already in use, or two offload jobs corrupting each other's swap files, or a stale lock.

Cause. DeepSpeed has no tenancy. Defaults like --master_port 29500, the shared extension cache and a common nvme_path are global to the host.

Fix. Template the port, TORCH_EXTENSIONS_DIR and NVMe path from the job identity, and prefer one pod or one allocation per job.

Features on the removal list

Symptom. After an upgrade a job refuses to start with DeepSpeedConfigError: DeepSpeed Sparse Attention has been removed; ... or the equivalent message for Nebula, the compression library, Mixture-of-Quantization or MiCS.

Cause. 0.19.7 removed those features and made leftover config blocks fail at parse time.

Fix. Delete the block, then evaluate the replacement: ZeRO++ hpZ for MiCS, external toolkits for compression, standard or node-local checkpointing for Nebula. Search your fleet now for what is still present but scheduled to go: amp, progressive_layer_drop, sparse_gradients, and the LoCo-ZeRO++ parameter.

Platform-ownership habits

Pin exactly. Pin DeepSpeed, torch and CUDA as one unit, in an image, with a version-controlled record of why. A range specifier on a 0.x project is a decision to be surprised.

Read every intermediate release note. Going from 0.19.4 to 0.19.7 means reading 0.19.5, 0.19.6 and 0.19.7, because each hides a default flip or removal. Make this a step in the upgrade ticket, not a hope.

Upgrade on a calendar, with a canary. A quarterly window is a reasonable start. The procedure: build the image, diff ds_report, run a config-parse smoke test over every production config, run a short convergence canary against a baseline, then roll out team by team with the previous image kept.

Treat the config as code. Version it, review it, generate it from templates, and run a validator on it. Use "auto" with Hugging Face so training arguments are the single source of truth.

Write rules from incidents. Each validator rule should point to an incident or a documented constraint, so people trust it and so you can retire it when the constraint disappears.

Make the paved road the fast road. A supported image, a template and a launcher wrapper that gets a team to a working multi-node run in an afternoon will out-compete any policy document.

Keep a support matrix and publish it. DeepSpeed, torch, CUDA or ROCm, Python (3.10 to 3.12 is the recommendation here, though PyPI declares no minimum), accelerators you can actually help with. What is not on the matrix is unsupported, and saying so kindly prevents anger later.

Own the smoke test. It constructs the engine from a config, runs a few steps on two GPUs, saves, converts and loads. Run it for every image and every template change.

Measure the platform, not just the jobs. Time to first successful multi-node run for a new team, fraction of jobs failing in the first ten minutes, fleet GPU utilization, allocated versus utilized GPU-hours, and open exceptions.

Attend the project's office hours. They run on the last Tuesday of each month at 12:00 America/New_York. The cheapest way to learn about a removal is to hear it before it ships.

Document where DeepSpeed stops. FSDP2 for PyTorch-native simplicity, a specialized stack for the largest 3D-parallel jobs, a managed service when you do not want to own nodes, and a dedicated server for inference. A platform that knows its limits earns more trust than one that claims none.

Be kind to researchers. They are not the problem. A fast, friendly answer in a shared channel prevents ten workarounds.

Review checklist for pull requests

Paste this into the template for any change that touches a training config, launcher script or image.

Config

• gradient_clipping is set explicitly.
• Exactly one of fp16, bf16, torch_autocast is enabled, and there is no amp block.
• Two of the three batch keys are set (or "auto" is used under Hugging Face).
• No removed or scheduled-for-removal keys are present.
• If offload is used, pin_memory is explicit and the host limit matches.
• The validator passes, and any exception has an owner and an expiry date.

Parallelism and memory

• The chosen stage is justified by an estimate or a measurement.
• TP and SP degrees fit within a node and divide the GPU count.
• MoE uses ZeRO-2 with native MoE, or AutoEP, not native MoE with ZeRO-3.
• Pipeline jobs avoid Accelerate and the newer accumulation controls.

Checkpoints

• The long-lived format is named, and the conversion has been tested.
• The resume plan survives a changed world size.
• No secrets are in client_state.
• Any externally sourced checkpoint is handled as untrusted.

Launch

• The image is pinned and has ops pre-built for our architectures.
• Port, extensions directory and NVMe path are unique per job.
• The timeout is set for the longest legitimate collective.
• .deepspeed_env contains no credentials.

Review checklist for design documents

A design document that proposes a new training setup should answer these before it is approved.

1. Memory arithmetic. What are the parameter count and precision, and what do the estimator functions report for the intended stage and offload?
2. Interconnect. What is the bandwidth within and between nodes, measured with ds_bench, and which parallelism forms cross nodes?
3. Throughput target. What step time or tokens per second is required, and what evidence says the design reaches it?
4. Cost. What is the cost per million tokens, and what are the alternatives that were considered (a smaller stage, more nodes, a parameter-efficient method)?
5. Checkpoint plan. Format, interval, retention, storage tier, save-time budget, and resume on a changed layout.
6. Failure plan. What happens when a rank hangs, when a node dies, when the loss goes NaN? Who is paged, and what is the first command?
7. Security. Network isolation, the launch method (pdsh versus no-SSH), where secrets come from, and where checkpoints are loaded from.
8. Upgrade plan. Which version is pinned, which deprecated features are in use, and when will they be migrated?
9. Exit. Under what measured conditions would the team move to another approach?
10. Ownership. Who owns the job, the image and the on-call for it?

A design that cannot answer items 2, 5 and 6 is not ready, whatever the other seven say.

Practice cards

Use these as short drills with a colleague.

Card 1. A bf16 job on 0.19.7 reports skipped steps that it never reported on 0.19.2. Why? Answer: the bf16 gradient-overflow check became default-on in 0.19.3, so steps with infinite or NaN gradients are now skipped and logged.

Card 2. A team's config contains "mics_shard_size": 8. What happens on 0.19.7, and what is the migration path? Answer: a DeepSpeedConfigError at parse time, since MiCS was removed; consider ZeRO-3 with zero_hpz_partition_size set to the GPUs per node, and evaluate the result.

Card 3. An engineer wants to run zero_to_fp32.py ckpt pytorch_model.bin. What is wrong? Answer: in 0.19.x the second argument is an output directory, and the script needs about twice the checkpoint size in CPU RAM.

Card 4. Two jobs share a node and one fails at start-up. What do you check first? Answer: the master port, then the extensions directory, then the NVMe path.

Card 5. A reviewer asks why you are building ops into the image rather than using the default install. Answer: JIT compilation on shared nodes causes slow starts, lock collisions and toolkit mismatches; pre-building in a controlled image makes the build reproducible and scannable.

Card 6. A job hangs and the watchdog names rank 17. Is rank 17 the culprit? Answer: not necessarily; it is the rank that noticed. Find which rank is in a different collective or stalled outside one.

Operational recipes

Check an install and its ops.

ds_report
python -c "import deepspeed; print(deepspeed.__version__); from deepspeed.accelerator import get_accelerator; print(get_accelerator()._name)"

Read the installed and compatible columns. [YES] means pre-built, [OKAY] means it can be built on demand, and [NO] under compatible means it cannot build here, which for async_io usually means a missing libaio-dev.

Build a production image's ops.

TORCH_CUDA_ARCH_LIST="8.0;9.0" DS_BUILD_OPS=1 pip install deepspeed==0.19.7 --no-build-isolation

Install torch first, use a devel CUDA image, and run ds_report as the last build step so a broken build fails the image, not the job.

Run a no-SSH multi-node launch (one command per node).

deepspeed --hostfile=hostfile --no_ssh --node_rank=$NODE_RANK \
  --master_addr=$MASTER_ADDR --master_port=$JOB_PORT \
  train.py --deepspeed --deepspeed_config ds_config.json

Set an explicit longer timeout for a known-slow step.

DEEPSPEED_TIMEOUT=30 deepspeed train.py --deepspeed --deepspeed_config ds_config.json

Consolidate a ZeRO checkpoint offline, without a GPU.

python ckpt/zero_to_fp32.py ckpt out_dir --safe_serialization --max_shard_size 5GB

Convert for a different world size.

python -m deepspeed.checkpoint.ds_to_universal --input_folder ckpt/global_step1000 --output_folder ckpt_universal/global_step1000

Benchmark communication and NVMe before promising numbers.

ds_bench
ds_nvme_tune --nvme_dir /local_nvme

A starting config template with the flipped defaults made explicit.

{
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": 1.0,
  "bf16": { "enabled": true, "check_grad_overflow": true },
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": { "device": "none" }
  },
  "steps_per_print": 100
}

The "auto" values are filled in by the Hugging Face integration; outside it, replace them with numbers, setting two of the three batch keys.

A closing checklist for the quarter

• Every production config passes the validator, and the exception list is short and dated.
• Every image has ops pre-built, is pinned to a triple, and has passed the smoke test.
• Every flipped default is explicit in the templates.
• At least one conversion to Universal Checkpointing has been rehearsed per model family.
• The hang alert fired in a game day and the on-call found the late rank.
• The fleet has been searched for amp, progressive_layer_drop, sparse_gradients and ZeRO-3 elastic_checkpoint.
• Secrets have been searched for in .deepspeed_env, configs and client_state.
• The next upgrade window is on the calendar, and the release notes for every intermediate version have an owner.
• You can say, in two sentences, when a team should not use DeepSpeed.