The Complete Axolotl Guide

This is part one of three. It covers everything you need to fine-tune a language model with Axolotl for the first time, not a teaser. By the end you can install the tool, write a configuration file, check that your data is being read the way you think it is, train a small model, talk to the result, and merge it into something you can hand to a colleague. Mid-level and Senior take the same ground further, into multi-GPU training, reinforcement learning and running Axolotl as a platform. Nothing here is thrown away.

This guide is written against Axolotl 0.20.0, released on 30 September 2026. The project moves quickly and renames configuration keys every few weeks, so a tutorial found on a blog from last year is probably wrong in at least one place. Where a key changed recently, this guide shows the current form and says what the old one was.

Each section ends with a Try it task. Do them in order. Fine-tuning is a craft you learn by watching a loss number fall (or refuse to), and no amount of reading replaces one real run.

What Axolotl is, and the problem it solves

A large language model such as Llama, Qwen or Gemma arrives from its publisher as a set of weights: billions of numbers that encode what the model learned from reading a vast amount of text. Out of the box it is a general-purpose assistant. Often that is not what you want. You may need it to answer in a particular tone, follow a strict output format, know the vocabulary of a specific industry, or handle Arabic dialect text better than the publisher's release does. Fine-tuning means continuing the model's training on your own examples so that its behaviour shifts towards them.

Doing that by hand is more work than it first appears. You must load the model, load and clean the data, turn each example into the exact token sequence the model expects, decide which tokens count towards the learning signal, pick a method that fits in your GPU's memory, choose an optimizer and a learning rate schedule, run the loop, log the results, save checkpoints, and later load everything back for testing. Each of those steps has a library behind it: transformers for models and the trainer, peft for efficient adapters, trl for preference and reinforcement methods, accelerate for running on several GPUs, datasets for data. Wiring them together in a Python script is possible, and many people do it, but every project ends up with its own slightly different script, and every script has its own bugs.

Axolotl is an open-source framework that wraps those libraries behind one YAML file. You describe the whole run in a configuration (which model, which data, which method, which hyperparameters, where to save) and Axolotl does the wiring. The same file then drives every other step: preparing the data, training, trying the model, merging the adapter, evaluating, quantizing and exporting.

That matters for a reader in the Gulf or Egypt. Many employers cannot send customer text to a third-party API, either because of data-residency rules or because the contract forbids it. A config-driven tool that runs entirely on hardware you control, including a rented GPU in a region you choose, is a practical answer to that constraint. Axolotl does collect anonymous usage telemetry by default, and we cover how to turn it off in the configuration section; in a regulated environment, turn it off from the first run.

Two facts about how the tool is built explain much of what follows:

The mental model: six nouns

Axolotl has a lot of keys in its configuration, hundreds in the reference page, but a beginner can work with six ideas. Learn these and the rest of the file becomes readable.

Base model. The pretrained model you start from, named by the key base_model. It is either a repository id on the Hugging Face Hub, such as NousResearch/Llama-3.2-1B, or a path to a folder on disk. It is the only key Axolotl strictly requires.

Adapter. A small set of extra trainable weights attached to a frozen base model. Instead of changing billions of numbers, you train a few million that sit alongside them. The most common kind is LoRA (low-rank adaptation). The variant QLoRA does the same thing but loads the frozen base in a compressed 4-bit form to save memory. If your config has no adapter: line, Axolotl does a full fine-tune, where every parameter is updated.

Dataset and its type. Your examples live in files or on the Hub, but a model cannot read a spreadsheet row. The dataset type tells Axolotl how to turn a row into tokens, and which of those tokens the model should learn to produce. Types include alpaca for instruction and response pairs, chat_template for multi-turn conversations, and completion for plain text.

Prepared dataset. Turning text into tokens takes time, so Axolotl does it once, in a step called preprocessing, and caches the result on disk. Training then reads the cache.

Sequence length and packing. sequence_len is the maximum number of tokens in one training row. Sample packing glues several short examples into one row so that the GPU is not spending its time on padding.

Output directory. Where checkpoints and the final adapter land, set with output_dir.

One more idea deserves a place in your head from the start: label masking. During training the model is shown a sequence of tokens and asked to predict each next token. The loss, the number that says how wrong it was, is computed over tokens whose label is not -100. Tokens labelled -100 are ignored. For instruction data you normally want the model to learn the answer, not to memorise the question, so the prompt tokens are masked. Axolotl's default, train_on_inputs: false, does exactly that. Most surprising results from a first fine-tune trace back to masking being different from what the author assumed, which is why the preprocessing step has a debug mode that lets you see it.

What you need before you start

Fine-tuning needs a GPU, and the first practical question is which one. Axolotl supports NVIDIA GPUs, and an AMD path exists, though its documentation is dated and you should treat it as community-grade. For the example in this guide you want an NVIDIA card from the Ampere generation or newer (RTX 30-series, A10, A100, L4, H100 and later). Ampere or newer matters because bf16, the 16-bit number format most recipes use, and Flash Attention both need it. Older cards such as the T4 or V100 can work with fp16 instead, but expect more friction.

How much memory? The project's own guidance for choosing a method gives rough figures for short sequences and a micro-batch of one or two:

Read that table as a reason to start small. This guide uses a one-billion-parameter model, which fits comfortably on a modest card, so you can finish every step without renting anything expensive. The method (LoRA instead of full fine-tuning) matters more than any other single decision for memory.

You also need:

• Linux, or Windows through WSL2. Axolotl does not support Windows natively; the documentation recommends WSL2 or Docker.
• Python 3.12 or newer. This is a hard minimum since version 0.20. Tutorials that say Python 3.10 are out of date.
• PyTorch 2.13 or newer. The release pins a range from 2.13.0 to 2.14.0.
• A Hugging Face account, for downloading models and, if you choose, uploading results. Some models are gated, meaning you must accept their licence on the model page before the download works.
• Disk space. Models and datasets are cached, and checkpoints are written during training. Leave tens of gigabytes free, more for larger models.

A Mac is for authoring, not training. Apple Silicon is partly supported: full training, LoRA and sample packing work, but 4-bit and 8-bit loading (so no QLoRA), mixed precision, Flash Attention and DeepSpeed do not. Use a Mac to write configs and check data on a tiny model, then train on a Linux GPU.

If you do not own a GPU, the documentation names providers such as RunPod, Vast.ai and Modal that work with Axolotl's Docker images, and guides for SkyPilot and Hugging Face Jobs. Rented machines bill while they run, so smoke-test on a few steps first.

Installing Axolotl

The recommended installer is uv, a fast Python package and environment manager. It replaces the old combination of pip, venv and friends for this purpose, and the project's own instructions and Docker images are built around it. On Linux the sequence is:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

export UV_TORCH_BACKEND=cu130
uv venv --python 3.12
source .venv/bin/activate
uv pip install --no-build-isolation 'axolotl[deepspeed]'

Take it line by line. The first two install uv and put it on your path. UV_TORCH_BACKEND tells uv which CUDA build of PyTorch to fetch; cu130 means CUDA 13.0, and the documentation also lists cu128 for CUDA 12.8. Pick the one that matches your driver. uv venv --python 3.12 creates an isolated environment so Axolotl's pinned libraries cannot clash with anything else on your machine, and source .venv/bin/activate switches into it. The final line installs Axolotl itself, together with the optional deepspeed extra.

The --no-build-isolation flag is required. Some of Axolotl's dependencies compile code against the PyTorch that is already installed, so the build must be allowed to see it. Leaving the flag out is the most common cause of a failed install.

The square brackets in axolotl[deepspeed] name an extra, an optional bundle of dependencies. Quote them (as above) because the zsh shell on macOS and many Linux setups treats square brackets as a pattern. Other extras you will meet later include flash-attn, vllm and ringmaster. Beginners can ignore them for now.

If you prefer plain pip, install PyTorch first, then:

pip3 install -U packaging setuptools wheel ninja
pip3 install --no-build-isolation 'axolotl[deepspeed]'

If you want the newest code instead of the last release, clone the repository and install it in editable mode:

git clone https://github.com/axolotl-ai-cloud/axolotl.git
cd axolotl
uv pip install --no-build-isolation -e '.[deepspeed]'

The Docker route avoids almost all of this. The project publishes images on Docker Hub that already contain a matching Python, CUDA, PyTorch and Axolotl:

docker run --gpus '"all"' --rm -it --ipc=host axolotlai/axolotl:main-latest

The --gpus '"all"' flag exposes your GPUs to the container (the odd double quoting is deliberate), and --ipc=host gives it the shared memory that data loaders need. The main-latest tag follows the development branch, which is fine for learning but wrong for anything you must reproduce; for that, use a release tag such as 0.20.0-py3.12-cu130-2.13.0. Since version 0.18 all published images are uv-based. If you are new to containers, the Docker guide explains what that command is doing. Rental providers usually want the axolotlai/axolotl-cloud image, which adds Jupyter and SSH access.

Checking that the setup works

Never start a three-hour run on an unverified install. Four quick commands tell you almost everything:

axolotl --version
axolotl --help
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
nvidia-smi

The first should print axolotl, version 0.20.0. The second lists the commands the tool offers. The third prints your PyTorch version and then True if PyTorch can see a GPU. If it prints False, training will not use your GPU: the cause is nearly always a PyTorch build that does not match the CUDA driver, or running inside a container started without --gpus.

Next, log in to Hugging Face so the tool can download models:

hf auth login

Paste an access token from your account settings when prompted. The older huggingface-cli login command does the same job. You can also set the environment variable HF_TOKEN. Axolotl checks the token when it starts, and if something is wrong you will see a message beginning "Error verifying HuggingFace token. Remember to log in using hf auth login". If you are working with no internet on purpose, setting HF_HUB_OFFLINE=1 skips that check.

One behaviour surprises people: when you run any axolotl command, it first prints an ASCII-art banner, and it loads a .env file from the current folder if one exists. That means a token or key in .env is picked up automatically. It also means you must never commit that file.

Telemetry. Axolotl sends anonymous usage data (operating system, library versions, hardware, and sanitized error types) by default, and on first start it pauses for ten seconds to tell you so. To switch it off, set an environment variable before running:

export AXOLOTL_DO_NOT_TRACK=1

DO_NOT_TRACK=1 works as well. Setting AXOLOTL_DO_NOT_TRACK=0 explicitly turns it on and silences the notice. If your employer works in a regulated sector, ask which you should do, and do it deliberately rather than by accident.

Finally, fetch the example configs that ship with the project. They are the best teaching material you have:

axolotl fetch examples

This downloads a folder called examples/ into the current directory, with ready-made YAML files for many models and methods. Use examples from the same release as your install. An example from the development branch may use a key your installed version does not know.

Your first run: a LoRA fine-tune of a small model

The official quickstart trains a LoRA adapter on NousResearch/Llama-3.2-1B, a one-billion-parameter model that is not gated, using a public instruction dataset. We will follow it, then take it apart. Here is the config, saved as lora-1b.yml:

base_model: NousResearch/Llama-3.2-1B

adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

datasets:
  - path: teknium/GPT4-LLM-Cleaned
    type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/lora-out

sequence_len: 2048
sample_packing: true

micro_batch_size: 2
gradient_accumulation_steps: 2
num_epochs: 1
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0002
warmup_ratio: 0.1

bf16: auto
gradient_checkpointing: true
attn_implementation: flash_attention_2

evals_per_epoch: 4
saves_per_epoch: 1
logging_steps: 1

special_tokens:
  pad_token: "<|end_of_text|>"

The quickstart page also shows a load_in_8bit: true line in its snippet, while the example file in the repository omits it. Both work; leaving it out is simpler and is what we do here.

The first run: do the cheap smoke test first, then the real run.

axolotl train lora-1b.yml --max-steps 5
axolotl train lora-1b.yml

If the five steps finish, your install, GPU, token and config all work. You can also take the file straight from the repository without saving it first, because the CLI accepts a local path or an HTTPS link to raw YAML. Pin such a link to a release tag, not to the moving main branch.

While it trains, watch the terminal. You will see a progress bar and, because logging_steps: 1, a line of numbers each step. The two you care about first are loss (how wrong the model currently is on the training data; it should trend downwards) and grad_norm (the size of the update signal; steady and modest is good). We read these properly in a later section. Also open a second terminal and run watch -n 2 nvidia-smi to see GPU memory in use. Watching it once teaches you more about the relationship between batch size and memory than any table.

A note on the first thing that may go wrong. If the log says the tokenizer has no padding token, the special_tokens block is what fixes it, and the quickstart config already includes it. A tokenizer is the component that turns text into token numbers, and training in batches requires one extra token to pad short rows; many base models do not define one, so you nominate an existing token for the job.

Reading the config, line by line

A config you cannot read is a config you cannot debug. Here is what every line in that file does, grouped by purpose.

The model. base_model names the starting point. Axolotl downloads it through the Hugging Face libraries on first use and caches it, so the second run starts faster.

The adapter. adapter: lora switches from full fine-tuning to LoRA. lora_r is the rank: the size of the small matrices that are trained. Higher rank means more capacity and more memory; 8 to 32 is the common range, and 16 is a sensible beginner default. lora_alpha scales how strongly the adapter's output is mixed in; a common convention is to set it to twice the rank, as the quickstart does. lora_dropout randomly zeroes a fraction of adapter activations during training to reduce overfitting, and 0.05 is a gentle value. lora_target_modules lists which layers of the model get an adapter; the seven names here cover the attention and feed-forward projections of a Llama-style model. The shortcut lora_target_linear: true targets every linear layer without you listing names.

The data. datasets is a list, so you can mix several sources. Each has a path (a Hub id or a local file) and a type. dataset_prepared_path is where the tokenized cache goes. val_set_size: 0.1 holds out ten percent of the data, never trained on, so the evaluation steps can measure how the model does on examples it has not seen.

Sequence handling. sequence_len: 2048 caps each row at 2,048 tokens. The default is only 512, which silently shortens many real examples, so always set it deliberately. sample_packing: true packs short examples together. With Flash Attention it does so safely, so packed examples cannot attend to each other.

Batching. micro_batch_size is how many examples each GPU processes in one forward pass; gradient_accumulation_steps is how many such passes are added up before one weight update. The effective batch size is the product of the two and the number of GPUs: here 2 x 2 x 1 = 4. If you run out of memory, lower the micro-batch and raise the accumulation to keep the effective size the same. Do not set batch_size yourself; it is derived.

The schedule. num_epochs: 1 means one pass over the data. If you also set max_steps, that wins. optimizer: adamw_8bit stores the optimizer's state in 8 bits to save memory. lr_scheduler: cosine shapes the learning rate along a cosine curve. learning_rate: 0.0002 is 2e-4, normal for LoRA; the project's stability guide gives 1e-4 to 3e-4 for LoRA and 1e-5 to 5e-5 for full fine-tuning. warmup_ratio: 0.1 ramps the rate up over the first tenth of training so early steps do not jolt the model.

Precision and speed. bf16: auto uses bfloat16 where the hardware supports it. gradient_checkpointing: true recomputes some values during the backward pass instead of storing them, trading roughly thirty percent more time for a large memory saving. attn_implementation: flash_attention_2 selects the attention algorithm. This is the current form. Older tutorials write flash_attention: true, which is still accepted but marked deprecated since version 0.17.

Evaluation and saving. evals_per_epoch: 4 evaluates four times per epoch; saves_per_epoch: 1 writes one checkpoint per epoch. These cannot be combined with the step-based versions (eval_steps, save_steps): Axolotl rejects a config that uses both of a pair.

Special tokens. The pad_token entry sets the padding token as discussed.

Your own data: formats and dataset types

The quickstart dataset is someone else's. Soon you will want to train on your own examples, and the single most important skill is matching your data's shape to the right type. Axolotl accepts local files in JSON Lines, CSV, Parquet and Arrow formats (it infers the format from the extension), folders of files, Hub repositories, and paths on cloud storage such as s3:// and gs://. JSON Lines, one JSON object per line, is the easiest to produce and inspect.

Instruction data (alpaca). Each row has an instruction, an optional input and an output. A file my_data.jsonl looks like this:

{"instruction": "Translate to French.", "input": "Good morning", "output": "Bonjour"}
{"instruction": "Name a prime number between 10 and 20.", "input": "", "output": "13"}

and the config entry is:

datasets:
  - path: my_data.jsonl
    type: alpaca

Conversation data (chat_template). For multi-turn chat, each row holds a list of messages in the OpenAI style:

{"messages": [{"role": "user", "content": "What is MLOps?"}, {"role": "assistant", "content": "MLOps is the practice of running machine learning reliably in production."}]}

with this config:

chat_template: tokenizer_default
datasets:
  - path: my_chats.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles_to_train: ["assistant"]
    train_on_eos: turn

A chat template is a small Jinja program, supplied by the model's publisher, that renders a list of messages into the exact text (with special marker tokens) that the model was trained to expect. chat_template: tokenizer_default means "use the one the tokenizer carries". Use it when your base model is a chat model that has one. If the base model has no template, you will get an error saying the tokenizer's chat_template is null, and the fix is to name a built-in one such as chatml or llama3, or to supply your own with chat_template_jinja.

roles_to_train: ["assistant"] is label masking for chat: only assistant turns count towards the loss, and system and user turns are context. train_on_eos: turn additionally trains the end-of-turn token for each assistant message, so the model learns when to stop talking.

If your data came from a ShareGPT-style export, with keys called conversations, from and value, you map them rather than renaming your file:

datasets:
  - path: sharegpt_export.jsonl
    type: chat_template
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value

Do not use type: sharegpt. It was removed, and today it stops the run with the message "type: sharegpt.* is deprecated. Please use type: chat_template instead." Many older blog posts still show it.

Plain text (completion). If you just want to continue pretraining on raw text, a completion type reads a text field. This is for domain adaptation, not for teaching a model to follow instructions.

Preprocessing, and looking at what the model will actually see

This is the step beginners skip and later wish they had not. axolotl preprocess runs only the data stage: it tokenizes your dataset, applies masking and packing, and writes the result to the cache.

axolotl preprocess lora-1b.yml --debug

With --debug, it prints a few examples as a stream of tokens, each shown with its label and its token id. Tokens the model will be trained to predict show a real label; tokens that are masked show -100. Read a few examples and ask the plain question: is the part I want the model to learn the part that is not masked, and do the special marker tokens look right? You can ask for more examples with --debug-num-examples.

Why do this separately when train would do it anyway? Three reasons. It surfaces data problems in seconds, before you have loaded a model. It can run on a machine with no GPU at all (setting CUDA_VISIBLE_DEVICES="" keeps it off the GPU), which is cheaper when your GPU is rented by the hour. And it makes the cache explicit, so you know where it is.

The cache has a subtlety that causes real bugs. The tokenized data is stored under dataset_prepared_path, in a folder named by a hash of the relevant config. When you leave dataset_prepared_path empty, output goes to ./last_run_prepared/ and the old cache is ignored, so every run re-tokenizes. When you set it explicitly, as the quickstart does, the cache is reused on the next run. That is faster, but if you edit your data file or a custom prompt strategy without changing the config, you may silently train on the stale version. When in doubt, delete the folder:

rm -rf last_run_prepared

Training and reading the logs

When axolotl train is running you are looking at the output of the Hugging Face trainer. The three numbers worth understanding at first are loss, gradient norm and learning rate.

Loss. It is the average error per predicted token. For chat fine-tuning on a reasonable dataset the project's stability guide says to expect values between roughly 0.5 and 2.0. What matters more than the absolute number is the trend. A loss that drops quickly and then flattens is normal. A loss that stays flat from the start suggests the learning rate is too low, the data is not what you think, or nearly everything is masked. A loss that hits nan (not a number) means numerical trouble, often an over-high learning rate or a precision problem.

Evaluation loss. Because you set val_set_size, Axolotl periodically reports eval_loss on held-out examples. Compare it to the training loss. If training loss keeps falling while evaluation loss rises, the model is overfitting: memorising the training set instead of learning something general. The cures are more data, fewer epochs, a lower learning rate or a smaller rank.

Gradient norm. grad_norm measures how large the update is. The stability guide gives a healthy range of about 0.1 to 10, with spikes above 100 indicating instability. The config key max_grad_norm (1.0 is the usual value) clips updates so a rare bad batch cannot wreck the model.

Learning rate. With a warmup and a cosine scheduler you will see it climb, peak, then glide towards zero. If the log shows it stuck at a constant, check that lr_scheduler is what you intended.

Training writes into output_dir. You will find checkpoint folders (one per save), and, at the end, the adapter weights and tokenizer files. A copy of your config is placed there too, which is why tokens must never be pasted into the YAML: the file travels with the output, and it can end up on the Hub if you push.

For tracking runs over time, Axolotl logs to Weights and Biases, MLflow, Comet, Trackio and TensorBoard. Setting up one tracker early is worth it, because comparing a dozen runs by scrolling terminal history is miserable. The MLflow guide and the Weights and Biases guide cover those tools. In Axolotl you enable them with a few keys, for example wandb_project: my-first-finetune, with your key supplied through the WANDB_API_KEY environment variable rather than the file.

Trying the model, then merging the adapter

A trained adapter is not a standalone model. It is a small file that only makes sense on top of its base model. To test it, Axolotl loads both:

axolotl inference lora-1b.yml --lora-model-dir ./outputs/lora-out

You get an interactive prompt: type something, press enter, and the model answers. Since version 0.18 there is a multi-turn chat mode, which keeps context across messages and suits a model trained on conversations:

axolotl inference lora-1b.yml --lora-model-dir ./outputs/lora-out --chat

and a browser interface through --gradio. The inference command reads the same YAML, so it knows the base model and the prompt format. It can also read prompts from standard input, which lets you script a batch of test questions.

Prompt your model the way you will use it in practice, including some inputs that look nothing like your training data. A model that handles your ten training-like examples and falls apart on everything else has memorised, not learned. Keep a small fixed list of test prompts, so you can compare run to run.

Once you are satisfied, merge the adapter into the base weights to produce a normal, standalone model:

axolotl merge-lora lora-1b.yml --lora-model-dir ./outputs/lora-out

The merged model is written to ./outputs/lora-out/merged. After this, it is an ordinary Hugging Face model folder that can be loaded by the transformers library or served by an inference engine such as vLLM. If your GPU is busy, setting CUDA_VISIBLE_DEVICES="" makes the merge run on CPU. Always merge with this command and not with a custom script: Axolotl handles details such as embeddings that grew during training.

There is one error to recognise here. Putting merge_lora: true into a config that also sets load_in_4bit produces "Can't merge qlora if loaded in 4bit". The merge command handles quantized bases itself, so run axolotl merge-lora separately, optionally with --dequant to get a bf16 result.

To use the merged model on a laptop with tools such as Ollama or llama.cpp, version 0.20 added a GGUF exporter, axolotl export. It needs a built copy of llama.cpp and it refuses an adapter-only folder, so merge first. That is a mid-level topic; for now just know the route exists.

If you want to share your work, hub_model_id in the config pushes the output to your Hugging Face account.

The everyday commands, grouped by purpose

Axolotl's command line follows one pattern: axolotl <command> <config.yml> [options]. The config can be a local path or an HTTPS URL. Options use dashes (--lora-model-dir). By contrast the legacy form python -m axolotl.cli.train uses underscores. Learn them by what you are trying to do.

Getting started.

Preparing data.

Training and measuring.

Using the result.

Two ideas are worth adopting immediately. First, any config key can be overridden on the command line as --key-name value, using dashes. That is how the --max-steps 5 smoke test worked without editing the file. Nested keys use dot notation, for example --trl.beta 0.1. Use this for one-off experiments and keep the file as the record of what you ran. Second, the launcher decides how worker processes start. The default is accelerate; with several GPUs you can pass --launcher torchrun, and arguments for the launcher go after a double dash:

axolotl train cfg.yml --launcher torchrun -- --nproc_per_node=2

You will not need that until you have more than one GPU, which is the subject of the next level. The older style, accelerate launch -m axolotl.cli.train cfg.yml, still works and appears in many tutorials.

When a run misbehaves, set AXOLOTL_LOG_LEVEL=DEBUG and keep a copy of the output:

AXOLOTL_LOG_LEVEL=DEBUG axolotl train cfg.yml 2>&1 | tee run.log

The tee command shows the output while also saving it, so you can search the log afterwards or paste it into a bug report.

Choosing a method: LoRA, QLoRA or full fine-tuning

The main decision in any run is how much of the model to train. Everything else follows from it.

LoRA is the right default. It uses far less memory, trains faster, and for teaching a style, a format or a narrow domain it usually gets you most of the way there. QLoRA is LoRA with the frozen base compressed to 4-bit, which lets a model fit on a smaller card at some cost in speed and a small cost in quality. Choose it when plain LoRA does not fit. To switch, change the adapter and ask for 4-bit loading:

adapter: qlora
load_in_4bit: true

Both lines are needed. If you set adapter: qlora alone, validation fails with "Require cfg.load_in_4bit to be True for qlora". QLoRA also relies on the bitsandbytes library, which is not available on macOS, so it is a Linux GPU technique. Full fine-tuning is the config with no adapter: line at all. It can teach the model things an adapter cannot, but it needs many times the memory, produces a full-size output and is easier to ruin the model with, because every weight can move. Reach for it only when LoRA has demonstrably failed.

How do you decide? If LoRA fits on your GPU, use it; if not, try QLoRA; only if LoRA demonstrably cannot learn your task, consider a larger rank and then a full fine-tune. The library behind adapters is covered in the PEFT and TRL guide, and DeepSpeed is what Axolotl uses to spread large runs across GPUs.

Common errors and how to read them

Error messages in Axolotl usually come from one of three places: the config validator (immediate, readable), the data pipeline (during preprocessing) or PyTorch (during model load or training). Identify the stage first, then match the message.

The padding error exists because a batch holds rows of different lengths and the filler token is the pad token. The bf16 error is a hardware fact: bfloat16 needs an Ampere-generation GPU or newer. For problems not in the table, read the first error in the log, decide which stage failed, and reproduce with the smallest config.

Putting it all together: a small end-to-end project

Here is a complete workflow that uses everything above. The goal is to teach a small model a consistent house style for support replies, from a dataset you build yourself.

1. Set up and pin.Create the virtual environment, install axolotl==0.20.0, run axolotl --version, log in with hf auth login, and set AXOLOTL_DO_NOT_TRACK=1 if your employer requires it.
2. Build the data.Write at least a few hundred examples of the style you want in support.jsonl, in the OpenAI messages format, with a system message that stays identical across rows. Quality beats quantity; ten careful examples beat a hundred sloppy ones.
3. Write the config.Copy the quickstart, set the dataset to type: chat_template with field_messages: messages, set chat_template: chatml if the base model has no template, and name the output ./outputs/support-lora.
4. Look at the data.Run axolotl preprocess support.yml --debug and confirm only assistant turns are trained.
5. Smoke test.Run axolotl train support.yml --max-steps 5, fix whatever appears, then launch the full run.
6. Watch it.Follow loss and eval loss. Stop and rethink if eval loss rises while training loss falls.
7. Test it.Run axolotl inference support.yml --lora-model-dir ./outputs/support-lora --chat with a fixed set of test prompts, including ones outside the style.
8. Merge and keep.Run axolotl merge-lora support.yml --lora-model-dir ./outputs/support-lora, and store the config, the data and a note of the version next to the result.

Here is the config for step three, compact enough to read in one go:

base_model: NousResearch/Llama-3.2-1B

adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

chat_template: chatml
datasets:
  - path: support.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles_to_train: ["assistant"]
    train_on_eos: turn

dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/support-lora

sequence_len: 2048
sample_packing: true
eval_sample_packing: false

micro_batch_size: 2
gradient_accumulation_steps: 4
num_epochs: 2
optimizer: adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0002
warmup_ratio: 0.1

bf16: auto
gradient_checkpointing: true
attn_implementation: flash_attention_2

evals_per_epoch: 4
saves_per_epoch: 1
logging_steps: 1

special_tokens:
  pad_token: "<|end_of_text|>"

When the result disappoints, the cause is almost always one of three things: not enough good data, a prompt format at test time that differs from training, or masking that is not what you assumed. Check them in that order, and change one thing per experiment.

What you can now do, and what comes next

You can now explain the problem Axolotl solves and the six nouns of its model. You can install it, verify the install, and avoid the early traps (the Python and PyTorch minimums, --no-build-isolation, the token login). You can read a LoRA config line by line, match your data to a dataset type, check masking before spending GPU time, run and monitor a training job, test the result and merge it, and recognise the common errors by their messages.

The beginner's rules of thumb: start small; smoke-test with a few steps; look at the tokens before you train; change one thing at a time; keep the config and the version with every result; never put secrets in the file. When a run runs out of memory, lower micro_batch_size and raise gradient_accumulation_steps, turn on gradient_checkpointing, switch to QLoRA, then shorten sequence_len. The dataset's size does not change peak memory.

The mid level goes deeper into the machinery: multi-GPU training with DeepSpeed and FSDP2, plugins and kernels, preference methods such as DPO, tracking and evaluation. Neighbouring guides worth reading: the Hugging Face guide, the PEFT and TRL guide and the vLLM guide. The senior level covers running Axolotl as a shared platform.

The quality of a fine-tuned model is decided mostly by the quality of the data and how carefully you checked it. Spend your next hour on the examples, not the knobs.

Sources

• Axolotl documentation home
• Installation
• Getting started
• Command line reference
• Config reference
• Dataset formats
• Conversation dataset format
• Instruction tuning format
• Dataset loading
• Dataset preprocessing
• Sample packing
• Batch size vs gradient accumulation
• Choosing a method
• LoRA and QLoRA
• Inference
• Docker images
• Running on a Mac
• Telemetry
• FAQ
• Debugging
• Training stability
• Axolotl releases on GitHub
• Axolotl v0.20.0 release
• Axolotl on PyPI
• Axolotl on Docker Hub

This file prepares you to talk about Axolotl in an interview for a junior MLOps or LLM engineering role. Interviewers at this level are not checking whether you have memorised every key. They want to know whether you understand what the tool is for, whether you have actually run a fine-tune, and whether you can reason about the things that go wrong. Read the detailed guide first; this file assumes you have, and it is about saying the same things aloud, briefly and correctly.

All the specifics below are for Axolotl 0.20.0, released on 30 September 2026. If an interviewer quotes a key you do not recognise, it may have been renamed since; saying "that was renamed in a recent release, I would check the current docs" is a strong answer, not a weak one.

The thirty-second answer

Axolotl is an open-source framework for fine-tuning large language models from a single YAML configuration file. It wraps the Hugging Face libraries (transformers, peft, trl, accelerate, datasets) and adds a data pipeline, sample packing, optimized kernels, a plugin system and a command line, so that one config describes the whole run and the same file drives preprocessing, training, inference, merging, evaluation and export.

If you have a minute more, add why people choose it: it is repeatable because the config is the record of the run, it runs on hardware you control (useful when data cannot leave a region or a company), and it scales from one GPU to many without rewriting code. Add what it is not: it is a batch trainer, with no server, scheduler or dashboard of its own.

A good delivery sounds like this: "It lets you describe a fine-tuning job in YAML, so instead of maintaining a custom training script you maintain a config. I used LoRA on a one-billion-parameter model, checked the label masking with the preprocess debug flag, trained, tested with the inference command and merged the adapter." Concrete experience, in one breath, beats a definition.

The model to hold in your head

Interviewers often probe the mental model, because it shows whether you can predict behaviour. Keep these six ideas ready.

Base model. The pretrained weights you start from, named by base_model. The only key strictly required.

Adapter. A small set of trainable weights attached to a frozen base. LoRA trains low-rank matrices; QLoRA does the same with the base loaded in 4-bit. No adapter line means full fine-tuning, where every parameter updates.

Dataset type. The recipe that turns raw rows into tokens and decides which tokens are learned. alpaca for instruction pairs, chat_template for conversations, completion for raw text.

Label masking. Tokens labelled -100 are ignored by the loss. With the default train_on_inputs: false, prompts are masked and only the answer is learned. For chat data, roles_to_train (default the assistant) controls which turns count.

Sample packing. Several short examples are glued into one row of sequence_len tokens, with attention arranged so they do not see each other, so the GPU wastes less time on padding.

Effective batch size. micro_batch_size times gradient_accumulation_steps times the number of GPUs. micro_batch_size is what fits in memory per pass; accumulation lets you reach a bigger batch without more memory.

One line that ties them together: "The config is validated first, then data is prepared and cached, then the model is loaded with an adapter and trained, and the result lands in output_dir." If you can say that and then say which of those steps a given bug belongs to, you are ahead of most candidates.

The commands to have ready

You may be asked to write a command on a whiteboard or recite the workflow. This is the sequence to know cold:

axolotl fetch examples
axolotl preprocess config.yml --debug
axolotl train config.yml
axolotl inference config.yml --lora-model-dir ./outputs/lora-out
axolotl merge-lora config.yml --lora-model-dir ./outputs/lora-out

What each does, in a phrase: fetch examples downloads maintained example configs; preprocess --debug tokenizes and caches the data and prints tokens with their labels so you can verify masking; train runs training; inference loads the base plus adapter for a try-out; merge-lora folds the adapter into the weights, writing to {output_dir}/merged.

Other things worth being able to say without notes:

• Any config key can be overridden on the command line as --key-name value, with dashes. axolotl train config.yml --max-steps 5 is the standard smoke test.
• Launching on several GPUs uses --launcher torchrun with arguments after a double dash, or the default accelerate launcher. The older accelerate launch -m axolotl.cli.train form still works.
• Checking the install: axolotl --version, and python -c "import torch; print(torch.cuda.is_available())".
• Logging in to the Hub: hf auth login, or the HF_TOKEN variable.
• Chat mode for testing: axolotl inference config.yml --lora-model-dir DIR --chat since version 0.18.
• Exporting to GGUF is axolotl export, new in 0.20, and it needs a merged model and a built llama.cpp.

And one install fact interviewers like because it catches out people who learnt from old tutorials: version 0.20 requires Python 3.12 or newer and PyTorch 2.13 or newer, and the install needs --no-build-isolation.

The classic traps, and why they happen

These are the things that go wrong in real first projects. Knowing the cause is what separates a candidate who has done it from one who has read about it.

The model trained on the wrong tokens. Training runs fine, loss falls, but the model behaves oddly. The cause is nearly always masking or a template mismatch: the prompt was trained when it should have been masked, or the chat template at training time differs from the one used at inference. Prevention is axolotl preprocess --debug, reading a few examples, and using the same template when you test.

Stale prepared data. When dataset_prepared_path is set explicitly, Axolotl reuses the cache. Edit your data file without touching the config, and you may silently train on the old data. When the path is empty, the cache in last_run_prepared is ignored, so it re-tokenizes every time. Delete the cache folder when in doubt.

Out of memory. The model, batch, sequence length and method decide VRAM use, not the dataset size. The fix is a ladder: smaller micro-batch with more accumulation, gradient checkpointing, QLoRA or 8-bit, shorter sequences, efficient attention.

QLoRA misconfigured. adapter: qlora requires load_in_4bit: true, or validation fails. QLoRA also depends on bitsandbytes, which is unavailable on macOS.

Deprecated keys from old tutorials. type: sharegpt now raises an error; the replacement is type: chat_template with field_messages: conversations and message_property_mappings: {role: from, content: value}. flash_attention: true still works but is deprecated in favour of attn_implementation: flash_attention_2. In 0.20 FSDP1 was removed, so fsdp_version: 1 is an error.

Overfitting. Training loss keeps falling but evaluation loss rises. Cause: the model is memorising a small dataset. Fixes: more or better data, fewer epochs, lower learning rate, smaller rank.

Missing pad token. "Asking to pad but the tokenizer does not have a padding token" appears because batches need a filler and many base models define none. The fix is special_tokens: {pad_token: ...}, commonly the end-of-sequence token.

Secrets in the YAML. The config is copied into the output folder and can be pushed to the Hub with the model. Tokens belong in environment variables or a login.

Common interview questions

What problem does Axolotl solve, and why not write a training script?

A custom script means you own the data handling, prompt formatting, label masking, optimizer setup, logging, checkpointing and multi-GPU launching, and you re-solve them each project. Axolotl packages these behind a validated YAML file, so a run is reproducible from the config and the data, and a colleague can read it. The trade-off is that you work within its abstractions, and when you need something unusual you write a plugin or drop to the underlying libraries. For a first project, or a team that wants a standard process, the config-driven route is the better bet.

Explain LoRA to someone who has not heard of it.

Instead of updating all of a model's billions of weights, LoRA freezes them and trains small extra matrices that are added to certain layers. The matrices are low-rank, meaning they are the product of two thin matrices, so they have far fewer numbers. The result needs much less memory, trains faster, and saves as a small file that sits on top of the base model. In the config: adapter: lora, lora_r for the rank (8 to 32 is typical), lora_alpha for scaling, and lora_target_modules or lora_target_linear: true for which layers get an adapter.

What is the difference between LoRA and QLoRA?

Both train an adapter on a frozen base. QLoRA also loads that frozen base in 4-bit precision, which cuts memory further so a larger model fits on a given GPU, at the cost of some speed and a small quality cost. Configuration: adapter: qlora and load_in_4bit: true, both required. It needs bitsandbytes, so it is a Linux GPU technique.

When would you choose a full fine-tune?

Rarely at first. When LoRA has been tried properly (adequate rank, enough target modules, good data) and demonstrably cannot capture the behaviour, and when you have the memory: roughly 60 to 80 GB for a 7 to 8 billion parameter model in bf16, according to the project's own guidance. A full fine-tune also uses lower learning rates (1e-5 to 5e-5) and produces a full-size output. A good answer names the cost and the risk, not just the capability.

What is label masking and why does it matter?

The loss ignores tokens labelled -100. For instruction tuning you usually want to learn the answer and not the question, so prompt tokens are masked, which is the default train_on_inputs: false. For chat data, roles_to_train selects which roles are trained, and train_on_eos controls whether the end-of-turn token is trained so the model learns when to stop. It matters because getting it wrong produces models that parrot prompts or never stop generating. You verify it with preprocess --debug.

What is sample packing?

It combines several short examples into one sequence_len row to avoid wasting compute on padding tokens. With flash attention or SDPA variable-length support, a bookkeeping array of cumulative sequence lengths keeps examples from attending to each other. It raises throughput substantially on short-sample datasets. It does not reduce memory per row. It is not supported with KTO, and with a tiny validation set you may need eval_sample_packing: false.

Explain micro-batch size and gradient accumulation.

micro_batch_size is how many examples one GPU processes in a forward pass. gradient_accumulation_steps is how many of those passes are summed before one weight update. The effective batch size is micro-batch times accumulation times GPU count. If you run out of memory, lower the micro-batch and raise accumulation, which keeps the effective batch (and so the training dynamics) roughly the same while using less memory at a cost in speed.

What does axolotl preprocess do, and why run it separately?

It tokenizes the dataset, applies masking and packing and caches the result, without loading the model. Run it separately to catch data problems in seconds, to inspect labels with --debug, and to do the CPU-only work on a cheap machine, since a rented GPU bills while it idles. It does not work with streaming datasets or skip_prepare_dataset.

What is a chat template?

A Jinja template, usually supplied with the tokenizer, that converts a list of role-tagged messages into the exact text, with special marker tokens, that the model expects. In Axolotl, chat_template: tokenizer_default uses the tokenizer's own; named ones such as chatml or llama3 are built in; chat_template_jinja takes a custom one. If the base model has no template and you use the default, validation fails saying the tokenizer's template is null. The template must be the same in training and in use, or quality drops.

How do you know if a run is going well?

Training loss should trend down (the project's stability guide says 0.5 to 2.0 is typical for chat fine-tuning). Evaluation loss on held-out data should follow it and not diverge. Gradient norm should stay in a modest range, roughly 0.1 to 10, with sudden spikes above 100 indicating instability. Memory and throughput should be steady. And, decisively, the generated outputs on a fixed set of test prompts should look right. A good candidate mentions both numbers and outputs.

What would you do if you ran out of GPU memory?

Work down a ladder and change one thing at a time: lower the micro-batch and raise accumulation; turn on gradient checkpointing (about 30 percent slower, much less memory); switch to QLoRA or 8-bit; shorten the sequence length; use efficient attention. Only then consider DeepSpeed offloading. Say explicitly that dataset size does not affect peak memory.

What does exitcode: -9 mean?

The operating system killed the process for using too much system RAM, typically during tokenization. It is different from CUDA out of memory, which is GPU memory. Reduce dataset_num_proc, use less data, or preprocess on a machine with more RAM.

How would you make a fine-tuning run reproducible?

Commit the config and name it for what it tests. Pin the Axolotl version (releases rename keys often) and the base model revision. Record the dataset version and the exact command. Use an explicit output_dir and, in Docker, a version-tagged image such as 0.20.0-py3.12-cu130-2.13.0 rather than main-latest. Add that exact bit-for-bit identity across hardware is not guaranteed, but the setup should reproduce the result closely.

How do you use the result?

Test it with axolotl inference against the adapter, then merge with axolotl merge-lora to produce a standalone model in {output_dir}/merged. That folder can be loaded with transformers or served by an inference engine; you can also export to GGUF with axolotl export for use with llama.cpp-based tools. Always merge with the Axolotl command, which handles quantized bases and resized embeddings, instead of a hand-written script.

What data-privacy considerations apply?

Training data often includes customer text. Running Axolotl on infrastructure you control keeps that data in the region and under the contracts that apply, which matters for many Gulf and Egyptian employers. Practical points: choose the rental region deliberately; remember anonymous telemetry is on by default and can be disabled with AXOLOTL_DO_NOT_TRACK=1; use offline variables if the machine must have no external access; keep tokens out of the config; and remember the base model's licence applies to what you produce.

Name something you would check before trusting an upgrade.

Read the release notes for renamed or removed keys, since releases arrive every few weeks. Recent examples: relora_steps became jagged_restart_steps, dataset_processes became dataset_num_proc, FSDP1 was removed in 0.20. Then re-run preprocess --debug and a short smoke test, because tokenization and masking fixes change loss curves.

Questions to ask the interviewer

A junior candidate who asks sharp questions stands out. A few good ones: How do you version and review training configs? How do you decide a fine-tuned model is better than the base? Where does the training data come from, and who checks it? How do you keep GPU costs under control? Do you pin tool versions, and how often do you upgrade? Each shows you think about the process around the tool, which is what the job is.

Sixty-second self-test

Answer aloud, without looking above. Any you cannot answer cleanly are the sections to reread.

This is the practical companion to the beginner guide. It lists the mistakes people actually make in their first weeks with Axolotl, each with the symptom you will see, the cause, and the fix, then the habits worth adopting on day one, then a short checklist you can paste into your notes. Everything refers to Axolotl 0.20.0 (30 September 2026). Keys are renamed often, so when a tutorial disagrees with this page, check the release notes and the config reference.

The mistakes beginners actually make

Following an old tutorial

Symptom. The install fails, or the config is rejected with a message about a deprecated or removed key, or the run starts and warns about several settings.

Cause. Axolotl changes quickly. Version 0.20 raised the minimum to Python 3.12 and PyTorch 2.13, removed FSDP1, and replaced the ring-flash-attn extra with ringmaster. Before that, type: sharegpt was removed, relora_steps was renamed jagged_restart_steps, and dataset_processes became dataset_num_proc. A tutorial more than a few months old probably uses at least one of these.

Fix. Check the date of anything you copy from. Start from the example configs that ship with your own release (axolotl fetch examples), and keep the config reference open. Read every warning the tool prints: a deprecation warning today can be an error in a later release.

Installing without --no-build-isolation

Symptom. The install stops with a long compilation error, often mentioning that torch cannot be found.

Cause. Some dependencies build code against the PyTorch already present, and build isolation hides it from them.

Fix. Use uv pip install --no-build-isolation 'axolotl[deepspeed]', in a fresh Python 3.12 environment. Quote the extras in brackets or your shell may treat them as a pattern. If the compiler complains about CUDA architectures on a very new GPU, check you are on CUDA 13.0, which Blackwell cards need.

Training before looking at the data

Symptom. Training completes and the loss looks healthy, but the model repeats the prompt, ignores its role, or never stops generating.

Cause. Masking or the chat template was not what you assumed. Perhaps the prompt was trained as well as the answer, perhaps the end-of-turn token was not trained, or perhaps the template used in training differs from the one at inference.

Fix. Run axolotl preprocess config.yml --debug and read several examples. Check which tokens show -100 and which carry labels. Confirm that the answer is trained and the instruction is not, and that the special tokens look right. Do this again every time you change the dataset type, the template, the special tokens or the trained roles.

Trusting the cache

Symptom. You corrected a typo in the dataset, or fixed a prompt format, and the model behaves as if you had not.

Cause. With dataset_prepared_path set explicitly, Axolotl reuses the cache on the next run. If the config did not change, it may not notice the data did.

Fix. Delete the prepared folder, for example rm -rf last_run_prepared, after any data change. Remember the opposite behaviour too: with the key left empty, the old cache in ./last_run_prepared/ is ignored and data is retokenized every run, which is slower but safe.

Skipping the smoke test

Symptom. A job that has been running for hours dies on a checkpoint write, a missing token, or a validation error that appears late.

Cause. Some failures only appear once a step runs, a checkpoint writes, or an evaluation starts.

Fix. Always begin with axolotl train config.yml --max-steps 5, and consider save_first_step: true so a checkpoint is written at once. Five steps cost seconds and have saved many nights of rented GPU time.

Misreading out-of-memory errors

Symptom. CUDA out of memory, or the process vanishing with exitcode: -9.

Cause. They are different problems. The first is GPU memory (VRAM), set by model size, method, micro-batch and sequence length. The second is system RAM, usually exhausted while tokenizing.

Fix. For VRAM, climb the ladder in this order, one rung at a time: lower micro_batch_size and raise gradient_accumulation_steps; set gradient_checkpointing: true; use QLoRA or 8-bit loading; shorten sequence_len; use attn_implementation: flash_attention_2. For RAM, lower dataset_num_proc or use less data. Shrinking the dataset does not help VRAM.

Setting QLoRA halfway

Symptom. "Require cfg.load_in_4bit to be True for qlora", or "Can't load qlora in 8bit".

Cause. QLoRA is an adapter on a 4-bit base, so it needs both lines.

Fix. Set adapter: qlora together with load_in_4bit: true, and do not add load_in_8bit. Remember bitsandbytes does not install on macOS, so QLoRA is for Linux GPUs.

Using type: sharegpt

Symptom. The run stops with "type: sharegpt.* is deprecated. Please use type: chat_template instead."

Cause. The old type was removed, but many posts still show it.

Fix. Use type: chat_template, field_messages: conversations and message_property_mappings with role: from and content: value. Then run preprocess --debug to confirm the roles were mapped as intended.

Forgetting the pad token

Symptom. "Asking to pad but the tokenizer does not have a padding token."

Cause. Training batches need padding, and many base models define no pad token.

Fix. Add special_tokens: {pad_token: "<|end_of_text|>"} or the end-of-sequence token your model uses. The quickstart config already has this.

Combining settings that clash

Symptom. A validation message such as "save_steps and saves_per_epoch are mutually exclusive and cannot be used together", or the same for eval_steps and evals_per_epoch.

Cause. Each pair expresses the same thing two ways, and Axolotl refuses to guess.

Fix. Keep one of each pair. The same logic applies to warmup_steps and warmup_ratio, which should not both be set, and to val_set_size together with test_datasets.

Using DeepSpeed on one GPU

Symptom. AttributeError: 'DummyOptim' object has no attribute 'step', or No module named 'mpi4py'.

Cause. A deepspeed: setting is present in a single-GPU run, often copied from a multi-GPU example.

Fix. Remove the deepspeed: key and any --deepspeed flag. Only reach for it when you really have several GPUs or a model too large for one.

Confusing training loss with quality

Symptom. The training loss looks great, but the model fails on new inputs.

Cause. Overfitting. The model memorised a small dataset.

Fix. Set val_set_size (0.1 is a fair start) so you get evaluation loss, and watch it. If training loss falls while evaluation loss rises, stop earlier, use fewer epochs, lower the learning rate, reduce the rank, or add data. Then test on prompts that look nothing like the training data.

Putting secrets in the YAML

Symptom. Nothing visible, until a token turns up in a public repository or on the Hub.

Cause. The config is copied into output_dir, and pushed with the model if you set hub_model_id. The CLI also loads a .env file automatically, which people then commit.

Fix. Log in with hf auth login or use the HF_TOKEN environment variable, and put tracker keys such as WANDB_API_KEY in the environment. Add .env and outputs/ to .gitignore.

Merging by hand

Symptom. A size mismatch error when loading or merging an adapter, or a merged model that behaves differently from the adapter.

Cause. If tokens were added during training, the embedding matrix grew, and a hand-written merge script may not account for it.

Fix. Use axolotl merge-lora config.yml --lora-model-dir DIR. If the config has load_in_4bit, do not use merge_lora: true in it (you would see "Can't merge qlora if loaded in 4bit"); run the merge command separately, with --dequant if you want a bf16 result.

Ignoring the learning rate

Symptom. The loss is flat from the first step, or it jumps to nan within a few steps.

Cause. The rate is far too low to move anything, or far too high. The project's stability guide gives 1e-4 to 3e-4 as the usual range for LoRA, and 1e-5 to 5e-5 for a full fine-tune. Copying a full fine-tune rate into a LoRA config, or the reverse, is a common slip.

Fix. Use the value from the same-method example config, keep warmup_ratio: 0.1 so early steps are gentle, and keep max_grad_norm: 1.0. Change the rate only by a factor of two or three between experiments.

Skipping the validation split

Symptom. You have training loss and nothing else, so you cannot tell learning from memorising.

Cause. val_set_size defaults to 0.0, so no data is held out unless you ask.

Fix. Set val_set_size: 0.1 and choose evals_per_epoch. If packing complains that the split is too small, set eval_sample_packing: false.

Habits worth adopting on day one

Start from a maintained example

The examples/ folder holds tested configs for many model families. Copy the nearest one and change only the dataset and the output path. A config assembled from three different blog posts mixes three different eras of key names.

Change one thing at a time

If you change the learning rate, the rank and the dataset together, you learn nothing from the result. Treat a fine-tune like an experiment: one hypothesis, one change, one comparison. The command-line override (--learning-rate 0.0001) makes this cheap, because you do not edit the file for a quick trial.

Spend your effort on data

The settings in a YAML file move quality a little; the data moves it a lot. Fifty carefully written examples in the exact format and tone you want will usually teach more than five thousand scraped ones. Read your data. Remove duplicates, wrong answers and examples whose style you would not want imitated. If you work in Arabic, check that a sample of your data tokenizes sensibly, because an inefficient tokenizer wastes your sequence budget.

Watch the right numbers

Learn four: training loss, evaluation loss, grad_norm and peak memory. A loss that falls and flattens is normal. Evaluation loss rising while training loss falls means overfitting. grad_norm mostly between 0.1 and 10 is healthy, and spikes above 100 mean instability. max_grad_norm: 1.0 clips damaging updates. If you rent GPUs, loss_watchdog_threshold and loss_watchdog_patience can end a diverging run on their own.

Rent wisely

If you do not own a GPU, rent one by the hour, but remember that a stopped idea still costs money while the machine runs. Do the CPU-only steps (writing configs, validating data, preprocess with CUDA_VISIBLE_DEVICES="") on your own machine. Smoke-test first. Save checkpoints somewhere that survives the machine: on a short-lived job, push to the Hub or mount storage, because a local disk may disappear when the job ends.

Keep notes that explain the run

Record the Axolotl version, the base model, the dataset version, the command and the final losses in a note beside each result. A month later, that note is the difference between a result you can reproduce and a number you cannot explain. If your fine-tune is later served somewhere, the same note tells operators what they are running.

Read the logs from the top

When something fails, read the first error, not the last. Later lines are usually consequences. Find which stage failed: config validation, data preparation, model loading or training. Then shrink the problem: a single GPU (CUDA_VISIBLE_DEVICES=0), dataset_num_proc: 1, a tiny model, max_steps: 1, micro_batch_size: 1, val_set_size: 0, sample_packing: false, and a cleared cache. Keep a copy of the output with AXOLOTL_LOG_LEVEL=DEBUG axolotl train config.yml 2>&1 | tee run.log.

Mind licences and gated models

Some models need you to accept a licence on the Hub page before downloading, and the licence follows your fine-tune. If you see "Error verifying HuggingFace token", log in, and then check you have accepted the terms for that specific model. Before you publish or deploy anything derived from a model, read the licence for use restrictions.

A starter checklist

Copy this into a note and tick it off for every run.

This is part two of three. It assumes you have already run a LoRA job from a YAML file, looked at a loss curve, and merged an adapter. Here we open the box: how a run is assembled, why configs fail in the ways they do, and how to spend GPU memory and time deliberately instead of by trial and error.

Where this picks up

Each section starts with the problem it solves, and most end with a Try it you can run in a few minutes. Everything here is checked against Axolotl v0.20.0, released on 2026-09-30. That release requires Python 3.12 or newer and torch 2.13 or newer, so any older tutorial that mentions Python 3.10 is out of date before you read the first line.

What actually happens when you run axolotl train

The problem this section solves is that a training job feels like one opaque command, so when it fails you do not know which stage failed. It is a pipeline of distinct stages, and each stage has its own failure signature.

1. The CLI starts. It loads a .env file if one exists, checks your Hugging Face token (unless HF_HUB_OFFLINE=1), and reads the config, which may be a local path or an HTTPS URL.
2. Validation. The YAML is parsed into a Pydantic model. Unknown combinations are rejected here, before any GPU is touched, with a ValueError that names the conflicting keys.
3. Dataset preparation. Rows are fetched, tokenized, labelled, optionally packed, and written as Arrow files into a prepared-dataset folder.
4. Launch. Unless you chose otherwise, axolotl train hands off to accelerate launch, which starts one process per GPU. You can pick --launcher torchrun or --launcher python instead. With use_ray: true, Ray Train starts the workers.
5. Model load. The base model is downloaded or opened, optionally quantized to 4 or 8 bit, wrapped in a PEFT adapter, and patched with kernels and attention implementations your config requested.
6. The training loop. This is a Hugging Face Trainer (or a TRL trainer for preference methods), so most knobs you know from transformers apply.
7. Checkpointing. Weights land in output_dir, and optionally on the Hub.

Because validation comes first and is cheap, the single most useful habit is to run a command that exercises stages 1 to 3 without stages 4 to 7. That is what axolotl preprocess does, and we use it heavily below.

There is no daemon, scheduler, web UI or API server in Axolotl. Each run is a batch job that starts, trains and exits. Anything that looks like a platform around it, such as queues, retries, quotas and dashboards, is something you add from outside. We come back to that in the production section.

The same YAML feeds every other command: evaluate, inference, merge-lora, quantize, export and lm-eval. That reuse is the design's main strength. It means the config you trained with is also the contract for how you evaluate and export, and it is why you should treat the YAML as the artifact worth versioning, more than any individual checkpoint.

The config as an interface you maintain

The problem: Axolotl is a fast-moving project, and its release notes routinely rename or remove keys. Releases arrived on 2026-01-30, 2026-03-06, 2026-04-02, 2026-06-03, 2026-07-17, 2026-09-10 and 2026-09-30, which is every three to seven weeks. A config that worked in the spring may refuse to load today. Mid-level competence is partly about making those breaks cheap.

Read the validator, not your memory. Validation lives in a Pydantic schema, so the error messages are precise. Dump the schema for the version you have installed:

axolotl config-schema --format yaml > schema.yml
axolotl config-schema --field sequence_len

Know the three failure behaviours. Deprecated keys fall into three classes, and telling them apart saves time. Some only warn and keep working, such as dpo_beta (use rl_beta) and evaluation_strategy (use eval_strategy). Some are accepted today but flagged for removal: dataset_processes logs "dataset_processes is deprecated and will be removed in a future version", and the replacement is dataset_num_proc. And some raise immediately, such as type: sharegpt, max_packed_sequence_len, a top-level rope_scaling, and in v0.20 anything that configures FSDP1.

Aliases exist. Four keys have a long internal name and a short name you write in YAML. You write model_config, model_kwargs, model_type and revision_of_model's alias model_revision; the schema stores them as overrides_of_model_config, overrides_of_model_kwargs, type_of_model and revision_of_model. If you grep logs for a name you did not write, this is why.

Override from the command line. Any config key can be supplied as --key-name value, and since v0.15 nested keys use dot notation. That is how you run a smoke test without editing the file:

axolotl train config.yml --max-steps 5 --output-dir ./outputs/smoke

Overrides are for experiments and sweeps. Anything you would want to reproduce belongs in the file, because the config is copied into the output directory and is the record of what ran.

Keep configs in git, with a version comment. A reliable convention is a header comment naming the Axolotl release the file was last validated against:

# validated against axolotl 0.20.0 on 2026-10-01
base_model: NousResearch/Llama-3.2-1B

When you upgrade, you run preprocess, read every warning, and update the comment. Warnings are the project telling you which key will break next quarter.

Datasets, templates and label masking

The problem: a model learns from whatever the loss is computed on. If masking is wrong, you train on the prompt, on padding, or on the wrong turn, and the loss curve still looks healthy. This is the most common source of silently bad fine-tunes, so it earns the most care.

Labels. Every token gets a label equal to its id, or -100, which the loss ignores. For instruction data, train_on_inputs: false (the default) masks the prompt so only the response is learned. For chat data you get finer control:

• roles_to_train defaults to ["assistant"].
• train_on_eos takes turn, all, last or none, and defaults to turn.
• eot_tokens lists tokens that end a turn, and train_on_eot controls whether they receive labels.
• Per-message training flags are available through message_field_training and message_field_training_detail, so a dataset can mark individual turns as trainable.

Choosing the template. chat_template can be tokenizer_default (the default, which errors if the tokenizer has no template), a built-in name such as chatml, llama3, gemma or phi_3, a fallback form tokenizer_default_fallback_chatml, or jinja together with chat_template_jinja. The fallback form is useful for a pipeline that fine-tunes several base models, since it uses the model's own template when it has one. The template you pick is written into the output tokenizer_config.json, which matters at serving time: the server must render prompts the way training did.

chat_template: tokenizer_default_fallback_chatml
datasets:
  - path: data/support_chats.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles_to_train: ["assistant"]
    train_on_eos: turn
special_tokens:
  eos_token: "<|im_end|>"

If your data uses the older ShareGPT shape, with conversations, from and value, the mapping changes but the type does not: field_messages: conversations and message_property_mappings: {role: from, content: value}. Writing type: sharegpt raises an error in v0.20.

The classic template errors, and what they mean.

• jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'content' means your property mapping names a key the rows do not have. Fix message_property_mappings.
• Could not find content start/end boundary for turn or EOS token ... not found in chat template means the EOS Axolotl expects differs from the one the template emits. Set special_tokens.eos_token to the template's turn-end token and list it in eot_tokens.
• EOT token ... is encoded as multiple tokens means the end-of-turn marker is not a single token. Prefer added_tokens_overrides: {id: "string"}, which repurposes a reserved added token, over adding new tokens.
• Conflict between train_on_eos and train_on_eot appears when EOS is also in eot_tokens and the two settings differ. Make them equal.

New tokens cost you more than you think. tokens: adds vocabulary. Axolotl expands the embedding matrix when the tokenizer is larger than the model, and never shrinks it unless shrink_embeddings: true. If you add tokens you also need lora_modules_to_save to include embed_tokens and lm_head, because frozen embeddings cannot learn the new rows. A vocabulary resized during training is also the cause of size-mismatch errors when you load an adapter later; merge with axolotl merge-lora rather than a hand-written script.

Look at the labels. The one check that replaces guesswork is the debug output of preprocessing:

axolotl preprocess config.yml --debug --debug-num-examples 3

It prints tokens as (label, token_id) pairs. Masked tokens show a label of -100. Read three examples with your own eyes: the system prompt and user turn should be masked, the assistant text should have real labels, and the end-of-turn token should have a label if train_on_eos says it should.

Data sources. path can be a Hub repo, a local file or directory, or a remote URL scheme: s3://, gs:// or gcs://, adl://, abfs:// or az://, oci://, and public https:// URLs. Credentials come from each cloud's standard chain. For employers in the Gulf or Egypt that must keep data in-region, this is the useful part: the dataset can live in a bucket in your own cloud region, and the training machine reads it directly, with no copy through the public Hub. Private Hub datasets need hf_use_auth_token: true.

The prepared-dataset cache and sample packing

The problem: tokenizing a large dataset is slow, and short samples waste compute when each is padded to a full row. Axolotl solves the first with a disk cache and the second with packing, and each has a rule that bites if you do not know it.

The cache. Tokenized data is stored as Arrow files under dataset_prepared_path, in a folder keyed by a hash of the relevant config. There are two behaviours, and they are opposites:

• If you leave the key empty, output goes to ./last_run_prepared/, but the existing cache there is ignored, so you retokenize every time.
• If you set it explicitly, for example dataset_prepared_path: last_run_prepared, the cache is reused.

Reuse is what you want for speed, and it is a hazard when you edit a custom prompt strategy, because the hash does not see changes inside your Python code. The symptom is training on stale data after you fixed the formatter. The cure is to delete the folder. Make that a reflex whenever you change a custom type:.

Packing. With sample_packing: true, several short samples are concatenated into a single row of sequence_len tokens. With flash attention or SDPA varlen, Axolotl passes cu_seqlens so samples inside one row do not attend to each other. The benefit is throughput: no tokens are spent on padding. Four consequences matter at this level:

1. Batches are no longer samples. A micro-batch of 2 means two packed rows, each holding many samples. Step counts per epoch drop accordingly.
2. Loss averaging changes. Per-token averaging across a packed row is not the same as per-sample averaging, so loss values are not comparable with an unpacked run.
3. pad_to_sequence_len defaults to true when packing. Rows have a fixed size, which also makes memory use predictable.
4. Evaluation has its own switch. eval_sample_packing controls it, and a tiny validation split with packing fails with "eval dataset split is too small for sample_packing. You should set eval_sample_packing: False in your config." Either turn it off or enlarge val_set_size.

Packing is not supported with KTO. And sequence_len defaults to 512, so a config that omits it silently drops or truncates long rows according to excess_length_strategy, which is drop unless you change it. Set it from your data's length distribution, not from habit.

Streaming for corpora too big to cache. pretraining_dataset: or streaming: true read data on the fly. This has hard rules: only one dataset, max_steps is mandatory because the length is unknown, val_set_size is rejected (use test_datasets or nothing), and axolotl preprocess does not apply, so you start directly with axolotl train. Running preprocess against a streaming config produces an IterableDataset error or KeyError: 'input_ids', which is the tool telling you that stage does not exist for this mode.

Spending a GPU memory budget deliberately

The problem: "CUDA out of memory" has a dozen possible fixes and people apply them randomly. Each fix trades something, and the right order is the one that costs you the least speed and quality first.

First, know where memory goes. Training holds four things: the model weights, the gradients for trainable parameters, the optimizer state, and activations. Adapters shrink the middle two, quantization shrinks the first, and checkpointing shrinks the last. The docs give rough guidance at short context and a micro-batch of 1 to 2: for a 7 to 8 billion parameter model, QLoRA needs about 10 to 14 GB, LoRA in bf16 about 16 to 24 GB, and a full bf16 fine-tune with AdamW 60 to 80 GB. For 70 to 72 billion, QLoRA needs 40 to 48 GB, LoRA in bf16 needs two 80 GB cards, and a full fine-tune needs four to eight. Use these to choose hardware before you rent it.

The order to apply levers, following the project's own advice:

1. Lower micro_batch_size and raise gradient_accumulation_steps to keep the effective batch. Effective batch is micro_batch_size × gradient_accumulation_steps × num_gpus. Do not set batch_size; it is derived.
2. Turn on gradient_checkpointing: true. It recomputes activations in the backward pass and costs roughly 30 percent in speed.
3. Move to 8-bit or to QLoRA (adapter: qlora with load_in_4bit: true).
4. Shorten sequence_len, or split long examples.
5. Use attn_implementation: flash_attention_2.
6. Shard with DeepSpeed ZeRO-3 or FSDP2, adding offload last.

Attention. Since v0.17 the legacy booleans (flash_attention: true, sdp_attention, xformers_attention, sage_attention) are soft-deprecated in favour of one key, attn_implementation. Valid values include eager, sdpa, flash_attention_2, flash_attention_3, flash_attention_4, flash_attention_torch, flex_attention, xformers, sage, fp8, or a Hub kernel path. Use the new key in every config you write.

Memory-saving plugins. Two are worth knowing. Cut Cross Entropy avoids materializing the full logits tensor, which is large for big vocabularies. Liger fuses several operations. Both are plugins, listed under plugins::

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

Also available are activation_offloading (values true, legacy, disk or hidden_states), selective_checkpointing, chunked_cross_entropy and tiled_mlp, the last useful for long context.

LoRA kernels turn on by themselves. For adapter: lora or qlora, the flags lora_mlp_kernel, lora_qkv_kernel and lora_o_kernel are switched on automatically, and the log says "Auto-enabling LoRA kernel optimizations for faster training." They are skipped when an rl: method is set, when trust_remote_code is set, for 8-bit LoRA, for unsupported models, or when you set a kernel flag yourself. If a run is slower than a colleague's, check whether one of those conditions turned the kernels off.

Checkpointing and DeepSpeed. With ZeRO-3 or with flex attention, non-reentrant checkpointing can raise a CheckpointError about tensor counts. The fix is gradient_checkpointing_kwargs: {use_reentrant: true}.

Optimizers, learning rates and reading a healthy run

The problem: most failed fine-tunes are not infrastructure failures, they are runs that quietly diverged or never learned, and the person only noticed when the evaluation was bad. You need ranges to compare against, and guards that stop a bad run early.

Defaults to know. The optimizer defaults to adamw_torch_fused, the scheduler to cosine, weight_decay to 0.0, bf16 to auto, and num_epochs to 1.0. optimizer accepts any Hugging Face optimizer name (adamw_8bit, paged_adamw_8bit, adafactor, lion_8bit, schedule_free_adamw) and a longer list of Axolotl-specific ones, among them optimi_adamw, muon, dion, came_pytorch, ademamix and sinkgd. The old ao_adamw_4bit name is deprecated in favour of adamw_torch_4bit. The practical advice is conservative: start with the default or adamw_8bit, and change optimizers only to solve a measured problem such as optimizer-state memory.

Learning-rate ranges. The project's training-stability guidance gives starting points: 1e-5 to 5e-5 for a full fine-tune, and 1e-4 to 3e-4 for LoRA, with lora_r between 8 and 32 and max_grad_norm: 1.0. A full fine-tune at a LoRA learning rate is the usual cause of an immediate blow-up, and a LoRA at a full fine-tune rate is the usual cause of a model that barely moves.

What healthy looks like. For chat fine-tuning, training loss is typically between 0.5 and 2.0 and grad_norm between 0.1 and 10. A grad_norm that spikes above 100 signals instability. Remember the packing caveat from earlier: compare loss only between runs with the same packing and template settings.

Guards. Two mechanisms stop wasted spending:

• loss_watchdog_threshold with loss_watchdog_patience aborts a run whose loss exceeds a threshold for a given number of steps.
• early_stopping_patience with load_best_model_at_end ends a run once evaluation stops improving, and returns the best checkpoint.

Evaluation cadence. Pick one of each pair: eval_steps or evals_per_epoch, and save_steps or saves_per_epoch. Setting both members of a pair fails with "save_steps and saves_per_epoch are mutually exclusive and cannot be used together." val_set_size holds out a fraction; test_datasets is the alternative and the two cannot be combined. generate_samples: true produces sample generations during training, which tells you more about a chat model than a loss number.

LoRA and QLoRA choices that change the result

The problem: adapters have many knobs, and defaults hide trade-offs. Knowing which knobs change quality, and which only change speed, keeps you from tuning the wrong one.

Where the adapter attaches. lora_target_linear: true targets every linear layer. The alternative is an explicit lora_target_modules list, such as the attention and MLP projections used in the quickstart. Targeting more modules raises capacity and memory together. lora_r sets rank and lora_alpha scales the update; a common pairing is alpha at twice the rank, as in the quickstart's 16 and 32. lora_dropout defaults to 0.0 once an adapter is set.

QLoRA. adapter: qlora quantizes the frozen base to 4-bit NF4 and trains a 16-bit adapter on top. It requires load_in_4bit: true; without it validation fails with "Require cfg.load_in_4bit to be True for qlora". Together with FSDP2 it lets a 70-billion-parameter model train across two 24 GB cards. For that case v0.20 added CPU-staged NF4 loading, and qlora_sharded_model_loading defaults on for FSDP2 QLoRA when cpu_ram_efficient_loading is set.

Merging. A merge folds the adapter into the base weights and writes {output_dir}/merged:

axolotl merge-lora config.yml --lora-model-dir ./outputs/lora-out
CUDA_VISIBLE_DEVICES="" axolotl merge-lora config.yml --lora-model-dir ./outputs/lora-out

The second form forces a CPU merge, which avoids needing a GPU big enough to hold the base in bf16. Do not put merge_lora: true in a config that also has load_in_4bit: true; the error is "Can't merge qlora if loaded in 4bit". Run merge-lora as a separate step, which handles quantized bases, and add --dequant if you want a bf16 result.

Multi-GPU training: DeepSpeed or FSDP2

The problem: a model, its gradients and its optimizer state do not fit on one card, or one card is too slow. Sharding spreads that state across GPUs. Axolotl supports DeepSpeed ZeRO and PyTorch FSDP2, and you pick one. Configuring both fails with "deepspeed and fsdp cannot be used together."

The ladder. Plain data parallelism (DDP) replicates the whole model on every GPU, so it speeds up training but saves no memory. DeepSpeed ZeRO stage 1 shards optimizer state, stage 2 adds gradients, and stage 3 adds parameters. FSDP2 shards parameters, gradients and optimizer state, and its reshard_after_forward setting decides whether parameters are freed after the forward pass: true behaves like ZeRO-3 and false like ZeRO-2. The guidance in the docs is to step up one stage at a time and to offload to CPU as little as possible, because offload costs speed.

DeepSpeed. Fetch the shipped JSON files, then reference one:

axolotl fetch deepspeed_configs

deepspeed: deepspeed_configs/zero2.json

Available files include zero1.json, zero2.json, zero3.json, zero3_bf16.json and two with CPU offload. Single-GPU runs must not set deepspeed:: leaving it in produces AttributeError: 'DummyOptim' object has no attribute 'step' or ModuleNotFoundError: No module named 'mpi4py'. An exitcode: -7 points to a DeepSpeed version problem; the deepspeed extra pins >=0.18.6,<0.20.0.

FSDP2. This is the only FSDP in v0.20. The FSDP1 implementation was removed, and fsdp_version: 1, the top-level fsdp: list, sharding_strategy and forward_prefetch now raise errors. A working block looks like this:

fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: LlamaDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true

Notice the keys have no fsdp_ prefix. Old configs wrote fsdp_state_dict_type and similar; v0.20 warns "Configuring FSDP fields with the fsdp_ prefix is deprecated" and you should drop the prefix. To find transformer_layer_cls_to_wrap, open the model's modeling_*.py and read _no_split_modules; for LlamaForCausalLM it is LlamaDecoderLayer. Other FSDP1-only keys (sync_module_states, use_orig_params, backward_prefetch, limit_all_gathers) are dropped with a warning, which is a sign the config predates v0.20.

Which one? FSDP2 is the docs' general recommendation and is the route for QLoRA at scale, for context parallelism and for the N-D parallelism features. DeepSpeed remains useful when you already have tuned JSON files or want its ZeRO offload variants. Teams migrating an old FSDP1 config should expect to rewrite it, not patch it.

Launchers. The default is accelerate. For direct control use torchrun, and pass launcher arguments after a double dash:

axolotl train config.yml --launcher torchrun -- --nproc_per_node=4 --nnodes=1

A stale accelerate config file at ~/.cache/huggingface/accelerate/default_config.yaml triggers a warning that it "can lead to unexpected errors", because it overrides what Axolotl sets up. Delete it unless you intentionally use accelerate for multi-node.

Multi-node. The recommended pattern is torchrun with a rendezvous endpoint:

axolotl train config.yml --launcher torchrun -- \
  --nnodes 2 --nproc_per_node 8 \
  --rdzv_id job42 --rdzv_backend c10d \
  --rdzv_endpoint "$HEAD_IP:29400"

Every node needs the same Axolotl commit and the same config, and the nodes must reach each other on that port. For InfiniBand fabrics, the NCCL doc names NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME and NCCL_BUFFSIZE. The alternatives are Ray Train (use_ray: true, with ray_num_workers) and SkyPilot's num_nodes. See the Ray guide if you already run a Ray cluster, and the DeepSpeed guide for ZeRO in more detail.

Preference training and GRPO

The problem: supervised fine-tuning teaches a model to imitate. To steer it toward preferred behaviour, or to optimize a measurable reward, you need preference or reinforcement learning. Axolotl exposes these through one key, rl:, and when it is absent the run is plain SFT.

Preference methods. rl: takes dpo, kto, orpo, simpo, grpo and ebft. The old ipo value is deprecated since v0.17: use rl: dpo with dpo_loss_type: ["ipo"]. It still runs but warns. Each method has its own dataset shape and options:

• DPO learns from chosen and rejected pairs. Key settings are rl_beta, dpo_loss_type (a list, default sigmoid), and dpo_loss_weights for mixing losses. Several older DPO keys (dpo_use_logits_to_keep, dpo_generate_during_eval, dpo_norm_loss, rpo_alpha) now raise, because the TRL version underneath dropped them.
• KTO learns from single examples labelled desirable or undesirable, with kto_desirable_weight. Packing is not supported.
• ORPO folds preference into SFT through orpo_alpha, and SimPO uses simpo_gamma and cpo_alpha.

Dataset types here are different from SFT. Names like chatml.intel, chatml.argilla, llama3.ultra and chat_template.default describe where the chosen and rejected fields live. With chat_template.default you name field_messages, field_chosen and field_rejected.

rl: dpo
rl_beta: 0.1
dpo_loss_type: ["sigmoid"]
datasets:
  - path: my_org/preferences
    split: train
    type: chat_template.default
    field_messages: prompt
    field_chosen: chosen
    field_rejected: rejected

Two memory levers matter in DPO: precompute_ref_log_probs computes reference-model log probabilities once up front, and rl_adapter_ref_model uses the adapter-disabled model as the reference so you do not load a second copy.

Check the field names against your data and TRL. Preference-data fields differ between sources, and a mismatch produces a confusing error late. Run axolotl preprocess config.yml --debug first, as you would for SFT.

GRPO. Group Relative Policy Optimization samples several completions per prompt, scores them with reward functions you write, and updates toward the better ones. In Axolotl it needs three things: a prompt dataset, reward functions in a Python module the process can import, and a vLLM server for fast generation. That second process means GRPO needs at least two GPUs, one to train and one to generate.

rl: grpo
vllm:
  host: 0.0.0.0
  port: 8000
  gpu_memory_utilization: 0.85
  max_model_len: 2048
trl:
  use_vllm: true
  vllm_mode: server
  vllm_server_host: 0.0.0.0
  vllm_server_port: 8000
  reward_funcs: [rewards.accuracy_reward, rewards.format_reward]
  num_generations: 16
  vllm_lora_sync: true

Start the two processes in separate terminals, giving the generation server the highest-index GPU:

CUDA_VISIBLE_DEVICES=1 axolotl vllm-serve config.yml
CUDA_VISIBLE_DEVICES=0 axolotl train config.yml

The traps are specific. The vLLM server must serve the same base model as training, or weight synchronization silently produces wrong results. The batch must divide evenly: a "generation batch size not divisible by num_generations" error means micro_batch_size times accumulation times data-parallel size must be a multiple of num_generations. unable to load strategy X followed by None is not callable means the reward module is not importable from the working directory, which you can check with python -c "import rewards". And a reward that stays at zero, or a reward_std that is always zero, means there is no learning signal: test your reward function by itself on a handful of known completions, and set trl.log_completions: true to read what the model actually produces.

Evaluating, quantizing and exporting

The problem: a training run ends with a loss number, but the thing you ship is a model that behaves well on your task and loads in your serving stack. Evaluation and export are where a fine-tune becomes something a team can use.

Three levels of evaluation. Use them in order of cost.

1. Loss on a held-out set. val_set_size or test_datasets give an evaluation loss during training, and axolotl evaluate config.yml computes loss on the train and eval sets afterwards. This tells you about overfitting and little else.
2. Generated samples. generate_samples: true, with num_generation_samples and generation_max_new_tokens, prints completions during training. The older eval_table_size and eval_max_new_tokens keys only warn now and point to these.
3. Benchmarks and task tests. axolotl lm-eval config.yml runs the EleutherAI harness through the LMEval plugin, with lm_eval_tasks and lm_eval_batch_size. Add the plugin under plugins: first. This is the check that catches a model that got better at your data and worse at general ability.

Beyond these, build a small fixed set of prompts from your real use case and run them after every candidate model. A fifty-prompt spreadsheet with a pass or fail column does more for release decisions than any single metric. Pair it with a judge or a tracker from the catalogue: Langfuse and MLflow both fit.

Quantization. axolotl quantize config.yml runs torchao post-training quantization and writes to {output_dir}/quantized. A quantization: block selects the scheme. Quantization-aware training uses a qat: block (weight_dtype, activation_dtype, group_size) so the model learns to tolerate low precision, then you run the same quantize command. The docs list NVFP4, MXFP4, FP8 and, since v0.19, ternary weights. The quantize doc says GPTQ and EXL2 formats are not supported.

GGUF export. New in v0.20, axolotl export produces GGUF files for llama.cpp-family runtimes. It needs a built llama.cpp, located through $LLAMA_CPP_DIR or --llama-cpp-dir, and it refuses adapter-only directories, so merge first:

axolotl merge-lora config.yml --lora-model-dir ./outputs/lora-out
axolotl export config.yml --quantize Q4_K_M,Q8_0 --outtype f16

The output feeds llama.cpp directly and Ollama through a Modelfile that points at the GGUF file. For GPU serving at scale, give vLLM the merged safetensors directory.

The serving contract. Whatever runtime you use, three things must match training: the chat template (it is saved in the output tokenizer_config.json), the special tokens (EOS and pad), and the base model's architecture. A model that scores well in axolotl inference and badly behind an API almost always has a template mismatch. Test the served endpoint with the same fixed prompts, not only the offline path.

Tracking, metrics and telemetry

The problem: when a run goes wrong at step 4,000, the console scrollback is not enough. You want curves, configs and artifacts attached to each run, and you want to know exactly what the tool itself reports about you.

Trackers. Axolotl logs to several, each enabled by its own keys:

• Weights & Biases: wandb_project, wandb_entity, wandb_name, and wandb_mode set to offline or disabled when you need it. See the Weights & Biases guide.
• MLflow: use_mlflow, mlflow_tracking_uri, mlflow_experiment_name, and hf_mlflow_log_artifacts.
• Comet (use_comet), Trackio (use_trackio) and TensorBoard (use_tensorboard), with SwanLab through its plugin.

Secrets stay out of the YAML. Use WANDB_API_KEY in the environment or a prior login, because the config is copied into output_dir and can be pushed to the Hub with the model.

OpenTelemetry metrics. Setting use_otel_metrics: true, with the axolotl[opentelemetry] extra installed, exposes a metrics endpoint at otel_metrics_host:otel_metrics_port. The default is localhost:8000, which is the same port the vLLM server uses by default, so change one of them when you run GRPO on the same host. This lets a Prometheus scrape see training metrics alongside the rest of your infrastructure.

Telemetry you should know about. Axolotl sends anonymous usage telemetry through PostHog, and it is on unless you opt out. It collects operating system, library versions, hardware details, runtime metrics, whitelisted model names, and sanitized error types and stack frames. Path-like values are redacted, and data is sent from rank 0 only. At startup, unless you set a preference explicitly, it pauses for ten seconds with a notice. Opt out with either of these:

export AXOLOTL_DO_NOT_TRACK=1
export DO_NOT_TRACK=1

Setting AXOLOTL_DO_NOT_TRACK=0 explicitly enables it and silences the notice. For regulated or air-gapped environments, which includes many banks and government workloads in the Gulf, combine the opt-out with HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1, and stage models and datasets locally beforehand. Put these in the container image or job template, not in someone's shell profile.

Containers, cloud launchers and secrets

The problem: a GPU setup that works on one engineer's machine is useless to a team. You need an environment that is identical everywhere, a way to launch it on rented hardware, and a clear policy for credentials.

Install for repeatability. The recommended Linux install uses uv with --no-build-isolation, and the guide extras are quoted for zsh:

uv venv --python 3.12
source .venv/bin/activate
uv pip install --no-build-isolation 'axolotl[deepspeed]==0.20.0'

Optional extras include flash-attn, vllm, ringmaster, ray, mlflow, opentelemetry, modal and baseten. Since v0.20 Modal and Baseten are extras, not defaults. On Blackwell cards (B100, B200, B300, RTX 50 series), use CUDA 13.0, because 12.8 cannot compile for the B300.

Containers. The project publishes images under axolotlai/axolotl. Since v0.18 only uv-based images exist; the plain names are the defaults and the -uv names remain as aliases. For a team, pin a release tag, not main-latest:

docker run --gpus '"all"' --rm -it --ipc=host \
  -v "$PWD":/workspace/project \
  -v "$HOME/.cache/huggingface":/root/.cache/huggingface \
  axolotlai/axolotl:0.20.0-py3.12-cu130-2.13.0

The --ipc=host flag matters, since PyTorch data loaders and NCCL use shared memory; the docs also suggest --shm-size 10g, --ulimit memlock=-1 and --ulimit stack=67108864 for demanding jobs. The axolotl-cloud image adds tmux, Jupyter and SSH, and it starts Jupyter by default. Set JUPYTER_PASSWORD or JUPYTER_DISABLE=1, or you have an open notebook server on a rented GPU. If you are new to image mechanics, the Docker guide covers volumes and caching.

Launching on rented compute. axolotl train config.yml --cloud cloud.yml sends the job to a provider. The provider: key is modal (the default), baseten, nebius, or a plugin-registered provider. Modal accepts gpu, gpu_count, memory, timeout, branch, docker_tag, volumes, env and secrets, where secrets is a list of environment variable names read from your host. Baseten and Nebius support training only, and Baseten rejects CLI overrides. Other routes the docs describe are SkyPilot (any cloud or Kubernetes, with spot instances and num_nodes), Hugging Face Jobs, and Docker-friendly providers such as RunPod and Vast.ai. HF Jobs bills per minute with a 30 minute default timeout, and the disk is lost on timeout, so push checkpoints to the Hub or mount storage.

Secrets. Rules that hold across all launchers:

• Tokens go in environment variables or the platform's secret store: HF_TOKEN for gated models, private datasets and Hub pushes; WANDB_API_KEY for W&B. Never in the YAML.
• .env files are loaded automatically by the CLI, so keep them out of version control and out of the image.
• Cloud storage credentials come from the usual chains: AWS environment variables or an instance role, with AWS_ENDPOINT_URL for S3-compatible stores; GCP metadata; Azure AZURE_STORAGE_* variables; OCI resource principals.

Upgrades, pinning and CI for a training repo

The problem: with a new release every few weeks and key renames in many of them, an unmanaged upgrade breaks a pipeline at the worst moment. You need a process that treats the tool version as a dependency with a test suite.

Pin exactly. Use axolotl==0.20.0 in requirements, or the container tag 0.20.0-py3.12-cu130-2.13.0. Only the latest release receives security fixes, so pinning is a trade between stability and patches; schedule a regular review. Vulnerabilities are reported by email to the address in the project's security policy, not as public issues.

A cheap CI job. You do not need GPUs to catch most upgrade breakage. A CI step can install the pinned version on a CPU runner and run validation and tokenization for every config in the repository:

for cfg in configs/*.yml; do
  CUDA_VISIBLE_DEVICES="" axolotl preprocess "$cfg" --debug-num-examples 1 \
    || { echo "FAILED: $cfg"; exit 1; }
done

This catches renamed keys, missing template fields and bad dataset mappings in minutes, long before a GPU is rented. Add a grep for the word "deprecated" in the output and fail on it if you want to enforce a clean config. Some configs (a GRPO job, say) cannot be preprocessed on CPU; mark them in a list and skip them.

Debugging beyond the basics

The problem: training failures present as one-line errors at the end of a long log, and the cause is often earlier or elsewhere. A routine beats intuition.

Shrink the failure first. The project's debugging guidance reduces a run to its smallest reproducer: one GPU with CUDA_VISIBLE_DEVICES=0, dataset_num_proc: 1, a tiny model, max_steps: 1, micro_batch_size: 1, val_set_size: 0, and sample_packing: false. Delete the prepared-dataset folder and the Hugging Face datasets cache so stale state cannot hide the bug. Then capture a full log:

AXOLOTL_LOG_LEVEL=DEBUG axolotl train config.yml 2>&1 | tee run.log

If the failure disappears in the reduced setup, add the features back one at a time until it returns. That bisection is faster than reading source code.

Hangs. A trainer that stops progressing for minutes with no error is usually GPU-to-GPU communication. Set NCCL_DEBUG=INFO and TORCH_DISTRIBUTED_DEBUG=INFO, check NVLink status, and look at whether one rank is far behind the others. With GRPO plus FSDP plus async prefetch, set async_prefetch: false.

Putting it all together

Here is the shape of a disciplined workflow for a team fine-tuning an assistant on internal support chats. The details are yours to change; the sequence is the point.

1. Pin the environment. A container tag such as 0.20.0-py3.12-cu130-2.13.0, with AXOLOTL_DO_NOT_TRACK=1 and offline variables set if your policy needs them.
2. Write the config for the target. QLoRA on a mid-size model, chat_template data, sample_packing, attn_implementation: flash_attention_2, the Liger plugin, an explicit dataset_prepared_path, and a tracker block. Add a version comment.
3. Validate on CPU in CI. The preprocessing loop runs on every pull request.
4. Inspect labels. preprocess --debug on three examples, reviewed by a person, with a screenshot in the pull request.
5. Smoke test. --max-steps 5 with save_first_step: true, on the real hardware.
6. Train with guards. A loss watchdog, an evaluation cadence, auto_resume_from_checkpoints: true, and checkpoints written to durable storage.
7. Evaluate in layers. Evaluation loss, sample generations, lm-eval on a small general benchmark, then the fixed prompt set from your real use case.
8. Merge and export. merge-lora on CPU, then safetensors for vLLM and GGUF for local use if needed.
9. Test the served endpoint. The same fixed prompts, through the real API, to catch a template mismatch.
10. Record provenance. Version, config hash, dataset version, base revision, commit.

A config that expresses steps 2 and 6 for a single node looks like this:

# validated against axolotl 0.20.0
base_model: NousResearch/Llama-3.2-1B
adapter: qlora
load_in_4bit: true
lora_r: 16
lora_alpha: 32
lora_target_linear: true

chat_template: tokenizer_default_fallback_chatml
datasets:
  - path: data/support_chats.jsonl
    type: chat_template
    field_messages: messages
    message_property_mappings: {role: role, content: content}
    roles_to_train: ["assistant"]
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
eval_sample_packing: false

sequence_len: 4096
sample_packing: true
attn_implementation: flash_attention_2
gradient_checkpointing: true
micro_batch_size: 2
gradient_accumulation_steps: 8
num_epochs: 2
learning_rate: 0.0002
optimizer: adamw_8bit
lr_scheduler: cosine
warmup_ratio: 0.1
max_grad_norm: 1.0
bf16: auto

evals_per_epoch: 4
saves_per_epoch: 2
auto_resume_from_checkpoints: true
loss_watchdog_threshold: 8.0
loss_watchdog_patience: 3
logging_steps: 1

output_dir: ./outputs/support-qlora
use_mlflow: true
mlflow_experiment_name: support-assistant

Where you are now

You can predict what Axolotl will do before it does it: validation first, then dataset preparation and its cache, then workers, model load and training. You can verify labels instead of trusting them, spend memory in a sensible order, choose between DeepSpeed and FSDP2, run preference and GRPO jobs without the usual traps, and evaluate and export a model in a way that survives contact with a serving stack. You also have a process for the project's fast release cadence: pin, validate on CPU, smoke-test, record provenance.

What this level leaves for the senior one is the platform view: running Axolotl for many teams, failure modes at scale such as which component breaks first, multi-tenant isolation, cost governance, the security review of plugins and reward code, and when a different tool is the right answer. For breadth around this guide, the PEFT and TRL guide covers the libraries Axolotl wraps, and the Unsloth guide covers a related approach to efficient fine-tuning.

Sources

• Axolotl documentation home
• Installation
• Getting started
• CLI reference
• Config reference
• Dataset formats
• Conversation dataset format
• Dataset loading
• Dataset preprocessing
• Streaming datasets
• Multipack and sample packing
• Batch size versus gradient accumulation
• Choosing a method
• LoRA optimizations
• Optimizers
• Inference
• Export
• Quantization
• Quantization-aware training
• Multi-GPU training
• Multi-node training
• N-D parallelism
• FSDP and QLoRA
• NCCL troubleshooting
• Ray integration
• RLHF and preference training
• GRPO
• vLLM serving
• Docker images
• Apple Silicon
• SkyPilot
• Hugging Face Jobs
• Telemetry
• FAQ
• Debugging
• Training stability
• Axolotl releases on GitHub
• Axolotl on PyPI

This is the mid-level interview file. At this level an interviewer is not checking whether you can name the commands. They want to hear how you reason when a training job misbehaves, how you weigh one approach against another, and whether you can say what you would measure before you change anything. Each answer below is written the way a strong candidate would speak it, followed by what separates it from a weak one.

How these interviews work

Mid-level questions come in three forms: a scenario ("a run crashed, what do you check first?"), a trade-off ("DeepSpeed or FSDP2?"), and a depth probe ("what exactly does sample packing do to the batch?"). In all three, the interviewer is listening for an ordered approach. Naming five possible causes in random order scores lower than naming the two most likely, saying how you would tell them apart, and saying what you would try first and why.

Two habits help throughout. First, state the version you are talking about, because Axolotl changes quickly: this guide is checked against v0.20.0, which needs Python 3.12 and torch 2.13 or newer. Second, say what you would verify rather than claiming certainty. "I would run preprocess with debug and read the labels" is a better answer than "the masking is probably fine".

Scenario questions

The training job was killed with exit code -9 after the data started processing. What do you do?

An exit code of -9 means the process received SIGKILL, and during dataset preparation the usual reason is the operating system's out-of-memory killer acting on host RAM, not GPU memory. Tokenizing a large dataset with many worker processes multiplies memory use, because each worker holds its own slice and the Arrow cache is built in memory before it is written.

I would confirm with the system log or by watching memory while preprocessing runs. Then I would lower dataset_num_proc, reduce the size of each shard, or split the work: run axolotl preprocess on a CPU-only machine with CUDA_VISIBLE_DEVICES="", with dataset_prepared_path set explicitly so the cache is reused, and copy the prepared folder to the GPU machine. That separation also saves money, because the expensive machine is not sitting idle while tokenization happens.

A weak answer says "reduce the batch size", which treats it as a GPU memory problem. A strong answer distinguishes host RAM from VRAM by the error: GPU memory problems print CUDA out of memory, while a bare kill with -9 is the host.

A run hits CUDA out of memory on the first step. Walk me through your order of fixes.

I would work from the cheapest trade to the most expensive. First, lower micro_batch_size and raise gradient_accumulation_steps so the effective batch stays the same, since effective batch is micro-batch times accumulation times the number of GPUs. Second, enable gradient_checkpointing, which costs roughly a third in speed. Third, move to 8-bit or QLoRA. Fourth, shorten sequence_len. Fifth, make sure I am using flash attention through attn_implementation: flash_attention_2. Sixth, shard with FSDP2 or DeepSpeed ZeRO-3, adding CPU offload only as a last resort because it slows everything.

Before any of that, I would ask a basic question: does the hardware fit the job at all? A full bf16 fine-tune of a 7 to 8 billion parameter model with AdamW wants roughly 60 to 80 GB, so no ordering of tweaks rescues it on a 24 GB card. Adapters and quantization change the arithmetic; tweaks do not.

If the OOM happens later in training and not on step one, I suspect a long example. With sample_packing rows are fixed in size so that is less likely; without packing, one very long sample sets the peak. I would check the length distribution and excess_length_strategy.

A weak answer lists levers with no order or cost. A strong answer orders them by what they sacrifice and mentions measuring memory/max_alloc, noting that it differs from nvidia-smi because the allocator caches memory.

The training loss looks great but the model gives poor answers. What are your hypotheses?

I would list them in order of how often they cause this, and say how to test each.

1. Masking is wrong. If the loss is computed on the prompt, or on the wrong role, loss can look low because prompts are easy to predict. Test: axolotl preprocess config.yml --debug and read the (label, token_id) pairs. User and system tokens should be -100.
2. Template mismatch between training and serving. The model was trained with one chat template and the server renders prompts with another. Test: compare the rendered prompt from the serving stack with the training prompt. The chosen template is saved in the output tokenizer_config.json.
3. Stale prepared data. An explicit dataset_prepared_path reuses its cache, and edits to a custom prompt strategy are not seen by the cache key. Test: delete the folder and re-run.
4. Overfitting or a learning rate that is too high. Test: look at evaluation loss against training loss and the sample generations. Learning rates above about 3e-4 for LoRA, or above 5e-5 for a full fine-tune, are suspicious.
5. Evaluation set leakage or a trivial task. Test: inspect the validation split.

A strong answer finishes by saying it would rank the hypotheses by cheapness to test, which puts masking first because it takes a minute. A weak answer jumps to "train longer" or "use a bigger model".

A multi-GPU job hangs for twenty minutes and then reports a collective operation timeout. What now?

The message Watchdog caught collective operation timeout with a 1,800,000 ms limit means one collective communication call ran past the 30 minute default. That is almost always a rank that stopped participating, or a slow or broken link between GPUs, not a modelling issue.

I would check, in order: whether all ranks started and are alive; the physical topology with nvidia-smi nvlink --status; and whether peer-to-peer traffic is behaving, trying NCCL_P2P_LEVEL=NVL, or a different level if the machine lacks NVLink. I would set NCCL_DEBUG=INFO and TORCH_DISTRIBUTED_DEBUG=INFO and rerun with fewer steps to get the diagnostic output. If one rank is much slower, I suspect a data problem such as one rank receiving a very long batch, or a failing GPU.

I would not simply raise ddp_timeout first. A bigger timeout hides the hang for longer. It is the right change only when a legitimate step, such as loading a very large model, takes longer than the limit. Since v0.19 that timeout is honoured during weight loading through AXOLOTL_NCCL_TIMEOUT, which is worth mentioning because it shows you read release notes.

Your team upgrades Axolotl and every config fails to validate. How do you handle it?

First, read the error rather than the whole log, because Pydantic validation names the exact key. Then check the release notes for the versions between the old and new one. In v0.20 the common causes are: FSDP1 keys (fsdp_version: 1, a top-level fsdp: list, sharding_strategy), which raise; fsdp_-prefixed keys inside fsdp_config, which warn; and the new Python 3.12 and torch 2.13 minimums, which fail at install time instead.

Earlier renames still bite configs that skipped versions: relora_steps became jagged_restart_steps in v0.17 with no compatibility shim, dataset_processes became dataset_num_proc in v0.15, and type: sharegpt raises in favour of chat_template.

To handle it as a team, not as a one-off, I would fix one config, put the migration into a note, and add a CI job that runs axolotl preprocess on CPU for every config. In future that job catches the break on the pull request that bumps the pin. I would also keep the version pinned exactly, and take example configs from the same release.

A weak answer is "downgrade". That is legitimate as a stopgap but leaves you on a version that no longer receives security fixes, since only the latest release is supported.

A GRPO run starts and the reward stays at zero. Where do you look?

A reward stuck at zero, or a reward_std that is always zero, means there is no learning signal. GRPO compares completions within a group, so if every completion gets the same score, the advantage is zero and nothing updates.

I would first test the reward function alone on hand-written good and bad completions, outside the trainer. Then I would turn on trl.log_completions: true and read what the model generates; often the output format never matches what the reward looks for, for example the reward expects an answer tag the model has never been shown. Remedies are a format reward that gives partial credit, a few SFT examples beforehand, or an easier prompt set so some completions succeed.

I would also check the plumbing: the vLLM server must serve the same base model as the trainer, or weight sync silently misbehaves; num_generations must divide the generation batch; and the reward module must be importable from the working directory. An error such as unable to load strategy X followed by None is not callable is the import problem.

Trade-off questions

LoRA, QLoRA or full fine-tuning?

The question is really about what you are trying to change and what hardware you have. LoRA trains small low-rank matrices on a frozen base. It uses far less memory than a full fine-tune, trains quickly, produces a small artifact, and is enough for most style, format and domain-adaptation tasks. QLoRA does the same on a 4-bit base, cutting weight memory again, at some cost in speed and a small quality risk from quantization. Full fine-tuning updates everything, which can teach more but needs far more memory, risks forgetting, and produces a full-size checkpoint.

For a 7 to 8 billion parameter model the rough memory guidance is 10 to 14 GB for QLoRA, 16 to 24 GB for LoRA in bf16, and 60 to 80 GB for a full fine-tune. My default is LoRA or QLoRA, and I move to full fine-tuning only when an adapter run with sensible rank and targets plateaus below what the task needs, and I have the hardware.

A strong answer adds that it depends on how many variants you maintain, since adapters are cheap to store and can be served on one base, and mentions evaluating the merged model, not just the adapter.

DeepSpeed ZeRO or FSDP2?

Both shard training state across GPUs, and you cannot use both in one run. FSDP2 is the only FSDP in v0.20, it is the project's general recommendation, and it composes with newer features such as context parallelism and QLoRA at scale. DeepSpeed is attractive when you already have tuned JSON configs, want its offload variants, or follow an existing team standard.

My approach is to start with the simplest stage that fits: ZeRO-1 or 2, or FSDP2 with reshard_after_forward, then move up only when memory demands it, since each step up adds communication. I try to avoid CPU offload unless I must, because it is slow.

Detail that earns credit: reshard_after_forward: true behaves like ZeRO-3 and false like ZeRO-2; DeepSpeed ZeRO-3 with non-reentrant checkpointing can raise a CheckpointError, fixed with use_reentrant: true; and migrating an old FSDP1 config is a rewrite because the keys lost their fsdp_ prefix and several were removed.

Sample packing on or off?

On, by default, for supervised data with many short samples, because it removes padding and raises throughput substantially. The cost is that a batch is no longer a set of samples, loss averaging per token differs from per-sample, and some features conflict: KTO does not support packing, and evaluation on a tiny split fails unless eval_sample_packing: false.

I would turn it off to debug, since it complicates reading individual examples, and for data where samples are already near the full sequence length, where packing buys nothing. I would never compare loss between a packed and an unpacked run and call one better.

Merge the adapter or serve it separately?

Merging produces one self-contained model that any runtime can load, which simplifies serving and distribution. Serving the adapter keeps one base in memory and swaps adapters, which is better when several teams or customers have their own fine-tunes. vLLM can serve LoRA adapters directly.

Either way, I test what I ship. If I trained on a 4-bit base and merge into bf16, I run my fixed prompts against the merged result. I also never merge inside a config that has load_in_4bit: true; Axolotl refuses with "Can't merge qlora if loaded in 4bit", and the right path is a separate axolotl merge-lora run.

Train in the cloud on a managed launcher or on your own cluster?

Managed launchers, such as Modal through --cloud, SkyPilot, Hugging Face Jobs or RunPod, remove cluster operations and let you pay per use, which suits bursty fine-tuning. Your own cluster, with Slurm or Kubernetes, suits sustained load, data that must not leave a network, and teams that already have the operations skills. For employers in the Gulf and Egypt, data residency often decides: if training data may not leave a regional boundary, check where the provider's machines and storage actually are, and consider reading data from a bucket in your own region.

I would also name the hidden costs: managed jobs may time out and lose their disk (HF Jobs has a 30 minute default timeout), so checkpoints must go to durable storage, and credentials must be passed as secrets, never baked into the config.

Depth probes

What does axolotl preprocess --debug give you that a training run does not?

It stops after validation and tokenization and prints each token with its label and id. You can see whether the prompt is masked, whether the end-of-turn token has a label, and whether special tokens are where you expect. A training run would show you only a loss number. I run it before every new dataset, after every template change, and after every upgrade, since masking fixes change loss values.

Why does an explicit dataset_prepared_path make caching both better and riskier?

With the key empty, output goes to ./last_run_prepared/ but the existing cache is ignored, so every run re-tokenizes. With the key set, the cache is reused, keyed by a hash of the relevant config. That is faster, but the hash cannot see changes inside custom Python, so editing a prompt strategy can silently reuse stale data. The fix is to delete the folder when custom code changes.

What changes about batch size when you add GPUs?

The effective batch is micro_batch_size × gradient_accumulation_steps × num_gpus. Doubling the GPUs with the same other settings doubles the effective batch, which changes the step count and may call for a different learning rate or accumulation. Axolotl derives batch_size itself, so you should not set it.

What does telemetry collect, and what do you do about it?

It is anonymous, on by default and opt-out. It sends operating system, library versions, hardware, runtime metrics, whitelisted model names and sanitized error information, with path-like values redacted, from rank 0. I set AXOLOTL_DO_NOT_TRACK=1 or DO_NOT_TRACK=1 in the job image, and for regulated work also run offline with HF_HUB_OFFLINE=1, after staging models and data.

What a strong answer covers that a weak one misses

Across these questions, the same differences recur.

• Order and cost. Strong answers sequence actions by cost and by likelihood. Weak ones list everything.
• A way to verify. Strong answers end each hypothesis with a command: preprocess --debug, nvidia-smi nvlink --status, a standalone call of the reward function. Weak ones assert.
• Version awareness. Strong answers mention the release: FSDP1 removal in v0.20, attn_implementation replacing the boolean flags, type: sharegpt raising. Weak ones teach advice from older tutorials.
• Separating host from device. Exit code -9 is host memory; CUDA out of memory is VRAM. Strong answers say which they are dealing with.
• Serving thinking. Strong answers connect training choices to the serving contract: the template, special tokens and the merge. Weak ones stop at the checkpoint.
• Security and secrets. Strong answers know that trust_remote_code, reward functions and plugins run arbitrary Python, and that tokens never belong in the YAML.
• Reproducibility. Strong answers pin versions, record provenance and keep configs in git.

Practice prompts to answer aloud

Try these with a timer, two minutes each, and speak in complete sentences.

1. Your colleague's QLoRA job uses 40 GB on a card where yours uses 22 GB with the same model. List what could differ.
2. You must fine-tune on 200,000 chat conversations stored in a regional cloud bucket. Describe the preprocessing and storage plan.
3. Validation loss falls for two epochs and then rises. What do you change, and how do you decide?
4. A teammate wants to enable DeepSpeed ZeRO-3 on a single GPU "to be safe". What do you say?
5. You inherit a config with fsdp_version: 1. Describe the migration and the verification.

For related material in the catalogue, see the PEFT and TRL guide for adapter and trainer concepts, the DeepSpeed guide for ZeRO, and the vLLM guide for serving.

This file collects the mistakes people make once they are past the beginner stage, the habits that pay off most, and recipes you can paste into a terminal. It is ordered by how often each item costs real time or money. Everything is checked against Axolotl v0.20.0.

Errors at this level

Beginner errors are about getting a job to start. Mid-level errors are subtler: the job starts, runs for hours, and the result is wrong or the cost is higher than it should be. Each entry gives the symptom, the cause and the fix.

Training on the prompt, not the answer

Symptom. Loss starts low and falls smoothly, yet the model parrots the user or ignores instructions. Cause. Labels are applied to prompt tokens. That happens when train_on_inputs is true, when roles_to_train includes roles you did not intend, or when a dataset type does not mask the way you assumed. Fix. Run axolotl preprocess config.yml --debug --debug-num-examples 3 and read the (label, token_id) pairs. Masked tokens show -100. Do this for every new dataset and after every template change, and keep the output in the pull request.

The end-of-turn token is wrong

Symptom. The model never stops generating, or errors such as Could not find content start/end boundary for turn or EOS token ... not found in chat template appear. Cause. The EOS the tokenizer reports differs from the token the chat template uses to end a turn. Fix. Set special_tokens.eos_token to the template's turn-end token, for example <|im_end|> for ChatML, and list it in eot_tokens. Keep train_on_eos and train_on_eot consistent, or you get the "Conflict between train_on_eos and train_on_eot" error. If the end token is split into several tokens, use added_tokens_overrides instead of adding new tokens.

Editing a prompt strategy and training on stale data

Symptom. You fixed a formatter in custom Python, but outputs behave as before. Cause. An explicit dataset_prepared_path is reused, and its hash cannot see changes inside your code. Fix. Delete the prepared-dataset folder after any custom-code change. Leaving the key empty avoids reuse but retokenizes every run, so use the explicit path for speed and the delete reflex for safety.

Old attention and FSDP keys

Symptom. Warnings about deprecated keys, or v0.20 errors such as "fsdp_version: 1 is no longer supported, FSDP1 has been removed from Axolotl". Cause. Configs copied from older blog posts. Since v0.17 the booleans flash_attention, sdp_attention, xformers_attention and sage_attention are soft-deprecated, and v0.20 removed FSDP1. Fix. Use attn_implementation: flash_attention_2 (or sdpa, xformers, sage). For FSDP use fsdp_version: 2 with an fsdp_config block whose keys have no fsdp_ prefix, and replace sharding_strategy with reshard_after_forward.

DeepSpeed left on in a single-GPU run

Symptom. AttributeError: 'DummyOptim' object has no attribute 'step', or ModuleNotFoundError: No module named 'mpi4py'. Cause. A deepspeed: key or --deepspeed flag in a run that has only one GPU. Fix. Remove it. If you see exitcode: -7 on multiple GPUs, update DeepSpeed within the range the extra pins, >=0.18.6,<0.20.0.

Two cadence keys set at once

Symptom. "save_steps and saves_per_epoch are mutually exclusive and cannot be used together", and the same for eval_steps and evals_per_epoch. Cause. Both members of a pair are present, often one from an example file and one you added. Fix. Keep one of each. Also remember warmup_steps and warmup_ratio are alternatives, and val_set_size cannot be combined with test_datasets.

Packing breaks evaluation

Symptom. "eval dataset split is too small for sample_packing". Cause. A tiny validation split with sample_packing: true. Fix. Set eval_sample_packing: false, or raise val_set_size. Also remember that KTO does not support packing.

Streaming without a step count

Symptom. "max_steps must be set when using streaming datasets", or a KeyError: 'input_ids' from preprocess. Cause. Streaming and pretraining_dataset have no known length, and there is no preprocessing stage for them. Fix. Set max_steps, use test_datasets or no evaluation instead of val_set_size, and skip preprocess.

Merge fails or gives size mismatches

Symptom. "Can't merge qlora if loaded in 4bit", or a torch.Size mismatch when loading an adapter. Cause. merge_lora: true in a config that also loads the base in 4 bit, or a vocabulary that was resized during training (new tokens). Fix. Run axolotl merge-lora as its own step; it handles quantized bases, and --dequant produces bf16. Add lora_modules_to_save for embed_tokens and lm_head when you add tokens, and avoid hand-written merge scripts.

GRPO batch arithmetic and silent mismatches

Symptom. "generation batch size not divisible by num_generations", or rewards that never move. Cause. micro_batch_size × gradient_accumulation_steps × data-parallel size must be a multiple of num_generations. And the vLLM server must serve the same base model as the trainer, or weight sync silently misbehaves. Fix. Adjust the batch settings, verify the model names match, and test the reward function by itself.

Telemetry and ports colliding

Symptom. A GRPO run on one host fails to bind a port after you enabled OpenTelemetry metrics. Cause. The metrics endpoint defaults to localhost:8000, the same port as the vLLM server. Fix. Change otel_metrics_port or vllm.port.

The practices that pay off most, in order of leverage

These are ordered by the ratio of time saved to effort spent.

1. Run preprocess --debug on every dataset change

It costs a minute and prevents the most expensive failure, a long run on mis-labelled data. Read three examples. Check the prompt is masked, the answer is labelled and the end token is handled the way you intend.

2. Smoke-test on the target hardware

Run --max-steps 5 with save_first_step: true before every long job. It checks the data path, model load, a real optimizer step and the checkpoint write. It also reveals memory problems on step one, not hour three.

3. Pin versions and validate configs on CPU in CI

Pin axolotl==0.20.0 or the container tag 0.20.0-py3.12-cu130-2.13.0, not main-latest. In CI, run axolotl preprocess on every config with CUDA_VISIBLE_DEVICES="". Release notes in this project routinely include renames and removals, and CI turns each into a failing pull request instead of a failed GPU job.

4. Spend memory in order

Micro-batch, then gradient checkpointing, then quantization, then sequence length, then flash attention, then sharding, with offload last. Measure memory/max_alloc at each step so you know what each lever bought.

5. Keep a fixed prompt set

Write fifty representative prompts from the real use case, with a pass or fail note for each. Run them against every candidate, and against the served endpoint. This catches template mismatches that no loss number reveals.

6. Put guards on long runs

Add loss_watchdog_threshold and loss_watchdog_patience to abort divergence, auto_resume_from_checkpoints: true to survive interruptions, and durable storage for output_dir. On spot instances, do not use save_only_model: true, because it disables resuming.

7. Keep secrets out of the config

The YAML is copied to the output directory and can be pushed to the Hub. Use HF_TOKEN, WANDB_API_KEY and platform secrets. The CLI loads .env files automatically, so keep them out of git and out of images.

8. Opt out of telemetry where policy requires it

Set AXOLOTL_DO_NOT_TRACK=1 in the image, and for restricted environments add HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1.

9. Record provenance

For every shipped model, keep the Axolotl version, config hash, dataset version, base model revision and the commit of any custom code. It makes a regression six weeks later reproducible.

10. Take examples from your own release

Run axolotl fetch examples with the version you pinned. Examples from main may use keys your version lacks.

Practice cards

Each card is a short drill. Do them on a small model so a run takes minutes.

Operational recipes

These recipes use only commands and keys that exist in v0.20.0. Replace paths and names.

Recipe: install a pinned environment with uv.

uv venv --python 3.12
source .venv/bin/activate
export UV_TORCH_BACKEND=cu130
uv pip install --no-build-isolation 'axolotl[deepspeed]==0.20.0'
axolotl --version
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Recipe: a five-step smoke test.

axolotl train config.yml --max-steps 5 --output-dir ./outputs/smoke

Add save_first_step: true to the file if you want to see a checkpoint written. Delete ./outputs/smoke afterwards.

Recipe: tokenize on a CPU machine, train on a GPU machine.

CUDA_VISIBLE_DEVICES="" axolotl preprocess config.yml
rsync -a last_run_prepared/ gpu-box:/workspace/project/last_run_prepared/

This works when dataset_prepared_path is set explicitly in the config, because the cache is reused only then. Keep the same config and the same Axolotl version on both machines, since the cache is keyed by a hash of the configuration.

Recipe: a reproducible debug run.

rm -rf last_run_prepared
CUDA_VISIBLE_DEVICES=0 AXOLOTL_LOG_LEVEL=DEBUG \
  axolotl train debug.yml 2>&1 | tee run.log

Where debug.yml sets dataset_num_proc: 1, max_steps: 1, micro_batch_size: 1, val_set_size: 0 and sample_packing: false, with a tiny model.

Recipe: two-GPU torchrun launch.

axolotl train config.yml --launcher torchrun -- --nproc_per_node=2 --nnodes=1

Recipe: a CPU merge and a quick comparison.

CUDA_VISIBLE_DEVICES="" axolotl merge-lora config.yml --lora-model-dir ./outputs/lora-out
axolotl inference config.yml --lora-model-dir ./outputs/lora-out --chat

The merged weights are written to ./outputs/lora-out/merged when output_dir is ./outputs/lora-out. Run the same fixed prompts against the merged directory.

Recipe: GGUF export for local use.

axolotl export config.yml --quantize Q4_K_M --outtype f16

This needs a built llama.cpp, found through $LLAMA_CPP_DIR, and a merged model directory, not an adapter-only one. See the llama.cpp guide for running the result.

Recipe: a GRPO pair of terminals.

CUDA_VISIBLE_DEVICES=1 axolotl vllm-serve config.yml
CUDA_VISIBLE_DEVICES=0 axolotl train config.yml

Check that the server host and port in the vllm: and trl: blocks agree, and that the reward module imports from the directory you launch in.

Recipe: an offline, no-telemetry environment.

export AXOLOTL_DO_NOT_TRACK=1
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export HF_DATASETS_OFFLINE=1

Stage the base model and datasets before you set these, since nothing will be downloaded afterwards.

Recipe: a CPU validation loop for CI.

for cfg in configs/*.yml; do
  CUDA_VISIBLE_DEVICES="" axolotl preprocess "$cfg" --debug-num-examples 1 \
    || { echo "FAILED: $cfg"; exit 1; }
done

Skip configs that cannot be preprocessed on CPU, such as streaming or GRPO jobs.

Recipe: a container run with sensible flags.

docker run --gpus '"all"' --rm -it --ipc=host --shm-size 10g \
  -v "$PWD":/workspace/project \
  -v "$HOME/.cache/huggingface":/root/.cache/huggingface \
  axolotlai/axolotl:0.20.0-py3.12-cu130-2.13.0

If you use the cloud image, set JUPYTER_PASSWORD or JUPYTER_DISABLE=1 so you do not expose an open notebook server.

Working with data well

Most of the quality in a fine-tune comes from the data, and most data mistakes are cheap to prevent. Before you tune any hyperparameter, spend an hour with the examples. Read fifty rows at random, not the first fifty, because files are often sorted by source or by date and the head of a file is rarely representative. Look for duplicates, truncated answers, boilerplate that appears in nearly every response, and rows where the assistant turn is empty or only a refusal. A model learns exactly what the data shows, including the bad habits.

Check lengths before you choose sequence_len. The default is 512 tokens, and rows beyond the limit are dropped unless you change excess_length_strategy. If a fifth of your conversations are longer than the limit and you did not notice, you have quietly trained on the short, easy fifth. Tokenize a sample, print the percentiles, and choose a length that covers the ninety-fifth percentile or deliberately accept the loss. Setting excess_length_strategy: raise in a pilot run turns the silent drop into a visible error.

Use dataset_exact_deduplication when you merge several sources, and use the per-dataset weight key from v0.20 when one source is much larger than the others. A weight between zero and one down-samples a source without editing files, so the mixing ratio lives in the config where reviewers can see it.

Keep a held-out set that never changes. val_set_size splits the training data, and if you regenerate the data between runs the split changes with it, which makes runs hard to compare. For anything you will compare across weeks, build a separate fixed evaluation file and point test_datasets at it. Remember that val_set_size and test_datasets cannot be combined.

If your data contains personal information, deal with it before it reaches the tokenizer. A prepared cache folder and a tracker upload are both copies of that data. For employers in the Gulf and Egypt that handle regulated customer records, keep the raw files in a bucket inside your own cloud region, read them with the s3://, gs://, abfs:// or oci:// path schemes, and keep the prepared cache on encrypted storage that is deleted when the project ends.

Reading a run as it happens

A training log is a stream of evidence, and a few habits turn it into early warning. Watch the first ten steps closely. The loss on step one tells you whether the model and template match the data: a loss far above the typical 0.5 to 2.0 range for chat data often means the template or the masking is wrong. If the loss on step one is near zero, suspect that almost everything is masked.

Watch grad_norm. Healthy values sit between about 0.1 and 10. Occasional bumps are normal. A value above 100, or a steady climb, means instability, and the usual causes are a learning rate that is too high, a bad batch, or a precision problem. With fp8: true, a loss of NaN points to a zero scale, and the training-stability documentation is the place to go.

Watch throughput. The logged tokens per second and train_per_sec_per_gpu should be steady. A sudden drop on a multi-GPU job points to one slow rank, a thermal problem or a noisy neighbour on shared hardware. If throughput is lower than a colleague's on the same model, check whether the automatic LoRA kernels were disabled: they are skipped for 8-bit LoRA, for trust_remote_code models, for unsupported architectures, and whenever an rl: method is set.

Watch memory across steps, not just on step one. The logged memory/max_reserved is higher than memory/max_alloc, and both differ from what nvidia-smi shows, because the torch allocator caches memory. Do not conclude a leak from a single number; look at the trend.

Finally, read the generated samples if you enabled generate_samples: true. A fluent but wrong answer at step 200 is information that a loss curve cannot give you. Write down what you see at a few checkpoints, because a model whose loss improves while its samples get worse is the classic sign of overfitting or mis-masked data.

Keeping the cost down

GPU hours dominate the bill, so the cheapest improvement is not spending them on mistakes. The practices above, smoke tests, CPU preprocessing and a CPU validation job in CI, already remove most waste. Beyond those, several levers matter.

Choose the smallest method that does the job. A LoRA run that fits on one 24 GB card costs a fraction of a full fine-tune across several larger ones, and for many formatting and domain tasks the quality is similar. Try an adapter first and escalate only when it plateaus below what you need.

Turn on the speed features that cost nothing. Sample packing raises throughput substantially on short samples. Flash attention reduces memory and time. The LoRA kernels turn on automatically when conditions allow. The Liger plugin and Cut Cross Entropy reduce memory pressure so you can use a larger micro-batch, which keeps the GPU busier.

Avoid idle machines. Tokenize on a CPU box. Keep base models in a shared cache or a volume so each job does not download them again. On managed platforms, check the default timeout before you start, because a job killed at the limit can lose its disk; Hugging Face Jobs, for example, defaults to 30 minutes. Push checkpoints to the Hub or to mounted storage as you go.

Use spot or preemptible capacity for long runs only if resumption is set up. With auto_resume_from_checkpoints: true and checkpoints on durable storage, an interrupted job restarts from its last save, and SkyPilot can manage spot instances across clouds. Without resumption, spot pricing is a false saving.

Stop bad runs early. A loss watchdog ends divergence, and early stopping ends a run whose evaluation has plateaued. A job that diverged in the first hour and ran for twelve is the most common avoidable cost.

Reviewing a config in a pull request

Treat the YAML as code and review it as code. A reviewer who knows what to look for catches problems in minutes. Use this order.

First, the identity of the run: the base_model and, if you use it, revision_of_model, so the base cannot change underneath you. Next, the dataset block: paths, type, field mappings and roles_to_train, with a debug preprocessing excerpt attached to the pull request. Then the template and special tokens, since these must match how the model will be served.

After that, the hardware assumptions: sequence_len, micro_batch_size, accumulation steps, the attention implementation and the sharding choice, with a note of the GPU count and memory the author tested on. Check that deprecated keys are absent, which the warnings in a preprocessing run will show.

Then the safety net: a watchdog, an evaluation cadence, resume settings and an output_dir on durable storage. Finally, the hygiene items: no secrets in the file, no trust_remote_code: true unless it is justified and pinned, no remote config URL that points at a moving branch, and a header comment with the validated version.

A short list of review questions helps. Could I reproduce this run from this file and a commit hash? Would a different engineer know why learning_rate has this value? What happens to this config on the next release? If the answers are no, ask for a comment or a smaller change.

Habits for teams

When several people fine-tune on shared hardware, small conventions prevent large collisions. Give every run its own output_dir that includes the date and a short name, and never let two jobs write to the same folder. Give each project its own dataset_prepared_path, because a shared cache with stale data is a hard bug to find. If people share one Hugging Face cache, agree on who downloads gated models, since the licence must be accepted for the token that downloads them.

Write the decision down. A short note beside each config, saying why the rank is 16, why the learning rate is 2e-4, and what you tried before, saves the next person days. The tracker holds the numbers and the note holds the reasons.

Share failures. A one-line entry in a team channel, "a stale prepared cache cost us a day", spreads the lesson further than a wiki page no one reads. The same applies to upgrade notes: when one person migrates a config to a new release, they should publish the diff.

Finally, rehearse the restore. At least once, take a checkpoint, delete the local state, and resume from it on a clean machine. Teams that have never done this find out during an outage that their checkpoints were incomplete or that the config needed a file that was never committed.

A pre-flight checklist

Before you start a long job, tick every item.

1. The Axolotl version is pinned and the config carries a validated-against comment.
2. preprocess --debug was reviewed by a person, and masking and the end token are right.
3. dataset_prepared_path is deliberate, and the folder was deleted if custom code changed.
4. sequence_len was chosen from the data's length distribution.
5. A five-step smoke test passed on the target hardware.
6. A loss watchdog, evaluation cadence and auto-resume are set, and output_dir is on durable storage.
7. A tracker is configured, and no secret appears in the YAML.
8. Telemetry policy is applied.
9. The fixed prompt set and the serving template are ready for the evaluation afterwards.

For more on the libraries underneath, see the PEFT and TRL guide, and for experiment tracking the MLflow guide.

This is part three of three. It takes every topic from the earlier levels one step further, to the point where the question is no longer "does my run train?" but "what happens when forty people I have never met run this on hardware I pay for?" It is written against Axolotl 0.20, released on 30 September 2026.

Up to here the questions were about getting one job to finish. From now on they are different: which part of the pipeline fails first when volume grows, what a config file is allowed to execute, how a rename in a release note becomes a broken Monday for another team, and how you prove to leadership that the spend produced a model worth serving.

Where this picks up

I am starting with architecture, because every operational decision in this level follows from one fact: Axolotl has no server.

Architecture and failure modes

Axolotl is a single-job batch trainer. There is no daemon, no scheduler, no API server and no user interface. A run is a process tree that starts, does its work and exits. That is a feature when you own a platform, because there is nothing to keep alive, patch or secure between jobs. It is also a constraint, because every property you expect from a platform (queueing, quotas, identity, audit, retries) has to come from somewhere else.

Each run walks the same stages, and each stage has its own characteristic failure. Knowing the stage is most of the diagnosis.

The first stage fails fast and cheaply. Configuration is validated by a Pydantic schema, so an invalid combination such as adapter: qlora without load_in_4bit: true, or deepspeed: together with FSDP settings, stops the run with a ValueError before any GPU is touched. For a platform owner this is the most valuable property the tool has, and the section on config contracts builds on it.

The second stage is where host memory gives out. Tokenization is CPU and RAM work, and the symptom of running out is a bare exitcode: -9, which is the kernel's OOM killer, not a Python error. Because this stage can run on a machine with no GPU (setting CUDA_VISIBLE_DEVICES="" forces that), the correct platform design runs it separately and cheaply. We will return to this.

The third stage is where VRAM and download problems appear, along with every compatibility issue between the pinned libraries. Axolotl wraps transformers, peft, trl, accelerate, datasets, DeepSpeed and PyTorch FSDP2, and v0.20.0 pins exact versions of several of them. When a model loads in one release and not the next, the cause is almost always in that pinned stack, not in Axolotl's own code.

The fourth stage is where distributed failures live: an NCCL collective that exceeds its timeout, a rank that dies and takes the others with it, a node whose network interface was chosen wrongly. These are slow to surface, because a hung collective can sit for the default 30 minutes before the watchdog speaks. The fifth stage fails in a different way: quietly. A checkpoint written to a disk that vanishes with the pod, or a Hub push with an expired token after eleven hours, is a failure you discover at the worst moment.

Finally, GRPO adds a second long-running process, a vLLM server that the trainer talks to over HTTP, with optional weight synchronization over NCCL. The moment you offer GRPO to other teams you are operating a small distributed system, with all the failure modes that implies.

The config as a contract

When other teams own the data and you own the platform, the YAML file is the interface between you. Treat it the way you would treat an API, because that is what it is: a typed input with versioning, defaults, deprecations and validation, and a consumer base that will resent surprises.

The schema is available to you directly. axolotl config-schema prints the JSON schema, with --format yaml and --field NAME to narrow it. That output is machine-readable, which means you can build the checks a platform needs without parsing documentation: a pre-submission linter that rejects configs failing validation, a diff of the schema between releases to see which keys appeared or disappeared, and generated documentation for your own users that lists only the keys your platform allows.

Do not rely on defaults you have not read. Several are surprising in a multi-team setting. sequence_len defaults to 512, so a team that forgets it trains on truncated data (rows over the limit are dropped by default under excess_length_strategy: drop, so the dataset also quietly shrinks). val_set_size defaults to 0.0, so a run with no held-out data produces no evaluation signal. output_dir defaults to ./model-out, so two runs launched from the same working directory write over each other. fsdp_version now defaults to 2. A platform template should set every one of these explicitly, and your validation layer should reject configs that rely on the defaults for the keys that matter.

Deprecations are the other half of the contract. The project removes and renames keys faster than most infrastructure you run. A few that matter to teams carrying older configs:

Notice the three behaviours: some keys raise and stop the run, some warn and continue, and some have no shim at all. A warning in a log nobody reads is how a team ends up on a deprecated path for six months and then hits the removal. Your platform should promote selected warnings to failures in CI, and should list the warning text of every submitted job somewhere searchable.

Scaling: which component breaks first

Ask "what breaks first?" about each resource in order, because the answer changes with scale and tends to be the cheapest resource, not the most expensive one.

Host RAM breaks first on large datasets. Tokenization with many processes multiplies memory, and the cheap fix is to lower dataset_num_proc, shard the data (preprocess_shards, shards), or move preparation to a larger CPU-only machine. Streaming datasets avoid the memory cost, but the constraints are real: streaming supports one dataset only, max_steps becomes mandatory, val_set_size is not supported, and axolotl preprocess does not work with pretraining_dataset. If a team's corpus is large enough to need streaming, they are doing continued pretraining, and you should treat it as a different workload class with its own template.

VRAM breaks next, and the docs give a useful planning table. At short context and micro-batch 1 to 2, a 7 to 8B model needs roughly 10 to 14 GB for QLoRA, 16 to 24 GB for bf16 LoRA, and 60 to 80 GB for a full fine-tune with AdamW. At 70 to 72B the figures are 40 to 48 GB for QLoRA, two 80 GB GPUs for LoRA, and four to eight 80 GB GPUs for a full fine-tune. Treat these as planning numbers, not guarantees; long context, large vocabularies and MoE models move them. The ordered list for an out-of-memory error stays the same at every level: lower micro_batch_size and raise gradient_accumulation_steps, enable gradient checkpointing (costing about 30 percent of speed), move to QLoRA or 8-bit, shorten sequence_len, use flash attention, and only then reach for ZeRO-3 offload. Platform owners should encode that ladder into a runbook so that a team does not escalate to a bigger instance when a config change would do.

Interconnect breaks at multi-node scale. Within a node, NVLink makes tensor parallelism viable. Across nodes it is the network, and the docs say tensor parallelism is not recommended across nodes. FSDP communicates heavily; DDP and HSDP communicate less across the slow link. When throughput per GPU drops as you add nodes, the usual cause is not Axolotl but the NIC, the NCCL interface selection, or an InfiniBand setting. The environment variables worth standardizing in your base image are NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME and NCCL_BUFFSIZE, with NCCL_DEBUG=INFO available when you need evidence.

Choosing a parallelism policy

At the platform level you do not choose parallelism per job; you choose a small number of supported shapes and make everything else an exception. The v0.20 release helps you here, because it removed an entire branch of the decision tree.

FSDP1 is gone. fsdp_version defaults to 2, and FSDP2 is the only supported version. A config that sets fsdp_version: 1, uses the top-level fsdp: list, or sets sharding_strategy or forward_prefetch now raises; keys like sync_module_states, use_orig_params, backward_prefetch, backward_prefetch_policy and limit_all_gathers are dropped with a warning. The mapping that teams need is mechanical: fsdp_sharding_strategy becomes reshard_after_forward, and the fsdp_ prefix goes away from the rest. A minimal modern block looks like this.

fsdp_version: 2
fsdp_config:
  offload_params: false
  cpu_ram_efficient_loading: true
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: LlamaDecoderLayer
  state_dict_type: FULL_STATE_DICT
  reshard_after_forward: true

reshard_after_forward: true behaves like ZeRO-3 and false like ZeRO-2, trading memory for communication. The wrap class comes from the model: find _no_split_modules in the model's modeling_*.py, for example LlamaDecoderLayer for LlamaForCausalLM. Getting this wrong is a common reason a team reports that FSDP "does nothing", so put the per-model values into your template library rather than leaving them to each team.

DeepSpeed remains available, with shipped ZeRO configs from zero1.json through zero3_bf16_cpuoffload_all.json, and the guidance is to offload as little as possible and step up from stage 1 to 2 to 3. DeepSpeed and FSDP are mutually exclusive: setting both fails with "deepspeed and fsdp cannot be used together." A sensible policy is FSDP2 as the default and DeepSpeed as a supported alternative for teams with existing ZeRO configurations. Maintaining both for new work doubles your support burden for no gain. If you need to keep either one honest, read the DeepSpeed guide for what ZeRO actually partitions.

Beyond data parallelism, the N-D keys compose: dp_shard_size and dp_replicate_size for HSDP, tensor_parallel_size (marked experimental), context_parallel_size for long sequences, and expert_parallel_size for MoE models. Three constraints belong in your validation layer. The context-parallel degree must divide the GPU count. Expert parallelism now requires fsdp_version: 2 with fsdp_config.state_dict_type: FULL_STATE_DICT, and combining it with tensor parallelism is rejected. DeepSpeed composes only with tensor parallelism. Context parallelism moved to a built-in implementation in 0.20, with the ringmaster extra replacing ring-flash-attn; the old ring_attn_func and heads_k_stride keys are ignored with a warning.

For multi-node, the documented recommendation is the torchrun launcher. Every node needs the same Axolotl commit and the same config, and a mismatch is a classic source of a hang that looks like a network fault.

axolotl train cfg.yaml --launcher torchrun -- \
  --nnodes $N --nproc_per_node $G \
  --rdzv_id $ID --rdzv_backend c10d \
  --rdzv_endpoint "$HEAD_IP:$PORT"

The other routes are an accelerate config per machine, Ray Train via use_ray: true, or SkyPilot's num_nodes. Pick one for the platform. A stale file at ~/.cache/huggingface/accelerate/default_config.yaml triggers a warning that it "can lead to unexpected errors", and in a shared image it can silently override the launch you thought you configured; remove it from the image.

The trust model

The most important security fact about Axolotl is that it executes what it is given. It runs with the privileges of the invoking user, and a configuration file is not data; it is a set of instructions that can lead to arbitrary Python being imported and run. Once you accept this, the controls follow.

The places where config becomes code are specific and short. trust_remote_code: true, at the model level and per dataset, executes code from a Hub repository; it defaults to false. GRPO reward_funcs, rollout_func and trainer_cls are importable Python. A custom type: prompt strategy is Python. Anything under plugins: is Python, and plugins can also register CLI subcommands and cloud launchers. Config files can be HTTPS URLs, which means the file you think you are running is whatever the server returns at that moment unless you pin it to a commit.

The consequence for a multi-team platform is that access to the config is access to the machine. A team that can submit arbitrary YAML to a shared runner can run arbitrary code on it, with whatever credentials the runner holds. So there are two coherent designs. Either you run every job in a sandbox that holds nothing worth stealing (a container, scoped credentials, no route to your internal network), or you restrict what configs may contain and review the Python that they import. In practice you do both: an allow-list for plugins:, a rule that trust_remote_code is false unless a reviewer has approved a specific repo at a specific revision, and a requirement that reward modules live in a repository your platform reviews.

Network exposure is the second area. The vLLM server for GRPO defaults to 0.0.0.0:8000 through vllm.host and trl.vllm_server_host, Bind it to localhost or a private network. The axolotlai/axolotl-cloud image starts Jupyter by default, and it reads JUPYTER_PASSWORD and PUBLIC_KEY or SSH_KEY; set a password or JUPYTER_DISABLE=1, because an open notebook is a remote shell.

The third area is what leaves the machine. Anonymous PostHog telemetry is on by default (opt-out). It collects operating system, library versions, hardware, runtime metrics, whitelisted model names, and sanitized error types, redacts path-like values, and sends from rank 0 only. Start-up pauses for ten seconds with a notice unless you set it explicitly. You disable it with AXOLOTL_DO_NOT_TRACK=1 or DO_NOT_TRACK=1, and setting AXOLOTL_DO_NOT_TRACK=0 enables it and silences the notice. For regulated or air-gapped environments, which includes many of the banks, telecoms and government-adjacent employers in the Gulf and Egypt that hire for these roles, disable it in the base image and also set HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1. Decide this deliberately and write the decision down; do not let it be a default someone discovers during an audit.

Secrets and identity

Credentials arrive from four directions: the Hub token for gated models and private datasets, experiment tracker keys, cloud storage credentials for remote datasets, and the credentials your orchestrator uses to launch the job. The recurring mistake is putting any of them in the YAML. The config is copied into output_dir, and output_dir is what you push to the Hub or hand to another team, so a token in the config travels with the model.

The CLI loads a .env file automatically via python-dotenv. That is convenient on a laptop and hazardous in a shared image or a repository, because the file is read from the working directory and is easy to commit. Use the environment instead: HF_TOKEN for the Hub (the CLI verifies it at start and prints "Error verifying HuggingFace token" when it cannot), WANDB_API_KEY for Weights and Biases, and the standard provider chains for storage. Remote datasets accept s3://, gs://, adl:// or abfs://, and oci:// paths, and credentials come from the usual places: AWS environment variables or an instance role (with AWS_ENDPOINT_URL for S3-compatible stores), Google metadata, AZURE_STORAGE_*, or OCIFS_* and resource principals for Oracle Cloud. Prefer instance or workload identity over static keys wherever the cloud supports it, since a role cannot be pasted into a chat.

The supply chain and upgrades

Your supply chain here is wide: Axolotl itself, the pinned libraries it pulls in, the CUDA and PyTorch layers, the base model and dataset repositories, and the image you build from all of it. v0.20.0 pins transformers==5.17.0, peft==0.21.0, trl==1.13.0, accelerate==1.15.0 and others exactly, with torch>=2.13.0,<=2.14.0, Python 3.12 or newer, and DeepSpeed in a bounded range. That is helpful, because it means a given Axolotl release is reproducible if you build from it. It also means that when you add your own packages to the image you can break a pin without noticing, so run pip check or the equivalent in CI.

The project itself has taken steps on its side: GitHub Actions workflows are pinned to commit SHAs since 0.18, and the official Docker images are uv-based since the same release. The older pip-based images were removed, so teams that referenced one need to move. Image names have a wrinkle worth knowing: the installation page still shows axolotlai/axolotl-uv, while the Docker page says the plain names are now the defaults and the -uv names are aliases. Use the plain names and check the Docker page when you write the platform's base image.

Pin tightly. The right reference is the exact release or a release image tag such as 0.20.0-py3.12-cu130-2.13.0, ideally resolved to a digest, never main-latest. A floating tag means two jobs launched on the same day can run different code. Scan the image with a tool like Trivy, keep a software bill of materials, and mirror it into your own registry so that a Hub or Docker Hub outage does not stop training.

Upgrades are where platform ownership earns its cost. The release cadence over the last year was roughly three to seven weeks apart, and the changes are often breaking. Look at what the last few releases required of you: 0.14 moved to Transformers v5, so any monkeypatch against v4 broke; 0.17 renamed relora_steps with no shim and fixed multimodal label masking, which changes loss values; 0.18 removed the non-uv images; 0.20 raised the floor to Python 3.12 and torch 2.13.0 and removed FSDP1. None of these are bugs. They are the project moving, and your job is to absorb the movement so your users do not.

That argues for a fixed upgrade procedure. Read the release notes and diff the schema. Build the candidate image. Run each team's golden config through axolotl preprocess --debug and a short training run. Compare the loss curves for the first few hundred steps against a stored baseline, because fixes in masking and gradient-accumulation normalization will legitimately move them (0.20 fixed a gradient accumulation mis-normalization on some models and a LoRA+ bug that had silently ignored weight_decay). Publish a migration note. Only then change the default image, and keep the previous image selectable for a defined window so that a team in the middle of a long run is not forced across a boundary.

Multi-tenancy

Axolotl has no multi-tenancy of its own: no authentication, no quotas, no job isolation. Everything you promise a team about separation, you build around it. The useful way to think about this is as a list of shared things that leak, and a deliberate decision about each.

The filesystem is the first leak. Two jobs with the same output_dir overwrite each other. Two jobs sharing dataset_prepared_path can reuse each other's cache; the rule here is subtle, and worth remembering exactly. If the key is left empty, output goes to ./last_run_prepared/ but the existing cache there is ignored. If you set the path explicitly, the cache is reused, keyed by a hash of the relevant config. A change to a custom prompt strategy may not change that hash, so a team can silently train on stale tokenization. Give each job its own prepared path, or include a code version in the path, and delete it when you change a strategy.

The Hugging Face cache is the second. The cloud images set HF_HOME=/workspace/data/huggingface-cache. A shared cache saves downloads, but it also means one team's gated model sits in a place another team's job can read, bypassing the licence acceptance and the per-team token you were careful about. For the models everyone may use, a read-only shared mirror is excellent. For gated or private ones, use a per-team cache.

Accelerator isolation is the third, and it belongs to your scheduler. On Kubernetes that means GPU resource requests and namespaces; see the Kubernetes guide for quotas, priority classes and node pools. On Slurm, the repository includes an examples/slurm directory as a starting point. Whatever you choose, the quota must be enforced in front of Axolotl: a team asking for eight GPUs gets them or waits, and Axolotl is never in the decision.

Cost control

GPU time is the expense, and nearly every lever is a configuration choice you can standardize. The ordering matters: spend your effort where it removes whole multiples, not percentages.

The largest lever is method choice. The docs' own guidance is that LoRA or QLoRA fits most adaptation tasks, and full fine-tuning is for cases where adapters demonstrably fall short. A platform that defaults to full fine-tuning for everything pays four to eight times the hardware for a result the team could not tell apart. Offer a decision aid, not a lecture: the planning VRAM table, a worked example, and a rule that full fine-tuning requires a short written justification.

The second lever is throughput per GPU. Sample packing raises token utilization sharply on short samples, because it fills each sequence_len row with several samples, and with flash attention or SDPA varlen it uses cu_seqlens so samples do not attend to each other. The plugins for Liger and Cut Cross Entropy reduce memory and time, and for lora and qlora the LoRA kernels turn on automatically unless you run 8-bit LoRA, set trust_remote_code, use an rl: method, or set a kernel flag explicitly. The log line "Auto-enabling LoRA kernel optimizations for faster training." tells you it happened. The right platform behaviour is to check the log for that line in your golden runs, because a configuration change that disables the kernels costs real money and no error appears.

The third lever is not wasting runs. A five-step smoke test, with max_steps and save_first_step: true to prove checkpoints write, catches most config errors for the price of a minute. The loss_watchdog_threshold and loss_watchdog_patience keys abort a diverging run early. Preprocessing on CPU-only machines keeps expensive accelerators out of a stage that does not use them. Spot or preemptible capacity, for example through SkyPilot, is cheap when combined with resumable checkpoints and a waste of money without them.

Observability and telemetry

A platform needs three kinds of signal: whether a job is healthy now, whether it was worth running, and whether the fleet is efficient. Axolotl contributes the first two and you assemble the third.

For job health, the trainer logs loss, learning rate and gradient norm at logging_steps. The ranges in the project's stability documentation are useful alert thresholds: chat fine-tuning loss is typically 0.5 to 2.0, grad_norm normally 0.1 to 10 with spikes above 100 meaning instability, learning rates 1e-5 to 5e-5 for full fine-tunes and 1e-4 to 3e-4 for LoRA, and max_grad_norm: 1.0 as the usual clip. Use these as starting points for alerts, not as laws, since a legitimate run on unusual data can sit outside them.

For GPU behaviour, Axolotl logs memory/max_alloc and memory/max_reserved. These are PyTorch allocator figures and differ from nvidia-smi, so a team comparing the two will think one is wrong. Teach the difference once. include_tkps defaults to true and logs tokens per second, which is your throughput metric. When you need to see inside a step, profiler_steps produces PyTorch profiler traces and memory snapshots. The dynamic_checkpoint block lets an operator touch axolotl_checkpoint.save in the output directory to force a checkpoint, which is the polite way to drain a node.

For metrics in your own stack, use_otel_metrics: true exposes an OpenTelemetry and Prometheus endpoint at otel_metrics_host:otel_metrics_port, which defaults to localhost:8000 and needs the axolotl[opentelemetry] extra. That default is the same port as the vLLM server's default, so a GRPO job that enables both will collide, and the failure looks like a vLLM start-up problem. Set one of them explicitly. See the OpenTelemetry and Prometheus guides for the collection side.

From checkpoint to serving

A model that cannot be served is not a deliverable, and the hand-off from training to serving is where many platforms discover a missing step. Treat it as a governed pipeline with explicit stages and recorded outputs.

For an adapter-trained model, the first step is merging. axolotl merge-lora folds the LoRA weights into the base model and writes the result under {output_dir}/merged; CUDA_VISIBLE_DEVICES="" forces a CPU merge, which avoids needing a GPU for a one-time operation. Do the merge with this command and not with a custom script. When tokens were added during training, the vocabulary was resized, and a hand-written merge fails with a torch.Size mismatch. Axolotl expands embeddings to match a larger tokenizer and never shrinks them unless shrink_embeddings: true, so the merge command is the supported path. A related error, "Can't merge qlora if loaded in 4bit", appears when merge_lora: true sits in a config that also loads 4-bit; merge separately, and use --dequant if you need bf16 output from a quantized base.

Quantization and export are the second step. axolotl quantize runs post-training quantization through torchao and writes to {output_dir}/quantized, and a qat: block supports quantization-aware training with NVFP4, MXFP4, FP8 and ternary options. The quantize documentation says GPTQ and EXL2 are not supported. For GGUF, v0.20 added axolotl export, which accepts --quantize Q4_K_M,Q8_0, --outtype, --outfile and --llama-cpp-dir, needs a built llama.cpp (found through $LLAMA_CPP_DIR), and refuses an adapter-only directory, so merge first. GGUF is the format that llama.cpp and Ollama consume.

Serving is the final stage and it sits outside Axolotl. Merged safetensors serve through vLLM, SGLang or Text Generation Inference; vLLM can also serve LoRA adapters directly, which lets many adapters share one base model in memory. That is often a better multi-team economic model than a merged model per team. Record the lineage for each promoted artifact: the Axolotl version and image digest, the config, the dataset revision, the base model revision (revision_of_model), and the evaluation results. If you later need to answer where a model came from, that record is the answer. Teams that also use PEFT and TRL directly will find the same lineage questions apply.

GRPO as a platform workload

Reinforcement learning with GRPO is the workload that most changes your operating model, so it deserves its own section. Axolotl's SFT jobs are one process group. A GRPO job is two cooperating services: a trainer, and a vLLM server that generates completions for it. The documentation says GRPO needs at least two GPUs, one for each, and that it almost always needs more.

The shape is concrete. The vLLM server runs through axolotl vllm-serve config.yaml, reading the vllm: block, and the trainer runs through axolotl train with rl: grpo and trl.use_vllm: true. In the server mode the two are separate processes, typically on different GPUs, with CUDA_VISIBLE_DEVICES assigning each. The colocate mode shares a GPU and is documented for a single GPU with models up to roughly 3B on 24 GB. The platform decisions follow from this: your launcher must start two coordinated processes, health-check the server before the trainer starts, and tear both down together.

Three failure modes deserve runbook entries. First, version consistency: vLLM must serve the same base model that is being trained, and if it does not, weight synchronization silently produces wrong results, with no error. Second, reward functions: the reward_funcs entries are importable Python modules, so an import failure shows as "unable to load strategy" followed by None is not callable when the module is not on the path, and a reward that is broken or uninformative shows as reward stuck at 0 or reward_std always 0. Set trl.log_completions: true for diagnosis, and test the reward function standalone before spending GPU time. Third, batch arithmetic: the generation batch must be divisible by num_generations, or the run stops with a divisibility error.

Governance and incident playbooks

Governance sounds bureaucratic, but for a training platform it reduces to four questions: who may train what, on which data, with which code, and with what evidence of the result. Keep each as a small, enforceable control rather than a policy document.

Who may train what is quotas and approvals in your scheduler. On which data is a registry of approved datasets with owners, revision pins and a licence note, and a rule that production training reads only from it. With which code is the allow-list for plugins, reward modules and trust_remote_code described earlier. What evidence is the lineage record and evaluation gate. For data in particular, think about residency. Many employers across the Gulf and Egypt require training data to stay in-country or in an approved region, and with remote dataset paths, Hub downloads and managed launchers, data can leave the boundary without anyone deciding it should. Choose regions deliberately, mirror the models, and set offline mode so that a missing mirror fails instead of falling back to the public internet.

Incidents follow patterns, and a playbook turns each pattern into a ten-minute task. Keep these in your runbook with the exact messages.

A job exits with exitcode: -9. Host memory ran out, usually during tokenization. Lower dataset_num_proc, reduce the dataset size or shard it, or run axolotl preprocess as a separate job on a larger CPU machine. Do not add GPUs.

A job stops making progress and later reports a Watchdog collective timeout. The NCCL collective ran past its 30-minute default. Check the NVLink status with nvidia-smi nvlink --status, try NCCL_P2P_LEVEL=NVL, check that PCI ACS is disabled, and turn on NCCL_DEBUG=INFO and TORCH_DISTRIBUTED_DEBUG=INFO. ddp_timeout is honoured at load time since 0.19 through AXOLOTL_NCCL_TIMEOUT, which helps for slow weight loading but is not a cure for a broken fabric. If the error is "NCCL communicator was aborted", the same investigation applies.

Tokens look wrong, or loss changed after an upgrade. Run axolotl preprocess --debug, which prints tokens as (label, token_id) pairs so you can see exactly which tokens are masked with -100. Check chat_template, the EOS or EOT tokens, and roles_to_train. Errors such as "Could not find content start/end boundary for turn" or "EOS token not found in chat template" mean the tokenizer's end token and the template disagree; set special_tokens.eos_token to the template's end token and eot_tokens accordingly.

Where Axolotl stops

A platform owner earns trust by saying clearly where a tool is the wrong choice. Axolotl is a post-training framework, and its edges are worth stating plainly.

It is not an orchestrator. Without a scheduler, queue, retry policy or workflow engine, it needs one around it: Slurm, Kubernetes, SkyPilot, Ray, or a managed service. If your problem is coordinating dozens of pipelines with dependencies, read about Airflow or Kubeflow and treat Axolotl as a step inside them.

It is not a serving system and has no registry. Inference commands exist for sanity checks and chat, not for production traffic. Pair it with a serving engine and with a model registry such as the one in MLflow.

It moves quickly. Weekly-to-monthly releases with breaking changes are a feature for teams that want new methods early, and a cost for teams that want stability. If you need a slow-moving, long-supported training stack, you will either maintain your own pinned fork of your configs or choose differently.

It is also not the only reasonable choice for adaptation. If the work is one engineer fine-tuning on one GPU and speed matters most, Unsloth is purpose-built for it (Axolotl's own LoRA kernels are credited in its docs as inspired by Unsloth). If the need is maximum control in code, using PEFT and TRL directly removes a layer of configuration indirection. If the team is already deep in DeepSpeed scripts, keeping them may cost less than a migration. LLaMA-Factory and torchtune are other alternatives that appear in the same comparisons, and you should evaluate them against your own requirements instead of against anyone's claims.

The honest summary for a design review is that Axolotl is a strong choice when many people need to run varied, reproducible post-training jobs from configuration files, and a weak choice when what you need is a long-lived service, a stable library API, or a single-researcher notebook workflow.

Running Axolotl as a platform

Pulling the sections together, here is what the service you offer actually consists of. It is smaller than people expect, because the tool brings so little infrastructure with it.

The base image is the product. Build one from a pinned official release image, add your CA certificates, your telemetry and offline settings, the NCCL environment defaults, and your tracker client, and publish it by digest. Everything else is configuration. Keep a second, separate image for GRPO with the vLLM extra. Rebuild on every upstream release, and rebuild on base-image vulnerability fixes.

The template library is the second product: validated YAMLs per workload class (LoRA SFT, QLoRA on one GPU, FSDP2 multi-GPU, long-context with context parallelism, DPO, GRPO), each with the defaults explicit, the wrap class for the supported model families, and the tracker and storage settings injected. A team starts from a template and overrides a short list of keys. This is how you get consistency without writing a tutorial for every change.

The submission path is the third. A pre-flight job validates the config against the schema, checks the allow-lists, scans for secrets, estimates memory from the planning table, and runs axolotl preprocess on CPU. If that passes, the training job is queued with an explicit wall-clock limit, a checkpoint location, a cost tag, and the per-team identity. The same path is used from a laptop and from CI, so there is one behaviour to learn. Platforms with GitOps habits can drive submission from pull requests, reviewing config changes the way they review infrastructure; the Argo CD guide and the Terraform guide cover that style for the surrounding infrastructure, and the Docker guide covers the image side of this section in depth.

The review checklist

Use this list on a platform change, a new template, or a team's first production config. Each item is a question that has an answer, not a reminder.

The config

• Does it set sequence_len, output_dir, val_set_size, micro_batch_size and gradient_accumulation_steps explicitly?
• Does it avoid every deprecated key, and does a log search show no deprecation warnings?
• Does it use attn_implementation rather than the legacy attention booleans, and type: chat_template rather than sharegpt?
• Is a custom prompt strategy paired with a fresh dataset_prepared_path?
• Does the axolotl preprocess --debug output show masking that matches the intent: prompts masked, assistant turns trained?

Parallelism and memory

• Is the strategy one the platform supports, and is fsdp_version: 2 with the correct wrap class set?
• Does context_parallel_size divide the GPU count, and is expert parallelism paired with FULL_STATE_DICT?
• Has the VRAM estimate been checked against the planning table, with a plan for the next rung of the out-of-memory ladder?

Security

• Is trust_remote_code false, or approved for a named repo and revision?
• Are all plugins, reward modules and custom strategies from reviewed sources?
• Is any config loaded from a URL pinned to a commit?
• Are tokens absent from the YAML and the .env, injected per team at launch?
• Is anything bound to 0.0.0.0 that should be private, and is Jupyter disabled or password protected?
• Is telemetry decided and set in the image?

Operations

• Is the image pinned by digest, from a release tag, not main-latest?
• Are the Axolotl commit and config identical on every node?
• Is there a wall-clock limit, a no-progress alert, and a checkpoint location that survives the job?
• Can the job resume, and has resume been tested?
• Does the job carry a cost tag and a team identity?

Outcome

• Is there a held-out evaluation, and does the pass criterion exist before the run?
• Is the lineage recorded: version, image digest, config, data revision, base model revision, results?
• Has the merged or exported artifact been loaded by the serving engine you actually use?

If a change cannot answer a question, it is not ready, however good the loss curve looks.

The complete picture

Where the series leaves you

You can now reason about Axolotl at three levels. At the first you could run a job and read its output. At the second you understood why it behaves as it does and tuned it for your hardware and data. At this one you can design a service around it, predict where it will fail as usage grows, defend the choice to people who control the budget, and say when to use something else.

The natural next steps are the neighbours: the PEFT and TRL guide for the libraries underneath, DeepSpeed for the sharding FSDP2 competes with, vLLM to serve what you train, and Kubernetes when placement and quotas outgrow a script. Whatever you build, keep it small enough that you can explain every part of it.

Sources

• Axolotl documentation
• Installation
• Command line interface
• Config reference
• Docker images
• Choosing a method
• Multi-GPU training
• Multi-node training
• N-D parallelism
• NCCL troubleshooting
• GRPO
• vLLM serving
• Custom integrations
• Dataset preprocessing
• Streaming datasets
• Quantization
• Export to GGUF
• Telemetry
• FAQ
• Debugging
• Training stability
• Axolotl releases on GitHub
• Axolotl v0.20.0 release
• Axolotl on PyPI

Senior interviews about a tool like Axolotl rarely ask what a flag does. They hand you a situation with money, people and a deadline attached, and watch how you reason. This file is built the same way: system-design prompts, incident questions, and the conversations you will have with people who control the budget. Each answer is written as a strong candidate might give it, followed by what separates it from a weaker one. The version discussed throughout is Axolotl 0.20.

The thirty-second answer

If you are asked "what is Axolotl and why would you run it as a platform?", a senior answer is short and names the trade-off. Axolotl is an open-source, configuration-driven post-training framework: one YAML file describes the model, data, adapter, hyperparameters, parallelism and logging, and the same file drives preprocessing, training, inference, merging, evaluation, quantization and export. It wraps Hugging Face libraries, DeepSpeed and PyTorch FSDP2, and adds sample packing, kernels, a plugin system and a command line.

You would run it as a platform because the YAML is a reviewable, diffable, validated contract, and because a single-job batch trainer with no server has very little to keep alive. The cost of the same property is that queues, quotas, identity, audit and retries all have to come from the systems around it, and that a config can execute code, so access to the config is access to the machine. A weak answer lists features. A strong one says what you get and what you now have to build.

System-design prompts

Design a fine-tuning platform for twelve teams

The prompt. Twelve product teams want to fine-tune open models on their own data. You have a pool of GPU nodes. Design the service.

A strong answer starts with questions, not boxes. How many jobs a week, and how long? Which model sizes? Is the data sensitive, and does it have residency rules? Who owns the cluster scheduler? Are teams expected to write YAML or click a form? Those answers decide almost everything. Suppose the answers are: a few dozen jobs a week, mostly 7 to 13B LoRA with a few larger runs, sensitive customer data, Kubernetes already in place, and engineers comfortable with YAML.

Then the design has four parts. The first is a pinned base image built from an official Axolotl release image and published by digest, with telemetry disabled, offline settings applied, NCCL defaults set, and a tracker client baked in. The second is a template library of validated YAMLs per workload class, each with the defaults that matter set explicitly: sequence_len, output_dir, val_set_size, fsdp_version: 2 with the right wrap class, and an attn_implementation. Teams override a short list of keys. The third is a submission path that runs a pre-flight before queuing: validate against the schema of the exact release, check allow-lists for plugins and trust_remote_code, scan for secrets, estimate memory from the planning figures, and run axolotl preprocess on a CPU-only machine. The fourth is the execution environment: each job runs as a Kubernetes job in a team namespace with a GPU quota, a per-team identity, a separate output_dir and dataset_prepared_path, a wall-clock limit, and egress restricted to your model mirror and tracker.

After training, an automated stage merges the adapter with axolotl merge-lora, runs evaluation, and registers the artifact with its lineage: image digest, config, dataset revision, base model revision, results. Teams serve through your serving platform, not from the training jobs.

What makes it strong. It begins by scoping, it treats the config as a contract, it identifies that Axolotl has no scheduler and puts one in front, and it names the security issue before being asked. What a weak answer misses. It describes "a VM with Axolotl installed", says nothing about isolation between teams, and does not mention how the platform survives an upgrade.

Likely follow-up: "Why not let teams submit any config?" Because a config can import arbitrary Python through plugins, reward functions, custom prompt strategies and trust_remote_code. You either sandbox every job so that it holds nothing worth stealing, restrict what configs may contain, or, as you should, do both.

Choose a parallelism strategy for a mixed fleet

The prompt. Teams train anything from 1B to 70B parameter models, on nodes with eight GPUs each. Which parallelism do you support?

A strong answer begins with the shape of the problem. Small models fit on one GPU and need data parallelism, if anything. Mid-size models need sharding. The largest need sharding across nodes, and possibly context parallelism for long sequences. The policy is a small set of supported shapes. FSDP2 is the default, because in 0.20 it is the only FSDP, and because the project's N-D parallelism guidance recommends it. DeepSpeed ZeRO stays supported for teams with existing configurations, but not as the default, because FSDP and DeepSpeed are mutually exclusive and two paths double your support burden. Tensor parallelism is experimental and wants NVLink, so it is offered only inside a node and only on request. Context parallelism is offered for long-context jobs, with the rule that the degree divides the GPU count. Expert parallelism for MoE models comes with its own requirement: fsdp_version: 2 and state_dict_type: FULL_STATE_DICT.

For memory, the candidate quotes the planning figures with a caveat: roughly 10 to 14 GB for a 7 to 8B QLoRA at short context, and around 40 to 48 GB for a 70B QLoRA, but long context and MoE move them. A 70B model on two 24 GB GPUs is possible with FSDP and QLoRA, which matters for teams without large accelerators.

What a weak answer misses. It picks DeepSpeed because it is familiar, or says "always use the biggest ZeRO stage", ignoring the communication cost. It also forgets that the cheapest answer to "it does not fit" is often a config change, not another node.

Offer GRPO as a service

The prompt. A team wants to train a model with reinforcement learning against a verifiable reward. What changes in your platform?

A strong answer says that a GRPO job is a different kind of workload. It runs two cooperating processes, a vLLM server generating completions and the trainer, with weight synchronization between them. That needs at least two GPUs, one per role, so the scheduler must place them together and tear them down together, and the launcher must health-check the server before the trainer starts. The reward functions are importable Python, so the trust model applies in full: they run with the trainer's privileges and need review. The vLLM server defaults to 0.0.0.0:8000, so it must be bound privately. Failure modes get runbook entries: vLLM serving a different base model than the one being trained, which silently gives wrong results; a reward that returns zeros or no variance; the generation batch not divisible by num_generations; and an async prefetch stall with FSDP.

It also isolates the support surface. GRPO gets its own image (the vLLM extra and -vllm tags exist for this) and its own template, and it is offered as a separate support tier. And it adds alerting on reward and step progress, because both processes can look alive while nothing is learning.

What a weak answer misses. It treats GRPO as "SFT with a different rl: value".

Move a hundred legacy configs to 0.20

The prompt. You inherit a hundred configs written across eighteen months. v0.20 is out. What is your plan?

A strong answer begins with inventory, not edits. Dump the schema for the old and new versions and diff them. Run every config through validation on the new release and sort the results into three groups: errors that stop the run, warnings that continue, and silent behaviour changes.

The errors in this release are specific. FSDP1 is removed, so fsdp_version: 1, the top-level fsdp: list, sharding_strategy and forward_prefetch raise. type: sharegpt raises and must become type: chat_template with field_messages: conversations and a role mapping. The warnings include dataset_processes, sequence_parallel_degree and fsdp_-prefixed keys. There is no shim for relora_steps, which became jagged_restart_steps in 0.17. Python 3.12 and torch 2.13 are the minimums, so anything pinned to older environments needs a new image.

The silent changes are the dangerous ones. Masking fixes (multimodal masking in 0.17), gradient accumulation normalization and the LoRA+ weight_decay bug in 0.20 change loss values legitimately, so a team that compares curves will think the migration broke something. The plan therefore includes a baseline comparison on a short run and a note explaining expected differences. axolotl preprocess --debug before and after shows whether the labels changed.

Then sequence the rollout: migrate templates first, then teams in order of risk, keep the previous image available for a fixed window, and publish a migration note before changing any default.

What a weak answer misses. It says "find and replace the keys", and does not mention the silent behavioural changes or the old image window.

Incident-response questions

A multi-node job hangs and then times out

The question. A sixteen-GPU job has shown no progress for twenty-five minutes, then fails with a Watchdog caught collective operation timeout. What do you do?

A strong answer orders the checks by cost. First, establish scope: did it ever make progress, or has it hung from the start? A hang at the start points at rendezvous or networking, such as the wrong interface, a blocked port, mismatched --nnodes, or the nodes running different Axolotl commits or configs. A hang after hours points at a rank dying, a data issue on one rank, or fabric trouble. Second, read the NCCL doc's checks: nvidia-smi nvlink --status, try NCCL_P2P_LEVEL=NVL, check that PCI ACS is disabled, then enable NCCL_DEBUG=INFO and TORCH_DISTRIBUTED_DEBUG=INFO. For InfiniBand the settings are NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME and NCCL_BUFFSIZE. Third, compare nodes: the image digest, the config hash and the Axolotl commit must be identical everywhere. Fourth, check for a stale accelerate config file, which the tool warns about, overriding the launch.

The timeout is 30 minutes by default, and ddp_timeout is honoured at weight loading since 0.19 through AXOLOTL_NCCL_TIMEOUT, which helps slow loading but does not repair a faulty fabric. Raising it is not a fix; say so.

Afterwards, the platform-level answer: why did a hung job run for twenty-five minutes before anyone knew? A no-progress alert on step count would have caught it in minutes.

Loss drops suspiciously fast, then the model is useless

The question. Training loss reaches near zero in the first epoch, and the model's outputs are poor. Where do you look?

A strong answer suspects the data pipeline before the optimizer. Run axolotl preprocess --debug and read the (label, token_id) pairs. Look for labels of -100 where there should be training signal, or the opposite: prompts left unmasked because of train_on_inputs, or an end-of-turn token mismatch between tokenizer and template. Look for duplicated data and for leakage between train and validation sets. Check whether sample_packing is combined with something that lets samples attend across boundaries. Check the chat template, roles_to_train, train_on_eos, and the special tokens. Check whether a stale prepared dataset was reused: with an explicit dataset_prepared_path, the cache is reused, and a change to a custom prompt strategy may not invalidate it.

A near-zero loss is usually a learning task that became trivially easy because the answer leaked into the input, or because most tokens are masked and the rest are easy. It is rarely a good result.

A team says an upgrade made their model worse

The question. After moving from 0.17 to 0.20 a team reports worse evaluation scores with an unchanged config. What do you do?

A strong answer refuses to guess in either direction. Compare like with like: run the old and new images on the same config, same seed, same data revision, and compare the tokenized output from axolotl preprocess --debug. If tokens or masks differ, the cause is a data pipeline change; the 0.17 notes document a fix to multimodal masking, for instance, and the effect on loss is expected. If tokens match, compare gradient norm and loss over the first few hundred steps, remembering the 0.20 fixes to gradient accumulation normalization and to LoRA+ ignoring weight_decay. Then compare evaluation setups, because an evaluation run that changed in the same release gives a score change that is nothing to do with training. Only then call it a regression, with a reproduction to report upstream.

The senior move is to point out that the platform should already have the golden-run baseline that makes this a ten-minute question.

Arguing trade-offs to leadership

Senior candidates are judged on whether they can explain a technical position to someone who does not care about the technical details. Use their language: money, risk, time and options.

"Why can't we just give every team a GPU and let them run it themselves?"

Lead with what they gain from your platform, not with what you object to. Shared infrastructure increases utilization: dedicated GPUs idle between experiments, and a pooled cluster with a queue keeps hardware busy. A platform also makes risk visible, because a config can run arbitrary code, and twelve teams with twelve credential sets and twelve configurations is twelve security reviews. Offer the number: if average utilization of dedicated GPUs is, say, a third of a pooled cluster's, you are buying three times the hardware for the same work. Use your own measured figure, not mine. Close with the option you are not removing: a team with a distinctive need can request an exception, with its own review.

"The project releases every few weeks. Can we freeze a version?"

Acknowledge the cost directly. Freezing avoids breakage but has two limits. The project's security policy supports only the latest release, so a frozen version is an unsupported one the longer you hold it, and new model support, such as the Gemma 4 support in 0.16.1, arrives in new releases. The alternative is a controlled cadence: adopt a release on a calendar (for example quarterly, or sooner for a security reason), absorb the breakage in the platform with a tested upgrade procedure, and keep the previous image for a fixed window. The cost of this is a fraction of an engineer's time; the cost of freezing is a larger, later, riskier migration. Put numbers on it where you can.

"How do we know fine-tuning was worth it?"

Never defend a training run by its loss curve. Define the success measure with the business owner before the run: task accuracy on a held-out set, a regression check against the current production model, and a cost per request compared with prompting a hosted model. Fine-tuning is sometimes the wrong tool. If a prompt and retrieval change reaches the target, say so, and save the GPU budget. This is the answer that builds trust, because it shows you are not defending your own tool.

What a strong answer covers that a weak one misses

Across all of these, a few habits separate senior answers from competent ones.

A strong answer scopes before it designs. It asks about volume, sensitivity, existing infrastructure and who writes configs, because the best design depends on them. A weak one jumps to a solution.

A strong answer names the failure stage. "The job failed" is not a diagnosis; "it failed in dataset preparation with a host-memory kill" is. Axolotl's stages are validation, preparation, model load, training under a launcher, and artifact writing, and each fails differently.

A strong answer treats configuration as code and as a security surface. It reviews, versions, validates and sandboxes. A weak one treats YAML as harmless.

A strong answer quantifies. It quotes the planning memory figures, the NCCL default timeout, the release cadence, and then says which of them it would measure for itself.

A strong answer admits limits. Axolotl has no server, no stable programmatic API, no multi-tenancy and no model registry, and the candidate says what fills each gap. It also says when it would choose something else.

A strong answer ends with an operational consequence: the alert to add, the test to automate, the doc to write. Incidents are only closed when the system is better afterwards.

Sixty-second self-test

Answer each aloud in under a minute. If you cannot, reread the matching section of the detailed guide.

1. Name the five stages of a run and give one failure that belongs to each.
2. Why is access to the config equivalent to access to the machine, and which keys make it so?
3. What does exitcode: -9 mean, and what is the first thing you change?
4. Which FSDP options raise in 0.20, which warn, and what replaces fsdp_sharding_strategy?
5. Why is dataset_prepared_path a multi-tenancy hazard when set explicitly?
6. What are the two processes in a GRPO job, and what silently breaks if their base models differ?
7. How do you merge a LoRA adapter after tokens were added, and why not by hand?
8. What is the default NCCL collective timeout, and why is raising it not a fix?
9. Which environment variables disable telemetry, and what else belongs in an air-gapped image?
10. Give two reasons not to freeze an Axolotl version indefinitely.
11. When would you recommend something other than Axolotl?
12. How would you show leadership that a fine-tune was worth its cost?

These are the lessons that only show up once many people run the tool on shared hardware for months. Each trap is described with its symptom, its cause and what to do about it, followed by the habits of a platform owner and review checklists for pull requests and design documents. Everything here assumes Axolotl 0.20.

Production traps that appear at scale

The stale prepared dataset

Symptom. A team edits a custom prompt strategy, reruns, and the model behaves as if nothing changed. Or a loss curve is subtly different from a colleague's with "the same" config.

Cause. Tokenized data is cached as Arrow under dataset_prepared_path, keyed by a hash of the relevant config. When the key is empty, output goes to ./last_run_prepared/ and the existing cache there is ignored. When you set the path explicitly, which every example config does, the cache is reused. A change inside a custom strategy's Python code does not change the config hash, so the old tokens are served.

Fix. Delete the prepared directory whenever a prompt strategy changes, or put a code version into the path. For a platform, make the path per job and ephemeral unless a team opts into reuse for a specific, pinned dataset. After any change to tokenization, run axolotl preprocess --debug and read the (label, token_id) pairs yourself.

Defaults that quietly shrink or overwrite your work

Symptom. A run completes with far fewer training samples than the dataset holds. Another run overwrites a colleague's checkpoints. A third has no evaluation at all.

Cause. sequence_len defaults to 512 and rows beyond it are dropped under excess_length_strategy: drop. output_dir defaults to ./model-out. val_set_size defaults to 0.0. None of these emits an error; each is a legitimate default for a demonstration.

Fix. Make your template set all three explicitly and reject configs that rely on the defaults. Log the dataset size after preparation, and alert when it is far below the source size.

The deprecation warning nobody read

Symptom. An upgrade breaks a config that "has always worked".

Cause. The project warns before it removes, but only some removals follow that pattern. dataset_processes warns, sequence_parallel_degree warns, fsdp_-prefixed keys warn, and flash_attention: true is soft-deprecated in favour of attn_implementation. Others raise immediately (type: sharegpt, max_packed_sequence_len, FSDP1 keys in 0.20), and relora_steps was renamed in 0.17 with no shim at all.

Fix. Collect the warning text from every job log into a searchable place, and promote chosen warnings to failures in CI. Treat a warning that has been present for two releases as an incident waiting for the release that removes the key.

One mismatched node in a multi-node run

Symptom. The job hangs at start, or ranks disagree and crash with a puzzling error. The network team finds nothing wrong.

Cause. The documentation is explicit that every node needs the same Axolotl commit and the same config. A node with a stale image, a different pinned library, or an edited config is a classic cause of failures that resemble network faults.

Fix. Pin the image by digest, print the digest and the config hash at the start of every rank's log, and compare them in your launcher before starting training. A mismatch should abort the job with a clear message.

The accelerate config that overrides you

Symptom. A job launched with one set of parameters behaves as if it were launched with another. The log contains a warning that an accelerate config file was found and "can lead to unexpected errors".

Cause. A file at ~/.cache/huggingface/accelerate/default_config.yaml, left over from a previous experiment or baked into an image, takes effect.

Fix. Remove it from the base image and from home directories on shared nodes. Keep it only if you deliberately use per-machine accelerate configuration for multi-node, and then manage it as code.

The port collision between metrics and vLLM

Symptom. A GRPO job fails to start its vLLM server, or metrics scraping returns the wrong content.

Cause. use_otel_metrics: true exposes metrics on otel_metrics_host:otel_metrics_port, which defaults to localhost:8000, the same port the vLLM server defaults to.

Fix. Set both ports explicitly in the GRPO template. This is a trap precisely because each feature works alone.

Checkpoint storage that vanishes

Symptom. A job succeeds, but the output directory is empty or the checkpoints are gone.

Cause. Output was written to ephemeral storage. Hugging Face Jobs, for example, has a default 30-minute timeout and loses the disk when it times out, so work must be pushed to the Hub or written to mounted storage. The same is true of any preemptible node or short-lived pod. A second variant: save_only_model: true saves space but disables resume, so a preempted job cannot continue.

Fix. Decide where checkpoints live before the first job, test a full resume from a checkpoint, and use save_first_step: true in smoke tests so you learn that writing fails in the first minute, not the tenth hour.

The quiet loss of memory optimizations

Symptom. A config change makes a job slower or hungrier, and nothing errors.

Cause. For lora and qlora, the LoRA kernels turn on automatically, unless you use 8-bit LoRA, trust_remote_code, an rl: method, an unsupported model, or set a kernel flag yourself. Flip any of these and the optimization disappears, with only a missing log line to show for it.

Fix. In your golden runs, assert that the log contains "Auto-enabling LoRA kernel optimizations for faster training." and track tokens per second per GPU so a regression of that kind is visible.

The merge that fails after tokens were added

Symptom. Merging an adapter fails with a size mismatch on a torch.Size, or with "Can't merge qlora if loaded in 4bit".

Cause. Training added tokens and the embeddings grew. Axolotl expands embeddings to match a larger tokenizer and never shrinks them unless shrink_embeddings: true, so a hand-written merge against the original base model has the wrong shape. The second message comes from merge_lora: true in a config that also loads 4-bit.

Fix. Merge with axolotl merge-lora, which handles quantized bases (use --dequant for bf16 output). When you add tokens, remember lora_modules_to_save for embed_tokens and lm_head. Better still, repurpose reserved tokens through added_tokens_overrides, which the docs recommend over adding new ones.

The tokenizer and template that disagree

Symptom. Errors such as "Could not find content start/end boundary for turn" or "EOS token not found in chat template", or a model that never stops generating.

Cause. The end-of-turn token in the template is not the tokenizer's EOS token. Related messages include "EOT token is encoded as multiple tokens" and a conflict between train_on_eos and train_on_eot.

Fix. Set special_tokens.eos_token to the template's turn-end token (for ChatML, <|im_end|>), set eot_tokens, and keep train_on_eos equal to train_on_eot. Verify with the debug output. Ship the chosen template with the model, since Axolotl saves it into the output tokenizer_config.json.

GRPO that looks alive and learns nothing

Symptom. Reward stays at 0, reward_std is always 0, or the run proceeds with no visible improvement.

Cause. A broken or uninformative reward, a vLLM server serving a different base model from the trainer (which produces wrong results without an error), or a reward module that did not import, which shows as "unable to load strategy" followed by None is not callable.

Fix. Test the reward function standalone with python -c "import my_rewards" from the job's working directory before queuing. Turn on trl.log_completions: true and read some completions. Confirm the server and trainer use the same base model, and make micro_batch_size times accumulation times data-parallel size divisible by num_generations.

The telemetry pause

Symptom. Every job in an isolated cluster pauses for ten seconds at start, and a security review asks what is leaving the network.

Cause. Anonymous PostHog telemetry is on by default and start-up blocks for ten seconds with a notice unless telemetry is explicitly set.

Fix. Set AXOLOTL_DO_NOT_TRACK=1 or DO_NOT_TRACK=1 in the image, with HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1 where required. Write down that this was a deliberate decision.

Platform-ownership habits

Pin everything, and pin it by digest

A tag moves; a digest does not. Pin the Axolotl image to a release tag such as 0.20.0-py3.12-cu130-2.13.0 and resolve it to a digest in your registry. Pin your own additions so that they do not violate Axolotl's pinned stack. Record the digest in every job's metadata. Reproducibility is the platform's most valuable promise, and it costs almost nothing to keep.

Treat the schema as your API documentation

axolotl config-schema is the authoritative description of what a config may contain in the version you run. Generate your validator, your template checks and part of your user documentation from it, and diff it between releases. This is cheaper and more accurate than reading every release note by hand, though you should still read them.

Keep golden runs

For each supported model family and workload class, keep a small run with its config, its first few hundred loss values, its gradient norms, its tokens per second and its tokenized debug output. Every upgrade, every new image and every infrastructure change runs the golden set and compares. When someone reports a regression, you have evidence in minutes. Update the baseline deliberately when a release fixes something, such as masking or gradient accumulation normalization, and note it.

Separate the stages and run each on the right hardware

Preprocess on CPU with CUDA_VISIBLE_DEVICES="", train on GPUs, merge on CPU, evaluate on whatever is cheapest that fits. Expressing the run as separate jobs with artifacts between them lets you retry only the stage that failed and stops idle accelerators from being billed for tokenization.

Make failure fast and cheap

Smoke test before every large run: a handful of steps with max_steps, save_first_step: true and a tiny dataset slice. Use loss_watchdog_threshold and loss_watchdog_patience to abort divergence. Set a wall-clock limit and a no-progress alert in the orchestrator. Most of what you save is not compute; it is the engineer who would otherwise discover the failure at the end.

Review the Python as well as the YAML

Anything importable is code: plugins, reward functions, custom prompt strategies, trainer_cls, remote-code model repositories. Keep an allow-list, review changes the way you review services, and keep trust_remote_code false unless a reviewer has approved a specific repository at a pinned revision. Pin config URLs to a commit, since an HTTPS config is whatever the server returns on the day.

Keep secrets out of artifacts

Tokens come from the environment or a workload identity, per team and short-lived where possible. Scan configs and .env files before submission, because the CLI loads .env automatically and the config is copied into output_dir, which may be pushed to the Hub. Make repository visibility a platform setting.

Own the upgrade calendar

The project supports only its latest release, ships every few weeks and breaks keys regularly. Put upgrades on a calendar, run the procedure (schema diff, candidate image, golden runs, migration note, previous image kept for a window), and report how many teams are on deprecated paths. An unowned upgrade becomes a forced one on the worst possible day.

Measure the platform like a product

Track queue wait, failure rate by stage, tokens per second per GPU by model family, GPU utilization versus allocation, cost per accepted model, time from first config to first successful run, and teams on deprecated paths. If you cannot see a number, you cannot defend the budget that supports it.

Write down what you do not support

A short, published list of supported workloads, models, Axolotl versions and parallelism shapes, with a process for exceptions, prevents most arguments. Put macOS, AMD and native Windows on it honestly: macOS is for authoring and smoke tests, AMD documentation is dated, and Windows means WSL2 or Docker.

Review checklist for a pull request that changes a config

Use this on any pull request that adds or edits a training configuration. Each item is a question with a yes or no answer.

Identity and reproducibility

• Is the base model pinned with a revision (revision_of_model, or the alias model_revision), not a moving branch?
• Is each dataset pinned with a revision, and does the path point to an approved source?
• Is the image tag or digest named, and does it match the Axolotl release the config was written for?
• Is a seed set where the team expects comparable runs?

Data handling

• Does axolotl preprocess --debug output show prompts masked and assistant turns trained, with the intended end-of-turn token?
• Is sequence_len set deliberately, and is the fraction of rows dropped or truncated known?
• Is val_set_size or test_datasets set, but not both?
• Are eval_steps and evals_per_epoch not both set, and likewise save_steps and saves_per_epoch?
• Is dataset_prepared_path unique to this job, or deliberately shared and pinned?

Method and memory

• Is the adapter choice justified, and for qlora is load_in_4bit: true present?
• For LoRA, are lora_target_modules or lora_target_linear and lora_modules_to_save consistent with any added tokens?
• Is attn_implementation used instead of the legacy attention booleans?
• Is the effective batch size computed (micro_batch_size times accumulation times GPUs) and sensible for the learning rate?
• Is learning_rate in the documented range for the method: roughly 1e-5 to 5e-5 for full fine-tunes and 1e-4 to 3e-4 for LoRA?

Distribution

• Is fsdp_version: 2 with fsdp_config (no fsdp_ prefixes) used, or a supported DeepSpeed file, but not both?
• Is transformer_layer_cls_to_wrap correct for this model?
• Does context_parallel_size divide the GPU count, and is expert parallelism paired with FULL_STATE_DICT?

Safety and operations

• Is trust_remote_code false, or approved for a named repository and revision?
• Are there any tokens in the file, and any plugin, reward module or strategy not on the allow-list?
• Where do checkpoints go, and has resume been tested? Is save_only_model avoided?
• Does the tracker configuration use platform-supplied project and run names?
• Is there a wall-clock limit and a cost tag in the submission, not only in the config?

If any answer is "I do not know", the pull request is not ready.

Review checklist for a design document

Use this when a team or a colleague proposes a new training workflow, a new model family or a change to the platform. A good design document answers these before it asks for approval.

Problem and alternatives

• What outcome is expected, and how will it be measured on held-out data before training starts?
• Was a prompting or retrieval approach tried, and how did it compare in quality and cost?
• Why this method (LoRA, QLoRA, full fine-tune, DPO, GRPO) and not a cheaper one? What evidence supports the choice?
• Which alternatives were considered, including other frameworks, and why were they rejected?

Scale and cost

• What is the model size, sequence length and dataset size, and what does the planning table say about memory?
• How many GPU-hours per run, how many runs for the project, and what is the budget?
• Which cost levers are applied: packing, kernels, preemptible capacity, CPU preprocessing?
• What is the plan if the estimate is wrong by a factor of two?

Failure and recovery

• Which stage is most likely to fail, and how is that detected?
• How does the job resume, and has that been tested?
• What are the wall-clock limit, the no-progress alert and the escalation path?
• For GRPO, how are the two processes started, health-checked and stopped together?

Security and data

• What data is used, who owns it, what is its sensitivity, and where may it be processed? Does it satisfy residency rules?
• What code runs: plugins, rewards, remote code? Who reviewed it?
• Which identities and credentials does the job hold, and what do they allow?
• Is telemetry decided, and is the network restricted as intended?

Outcome and lifecycle

• How is the result merged, evaluated, quantized or exported, and served? Has the serving engine loaded a sample?
• What lineage is recorded for the artifact?
• Who owns the model and its retraining, and what triggers retirement?
• How does the design survive the next two Axolotl releases?