The Complete PEFT & TRL Guide

This is part one of three. It assumes you have never used PEFT or TRL, and it takes you from an empty folder to a language model that you fine-tuned yourself, saved as a small adapter, reloaded, and merged into a standalone model. By the end you will be able to read a training log, recognise the handful of errors that stop most first runs, and know which of the many trainers in TRL to reach for next. Mid-level and Senior go further on the same topics; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. Fine-tuning is a skill you learn by watching a loss curve fall, and a loss curve that falls on your own data teaches more than any amount of reading.

A word on versions before we start. This guide is written against TRL 1.14 and PEFT 0.21, both of which were current at the end of September 2026. Both libraries changed a great deal between 2024 and 2026. Blog posts, forum answers and the memory of many language models still describe APIs that no longer exist, such as PPOTrainer, DataCollatorForCompletionOnlyLM, and SFTTrainer(tokenizer=..., dataset_text_field=..., max_seq_length=...). If you paste an old snippet and it fails, that is almost certainly why. Every example below uses the current form.

What PEFT and TRL are, and what you can do with them

A language model such as Qwen3 or Llama arrives from its creators as a file of billions of numbers called weights. Those numbers were learned from a vast amount of text, and they encode what the model knows and how it writes. Fine-tuning means continuing the training of that model on your own, much smaller dataset so that it behaves the way you want: answering in a house style, following a particular output format, speaking Arabic dialect more naturally, or handling the vocabulary of your industry.

Two libraries from Hugging Face do most of the work in the open-source world, and they are separate packages that cooperate.

TRL stands for Transformers Reinforcement Learning, though most of what beginners do with it is not reinforcement learning at all. It is Hugging Face's library for post-training, a word that covers everything you do to a model after the expensive pretraining is finished: supervised fine-tuning, preference tuning, and reinforcement learning. TRL gives you ready-made trainer classes. You hand a trainer a model and a dataset, call train(), and it runs the loop for you.

PEFT stands for Parameter-Efficient Fine-Tuning. Instead of updating every one of the billions of weights, PEFT freezes the original model and trains a tiny set of extra parameters, called an adapter, that sit alongside it. The best-known technique is LoRA. The adapter is typically a few megabytes to a few hundred megabytes, and it can be trained on a single modest GPU.

Put together, the two libraries let you do things that used to need a cluster. With a small model and a free or cheap GPU you can teach a model a new format in under an hour. With a rented GPU you can fine-tune a seven-billion-parameter model. And because the result is an adapter rather than a full copy of the model, you can keep one base model and a dozen adapters for a dozen different customers or tasks.

What people use them for:

That last card matters in this region. Many employers in the Gulf and Egypt have data-residency rules that make sending customer text to an overseas API awkward. Fine-tuning an open model inside a cloud region you control, or on a machine in your own office, is a legitimate answer, and these two libraries are how it is usually done.

You need very little to follow along: Python 3.10 or newer, a terminal, and ideally an NVIDIA GPU with a few gigabytes of memory. A free notebook GPU is enough for everything in this guide, because we use a model with only 0.6 billion parameters. A Mac can also run the examples on its own accelerator, only slowly.

The problem these libraries solve

To see why PEFT exists, you need a feel for the cost of training a model the obvious way.

Training adjusts every weight a little, many thousands of times. For each weight the training process must keep not only the weight itself but also its gradient (the direction to nudge it) and, with the common AdamW optimiser, two further running statistics per weight. A rough rule is that full fine-tuning needs several times the memory of the model itself. A seven-billion-parameter model that occupies around fourteen gigabytes just to sit in memory in half precision can need well over a hundred gigabytes to train fully. That means a rack of expensive accelerators, and every fine-tuned variant is another complete multi-gigabyte copy of the model that you have to store and ship.

That is the situation PEFT was designed for. The key observation behind LoRA is that when you fine-tune a large model for a specific purpose, the change to each weight matrix does not need to be complicated. It can be well approximated by the product of two thin matrices. If a weight matrix is a big square grid of numbers, LoRA leaves that grid alone and learns two skinny rectangles that multiply together to form a correction of the same size. The skinniness is controlled by a number called the rank, written r. With a small rank, the skinny matrices contain a tiny fraction of the numbers in the original, so there is far less to train and store.

Before PEFT, practitioners had a few poor options. They could fine-tune the whole model if they had the hardware. They could freeze most layers and train only the last few, which is cheap but often disappointing. Or they could skip fine-tuning and rely on careful prompting, which is cheap and flexible but cannot make a model reliably follow a format it does not naturally produce, and makes every request longer and more expensive.

TRL solves a different problem. Even with the memory question answered, writing a correct training loop for a language model is surprisingly fiddly. You must turn raw text into token IDs, decide which tokens count toward the loss, pad or pack examples into batches, schedule the learning rate, log metrics, save checkpoints, and handle several GPUs. Hugging Face's transformers.Trainer already handles much of this for generic models. TRL builds on top of it and adds the language-model-specific parts. If you have used Trainer before, TRL's trainers will feel familiar, because they are subclasses of it.

The mental model: layers and nouns

Fine-tuning vocabulary comes at you fast, so it helps to see how the pieces stack before you meet any code. Think of five layers, from the bottom up.

You will mostly touch the top two layers, but the lower ones leak into error messages, so knowing they exist saves confusion. When a message mentions accelerate, it is the layer that decides which device your tensors live on. When it mentions transformers, it is about the model or the tokenizer.

Here are the nouns you need, defined once so that you can recognise them everywhere.

Base model. The pretrained model you start from, for example Qwen/Qwen3-0.6B. It is hosted on the Hugging Face Hub and downloaded the first time you use it. During LoRA training it stays frozen.

Tokenizer. The component that splits text into tokens, the integer pieces the model actually reads. Every model comes with its own tokenizer, and mixing a model with the wrong tokenizer produces nonsense. In TRL the tokenizer is passed as processing_class. Older code called it tokenizer; that name is gone from the trainers.

Dataset. A table of training examples, loaded with the Hugging Face datasets library. The columns it must have depend on the trainer, which is the subject of its own section below.

Trainer and config pairs. Every TRL method comes as two classes. The config holds the settings, and the trainer does the work. SFTConfig pairs with SFTTrainer, DPOConfig with DPOTrainer, GRPOConfig with GRPOTrainer, and so on. The config classes subclass transformers' TrainingArguments, so everything you may know about TrainingArguments, such as output_dir, num_train_epochs and learning_rate, still applies.

SFT. Supervised fine-tuning. You show the model examples of the text you want it to produce, and it learns to predict each next token. The loss is called cross-entropy or negative log-likelihood (NLL): roughly, how surprised the model is by the correct next token. Lower is better. SFT is where every beginner starts, and it is what this guide covers in depth.

LoRA. Low-Rank Adaptation. Next to each chosen weight matrix W, PEFT adds a pair of thin matrices A and B. The layer's output becomes the original output plus a scaled correction from B times A. B starts at zero, so at step zero the model behaves exactly like the base model, and training gradually grows the correction.

Adapter. The set of those added matrices. It is the only thing training changes, and the only thing you save.

Rank (r) and alpha (lora_alpha). The rank sets how thin the matrices are, and so how much capacity the adapter has. Alpha scales the correction; the scale applied is alpha divided by rank. A common starting point is rank 16 and alpha 32.

Target modules. Which layers of the model get adapters. You can name them, but the beginner-friendly choice is the string "all-linear", which means every linear layer except the final output layer.

Offline and online methods. TRL's trainers split into two families. Offline methods, which are SFT, DPO, KTO and reward modelling, learn from a fixed dataset that already exists. Online methods, GRPO and RLOO, make the model generate answers during training and then score them. Online methods are heavier and need more setup. This guide stays offline.

Installing and checking the setup

The install is short but has one trap, and the trap is PyTorch.

Install PyTorch first, using the selector on pytorch.org to pick the build that matches your hardware (a specific CUDA version for NVIDIA, ROCm for AMD, or the default for Apple Silicon and CPU). If you skip this step, installing TRL will pull in whatever PyTorch pip chooses, which on some systems is a CPU-only build, and you will wonder why training is a hundred times slower than expected.

Then create an isolated environment and install the libraries.

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install "trl[peft]"

The [peft] part is an extra: TRL on its own does not install PEFT, so pip install trl followed by a first LoRA run would stop with an ImportError telling you to install trl[peft]. Asking for the extra up front avoids that. If you prefer uv, the same command works as uv pip install "trl[peft]". Python 3.10 or newer is required by both libraries. One page of the PEFT documentation still says 3.9, but the package metadata and the release notes say 3.10, and 3.10 is what you should assume.

There are other extras you will meet later, and it helps to know their names now. trl[quantization] adds bitsandbytes for 4-bit loading. trl[vllm] adds the fast generation engine used by online methods. trl[deepspeed] and trl[liger] add memory and speed helpers. You do not need any of them for this guide except bitsandbytes in the QLoRA section.

What works on which operating system

On a Mac, stick to a tiny model and expect minutes where a GPU takes seconds. On Windows, native Python works for the basics, and WSL2 is the smoother path once you want anything more. A free cloud notebook with an NVIDIA GPU is a perfectly good place to do this whole guide.

Verify the install

TRL ships a command that prints everything relevant about your environment.

trl env
python -c "import trl, peft, transformers, torch; print(trl.__version__, peft.__version__, transformers.__version__, torch.__version__, torch.cuda.is_available())"
trl sft --help

trl env prints your platform, Python, PyTorch, accelerator, and the versions of transformers, accelerate, datasets, TRL, PEFT and others. The official documentation asks you to paste this output when you report an issue, so get used to running it. The second command prints five values on one line; the last one, True or False, tells you whether PyTorch can see an NVIDIA GPU. False on a machine with a GPU means you have the wrong PyTorch build. The third command proves that the trl command-line tool is installed.

Some models on the Hub are gated: you must accept a licence on the model page and then prove who you are. The tiny Qwen model we use is not gated, so you can skip this today, but when you move to a gated model, run hf auth login or set the HF_TOKEN environment variable.

Understanding your data: dataset formats

Most fine-tuning failures are data failures, so it pays to learn the shape of the data before running anything. Training data in TRL comes in two formats and several types.

The format is about how the text is written. In standard format each field is a plain string. In conversational format each field is a list of messages, and each message is a small dictionary with a role (such as system, user or assistant) and a content. Conversational format is how chat models are used in practice, and it is what most modern datasets use.

The type is about which fields exist, and each TRL trainer expects a particular type.

A single row of a conversational language-modelling dataset looks like this.

example = {
    "messages": [
        {"role": "user", "content": "Summarise: the meeting moved to Thursday."},
        {"role": "assistant", "content": "The meeting is now on Thursday."},
    ]
}

A prompt-completion row separates the two halves, which gives you more control, because SFT then computes its loss only on the completion by default. The model is not rewarded for reproducing the question, only for producing the answer.

example = {
    "prompt": [{"role": "user", "content": "Summarise: the meeting moved to Thursday."}],
    "completion": [{"role": "assistant", "content": "The meeting is now on Thursday."}],
}

If you use plain messages data, the loss by default covers every token, including the user's turn. That is acceptable for a first run, and later you can switch on assistant_only_loss=True to restrict the loss to the assistant's turns. That option needs a chat template that marks the assistant tokens, and TRL patches the templates of well-known model families such as Qwen3 for you.

A chat template deserves a sentence of its own. It is a small Jinja program, stored with the tokenizer, that turns a list of messages into the exact string of special tokens that model was trained to expect. You never need to write it for a mainstream model; the trainer applies it for you when it sees conversational data. This is the main reason conversational format is convenient: the formatting details are handled once, correctly.

You can load a dataset from the Hub in one line and inspect it before spending a minute of GPU time.

from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")
print(dataset)               # column names and number of rows
print(dataset[0])            # the first example

Read the output. trl-lib/Capybara is a conversational dataset with a messages column, which is why SFT accepts it with no preparation. Whenever you swap in your own data, run these two lines first and check that the column names match the table above. If they do not, rename the columns with dataset.rename_column("old", "new") before you hand the dataset to a trainer. If you only have plain strings, put them in a text column.

For your own data, the simplest route is a JSON Lines file with one example per line, which you can load like this.

dataset = load_dataset("json", data_files="my_data.jsonl", split="train")

Your first training run: SFT in a few lines

Now the payoff. The smallest useful TRL program is this, taken from the official quickstart pattern.

from datasets import load_dataset
from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()

Four things are worth noticing. First, model is just a string. TRL downloads the model and its tokenizer from the Hub and builds them for you, so you do not need to import AutoModelForCausalLM for the simple case. Second, there is no args at all, so TRL uses defaults. Third, there is no peft_config, so this run is full fine-tuning: every weight is trained. For a 0.6-billion-parameter model that is just about possible on a good GPU, but it is not what we want to teach you, and the full Capybara dataset is large enough to take a long time. Fourth, the tokenizer is quietly loaded and wired in as the trainer's processing_class.

TRL's configs differ from plain TrainingArguments in a few defaults that are worth knowing, because they explain behaviour you might otherwise find surprising.

• logging_steps is 10, not 500, so you see the loss early and often.
• gradient_checkpointing is on. This trades about a fifth of your speed for a large memory saving by recomputing some values instead of storing them.
• bf16 is on unless you set fp16. This is a 16-bit number format that is stable for training on modern GPUs.
• The default learning rate for SFT is 2e-5.
• max_length defaults to 1024 tokens. Examples longer than that are truncated.

Do not run the basic script on the whole dataset as your first experiment. Let us shrink it so that the whole cycle takes a few minutes, and let us also use LoRA, which is the next section. For now, the point is only to know the shortest path exists.

To keep your first experiment short, you can limit the work with the standard max_steps setting, and slice the dataset. This is the structure of every script you will write from now on.

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train[:1000]")

config = SFTConfig(
    output_dir="qwen3-sft-full-test",
    max_steps=20,
    per_device_train_batch_size=2,
    report_to="none",
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=config,
    train_dataset=dataset,
)
trainer.train()

Run it with python train_small.py. The slice train[:1000] is datasets syntax for the first thousand rows. Setting report_to="none" turns off experiment tracking, which we will come back to. The first run downloads the model, which takes a minute depending on your connection; later runs reuse the local cache.

You will see a progress bar and, every ten steps, a dictionary of numbers. We read those numbers in a later section. For now, confirm that the loss is a finite number and that the run finishes without an error.

Adding LoRA with PEFT

Now the central technique. To switch the run above from full fine-tuning to LoRA, you add one argument: a LoraConfig passed as peft_config. TRL sees it, wraps the model with adapters, freezes everything else, and trains only the adapters.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train[:2000]")

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

config = SFTConfig(
    output_dir="qwen3-sft-lora",
    learning_rate=2e-4,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    report_to="none",
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=config,
    train_dataset=dataset,
    peft_config=peft_config,
)

trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model("qwen3-sft-lora")

Let us walk through each choice, because each is a lever you will turn later.

r=16 is the rank. Higher means more capacity and more memory. For learning a style or a format, 8 to 32 is typical. lora_alpha=32 is the scale numerator; with alpha at twice the rank, the correction is multiplied by 2. A common habit is to keep alpha at one or two times the rank and to tune the learning rate instead. lora_dropout=0.05 randomly drops a small share of the adapter's activations during training, a mild guard against overfitting.

target_modules="all-linear" adds adapters to every linear layer of the model except the output layer. Older tutorials target only the attention projections, such as q_proj and v_proj. TRL's own guidance, summarised in its "LoRA Without Regret" page, is that applying LoRA to all weight matrices works better than attention-only even when the total parameter count is matched, so "all-linear" is both the easiest and the recommended choice.

task_type="CAUSAL_LM" tells PEFT this is a next-token-prediction model, so it saves the right things. The other task types, such as sequence classification, matter when you build a classifier.

There is one detail that trips people up. LoraConfig has its own defaults, which are r=8, lora_alpha=8 and lora_dropout=0.0. The TRL command-line tool uses different defaults, 16, 32 and 0.05. So a LoraConfig() with no arguments is not the same as what you get from the CLI with --use_peft. In your own scripts, always write the numbers down explicitly so that you never depend on a default.

The learning rate is the other important change. For full fine-tuning the SFT default is 2e-5. For LoRA the TRL documentation recommends roughly ten times higher, 2e-4. This is a very common surprise: the adapter starts from zero effect and needs bigger steps to move. If you copy a full fine-tuning learning rate into a LoRA run, the loss will crawl.

per_device_train_batch_size=2 with gradient_accumulation_steps=8 gives an effective batch of 16 examples per update on one GPU, while only ever holding two in memory at once. Gradient accumulation means the trainer runs eight small batches, adds up their gradients, and only then updates the weights. It is the standard trick for getting a big batch on a small GPU. The effective batch size is per-device batch size times the number of devices times the accumulation steps.

When the script starts, print_trainable_parameters() prints a line like the following. The exact numbers depend on the model.

trainable params: 10,092,544 || all params: 606,142,464 || trainable%: 1.6652

That single line is the whole point of PEFT. You are about to train under two percent of the model's weights. If it prints trainable%: 100, no adapter was attached, so check that you passed peft_config. If it prints a tiny number such as 0.0, your target_modules matched nothing useful.

Reading the training log

While the training runs, a line is printed every ten steps. A representative line looks like this.

{'loss': 1.4213, 'grad_norm': 0.6821, 'learning_rate': 0.000187, 'entropy': 1.31, 'num_tokens': 163840.0, 'mean_token_accuracy': 0.6702, 'epoch': 0.08}

Here is how to read the SFT numbers.

loss is the number to watch. It should start somewhere around 1 to 3 for a chat model on general data and trend downward. It will jump around from line to line, because each line reflects different examples, so judge the trend over many lines rather than any single value. A loss that never moves at all is a warning sign, and so is a loss that falls to nearly zero on a small dataset, which usually means the model is memorising it.

mean_token_accuracy is the fraction of next tokens the model predicts exactly right. It rises as the model learns your data's patterns. It is a friendlier number than loss for a quick sanity check.

grad_norm is the size of the gradient. You mostly care that it is a finite, reasonable number. If it becomes nan or enormous, training has gone unstable, and lowering the learning rate is the first remedy.

To see these over time instead of scrolling a terminal, plug in an experiment tracker. TRL's documentation recommends Trackio, an open-source tracker from Hugging Face; install it and set report_to="trackio". Weights & Biases, TensorBoard and MLflow work through the same report_to setting. Our guides on MLflow and Weights & Biases cover those tools in depth. For a first run, the terminal is enough, and report_to="none" keeps it simple.

One more habit worth forming now: hold some data back. A model can drive its training loss down simply by memorising the examples. If you give the trainer an eval_dataset and set eval_strategy="steps" with an eval_steps value, it will also report an eval_loss on data it never trained on. When training loss keeps falling while eval loss rises, you are overfitting, and you should stop earlier, use less rank, or add more varied data.

Saving, loading and merging adapters

When training finishes, trainer.save_model("qwen3-sft-lora") writes the result. Because you used PEFT, it saves only the adapter, not the base model. Look inside the folder.

ls -lh qwen3-sft-lora

You will find adapter_config.json, which records the LoRA settings and the name of the base model, and adapter_model.safetensors, which holds the trained numbers. The folder is a few tens of megabytes instead of gigabytes. That is the practical magic: you can email it, keep fifty of them, or push each to the Hub.

An adapter is useless without its base model, so to use it you load both. The right tool for loading a trained adapter is PeftModel.from_pretrained.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
model = PeftModel.from_pretrained(base, "qwen3-sft-lora")
model.eval()

messages = [{"role": "user", "content": "Explain gradient accumulation in two sentences."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Note dtype="auto". Older code wrote torch_dtype; the current name is dtype. Note also model.eval(), which switches off dropout so that the output is deterministic in character. A shortcut is AutoPeftModelForCausalLM.from_pretrained("qwen3-sft-lora"), which reads the base model name from adapter_config.json and loads both in one call.

Sometimes you want a single standalone model with no adapter, perhaps to hand to a serving system that does not know about PEFT. For that you merge: PEFT folds the adapter's correction into the base weights and removes the adapter machinery.

merged = model.merge_and_unload()
merged.save_pretrained("qwen3-merged")
tokenizer.save_pretrained("qwen3-merged")

The qwen3-merged folder is now an ordinary Hugging Face model that loads with AutoModelForCausalLM.from_pretrained("qwen3-merged") and that tools such as vLLM can serve. The trade-off is size: a merged model is a full copy, gigabytes instead of megabytes. Keep the small adapter as your source of truth and merge only when you need to.

If you trained with a quantized base model (the QLoRA technique later in this guide), reload the base in full or bf16 precision before merging. Merging into a 4-bit base introduces rounding error.

Using the command line instead of a script

You do not have to write Python for standard runs. The trl command wraps the trainers, and it is ideal for quick experiments and for runs you want to record as a one-line command in a README. The subcommands are trl sft, trl dpo, trl grpo, trl rloo, trl kto, trl reward, trl distillation, and trl env, which you have already met.

The same LoRA run as our script looks like this.

trl sft \
  --model_name_or_path Qwen/Qwen3-0.6B \
  --dataset_name trl-lib/Capybara \
  --output_dir qwen3-sft-cli \
  --use_peft \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_target_modules all-linear \
  --learning_rate 2e-4 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 8 \
  --num_train_epochs 1

The flag names are the same as the Python argument names, which is easy to remember. For longer runs, put the settings in a YAML file whose keys are the flag names without dashes, then point the command at it.

model_name_or_path: Qwen/Qwen3-0.6B
dataset_name: trl-lib/Capybara
output_dir: qwen3-sft-cli
use_peft: true
lora_r: 16
lora_alpha: 32
lora_target_modules: all-linear
learning_rate: 2.0e-4
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
num_train_epochs: 1

trl sft --config sft_config.yaml

The settings that matter most

Beyond the LoRA choices, a handful of config settings account for most of what you will tune. Learn these before the hundred others.

Learning rate. The single most influential number. For LoRA SFT, start at 2e-4. If the loss spikes or turns into nan, halve it. If the loss barely moves after a hundred steps, double it.

Epochs or steps. num_train_epochs counts passes over the data. max_steps counts updates and, if set, wins over epochs. For small datasets, one to three epochs is typical. More epochs on a small dataset are the commonest route to overfitting.

Batch size and accumulation. per_device_train_batch_size is limited by memory, so make it as large as fits, then use gradient_accumulation_steps to reach an effective batch of about 16 to 32. TRL's guidance for LoRA is to keep the effective batch below about 32.

max_length. The longest sequence, in tokens, that the trainer will keep. Default 1024. Longer examples are cut off at the end, which silently removes the end of your answers if your data is long. Memory grows quickly with this number, so raise it deliberately, not by habit.

packing. Setting packing=True concatenates several short examples into one max_length sequence so that no compute is wasted on padding. It can speed up training a lot when your examples are short. It needs max_length to be set, and it is not supported for image data.

loss_type. Since TRL 1.7 the SFT default is chunked_nll, which has the same maths as plain NLL but computes the loss in chunks so that peak memory is lower, by around 30 percent on average. You normally leave it alone. It does have one interaction worth knowing: it does not work if you put an adapter on the lm_head layer, which target_modules="all-linear" avoids by excluding the output layer.

Here is a config that sets the important choices explicitly, which is a good habit: it documents your run and protects you when a default changes in a future release. Several TRL defaults did change in the past year, which is the practical reason to be explicit.

from trl import SFTConfig

config = SFTConfig(
    output_dir="qwen3-sft-lora",
    learning_rate=2e-4,
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    max_length=1024,
    packing=False,
    logging_steps=10,
    save_steps=200,
    seed=42,
    report_to="none",
)

QLoRA: fitting a bigger model on a small GPU

LoRA shrinks what you train, but the frozen base model still has to sit in memory. For a small model that is fine. For a seven-billion-parameter model it is not, and that is where QLoRA comes in. QLoRA first quantizes the frozen base model to 4-bit numbers, which cuts its memory to roughly a quarter, and then trains LoRA adapters on top of it in higher precision. Quantization means storing each weight with fewer bits, accepting a small loss of precision in exchange for a large saving of memory. The library that does the 4-bit work is bitsandbytes.

In current TRL the way to ask for this is the quantization_config argument on the trainer, available since TRL 1.8. Install the extra first.

pip install "trl[quantization]"

Then pass a BitsAndBytesConfig.

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=SFTConfig(output_dir="qwen3-qlora", learning_rate=2e-4, report_to="none"),
    train_dataset=load_dataset("trl-lib/Capybara", split="train[:2000]"),
    quantization_config=bnb,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()

nf4 is a 4-bit number format designed for the bell-shaped distribution of neural network weights. Double quantization squeezes the bookkeeping numbers too. bnb_4bit_compute_dtype says that the arithmetic itself is done in bf16, even though the stored weights are 4-bit.

Two rules keep you out of trouble. Set quantization_config as a trainer argument only, not also inside model_init_kwargs, or the trainer raises an error. And if you use raw PEFT without TRL, you must call prepare_model_for_kbit_training(model) before get_peft_model; TRL does that preparation for you.

Common errors and how to read them

Every beginner meets the same handful of failures. Read the last line of a Python traceback first, since the exception type and message sit there, then match it here.

torch.OutOfMemoryError: CUDA out of memory. The model, the sequence length or the batch is too big for the GPU. Work down this list: set per_device_train_batch_size=1 and raise gradient_accumulation_steps to compensate; lower max_length; use LoRA, then QLoRA; pick a smaller model. Check that nothing else is using the GPU with nvidia-smi, since a forgotten notebook can hold all the memory.

ImportError: You passed peft_config but the peft library is not installed. Install it with pip install trl[peft]. You installed trl without the extra. Run pip install "trl[peft]".

TypeError: peft_config must be a peft.PeftConfig instance (e.g. peft.LoraConfig), got dict. You passed a dictionary. Build a LoraConfig(...) object instead.

KeyError: The input examples must contain either 'messages' for conversational data or 'text' for standard data. Your dataset's columns are not what SFT expects. Print dataset.column_names and rename a column to text or messages, or use prompt and completion.

ValueError: You passed a PeftModel instance together with a peft_config to the trainer. Pick one: give a plain model name with peft_config, or give an already-wrapped model without it.

ValueError: loss_type='chunked_nll' is not supported when lm_head is wrapped by a PEFT adapter. Something targeted the output layer. Remove lm_head from target_modules, or set loss_type="nll".

ValueError: When packing is enabled, max_length can't be None. Give max_length a number when packing=True.

RuntimeError: You're using assistant_only_loss=True, but at least one example has no assistant tokens. The chat template has no markers for assistant tokens. Leave assistant_only_loss off, or use a model family whose template TRL patches.

ValueError: Attempting to unscale FP16 gradients. The model was loaded in 16-bit float and trained with fp16 mixed precision, leaving trainable weights in 16-bit. Prefer bf16 on modern GPUs, which is TRL's default.

ModuleNotFoundError: No module named 'trl.losses' or ImportError: cannot import name 'PPOTrainer' from 'trl'. You are running code written for an older TRL. Those pieces were removed in 2026. PPO was dropped in TRL 1.13, and for online reinforcement learning the library now points you to GRPO or RLOO. Do not fight this; port the code to a current trainer.

Then there are the silent problems, which print no error at all.

• The loss barely moves. Some of your target_modules names did not match any layer. PEFT raises an error only if none match, and quietly skips partial mismatches. Prefer "all-linear", and check print_trainable_parameters().
• The tuned model acts like the base model. You probably used get_peft_model where you needed PeftModel.from_pretrained, or forgot model.eval(), or pointed at the wrong folder.

Finally, a pragmatic tip: when you must ask for help, run trl env and paste its output together with the full traceback and the exact versions. The maintainers ask for that.

A first look beyond SFT: DPO and GRPO

SFT teaches a model to imitate. Sometimes you want to teach it to prefer. That is what the other TRL trainers are for, and you only need a taste of them now.

DPO, Direct Preference Optimization, learns from pairs. Each training row has a prompt, a chosen answer people liked and a rejected answer they did not. The model learns to make the chosen one more likely than the rejected one, without any separate scoring model. A frozen reference model acts as an anchor so that the model does not wander too far from where it started, and a number called beta, default 0.1, controls how tight that anchor is. When you use PEFT, TRL does not load a second copy of the model: the reference is simply the base model with the adapter switched off.

from datasets import load_dataset
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    args=DPOConfig(output_dir="dpo-lora", learning_rate=5e-6, beta=0.1, report_to="none"),
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train[:1000]"),
    peft_config=LoraConfig(task_type="CAUSAL_LM"),
)
trainer.train()

Notice that the learning rate is far smaller than in SFT. DPO is sensitive, and TRL's recommendation for LoRA is about 5e-6.

GRPO, Group Relative Policy Optimization, is online. For each prompt the model writes several answers, a reward function that you write scores each one, and the model learns to produce more of the answers that scored above the group average. It is how models are taught to reason in ways that can be checked automatically, such as getting a maths answer right. A reward function is just Python: it receives the completions and returns one number per completion. GRPO needs more memory and, ideally, a generation engine such as vLLM, so leave it until the basics are second nature. The mid-level guide covers it properly.

Putting it all together

Here is one small, complete project that uses every piece above. The goal: take your "behaviour sentence" from the first section, teach it to a small model, and prove that the change took.

First, write the data. Make about forty examples in a JSON Lines file called my_data.jsonl, one JSON object per line, each with a messages list. If your behaviour is "reply in two short sentences", write forty varied questions with two-sentence answers. Vary the topics, because a model trained on near-duplicates learns only the duplicates. Here are the first lines of such a file.

{"messages": [{"role": "user", "content": "What is a GPU?"}, {"role": "assistant", "content": "A GPU is a chip built to do many simple calculations in parallel. That makes it ideal for graphics and for training neural networks."}]}
{"messages": [{"role": "user", "content": "What is Docker?"}, {"role": "assistant", "content": "Docker packages an application with everything it needs into a portable image. You can then run that image as a container on any machine."}]}

Second, the training script, which uses LoRA and holds back a few rows for evaluation.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files="my_data.jsonl", split="train")
splits = data.train_test_split(test_size=0.1, seed=42)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=SFTConfig(
        output_dir="two-sentence-adapter",
        learning_rate=2e-4,
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=2,
        eval_strategy="epoch",
        logging_steps=5,
        seed=42,
        report_to="none",
    ),
    train_dataset=splits["train"],
    eval_dataset=splits["test"],
    peft_config=LoraConfig(
        r=16, lora_alpha=32, lora_dropout=0.05,
        target_modules="all-linear", task_type="CAUSAL_LM",
    ),
)

trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model("two-sentence-adapter")

Third, compare the base model and the tuned model on a question that is not in your data. Reuse load_adapter.py with the new folder name, ask something new, then ask the same thing again with the adapter removed, by using the plain base object. If the tuned answer is two short sentences and the base answer rambles, your fine-tune worked.

Fifth, write down what you did. Record the library versions from trl env, the dataset file, the config values and the result of your before-and-after check in a short README beside the adapter. Six weeks from now, with different library versions installed, this note is the difference between a repeatable result and a mystery.

What you can now do, and what comes next

You can now install TRL and PEFT and verify the setup, explain in plain words what a base model, an adapter, LoRA, rank and a trainer are, and prepare a dataset in the shape each trainer expects. You can run supervised fine-tuning with LoRA or QLoRA from a script or from the command line, read the loss and token accuracy in the training log, and hold out data to spot overfitting. You can save an adapter, load it correctly, merge it into a standalone model, and recognise the errors that stop most first attempts. And you know where DPO and GRPO fit, so you will not mistake them for beginner topics or for things you must learn today.

The natural next steps follow from here. Mid-level covers preference tuning with DPO in earnest, GRPO with your own reward functions, memory tuning, multi-GPU training with accelerate, and experiment tracking. Senior covers running training and serving adapters as a platform for a team. Alongside this guide, the Hugging Face guide covers the Hub, datasets and transformers underneath, Unsloth offers a faster fine-tuning stack built around the same ideas, DeepSpeed is how very large models are spread over many GPUs, and vLLM is how you serve the model you just trained.

Sources

• TRL documentation
• TRL installation
• TRL quickstart
• TRL command-line interface
• SFT Trainer
• DPO Trainer
• GRPO Trainer
• Dataset formats and types
• PEFT integration in TRL
• LoRA Without Regret
• Reducing memory usage
• TRL usage statistics
• TRL releases on GitHub
• TRL migration guide
• PEFT documentation
• PEFT installation
• PEFT quicktour
• PEFT troubleshooting
• PEFT checkpoint format
• PEFT quantization guide
• PEFT LoRA reference
• PEFT releases on GitHub

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about fine-tuning with PEFT and TRL. Mid-level reviews this plus its own material; Senior reviews all three. Versions in this guide: TRL 1.14 and PEFT 0.21.

The thirty-second answer

PEFT is Hugging Face's library for parameter-efficient fine-tuning: it freezes a pretrained model and trains a small set of extra weights, called an adapter, most often with LoRA. TRL is Hugging Face's post-training library: it provides trainers such as SFTTrainer and DPOTrainer that run the training loop. Together they let you fine-tune a large model on one modest GPU and ship a few-megabyte adapter instead of a full copy of the model.

Then the line that shows you have really used them: "With PEFT the trainer saves only the adapter, so to use it I load the base model and attach the adapter with PeftModel.from_pretrained, and I never call get_peft_model on a model I have already trained."

The model

Why LoRA works, in one breath

When you fine-tune for a specific purpose, the change needed in each weight matrix is usually simple. LoRA exploits that by representing the change as the product of two thin matrices instead of a full grid. B starts at zero, so training begins exactly at the base model's behaviour. Only the thin matrices receive gradients and optimiser state, so memory and storage drop sharply. The price is a ceiling on how much the model can change, which rarely matters for style and format tasks and can matter for teaching large amounts of new knowledge.

The commands and calls to have ready

pip install "trl[peft]"            # TRL plus PEFT; install PyTorch for your hardware first
trl env                            # versions and hardware, paste this into bug reports
trl sft --model_name_or_path Qwen/Qwen3-0.6B --dataset_name trl-lib/Capybara \
        --output_dir out --use_peft --lora_r 16 --lora_alpha 32 --lora_target_modules all-linear

from peft import LoraConfig, PeftModel
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=SFTConfig(output_dir="out", learning_rate=2e-4),
    train_dataset=dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model("out")                          # saves only the adapter

model = PeftModel.from_pretrained(base, "out")     # load a trained adapter
merged = model.merge_and_unload()                  # standalone model

Be able to explain each line in words. Interviewers often hand you a snippet like this and ask what is wrong with it, so also know the removed forms: tokenizer=, max_seq_length=, dataset_text_field= as trainer arguments, torch_dtype, PPOTrainer, and DataCollatorForCompletionOnlyLM are all stale.

The classic traps, and why they happen

You forget that LoRA wants a bigger learning rate. Full fine-tuning in TRL defaults to 2e-5 for SFT. With LoRA the documentation recommends about ten times more, 2e-4, because the adapter starts at zero effect and needs larger steps to move. Symptom: the loss barely drops. Cause: a learning rate copied from a full fine-tune.

You call get_peft_model on a trained adapter. That function builds a new, randomly initialised adapter. The model then acts like the base model and nothing crashes. The right call to load a trained adapter is PeftModel.from_pretrained.

You expect peft_config to be optional in the CLI. On the command line, LoRA is off unless you pass --use_peft. Without it, you do a full fine-tune and probably run out of memory.

You trust LoraConfig() defaults. LoraConfig defaults to r=8, lora_alpha=8 and lora_dropout=0.0, while the TRL command line defaults to 16, 32 and 0.05. Always write the numbers down.

Your dataset columns do not match the trainer. SFT expects text, messages, or prompt plus completion. DPO expects prompt, chosen and rejected. GRPO expects prompt. The error is usually a KeyError naming the columns it wanted.

You install trl without the peft extra. TRL does not install PEFT by default, so using peft_config raises an ImportError that tells you to install trl[peft].

You pass a PEFT model and a peft_config together. TRL refuses. Pass one or the other.

You merge into a quantized base. A QLoRA adapter merged into a 4-bit model picks up rounding error. Reload the base in bf16 and merge there.

You copy old code. trl.losses and PPOTrainer are gone in current TRL. Many tutorials still use them. The trainers that are stable in 1.14 are SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer, KTOTrainer, RewardTrainer and DistillationTrainer.

Common interview questions

What is the difference between PEFT and TRL? They solve different problems. PEFT decides what to train: it freezes the model and adds a small adapter. TRL decides how to train it: the loss, the data handling and the loop, through trainer classes built on transformers.Trainer. You can use either alone, but they are usually paired by passing peft_config to a TRL trainer.

Explain LoRA to someone who has not seen it. Instead of changing a big weight matrix directly, we leave it frozen and learn two small matrices whose product is a correction of the same shape. The layer's output is the original output plus a scaled correction. Because the small matrices hold a fraction of the parameters, we need far less memory for gradients and optimiser state, and the adapter file is tiny. B starts at zero so we begin from the base model's behaviour.

What are rank and alpha, and how do you pick them? Rank is the inner dimension of the two matrices and sets capacity. Alpha scales the correction by alpha over rank. For style and format tasks I start with rank 16 and alpha 32, then adjust the learning rate before I touch the rank. A bigger rank costs memory and storage and helps when the task is complex or the dataset is large.

What does target_modules="all-linear" do, and why use it? It places adapters on every linear layer except the output layer. Older recipes targeted only attention projections. TRL's LoRA guidance says that applying LoRA to all weight matrices works better than attention-only, even at matched parameter counts, so it is the simple and recommended choice.

What is QLoRA and when would you use it? QLoRA quantizes the frozen base model to 4-bit with bitsandbytes, using the NF4 format, and trains LoRA adapters on top in higher precision. I use it when the base model will not fit in memory otherwise, for example a 7B model on a single small GPU. It is slightly slower than plain LoRA, it needs an NVIDIA GPU for 4-bit training, and in current TRL I pass it as the quantization_config argument on the trainer.

What does SFT actually optimise? Token-level cross-entropy, also called negative log-likelihood: how surprised the model is by each correct next token. With prompt-completion data the loss is computed on the completion only by default. With plain messages data the loss covers all tokens unless I set assistant_only_loss=True, which needs a chat template that marks assistant tokens.

What are the two dataset formats and the common types? Standard format uses plain strings, and conversational format uses lists of role and content messages. Types are language modelling, prompt-completion, preference with chosen and rejected, unpaired preference with a label, and prompt-only. Each trainer expects a particular type.

How do you know training is working? The loss trends downward over many log lines, though it is noisy line to line. mean_token_accuracy rises. grad_norm stays finite. I hold out an eval set and watch eval_loss; if training loss keeps falling while eval loss rises, I am overfitting. And I read some real generations, because a good-looking loss does not guarantee good behaviour.

What gets saved when you train with PEFT, and how do you use it? Only the adapter: adapter_config.json and adapter_model.safetensors. The config records the base model name. To use it I load the base model and call PeftModel.from_pretrained(base, path), or use AutoPeftModelForCausalLM.from_pretrained(path), and call .eval().

What does merging do, and what is the trade-off? merge_and_unload() folds the correction into the base weights and returns an ordinary model with no adapter overhead, which is easy to serve. The cost is that every merged variant is a full model copy, gigabytes instead of megabytes. I keep the adapter as the source of truth and merge when a serving system needs a plain model.

What is the effective batch size? Per-device batch size times the number of devices times the gradient accumulation steps. Accumulation lets a small GPU act like a larger batch by summing gradients over several small batches before one weight update.

What is DPO, and what does beta do? Direct Preference Optimization trains on prompt, chosen and rejected triples without a separate reward model. A frozen reference model anchors the policy, and beta, default 0.1, controls how far the policy may drift from it. With PEFT the reference is the base model with the adapter disabled, so no second copy is loaded.

What is the difference between offline and online methods? Offline methods such as SFT, DPO and KTO learn from a dataset that already exists. Online methods such as GRPO and RLOO generate completions from the current model during training, score them with reward functions, and learn from those scores. Online methods need more memory and usually a fast generation engine such as vLLM.

Why is gradient checkpointing on by default in TRL, and what does it cost? It saves memory by recomputing some activations during the backward pass instead of storing them. The cost is roughly a fifth of training speed. TRL's configs differ from plain TrainingArguments here, along with logging_steps=10 and bf16 on by default.

Which TRL trainers would you avoid, and why? Anything under trl.experimental. The official contract says those APIs can change or be removed in any release without deprecation, so they are poor foundations for production. PPO has been removed entirely in TRL 1.13, and for online reinforcement learning I would use GRPO or RLOO.

Does fine-tuning teach a model new facts? Poorly and unreliably. It is excellent at behaviour: format, tone, task framing. For facts that change or must be accurate, retrieval is generally a better fit, and the two combine well.

What would you check before fine-tuning at all? Whether prompting with a few examples already works, whether I have enough clean data in the right format, how I will evaluate the result, and whether data-residency rules mean the training must run in a particular country or region.

A colleague's tuned model gives the same answers as the base model. What do you check? First whether they used get_peft_model where PeftModel.from_pretrained was needed. Then whether they pointed at the right adapter folder and called .eval(). Then whether training ever moved: print_trainable_parameters should show a small non-zero percentage, and the loss should have fallen. A target_modules list that matched nothing, or only partly, can also leave the adapter doing very little.

What do you do about telemetry in a regulated environment? TRL sends one anonymous ping per trainer instantiation since version 1.5. I set HF_HUB_DISABLE_TELEMETRY=1, or HF_HUB_OFFLINE=1 for an offline cluster with a pre-populated cache, and I document that choice.

What is packing, and when does it help? Setting packing=True concatenates several short examples into one sequence of max_length tokens, so the GPU does not spend time on padding. It can speed training substantially when examples are short. It requires max_length to be set, and it is not supported for image data.

Why do some people call the learning rate the most important setting? Because it decides whether training moves at all. Too low and the loss crawls, too high and it spikes or turns into nan. I set it deliberately for the method: about 2e-4 for LoRA SFT, and far smaller values for DPO, where the documented LoRA rate is 5e-6.

How would you explain overfitting to a manager? The model has memorised the training examples instead of learning the pattern, like a student who memorised last year's exam. Training loss looks excellent while answers to new questions get worse. I catch it with a held-out set, and I fix it with fewer epochs, a smaller rank or more varied data.

Questions to ask back

Strong candidates ask about the situation as well as answering. Good questions: How much data do we have, and who checked its quality? What does success look like, and how will we measure it? Where does the training data live, and which regions may it pass through? Which model licence applies? Is an adapter acceptable for serving, or must we merge? These show you treat fine-tuning as an engineering decision, not a button.

What a strong answer covers that a weak one misses

Sixty-second self-test

Answer each aloud, then check yourself against the sections above.

1. What are PEFT and TRL, and which one decides what is trained?
2. Why does LoRA need so much less memory than full fine-tuning?
3. What are r, lora_alpha and target_modules, and what are sensible starting values?
4. Which learning rate would you start with for LoRA SFT, and why is it higher than for a full fine-tune?
5. What does trainer.save_model write when PEFT is in use?
6. How do you load a trained adapter, and which function must you not use for that?
7. What does merge_and_unload() return and what is the cost?
8. What does the CLI need to switch LoRA on?
9. Name three SFT dataset shapes and the DPO shape.
10. What do you watch in the log to know training is healthy?
11. Which two things have been removed from current TRL that old tutorials still use?
12. When would you use QLoRA instead of LoRA?

If you can answer all twelve without looking, you are ready for a beginner-level conversation on this topic. If you stumbled on more than three, reread the section that covers them, then run the small project at the end of the detailed track once more.

For neighbouring topics, the Hugging Face guide covers the Hub and transformers underneath, and the vLLM guide covers serving the model you tuned.

Part one of three. Most beginner failures in PEFT and TRL come from four causes: stale code from an older release, data that does not match the trainer, a learning rate chosen for the wrong kind of training, and confusion between a trainable adapter and a finished model. Start with the table, then the longer explanations, the habits, and the checklist. Versions here are TRL 1.14 and PEFT 0.21.

Common errors at this level

The mistakes beginners actually make

Each of these follows the same shape: the symptom, the cause, the fix.

Copying a tutorial that describes an older release

Symptom. A snippet from a blog post, a forum answer or an AI assistant fails with an import error, or an unexpected-keyword error on tokenizer, max_seq_length or dataset_text_field.

Cause. Both libraries changed a lot between 2024 and 2026. The tokenizer now goes in as processing_class. Sequence length lives in the config as max_length, and the text column name lives there as dataset_text_field rather than on the trainer. DataCollatorForCompletionOnlyLM is gone, replaced by the config option completion_only_loss, which already defaults to applying the loss to the completion for prompt-completion data. PPOTrainer was removed in TRL 1.13, and the trl.losses module, which existed only briefly, was removed in 1.14.

Fix. Check the date and the version of any snippet before trusting it. Compare against the official documentation for the version you have installed, and print your versions with trl env. When something fails on import, search the release notes on GitHub for the name before you try to patch the code.

Not pinning versions

Symptom. A script that worked last month fails after a reinstall, or a colleague gets different results with the "same" code.

Cause. TRL releases about every two weeks, and PEFT is still pre-1.0. Defaults change. For example, the default SFT loss became chunked_nll in TRL 1.7, and several GRPO defaults flipped in 1.10.

Fix. Write your versions into a requirements.txt with exact pins, such as trl==1.14.1 and peft==0.21.1, and also pin transformers, accelerate and torch. Upgrade on purpose, one at a time, after reading the release notes.

Using a full-fine-tuning learning rate with LoRA

Symptom. The loss decreases very slowly and the tuned model behaves almost like the base model after a full epoch.

Cause. The SFT default learning rate in TRL is 2e-5, which is tuned for updating all weights. A LoRA adapter starts at zero effect and needs bigger steps. The TRL documentation recommends roughly ten times the full fine-tuning rate with LoRA: 2e-4 for SFT. It is the other direction for DPO, where the documented LoRA rate is a small 5e-6.

Fix. Set the learning rate explicitly in your config for every run, with a comment on why. If the loss is flat, double it. If the loss spikes or turns into nan, halve it.

Loading a trained adapter with get_peft_model

Symptom. You trained, saved, reloaded, and the model acts like the untuned base. Nothing raised an error.

Cause. get_peft_model(model, config) creates a new, randomly initialised adapter. It is for starting training. A trained adapter is loaded with PeftModel.from_pretrained(base, path).

Fix. Use PeftModel.from_pretrained, or AutoPeftModelForCausalLM.from_pretrained(path), and call .eval() before generating. If you want to keep training the adapter, pass is_trainable=True to AutoPeftModelForCausalLM.from_pretrained, and then hand that model to the trainer without a peft_config.

Pointing the trainer at data with the wrong columns

Symptom. A KeyError listing messages and text, or training that runs but learns the wrong thing.

Cause. Each trainer expects a particular dataset type. SFT wants text, messages, or prompt plus completion. DPO wants prompt, chosen and rejected. GRPO wants prompt. A dataset with a column called question and one called answer is none of these.

Fix. Print dataset.column_names and dataset[0] before you build a trainer. Rename columns with dataset.rename_column, or convert rows to the messages structure with dataset.map. This takes two minutes and prevents most data errors.

Training on data you never read

Symptom. The model produces odd artefacts: stray labels, repeated phrases, inconsistent formatting, answers that stop mid-sentence.

Cause. The model copies the data. If your examples contain HTML leftovers, inconsistent answer styles or truncated text, the model learns them.

Fix. Read at least fifty random rows. Remove duplicates. Make sure the answers in your examples are the style you want everywhere. Keep a held-out set so you can compare the tuned model on questions it never trained on. A few hundred clean, varied examples usually beat thousands of noisy ones.

Truncating your own answers without noticing

Symptom. Tuned answers cut off early, or the model never learned to finish a long reply.

Cause. max_length defaults to 1024 tokens in the SFT config. Longer examples are truncated from the end, which removes the end of the answer. The model then never sees how to finish.

Fix. Measure your example lengths in tokens, using the tokenizer, before training. Raise max_length if needed, accepting the memory cost, or shorten the examples. Remember that memory grows quickly with sequence length.

Out-of-memory on the very first run

Symptom. torch.OutOfMemoryError: CUDA out of memory, often right after the model loads or at the first step.

Cause. Training needs memory for the weights, the gradients, the optimiser state and the activations. Activations scale with batch size and sequence length.

Fix, in order. Check nvidia-smi for other processes holding memory, such as a forgotten notebook. Set per_device_train_batch_size=1 and raise gradient_accumulation_steps to keep the effective batch. Lower max_length. Use LoRA if you are doing a full fine-tune, then QLoRA if you are already on LoRA. Choose a smaller model. Advanced options such as activation_offloading=True exist, but the first five steps solve almost every beginner case.

Forgetting that the CLI does not use LoRA by default

Symptom. A command-line run behaves nothing like your script, uses far more memory, and writes a multi-gigabyte output folder.

Cause. trl sft trains the full model unless you pass --use_peft. The LoRA flags, such as --lora_r, have no effect on their own.

Fix. Add --use_peft, and put every setting in a YAML file so that the run is reproducible. Remember the CLI defaults for r, alpha and dropout, which are 16, 32 and 0.05, differ from those of a bare LoraConfig().

Judging success by the loss alone

Symptom. The loss fell nicely, but the model's answers are bad, or worse than before on cases outside your data.

Cause. A low training loss can mean the model memorised your examples. It says nothing about questions it never saw.

Fix. Always hold out an evaluation set, set eval_strategy, and watch eval_loss. Then read generations for a dozen prompts, including a few unlike anything in the training data. Compare against the base model on the same prompts.

Merging into a quantized model

Symptom. The merged model is a little worse than the adapter in use, or merging fails to produce what you expected.

Cause. Merging a QLoRA adapter into a 4-bit base adds rounding error, because the base weights have already lost precision.

Fix. Load the base model in bf16 or float16 for the merge, attach the adapter, call merge_and_unload(), and save. Keep the original adapter.

Pushing private data to a public place

Symptom. A colleague discovers your adapter or dataset on the Hub, visible to everyone.

Cause. A repository created by push_to_hub may be public depending on how it is created and on your account defaults, and a model trained on confidential text can reveal fragments of it.

Fix. Create private repositories on purpose for company data, check the visibility on the Hub page after every push, and never put an access token in a script or a YAML file that goes into git. Use hf auth login or the HF_TOKEN environment variable. Where residency rules apply, train and store inside the permitted region.

Habits worth adopting on day one

Run a tiny experiment first. Before you start a long run, slice the dataset to a few hundred rows and set max_steps=20. If the script will fail on data, memory or a typo, you find out in two minutes instead of two hours. Then scale up.

Write every setting down. Put learning rate, rank, alpha, epochs, batch settings and max_length explicitly in the config, even when they match a default. Defaults in these libraries have changed more than once, and an explicit config is documentation that does not go stale.

Change one thing at a time. If you alter the rank, the learning rate and the data together, you will never learn which change helped. Keep a short log: what changed, what the loss did, what a few sample outputs looked like.

Look at the trainable-parameter line. print_trainable_parameters() takes one line and catches a missing peft_config (100 percent trainable) or a mismatched target_modules (almost nothing trainable). Make it a reflex.

Prefer "all-linear" for target_modules. It sidesteps the layer-naming differences between model families, and TRL's guidance favours it over attention-only adapters.

Hold out data and read outputs. An eval split and a handful of real prompts, compared against the base model, tell you more than any single loss number.

Use a tracker once runs matter. Set report_to="trackio" after installing Trackio, which TRL's docs recommend, or use MLflow or Weights & Biases through the same setting. Charts of several runs side by side make it easy to compare. For one-off experiments report_to="none" is fine.

Keep the adapter as the source of truth. Adapters are small, so store each one with the config, the dataset version and the output of trl env. Merge only when a serving system needs a standalone model.

Save checkpoints on long runs. Set save_steps to something sensible. If a run dies at 90 percent, trainer.train(resume_from_checkpoint=True) picks up from the last checkpoint instead of starting again.

Set a seed. seed=42 in the config makes comparisons between runs fairer, since you are not also comparing different random starts.

Respect licences. Models and datasets on the Hub carry licences. Read them before building a product, and note that gated models require accepting terms and logging in.

Mind where you run. For employers with data-residency rules, choose the training location deliberately, whether that is a machine you control or a cloud region inside the required country or economic zone, and keep checkpoints and datasets there too.

Practice cards

Spend a few minutes on each. They are small enough to do in a notebook.

Card 1: see the adapter. Train LoRA for twenty steps on any small dataset, then run ls -lh on the output folder. Find adapter_config.json and adapter_model.safetensors. Open the JSON and find the base model name and the rank.

Card 2: break it on purpose. Run once with per_device_train_batch_size=64 and max_length=4096. Read the out-of-memory message. Then fix it and run again.

Card 3: the learning-rate experiment. Train for sixty steps at 2e-5 and again at 2e-4. Compare the final loss. Write one sentence explaining what you saw.

Card 4: the wrong loader. Train and save an adapter. Load it once with PeftModel.from_pretrained and once as the plain base model. Ask both the same question that your data covers. Note the difference. Then try get_peft_model on the base in a third cell and see why it is the wrong tool.

Card 5: read your data. Load a dataset, print twenty random rows, and write down three things you would fix.

Card 6: merge and measure. Merge the adapter, save the merged model, and compare folder sizes. Load the merged model with AutoModelForCausalLM.from_pretrained alone and confirm that it still gives the tuned answer.

Card 7: the CLI twin. Reproduce your best script as a trl sft --config YAML file, using use_peft: true. Check that the output adapter is the same size as the script version.

Card 8: a stale snippet. Find an old tutorial that uses PPOTrainer or tokenizer=. Write what you would replace each part with in current TRL. Check your answer against the official docs.

When to ask for help, and how

Before you post a question to a forum or open an issue, collect the facts. Run trl env and copy the output. Copy the full traceback, not only the last line. Keep the smallest script that reproduces the problem, with a small public dataset instead of your private one. Note what you already tried. The TRL documentation explicitly asks for the trl env output when reporting a bug, and a report that includes it gets answered faster.

Search first. The error messages in TRL and PEFT are usually written in plain English and often contain the fix in the sentence itself, such as "Install it with pip install trl[peft]" or "Please first merge and unload the existing adapter". Read the whole message twice before you search.

A short pre-flight checklist

Run through this before every training job that matters.

• Python is 3.10 or newer, and PyTorch matches my hardware.
• trl env shows the versions I intended, and they are pinned in a requirements file.
• I printed dataset[0] and the columns match the trainer I am using.
• I read a sample of the rows and removed duplicates and obvious junk.
• I have a held-out evaluation split.
• My LoraConfig has explicit r, lora_alpha, lora_dropout, target_modules and task_type.
• The learning rate is set on purpose: about 2e-4 for LoRA SFT.
• per_device_train_batch_size fits in memory, and accumulation gives a sensible effective batch.
• max_length is long enough for my longest example.
• print_trainable_parameters() shows a small, non-zero percentage.
• I ran a twenty-step test before the full run.
• The output folder, the seed and the tracking setting are what I want.
• I know where the data and the checkpoints are stored, and that location is allowed.
• No tokens or passwords are in my script, my YAML or my git history.

A short post-flight checklist

Run through this after the job finishes.

• The loss trended down and the final lines of the log look healthy.
• eval_loss did not start rising while the training loss kept falling.
• I loaded the adapter with PeftModel.from_pretrained, called .eval(), and compared it with the base on unseen prompts.
• I saved the adapter together with its config, the dataset version, the library versions and a note on what I changed.
• If I merged, I merged from a bf16 base, and I kept the adapter.
• If I pushed to the Hub, the repository is private unless I meant otherwise.

When all of those boxes are ticked, you have done what many people skip: a fine-tune that you can explain, reproduce and hand to someone else. For the next step, try one of the practice cards again on your own data, and then move on to the mid-level track, where preference tuning with DPO, reward functions for GRPO and multi-GPU training build directly on these habits. The Unsloth guide is a good companion if you want a faster fine-tuning stack that follows the same ideas.

This is part two of three. It picks up where Beginner ended: you can run an SFTTrainer with a LoraConfig, save an adapter and load it again. Here we explain why those pieces behave the way they do, add preference tuning and reinforcement learning, and cover the production concerns that decide whether a fine-tuning job is repeatable. Everything below is written against TRL 1.14 and PEFT 0.21. Both libraries moved fast, so older tutorials and model answers from memory are often wrong. Where something was removed, we say so.

Where this picks up

Each section says what problem it solves before showing code, and ends with a Try it you can run on a small model in a few minutes.

How the stack fits together under the hood

Beginner treated TRL and PEFT as one thing. They are two packages, and knowing where the seam is lets you predict which library owns a given error.

The layers, from the bottom up, are PyTorch, transformers (models, tokenizers and the Trainer loop), accelerate (device placement and distributed backends), PEFT (adapters injected into a model), and TRL on top. Every TRL config class, such as SFTConfig or DPOConfig, subclasses transformers.TrainingArguments. Every TRL trainer subclasses transformers.Trainer. That is why report_to, save_steps, resume_from_checkpoint, gradient_accumulation_steps and the rest behave exactly as in plain transformers: they are the same code. TRL adds the method-specific pieces: how a batch is built, how the loss is computed, and, for online methods, how completions are generated.

TRL does not install PEFT by default. When you pass peft_config, the trainer imports PEFT, wraps the model with get_peft_model internally, and from then on training updates only the adapter weights. If PEFT is missing you get an ImportError that tells you to run pip install trl[peft]. If you hand the trainer a dictionary instead of a config object, you get a TypeError saying peft_config must be a peft.PeftConfig instance. Build a real LoraConfig.

Three defaults in every TRL config differ from TrainingArguments, and they surprise people who move between plain transformers and TRL:

The learning rate default also differs per method: 2e-5 for SFT, 1e-6 for DPO and 1e-6 for GRPO, against transformers' 5e-5. These defaults are tuned for full fine-tuning. With LoRA you raise them, which we cover below.

The trainers also accept a model as a string. SFTTrainer(model="Qwen/Qwen3-0.6B", ...) loads the model and its tokenizer for you. When you need control over loading, pass model_init_kwargs in the config, for example {"dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"}. Use dtype, not torch_dtype: the old name was deprecated in transformers and TRL moved to dtype in v1.10. The tokenizer argument is processing_class, not tokenizer.

One more piece of vocabulary matters for everything that follows. Methods split into offline and online. Offline methods (SFT, DPO, KTO and reward modelling) learn from a fixed dataset. Online methods (GRPO, RLOO) generate completions from the current model during training and score them. Offline runs are cheap and predictable. Online runs spend most of their time generating, which is why vLLM appears in this guide.

The stable trainers in TRL 1.14 are SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer, KTOTrainer, RewardTrainer and DistillationTrainer, each with a matching Config. Everything under trl.experimental carries an explicit contract: it may change or disappear in any release, including a patch release, without a deprecation period. Treat experimental trainers as prototypes, never as a dependency of a production pipeline.

Data: formats, chat templates and what the loss sees

Most failed fine-tunes are data problems that look like training problems. TRL's contract is that the columns in your dataset decide the type of training example, and the trainer decides from that what to mask.

Datasets come in two formats. Standard data uses plain strings. Conversational data uses lists of {"role": ..., "content": ...} messages, and the trainer applies the tokenizer's chat template. There are five types:

Tool-calling data adds a tools column, and vision data adds image or images. A mismatch between type and trainer produces errors such as KeyError: The input examples must contain either 'messages' for conversational data or 'text' for standard data. The fix is to rename columns, or set dataset_text_field in SFTConfig if your text lives in a column that is not called text.

What counts toward the loss

SFT minimises cross-entropy over target tokens. The label sequence is shifted by one position, and positions you want ignored carry the label -100, which PyTorch's loss skips. Which positions are ignored depends on your data and two config switches.

For prompt-completion data, SFT computes loss on the completion only by default. The field is completion_only_loss, which is None and acts as True in that case. This replaced the old DataCollatorForCompletionOnlyLM pattern, which no longer exists, so do not go looking for response-template tricks. For conversational messages data, the default is loss on every token, prompt included. To train only on assistant turns, set assistant_only_loss=True. That switch needs a chat template containing {% generation %} and {% endgeneration %} markers around the assistant content, so TRL can build a mask. TRL patches the templates of known model families such as Qwen3. For anything else, you supply a training template through chat_template_path. If the template has no markers you get a RuntimeError saying at least one example has no assistant tokens. If you set assistant_only_loss=True on a non-conversational dataset, you get a ValueError saying the dataset is not conversational.

Two traps live here. First, a formatting function is incompatible with completion_only_loss=True, and the trainer raises a ValueError rather than guess. Apply your formatting to the dataset with datasets.map before passing it in. Second, v1.0 stopped stripping None values from SFT, DPO and reward datasets automatically. If your data has ragged None fields from a JSON source, apply dataset.with_transform(remove_none_values) from trl.trainer.utils yourself.

The practical habit that saves the most time is to decode a batch before you train. Pull one example through the trainer's dataset, decode input_ids with the tokenizer, and decode only the positions where labels != -100. If the second decode includes your system prompt, your mask is wrong, and no hyperparameter will fix it.

Inspect trainer.train_dataset.column_names to see what the trainer produced in your version, and check the mask columns against what you expect.

SFT in depth: loss path, packing and memory

At beginner level, memory problems meant lowering the batch size. At this level you know which parts of the memory bill are under your control.

The loss path

The expensive tensor in language-model training is the logits: batch times sequence length times vocabulary size. For a model with a vocabulary of over 150,000 tokens, that tensor dwarfs everything else. Since TRL 1.7 the SFT default loss_type is "chunked_nll". It computes the same maths as plain NLL, but it skips ignored tokens before the lm_head projection and computes cross-entropy in chunks of 256 tokens, so the full logits tensor is never materialised. The release notes report about 30% lower peak memory on average and up to 50% on large-vocabulary models. Version 1.13 sped it up further by running the projection on bf16 tensor cores, for a 1.2x to 1.7x end-to-end gain.

You normally do nothing to benefit. You do need to know its two constraints, because each produces a precise error:

• With a PEFT adapter on lm_head, including target_modules="all-linear" when it ends up wrapping lm_head, chunked loss is refused with a ValueError. Remove lm_head from target_modules, or set loss_type="nll".
• With use_liger_kernel=True, leaving loss_type unset resolves to "nll". Setting "chunked_nll" explicitly raises a ValueError.

The config value is None by default and resolves at runtime, which is why a log of your config may show None while the run used chunked_nll. If you need exact reproducibility against an older TRL, set loss_type="nll" explicitly. A third value, "dft", also exists.

Padding, packing and padding-free

Padding wastes compute: short examples are padded to the longest in the batch. Packing concatenates several examples into one sequence of max_length tokens, so almost no compute goes to padding. Turn it on with packing=True. The default strategy is packing_strategy="bfd" (best-fit decreasing), and the alternative is "bfd_split". The old spelling "bfd-requeue" was renamed in v1.0 and still works, with removal planned for v2.0.0. Packing needs max_length to be set, or you get a ValueError, and it is not supported for vision datasets. It is also incompatible with context parallelism.

Padding-free training, padding_free=True, flattens a batch into one long sequence without pad tokens and tells attention where the example boundaries are. Both approaches aim at the same waste; packing changes how many examples fit in a step, while padding-free changes how a batch is laid out. Test them on your data and compare tokens per second rather than trusting folklore.

max_length defaults to 1024 and truncation_mode to "keep_start". The value "keep_end" is deprecated. For vision-language models, set max_length=None, otherwise image placeholder tokens can be truncated and you get a ValueError explaining exactly that.

The memory levers, in order

When a run runs out of memory, work down this list rather than guessing:

1. Use LoRA or QLoRA, so optimizer state exists only for the adapter.
2. Keep gradient_checkpointing=True, which is already the TRL default. It costs roughly 20% speed for a large memory saving.
3. Keep the default chunked_nll loss.
4. Reduce max_length to what your data needs. Check the length distribution first.
5. Use per_device_train_batch_size=1 with gradient_accumulation_steps to keep the effective batch size.
6. Try packing or padding_free for compute efficiency.
7. Turn on activation_offloading=True.
8. Try use_liger_kernel=True, with the caveats in the Liger paragraph below.
9. Shard with ZeRO-3 or FSDP when the model itself does not fit.

The effective batch size is per_device_train_batch_size × number of devices × gradient_accumulation_steps. When you change the number of GPUs, change accumulation to keep it constant, or your learning-rate tuning stops applying.

On Liger: since TRL 1.14, use_liger_kernel=True still enables Liger's model kernels through transformers, but in DPO, KTO and GRPO it selects TRL's own chunked log-probability path. The old trl.losses module with FusedLinearDPOLoss and friends was removed. Liger is still refused with use_weighting, compute_metrics, an adapter on lm_head, and prompt-learning PEFT methods such as prompt tuning, prefix tuning and P-tuning. A fused Triton kernel for log-probabilities and entropy ships in-tree as trl.kernels and is on by default on CUDA, ROCm and XPU, so you do not need to install anything extra for it.

LoRA mechanics: rank, alpha, targets and initialisation

LoRA freezes a weight matrix W and learns a low-rank update beside it. The adapted layer computes W·x plus a scaled B·A·x, where A and B are small matrices with inner dimension r. B is initialised to zero, so at step zero the model is exactly the base model, and training moves it from there. The scale is lora_alpha / r, or lora_alpha / sqrt(r) if you set use_rslora=True.

Two practical consequences follow. First, alpha and rank are coupled through the scale. Doubling r while holding lora_alpha fixed halves the scale, which is why people set alpha to a multiple of rank and then change both together. Second, because B starts at zero, the learning rate that works for LoRA is higher than for full fine-tuning. The TRL docs give about ten times the full fine-tuning rate: SFT 2e-4 against 2e-5, DPO 5e-6 against 5e-7, GRPO 1e-5 against 1e-6. If you leave the TRL default of 2e-5 with LoRA, the run looks healthy and learns almost nothing.

Which layers

target_modules accepts a list of names (exact or suffix match), a regular-expression string, or "all-linear". The special value means every linear layer except the output layer. The TRL docs, citing the "LoRA Without Regret" work, recommend "all-linear": attention-only LoRA underperforms even when parameter counts are matched. For the same source, rank 256 is suggested for SFT at post-training scale and ranks of 1 to 32 for reinforcement learning, with the effective batch size kept below 32.

If you leave target_modules=None, PEFT uses a per-architecture default table. That works for common architectures and fails with a clear error for unknown ones. For mixture-of-experts models, expert weights are often fused parameters rather than nn.Linear modules. target_parameters (PEFT 0.17 and later) lets LoRA target them directly.

modules_to_save lists modules that are trained fully and saved with the adapter. You need it when you add a new classification head, and when you add new tokens and must train embed_tokens or lm_head. PEFT 0.19 changed it so the original module's requires_grad is no longer flipped, so do not rely on old behaviour. For new tokens, trainable_token_indices is a lighter alternative to training the whole embedding matrix.

Check that your config matched

This is the most useful PEFT habit at this level. PEFT raises an error only if none of your target_modules match. If some entries match and others do not, the unmatched ones are skipped silently, and the symptom is a loss that barely moves. Verify with:

from peft import get_peft_model, LoraConfig

model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"))
model.print_trainable_parameters()      # trainable params: ... || all params: ... || trainable%: ...
print(model.get_model_status())         # adapter state summary
print(model.get_layer_status()[:3])     # per-layer detail

Look for a trainable percentage that is plausible for your rank, and for layers flagged as irregular. When using TRL you rarely call get_peft_model yourself, but trainer.model.print_trainable_parameters() works on the wrapped model.

Remember the CLI/API difference: the trl CLI's model options default to lora_r=16, lora_alpha=32 and lora_dropout=0.05, while a bare LoraConfig() defaults to r=8, lora_alpha=8 and lora_dropout=0.0. The same adapter "settings" in the two places are not the same unless you spell them out.

QLoRA and quantized training the current way

QLoRA is LoRA on top of a base model quantized to 4 bits with bitsandbytes, using the NF4 data type, double quantization and bf16 compute. The base weights are frozen and stored in 4 bits, so a model that needs 16 GB in bf16 fits in roughly a quarter of that for weights. The adapters and activations are still higher precision. It trades some speed for memory and is the standard way to fine-tune a 7 to 14 billion parameter model on a single GPU.

Older tutorials load a quantized model by hand, call prepare_model_for_kbit_training and pass the model in. Since TRL 1.8 there is a shorter path: pass quantization_config straight to the trainer. It works on SFTTrainer, DPOTrainer, GRPOTrainer and RLOOTrainer.

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

trainer = SFTTrainer(
    model="meta-llama/Llama-3.1-8B",
    args=SFTConfig(output_dir="llama-qlora", learning_rate=2e-4),
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
    quantization_config=bnb,
    peft_config=LoraConfig(target_modules="all-linear", task_type="CAUSAL_LM"),
)

There are two guard rails. Setting quantization_config both as a trainer argument and inside model_init_kwargs raises a ValueError; choose one place. On DPO, passing quantization_config together with an already-instantiated model also raises an error, because the trainer can only quantize at load time. If you write raw PEFT code without TRL, call peft.prepare_model_for_kbit_training(model) before get_peft_model.

Platform honesty matters here. bitsandbytes needs Python 3.10 or later and PyTorch 2.4 or later. On Linux with an NVIDIA GPU it works as documented. On Windows, x86-64 CUDA wheels exist. On macOS with Apple Silicon only a CPU build exists, so there is no 4-bit GPU training on a Mac. If your team uses MacBooks for development, develop the pipeline with a small unquantized model and run QLoRA on a Linux GPU host.

Merging has a quantization-specific trap. If you merge a QLoRA adapter into the 4-bit base, you introduce rounding error. Load the base model in bf16 for the merge instead, and accept that the merged artifact is full size.

Preference tuning with DPO

SFT teaches a model to imitate. Preference tuning teaches it to prefer one answer over another, which is how you move style, safety behaviour and helpfulness without writing perfect demonstrations. Direct Preference Optimization learns from (prompt, chosen, rejected) rows and needs no separate reward model. The trick is a frozen reference model: the loss rewards the policy for raising the likelihood of the chosen answer relative to the reference, more than it raises the rejected one. The coefficient beta, default 0.1, controls how far the policy may drift from the reference. A higher beta stays closer to it.

With PEFT, you do not load a second model. The reference is the base model with the adapter disabled, computed in the same process. This is a large memory saving, and it explains two errors you may meet: passing the same object as model and ref_model raises a ValueError, and sync_ref_model=True with a PEFT model raises NotImplementedError, because there is no separate reference to synchronise. Leave ref_model out and let TRL handle it.

from datasets import load_dataset
from peft import LoraConfig
from trl import DPOConfig, DPOTrainer

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    args=DPOConfig(output_dir="qwen-dpo", beta=0.1, learning_rate=5e-6, loss_type=["sigmoid"]),
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
    peft_config=LoraConfig(task_type="CAUSAL_LM"),
)
trainer.train()

Notice loss_type is a list. The default is ["sigmoid"], the original DPO loss. You can combine several losses with loss_weights, which is how mixed-preference-optimisation recipes are expressed. Other accepted values include hinge, ipo, robust, bco_pair, apo_zero, sppo_hard, sft, and sigmoid_norm. aot pairs and aot_unpaired exist too; the name aot_pair was renamed in v0.28. The divergence is configured with a string: f_divergence_type="reverse_kl" is the default, and FDivergenceType enums were replaced by strings.

Other keys to know: max_length (default 1024), label_smoothing (0.0), precompute_ref_log_probs (False), disable_dropout (True) and use_weighting (False). precompute_ref_log_probs=True computes reference log-probabilities once before training. It is worth it when you do not use PEFT and the reference model would otherwise take GPU memory for the whole run. Several older DPO settings, including max_prompt_length and max_completion_length, were deprecated in v0.28, so check before copying them from a blog post. The recommended dataset shape is an explicit prompt column; implicit prompts work but are less transparent.

Reading a DPO run

DPO logs four metrics you should learn to read: rewards/chosen, rewards/rejected, rewards/accuracies and rewards/margins. These are implicit rewards, computed from the log-ratio against the reference scaled by beta. In a healthy run, accuracy rises above 0.5 and the margin grows. If both rewards/chosen and rewards/rejected fall together while the margin grows, the policy is lowering the likelihood of everything, which can degrade generation quality even though the preference metric looks fine. When that happens, lower the learning rate, raise beta, or add an SFT term through loss_type with loss_weights. Always spot-check generations; a preference metric is a proxy.

GRPO and reward functions

DPO needs pairs somebody has already judged. Group Relative Policy Optimization needs only prompts and a way to score answers. For each prompt the trainer samples a group of completions (num_generations, default 8), scores each with your reward functions, and computes each completion's advantage as its reward minus the group mean, optionally divided by the group standard deviation. Completions better than their siblings are reinforced, and worse ones are discouraged. No value model is needed. This is why GRPO suits tasks with a checkable answer: maths, code that must pass tests, structured output that must match a schema.

A reward function is a plain Python callable. TRL calls it with prompts, completions, completion_ids, trainer_state and every dataset column except prompt, as keyword arguments. It must return a list of floats with exactly one value per completion. Always accept **kwargs, because the set of arguments passed can grow. A function can also be async def, which helps when scoring calls out to a service.

import re
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

def format_reward(completions, **kwargs):
    # completions are plain strings for standard data, message lists for conversational data
    texts = [c if isinstance(c, str) else c[0]["content"] for c in completions]
    return [1.0 if re.search(r"<answer>.*</answer>", t, re.S) else 0.0 for t in texts]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=[format_reward],
    args=GRPOConfig(output_dir="qwen-grpo", num_generations=8, max_completion_length=512),
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
)
trainer.train()

Built-in reward helpers live in trl.rewards, including accuracy_reward, which checks maths answers and needs the math-verify extra. On the CLI you can name one directly: trl grpo --model_name_or_path ... --dataset_name ... --reward_funcs accuracy_reward. With several reward functions, reward_weights sets their relative weight, and multi_objective_aggregation controls whether rewards are summed and then normalised ("sum_then_normalize", the default) or normalised first.

The defaults that matter

Because several of these defaults flipped across versions, write them down in your config file rather than relying on them. A run reproduced six months later on a newer TRL should not silently change meaning.

The batch arithmetic is the most common first error. Completions are generated in batches whose size must be a multiple of num_generations. The relevant check is that generation_batch_size is divisible by num_generations, which with defaults means per-device batch size times number of processes times steps_per_generation must be a multiple of the group size. The errors say so directly. generation_batch_size and steps_per_generation are mutually exclusive; set one. Also, auto_find_batch_size is rejected by GRPO, so size batches by hand.

Is it learning?

A reward that goes up is necessary but not sufficient. The signals to watch are listed in the monitoring section, but two deserve a mention now. reward_std near zero means every completion in each group scores the same, so advantages are zero and there is no learning signal; your reward is too coarse or your task is too easy or too hard. And completions/clipped_ratio shows how many completions hit max_completion_length; a high value means you are truncating answers before they can earn reward. log_completions=True prints a table of sampled completions with their rewards, and you should read it. Reward hacking, where the model finds a cheap way to satisfy the function, is visible in samples long before it is visible in curves.

vLLM for online methods

In GRPO and RLOO, generation dominates the wall clock. Plain model.generate from transformers works and is the right choice for a quick experiment. For anything bigger, set use_vllm=True so generation runs in vLLM. It is off by default, and it is Linux-only: vLLM supports Linux as its operating system and does not run natively on Windows (use WSL2) or on a Mac. TRL 1.14 supports vLLM versions 0.20.0 to 0.30.0. Outside that window you get crashes, not warnings, so pin the version.

There are two modes, set by vllm_mode.

Colocate is the default since v1.0. vLLM runs inside the trainer process on the same GPUs. You need no extra infrastructure, and it is the right mode for one node. Memory is shared, and vllm_gpu_memory_utilization (default 0.3) sets the fraction vLLM may take. If a colocated run runs out of memory, lower that value first. vllm_enable_sleep_mode=True lets vLLM release memory while the trainer works.

Server mode runs a separate vLLM process on dedicated GPUs, and the trainer pushes updated weights to it after each step over NCCL. It is the right choice when generation needs more GPUs than a trainer rank can spare, or when you want to scale the two sides independently. The official recipe splits eight GPUs between the two:

CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-7B \
  --tensor-parallel-size 4 \
  --weight-transfer-config '{"backend": "nccl"}' \
  --logprobs-mode processed_logprobs --max-logprobs -1

CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py   # GRPOConfig(use_vllm=True, vllm_mode="server")

Since v1.11, trl vllm-serve is a deprecated thin wrapper that prints and runs the equivalent vllm serve command. Learn and script the vllm serve form. Each flag has a job. VLLM_SERVER_DEV_MODE=1 exposes the weight-update endpoints. --weight-transfer-config selects NCCL transfer (use "ipc" when trainer and server share a GPU). --logprobs-mode processed_logprobs is what importance-sampling correction expects. The trainer and the server must use disjoint CUDA_VISIBLE_DEVICES. To scale generation, raise --tensor-parallel-size: data parallelism no longer helps dense models.

The trainer-side keys are vllm_server_base_url, vllm_server_host (default 0.0.0.0), vllm_server_port (8000), vllm_group_port (51216) and vllm_server_timeout (240 seconds). Start the server first, and make sure the trainer can reach both ports.

Because vLLM samples with its own numerics, its token probabilities can differ slightly from the trainer's. TRL corrects for that with importance sampling: vllm_importance_sampling_correction is on by default, with vllm_importance_sampling_mode="sequence_mask" and a clip value vllm_importance_sampling_clip_max of 3.0. The old vllm_importance_sampling_cap is deprecated in favour of the clip name.

If a run fails with RuntimeError: No completions were generated, the generation backend failed. The cause is in the vLLM logs above that line: usually a version outside the window, overlapping GPUs, a missing dev-mode variable, or a server that never became ready within the timeout.

Multi-GPU launches and the CLI

Single-GPU scripts become multi-GPU through accelerate. Run accelerate config once to answer the prompts, then accelerate launch train.py. Under the hood, accelerate starts one process per device. With plain data parallelism (DDP), every GPU holds a full copy of the model, so DDP helps throughput but not capacity. When the model does not fit, you move to sharding: FSDP (versions 1 and 2) or DeepSpeed ZeRO stages 1, 2 and 3. See the DeepSpeed guide for what each ZeRO stage shards.

TRL bundles accelerate presets so you do not hand-write configs for common cases: single_gpu, multi_gpu, zero1, zero2, zero3, fsdp1 and fsdp2. On the CLI you select one by name:

trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara \
  --output_dir out --num_processes 4 --accelerate_config zero2

Choose by memory pressure. If the model plus adapter plus activations fit on one GPU, multi_gpu (DDP) is simplest and fastest. If optimizer state is the problem, zero2 shards it. If the weights themselves do not fit, zero3 or FSDP is needed. With PEFT, the trainable state is small, so DDP carries you further than it would in full fine-tuning.

YAML configs

The CLI accepts the same fields from a YAML file, where the keys are the flag names:

model_name_or_path: Qwen/Qwen2.5-0.5B
dataset_name: trl-lib/Capybara
output_dir: runs/sft-001
learning_rate: 2.0e-4
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
use_peft: true
lora_r: 16
lora_alpha: 32
lora_target_modules: all-linear
report_to: trackio

Run it with trl sft --config sft_config.yaml. The PEFT-related flags are --use_peft, --lora_r, --lora_alpha, --lora_dropout, --lora_target_modules (space separated), --lora_task_type, --use_rslora, --use_dora, --load_in_4bit, --load_in_8bit and --bnb_4bit_quant_type. A config file you can commit and review is far better than a long command in shell history. Subcommands exist for sft, dpo, grpo, rloo, kto, reward and distillation, plus trl env.

For multi-node jobs, the accelerate config carries num_machines, machine_rank, main_process_ip, main_process_port and num_processes, which is the total across nodes. Schedulers such as SLURM, or Kubernetes with a job controller, launch the same command on each node; there is no TRL-specific operator. See the Kubernetes guide for scheduling GPU workloads.

Monitoring runs and debugging beyond the basics

A long run is only as good as your ability to tell that it has gone wrong early. TRL's recommended tracker is Trackio, enabled with report_to="trackio" and configured with the TRACKIO_PROJECT and TRACKIO_SPACE_ID environment variables. Weights & Biases, TensorBoard and MLflow work through the standard report_to option; see the MLflow guide and the Weights & Biases guide if your team already uses one. Metrics are logged every 10 steps by default.

What to watch, by method:

For GRPO, kl is logged only when beta is above zero. A collapsing entropy means the policy is becoming deterministic and exploration is dying.

A debugging order that works

When something is wrong, ask these questions in order, because each is cheaper than the next:

1. Is it the data? Decode a batch and its labels as described earlier. Check for empty completions, wrong columns and a template mismatch.
2. Is it the config? Print the resolved config. Remember that None for loss_type resolves later, and that CLI LoRA defaults differ from LoraConfig.
3. Did the adapter attach? Call print_trainable_parameters(), then get_model_status() and get_layer_status(). Partial target_modules mismatches are silent.
4. Is the learning rate right for LoRA? A full fine-tuning rate with LoRA barely moves the loss.
5. Is it a version problem? Run trl env, then compare against the pinned versions. A TypeError about an unexpected keyword argument in LoraConfig means the adapter was saved by a newer PEFT; upgrade PEFT.
6. Is it numerics? ValueError: Attempting to unscale FP16 gradients means trainable weights stayed in fp16 under fp16 mixed precision. Use bf16 where the hardware allows, or peft.cast_mixed_precision_params.

For multi-adapter work there is a further trap: an inactive adapter has requires_grad=False, so if you want to train two, call model.base_model.set_adapter(["default", "other"]) before building the optimizer. And merging only some of several active adapters silently drops the others; merge all, or unmerge first.

Finally, TRL sends one anonymous telemetry ping per trainer instantiation since v1.5. It contains the library version, trainer class, a model architecture name if known, whether PEFT is used, hardware and distributed-backend information. Opt out with HF_HUB_DISABLE_TELEMETRY=1, or HF_HUB_OFFLINE=1, which air-gapped clusters should set anyway. No ping is sent when CI is set. This matters for employers in the Gulf and Egypt with data-residency or network-egress rules: put the variable in your job template so it is a decision, not an accident.

Saving, merging and serving adapters

A PEFT checkpoint is a small directory: adapter_model.safetensors and adapter_config.json. It does not contain the base model. trainer.save_model(path) saves only the adapter when PEFT is used. Prefer safetensors over the legacy adapter_model.bin, which is a pickle and can execute code when loaded, so be careful about adapters from sources you do not control.

To resume an interrupted run, use trainer.train(resume_from_checkpoint=True) or pass a path. To continue training an existing adapter in a new run, load it as trainable and pass the model with no peft_config:

from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained("qwen3-sft-lora", is_trainable=True)
# pass `model` to the trainer without peft_config

Passing both a PeftModel and a peft_config raises a ValueError telling you to merge, save and start from the merged base instead.

Merge or hot-swap

You have two serving options, and the trade-off is real.

Merge with merge_and_unload(), which folds the update into the base weights and returns a plain transformers model. There is no adapter latency, any server can load the result, and the cost is a full model copy per fine-tune.

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype="auto")
merged = PeftModel.from_pretrained(base, "qwen3-sft-lora").merge_and_unload()
merged.save_pretrained("qwen3-merged")

Hot-swap keeps one base model in memory and loads adapters per request. In PEFT you can hold several adapters with load_adapter(path, adapter_name=...) and switch with set_adapter, and disable_adapter() is a context manager that gives you the base model's behaviour for comparison. LoRA and RoAd also support mixed-adapter batches through adapter_names=[...]. In production, vLLM's multi-LoRA serving does this for you, and it is how many tenants share one base model. See the vLLM guide. vLLM loads only LoRA, which is why PEFT 0.19 added converters from other adapter types to LoRA.

Pick merge when you have a handful of models and want the simplest serving stack. Pick hot-swap when you have many adapters on one base and the per-copy storage and memory would be wasteful.

Combining adapters, with add_weighted_adapter, supports linear, SVD, concatenation, TIES and DARE combinations. After PEFT 0.21 fixed the SVD path, which had applied LoRA scaling twice, merged results can differ from those produced by older versions, so re-evaluate any merge recipe you saved before upgrading.

Publish artifacts with push_to_hub, which needs HF_TOKEN or hf auth login. Keep the token in an environment variable or secret manager and never in a YAML file committed to git.

Configuration management, CI and upgrades

Fine-tuning code rots quickly because both libraries release often. TRL ships roughly every two weeks, and PEFT, still pre-1.0, ships a minor release every one to three months and says some behaviours will change at v1.0. Treat versions and defaults as part of your configuration.

Pin the whole stack. TRL, PEFT, transformers, accelerate, vLLM, torch and DeepSpeed move together, and TRL raises its dependency floors regularly. TRL 1.14 requires transformers>=4.56.2 and datasets>=4.7.0, and PEFT 0.21.1's tensor-parallel support needs transformers 5.17 or later. Commit a lock file and record the output of trl env with every run.

Make defaults explicit. The defaults that changed in the last year include vllm_mode (to colocate in 1.0), SFT loss_type (to chunked in 1.7) and GRPO max_completion_length and use_bias_correction_kl (in 1.10). A run that spells these out in its YAML does not change meaning on upgrade.

Read the release notes' breaking sections and MIGRATION.md before bumping. The things most likely to bite are removed experimental trainers, renamed values such as bfd-requeue, and deprecated names such as pad_token in configs (set tokenizer.pad_token and pass processing_class) and use_transformers_paged. Deprecations warn now and are removed in v2.0.0, so warnings in your logs are your upgrade to-do list. In CI, treat FutureWarning from trl as a failure on a branch you are preparing to upgrade.

Test with a tiny model. A smoke test that builds each trainer with a very small model, runs two steps and saves an adapter catches most breakage from an upgrade in minutes, on CPU, without a GPU runner. Add assertions that matter: trainable parameters are non-zero and below a fraction you expect, the adapter directory contains the files you serve, and a reload through PeftModel.from_pretrained produces different logits from the base model.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

def test_adapter_changes_outputs(tmp_path_factory):
    name = "Qwen/Qwen3-0.6B"
    tok = AutoTokenizer.from_pretrained(name)
    base = AutoModelForCausalLM.from_pretrained(name, dtype="auto")
    adapted = PeftModel.from_pretrained(base, "artifacts/adapter")
    ids = tok("Hello", return_tensors="pt")
    with adapted.disable_adapter():
        off = adapted(**ids).logits
    on = adapted(**ids).logits
    assert not torch.allclose(on, off)

Keep runs reproducible. Store the YAML, the pinned environment, the dataset revision and the base model revision (model_revision). If you enable trust_remote_code, which is False by default and runs Python from a Hub repository, pin the revision to a commit SHA. For secrets and egress: tokens in environment variables, HF_HUB_OFFLINE=1 with a pre-populated cache on restricted networks, and no tracker keys in committed files.

For managed compute, Hugging Face Jobs can run a TRL script on rented hardware, for example hf jobs uv run --flavor a10g-small --timeout 30m --secrets HF_TOKEN -- <script URL> .... The default timeout is 30 minutes, so set it for real runs. For containerised training, see the Docker guide, and for the larger tuning ecosystem see Hugging Face and Unsloth.

Putting it all together

Here is how the pieces combine into one repeatable pipeline for a team that wants a domain-tuned assistant, written as the order you would do it in.

First, fix the environment. Create a pinned environment with TRL, PEFT, transformers, accelerate and, on Linux GPU hosts, bitsandbytes and vLLM, and record trl env. Set HF_HUB_DISABLE_TELEMETRY=1 and secrets through environment variables.

Second, prepare and inspect the data. Convert your examples to conversational messages or prompt/completion form. Check the chat template for generation markers if you want assistant-only loss. Decode a batch and its trained-on span before any real run.

Third, run SFT with LoRA from a YAML file: target_modules: all-linear, a learning rate near 2e-4, effective batch size set explicitly, lm_head left out of the targets, and report_to: trackio. Verify trainable parameters, watch loss and mean_token_accuracy, and evaluate on held-out prompts. If memory is tight, switch to QLoRA through quantization_config.

Fourth, optionally align. If you can collect preference pairs, run DPO from the SFT result with a learning rate near 5e-6 and beta=0.1, and watch rewards/accuracies and rewards/margins. If you have a verifiable task instead, write a reward function, sandbox it, run GRPO with log_completions=True, and watch reward_std and entropy. Turn on vLLM in colocate mode on one node, or server mode with disjoint GPUs and a firewalled port when you need more.

Fifth, package. Save the adapter in safetensors, merge in bf16 if you need a standalone model, or serve the adapter through vLLM multi-LoRA. Run the reload test that proves the adapter changes outputs.

Sixth, keep it alive. Upgrade deliberately, with a smoke test and release notes, and with every default you rely on written out in the config.

Where you are now

You can now explain why a TRL run behaves as it does rather than only that it works. You know where TRL ends and PEFT begins, what the loss actually sees and how to check it, how LoRA rank, alpha and target modules interact, and how to verify that an adapter really attached. You can run QLoRA with the current trainer argument, tune a model with DPO and read its metrics, write a GRPO reward function, choose between vLLM colocate and server modes, and launch on several GPUs from a reviewed YAML file. You can ship the result by merge or hot-swap, and you have a plan for the upgrades that will otherwise break you.

What the Senior level adds is the platform view: failure modes at scale, multi-node online reinforcement learning, security and tenancy, cost, and when to choose a different tool. If your day job is moving towards owning this for a team, continue there.

Sources

• TRL documentation
• TRL SFT trainer
• TRL DPO trainer
• TRL GRPO trainer
• TRL RLOO trainer
• TRL KTO trainer
• TRL dataset formats
• TRL PEFT integration
• TRL LoRA Without Regret
• TRL vLLM integration
• TRL reducing memory usage
• TRL distributing training
• TRL command line interface
• TRL experimental overview
• TRL usage statistics
• TRL Trackio integration
• TRL training with Jobs
• TRL releases
• TRL migration guide
• PEFT documentation
• PEFT quicktour
• PEFT troubleshooting
• PEFT checkpoint format
• PEFT quantization
• PEFT model merging
• PEFT LoRA reference
• PEFT releases
• Accelerate launch tutorial

This is the mid-level interview file. Beginner interviews ask what LoRA is. Mid-level interviews give you a broken or half-specified situation and watch how you reason. The questions below are scenarios and trade-offs, each with a model answer and a note on what separates a strong answer from a weak one. Everything is written against TRL 1.14 and PEFT 0.21.

How mid-level interviews work

Interviewers at this level are checking three things. Can you predict behaviour from the mechanism rather than from memory of a tutorial? Do you check before you change things? And do you know which advice is out of date? Both libraries removed or renamed a lot in the last year, so a candidate who confidently describes PPOTrainer, DataCollatorForCompletionOnlyLM or tokenizer= in a trainer signals that their knowledge is stale.

A good pattern for any scenario answer is: state the first thing you would look at and why, name the cheap checks before the expensive ones, and say what result would change your mind. Interviewers reward visible ordering of hypotheses.

Scenario questions

1. The loss barely moves

"You fine-tune with LoRA on 20,000 examples. After 500 steps the loss has dropped from 1.9 to 1.85. What do you check?"

Model answer. I would go in order of cost. First the learning rate: TRL's SFT default is 2e-5, tuned for full fine-tuning, and LoRA wants roughly ten times that, so 2e-4. A full fine-tuning rate with an adapter looks healthy and learns very little, because B starts at zero. Second, whether the adapter attached to enough layers. PEFT raises an error only if no target_modules match; partial mismatches are skipped silently. I would call print_trainable_parameters(), get_model_status() and get_layer_status() and check the trainable percentage is plausible. I would use target_modules="all-linear". Third, the data: decode a batch and its trained-on span to confirm the mask covers what I think it does. Fourth, the effective batch size and learning-rate schedule. Fifth, rank and alpha, since their ratio sets the scale.

Strong versus weak. A strong answer gives the order and the reason for each step, and mentions the silent partial mismatch. A weak answer says "train longer" or "increase the rank" without checking that the adapter is training at all.

2. Out of memory on a single GPU

"A 7B SFT job with LoRA runs out of memory at max_length=4096. Walk me through it."

Model answer. I would look at the memory bill. Weights in bf16 are the fixed part. LoRA already removes most optimizer state. Activations and the logits tensor scale with sequence length and batch size. TRL already enables gradient checkpointing and the chunked loss by default, so the usual first moves are done. I would confirm that, then reduce the per-device batch size to 1 and compensate with gradient_accumulation_steps. I would check whether the data really needs 4096 tokens by looking at the length distribution, because truncating to what the data needs is the cheapest saving. Then I would consider activation_offloading=True, and QLoRA through the quantization_config trainer argument if the weights themselves are the problem. If this is still not enough, the model needs sharding with ZeRO-3 or FSDP across several GPUs.

Watch for. Candidates should know that gradient_checkpointing=True is already the default in TRL configs, and that chunked_nll is the SFT default since 1.7. Suggesting "enable gradient checkpointing" as a fix shows they have not read the defaults. Also check they keep the effective batch size constant when changing the per-device size.

3. The error mentions lm_head

"Training fails with a ValueError saying chunked_nll is not supported when lm_head is wrapped by a PEFT adapter. What happened and what are your options?"

Model answer. The SFT default loss skips ignored tokens before the output projection and computes cross-entropy in chunks, so it needs direct access to the lm_head weights. An adapter on lm_head breaks that. This usually happens because target_modules includes lm_head explicitly, or a broad pattern ends up covering it. The options are to remove lm_head from the targets, which is what I would do unless I truly need it adapted, or to set loss_type="nll" and pay the extra memory. If I need to train the output layer for new tokens, I would consider trainable_token_indices rather than adapting the whole head.

Strong versus weak. The strong answer explains why the loss and the adapter conflict. The weak one just changes the loss type without understanding the cost.

4. The model after loading is as good as the base model

"You trained an adapter that looked good in the logs. You load it the next day for evaluation and it behaves like the base model, or like noise."

Model answer. The two classic causes are loading with get_peft_model instead of PeftModel.from_pretrained, which creates a fresh, randomly initialised adapter, and forgetting .eval() so dropout is active. Behaving like the base model is also what a freshly initialised LoRA does, because B starts at zero. If I used a classification task, I would check that the new head was in modules_to_save, otherwise it is reinitialised on load. I would also check that the adapter directory has the adapter_model.safetensors file and is not empty, and that the base model name in adapter_config.json matches what I loaded. Finally I would write a test that compares logits with disable_adapter() against logits with the adapter on.

5. DPO metrics look good, generations look worse

"rewards/accuracies climbs to 0.9 and margins grow. But the model's answers have become short and strange."

Model answer. DPO optimises relative likelihood between chosen and rejected answers. It is possible for both to fall, with the margin still growing, which means the policy is shifting probability mass away from everything the reference liked. I would plot rewards/chosen and rewards/rejected separately; if both sink, that is the signature. Remedies: lower the learning rate (5e-6 is the LoRA recommendation, and DPO is sensitive), raise beta so the policy stays closer to the reference, and consider adding an SFT term through the list-valued loss_type with loss_weights. I would also check the data: if chosen answers are consistently shorter or the pairs differ in length more than in quality, the model learns length rather than preference. And I would start DPO from an SFT checkpoint rather than a raw base.

Watch for. Strong candidates know loss_type is a list in current TRL. They treat preference accuracy as a proxy and insist on reading samples.

6. GRPO starts and then nothing improves

"Your GRPO run has run for 300 steps. Mean reward is flat. What do you look at?"

Model answer. First reward_std. If it is near zero, each group of eight completions scores identically, so the advantages are zero and there is no gradient signal. That means the reward is too coarse, or the task is too hard for the model to ever get partial credit, or too easy. Second, completions/clipped_ratio and completions/mean_length, to see whether answers are being truncated at max_completion_length before they can earn reward. Third, the entropy: a collapse means the policy has become deterministic. Fourth, I would turn on log_completions=True and read samples, because a parsing bug in the reward function, such as a regular expression that never matches, looks exactly like a model that is not learning. Fifth, num_generations and temperature: too few samples or too low a temperature reduces diversity within the group.

7. A reward function crashes the job

"The job dies with an error that the reward function returned the wrong number of rewards."

Model answer. TRL requires exactly one float per completion. The error names the function and gives the count it expected. Typical causes are a function that filters instead of mapping, or one that returns a single aggregate. I would fix the function so it always returns a list the same length as completions, accept **kwargs for forward compatibility, and add a unit test that calls it with a small fake batch. I would mention that reward functions can be async def, and that they run arbitrary code, so anything executing model output needs a sandbox.

8. The batch size error in GRPO

"generation_batch_size must be divisible by num_generations. Explain it."

Model answer. GRPO samples a group of num_generations completions per prompt and normalises rewards within each group. A generation batch has to contain whole groups. The batch size is per-device size times number of processes times steps_per_generation, so changing GPU count can break a config that used to work. I would adjust the per-device size or steps_per_generation until the product is a multiple of the group size. generation_batch_size and steps_per_generation are mutually exclusive, so I would set only one. I would not use auto_find_batch_size, which GRPO rejects.

9. Two GPUs, vLLM, and a hang

"Server-mode GRPO starts, then fails with No completions were generated."

Model answer. That error means the generation backend failed to return results; the cause is in the vLLM server logs. My checklist: Is vLLM inside the supported window, which is 0.20.0 to 0.30.0 for TRL 1.14? Are the trainer and the server on disjoint CUDA_VISIBLE_DEVICES? Did the server start with VLLM_SERVER_DEV_MODE=1 and the NCCL weight-transfer config? Did it come up before the trainer's timeout of 240 seconds? Do host and port match vllm_server_host and vllm_server_port, and is the group port 51216 reachable? I would also mention that trl vllm-serve is deprecated and I would launch vllm serve directly.

10. A teammate upgrades TRL and the nightly job fails

"Your fine-tuning pipeline worked on Friday. On Monday it fails with an import error."

Model answer. I would run trl env to see what changed, then read the release notes for breaking changes. The most likely causes in recent versions are removed experimental trainers, trl.losses going away in 1.14, PPO removed in 1.13, or an import path that moved from experimental to stable. The cure for the incident is to pin the previous versions and restore the job. The cure for the cause is to pin the whole stack in a lock file, make upgrades a deliberate change with a smoke test on a tiny model, and treat deprecation warnings as a to-do list. I would also write down every default the job relies on, because defaults flip across versions: vllm_mode, the SFT loss type, GRPO's max_completion_length.

Trade-off questions

LoRA or full fine-tuning?

Model answer. LoRA wins on memory, storage and iteration speed. Adapters are megabytes, many can share one base model, and you can swap them at serving time. Full fine-tuning can win on capacity when the task needs large behavioural change or lots of new knowledge, and the "LoRA Without Regret" guidance in the TRL docs says LoRA matches full fine-tuning in many post-training settings when applied to all layers, with enough rank for the data size. I would start with LoRA on all linear layers, measure against the target metric, and only move to full fine-tuning if there is a measured gap that rank and data do not close.

Merge the adapter or serve it separately?

Model answer. Merge for simplicity: no adapter overhead, any inference stack loads the result, but it costs a full model copy per fine-tune and fixes you to one behaviour per deployment. Serve adapters separately when many variants share a base: storage and GPU memory are shared, and vLLM supports multi-LoRA. With QLoRA, merge into a bf16 copy of the base, not the 4-bit one, to avoid rounding error. I would choose by count: a few models, merge; dozens of tenants, hot-swap.

SFT, DPO or GRPO?

Model answer. The data I can get decides it. If I have good demonstrations, SFT. If I have pairs of better and worse answers and the model already follows my format, DPO, typically after SFT. If I have prompts and a reliable programmatic check, such as maths answers or schema-valid output, GRPO. SFT and DPO are offline and cheap; GRPO generates during training, so its cost is dominated by generation length and group size. For unpaired thumbs-up and thumbs-down data there is KTO.

Colocate or server mode for vLLM?

Model answer. Colocate is the default and needs no extra infrastructure, but trainer and generator share GPU memory, so vllm_gpu_memory_utilization competes with training. It suits one node. Server mode separates them, lets generation scale with tensor parallelism and avoids memory contention, at the cost of two components, a weight-sync network path and a dev-mode endpoint that has no authentication and must sit on a private network. I would choose colocate until generation becomes the bottleneck and I have GPUs to dedicate.

Rank 8 or rank 256?

Model answer. Rank limits how much the adapter can learn. Small datasets and reinforcement learning need little capacity; the TRL docs suggest ranks of 1 to 32 for RL. Large SFT datasets benefit from larger ranks; the same docs cite 256 for post-training scale. I would also remember that alpha and rank are coupled through the scale, so I change them together and re-tune the learning rate.

What strong answers have in common

A strong answer names the mechanism: the chunked loss needing lm_head, the reference model being the base with the adapter disabled, the group needing whole multiples of num_generations. It checks before changing, using trl env, print_trainable_parameters(), decoded batches and logged completions. It is version-aware, mentioning that PPO is gone and that defaults moved. And it knows the blast radius: a dev-mode vLLM server is an unauthenticated endpoint, a reward function runs untrusted code, and trust_remote_code runs Python from a repository.

A weak answer lists hyperparameters to try, quotes an old API, or treats a rising metric as proof of success.

Rapid-fire checks

Interviewers sometimes finish with short factual probes. Have these ready.

• What does trl env do? It prints platform, Python and library versions, and the docs ask for it in bug reports.
• What replaced tokenizer= in the trainers? processing_class.
• What is the SFT default loss? chunked_nll, resolved from None; with Liger it resolves to nll.
• Which dtype argument is current? dtype, not torch_dtype.
• What is GRPO's default beta? 0.0, so no reference model is loaded.
• What is GRPO's default group size? 8, and it must be at least 2.
• How does DPO get its reference with PEFT? It disables the adapter on the same model.
• What does target_modules="all-linear" exclude? The output layer.
• Which vLLM versions does TRL 1.14 support? 0.20.0 to 0.30.0.
• How do you opt out of telemetry? HF_HUB_DISABLE_TELEMETRY=1 or HF_HUB_OFFLINE=1.

Cross-topic questions

Interviewers like to connect this tool to its neighbours. Expect questions such as how you would track runs (see the MLflow guide), how a merged model gets served (see the vLLM guide), how sharding with ZeRO relates to LoRA (see the DeepSpeed guide), and how a training image is built and pinned (see the Docker guide). A concise answer to the DeepSpeed question: LoRA shrinks the trainable state, so optimizer sharding matters less than in full fine-tuning, but the frozen weights still need to fit, and ZeRO-3 or FSDP is what lets a model larger than one GPU train at all.

Another follow-up concerns regional constraints. If your employer is in the Gulf or Egypt and the training data is regulated, say where the data and the checkpoints live, that you run in a region that satisfies the rules, that HF_HUB_OFFLINE=1 with a pre-populated model cache keeps the job from reaching out, and that telemetry is disabled by policy. Those details show you have operated this in a real organisation.

Self-test

Answer these aloud in under a minute each.

1. Explain why a LoRA run with the default SFT learning rate can look healthy and learn nothing.
2. Give the three checks you run before blaming hyperparameters.
3. Why does a PEFT-backed DPO run need no second model in memory?
4. What does reward_std near zero mean in GRPO, and what do you do about it?
5. When would you choose vLLM server mode over colocate?
6. Why merge a QLoRA adapter into a bf16 base?
7. Name two things removed from TRL in the last year that tutorials still teach.
8. What goes into your run record so that an upgrade does not change its meaning?
9. Why is the dev-mode vLLM server a security concern, and how do you contain it?
10. How do you prove that a loaded adapter is actually active?

This is the mid-level tips file. You can already run fine-tuning jobs. These pages cover the errors you will actually meet at work, the practices that pay off most in order of leverage, practice cards to drill, and operational recipes to copy. Everything is written against TRL 1.14 and PEFT 0.21. Many error messages below are quoted from the source, so you can search your logs for them.

Errors at this level

Each entry gives the symptom, the cause and the fix.

The adapter exists but nothing is learning

Symptom. Loss drops a few percent and plateaus. No error. Cause. Usually one of three things. The learning rate is the full fine-tuning default (2e-5) rather than a LoRA rate. Some target_modules entries matched nothing and were skipped silently, because PEFT only raises when none match. Or the mask excludes the tokens you care about. Fix. Raise the rate to around 2e-4 for SFT, use target_modules="all-linear", then call model.print_trainable_parameters(), get_model_status() and get_layer_status(). Decode a batch and check the trained-on span.

ValueError about chunked_nll and lm_head

Symptom. SFT stops at startup, saying chunked_nll is not supported when lm_head is wrapped by a PEFT adapter. Cause. An adapter, explicit or via a broad pattern, covers the output projection, which the chunked loss needs untouched. Fix. Remove lm_head from target_modules, or set loss_type="nll" and accept higher memory.

chunked_nll with Liger

Symptom. ValueError saying chunked_nll is not compatible with use_liger_kernel=True. Cause. You set both explicitly. Fix. Leave loss_type unset. It resolves to nll when Liger is on.

ImportError or ModuleNotFoundError for things that existed last year

Symptom. cannot import name 'PPOTrainer' from 'trl', No module named 'trl.losses', or failures for AutoModelForCausalLMWithValueHead, BCOTrainer, XPOTrainer and NashMDTrainer. Cause. Those APIs were removed: PPO in 1.13, trl.losses and several experimental trainers in 1.14. Fix. Move to GRPO or RLOO for online reinforcement learning, or pin an older TRL for a legacy job. Do not mix: pin the whole stack.

Assistant-only loss errors

Symptom. RuntimeError saying at least one example has no assistant tokens, or ValueError saying the dataset is not conversational. Cause. assistant_only_loss=True needs conversational messages data and a chat template with {% generation %} markers. Fix. Convert the data to messages, use a model family whose template TRL patches, or supply a training template with chat_template_path.

A formatting function with completion-only loss

Symptom. ValueError: A formatting function was provided while completion_only_loss=True. Cause. The two are incompatible. Fix. Apply the formatting with datasets.map before the trainer, or set completion_only_loss=False.

Wrong columns

Symptom. KeyError: The input examples must contain either 'messages' for conversational data or 'text' for standard data. Cause. The dataset has neither column. Fix. Rename columns, or set dataset_text_field.

Packing without max_length, or on images

Symptom. When packing is enabled, max_length can't be None, or Packing is not supported for vision datasets. Fix. Set max_length when packing. For vision-language models, set packing=False and max_length=None, so image placeholder tokens are not truncated.

Both a PeftModel and a peft_config

Symptom. ValueError: You passed a PeftModel instance together with a peft_config to the trainer. Fix. Pass one or the other. To continue training an adapter, load it with is_trainable=True and pass no peft_config. To add a new adapter on top of an old one, merge and unload first.

peft_config problems

Symptom. ImportError telling you to pip install trl[peft], or TypeError saying peft_config must be a peft.PeftConfig, got dict. Fix. Install the extra. Build a LoraConfig object.

Quantization configured twice

Symptom. ValueError: You set quantization_config both as a trainer argument and in args.model_init_kwargs. Fix. Set it once. On DPO, do not pass quantization_config together with an already-instantiated model.

The reference model complaints

Symptom. ValueError: model and ref_model cannot be the same object, or NotImplementedError about sync_ref_model=True with a PEFT model, or a ValueError about sync_ref_model=True while beta=0.0. Cause. With PEFT the reference is the base with the adapter disabled, and GRPO's beta defaults to 0.0, so there is no reference to sync. Fix. Leave out ref_model. Drop sync_ref_model, or set beta above zero for a non-PEFT GRPO run.

GRPO batch and group errors

Symptom. A minimum-two-generations error, a generation_batch_size must be divisible by num_generations error, or auto_find_batch_size is not supported by GRPO. Fix. Use num_generations of at least 2, make the batch arithmetic a multiple of it, set auto_find_batch_size=False and size by hand.

Reward function returns the wrong count

Symptom. The reward function returned k rewards, but n were expected. Fix. Return exactly one float per completion, and accept **kwargs.

Streaming datasets

Symptom. Iterable datasets require dispatch_batches=False, or a failure about max_steps. Fix. With iterable datasets in GRPO or RLOO, set max_steps and dispatch_batches=False.

No completions generated

Symptom. RuntimeError: No completions were generated. Cause. vLLM failed or was never reachable. Fix. Read the server logs. Check the version window (0.20.0 to 0.30.0), disjoint GPUs, VLLM_SERVER_DEV_MODE=1, the NCCL weight-transfer flag, host, port and the 240-second timeout.

Unexpected keyword when loading an adapter

Symptom. TypeError: LoraConfig.__init__() got an unexpected keyword argument. Cause. The adapter was saved with a newer PEFT. Fix. pip install -U peft. Deleting the key from adapter_config.json is only safe if it held the default value.

Unscaling fp16 gradients

Symptom. ValueError: Attempting to unscale FP16 gradients. Cause. The model was loaded in fp16 and trained with fp16 mixed precision, leaving trainable weights in fp16. Fix. Use bf16 where supported (TRL enables it by default unless you set fp16), or cast with peft.cast_mixed_precision_params.

The second adapter is never trained

Symptom. Multi-adapter training updates only the first. Cause. An inactive adapter has requires_grad=False. Fix. Call model.base_model.set_adapter(["default", "other"]) before creating the optimizer or trainer.

Merging a subset of adapters

Symptom. After merging, one adapter's behaviour has vanished. Cause. Merging only some of several active adapters silently drops the others; PEFT says this is planned to change at v1.0. Fix. Merge all, or unmerge first.

add_adapter raises after an upgrade

Symptom. A peft.PeftError on PEFT 0.21. Cause. It now raises when the new adapter targets layers that do not exist. Fix. Correct the targets, or catch PeftError where a mismatch is expected.

The practices that pay off most, in order of leverage

1. Decode before you train. One minute spent reading the trained-on span of a batch prevents more wasted runs than any hyperparameter search.
2. Pin the whole stack and record trl env. TRL ships about every two weeks and removes things. A pinned environment turns upgrades into decisions.
3. Write every default you depend on into the config. Defaults flipped in the last year: vllm_mode, the SFT loss, GRPO's length and KL correction.
4. Verify the adapter attached. print_trainable_parameters() takes a second and catches the silent partial mismatch.
5. Use LoRA learning rates. About ten times the full fine-tuning rate: SFT 2e-4, DPO 5e-6, GRPO 1e-5.
6. Target all linear layers. Attention-only adapters underperform even at equal parameter counts, per the TRL docs.
7. Smoke test on a tiny model in CI. Two steps on CPU catch most upgrade breakage.
8. Read completions, not only curves. For GRPO use log_completions=True. For DPO sample generations.
9. Start preference tuning from an SFT checkpoint.
10. Keep the effective batch size explicit and recompute it whenever the GPU count changes.
11. Treat deprecation warnings as a backlog. Everything deprecated is removed in v2.0.0.
12. Opt out of telemetry deliberately where policy requires it.

Practice cards

Each card is a ten-minute exercise on a small model. Do them on a Linux GPU box where noted, otherwise a CPU or Apple Silicon laptop with a 0.5B model is enough.

Card 1: Break the mask. Train 20 steps of SFT on prompt-completion data. Print the decoded trained-on span. Switch completion_only_loss=False and compare. Write down what each setting trains on.

Card 2: Break the targets. Wrap a model with target_modules=["q_proj", "v_proj"], then "all-linear", then a list with one real and one fake name. Record the trainable percentage each time and what PEFT said.

Card 3: Learning-rate sweep. Run 100 steps of LoRA SFT at 2e-5, 1e-4 and 2e-4. Plot the three loss curves.

Card 4: The lm_head error. Add lm_head to the targets and read the error. Fix it two ways and compare memory.

Card 5: DPO read-out. Run DPO for 100 steps. Plot the four reward metrics. Change beta to 0.5 and compare.

Card 6: Reward function unit test. Write a GRPO reward function and a pytest that feeds it three fake completions, in both string and message format, and asserts the output length.

Card 7: Group arithmetic. For your GPU count, per-device batch size and steps_per_generation, compute the generation batch size by hand. Choose values that pass the divisibility check, then run to confirm.

Card 8: Reload test. Save an adapter, reload with PeftModel.from_pretrained, and assert the logits with the adapter differ from those under disable_adapter().

Card 9: Merge comparison. Merge the adapter, save, and compare outputs on five prompts with the adapter version.

Card 10: Upgrade rehearsal. In a throwaway environment, bump TRL by one minor version, run your smoke test, and list every warning.

Operational recipes

Recipe: a pinned, reviewable training environment

Create the environment, install PyTorch first with the selector at pytorch.org for your CUDA version, then the libraries, and freeze:

python -m venv .venv && source .venv/bin/activate
# install PyTorch first, using the command from pytorch.org for your CUDA version
pip install "trl[peft]"
pip freeze > requirements.lock
trl env | tee env-report.txt

Commit requirements.lock and keep env-report.txt as a run artifact. Python 3.10 or later is required by both libraries.

Recipe: the SFT-with-LoRA YAML

model_name_or_path: Qwen/Qwen2.5-0.5B
dataset_name: trl-lib/Capybara
output_dir: runs/sft-001
learning_rate: 2.0e-4
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
num_train_epochs: 1
max_length: 1024
use_peft: true
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules: all-linear
report_to: trackio

Launch it with trl sft --config sft_lora.yaml. Add --num_processes 4 --accelerate_config zero2 to go multi-GPU.

Recipe: QLoRA, in code

Pass quantization_config to the trainer and a LoraConfig as peft_config, as in the Detailed guide. Use bnb_4bit_quant_type="nf4", bf16 compute and double quantization. Remember it needs a Linux or Windows CUDA machine; macOS gets only a CPU build of bitsandbytes.

Recipe: a safe vLLM server for GRPO

CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-7B \
  --tensor-parallel-size 4 \
  --weight-transfer-config '{"backend": "nccl"}' \
  --logprobs-mode processed_logprobs --max-logprobs -1

Run the trainer on the other GPUs with GRPOConfig(use_vllm=True, vllm_mode="server"). Bind to a private interface, firewall ports 8000 and 51216 to the trainer's address only, and never expose the server through a public ingress, because dev mode has no authentication. On Kubernetes, use a ClusterIP Service plus a NetworkPolicy. See the Kubernetes guide.

Recipe: merge for serving

from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype="auto")
model = PeftModel.from_pretrained(base, "runs/sft-001").merge_and_unload()
model.save_pretrained("serve/qwen3-merged")

Then serve serve/qwen3-merged as an ordinary model; see the vLLM guide. For a QLoRA adapter, load the base in bf16 as shown, not in 4-bit.

Recipe: offline and air-gapped runs

export HF_HUB_OFFLINE=1            # also disables TRL telemetry
export HF_HUB_DISABLE_TELEMETRY=1  # explicit, in case offline mode is later removed

Pre-populate the Hugging Face cache on a connected machine, copy it to the cluster, and reference models by local path or cached name. This is the usual pattern when data-residency rules in the Gulf or Egypt keep training hosts off the public internet.

Recipe: a CI smoke test

Build each trainer you use with the smallest model that loads, run max_steps=2, and save the adapter. Assert that trainable parameters are non-zero, that the adapter directory contains adapter_config.json and adapter_model.safetensors, and that a reload through PeftModel.from_pretrained changes the logits. Run it on every dependency bump, with FutureWarning from trl promoted to an error on the upgrade branch.

Recipe: a managed Job

hf jobs uv run --flavor a10g-small --timeout 30m --secrets HF_TOKEN -- \
  https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/trl/scripts/sft.py \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct --dataset_name trl-lib/Capybara \
  --output_dir out --push_to_hub

The default timeout is 30 minutes, so raise it for a real run. Billing stops when the script exits. Pass the token with --secrets, never inline.

Recipe: a triage checklist for a sick run

1. Run trl env and compare against the pinned environment.
2. Print the resolved config and check the explicit defaults.
3. Decode one batch and its trained-on span.
4. Check print_trainable_parameters() and get_layer_status().
5. Read the first five logged metrics for the method, then read completions.
6. For online runs, read the vLLM server logs before the trainer logs.

A short weekly checklist

• Are all training environments pinned, with trl env stored?
• Does every run have a committed YAML with explicit defaults?
• Does CI run a tiny smoke test on dependency bumps?
• Are reward functions sandboxed and unit tested?
• Are vLLM servers on a private network with firewalled ports?
• Are tokens in environment variables or a secret manager, not in files?
• Is telemetry configured on purpose?
• Is anyone reading the deprecation warnings?
• Are adapters stored as safetensors, with the base model revision recorded?
• Do you know who will review the next upgrade?

Reading a run like a reviewer

Most of the value in this level comes from reading evidence correctly, so here is how a careful reviewer approaches a finished or failing run.

Start with the environment. A run without a recorded trl env cannot be compared with another, because the defaults may have changed between them. Next read the resolved config, not the YAML you meant to write. Fields left at None, such as the SFT loss_type, resolve at runtime, and the CLI's LoRA defaults of rank 16, alpha 32 and dropout 0.05 differ from a bare LoraConfig. Many "mysterious" differences between two runs are one of these.

Then look at the shape of the curves before the values. A loss that is flat from step one points at masks, learning rate or a detached adapter. A loss that falls and then spikes points at the learning rate or a data problem such as a very long example. In DPO, plot chosen and rejected rewards separately, because a growing margin built from two falling lines is a warning, not a success. In GRPO, check whether reward_std is above zero before reading anything else, because a zero value means no learning signal whatever the mean reward does.

Finally read samples. Take ten prompts the model never saw, generate with the base model and with the adapter, and put the outputs side by side. Reviewers who do this catch length drift, format collapse and reward hacking that no scalar shows. Keep the same ten prompts across runs so improvements are comparable, and keep them out of the training data.

When you report a result to a team, give the pinned versions, the config file, the data revision, the base model revision, the metric you chose and the samples you read. A result without those is an anecdote.

This is part three of three. It takes everything from Beginner and Mid, the trainers, the adapters, the memory levers and the vLLM wiring, and asks what changes when you are no longer the person running the job but the person responsible for the system that other teams run their jobs on. Every topic gains a failure mode, a cost line or a security question.

The numbers below are checked against TRL 1.14 and PEFT 0.21, the current releases when this was written. Both libraries move fast: TRL ships roughly every two weeks and PEFT is still pre-1.0. Treat every default quoted here as something to pin and test, not something to trust.

Where this picks up

I start with the shape of the system you own, because the failure modes follow from the shape.

What you are actually operating

PEFT and TRL are libraries, not services. There is no TRL server, no operator, and no control plane. The Hugging Face documentation is explicit that Kubernetes and SLURM reach TRL only through accelerate launch or torchrun. So "owning PEFT and TRL as a platform" means owning a set of conventions around a Python process: the images it runs in, the version pins, the launch wrapper, the data it reads, the artefacts it writes, and the cluster it is scheduled on.

It helps to name the four architectures that appear in practice, because each has a different bottleneck and a different blast radius.

The first lane is forgiving. A failed SFT run costs you the GPU hours and nothing else, it can be restarted from a checkpoint with trainer.train(resume_from_checkpoint=True), and its output is a small file. The second lane is where platform engineering earns its salary. Online methods generate completions from the current policy during training, score them with reward functions that run arbitrary code, and then push updated weights to the generator after every optimizer step. Throughput is dominated by generation: max_completion_length multiplied by num_generations sets the cost of each step, not the backward pass.

Notice also what is not in the picture: a model server, a feature store, a scheduler. TRL has no opinion about any of them. That is a strength, since you can plug it into the scheduler and registry you already run, and a risk, since nothing in the library enforces the conventions you will need.

The stability contract and the version treadmill

The single most important fact for a platform owner is that these libraries do not promise what an enterprise dependency usually promises. TRL's own documentation says that anything under trl.experimental "may change or be removed in any release (including patch versions) without prior deprecation", and tells you not to rely on it for production workloads. That is a quotable policy, and you should quote it in your platform's onboarding document.

The stable surface in 1.14 is seven trainer and config pairs: SFTTrainer, DPOTrainer, GRPOTrainer, RLOOTrainer, KTOTrainer, RewardTrainer and DistillationTrainer. Everything else, including OnlineDPOTrainer, CPOTrainer, ORPOTrainer, GKDTrainer, GOLDTrainer, AsyncGRPOTrainer and others, lives under trl.experimental. Two promotions in recent releases show the pattern: DistillationTrainer became stable in 1.10 and KTOTrainer in 1.8, each leaving a deprecated import path that warns and will be removed in v2.0.0.

The removals show the other half of the pattern. Version 1.13 deleted PPO entirely: PPOTrainer, PPOConfig and the value-head wrappers are gone. Version 1.14 deleted six experimental trainers (GRPOWithReplayBufferTrainer, BCOTrainer, PRMTrainer, XPOTrainer, NashMDTrainer and the GSPO-token trainer) and the short-lived trl.losses module. A team that built its workflow on any of these found out from an ImportError. If someone on your team proposes PPO because a tutorial uses it, the answer is GRPO or RLOO, and the reason is on the record.

Defaults move too, and these changes are quieter than removals because nothing fails. Here are the ones a platform must pin:

The practical response is a lockfile plus an explicit config. Pin trl, peft, transformers, accelerate, vllm, torch and deepspeed together, because TRL raises dependency floors often (1.13 raised peft>=0.13.0 and deepspeed>=0.18.6) and because the vLLM window is a hard range: 0.20.0 to 0.30.0 for TRL 1.14. Then write every default you depend on into the config file instead of inheriting it. A config that says vllm_mode: colocate and loss_type: chunked_nll survives a version bump that changes the default; a config that says nothing does not.

pip install "trl[peft,vllm]==1.14.1" "peft==0.21.1"
pip freeze | grep -iE '^(trl|peft|transformers|accelerate|vllm|torch|deepspeed|bitsandbytes)=='
trl env          # paste this into every run record and every bug report

PEFT has its own coupling: transformers v5 needs PEFT 0.18.0 or later, and tensor-parallel LoRA needs PEFT 0.21.1 with transformers 5.17 or later. An adapter saved by a newer PEFT can fail to load in an older one with TypeError: LoraConfig.__init__() got an unexpected keyword argument, which means your training image and your serving image cannot drift independently.

Scaling: which component breaks first

Scaling these libraries is not one problem but several, each with its own first-to-fail component. Walking them in the order you meet them as jobs grow is the most useful way to plan capacity.

A single GPU runs out of memory first. This is the case Beginner and Mid taught you to solve with LoRA, gradient checkpointing (on by default in every TRL config), the chunked_nll default, a smaller max_length, and a batch of one with gradient accumulation. By the time a team asks you for help, they have usually pulled those levers. The next steps are activation_offloading=True, use_liger_kernel=True, or moving to a sharded strategy.

DDP breaks next, because it replicates the model. Plain data parallelism puts a full copy of the weights on every GPU, so adding GPUs adds throughput but never lets a bigger model fit. When the model itself does not fit, the options are FSDP2 or DeepSpeed ZeRO-3. TRL bundles accelerate presets for this (fsdp1, fsdp2, zero1, zero2, zero3, multi_gpu, single_gpu), selectable with --accelerate_config on the CLI. For the DeepSpeed side, see the DeepSpeed guide. One interaction deserves a platform-level note: with PEFT, ZeRO-3 and gradient checkpointing together, TRL forces use_reentrant=True, overriding the modern default, so a "why is this slow" ticket may trace back to that interaction rather than to your cluster.

Long context breaks the logits tensor. The full [batch x seq x vocab] logits matrix is what kills long-sequence SFT, and chunked_nll exists precisely to avoid materialising it by computing cross-entropy in chunks of 256 tokens. Beyond that, TRL's long-context guide lists gradient-checkpointing offload (which needs transformers 5.16 or later), YaRN RoPE scaling and sequence parallelism. The limits are written down and you should publish them to your users: GRPO does not support context or sequence parallelism, and packing is incompatible with context parallelism. DPO padding-free was temporarily unavailable as of v1.13. Naive pipeline parallelism through device_map does not combine with data parallelism, and requires the lm_head on GPU 0.

Mixture-of-experts models break the adapter targeting. Expert weights are fused tensors rather than Linear layers, so target_modules misses them. PEFT's target_parameters, added in 0.17.0, targets raw nn.Parameter objects instead, and TRL's MoE guidance adds model_init_kwargs={"output_router_logits": True} with router_aux_loss_coef. Make your platform's reference configs for MoE explicit, because the failure here is silent: a LoRA that adapts only attention and trains happily.

Online RL breaks on generation. Everything in the previous paragraphs applies, and then the generator becomes the bottleneck. The next section covers it in full, because topology is the biggest decision you will make for these workloads.

Multi-node breaks on the network and on coordination. The accelerate config needs num_machines, machine_rank, main_process_ip, main_process_port (29500 by default) and num_processes, the total across nodes. TRL's docs show a SLURM layout with four training nodes and one node running vllm serve. There is no TRL scheduler: gang scheduling, preemption and retry are yours to provide through Kubernetes, SLURM or Ray. See the Ray guide if that is your route.

# the effective batch size is the quantity to reason about, not the per-device value
# effective = per_device_train_batch_size x num_devices x gradient_accumulation_steps
accelerate launch --config_file zero3.yaml train.py \
  --per_device_train_batch_size 1 --gradient_accumulation_steps 16

Online RL topology: colocate, server and async

In GRPO and RLOO every optimizer step needs fresh completions from the current weights, so generation speed decides cost. TRL lets vLLM do that generation in three arrangements. Choosing between them is a capacity-planning decision, and because the default flipped in 1.0, it is also a pinning decision.

In colocate mode, the default since 1.0, vLLM runs inside the trainer process on the training GPUs. vllm_gpu_memory_utilization defaults to 0.3, so vLLM reserves about a third of each card, and the trainer must live in the rest. There is no extra hardware and no network hop, which makes colocation the right default for small and medium jobs. The cost is contention: when GRPO runs out of memory, the first lever is lowering vllm_gpu_memory_utilization, and vllm_enable_sleep_mode=True lets vLLM release memory between generation phases.

In server mode, a separate vllm serve process runs on dedicated GPUs and the trainer pushes weights to it over NCCL after each step. Dedicated generators let you size each side independently and use tensor parallelism for large models, since data parallelism no longer helps dense models in vLLM. The official layout splits eight GPUs four and four:

CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-7B \
  --tensor-parallel-size 4 \
  --weight-transfer-config '{"backend": "nccl"}' \
  --logprobs-mode processed_logprobs --max-logprobs -1

CUDA_VISIBLE_DEVICES=4,5,6,7 accelerate launch train.py   # GRPOConfig(use_vllm=True, vllm_mode="server")

Each flag has a job you should be able to defend in a review. VLLM_SERVER_DEV_MODE=1 exposes the weight-transfer endpoints (/start_weight_update, /update_weights, /finish_weight_update), and also vLLM's other development endpoints. --weight-transfer-config selects the NCCL backend, or "ipc" when trainer and server share a GPU. --logprobs-mode processed_logprobs is what importance sampling expects. --max-logprobs -1 is required for distillation's top-k. The trainer and the server need disjoint CUDA_VISIBLE_DEVICES, and the trainer waits vllm_server_timeout, 240 seconds by default, for the server to appear. Since v1.11 the older trl vllm-serve is a deprecated wrapper that prints the equivalent vllm serve command; teach and automate the direct form. See the vLLM guide for the serving side itself.

The third arrangement is async GRPO, trl.experimental.async_grpo.AsyncGRPOTrainer. Rollouts run in a separate process against a vLLM server and overlap with training, which raises GPU utilisation. The price is staleness: completions come from slightly older weights, which is why importance-sampling correction exists. The defaults are vllm_importance_sampling_correction=True, mode "sequence_mask" and a clip maximum of 3.0. It is experimental, so the stability contract applies in full, and since v1.6 its reward_funcs, tools and environment_factory must be picklable, which rules out closures over open connections.

Two further constraints belong in the platform contract. vLLM is Linux only: its documentation lists Linux as the supported OS, and native Windows is unsupported, so Windows users go through WSL2. And the version window is hard, so your trainer image and your generator image must resolve vLLM from the same constraint file. A mismatch tends to surface as RuntimeError: No completions were generated, which sends people to the wrong logs.

A cost and memory model per job class

A platform needs cost to be predictable enough to budget, and these libraries make that easier than most because the drivers are few. The skill is knowing which driver belongs to which job class.

For offline jobs with LoRA, cost is GPU hours, which is sequence length times dataset size divided by throughput, and the memory ceiling is set by the base model, max_length and the batch. The big wins have already shipped as defaults: chunked_nll cut peak SFT memory by about 30 percent on average and up to 50 percent on large-vocabulary models, and gradient checkpointing, also on by default, trades roughly 20 percent speed for memory. Packing, with packing=True and the default "bfd" strategy, removes padding waste, and is the cheapest throughput gain for short-example datasets. Checkpoints are the storage side: adapters are megabytes to a few hundred megabytes, against gigabytes for a full model, which changes the economics of keeping every experiment.

For online jobs, three numbers set the cost: num_generations (default 8), max_completion_length (default 512 since 1.10) and the vLLM GPU split. Doubling num_generations doubles generation per step. That makes the 1.10 default change from 256 to 512 completion tokens a silent doubling for any config that inherited it. Make your intake form ask for those three numbers.

Hugging Face Jobs offers a hosted alternative worth knowing as a benchmark and as an overflow valve. The documented command runs a script on a named hardware flavour, bills until the script exits, and has a default timeout of 30 minutes:

hf jobs uv run --flavor a10g-small --timeout 30m --secrets HF_TOKEN -- \
  https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/trl/scripts/sft.py \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct --dataset_name trl-lib/Capybara \
  --output_dir out --push_to_hub

Jobs are not a substitute for your cluster when data cannot leave your environment. For employers in the Gulf and Egypt with data-residency obligations, check where the data and the Hub repositories live before you route anything sensitive there. The point stands as a price check: if an internal GPU hour costs more than the managed flavour once you include idle time, your utilisation is the problem.

Security: the trust model, secrets and the supply chain

The security posture of a training platform differs from that of a web service, because the dangerous inputs are not requests but artefacts: models, datasets, adapters, reward code and kernels. Start from the question the Docker guide taught you to ask, namely what a compromised component can reach.

Remote code. trust_remote_code=False is the default on every TRL config. Turning it on executes Python from a Hub repository on your training node, with whatever credentials that node holds. Since v1.7 the flag lives on every trainer config and covers model, processor, reference, reward and teacher; ModelConfig.trust_remote_code was removed. If a team must enable it, require pinning model_revision to a commit SHA, never a branch, and run the job in a network-restricted sandbox. TRL itself moved its log-probability kernel in-tree in 1.14, shipped as trl.kernels, precisely so users would not need a Hub kernel with remote code.

Pickle. Adapters are either adapter_model.safetensors or the legacy adapter_model.bin, and .bin is a pickle file that can run code on load. Make safetensors a hard policy at your registry: reject uploads containing .bin, and scan before anything is load_adapter-ed in a shared process. Adapter provenance matters because an adapter is just weights, and weights from an untrusted source can encode behaviour you did not evaluate.

Reward functions and environments run arbitrary code. A GRPO reward function is a Python callable that the trainer invokes with prompts and completions; an agent environment or tool executes actions the model proposes. If the model's output reaches a shell, an interpreter or a network, you have a code-execution surface fed by a stochastic generator. Sandbox them: separate containers or VMs, no credentials mounted, no egress, strict timeouts. TRL's OpenEnv integration exists for this purpose.

Secrets. Use HF_TOKEN, from hf auth login or the environment, for gated models, push_to_hub and Jobs, and prefer fine-grained tokens scoped to the repositories a job needs. Tracker keys such as WANDB_API_KEY belong in environment variables or a secret manager, injected at launch. The one habit to ban is committing YAML run configs with keys in them; the CLI reads configuration from a YAML file through --config, which makes that file easy to commit by accident. See the Vault guide for injection patterns.

Telemetry. Since v1.5, TRL sends one anonymous ping per trainer instantiation, containing the TRL version, trainer class, model architecture (only if transformers knows it), whether PEFT is used, the dataset source kind, the distributed backend, a bucketed world size, and the accelerator and GPU model. The source IP is visible to the receiving server. Opt out with HF_HUB_DISABLE_TELEMETRY=1 or HF_HUB_OFFLINE=1, and nothing is sent when CI is set. For air-gapped or regulated clusters, set HF_HUB_OFFLINE=1 and pre-populate the cache, which also removes the dependency on Hub availability during a run.

Supply chain. The dependency set is wide: torch, transformers, accelerate, vllm, bitsandbytes, deepspeed and liger-kernel each release independently. Build one base image per pin set, scan it as you would any other (see the Docker guide for digest pinning and provenance), and install from an internal mirror. Treat an upgrade as a supply-chain event with an owner and a changelog review.

Adapter lifecycle: registry, merging and serving

Small artefacts make adapters easy to produce and easy to lose track of. A platform needs a lifecycle for them: where they live, what metadata they carry, how they are promoted, and how they are served.

An adapter is two files that matter: adapter_model.safetensors and adapter_config.json. The config records the PEFT configuration and the name of the base model it was trained against. It does not contain the base model. The consequence is that an adapter is meaningless without the exact base weights, and "exact" includes the revision. Your registry entry should record the base model identifier and revision, the PEFT and TRL versions, the config hash, the dataset version and the training run link. If you already use MLflow or Weights & Biases, register the adapter as an artefact there and link it to the run, rather than building a second system.

At serving time you have two options, and the decision is about economics and operations.

Merge and serve a plain model. merge_and_unload() folds the low-rank update into the base weights and returns a standard transformers model, with zero adapter overhead at inference. The cost is one full model copy per fine-tune, in storage and in serving memory. For a single high-traffic model, merging is the right choice.

Serve one base with hot-swapped adapters. vLLM supports multi-LoRA serving, so many tenants share one base model and each request names an adapter. Storage and GPU memory scale with adapters, not full models. Note that vLLM and Diffusers load only LoRA, which is why PEFT 0.19 added converters from non-LoRA adapters to LoRA; if a team trains with a more exotic method, conversion is how they reach your serving tier. This is the better fit for many small, per-tenant fine-tunes.

Merging has three traps worth building into your promotion checks. First, merging a QLoRA adapter into a quantized base introduces rounding error; load the base in bf16 before merging. Second, merging only some of several active adapters silently drops the others, and the docs say that behaviour is planned to change in PEFT v1.0, so merge all of them or unmerge first. Third, add_weighted_adapter with the SVD combination previously applied LoRA scaling twice, fixed in PEFT 0.21.0, so merged results can differ from older versions.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "adapters/support-bot/v7")
merged = model.merge_and_unload()
merged.save_pretrained("merged/support-bot-v7")

A promotion gate should run the merged or adapter-served model on a fixed evaluation set and compare against the previous version, because the merge itself is a change. Pair this with serving-side checks from the KServe guide or whatever your serving tier is.

Multi-tenancy and isolation

Multi-tenancy for these tools splits cleanly into training and serving, and the two have different answers.

Training is single-tenant by design. A training job is one Python process group with one set of credentials, one dataset and one output directory. There is no in-library notion of tenants, quotas or isolation. So tenant isolation at training time comes from the layer below: a namespace per team in Kubernetes, a resource quota on GPUs, a service account per team with access only to that team's data and registry paths, and network policy that stops one team's vLLM server from being reachable from another's pods. Queueing and fairness come from your scheduler. TRL gives you no hook for them.

Serving is where tenants share a base model. PEFT can hold many adapters on a model with load_adapter, set_adapter and delete_adapter, and can run mixed-adapter batches with model(**inputs, adapter_names=[...]), where "__base__" selects the base model. Only LoRA and RoAd support mixed-adapter batches. The isolation is weaker than people assume: each tenant's adapter is just weights, and the serving layer is responsible for ensuring tenant A cannot request tenant B's adapter name. Authorise adapter selection at the gateway, from the caller's identity, rather than trusting a client-supplied field.

Two PEFT behaviours matter when you build on adapter management. An inactive adapter has requires_grad=False, so training a second adapter requires model.base_model.set_adapter(["default", "other"]) before the optimizer is created. And from 0.21.0, add_adapter raises peft.PeftError when the new adapter targets layers that do not exist, which is better than the old silent behaviour but is a behaviour change your code must handle.

Observability: run health, not only GPU health

GPU utilisation tells you a job is busy. It does not tell you whether the job is learning, and for post-training that distinction is most of the operational difficulty. The libraries log what you need; a platform's job is to collect it consistently and alert on it.

Every TRL config logs every 10 steps by default (logging_steps=10), and report_to selects the backend. Trackio is TRL's recommended tracker (report_to="trackio", configured with TRACKIO_PROJECT and TRACKIO_SPACE_ID); W&B, TensorBoard and MLflow work through the standard transformers mechanism. Pick one as the platform default, and make the run name, config hash and trl env output part of every record.

The signals differ by method, so the dashboard template should too:

For GRPO specifically, turn on log_completions=True for a completions table, and give reward functions the log_extra and log_metric callables so per-component scores show up. A reward function that returns a constant, or that is trivially gamed, is the most common cause of a run that looks healthy and is worthless, and per-function reward curves are how you catch it.

On the PEFT side, silent misconfiguration is the main hazard. PEFT raises an error only when none of your target_modules match; a partial mismatch is skipped without a word. Catch it with inspection calls that belong in a startup check:

model.print_trainable_parameters()      # trainable params and their share of the total
status = model.get_model_status()       # adapter-level summary, including irregular state
layers = model.get_layer_status()       # per-layer view; look for "irregular"

Log the trainable-parameter percentage at job start, and fail fast when it falls outside the expected band for the chosen rank and targets. That one check prevents the whole class of "trained for nine hours and nothing changed" tickets.

Finally, do not forget infrastructure metrics, which belong in Prometheus like everything else: GPU memory and utilisation per rank, NCCL timeouts, checkpoint write latency, and for server-mode GRPO, the vLLM server's request latency and queue depth. Alert on reward_std and entropy through the tracker's own alerting or an export, because those are the signals your GPU dashboards will never show.

Governance, reproducibility and change control

Governance in this context is mostly about being able to answer, months later, "why does this model behave this way, and can we rebuild it?" The libraries give you the ingredients; you have to insist on recording them.

A reproducible post-training run needs five things captured together: the exact package set (the lockfile and trl env output), the resolved config with every default written out, the dataset identity and version, the base model identifier and revision, and the random seeds. Defaults are the weak point because they change under you, as the table earlier showed. Make the launch wrapper write the resolved config, meaning the config object after defaults are applied, next to the adapter, not the YAML the user typed.

Datasets deserve their own controls. Since TRL 1.0, the SFT, DPO and Reward trainers no longer strip None values from datasets automatically, so data prepared for older versions can behave differently. If you need the old behaviour you apply dataset.with_transform(remove_none_values) from trl.trainer.utils yourself. Dataset type also matters: each trainer expects a particular type (language modelling or prompt-completion for SFT, preference for DPO, prompt-only for GRPO and RLOO, unpaired preference or preference for KTO), and a schema validator at intake that rejects the wrong shape prevents the KeyError: The input examples must contain either 'messages'... class of failure before it costs GPU time.

Change control comes down to three gates. An upgrade gate runs a reference SFT, a reference DPO and a reference GRPO job on the new pin set and compares curves and evaluation scores to the previous set, which catches default drift. A config gate checks submitted configs against policy: allowed trust_remote_code, rank bounds, maximum num_generations, required tracker. A promotion gate runs the evaluation set against the adapter before it is registered as deployable. Write the policy down once, enforce it in the launcher, and reviewers stop arguing about it in pull requests.

Data governance has a regional dimension in the Middle East and North Africa. If prompts or completions contain personal data, the residency rules of the employer's jurisdiction apply to the training data, the checkpoints (which can memorise) and the logs, including the completions table that log_completions=True writes. The option log_completions_hub_repo pushes completions to a Hub repository; make sure it is off or private by default for sensitive workloads.

Incident playbooks

Incidents on a training platform fall into a small number of shapes. For each, know the first three questions, because the quoted error is rarely the real cause.

Out of memory. torch.OutOfMemoryError: CUDA out of memory is the most common ticket. Ask: is it the first step or step N? First-step failures mean the static footprint (model, batch, max_length) is too big, so reduce per_device_train_batch_size to 1 with gradient accumulation, cut max_length, or move to QLoRA. Failures at step N point to a long example or, in GRPO, a long completion, so check the length distribution. In colocate GRPO, lower vllm_gpu_memory_utilization or enable sleep mode, and reduce num_generations.

GRPO refuses to start. The messages are explicit and map straight to config fixes. GRPO requires at least 2 generations per prompt means num_generations is below 2. generation_batch_size (X) must be divisible by num_generations (Y) means per-device batch, process count and steps_per_generation do not multiply to a multiple of num_generations. auto_find_batch_size is not supported by GRPO means exactly that. And The reward function '{name}' returned {k} rewards, but {n} were expected means a reward function returned the wrong number of values; it must return one float per completion.

No completions in server mode. RuntimeError: No completions were generated usually hides a vLLM-side failure. Check, in order: the server logs for the underlying error; whether the installed vLLM is inside 0.20.0 to 0.30.0; whether trainer and server use disjoint CUDA_VISIBLE_DEVICES; whether VLLM_SERVER_DEV_MODE=1 is set; and whether the trainer can reach vllm_server_host and vllm_server_port within the 240-second timeout.

Import errors after an upgrade. ModuleNotFoundError: No module named 'trl.losses' and ImportError: cannot import name 'PPOTrainer' from 'trl' are not environment problems; the API was removed. The fix is a code migration, or a pin to an older TRL for a bounded time.

Chat-template and loss-masking failures. RuntimeError: You're using assistant_only_loss=True, but at least one example has no assistant tokens means the chat template lacks {% generation %} markers; use a training template with them, or one of the families TRL patches automatically, such as Qwen3. loss_type='chunked_nll' is not supported when lm_head is wrapped by a PEFT adapter means remove lm_head from the targets or set loss_type="nll". loss_type='chunked_nll' is not compatible with use_liger_kernel=True means leave loss_type unset, since it resolves to nll with Liger.

A trained adapter that does nothing. The model behaves like the base. Ask: was it loaded with get_peft_model instead of PeftModel.from_pretrained? Is the model in .eval()? Did a new classification head go missing from modules_to_save? Did the targets partially miss? Then run the status calls above. For a model that trained but did not move, check the trainable-parameter share first.

Adapters that fail to load in serving. TypeError: LoraConfig.__init__() got an unexpected keyword argument means the adapter came from a newer PEFT than the server runs. Upgrade serving; deleting the key from adapter_config.json is a last resort that is only safe if it held the default value.

Where PEFT and TRL stop

A platform owner's most valuable skill is knowing when the tool is the wrong one. These libraries have clear edges.

They stop at Linux for the online path. vLLM supports Linux only, so GRPO with vLLM cannot run natively on Windows or macOS. On Apple Silicon, TRL and PEFT core work on MPS or CPU but slowly, and bitsandbytes offers only a CPU wheel there, so there is no GPU 4-bit training. For laptops, that means small models for learning and a Linux target for real runs.

They stop at the stable surface. If the method you need lives under trl.experimental, you are adopting a moving target. That can be right for research, and wrong for a platform commitment. A reasonable policy is that nothing experimental ships in a supported template.

They stop where a faster stack wins. Unsloth is a faster drop-in fine-tuning stack built on TRL, and Axolotl wraps training behind YAML configuration. If your teams want recipes over code, or a particular speed profile on a single GPU, evaluating them is reasonable, and since both sit on or alongside the same ecosystem, the adapters usually remain portable. For framework-level orchestration and trainer abstractions, PyTorch Lightning is a different shape of tool. For very large sharded training, go deeper into DeepSpeed directly.

They stop at the scheduler. There is no TRL operator, queue or gang scheduler. If you need fair-share GPU scheduling across dozens of teams, that comes from Kubernetes with a queueing layer, SLURM, or Ray, and TRL is a workload inside it.

They stop at serving. Neither library serves models. Production serving is vLLM, KServe or similar, and the lifecycle in this guide is about producing artefacts those systems consume.

They stop at the pre-1.0 promise. PEFT documents several behaviours as planned to change in v1.0, and TRL removes experimental code without notice. If your organisation needs a multi-year API stability guarantee, wrap the libraries behind your own thin interface so that churn is absorbed in one place.

Running this as a platform for other teams

Everything above converges on a product: a paved road that lets another team fine-tune a model without learning every hazard in this guide. Its parts are modest.

A launch wrapper. One entry point that takes a small config, resolves defaults into an explicit file, validates against policy, selects an accelerate preset, injects secrets, sets telemetry variables and submits to the scheduler. The wrapper is where the review checklist below becomes code. It can call the trl CLI underneath: trl sft, trl dpo, trl grpo, trl rloo, trl kto, trl reward and trl distillation all accept --config with a YAML file whose keys are the flag names, and --num_processes with --accelerate_config for multi-GPU.

model_name_or_path: Qwen/Qwen3-0.6B
dataset_name: my-org/support-conversations
output_dir: runs/support-bot-v8
use_peft: true
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules: all-linear
learning_rate: 2.0e-4
loss_type: chunked_nll
report_to: trackio

The CLI's LoRA defaults (r=16, alpha=32, dropout 0.05) differ from LoraConfig's own (r=8, alpha=8, dropout 0.0), so a team that moves from the CLI to Python code changes its adapter without noticing. Your templates should state the values in both forms.

Golden configs. A small library of known-good recipes per job class: SFT with LoRA, QLoRA on a single GPU, DPO, a colocated GRPO job and a server-mode GRPO job. Each has a tested accelerate preset, a published memory footprint, an expected cost and a reference evaluation. Teams copy and change one number at a time. The TRL documentation's recommended LoRA learning rates, about ten times the full fine-tuning rate (2e-4 for SFT, 5e-6 for DPO, 1e-5 for GRPO), are the starting values.

An onboarding path. Point new users to the earlier levels, then to the Hugging Face guide for the surrounding ecosystem and the Docker guide for the images. Use Terraform for the cluster definitions themselves if you provision GPU capacity as code.

Metrics for the platform itself. Time from request to first training step, percent of jobs that complete without human intervention, GPU utilisation, cost per experiment, and the share of runs using a golden config. When the last number climbs, your paved road is working.

The review checklist

Use this list in design reviews and when approving a new workload. A "no" is not a veto; it is a conversation that needs an owner.

• Versions: Are trl, peft, transformers, accelerate, vllm, torch and deepspeed pinned together, and is the vLLM version inside TRL's window?
• Defaults: Does the config write out every default the run depends on, including vllm_mode, loss_type, max_completion_length and beta?
• Stability: Is every trainer in the stable list? If not, who owns the migration when it is removed?
• Security: Is trust_remote_code false, or pinned to a commit with a sandbox? Are adapters safetensors? Do reward functions and tools run without credentials or egress?
• Network: Is the vLLM server reachable only from trainer pods, with ports 8000 and 51216 restricted?
• Capacity: Are num_generations, completion length and GPU split recorded, and is the cost estimate within budget?
• Targeting: Does the trainable-parameter share match expectation, and have get_layer_status() and get_model_status() been checked?
• Quality: Is there a held-out evaluation, and are reward_std, entropy and rewards/margins charted?
• Artefacts: Does the registry record base revision, PEFT and TRL versions, resolved config and dataset version?
• Serving: Is the merge-or-multi-LoRA decision justified, and was a QLoRA merge done from a bf16 base?
• Privacy: Is telemetry set deliberately, and are completion logs kept off public repositories?
• Recovery: Can the run resume from a checkpoint, and has that been tested?

The complete picture

Put the series together. Beginner showed you a trainer, a model string and an adapter. Mid showed you why the defaults are what they are, how DPO and GRPO differ, and how to spend memory wisely. This level asks what happens when forty people use them.

The answer is mostly discipline around a small number of moving parts. Pin the versions together and write the defaults down. Choose the vLLM topology deliberately and firewall it. Treat every artefact, whether model, dataset, adapter or reward function, as a potential carrier of code or bad behaviour. Watch the run, not just the GPU. Keep adapters small, registered and reproducible. Know where the libraries stop, and put your own thin layer at those edges so that the churn of a pre-1.0 ecosystem lands in one place.

Where the series leaves you

You can now own PEFT and TRL as a platform: design the topology for a workload, predict which component breaks first, set the security and cost policies, write the runbook, and say no to the wrong tool with a reason. The next step is the serving and operations side of the same story. Read the vLLM guide for the generation engine you just operated, the DeepSpeed guide for the sharding underneath large runs, and the MLflow guide for the registry that makes all of this auditable.

Sources

• TRL documentation
• TRL experimental overview
• TRL vLLM integration
• TRL reducing memory usage
• TRL distributing training
• TRL long context training
• TRL usage statistics
• TRL LoRA without regret
• TRL jobs training
• TRL releases
• TRL migration guide
• PEFT documentation
• PEFT troubleshooting
• PEFT checkpoint format
• PEFT model merging
• PEFT mixed adapter types
• PEFT releases
• vLLM GPU installation

Senior interviews for a post-training platform are not trivia contests. The interviewer wants to see whether you can design a system around PEFT and TRL, diagnose it at three in the morning, and argue for a budget without hiding the risks. This file gives you prompts of each kind, with the reasoning a strong answer contains and the gaps that give a weak one away. The facts are checked against TRL 1.14 and PEFT 0.21.

How to use this file

Read each prompt, answer it aloud in two minutes before reading the model answer, and compare. Senior answers have a common structure: they name the constraint first, state the decision, give the reason, and close with the failure mode they are accepting. Candidates who list features lose to candidates who commit to a choice and say what it costs.

Two habits help in every answer. First, ask one clarifying question about scale or data sensitivity before designing anything; it shows you know the answer depends on it. Second, mention version pinning once, early, because it signals that you have operated these libraries through a release cycle and not just read the quick start.

System design: a self-service fine-tuning platform

Prompt: Forty engineers across six teams want to fine-tune open models on company data. Design the platform.

Start with the clarifying questions: what model sizes, what data sensitivity, is online RL in scope, and what GPU budget exists. Assume 7 to 14B models, internal data, and SFT plus DPO now with GRPO later.

A strong answer builds four layers. At the bottom, a scheduler: Kubernetes with a queueing layer, SLURM or Ray, because TRL has no scheduler, operator or control plane and reaches clusters only through accelerate launch or torchrun. Above it, a pinned base image containing trl, peft, transformers, accelerate, torch and deepspeed resolved together, and vllm only in the online image. Above that, a launch wrapper that accepts a small YAML, resolves every default into an explicit config, validates policy and submits the job. At the top, a registry and evaluation gate for the outputs.

Then the details that show seniority. Offline jobs use LoRA or QLoRA on one GPU or DDP and produce adapters of megabytes, so storage is cheap and every experiment can be kept. Secrets arrive through the secret manager, not YAML. trust_remote_code is false and enforced in the wrapper. Telemetry is set deliberately, with HF_HUB_OFFLINE=1 and a pre-populated cache on restricted networks. Each team gets a namespace, a GPU quota and its own object-store prefix, because training has no in-library tenancy.

A weak answer says "we will use Kubeflow and MLflow" without saying what the unit of work is, or forgets that defaults move between versions. The follow-up you should expect is "what breaks first at ten times the load?" The answer is almost always the scheduler policy and GPU fairness, not the library.

System design: a GRPO pipeline for a reasoning model

Prompt: A team wants to train a 7B model with GRPO on math problems with verifiable answers. Design the training topology.

Begin with the cost drivers: num_generations (default 8), max_completion_length (default 512 since TRL 1.10) and the vLLM share of GPUs. Throughput is dominated by generation, so the topology question is where vLLM runs.

For eight GPUs, I would start with colocate mode, which has been the default since TRL 1.0: no extra hardware, no network hop, and vllm_gpu_memory_utilization at 0.3 by default. I would move to server mode if generation dominated step time or the model needed tensor parallelism: four GPUs for vllm serve with --tensor-parallel-size 4, four for the trainer, disjoint CUDA_VISIBLE_DEVICES, NCCL weight transfer via --weight-transfer-config '{"backend": "nccl"}'. I would mention that data parallelism no longer helps dense models in vLLM, which is why the server scales with tensor parallelism.

Then the safeguards. The server uses VLLM_SERVER_DEV_MODE=1, which exposes development and weight-update endpoints without authentication, so it sits on a private network with a policy that admits only trainer pods and with ports 8000 and 51216 closed to everything else. The vLLM version must stay inside 0.20.0 to 0.30.0 for TRL 1.14. The reward function runs an answer checker, and any custom reward code runs sandboxed.

Monitoring follows: reward, reward_std, entropy, clip_ratio/* and completions/clipped_ratio. A near-zero reward_std means every completion in a group scores the same, so no learning signal exists. Finally, say what you would not do: use PPO, which was removed in TRL 1.13, or build on AsyncGRPOTrainer for a production commitment, because it is experimental and may change in any release.

System design: serving many fine-tunes for many tenants

Prompt: Thirty customers each have their own fine-tuned variant of one base model. How do you serve them?

The answer is a shared base with hot-swapped LoRA adapters rather than thirty merged models. Merging with merge_and_unload() gives zero adapter overhead but costs one full model copy per fine-tune, in storage and GPU memory. Thirty copies of a 7B model is a poor trade for low-traffic tenants. vLLM supports multi-LoRA serving, so one base serves all, with each request naming an adapter.

Cover the constraints. vLLM and Diffusers load only LoRA, so a customer trained with another PEFT method needs a conversion, which PEFT 0.19 added as converters. Adapter selection must be authorised at the gateway from the caller's identity, never from a client-supplied name, because an adapter is just weights and the serving layer owns isolation. Adapters must be safetensors, since .bin is a pickle file that can execute code on load. And every adapter's adapter_config.json names its base model, so the registry must record the base revision and the PEFT version; an adapter saved by a newer PEFT can fail to load in an older server with an unexpected-keyword TypeError.

Offer the alternative honestly. For the two or three highest-traffic customers, merging is better: the hot path has no adapter cost and you can quantize or compile the result. A hybrid is the senior answer: merge the heavy tenants, multi-LoRA the long tail, and decide by traffic threshold.

One more point separates strong candidates. Mixed-adapter batches through adapter_names=[...] work for LoRA and RoAd only, so if the design depends on batching several adapters in one forward pass, the method choice is constrained.

System design: governance for adapter promotion

Prompt: How do you stop a bad adapter reaching production?

Describe a pipeline, not a policy document. The run produces an adapter, a resolved config and the trl env output. A fixed held-out evaluation runs automatically and records scores. The registry entry links base model revision, dataset version, PEFT and TRL versions and the run. A promotion gate compares the candidate against the incumbent on the same evaluation, and a human approves exceptions.

Strong answers add that the merge is itself a change that needs evaluating. Merging into a quantized base introduces rounding error, so merge from a bf16 base. Merging only some of several active adapters silently drops the others. And add_weighted_adapter with SVD applied LoRA scaling twice before PEFT 0.21.0, so results merged on older versions are not reproducible. Each of these is a reason to evaluate the artefact that is actually served.

Incident response: nothing is learning

Prompt: A team reports their SFT run has trained for nine hours and the model behaves exactly like the base. What do you do?

Check the cheapest hypotheses first. Was the adapter loaded with get_peft_model rather than PeftModel.from_pretrained? The former creates a new, randomly initialised adapter, so the loaded model behaves like the base. Was the model put in .eval()? For classification tasks, was a new head missing from modules_to_save?

Then the training side. Run model.print_trainable_parameters(), get_model_status() and get_layer_status(). PEFT raises an error only when none of the target_modules match; a partial mismatch is skipped silently, so a typo can leave most layers untouched and the loss barely moving. Check the loss curve and grad_norm, and confirm the learning rate: TRL's default for SFT is 2e-5, which is a full fine-tuning value, and LoRA usually wants about ten times that, around 2e-4. A LoRA run left at 2e-5 learns slowly and looks as if it did nothing.

Finally check what the loss measured. If the data is conversational and assistant_only_loss is in play, a template without {% generation %} markers raises an error, but a completion-only setup that masks everything is subtler. The stronger answer ends with prevention: a launch-time check on trainable-parameter share, and an automatic held-out evaluation, so the nine hours become nine minutes.

Incident response: GRPO stalls or diverges

Prompt: A GRPO run's reward flatlines at step 200 and then entropy collapses. Walk me through it.

Two signals, two hypotheses. A flat reward with reward_std near zero means all completions in each group score alike, so advantages are zero and nothing is learned. Causes: the reward function is saturated or constant, prompts are too easy or too hard, or num_generations is too small to produce variance. Look at the completions table with log_completions=True and at per-function rewards using log_metric.

Collapsing entropy means the policy is becoming deterministic. Causes include a reward that is easy to game, a learning rate too high (the recommended LoRA rate for GRPO is about 1e-5, ten times the full fine-tune 1e-6), or a KL penalty that is absent: beta defaults to 0.0, so no reference model is loaded and nothing anchors the policy. Mitigations: lower the learning rate, set beta above zero at the memory cost of a reference model (or, with PEFT, the base with the adapter disabled), use the entropy regularisation options added in TRL 1.8 such as entropy_coef, and check clip_ratio/* for large policy movement.

Add the platform-level point: a version upgrade can change these dynamics. GRPO's use_bias_correction_kl flipped to true and max_completion_length doubled in TRL 1.10, and loss values changed in 1.9 when num_items_in_batch began counting generated tokens. Always ask what changed in the stack before blaming the data.

Incident response: the server-mode run never starts

Prompt: Server-mode GRPO fails with "No completions were generated". Triage it.

The message hides a backend failure, so go to the server, not the trainer. Read the vLLM logs for the underlying exception. Confirm the installed vLLM is inside TRL 1.14's window of 0.20.0 to 0.30.0; TRL 1.14.1 fixed crashes on vLLM 0.20 to 0.25, which is itself a hint that version mismatch is a common cause. Confirm trainer and server have disjoint CUDA_VISIBLE_DEVICES, that VLLM_SERVER_DEV_MODE=1 is set, that --weight-transfer-config is NCCL or IPC as appropriate, and that --logprobs-mode processed_logprobs is present for importance sampling. Check that the trainer can reach the host and port within the 240-second timeout.

If someone launched with trl vllm-serve, note that it is a deprecated wrapper since TRL 1.11 and the direct vllm serve command is the supported form. Close with the prevention step: a startup smoke test that requests one completion and fails the job early.

Incident response: an import error after the upgrade

Prompt: Monday's upgrade broke twelve pipelines with ImportError: cannot import name 'PPOTrainer'. What now, and what do you change?

Roll back first: re-deploy the previous pin set, because the platform's contract is that upgrades are reversible. Then explain, calmly, that this is not a regression: PPO, PPOConfig and the value-head wrappers were removed in TRL 1.13 after living in trl.experimental, and from trl import PPOTrainer had stopped working earlier, in 1.10. The migration is to GRPO or RLOO for online methods, which need no value model.

Then the process fix, which is what the interviewer is really testing. The upgrade should have run through a gate: reference SFT, DPO and GRPO jobs on the new pin set, a grep for removed APIs across repositories, and a read of the release's breaking changes. Experimental APIs carry an explicit promise that they may be removed in any release without deprecation, so nothing experimental should be in a supported template. Also mention the quieter removals in 1.14: six experimental trainers and the trl.losses module.

Trade-offs: how to argue them to leadership

Leadership questions test whether you can translate technical risk into decisions. Use the same shape each time: the options, the cost of each in money and risk, your recommendation, and what would change your mind.

Prompt: Why should we run our own training rather than use a managed service?

Be fair to the managed option. Hugging Face Jobs lets you run a TRL script on a named hardware flavour, bills until the script exits, and is a fast way to start. The case for running your own rests on three things: data residency, since regulations in Gulf and Egyptian jurisdictions may restrict where training data and checkpoints live; utilisation economics at steady load; and control over the stack. The case against is the operating burden of a pre-1.0 ecosystem. A senior answer proposes a hybrid: managed capacity for bursts and experiments with non-sensitive data, internal capacity for sensitive workloads, one wrapper for both so teams do not care.

Prompt: Why should we pay for an engineer to pin and test versions? Just upgrade.

Show the cost of not doing it. TRL ships about every two weeks. In a year the default for vllm_mode, SFT's loss_type and GRPO's completion length each changed, with no code change required from users. Run-to-run comparisons stop being valid, and cost can double silently. The pin-and-test discipline costs a fraction of one engineer and protects every team's results. Offer a cadence, such as upgrading monthly through a reference-job gate, so leadership sees a plan rather than a veto.

Prompt: Should we standardise on LoRA?

Recommend LoRA as the default and name the exceptions. LoRA cuts memory and makes adapters megabytes, which transforms storage and multi-tenant serving. TRL's "LoRA Without Regret" guidance supports it with conditions: apply it to all linear layers with target_modules="all-linear", use rank 256 for SFT at post-training scale and 1 to 32 for reinforcement learning, use about ten times the full fine-tune learning rate, and keep the effective batch size under 32. Say plainly that attention-only LoRA underperforms. The exception is a workload where capacity limits show up in evaluation, in which case full fine-tuning is the fallback and the cost conversation changes.

Prompt: Can we adopt the experimental trainer that a paper just recommended?

Say yes for research, no for the supported platform. The official stability contract allows removal in any release including patches. Offer a path: run it in a research namespace, wrap it in a thin interface, and promote it if it is moved to the stable surface, as KTO was in 1.8 and distillation in 1.10.

Common weak answers and what replaces them

Interviewers hear the same failures repeatedly. Knowing them lets you avoid them.

A weak answer says "we will use QLoRA for everything to save memory." The stronger answer notes that QLoRA saves memory at the cost of speed and merge fidelity, and that merging into a quantized base adds rounding error, so adapters are merged from a bf16 base.

A weak answer says "turn on trust_remote_code if the model needs it." The stronger answer says it executes arbitrary Python from a Hub repository on a node that holds credentials, so it is off by default, pinned to a commit SHA if enabled, and sandboxed.

A weak answer treats the vLLM server as an internal detail. The stronger answer points out the unauthenticated dev endpoints and the ports to restrict.

A weak answer describes PPO with a value head. The stronger answer says PPO was removed from TRL in 1.13 and chooses GRPO or RLOO.

A weak answer relies on documentation memorised from a blog post, with tokenizer= and max_seq_length= arguments on SFTTrainer. The stronger answer uses processing_class= and max_length, and says to check the release notes because both libraries changed a lot between 2024 and 2026.

Practice drills

Try these without notes. Each should take under three minutes.

1. State the stability promise for trl.experimental and name two trainers that moved from experimental to stable.
2. List four things you must record so an adapter can be rebuilt.
3. Explain, in two sentences, why LoRA's output is W·x + (lora_alpha/r)·B·A·x and why B starts at zero, and what use_rslora=True changes.
4. Give the three cost drivers of a GRPO job and the default value of each.
5. Name three ways a trained adapter can behave like the base model.
6. Describe how you would verify a tenant cannot read another tenant's adapter.
7. Say which parts of the online path are Linux only and why.

For the content behind these answers, return to the vLLM guide for the generation side, the DeepSpeed guide for sharding, and the MLflow guide for the registry and run tracking.

These are the traps that only show up when many people run many jobs, and the habits that keep a PEFT and TRL platform boring. Each item states the symptom, the cause and the fix, checked against TRL 1.14 and PEFT 0.21. Boring is the goal: a platform that nobody has to think about is the one that is working, and every habit below exists to move surprises from production into a reference run you control.

Production traps that only appear at scale

The default that moved under you. Symptom: a recurring job gets slower, or its loss curve shifts, after an unrelated rebuild. Cause: a default changed between releases. GRPO's max_completion_length went from 256 to 512 in TRL 1.10, use_bias_correction_kl became true in the same release, vllm_mode flipped to "colocate" in 1.0, and SFT's loss_type resolved to chunked_nll from 1.7. Fix: write every default you depend on into the config, and diff resolved configs when you upgrade. A job whose config says nothing about these values is a job whose behaviour belongs to the next release.

The step-zero failure on a big allocation. Symptom: a four-node job dies immediately with a ValueError about batch divisibility. Cause: GRPO requires generation_batch_size to be divisible by num_generations, and the global batch (per-device batch times processes times steps_per_generation) changes whenever someone changes node count. Fix: validate it in the launch wrapper before the scheduler sees the job, and publish the formula. generation_batch_size and steps_per_generation are mutually exclusive, so set one, not both.

The adapter that silently trains half the model. Symptom: a loss that falls slowly and an evaluation that barely improves. Cause: PEFT raises an error only if none of your target_modules match; a partial match is accepted quietly. This is common after an architecture change when layer names differ. Fix: log print_trainable_parameters() at start, compare the trainable share to an expected band, and check get_layer_status() for irregular entries.

The lm_head conflict. Symptom: ValueError: loss_type='chunked_nll' is not supported when lm_head is wrapped by a PEFT adapter. Cause: target_modules="all-linear" normally excludes the output layer for PreTrainedModels, but an explicit list that includes lm_head wraps it, and the default SFT loss cannot run chunked over a wrapped head. Fix: remove lm_head from the targets, or set loss_type="nll" knowingly, accepting the higher memory. Liger has the same limitation, and GRPO refuses an adapter on lm_head outright.

The vLLM version window. Symptom: weight sync or generation crashes after someone upgraded vLLM for another team's serving reasons. Cause: TRL 1.14 supports vLLM 0.20.0 to 0.30.0 only, and other versions fail in ways that look like network issues. Fix: one constraint file shared by trainer and generator images, and a startup smoke test that requests a single completion.

The shared-GPU colocate squeeze. Symptom: a colocated GRPO job that ran fine on a 80 GB card fails on 40 GB. Cause: vLLM reserves vllm_gpu_memory_utilization, 0.3 by default, leaving the trainer the remainder. Fix: reduce the fraction, enable vllm_enable_sleep_mode, reduce num_generations, or move to server mode on dedicated GPUs. Do not use auto_find_batch_size with GRPO; it raises an error.

The unauthenticated dev server. Symptom: none, until a security review. Cause: VLLM_SERVER_DEV_MODE=1 exposes development and weight-update endpoints with no authentication. Fix: network policy that admits only trainer pods; ports 8000 and 51216 closed elsewhere; never an ingress.

The silent telemetry ping. Symptom: an air-gapped job logs connection timeouts at startup, or a privacy review asks what leaves the cluster. Cause: since TRL 1.5, each trainer instantiation sends one anonymous ping. Fix: set HF_HUB_DISABLE_TELEMETRY=1 or HF_HUB_OFFLINE=1 in the base image, and pre-populate the cache for offline clusters. Nothing is sent when the CI variable is set, so do not rely on that for production.

Traps in artefacts and serving

The wrong loader. Symptom: a deployed adapter behaves like the base model. Cause: get_peft_model creates a new, randomly initialised adapter; trained adapters load with PeftModel.from_pretrained or AutoPeftModelForCausalLM.from_pretrained. Fix: put the loader in your serving template and add a smoke test that compares outputs with the base.

The newer-adapter-older-server mismatch. Symptom: TypeError: LoraConfig.__init__() got an unexpected keyword argument. Cause: training ran a newer PEFT than serving. Fix: keep training and serving PEFT versions in the same pin file; deleting the key from adapter_config.json is a last resort that is safe only if it held the default.

The quantized merge. Symptom: a merged model is subtly worse than the adapter it came from. Cause: merging into a quantized base introduces rounding error. Fix: load the base in bf16 and merge there, then quantize the result if serving needs it.

The partial merge. Symptom: behaviour from one of several adapters vanishes after merging. Cause: merging only some active adapters silently drops the others, which the PEFT docs say is planned to change in v1.0. Fix: merge all with merge_adapter() and no names, or unmerge first.

The second adapter that does not train. Symptom: loss unchanged after adding an adapter. Cause: an inactive adapter has requires_grad=False. Fix: call model.base_model.set_adapter(["default", "other"]) before creating the optimizer or trainer.

The pickle adapter. Symptom: none, until it executes. Cause: adapter_model.bin is a pickle. Fix: reject .bin at the registry and scan what is already stored.

The old merged result. Symptom: a merged adapter built a year ago cannot be reproduced. Cause: add_weighted_adapter with the SVD path applied LoRA scaling twice before PEFT 0.21.0. Fix: record the PEFT version with every merge, and re-evaluate merges built on older versions before you rebuild them.

The OFT retrain. Symptom: an OFT adapter from before PEFT 0.18 loads but behaves differently. Cause: OFT numerics changed in 0.18, so adapters need retraining. Fix: retrain, and keep method and version in the registry so you can find affected adapters.

Platform ownership habits

Own a pin set, not a package list. One versioned manifest names trl, peft, transformers, accelerate, vllm, torch, deepspeed and bitsandbytes together, with a reference run proving that set works. Teams adopt the manifest, not individual versions. TRL's dependency floors rise often, for instance peft>=0.13.0 and deepspeed>=0.18.6 in 1.13, and transformers v5 needs PEFT 0.18.0 or later, so independent upgrades fail in combination.

Make the launcher the policy. Every rule that matters, such as trust_remote_code=False, tracker required, rank limits, num_generations ceiling, telemetry variables and secret injection, lives in the wrapper. Policy that lives only in a wiki gets ignored by the first deadline.

Record the resolved config. Save the config after defaults are applied, beside the adapter, together with trl env output. It is the only record that survives a default change.

Keep an upgrade cadence. Pick a rhythm, perhaps monthly, and run reference SFT, DPO and GRPO jobs through the new pin set before anyone else sees it. TRL's roughly fortnightly releases make "upgrade when needed" a recipe for emergency migrations. Read each release's breaking section and MIGRATION.md.

Grep for removed APIs. Maintain a list of names that no longer exist: PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead, trl.losses, BCOTrainer, XPOTrainer, NashMDTrainer, trl vllm-serve. Run it in CI across every team's repository. It finds migrations before an import error does.

grep -rnE "PPOTrainer|PPOConfig|AutoModelForCausalLMWithValueHead|trl\.losses|BCOTrainer|XPOTrainer|NashMDTrainer|vllm-serve|use_transformers_paged|torch_dtype" \
  --include='*.py' --include='*.yaml' --include='*.sh' .

Watch deprecation warnings. Several things warn today and will be removed in TRL v2.0.0: pad_token on SFT and DPO configs, truncation_mode="keep_end", packing_strategy="bfd-requeue" (now "bfd_split"), use_transformers_paged, vllm_importance_sampling_cap, and the old experimental import paths for KTO and distillation. Treat warnings in job logs as backlog, and have CI fail on FutureWarning from trl in your reference jobs.

Publish a support matrix. Methods against parallelism strategies, and operating systems against features. Examples that belong in it: GRPO does not support context or sequence parallelism; packing is incompatible with context parallelism; vLLM is Linux only; DPO padding-free was temporarily unavailable as of 1.13; and macOS gets CPU-only bitsandbytes.

Keep a golden-config library. SFT with LoRA, QLoRA on one GPU, DPO, GRPO colocated and GRPO with a server. Each has a tested accelerate preset, a measured memory footprint and a reference evaluation. When a team deviates, they change one value at a time.

Instrument the platform, not only the jobs. Track time to first training step, the share of jobs that finish unattended, cost per experiment and the percentage using golden configs. They tell you whether the road is paved.

Pull request review checklist

Use this for changes to training code, configs or the launcher. Each line should be answerable from the diff.

• Does the PR change a version pin? If so, are all coupled packages updated together, and is there a reference run attached?
• Does the config set explicitly every default the job depends on (vllm_mode, loss_type, max_completion_length, beta, learning_rate)?
• Is learning_rate appropriate for the method? With LoRA, TRL's documented starting points are 2e-4 for SFT, 5e-6 for DPO and 1e-5 for GRPO, roughly ten times the full fine-tuning values.
• Are target_modules and rank justified? Is lm_head absent from them?
• Does the PR use processing_class= and max_length, not the removed tokenizer= and max_seq_length= forms?
• Does it use dtype, not the deprecated torch_dtype?
• Is quantization_config passed in exactly one place, as a trainer argument or in model_init_kwargs, never both?
• Is trust_remote_code false? If true, is model_revision a commit SHA and is the job sandboxed?
• Do reward functions return one float per completion, and do they avoid shelling out or network calls with model text?
• Do secrets come from the environment or secret manager, and does the YAML contain none?
• Does the change write a resolved config and trl env output with the run?
• Does anything import from trl.experimental? If so, who owns it and is it outside the supported templates?

Design document checklist

Use this for a new workload or a change of topology. The document should answer these before anyone requests GPUs.

• Goal and method. Which trainer, and why that one instead of the neighbouring one? If online, why GRPO or RLOO rather than a simpler offline method?
• Data. Source, size, schema and dataset type matching the trainer, and whether it contains personal data that falls under a residency rule.
• Capacity. For online jobs: num_generations, completion length, vLLM mode and GPU split, with an estimate of GPU hours.
• Topology. Colocate or server, and the reason. If server: network policy, ports, version window and failure behaviour.
• Scaling plan. What happens at ten times the data: which component breaks first, and the accelerate preset chosen.
• Security. Remote code, reward-code sandboxing, secrets, telemetry, adapter format and supply chain.
• Artefacts. Where adapters live, what metadata they carry, how they are promoted, and how they are served: merged or multi-LoRA.
• Evaluation. The held-out set, the metrics, the regression threshold and who owns it.
• Observability. Dashboards and alerts for reward_std, entropy, rewards/margins and infrastructure metrics.
• Rollback and upgrade. How the pin set changes and how a bad adapter is reverted.
• Exit. What would make the team leave PEFT and TRL, and what the migration cost would be.

Operational recipes

Reproduce an environment report. Start every investigation with the environment report. It prints platform, Python, PyTorch, accelerators, transformers, accelerate, datasets, TRL, bitsandbytes, DeepSpeed, Liger, PEFT and vLLM versions, and the TRL docs say it is required when reporting an issue.

trl env
python -c "import trl, peft, transformers, torch; print(trl.__version__, peft.__version__, transformers.__version__, torch.__version__)"

Launch with a bundled accelerate preset.

trl sft --config sft_config.yaml --num_processes 4 --accelerate_config zero2

Resume a failed run.

trainer.train(resume_from_checkpoint=True)   # or pass an explicit checkpoint path

Continue training an existing adapter. Load with AutoPeftModelForCausalLM.from_pretrained(path, is_trainable=True) and pass that model to the trainer without peft_config. Passing a PeftModel together with a peft_config raises an error that tells you to merge and unload first.

Silence the experimental notice deliberately. Setting TRL_EXPERIMENTAL_SILENCE=1 removes the runtime notice for trl.experimental APIs. Do it only in a research environment, since the notice is the reminder that the API may vanish.

Serve through the supported command.

CUDA_VISIBLE_DEVICES=0,1,2,3 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen2.5-7B --tensor-parallel-size 4 \
  --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-logprobs -1

Run overflow on hosted compute. Hugging Face Jobs accepts a script URL, a hardware flavour, a timeout and secrets, and bills until the script exits.

hf jobs uv run --flavor a10g-small --timeout 30m --secrets HF_TOKEN -- \
  https://raw.githubusercontent.com/huggingface/trl/refs/heads/main/trl/scripts/sft.py \
  --model_name_or_path Qwen/Qwen2-0.5B-Instruct --dataset_name trl-lib/Capybara --output_dir out --push_to_hub

Check adapter health after load. Run get_model_status() and get_layer_status(), and look for entries marked irregular. Do this in the serving smoke test as well as in training.

Practice cards

Card 1: the upgrade drill. Take a copy of your reference GRPO config. Upgrade TRL by one minor version in a sandbox. Dump the resolved configs before and after, and list every difference. Decide, for each, whether to pin it or accept it.

Card 2: the tenant test. Create two adapters under two service accounts. Attempt to read the other's artefact path, and to select the other's adapter name through the serving gateway. Both attempts must fail. Fix the gateway until they do.

Card 3: the blind run. Start a GRPO job with a constant reward function. Observe reward_std. Write the alert that would have caught it in the first fifty steps.

Card 4: the partial target. Define a target_modules list with one deliberate typo. Observe that no error is raised, then catch it with print_trainable_parameters() and get_layer_status().

Card 5: the merge audit. Merge one adapter from a bf16 base and from a quantized base. Evaluate both against the unmerged adapter. Record the difference as the number you quote when someone asks whether merging is free.

Card 6: the removed API sweep. Run the grep above across every repository you own and open one issue per team with the hits.

Habits that pay off, in order of leverage

1. Pin the whole set, write the defaults down, and test upgrades on reference jobs. This prevents more incidents than everything else on this page combined.
2. Put policy in the launcher. Every rule you cannot enforce in code will be broken, politely, by someone under pressure.
3. Record resolved configs and environment reports with every run, so any result can be reproduced or explained.
4. Watch the run's health signals, not only utilisation. reward_std, entropy and rewards/margins find problems that GPU graphs cannot.
5. Treat artefacts as untrusted: safetensors only, sandboxed reward code, remote code off, the dev server on a private network.
6. Keep adapters small, registered and linked to their base revision, and decide merge versus multi-LoRA by traffic.
7. Publish the support matrix and the runbook, and update both after every incident.

For neighbouring topics, see the vLLM guide for the generation engine, the DeepSpeed guide for ZeRO sharding, and the MLflow guide for tracking and a registry.