The Complete DSPy Guide

This is part one of three. It covers everything you need to do real work with DSPy, not a teaser. By the end you can install DSPy, point it at a model, declare a task as a Python signature, run it, read the exact prompt DSPy built for you, score the results against a small dataset, and let an optimizer improve the prompt automatically. Mid-level and Senior take the same topics further; nothing here is wasted.

Everything below is written against DSPy 3.4.0 (released 25 September 2026). DSPy moves fast and a lot of blog posts you will find are written against versions that removed the APIs they teach, so where something changed recently this guide says so and shows the current form.

Each section ends with a Try it task. Do them as you go. They take a few minutes, and these ideas only stick once you have watched your own program run, print a wrong answer, and then print a better one after you changed a single line.

What DSPy is, and the problem it solves

DSPy — the name expands to "Declarative Self-improving Python" — is a Python library for programming language models instead of prompting them. You describe a task by declaring its inputs and outputs in Python. DSPy writes the prompt, sends it to the model, parses the reply back into typed Python values, and gives you an object whose attributes are the fields you asked for. Then, if you give it a handful of examples and a function that scores an answer, DSPy's optimizers will rewrite the instructions and pick few-shot examples for you, measured against that score.

To see why that matters, look at what it replaces. The ordinary way to build a feature on a language model is to write a long string. You ask for JSON, you add "respond only with valid JSON", you add an example, you add "do not include markdown fences", you add a note about the edge case a tester found on Tuesday. Then you call the model, get a string back, and write parsing code that handles the three shapes the model actually produces. The prompt becomes a 600-word artefact that nobody dares edit, because nobody can tell which sentence is load-bearing.

The deeper problem is not that the string is ugly. It is that the string is tangled up with everything else. Three separate decisions live inside it: what the task is (classify this ticket into one of five categories), how to attack it (think step by step first, or just answer), and how to talk to this particular model (this one needs the JSON schema repeated, that one does not). Because the three are glued together, you cannot change one without risking the others. Swap gpt-4o-mini for a local Llama and your careful formatting instructions are suddenly wrong. Decide to add chain-of-thought reasoning and you have to rewrite the output contract by hand.

DSPy separates those three decisions into three different objects, and that separation is the whole value proposition.

Because the pieces are separate, each one can be changed on its own. You keep the signature and switch ChainOfThought for Predict to make the program cheaper. You keep both and switch the LM from OpenAI to a model running on your own machine. And — the part people come to DSPy for — you keep all of it and hand the program to an optimizer, which searches over instructions and examples and gives you back a measurably better program without you editing a single word of prompt text.

One thing to be clear about before you start: DSPy has no command line tool of its own. There is no dspy run. Everything is the Python API, which is why this guide is full of Python rather than shell. The only shell commands you need are for installing things and setting a key.

Prompting versus programming: the shift in mindset

It is worth slowing down on the mental shift, because people who skip it write DSPy that looks like prompt engineering with extra imports, and then wonder why the optimizers do nothing for them.

When you prompt, you are the optimizer. You read an output, decide it is bad, guess which words caused it, edit them, and re-read. Your loop is intuition-driven, your sample size is however many examples you looked at this morning, and your record of what you tried is in your head. It works surprisingly well for the first prototype and stops working the moment the task gets specific enough that two sensible people disagree about what a good answer looks like.

When you program with DSPy, you write down the task once and move your effort to two other places: the data (twenty or fifty examples of input and the answer you want) and the metric (a Python function that looks at an example and a prediction and returns a number). Those two artefacts are the real deliverable. Given them, the library can do the editing loop for you, thousands of times, with a sample size you choose.

This is a genuine trade. You give up the illusion of control over the exact words and you take on the obligation to define success numerically. Beginners resist the second half, and it is the half that pays. If you cannot write a metric, you do not yet know what you are asking the model for — which means you also cannot tell whether yesterday's prompt edit helped.

The practical consequence for your first week: resist writing long instructions. Write a short, honest signature, run it, and look at what DSPy produced. Nine times out of ten the output is already reasonable, and the interesting work is building the twenty examples that will tell you whether it is right.

The four nouns you need: signature, module, adapter, prediction

DSPy has a large API surface but a small vocabulary. Four nouns carry most of the weight, and once you can say what each one is, the documentation stops being confusing.

A signature is a declaration of a task's inputs and outputs: their names, their types, and optionally a sentence describing each. It is not a prompt. The shortest form is a string, "question -> answer", where the names before the arrow are inputs and the names after it are outputs. You can add types: "context: list[str], question: str -> answer: str, confidence: float". For anything beyond a line or two you write a class instead, which gives you a place to put the instructions (the class docstring) and per-field descriptions.

A module is a strategy for satisfying a signature. dspy.Predict is the simplest: ask the model, parse the answer. dspy.ChainOfThought adds a reasoning output field before your declared fields, so the model works through the problem before committing to an answer. dspy.ReAct gives the model tools it can call in a loop. All of them take the same signature, which is the point — you change the strategy without rewriting the task. Modules compose: you subclass dspy.Module, build submodules in __init__, and write the logic in forward().

An adapter sits between the signature and the model and does the mechanical work: turn fields into messages the model will understand, then turn the reply back into typed values. The default is ChatAdapter, which uses [[ ## field_name ## ]] markers in the prompt and, if parsing fails, falls back to JSONAdapter. There are also XMLAdapter, TwoStepAdapter and BAMLAdapter. You will rarely touch adapters as a beginner; knowing they exist explains where the prompt's formatting instructions come from.

A prediction is what comes back: a dspy.Prediction whose attributes are your output fields. For "question -> answer" you read pred.answer. With ChainOfThought you also get pred.reasoning.

Beneath those four sits the LM: dspy.LM("openai/gpt-4o-mini"), a configured client holding the model name, sampling parameters, retry count and a cache flag. Model strings follow LiteLLM's naming convention — openai/..., anthropic/..., gemini/..., vertex_ai/..., ollama_chat/..., azure/<deployment> — which is handy, because it means the same string works in LiteLLM itself.

Three more words appear constantly and are worth learning now. A predictor is a dspy.Predict instance — the leaf that actually calls the model, and the thing that holds the learnable state (instructions plus few-shot examples). An example is a dspy.Example, one data point. Demos are the few-shot examples attached to a predictor; optimizers mostly work by choosing good ones.

Installing DSPy and checking your setup

DSPy is a pure-Python wheel, so installation is identical on Linux, macOS and Windows, and there is nothing to compile. It supports Python 3.10 through 3.14. Create a virtual environment first — DSPy pulls in a reasonably large dependency tree (openai, litellm, pydantic, diskcache, tenacity and others) and you do not want those in your system Python.

python -m venv .venv
source .venv/bin/activate          # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install -U dspy

If you prefer uv, uv add dspy does the same thing. Pin the version in anything you will come back to, with pip install "dspy==3.4.0", because DSPy has shipped breaking changes in minor releases and a floating dependency means your program can stop working after an unrelated pip install -U.

Several features live behind optional extras, because the maintainers moved heavyweight dependencies out of the base install. If you hit a ModuleNotFoundError for numpy or optuna, this is why — nothing is broken, you just need the extra. Quote the brackets; zsh and PowerShell both treat bare square brackets as pattern syntax.

pip install "dspy[numpy]"      # embeddings, KNN / KNNFewShot, SIMBA
pip install "dspy[optuna]"     # MIPROv2, BootstrapFewShotWithOptuna
pip install "dspy[deno]"       # managed Deno runtime for the sandboxed interpreter
pip install "dspy[mcp]"        # Model Context Protocol tools

Next, credentials. DSPy reads the standard provider environment variables, so you do not pass keys in code: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, or for Azure the trio AZURE_API_KEY, AZURE_API_BASE and AZURE_API_VERSION. You can also pass api_key= to dspy.LM(...) if you are loading it from a secret manager.

export OPENAI_API_KEY=sk-...        # Windows PowerShell: $env:OPENAI_API_KEY="sk-..."

Now verify, in two stages. First that the library imports and is the version you think it is:

python -c "import dspy; print(dspy.__version__)"     # 3.4.0

Then that a real call works end to end. Save this as check.py and run it:

import dspy

lm = dspy.LM("openai/gpt-4o-mini")      # reads OPENAI_API_KEY
dspy.configure(lm=lm)

print(lm("Say this is a test!"))        # ['This is a test!']
print(dspy.Predict("question -> answer")(question="2+2?").answer)

Two details in that output are worth noticing now. Calling the LM directly with a plain string returns a list of strings, not a string — one entry per sample — so print(lm("...")) shows brackets. And the Predict call returns a prediction, which is why you read .answer off it.

If you have no API key, you can still follow every example locally with Ollama. Run ollama run llama3.2 once to pull the model, then point DSPy at it. Expect weaker answers than a frontier model gives, which actually makes the evaluation sections of this guide more interesting, not less.

lm = dspy.LM("ollama_chat/llama3.2", api_base="http://localhost:11434", api_key="")
dspy.configure(lm=lm)

Your first real program, step by step

Let us build something small but complete: a classifier for incoming support messages. The task is to read a message and return a category, an urgency flag, and a one-line reason. We will build it in four passes, each of which teaches one thing.

Pass one: the shortest thing that works. An inline signature with types.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

triage = dspy.Predict("message -> category: str, urgent: bool, reason: str")

pred = triage(message="My card was charged twice for the same order this morning.")
print(pred.category, pred.urgent, pred.reason)

That already does more than it looks. urgent comes back as a genuine Python bool, because the adapter told the model what shape to produce and parsed it on the way back. You wrote no formatting instruction and no parsing code.

Pass two: constrain the category. A free-form str category is useless for anything downstream, because the model will invent synonyms. Switch to a class-based signature and a Literal, which pins the allowed values.

from typing import Literal
import dspy

class Triage(dspy.Signature):
    """Classify an incoming customer support message for routing."""

    message: str = dspy.InputField(desc="The raw message as the customer wrote it")
    category: Literal["billing", "technical", "account", "shipping", "other"] = dspy.OutputField()
    urgent: bool = dspy.OutputField(desc="True only if the customer is blocked right now")
    reason: str = dspy.OutputField(desc="One short sentence justifying the category")

triage = dspy.Predict(Triage)
pred = triage(message="My card was charged twice for the same order this morning.")
print(pred.category, pred.urgent, pred.reason)

Three things changed. The docstring became the task instructions. The Literal became an enumerated constraint the adapter enforces in the prompt and the parser. And each desc= became a per-field hint — note that urgent has a definition attached, which is exactly where a human reviewer's disagreement would otherwise live.

Pass three: change the strategy without touching the task. Suppose urgency is being judged badly. Give the model room to think first:

triage = dspy.ChainOfThought(Triage)
pred = triage(message="My card was charged twice for the same order this morning.")
print(pred.reasoning)      # the model's working, added as an extra output field
print(pred.category, pred.urgent)

The signature did not move. ChainOfThought prepends a reasoning output field, so the model produces its working before the fields you declared, and you get it back on the prediction. This costs more tokens and takes longer. Whether it is worth it is an empirical question, which is the subject of two sections' time.

Pass four: wrap it in a module. Real programs do more than one model call. A module is where that composition lives: submodules in __init__, logic in forward().

class SupportTriage(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.ChainOfThought(Triage)
        self.reply = dspy.Predict("message, category -> draft_reply")

    def forward(self, message):
        t = self.classify(message=message)
        draft = self.reply(message=message, category=t.category)
        return dspy.Prediction(
            category=t.category, urgent=t.urgent,
            reason=t.reason, draft_reply=draft.draft_reply,
        )

program = SupportTriage()
out = program(message="I still have not received order 4471, it was due Sunday.")
print(out.category, out.urgent)
print(out.draft_reply)

Notice what the module gave us: a single object that takes a message and returns everything the rest of the application needs, with two tunable predictors inside it. When we optimize later, both predictors get tuned, and we will not have to think about which.

Signatures in depth

Signatures are where beginners get the most value from a little extra care, so here is the fuller picture.

Inline signatures are strings of the form "inputs -> outputs". Names are comma-separated and may carry types after a colon. A name with no type is treated as a string. The names matter: they are shown to the model, so "passage, question -> answer" communicates more than "a, b -> c". Use inline signatures for anything you can describe in one line, and for throwaway experiments.

Class-based signatures earn their extra lines as soon as you want instructions or per-field descriptions. Subclass dspy.Signature, put the task instruction in the docstring, and declare fields with dspy.InputField() and dspy.OutputField(). The docstring is real: it is the instruction block in the prompt, and it is the thing optimizers rewrite.

Output types can be plain Python types, Literal[...] for an enumeration, a list such as list[str], or a Pydantic model when you need nested structure. DSPy also ships its own field types — dspy.Image, dspy.History for conversation turns, dspy.Tool, dspy.Code — which behave as fields the adapter knows how to render.

from pydantic import BaseModel
import dspy

class LineItem(BaseModel):
    description: str
    amount: float

class ParseInvoice(dspy.Signature):
    """Extract the supplier and all line items from an invoice's plain text."""

    invoice_text: str = dspy.InputField()
    supplier: str = dspy.OutputField()
    items: list[LineItem] = dspy.OutputField(desc="One entry per billed line")

parse = dspy.Predict(ParseInvoice)
pred = parse(invoice_text=open("invoice.txt").read())
for item in pred.items:
    print(item.description, item.amount)      # real LineItem objects

pred.items is a list of validated LineItem instances. Pydantic 2.11 or newer is required in DSPy 3.4, and this is where you see the reason: the models in your signature are validated on the way out of the adapter, so a reply with a non-numeric amount fails loudly at the parse step instead of quietly three functions later.

Two habits make signatures work well. First, name the output you actually want, not a container for it — declare supplier: str rather than result: dict, because the field name is a hint to the model and a dict tells it nothing. Second, put the disputable definitions in desc=. Any rule a human reviewer would have to ask about ("does a refund request count as billing?") belongs in a description, where the optimizer can also see and refine it.

The modules you will actually use

There are more modules than you need. Here is the short list, in the order you are likely to reach for them, with the reason each exists.

dspy.Predict is one call to the model: format the inputs, read back the output fields. Start here always. It is the cheapest, the fastest, and the easiest to debug, and for extraction and classification it is often as good as anything else.

dspy.ChainOfThought is Predict plus an automatically added reasoning output that the model fills in before your fields. Reach for it when the task involves a decision with steps — arithmetic, multi-condition rules, anything where you would want to see the working. The cost is extra output tokens on every call, and for a short classification that can double your bill for no measurable gain. Measure it rather than assuming.

dspy.ReAct gives the model tools and lets it loop: think, call a tool, read the result, think again, until it has an answer. A tool is any Python callable wrapped in dspy.Tool(func), which introspects the name, docstring and argument types to tell the model how to call it. The default iteration cap is max_iters=20.

import dspy

def get_order_status(order_id: str) -> str:
    """Look up the delivery status of an order by its ID."""
    return {"4471": "in transit, due Thursday"}.get(order_id, "no such order")

agent = dspy.ReAct("question -> answer", tools=[dspy.Tool(get_order_status)], max_iters=6)
print(agent(question="Where is order 4471?").answer)

Write the tool's docstring for the model, not for your colleagues: it is the only description the model gets. A tool called lookup whose docstring says "looks things up" will be called at the wrong times, and no amount of optimization fixes that.

dspy.BestOfN and dspy.Refine buy quality with repeated calls. Both take a module, a number N, a reward_fn and a threshold. BestOfN runs the module up to N times and keeps the highest-scoring result; Refine feeds the previous attempt back for improvement. They are the supported replacement for the old dspy.Assert and dspy.Suggest API, which is deprecated and no longer supported — if a tutorial shows you dspy.Assert, it predates this change.

dspy.Parallel and the .batch() method on any module run many inputs concurrently, which is how you evaluate a dev set in seconds instead of minutes.

# Run 200 inputs with 8 worker threads
examples = [dspy.Example(message=m).with_inputs("message") for m in messages]
results = program.batch(examples, num_threads=8)

Seeing what DSPy actually sent

This is the section that converts scepticism into confidence, and it is one line of code. DSPy builds prompts for you, which is wonderful until something goes wrong and you have no idea what the model was asked. So look.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
dspy.Predict("message -> category, urgent: bool")(message="My card was charged twice.")

dspy.inspect_history(n=1)

dspy.inspect_history(n) prints the last n LM interactions: the system message DSPy composed, the field markers it used, any few-shot demos attached to the predictor, your input, and the raw reply. Read it once in full. You will see where your docstring went, where each desc= went, and how the output fields were requested. After that, the library stops feeling magical and starts feeling like code.

For programmatic access, every LM keeps its own history:

lm = dspy.LM("openai/gpt-4o-mini")
dspy.configure(lm=lm)
dspy.Predict("question -> answer")(question="2+2?")

entry = lm.history[-1]
print(entry.keys())        # prompt, messages, kwargs, response, outputs, usage, cost, timestamp, model, ...
print(entry["usage"], entry["cost"])

usage gives you token counts and cost an estimate. Be careful with cost: DSPy reports unknown cost as unknown rather than as zero, so do not sum it blindly across a run. A cached call also reports empty usage, which is correct — nothing was spent — but surprising the first time you see a zero.

For tidier accounting, turn on usage tracking and read it off the prediction:

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"), track_usage=True)
pred = dspy.ChainOfThought("question -> answer")(question="Why is the sky blue?")
print(pred.get_lm_usage())

When you want this permanently, rather than at a debugging prompt, DSPy integrates with MLflow tracing via mlflow.dspy.autolog(), which records every module and LM call as a trace you can browse later. Third-party platforms including Langfuse and Arize Phoenix also have DSPy integrations; MLflow is the one the official DSPy docs cover.

Data, metrics and evaluation

Now the part that makes DSPy more than a prompt wrapper. To improve a program you need a dataset and a score.

A data point is a dspy.Example. You build it with keyword arguments and then mark which keys are inputs with .with_inputs(...); everything you did not mark is treated as a label.

import dspy

trainset = [
    dspy.Example(message="My card was charged twice.", category="billing").with_inputs("message"),
    dspy.Example(message="The app crashes when I open settings.", category="technical").with_inputs("message"),
    dspy.Example(message="Order 4471 has not arrived.", category="shipping").with_inputs("message"),
]

The .with_inputs() call is not optional bookkeeping. It is how DSPy knows what to feed the program and what to hold back as the answer, so forgetting it is one of the most common first-day errors — your program receives the label as an input and scores suspiciously well.

A metric is a plain function with the signature metric(example, pred, trace=None) returning a float or a bool. Higher is better.

def category_match(example, pred, trace=None):
    return example.category == pred.category

The trace parameter deserves a word, because it looks like noise and is not. During ordinary evaluation trace is None. During an optimizer's bootstrapping phase, DSPy passes the run's trace, and the convention is to return a stricter boolean in that case — the optimizer is deciding whether a generated example is good enough to keep as a demo, and you want only clean successes. A metric that returns partial credit during evaluation and a hard pass/fail during bootstrapping is a good default:

def quality(example, pred, trace=None):
    correct = example.category == pred.category
    reasoned = len(pred.reason.split()) >= 4
    if trace is not None:          # bootstrapping: be strict
        return correct and reasoned
    return 0.8 * correct + 0.2 * reasoned

DSPy also ships metrics you can use directly: dspy.evaluate.answer_exact_match and answer_passage_match for question answering, and SemanticF1 and CompleteAndGrounded, which use a model to judge the answer. For evaluating retrieval-augmented systems in more depth, the companion toolkit is RAGAS.

With data and a metric, evaluation is three lines:

evaluate = dspy.Evaluate(devset=devset, metric=category_match, num_threads=8, display_progress=True)
result = evaluate(program)

print(result.score)          # a percentage, 0-100, rounded to two decimals
for example, prediction, score in result.results:
    if not score:
        print(example.message, "->", prediction.category)

Two details to internalise. result.score is a percentage, not a fraction: a program that gets two-thirds right scores 66.67, not 0.67. And the per-example detail is in result.results, a list of (example, prediction, score) triples. If you find a tutorial passing return_outputs=True, that argument now raises an error telling you the results are always in results.

Printing the failures, as the loop above does, is the highest-value habit in this guide. The score tells you where you are; the failures tell you what to do next. Ten minutes reading wrong predictions usually beats an hour of prompt speculation.

Your first optimizer

An optimizer (older documentation calls it a teleprompter) takes your program, your training examples and your metric, and returns a new program with better instructions and better few-shot demos. Your original is not modified, which makes experimenting safe.

Start with dspy.BootstrapFewShot. It works by running your program on the training examples, keeping the runs the metric approves of, and attaching those successful runs to the predictors as demos. In other words, it writes your few-shot examples by using your own program's good days as the teacher.

import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

program = SupportTriage()

baseline = dspy.Evaluate(devset=devset, metric=category_match, num_threads=8)
print("before:", baseline(program).score)

optimizer = dspy.BootstrapFewShot(metric=category_match, max_bootstrapped_demos=4, max_labeled_demos=16)
optimized = optimizer.compile(program, trainset=trainset)

print("after:", baseline(optimized).score)

optimized.save("triage.json")

Three things to understand about what just happened.

It returned a new object. optimized is a separate program; program is untouched. Compare them on the same dev set, and keep the dev set separate from the training set, or you are measuring your own homework.

It cost real money and real time. Compiling runs your program many times. BootstrapFewShot is at the cheap end; the heavier optimizers are not. The official documentation is explicit that a GEPA or MIPROv2 run can cost hundreds of dollars in model calls on a large task. The economics of DSPy assume you compile once, offline, save the result, and serve it many times — not that you optimize per request.

The result is small. optimized.save("triage.json") writes state only: the tuned instructions, the chosen demos, the signature field metadata, and the LM configuration minus the API key, which is never saved. The file is human-readable, belongs in version control, and loads into a fresh instance of the same class:

program = SupportTriage()
program.load("triage.json")

Which optimizer next? The official cheat sheet gives a simple ladder, and it is worth following rather than jumping to the newest name. Starting out, use BootstrapFewShot. If the quality of your demos varies a lot between runs, move to BootstrapFewShotWithRandomSearch (aliased dspy.BootstrapRS), which tries several candidate demo sets and keeps the best. If the problem is that your instructions are wrong rather than your examples, use dspy.COPRO or dspy.GEPA, both of which rewrite instruction text. If both are weak and you have budget, dspy.MIPROv2 tunes instructions and demos together — it needs pip install "dspy[optuna]" and takes auto="light", "medium" or "heavy" to set the budget.

# A step up, once BootstrapFewShot has plateaued
optimizer = dspy.MIPROv2(metric=category_match, auto="light")
optimized = optimizer.compile(program, trainset=trainset, valset=valset)

Configuration, context and the cache

Three configuration mechanisms cover nearly everything a beginner needs.

dspy.configure(...) sets process-wide defaults: the LM, the adapter, usage tracking, the default thread count, the error budget. It is the thing you call once at the top of your program.

dspy.configure(
    lm=dspy.LM("openai/gpt-4o-mini", temperature=0.7, max_tokens=1000),
    track_usage=True,
    num_threads=8,
)

with dspy.context(...) makes a scoped override that is safe across threads and async tasks. Use it when one part of your program needs a different model — a cheap model for the bulk of the work, a strong one for a hard step.

with dspy.context(lm=dspy.LM("anthropic/claude-sonnet-4-5-20250929")):
    hard = careful_step(question=q)

And predictor.set_lm(other_lm) pins a model to one predictor permanently, which is the cleanest way to express "this step always uses the big model".

Now the cache, because it will confuse you before it helps you. DSPy caches LM responses by default, in memory and on disk. The disk cache lives in ~/.dspy_cache unless you set DSPY_CACHEDIR, and its size limit is controlled by DSPY_CACHE_LIMIT (30 GB by default). The key is a hash of the request arguments, excluding credentials.

This is excellent for development: rerunning a script is instant and free. It is also the explanation for the most common "DSPy is broken" report — you call the model three times with the same input and get the same answer three times, even at a high temperature. Nothing is broken; you are reading the cache. Two ways to get genuinely fresh calls:

lm = dspy.LM("openai/gpt-4o-mini", cache=False)          # this LM never caches
lm("Tell me a joke", rollout_id=1, temperature=1.0)      # cached, but a distinct key per rollout_id

The rollout_id form is usually what you want: you still get caching (so a rerun is free) but each rollout_id is a separate cache entry, which is how you collect several independent samples reproducibly.

Errors you will meet, and how to read them

DSPy's error messages are unusually good: most of them tell you the fix in the message. Here are the ones you will actually hit, and what is really going on behind each.

ValueError: No LM is loaded. The full message suggests dspy.configure(lm=dspy.LM('openai/gpt-4o-mini')), which is the fix. You called a module before configuring a model. In a notebook this usually means the configure cell has not been run since the kernel restarted.

ValueError: LM must be an instance of dspy.BaseLM, not a string. You wrote dspy.configure(lm="openai/gpt-4o-mini"). Wrap the string: dspy.configure(lm=dspy.LM("openai/gpt-4o-mini")). The model string names a model; dspy.LM is the client that talks to it.

dspy.LMAuthError. Missing or invalid key. Check the environment variable name matches the provider in your model string — an anthropic/ model will not read OPENAI_API_KEY. Worth knowing: since 3.3, DSPy normalises provider errors into its own exception hierarchy. Catch dspy.LMError or a subclass, not LiteLLM or OpenAI exceptions, because the provider exception will not reach you. The subclasses are descriptive: LMRateLimitError (with .retry_after), LMTimeoutError, LMServerError, LMInvalidRequestError, LMBillingError, ContextWindowExceededError and others, and dspy.is_retryable_lm_error(e) tells you whether retrying is sensible.

AdapterParseError: Adapter <Name> failed to parse the LM response ... Expected to find output fields in the LM response: [...] This is the characteristic DSPy error and it has three common causes. The reply was truncated because max_tokens is too low — the most frequent cause by a distance, especially with ChainOfThought, where the reasoning eats the budget before your fields are emitted. The model ignored the format, which smaller local models do more often; switching to dspy.JSONAdapter() or dspy.TwoStepAdapter(extraction_model=...) usually fixes it. Or the response was empty. In all three cases, dspy.inspect_history(n=1) shows you which it is in one glance.

dspy.LMRateLimitError. A provider 429. Lower num_threads, lean on the built-in retries (num_retries defaults to 3), and remember that an Evaluate with num_threads=16 against a new API key will trip limits that a single call never would.

dspy.ContextWindowExceededError. Your prompt is too long. For a beginner the usual culprit is an optimizer attaching more demos than you expected, or a long retrieved context. Reduce max_bootstrapped_demos, shorten the context, or move to a larger-context model.

LMConfigurationError for reasoning models. OpenAI's reasoning models (gpt-5 and the o-series) require temperature=1.0 or None and max_tokens of at least 16000, or None. The message says so explicitly. This catches everyone who copies a temperature=0.0, max_tokens=1000 line from a tutorial written for a chat model.

ModuleNotFoundError for numpy or optuna. Those are optional extras now, as covered in the install section: pip install "dspy[numpy]" or "dspy[optuna]".

ValueError: Loading .pkl files can run arbitrary code ... You tried to load a pickled program without opting in. Prefer .json saves. If you must load a pickle, and you trust where it came from, pass allow_pickle=True.

Deprecation warnings. If you see them for lm(messages=...), BaseLM.forward, CodeAct, ProgramOfThought or Image.from_file, you are using an API that goes away in 3.5. Fix them while they are warnings. Run python -W default::DeprecationWarning your_program.py to see the ones Python hides by default.

Explicit messages, when you need them

Most of the time you let modules talk to the model for you. Occasionally you want to send a raw request with a specific system message — for a sanity check, or to compare a DSPy program against a hand-written prompt. The 3.4 way to do that is an explicit request object, and it is worth seeing once because it is also the migration target for anyone with lm(messages=[...]) in their codebase.

import dspy
from dspy.lm15 import Config, Message, Request

lm = dspy.LM("openai/gpt-4o-mini")
response = lm(Request(
    model=lm.model,
    system="Be concise.",
    messages=(Message.user("What is DSPy?"),),
    config=Config(max_tokens=200),
))
print(response.text)

Two rules apply. The request's model must match the LM's model, and generation options go in Config — the LM's own defaults are not merged into an explicit request, so if you need max_tokens you must state it here. One more subtlety: Config.cache controls the provider's prompt caching, not DSPy's response cache, which is the cache= flag on dspy.LM.

The names lm15, Request and Response come from 3.4's biggest change. DSPy now ships native LM engines and dspy.LM(..., engine="auto") is the default: it prefers the native path where it can and falls back to LiteLLM before inference where it cannot. You can force either with engine="lm15" or engine="litellm". As a beginner you should leave engine alone — the default is correct — but knowing the word helps you read the release notes and the error messages. LiteLLM is still installed and is not deprecated.

Putting it all together

Here is the whole beginner arc in one file: a signature, a module, a dataset, a metric, a baseline, an optimizer, a saved artefact, and a reload. Run it, change one thing, run it again.

from typing import Literal
import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", max_tokens=1000), track_usage=True)


class Triage(dspy.Signature):
    """Classify an incoming customer support message for routing."""

    message: str = dspy.InputField(desc="The raw message as the customer wrote it")
    category: Literal["billing", "technical", "account", "shipping", "other"] = dspy.OutputField()
    urgent: bool = dspy.OutputField(desc="True only if the customer is blocked right now")
    reason: str = dspy.OutputField(desc="One short sentence justifying the category")


class SupportTriage(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.ChainOfThought(Triage)

    def forward(self, message):
        return self.classify(message=message)


def make(message, category, urgent):
    return dspy.Example(message=message, category=category, urgent=urgent).with_inputs("message")


trainset = [
    make("My card was charged twice for order 4471.", "billing", True),
    make("The app crashes when I open settings.", "technical", True),
    make("How do I change the email on my account?", "account", False),
    make("Order 4471 was due Sunday and has not arrived.", "shipping", True),
    make("Do you ship to Riyadh?", "shipping", False),
    make("Just wanted to say the new version is great.", "other", False),
]
devset = [
    make("I was billed in the wrong currency.", "billing", False),
    make("Login fails with a server error.", "technical", True),
    make("Please delete my account.", "account", False),
    make("Where is my parcel?", "shipping", True),
]


def score(example, pred, trace=None):
    correct = example.category == pred.category
    if trace is not None:
        return bool(correct and example.urgent == pred.urgent)
    return 0.7 * correct + 0.3 * (example.urgent == pred.urgent)


if __name__ == "__main__":
    program = SupportTriage()
    evaluate = dspy.Evaluate(devset=devset, metric=score, num_threads=4, display_progress=True)

    before = evaluate(program)
    print("baseline:", before.score)
    for example, prediction, s in before.results:
        if s < 1:
            print("  miss:", example.message, "->", prediction.category, prediction.urgent)

    optimizer = dspy.BootstrapFewShot(metric=score, max_bootstrapped_demos=3, max_labeled_demos=6)
    optimized = optimizer.compile(program, trainset=trainset)

    after = evaluate(optimized)
    print("optimized:", after.score)

    optimized.save("triage.json")

    reloaded = SupportTriage()
    reloaded.load("triage.json")
    print(reloaded(message="You charged me for a plan I cancelled.").category)

Run it and you get a baseline percentage, a list of misses, an optimized percentage, a triage.json on disk, and a prediction from the reloaded program. Then do the experiment that teaches the most: swap ChainOfThought for Predict and run it again. On a short classification like this one, reasoning often buys nothing and costs a third of your tokens — and now you have the number instead of an opinion.

Two notes for anyone building this for a real employer. Because the model string is the only thing naming a provider, the same program runs against an Azure OpenAI deployment in the UAE North region or against a model you host yourself, which is how teams in the Gulf and Egypt usually satisfy a data-residency requirement: dspy.LM("azure/<your-deployment>") with the AZURE_API_BASE of a regional endpoint, or dspy.LM("openai/...", api_base=...) pointed at a self-hosted OpenAI-compatible server. And because program.json contains no API key — dump_state excludes it unconditionally — the optimized artefact is safe to commit.

What you can now do, and what comes next

You can install DSPy, point it at a hosted or local model, and confirm the setup works. You can declare a task as an inline or class-based signature with typed outputs including Literal enumerations and Pydantic models. You can choose between Predict, ChainOfThought and ReAct on evidence rather than fashion, and compose them into a custom dspy.Module. You can read the exact prompt DSPy built with inspect_history, and the token cost with get_lm_usage(). You can build a dataset of dspy.Example objects, write a metric that behaves correctly during bootstrapping, score a program with dspy.Evaluate, and read the per-example failures. You can run BootstrapFewShot, measure the improvement, save the result as JSON and load it back. And you can diagnose the dozen errors that account for most beginner time loss.

What you have not touched is everything to do with running this in production. The Mid-level guide picks up at the boundary: how settings, context and threads actually interact; how the adapters format and parse, precisely enough to predict a parse failure; the full optimizer ladder including GEPA and when each one is the right spend; caching strategy and usage accounting for a service rather than a script; serving a program behind FastAPI with dspy.asyncify and streaming with dspy.streamify; and integrating tracing properly rather than printing history. The Senior guide goes further into multi-tenancy, the security model around pickles and code interpreters, cost control at scale, and the upgrade path to 3.5, where the APIs deprecated in 3.4 disappear.

Two things to do before you move on. First, finish a small project end to end, with a committed dev set and a score in the commit message — the habit is worth more than any additional API you could learn. Second, since you will want to see your traces rather than print them, read the MLflow guide next and turn on mlflow.dspy.autolog(); if your team already runs an LLM observability platform, the Langfuse guide covers the same ground from that side. When you start evaluating retrieval quality seriously, RAGAS is the natural companion.

Sources

• DSPy documentation
• Installation
• Language models
• Signatures
• Modules
• Adapters
• Tools
• Metrics
• Optimizers
• Choosing an optimizer
• Settings and context
• Saving and loading
• Caching
• Observability
• Normalized LM API migration
• Cheatsheet
• FAQs
• DSPy 3.4.0 release notes
• DSPy 3.3.1 release notes
• DSPy 3.3.0 release notes
• dspy on PyPI

This is the interview companion to the beginner guide. It assumes you have run a DSPy program at least once. Everything here is written against DSPy 3.4.0, and several of the traps exist precisely because interviewers (and candidates) learned DSPy from material written for older versions.

The thirty-second answer

When someone asks "what is DSPy?", say this:

DSPy is a Python framework for programming language models instead of prompting them. You declare a task's inputs and outputs as a typed signature, pick a module that decides the strategy, and DSPy builds the prompt and parses the response for you — then optimizers tune the instructions and few-shot examples automatically against a metric you define on a small dataset.

Two sentences, and they contain the two claims that matter: the separation of task from prompt text, and the measured self-improvement. If you have time for a third sentence, make it the consequence rather than more features: "so the prompt becomes generated output rather than hand-maintained source, and improvement becomes a number on a dev set instead of a judgement call."

Avoid two framings that interviewers hear as a red flag. Do not call DSPy "a prompt template library" — templating is the part it replaces, and describing it that way signals you only skimmed the README. And do not say it "makes prompts unnecessary"; prompts still exist, DSPy just generates them, and an interviewer who knows the tool will test whether you understand that by asking what the adapter does.

The model, in four nouns

Interviews at this level reward a clean vocabulary more than breadth. Be able to define these without hedging.

A signature declares a task: input and output field names, types, and instructions. Inline form "question -> answer" or "context: list[str], question: str -> answer: str"; class form subclasses dspy.Signature with dspy.InputField() and dspy.OutputField(), and the docstring carries the instructions. It is a specification, not a prompt.

A module is the strategy that satisfies a signature. dspy.Predict is a single call. dspy.ChainOfThought adds a reasoning output field so the model works before answering. dspy.ReAct loops over tools. Modules compose: subclass dspy.Module, build submodules in __init__, implement forward().

An adapter turns a signature plus inputs into messages and turns the reply back into typed values. The default ChatAdapter uses [[ ## field ## ]] markers and falls back to JSONAdapter if parsing fails. XMLAdapter, TwoStepAdapter and BAMLAdapter also exist.

An optimizer — older docs say teleprompter — takes a program, a trainset and a metric, and returns a new program with tuned instructions and demos. .compile(student, trainset=...) is the universal entry point, and the original program is not mutated.

Beneath those, the LM (dspy.LM("openai/gpt-4o-mini")) is the configured provider client, the predictor is the dspy.Predict leaf that holds the learnable state, demos are the few-shot examples attached to a predictor, and a prediction is the returned object whose attributes are your output fields.

If you can draw the pipeline — signature to module to adapter to LM, giving a prediction; then examples plus a metric into Evaluate, then into an optimizer, then saved and served — you have shown the mental model more convincingly than any list of class names.

The code to have ready

There is no DSPy CLI, so "commands to have ready" means API you can write on a whiteboard without hesitating. Six snippets cover almost every beginner question.

Configure and call:

import dspy
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", temperature=0.7, max_tokens=1000))
pred = dspy.ChainOfThought("question -> answer")(question="Why is the sky blue?")
print(pred.reasoning, pred.answer)

A class signature with a constrained output:

from typing import Literal
import dspy

class Emotion(dspy.Signature):
    """Classify the emotion of a sentence."""
    sentence: str = dspy.InputField()
    emotion: Literal["sadness", "joy", "love", "anger", "fear", "surprise"] = dspy.OutputField()

A custom module:

class RAG(dspy.Module):
    def __init__(self):
        super().__init__()
        self.respond = dspy.ChainOfThought("context, question -> response")

    def forward(self, question):
        return self.respond(context=search(question), question=question)

Data and a metric:

trainset = [dspy.Example(question="...", answer="...").with_inputs("question")]

def exact_match(example, pred, trace=None):
    return example.answer.lower() == pred.answer.lower()

Evaluation:

evaluate = dspy.Evaluate(devset=devset, metric=exact_match, num_threads=8, display_progress=True)
result = evaluate(program)
print(result.score)      # a percentage, 0-100

Optimize, save, load:

optimized = dspy.BootstrapFewShot(metric=exact_match, max_bootstrapped_demos=4).compile(program, trainset=trainset)
optimized.save("program.json")

fresh = RAG(); fresh.load("program.json")

And one debugging line you should offer before being asked: dspy.inspect_history(n=1), which prints the prompt DSPy actually sent. Volunteering it tells an interviewer you have debugged a real program rather than read a tutorial.

The classic traps, and why they happen

Each of these is a question in disguise. Knowing the why is what separates a memorised answer from an understood one.

Identical answers on repeated calls. DSPy caches LM responses by default, in memory and on disk under ~/.dspy_cache (override with DSPY_CACHEDIR). The key is a hash of the request arguments, so the same input at any temperature returns the same cached response. The fix is dspy.LM(..., cache=False) for a non-caching client, or lm("...", rollout_id=1, temperature=1.0) to keep caching while getting a distinct entry per rollout. This exists because compiling an optimizer re-runs your program hundreds of times; without a cache, development would be unaffordable.

AdapterParseError. The adapter could not find the declared output fields in the reply. The most common cause is truncation: max_tokens too low, especially with ChainOfThought, where reasoning consumes the budget before the real fields are emitted. Other causes are a model that ignores the format (common with small local models) or an empty response. Fixes, in order: raise max_tokens, inspect the history, switch to JSONAdapter or TwoStepAdapter, use a stronger model.

Forgetting .with_inputs(). dspy.Example(question=..., answer=...) without .with_inputs("question") leaves DSPy unable to tell inputs from labels, so the label can be fed to the program as an input. The symptom is a suspiciously high score. It happens because Example is deliberately a loose bag of keys; with_inputs is the only thing that partitions it.

Reading result.score as a fraction. Evaluate returns a percentage rounded to two decimals, so two-thirds correct is 66.67, not 0.67. Candidates who say "0.67" get a follow-up.

Expecting the optimizer to mutate in place. compile() returns a new program. Writing optimizer.compile(program, trainset=trainset) and then evaluating program measures the unoptimized version, and the candidate concludes the optimizer does nothing.

Calling forward() directly. program.forward(x=...) bypasses DSPy's wrapper, which records the trace that optimizers learn from and fires callbacks and usage tracking. Always call the module.

dspy.configure from a worker thread. It raises RuntimeError: dspy.settings can only be changed by the thread that initially configured it. Only the thread that first configured may configure again; workers use with dspy.context(...), which is ContextVar-scoped and async-safe.

Reasoning-model parameters. gpt-5 and the o-series require temperature=1.0 or None and max_tokens >= 16000 or None; anything else raises LMConfigurationError with the required values in the message. Copying temperature=0.0, max_tokens=1000 from a chat-model tutorial is the usual cause.

Catching the wrong exception. Since 3.3, DSPy normalises provider failures into its own hierarchy. except openai.RateLimitError will not fire; catch dspy.LMError or dspy.LMRateLimitError. dspy.is_retryable_lm_error(e) answers whether to retry.

Teaching removed APIs. dspy.Assert and dspy.Suggest are deprecated and unsupported — use dspy.Refine or dspy.BestOfN. dspy.TypedPredictor and TypedChainOfThought are gone; typed signatures work directly. dspy.CodeAct and dspy.ProgramOfThought warn on construction and go away in 3.5, replaced by dspy.RLM. lm(messages=[...]) is deprecated in 3.4 and removed in 3.5. Mentioning any of these as current is an instant signal that your knowledge is second-hand.

ModuleNotFoundError for numpy or optuna. Those moved to optional extras: pip install "dspy[numpy]" covers embeddings, KNN/KNNFewShot and SIMBA; "dspy[optuna]" covers MIPROv2 and BootstrapFewShotWithOptuna.

Common interview questions, with model answers

"Why would I use DSPy instead of just writing a prompt?"

Because a hand-written prompt tangles three decisions into one string: what the task is, what strategy to use, and how this particular model wants its output formatted. That tangle means you cannot change the model without rewriting your instructions, and you cannot add reasoning without rewriting the output contract. DSPy puts those three in separate objects, so each changes independently. The second reason is the one that actually pays: once the task is data rather than prose, an optimizer can search over instructions and few-shot examples against a metric, which is a loop you cannot run by hand at any useful sample size.

"What is the difference between Predict and ChainOfThought?"

ChainOfThought is Predict with an extra reasoning output field prepended to your declared fields, so the model produces its working before committing. You read it back as pred.reasoning. It tends to help on multi-step or multi-condition tasks and costs extra output tokens on every call. For short classification it is often pure cost, which is why the right answer ends with "and I would measure it on a dev set rather than assume."

"How do you know a DSPy program got better?"

You need two artefacts: a devset of dspy.Example objects with .with_inputs() marking the inputs, and a metric function metric(example, pred, trace=None) returning a float or bool. Then dspy.Evaluate(devset=..., metric=..., num_threads=8) called on the program returns an EvaluationResult whose .score is a percentage and whose .results is a list of (example, prediction, score) triples. You score before and after, on a devset held separate from the trainset, and you read the failures in .results — the score tells you where you are, the failures tell you what to fix.

"What does the trace argument in a metric do?"

It distinguishes evaluation from bootstrapping. During normal evaluation trace is None. During an optimizer's bootstrapping phase, DSPy passes the run's trace, and the convention is to return a stricter boolean then, because the optimizer is deciding whether a generated run is clean enough to keep as a few-shot demo. A typical metric returns partial credit when trace is None and a hard pass/fail otherwise.

"Which optimizer would you start with?"

dspy.BootstrapFewShot. It runs your program on the trainset, keeps the runs the metric approves, and attaches them as demos — cheap, and it needs nothing beyond what you already have. From there the official ladder is: BootstrapFewShotWithRandomSearch (alias BootstrapRS) if demo quality varies between runs; COPRO or GEPA if the instructions rather than the examples are the problem; MIPROv2 if both are weak and you have budget; BootstrapFinetune once prompt-only work has plateaued and the model is tunable; BetterTogether to chain optimizers. I would also say out loud that compiling is an offline, paid step — the official docs note GEPA or MIPROv2 runs can cost hundreds of dollars — so the pattern is compile once, save, serve many times.

"How do you save and ship an optimized program?"

program.save("program.json") writes state only: tuned instructions, chosen demos, signature field metadata, and the LM config with the API key excluded — it is never serialized. In production you instantiate the same module class and call program.load("program.json"). JSON is the recommended mode; .pkl and whole-program saves exist but require allow_pickle=True on load because they can execute arbitrary code. A good answer adds that the JSON is small and readable, so it belongs in version control next to the devset, with the score in the commit message.

"The model keeps returning the same thing. What do you check?"

The cache, first and almost always. DSPy caches by default and the key ignores temperature in the sense that an identical request hits the same entry. Confirm with dspy.inspect_history(n=2) — if both calls show the same response and no new provider call happened, it is the cache. Then either cache=False or distinct rollout_id values with a non-zero temperature.

"How do you point DSPy at a model that is not OpenAI?"

Change the model string. They follow LiteLLM naming: anthropic/..., gemini/..., vertex_ai/..., azure/<deployment>, ollama_chat/.... Credentials come from the matching environment variables, or api_key= on dspy.LM. For a local model, dspy.LM("ollama_chat/llama3.2", api_base="http://localhost:11434", api_key=""). For a self-hosted OpenAI-compatible server such as vLLM or SGLang, an openai/-prefixed model with api_base pointed at it. This is also the answer to a data-residency question: a regional Azure deployment or a self-hosted endpoint is a one-line change, and nothing about the signature or the saved program moves.

"What is new in DSPy 3.4, and what breaks in 3.5?"

3.4 introduced native LM engines: dspy.LM(..., engine="auto") is the default and prefers the vendored native path, falling back to LiteLLM before inference when needed, with engine="lm15" and engine="litellm" to force either. It also added dspy.LocalInterpreter — explicitly not a sandbox, unlike the default PythonInterpreter — and async ReActV2 via acall. The things removed in 3.5 are worth naming: lm(messages=[...]), custom BaseLM.forward() integrations, LegacyEngine, and dspy.CodeAct / dspy.ProgramOfThought. Run with python -W default::DeprecationWarning to find them.

"What does an adapter actually do, and when would you change it?"

It is the layer between the signature and the model. On the way out it turns your fields and instructions into chat messages — the default ChatAdapter marks each field with [[ ## field_name ## ]] so the reply is unambiguous to parse. On the way back it extracts those fields and coerces them to the declared types, validating Pydantic models and enforcing Literal options. ChatAdapter falls back to JSONAdapter if parsing fails, which is why many format problems fix themselves. You change it when a model consistently refuses the marker format: dspy.configure(adapter=dspy.JSONAdapter()) for models that are better at JSON, or dspy.TwoStepAdapter(extraction_model=dspy.LM(...)) for reasoning models that answer well but format badly — it lets the strong model answer freely and a second, cheap model pull the fields out. For strict tests you can disable the fallback with dspy.ChatAdapter(use_json_adapter_fallback=False) so a format failure is loud rather than silently repaired.

"Where does DSPy fit in a wider stack?"

It is an in-process Python library, not a service — there is no server and no CLI. So it sits inside whatever already serves your application: a FastAPI handler, a batch job, a worker. The neighbouring pieces are a provider gateway if you have one (the model strings are LiteLLM-compatible, so a LiteLLM proxy drops straight in), a tracing backend such as MLflow for observability, and an evaluation toolkit if you are measuring retrieval quality rather than just final answers. The one architectural rule to state is the split between compile time and serve time: optimization is an offline job whose output is a versioned program.json, and the serving process only loads that file and configures an LM.

Sixty-second self-test

Answer out loud, without looking. Each should take about five seconds.

1. Name the four nouns and what each one is responsible for.
2. Write "context: list[str], question: str -> answer: str" and say which names are inputs.
3. What extra field does ChainOfThought add, and what does it cost?
4. What does .with_inputs() do, and what is the symptom of forgetting it?
5. Is result.score a fraction or a percentage?
6. What does compile() return, and what happens to the program you passed in?
7. Two ways to defeat the response cache.
8. Three causes of AdapterParseError, in order of likelihood.
9. Which exception class do you catch for a provider 429?
10. What does program.save("p.json") include, and what does it deliberately exclude?
11. Why must you call the module rather than forward()?
12. Which optimizer do you start with, and why that one?
13. Name two APIs that older tutorials teach and 3.4 has deprecated or removed.
14. Which parameters do OpenAI reasoning models require?
15. What single line shows you the prompt DSPy actually sent?

If you hesitated on more than three, reread the corresponding section of the detailed guide rather than this page — the traps make sense only once you have caused them yourself. If you were comfortable with all fifteen, the next things an interviewer will probe are the mid-level topics: how adapters parse, how context interacts with threads, and how you would serve a compiled program behind an API.

This page is the collected mistakes. Everything here is something beginners actually do with DSPy 3.4.0, written as symptom, cause and fix, followed by the habits worth adopting on your first day and a checklist to run before you call a program finished.

The mistakes beginners actually make

Installing the wrong package

Symptom. pip install dspy-ai succeeds, then half the tutorial's imports fail or behave oddly.

Cause. The project was once published as dspy-ai. That name is legacy. Any guide telling you to install it is old enough that other things in it are also wrong.

Fix. pip install -U dspy, and import dspy. Then pin it: pip install "dspy==3.4.0". DSPy has shipped breaking changes in minor releases, so a floating dependency means an unrelated pip install -U can break your program next month.

Passing a model string to configure

Symptom. ValueError: LM must be an instance of dspy.BaseLM, not a string.

Cause. You wrote dspy.configure(lm="openai/gpt-4o-mini"). The string names a model; it is not a client.

Fix. Wrap it: dspy.configure(lm=dspy.LM("openai/gpt-4o-mini")). The error message spells out exactly this, which is a good reminder that DSPy's errors are worth reading to the end before searching the web.

Being surprised by the cache

Symptom. You call the model five times at temperature=1.0 expecting five different answers and get the same one five times. Or an evaluation finishes impossibly fast and reports zero token usage.

Cause. DSPy caches LM responses by default, in memory and on disk under ~/.dspy_cache (override with DSPY_CACHEDIR; the disk limit is DSPY_CACHE_LIMIT, 30 GB by default). The cache key is a hash of the request arguments, so an identical request is served from the cache regardless of temperature. Cached calls correctly report empty usage.

Fix. For genuinely independent samples, either dspy.LM(..., cache=False) or — usually better — lm("...", rollout_id=1, temperature=1.0) with a different rollout_id per sample, which keeps caching for reruns while giving each sample its own entry.

Forgetting .with_inputs()

Symptom. Your baseline score is implausibly high, or the program answers correctly on inputs it clearly should not handle.

Cause. dspy.Example(question=..., answer=...) is an untyped bag of keys. Only .with_inputs("question") tells DSPy which keys are inputs; without it, the label can be handed to the program as an input and the program is effectively reading the answer.

Fix. Always chain it: dspy.Example(question=q, answer=a).with_inputs("question"). Build a small helper function so you cannot forget — that is what the detailed guide's make() function is for.

Reading Evaluate's score as a fraction

Symptom. You report that your program scores "0.67" and your colleague assumes you mean 67%, when in fact DSPy said 66.67.

Cause. EvaluationResult.score is a percentage from 0 to 100, rounded to two decimals.

Fix. Treat it as a percentage, and label it that way in logs. Also stop using return_outputs=True if you picked it up from an old tutorial: it now raises, and the per-example detail is always in result.results as (example, prediction, score) triples.

Expecting compile() to modify your program

Symptom. You run an optimizer, evaluate, and the score has not moved at all.

Cause. optimizer.compile(program, trainset=...) returns a new program. The one you passed in is unchanged by design, so evaluating it measures the baseline again.

Fix. Assign the result: optimized = optimizer.compile(program, trainset=trainset) and evaluate optimized. Keep both around and score them on the same devset, which is also the only honest way to report an improvement.

Training and evaluating on the same examples

Symptom. A large improvement that evaporates on the first real input.

Cause. The optimizer chose demos using the very examples you then scored on. The score measures memorisation.

Fix. Split before you optimize. A trainset for compile(), a devset for Evaluate, no overlap. With only twenty examples this feels wasteful and still matters more than anything else you will do.

Calling forward() directly

Symptom. An optimizer runs, consumes calls, and attaches no demos. Callbacks never fire. get_lm_usage() is empty.

Cause. program.forward(x=...) skips the wrapper that DSPy puts around a module call — the part that records the trace optimizers learn from, fires callbacks, and tracks usage.

Fix. Call the module: program(x=...). This applies inside your own forward() too — call submodules, never their forward.

max_tokens too low, especially with ChainOfThought

Symptom. AdapterParseError: Adapter ChatAdapter failed to parse the LM response ... Expected to find output fields in the LM response: [...]

Cause. The reply was truncated. ChainOfThought emits a reasoning field before your declared fields, so a tight token budget gets spent on reasoning and the real fields never appear. Other causes of the same error are a model that ignores the format and an empty response.

Fix. Raise max_tokens first — it is the most common cause by a wide margin. Then run dspy.inspect_history(n=1) to see which of the three you have. If the model is genuinely ignoring the format, dspy.configure(adapter=dspy.JSONAdapter()) or dspy.TwoStepAdapter(extraction_model=dspy.LM(...)) usually fixes it.

Using reasoning-model parameters from a chat-model tutorial

Symptom. LMConfigurationError mentioning temperature=1.0 or None and max_tokens >= 16000 or None.

Cause. OpenAI's reasoning models (gpt-5, the o-series) refuse the temperature=0.0, max_tokens=1000 settings that are standard for chat models.

Fix. Use what the message says: dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=16000). For OpenAI's Responses API, add model_type="responses".

Catching provider exceptions

Symptom. Your except openai.RateLimitError block never runs and the program crashes anyway.

Cause. Since 3.3, DSPy normalises LM failures into its own hierarchy, so the provider's exception never reaches your code.

Fix. Catch dspy.LMError or a specific subclass: LMRateLimitError (which carries .retry_after and .provider), LMAuthError, LMTimeoutError, LMServerError, LMInvalidRequestError, LMBillingError, ContextWindowExceededError. Use dspy.is_retryable_lm_error(e) to decide whether a retry makes sense.

Hammering the provider with threads

Symptom. dspy.LMRateLimitError as soon as you raise num_threads, or an evaluation that mostly fails.

Cause. Evaluate, Parallel and .batch() run concurrently (default num_threads=8). A new API key with a low tier will not take that, let alone 16.

Fix. Lower num_threads, let the built-in retries work (num_retries defaults to 3), and remember max_errors defaults to 10 — a run can abort because the error budget ran out, not because your code is wrong.

Calling dspy.configure from a worker thread

Symptom. RuntimeError: dspy.settings can only be changed by the thread that initially configured it. In async code, a similar message pointing at dspy.context.

Cause. configure sets process-wide defaults and is restricted to the thread (or async task) that first called it.

Fix. Configure once at the top of your program. Everywhere else use with dspy.context(lm=..., ...), which is scoped and safe across threads and async tasks, or pin a model to one predictor with predictor.set_lm(other_lm).

Copying APIs that no longer exist

Symptom. AttributeError, or a DeprecationWarning you ignore until an upgrade breaks you.

Cause. A large amount of DSPy writing on the web predates the current API.

Fix. Know the list. dspy.Assert and dspy.Suggest are deprecated and unsupported — use dspy.Refine or dspy.BestOfN. dspy.TypedPredictor and TypedChainOfThought are gone; typed signatures work directly with Predict and ChainOfThought. dspy.OpenAI(...) and the old client classes are replaced by dspy.LM("provider/model"). dspy.CodeAct and dspy.ProgramOfThought warn on construction and go away in 3.5, replaced by dspy.RLM. The prefix=, format= and parser= field kwargs are deprecated. And lm(messages=[...]) is deprecated in 3.4 and removed in 3.5 — even though some official pages still show it — so use lm("...") for quick checks and an explicit dspy.lm15.Request(...) when you need system and user messages.

Confusing optional extras with a broken install

Symptom. ModuleNotFoundError: No module named 'numpy' from KNNFewShot, or optuna from MIPROv2.

Cause. The maintainers moved heavy dependencies out of the base install.

Fix. pip install "dspy[numpy]" for embeddings, KNN/KNNFewShot and SIMBA; "dspy[optuna]" for MIPROv2 and BootstrapFewShotWithOptuna; "dspy[deno]" for the sandboxed interpreter; "dspy[mcp]" for MCP tools. Quote the brackets — zsh and PowerShell both treat bare brackets as patterns.

Writing a 300-word docstring instead of a metric

Symptom. Hours spent editing instructions, no reproducible evidence that anything improved.

Cause. The instinct carried over from prompt engineering. It is the single biggest waste of time in a first DSPy week.

Fix. Write a short honest signature, then spend the time on twenty examples and a metric. The metric is what lets the library do the editing for you — and if you cannot write one, you do not yet know what a good answer looks like, which is a more important discovery than any prompt edit.

Loading a pickled program without thinking

Symptom. ValueError: Loading .pkl files can run arbitrary code, which may be dangerous...

Cause. You tried to load a pickle without allow_pickle=True, or copied a save_program=True example.

Fix. Prefer program.save("p.json") and program.load("p.json"): state only, human readable, API key never included. Only pass allow_pickle=True for a file you produced yourself and can vouch for. This matters more than it looks — a pickle you downloaded is executable code.

Habits worth adopting on day one

Print the prompt before you theorise. dspy.inspect_history(n=1) after any surprising output. Roughly half the time the cause is visible immediately: a field description that says something you did not mean, or a reply cut off mid-field. Guessing first and looking second is how an afternoon disappears.

Keep a growing devset in the repo. Every input your program gets wrong goes into a file with the answer you wanted. Twenty examples is enough to run an optimizer. Collecting them as you notice them is free; collecting them in a hurry when someone asks for a quality number costs a day.

Score before you change anything. Run dspy.Evaluate on the unmodified program and write the number down. Without a baseline every later number is uninterpretable, and you will find yourself arguing from memory about whether last Tuesday's version was better.

Read the failures, not just the score. Loop over result.results and print every example where the score is below one. One sentence describing what the failures have in common is worth more than another optimizer run.

Start with Predict. Add ChainOfThought only when you have measured that reasoning helps this task. It roughly doubles output tokens, and on short classification it frequently buys nothing. The same discipline applies to ReAct: do not give the model tools until a tool-free version has demonstrably failed.

Write tool docstrings for the model. dspy.Tool(func) introspects the name, docstring and argument types, and that is all the model gets. A tool named lookup whose docstring says "looks things up" will be called at the wrong moments, and no optimizer fixes that.

Put disputable definitions in desc=. Any rule a human reviewer would have to ask about ("does a refund request count as billing?") belongs in a field description, where it is explicit and where an optimizer can see and refine it. Rules in your head are rules the program does not have.

Pin everything. dspy==3.4.0 in your requirements, and the Python version alongside it (DSPy 3.4 supports 3.10 through 3.14). A reproducible optimizer run depends on a reproducible environment.

Commit the artefact with its score. When you save program.json, commit it together with the devset that produced the number, and put the number in the commit message. An optimized program without its score and its data is an unverifiable claim.

Treat compiling as an offline, paid step. BootstrapFewShot is cheap; MIPROv2 and GEPA are not — the official docs are explicit that a large run can cost hundreds of dollars in model calls. Compile once, save, serve many times. Never optimize inside a request handler.

Turn on usage tracking while you learn. dspy.configure(track_usage=True) and pred.get_lm_usage() turn "this feels expensive" into token counts. Note that lm.history[-1]["cost"] can be unknown rather than zero, so do not blindly sum it.

Graduate from printing to tracing early. inspect_history is a debugging prompt, not an observability story. The integration the official docs cover is MLflow — mlflow.dspy.autolog() records every module and LM call. The MLflow guide covers the setup; if your team already runs an LLM platform, Langfuse has a DSPy integration too.

Keep the provider in one place. The model string is the only thing in your program that names a vendor. Load it from configuration rather than hard-coding it, and switching to a regional Azure deployment or a self-hosted Ollama or vLLM endpoint — which is how most teams in the Gulf and Egypt satisfy data-residency rules — stays a one-line change.

The checklist

Run through this before you call a beginner DSPy program finished.

Setup. dspy installed (not dspy-ai), pinned to an exact version. Python 3.10-3.14. dspy.__version__ prints what you expect. Provider key in an environment variable, never in the source. Any optional extra your code paths need is installed.

Program. The LM is wrapped in dspy.LM(...), not passed as a string. dspy.configure is called once, at the top, from the main thread; everything scoped uses dspy.context. Output fields are typed, with Literal for anything enumerable. Disputable rules live in docstrings and desc=, not in your head. You call modules, never forward(). No deprecated API in the file, confirmed by a run under -W default::DeprecationWarning.

Data and metric. Every dspy.Example has .with_inputs(). Trainset and devset do not overlap. The metric returns a stricter boolean when trace is not None. You have at least twenty examples, including the failures you have noticed so far.

Measurement. A baseline score, recorded. The failing examples printed and read. An optimizer run assigned to a new variable and scored on the same devset. The improvement reported as a percentage, with the devset size next to it.

Shipping. Saved as .json, not .pkl. The file committed alongside the devset, with the score in the commit message. The reload path tested — instantiate the class, load(), run one input. num_threads set to something your provider tier tolerates, and max_errors understood. Caching decided deliberately: on for development, and consciously chosen for whatever runs in production.

Work through that list once by hand and most of it becomes automatic. The two items that stay worth checking every single time are the trainset/devset split and the baseline score, because those are the two that make every other number you report either meaningful or worthless.

This is part two of three. Beginner left you able to declare a signature, call a module, write a metric and run an optimiser once. This level takes every one of those further and adds the machinery that turns a working notebook into something a team can deploy, measure and keep running on DSPy 3.4.

Where this picks up

Each section opens with the problem it solves and closes with a Try it you can run on a real project in a few minutes. Everything here is checked against DSPy 3.4.0 (released 25 September 2026), which runs on Python 3.10 to 3.14.

The call path, end to end

Beginner treated a module call as a black box: you pass inputs, you get a Prediction. To debug anything beyond a missing API key, you need the five stages inside that box, because each one fails in its own recognisable way.

The signature is pure declaration: field names, types, descriptions, and the docstring that becomes the task instructions. It is the only part an optimiser edits when it rewrites instructions.

The module is the strategy. Predict is the leaf that actually calls the LM; everything else composes predictors. ChainOfThought is a Predict with a reasoning output field prepended to your declared outputs, which is why pred.reasoning exists on its output and not on a plain Predict. ReAct is a loop that appends tool calls and observations to a growing trajectory. Knowing which predictors a module contains matters, because optimisers tune predictors, not modules:

import dspy

class Pipeline(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.Predict("ticket -> category: str")
        self.reply = dspy.ChainOfThought("ticket, category -> reply: str")

    def forward(self, ticket):
        category = self.classify(ticket=ticket).category
        return self.reply(ticket=ticket, category=category)

for name, predictor in Pipeline().named_predictors():
    print(name, type(predictor).__name__)

classify Predict
reply.predict Predict

Two predictors, so an optimiser has two sets of instructions and two sets of demos to tune. That count is the single best predictor of how expensive a compile will be.

The adapter turns the signature plus the inputs into messages, and turns the response text back into typed fields. It gets its own section below.

The LM and its engine perform the HTTP call. The LM holds the model string, sampling parameters, the cache flag, retry count and the call history; the engine is the backend that executes the request. In 3.4 that backend is selectable.

The Prediction is a plain object whose attributes are your declared output fields. If a field is missing, the adapter failed to find it, so the problem is upstream in formatting or truncation, never in your code reading the attribute.

The practical payoff: decide which stage owns a symptom before changing anything. An LMAuthError is stage four. An AdapterParseError is stage three, almost always caused by stage four truncating the response. A present-but-nonsense field is stage one, your instructions. Correct per-step but wrong overall is stage two, your composition.

Adapters, and what a parse failure really means

The problem adapters solve: language models emit text, but your program wants a float, a Literal, a list of strings or a Pydantic object. Something has to specify a format in the prompt and enforce it on the way back. That something is the adapter, and it is a configurable, swappable layer.

DSPy 3.4 ships five. ChatAdapter is the default: fields delimited by [[ ## field_name ## ]] markers, cheap in tokens and robust with most instruction-following models, falling back to JSONAdapter for a call whose parse fails. JSONAdapter asks for a JSON object and can use provider-side response-schema enforcement where it exists. XMLAdapter uses XML tags and, since 3.3.1, handles nested structures. TwoStepAdapter runs the task with your main model and then uses a separate extraction model to pull the fields out of whatever it said. BAMLAdapter is the fifth, for BAML-style formatting.

import dspy

# Process-wide
dspy.configure(adapter=dspy.JSONAdapter())

# Scoped to a block
with dspy.context(adapter=dspy.XMLAdapter()):
    result = program(ticket="card declined twice")

# Reasoning models that answer well but format badly
dspy.configure(adapter=dspy.TwoStepAdapter(extraction_model=dspy.LM("openai/gpt-4o-mini")))

The choice is a trade-off, not a preference. ChatAdapter costs the fewest tokens and its fallback hides format drift. JSONAdapter is stricter and suits provider-side schema enforcement, but braces inside free-text fields are a recurring source of malformed responses. TwoStepAdapter doubles your call count per prediction, so it fixes a specific class of model rather than being a default.

When a parse does fail you get a precise error:

AdapterParseError: Adapter ChatAdapter failed to parse the LM response.
...
Expected to find output fields in the LM response: ['category', 'reply']

Four common causes, in this order. Truncation: max_tokens too low for the reasoning plus every output field, so the response stops mid-field. An empty response, usually a content filter or a provider hiccup. A model too weak for the format, showing up as prose where fields should be. A long or awkward schema, where ten similarly named output fields confuse the model. Since 3.3, ChatAdapter and JSONAdapter no longer retry an alternate format after an LMError — only after a parse failure — so a rate-limit error is never masked by a format retry.

One 3.3.1 detail matters for schema design: adapters apply a field's declared default when an optional output field is omitted. A missing required output still raises AdapterParseError. Giving an output field a default is therefore a decision to accept silence as a valid answer.

Typed outputs are where adapters earn their keep: the type you declare becomes part of the prompt and part of the validation.

from typing import Literal
from pydantic import BaseModel
import dspy

class Entity(BaseModel):
    name: str
    kind: Literal["person", "org", "place"]

class Extract(dspy.Signature):
    """Extract every named entity from the passage."""
    passage: str = dspy.InputField()
    entities: list[Entity] = dspy.OutputField(desc="one item per distinct entity")
    confidence: float = dspy.OutputField(desc="0.0 to 1.0")

extract = dspy.Predict(Extract)

Note that prefix, format and parser on InputField/OutputField were deprecated in 3.2 — if you inherited code that uses them, move the guidance into desc and the docstring. Duplicate input/output field names are now rejected outright.

Engines: what 3.4 changed, and what you must migrate

The problem: for years every DSPy call went through LiteLLM, which gave DSPy a hundred providers for free but put a large dependency on the hot path of every request. DSPy 3.4 introduces native LM engines. DSPy vendors lm15 and exposes it as dspy.lm15, with no separate install.

The selection is a constructor argument:

import dspy

lm_auto = dspy.LM("openai/gpt-4o-mini")                     # engine="auto" is the default
lm_native = dspy.LM("openai/gpt-4o-mini", engine="lm15")    # force native; raises if impossible
lm_litellm = dspy.LM("openai/gpt-4o-mini", engine="litellm")  # force the compatibility path

engine="auto" prefers native execution for supported routes and falls back to LiteLLM before inference when it cannot run natively. The timing is the important part: the fallback decision happens before any request is sent. Authentication failures, timeouts and provider errors never trigger a backend switch, so you will not see a request silently re-run on a different backend after it failed. LiteLLM remains installed and is not deprecated.

Four engine behaviours will surprise you if nobody tells you.

n>1 differs between engines. The native engine issues separate sequential requests, so input tokens are billed once per sample. If you rely on provider-native multi-sampling for cost reasons — BestOfN, self-consistency, majority voting — use engine="litellm" for that LM and measure.

Native timeouts bound individual transport waits, not total generation time. A long generation that keeps streaming is not cut off by a timeout you set expecting a wall-clock bound; impose a hard overall limit in your own code.

A stream is not retried once it has emitted a chunk, and incomplete responses are not cached. So a streaming endpoint has a failure mode a non-streaming one does not: partial output, no retry.

Unknown cost stays unknown. lm.history[-1]["cost"] can legitimately be absent rather than zero, and a dashboard that coerces missing to zero under-reports by exactly what it cannot see.

To add a provider that speaks the OpenAI or Anthropic HTTP shape — a self-hosted vLLM or SGLang server, a LiteLLM proxy, a gateway in your own VPC — register the endpoint instead of writing an engine:

import dspy

dspy.lm15.register_provider(
    "internal",
    base_url="https://llm.internal.example/v1",
    api_key_env="INTERNAL_LLM_KEY",
)
lm = dspy.LM("internal/qwen3-32b", engine="lm15")

Each LM binds the registrations that exist at construction time, so register before you build your LMs, not after. For anything stranger than an OpenAI- or Anthropic-compatible endpoint, implement an object with complete(request) -> Response and pass it as engine= (async_engine= for async). With a custom engine, api_key, api_base and timeout belong on the engine object: DSPy rejects them on dspy.LM(...).

Now the migration work, because 3.4 named 3.5 as the deadline. Deprecated in 3.4 and removed in 3.5: OpenAI-style lm(messages=[...]) calls, custom BaseLM.forward()/aforward() integrations, and LegacyEngine/AsyncLegacyEngine/complete_legacy(). Still supported in 3.5: lm("hello") returning a list of strings, and ordinary DSPy programs — the overwhelming majority of normal code.

import dspy
from dspy.lm15 import Config, Message, Request

lm = dspy.LM("openai/gpt-4o-mini")
response = lm(Request(
    model=lm.model,
    system="Be concise.",
    messages=(Message.user("What is DSPy?"),),
    config=Config(max_tokens=200),
))
print(response.text)

Two rules on that form. The request's model must match the LM. And generation options go in Config — the LM's own defaults are not merged in, so max_tokens set on dspy.LM(...) does not apply to an explicit Request. Also note that Config.cache controls provider prompt caching, not DSPy's own response cache.

If you are carrying 3.3 code that used the experimental typed LM API, those types were removed rather than aliased: dspy.LMRequest/LMResponse become dspy.lm15.Request/Response, LMMessage/LMConfig become lm15.Message/Config, dspy.System(text) becomes Request(system=text, ...), dspy.User/Assistant(text) become Message.user(...)/Message.assistant(...), and response.outputs[0].parts becomes response.message.parts. Importing dspy.core.types raises a migration error.

Settings, context, and the thread rules

The problem: a real pipeline uses more than one model. A cheap model for classification, a strong one for synthesis, a different one per tenant, a big-context one for a single step. DSPy gives you three levels of control, and the differences are not cosmetic.

import dspy

# 1. Process-wide defaults
dspy.configure(
    lm=dspy.LM("openai/gpt-4o-mini"),
    adapter=dspy.JSONAdapter(),
    track_usage=True,
    num_threads=8,
    max_errors=10,
)

# 2. Scoped, thread- and async-safe override
with dspy.context(lm=dspy.LM("anthropic/claude-sonnet-4-5-20250929")):
    hard = program(ticket=text)

# 3. Pinned to one predictor, permanently
program.reply.predict.set_lm(dspy.LM("openai/gpt-4o"))

dspy.configure is the process default. Its keys include lm, adapter (which defaults to ChatAdapter when unset), rm, callbacks, track_usage=False, num_threads=8, async_max_workers=8, max_errors=10, provide_traceback=False, disable_history=False, warn_on_type_mismatch=True, max_history_size=10000, max_trace_size=10000, and interpreter_factory=None. A handful of keys are internal and you should never set them: caller_predict, caller_modules, usage_tracker and trace.

Now the rule that produces the most confusing production bug in DSPy. dspy.configure may only be called from the thread, or async task, that first called it. Anywhere else you get:

RuntimeError: dspy.settings can only be changed by the thread that initially configured it.

or, in an async task:

dspy.configure(...) can only be called from the same async task that called it first.
Please use `dspy.context(...)` in other async tasks instead.

The fix is always dspy.context, which is ContextVar-scoped and therefore safe in threads and async tasks alike. This is why a FastAPI handler must never call configure — it runs per-request, often on a different task — and must use with dspy.context(...) instead. Configure once at startup; context everywhere else.

The same rule has a quieter half. dspy.Parallel, Module.batch and dspy.asyncify copy your context overrides into their workers; a plain threading.Thread or ThreadPoolExecutor does not, so hand-rolled workers silently run with process defaults and ignore the dspy.context block you are standing inside. Either use DSPy's primitives, or capture and re-set the overrides:

import dspy
from concurrent.futures import ThreadPoolExecutor

overrides = dspy.settings.thread_local_overrides.get()

def work(item):
    token = dspy.settings.thread_local_overrides.set(overrides)
    try:
        return program(**item)
    finally:
        dspy.settings.thread_local_overrides.reset(token)

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(work, items))

For a multi-tenant service, dspy.context is the isolation mechanism but not a complete one. The response cache is keyed on the request, not the tenant, so two tenants sending identical text share a cached answer. Where that is unacceptable — and under Saudi and UAE expectations for customer content it often is — set cache=False on tenant LMs or give each tenant a distinct rollout_id. Also, lm.history and the global history hold prompts in memory, so an untrimmed history in a long-lived process is a copy of every prompt you have handled: set disable_history=True or lower max_history_size.

Reasoning models carry their own parameter constraints, enforced at construction:

LMConfigurationError: OpenAI's reasoning models require passing temperature=1.0 or None
and max_tokens >= 16000 or None to `dspy.LM(...)`, e.g.,
dspy.LM('openai/gpt-5', temperature=1.0, max_tokens=16000)

strong = dspy.LM("openai/gpt-5-mini", model_type="responses", temperature=1.0, max_tokens=16000)

Data and metrics an optimiser can learn from

The problem: an optimiser is a search procedure, and a search is only as good as its objective. Most disappointing DSPy results are metric problems wearing an optimiser costume.

Start with the data split: three sets, not one. A trainset the optimiser may bootstrap demos from, a valset it may score candidates against, and a test set it never touches, which is the only number you may quote. Skipping the third is how teams report a score that does not survive production.

import dspy, random

examples = [
    dspy.Example(ticket=t, category=c, reply=r).with_inputs("ticket")
    for t, c, r in rows
]
random.Random(0).shuffle(examples)
trainset, valset, testset = examples[:150], examples[150:250], examples[250:]

with_inputs("ticket") marks which keys are inputs; everything else is treated as a label. Get that wrong and the optimiser will happily feed the answer in as an input, score perfectly, and collapse in production. A fixed seed matters more than it looks: without one, "the optimiser improved things" and "the split changed" are indistinguishable.

Now the metric. Its full signature is metric(example, pred, trace=None) -> float | bool. The trace argument is the part people miss. It is None during plain evaluation and not None during bootstrapping, when the optimiser is deciding whether a generated trace is good enough to keep as a demonstration. That gives you one function with two personalities: a graded score for ranking, and a strict pass/fail gate for demo selection.

import dspy

judge = dspy.ChainOfThought("question, reference, answer -> correct: bool, reason: str")

def quality(example, pred, trace=None):
    if pred.category != example.category:
        return False
    with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", temperature=0.0)):
        verdict = judge(question=example.ticket, reference=example.reply, answer=pred.reply)
    if trace is not None:
        return bool(verdict.correct)       # strict gate while bootstrapping demos
    return 1.0 if verdict.correct else 0.3  # graded signal while ranking candidates

Three properties separate a metric that drives learning from one that does not. Discriminative: one returning 1.0 for 95% of examples has nothing left to optimise. Cheap enough to run thousands of times, because an optimiser runs it across the set repeatedly — an LM judge on a strong model can cost more than the compile, which is why the judge above is pinned to a cheap model inside dspy.context. And deterministic, so set temperature=0.0 on any judge; a noisy metric makes the optimiser chase randomness.

DSPy ships metrics worth reaching for before you write your own: dspy.evaluate.answer_exact_match, answer_passage_match, SemanticF1, and CompleteAndGrounded — the last two being the right starting point for retrieval-augmented answers, where exact match is meaningless. If you need a broader battery of RAG-specific scores, the RAGAS guide covers that ground and composes cleanly with a DSPy program as the thing under test.

GEPA metrics are a special, more powerful case. They additionally accept pred_name and pred_trace (and optionally program_trace), and they may return a dspy.Prediction(score=..., feedback="..."). That feedback string is natural-language guidance handed to the reflection model, and it is the single highest-leverage thing you can write when using GEPA:

import dspy

def feedback_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    if pred.category == gold.category:
        return dspy.Prediction(score=1.0, feedback="Category correct.")
    return dspy.Prediction(
        score=0.0,
        feedback=(f"Predicted {pred.category!r} but the ticket is {gold.category!r}. "
                  "Billing tickets mention charges, refunds or invoices; "
                  "access tickets mention logins, passwords or permissions."),
    )

Generic feedback ("wrong answer") teaches the reflection model nothing. Feedback that names the distinguishing feature is what makes GEPA worth its cost.

Evaluation you can trust

The problem: a single number from a single run tells you almost nothing, and teams routinely ship regressions because 72 looked better than 69 on a 40-example dev set.

import dspy

evaluate = dspy.Evaluate(
    devset=testset,
    metric=quality,
    num_threads=16,
    display_progress=True,
)
result = evaluate(program)
print(result.score)            # a percentage, 0-100, rounded to 2 dp
worst = sorted(result.results, key=lambda r: r[2])[:10]

Two API facts to internalise. result.score is a percentage, not a fraction — a mean of 0.72 prints as 72.0, and mixing the two up is how a dashboard ends up off by a factor of a hundred. And result.results is the per-example list of (example, prediction, score) tuples. The old return_outputs=True argument is gone and raises:

ValueError: `return_outputs` is no longer supported. Results are always returned inside
the `results` field of the `EvaluationResult` object.

Since 3.4, an empty devset also fails fast rather than reporting a meaningless zero:

ValueError: devset must contain at least one example, got an empty devset.

That results list is the most useful part and the most ignored. Reading the ten worst cases beats staring at the aggregate, because it tells you whether you have a systematic failure on one input category — a signature or routing problem — or uniform mediocrity, which is a model or demo problem.

On the mechanics: num_threads governs concurrency and your ceiling is the provider's rate limit, not your CPU. Push it too high and you trade throughput for dspy.LMRateLimitError, which carries .retry_after and .provider and only surfaces after num_retries attempts (default 3). max_errors (default 10) caps tolerated per-example failures before the run aborts — raise it for an exploratory sweep, keep it low for a CI gate.

Variance deserves its own paragraph. On a fifty-example set, a five-point swing is noise. Size the set so a difference you care about exceeds the run-to-run spread — a hundred examples is a reasonable floor for a go/no-go decision. Keep the cache on during comparisons. And when the program is non-deterministic, run it several times with distinct rollout_id values and compare means.

scores = []
for i in range(5):
    with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", temperature=1.0)):
        scores.append(evaluate(program).score)
print(sum(scores) / len(scores), max(scores) - min(scores))

If the spread is wider than the improvement you are claiming, you have not measured an improvement.

Choosing and running an optimiser

The problem: there are more than a dozen optimisers, they differ by orders of magnitude in cost, and the expensive ones can run into hundreds of dollars of LM calls. Picking by name recognition is how budgets disappear.

All of them share one interface. .compile(student, trainset=..., [valset=...]) returns a new optimised program; the original is not mutated. Compiled modules are flagged _compiled=True so an outer optimiser skips them.

The official decision path, compressed:

Start at the top and move down only when a measurement tells you to. BootstrapFewShot costs a handful of runs over your trainset and often captures most of the available gain, because the gap between zero-shot and four good demonstrations is usually larger than anything a search adds on top.

Some optimisers now need extras, which is a change people hit on upgrade: MIPROv2 and BootstrapFewShotWithOptuna need pip install "dspy[optuna]"; KNNFewShot, SIMBA and the embedding paths need pip install "dspy[numpy]". A bare ModuleNotFoundError from inside an optimiser almost always means a missing extra, not a broken install.

import dspy

tp = dspy.MIPROv2(metric=quality, auto="light")   # auto in {"light","medium","heavy",None}
optimised = tp.compile(program, trainset=trainset, valset=valset)
print(evaluate(optimised).score)
optimised.save("artifacts/support-v3.json")

A typo in auto fails immediately rather than wasting a run:

ValueError: Invalid value for auto: ... Must be one of {None, 'light', 'medium', 'heavy'}.

GEPA is the reflective optimiser and the one most worth understanding properly, because it rewrites instructions using a second model that reads your metric's feedback. Its budget is explicit and exactly one of three arguments must be set:

import dspy

gepa = dspy.GEPA(
    metric=feedback_metric,
    auto="light",
    reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000),
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",
    num_threads=8,
    track_stats=True,
    log_dir="artifacts/gepa-run-3",
)
optimised = gepa.compile(program, trainset=trainset, valset=valset)

AssertionError: Exactly one of max_metric_calls, max_full_evals, auto must be set. You set ...

The reflection_lm should be a strong model; it is called a small number of times to write instructions that then run on your cheap task model forever. That asymmetry — expensive at compile time, cheap at serve time — is the core economics of DSPy optimisation. Since 3.3 you cannot pass reflection_prompt_template through gepa_kwargs (it raises ValueError); use instruction_proposer, or code_proposer in 3.4. GEPA result shapes also changed in 3.3: DspyGEPAResult.candidates are compiled modules and best_candidate is a module, not a state dict. max_reflection_cost is unsupported and raises.

Three operational rules for any compile. Set log_dir so a run that cost real money leaves an audit trail. Cap the budget before you start — auto="light" or an explicit max_metric_calls — because an unbounded sweep against a paid API is the most common way a DSPy project embarrasses itself. And always score the result on the untouched test set, not on the valset the optimiser was allowed to see.

Saving, versioning and loading an optimised program

The problem: an optimised program is two things — your Python code and a learned state of instructions and demonstrations. Deploying one without the matching other produces a program that runs and quietly performs like an unoptimised one.

There are three save modes and the choice has security consequences:

# 1. State only: JSON. Recommended.
program.save("artifacts/support-v3.json")
restored = Pipeline()
restored.load("artifacts/support-v3.json")

# 2. State via cloudpickle
program.save("artifacts/support-v3.pkl")
restored = Pipeline()
restored.load("artifacts/support-v3.pkl", allow_pickle=True)

# 3. Whole program, including the class
program.save("artifacts/support-v3/", save_program=True, modules_to_serialize=[my_helper])
restored = dspy.load("artifacts/support-v3/", allow_pickle=True)

Mode one is the default answer. The JSON holds instructions, demos, signature field metadata and the LM configuration minus the api_key, plus a metadata block recording the Python, DSPy and cloudpickle versions. It is diffable, reviewable in a pull request, and safe to load from a build artefact. Mode three writes program.pkl plus metadata.json into a directory and is for when the structure cannot be reconstructed from code — but loading it executes arbitrary code, which is why the flag exists:

ValueError: Loading with pickle is not allowed. Please set `allow_pickle=True` if you are
sure you trust the source of the model.

Path shape is validated, so these two are worth recognising rather than debugging: path must end with .json or .pkl when save_program=False, and must be a suffix-less directory when save_program=True. A save that fails with RuntimeError: Failed to save state to ... Your DSPy program may contain non json-serializable objects means something in your state is not JSON — move to .pkl or save_program=True, or better, keep the unserialisable object out of the module's state.

Loading has a deliberate safety behaviour worth knowing because it occasionally looks like a bug. By default (allow_unsafe_lm_state=False) load strips api_base, base_url and model_list from the saved LM config, and api_key is never saved at all. The reasoning is direct: a saved artefact must not be able to redirect your traffic to another endpoint. So if you save a program pointed at a self-hosted endpoint and load it expecting that endpoint, you will get your configured default instead. Configure the LM at load time in your serving code; treat the artefact as prompts and demos only. Custom-engine state additionally requires allow_unsafe_lm_state=True. load_state is transactional, so a failed load leaves the program unchanged rather than half-populated.

Version-skew handling is lenient by design: a mismatch between saved and running versions logs a warning only, and a warning in a container log is a warning nobody reads. Treat the artefact as part of your release: pin dspy==3.4.0 in the image that produced it and the one that serves it, and record the DSPy version, model string and test-set score beside the file. Old pickles containing the removed 3.3 experimental LM types are not migrated, so a 3.3 pickle may simply fail to load.

Serving a DSPy program

The problem: DSPy is an in-process Python library, not a server. There is no dspy serve. The production shape is therefore always the same — compile offline, version the artefact, and load it inside a process you operate.

The documented FastAPI pattern:

import dspy
from fastapi import FastAPI
from pydantic import BaseModel

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"), track_usage=True)

program = Pipeline()
program.load("artifacts/support-v3.json")
served = dspy.asyncify(program)

app = FastAPI()

class Ask(BaseModel):
    ticket: str

@app.post("/reply")
async def reply(body: Ask):
    pred = await served(ticket=body.ticket)
    return {"category": pred.category, "reply": pred.reply, "usage": pred.get_lm_usage()}

dspy.asyncify runs a synchronous program in worker threads so it does not block the event loop. The cap is async_max_workers, default 8 — the ninth concurrent call waits. That is the single most common cause of "our DSPy service has terrible tail latency under load": not the model, a queue of depth eight in your own process. Raise it to match your provider concurrency, and remember it is per process, so your real ceiling is async_max_workers × replicas.

If your program is async all the way down, skip asyncify and implement aforward:

import dspy

class AsyncPipeline(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classify = dspy.Predict("ticket -> category: str")
        self.reply = dspy.ChainOfThought("ticket, category -> reply: str")

    async def aforward(self, ticket):
        cat = (await self.classify.acall(ticket=ticket)).category
        return await self.reply.acall(ticket=ticket, category=cat)

Most built-in modules support acall, and 3.4 added async ReActV2 via await agent.acall(...), though its tools still run sequentially. One 3.4 breaking change to watch: calling an async tool synchronously inside a running event loop now raises ValueError. Use await tool.acall(...).

Streaming is the third piece, because a user watching a blank box for eight seconds is a product problem regardless of your p50:

import dspy
from dspy.utils.streaming import streaming_response

stream_program = dspy.streamify(
    program,
    stream_listeners=[dspy.streaming.StreamListener(signature_field_name="reply")],
)

@app.post("/reply/stream")
async def reply_stream(body: Ask):
    return streaming_response(stream_program(ticket=body.ticket))

The listener names which output field to stream. Remember the engine behaviour from earlier: once a stream has emitted a chunk it is not retried, and incomplete responses are not cached. A streaming endpoint needs its own client-side handling of a truncated answer.

The alternative to rolling your own is MLflow, which can log and serve the program directly:

import mlflow

mlflow.dspy.log_model(program, "dspy_program", input_example={"ticket": "..."}, task="llm/v1/chat")

mlflow models serve -m runs:/<run_id>/model -p 6000
mlflow models build-docker -m "runs:/<run_id>/model" -n "dspy-program"

This needs mlflow>=2.18.0, and the DSPy docs note that as of MLflow 2.22 you must wrap built-in modules in a custom dspy.Module because MLflow passes positional arguments. The honest trade-off: MLflow gives you the registry, the tracing and the packaging for free, and takes away control of the serving process. A FastAPI container gives you full control and makes the versioning your problem. For teams already running MLflow for classical models, logging DSPy programs into the same registry is usually the right call; see the MLflow guide for the registry mechanics.

Two container details matter. The disk cache lives at ~/.dspy_cache with a 30 GB default limit, which a read-only root filesystem rejects outright — set DSPY_CACHEDIR to a writable path, or disable it with dspy.configure_cache(enable_disk_cache=False). And if data residency drives you to a Gulf or Egyptian region, the LM endpoint determines where prompts are processed: use the vertex_ai/ prefix with vertex_project and vertex_location rather than gemini/ with project/location, because the wrong kwargs send requests to the wrong region silently.

Caching, usage and cost

The problem: LLM calls are the dominant cost and latency in a DSPy program, and DSPy caches aggressively by default — which is excellent for development and occasionally wrong in production.

There are three independent cache layers. An in-memory cachetools.LRUCache, defaulting to 1,000,000 entries. An on-disk diskcache.FanoutCache at ~/.dspy_cache, defaulting to 30 GB (DSPY_CACHEDIR and DSPY_CACHE_LIMIT override both). And the provider's own prompt cache, which is a different thing entirely and is controlled separately. The DSPy cache key is a hash of the request arguments, excluding credentials.

import dspy

dspy.configure_cache(
    enable_disk_cache=True,
    enable_memory_cache=True,
    memory_max_entries=200_000,
    restrict_pickle=True,
)

lm = dspy.LM("openai/gpt-4o-mini", cache=False)      # per-LM opt out
fresh = lm("summarise this", rollout_id=1, temperature=1.0)  # bypass, keep caching enabled

"Why do I get the same answer every time?" is nearly always the cache, not the model. rollout_id with a non-zero temperature is the clean way to force variation while leaving caching in place for everything else — this is what you want for self-consistency sampling and for variance measurement.

restrict_pickle=True is worth enabling in production: the disk cache uses pickle, and restricting what it will deserialise closes a real hole when the cache directory is shared or persisted. The maintainers have said it will become the default later. The cost is a warning when a type is rejected:

WARNING dspy.clients.cache: Failed to deserialize disk cache entry <key>

Register legitimate types with safe_types=[...] rather than turning the restriction back off.

Provider prompt caching is the second lever and is often the bigger win, because it discounts the shared prefix of your prompts — instructions and demos, which after optimisation can be most of your input tokens:

import dspy

native = dspy.LM("anthropic/claude-sonnet-4-5-20250929", engine="lm15",
                 prompt_cache=dspy.lm15.CacheConfig(prefix="stable"))

via_litellm = dspy.LM("anthropic/claude-sonnet-4-5-20250929", engine="litellm",
                      cache_control_injection_points=[{"location": "message", "role": "system"}])

The configuration differs by engine, so this is one place where forcing an engine explicitly is justified. The DSPy docs note that ReActV2's history handling improves prefix reuse, citing up to 50% savings in internal tests.

Measurement closes the loop:

import dspy

dspy.configure(track_usage=True)
pred = program(ticket="card declined twice")
print(pred.get_lm_usage())

with dspy.track_usage() as usage:
    for ex in testset[:20]:
        program(**ex.inputs())
print(usage)

Cached calls report empty usage, which is correct but means a cached evaluation run looks free. lm.history[i]["cost"] gives per-call cost where the provider reports it, and may legitimately be unknown — never coerce that to zero.

The cost levers in order of leverage: a cheap task model with a strong one only at compile time or for the step that needs it; provider prompt caching; capped optimiser budgets; a lower max_bootstrapped_demos, since every demo is paid for on every request forever; and the DSPy cache left on in development. For centralised budgeting and key rotation across teams, a proxy in front of DSPy is the usual answer — see the LiteLLM guide, the same library DSPy's compatibility engine uses.

Observability and debugging beyond inspect_history

The problem: dspy.inspect_history(n=3) is perfect at your terminal and useless in production. You cannot grep a notebook for yesterday's p99.

Three mechanisms, in increasing order of production fitness.

History. dspy.inspect_history(n) prints the global record of LM calls; lm.history[-1] is the structured version, with keys including prompt, messages, kwargs, response, outputs, usage, cost, timestamp, uuid and model. It answers "what did DSPy actually send?", the right first question for almost every quality complaint, because an optimiser's prompt is rarely what you imagine.

Callbacks. Subclass dspy.utils.callback.BaseCallback and register it process-wide. Handlers cover on_module_start/end, on_lm_start/end, on_adapter_format_start/end, on_adapter_parse_start/end, on_tool_start/end, on_evaluate_start/end, on_compile_start/end (added in 3.3.1), and interpreter lifecycle events.

import time, logging
import dspy
from dspy.utils.callback import BaseCallback

log = logging.getLogger("dspy.metrics")

class Timing(BaseCallback):
    def __init__(self):
        self.started = {}

    def on_lm_start(self, call_id, instance, inputs):
        self.started[call_id] = time.perf_counter()

    def on_lm_end(self, call_id, outputs, exception):
        ms = (time.perf_counter() - self.started.pop(call_id, time.perf_counter())) * 1000
        log.info("lm_call ms=%.0f error=%s", ms, exception)

dspy.configure(callbacks=[Timing()])

Two rules: callbacks are process-wide, so in a multi-tenant service do not put tenant state in them; and do not mutate inputs or outputs in place, because you are sitting on the hot path of every call.

Tracing. MLflow Tracing is the official integration and is OpenTelemetry-based, which means the spans land somewhere you already have:

import mlflow

mlflow.dspy.autolog()

mlflow server --backend-store-uri sqlite:///mydb.sqlite

One line gives you a span tree per request: nested module calls, each LM call with its prompt, response and token counts, and adapter parse steps as their own spans. That nesting is what makes a slow multi-step program diagnosable — one slow predictor, a ReAct loop that ran nineteen iterations, or a retrieval step that was never the suspect. MLflow also covers optimiser tracking, so a compile becomes a run with score history attached.

Because the tracing is OpenTelemetry-based it composes with the rest of your stack; if your traces already go to a collector, see the OpenTelemetry guide. Third-party integrations exist too — Langfuse, Arize Phoenix and Weave — maintained outside the DSPy docs, so check their own 3.4 compatibility notes.

The error taxonomy is the fourth piece of debugging, and it changed in 3.3 in a way that affects every try/except you have written. LM errors are now normalised to DSPy exceptions, so catching LiteLLM or provider exceptions no longer works:

import dspy

try:
    pred = program(ticket=text)
except dspy.LMRateLimitError as e:
    retry_after = e.retry_after
except dspy.ContextWindowExceededError:
    pred = program(ticket=text[:4000])
except dspy.LMError as e:
    log.error("lm failure code=%s model=%s status=%s request_id=%s",
              e.code, e.model, e.status, e.request_id)
    if dspy.is_retryable_lm_error(e):
        raise

dspy.LMError is the base and carries .code, .model, .provider, .status, .request_id and .retry_after. The subclasses are LMRateLimitError, LMAuthError, LMTimeoutError, LMServerError, LMInvalidRequestError, LMUnsupportedModelError, LMBillingError, LMTransportError, LMUnexpectedError, LMNotConfiguredError, plus ContextWindowExceededError and the helper dspy.is_retryable_lm_error(e). ContextWindowExceededError deserves special mention because it is the characteristic failure of a successful optimisation: more demos plus a longer retrieval context eventually overflows. ReAct truncates its trajectory before re-raising, so a long agent loop degrades before it dies.

Two more switches: dspy.enable_logging() / dspy.disable_logging() control DSPy's own logger, and dspy.configure(provide_traceback=True) makes per-example failures inside Evaluate and Parallel print full tracebacks rather than a one-line summary.

Testing a DSPy program in CI

The problem: the output is non-deterministic and the dependency is a paid network service. Both of those are solvable, and a DSPy project without tests regresses on every prompt edit.

Split your tests into three layers.

Layer one: no LM at all. Most of your code is structure, and structure is testable offline: field names and types, module composition, named_predictors() counts, that program.load() restores the demos you expect, and that your metric behaves on hand-written pairs. These run in milliseconds and catch most refactoring mistakes.

import dspy
from pipeline import Pipeline, quality

def test_predictor_count_is_stable():
    assert [n for n, _ in Pipeline().named_predictors()] == ["classify", "reply.predict"]

def test_metric_is_strict_while_bootstrapping():
    gold = dspy.Example(ticket="t", category="billing", reply="r").with_inputs("ticket")
    pred = dspy.Prediction(category="access", reply="r")
    assert quality(gold, pred, trace=object()) is False

def test_optimised_artifact_has_demos():
    p = Pipeline()
    p.load("artifacts/support-v3.json")
    assert any(pred.demos for _, pred in p.named_predictors())

Layer two: a fake LM. Replace the engine and your program becomes deterministic. A custom engine is an object with complete(request) -> Response, which is exactly what you want for a stub:

import dspy

class ScriptedEngine:
    """Implements the 3.4 engine protocol: complete(request) -> dspy.lm15.Response."""

    def __init__(self, make_response):
        self.make_response = make_response
        self.calls = []

    def complete(self, request):
        self.calls.append(request)
        return self.make_response(request)

def scripted(make_response):
    return dspy.LM("custom/scripted", engine=ScriptedEngine(make_response))

Build the dspy.lm15.Response your make_response returns following the shape in the custom-LM-engines tutorial; the engine protocol itself is just that one method (plus an async counterpart passed as async_engine=).

This gives you fast, repeatable tests of the parts that are genuinely yours: that the adapter parses your format, that forward routes correctly, that an error path is handled. Use dspy.context(lm=scripted(...)) inside the test rather than mutating global settings, which would also trip the configure-owner-thread rule under pytest-xdist. With a custom engine, credentials and timeouts live on the engine, not on dspy.LM.

Layer three: a scored evaluation gate. This one does call the real provider, so it is the job you run on a schedule or on a label, not on every commit. Assert on a threshold relative to your recorded baseline and your measured noise floor, never on exact equality:

import dspy

BASELINE, NOISE = 72.0, 3.0

def test_no_regression(program, testset):
    score = dspy.Evaluate(devset=testset, metric=quality, num_threads=8)(program).score
    assert score >= BASELINE - NOISE, f"regression: {score} vs {BASELINE}"

Three practices make layer three affordable. Keep the test set small but fixed — the same hundred examples, committed, so a score change means a program change. Cache ~/.dspy_cache between CI runs, which turns most of the evaluation into a free replay. And run with dspy.ChatAdapter(use_json_adapter_fallback=False) so a format regression fails the build.

Add one job that costs nothing: python -W default::DeprecationWarning -m pytest tells you when a dependency bump has put you on a path 3.5 removes. Whatever runner you use — GitHub Actions is the common choice — the shape is the same: structural tests on every push, scored evaluation nightly, deprecation warnings on the pull request.

Putting it all together

Here is the whole shape in one place: a support-triage program with two predictors on two different models, a feedback-bearing metric, a capped compile, a versioned artefact and a served endpoint.

import dspy
from typing import Literal

Category = Literal["billing", "access", "bug", "other"]

class Triage(dspy.Signature):
    """Classify a customer support ticket into exactly one category."""
    ticket: str = dspy.InputField()
    category: Category = dspy.OutputField()

class Reply(dspy.Signature):
    """Write a short, specific reply to the ticket. Do not invent policy."""
    ticket: str = dspy.InputField()
    category: Category = dspy.InputField()
    reply: str = dspy.OutputField(desc="under 80 words, no greeting")

class Support(dspy.Module):
    def __init__(self):
        super().__init__()
        self.triage = dspy.Predict(Triage)
        self.compose = dspy.ChainOfThought(Reply)

    def forward(self, ticket: str):
        category = self.triage(ticket=ticket).category
        out = self.compose(ticket=ticket, category=category)
        return dspy.Prediction(category=category, reply=out.reply, reasoning=out.reasoning)

def feedback_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    if pred.category != gold.category:
        return dspy.Prediction(
            score=0.0,
            feedback=(f"Category {pred.category!r} is wrong; it is {gold.category!r}. "
                      "Billing mentions charges or refunds; access mentions logins."),
        )
    if trace is not None:
        return True
    return dspy.Prediction(score=1.0, feedback="Category correct.")

import dspy
from pipeline import Support, feedback_metric
from data import trainset, valset, testset

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", temperature=0.0), track_usage=True)

program = Support()
program.compose.predict.set_lm(dspy.LM("openai/gpt-4o", temperature=0.0))

evaluate = dspy.Evaluate(devset=testset, metric=feedback_metric, num_threads=16,
                         display_progress=True)
print("baseline", evaluate(program).score)

gepa = dspy.GEPA(
    metric=feedback_metric,
    auto="light",
    reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000),
    num_threads=8,
    track_stats=True,
    log_dir="artifacts/gepa-2026-10-08",
)
optimised = gepa.compile(program, trainset=trainset, valset=valset)

print("optimised", evaluate(optimised).score)
optimised.save("artifacts/support-2026-10-08.json")

import dspy
from fastapi import FastAPI
from pydantic import BaseModel
from pipeline import Support

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", temperature=0.0),
               track_usage=True, async_max_workers=32, disable_history=True)
dspy.configure_cache(enable_disk_cache=False, restrict_pickle=True)

program = Support()
program.load("artifacts/support-2026-10-08.json")
program.compose.predict.set_lm(dspy.LM("openai/gpt-4o", temperature=0.0))
served = dspy.asyncify(program)

app = FastAPI()

class Ask(BaseModel):
    ticket: str

@app.post("/triage")
async def triage(body: Ask):
    try:
        pred = await served(ticket=body.ticket)
    except dspy.ContextWindowExceededError:
        pred = await served(ticket=body.ticket[:4000])
    return {"category": pred.category, "reply": pred.reply, "usage": pred.get_lm_usage()}

Read what each decision buys. Two models, because triage is easy and composition is not. A metric with feedback, because GEPA's reflection model is only as good as what you tell it. auto="light" and a log_dir, because an uncapped compile is an uncapped invoice and an unlogged one is unreproducible. A date-stamped artefact, because a prompt with no provenance is not an artefact. set_lm repeated in serve.py, because loading strips endpoint details on purpose. async_max_workers=32, because the default eight is a queue. disable_history=True and no disk cache, because a container holding every prompt in memory and writing to a read-only filesystem are both avoidable. And a ContextWindowExceededError branch, because that is the failure an optimised program grows into.

Where you are now

You can trace a call through signature, module, adapter, engine and prediction, and name which stage owns a symptom. You can choose an adapter on token cost and strictness rather than habit, and read an AdapterParseError as the truncation it usually is. You know what 3.4's engines changed, which APIs 3.5 removes, and how to find your own deprecations in one command. You can isolate per-request state with dspy.context and explain why plain threads lose it. You can build a three-way split, write a metric with a bootstrapping gate and GEPA feedback, and measure a noise floor before claiming an improvement. You can pick an optimiser on budget, cap it, log it, and version its output as a real artefact. You can serve it behind FastAPI with sane concurrency, stream a field, track usage, catch the right exceptions and see the spans.

Senior takes every one of these further, with an ownership dimension. The architecture as a platform: who owns the compile budget, who signs off an artefact, and what the promotion path looks like. Which component breaks first under load, and why it is almost never DSPy. The trust model in full — pickle in the disk cache, LocalInterpreter being explicitly not a sandbox, from_url() having no SSRF allowlist, prompt injection through tool output and retrieved text. Multi-tenancy without a built-in tenant model. Custom engines as a first-class integration surface. Upgrade and migration planning across a fleet of compiled artefacts. RLM and sandboxed code execution. Cost governance when several teams share a provider quota. And where DSPy stops, and a different tool starts.

Sources

• DSPy documentation
• Language models
• Signatures
• Modules
• Adapters and adapters in depth
• Metrics
• Optimizers and choosing an optimizer
• GEPA in depth
• Settings and context
• Saving and loading
• Normalized LM API migration
• Custom LM engines
• Caching
• Async and streaming
• Deployment
• Observability and optimizer tracking
• Production
• Cheatsheet
• DSPy 3.4.0 release notes
• DSPy 3.3.1 release notes
• DSPy 3.3.0 release notes
• dspy on PyPI
• MLflow DSPy integration

At this level the questions stop being "what is a signature?" and start being "your optimised program regressed after a model swap — what do you check?" Interviewers are testing whether you have operated DSPy, not read about it. The pattern below is the same across every answer: name the mechanism, name the measurement, then name the fix. Everything here is current for DSPy 3.4.

How mid-level DSPy questions are graded

A weak answer names a feature. A strong answer names a mechanism and the observation that distinguishes it from the alternatives.

Asked "how do you make a DSPy program better?", a weak answer says "run MIPROv2". A strong one says: "First I check whether I have a baseline on a held-out test set and whether my metric is discriminative, because an optimiser cannot improve what the metric cannot see. Then BootstrapFewShot, which is cheap and usually captures most of the gain. Only if that plateaus do I spend on MIPROv2 or GEPA with a capped budget." That answer demonstrates cost awareness, measurement discipline and an ordering, which is what the question is actually probing.

Three habits separate strong candidates. They quote a noise floor — "on my hundred-example test set run-to-run spread is about three points, so I don't act on a two-point change." They distinguish compile-time from serve-time, because almost every DSPy cost and reliability question resolves on that axis. And they name exceptions precisely: dspy.LMError and its subclasses, not "the OpenAI error".

Scenario: the program is slow in production

"Your DSPy service has a p50 of 2 seconds but a p99 of 40 seconds under load. Where do you look?"

Start by stating that DSPy itself is almost never the latency. It is an in-process library; the time is in LM calls, in your own queueing, or in the number of calls per request.

Check in this order. How many LM calls does one request make? Print named_predictors() and count, then remember that ChainOfThought is one call, ReAct is a loop with max_iters=20 by default, and a TwoStepAdapter doubles every prediction. A p99 that is twenty times the p50 is the shape of a loop that usually terminates in two iterations and occasionally runs to its cap.

Then check async_max_workers. If you are serving with dspy.asyncify, the default is 8 per process. The ninth concurrent request waits, and the wait compounds, which produces exactly this p50/p99 split. Raising it and re-measuring is the single highest-yield check in this scenario and the one most candidates miss.

Then the prompt size. An optimised program carries demos on every request. Four long bootstrapped demos plus a retrieval context can triple your input tokens, and time-to-first-token scales with input length. track_usage=True plus pred.get_lm_usage() tells you the split between demo tokens and real input.

Then retries. num_retries defaults to 3. A provider that is soft-failing turns one slow call into four, and the symptom is a long tail with no corresponding error rate. dspy.LMRateLimitError carries .retry_after, so your logs should show whether you are being throttled.

Finally, turn on mlflow.dspy.autolog() and read one slow trace. The span tree tells you in seconds what the above tells you in twenty minutes: whether it is one slow predictor, a long loop, or queueing in your own process.

A strong answer ends with the mitigation menu and its costs: cut max_bootstrapped_demos, enable provider prompt caching to discount the stable prefix, lower max_iters, switch the easy predictor to a cheaper and faster model with set_lm, and raise async_max_workers to match your provider concurrency.

Scenario: the optimised program got worse

"You compiled with GEPA, the score went from 71 to 84, you deployed, and quality complaints went up. What happened?"

This is a measurement-integrity question and there are four standard answers. Say that you check them in this order because each is cheaper to rule out than the last.

You reported the valset score. The optimiser was allowed to see the valset while selecting candidates, so that number is contaminated by construction. Only a test set the optimiser never touched is quotable. If the 84 came from the valset, there may be no real improvement at all.

Your label leaked into the inputs. If with_inputs() names a field that is actually a label, the program is being handed the answer during evaluation and not in production. This produces exactly this symptom — a spectacular offline score and a flat online one.

The artefact and the code drifted. program.load() restores instructions and demos but a version mismatch between the saving and running DSPy only logs a warning. If the serving image is a different DSPy version, or the class changed shape since the compile, you can be serving something that is not what you measured.

The endpoint silently changed. By default load() strips api_base, base_url and model_list from the saved LM config, and api_key is never saved, deliberately, so an artefact cannot redirect traffic. If you compiled against a self-hosted endpoint and expected the artefact to carry it, production is running against your configured default model instead. Configure the LM explicitly in the serving code.

A strong answer adds the preventative: record provenance beside every artefact — DSPy version, model string, data snapshot, test-set score — and assert the score in a nightly job against a threshold set to baseline minus your measured noise, so a drift is caught by CI rather than by customers.

Scenario: AdapterParseError in production only

"It works in development and raises AdapterParseError in production a few per cent of the time. Why?"

Name the layer first: the adapter is the component that formats the prompt and parses the response into your declared fields. The error means it could not find the output fields it expected.

Then give the ranked causes. Truncation is the most common: max_tokens is too low for the reasoning plus every output field, so the response stops mid-field. The reason this appears in production and not development is that production inputs are longer, so the reasoning is longer, so the output gets cut. An empty response, usually a content filter on real user text that your curated dev set never triggers. Model drift, where a provider rolls a model version and format compliance changes under you — a reason to pin model versions. A schema that is too much for the model, which is load-independent but input-length dependent.

The best answer makes a point about why the rate is only a few per cent: ChatAdapter falls back to JSONAdapter when a parse fails, so most format drift is invisible. What reaches you as an error is the residue after that fallback also failed. The diagnostic move is to construct dspy.ChatAdapter(use_json_adapter_fallback=False) in your test suite, so format drift fails loudly where you can see it, while leaving the fallback on in production where you want the resilience.

Then the fixes, in order: raise max_tokens; inspect dspy.inspect_history() on a failing case to see what came back; switch to JSONAdapter for provider-side schema enforcement; use TwoStepAdapter with a cheap extraction model if the task model answers well but formats badly; and give genuinely optional output fields a declared default, since since 3.3.1 adapters apply the default for a missing optional field while a missing required field still raises.

Scenario: the worker threads ignore my model

"A batch job wraps program(...) in a ThreadPoolExecutor inside a with dspy.context(lm=big_model) block, and every call goes to the default model. Explain."

This question separates people who have read the settings page from people who have debugged it.

DSPy settings have two layers. dspy.configure sets process-wide defaults and may only be called from the thread or async task that first called it; anywhere else you get RuntimeError: dspy.settings can only be changed by the thread that initially configured it. dspy.context sets ContextVar-scoped overrides and is safe in threads and async tasks.

The subtlety is propagation. dspy.Parallel, Module.batch and dspy.asyncify copy the current overrides into their workers. A plain threading.Thread or ThreadPoolExecutor does not, so the workers see only the process defaults and the surrounding dspy.context has no effect on them. That is the whole bug.

Two fixes. Use DSPy's concurrency — dspy.Parallel(num_threads=8) or program.batch(examples, num_threads=8). Or capture dspy.settings.thread_local_overrides.get() in the parent and set() it at the top of each worker.

Follow-ups to be ready for: why a FastAPI handler must never call configure (it runs per request, often on a different task, and will raise), and how you isolate per-tenant models (dspy.context per request, or set_lm per predictor — with the caveat that the response cache is keyed on the request rather than the tenant, so identical text from two tenants shares a cached answer unless you set cache=False or distinct rollout_id values).

Trade-off: which optimiser, and why not the best one

"Why wouldn't you just always run the strongest optimiser?"

Because compile cost is real money and the gains are not monotonic in spend. GEPA and MIPROv2 runs can cost hundreds of dollars in LM calls, and a strong reflection_lm is deliberately an expensive model.

Give the ordering and the condition that moves you down it. BootstrapFewShot when you have no baseline yet, because it is cheap and the jump from zero-shot to four good demonstrations is usually the largest single gain available. BootstrapRS when demo quality swings between runs. COPRO or GEPA when the diagnosis is that the instructions are wrong rather than the examples — you can tell by reading inspect_history and seeing a well-demonstrated task described badly. MIPROv2 when both are weak and you have budget. BootstrapFinetune only when prompt optimisation has plateaued and the model is actually tunable. BetterTogether to chain prompt and weight optimisation, with strategy="p -> w -> p".

Then mention the operational constraints, which is where experience shows. Exactly one of auto, max_full_evals or max_metric_calls must be set on GEPA, or it raises. MIPROv2 and BootstrapFewShotWithOptuna need pip install "dspy[optuna]"; SIMBA, KNNFewShot and the embedding paths need "dspy[numpy]". Set log_dir so a run that cost money leaves a record. And more demos is not free at serve time — every demo is paid for on every request forever, so a two-point gain from eight demos instead of four may be a bad trade.

Trade-off: engines, and whether to upgrade now

"DSPy 3.4 added native engines. Do you switch, and what breaks?"

Describe the mechanism accurately first. DSPy 3.4 vendors lm15 and exposes it as dspy.lm15. dspy.LM(..., engine="auto") is the default: it prefers native execution for supported routes and falls back to LiteLLM before inference when it cannot run natively. engine="lm15" forces native and raises if impossible; engine="litellm" forces the compatibility path. Auth failures, timeouts and provider errors never trigger a backend switch, and LiteLLM is still installed and not deprecated.

So the answer to "do you switch" is that you are already on auto after upgrading, and the real work is knowing the behaviour differences. The one with a bill attached: native n>1 makes separate sequential requests, so input tokens are billed per sample — if you use provider-native multi-sampling, force engine="litellm" for that LM. Also, native timeouts bound individual transport waits rather than total generation time, a stream is not retried once it has emitted a chunk, incomplete responses are not cached, and unknown cost stays unknown rather than being reported as zero.

The migration half of the question is the removals. Deprecated in 3.4 and gone in 3.5: OpenAI-style lm(messages=[...]) calls, custom BaseLM.forward()/aforward() integrations, LegacyEngine/AsyncLegacyEngine/complete_legacy(), plus dspy.CodeAct and dspy.ProgramOfThought, which were deprecated in 3.3.1 in favour of dspy.RLM. Still fine in 3.5: lm("hello") and ordinary DSPy programs. Mention that the 3.4 docs themselves still show lm(messages=[...]) on the installation and language-models pages, so you teach lm("...") for a quick check and lm(Request(...)) when you need explicit messages.

The habit that lands this answer: python -W default::DeprecationWarning on your test suite, in CI, so new deprecations surface on the pull request that introduces them.

Trade-off: testing something non-deterministic

"How do you put a DSPy program under test without a flaky suite and a large API bill?"

Answer in three layers, because a candidate who gives only the third has not solved the problem.

Structural tests with no LM at all: field names and types, module composition, the predictor list from named_predictors(), that program.load() restores the demos you expect, and that your metric returns the right thing on hand-written pairs — including that it is strict when trace is not None, which is the bootstrapping path.

Fake-LM tests: the 3.4 engine protocol is an object with complete(request) -> Response, so a scripted engine passed as engine= makes the program fully deterministic offline. Use it inside dspy.context so you never mutate global settings from a test. Note that with a custom engine, api_key, api_base and timeout live on the engine — DSPy rejects them on dspy.LM(...).

A scored gate that does hit the provider: a fixed, committed test set; a threshold at baseline minus the measured noise floor, never equality; ~/.dspy_cache cached between CI runs so most of it replays for free; and use_json_adapter_fallback=False so format regressions fail the build. Run it nightly, not per commit.

Common interview questions, with model answers

"What does an optimiser actually change?" The learnable state on predictors: instructions (the signature docstring) and demos (few-shot examples). Some optimisers tune weights instead, via BootstrapFinetune. It returns a new program; the original is not mutated, and compiled modules are flagged _compiled=True so an outer optimiser skips them.

"What does trace mean in a metric?" metric(example, pred, trace=None) is called with trace=None during plain evaluation and with a non-None trace during bootstrapping, when the optimiser is deciding whether to keep a generated run as a demonstration. The idiom is a strict bool when trace is not None and a graded float otherwise.

"What unit is EvaluationResult.score?" A percentage, 0 to 100, rounded to two decimal places. .results is the per-example list of (example, prediction, score). The old return_outputs=True raises.

"Why do I get the same answer every time?" The DSPy cache is on by default, in memory and on disk at ~/.dspy_cache. Use cache=False per LM, or rollout_id with a non-zero temperature to force a fresh call while keeping caching on elsewhere.

"Which exception do you catch?" dspy.LMError, the normalised base class since 3.3, with .code, .model, .provider, .status, .request_id and .retry_after. Subclasses include LMRateLimitError, LMAuthError, LMTimeoutError, ContextWindowExceededError; the helper is dspy.is_retryable_lm_error(e). Catching LiteLLM or provider exceptions no longer works.

"How do you save and ship an optimised program?" program.save("p.json") for state only — instructions, demos, field metadata, LM config minus the key — which is diffable and safe to load. .pkl and save_program=True directory saves need allow_pickle=True on load because they execute code. Then version it with the DSPy version, model string and test score recorded beside it.

"Why does my reasoning model fail at construction?" OpenAI reasoning models require temperature=1.0 or None and max_tokens >= 16000 or None; DSPy raises LMConfigurationError with exactly that guidance rather than letting the provider reject it later.

"What are the assertions, dspy.Assert and dspy.Suggest?" Deprecated and not supported; the docs page carries the warning. Use dspy.Refine or dspy.BestOfN, both of which take a reward_fn and a threshold.

Sixty-second self-test

Answer out loud, in one sentence each.

What are the five stages of a module call? Which stage does an AdapterParseError belong to, and which stage usually caused it? What is the default engine and when does its fallback decision happen? Name one thing that costs more on the native engine than the LiteLLM one. Why must a FastAPI handler use dspy.context rather than dspy.configure? What propagates overrides into workers, and what does not? What does trace is not None tell your metric? What unit is result.score? Which optimiser do you run first, and why? How many of auto, max_full_evals and max_metric_calls may GEPA have set? What does load() strip by default, and why? What number does a service stall at if you forget async_max_workers? Which exception base class replaced provider exceptions in 3.3? Which two modules were deprecated in 3.3.1 and replaced by dspy.RLM? What is the one-flag command that reveals your 3.5 migration work?

If more than three of those are vague, re-read the matching section of the detailed guide rather than trying to memorise the answer — these questions have follow-ups, and a memorised answer with no mechanism behind it collapses on the second question.

These are the mistakes that show up once DSPy is in a repository other people depend on, in roughly the order they cost you money. Each one has the symptom you will actually see, the mechanism underneath it, and the fix. Current for DSPy 3.4.

The errors that bite at this level

You quote the valset score. The symptom is a glorious offline number and a flat production one. The optimiser was allowed to read the valset while ranking candidates, so that score is contaminated by construction. Split three ways with a fixed seed — trainset, valset, test — and treat the test set as untouchable. The only number that may appear in a slide is the test-set number.

Your label is in with_inputs. The symptom is a near-perfect score that collapses in production. .with_inputs("question") declares which keys are inputs; everything else is a label. Name a label there and the program is handed the answer during evaluation. Check the list by eye before every compile, because this failure is silent and expensive.

You act on noise. The symptom is a change that "improved things by two points" and then does not reproduce. Run your evaluation five times unchanged and record the spread; that number is your significance threshold. Below it you have measured nothing. On a fifty-example set a five-point swing is routine.

Your metric is not discriminative. The symptom is an optimiser that runs for an hour and returns a program indistinguishable from the input. If the metric returns 1.0 for most examples there is no gradient to follow. Plot the distribution of per-example scores from result.results before you spend anything on a compile.

You call dspy.configure in a request handler or a worker. The symptom is RuntimeError: dspy.settings can only be changed by the thread that initially configured it., or the async variant telling you to use dspy.context. Configure once at startup; use with dspy.context(...) everywhere else.

You hand-roll threads inside a dspy.context block. The symptom is worse: no error, and every worker quietly using the process-default model. dspy.Parallel, Module.batch and dspy.asyncify copy overrides into workers; threading.Thread and ThreadPoolExecutor do not. Use DSPy's primitives, or capture dspy.settings.thread_local_overrides.get() and re-set it in each worker.

You leave async_max_workers at 8. The symptom is a p50 of two seconds and a p99 of forty under load, with no errors anywhere. dspy.asyncify caps concurrent in-flight programs at 8 per process by default, and the ninth call waits. Raise it to match your provider concurrency and remember your real ceiling is per-process times replicas.

You catch provider exceptions. The symptom is an except clause that stopped firing after an upgrade. Since 3.3, LM errors are normalised to DSPy exceptions, so LiteLLM and OpenAI exception classes no longer reach your handler. Catch dspy.LMError or a subclass, and use dspy.is_retryable_lm_error(e) for the retry decision.

You expect load() to restore your endpoint. The symptom is production traffic going to the wrong model after a deploy. By default load() strips api_base, base_url and model_list from the saved LM config, and api_key is never saved — deliberately, so an artefact cannot redirect traffic. Configure the LM in your serving code and treat the artefact as prompts and demos only.

You read an AdapterParseError as a parsing bug. It is almost always truncation: max_tokens too low for the reasoning plus every output field. The other causes, in order, are an empty response from a content filter, a model too weak for the format, and a schema with too many similar output fields. Raise max_tokens first, then look at dspy.inspect_history().

You ignore the DSPy cache during a comparison. Two symptoms, opposite directions: identical answers on repeated calls ("the model is broken") and a free-looking evaluation that reports empty usage. Cached calls report no usage. Use cache=False or rollout_id plus a non-zero temperature when you need genuine variation, and keep the cache on when you want two runs to be comparable.

You let a compile run uncapped. The symptom arrives on the invoice. Exactly one of auto, max_full_evals or max_metric_calls must be set on GEPA. Set auto="light" first, set log_dir, and know your metric's per-call cost before you multiply it by a budget.

You hit ModuleNotFoundError from inside an optimiser. Numpy and Optuna are optional extras now: pip install "dspy[optuna]" for MIPROv2 and BootstrapFewShotWithOptuna, "dspy[numpy]" for SIMBA, KNNFewShot and the embedding paths. It is a missing extra, not a broken install.

Your container cannot write the cache. The symptom is a crash on a read-only root filesystem. The disk cache defaults to ~/.dspy_cache with a 30 GB limit. Set DSPY_CACHEDIR to a writable path or dspy.configure_cache(enable_disk_cache=False).

Your long-lived process grows. lm.history and the global history hold prompts in memory, default max_history_size=10000. In a service handling customer data that is both a memory leak and a privacy problem. Set disable_history=True in production.

The practices that pay off, in order of leverage

1. Three-way split with a fixed seed, before anything else. Every other practice depends on having a number you can trust. Costs ten minutes, prevents the single most common way a DSPy project misleads its own team.

2. A measured noise floor. Five identical runs, record the spread, write it in the README. It converts every subsequent "did this help?" from a debate into an arithmetic check.

3. A baseline before a compile. A zero-shot test-set score is the denominator for every claim you will make. Without it, a compile result is a number with nothing to compare to.

4. Cheapest optimiser first. BootstrapFewShot before BootstrapRS before MIPROv2 or GEPA. The step from zero-shot to four good demonstrations is usually the largest gain available, and it costs a fraction of a search.

5. Pin models and pin DSPy. dspy==3.4.0 exactly, in the compile image and the serving image, and explicit model versions on your LMs. DSPy pins gepa[dspy]==0.1.4 exactly for the same reason. Version skew between an artefact and the library that reads it only logs a warning.

6. Provenance beside every artefact. The DSPy version, the model string, the data snapshot and the test-set score, in a sidecar file or an MLflow run. Six months later this is the difference between a rollback and an archaeology project.

7. Tracing on from day one. mlflow.dspy.autolog() is one line and gives you a span tree per request. Every latency and quality investigation starts there instead of in a terminal.

8. Structural tests that need no LM. Predictor lists, field types, metric behaviour on hand-written pairs, that an artefact loads with demos present. Milliseconds to run, catches most refactoring breakage.

9. -W default::DeprecationWarning in CI. Python hides deprecations, and 3.5 removes real APIs. One flag on one job turns a future breakage into a visible item on a pull request.

10. Per-predictor models. set_lm on the easy predictor is usually a free cost cut; measure it on the test set and keep it if the score holds within your noise floor.

11. Provider prompt caching. After optimisation most of your input tokens are a stable prefix of instructions and demos, which is exactly what a prompt cache discounts. Configure it per engine: prompt_cache=dspy.lm15.CacheConfig(...) on the native engine, cache_control_injection_points=[...] on the LiteLLM one.

12. A nightly scored gate. A fixed committed test set, a threshold at baseline minus noise, the DSPy cache warm between runs. It catches model drift, which is otherwise discovered by users.

Practice cards

Card 1 — find the invisible queue. Serve a program with dspy.asyncify behind FastAPI. Fire 8, then 16, then 32 concurrent requests and plot p50 and p99. Raise async_max_workers to 32 and repeat. Done when you can explain the shape of both curves from the worker cap alone.

Card 2 — break and fix a parse. Set max_tokens=20 on a program with three output fields and trigger AdapterParseError. Read dspy.inspect_history(n=1) and find the exact point of truncation. Then raise max_tokens and confirm. Done when you read that error as truncation reflexively.

Card 3 — prove the fallback is hiding something. Run your dev set under the default ChatAdapter, then under dspy.ChatAdapter(use_json_adapter_fallback=False). Done when you know your real format-compliance rate for your model and schema.

Card 4 — measure the noise floor. Run the same evaluation five times with temperature=1.0 and distinct rollout_id values. Record mean and spread. Done when you have a number you would defend in a review.

Card 5 — lose your overrides. Run a program inside with dspy.context(lm=model_a) using a ThreadPoolExecutor, and log which model each call used. Then do the same with dspy.Parallel. Done when you have seen the difference first-hand rather than read about it.

Card 6 — write feedback a reflection model can use. Take a classification metric and replace "wrong" with dspy.Prediction(score=0.0, feedback="...") naming the distinguishing feature of the correct class. Run GEPA with auto="light" both ways. Done when you can see the difference in the instructions it writes.

Card 7 — build a fake engine. Implement an object with complete(request) -> Response and pass it as engine=. Write three deterministic tests with it. Done when your suite tests your own logic with no network.

Card 8 — audit your deprecations. Run the whole suite with python -W default::DeprecationWarning -m pytest and turn the output into a migration checklist for 3.5. Done when the list is empty or ticketed.

Card 9 — do the demo-cost arithmetic. With track_usage=True, print get_lm_usage() for one request. Work out what fraction of input tokens are demos, then halve max_bootstrapped_demos, recompile and compare score against cost. Done when your demo count is a decision rather than a default.

Card 10 — survive a hardened container. Run your service with a read-only root filesystem and disable_history=True. Done when it starts, serves, and does not grow.

Operational recipes

Promote an optimised program. Compile offline with a capped budget and log_dir set. Score on the untouched test set. Save state as JSON named after the run (support-2026-10-08.json). Write a sidecar recording DSPy version, model strings, data snapshot and score. Build an image pinned to dspy==3.4.0 containing both the code and the artefact. At startup, dspy.configure(lm=...), instantiate the class, load() the artefact, re-apply any set_lm, and assert a smoke-test score before the readiness probe passes. Roll back by deploying the previous image, not by editing the artefact.

Triage a quality complaint. Reproduce the exact input. Run it once and read dspy.inspect_history(n=1) — the first question is always what DSPy actually sent, because an optimised prompt is rarely what you remember. If the prompt looks right and the answer is wrong, it is a model or instruction problem: check whether the input resembles anything in your test set, and add it if not. If the prompt looks wrong, check that the artefact loaded (demos present) and that the serving DSPy version matches the compile one. If the output is missing a field, it is the adapter, and the cause is probably truncation.

Respond to a rate-limit storm. dspy.LMRateLimitError carries .retry_after and .provider. Lower num_threads on Evaluate, Parallel and batch — your ceiling is the provider quota, not your CPU. Leave num_retries at 3 rather than raising it, because a higher retry count converts throttling into latency. Lower max_errors on CI gates so you find out immediately; raise it on exploratory sweeps so a long run is not destroyed by three bad examples.

Add a self-hosted model. For an OpenAI- or Anthropic-compatible endpoint — vLLM, SGLang, a LiteLLM proxy, a gateway in your own VPC — call dspy.lm15.register_provider(...) before constructing any LM, since each LM binds the registrations present at construction time, then use dspy.LM("yourname/model", engine="lm15"). For a genuinely different protocol, implement complete(request) -> Response and pass engine=, putting api_key, api_base and timeout on the engine, because DSPy rejects them on dspy.LM(...). For local experimentation, dspy.LM("ollama_chat/llama3.2", api_base="http://localhost:11434", api_key="") needs no keys at all; see the Ollama guide.

Isolate tenants. Configure a default LM at startup and use with dspy.context(lm=tenant_lm) per request, which is ContextVar-scoped and async-safe. Then close the two gaps DSPy does not close for you: the response cache is keyed on the request rather than the tenant, so identical text from two tenants shares an answer unless you set cache=False or distinct rollout_id values; and the global history holds prompts in memory, so set disable_history=True. Callbacks are process-wide, so never put tenant state in one.

Keep prompts in the region you promised. The LM endpoint decides where prompts are processed, which is the whole of your data-residency story for Gulf and Egyptian customers. Pin it explicitly. On Vertex, use the vertex_ai/ prefix with vertex_project and vertex_location — not gemini/ with project/location, which sends requests to the wrong region silently. On Azure, set AZURE_API_BASE and AZURE_API_VERSION alongside the key and use the deployment name. Record the endpoint in the artefact's sidecar so a review can check it.

Decide whether to run code. dspy.RLM is the current module for letting the model write and run Python; dspy.CodeAct and dspy.ProgramOfThought were deprecated in 3.3.1 and go away in 3.5. The default interpreter is PythonInterpreter, a Deno plus Pyodide sandbox — install it with pip install "dspy[deno]", or you will meet CodeInterpreterError: Deno executable not found. dspy.LocalInterpreter, new in 3.4, runs in a persistent CPython subprocess and is explicitly not a security sandbox: generated code keeps your filesystem, environment, credentials, subprocess and network access. Use it only for code you would have run yourself. Note also that the 3.4 call-time override is keyword-only — rlm(query=q, interpreter_factory=dspy.PythonInterpreter) — and a positional factory or a live instance raises TypeError.

The checklist

Before you compile: three-way split with a fixed seed · with_inputs contains only real inputs · metric distribution plotted and discriminative · metric strict when trace is not None · zero-shot baseline recorded on the test set · noise floor measured · budget capped · log_dir set.

Before you deploy: score read from the test set, not the valset · artefact saved as JSON and named after the run · sidecar with DSPy version, model strings and score · dspy==3.4.0 pinned in both images · LM configured in the serving code, not expected from the artefact · set_lm re-applied after load() · async_max_workers sized by load test · disable_history=True · cache directory writable or disabled · dspy.LMError handled, including a ContextWindowExceededError branch.

Every week: tracing on and one slow trace actually read · nightly scored gate green against baseline minus noise · deprecation job clean · token spend reviewed with the demo fraction broken out.

This is part three of three. Beginner taught you signatures, modules and a first optimizer run. Mid taught you how DSPy actually builds a prompt, how to evaluate a program honestly, and how to fit it into a service. This level is about the part nobody warns you about: the moment your team's DSPy experiment becomes three teams' production dependency, and you are the person who owns the version pin, the cache, the API keys, the compile budget and the pager.

The shift in question is specific. Up to now you have been asking "does this program score well enough". From here you are asking what happens when a tenant's prompt injection reaches a tool, who pays for a GEPA run that cost nine hundred dollars, whether a saved program.json someone emailed you can redirect traffic to an attacker's endpoint, and what breaks first when the provider starts returning 429 at four in the afternoon every day.

Where this picks up

I am starting with architecture, because almost every operational surprise in DSPy traces back to one structural fact: DSPy is a library, not a server, and the thing you deploy is a Python process that happens to hold a tuned prompt.

The architecture of a DSPy platform

DSPy is an in-process Python library. There is no DSPy daemon, no scheduler, no control plane, and no CLI of its own. That is worth saying plainly because it determines where every platform responsibility lands: on you, in the surrounding infrastructure.

The production shape that works is a hard split between a compile plane and a serve plane. Compiling is where optimizers run: expensive, non-deterministic in wall-clock cost, and dependent on a training set and a metric. Serving is where a program is instantiated, has previously-optimized state loaded into it, and answers requests. The artifact that crosses the boundary is a saved state file plus the code that defines the program class. Both halves of that pair matter: program.json holds instructions, demonstrations and field metadata, but the class that gives those predictors their structure lives in your repository. Ship one without the other and you have nothing.

Inside the serve plane, the layering is worth internalizing because it tells you where to look when something is wrong. A request enters a module, which delegates to one or more predictors. Each predictor hands its signature and inputs to an adapter, which turns them into messages and later parses the response back into typed fields. The adapter hands the request to an LM, which hands it to an engine — and since 3.4 the engine is a real, swappable component rather than an implementation detail. The engine talks to the provider. On the way back, parsing failures surface as AdapterParseError, transport and provider failures surface as dspy.LMError subclasses, and everything in between is your problem to classify.

The engine transition, and the clock you are already on

DSPy 3.4.0, released on 2026-09-25, is the release where the backend stopped being LiteLLM by assumption. DSPy now vendors lm15 and exposes it as dspy.lm15, with no separate install. dspy.LM(..., engine="auto") is the default: it prefers native lm15 execution for routes it supports and falls back to the LiteLLM compatibility path before inference when it has to. You can force either side — engine="lm15" raises if native execution is impossible, engine="litellm" pins the compatibility path — and LiteLLM is still installed and explicitly not deprecated.

Several engine behaviours change capacity planning. Native n>1 issues separate sequential requests, so input tokens are billed once per sample; if you want provider-native multi-sample generation, you need engine="litellm". Native timeouts bound individual transport waits, not total generation time, so a long generation can exceed your nominal timeout without any single wait doing so. A stream is not retried once it has emitted a chunk, and incomplete responses are not cached — both of which are correct and both of which mean a partial failure in a streaming endpoint is visible to the user rather than quietly healed. And unknown cost stays unknown rather than being reported as zero, which matters enormously the first time you build a cost dashboard.

For platform teams, the most useful new primitive is provider registration. Instead of writing a custom engine for an OpenAI- or Anthropic-compatible endpoint, you declare it:

import dspy

dspy.lm15.register_provider(
    "internal-gateway",
    base_url="https://llm-gw.internal.example/v1",
    api_key_env="INTERNAL_GATEWAY_KEY",
)

# Each LM binds the registrations present at construction time.
lm = dspy.LM("internal-gateway/qwen-32b-instruct", engine="lm15")

That binding rule — each LM binds registrations present at construction time — is a real footgun in a long-lived server. Register everything at import time, before any LM is built, or you will have LMs in the process that cannot see a provider you added later.

If you genuinely need a custom backend, implement an object with complete(request) -> Response and pass engine=; pass async_engine= for the async path. Two constraints follow. First, api_key, api_base and timeout belong on the engine, and DSPy rejects them on dspy.LM(...) when a custom engine is in play — the engine owns its credentials. Second, restoring custom-engine state from a saved program requires allow_unsafe_lm_state=True, which is a deliberate speed bump you should not paper over with a default.

The explicit form is the one to standardize on:

import dspy
from dspy.lm15 import Config, Message, Request

lm = dspy.LM("openai/gpt-4o-mini")
response = lm(Request(
    model=lm.model,
    system="Be concise.",
    messages=(Message.user("What is DSPy?"),),
    config=Config(max_tokens=200),
))
print(response.text)

The 3.4 migration also removed, rather than aliased, the experimental 3.3 LM types. dspy.LMRequest/LMResponse became dspy.lm15.Request/Response; LMMessage/LMConfig became lm15.Message/Config; dspy.System(text) became Request(system=text, ...); dspy.User/Assistant became Message.user/Message.assistant; and response.outputs[0].parts became response.message.parts. Importing dspy.core.types raises a migration error outright, forward_contract="typed_lm" is rejected, and experimental=True no longer makes prompt calls return typed responses. The dspy.clients.openai_format module is gone. If your platform has any code written against the 3.3 experimental surface, it does not run on 3.4 at all — this is a hard break, not a warning.

Failure modes, and which one you are in

A DSPy program has a small number of ways to fail and they are easy to confuse under pressure. The discipline is to classify before you act.

Prompt-layer failures look like a quality regression with no error. The score drops, nothing throws, and the cause is usually that demos or instructions changed — a new compile was promoted, or a program loaded state it was not supposed to. The tell is that dspy.inspect_history(n=1) shows a prompt you do not recognize. The fix is a rollback of the artifact, not a code change.

Parse-layer failures throw AdapterParseError, with a message of the form "Adapter <Name> failed to parse the LM response. ... Expected to find output fields in the LM response: [...]". There are exactly three realistic causes: the model ignored the format, the response was truncated because max_tokens was too low, or the response was empty. Truncation is the most common in production and the least obvious, because a half-written answer looks like a formatting problem. Check the finish reason and the token count before you blame the adapter. If the model genuinely formats badly — reasoning models are the usual suspects — JSONAdapter or TwoStepAdapter with a separate extraction_model is the structural fix. Note that since 3.3, ChatAdapter and JSONAdapter no longer retry an alternate format after an LMError, so a transport failure will not be disguised as a parse retry.

Transport-layer failures are the normalized error tree, and it is worth memorizing because it replaced catching provider exceptions directly. The base is dspy.LMError, carrying .code, .model, .provider, .status, .request_id and .retry_after. Subclasses include LMRateLimitError (with .retry_after and .provider), LMAuthError, LMTimeoutError, LMServerError, LMInvalidRequestError, LMUnsupportedModelError, LMBillingError, LMTransportError, LMUnexpectedError and LMNotConfiguredError. There is a helper, dspy.is_retryable_lm_error(e), and you should use it rather than inventing your own status-code table. dspy.ContextWindowExceededError is its own case: too many demos, too much retrieved context, or a ReAct trajectory that grew — and ReAct truncates the trajectory before re-raising, so by the time you see it, it has already tried.

import dspy

def classify(exc: Exception) -> str:
    """Map a DSPy failure to the on-call action, not to a log level."""
    if isinstance(exc, dspy.AdapterParseError):
        return "check max_tokens and finish_reason, then the adapter"
    if isinstance(exc, dspy.ContextWindowExceededError):
        return "shrink demos/context; check for trajectory growth"
    if isinstance(exc, dspy.LMAuthError):
        return "page the platform team: credential rotation or scope"
    if isinstance(exc, dspy.LMBillingError):
        return "page finance owner: provider account state"
    if isinstance(exc, dspy.LMRateLimitError):
        return f"back off {getattr(exc, 'retry_after', None)}s; lower num_threads"
    if isinstance(exc, dspy.LMError) and dspy.is_retryable_lm_error(exc):
        return "transient: rely on num_retries, alert only on rate"
    return "unknown: capture request_id and escalate"

Configuration failures are the fourth family and they are almost always a threading mistake. dspy.configure may only be called from the thread — or async task — that first called it. From anywhere else you get RuntimeError: dspy.settings can only be changed by the thread that initially configured it. or, in the async case, a message telling you to use dspy.context(...) in other async tasks. There is no way around this and no reason to want one: process-wide defaults are set once at startup, and everything per-request uses with dspy.context(...), which is ContextVar-scoped and safe across threads and tasks.

The subtle version of this bug is context that silently does not propagate. dspy.Parallel, Module.batch and asyncify copy context overrides into their workers. A plain threading.Thread or ThreadPoolExecutor does not. If you hand work to your own executor, you must capture dspy.settings.thread_local_overrides.get() on the calling side and re-set it inside the worker — or use dspy.Parallel and let DSPy do it. A tenant-scoped LM override that evaporates in a worker thread means a tenant's traffic quietly running on the default model, which is the kind of bug that gets found in a bill rather than a log.

Scaling: which component breaks first

The honest answer is that the provider breaks first, and almost everything you can tune in DSPy is really about not making that worse.

DSPy's concurrency comes from two places. The threaded path — dspy.Evaluate, dspy.Parallel, Module.batch — uses num_threads, which defaults to 8. The async path uses acall/aforward natively, and dspy.asyncify(program) runs a synchronous program in worker threads bounded by async_max_workers, which also defaults to 8. That second default is the one that bites: in a FastAPI service built on asyncify, async_max_workers is a hard ceiling on concurrent in-flight programs per process, and the ninth concurrent call waits. Nothing errors, nothing logs, latency simply doubles under load. If your p99 chart has a cliff exactly where concurrency crosses eight, that is this.

The ordering of what to tune is: raise async_max_workers and num_threads until the provider starts returning 429, then stop and scale processes instead. num_retries defaults to 3 and handles ordinary transient failures. max_errors defaults to 10 and governs how many failures an evaluation or batch tolerates before giving up — raise it for exploratory evaluation runs, and keep it tight for release gates so a broken devset fails loudly.

Settings reads are lock-free, so dspy.context is cheap enough to use on every request. That is a deliberate design choice and you should lean on it rather than building your own request-scoped configuration layer.

Native async is strictly better than asyncify when you can get it, because asyncify is a thread pool wearing an async coat. Writing aforward on your own modules and using acall on built-ins keeps the work on the event loop. One 3.4 constraint to know: calling an async tool synchronously inside a running event loop now raises ValueError, so you must await tool.acall(...). There is a settings flag, allow_tool_async_sync_conversion, defaulting to False; leaving it off is the right platform default because the conversion hides a blocking call inside your loop.

import dspy
from fastapi import FastAPI
from dspy.utils.streaming import streaming_response

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"), async_max_workers=32, track_usage=True)

program = MyProgram()
program.load("artifacts/program-2026-09-30.json")

api = FastAPI()
served = dspy.asyncify(program)
streamed = dspy.streamify(
    program,
    stream_listeners=[dspy.streaming.StreamListener(signature_field_name="answer")],
)

@api.post("/answer")
async def answer(question: str, tenant: str):
    with dspy.context(lm=lm_for(tenant)):
        pred = await served(question=question)
    return {"answer": pred.answer, "usage": pred.get_lm_usage()}

Streaming deserves its own note. dspy.streamify plus a StreamListener on a named output field gives you token-level streaming, and dspy.utils.streaming.streaming_response is the SSE helper for FastAPI. The operational catch is the one from the engine section: once a stream has emitted a chunk it is not retried, and incomplete responses are not cached. A streaming endpoint therefore has a genuinely worse failure profile than a buffered one, and if your SLO is about correctness rather than perceived latency, buffering is the defensible choice.

The cache is shared infrastructure

DSPy caches by default, at three layers: an in-memory cachetools.LRUCache, an on-disk diskcache.FanoutCache, and whatever prompt caching the provider offers. The key is a hash of the request arguments, excluding credentials. The disk cache lives at ~/.dspy_cache unless DSPY_CACHEDIR says otherwise, and its size limit comes from DSPY_CACHE_LIMIT, defaulting to 3e10 bytes — 30 GB. The in-memory cache defaults to 1,000,000 entries.

import dspy

# Read-only containers: no disk cache at all.
dspy.configure_cache(enable_disk_cache=False, enable_memory_cache=True,
                     memory_max_entries=50_000)

# Or: a bounded, explicitly mounted, writable cache volume.
dspy.configure_cache(
    enable_disk_cache=True,
    disk_cache_dir="/var/cache/dspy",
    disk_size_limit_bytes=2 * 1024**3,
    memory_max_entries=50_000,
    restrict_pickle=True,
)

restrict_pickle=True matters and is covered properly in the security section; the short version is that the disk cache uses pickle, this flag opts into safer unpickling, and the maintainers have said it will become the default later. Turn it on now. When it rejects a type you actually need, you will see WARNING dspy.clients.cache: Failed to deserialize disk cache entry <key> and the fix is to register the type via safe_types=[...] rather than to turn the flag off.

The behaviour that generates the most support tickets is the simplest: identical calls return identical results because the cache is on. Someone reports "the model is stuck", and nothing is stuck. Two legitimate escapes exist. cache=False on a specific LM turns it off for that client. Or keep caching and force a fresh sample with a distinct rollout_id plus a non-zero temperature:

lm("Draft a subject line", rollout_id=1, temperature=1.0)
lm("Draft a subject line", rollout_id=2, temperature=1.0)

The cache is also where multi-tenancy leaks, and I will come back to that. For now, hold onto the fact that the key is derived from the request, not from the tenant.

The trust model

DSPy's security posture is better than most LLM frameworks in a few specific, deliberate places, and silent in others. Knowing which is which is the job.

API keys are never serialized. LM.dump_state hard-excludes api_key. More interestingly, loading with the default allow_unsafe_lm_state=False strips api_base, base_url and model_list from the saved LM config. That is a real attack mitigated by design: a tampered program.json cannot redirect your traffic to an attacker's endpoint. Which also tells you exactly what allow_unsafe_lm_state=True means, and why it should never appear in a serving path that loads artifacts it did not produce.

Pickle is the sharpest edge. program.save("p.pkl") and whole-program saves (save_program=True, producing program.pkl plus metadata.json in a directory) both require allow_pickle=True to load, with the explicit message that loading .pkl files can run arbitrary code. The JSON path is the recommended one for a reason: it is state-only, inspectable, reviewable in a pull request, and incapable of executing. Make JSON the only format your platform accepts across a trust boundary, and treat allow_pickle=True as a code smell that needs a named owner.

Resource loading no longer does implicit I/O. Since 3.3, constructing dspy.Image(path) does not read the file; you call dspy.Image.from_path(p), dspy.Image.from_url(u) (which downloads and embeds a data URI), or dspy.Image(u) to pass a URL for the provider to fetch, with the same pattern for Audio and File.from_path. The security value is direct: an LM's output cannot trigger a filesystem read just by being shaped like a path. But from_url() follows redirects and has no SSRF allowlist, so a URL that arrived from a user or a tool output must be validated by you before it reaches that call. Image.from_url() and Audio.from_url() default to timeout=30 seconds, with timeout=None for unbounded; leave that bounded.

Code execution has two very different modes. PythonInterpreter is a Deno plus Pyodide sandbox, hardened in 3.3.1 with unpredictable JSON-RPC request IDs, rejection of recursive host-tool execution, revocation of Deno cache access after startup, and rejection of mount collisions. It remains the default interpreter. dspy.LocalInterpreter, new in 3.4, runs generated Python in a persistent CPython subprocess with no Deno — and the release notes say it plainly: it is not a security sandbox. Generated code keeps the host user's filesystem, environment, credentials, subprocess and network access.

The 3.4 interpreter override API is keyword-only and takes a zero-arg factory. Passing a live interpreter instance, or a positional factory, raises TypeError. The name interpreter_factory is also reserved: declaring it as a signature input raises ValueError at RLM construction. This applies to RLM only — the deprecated CodeAct and ProgramOfThought keep their 3.3 call API, and are scheduled for removal in 3.5 anyway.

import dspy

rlm = dspy.RLM("context, query -> answer", max_iters=20, max_llm_calls=50,
               max_output_chars=10_000, interpreter_factory=dspy.PythonInterpreter)

result = rlm(query="How many rows mention Riyadh?", context=big_text)
# async form, same keyword-only override:
result = await rlm.acall(query="...", interpreter_factory=dspy.PythonInterpreter)

Those three caps — max_iters, max_llm_calls, max_output_chars — are your blast radius controls for a module whose whole premise is that the LM decides what to do next. Set them explicitly per tenant rather than inheriting defaults.

Prompt injection is not solved, and DSPy does not claim to solve it. Treat retrieved text and tool outputs as data. Concretely: never let a tool output decide which tool to call next without a policy check in your own code, keep destructive tools behind an allowlist that is a function of the request's authenticated identity rather than of the model's intent, and remember that a signature's instructions carry no more authority with the model than the context you paste below them.

Vulnerability reporting goes through GitHub Private Vulnerability Reporting, per the project's SECURITY.md. If you are running DSPy as internal platform infrastructure, put that in your own security documentation so a finding from one of your tenant teams has somewhere to go.

Secrets and the supply chain

Secrets handling is mostly straightforward because DSPy refuses to persist keys. Use environment variables or a secret manager, and pass api_key= at LM construction if you are not relying on the provider's conventional variable. The conventional ones are OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, the Azure trio of AZURE_API_KEY, AZURE_API_BASE and AZURE_API_VERSION, DATABRICKS_API_KEY with DATABRICKS_API_BASE, and for Vertex, GOOGLE_APPLICATION_CREDENTIALS alongside VERTEXAI_PROJECT and VERTEXAI_LOCATION.

Region matters here for Gulf and Egyptian employers in a way that is easy to get wrong silently. If you are routing to Vertex AI in a specific region for data-residency reasons, use the vertex_project and vertex_location kwargs with the vertex_ai/ model prefix. Passing project/location instead, or using the gemini/ prefix, sends requests to the wrong region silently — no error, just inference happening somewhere your compliance document says it does not. That failure has no log line, so it needs a test: assert on the resolved endpoint in a startup check rather than trusting the configuration.

The dependency supply chain is unusually concrete for a Python library because 3.4.0's base dependencies are specified and one of them is pinned exactly. The floors are openai>=1.66.2, litellm>=1.65.8, pydantic>=2.11.0, diskcache>=5.6.0, cachetools>=5.5.0, cloudpickle>=3.1.2, json-repair>=0.54.2, tenacity>=8.2.3, orjson>=3.9.0, plus anyio, regex, requests and tqdm. The exact pin is gepa[dspy]==0.1.4. That pin is a feature for reproducibility and a constraint when another dependency in your environment wants a different GEPA.

# Pin exactly. gepa[dspy] is already pinned exactly by dspy itself.
dspy[numpy,optuna,deno,mcp]==3.4.0

Python support in 3.4.0 is >=3.10,<3.15, so 3.10 through 3.14. The package name is dspy; dspy-ai is the legacy name and should not appear in any new requirements file. Finally, prefer the dspy PyPI distribution with hashes in a lockfile — the artifact is a pure-Python wheel, which makes reproducible installs genuinely easy and removes any excuse not to.

Multi-tenancy

DSPy has no built-in tenant model. Everything below is a pattern you implement and own.

Per-request isolation is with dspy.context(lm=tenant_lm, ...). It is ContextVar-scoped and async-safe, which makes it the correct primitive. Per-module isolation is predictor.set_lm(other_lm), useful when one stage of a pipeline must run on a different model for every tenant — a cheap extractor on a shared model, a tenant-specific generator.

Three pitfalls deserve explicit controls.

The cache crosses tenants. The cache key is a hash of the request arguments, so two tenants sending the same prompt share a cache entry. For most workloads that is a cost win and harmless. For a workload where one tenant's output must never be derivable from another's request, it is a data-sharing bug wearing a performance costume. The mitigations are cache=False on tenant LMs, or distinct rollout_id values per tenant so the keys differ. Decide this per tenant class, write the decision down, and make it a field in your tenant configuration rather than a tribal convention.

History holds prompts in memory. lm.history and the global history retain prompts and responses. Defaults are max_history_size=10000 and disable_history=False. In a multi-tenant process that is a cross-tenant memory of request content sitting in RSS, available to anything that can read the process — including a crash dump that lands in your logging system. Set disable_history=True in production serving and rely on structured telemetry you control, or keep history on with a small max_history_size only in environments where you accept the exposure.

Callbacks are process-wide. A callback registered through dspy.configure(callbacks=[...]) sees every module, LM, adapter, tool, evaluation and compile event in the process, for every tenant. That is exactly what you want for platform telemetry and exactly what you must not let tenant code register. Own the callback list in your platform package.

import dspy

dspy.configure(
    lm=SHARED_LM,
    disable_history=True,          # no cross-tenant prompt retention in RSS
    track_usage=True,
    callbacks=[PlatformTelemetry()],   # platform-owned, not tenant-registered
    max_errors=5,
)

def run_for_tenant(tenant: Tenant, **inputs):
    lm = dspy.LM(tenant.model, api_key=tenant.key,
                 cache=tenant.allow_shared_cache)
    with dspy.context(lm=lm):
        pred = PROGRAM(**inputs)
    meter(tenant.id, pred.get_lm_usage())
    return pred

Cost, and who owns it

Cost in a DSPy platform splits cleanly into serving cost, which is roughly proportional to traffic and therefore forecastable, and compile cost, which is lumpy, discretionary and the thing that produces surprise invoices. The official documentation is direct that GEPA and MIPROv2 runs can cost hundreds of dollars in LM calls.

Serving levers, in order of leverage: the DSPy cache; the provider's prompt cache; fewer and shorter demos; and a cheaper task model. Provider prompt caching is configured differently per engine — on the native engine it is dspy.LM(..., engine="lm15", prompt_cache=dspy.lm15.CacheConfig(prefix="stable")), while the LiteLLM engine uses cache_control_injection_points=[{"location": "message", "role": "system"}] with engine="litellm". The docs note that ReActV2's history handling improves prefix reuse, citing up to 50% savings in internal tests. Treat that as a documented internal result rather than a number to put in your own forecast, and measure your own.

Measurement is the part teams skip. dspy.configure(track_usage=True) then pred.get_lm_usage() gives per-prediction usage; with dspy.track_usage() as usage: gives a scoped total. lm.history[i]["cost"] carries per-call cost where the engine knows it. Two caveats that will otherwise corrupt your dashboard: cached calls report empty usage, so a cache-heavy workload will look cheaper than it is in a way that breaks per-tenant chargeback unless you count cache hits separately; and unknown cost stays unknown rather than zero, so your aggregation must handle nulls instead of summing them into a confidently wrong total. Native n>1 bills input tokens per sample, which is the one place where the engine choice shows up directly on the bill.

Compile cost is governed by budgets, and the budget argument is mandatory for GEPA: exactly one of auto, max_full_evals or max_metric_calls must be set, and setting none or more than one raises AssertionError: Exactly one of max_metric_calls, max_full_evals, auto must be set. For MIPROv2, auto takes "light", "medium", "heavy" or None, defaulting to "light", and a typo raises a ValueError naming the valid set. The strongest platform control is to prefer max_metric_calls over auto for anything that runs in CI, because a metric-call ceiling is a number a finance owner can reason about while "heavy" is not.

import dspy

optimized = dspy.GEPA(
    metric=feedback_metric,
    max_metric_calls=4_000,                 # a hard, reviewable ceiling
    reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=32000),
    reflection_minibatch_size=3,
    candidate_selection_strategy="pareto",
    num_threads=8,
    track_stats=True,
    log_dir="artifacts/gepa/2026-09-30",
).compile(program, trainset=trainset, valset=valset)

Reasoning models carry their own parameter constraints that double as cost constraints. OpenAI's reasoning models require temperature=1.0 or None and max_tokens >= 16000 or None, and violating that raises LMConfigurationError with the fix in the message. A sixteen-thousand-token floor is a real cost floor; budget for it rather than discovering it.

Upgrades and migrations

Pin exact versions. dspy==3.4.0, not a range, for every environment including developer laptops — because the thing you are protecting is not just runtime behaviour but the prompts an optimizer produces, and those are a function of the library.

Saved JSON state is designed to be backward compatible, and a version mismatch between the metadata in a saved file and the running library logs a warning only. That is a convenience and a trap: a warning in a serving log is not a gate. If your platform promotes artifacts across environments, assert on the metadata explicitly at load time and refuse a mismatch you have not blessed. Also note load_state is transactional, so a failed load leaves the program in its prior state rather than half-applied, which makes a strict check safe to add.

The upgrade procedure that works:

1. Sweep for deprecations firstRun the full test suite under python -W default::DeprecationWarning on the current version. Every warning is work that becomes an error in 3.5.
2. Fix the known 3.5 listRemove lm(messages=...) calls, custom BaseLM.forward integrations, LegacyEngine/AsyncLegacyEngine/complete_legacy(), CodeAct and ProgramOfThought usage, and the Image.from_file/from_PIL/Audio.from_file aliases.
3. Check for the 3.3 experimental LM typesIf any code imports dspy.core.types, uses dspy.LMRequest/LMResponse/LMMessage/LMConfig/System/User/Assistant, reads response.outputs[0].parts, or sets forward_contract="typed_lm", it does not run on 3.4 at all.
4. Re-run evaluation, do not assumeEvaluate every production artifact on the new version against the frozen devset before promoting the version. Adapter and parsing changes between releases can move scores without touching your code.
5. Rebuild rather than migrate old picklesOld pickles containing the removed 3.3 experimental LM types are not migrated automatically. Treat a pickle that will not load as a reason to recompile, which is also a good reason to have kept the compile job reproducible.
6. Promote one tier at a timeDevelopment, then a canary tenant, then the shared tier, then the compliance tier. Keep the previous artifact and the previous pin available for a one-command rollback.

Other breaking changes from recent releases worth checking against your code: Pydantic's minimum is now 2.11.0; ReActV2 output fields may not be named history or termination_reason; Evaluate rejects an empty devset with ValueError: devset must contain at least one example, got an empty devset.; RLM(max_iterations=...) became RLM(max_iters=...) in 3.3; BaseLM direct defaults for temperature and max_tokens became None rather than 0.0/1000; ToolCalls.format() now serializes each call as {"name": ..., "args": ...}; and GEPA result shapes changed so that DspyGEPAResult.candidates are compiled modules and best_candidate is a module. Evaluate(..., return_outputs=True) raises — the results are always in result.results. On the older side, dspy.Assert/dspy.Suggest are deprecated and not supported, with dspy.Refine or dspy.BestOfN as the replacement, and dspy.TypedPredictor/TypedChainOfThought are gone because typed signatures work directly with Predict and ChainOfThought. If you are reading a blog post that uses any of those, the post is stale.

Governance and incident playbooks

Governance for a DSPy platform is mostly governance of two things nobody initially thinks of as governed: the devset and the compile budget.

The devset is your release gate, which makes it a controlled artifact. It belongs in version control, with an owner, a changelog and a rule that changing the devset and promoting a new program are separate pull requests. The reason is uncomfortably practical: if both change at once, a score improvement is uninterpretable, and the first time a regression slips through on a quietly relaxed devset, you lose the ability to trust any historical number. dspy.Evaluate returns an EvaluationResult whose .score is a percentage from 0 to 100 rounded to two decimal places, and whose .results is the per-example list of example, prediction and score — keep both, because the per-example list is what lets you show which cases changed rather than only that the aggregate moved.

For incidents, three playbooks cover most of what will actually happen.

Quality regression with no errors. First question: did the artifact change? Compare the loaded artifact's hash against the promoted one, then run dspy.inspect_history(n=1) and read the instructions and demos. If the prompt is the expected one, the provider changed the model behind a floating alias — which is why you pin model versions in the model string where the provider permits it. Rollback is the artifact, then the version pin, then the model string, in that order of likelihood.

Rate limiting and latency cliffs. LMRateLimitError carries .retry_after and .provider. Lower num_threads and async_max_workers, let num_retries do its job, and if the cliff appears at a fixed concurrency rather than a fixed rate, suspect async_max_workers rather than the provider. Capture .request_id from errors — 3.4 retains Retry-After, request IDs and headers on rate-limit errors — because a provider support ticket without request IDs goes nowhere.

Parse failures appearing in a burst. A burst of AdapterParseError from a program that was fine yesterday is almost always truncation, caused by longer inputs or more demos pushing the completion against max_tokens. Check the token counts first. If the content genuinely changed shape, JSONAdapter is the next step, and dspy.ChatAdapter(use_json_adapter_fallback=False) is how you make a test environment fail loudly instead of silently repairing itself through the fallback. Related: since 3.3.1, adapters apply declared defaults when an optional output field is omitted, while missing required outputs still raise — so a field that silently gets its default is a signature design decision, not a bug.

Observability is the foundation all three stand on. dspy.inspect_history(n) and lm.history are the interactive tools. BaseCallback subclasses are the programmatic one, with handlers covering on_module_start/end, on_lm_start/end, on_adapter_format_start/end, on_adapter_parse_start/end, on_tool_start/end, on_evaluate_start/end, on_compile_start/end, plus interpreter lifecycle events added in 3.3.1. The documented rule is not to mutate inputs or outputs in place inside a callback; a callback that edits what it observes turns your telemetry into a source of bugs. For traces, MLflow Tracing via mlflow.dspy.autolog() is the officially documented path and is OpenTelemetry based, which means it lands in whatever backend you already run. Third-party integrations exist — Langfuse, Arize Phoenix and Weave all have DSPy support — but they are not official DSPy documentation, so verify behaviour against your own version rather than against a vendor blog post.

import dspy

class PlatformTelemetry(dspy.utils.callback.BaseCallback):
    def on_lm_end(self, call_id, outputs, exception):
        # Do not mutate outputs. Emit, and move on.
        emit("dspy.lm", call_id=call_id, error=type(exception).__name__ if exception else None)

    def on_compile_end(self, call_id, outputs, exception):
        emit("dspy.compile", call_id=call_id, ok=exception is None)

dspy.enable_logging()
dspy.configure(callbacks=[PlatformTelemetry()])

Where DSPy stops

Owning a tool well includes knowing the shape of problems it is the wrong answer to.

DSPy stops being the right tool when there is nothing to optimize. If your task is a single stable prompt against a single model with no metric and no dataset, DSPy's value proposition — optimizers that tune instructions and demos against a metric — does not apply, and you are paying an abstraction cost for a prompt you could have written in a string. The honest recommendation in that case is a thin client, or LiteLLM directly.

It stops when the orchestration, not the prompt, is the hard part. Long-running, stateful, human-in-the-loop workflows with branching, retries and durable checkpoints are a workflow-engine problem. LangGraph exists for that shape, and the sane architecture is often LangGraph for control flow with DSPy-optimized modules as nodes inside it. Similarly, if your scheduling needs are "this runs nightly and must be observable and retryable", that is Prefect or Airflow around a compile job, not something to build inside DSPy.

It stops when you need a serving platform rather than a library. DSPy has documented serving paths — FastAPI with asyncify and streamify, and MLflow's mlflow.dspy.log_model(program, "dspy_program", input_example=..., task="llm/v1/chat") followed by mlflow models serve -m runs:/<run_id>/model -p 6000 or mlflow models build-docker — but those are integrations, not a model-serving control plane. If you need autoscaling, canaries, multi-model routing and traffic splitting as first-class concepts, that is KServe or BentoML territory, with DSPy inside the container. One documentation note if you take the MLflow route: as of MLflow 2.22 you must wrap built-in modules in a custom dspy.Module because MLflow passes positional arguments, and the docs require mlflow>=2.18.0. See the MLflow guide for the surrounding machinery.

It stops at evaluation breadth. dspy.Evaluate with a metric is a good release gate for the task you compiled against. It is not a RAG evaluation suite, a red-teaming harness, or a drift monitor. RAGAS and DeepEval cover evaluation dimensions DSPy does not model, and Evidently covers the production-drift question that Evaluate cannot see because it only knows your devset.

And it stops at guardrails and policy. DSPy does not solve prompt injection and does not pretend to. Input and output policy enforcement, PII handling and refusal behaviour live in a layer you add — Guardrails is one option — or in your own code. A platform that tells tenants "DSPy handles safety" is making a claim the project explicitly does not make.

Running it as a platform for other teams

Everything above becomes a platform when you turn it into defaults other teams inherit without thinking about them. The pattern that works is a small internal package — call it yourco_dspy — that owns the decisions and leaves applications with nothing to configure.

That package should own, at import time: provider registrations (before any LM is constructed, because of the binding rule); the cache configuration, with the disk cache either disabled or pointed at a bounded writable volume with restrict_pickle=True; the callback list; disable_history=True; the default adapter; max_errors; concurrency defaults; and the interpreter factory. It should expose a narrow surface: a function to build a tenant-scoped LM, a context manager that wraps a request, and a loader that refuses artifacts whose metadata does not match the pinned version.

import dspy
from dspy.lm15 import Config, Message, Request   # re-export the supported call shape

dspy.lm15.register_provider("internal-gateway", base_url=GATEWAY_URL,
                            api_key_env="INTERNAL_GATEWAY_KEY")

dspy.configure_cache(enable_disk_cache=False, memory_max_entries=50_000,
                     restrict_pickle=True)

dspy.configure(
    lm=dspy.LM("internal-gateway/qwen-32b-instruct", engine="lm15"),
    adapter=dspy.ChatAdapter(),
    disable_history=True,
    track_usage=True,
    max_errors=5,
    num_threads=8,
    async_max_workers=32,
    warn_on_type_mismatch=True,
    callbacks=[PlatformTelemetry()],
)

def load_program(cls, path: str):
    """JSON-only loader that refuses a version it was not blessed for."""
    program = cls()
    program.load(path, allow_pickle=False, allow_unsafe_lm_state=False)
    return program

A compile service, not a compile convention. Give teams one way to run an optimizer: a job with a budget argument, a named owner, a trainset reference, a log directory and an artifact output path. GEPA's log_dir and track_stats exist to make that auditable. A platform where every team writes its own compile script has no cost ceiling and no reproducibility, and you will find out during a budget review.

An artifact registry with promotion. Artifacts are JSON, small and reviewable. Store them with their DSPy version, their devset version, their score, the model string they were compiled against and their compile cost. Promotion is a pull request. Rollback is reverting that pull request. This is the single highest-leverage thing you can build, because it converts "the prompts changed" from an archaeology problem into a diff.

A golden-path example and an optimizer recommendation. Teams will ask which optimizer to use, and the official cheat sheet gives a defensible default path: start with BootstrapFewShot; move to BootstrapFewShotWithRandomSearch (aliased BootstrapRS) when demo quality varies; use COPRO or GEPA when the instructions are the problem; use MIPROv2 or GEPA when both are weak and you have budget; use BootstrapFinetune when prompt-only work has plateaued and the model is tunable; and combine with BetterTogether. Publish that as your platform's recommendation with your own budget caps attached, and most teams will never need to think about it again. One useful implementation detail: compiled modules are flagged _compiled=True so outer optimizers skip them, which is what makes composing optimizers safe.

The review checklist

This is what I actually read for, in order, on a pull request or a design document that touches a DSPy service.

Configuration and threading. Is dspy.configure called exactly once, at startup, from the main thread? Is every per-request override a with dspy.context(...)? If work is handed to a plain ThreadPoolExecutor, are thread_local_overrides captured and re-set in the worker, or has the author used dspy.Parallel instead? Is async_max_workers set to something chosen rather than the default 8?

Artifacts. Is the artifact JSON? Is it loaded with allow_pickle=False and allow_unsafe_lm_state=False? Is there a version assertion on the saved metadata rather than a reliance on the warning? Is the artifact's provenance — DSPy version, devset version, score, model string, compile cost — recorded somewhere a human can find it during an incident?

Secrets and regions. Are keys from the environment or a secret manager, never in code or artifacts? For Vertex, are vertex_project, vertex_location and the vertex_ai/ prefix used, with a startup assertion on the resolved region? Is there a CI check that artifacts contain no key-shaped strings?

Code execution and tools. Which interpreter factory is in force, and was it set by the platform package or by application code? Is LocalInterpreter absent from anything reachable by untrusted input? Are max_iters, max_llm_calls and max_output_chars set explicitly on RLM? Do destructive tools check the authenticated identity rather than the model's intent? Is every from_url() on an externally-supplied URL preceded by validation, given there is no SSRF allowlist?

Cost and budgets. Does every optimizer call set exactly one budget argument, and is it max_metric_calls rather than auto for anything in CI? Is the reflection or prompt model confined to compile time? Is track_usage=True on, and does the usage aggregation handle empty usage from cache hits and unknown cost without turning either into a zero?

Errors. Does the code catch dspy.LMError and subclasses rather than provider exceptions? Does retry classification use dspy.is_retryable_lm_error? Is max_errors tight on release gates and loose only on exploratory runs?

Evaluation. Is the devset in version control with an owner? Does this pull request change both the devset and the program? Is the metric strict when trace is not None? For GEPA metrics, is the feedback string specific enough to be useful to a reflecting model?

Deprecations. Does the diff introduce lm(messages=[...]), a custom BaseLM.forward, CodeAct, ProgramOfThought, dspy.Assert/Suggest, Image.from_file, field prefix/format/parser kwargs, or Evaluate(return_outputs=True)? Any of those is a change request, because all of them are on a removal path or already removed.

The complete picture

Put the three levels together and the shape is this. A signature declares a task. A module chooses a strategy for it. An adapter turns that into messages and parses the result back into typed fields. An LM, through an engine, talks to a provider. That is the whole inference path, and it is small enough to hold in your head — which is the point of the framework.

And around that sits everything this level added, which is almost entirely about the seam being a trust and cost boundary rather than a file path. The artifact must be JSON, loaded with pickle off and unsafe LM state off, with its provenance recorded. The compile must have a budget and an owner. The serving process must configure once, override per request through dspy.context, bound its caches, disable its history, and own its callbacks. The tenant boundary must be explicit in the cache key, the history setting and the process topology, because DSPy provides no tenant model of its own. The interpreter choice must be a platform decision, because one of the two options is not a sandbox. And the version pin must be exact, with a deprecation sweep standing between you and 3.5.

The failure modes reduce to three questions you can ask in any order and answer in minutes: did the prompt change, did the parse fail, or did the transport fail. Everything in your telemetry should exist to answer one of those three faster than a human can guess.

Where the series leaves you

You can now take a DSPy program from a notebook to a platform that other teams build on without you being in the loop for every decision. That means you can design the compile and serve split, defend the budget, set the trust boundaries, name the component that breaks first under load, run the three incidents that will actually happen, and say — with specifics — when DSPy is the wrong tool and what to reach for instead.

One last habit to keep: re-read the release notes for every DSPy minor version before you pin it, and run the deprecation sweep before you believe the upgrade was clean. The framework is moving quickly and deliberately, the maintainers document breaks precisely, and the half-hour that costs you is the cheapest insurance in this entire guide.

Sources

• DSPy documentation
• Installation
• Language models
• Modules
• Adapters
• Tools
• MCP
• Metrics
• Optimizers
• Settings and context
• Saving and loading
• Choosing an optimizer
• GEPA in depth
• RLM
• Normalized LM API migration
• Cache tutorial
• Async tutorial
• Streaming tutorial
• Deployment tutorial
• Observability tutorial
• Custom LM engines
• Production
• Cheatsheet
• DSPy 3.4.0 release notes
• DSPy 3.3.1 release notes
• DSPy 3.3.0 release notes
• SECURITY.md
• dspy on PyPI
• MLflow DSPy integration

At this level the interview stops being about syntax. Nobody is going to ask you what a signature is. They are going to hand you an ambiguous situation — a bill that tripled, a team that wants to run optimizers in production, a compliance question about where inference happens — and watch how you reason. The version in play is DSPy 3.4.0, which matters because the engine transition and the 3.5 removal deadline are live topics an informed interviewer will probe.

What they are actually testing

Three things, in roughly this order of weight.

Do you know what DSPy is and is not? DSPy is an in-process Python library with no server, no scheduler and no CLI. Every candidate who treats it as a platform gets caught within two questions, because the follow-up is always "so where does the autoscaling come from". The strong answer names the surrounding infrastructure explicitly.

Can you draw the trust and cost boundary? The seam between compiling and serving is where every senior-level question lives. If you can explain why compile must be offline, what the artifact contains, what it refuses to contain, and what that implies for security and cost, you have answered half the interview before it is asked.

Do you own failures or describe them? Incident questions are not testing whether you know the error names. They are testing whether you classify before acting. The three families — prompt, parse, transport — are the frame worth bringing to every scenario.

System-design prompts

"Design a shared DSPy platform for five product teams"

Open with the split, because it is the load-bearing decision. A compile plane that runs optimizers as budgeted offline jobs, and a serve plane that instantiates a program class, loads saved state and answers requests. The artifact crossing the boundary is a program.json plus the code that defines the program — and say out loud that the pair is required, because the state file holds instructions, demos and field metadata but is meaningless without the class.

Then name what the platform owns versus what teams own. The platform owns the version pin (exactly dspy==3.4.0, not a range, because optimizer output is a function of the library), the dependency set in the base image, provider credentials, cache configuration, telemetry callbacks and the serving runtime. Teams own signatures, metrics, devsets and artifact quality. Most internal-platform friction is two people both assuming the other owns the metric; saying so signals you have run one of these.

Describe a small internal package that sets the defaults at import time: dspy.lm15.register_provider(...) before any LM is constructed, because each LM binds the registrations present at construction time; dspy.configure_cache(...) with the disk cache either disabled or pointed at a bounded writable volume; and a single dspy.configure(...) with disable_history=True, track_usage=True, a chosen async_max_workers, platform-owned callbacks and the interpreter factory. Note that dspy.configure(interpreter_factory=...) is global, so one team setting it changes the trust model for every program in the process — which is exactly why it belongs to the platform.

Close with the artifact registry and promotion flow: artifacts stored with their DSPy version, devset version, score, model string and compile cost; promotion as a pull request; rollback as a revert. That converts "the prompts changed" from archaeology into a diff, and it is the highest-leverage thing you can build.

"A team wants to run GEPA inside a request handler. Argue them out of it."

Do not lead with "that's wrong". Lead with the economics, because that is the argument that wins. DSPy's optimization model assumes you compile once and serve many times; the official docs are direct that GEPA and MIPROv2 runs can cost hundreds of dollars in LM calls. The standard cost structure is a cheap task model at serving time paired with a strong reflection_lm only at compile time. Running the optimizer in the request path puts the expensive model in the hot path and removes the ceiling the whole design depends on.

Then the operational argument: there is no budget, no reproducibility and no release gate. GEPA requires exactly one of auto, max_full_evals or max_metric_calls — it raises AssertionError: Exactly one of max_metric_calls, max_full_evals, auto must be set. otherwise — and a request handler has nowhere sensible to put that number. Offer max_metric_calls as the platform default over auto, because a metric-call ceiling is a figure a finance owner can reason about while "heavy" is not.

Finally, offer the thing they actually wanted. If the goal is per-tenant adaptation, that is a per-tenant compile job producing a per-tenant artifact, not an inline optimizer.

"We have a data-residency requirement. Where does inference happen, and how do you prove it?"

This one rewards specificity, especially for Gulf and Egyptian employers. If you route to Vertex AI in a particular region, you must use the vertex_project and vertex_location kwargs together with the vertex_ai/ model prefix. Using project/location, or the gemini/ prefix, sends requests to the wrong region silently — no exception, no log line. That is the detail that separates someone who has shipped this from someone who has read about it.

Because the failure is silent, the proof has to be a startup assertion on the resolved configuration plus a test, not a configuration review. Add that API keys are never serialized (LM.dump_state excludes api_key) and that loading with the default allow_unsafe_lm_state=False strips api_base, base_url and model_list from saved state — so a tampered artifact cannot redirect traffic to an out-of-region endpoint. That is a genuine architectural control, and naming it is much stronger than promising vigilance.

"Design multi-tenancy on top of DSPy"

Start by saying DSPy has no built-in tenant model, so everything is a pattern you implement and own. Per-request isolation is with dspy.context(lm=tenant_lm, ...), which is ContextVar-scoped and async-safe; per-stage isolation is predictor.set_lm(other_lm).

Then name the three leaks, because this is where interviewers are looking for depth. The cache key is a hash of the request arguments, not the tenant, so two tenants sending the same prompt share an entry — a cost win for most workloads and a data-sharing bug for any workload where one tenant's output must not be derivable from another's request; mitigate with cache=False or distinct rollout_id values. lm.history and the global history retain prompts in memory with max_history_size defaulting to 10,000, so disable_history=True belongs in production serving. And callbacks registered via dspy.configure are process-wide and see every tenant, which is right for platform telemetry and unacceptable for tenant-registered code.

End on topology. A process per trust tier — not per tenant, which does not scale — physically separates credentials, caches, history and blast radius, and makes every compliance conversation shorter.

Incident-response questions

"Quality dropped overnight. No errors. Go."

The frame: prompt, parse or transport. No errors means you are in the prompt family. First action is to establish whether the artifact changed — compare the loaded artifact against the promoted one — then run dspy.inspect_history(n=1) and read the instructions and demos. If the prompt is the one you expect, the provider changed the model behind a floating alias, which is the argument for pinning model versions in the model string wherever the provider permits it. Rollback order: artifact, then version pin, then model string.

What a weak answer does here is start tuning. A strong answer establishes what changed before changing anything, and knows that the artifact is small, JSON and diffable precisely so this takes minutes.

"A burst of AdapterParseError on a program that was fine yesterday"

Name the three realistic causes: the model ignored the format, the response was truncated because max_tokens was too low, or the response was empty. In production the overwhelming favourite is truncation, because longer inputs or more demos pushed the completion against the limit — and a half-written answer looks exactly like a formatting problem. So: finish reason and token counts first, adapter second.

If the content genuinely changed shape, JSONAdapter or TwoStepAdapter with a separate extraction_model is the structural fix. Add the detail that since 3.3, ChatAdapter and JSONAdapter no longer retry an alternate format after an LMError, so a transport failure is not disguised as a parse retry — and that dspy.ChatAdapter(use_json_adapter_fallback=False) is how you make a test environment fail loudly instead of silently repairing itself.

"p99 latency has a cliff and the provider says we are nowhere near our rate limit"

This is the async_max_workers question and it is a favourite because the symptom is so misleading. dspy.asyncify(program) runs a synchronous program in worker threads bounded by async_max_workers, which defaults to 8. The ninth concurrent call waits. Nothing errors and nothing logs; latency simply doubles. The tell is that the cliff sits at a fixed concurrency rather than a fixed rate.

Fix: raise it deliberately, then raise num_threads too if the threaded paths are involved, until the provider starts returning LMRateLimitError, and scale processes from there. The better long-term answer is native async — write aforward on your modules and use acall on built-ins — because asyncify is a thread pool wearing an async coat.

"A tenant's traffic is running on the wrong model and nobody can reproduce it"

Context propagation. dspy.Parallel, Module.batch and asyncify copy context overrides into their workers; a plain threading.Thread or ThreadPoolExecutor does not. A tenant-scoped dspy.context(lm=...) override evaporates in a hand-rolled worker, and the tenant's traffic quietly runs on the process default. Mention the fix precisely: capture dspy.settings.thread_local_overrides.get() on the calling side and re-set it inside the worker, or use dspy.Parallel and let DSPy handle it.

The reason this question works as a filter is that the bug has no error message. It gets found in a bill.

"Rate limiting at the same time every afternoon"

dspy.LMRateLimitError carries .retry_after and .provider, and 3.4 retains Retry-After, request IDs and headers on rate-limit errors. Lower num_threads and async_max_workers, let num_retries (default 3) absorb the transients, and alert on the rate of retryable errors rather than on each one. Use dspy.is_retryable_lm_error(e) for classification rather than inventing a status-code table, and catch dspy.LMError subclasses rather than provider exceptions — since 3.3, LM errors are normalized into DSPy's own tree.

The senior addition: capture .request_id on every error, because a provider support ticket without request IDs goes nowhere, and the moment you need one is the moment you have stopped logging them.

Arguing a trade-off to leadership

Leadership conversations about DSPy reduce to three recurring arguments. Rehearse all three.

"Why pin an exact version when we pin ranges everywhere else?" Because the artifact you ship is a prompt an optimizer produced, and optimizer output is a function of the library. A range means prompts can drift when a dependency resolves differently, with no code change to point at during the incident. Add that the saved JSON state is designed to be backward compatible and a version mismatch only logs a warning — a warning is not a gate, so you assert on the metadata at load time. That is the kind of detail that makes the pin sound like engineering rather than caution.

"Why does compiling cost so much, and why can't we do it continuously?" Frame it as capital versus operating expense. Compile is lumpy, discretionary and expensive by design because an expensive reflection model is in the loop; serving is cheap and proportional to traffic. The lever you offer is not "spend less" but "spend deliberately": max_metric_calls as a hard ceiling, a named owner per compile job, and a release gate so a compile that does not beat the incumbent does not ship. Then show the measurement: track_usage=True with pred.get_lm_usage(), and lm.history[i]["cost"]. Be honest about the two caveats, because being honest about them is what makes the number credible — cached calls report empty usage, so a cache-heavy workload looks cheaper than it is, and unknown cost stays unknown rather than zero, so the aggregation has to handle nulls instead of summing them into a confidently wrong total.

"Can we let teams use the local interpreter? It's faster." This is the one to hold the line on, and the argument is a sentence from the release notes rather than your opinion: dspy.LocalInterpreter runs generated Python in a persistent CPython subprocess and is not a security sandbox — generated code keeps the host user's filesystem, environment, credentials, subprocess and network access. The default PythonInterpreter is a Deno plus Pyodide sandbox, hardened in 3.3.1. So the framing to leadership is: this is a request to let a prompt injection become shell access with the service's credentials, in exchange for a startup-time saving. Offer the compromise that actually works — LocalInterpreter in developer environments with trusted inputs, never in a path reachable by tenant data — and note that because the interpreter factory is set globally, the control has to live in the platform package.

A fourth argument shows up more and more: "Should we build this on DSPy at all?" Knowing where DSPy stops is a senior signal, not a disloyalty. If there is no metric and no dataset, there is nothing to optimize and a thin client is more honest. If the hard part is long-running stateful orchestration, that is a workflow engine with DSPy modules as nodes. If you need autoscaling, canaries and traffic splitting as first-class concepts, that is a serving platform with DSPy inside the container. Saying so, unprompted, usually ends the technical portion of the interview well.

The sixty-second self-test

Answer these out loud. Hesitation on any of them is a reading assignment, not a failure.

What is the default engine in 3.4, and when does the fallback decision get made? What never triggers a backend switch? What does native n>1 do differently from engine="litellm", and how does that show on the bill? Which thread may call dspy.configure, and what is the message when another one tries? Which DSPy constructs propagate context into workers, and which one famously does not? What is async_max_workers' default and what is the symptom when you hit it? What are the three cache layers, what is the key derived from, and what are the two disk-cache defaults a server should never inherit? What does program.save("p.json") deliberately refuse to store, and what does loading strip by default? Which flag do you need to load a .pkl, and why is that a code smell? Which interpreter is not a sandbox? Which four APIs are deprecated in 3.4 and removed in 3.5? What exactly must be set for a GEPA budget, and what is raised if you get it wrong? What is EvaluationResult.score's range and precision? What happens if you pass return_outputs=True to Evaluate? What is the one-line command that surfaces your 3.5 migration backlog?

That last one is the answer most worth having ready, because it is both a fact and a habit: python -W default::DeprecationWarning your_program.py. Candidates who volunteer it tend to be the ones who have actually done an upgrade.

These are the things that do not show up in a tutorial because they only appear once a DSPy program has other teams depending on it, a bill attached to it, and a pager pointed at it. Everything below is current for DSPy 3.4.0.

Traps that only appear at scale

The eight-concurrency cliff. dspy.asyncify(program) runs a synchronous program in worker threads bounded by async_max_workers, which defaults to 8. The ninth concurrent request waits. There is no error and no log line — latency simply doubles. Teams spend days blaming the provider for a ceiling that is in their own configuration. Set async_max_workers explicitly in your platform defaults, so the number in production is one somebody chose. The same applies to num_threads (default 8) on the threaded paths: Evaluate, Parallel and Module.batch.

Thirty gigabytes of disk cache in a container. The on-disk diskcache.FanoutCache defaults to ~/.dspy_cache with a limit from DSPY_CACHE_LIMIT of 3e10 bytes — 30 GB — and the in-memory LRU defaults to 1,000,000 entries. Neither default is wrong on a laptop and neither is right in a fleet. In a read-only container the disk cache fails to write at all; in a writable one it will happily eat the volume. Decide once, in the platform package: either enable_disk_cache=False with a bounded memory_max_entries, or a mounted volume with an explicit disk_size_limit_bytes.

The cache that crosses tenants. The cache key is a hash of the request arguments, excluding credentials. It is not keyed on the tenant. Two tenants sending the same prompt share an entry. For most workloads that is free money; for any workload where one tenant's output must not be derivable from another's request, it is a data-sharing bug that looks like a performance feature. Mitigate with cache=False on tenant LMs, or distinct rollout_id values so the keys differ. Make it a field in tenant configuration, not a tribal convention.

Prompts living in RSS. lm.history and the global history retain prompts and responses, with max_history_size defaulting to 10,000 and disable_history defaulting to False. In a multi-tenant process that is cross-tenant request content sitting in memory, available to anything that can read the process — including a crash dump that lands in your logging system. disable_history=True in production serving, with structured telemetry you own instead.

Context that evaporates in a worker thread. dspy.Parallel, Module.batch and asyncify copy context overrides into their workers. A plain threading.Thread or ThreadPoolExecutor does not. A tenant-scoped dspy.context(lm=...) silently reverts to the process default inside a hand-rolled worker. There is no error. The bug gets found in a bill. Either capture dspy.settings.thread_local_overrides.get() and re-set it in the worker, or use dspy.Parallel.

Usage that under-reports. Cached calls report empty usage, and unknown cost stays unknown rather than being reported as zero. A cache-heavy workload therefore looks dramatically cheaper than it is, and an aggregation that sums costs without handling nulls produces a number that is confidently wrong. Count cache hits as a separate series, and make your cost pipeline null-aware before anyone builds a chargeback on it.

Silent region drift. Using project/location instead of vertex_project/vertex_location, or the gemini/ prefix instead of vertex_ai/, sends Vertex requests to the wrong region with no error. For anyone with a data-residency commitment — which in practice means most Gulf and Egyptian employers — this is the trap with the worst consequence-to-visibility ratio in the whole framework. It needs a startup assertion, not a configuration review.

Streams that cannot be healed. A stream is not retried once it has emitted a chunk, and incomplete responses are not cached. A streaming endpoint has a genuinely worse failure profile than a buffered one. If your SLO is about correctness rather than perceived latency, buffering is the defensible choice — and if you stream anyway, make sure the client can tolerate a truncated response.

Native n>1 billing per sample. The native lm15 engine makes separate sequential requests for n>1, so input tokens are billed once per sample. If you want provider-native multi-sample generation, you need engine="litellm". Teams who switch engines for latency and keep an n=5 call pattern discover this on an invoice.

Registrations that bind at construction. dspy.lm15.register_provider(...) is bound by each LM at construction time. Register a provider after an LM exists and that LM cannot see it. In a long-lived server this presents as "the new gateway works in tests and not in production". Register everything at import time, before any LM is built.

Optional extras discovered in production. Several things that used to be automatic are now extras, which turns a missing capability into a ModuleNotFoundError at the worst moment. NumPy became optional in 3.3, so dspy[numpy] is required for embeddings, KNN/KNNFewShot and SIMBA. optuna became optional in 3.2, so MIPROv2 and BootstrapFewShotWithOptuna need dspy[optuna]. dspy[deno] installs the managed Deno runtime that PythonInterpreter and RLM depend on, validating Deno >=2.0.0,<3.0.0. dspy[mcp] covers MCP tools. Pick one dependency set for the fleet and bake it into the base image; the failure mode otherwise is a team reaching for SIMBA on a Friday and finding out NumPy is not installed.

Config that does not inherit LM defaults. When you move to the explicit Request form, generation options go in Config and the LM's own defaults are not merged in. Set max_tokens on the LM, send a Request without it, and you did not inherit it. The related name collision is worth internalizing too: Config.cache controls the provider's prompt caching, not DSPy's response cache. Two different mechanisms, one confusing word.

Reasoning-model parameter floors. OpenAI's reasoning models require temperature=1.0 or None and max_tokens >= 16000 or None, and violating that raises LMConfigurationError with the correct values in the message. A sixteen-thousand-token floor is a real cost floor, not a formality — budget for it before you route a high-volume path at a reasoning model, and remember that model_type="responses" is its own code path with its own behaviour.

A warning where you wanted a gate. A DSPy version mismatch between a saved artifact's metadata and the running library logs a warning only. In a serving log, a warning is invisible. Assert on the metadata at load time and refuse a combination you have not blessed. load_state is transactional, so a strict check is safe to add — a failed load leaves the program in its prior state.

Platform-ownership habits

Pin exactly, everywhere. dspy==3.4.0, including on developer laptops. The thing you are protecting is not only runtime behaviour but the prompts an optimizer produced, which are a function of the library. Decide one dependency set for the fleet — the extras you need from numpy, optuna, deno, mcp, anthropic, typesafe, langchain, weaviate — and bake it into the base image, so nobody discovers an optional extra as a ModuleNotFoundError in production. Remember gepa[dspy]==0.1.4 is already pinned exactly by DSPy itself, which is a reproducibility win and an occasional dependency conflict.

Run the deprecation sweep on a schedule, not on an upgrade. python -W default::DeprecationWarning over your test suite, in CI, with the output tracked. The list it produces is your 3.5 backlog: lm(messages=[...]), custom BaseLM.forward()/aforward(), LegacyEngine/AsyncLegacyEngine/complete_legacy(), dspy.CodeAct, dspy.ProgramOfThought, the Image.from_file/from_PIL/Audio.from_file aliases, and the prefix/format/parser field kwargs. Doing this continuously turns an upgrade into a non-event.

Make JSON the only artifact format that crosses a trust boundary. program.save("p.json") is state-only: instructions, demos, signature field metadata, and LM config minus api_key, plus metadata recording the Python, DSPy and cloudpickle versions. It is small, inspectable and reviewable in a pull request, and it cannot execute. .pkl and whole-program saves require allow_pickle=True to load, with an explicit warning that loading them can run arbitrary code. Treat allow_pickle=True as a code smell that needs a named owner and a reason.

Load defensively by default. program.load(path, allow_pickle=False, allow_unsafe_lm_state=False). The second flag is doing real work: at its default, loading strips api_base, base_url and model_list from saved LM config, so a tampered artifact cannot redirect your traffic to someone else's endpoint. Put a one-line loader in your platform package and make it the only sanctioned path.

Own the global knobs and take them away from application code. dspy.configure is process-wide and may only be called from the thread or async task that first called it. dspy.configure(interpreter_factory=...) replaces the interpreter factory globally. Callbacks registered through configure see every module, LM, adapter, tool, evaluation and compile event for every tenant. All three belong to the platform package, set once at import. Application code gets with dspy.context(...) and nothing else.

Treat the devset as a controlled artifact. It is your release gate, so it belongs in version control with an owner and a changelog, and changing the devset must be a separate pull request from promoting a program. If both move at once, a score improvement is uninterpretable — and the first time a regression slips through on a quietly relaxed devset, every historical number becomes untrustworthy. Keep .results, not just .score, because the per-example list is what lets you show which cases changed.

Budget every compile with a number, not an adjective. Prefer max_metric_calls over auto for anything in CI. GEPA requires exactly one of auto, max_full_evals or max_metric_calls, and a metric-call ceiling is something a finance owner can reason about while "heavy" is not. Pair it with log_dir and track_stats so the run is auditable afterwards.

Keep the expensive model at compile time only. A cheap task model with a strong reflection_lm or prompt model used only during optimization is the standard cost structure, and it only holds if compile never runs in a request path. Guard that with a lint rule if you have to.

Alert on error rates, not errors. num_retries defaults to 3 and absorbs ordinary transients. Classify with dspy.is_retryable_lm_error(e) rather than your own status-code table, catch dspy.LMError and subclasses rather than provider exceptions, and page on LMAuthError and LMBillingError immediately because those are never transient. Capture .request_id on everything — provider support tickets without them go nowhere.

Link runbooks at a pinned docs snapshot. Since 3.4 the documentation site has a version picker: "Current" tracks main and each stable release has a snapshot. A runbook pointed at "Current" drifts away from your fleet without anybody editing it.

Drill the rollback before you need it. The artifact is JSON and small, which makes rollback genuinely fast — but only if someone has done it once. Keep the previous artifact and the previous version pin available as a single revert, and practise the sequence: artifact first, then the version pin, then the model string, in that order of likelihood. Teams that skip the drill discover during an incident that the previous artifact was never actually stored anywhere durable, only in the CI job that produced it and has since been garbage-collected.

Adapter defaults that quietly fill themselves in. Since 3.3.1, adapters apply declared defaults when an optional output field is omitted, while missing required outputs still raise AdapterParseError. That is sensible behaviour and it means an output field silently taking its default is a signature design decision rather than a bug. Review optional output fields with that in mind, because a field that is optional with a plausible default will never tell you the model stopped producing it.

Checklist: reviewing a pull request

• dspy.configure is called once, at startup, from the main thread; every per-request override is with dspy.context(...).
• No work is handed to a bare ThreadPoolExecutor without re-setting thread_local_overrides — or dspy.Parallel is used instead.
• async_max_workers and num_threads are set deliberately, not inherited.
• Artifacts are JSON, loaded with allow_pickle=False and allow_unsafe_lm_state=False, and the saved metadata version is asserted rather than warned about.
• Artifact provenance is recorded: DSPy version, devset version, score, model string, compile cost.
• No secret appears in code or in an artifact; there is a CI check that greps artifacts for key-shaped strings.
• Vertex routing uses vertex_project, vertex_location and the vertex_ai/ prefix, with a startup assertion on the resolved region.
• dspy.LocalInterpreter is absent from anything reachable by untrusted input; interpreter_factory is set by the platform package only.
• RLM calls set max_iters, max_llm_calls and max_output_chars explicitly, and use the keyword-only interpreter_factory= with a zero-arg factory.
• Any from_url() on an externally supplied URL is preceded by validation, because there is no SSRF allowlist, and timeout is left bounded.
• Destructive tools check the authenticated identity, not the model's intent.
• Every optimizer call sets exactly one budget argument.
• track_usage=True is on, and the usage aggregation handles empty usage from cache hits and unknown cost without turning either into zero.
• Errors are caught as dspy.LMError subclasses; retry classification uses dspy.is_retryable_lm_error.
• max_errors is tight on release gates and loose only on exploratory runs.
• The metric is stricter when trace is not None; GEPA feedback strings are specific enough to be useful to a reflecting model.
• The diff does not introduce lm(messages=[...]), a custom BaseLM.forward, CodeAct, ProgramOfThought, dspy.Assert/Suggest, Image.from_file, field prefix/format/parser kwargs, or Evaluate(return_outputs=True).
• The devset and the program it gates do not change in the same pull request.

Checklist: reviewing a design document

• Is there an explicit compile plane and serve plane, with a named artifact crossing between them?
• Who owns the compile budget, and what is the number?
• What is the release gate, and what happens when a new compile does not beat the incumbent?
• What is the rollback procedure, and has it been executed once in a drill?
• Is the tenant boundary stated in terms of the cache key, the history setting and the process topology — or is it hand-waved?
• Does the design say what the platform owns and what the team owns, in enough detail that neither side can assume the other owns the metric?
• Which component is expected to break first under load, and what is the evidence? "The provider" is the right answer; the number attached to it should come from a load test, not a guess.
• Is there a stated position on streaming versus buffering, given that streams are not retried after the first chunk?
• Does the observability plan cover all three failure families — prompt, parse, transport — and can it distinguish them without a human guessing?
• Is there a statement of where DSPy stops for this system, and what the neighbouring tool is?

Habits that compound

Read the release notes for every minor version before you pin it. The maintainers document breaks precisely — the 3.4 notes name every removed experimental type and its replacement — and thirty minutes of reading has repeatedly saved teams a week of mystery.

Reconcile your cost model against two independent sources before you present it: with dspy.track_usage() totals against the sum of lm.history[i]["cost"] for the same window. If they disagree, understand why before anyone builds a budget on either.

Keep one golden-path example in the platform repository that a new team can copy — configuration, a program class, a metric, a devset, a compile job with a budget, an artifact load, a served endpoint — and keep it running in CI. An example that still works is worth more documentation than any document.

And run one incident drill a quarter: roll an artifact back, point a tenant at a different model, and prove the deprecation sweep is clean. All three are cheap when nothing is on fire, and all three are what you will be asked to do when something is.