The Complete Haystack Guide

This is part one of three. By the end of it you can install Haystack, build a pipeline out of typed components, index a folder of your own documents, ask questions about them with a language model, save the whole thing to a YAML file and load it back, and read the error messages Haystack produces well enough to fix them yourself. Nothing here is a teaser for the paid version of anything: Haystack is an open-source Python library, and everything below runs on a laptop.

Each section ends with a Try it task. Do them as you go. Pipelines are one of those ideas that look obvious on a page and then behave unexpectedly the first time you wire two components together, and the only cure is to have wired two components together.

One thing to settle before the first line of code. This guide targets Haystack 3.2.0, the current stable release. Haystack 3.0 arrived in July 2026 and removed a long list of things that 2.x code used daily. Most blog posts, most Stack Overflow answers and most of what a language model will confidently tell you about Haystack describe 2.x. If you copy an example that imports AsyncPipeline or OpenAIGenerator, it will not run, and the error will not explain why. Section after section below flags the places where 2.x habits break, because recognising them is most of what separates a frustrating first week from a productive one.

What Haystack is, and the problem it solves

Haystack is a Python framework from deepset for building applications on top of language models: question answering over your own documents, search, chatbots, agents that call tools. Its central claim is small and concrete. Instead of writing one long script that fetches documents, builds a prompt string, calls an API and parses the reply, you build a graph of small typed parts, and the framework checks that the parts fit together before you run anything.

To see why that matters, write the long script first, in your head. A retrieval-augmented question answering app — the standard first project in this field, usually shortened to RAG, for retrieval-augmented generation — does roughly this: take the user's question, find the handful of passages in your corpus most likely to contain the answer, paste those passages into a prompt along with the question, send that prompt to a language model, and return the model's reply. Five steps, maybe sixty lines.

The sixty lines are fine until you need to change something. Swap the keyword search for a vector search and the retrieval step's inputs change shape. Add a reranking step between retrieval and the prompt and you are reordering function calls and hoping you passed the right variable. Serve it behind a web API and you discover the model client was built at import time with a key that was not set yet. Run the same app against a second corpus and you copy the file. Each of these is a small problem, and they compound into the thing that makes LLM codebases unpleasant: nothing has a declared shape, so nothing can be checked, so every change is a guess verified by running it.

Haystack's answer is to make the shape explicit. A component declares what types it accepts and what types it emits. A pipeline is a set of components plus a set of connections between their named inputs and outputs, and connections are type-checked when you create them, not when you run the pipeline. If you connect a component that emits list[Document] to one that expects str, you get an error at wiring time, with both type names in the message, before you have spent a single token of API budget.

The second half of the claim is that the graph is data. A pipeline can be serialised to YAML and loaded back, which means the structure of your application can live in a file that is reviewed, versioned and deployed separately from the code that runs it. That is an unusual property for this kind of library and it is the reason Haystack shows up in teams that already have opinions about configuration management.

What came before is worth knowing, because it explains some of Haystack's shape. Haystack 1.x, released long before the current wave of chat models, was built around extractive question answering: a reader model pointed at a span of text inside a retrieved document and said "the answer is these words, here". It had Node classes, a YAML-first design and a quite rigid pipeline. Haystack 2.x, a full rewrite, replaced that with the component-and-socket model you will learn here and made generative models first-class. Haystack 3.x keeps the 2.x model and cleans up the parts that had accumulated duplication — most visibly by deleting the old non-chat generators and the separate asynchronous pipeline class. The 1.x package is still on PyPI under the name farm-haystack, which matters for one practical reason covered in the install section: it claims the same haystack import name, so the two cannot live in one environment.

The four nouns

Almost everything in Haystack is one of four things. Learn these names properly and the API documentation stops being a maze.

A Component is a Python class decorated with @component that has a run() method returning a dictionary. The parameters of run() are its inputs. The keys of the returned dictionary are its outputs, declared ahead of time with the @component.output_types(...) decorator. Each named input and output is called a socket, and a socket has a type. That is the whole contract. A component can be used entirely on its own — call run() yourself — or dropped into a pipeline.

A Pipeline is a directed multigraph of components. "Multigraph" matters: two components can be joined by more than one connection, because connections are between sockets, not between components. Data travels only along the connections you declare. A component sitting in the pipeline with nothing wired into it does not magically receive the inputs you passed to its neighbour. This is the single most common early misunderstanding, and the cure is to read the graph as a set of wires rather than a sequence of steps.

A Document is a dataclass holding a piece of content plus everything you know about it: id, content, blob for binary data, meta for your own metadata dictionary, score for relevance after retrieval, and embedding and sparse_embedding for vectors. Documents are the currency of the retrieval half of Haystack. Converters produce them, splitters cut them up, embedders add vectors to them, retrievers return them with scores attached, and prompt builders loop over them to build text.

A Document Store is a database interface — and it is not a component. It has no run() method and cannot be added to a pipeline. Its protocol has exactly four methods: count_documents, filter_documents, write_documents and delete_documents. Components that need storage hold a reference to one. So a retriever takes document_store=... in its constructor, and the thing you add to the pipeline is the retriever. Beginners try to add the store and get a confusing error; now you know why.

Two more words you will meet within the hour. A ChatMessage is a message with a role (system, user, assistant or tool) and content parts; you build one with ChatMessage.from_user(...), ChatMessage.from_system(...) and friends, and you read its text with .text. A Chat Generator is the component that actually calls a language model, for example OpenAIChatGenerator. It takes messages and returns {"replies": [ChatMessage, ...]}.

That diagram is the shape of nearly every Haystack application you will build this year. The top lane runs occasionally and writes into the store. The bottom lane runs on every request and reads from it. They are separate pipelines that happen to share one store object, and keeping them separate is a deliberate choice rather than an accident: indexing is slow, batchy and idempotent, while querying is fast and latency-sensitive.

There is also a lifecycle worth naming now because it explains an error you will hit. Components acquire expensive resources — model weights, HTTP clients — in warm_up(), not in __init__. Haystack calls warm_up() automatically before the first run, and close() releases the resources. The consequence: constructing a chat generator with no API key set succeeds silently, and the failure appears later, at warm-up or on the first run. In Haystack 2.x it failed at construction. If you remember that change you will save yourself a confused ten minutes.

Installing Haystack and checking it works

Haystack needs Python 3.10 or later. The package on PyPI is called haystack-ai, which is a common tripwire: the import name is haystack but the install name is not.

Use a virtual environment. Not as a matter of taste — Haystack's optional dependencies are numerous and you will install several of them experimentally, and you want that mess scoped to one folder.

python3 -m venv .venv
source .venv/bin/activate
pip install haystack-ai

On Windows PowerShell the equivalent is three commands:

py -m venv .venv
.venv\Scripts\Activate.ps1
pip install haystack-ai

If your team standardises on uv, either form works:

uv pip install haystack-ai     # as an installer, inside an existing environment
uv add haystack-ai             # as a project dependency, writing to pyproject.toml

And for conda users:

conda install conda-forge::haystack-ai

Now verify. Do not trust pip install output; ask the library what it thinks its own version is.

python -c "from haystack.version import __version__; print(__version__)"

3.2.0

If that prints something starting with 2., your environment has an older release and roughly a third of the code in this guide will fail. Upgrade with pip install --upgrade haystack-ai — the package name did not change between 2.x and 3.x, only the contents. pip show haystack-ai gives you the full picture including where it was installed, which is useful when you suspect you are in the wrong environment.

There is a Docker image if you prefer not to install Python things on your laptop at all. The only flavour is base, which is equivalent to pip install haystack-ai and nothing more.

docker pull deepset/haystack:base-v3.2.0
docker run -it --rm deepset/haystack:base-v3.2.0 \
  python -c "from haystack.version import __version__; print(__version__)"

To add integrations to that image, extend it:

FROM deepset/haystack:base-v3.2.0
RUN pip install sentence-transformers-haystack qdrant-haystack

If containers are new to you, the Docker guide in this series covers that FROM line and everything around it.

Optional dependencies are lazy, and this is a feature. Haystack does not install a PDF parser, an HTML extractor or a tokeniser unless you ask. The first time you use a component that needs one, you get a precise message:

Haystack failed to import the optional dependency 'pypdf'. Run 'pip install pypdf'.

Do what it says. The common ones are pypdf for PDF conversion, trafilatura for HTML extraction, tiktoken for token-based splitting, and rich for the fancy console confirmation UI.

Last piece of setup: an API key, if you want to call a hosted model. OpenAIChatGenerator reads OPENAI_API_KEY from the environment.

export OPENAI_API_KEY="sk-..."

$env:OPENAI_API_KEY="sk-..."    # this session only
setx OPENAI_API_KEY "sk-..."    # persisted for future sessions

Two related environment variables are worth knowing early because they explain hangs: OPENAI_TIMEOUT defaults to 30 seconds and OPENAI_MAX_RETRIES defaults to 5. A badly behaving endpoint can therefore keep your process busy for well over two minutes before raising.

You can do real work with no key at all. Haystack 3.0 added MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder precisely so that tests and tutorials can exercise pipeline structure without network calls or spend. A keyword-search pipeline over InMemoryDocumentStore also needs nothing external. Both appear below.

Your first component

Before pipelines, build a component on its own. It is ten lines and it makes the contract concrete.

from haystack import component

@component
class WelcomeTextGenerator:
    @component.output_types(welcome_text=str, note=str)
    def run(self, name: str):
        return {"welcome_text": f"Hello {name}".upper(), "note": "ready"}

greeter = WelcomeTextGenerator()
print(greeter.run(name="Layla"))

{'welcome_text': 'HELLO LAYLA', 'note': 'ready'}

Read every line of that against the rules from the previous section. The @component decorator registers the class. run() takes one parameter, name, typed as str — that is one input socket called name. The decorator declares two output sockets, welcome_text and note, both strings. The returned dictionary's keys match the declared outputs exactly; if they do not, Haystack raises rather than silently dropping data. And because the class is a normal Python object, you can call run() directly, which makes components genuinely unit-testable without a pipeline anywhere in sight.

Three rules about components that will save you debugging time later. Do not mutate your inputs. If you receive a list of documents and want to change them, copy the list or use dataclasses.replace() on each document; the same objects may be handed to another component. Store every constructor argument on an attribute of the same name — self.model = model for a model= parameter — because the default serialisation walks the constructor signature and reads the matching attributes, and a mismatch produces an error at save time rather than a wrong file. And if you add an asynchronous version, it must be called run_async, must be a coroutine, and must declare exactly the same parameters and output types as run; Haystack checks all three and the error messages say so plainly.

Documents, stores and a first pipeline

Now the retrieval half. Start with the simplest store there is.

from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore

store = InMemoryDocumentStore()
store.write_documents([
    Document(content="Riyadh is the capital of Saudi Arabia.", meta={"topic": "geography"}),
    Document(content="Cairo sits on the Nile and is the capital of Egypt.", meta={"topic": "geography"}),
    Document(content="Haystack pipelines are directed multigraphs of components.", meta={"topic": "software"}),
])
print(store.count_documents())

3

InMemoryDocumentStore keeps documents in process memory. It is for prototyping, tutorials and tests; it disappears when the process exits and is not shared between server replicas. It does have save_to_disk and load_from_disk if you want a prototype to survive a restart. For anything real you swap in an integration package — Qdrant, pgvector, Weaviate, Chroma and many others are supported — and the rest of your pipeline does not change, which is the point of the store being an interface.

Notice the meta dictionary. You will use it constantly: to filter retrieval by language or department or date, to show the source of an answer, and to separate one tenant's documents from another's. Metadata filters in Haystack have a uniform shape. A comparison filter is a dictionary with three keys:

{"field": "meta.topic", "operator": "==", "value": "geography"}

The operators are ==, !=, >, >=, <, <=, in and not in. A logic filter combines them:

{"operator": "AND", "conditions": [
    {"field": "meta.topic", "operator": "==", "value": "geography"},
    {"field": "meta.year", "operator": ">=", "value": 2024},
]}

The logical operators are AND, OR and NOT. You can pass filters straight to store.filter_documents(filters=...) or to any retriever's filters argument. Support varies by store — Chroma, for example, has no NOT but adds a contains operator — and an operator a store does not support raises FilterError. Writing "gt" instead of ">" raises the same error, and that typo is common enough to be worth memorising.

Now the first pipeline. Keyword search, no model, no API key, nothing to pay for.

from haystack import Pipeline
from haystack.components.retrievers import InMemoryBM25Retriever

pipe = Pipeline()
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=store, top_k=2))

result = pipe.run({"retriever": {"query": "Which city is on the Nile?"}})
for doc in result["retriever"]["documents"]:
    print(round(doc.score, 3), doc.content)

3.147 Cairo sits on the Nile and is the capital of Egypt.
1.882 Riyadh is the capital of Saudi Arabia.

Your scores will differ; BM25 scores are not normalised and depend on the corpus. What matters is the shape of the call. pipe.run() takes a dictionary keyed by component name, whose values are dictionaries of that component's inputs. The result comes back keyed the same way, then by output socket name. result["retriever"]["documents"] is a list of Document objects with .score filled in.

BM25 is a keyword-ranking algorithm: it scores documents by how often the query's words appear in them, discounted by how common those words are across the corpus. It finds nothing when the user's wording does not overlap the document's. That is the limitation embeddings fix, later in this guide.

Prompts, chat messages and your first RAG pipeline

Three components, two connections, and you have the application everyone wants to build.

from haystack import Pipeline
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

template = [ChatMessage.from_user(
    "Answer the question using only these documents. If they do not contain the "
    "answer, say you do not know.\n\n"
    "{% for doc in documents %}- {{ doc.content }}\n{% endfor %}\n"
    "Question: {{ question }}"
)]

rag = Pipeline()
rag.add_components({
    "retriever": InMemoryBM25Retriever(document_store=store, top_k=3),
    "prompt_builder": ChatPromptBuilder(template=template, required_variables="*"),
    "llm": OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY"),
        model="gpt-5-mini",
    ),
})
rag.connect_many([
    ("retriever", "prompt_builder.documents"),
    ("prompt_builder", "llm"),
])

question = "Which city is on the Nile?"
result = rag.run({
    "retriever": {"query": question},
    "prompt_builder": {"question": question},
})
print(result["llm"]["replies"][0].text)

Cairo sits on the Nile and is the capital of Egypt.

Several things in that listing deserve attention.

add_components({...}) and connect_many([...]) are new in Haystack 3.2. They are convenience methods: add_components is atomic, so if any one component fails a check nothing is added, and connect_many connects in order and stops at the first failure, keeping the connections it already made. add_component now returns the pipeline, so you can chain calls. On any version before 3.2 you write add_component and connect one at a time, which is exactly the same graph in more lines.

The connection strings use "sender.output" and "receiver.input" syntax. ("retriever", "prompt_builder.documents") leaves the sender's socket unnamed because the retriever has only one output, so there is no ambiguity. ("prompt_builder", "llm") names neither side for the same reason. When more than one pairing is possible Haystack refuses and tells you to be explicit, which is far better than guessing.

The template is a list[ChatMessage] containing Jinja. {% for doc in documents %} loops over the documents the retriever sent; {{ question }} is a variable you supply at run time. The two template variables map to two inputs of prompt_builder, which is why the question appears twice in the run() dictionary — once as the retriever's search query, once as the prompt's variable. They are genuinely two different inputs that happen to hold the same string, and seeing that clearly is a good sign you have understood the graph.

The generator is worth a paragraph of its own. OpenAIChatGenerator's default model is gpt-5-mini, so the explicit model= above is documentation rather than necessity. Its api_key takes a Secret, and Secret.from_env_var("OPENAI_API_KEY") is what you should almost always use: it stores only the variable name, so a serialised pipeline contains no credential. Its sibling Secret.from_token("sk-...") holds the literal value and cannot be serialised at all — attempting it raises Cannot serialize token-based secret. Use an alternative secret type like environment variables. Treat from_token as a thing for throwaway notebooks.

api_base_url is the other argument worth knowing on day one. Point it at any OpenAI-compatible server — vLLM, Ollama, LM Studio, LiteLLM — and the same component talks to a local or self-hosted model. For teams in the Gulf and Egypt working under data-residency rules that forbid sending customer text to a model hosted abroad, this single argument is often the difference between a pilot that ships and one that stalls in legal review. Everything else in your pipeline stays identical.

The reply is a ChatMessage, not a string. .text gives the text, .meta gives metadata such as finish reason and token counts, and .tool_calls gives tool calls when the model requests them. Haystack 2.x returned a separate result["meta"] list alongside the replies; that list is gone, and the metadata now lives on each message where it belongs.

Indexing real files

Three hand-written documents are not a corpus. A real indexing pipeline converts files into documents, cleans them, cuts them into retrievable chunks and writes them to the store.

from pathlib import Path
from haystack import Pipeline
from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.types import DuplicatePolicy

indexing = Pipeline()
indexing.add_components({
    "converter": TextFileToDocument(),
    "cleaner": DocumentCleaner(),
    "splitter": DocumentSplitter(split_by="word", split_length=200, split_overlap=40),
    "writer": DocumentWriter(document_store=store, policy=DuplicatePolicy.OVERWRITE),
})
indexing.connect_many([
    ("converter", "cleaner"),
    ("cleaner", "splitter"),
    ("splitter", "writer"),
])

files = list(Path("corpus").glob("*.txt"))
report = indexing.run({"converter": {"sources": files}})
print(report["writer"]["documents_written"])

184

Each stage earns its place. Converters turn files or byte streams into documents; besides TextFileToDocument there are PyPDFToDocument (needs pypdf), HTMLToDocument (needs trafilatura) and MarkdownToDocument. DocumentCleaner strips the debris that wrecks retrieval: repeated headers and footers, runs of whitespace, empty lines. Haystack 3.2 added min_content_length= so it can drop chunks too short to be useful.

DocumentSplitter is where the thinking happens. Retrieval works on chunks, and chunk size is the one knob with the largest effect on answer quality. split_by accepts word, sentence, line, page, passage, period, function, and — new in 3.2 — token, which needs pip install tiktoken and takes a tokenizer_encoding such as "o200k_base". Token splitting is the honest option when you care about fitting a model's context window, because that is the unit the model actually counts.

split_overlap repeats a tail of each chunk at the head of the next, so a sentence that straddles a boundary is not lost to both sides. Note the guard rail: split_overlap >= split_length raises ValueError at construction, not at run time, which is a small kindness. Richer splitters exist when the defaults are not enough — RecursiveDocumentSplitter, MarkdownHeaderSplitter, EmbeddingBasedDocumentSplitter — and DocumentPreprocessor bundles cleaning and splitting into one component.

DocumentWriter takes the store and a DuplicatePolicy. The default is NONE, which defers to whatever the store does. The others are OVERWRITE, SKIP and FAIL. Choose deliberately: re-running an indexing pipeline with OVERWRITE is idempotent and safe, while the default can leave you with two copies of everything and silently halved retrieval precision.

Embeddings and semantic retrieval

BM25 matches words. Ask "what is the capital of Egypt?" of a corpus that says "Cairo is Egypt's seat of government" and keyword search may well miss it. Embeddings fix this by mapping text to a vector such that similar meanings land near each other, and retrieval becomes a nearest-neighbour search.

This needs two components rather than one, and the distinction confuses everybody once. A document embedder takes documents and returns them with .embedding filled in; it runs at indexing time. A text embedder takes a query string and returns a bare embedding; it runs at query time. They must use the same model, or the two sets of vectors live in incompatible spaces and your retrieval is noise that looks plausible.

from haystack import Pipeline
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy

vector_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
MODEL = "text-embedding-ada-002"

index = Pipeline()
index.add_components({
    "embedder": OpenAIDocumentEmbedder(model=MODEL),
    "writer": DocumentWriter(document_store=vector_store, policy=DuplicatePolicy.OVERWRITE),
})
index.connect("embedder", "writer")
index.run({"embedder": {"documents": my_documents}})

query = Pipeline()
query.add_components({
    "text_embedder": OpenAITextEmbedder(model=MODEL),
    "retriever": InMemoryEmbeddingRetriever(document_store=vector_store, top_k=3),
})
query.connect("text_embedder.embedding", "retriever.query_embedding")

hits = query.run({"text_embedder": {"text": "What is Egypt's seat of government?"}})
for doc in hits["retriever"]["documents"]:
    print(round(doc.score, 3), doc.content[:60])

OpenAIDocumentEmbedder and OpenAITextEmbedder both default to the model text-embedding-ada-002. Set the model explicitly anyway, as above, in one constant used by both pipelines — that single MODEL variable is the cheapest possible insurance against the mismatch described a moment ago.

InMemoryDocumentStore has an embedding_similarity_function argument, "dot_product" by default with "cosine" as the alternative. Cosine ignores vector length and compares direction only, which is usually what you want for text. InMemoryEmbeddingRetriever takes top_k (10 by default), filters, scale_score and return_embedding, and its run input is query_embedding rather than query — that difference in socket name is precisely what the explicit connect("text_embedder.embedding", "retriever.query_embedding") is spelling out.

The column headings say "good" and "bad" because the styling needs two columns, not because one wins. Mature systems run both and combine the results. Building that is mid-level work; knowing that it is the normal destination keeps you from over-investing in either one now.

If you would rather not pay per embedding, the Sentence Transformers integration runs models locally: pip install sentence-transformers-haystack, then import from haystack_integrations.components.embedders.sentence_transformers. Note the import path carefully — these components lived in haystack in 2.x and moved out in 3.0. About thirty components made that move, and a stale import is the single most common 3.0 migration failure.

Reading Haystack's errors

Haystack's error messages are unusually informative, and learning to read four or five families of them turns most problems into two-minute fixes.

Connection errors. PipelineConnectError fires at connect() time, before anything runs. The messages are specific. Cannot connect 'a.out' with 'b.in': their declared input and output types do not match. means exactly what it says; fix the types or insert an OutputAdapter. more than one connection is possible between these components means you left both socket names off and the pipeline will not guess — name them. Component 'b' cannot accept multiple inputs to 'x'. It is already connected to component 'a', and it can only accept inputs from multiple senders if its type is list, Optional[list], or union of list types. is the rule about variadic sockets: only list-typed inputs can take several senders, and the pipeline then concatenates the lists. And 'a' does not have any output connections. Please check that the output types of 'a.run' are set, for example by using the '@component.output_types' decorator. is the missing-decorator mistake you provoked deliberately earlier.

Naming errors. ValueError: A component named 'x' already exists in this pipeline: choose another name. Names cannot contain a dot — it is the socket separator — and _debug is reserved. One more, easy to hit and hard to guess: Component instance cannot be added to the pipeline more than once. A single instance belongs to one pipeline under one name. If you want the same component twice, construct it twice.

Missing input errors. ValueError: Missing mandatory input '<var>' for component '<name>'. Either a template variable is required and absent, or a run parameter without a default was neither passed nor connected. Check both before changing required_variables.

Runtime errors. PipelineRuntimeError wraps whatever a component actually threw:

The following component failed to run:
Component name: 'llm'
Component type: 'OpenAIChatGenerator'
Error: ...

Read the inner error; the wrapper only tells you where. The exception also carries .pipeline_snapshot, so you can resume rather than restart — genuinely useful when the failure happened after an expensive embedding step. A related message, returned an invalid output ... Expected a dictionary, but got X instead, means one of your own components forgot to return a dict.

Authentication errors. ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',). A strict environment-variable Secret with nothing in the variable. Since 3.0 this appears at warm_up() or the first run rather than at construction, because clients are built at warm-up. Export the key, or use Secret.from_env_var(..., strict=False) for genuinely optional credentials.

Loop errors. PipelineMaxComponentRuns: Maximum run count 100 reached for component '<name>' means a loop never exited. Pipeline(max_runs_per_component=...) sets that limit, 100 by default, and lowering it while debugging gets you to the problem faster. Its cousin is PipelineComponentsBlockedError, whose message lists the suspects and says the cause is either no valid entry point or a circular dependency. In practice it is usually a required template variable that only a later component produces, so nothing can start.

Deserialisation errors. DeserializationError: Refusing to deserialize a class from module '<m>': the module is not on the trusted-module allowlist. That is the 3.0 security model, covered next. Its sibling, Refusing to deserialize unknown parameter '<key>' for '<Class>', is what stale YAML looks like now — a parameter that was removed from a class in a newer release will stop a load instead of being quietly ignored.

Configuration, secrets and saving a pipeline

A pipeline can be written out as YAML and read back. This is one of Haystack's distinguishing features and it is worth understanding properly, including the safety machinery around it.

yaml_text = rag.dumps()
with open("rag.yaml", "w") as fp:
    rag.dump(fp)

The resulting file has a predictable structure: a components mapping of names to {type, init_parameters}, a connections list of {sender, receiver} pairs, plus max_runs_per_component, metadata and connection_type_validation. Secrets serialise by reference, never by value:

components:
  llm:
    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
    init_parameters:
      model: gpt-5-mini
      api_key:
        type: env_var
        env_vars: [OPENAI_API_KEY]
        strict: true
connections:
  - sender: retriever.documents
    receiver: prompt_builder.documents

Only the variable name is stored. That file is safe to commit, which is the entire reason to prefer Secret.from_env_var over Secret.from_token.

Loading is where Haystack 3.0 changed behaviour deliberately:

from haystack import Pipeline

with open("rag.yaml") as fp:
    loaded = Pipeline.load(fp)

Loading a pipeline is gated by a trusted-module allowlist. The defaults are haystack, haystack_integrations, haystack_experimental, builtins, typing and collections. Dangerous builtins — eval, exec, compile, __import__, open, getattr — are blocked outright. And nested init_parameters keys are validated against the parent class's constructor signature, so a stale or misspelled key fails loudly.

The reason is straightforward once stated: a pipeline file names Python classes to import and arguments to pass them, so a YAML file is executable code. Before 3.0, loading an untrusted pipeline file was equivalent to running an untrusted script. If your own components live outside the allowlist, extend it explicitly with allowed_modules=["mypkg.*"], or set HAYSTACK_DESERIALIZATION_ALLOWLIST="mypkg.*". There is a global escape hatch, unsafe=True, and you should treat it as off-limits for any file you did not write yourself.

Default serialisation has one requirement that catches people: constructor parameter names must match instance attribute names, one to one. A component whose __init__ takes model_name but stores self.model cannot be serialised by the default machinery. Assign self.<param> = <param> for every argument and the problem disappears.

The environment variables worth knowing on day one:

On telemetry: Haystack sends anonymous component-usage events to PostHog on an EU host, identified by a random UUID, with no IP address, hostname, file paths, queries or content. Many employers in the region will still want it off, and one environment variable does it. The configuration file lives at ~/.haystack/config.yaml.

Logging deserves a note because 3.0 changed it for the better. Importing Haystack no longer reconfigures the root logger or global structlog, so the library can no longer silently reformat your application's logs. The default level is WARNING. To see more:

import logging
logging.getLogger("haystack").setLevel(logging.DEBUG)

If you actually wanted the old global behaviour, configure_logging(logger_name="") restores it, and configure_logging(propagate=False) is the fix for duplicated log lines.

Seeing what the pipeline did

Two tools: a picture of the graph, and a trace of the run.

pipe.show()                              # renders inline in Jupyter
pipe.draw(path=Path("pipeline.png"))     # writes a PNG

from pathlib import Path
rag.draw(path=Path("rag.png"))

draw and show take keyword-only arguments in 3.2. You will find documentation examples written as draw("my_pipeline.png", ...); that positional form fails against the current signature. Write draw(path=...) and move on.

Both methods render the graph by sending it to mermaid.ink, a public service on the internet. That is a privacy problem for a pipeline whose component names and configuration you would rather not publish, and it is also why draw() fails on a machine with no network access. Run the renderer yourself:

docker run --platform linux/amd64 --publish 3000:3000 --cap-add=SYS_ADMIN \
  ghcr.io/jihchi/mermaid.ink

rag.draw(path=Path("rag.png"), server_url="http://localhost:3000")

draw accepts mermaid-image (the default) or mermaid-text, and the text form needs no server at all — useful in CI, where you can diff the graph as text.

For understanding a run rather than a structure, start with include_outputs_from:

result = rag.run(
    {"retriever": {"query": q}, "prompt_builder": {"question": q}},
    include_outputs_from={"retriever", "prompt_builder"},
)
print(result["prompt_builder"]["prompt"][0].text)

That prints the exact prompt sent to the model, with the retrieved passages interpolated. Get into the habit of looking at it whenever an answer is wrong, because the most common cause of a bad answer is a prompt full of the wrong documents, and no amount of prompt engineering fixes a retrieval problem.

pipeline.inputs() is the other small tool worth knowing: it lists what the pipeline expects and which of those inputs are mandatory. When you cannot work out what key run() wants, ask.

For anything beyond this — span-level traces, token accounting, prompt history across sessions — you want a real observability tool, and Haystack integrates with several. Langfuse is the common choice in this series; note that it needs HAYSTACK_CONTENT_TRACING_ENABLED=true, set before importing Haystack, because recording inputs and outputs is off by default for the good reason that spans would otherwise contain prompts and personal data. Tracing is never enabled automatically in 3.x; if you remember HAYSTACK_AUTO_TRACE_ENABLED from 2.x, it was removed and now does nothing.

Putting it all together

One script, two pipelines, one store: index a folder of text files and answer questions about them with citations. This is the thing to keep and modify.

import logging
from pathlib import Path

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.converters import TextFileToDocument
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.components.writers import DocumentWriter
from haystack.dataclasses import ChatMessage
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack.utils import Secret

logging.getLogger("haystack").setLevel(logging.INFO)

store = InMemoryDocumentStore()

indexing = Pipeline()
indexing.add_components({
    "converter": TextFileToDocument(),
    "cleaner": DocumentCleaner(),
    "splitter": DocumentSplitter(split_by="word", split_length=180, split_overlap=30),
    "writer": DocumentWriter(document_store=store, policy=DuplicatePolicy.OVERWRITE),
})
indexing.connect_many([
    ("converter", "cleaner"),
    ("cleaner", "splitter"),
    ("splitter", "writer"),
])

template = [
    ChatMessage.from_system(
        "You answer strictly from the supplied documents. "
        "If they do not contain the answer, say so plainly."
    ),
    ChatMessage.from_user(
        "{% for doc in documents %}"
        "[{{ loop.index }}] {{ doc.meta.file_path }}\n{{ doc.content }}\n\n"
        "{% endfor %}"
        "Question: {{ question }}\n"
        "Cite the sources you used by their bracketed number."
    ),
]

answering = Pipeline()
answering.add_components({
    "retriever": InMemoryBM25Retriever(document_store=store, top_k=4),
    "prompt_builder": ChatPromptBuilder(template=template, required_variables="*"),
    "llm": OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY"),
        model="gpt-5-mini",
    ),
})
answering.connect_many([
    ("retriever", "prompt_builder.documents"),
    ("prompt_builder", "llm"),
])


def ask(question: str, show_prompt: bool = False) -> str:
    extras = {"prompt_builder"} if show_prompt else set()
    result = answering.run(
        {"retriever": {"query": question}, "prompt_builder": {"question": question}},
        include_outputs_from=extras,
    )
    if show_prompt:
        print("--- prompt ---")
        print(result["prompt_builder"]["prompt"][-1].text)
        print("--- /prompt ---")
    return result["llm"]["replies"][0].text


if __name__ == "__main__":
    files = sorted(Path("corpus").glob("*.txt"))
    if not files:
        raise SystemExit("Put some .txt files in ./corpus first.")

    report = indexing.run({"converter": {"sources": files}})
    print(f"indexed {report['writer']['documents_written']} chunks from {len(files)} files")

    answering.warm_up()
    answering.dump(open("answering.yaml", "w"))

    while True:
        try:
            question = input("\nquestion> ").strip()
        except (EOFError, KeyboardInterrupt):
            break
        if not question:
            break
        print(ask(question, show_prompt=question.startswith("?")))

    answering.close()

Walk through the deliberate choices, because each one is a habit worth keeping.

The two pipelines share store and nothing else. Indexing runs once at startup; in a real deployment it would be a separate job run when the corpus changes. DuplicatePolicy.OVERWRITE makes re-running it safe.

The template is two messages, not one. The system message carries the instruction that must not be negotiable — answer only from the documents — and the user message carries the data. Separating them is the right default: instructions that live in the same turn as retrieved text are easier for that text to override.

The prompt includes doc.meta.file_path and a loop.index number, and asks the model to cite the brackets. That metadata is there because TextFileToDocument put it there, and it travels through cleaning and splitting untouched. Citations are the cheapest trust mechanism a RAG system has, and they cost one line of Jinja.

answering.warm_up() is called explicitly before the loop rather than being left to the first run. In a server you do this at startup so the first user request is not the one that pays for client construction and model loading. answering.close() releases HTTP clients on the way out. Both methods exist on Pipeline as well as on components.

show_prompt is wired to a leading ? on the question. Build the debug path in from the start; you will use it on the first bad answer, and a flag you already have beats an edit under pressure.

And the pipeline is dumped to answering.yaml on every run, which gives you a reviewable artefact of exactly what shipped — a file a colleague can read without reading your Python.

What you can now do, and what comes next

You can install Haystack and confirm its version. You can write a component with declared input and output sockets and test it without a pipeline. You can build a pipeline, connect named sockets, read the connection errors when the types do not fit, and get at the intermediate outputs that run() hides by default. You can index a folder of files through a converter, cleaner, splitter and writer, with a duplicate policy you chose on purpose. You can retrieve by keyword and by embedding, filter by metadata, and explain to a colleague when each kind of retrieval wins. You can build a chat prompt from a template, call a model through a Secret that never lands in a file, and read the reply off a ChatMessage. You can serialise the whole graph to YAML and load it back, and you know why loading is gated by an allowlist. You can read the six or seven error families Haystack actually produces. And you know which of your 2.x instincts are now wrong.

What you have not touched is the other half of the library. Agents are a loop — call the model, run the tools it asked for, check whether to exit — and Agent is the component that implements it, with max_agent_steps as a safety limit and an exit_reason output explaining how it stopped. Tools are functions the model can call, created with the @tool decorator or by wrapping a component with ComponentTool. State is the agent's shared scratchpad. Hooks let you intervene at defined points in that loop, including the human-in-the-loop confirmation you want before a tool sends an email or writes to a database. Haystack's Agent is deliberately not a graph framework; if you have met LangGraph, the trade-off between an explicit state machine and a bounded tool loop is worth thinking about once you have used both.

The production side is also ahead of you. Pipeline.run_async and Pipeline.stream replace the AsyncPipeline class that 3.0 removed, and they are what you want inside a FastAPI endpoint. Hayhooks is a separate package that serves pipelines and agents as REST, OpenAI-compatible, MCP and A2A endpoints, which is how a Haystack pipeline becomes something a front end can talk to. MCP works in both directions: mcp-haystack lets your agent consume MCP tools, and hayhooks mcp run exposes your pipelines as MCP tools for someone else's agent. For judging whether a change made answers better rather than merely different, Haystack ships evaluators and integrates with RAGAS.

Before any of that, do one thing: take app.py, point it at a corpus you actually care about, and use it for a week. Every concept in the mid-level guide — hybrid retrieval, reranking, async serving, loops and routers, agents with tools — is an answer to a problem you will meet in that week. Meeting the problem first is what makes the answer stick.

Sources

• Haystack documentation: Intro
• Installation
• Components
• Custom components
• Pipelines
• Creating pipelines
• Smart pipeline connections
• Document store
• Metadata filtering
• Secret management
• Serialization
• Visualizing pipelines
• Debugging pipelines
• Logging
• Tracing
• Telemetry
• Breaking change policy
• Migration guide
• FAQ
• Docker
• Haystack 3.2.0 release notes
• Haystack 3.0.0 release notes
• MIGRATION.md
• haystack-ai on PyPI
• deepset/haystack on Docker Hub

The thirty-second answer

Haystack is an open-source Python framework from deepset for building LLM applications — retrieval-augmented question answering, search, chatbots and agents — out of small typed components wired into pipelines, where the connections between components are type-checked when you declare them rather than when you run them. The whole pipeline can be serialised to YAML and loaded back, so the structure of the application is configuration rather than code.

If the interviewer wants one more sentence, add the version boundary: the current stable release is Haystack 3.2.0, Haystack 3.0 removed a long list of 2.x APIs, and knowing which ones is the practical difference between code that runs and code that does not.

The model in four nouns

Say these four and you have answered most "explain Haystack's architecture" questions.

A component is a class decorated with @component whose run() method returns a dictionary. The parameters of run() are its inputs. The outputs are declared ahead of time with @component.output_types(...). Each named input or output is a socket, and a socket has a type.

A pipeline is a directed multigraph of components plus explicit connections between sockets. Data flows only along those connections. Types are validated at connect() time.

A Document is a dataclass with id, content, blob, meta, score, embedding and sparse_embedding. It is the unit the retrieval half of Haystack moves around.

A Document Store is a database interface and not a component — no run(), cannot be added to a pipeline. Its protocol is four methods: count_documents, filter_documents, write_documents, delete_documents. Components that need storage hold a reference to one.

Two supporting nouns usually come up in the same breath. A ChatMessage carries a role (system, user, assistant, tool) and content parts; .text gets the text. A Chat Generator such as OpenAIChatGenerator takes messages and returns {"replies": [ChatMessage]}.

If asked to draw the architecture, draw two lanes sharing one store: an indexing lane of converter, cleaner, splitter, embedder, writer; and a querying lane of retriever, prompt builder, chat generator. Say out loud that they are two pipelines, not one, because indexing is slow and occasional while querying is fast and per-request.

The APIs to have ready

Pipeline construction, including the 3.2 additions:

from haystack import Pipeline

pipe = Pipeline(max_runs_per_component=100)
pipe.add_component("retriever", retriever)          # returns the pipeline (3.2+)
pipe.add_components({"a": comp_a, "b": comp_b})     # atomic (3.2+)
pipe.connect("retriever.documents", "prompt_builder.documents")
pipe.connect_many([("a", "b"), ("b", "c")])         # (3.2+)
pipe.inputs()                                       # what is expected, and what is mandatory

Running, with the debugging argument that matters most:

result = pipe.run(
    {"retriever": {"query": q}, "prompt_builder": {"question": q}},
    include_outputs_from={"retriever", "prompt_builder"},
)

The generator and its secret:

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret

llm = OpenAIChatGenerator(
    api_key=Secret.from_env_var("OPENAI_API_KEY"),   # serialisable: stores the name only
    model="gpt-5-mini",                              # this is already the default
)

Serialisation:

yaml_text = pipe.dumps()
pipe.dump(open("pipe.yaml", "w"))
loaded = Pipeline.load(open("pipe.yaml"), allowed_modules=["mypkg.*"])

A custom component, which many interviewers ask you to write on a whiteboard:

from haystack import component

@component
class Shouter:
    @component.output_types(text=str)
    def run(self, text: str):
        return {"text": text.upper()}

And the install facts, because a surprising number of interviews start there: the PyPI package is haystack-ai, the import namespace is haystack, integrations import from haystack_integrations, and Python 3.10 or later is required. Verify with python -c "from haystack.version import __version__; print(__version__)".

The classic traps and why they happen

Trap: pasting 2.x code. from haystack import AsyncPipeline fails — one Pipeline class now has run, run_async, run_async_generator and stream. The non-chat generators OpenAIGenerator, AzureOpenAIGenerator, HuggingFaceAPIGenerator and HuggingFaceLocalGenerator were removed; use the Chat Generators, which accept a plain str for messages. ToolInvoker was removed and the Agent runs tools itself. DALLEImageGenerator is now OpenAIImageGenerator. Why it happens: 2.x was the stable release for nearly two years, so almost all published example code is 2.x.

Trap: a stale import for a component that moved out of core. About thirty components became separate integration packages in 3.0. The ones you will hit first are the Sentence Transformers embedders and rankers (pip install sentence-transformers-haystack, import from haystack_integrations.components.embedders.sentence_transformers), HuggingFaceLocalChatGenerator which became TransformersChatGenerator in transformers-haystack, and ExtractiveReader which became TransformersExtractiveReader. Why it happens: the import path changed but the class name often did not, so the error looks like a broken install rather than a move.

Trap: a missing template variable. ValueError: Missing mandatory input '<var>' for component '<name>'. In 3.x ChatPromptBuilder, PromptBuilder, Agent and LLM all default to required_variables="*". Why it happens: 2.x treated every variable as optional, so a typo rendered an empty string and the model answered from nothing at all. The fix is to supply the variable; required_variables=None restores the old behaviour and you should say you would rather not.

Trap: expecting the construction of a generator to fail without a key. It does not. Since 3.0, resources are created in warm_up() rather than __init__, so a missing key surfaces as ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',) at warm-up or on the first run.

Trap: adding the document store to the pipeline. It is not a component. Pass it to the retriever or writer's constructor.

Trap: reusing one component instance in two places. Component instance cannot be added to the pipeline more than once. An instance belongs to one pipeline under one name. Construct a second instance.

Trap: expecting every component's output back from run(). You get only the last output of components whose outputs are unconnected. Use include_outputs_from={...}.

Trap: connecting a non-list input to two senders. Only a list-typed input, list | None, or a union of list types can accept several senders, in which case the pipeline concatenates the lists — "implicit list joining". Anything else raises PipelineConnectError with a message that spells out the rule.

Trap: mismatched embedding models. A document embedder at index time and a text embedder at query time must use the same model, or the vectors live in different spaces and retrieval returns plausible-looking noise with no error anywhere.

Trap: deduplicating 3.x documents against data indexed under 2.x. The Document.id hash changed in 3.0: it now uses canonical key-sorted JSON of meta, with non-JSON values passed through str() rather than repr(). Any document with non-empty metadata therefore gets a different ID, and DuplicatePolicy.SKIP or FAIL silently stops working. Re-ingest, or pass explicit IDs.

Trap: assuming Haystack follows semantic versioning. The docs say outright that it does not. A minor release can remove functionality, and removing a deprecated component is described as the most common breaking change in a minor release. A FutureWarning in 3.1 means removal in 3.2.

Trap: calling draw("file.png"). In 3.2 draw and show take keyword-only arguments, so it is draw(path=Path("file.png")). Some documentation examples still show the positional form.

Common interview questions

"What is the difference between a component and a pipeline?"

A component is one unit of work with declared, typed inputs and outputs; it has a run() method returning a dictionary and works perfectly well on its own, which makes it unit-testable without any framework machinery. A pipeline is a graph of components plus the connections between their sockets. The pipeline adds three things the components do not have individually: type validation at wiring time, ordered execution that respects the data dependencies, and serialisation of the whole structure to YAML.

"Why isn't a Document Store a component?"

Because it has no run-time behaviour in the dataflow sense. It is a database interface with four methods — count, filter, write, delete — and no run(). Components that need persistence hold a reference to it: a retriever is constructed with document_store=, and the retriever is what goes in the pipeline. Keeping the store outside the graph is also what lets two pipelines, one for indexing and one for querying, share the same data.

"When would you use BM25 and when embeddings?"

BM25 scores documents by term overlap discounted by term frequency across the corpus. It is free, instant, needs no model, and is exact on names, identifiers and part numbers — the things embeddings are worst at. It fails completely when the user's wording does not overlap the document's. Embedding retrieval maps text to vectors so that similar meanings are near each other, which finds paraphrases and synonyms, but it costs an embedding call per document at index time and per query at run time, and it blurs exact codes. Production systems usually run both and combine the results. For a first prototype I would start with BM25 on InMemoryDocumentStore, because it needs no key and no spend, and add embeddings once I have seen which queries it misses.

"How do you keep an API key out of a serialised pipeline?"

Use Secret.from_env_var("OPENAI_API_KEY"). The YAML then records only the variable name, the type env_var, and whether it is strict — never the value. Secret.from_token("...") holds the literal value and cannot be serialised at all; attempting it raises Cannot serialize token-based secret. Use an alternative secret type like environment variables. In Kubernetes the environment variable is injected from a Secret with secretKeyRef, so the pipeline file is the same in every environment.

"Why does loading a pipeline from YAML need an allowlist?"

Because the YAML names Python classes to import and arguments to pass them, which makes it executable code. Haystack 3.0 gated loading behind a trusted-module allowlist whose defaults are haystack, haystack_integrations, haystack_experimental, builtins, typing and collections, blocked dangerous builtins such as eval, exec, compile, __import__, open and getattr, and started validating nested init_parameters keys against the parent class's constructor signature. The third change also catches ordinary typos and leftovers from removed parameters, which now raise DeserializationError instead of being ignored. For your own packages, extend the list with allowed_modules=["mypkg.*"] or the HAYSTACK_DESERIALIZATION_ALLOWLIST environment variable. There is an unsafe=True escape hatch and you should never point it at a file you did not write.

"A pipeline raises PipelineMaxComponentRuns. What happened?"

A component hit its run limit, which defaults to 100 and is set with Pipeline(max_runs_per_component=...). It means a loop never reached its exit condition — almost always a router condition that is never true, or a joiner that keeps re-feeding the loop. I would lower the limit while debugging to fail faster, add include_outputs_from on the router to see what it is actually deciding, and check the condition against the real values rather than the ones I assumed.

"What does include_outputs_from do, and why would you need it?"

By default a pipeline returns only the last output of each component whose outputs are not connected to anything, so in a three-stage RAG pipeline you get the generator's reply and nothing else. include_outputs_from={"retriever", "prompt_builder"} adds those components' outputs to the result. It is the first thing I reach for on a bad answer, because printing the rendered prompt almost always shows that retrieval returned the wrong passages and the model was doing its best with poor material.

"How would you test a Haystack pipeline without spending money?"

Three ways, in order of preference. Call each custom component's run() directly in a unit test, since components are ordinary objects. Use MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder, added in 3.0 precisely for this, to exercise the full graph with no API key. And build structural tests that only wire the pipeline and assert on inputs() — because connections are validated at connect() time, a wiring test catches type mismatches without running anything.

"What breaks when you upgrade from 2.x to 3.x?"

I would name the five that cost the most time: the removed AsyncPipeline and non-chat generators; the thirty-odd components that moved to integration packages; required template variables by default; the changed Document.id hash, which means re-ingesting anything with metadata; and the deserialisation allowlist plus nested-parameter validation, which makes old YAML fail loudly. The recommended path is to go to 2.31 first, clear every FutureWarning, then upgrade, then re-check tracing because span names changed and tracing is no longer auto-enabled.

"Walk me through what happens when you call pipeline.run()."

The pipeline already knows the graph, because every connection was validated when it was declared. At run time it takes your input dictionary, which is keyed by component name and then by input socket name, and works out which components have everything they need. A flat dictionary is also accepted, in which case each key is routed to every component that has an input of that name. Components run as their inputs become available, so the order follows the data dependencies rather than the order you added them. Before the first run, Haystack calls warm_up() on anything that has not been warmed, which is where models load and HTTP clients are built. Each component returns a dictionary whose keys must match its declared output sockets; a component that returns something other than a dictionary raises. Values travel along the declared connections and nowhere else — a component with nothing wired into it receives nothing, however convenient that would be. Finally the pipeline assembles the result, keeping only the last output of components whose outputs are unconnected, unless include_outputs_from says otherwise. If a component throws, the exception is wrapped in a PipelineRuntimeError that names the component and its type, and carries a pipeline_snapshot you can resume from.

"How do you filter retrieval by metadata?"

A comparison filter is {"field": "meta.topic", "operator": "==", "value": "geography"}, with the operators ==, !=, >, >=, <, <=, in and not in. A logic filter is {"operator": "AND"|"OR"|"NOT", "conditions": [...]}. Pass it to a retriever's filters argument or to store.filter_documents(filters=...). Support varies by store, and an unsupported operator raises FilterError — as does writing "gt" instead of ">", which is the mistake I have made most. Metadata filters are also the standard way to separate tenants in a shared store, applied server-side rather than trusted from the client.

Sixty-second self-test

Answer out loud. Any hesitation marks a section of the detailed guide to re-read.

1. Name the four nouns, and say which one is not a component.
2. What two decorators does a custom component need, and what does each declare?
3. When are pipeline connection types checked — at connect() or at run()?
4. What does run() return for a component whose output is connected to another component?
5. What is the PyPI package name, and what is the import name?
6. What is OpenAIChatGenerator's default model?
7. Which Secret constructor can be serialised, and what does the YAML contain?
8. What is the default value of required_variables on ChatPromptBuilder in 3.x?
9. Name three APIs that existed in 2.x and do not exist in 3.x.
10. Why must a document embedder and a text embedder use the same model?
11. What are the four DuplicatePolicy values, and which is the DocumentWriter default?
12. What does PipelineComponentsBlockedError tell you about your graph?
13. Where does a component acquire its HTTP client, and what does that change about missing-key errors?
14. What is max_runs_per_component's default, and what is it for?
15. Which external service does pipeline.draw() contact by default?

Model answers, in order: component, pipeline, Document, Document Store, and the store is not a component. @component on the class and @component.output_types(...) on run, declaring registration and the output sockets. At connect(). Nothing — connected outputs are excluded, which is what include_outputs_from exists to override. haystack-ai and haystack. gpt-5-mini. Secret.from_env_var, and the YAML holds the type env_var, the variable names and the strict flag. "*", meaning all variables are required. Any three of AsyncPipeline, OpenAIGenerator, ToolInvoker, DALLEImageGenerator, HuggingFaceLocalGenerator, TransformersSimilarityRanker. Because otherwise the document vectors and query vectors are in different spaces and retrieval silently returns noise. NONE, OVERWRITE, SKIP, FAIL, with NONE the default. That no component can start: either there is no valid entry point or there is a circular dependency. In warm_up(), so a missing key fails at warm-up or the first run rather than at construction. 100, as the safety net for loops. mermaid.ink, which is why draw() fails offline and why you self-host the renderer for a pipeline you would rather not publish.

The mistakes beginners actually make

Each one here is a real failure with its symptom, its cause and its fix. They are roughly in the order you will meet them.

Installing the wrong package

Symptom: pip install haystack succeeds, then from haystack import Pipeline raises ImportError, or imports something unrecognisable.

Cause: the PyPI package you want is haystack-ai. The import namespace is haystack, which makes the mismatch easy to guess wrong. farm-haystack is Haystack 1.x — a different architecture entirely — and it claims the same import namespace.

Fix: pip install haystack-ai, in a fresh virtual environment, and never alongside farm-haystack. If you suspect contamination, delete the environment rather than trying to repair it: a half-overwritten namespace package produces errors that waste more time than a reinstall.

Trusting example code from the internet

Symptom: ImportError: cannot import name 'AsyncPipeline' from 'haystack', or the same for OpenAIGenerator, ToolInvoker, DALLEImageGenerator or SentenceTransformersTextEmbedder.

Cause: the example was written for Haystack 2.x. Haystack 3.0 removed the separate AsyncPipeline class, all four non-chat generators, ToolInvoker, and about thirty components that became separate integration packages.

Fix: check the version in every example before you copy it, and prefer the official documentation to blog posts. For the specific names: use the Chat Generators instead of the legacy ones, OpenAIImageGenerator instead of DALLEImageGenerator, pipeline.run_async() instead of AsyncPipeline, and pip install sentence-transformers-haystack with imports from haystack_integrations.components.embedders.sentence_transformers. There is one documentation inconsistency to watch: a 3.0 upgrade note suggests importing SentenceTransformersSimilarityRanker from haystack.components.rankers, and that path does not exist in 3.x. Use the haystack_integrations path.

Adding the document store to the pipeline

Symptom: an error when you try to add a store with add_component, or confusion about why the store never appears in your run() dictionary.

Cause: a Document Store is not a component. It has no run(). It is a database interface with four methods.

Fix: pass it to the constructor of whichever component needs it — InMemoryBM25Retriever(document_store=store), DocumentWriter(document_store=store) — and add that component.

Expecting all the outputs back

Symptom: KeyError on result["retriever"], or an empty-looking result from a pipeline you know ran.

Cause: a pipeline returns only the last output of components whose outputs are not connected to anything. In a retriever-to-builder-to-generator pipeline, two of the three are connected, so you get one entry.

Fix: pass include_outputs_from={"retriever", "prompt_builder"} to run(). Make it a habit, not a last resort.

Forgetting the output_types decorator

Symptom: 'a' does not have any output connections. Please check that the output types of 'a.run' are set, for example by using the '@component.output_types' decorator.

Cause: your custom component has no declared output sockets, so there is nothing for the pipeline to connect from. The component still works standalone, which is why this surprises people.

Fix: add @component.output_types(name=type, ...) to run, with keys matching exactly what the returned dictionary contains.

Reusing one component instance twice

Symptom: Component instance cannot be added to the pipeline more than once.

Cause: an instance belongs to one pipeline under one name. This bites hardest when you build a generator once and try to use it in both an indexing and a querying pipeline.

Fix: construct a second instance. If construction is expensive, remember that the expense is in warm_up(), not __init__, so building two instances is cheaper than it looks.

Missing template variables

Symptom: ValueError: Missing mandatory input '<var>' for component '<name>'.

Cause: in 3.x every template variable is required by default (required_variables="*" on ChatPromptBuilder, PromptBuilder, Agent and LLM). Either you did not pass it, or you misspelled it so Haystack sees an extra required name.

Fix: pass the variable. Resist the urge to set required_variables=None — in 2.x that was the default and a typo'd variable rendered as an empty string, so the model answered confidently from nothing. The error is the feature.

Passing the question only once

Symptom: the retriever searches correctly but the prompt has no question in it, or the reverse.

Cause: in a RAG pipeline the question is genuinely two inputs — the retriever's query and the prompt builder's template variable. They happen to hold the same string.

Fix: pass both: {"retriever": {"query": q}, "prompt_builder": {"question": q}}. If you find that repetitive, know that a flat dictionary such as {"query": q} is routed to every component that has an input of that name, which is a tidy shortcut once your socket names line up deliberately.

Mismatched embedding models

Symptom: retrieval returns documents with reasonable-looking scores that have nothing to do with the question. No error anywhere.

Cause: the document embedder at indexing time and the text embedder at query time used different models, so the two sets of vectors are in incompatible spaces.

Fix: declare the model name in one constant and use it in both pipelines. This is the only failure in this list that produces no exception at all, which makes it the most expensive one to find.

Re-indexing and doubling the corpus

Symptom: count_documents() grows every time you run the indexing pipeline, and retrieval precision drops.

Cause: DocumentWriter's default policy is DuplicatePolicy.NONE, which defers to the store.

Fix: pass policy=DuplicatePolicy.OVERWRITE so re-running is idempotent. If you upgraded from 2.x and dedup suddenly stopped working, that is the separate 3.0 change: the Document.id hash now uses canonical key-sorted JSON of meta, so any document with metadata gets a different ID than before. Re-ingest, or pass explicit id= values.

A splitter that raises at construction

Symptom: ValueError from DocumentSplitter(...) before you have run anything.

Cause: split_overlap >= split_length. The check is at init time deliberately.

Fix: keep overlap well below length — a fifth is a reasonable starting point.

Filter operators written as words

Symptom: FilterError.

Cause: {"operator": "gt"} instead of {"operator": ">"}. The valid set is ==, !=, >, >=, <, <=, in, not in, with AND, OR and NOT for logic filters.

Fix: use the symbols. Note also that store support varies — Chroma has no NOT but adds contains — so an operator that works in memory can fail against a real store.

A missing key that fails later than expected

Symptom: the generator constructs fine, and then ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',) appears at warm-up or on the first run.

Cause: since 3.0, resources including API clients are created in warm_up(), not __init__.

Fix: export the key. For a genuinely optional credential use Secret.from_env_var(..., strict=False). In a server, call warm_up() explicitly at startup so this kind of failure happens at deploy time rather than on a user's request.

A secret that will not serialise

Symptom: Cannot serialize token-based secret. Use an alternative secret type like environment variables.

Cause: the component was built with Secret.from_token("sk-...").

Fix: Secret.from_env_var("OPENAI_API_KEY"). Treat from_token as a notebook convenience and nothing more.

An optional dependency you have not installed

Symptom: Haystack failed to import the optional dependency 'pypdf'. Run 'pip install pypdf'.

Cause: Haystack keeps heavy dependencies optional and imports them lazily.

Fix: do what the message says. The common ones are pypdf for PDFs, trafilatura for HTML, tiktoken for token splitting, and rich for the rich console UI.

draw() called positionally, or offline

Symptom: a TypeError from draw(), or a network error when you are not connected.

Cause: in 3.2 draw and show take keyword-only arguments, and both render by sending the graph to mermaid.ink.

Fix: write draw(path=Path("p.png")). For offline or sensitive work, run ghcr.io/jihchi/mermaid.ink locally and pass server_url="http://localhost:3000", or use mermaid-text output, which needs no server.

Missing traces after upgrading

Symptom: OpenTelemetry or Datadog is installed and configured, and no Haystack spans appear.

Cause: tracing is never auto-enabled in 3.x. auto_enable_tracing was removed and HAYSTACK_AUTO_TRACE_ENABLED has no effect. The tracers also moved to opentelemetry-haystack and datadog-haystack.

Fix: call tracing.enable_tracing(...) or add the matching connector component. For Langfuse, also set HAYSTACK_CONTENT_TRACING_ENABLED=true before importing Haystack, and flush the tracer before the process exits.

Habits worth adopting on day one

Pin the version and read the upgrade notes. Haystack does not follow semantic versioning, and the docs say so. A minor release can remove functionality; deprecating a component in one minor release and deleting it in the next is normal, which gives you about a month. Pin haystack-ai==3.2.*, pin each integration package separately since they are versioned independently, and treat a FutureWarning from this library as a ticket rather than noise.

Print the prompt before you blame the model. Most bad answers are retrieval failures wearing a disguise. include_outputs_from={"prompt_builder"} and one print tells you whether the model was given the right material. Build that switch into your script from the first version — a flag you already have beats an edit made under pressure.

Keep indexing and querying as separate pipelines sharing one store. They have different lifecycles: indexing is batchy, slow and idempotent; querying is per-request and latency-sensitive. Separating them is what lets you re-index without touching the serving path, and it makes both pipelines small enough to reason about.

Always use Secret.from_env_var. Then you can dump any pipeline to YAML at any time without thinking about what leaks. Make dumping part of your run: a pipe.dump(open("pipeline.yaml", "w")) on startup gives you a reviewable artefact of exactly what shipped, in a form a colleague can read without reading your Python.

Choose a duplicate policy on purpose. The default defers to the store, which means you have not decided. OVERWRITE for an idempotent re-index, SKIP when you know your IDs, FAIL when a duplicate indicates a bug upstream.

Name one embedding model constant. One MODEL = "..." used by the document embedder and the text embedder. It costs a line and removes the only silent failure in this guide.

Test components directly. They are ordinary Python objects with a run() method, so a unit test needs no pipeline, no store and no network. Then use MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder for graph-level tests. CI that costs nothing gets run; CI that costs tokens gets disabled.

Write a wiring test. Because connections are validated at connect() time, a test that only builds the pipeline catches every type mismatch without executing anything. It is the cheapest test in the repository and it fails on exactly the mistakes that are hardest to spot by reading.

Call warm_up() and close() explicitly in anything long-lived. Warm up at startup so the first request does not pay for client construction and model loading; close on shutdown so HTTP clients, connections and GPU memory are released. Pipeline has both methods, as do components.

Turn telemetry off if your employer would want it off. HAYSTACK_TELEMETRY_ENABLED=False. What is sent is anonymous component-usage data to PostHog on an EU host, with no IP, hostname, paths, queries or content — but for teams in the Gulf and Egypt working under data-residency rules, "we disabled it" is a shorter conversation than explaining what it collects. The config file is ~/.haystack/config.yaml.

Know that api_base_url exists. It points any OpenAI-compatible component at vLLM, Ollama, LM Studio or a gateway, which is often the whole answer to "the data cannot leave the country". Everything else in the pipeline stays the same.

Reach for the error message, not the search engine. Haystack's exceptions name the component, the socket, the types and usually the fix. PipelineConnectError tells you which pair does not match. PipelineRuntimeError wraps the real exception and tells you where. DeserializationError names the offending module or parameter. Reading them properly is faster than any search.

Decide your chunking by experiment, not by default. Chunk size is the single knob with the largest effect on answer quality, and no blog post's number is right for your corpus. Run the same five questions against split_length of 50, 180 and 500 and read the retrieved passages each time. Short chunks retrieve precisely and hand the model fragments with no surrounding context; long chunks carry context but dilute the keyword signal and spend tokens. Twenty minutes of this beats a week of prompt tuning, and Haystack makes the experiment cheap because only one constructor argument changes.

Put the non-negotiable instruction in a system message. ChatPromptBuilder takes a list of messages, so build two: a system message carrying the rule — answer only from the supplied documents, say so when they do not contain the answer — and a user message carrying the retrieved text and the question. Instructions that share a turn with retrieved content are easier for that content to override, and separating them costs nothing.

Carry metadata into the prompt and ask for citations. TextFileToDocument and the other converters put source information in meta, and it survives cleaning and splitting untouched. One line of Jinja that prints doc.meta.file_path with a bracketed number, plus one sentence asking the model to cite the brackets, gives you the cheapest trust mechanism a retrieval system has. A reviewer who can click through to the source will forgive an imperfect answer; one who cannot will not trust a correct one.

Test the empty case on purpose. Ask a question whose answer is genuinely not in the corpus and see what happens. A system that invents an answer when retrieval fails is worse than no system, because it is wrong at exactly the moments nobody is checking. This is a one-minute test and almost nobody runs it before a demo.

Checklist

Before you call a beginner Haystack project finished:

• The installed package is haystack-ai, Python is 3.10 or later, and from haystack.version import __version__ prints 3.2.0 or later.
• No import in the project comes from a 2.x example: no AsyncPipeline, no non-chat generators, no ToolInvoker, no core-namespace Sentence Transformers imports.
• Every API key goes through Secret.from_env_var, and the dumped YAML has been read to confirm no credential appears in it.
• Indexing and querying are separate pipelines sharing one store.
• DocumentWriter has an explicitly chosen DuplicatePolicy.
• The document embedder and text embedder reference the same model constant, if you use embeddings at all.
• One include_outputs_from debug path exists and has been used at least once to read the real prompt.
• The empty case has been tested: a question whose answer is genuinely absent produces an admission, not an invention.
• A wiring test builds the pipeline in CI, and model-dependent tests use the mock components.
• warm_up() is called at startup and close() at shutdown in anything that serves requests.
• haystack-ai and each integration package are pinned, and the 3.2 release notes have been read once.
• Telemetry is set deliberately, one way or the other, and the decision is written down.
• If draw() is used on anything sensitive, it points at a self-hosted mermaid renderer rather than mermaid.ink.

This is part two of three. It assumes you have built a working Haystack pipeline and now have to keep one running for other people. Everything here is about predicting behaviour rather than discovering it: how the engine schedules components, when a loop stops, what the Agent actually sends to the model, and which of your design choices will hurt on the day traffic arrives.

Everything below is verified against Haystack 3.2.0 (the haystack-ai package, released 2026-09-24), which requires Python 3.10 or later. Haystack 3.0 removed a long list of 2.x APIs, so most blog posts you will find are wrong in ways that fail loudly. Where that matters, this guide says so.

Where this picks up

Each section opens with the problem it solves and ends with a Try it you can run against a real pipeline in a few minutes.

How the pipeline engine actually runs

Beginner treated a pipeline as a chain. It is not one. A Pipeline is a directed multigraph of components, and the engine walks that graph by data availability, not by the order you called add_component. Understanding that one sentence removes most of the surprises at this level.

Three facts follow from it, and all three show up in real code.

Data travels only along explicit connections. Nothing is broadcast. If a component needs the user's question and you only connected the retriever's documents to it, the question never arrives and the run fails with a missing-input error. The one exception is the shape of the run data itself: pipeline.run() accepts either the nested form {"retriever": {"query": q}} or a flat form {"query": q}, and a flat key is routed to every component that declares an input with that name. Flat input is convenient in a notebook and a liability in production, because adding a component later silently changes which components receive the value. Use the nested form in anything you deploy.

Types are checked when you connect, not when you run. connect("a.out", "b.in") raises PipelineConnectError immediately if the declared output type cannot feed the declared input type. This is the single most valuable property of the framework for a team: a wiring mistake fails at import time, in CI, not at 2 a.m. in front of a customer. You can switch the check off with Pipeline(connection_type_validation=False), and you should not.

The pipeline returns less than you expect. For each component the result contains only its last output, and only for components whose outputs are not connected to anything downstream. A retriever in the middle of a RAG pipeline produces nothing in the result dict, because its documents were consumed. That is not a bug and it is not configurable per component — it is how you ask for intermediate values:

result = rag.run(
    {"retriever": {"query": q}, "prompt_builder": {"question": q}},
    include_outputs_from={"retriever", "prompt_builder"},
)
print(len(result["retriever"]["documents"]), "documents retrieved")
print(result["prompt_builder"]["prompt"][0].text[:400])

Haystack 3.2 added two conveniences for building graphs. add_components({...}) registers several components in one call and is atomic: if any name clashes or any instance is already owned by another pipeline, nothing is added at all. connect_many([...]) connects pairs in order but is not atomic — it stops at the first failure and keeps the earlier connections. Both add_component and add_components now return the pipeline, so construction can be chained:

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.dataclasses import ChatMessage
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.utils import Secret

store = InMemoryDocumentStore()
template = [ChatMessage.from_user(
    "Context:\n{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
    "Question: {{ question }}\nAnswer only from the context."
)]

rag = Pipeline(max_runs_per_component=20).add_components({
    "retriever": InMemoryBM25Retriever(document_store=store, top_k=5),
    "prompt_builder": ChatPromptBuilder(template=template, required_variables="*"),
    "llm": OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-5-mini"
    ),
})
rag.connect_many([("retriever", "prompt_builder.documents"), ("prompt_builder", "llm")])

Two naming rules bite people who generate pipelines programmatically: a component name cannot contain a dot, because the dot separates component from socket, and _debug is reserved. A component instance also belongs to exactly one pipeline under exactly one name — reusing the same object twice raises "Component instance cannot be added to the pipeline more than once." Build a fresh instance, or use Agent.clone() for agents.

Sockets, variadic inputs and smart connections

A socket is a named, typed input or output. Inputs are the parameters of run(); outputs are declared with the @component.output_types(...) decorator, or dynamically with set_output_types. That is the whole contract — but the rules for multiple senders into one input are where real pipelines go wrong.

An input can accept more than one sender only if its declared type is a list, an optional list, or a union of list types. When it is, the pipeline concatenates what arrives; the docs call this implicit list joining. So two retrievers can both feed one documents: list[Document] input and the downstream component sees one combined list, with no joiner component in between. If the input is not list-typed, you get a precise error instead:

Component 'ranker' cannot accept multiple inputs to 'documents'. It is already
connected to component 'bm25', and it can only accept inputs from multiple
senders if its type is list, Optional[list], or union of list types.

Multi-sender inputs are variadic, and variadic sockets come in two flavours that behave very differently in a loop. A lazy variadic socket accumulates values and waits; a greedy one consumes a single value per run and proceeds. When you build a cycle, this is the difference between a component that fires once per iteration and one that never fires at all because it is still waiting for a sender that will never produce anything.

The second mechanism worth learning precisely is smart connections, Haystack's type adaptation at connect time. The engine will bridge a small, fixed set of mismatches for you: a str becomes a ChatMessage with the user role, a ChatMessage becomes a str via .text, a T becomes a list[T], and a list[str] or list[ChatMessage] can be narrowed to its first item. An exact type match always wins over an adaptation, so adding an exactly-typed socket later can quietly change which conversion happens. For anything beyond that set, insert an OutputAdapter with an explicit Jinja expression so the transformation is visible in the graph and in the serialized YAML.

If you write custom components — and at this level you will — four rules keep them from becoming the broken part of the system. First, run() must return a dict; anything else raises at runtime with "returned an invalid output ... Expected a dictionary, but got X instead." Second, do not mutate your inputs. A pipeline may hand the same Document list to two branches, so copy it or use dataclasses.replace(). Third, if you add run_async, it must be a coroutine with exactly the same parameters and output types as run, or construction fails with a ComponentError saying so. Fourth, every init parameter must be stored on an attribute of the same name, because default serialization reads the attributes back out — break the 1:1 mapping and to_dict() produces YAML that cannot be loaded.

from haystack import component, Document


@component
class MetadataStamp:
    """Adds a tenant tag to every document without mutating the input list."""

    def __init__(self, tenant: str):
        self.tenant = tenant  # init name == attribute name, or serialization breaks

    @component.output_types(documents=list[Document])
    def run(self, documents: list[Document]) -> dict:
        stamped = [
            Document(content=d.content, meta={**d.meta, "tenant": self.tenant})
            for d in documents
        ]
        return {"documents": stamped}

When a sub-graph becomes a reusable unit, promote it to a SuperComponent rather than copying wiring between pipelines. Either wrap an existing pipeline with SuperComponent(pipeline, input_mapping=..., output_mapping=...), or write a class decorated with @super_component that sets self.pipeline. The built-in DocumentPreprocessor and MultiFileConverter are exactly this, and pipeline.draw(path=..., super_component_expansion=True) will expand them in the diagram when you need to see inside.

Branches, loops and the two limits that save you

Straight pipelines do not need routers. Anything that retries, validates, falls back, or picks a path per request does. Haystack gives you three pieces for that, and two limits that keep a mistake from becoming a bill.

ConditionalRouter takes a list of routes, each with a Jinja condition, an output, an output_name and an output_type. The first condition that evaluates true wins, and the value is emitted on that named socket, which you then connect onward. By default the router only emits primitives; to pass a ChatMessage or a list of Documents you need unsafe=True, and since 2.31 output_passthrough: True on a route passes typed objects through. Treat unsafe=True as what it says: the router evaluates Jinja, so never hand it a template that came from a user.

BranchJoiner is the component that makes cycles legal. It merges a first input with an input that loops back, so the component after it does not need a multi-sender socket. The canonical shape is a self-correcting loop: build a prompt, call the model, validate the output, and either emit the answer or send the failure back round with the error appended to the prompt.

Now the limits. Pipeline(max_runs_per_component=...) defaults to 100: when any single component has been executed that many times in one run, the engine raises PipelineMaxComponentRuns naming the component. That is your loop fuse, and 100 is far too generous for a loop that calls an LLM each time. Set it to the number of retries you actually intend — three or five — so a broken condition costs you three model calls instead of a hundred.

The second failure mode is the opposite one: nothing runs at all.

PipelineComponentsBlockedError: Cannot run pipeline - all components are blocked.
This typically happens when: 1. There is no valid entry point for the pipeline
2. There is a circular dependency preventing the pipeline from running

In practice this almost always means a cycle whose entry point is missing a value, and most often it is a required template variable that only the loop produces. Since 3.0, ChatPromptBuilder, PromptBuilder, Agent and LLM default to required_variables="*" — every variable in the template is mandatory. In 2.x they were all optional, so the same graph used to start and now deadlocks. The fix is one of three: give the loop a real entry point through a BranchJoiner, list only the variables that are genuinely required with required_variables=["question"], or pass required_variables=None to restore the old all-optional behaviour.

Async, streaming and what cancellation does not cancel

Haystack 3.0 deleted AsyncPipeline. There is one Pipeline class with four ways to execute it, and from haystack import AsyncPipeline now raises ImportError. If you inherited 2.x code, this is the migration you will hit first.

• run(data, include_outputs_from=None, ...) is synchronous and runs components sequentially. It has no concurrency_limit.
• run_async(data, include_outputs_from=None, concurrency_limit=4) is the async path and the only one that executes independent components in parallel.
• run_async_generator(...) yields partial outputs as components finish, which is how you show progress through a long indexing or multi-branch run.
• stream(data, *, streaming_components=None, include_outputs_from=None, concurrency_limit=4, cancel_on_abandon=True) returns a handle you iterate for StreamingChunks, with the full result available on handle.result afterwards.

The old AsyncPipeline.run(...) becomes asyncio.run(pipeline.run_async(...)). Both modes now share one tracing operation, haystack.pipeline.run, distinguished by the tag haystack.pipeline.execution_mode set to sync or async; the former haystack.async_pipeline.run span name is gone, so any dashboard or alert built on it is silently empty until you update it.

Streaming is the feature users notice, so get its shape right once:

import asyncio


async def answer(question: str) -> str:
    handle = rag.stream(data={"retriever": {"query": question},
                             "prompt_builder": {"question": question}})
    async for chunk in handle:
        print(chunk.content, end="", flush=True)
    return handle.result["llm"]["replies"][0].text


asyncio.run(answer("Which regions store customer data?"))

Two traps live in here. First, if you pass a synchronous streaming_callback into run_async, Haystack only logs a warning — it does not stop you — and your callback blocks the event loop for the whole generation. Pass an async callback in async code. Second, cancellation is partial. Cancelling a run_async task interrupts components that are natively async; synchronous components execute in worker threads, and their side effects complete regardless. A sync component that writes to a database will finish that write after the client has disconnected. If a step must be cancellable, it has to be genuinely async.

The lifecycle matters as much as the execution mode. Since 3.0, components build their expensive resources — model handles, HTTP clients — in warm_up() rather than __init__, and Pipeline exposes warm_up, warm_up_async, close and close_async too. Warm-up happens automatically on the first run, which means the first request after a deploy pays for it and a missing API key surfaces at warm-up instead of at construction. In a server, build the pipeline once at startup, call warm_up_async() explicitly during startup, and call close_async() on shutdown so HTTP clients, database connections and GPU memory are released.

The Agent loop under the hood

Agent is not magic and it is not a graph. It is a loop: call the chat generator, run any tools the model asked for, check the exit conditions, repeat. Haystack 3.0 removed the separate ToolInvoker component — the Agent runs tools itself now — and with it tool_invoker_kwargs. Knowing the loop's exact stopping rules is what lets you reason about cost.

Every init argument is keyword-only:

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-5-mini"),
    system_prompt="You answer from the knowledge base. Cite document ids.",
    tools=[search_docs, calculator],
    exit_conditions=["text"],
    max_agent_steps=8,
    tool_concurrency_limit=4,
    raise_on_tool_invocation_failure=False,
)

A run ends for exactly one of a few reasons, and the Agent tells you which in its exit_reason output (added in 3.1): "text" when the model replies without calling a tool, which is the default exit condition; a tool name, when you listed that tool in exit_conditions and it was called; "max_agent_steps"; "length" or "content_filter" (both new in 3.2) when the provider truncated or blocked the output; or a custom string that a hook wrote into the stop_run state key. Alongside it you get messages, last_message, step_count, token_usage, tool_call_counts, and one output per key in your state_schema.

max_agent_steps defaults to 100. That default exists so tutorials never hit it, and it is wrong for production: an agent that loops a hundred times calling a model is a bill, not an outage, which is why nobody pages you. Set it to the number of steps your task genuinely needs. Note also that exit_conditions are no longer validated against tool names at init — a typo in a tool name is now discovered at runtime, when the condition simply never fires.

Tool failures are, by default, not exceptions. With raise_on_tool_invocation_failure=False, an error is serialized and fed back to the model as a tool message so it can try something else. That is usually what you want for a research agent and exactly what you do not want for a payment tool, where a silent retry is worse than a crash. Choose per agent, and if you set it to True, be ready for ToolNotFoundException when the model invents a tool name. Duplicate tool names are detected at the start of every step.

Two other behaviours changed in ways that break 2.x code. user_prompt and system_prompt can no longer be passed to Agent.run (or to LLM.run); they are init-time only. To vary instructions per request, pass a ChatMessage.from_system(...) as part of messages. And messages is required — pass [] rather than omitting it. Runtime tools= on run() is supported, taking a list of tools, toolsets, or plain tool-name strings, which is the cleanest way to give different callers different capabilities from one shared agent.

Finally, a subtle 3.1 change with real consequences: the Agent warms up its hooks, tools and toolsets before every run, not once. Any warm_up() you write must be idempotent, or you will open a new client per request:

def warm_up(self) -> None:
    if self._client is not None:   # idempotent: 3.1+ warms up before every run
        return
    self._client = build_client()

Tools, State and controlling what the model sees

A Tool is a dataclass: name, description, a JSON-schema parameters, a function, optionally an async_function, plus three fields that control the plumbing — inputs_from_state, outputs_to_state and outputs_to_string. Those three are where mid-level work happens, because they decide what the model is allowed to know and what it is allowed to influence.

from typing import Annotated
from haystack.tools import tool


@tool(outputs_to_state={"ticket_id": {"source": "id"}})
def open_ticket(
    summary: Annotated[str, "One-line description of the problem"],
    severity: Annotated[str, "one of low, medium, high"],
) -> dict:
    """Open a support ticket and return its id."""
    return {"id": create_ticket(summary, severity)}

The docstring becomes the description the model reads, and the Annotated metadata becomes the parameter descriptions, so both are prompt engineering rather than documentation. Besides @tool, you can use create_tool_from_function, construct Tool(...) directly, or wrap existing Haystack objects: ComponentTool wraps a component, PipelineTool a whole pipeline, and AgentTool (3.1) another agent, returning only the sub-agent's final reply to the caller. MCPTool and MCPToolset from mcp-haystack consume external MCP servers — see the MCP guide for the protocol itself. Note that SSEServerInfo is deprecated along with SSE transport in the MCP spec; use StreamableHttpServerInfo.

State is the Agent's shared store for a run. Its schema is {"key": {"type": T, "handler": fn}}, and the default handler is merge_lists for list types (so each write appends) and replace_values otherwise. A custom handler is how you make a counter accumulate or a dict merge instead of being overwritten. The API is get, set, has and data, and one detail saves hours: State.get returns a deep copy. If you need the live object — a database client, a WebSocket — read state.data["key"] instead.

Haystack 3.0 removed implicit State injection by name. A tool now receives a State value only if it declares inputs_from_state={"tenant_id": "tenant"}, which both injects the value and hides that parameter from the schema the model sees. That is the mechanism for per-request authority: the tenant comes from your request context, never from the model, and the model cannot even name it.

Reserved state keys raise ValueError at construction if you redefine them: step_count, token_usage, tool_call_counts, continue_run, exit_reason, stop_run, tools, hook_context and context_tokens. If you want your own token figure, call it my_token_usage.

Hooks replaced several ad-hoc parameters. A hook is any object with run(state) — optionally run_async, warm_up, close — or a function decorated with @hook from haystack.hooks, registered at a named point. The points are before_run (once), before_llm (every step), before_tool (after calls are requested, before they execute), after_tool, on_exit (fires on an exit condition but not on max_agent_steps, and can set continue_run=True to keep going), and after_run (always once, where continue_run is ignored). An unknown hook point raises ValueError at construction.

Four ready-made hooks cover the problems you will actually hit:

ConfirmationHook lives in haystack.hooks.human_in_the_loop — it moved there in 3.0, and the old confirmation_strategies parameter on Agent is gone. It takes a mapping of tool name, tuple of names, or "*" to a BlockingConfirmationStrategy, which combines a policy (AlwaysAskPolicy, AskOncePolicy, NeverAskPolicy) with a UI (SimpleConsoleUI or RichConsoleUI). One caveat with security consequences: confirmation strategies see only the arguments the model supplied, not values injected from State, so an approval screen will not show the tenant you injected.

TokenBudgetHook(max_total_tokens=100_000) ends the run with exit_reason="token_budget_exceeded". CompactionHook takes a compactor — SlidingWindowCompactor, ToolResultPruningCompactor or the 3.2 SummarizationCompactor (give it a cheap model) — with context_window, compact_at=0.7 and compact_to=0.4. Both are marked experimental, so pin your version and read the upgrade notes.

Serialization: pipelines as YAML, safely

Every component has to_dict and from_dict, and Pipeline has dumps, dump, loads and load, defaulting to YAML through a pluggable Marshaller. A dumped pipeline is a flat description — components with a qualified type and init_parameters, plus a connections list, max_runs_per_component, metadata and connection_type_validation:

components:
  retriever:
    type: haystack.components.retrievers.in_memory.InMemoryBM25Retriever
    init_parameters:
      top_k: 5
  llm:
    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
    init_parameters:
      model: gpt-5-mini
      api_key:
        type: env_var
        env_vars: [OPENAI_API_KEY]
        strict: true
connections:
  - sender: retriever.documents
    receiver: prompt_builder.documents
max_runs_per_component: 20

Note what the secret became: a reference to an environment variable name, not a value. That only works if you used Secret.from_env_var(...). Secret.from_token("sk-...") is deliberately unserializable and dumps() refuses it with "Cannot serialize token-based secret. Use an alternative secret type like environment variables." Treat that error as the framework protecting you from committing a key.

Haystack 3.0 made loading a security boundary, because a crafted YAML file can name any importable class. Deserialization is now gated by a trusted-module allowlist whose defaults are haystack, haystack_integrations, haystack_experimental, builtins, typing and collections. Dangerous builtins — eval, exec, compile, __import__, open, getattr — are blocked outright, and nested init_parameters keys are validated against the parent class's __init__ signature. Three error messages follow from that, and all three are common during an upgrade:

DeserializationError: Refusing to deserialize a class from module 'mypkg': the
module is not on the trusted-module allowlist.

DeserializationError: ... resolves to the builtin 'eval', which is blocked ...

DeserializationError: Refusing to deserialize unknown parameter
'tool_invoker_kwargs' for 'Agent'. Correct the parameter name or remove it
from the serialized data.

The third one is the useful one: stale YAML holding a parameter that no longer exists — tool_invoker_kwargs, confirmation_strategies — now fails loudly instead of being ignored. For your own components, extend the allowlist deliberately with Pipeline.loads(yaml, allowed_modules=["mypkg.*"]), or allow_deserialization_module("mypkg") from haystack.core.serialization, or the environment variable HAYSTACK_DESERIALIZATION_ALLOWLIST="mypkg.*,other.*", which is read on every call. Patterns are prefix matches, with fnmatch globbing if the pattern contains *, ? or [ somewhere other than a trailing .*.

There is also a blunt escape hatch: unsafe=True, or HAYSTACK_UNSAFE_DESERIALIZATION=1 (3.1+), which disables every check process-wide and logs a warning. You will meet it legitimately in one narrow case — a serialized OutputAdapter or ConditionalRouter that carries unsafe: true or Jinja custom_filters requires Pipeline.load(fp, unsafe=True) since 3.1. Use it for files you produced and store in your own repository, never for a file a user uploaded or a tenant configured.

Indexing pipelines that are safe to re-run

Retrieval quality is mostly an indexing problem, and indexing in production is mostly an idempotency problem. A Document carries id, content, blob, meta, score, embedding and sparse_embedding, and the id is a hash of content and meta unless you set it. Haystack 3.0 changed that hash: it now uses canonical key-sorted JSON of meta, and non-JSON values go through str() rather than repr().

Passing explicit IDs is the better habit regardless of version. A stable id like f"{source_system}:{record_id}:{revision}" makes re-ingestion idempotent, makes deletion possible, and decouples you from any future hashing change.

DuplicatePolicy lives in haystack.document_stores.types and has four values: NONE (the DocumentWriter default, which defers to whatever the store does), OVERWRITE, SKIP and FAIL. For an incremental re-index, OVERWRITE with stable IDs is almost always right.

A document store is not a component: it has no run(). It is a database interface with four protocol methods — count_documents, filter_documents, write_documents, delete_documents — that components hold a reference to. InMemoryDocumentStore is for prototyping and tests, and one default surprises people: it is process-global per index, because shared=True by default. Pass shared=False when you want a request-local store, or memory grows with traffic. For anything real, use a store integration — Qdrant, pgvector, Weaviate, Elasticsearch, OpenSearch, Pinecone, Chroma, Milvus and many others ship as *-haystack packages. If you are choosing, the Qdrant, pgvector and Weaviate guides cover the trade-offs, and FAISS covers the library underneath several of them.

Splitting is the other lever. DocumentSplitter(split_by=..., split_length=200, split_overlap=0, split_threshold=0) accepts function, page, passage, period, word, line, sentence, and — new in 3.2 — token with a tokenizer_encoding such as "o200k_base", which needs pip install tiktoken. Token splitting is worth the extra dependency because it bounds chunks in the same unit your model bills in. An split_overlap >= split_length raises ValueError at init, which is a kindness. Also new in 3.2: DocumentCleaner(min_content_length=...) to drop fragments that are too short to retrieve usefully.

from haystack import Pipeline
from haystack.components.converters import PyPDFToDocument
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.types import DuplicatePolicy

indexing = Pipeline().add_components({
    "converter": PyPDFToDocument(),
    "cleaner": DocumentCleaner(min_content_length=40),
    "splitter": DocumentSplitter(
        split_by="token", tokenizer_encoding="o200k_base",
        split_length=400, split_overlap=40,
    ),
    "writer": DocumentWriter(document_store=store, policy=DuplicatePolicy.OVERWRITE),
})
indexing.connect_many([("converter", "cleaner"), ("cleaner", "splitter"),
                       ("splitter", "writer")])

Metadata filters are how one index serves many tenants or many document types. A comparison filter is {"field": "meta.tenant", "operator": "==", "value": "acme"} with the operators ==, !=, >, >=, <, <=, in and not in; a logic filter is {"operator": "AND", "conditions": [...]} with AND, OR and NOT. An invalid operator — "gt" instead of ">" — raises FilterError, and so does an operator the store does not support; Chroma, for example, has no NOT but adds contains. Retrievers take filters, top_k and filter_policy, which is FilterPolicy.REPLACE by default or MERGE. For multi-tenant retrieval, MERGE plus a server-side tenant filter means a caller-supplied filter narrows the result rather than replacing your isolation.

Testing a pipeline without an API key

The thing that stops teams testing LLM pipelines is the belief that every test needs a model. Haystack 3.0 removed that excuse by shipping MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder. Combined with the type checking at connect() time, you can cover most of what breaks for free and in milliseconds.

Build your tests in four layers, cheapest first.

Wiring tests assert that the graph is the graph you meant. Construct the pipeline, assert on pipeline.inputs(), and round-trip dumps()/loads(). No model, no network. These catch the whole class of failure where someone adds a component and forgets a connection.

Component tests exercise a custom component's run() directly as an ordinary Python function. Assert the returned dict's keys match the declared output types, and assert the inputs were not mutated.

Pipeline tests with mocks substitute MockChatGenerator for the real generator and a BM25-only retrieval path over InMemoryDocumentStore, which needs no embeddings at all. You are testing your prompt assembly, your routing and your parsing, which is where your bugs are. Keep these in CI on every pull request.

A small live suite runs against the real provider on a schedule, not on every commit, and asserts on properties rather than exact strings: that an answer cites at least one document id, that exit_reason is "text", that token_usage is under a threshold. Those three assertions catch prompt regressions without being flaky.

from haystack.components.generators.chat import MockChatGenerator


def build_rag(generator):
    pipe = Pipeline().add_components({
        "retriever": InMemoryBM25Retriever(document_store=store),
        "prompt_builder": ChatPromptBuilder(template=template,
                                            required_variables=["question"]),
        "llm": generator,
    })
    pipe.connect_many([("retriever", "prompt_builder.documents"),
                       ("prompt_builder", "llm")])
    return pipe


def test_prompt_includes_retrieved_context():
    store.write_documents([Document(content="Backups run nightly at 02:00 UTC.")])
    pipe = build_rag(MockChatGenerator())
    out = pipe.run({"retriever": {"query": "backups"},
                    "prompt_builder": {"question": "When do backups run?"}},
                   include_outputs_from={"prompt_builder"})
    assert "02:00 UTC" in out["prompt_builder"]["prompt"][0].text

Note the pattern: the generator is injected, so the same builder function serves tests and production. For quality rather than correctness, Haystack ships evaluators — FaithfulnessEvaluator, ContextRelevanceEvaluator, LLMEvaluator, and the retrieval metrics DocumentRecallEvaluator, DocumentMRREvaluator, DocumentMAPEvaluator and DocumentNDCGEvaluator — plus Ragas and DeepEval integrations. DocumentMAPEvaluator scores changed in 3.1, so a baseline is not comparable across that boundary. The RAGAS guide goes deeper.

Debugging beyond print

When a run misbehaves, you need three things in order: the data that moved between components, the point at which it went wrong, and a way to resume from there without replaying the expensive parts. Haystack has a tool for each.

For data, include_outputs_from is still the first move, and it costs nothing. For failures, read the error envelope carefully, because PipelineRuntimeError wraps the real exception:

PipelineRuntimeError: The following component failed to run:
Component name: 'llm'
Component type: 'OpenAIChatGenerator'
Error: ...

The inner error is the one to act on. Crucially, the exception carries e.pipeline_snapshot — the full state of the run at the moment of failure — and you can resume from it rather than starting over.

Logging is scoped. Importing Haystack no longer reconfigures the root logger or global structlog, which is why your app's log format survives. The default level is WARNING, so turn it up when you need detail: logging.getLogger("haystack").setLevel(logging.DEBUG). If you see duplicated lines, call configure_logging(propagate=False); if you want Haystack's formatting applied to every record, configure_logging(logger_name="") restores the old global behaviour, and configure_logging(configure_structlog=True) wires up structlog.

For a line-by-line view of a run, LoggingTracer prints each component's inputs and outputs. Content tracing is off by default — spans would contain prompts and personal data — so you enable it explicitly:

import logging
from haystack import tracing
from haystack.tracing.logging_tracer import LoggingTracer

logging.getLogger("haystack").setLevel(logging.DEBUG)
tracing.tracer.is_content_tracing_enabled = True
tracing.enable_tracing(LoggingTracer())

Breakpoints are the heavier instrument. A Breakpoint(component_name=..., visit_count=..., snapshot_file_path=...) pauses the run before that component on that visit and raises BreakpointException, carrying e.component, e.inputs and e.results. You then resume with pipeline.run(data={}, pipeline_snapshot=load_pipeline_snapshot(path)):

from haystack.core.errors import BreakpointException
from haystack.core.pipeline.breakpoint import load_pipeline_snapshot
from haystack.dataclasses.breakpoints import Breakpoint

bp = Breakpoint(component_name="llm", visit_count=0, snapshot_file_path="/snaps")
try:
    pipeline.run(data, break_point=bp)
except BreakpointException as exc:
    print(exc.component, exc.inputs)

pipeline.run(data={}, pipeline_snapshot=load_pipeline_snapshot("/snaps/llm_0_2026_10_03_11_02_44.json"))

Snapshot files are only written when HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED=true. The directory is chosen in order: ~/.haystack/pipeline_snapshot, then /tmp/haystack/pipeline_snapshot, then ./.haystack/pipeline_snapshot. Names follow {component}_{visit}_{YYYY_MM_DD_HH_MM_SS}.json, and snapshot_callback=fn replaces file saving if you would rather ship snapshots somewhere else. Since 3.0 you can combine break_point and pipeline_snapshot for stepped debugging: resume, then break again at the next component.

One honest limitation: Agent breakpoints were removed in 3.0. AgentBreakpoint, ToolBreakpoint, AgentSnapshot and the break_point, snapshot and snapshot_callback parameters on Agent.run are gone, as is PipelineSnapshot.agent_snapshot. Pipeline-level breakpoints still work, so to inspect an agent step by step you instrument it with hooks instead — a before_llm hook that logs the message list is the practical replacement.

Observability, cost and the numbers that matter

Tracing in 3.x is never automatic. auto_enable_tracing was removed and HAYSTACK_AUTO_TRACE_ENABLED has no effect, so a 2.x deployment that upgraded will have an installed OpenTelemetry SDK, no errors, and no traces. Enable it explicitly with tracing.enable_tracing(...) or by adding a connector component to the pipeline. The OpenTelemetry and Datadog tracers moved out of core into opentelemetry-haystack and datadog-haystack.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from haystack import tracing
from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))

Learn the span names, because every dashboard depends on them: haystack.pipeline.run with the haystack.pipeline.execution_mode tag; per-component spans carrying haystack.component.name, .type, .input and .output; haystack.agent.run; haystack.agent.step with children haystack.agent.step.llm and one haystack.agent.step.tool per tool call; haystack.agent.hook (3.1); and haystack.chat_generator.run inside FallbackChatGenerator (3.2). The agent spans deliberately carry no haystack.component.* tags, so a custom SpanHandler that matched on component_type == "ToolInvoker" is now dead code.

Langfuse, MLflow, Datadog, Arize/Phoenix, Weave and Rhesis all work; Langfuse through langfuse-haystack is the most common choice for LLM-specific views. Two requirements trip everyone up: HAYSTACK_CONTENT_TRACING_ENABLED=true must be set before importing Haystack, and you must call tracer.flush() before the process exits or short-lived jobs lose their traces. See Langfuse and OpenTelemetry for the backends themselves.

For logs, use JSON in production: HAYSTACK_LOGGING_USE_JSON=true, or configure_logging(use_json=True). JSON is automatic when there is no TTY, so it is probably already on in your containers.

Cost is dominated by tokens, and the controls are all things you have now met. The Agent's token_usage output and tool_call_counts give you per-request numbers worth recording as metrics; max_agent_steps and TokenBudgetHook cap the worst case; CompactionHook and ToolResultOffloadHook stop context growth from multiplying the cost of every subsequent step; generation_kwargs sets per-run output limits. FallbackChatGenerator fails over across providers or models, which is availability as much as cost. On the embedding side, batch document embedders at indexing time, and consider local Sentence Transformers through sentence-transformers-haystack instead of a paid API. And keep CI free with the mock components.

Two privacy details are worth raising in a Gulf or Egyptian data-residency review. Haystack sends anonymous telemetry — component-usage events with a random UUID, no IP, hostname, paths, queries or content — to PostHog on an EU host; opt out with HAYSTACK_TELEMETRY_ENABLED=False. And pipeline.show() and draw() send your graph to the public mermaid.ink service unless you point server_url at a local renderer.

Serving: Hayhooks, Docker and Kubernetes

Haystack is a library, so serving is your decision. There are three defensible answers: your own FastAPI app, Hayhooks, or deepset's commercial Haystack Enterprise Platform.

In your own app, the rules are the ones from the async section: build pipelines once at startup, await warm_up_async() there, use await pipeline.run_async(...) in handlers, pipeline.stream(...) for server-sent events, and await close_async() on shutdown. The FastAPI guide covers the lifespan plumbing.

Hayhooks is a separate package and FastAPI server that serves pipelines and agents as REST, OpenAI-compatible, MCP and A2A endpoints. The current version is 1.25.0, supporting Python 3.10 up to but not including 3.15. A deployed pipeline is a directory containing pipeline_wrapper.py with a PipelineWrapper(BasePipelineWrapper) class: setup() builds or loads the pipeline, run_api(...) defines the /{name}/run endpoint with its type hints becoming the request schema, and run_chat_completion(self, model, messages, body) provides the OpenAI-compatible endpoint. Add files: list[UploadFile] | None = None to accept multipart uploads, and there are async variants of each method. create_app() embeds the whole thing inside your own FastAPI application.

pip install hayhooks
hayhooks run --host 0.0.0.0 --port 1416 --pipelines-dir ./pipelines
hayhooks status                     # -> Hayhooks server is up and running.
hayhooks pipeline deploy-files -n support-rag ./pipelines/support_rag
hayhooks pipeline run support-rag --param 'question="When do backups run?"'
hayhooks mcp run                    # Streamable HTTP MCP server on port 1417
hayhooks a2a run                    # A2A server on port 1418

Configuration comes from a .env file, environment variables or CLI flags. The ones that matter in production: HAYHOOKS_HOST and HAYHOOKS_PORT (1416); HAYHOOKS_PIPELINES_DIR (./pipelines, auto-deployed at startup); HAYHOOKS_SHOW_TRACEBACKS, which defaults to false and must stay false; HAYHOOKS_CORS_ALLOW_ORIGINS, which defaults to ["*"] and must not stay that way; HAYHOOKS_STREAMING_COMPONENTS (empty means only the last component streams); HAYHOOKS_LOG_LEVEL; and the standard OTEL_* variables.

For containers, start from the official image and bake your dependencies in. The only flavour is base, which is equivalent to pip install haystack-ai:

FROM deepset/hayhooks:v1.25.0
RUN pip install --no-cache-dir qdrant-haystack sentence-transformers-haystack \
    opentelemetry-haystack tiktoken
COPY pipelines /app/pipelines
ENV HAYHOOKS_PIPELINES_DIR=/app/pipelines \
    HAYHOOKS_HOST=0.0.0.0 \
    HAYSTACK_LOGGING_USE_JSON=true \
    HAYSTACK_TELEMETRY_ENABLED=False
EXPOSE 1416

Installing packages in the container command is a demo pattern from the docs, not a deployment pattern: it makes startup slow, non-reproducible and dependent on PyPI being reachable. Pin the image tag and the integration versions.

On Kubernetes, pipelines are described as "usually stateless", which makes a Deployment of N replicas behind a Service the natural shape. The two things to get right are secrets and readiness. Inject keys with secretKeyRef so the container only ever sees an environment variable that Secret.from_env_var can resolve — the official example uses a Secret named openai-secret with key api-key:

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: openai-secret
        key: api-key

And remember that warm-up happens on first use. If your readiness probe passes before the pipeline is warm, the first requests to every new pod are slow; warm up explicitly during startup so readiness means ready. If you also run an in-process InMemoryDocumentStore, note that it is per-replica and not shared — scaling to two pods gives you two different corpora, which is a correctness bug, not a performance one. See the Kubernetes and Docker guides for the surrounding mechanics.

Upgrades and version discipline

This section exists because of one sentence in the official docs: "Haystack DOES NOT follow the principles of Semantic Versioning." If you treat 3.2 to 3.3 as a safe patch, you will be wrong, and you will be wrong in production.

The policy in practice:

• Major releases bring fundamental incompatible changes, happen at most once a year, and come with a migration guide.
• Minor releases may add or remove functionality. The docs are explicit: "Deprecated components are the most common breaking change shipped in a Minor version release."
• Patch releases are bug fixes only.
• A feature deprecated in X.Y raises FutureWarning and is removed in X.Y+1 — roughly one month of notice.

Three habits follow. Pin exactly: haystack-ai==3.2.*, and pin every integration separately, because *-haystack packages are versioned independently. Treat FutureWarning as an error in CI, since a warning today is a removal next month. And read the "Upgrade Notes" section of every minor release before you take it, not after.

Integrations have their own lifecycle: a discontinued one moves to a staging branch with a three-month grace period, after which it is adopted or discontinued. The PyPI package is never deleted, so a pinned dependency keeps working — it just stops receiving fixes.

For a 2.x codebase, the migration path is: go to 2.31.0 first, fix every deprecation warning there (2.31 deprecated everything that 3.0 removed), then upgrade to 3.x. The work that remains after the imports compile is the part people underestimate: re-ingest documents that have meta, because the ID hash changed; audit every YAML file against the allowlist and the nested-parameter validation; audit prompt templates, because required_variables="*" is now the default; and re-check tracing setup and dashboards, because tracing is no longer auto-enabled and the span names changed. Legacy non-chat generators (OpenAIGenerator, HuggingFaceAPIGenerator and friends) are gone — use the Chat Generators, which now also accept a plain str for messages, and read replies[0].text and replies[0].meta instead of a separate result["meta"].

Roughly thirty components moved into separate integration packages in 3.0. The Sentence Transformers embedders and rankers are in sentence-transformers-haystack; HuggingFaceAPIChatGenerator and the HF API embedders are in huggingface-api-haystack; HuggingFaceLocalChatGenerator became TransformersChatGenerator and ExtractiveReader became TransformersExtractiveReader, both in transformers-haystack. One docs inconsistency to ignore: an upgrade note suggests importing SentenceTransformersSimilarityRanker from haystack.components.rankers, and that path does not exist in 3.x. Also note that haystack-experimental is no longer installed automatically, and DALLEImageGenerator is now OpenAIImageGenerator.

Looking forward, 3.3.0-rc1 is published but is a pre-release — do not ship it. When it lands it changes in-memory BM25 so that BM25L (the default) or BM25Plus returns only documents containing at least one query term, meaning results can be fewer than top_k, and it adjusts sentence-split boundaries around quoted sentences. Both change retrieval output, so plan to re-baseline your evaluations and re-index when you take it.

Putting it all together

Here is the shape of a production Haystack service with the decisions from every section above made explicitly. It is a support-answering RAG agent: a real document store, a retrieval tool, per-tenant isolation that the model cannot influence, bounded steps and tokens, async execution, and a startup that warms up and a shutdown that closes.

import os
from contextlib import asynccontextmanager
from typing import Annotated

from fastapi import FastAPI
from haystack import Pipeline
from haystack.components.agents import Agent
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.retrievers import InMemoryBM25Retriever
from haystack.dataclasses import ChatMessage
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.hooks.budget import TokenBudgetHook
from haystack.tools import tool
from haystack.utils import Secret

store = InMemoryDocumentStore(shared=True)
RETRIEVAL = Pipeline().add_components({
    "retriever": InMemoryBM25Retriever(document_store=store, top_k=5),
    "prompt_builder": ChatPromptBuilder(
        template=[ChatMessage.from_user(
            "{% for doc in documents %}[{{ doc.id }}] {{ doc.content }}\n{% endfor %}"
        )],
        required_variables=None,
    ),
})
RETRIEVAL.connect("retriever", "prompt_builder.documents")


@tool(inputs_from_state={"tenant": "tenant"})
def search_kb(
    query: Annotated[str, "What to look for in the knowledge base"],
    tenant: str = "",
) -> dict:
    """Search the support knowledge base and return passages with their ids."""
    out = RETRIEVAL.run({
        "retriever": {
            "query": query,
            "filters": {"field": "meta.tenant", "operator": "==", "value": tenant},
        }
    })
    return {"passages": out["prompt_builder"]["prompt"][0].text}


agent = Agent(
    chat_generator=OpenAIChatGenerator(
        api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-5-mini"
    ),
    system_prompt=(
        "Answer support questions using search_kb only. "
        "Cite the bracketed document ids you used. If nothing matches, say so."
    ),
    tools=[search_kb],
    state_schema={"tenant": {"type": str}},
    exit_conditions=["text"],
    max_agent_steps=6,
    tool_concurrency_limit=4,
    hooks={"before_llm": [TokenBudgetHook(max_total_tokens=40_000)]},
)


@asynccontextmanager
async def lifespan(app: FastAPI):
    await RETRIEVAL.warm_up_async()
    await agent.warm_up_async()
    yield
    await RETRIEVAL.close_async()


app = FastAPI(lifespan=lifespan)


@app.post("/ask")
async def ask(question: str, tenant: str) -> dict:
    result = await agent.run_async(
        messages=[ChatMessage.from_user(question)], tenant=tenant
    )
    return {
        "answer": result["last_message"].text,
        "exit_reason": result["exit_reason"],
        "steps": result["step_count"],
        "tokens": result["token_usage"],
    }

Read it as a set of decisions rather than code. The tenant arrives from the request and enters State, and because search_kb declares inputs_from_state, the parameter is hidden from the model's view of the tool — the model cannot ask to search another tenant's data. The retrieval pipeline is a reusable unit called from a tool, which keeps retrieval testable on its own. max_agent_steps=6 and a 40k token budget bound the worst case, and the response returns exit_reason, step_count and token_usage so cost per request is visible without extra instrumentation. The one thing you must change before real traffic is InMemoryDocumentStore: it is per-process, so swap it for a store integration the moment you run more than one replica.

Where you are now

You can predict what a Haystack pipeline will do. You know that the engine schedules by data availability, that types are checked at connect time, that outputs you do not ask for are discarded, and that multiple senders need a list-typed socket. You know which run method parallelises and which does not, that cancellation does not reach synchronous components, and that warm-up is where resources appear. You know the Agent's exit reasons by name, how to bound its steps and its tokens, and how to inject per-request authority that the model cannot see or change. You know that YAML is code, that the allowlist exists for a reason, and that unsafe=True belongs only on files you wrote. You can test most of a pipeline with no API key, debug one with snapshots rather than reruns, and trace one into Jaeger or Langfuse. And you know that a minor release can remove things, so you pin exactly and treat FutureWarning as an error.

What is left is the work of owning Haystack for other teams: architecture and failure modes under load, which component breaks first, the full trust model for multi-tenant deployments, upgrade and migration programmes, governance, and the honest question of when a pipeline framework is the wrong tool. That is part three. Natural neighbours from here are LangGraph for a graph-first alternative, LlamaIndex for a retrieval-first one, Langfuse for LLM observability, and RAGAS for turning "it seems better" into a number.

Sources

• Haystack documentation: Intro
• Pipelines and Creating Pipelines
• Components and Custom Components
• SuperComponents
• Smart Pipeline Connections
• Pipeline Loops
• Serialization
• Pipeline Breakpoints and Debugging Pipelines
• Visualizing Pipelines
• Secret Management
• Document Store and Metadata Filtering
• Agent, State and Hooks
• Tool, MCPTool and AgentTool
• Human in the Loop, Compaction Hook and Token Budget
• Logging, Tracing, OpenTelemetry and Langfuse
• Telemetry
• Deployment, Docker and Kubernetes
• Hayhooks and the Hayhooks CLI reference
• Hayhooks environment variables
• Breaking change policy and Migration guide
• Haystack 3.2.0 release notes and 3.0.0 release notes
• MIGRATION.md
• haystack-ai on PyPI and hayhooks on PyPI

Mid-level Haystack interviews are rarely about naming components. They are about whether you can reason about a running system: where latency comes from, why a loop never ends, what happens when a model calls a tool it should not, and how you would know. The questions below are the ones that actually get asked, with the reasoning a strong answer shows. Everything is against Haystack 3.2.0.

A note on framing before the questions. Haystack 3.0, released in July 2026, removed a long list of 2.x APIs. Interviewers who have done an upgrade will probe for it, because answering with AsyncPipeline or OpenAIGenerator tells them your knowledge predates the version they run. Saying "in 3.x that changed to…" is one of the cheapest ways to sound current.

Scenario: the RAG endpoint got slow after we added a reranker

Start by separating the three places time can go, because the fix differs for each: retrieval, the model call, and the framework's own scheduling.

The first thing to check is which run method the endpoint uses. pipeline.run() is synchronous and executes components sequentially — it has no concurrency_limit at all. If the endpoint is async and calls run(), two independent branches that could overlap are running one after the other, and the thread is blocked for the whole generation. Moving to await pipeline.run_async(data, concurrency_limit=4) is often the single largest win and requires no change to the graph.

The second is warm-up. Since 3.0, components build models and HTTP clients in warm_up(), not __init__, and warm-up runs automatically on first use. A reranker usually means a model to load. If the pipeline is constructed per request, or constructed at startup but never warmed, every cold pod pays model-loading cost on real traffic. Build once at startup, call warm_up_async() there, and close_async() on shutdown.

The third is measurement, not guessing. Enable tracing — it is never automatic in 3.x; auto_enable_tracing was removed and HAYSTACK_AUTO_TRACE_ENABLED does nothing — and look at the per-component spans, which carry haystack.component.name and .type. That tells you in one look whether the reranker, the retriever or the generator owns the time. For a quick local answer, LoggingTracer with tracing.tracer.is_content_tracing_enabled = True prints each component's inputs and outputs.

Only then talk about tuning: a smaller top_k into the reranker, batching at indexing time rather than query time, or a local embedder through sentence-transformers-haystack instead of a paid API call per query.

What a weak answer misses: it jumps straight to "reduce top_k" without establishing where the time went, and it does not know that run() is sequential. The interviewer is testing whether you measure before you tune.

Scenario: the pipeline raises PipelineMaxComponentRuns in production

The message names the component and the limit: Maximum run count 100 reached for component '<name>'. That is max_runs_per_component, a Pipeline(...) argument defaulting to 100, and it is the loop fuse rather than the bug.

The bug is a cycle whose exit condition never becomes true. So: find the ConditionalRouter in the loop and ask what its conditions evaluate against. The usual causes are a condition that depends on a field the model stopped producing, a validation that can never pass because the prompt does not ask for the right format, or a route whose output_type quietly coerces the value so the comparison never matches.

Three concrete moves. First, lower the limit while debugging — set it to 2 or 3 so each reproduction costs two model calls instead of a hundred. Second, use include_outputs_from covering the router and the generator, so you can see the actual value the condition tested. Third, after fixing it, leave the limit low: a retry loop that is designed for three attempts should have a fuse near three, not a hundred, because the default turns a logic bug into a bill nobody gets paged for.

Mention the sibling error, because interviewers like the pairing: PipelineComponentsBlockedError is the same root cause in the other direction — a cycle with no valid entry point, where a component is waiting for an input that only the loop produces. In 3.x that is very often a required template variable, because ChatPromptBuilder, PromptBuilder, Agent and LLM now default to required_variables="*". The fixes are a BranchJoiner to provide the first value, an explicit required_variables=[...] list, or required_variables=None to restore the 2.x all-optional behaviour.

A strong answer names both errors, explains that they share a cause, and treats the limit as a safety net rather than something to raise.

Scenario: an agent ran for twenty minutes and cost a fortune

Ask for four numbers first, because the Agent produces all of them: exit_reason, step_count, token_usage and tool_call_counts. They narrow the diagnosis immediately.

If exit_reason is "max_agent_steps", the agent never satisfied its exit condition. The default max_agent_steps is 100, which is far too high for production — it exists so tutorials never hit it. Set it to what the task needs. Also check exit_conditions: since 3.0 they are no longer validated against tool names at init, so a typo'd tool name means the condition simply never fires and the agent loops until the step cap.

If step_count is small but tokens are large, context growth is the problem. Each step resends the whole message history, so one tool that returns a 200 KB payload multiplies the cost of every subsequent step. Two hooks address it: ToolResultOffloadHook at after_tool, which moves large results out of the context and leaves a reference, and CompactionHook at before_llm with SlidingWindowCompactor, ToolResultPruningCompactor or the 3.2 SummarizationCompactor pointed at a cheap model. Both are marked experimental, so pin the version.

For a hard ceiling, TokenBudgetHook(max_total_tokens=...) ends the run with exit_reason="token_budget_exceeded". That is the control you want on a user-facing endpoint, because it converts an unbounded cost into a predictable failure you can handle.

Finally, record those four numbers as metrics per request. Cost questions are unanswerable without them, and they are already in the output dict — there is no instrumentation to write.

Scenario: the model called a tool it should not have

This question is really about where authority lives, and the correct answer is architectural rather than a prompt tweak.

The mechanism is inputs_from_state={"state_key": "param_name"} on the tool. It injects a value from State and hides that parameter from the schema the model sees. So a tenant id, a user id or an account number should arrive from your request context into State, never as a model-supplied parameter. Haystack 3.0 removed implicit State injection by name precisely so that this is explicit: a tool reads State only if it declares it.

For tools that have side effects — sending email, writing to a database, moving money — add ConfirmationHook at the before_tool point. It takes a mapping from tool name, tuple of names, or "*" to a BlockingConfirmationStrategy, which pairs a policy (AlwaysAskPolicy, AskOncePolicy, NeverAskPolicy) with a UI (SimpleConsoleUI, RichConsoleUI). Note the limitation and say it out loud, because it is exactly the kind of detail that distinguishes a careful answer: confirmation strategies see only the arguments the model supplied, not the values injected from State, so an approval screen will not display the injected tenant.

Also mention raise_on_tool_invocation_failure, which defaults to False. A tool error is serialized and fed back to the model as a message so it can retry. For a research tool that is desirable; for a payment tool a silent retry is worse than a crash, so set it to True there and handle ToolNotFoundException.

And the per-caller version of the same idea: Agent.run(tools=[...]) accepts tools, toolsets or plain tool-name strings at runtime, so one shared agent can expose different capabilities to different tenants without a second agent instance.

Scenario: it works locally and fails on the second pod

Two causes account for almost all of these, and naming both is the strong answer.

InMemoryDocumentStore is per process. Scaling to two replicas gives you two different corpora, so the same query returns different answers depending on routing. It is also process-global per index by default, because shared=True; pass shared=False for a request-local store, or memory grows with traffic. The fix for production is a real store integration — Qdrant, pgvector, Elasticsearch, OpenSearch, Weaviate, Pinecone and many others ship as *-haystack packages.

Secrets and serialization. If the pipeline is loaded from YAML on the pod, a Secret.from_token(...) could never have been serialized in the first place — dumps() refuses it with "Cannot serialize token-based secret." So the YAML holds a Secret.from_env_var reference, which means the pod must have that variable. In Kubernetes that is a secretKeyRef into the container's environment. If the variable is missing, the failure now appears at warm_up() or the first run rather than at construction, which is why it looks like a runtime bug rather than a config bug.

A third, less common cause worth a sentence: if the YAML was written by an older version, loading may fail with a DeserializationError about an unknown parameter — tool_invoker_kwargs or confirmation_strategies, for instance — because 3.0 validates nested init_parameters against the parent's __init__ signature. Stale YAML that used to be ignored now fails loudly.

Trade-off: Hayhooks or your own FastAPI app

Both are defensible; the interviewer wants to hear the axis you are choosing on.

Hayhooks gives you REST, an OpenAI-compatible endpoint, MCP and A2A for free, plus a CLI for deploying and undeploying pipeline directories. The OpenAI-compatible endpoint is the real argument: it means Open WebUI, or anything else that speaks that protocol, works without you writing an adapter. The cost is that Hayhooks has no authentication setting in its environment-variable reference, and HAYHOOKS_CORS_ALLOW_ORIGINS defaults to ["*"]. You must put it behind an ingress or gateway that terminates TLS and authenticates, and keep HAYHOOKS_SHOW_TRACEBACKS false.

Your own FastAPI app gives you auth, rate limiting, request validation, multi-tenancy and a response schema that matches the rest of your API. You write the lifespan plumbing: build pipelines once at startup, await warm_up_async(), await run_async(...) or stream(...) in handlers, await close_async() on shutdown. create_app() also lets you embed Hayhooks inside your own app, which is a reasonable middle path.

The honest summary: Hayhooks for internal tools and for exposing pipelines to chat front ends quickly; your own app when the endpoint is part of a product with existing auth and SLA conventions.

Trade-off: one Agent or a pipeline with a router

An agent delegates control flow to the model; a pipeline with a ConditionalRouter keeps control flow in code you can read, test and diff.

Choose the pipeline when the decision tree is known. It is cheaper, because there is no extra model call to decide what to do; it is deterministic, so a test asserts a path rather than a tendency; and the type checking at connect() time catches wiring mistakes in CI. Choose an agent when the number of steps genuinely depends on what the data turns out to be — multi-hop research, iterative refinement, tool selection over a large surface.

The useful nuance is that these compose. ComponentTool, PipelineTool and AgentTool (3.1) wrap a component, a pipeline or another agent as a tool. So the common production shape is a deterministic retrieval pipeline wrapped as a tool and handed to a small agent, which keeps the expensive part testable in isolation while leaving the model free to decide how many times to call it.

What a weak answer misses: treating "agentic" as a quality rather than a cost. Every agent step is a model call, and the default max_agent_steps=100 means an unbounded one unless you change it.

Trade-off: how much should live in YAML

Serializing pipelines to YAML is attractive because non-engineers can edit them and you can deploy a change without a release. The counter-argument is the one Haystack's own 3.x security work makes: YAML is code.

Loading is gated by a trusted-module allowlist (haystack, haystack_integrations, haystack_experimental, builtins, typing, collections), dangerous builtins like eval and __import__ are blocked, nested parameters are validated, and several releases fixed allowlist-bypass gadgets. All of that exists because a crafted YAML file naming an arbitrary importable class is remote code execution. So YAML that lives in your repository and goes through review is fine; YAML that a tenant can edit is not, and unsafe=True or HAYSTACK_UNSAFE_DESERIALIZATION must never touch untrusted input.

A sound position: keep structure in Python, where it is type-checked and testable, and put values — model names, top_k, prompt text — in configuration. If you do serialize whole pipelines, extend the allowlist narrowly with allowed_modules=["mypkg.*"] rather than switching the check off, and add a test that round-trips dumps() and loads() so serialization breakage is caught before deployment.

Trade-off: when to upgrade

The answer starts with a quotation from the docs: "Haystack DOES NOT follow the principles of Semantic Versioning." Minor releases may add or remove functionality, and the docs state that deprecated components are the most common breaking change in a minor release. A feature deprecated in X.Y raises FutureWarning and is removed in X.Y+1 — about one month of notice.

So: pin exactly (haystack-ai==3.2.*), pin every integration separately because *-haystack packages are versioned independently, treat FutureWarning as an error in CI, and read the "Upgrade Notes" of each minor release before taking it.

For 2.x to 3.x specifically, the migration is: move to 2.31 first and clear every deprecation warning there, then upgrade. The work that remains after imports compile is the part teams underestimate — re-ingest documents that have meta, because the Document.id hash changed and deduplication against 2.x data silently fails; audit YAML against the allowlist; audit prompt templates for the new required_variables="*" default; and re-check tracing, because it is no longer auto-enabled and the span names changed.

Looking ahead, 3.3.0-rc1 exists but is a pre-release. When it lands, in-memory BM25 returns only documents containing at least one query term, so results can be fewer than top_k, and sentence-split boundaries change around quoted sentences. Both change retrieval output, so plan to re-index and re-baseline evaluations.

What separates a strong answer from a weak one

Across every question above, the same four habits mark the strong candidate.

They name the version and what changed in it. Saying "AsyncPipeline was removed in 3.0; it is one Pipeline with run, run_async, run_async_generator and stream" immediately establishes that your knowledge is current. So does mentioning that the legacy non-chat generators are gone, that ToolInvoker was removed because the Agent runs tools itself, and that about thirty components moved into *-haystack integration packages.

They measure before tuning. Every performance answer should reach for traces, include_outputs_from, or the Agent's token_usage before it reaches for a parameter. "I would look at the component spans" is a better opening than any specific fix.

They treat defaults as decisions. max_runs_per_component=100, max_agent_steps=100, tool_concurrency_limit=4, raise_on_tool_invocation_failure=False, shared=True on the in-memory store, HAYHOOKS_CORS_ALLOW_ORIGINS=["*"]: each of those is a choice the framework made for a tutorial, and a strong candidate knows which ones to change for production and why.

They keep authority out of the model. Per-request values arrive through State with inputs_from_state, side effects go through ConfirmationHook, and untrusted YAML never reaches Pipeline.load. Candidates who answer the tool-misuse question with "I'd improve the system prompt" and nothing else are telling you they have not run an agent in front of real users.

One last piece of practical advice: have one real failure ready to describe. A blocked pipeline you traced to a required template variable, an agent that burned tokens until you added a budget hook, a duplicate corpus after a 3.x upgrade. The specifics are what make the rest of your answers credible.

These are the mistakes that cost mid-level Haystack users real time, the practices that pay back most, and a set of recipes you can lift straight into a project. Everything is against Haystack 3.2.0 on Python 3.10 or later.

Errors you will actually hit, and what they mean

ImportError: cannot import name 'AsyncPipeline' from 'haystack' — and the same for OpenAIGenerator, ToolInvoker, DALLEImageGenerator or SentenceTransformersTextEmbedder. All of these are 3.0 removals, not bugs. There is one Pipeline class with run, run_async, run_async_generator and stream; the legacy non-chat generators are replaced by Chat Generators; the Agent runs tools itself; DALLEImageGenerator is now OpenAIImageGenerator; and the Sentence Transformers components live in sentence-transformers-haystack under haystack_integrations.

ValueError: Missing mandatory input '<var>' for component '<name>'. This is the most common 3.x surprise. ChatPromptBuilder, PromptBuilder, Agent and LLM now default to required_variables="*", so every variable in your template is mandatory. In 2.x they were all optional. Either supply or connect the variable, pass a list of the ones that are genuinely required, or pass required_variables=None to restore the old behaviour.

PipelineComponentsBlockedError — "Cannot run pipeline - all components are blocked." A cycle with no valid entry point, or a required input nothing produces. Nearly always a loop whose first iteration needs a value that only the loop produces. Add a BranchJoiner, or make the variable optional.

PipelineMaxComponentRuns: Maximum run count 100 reached for component '<name>' A loop that never exits. The limit is the fuse, not the bug: fix the router condition, and while you debug set max_runs_per_component=2 so each attempt is cheap.

PipelineConnectError comes in several shapes, each telling you exactly what to do. "their declared input and output types do not match" means fix the types or insert an OutputAdapter. "more than one connection is possible between these components" means name the sockets: connect("a.x", "b.y"). "cannot accept multiple inputs to 'x' … only if its type is list, Optional[list], or union of list types" means your fan-in target is not list-typed. "does not have any output connections … for example by using the '@component.output_types' decorator" means you forgot the decorator on a custom component.

ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',) Since 3.0 this appears at warm_up() or the first run, not at construction, because resources are created at warm-up. Export the key, or use Secret.from_env_var(..., strict=False) for genuinely optional secrets.

Cannot serialize token-based secret. Use an alternative secret type like environment variables. You called dumps() on a component built with Secret.from_token. This is the framework stopping you from writing a key into a file. Use Secret.from_env_var.

DeserializationError: Refusing to deserialize a class from module '<m>': the module is not on the trusted-module allowlist. Your own package is not on the default list (haystack, haystack_integrations, haystack_experimental, builtins, typing, collections). Add it narrowly with allowed_modules=["mypkg.*"]. The sibling message "Refusing to deserialize unknown parameter '' for ''" means stale YAML — tool_invoker_kwargs or confirmation_strategies on an Agent are the usual culprits — because nested init_parameters are now validated against the parent's __init__.

DeserializationError when loading an OutputAdapter or ConditionalRouter that carries unsafe: true or Jinja custom_filters. This one is legitimate and needs Pipeline.load(fp, unsafe=True) — only for files you wrote yourself.

YAML containing _ToolsetWrapper fails to load in 3.2. The + operator for combining Toolsets was removed. Rebuild with tools=[toolset_a, toolset_b] and re-dump.

ValueError at Agent(...) construction when state_schema uses a reserved key: step_count, token_usage, tool_call_counts, continue_run, exit_reason, stop_run, tools, hook_context, context_tokens. Rename yours, for example my_token_usage.

PipelineRuntimeError wraps the real exception — read the inner Error: line, and remember e.pipeline_snapshot lets you resume rather than re-run.

Duplicate or ghost documents after upgrading to 3.x. The Document.id hash changed: it now uses canonical key-sorted JSON of meta, with non-JSON values via str() rather than repr(). Any document with non-empty meta gets a different auto-id than in 2.x, so DuplicatePolicy.SKIP or FAIL against 2.x data silently does nothing. Re-ingest, or pass explicit ids.

Haystack failed to import the optional dependency 'pypdf'. Run 'pip install pypdf'. Optional dependencies are lazy. Install the named package: pypdf for PyPDFToDocument, trafilatura for HTMLToDocument, tiktoken for token splitting and TiktokenCounter, rich for RichConsoleUI.

Traces are missing even though the SDK is installed. Tracing is never auto-enabled in 3.x. Call tracing.enable_tracing(...) or add a connector component. For Langfuse, also set HAYSTACK_CONTENT_TRACING_ENABLED=true before importing Haystack, and call tracer.flush() before the process exits.

Duplicated log lines, or your app's log format changed. Use configure_logging(propagate=False). The opposite need — format every record in the process — is configure_logging(logger_name="").

A sync streaming_callback in run_async only logs a warning while blocking the event loop for the whole generation. Pass an async callback in async code.

The practices that pay off most, in order of leverage

1. Bound every loop deliberately. Change two defaults on day one: Pipeline(max_runs_per_component=...) from 100 to your intended retry count, and Agent(max_agent_steps=...) from 100 to the number of steps the task needs. These are the two numbers that turn a logic bug into an invoice, and nobody gets paged for an invoice. Add TokenBudgetHook(max_total_tokens=...) on any user-facing agent so an unbounded cost becomes a handled failure with exit_reason="token_budget_exceeded".

2. Use stable document ids. Derive them from your source — f"{system}:{record_id}:{revision}" — and write with DuplicatePolicy.OVERWRITE. This makes re-ingestion idempotent, makes deletion possible, and makes you immune to any future change in the hashing scheme. It is five minutes of work that removes a whole category of retrieval bugs.

3. Build once, warm up explicitly, close on shutdown. Construct pipelines and agents at startup, call warm_up_async() there so the first real request is not slow, and close_async() on shutdown so HTTP clients, database connections and GPU memory are released. If you write a custom warm_up(), make it idempotent — since 3.1 the Agent warms up its tools and hooks before every run, so a non-idempotent warm-up opens a client per request.

4. Keep per-request authority out of the model. Any value that identifies a tenant, user or account goes through State and inputs_from_state={"tenant": "tenant"}, which injects it and hides the parameter from the model's schema. Pair it with a server-side metadata filter and FilterPolicy.MERGE so a caller-supplied filter narrows results rather than replacing your isolation.

5. Test the wiring, mock the model. MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder (3.0) plus a BM25-only path over InMemoryDocumentStore let you run most of your suite with no API key and no network. Inject the generator into a builder function so the same code serves tests and production. Round-trip dumps()/loads() in a test too, so serialization breakage fails in CI rather than on a pod.

6. Measure before you tune. include_outputs_from costs nothing and shows you what actually moved between components. Enable OpenTelemetry in staging and learn the span names — haystack.pipeline.run, the per-component spans, haystack.agent.step with its .llm and .tool children — so a latency question has an answer rather than a theory.

7. Treat YAML as code. Keep it in the repository, review it, pin it. Extend the deserialization allowlist narrowly with allowed_modules rather than reaching for unsafe=True, and never set HAYSTACK_UNSAFE_DESERIALIZATION in a deployment manifest.

8. Pin exactly and fail on FutureWarning. haystack-ai==3.2.*, every integration pinned separately, and -W error::FutureWarning in your test command. A minor release can remove functionality, and a deprecation in X.Y is gone in X.Y+1.

9. Prefer nested run data over flat. pipeline.run({"query": q}) routes that value to every component declaring a query input, which is convenient in a notebook and a trap in a service — adding a component later silently changes who receives it. Write {"retriever": {"query": q}} in anything you deploy.

10. Insert an OutputAdapter rather than relying on smart connections. Haystack adapts a small fixed set of mismatches at connect time: str to ChatMessage, ChatMessage to str, T to list[T], and a list to its first item. An exact type match always wins over an adaptation, so adding a precisely-typed socket later can change which conversion happens. An explicit adapter is visible in the graph, in the diagram and in the serialized YAML.

11. Read live State objects through state.data. State.get returns a deep copy, which is right for values and wrong for anything you need to mutate or reuse — a database client, a WebSocket, a request-scoped resource passed in through hook_context. For those, read state.data["hook_context"] directly.

Practice cards

Card 1 — Make the pipeline tell you what it wants. Take an existing pipeline, print pipeline.inputs(), then write a test asserting that exact set. Now delete one connect() call and watch the test fail. You have just made the whole class of "someone added a component and forgot a wire" a CI failure instead of a production one.

Card 2 — Break the loop on purpose. Build a self-correcting loop: BranchJoiner into ChatPromptBuilder into a Chat Generator into a ConditionalRouter whose failure route goes back to the joiner. Set max_runs_per_component=3. Force the validation to always fail and read the PipelineMaxComponentRuns message. Then remove the BranchJoiner and read PipelineComponentsBlockedError. Those two errors will cost you an hour each in production if you have not seen them before.

Card 3 — Hide a parameter from the model. Take a tool that accepts a tenant id as a parameter. Move it to inputs_from_state, pass the tenant as a run keyword, and print the tool's schema before and after. Confirm the parameter has disappeared from what the model can see. This is the exercise that makes multi-tenant agent design click.

Card 4 — Resume instead of re-running. Set HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED=true, add a Breakpoint(component_name="llm", visit_count=0, snapshot_file_path="/snaps"), catch the BreakpointException, inspect exc.inputs, then resume with pipeline.run(data={}, pipeline_snapshot=load_pipeline_snapshot(path)). Compare the time with re-running the expensive retrieval step.

Card 5 — Account for an agent run. Run an agent on a task it cannot complete with max_agent_steps=3. Print exit_reason, step_count, tool_call_counts and token_usage. Then add a tool that returns a very large string and watch token_usage grow non-linearly across steps. Add ToolResultOffloadHook at after_tool and measure again.

Card 6 — Survive a round trip. Serialize a pipeline that uses a custom component with dumps(), load it in a fresh Python process, and watch the allowlist refuse it. Fix it with allowed_modules=["mypkg.*"]. Then deliberately break the 1:1 mapping between an init parameter and its attribute name and see serialization fail.

Card 7 — Catch a deprecation before it removes itself. Run your test suite with -W error::FutureWarning. Every failure is something that will disappear in the next minor release, which is roughly a month away. Fix them while the old and new forms both work, rather than during an outage when only one does. Then read the current release's upgrade notes and check each item against your code; it takes twenty minutes and it is the whole of upgrade safety on a framework that does not follow SemVer.

Card 8 — Promote a sub-graph to a SuperComponent. Find wiring you have copied between two pipelines — a converter, cleaner and splitter chain is the usual one — and wrap it with SuperComponent(pipeline, input_mapping=..., output_mapping=...) or a class decorated with @super_component. Drop it into both pipelines as a single component, then render the diagram with super_component_expansion=True to confirm the inside is what you expect. The built-in DocumentPreprocessor and MultiFileConverter are exactly this pattern, so you are following the framework rather than working around it.

Card 9 — Index twice, count once. Run your indexing pipeline twice and call store.count_documents() after each pass. If the number doubles, you do not have stable ids. Add explicit ids plus DuplicatePolicy.OVERWRITE and confirm the second run changes nothing.

Operational recipes

Get intermediate values out of a run.

result = pipe.run(
    {"retriever": {"query": q}, "prompt_builder": {"question": q}},
    include_outputs_from={"retriever", "prompt_builder"},
)

Only unconnected outputs appear in the result by default, and only the last one per component. This is the first debugging move, every time.

Turn on verbose tracing locally.

import logging
from haystack import tracing
from haystack.tracing.logging_tracer import LoggingTracer

logging.getLogger("haystack").setLevel(logging.DEBUG)   # default is WARNING
tracing.tracer.is_content_tracing_enabled = True        # off by default: prompts in spans
tracing.enable_tracing(LoggingTracer())

Leave content tracing off outside development — spans would carry prompts and personal data, which matters for any data-residency review.

Production environment variables worth setting.

export HAYSTACK_LOGGING_USE_JSON=true        # JSON logs (automatic when there is no TTY)
export HAYSTACK_TELEMETRY_ENABLED=False      # opt out of anonymous PostHog events
export OPENAI_TIMEOUT=30                     # default 30.0 seconds
export OPENAI_MAX_RETRIES=5                  # default 5

Telemetry sends component-usage events with a random UUID and no IP, hostname, paths, queries or content, to an EU-hosted PostHog. The config lives at ~/.haystack/config.yaml. Opting out is still the right default for a regulated environment, and it is a question you will be asked in a Gulf or Egyptian data-review.

Draw a pipeline without leaking it.

docker run --platform linux/amd64 --publish 3000:3000 --cap-add=SYS_ADMIN \
  ghcr.io/jihchi/mermaid.ink

pipe.draw(path="pipeline.png", server_url="http://localhost:3000")

draw and show send your graph to the public mermaid.ink service by default. Note also that both take keyword-only arguments in 3.2 — draw("file.png") positionally will fail, so always write path=.

Harden a Hayhooks deployment.

export HAYHOOKS_HOST=0.0.0.0
export HAYHOOKS_PORT=1416
export HAYHOOKS_PIPELINES_DIR=/app/pipelines
export HAYHOOKS_CORS_ALLOW_ORIGINS='["https://support.example.com"]'
export HAYHOOKS_SHOW_TRACEBACKS=false
hayhooks run

Hayhooks has no authentication setting of its own, and CORS defaults to ["*"]. Put it behind an ingress or gateway that terminates TLS and authenticates callers, and never expose 1416 directly. The MCP server defaults to 1417 and A2A to 1418.

Bake dependencies into the image.

FROM deepset/hayhooks:v1.25.0
RUN pip install --no-cache-dir qdrant-haystack opentelemetry-haystack tiktoken
COPY pipelines /app/pipelines

Installing packages in the container command is a docs demo, not a deployment pattern — it makes startup slow and dependent on PyPI being up. The Haystack base image is deepset/haystack:base-v3.2.0, whose only flavour is base, equivalent to pip install haystack-ai.

Inject a key in Kubernetes.

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: openai-secret
        key: api-key

This is the whole reason to use Secret.from_env_var: the serialized pipeline stores only the variable name, and the platform supplies the value.

Split by token rather than by word.

from haystack.components.preprocessors import DocumentSplitter

splitter = DocumentSplitter(
    split_by="token", tokenizer_encoding="o200k_base",
    split_length=400, split_overlap=40,
)

split_by="token" is new in 3.2 and needs pip install tiktoken. It bounds chunks in the unit your model bills in. split_overlap >= split_length raises ValueError at init, which is a useful guard rail.

A zero-cost smoke test for CI.

from haystack.components.generators.chat import MockChatGenerator

def test_pipeline_runs_without_credentials():
    pipe = build_rag(MockChatGenerator())
    out = pipe.run({"retriever": {"query": "backups"},
                    "prompt_builder": {"question": "When do backups run?"}})
    assert out["llm"]["replies"][0].text is not None

Run it with OPENAI_API_KEY unset. If it fails, something reaches for a credential at construction time that should only need it at warm-up.

The checklist before you call it production

Nine things, each a failure somebody has already had.

1. max_runs_per_component and max_agent_steps lowered from 100 to what the work needs, and a token budget on any user-facing agent.
2. No InMemoryDocumentStore behind more than one replica; a store integration instead.
3. Stable document ids and DuplicatePolicy.OVERWRITE, verified by indexing twice and counting.
4. Every secret through Secret.from_env_var, injected by the platform; no from_token anywhere near dumps().
5. Pipelines built once at startup, warmed up explicitly, closed on shutdown; custom warm_up() idempotent.
6. Async endpoints using run_async or stream, not run; async streaming callbacks in async code.
7. Tracing explicitly enabled, span names checked against your dashboards, content tracing off unless you decided otherwise.
8. The deserialization allowlist on, extended narrowly; unsafe=True only for files from your own repository.
9. Versions pinned exactly, integrations pinned separately, FutureWarning an error in CI, and the next release's upgrade notes read before you take it.

If you want the depth behind any of these, the detailed guide covers each mechanism, and Langfuse, RAGAS and Qdrant cover the neighbours you will reach for first.

This is part three of three. It takes every topic from Beginner and Mid one level further, and then adds the work that only exists when Haystack is a platform you run for other people rather than a library you import into your own service.

Up to now the questions have been "does this retrieve the right documents" and "does the agent finish". From here they are different: whose YAML is allowed to load in your process, what a tenant can see through a filter they control, which component falls over first when traffic triples, what the token bill looks like when four teams share one Agent, and what you do at 02:00 when every request to the RAG endpoint returns a 500 and the only clue is a wrapped PipelineRuntimeError.

Everything here is verified against Haystack 3.2.0 (haystack-ai, released 2026-09-24) and Hayhooks 1.25.0. That matters more than usual, because 3.0 landed in July 2026 and broke much of what the internet still says about Haystack. If you inherited a 2.x platform, read the upgrade section first.

Where this picks up

I am starting with architecture, because almost every failure mode below is a consequence of one structural fact: Haystack is a library, not a server, and the server is yours.

The architecture you actually own

Haystack ships no daemon, no scheduler, no control plane and no queue. A Pipeline is a directed multigraph of Python objects living in one process. That is a genuine strength — nothing to operate, nothing to upgrade separately — and the source of the biggest misunderstanding senior engineers bring to it: there is no Haystack cluster to size, only your service, which happens to contain Haystack. So the architecture diagram you own has four layers, and only the third one is Haystack's.

Two architectural decisions follow from that picture, and both are yours to make before anyone writes a pipeline.

Decide whether pipelines are code or configuration. A pipeline can be Python assembled in your repository, or YAML loaded at runtime via Pipeline.load. Code gives you review, typing, tests and a deploy as the unit of change. YAML lets consuming teams ship a prompt change without touching your service — and, unless you are careful, execute arbitrary classes inside your process. There is no wrong answer, but there is a wrong answer for untrusted authors: if the YAML's authors could not already merge to your service, treat the file as a privileged artifact with its own review gate.

Decide the unit of deployment. One service per pipeline is easy to bill and isolates failure. One service hosting many pipelines — the Hayhooks model, where HAYHOOKS_PIPELINES_DIR is auto-deployed at startup — is cheaper and lets teams self-serve, at the cost of shared fate: a memory-hungry local embedder in one tenant's pipeline can OOM the pod that serves everyone. My default past three consumer teams is Hayhooks per domain.

Failure modes, ranked by how often they bite

Senior work is largely pattern recognition, so here are the failures that recur, in rough order of frequency, with the mechanism rather than just the fix.

A loop that never exits. Both safety nets default generously: Pipeline(max_runs_per_component=100) raises PipelineMaxComponentRuns, and Agent(max_agent_steps=100) stops with exit_reason="max_agent_steps". A hundred LLM calls is not a safety net, it is an invoice. Lower both — 10 to 20 for an interactive agent — and treat hitting the ceiling as alertable. Watch exit_reason: a rising share of max_agent_steps usually means a tool is returning an error string the model cannot act on, because raise_on_tool_invocation_failure defaults to False and feeds tool failures back to the LLM as a message.

Context growth until the provider refuses. An agent calling tools that return large payloads grows its message list every step. The failure is loud — a provider error or exit_reason="length" — but intermittent, because it depends on how much the tool returned that day. Haystack has three composable answers: CompactionHook on before_llm (with SlidingWindowCompactor, ToolResultPruningCompactor or SummarizationCompactor), ToolResultOffloadHook on after_tool to push big results to a FileSystemToolResultStore, and TokenBudgetHook to end the run cleanly with exit_reason="token_budget_exceeded". Compaction and token budgets are marked experimental in 3.2, which for a platform means pin the version and read the upgrade notes, not avoid them.

A pipeline that cannot start at all. PipelineComponentsBlockedError says every component is blocked, and names the two causes: no valid entry point, or a circular dependency. In practice it is a loop whose entry value is produced only by the loop itself, often a required template variable — 3.x made every template variable required by default (required_variables="*"). Fix it with a BranchJoiner at the loop entrance, or by requiring only the variables you truly need.

Deserialization refusals after an upgrade. DeserializationError saying "the module is not on the trusted-module allowlist" or "Refusing to deserialize unknown parameter" means your YAML is older than your code. The second is the useful one: nested init_parameters are validated against the parent's __init__ signature, so a leftover tool_invoker_kwargs or confirmation_strategies on an Agent fails loudly instead of being ignored.

Resources that fail late. Since 3.0, components build their clients in warm_up(), not __init__, so a missing OPENAI_API_KEY raises ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',) at warm-up or first run. That moves the failure from deploy time to first-request time unless you call warm_up() explicitly at startup. Do call it explicitly: it is also the cheapest readiness probe you have.

Duplicate or ghost documents after the 3.0 upgrade. Document.id is auto-hashed from content and meta, and in 3.0 the hash changed to canonical key-sorted JSON of meta, with non-JSON values through str() rather than repr(). Any document with non-empty meta gets a different auto-ID than in 2.x, so DuplicatePolicy.SKIP or FAIL dedup against 2.x data silently stops working and the index doubles. Re-ingest, or pass explicit id= values.

Scaling: which component breaks first

Because pipelines are described as usually stateless, horizontal scaling is the easy part: run N replicas of your serving process behind a Service or Ingress, and let the load balancer do the work. The interesting question is what stops improving when you add replicas.

First to break: the document store. Every replica queries the same store, so replicas multiply query load without adding query capacity. InMemoryDocumentStore must never reach a multi-replica deployment: it lives in process memory and is not shared, so two pods answer from two different corpora. Worse, it is process-global per index by default (shared=True), so a store you create per call is not actually request-scoped and memory grows for the life of the worker. Use shared=False when you genuinely want a per-request store, and reach for a real store integration — Qdrant, pgvector, Weaviate, Pinecone, Milvus, OpenSearch, Elasticsearch and a long list of others — for anything a user sees.

Second: the model provider. You are rate-limited by tokens per minute, per account or deployment rather than per pod, so adding replicas makes you hit it faster. The controls are OPENAI_TIMEOUT (default 30 seconds) and OPENAI_MAX_RETRIES (default 5), both overridable per component, plus FallbackChatGenerator to fail over across models or providers. Five retries at thirty seconds is a 150-second worst case, longer than most gateways will wait — tune both deliberately.

Third: the event loop. Pipeline.run is synchronous and, since 3.0, runs components sequentially with no concurrency_limit, blocking a worker thread for the whole pipeline. Use run_async, whose concurrency_limit defaults to 4, or stream() for SSE. Be precise about cancellation: it interrupts native-async components only. Sync components run in worker threads and their side effects complete after the client has gone — a tool that charges a card will still charge it.

Fourth: warm-up cost. Since 3.1 the Agent warms up hooks, tools and toolsets before every run, so an unguarded warm_up() that loads a model or opens a pool costs you that on every request. The fix is a one-line idempotency guard, and it belongs in your component template.

from haystack.tools import Tool


class WarmableToolBackend:
    """Shared base for tool backends. warm_up must be cheap on every call
    after the first, because the Agent warms tools up before every run."""

    def __init__(self, endpoint: str) -> None:
        self.endpoint = endpoint
        self._client = None

    def warm_up(self) -> None:
        if self._client is not None:   # idempotent, deliberately
            return
        self._client = build_http_client(self.endpoint)

    def close(self) -> None:
        if self._client is not None:
            self._client.close()
            self._client = None

Fifth: tool fan-out. Tool calls run in parallel up to tool_concurrency_limit, default 4. Raising it raises throughput and simultaneous load on whatever those tools call. Since 3.0, runs sharing a Toolset get isolated per-run copies via Toolset.spawn(), so concurrency is correct — the risk is purely that you have multiplied your own backend's traffic by the number of in-flight runs.

The trust model: pipeline YAML is code

This is the section to read if you read only one. Haystack serialization is genuinely useful — pipe.dumps() gives YAML, Pipeline.loads() reads it back, and the format is readable enough to diff in a pull request. It is also, by construction, a description of which Python classes to import and instantiate with which arguments. A crafted file is therefore a request to run code in your process.

Haystack 3.x takes this seriously, and you should understand exactly what the defences are, because they define the boundary of what is safe to accept.

A trusted-module allowlist. Deserialization is restricted to haystack, haystack_integrations, haystack_experimental, builtins, typing and collections; anything else raises DeserializationError: Refusing to deserialize a class from module '<m>': the module is not on the trusted-module allowlist.

Dangerous builtins blocked. eval, exec, compile, __import__, open and getattr are refused even though builtins is listed.

Nested parameter validation. Keys inside init_parameters are checked against the parent class's __init__ signature, so stale or misspelled YAML fails rather than being ignored.

Jinja hardening. ConditionalRouter and OutputAdapter evaluate Jinja. Since 3.1, loading a serialized one with unsafe: true or Jinja custom_filters requires Pipeline.load(..., unsafe=True); in 3.2 those callables no longer run during safe-mode loading, and callable-path deserialization was hardened further.

To load your own components you extend the allowlist, and there are three ways to do it, in descending order of how much I like them:

from haystack import Pipeline

# 1. Explicit, per-call, visible in review. Prefer this.
pipe = Pipeline.load(open("rag.yml"), allowed_modules=["haystack_platform.components.*"])

# 2. Process-wide, registered once at startup.
from haystack.core.serialization import allow_deserialization_module
allow_deserialization_module("haystack_platform")

# 3. Environment variable, read on every call. Easiest to change without review.
#    HAYSTACK_DESERIALIZATION_ALLOWLIST="haystack_platform.*,shared_tools.*"

Allowlist patterns are prefix matches, with fnmatch globs when the pattern contains *, ? or [ other than a trailing .*. The third form needs a policy decision: read on every call and living in the environment, it can be widened by anyone who can edit a deployment manifest. On a regulated platform keep it unset, so the allowlist stays reviewable code.

Two more trust boundaries are easy to miss. Prompt injection through retrieved content: ChatPromptBuilder sanitizes template variables so user input cannot become structured parts such as images or tool calls (hardened in 2.31), but it cannot stop retrieved text from containing instructions. The defence is architectural — guard tool side effects, not the prompt. That is what ConfirmationHook is for, under before_tool, with a BlockingConfirmationStrategy and AlwaysAskPolicy, AskOncePolicy or NeverAskPolicy per tool. The sharp edge: strategies see only the arguments the model supplied, not values injected from State via inputs_from_state, so if the dangerous parameter comes from State the approver is not seeing it.

Pipeline visualization: show() and draw(path=...) send the graph to the public mermaid.ink service by default, which for non-public component names and prompts is an unintended disclosure. Run the renderer yourself:

docker run --platform linux/amd64 --publish 3000:3000 --cap-add=SYS_ADMIN \
  ghcr.io/jihchi/mermaid.ink

from pathlib import Path

pipe.draw(path=Path("rag.png"), server_url="http://localhost:3000")

draw and show take keyword-only arguments in 3.2, so the older positional form from some examples fails.

Secrets and the supply chain

The rule is short: always Secret.from_env_var(...), never Secret.from_token(...) in anything serialized. The reason is mechanical. An env-var secret serializes as a reference to the variable name only:

api_key:
  type: env_var
  env_vars: [OPENAI_API_KEY]
  strict: true

A token secret cannot be serialized at all, and attempting it raises Cannot serialize token-based secret. Use an alternative secret type like environment variables. Since 2.31 its repr prints <redacted>, which stops the most common accidental leak into logs but does not make it safe to store. Secret.from_env_var(["A", "B"]) tries several variables in order, which is the clean way to support a migration from one key name to another without a flag day.

In Kubernetes the variable comes from a Secret via secretKeyRef, and the deployment is the only place the mapping lives:

env:
  - name: OPENAI_API_KEY
    valueFrom:
      secretKeyRef:
        name: openai-secret
        key: api-key
  - name: HAYSTACK_TELEMETRY_ENABLED
    value: "False"
  - name: HAYSTACK_LOGGING_USE_JSON
    value: "true"

For custom components, resolve secrets in warm_up() rather than __init__, serialize them with .to_dict() and deserialize with deserialize_secrets_inplace. Resolving in the constructor reintroduces the early-failure behaviour 3.0 deliberately moved, and it means a secret's value can end up captured in an object that gets pickled or logged.

The supply chain has three parts worth owning. Pinning: because the release policy allows removals in minor versions, pin haystack-ai==3.2.* and every integration package separately, since integrations are versioned independently. Images: build from deepset/haystack:base-v3.2.0 or deepset/hayhooks:v1.25.0 and bake integrations in at build time; the docs install packages in the container command only as a demonstration, and doing that in production makes a registry outage your outage. Telemetry: Haystack sends anonymous component-usage events to PostHog on an EU host, with a random UUID as distinct_id and no IP, hostname, paths, queries or content. That is a defensible default and still something to decide rather than discover. For Gulf and Egyptian employers with data-residency language in their contracts, set HAYSTACK_TELEMETRY_ENABLED=False in the base image so every pipeline inherits it, and record the decision where a security reviewer will find it.

FROM deepset/hayhooks:v1.25.0

ENV HAYSTACK_TELEMETRY_ENABLED=False \
    HAYSTACK_LOGGING_USE_JSON=true \
    HAYHOOKS_SHOW_TRACEBACKS=false

RUN pip install --no-cache-dir \
      "haystack-ai==3.2.*" \
      "qdrant-haystack==11.0.0" \
      "langfuse-haystack==7.0.0"

Multi-tenancy without a tenant model

Haystack has no tenant concept. There is no tenant_id anywhere in the framework, no row-level security, no per-tenant quota. Everything is a pattern you implement, and the patterns are good, but the enforcement point has to be yours.

Isolate the data. The choice is about blast radius. A separate store index or collection per tenant gives real isolation: a filter bug cannot leak across tenants because the data is not in the same place. Shared indexes with per-tenant metadata filters scale further and cost less, at the price that one wrong filter is a data breach. The filter is ordinary Haystack: {"field": "meta.tenant", "operator": "==", "value": tenant_id}, combined with any user-supplied filter under an AND.

The subtlety is filter_policy, which defaults to FilterPolicy.REPLACE: filters passed at run time replace those set at init. If your tenant filter lives in the component's filters argument and a caller passes their own at run time, the tenant filter disappears. That is the most dangerous default in a multi-tenant deployment. Use FilterPolicy.MERGE, or better, never let a caller's filter reach the retriever unmerged:

def scoped_filters(tenant_id: str, caller_filters: dict | None) -> dict:
    """Build the only filter object allowed to reach a retriever."""
    tenant = {"field": "meta.tenant", "operator": "==", "value": tenant_id}
    if not caller_filters:
        return tenant
    return {"operator": "AND", "conditions": [tenant, caller_filters]}

Call that at the edge, from an identity your gateway established, and never from a request body. Also remember that filter support varies per store — Chroma, for instance, has no NOT but adds contains — and that an unsupported operator raises FilterError rather than silently matching everything. Test your tenant filter against the store you actually run.

Isolate the capabilities. One shared Agent can give each tenant a different tool subset, because Agent.run accepts tools= at run time, as Tool/Toolset objects or as tool names — a stated 3.0 use case, and much cheaper than an Agent per tenant. Note the 3.2 change: the + operator for combining Toolsets was removed, so pass tools=[toolset_a, toolset_b], and YAML containing _ToolsetWrapper no longer loads.

Isolate request-scoped resources. Pass per-request objects — a user ID, a database client, a WebSocket — through hook_context= and read them from state.data["hook_context"], not state.get, because State.get returns a deep copy and a deep-copied database client is not a database client. Reserved keys (step_count, token_usage, tool_call_counts, exit_reason, continue_run, stop_run, tools, hook_context, context_tokens) raise ValueError at construction if you redefine them in state_schema.

Cost control that survives a budget review

On a Haystack platform, LLM tokens dominate everything else. Compute is noise by comparison, the vector store is predictable, and the bill that gets escalated is always the model provider's. The useful framing is that you have four levers, and they work at different time scales.

Bound each run. max_agent_steps lowered from 100 to something you can justify. TokenBudgetHook(max_total_tokens=..., add_final_message=False), which ends the run with exit_reason="token_budget_exceeded". Per-run generation_kwargs to cap output tokens. These hard ceilings are the only thing between a prompt-injection loop and a five-figure day.

Shrink what you send. SlidingWindowCompactor is nearly free and drops old turns. ToolResultPruningCompactor(min_keep_steps=2, min_tokens=200) keeps recent tool output only. SummarizationCompactor costs tokens to save tokens, so give it a cheap model. ToolResultOffloadHook with an OffloadOverChars policy moves large results out of the message list; in 3.2 the tool_result_offloaded meta key became a list of references, which matters if you parse it.

Choose cheaper paths. FallbackChatGenerator fails over across providers or models, a reliability feature that doubles as a cost one if the fallback is cheaper. api_base_url points any OpenAI-compatible component at a self-hosted endpoint — vLLM, Ollama or a LiteLLM gateway — which is how teams in the Gulf and Egypt reconcile a token budget with data residency, since inference stays in a region they control. Embedding is the other half: batch document embedders at indexing time, and consider local Sentence Transformers through sentence-transformers-haystack, because embeddings are a volume game the local option wins early.

Spend nothing in CI. MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder arrived in 3.0 so tests need no API keys. Leave this on the table and you end up with either a test suite nobody runs or a test suite with a budget.

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import MockChatGenerator
from haystack.dataclasses import ChatMessage


def test_rag_pipeline_wires_up_and_runs_without_a_key():
    template = [ChatMessage.from_user("Answer: {{question}}")]
    pipe = (Pipeline()
            .add_components({"prompt_builder": ChatPromptBuilder(template=template),
                             "llm": MockChatGenerator()})
            .connect_many([("prompt_builder", "llm")]))

    result = pipe.run({"prompt_builder": {"question": "what is retrieval?"}})

    assert result["llm"]["replies"]

That test costs nothing and catches the two mistakes that break deploys: a connection that no longer type-checks, and a template variable that is now required.

Observability you can run a rotation on

Haystack's tracing is good and never automatic. auto_enable_tracing was removed in 3.0 and HAYSTACK_AUTO_TRACE_ENABLED has no effect, so a platform upgraded from 2.x loses its traces silently unless someone calls tracing.enable_tracing(...) or adds a connector component. The OpenTelemetry and Datadog tracers also moved out of core into opentelemetry-haystack and datadog-haystack.

from haystack import tracing
from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "rag-api"}))
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)

tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("rag-api")))

Treat span names as an API, because 3.0 changed them. One operation haystack.pipeline.run now covers sync and async, tagged with haystack.pipeline.execution_mode; the old haystack.async_pipeline.run is gone. Agent spans became one haystack.agent.step per iteration with haystack.agent.step.llm and haystack.agent.step.tool children, carrying no haystack.component.* tags — so a custom SpanHandler matching component_type == "ToolInvoker" matches nothing. 3.1 added haystack.agent.hook; 3.2 added haystack.chat_generator.run inside FallbackChatGenerator. Each is a dashboard that needs re-checking after an upgrade, which argues for keeping dashboards in version control.

The privacy switch deserves a paragraph. "Content tracing" — recording component inputs and outputs in spans — is off by default, correctly, because those spans would contain prompts, retrieved documents and whatever your users typed. Langfuse requires it, so enabling Langfuse means accepting that your prompts live in a tracing backend. Set HAYSTACK_CONTENT_TRACING_ENABLED=true before importing Haystack, and call tracer.flush() before the process exits or you lose the last spans of every run. For regulated data the defensible position is content tracing on in staging and off in production, with sampled redacted traces if you need production insight — OpenTelemetry with your own collector gives you the redaction hook.

Logs changed in your favour: importing Haystack no longer reconfigures the root logger or global structlog, so it cannot reformat your application's logs. configure_logging(logger_name="") restores the old global behaviour; configure_logging(propagate=False) fixes duplicated lines. Use JSON in production via HAYSTACK_LOGGING_USE_JSON=true — automatic when there is no TTY, but being explicit keeps the format the same when someone runs the container interactively. The default level is WARNING, so a quiet log is not evidence of a healthy pipeline.

Four signals belong on a Haystack dashboard: request latency by pipeline, exit_reason distribution per agent, tokens per request with a tenant dimension, and store query latency as a series separate from total latency. The last split tells you in thirty seconds whether a slow endpoint is the retriever or the model. Wire them into Prometheus and Grafana if that is your stack.

Upgrades and migrations

Haystack states plainly that it does not follow Semantic Versioning, and a platform owner has to internalise what that means. Major releases bring fundamental incompatible changes, happen at most once a year, and come with a migration guide. Minor releases can add or remove functionality, and the docs say directly that deprecated components are the most common breaking change shipped in a minor release. Patch releases are bug fixes only. The deprecation window is one minor version: a feature deprecated in X.Y raises FutureWarning and is removed in X.Y+1, which in practice is about a month.

That timeline turns FutureWarning into your early-warning system: surface warnings in CI output and treat a new one as a ticket with a one-month SLA.

The 2.x-to-3.x migration is the big one, and the order matters. Move to 2.31 first — it deprecated everything 3.0 removed, so a clean 2.31 run with no FutureWarning is your gate. Then the five changes that break real code. AsyncPipeline is gone; one Pipeline has run, run_async, run_async_generator and stream, and AsyncPipeline.run becomes asyncio.run(pipeline.run_async(...)). The legacy non-chat generators are gone — OpenAIGenerator, AzureOpenAIGenerator, HuggingFaceAPIGenerator, HuggingFaceLocalGenerator — replaced by Chat Generators that accept a plain str for messages and whose replies[0] is a ChatMessage read with .text and .meta, with no separate result["meta"] list. ToolInvoker is gone and the Agent runs tools itself, so tool_invoker_kwargs becomes tool_concurrency_limit. About thirty components moved into integration packages, so your image needs sentence-transformers-haystack, transformers-haystack, huggingface-api-haystack and friends, with imports under haystack_integrations. And haystack-experimental is no longer installed automatically — pip install haystack-experimental, pinned to 0.19.0.post1 per the migration guide, if you depend on it.

Then there are the four quieter ones that cause incidents rather than import errors. Template variables are required by default; pass required_variables=None to restore 2.x behaviour or a list to require only some. user_prompt and system_prompt can no longer be passed to Agent.run or LLM.run — set them at __init__, and build runtime prompts by passing ChatMessage.from_system(...) in messages. State is no longer injected into tools by name, so each tool must declare inputs_from_state={"state_key": "param_name"}. And the Document.id hash changed, so re-ingest anything with meta.

# the gate: 2.31 with zero FutureWarnings before you touch 3.x
pip install "haystack-ai==2.31.*"
python -W error::FutureWarning -m pytest

# then move, pinning the minor
pip install "haystack-ai==3.2.*" haystack-experimental==0.19.0.post1
python -c "from haystack.version import __version__; print(__version__)"   # -> 3.2.0

Looking forward, 3.3 is a release candidate and must not be deployed as stable, but two changes move your numbers: in-memory BM25 with BM25L or BM25Plus will return only documents containing at least one query term, so results can be fewer than top_k, and SentenceSplitter chunk boundaries change around quoted sentences. Both mean re-baselining evaluations and re-indexing — schedule that as part of the upgrade, with RAGAS or Haystack's own evaluators, not after someone reports a regression. One 3.1 trap to remember: PipelineSnapshot.pipeline_state.inputs changed shape to {component: {socket: [{"sender":..., "value":...}]}}, which breaks tooling built on snapshots.

Governance: the contract with consuming teams

A platform is a contract, and vague contracts produce the pager. Here is the contract I would write for a shared Haystack platform, stated as what each side owns.

You own the runtime: the image, the pins, the allowlist, the secret wiring, the tracing setup, the ingress, the quotas, the dashboards, the upgrade schedule. Consumers do not pin their own Haystack version, because minor releases remove things and you cannot support four runtimes.

They own the pipeline and the prompts, as a Python module in a repository you depend on or a pipeline_wrapper.py in a Hayhooks directory. Either way the artifact goes through review, using the checklist below.

Both of you own the budget, explicitly. Each pipeline declares a token budget and a step ceiling at onboarding, enforced by TokenBudgetHook and max_agent_steps rather than by goodwill, and a pipeline without declared limits does not get deployed. This is the policy that prevents the escalation where finance asks why the bill tripled and nobody can say which team caused it.

Three governance mechanisms pay for themselves. A component registry: a small internal package of approved components with your conventions baked in — idempotent warm_up, secrets resolved correctly, the scoped_filters helper, standard logging and tracing — plus the allowlist entry that makes them loadable; the deepset-ai/custom-component template repository is the right packaging starting point. Golden evaluations per pipeline, cheap enough for every pull request: the built-in evaluators (FaithfulnessEvaluator, ContextRelevanceEvaluator, LLMEvaluator, the DocumentRecall, MRR, MAP and NDCG evaluators, SASEvaluator) cover most needs, with Ragas and DeepEval integrations if you standardised on those. And a deprecation channel: one message per release telling consumers which FutureWarning applies to their pipeline and by when.

What governance should not do is prevent every mistake through policy. The filter_policy default, the reserved state keys and the required template variables are all better handled by a shared helper or a test than by a document nobody reads.

Incident playbooks

Four playbooks cover most of what you will be paged for, each starting with the signal rather than the hypothesis.

All requests to one pipeline fail with a 500. Read the inner error: PipelineRuntimeError wraps the real exception and names the component and type. Usual causes, in order: a dependency outage, a missing environment variable after a deployment change, a model the provider retired. A failed run attaches a snapshot to PipelineRuntimeError.pipeline_snapshot, the fastest way to see what inputs the failing component received. Snapshots reach disk only when HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED=true, under ~/.haystack/pipeline_snapshot, then /tmp/haystack/pipeline_snapshot, then ./.haystack/pipeline_snapshot, named {component}_{visit}_{YYYY_MM_DD_HH_MM_SS}.json. On always in staging, in production behind a flag — the snapshot contains inputs, which means user data.

Latency has doubled and nothing was deployed. Split total latency into store time and model time — this is why the dashboard keeps them separate. Model-side: provider status, then retry and timeout settings, because five retries at thirty seconds turns a degraded provider into a timeout storm. Store-side: index size, and whether someone started a backfill. If neither moved, look at agent step counts: one extra tool call per run looks exactly like a latency regression.

The token bill spiked. exit_reason distribution first: a jump in max_agent_steps means loops, a jump in length means context growth. Then tokens per request by tenant, to see whether it is one consumer or everyone. Mitigate with a tighter TokenBudgetHook and a lower step ceiling; fix whichever tool returns payloads that need offloading or pruning.

A pipeline stopped loading after a deploy. Nearly always DeserializationError, and the message distinguishes the cases: a module not on the allowlist, a blocked builtin, an unknown nested parameter, a ConditionalRouter or OutputAdapter needing unsafe=True, or _ToolsetWrapper from the removed + syntax. Resist resolving the incident with unsafe=True: it works, and you will never remove it.

For deep debugging outside production: include_outputs_from={...} returns intermediate outputs the pipeline otherwise discards, since it returns only the last output of components whose outputs are unconnected. LoggingTracer prints the trace inline. Breakpoint pauses at a named component and visit count, raising BreakpointException carrying component, inputs and results, and you resume from the snapshot:

import logging
from haystack import tracing
from haystack.tracing.logging_tracer import LoggingTracer
from haystack.dataclasses.breakpoints import Breakpoint
from haystack.core.errors import BreakpointException
from haystack.core.pipeline.breakpoint import load_pipeline_snapshot

logging.getLogger("haystack").setLevel(logging.DEBUG)   # default is WARNING
tracing.tracer.is_content_tracing_enabled = True
tracing.enable_tracing(LoggingTracer())

bp = Breakpoint(component_name="llm", visit_count=0, snapshot_file_path="/snaps")
try:
    pipeline.run(data, break_point=bp)
except BreakpointException as e:
    print(e.component, e.inputs)

pipeline.run(data={}, pipeline_snapshot=load_pipeline_snapshot("/snaps/llm_0_2026_10_03_09_12_44.json"))

Agent-level breakpoints were removed in 3.0 — AgentBreakpoint, ToolBreakpoint, AgentSnapshot and the break_point, snapshot and snapshot_callback parameters on Agent.run are gone — while pipeline-level Breakpoint and PipelineSnapshot remain. Rebuild an agent-breakpoint habit on hooks instead: a before_tool hook logging tool name and arguments gives most of the same visibility and works in production.

Where Haystack stops

Knowing a tool's edges is what makes a recommendation credible, so here are the places where I would reach for something else, or for something alongside.

You need durable, long-running orchestration. Pipelines run inside one process and finish. There is no persistent state machine, no durable retry across restarts, no approval step that can wait three days; breakpoints and snapshots are a debugging tool, not a workflow engine. If your flow spans days or must survive a deploy, put a real orchestrator above Haystack — Prefect, Dagster or Airflow for data work, or LangGraph if what you want is specifically a durable, checkpointed graph of agent steps — and let Haystack be the thing each step calls.

You want a managed service. Running it is your job; deepset's commercial Haystack Enterprise Platform is the managed option, and the comparison is build-versus-buy on exactly the work this guide describes.

You need a model gateway. Haystack can point any OpenAI-compatible component at a different base URL and FallbackChatGenerator handles failover, but it is not a gateway: there is no central key management, no per-team spend caps, no cross-team caching. If four teams call three providers, a gateway like LiteLLM sitting between Haystack and the providers is less work than building that inside your pipelines.

Your problem is the data pipeline, not the query path. Ingestion at scale — change capture, scheduling, backfills, incremental re-embedding — is a data-engineering problem. The converters, cleaners and splitters are good building blocks and DocumentPreprocessor packages them neatly, but the schedule, the retries and the lineage belong elsewhere.

You need model serving. Haystack calls models; it does not host them. vLLM or TGI sit behind api_base_url, and BentoML or KServe serves your own models alongside, with MLflow tracking what you trained.

One place where Haystack wins outright: composition is typed and checked before it runs. Connections are validated at connect() time, so a wiring mistake fails at import rather than in front of a user. For a platform with many contributors of varying experience, that is worth a great deal.

Running it as a platform for other teams

Everything above assembles into a small number of concrete platform artifacts. If you build these five, consuming teams can move quickly and you can sleep.

A base image. FROM deepset/hayhooks:v1.25.0 with your pins, your integrations, telemetry decided, JSON logging on, HAYHOOKS_SHOW_TRACEBACKS=false. Everyone builds from it: this is where version policy becomes real rather than documented.

A serving template. Either a FastAPI application that builds pipelines at startup, warms them up, closes them on shutdown and exposes run_async endpoints, or a Hayhooks pipelines directory. Hayhooks is the faster self-service path: a deployed pipeline is a directory with a PipelineWrapper(BasePipelineWrapper) class whose setup() builds the pipeline, whose run_api(...) type hints become the request schema, and whose optional run_chat_completion(self, model, messages, body) gives an OpenAI-compatible endpoint — which is how Open WebUI or the built-in Chainlit UI goes in front of a pipeline with no front-end code.

from pathlib import Path
from hayhooks import BasePipelineWrapper
from haystack import Pipeline


class PipelineWrapper(BasePipelineWrapper):
    def setup(self) -> None:
        yaml_path = Path(__file__).parent / "pipeline.yml"
        with yaml_path.open() as fp:
            self.pipeline = Pipeline.load(
                fp, allowed_modules=["haystack_platform.components.*"]
            )
        self.pipeline.warm_up()

    def run_api(self, question: str, tenant: str) -> str:
        result = self.pipeline.run({
            "retriever": {"query": question, "filters": scoped_filters(tenant, None)},
            "prompt_builder": {"question": question},
        })
        return result["llm"]["replies"][0].text

An ingress with authentication. Not optional, and the most commonly skipped step. Hayhooks has no authentication setting in its environment-variable reference, and HAYHOOKS_CORS_ALLOW_ORIGINS defaults to ["*"]. A Hayhooks pod with a public Service is an open LLM endpoint billed to you. Put it behind an ingress or API gateway doing TLS and authn, restrict CORS to the front ends you run, and keep HAYHOOKS_SHOW_TRACEBACKS false so stack traces are not part of your API. On Kubernetes that is a Deployment of N replicas behind a Service on port 1416, an Ingress with auth, secrets via secretKeyRef, and a readiness probe that fails until warm-up completes. The knobs worth knowing: HAYHOOKS_HOST, HAYHOOKS_PORT (1416), HAYHOOKS_PIPELINES_DIR (./pipelines, auto-deployed at startup), HAYHOOKS_STARTUP_DEPLOY_STRATEGY (parallel) with HAYHOOKS_STARTUP_DEPLOY_WORKERS (4), HAYHOOKS_STREAMING_COMPONENTS and HAYHOOKS_LOG_LEVEL. The MCP server runs separately on 1417 and A2A on 1418, each needing its own ingress and auth decision.

A paved path for tools. Most consumer requests are "make the agent call my service". Give them ComponentTool and PipelineTool for in-process work, @tool with Annotated parameters for plain functions, and MCPTool/MCPToolset from mcp-haystack for anything speaking MCP — with StreamableHttpServerInfo, since SSEServerInfo is deprecated along with SSE in the MCP spec. For multi-agent designs, AgentTool (3.1+) wraps a whole Agent as a tool and returns only its final reply.

A CLI and a deploy path. The Hayhooks CLI covers the verbs: hayhooks run, hayhooks status, hayhooks pipeline deploy-files -n NAME PATH, deploy-yaml -n NAME FILE, undeploy NAME, and hayhooks pipeline run NAME --param 'question="..."' to smoke-test. Wrap the deploy verbs in CI so nobody deploys from a laptop, and make -o/--overwrite an explicit, logged decision.

The review checklist

Use this on pull requests that touch a pipeline, and on design documents for new ones. It is deliberately ordered so the dangerous things come first.

A design document gets four extra questions, the ones expensive to change later. Is the pipeline code or configuration, and who may author it? Shared store with per-tenant filters, or one index per tenant? What is the token budget, in numbers? And what happens to an in-flight run when the pod is terminated — cancellation does not stop sync components, so any side-effecting tool must be idempotent.

The complete picture

Beginner taught the nouns: components with typed sockets, pipelines as graphs, document stores that are not components, ChatMessage as the currency of every LLM call. Mid taught prediction: data flows only along explicit connections, connect() fails at build time rather than run time, an Agent's loop terminates for one of a few named reasons. Senior adds a different kind of question, and all of them are about boundaries. The boundary between your process and a YAML file, policed by an allowlist that exists because that file is executable. The boundary between tenants, which Haystack does not draw for you and which FilterPolicy.REPLACE will quietly erase if you let a caller's filters through. The boundary between a bounded run and an unbounded bill, drawn with step ceilings, token budgets and compaction. The boundary between your service and its dependencies, where every real incident lives. And the boundary between you and the teams you serve, which is a written contract or it is a pager rotation.

The through-line is that Haystack gives you excellent primitives and very little opinion. required_variables="*", the deserialization allowlist, resources created in warm_up, tracing never auto-enabled, content tracing off by default — each 3.x default chose safety over convenience. Your job is to supply the opinions it withholds: which store, which models, which filters, which budget, which ingress, which version, and who may change each one.

Where the series leaves you

You can now design a Haystack platform rather than a Haystack script: draw the four-layer architecture and name the dependency behind every box; decide whether pipelines are code or configuration and defend it; state what the allowlist protects and what unsafe=True gives away; implement tenant isolation that survives a caller's own filters; bound an agent's cost three independent ways; enable tracing without leaking prompts; run a 2.x-to-3.x migration in the right order, including the re-index nobody expects; and hand a consuming team a base image, a template, a contract and a checklist instead of a conversation.

Where to go next depends on which boundary costs you most now. Retrieval quality: a vector store you own — Qdrant or pgvector — plus evaluation discipline from RAGAS. Observability: Langfuse for LLM traces, OpenTelemetry for everything else. Cost or provider sprawl: LiteLLM as a gateway, vLLM if self-hosting suits your region and volume. Orchestration beyond one process: LangGraph for durable agent graphs, Prefect for data flows. And if the comparison question is live where you work, the official "Migrating from LangGraph/LangChain to Haystack" guide is worth reading in both directions.

The interview and tips files at this level cover the same ground from two other angles: how to argue these decisions, and the traps that appear only once you run this for other people.

Sources

• Haystack documentation: Intro
• Migration guide
• Breaking change policy
• Pipelines
• Serialization
• Pipeline breakpoints
• Debugging pipelines
• Secret management
• Document store
• Metadata filtering
• Agent
• Hooks
• Human in the loop
• Compaction hook
• MCPTool
• Logging
• Tracing
• OpenTelemetry
• Langfuse
• Telemetry
• Hayhooks
• Deployment
• Kubernetes
• Haystack 3.2.0 release notes
• Haystack 3.0.0 release notes
• MIGRATION.md
• Hayhooks CLI commands
• Hayhooks environment variables

At this level the interview stops asking whether you can build a RAG pipeline. It asks whether you can be trusted with a shared one. The questions are system design, incident response, and the argument you make to a director who has just seen the model bill. Everything below assumes Haystack 3.2.0 and Hayhooks 1.25.0, and a few answers only make sense in 3.x — saying "AsyncPipeline" out loud in 2026 is a dating signal, so know that it was removed in 3.0.

How senior answers are judged

Three things separate a strong answer from a competent one. A strong answer names the failure mode before the feature: it says "the risk is a caller's run-time filters replacing my tenant filter, because filter_policy defaults to REPLACE", not "I'd use metadata filters". It states a default and then changes it, which proves you have read the code rather than a blog post: max_agent_steps is 100, tool_concurrency_limit is 4, OPENAI_MAX_RETRIES is 5, and none of those are the values you want in production. And it separates what Haystack does from what you must build, because Haystack is a library with no tenant model, no quotas, no auth and no scheduler, and pretending otherwise is the fastest way to lose the room.

System design: a shared RAG platform for four teams

"Design a Haystack platform that four product teams will use. Walk me through it."

Start with the layers, because it frames everything else. At the edge, an ingress or API gateway doing TLS, authentication and rate limiting — this is the only place tenant identity is established, and nothing downstream trusts a tenant ID from a request body. In the middle, a stateless serving process in N replicas, either FastAPI or Hayhooks, building pipelines once at startup, calling warm_up() at boot and close() on shutdown. Below that, the dependencies that actually generate your incidents: the document store, the model providers, the tool backends.

Then make the two decisions the interviewer is listening for.

The first is whether pipelines are code or configuration. Code means Python in a repository, reviewed and deployed as a unit. Configuration means YAML loaded with Pipeline.load, which lets teams ship prompt changes without a deploy — and which is a code-execution surface, because YAML names classes to import and instantiate. Say that plainly, then say what protects you: the 3.x trusted-module allowlist (haystack, haystack_integrations, haystack_experimental, builtins, typing, collections), blocked builtins like eval and open, and nested init_parameters validated against the parent's __init__. Extend it narrowly with allowed_modules=["mypkg.*"] rather than globally, and refuse unsafe=True outside a process that handles nothing but your own CI artifacts.

The second is the unit of deployment. One service per pipeline gives isolation and clean cost attribution; one Hayhooks instance hosting many gives self-service at the price of shared fate — a local embedder in one team's pipeline can OOM the pod serving everyone. A defensible answer is Hayhooks per domain, with the note that Hayhooks has no authentication setting and defaults HAYHOOKS_CORS_ALLOW_ORIGINS to ["*"], so it never faces the internet directly.

Finish with what you make mandatory at onboarding: a declared token budget enforced by TokenBudgetHook, a max_agent_steps ceiling well below 100, a real document store rather than InMemoryDocumentStore, and tracing enabled explicitly, since 3.0 removed auto_enable_tracing and HAYSTACK_AUTO_TRACE_ENABLED has no effect.

System design: multi-tenant retrieval

"Ten tenants share one index. How do you guarantee one cannot read another's documents?"

The honest opening is that Haystack has no tenant model, so the guarantee is yours to build. Then give the two architectures. A separate index or collection per tenant gives real isolation, because a filter bug cannot cross a boundary that is physical; it costs more and scales to fewer tenants. A shared index with per-tenant metadata filters is cheaper and scales further, and one wrong filter is a breach.

The detail that marks a senior answer is filter_policy. Retrievers default to FilterPolicy.REPLACE, which means filters passed at run time replace those set at init. If the tenant filter lives in the retriever's filters argument and a caller supplies their own at run time, the tenant filter silently disappears. The fix is either FilterPolicy.MERGE or, better, a single helper at the edge that composes the authenticated tenant filter with any caller filter under an AND and is the only thing allowed to reach the retriever. Add that filter support varies per store — Chroma has no NOT but adds contains — and that an unsupported operator raises FilterError rather than matching everything, so the test must run against the store you actually deploy.

Close with the test you would write: call the endpoint as tenant A with a filter crafted to match tenant B's documents, assert zero results, and run it in CI. Interviewers remember candidates who propose the test.

System design: agents that cannot bankrupt you

"A team wants an agent with six tools, one of which writes to a customer database. What do you require before it ships?"

Four requirements, each tied to a mechanism.

Bounded steps. max_agent_steps lowered from its default of 100 to something near observed p95. Log exit_reason on every run — its values are "text", "length", "content_filter", a tool name, "max_agent_steps", or a custom stop_run reason — and alert on a rising share of max_agent_steps, which usually means a tool is returning an error string the model cannot act on. Mention why: raise_on_tool_invocation_failure defaults to False, so tool errors are fed back to the LLM rather than raised.

A token ceiling. TokenBudgetHook(max_total_tokens=...) ends the run with exit_reason="token_budget_exceeded". Note that it is experimental in 3.2, so you pin the version and read upgrade notes.

Context management. CompactionHook under before_llm with SlidingWindowCompactor or ToolResultPruningCompactor(min_keep_steps=2, min_tokens=200), or ToolResultOffloadHook under after_tool with an OffloadOverChars policy for genuinely large results.

Human approval on the write. ConfirmationHook under before_tool, with a BlockingConfirmationStrategy and AlwaysAskPolicy for the database tool. Then deliver the sharp edge, because this is the detail that wins the question: confirmation strategies see only the arguments the model supplied, not values injected from State via inputs_from_state. If the destructive parameter comes from State, the human approving the call is not seeing it, and your approval screen is lying to the approver.

Incident response

"Every request to the RAG endpoint returns a 500. Go."

Read the inner error first. PipelineRuntimeError wraps the real exception and its message gives the failing component's name and type, so the first ninety seconds are reading, not hypothesising. The usual causes in order are a dependency outage, a missing environment variable after a deployment change, and a retired model. Since 3.0, resources are created in warm_up(), so a missing key raises ValueError: None of the following authentication environment variables are set: ('OPENAI_API_KEY',) at warm-up or first run rather than at construction — which is why you call warm_up() at startup and wire it to the readiness probe. For the inputs the component actually received, use PipelineRuntimeError.pipeline_snapshot, remembering that snapshots reach disk only with HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED=true and that they contain user data.

"Latency doubled and nothing was deployed."

Split latency into store time and model time, which is why those are separate series on the dashboard. Model side: provider status, then timeouts and retries, because OPENAI_MAX_RETRIES=5 at OPENAI_TIMEOUT=30 is a 150-second worst case per call and turns a degraded provider into a timeout storm. Store side: index size and any running backfill. If neither moved, check agent step counts — one extra tool call per run looks exactly like a latency regression. One more: a sync streaming_callback passed to run_async only logs a warning while blocking the event loop, so it presents as tail latency on unrelated endpoints.

"A pipeline stopped loading after a deploy."

Almost certainly DeserializationError, and the message distinguishes the cases: a module not on the allowlist, a blocked builtin, an unknown nested parameter (often leftover tool_invoker_kwargs or confirmation_strategies, both removed in 3.0), a ConditionalRouter or OutputAdapter needing unsafe=True, or a _ToolsetWrapper left over from the Toolset + operator removed in 3.2. Say explicitly that you will not resolve the incident with unsafe=True, because it works and then never gets removed.

"Documents are duplicated since the upgrade."

The Document.id hash changed in 3.0 — canonical key-sorted JSON of meta, with non-JSON values through str() rather than repr() — so any document with non-empty meta gets a different auto-ID than in 2.x, and DuplicatePolicy.SKIP or FAIL dedup against 2.x data stops working. Re-ingest, or pass explicit IDs and stop depending on the hash.

System design: migrate 2.x to 3.x without a freeze

"We are on Haystack 2.28 in production and cannot stop shipping. Plan the upgrade."

Give the order, because the order is the answer. First go to 2.31, the last 2.x release, which deprecated everything 3.0 removed. Run the test suite with FutureWarning promoted to an error, and treat a clean run as the gate. That step is cheap, reversible and does the discovery for you.

Then the import-level work, which is mechanical and reviewable: AsyncPipeline is gone, replaced by one Pipeline with run, run_async, run_async_generator and stream, so the old AsyncPipeline.run becomes asyncio.run(pipeline.run_async(...)). The legacy non-chat generators are gone, so OpenAIGenerator and friends become Chat Generators, which accept a plain str for messages and whose replies[0] is a ChatMessage read with .text and .meta — there is no separate result["meta"] list any more. ToolInvoker is gone and the Agent runs tools itself, so tool_invoker_kwargs becomes tool_concurrency_limit. About thirty components moved into *-haystack integration packages, so the image gains sentence-transformers-haystack, transformers-haystack and similar, with imports under haystack_integrations. And haystack-experimental is no longer installed automatically.

Then the two items that are not code changes, and this is what separates a plan from a list. Re-ingest any documents with non-empty meta, because the Document.id hash changed. And re-check the observability stack, because span names changed and tracing is no longer auto-enabled, so dashboards and alerts built on haystack.async_pipeline.run or on ToolInvoker spans quietly show nothing.

Then the runtime behaviours that bite later. Template variables are now required by default (required_variables="*"), tools read State only through a declared inputs_from_state, user_prompt and system_prompt can no longer be passed to Agent.run, and from 3.1 the Agent warms up hooks, tools and toolsets before every run, so a non-idempotent warm_up() becomes a per-request cost.

To ship without a freeze, run 2.31 and 3.2 behind the same interface for one release — the pipelines are built in code at startup, so a feature flag choosing which module builds them is enough — and cut over per pipeline rather than all at once. Say that the re-index is the long pole and schedule it first.

Arguing trade-offs to leadership

"Convince me we should not just use the managed platform."

Do not argue from preference. Lay out what self-hosting means in work: an image and a pinning policy, an ingress with auth, secret wiring, tracing, dashboards, the upgrade schedule, and an on-call rotation. Then name the reasons to own it anyway — data residency, which is the decisive one for Gulf and Egyptian employers with contractual language about where inference and prompts may go, and the ability to put inference behind api_base_url on infrastructure in a region you control. deepset's Haystack Enterprise Platform is the managed option and the comparison is an ordinary build-versus-buy. Senior credibility here comes from costing your own option honestly.

"Why did our bill triple?"

Answer with a mechanism and a control, not an apology. Show the exit_reason chart: a spike in max_agent_steps means loops, a spike in length means context growth. Show tokens per request by tenant, built from the Agent's token_usage output, which is the artifact that makes this question answerable at all — and say that you emit it with a tenant ID precisely so this conversation takes minutes. Then the controls, in the order you would apply them: a tighter budget hook as configuration today, compaction or offloading for the tool returning large payloads this week, and a cheaper model or a self-hosted endpoint behind api_base_url if the volume justifies it.

"Why do we pin a minor version?"

Because Haystack explicitly does not follow Semantic Versioning. Minor releases can add or remove functionality, and the docs state that deprecated components are the most common breaking change in a minor release; a feature deprecated in X.Y raises FutureWarning and is gone in X.Y+1, roughly a month later. So you pin haystack-ai==3.2.*, pin each *-haystack integration separately because they version independently, run the test suite with FutureWarning as an error, and treat each new warning as a ticket with a one-month deadline. That is a governance answer, and it is the one platform leadership actually wants.

"Should consuming teams be allowed to deploy their own YAML?"

Frame it as a trust decision rather than a tooling one. If the YAML's authors could already merge to your service, YAML adds velocity at no new risk. If they could not, YAML hands them code execution in your process, and the allowlist narrows that rather than removing it. The middle path is a review gate on the pipelines directory plus a narrow allowlist for your own component package, which is exactly how you would treat any other privileged artifact.

Questions to have an answer ready for

• "What is a Document Store in Haystack?" A database interface, not a component — no run() — with four protocol methods: count_documents, filter_documents, write_documents, delete_documents. Components hold one.
• "Why does connect() fail before the pipeline runs?" Connections are type-checked at connect time, so a wiring mistake fails at import rather than in front of a user. For a platform with many contributors that property is worth a lot.
• "When is InMemoryDocumentStore acceptable?" Tests, notebooks, single-process prototypes. It is process-global per index by default (shared=True), so a "per-request" store is not request-scoped and memory grows for the worker's lifetime.
• "What does cancellation actually cancel?" Native-async components only. Sync components run in worker threads and their side effects complete after the client is gone, so side-effecting tools must be idempotent.
• "How do you give each tenant different tools?" Agent.run(tools=[...]) at run time, as objects or names, on one shared Agent. In 3.2 combine toolsets with tools=[a, b], since the + operator was removed.
• "Where does request-scoped state go?" hook_context= on the run call, read from state.data["hook_context"] rather than state.get, which returns a deep copy.
• "What changed about tracing in 3.0?" It is never automatic, the OTel and Datadog tracers moved to separate packages, haystack.async_pipeline.run is gone in favour of haystack.pipeline.run with an execution_mode tag, and agent spans became haystack.agent.step with .llm and .tool children carrying no haystack.component.* tags.
• "How do you keep CI free?" MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder, which exist in 3.0 for exactly this.

A sixty-second self-test

Answer out loud, in one sentence each. What are the four layers of a Haystack platform, and which one holds authentication? What does the deserialization allowlist permit by default, and what do eval and open do inside it? Which default silently erases a tenant filter? What are the three independent ways to bound an agent's cost? Why does a missing API key now fail at warm-up instead of construction? What does exit_reason tell you during a cost incident? Which two things must you redo when upgrading 2.x to 3.x besides changing imports? And what is the one switch that disables every deserialization safety check, in both its forms?

If any answer took longer than a sentence, that is the section of the detailed guide to reread before the interview. The tips file at this level covers the traps behind these answers — the ones that only show up once the platform is real.

These are the traps that do not appear in a tutorial, in a code review, or in staging. They appear the first week a second team depends on your Haystack service, or the first month after an upgrade, or the first time a tool returns ten times more text than it did yesterday. Everything here is against Haystack 3.2.0 and Hayhooks 1.25.0.

Traps that only appear at scale

The default that erases tenant isolation. Retrievers take filter_policy=FilterPolicy.REPLACE, so filters passed at run time replace the ones set at init. A tenant filter configured on the component disappears the moment a caller passes their own filters. Nothing errors, nothing logs, and the results look plausible. Treat the retriever's own filters argument as untrustworthy for security purposes, compose the tenant filter at the edge from an authenticated identity, and keep a CI test that queries as tenant A with a filter crafted for tenant B's data and asserts zero rows.

InMemoryDocumentStore escaping into production. It lives in process memory and is not shared across replicas, so two pods answer the same question from two corpora. The subtler half is that it is process-global per index by default (shared=True), so a store you create inside a request handler is not request-scoped — it accumulates for the lifetime of the worker and presents as a slow memory leak. Pass shared=False when you genuinely want per-request, and add a grep for the class name to your review checklist.

Warm-up cost paid on every request. Since 3.1 the Agent warms up its hooks, tools and toolsets before every run. A tool whose warm_up() loads a model or opens a connection pool without a guard turns a one-off startup cost into per-request latency, and the symptom is a latency floor nobody can explain. Make if self._client is not None: return the first line of every warm_up() you write, and put it in the base class so nobody has to remember.

Retry storms from inherited defaults. OPENAI_MAX_RETRIES is 5 and OPENAI_TIMEOUT is 30 seconds, which is a 150-second worst case per call. Under provider degradation your own service becomes the amplifier, and your gateway's timeout decides what the user sees. Choose both numbers deliberately and write the reasoning next to them.

Sync callbacks in async paths. A synchronous streaming_callback passed to run_async only logs a warning while it blocks the event loop. It is invisible at staging volumes and shows up in production as tail latency on endpoints that have nothing to do with streaming. Same family of problem: Pipeline.run is synchronous and, since 3.0, runs components sequentially with no concurrency_limit, so it blocks a worker thread for the whole pipeline.

Cancellation that does not cancel. A dropped client interrupts native-async components only. Sync components keep running in worker threads and their side effects complete, so a payment tool still charges the card. Every side-effecting tool must be idempotent, and that is a design requirement rather than a nice-to-have.

Context growth that is intermittent by nature. An agent with tools that return variable-size payloads will cross the provider's context limit on the day a tool returns more than usual. You will see exit_reason="length" or a provider error, in a pattern that defeats reproduction. Configure compaction or offloading before you observe the failure, not after.

Loop ceilings used as business logic. max_runs_per_component is 100 and max_agent_steps is 100. Hitting either is not a normal outcome, but with defaults that high you will absorb a great deal of cost before anything complains. Lower both and alert on the ceiling.

A dashboard that silently empties after an upgrade. 3.0 renamed and restructured spans: haystack.async_pipeline.run is gone, haystack.pipeline.run carries an execution_mode tag, and agent spans became haystack.agent.step with .llm and .tool children that carry no haystack.component.* tags. A panel keyed on the old names shows no data rather than an error, which is the worst possible failure for an observability tool.

Hayhooks exposed without auth. Hayhooks has no authentication setting in its environment-variable reference, and HAYHOOKS_CORS_ALLOW_ORIGINS defaults to ["*"]. A Service with a public address is an open LLM endpoint billed to you. The MCP server on 1417 and A2A on 1418 are separate listeners and need their own decisions.

Pipeline diagrams leaving the building. show() and draw() send the graph to the public mermaid.ink service by default. Component names, and sometimes prompts, are not always things you want posted to a third party. Self-host the renderer and pass server_url.

Platform-ownership habits

Own the image, not the instructions. A base image with your pins, your integrations, telemetry decided and JSON logging on is the only version policy that actually holds. Documentation asking teams to pin correctly does not survive contact with a deadline. Pin haystack-ai==3.2.* and each *-haystack integration separately, because integrations version independently.

Treat FutureWarning as a dated ticket. The release policy is explicit: Haystack does not follow SemVer, minor releases may remove functionality, and a feature deprecated in X.Y is gone in X.Y+1 — about a month. Run the test suite with -W error::FutureWarning so the warning cannot be scrolled past, and send one message per release telling each consuming team which warning is theirs.

Make the safe thing the easy thing. The recurring Haystack mistakes are all preventable by a shared helper rather than a policy: a scoped_filters function that is the only way to build a retriever filter, a tool base class with an idempotent warm_up, a loader that passes a narrow allowed_modules list, a logging and tracing setup imported rather than copied. A small internal package beats a long wiki page every time, and the deepset-ai/custom-component template is the right starting point for packaging it.

Keep two numbers per pipeline. A token budget and a step ceiling, declared at onboarding and enforced by TokenBudgetHook and max_agent_steps. A pipeline without both does not deploy. This single rule prevents the escalation where the bill tripled and nobody can say whose pipeline did it.

Emit exit_reason and token_usage on every agent run, with the tenant ID attached. These two outputs are what turn a cost incident from an investigation into a query, and they cost nothing to log.

Decide content tracing per environment, in the manifest. Recording component inputs and outputs is off by default because those spans contain prompts, retrieved documents and user data. Langfuse requires it. On in staging and off in production, with sampled redacted traces where you need production insight, is a position you can defend to a security reviewer — and HAYSTACK_CONTENT_TRACING_ENABLED must be set before Haystack is imported, with tracer.flush() before exit or you lose the last spans of every run.

Call warm_up() at startup and wire it to readiness. Since 3.0 resources are built at warm-up, so a missing key fails there rather than at construction. Warming up explicitly converts a first-user failure into a deploy failure, which is the cheapest trade available to you.

Keep the free test path free. MockChatGenerator, MockTextEmbedder and MockDocumentEmbedder exist so CI needs no API keys. A platform whose tests cost money ends up with tests nobody runs.

Write the runbook before the incident. Four pages cover most pages received: all requests failing, latency doubled, bill spiked, pipeline stopped loading. Each one needs your dashboard links in it, not generic advice.

Review checklist for a pull request

Read in this order, because the first three can be security incidents and the rest are only outages.

1. Loading. No unsafe=True, no HAYSTACK_UNSAFE_DESERIALIZATION. Any allowlist extension is as narrow as the package it needs.
2. Secrets. Secret.from_env_var only; Secret.from_token cannot be serialized and has no place outside a test. Nothing resolved in __init__. Running dumps() yields variable names, never values.
3. Tenancy. The filter reaching the retriever is composed at the edge from an authenticated identity. filter_policy is set explicitly. A cross-tenant test exists.
4. Bounds. max_agent_steps and max_runs_per_component lowered from their defaults, with a declared token budget.
5. Context. Compaction or offloading configured wherever tools can return large payloads.
6. Lifecycle. Pipeline built once at startup, warm_up() explicit, close() on shutdown, every custom warm_up idempotent.
7. Async. run_async or stream in async endpoints, no sync streaming_callback in an async path, concurrency_limit chosen rather than inherited.
8. Store. A real store integration; any InMemoryDocumentStore is a test or has a written justification and an explicit shared.
9. State. No reserved keys in state_schema; request-scoped objects read from state.data["hook_context"], not state.get.
10. Observability. Tracing explicitly enabled, content tracing decided, JSON logs, exit_reason logged.
11. Tests. Mock generators so CI needs no keys, and a golden evaluation that runs on every pull request.
12. Versions. Everything pinned, and no new FutureWarning in the test output.

Review checklist for a design document

A design document is the cheapest place to fix the decisions that are expensive later, so ask these six and insist on answers in numbers where numbers apply.

Are pipelines code or configuration, and who may author them? If the authors could not already merge to your service, YAML is a privileged artifact and needs a review gate; the allowlist narrows that risk rather than removing it.

Is the store shared with per-tenant filters, or one index per tenant? Name the blast radius of a filter bug under the chosen option, and say which you are accepting.

What is the token budget, in numbers, and what happens when it is hit? exit_reason="token_budget_exceeded" returned to the caller is a design decision, not an error — decide what the user sees.

What happens to an in-flight run when the pod is terminated? Cancellation does not stop sync components, so list the side-effecting tools and state how each is made idempotent.

Which dependency is expected to break first, and what does the service do then? A FallbackChatGenerator across models, a degraded retrieval-only answer, or an honest error are all acceptable; having no answer is not.

How is this upgraded? Given a release policy that allows removals in minor versions, the document should name who watches release notes, how FutureWarning reaches the team, and what re-indexing or re-baselining the next upgrade is expected to need — 3.3, once final, changes in-memory BM25 results and sentence-split boundaries, which moves retrieval numbers.

Operational recipes

Find out, today, whether your traces are real. After a 2.x upgrade the most common silent loss is tracing, because 3.0 removed auto_enable_tracing and HAYSTACK_AUTO_TRACE_ENABLED now does nothing. Grep the service for enable_tracing or a *Connector component. If neither appears, you have no traces, regardless of what is installed. The same pass should search your dashboards for haystack.async_pipeline.run and for any SpanHandler matching component_type == "ToolInvoker", since ToolInvoker no longer exists.

Audit secrets in ten minutes. Call dumps() on every pipeline you serve and read the output. Env-var secrets serialize as the variable name with type: env_var, so anything resembling a key is a finding. A token secret cannot serialize at all and raises Cannot serialize token-based secret. Use an alternative secret type like environment variables., which makes the audit self-enforcing once you add a CI check that fails on Secret.from_token outside tests.

Prove the tenant boundary rather than asserting it. Write one test that queries as tenant A using a filter crafted to match tenant B's documents and asserts an empty result, and run it against the store you actually deploy, because operator support varies — Chroma has no NOT but adds contains, and an unsupported operator raises FilterError. A passing test against InMemoryDocumentStore proves nothing about Qdrant.

Make an agent's cost legible before it matters. Add exit_reason, step_count, token_usage and tool_call_counts to the structured log line for every run, with the tenant ID. Chart exit_reason by value. A flat, boring chart dominated by "text" is a healthy agent; a rising share of max_agent_steps means a tool is handing the model something it cannot act on, which happens quietly because raise_on_tool_invocation_failure defaults to False.

Set up a snapshot path you can use under pressure. Breakpoints and failure snapshots are written to disk only when HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED=true, into ~/.haystack/pipeline_snapshot, then /tmp/haystack/pipeline_snapshot, then ./.haystack/pipeline_snapshot, named {component}_{visit}_{YYYY_MM_DD_HH_MM_SS}.json. Leave it on in staging permanently. In production put it behind a flag you can flip during an incident, and document that the files contain component inputs, which means user data with a retention obligation attached.

Replace agent breakpoints with a hook. 3.0 removed AgentBreakpoint, ToolBreakpoint, AgentSnapshot and the break_point, snapshot and snapshot_callback parameters on Agent.run, while keeping pipeline-level Breakpoint and PipelineSnapshot. A before_tool hook that logs tool names and argument keys — not values — gives most of the same visibility, works in production, and does not turn your logs into a data-protection problem.

Keep a local renderer for diagrams. Run the mermaid.ink image yourself and pass server_url to draw, remembering that draw and show are keyword-only in 3.2, so older positional examples fail. Put the working invocation in your internal docs; otherwise everyone rediscovers the same two errors.

Rehearse the dependency failure. Point a staging pipeline at a model endpoint that returns errors and watch what your service does with five retries at a thirty-second timeout. Then set the numbers you actually want. Do the same with the store: block it at the network level and confirm the error surfaces as a clean failure rather than a hung request.

Onboard a team end to end, and write down every question they ask you. Each question is a gap in the paved path — a missing template, a missing helper, a default you never documented. Two onboardings is usually enough to turn a collection of scripts into a platform, and nothing else tells you as reliably whether you have one.

The habit that matters most

Everything above reduces to one practice: write down the default, then write down your value. Haystack's defaults are deliberately permissive in exactly the places a platform must be strict — one hundred agent steps, one hundred component runs, five retries, thirty-second timeouts, FilterPolicy.REPLACE, shared=True, CORS *, four concurrent tool calls. None of them are wrong for a library. All of them are wrong for a platform serving four teams. Keeping a short table of "default versus ours, and why" in the repository is the single highest-leverage artifact a Haystack platform owner can maintain, because it is simultaneously the onboarding document, the review checklist and the first thing you read during an incident.