The Complete LlamaIndex Guide

This is part one of three. It covers everything you need to build a working question-answering application over your own documents, and a first agent that can call your own Python functions. By the end you can install LlamaIndex correctly, load a folder of files, build an index, ask questions and get answers grounded in those files, save the index so you never pay to rebuild it twice, swap OpenAI for a model running on your own laptop, and read the error messages the framework throws at you. Mid-level and Senior take the same ideas into production; nothing here is wasted.

Every section ends with a Try it task. Do them while you read. Retrieval is one of those subjects that feels obvious on the page and surprises you the moment you run it on real documents, so the sooner you see your own output the better.

What LlamaIndex is, and the problem it solves

LlamaIndex is a Python framework for building applications that let a large language model answer questions about your data. The official description is "a framework for building LLM-powered applications over your data". That phrase is doing a lot of work, so unpack it.

A language model such as GPT-4o or Claude knows what was in its training data and nothing else. It has never seen your company's HR handbook, the 400 PDFs your compliance team keeps on a shared drive, your product's API documentation, or the Arabic-language policy circulars your bank published last quarter. Ask it a question about any of those and one of two things happens: it tells you it does not know, or — worse, and far more common — it produces a confident, fluent, completely invented answer. The industry word for that is hallucination, and for an internal tool it is fatal. A chatbot that is right 85% of the time and gives no hint about which 15% is wrong is a liability, not a product.

There are two ways to fix this. One is to train the model on your data, which is expensive, slow, has to be repeated every time a document changes, and still does not give you a citation you can show a reviewer. The other is to look the answer up at the moment the question arrives, and hand the model the relevant passages along with the question. The second approach is called RAG, retrieval-augmented generation. The LlamaIndex docs define it as a technique that "allows LLMs to answer questions about your private data by providing it to the LLM at query time, rather than training the LLM on your data", sending "only the relevant parts along with your query".

That last clause is the engineering problem in a nutshell. You cannot paste 400 PDFs into a prompt; models have a context window, and even when the window is large, cost and latency scale with every token you send, and accuracy degrades when the useful sentence is buried in a hundred pages of noise. So you need machinery that splits documents into pieces, works out which pieces are relevant to a question, and sends only those. LlamaIndex is that machinery, plus a consistent interface over the dozens of model providers, vector databases and file formats you might plug into it.

What came before, and why this shape won

Before frameworks like this existed, every team built the same pipeline by hand. You wrote a loop over a directory, picked a PDF library, invented a chunking rule, called an embedding API, pushed the vectors into whatever database you had, wrote a similarity query, and glued a prompt template on the end. It worked, and it took two weeks, and then you discovered your chunking cut tables in half, and the rewrite took another week.

LlamaIndex, which started life in 2022 as "GPT Index", packaged that pipeline. Its sibling frameworks — most obviously LangChain — made similar bets. The practical difference in 2026 is one of centre of gravity: LangChain grew out of chaining prompts and generalised towards agents, while LlamaIndex grew out of indexing documents and generalised towards agents from the data side. Both arrive in roughly the same place now, and the choice between them is usually about which abstractions you find clearer rather than what is possible.

What you get by using a framework instead of hand-rolling is substitution. Every component is behind an interface, so changing your embedding model from OpenAI's hosted one to a local HuggingFace model is one line, and moving from an in-memory store to Qdrant or pgvector is two. That matters enormously for employers in the Gulf and in Egypt, where "the documents may not leave the country" is a routine constraint: the same application code can run against a hosted model for a prototype and against a self-hosted model in an on-premise or in-region deployment for production, because the retrieval layer does not care.

The mental model: six nouns

Almost everything in LlamaIndex is one of six things. Learn these names properly now and the API documentation stops being a maze.

A Document is a container for one whole source: a file, a web page, a database row. It holds text, a dictionary of metadata, and an id (id_, also exposed as doc_id). When you load a folder of twenty PDFs you get Documents back, one per page or per file depending on the reader.

A Node is a chunk of a Document. The docs call it "a lightweight abstraction over text strings that keep track of metadata and relationships". The common subclass is TextNode; there are also ImageNode and IndexNode. A Node inherits its parent's metadata and keeps a pointer home in node.ref_doc_id. Nodes, not Documents, are what actually gets embedded, retrieved and shown to the model. This is the single most important distinction in the framework: you load Documents, but you retrieve Nodes.

A Reader, also called a data connector, turns a source into Documents. The built-in one for local files is SimpleDirectoryReader. Hundreds more — Notion, Google Drive, S3, SharePoint, web pages — ship as separate packages.

A node parser (equivalently a text splitter, and more generally a transformation) turns Documents into Nodes. The default is SentenceSplitter, which tries to break on sentence boundaries. Others include TokenTextSplitter, SemanticSplitterNodeParser, CodeSplitter and MarkdownElementNodeParser.

An Index is a data structure over Nodes that serves retrieval. VectorStoreIndex is the default and the only one you need at this level: it embeds each Node into a vector and finds matches by similarity. Others exist for other shapes of question — SummaryIndex, DocumentSummaryIndex, PropertyGraphIndex, KeywordTableIndex, TreeIndex.

A query engine is the end-to-end single-turn flow: take a question, retrieve Nodes, optionally post-process them, and synthesise an answer. You get one with index.as_query_engine() and call it with .query(). Its stateful multi-turn cousin is the chat engine, index.as_chat_engine().

Three more terms complete the picture. A retriever is just the retrieval half of a query engine, returning NodeWithScore objects, obtained with index.as_retriever(). A node postprocessor filters or re-ranks retrieved Nodes before synthesis, for example SimilarityPostprocessor(similarity_cutoff=0.75). A response synthesizer turns Nodes plus the question into prose; its default mode for query engines is compact, with refine and tree_summarize as the main alternatives.

The second half of the framework

Everything above is the data layer, which lives in llama_index.core. There is a second, independent half: Workflows, shipped as llama-index-workflows and re-exported as llama_index.core.workflow. A Workflow is "an event-driven, step-based way to control the execution flow of an application". You will not write one by hand in this guide, but you need to know it exists for one reason: every agent in LlamaIndex is a Workflow underneath. That is why agents are asynchronous, and why their errors mention workflow concepts like timeouts and iteration limits.

An agent is, in the docs' words, "a specific system that uses an LLM, memory, and tools, to handle inputs from outside users". A tool is a callable plus metadata — a name, a description, a parameter schema — that the agent is allowed to invoke. The loop is simple: the agent sends the user's message plus the tool schemas to the model; the model either answers or asks for one or more tool calls; your code runs them and appends the results; repeat until the model answers. The classes you will meet are FunctionAgent (uses the provider's native tool-calling), ReActAgent (uses Thought/Action/Observation prompting, so it works with models that have no tool-calling API), CodeActAgent (writes and runs code) and AgentWorkflow (several agents handing off to each other).

Installing it, and the trap in the install docs

LlamaIndex needs Python 3.10 to 3.14. Python 3.9 was dropped during the 0.14 line and will not work. Always install into a virtual environment; the package pulls in a lot of transitive dependencies and you do not want them in your system Python.

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install llama-index llama-index-readers-file
export OPENAI_API_KEY="sk-..."

On Windows PowerShell:

py -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -U pip
pip install llama-index llama-index-readers-file
$env:OPENAI_API_KEY = "sk-..."

In cmd.exe, activate with .venv\Scripts\activate.bat and set the key with set OPENAI_API_KEY=sk-.... To make it survive a new shell, use setx OPENAI_API_KEY "sk-...", which affects future shells only. If you use uv, the equivalent is uv venv && source .venv/bin/activate && uv pip install llama-index llama-index-readers-file.

What the llama-index bundle actually contains as of 0.14.25 is llama-index-core, llama-index-embeddings-openai, llama-index-llms-openai and nltk. That tells you something useful: the bundle is opinionated towards OpenAI. If you are not using OpenAI, do not install the bundle at all.

Verifying the install

python -c "import llama_index.core as c; print(c.__version__)"
pip show llama-index-core llama-index-workflows
python -c "from llama_index.core.agent.workflow import FunctionAgent; print('ok')"

The first command should print 0.14.25. The third is worth running because it is the import that breaks when you have followed an old tutorial: if it succeeds, your installation has the modern, workflow-based agents.

On Linux and macOS, pip list | grep -i llama shows every LlamaIndex package you have; on Windows use pip list | findstr llama.

Installing without OpenAI

If you have no OpenAI account, or your employer will not let documents leave the building, run both models locally. You need two of them: one LLM to write answers and one embedding model to turn text into vectors.

pip install llama-index-core llama-index-readers-file llama-index-llms-ollama llama-index-embeddings-huggingface
ollama pull llama3.1

from llama_index.core import Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.llm = Ollama(model="llama3.1", request_timeout=360.0, context_window=8000)
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")

The docs note that running llama3.1 8B locally wants roughly 32 GB of RAM. The embedding model is far lighter and downloads once to a local cache. Ollama has its own guide in this catalogue if you want the model-serving side in depth.

Note the request_timeout=360.0. Local models are slow, and the default timeouts assume a hosted API. Note also that you must set both Settings.llm and Settings.embed_model. Setting only one is the most frequent beginner mistake with local setups, and the next section explains exactly what it costs you.

Your first application, in five lines

Make a folder, put something in it, and point LlamaIndex at it.

mkdir -p rag-demo/data && cd rag-demo
curl -o data/essay.txt https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt

Any text file will do; use your own if you prefer. Now the program:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents, show_progress=True)
query_engine = index.as_query_engine()
print(query_engine.query("What did the author work on before college?"))

Run it with python starter.py. Expect a progress bar for the embedding step and then a paragraph of prose. This is a complete RAG application, and it is worth being precise about what those five lines did, because each one maps onto a noun from the previous section.

1. LoadSimpleDirectoryReader("data").load_data() walked the data folder, read each file with a reader chosen by extension, and returned a list of Document objects. By default it does not recurse into subfolders.
2. Splitfrom_documents applied the default transformations. SentenceSplitter cut each Document into Nodes of about 1024 tokens with 200 tokens of overlap between neighbours.
3. EmbedEach Node's text went to the embedding model and came back as a vector. This is the step that costs money and takes time, which is why you asked for a progress bar.
4. StoreThe vectors and the Node text went into the default in-memory store. Nothing is on disk yet.
5. Queryas_query_engine() built a retriever plus a synthesiser. .query() embedded your question, found the two most similar Nodes, pasted them into a prompt with your question, and sent it to the LLM.

Two defaults in there deserve to be dragged into the light, because they explain most first-day disappointments.

You retrieved two chunks, not ten. similarity_top_k defaults to 2. For a question whose answer lives in one paragraph that is plenty. For "summarise the main themes" it is hopeless, and the model will dutifully summarise the two paragraphs it was given as though they were the whole document. Raise it: index.as_query_engine(similarity_top_k=5).

You used models you never chose. With no Settings configured, the LLM resolves to OpenAI(), whose default model is gpt-3.5-turbo, and the embedding model to OpenAIEmbedding(), whose default is text-embedding-ada-002. Both are old. Always set them explicitly.

Seeing what was actually retrieved

The response object is not just a string. Print its sources and you can audit every answer.

response = query_engine.query("What did the author work on before college?")
print(response.response)
for node in response.source_nodes:
    print(f"--- score={node.score:.3f} file={node.metadata.get('file_name')}")
    print(node.text[:300].replace("\n", " "))

response.source_nodes is a list of NodeWithScore. The score is the similarity between the question's vector and the Node's vector, so higher is better, and the absolute value depends on the embedding model. Get into the habit of printing this. When an answer is wrong, these few lines tell you immediately whether retrieval failed (the right text is not in the list) or synthesis failed (the right text is there and the model still got it wrong). Those two failures have completely different fixes, and guessing which one you have is the most common waste of an afternoon in this field.

Configuration: Settings and local overrides

Choosing models explicitly is the first thing to do in any real project. LlamaIndex gives you two levels: a process-wide singleton called Settings, and per-call arguments that override it.

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small", embed_batch_size=100)
Settings.text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
Settings.context_window = 4096
Settings.num_output = 256

Import this module before you build an index and everything downstream picks up these choices. temperature=0.1 is deliberate: for factual question answering you want the model boring and repeatable, not creative.

The attributes Settings holds are llm, embed_model, text_splitter (also reachable as node_parser), chunk_size, chunk_overlap, transformations, tokenizer, callback_manager, context_window and num_output. The verified defaults are worth memorising:

One piece of history you will trip over in older tutorials: configuration used to live in a ServiceContext object. It still imports, but using it raises ValueError: ServiceContext is deprecated. Use llama_index.settings.Settings instead, or pass in modules to local functions/methods/interfaces. If a blog post hands you ServiceContext.from_defaults(...), the post predates mid-2024 and the rest of its code is probably stale too.

Chunk size is a real decision

chunk_size controls how much text goes into each Node, and it is the knob with the largest effect on answer quality. Small chunks (256–512 tokens) are precise: a match means the chunk is genuinely about your question, and you can afford to retrieve several. Large chunks (1024–2048) carry more surrounding context, which helps when the answer needs a paragraph of setup, but they dilute the embedding — a 2000-token chunk covering five topics has a vector that is the average of five topics and matches none of them strongly.

chunk_overlap exists so that a sentence straddling a boundary still appears whole in one of the two chunks. Keep it well below the chunk size; the framework enforces this and will tell you off: ValueError: Got a larger chunk overlap (200) than chunk size (100), should be smaller. If you shrink chunk_size, shrink the overlap too.

There is no universally right answer. Start at 512 with 50 overlap for prose documents, measure, and adjust. The measuring part is not optional, and tools like RAGAS exist to make it systematic once you are past the prototype.

Loading files properly

SimpleDirectoryReader does more than its name suggests, and its defaults will bite you at least once. The two that matter most: it reads only the top level of the directory (recursive=False), and it assumes UTF-8.

from llama_index.core import SimpleDirectoryReader

reader = SimpleDirectoryReader(
    input_dir="data",
    recursive=True,
    required_exts=[".pdf", ".md", ".txt"],
    exclude=["drafts/*"],
    num_files_limit=200,
)
documents = reader.load_data()
print(f"{len(documents)} documents")
print(documents[0].metadata)

The options you will reach for are input_dir, input_files=[...] for an explicit list, exclude=[...], recursive=True, required_exts=[...], num_files_limit, encoding="latin-1" for legacy files, file_metadata=fn to attach your own metadata, file_extractor={".myfile": MyReader()} to override the reader for an extension, and fs= to read from a remote filesystem such as S3. load_data(num_workers=4) parallelises across processes, and iter_data() streams instead of loading everything into memory.

Supported extensions out of the box include .csv, .docx, .epub, .hwp, .ipynb, .jpeg, .jpg, .mbox, .md, .mp3, .mp4, .pdf, .png, .ppt, .pptm and .pptx — all of which need llama-index-readers-file installed. JSON is handled by a separate package, llama-index-readers-json.

Every Document arrives with metadata filled in: file_path, file_name, file_type, file_size, creation_date, last_modified_date and last_accessed_date. The dates are UTC in %Y-%m-%d form. Print it on your first run of any new dataset. If file_name is right but the text looks like %PDF-1.4 followed by binary noise, you have the missing-readers problem from the install section.

Windows users have two extra notes from the docs. load_data(num_workers=N) uses multiprocessing, so "Windows users may see less or no performance gains", and any multiprocessing code must sit under if __name__ == "__main__": or it will spawn copies of itself. Several Windows encoding bugs were fixed during the 0.14 line by forcing UTF-8 for text-mode file I/O.

Persisting the index so you build it once

Everything so far lives in RAM and dies with the process. Embedding is the expensive step — in money with a hosted model, in time with a local one — so rebuilding an index on every run is the first thing to fix.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
index.storage_context.persist("storage")

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage_context)
print(index.as_query_engine().query("What is this about?"))

persist() with no argument uses ./storage. Look inside the folder afterwards: docstore.json holds the Nodes, index_store.json the index metadata, and default__vector_store.json the embeddings. They are plain JSON, so you can open them and see exactly what your chunking produced. That is a genuinely useful debugging move the first time chunking surprises you.

A StorageContext bundles four or five stores: a docstore for Nodes, an index store for index metadata, a vector store for embeddings, and optionally a property graph store and chat stores. At this level the defaults are all in-memory-plus-JSON, which is right for prototypes and wrong for production.

If you keep several indexes in one directory, name them, or loading becomes ambiguous:

index.set_index_id("vector_index")
index.storage_context.persist("storage")

# later
index = load_index_from_storage(storage_context, index_id="vector_index")

The three errors you will meet here are all informative. A wrong or missing directory gives FileNotFoundError: [Errno 2] No such file or directory: '.../docstore.json'. An empty store gives ValueError: No index in storage context, check if you specified the right persist_dir. Several unnamed indexes give ValueError: Expected to load a single index, but got N instead. Please specify index_id.

import os
from llama_index.core import (VectorStoreIndex, SimpleDirectoryReader,
                              StorageContext, load_index_from_storage)

PERSIST_DIR = "storage"

if os.path.exists(PERSIST_DIR):
    index = load_index_from_storage(StorageContext.from_defaults(persist_dir=PERSIST_DIR))
else:
    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)
    index.storage_context.persist(PERSIST_DIR)

One thing to understand about persistence before you move on: the JSON stores are versioned with the library, so test that your saved index still loads after you upgrade. And re-index from scratch whenever you change the embedding model or the chunking, because old vectors from a different model are meaningless against new queries. When you graduate to a real vector database — Chroma, Qdrant, pgvector — most of them store text alongside vectors and persist() becomes unnecessary; FAISS is the notable exception, because it stores vectors only and still needs a docstore beside it.

Asking questions well: query engines and chat engines

A query engine answers one question with no memory of the last one. That is the right shape for a search box and the wrong shape for a conversation. Both are one method call away.

from llama_index.core.postprocessor import SimilarityPostprocessor

query_engine = index.as_query_engine(
    similarity_top_k=5,
    response_mode="compact",
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)],
)
response = query_engine.query("What are the main themes?")
print(response)

Three choices there, each solving a specific problem. similarity_top_k=5 gets you past the default of two. The SimilarityPostprocessor drops Nodes whose score is below 0.7, which is how you stop the engine answering from irrelevant text — the gap you found in the fourth Try it. And response_mode picks the synthesis strategy.

The synthesis modes you should know are three. compact, the default for query engines, stuffs as many retrieved Nodes as fit into one prompt and asks once, then repeats with the leftovers if there are any. It is the cheapest and right for most questions. refine sends the first Node with the question, then shows the model its own draft answer plus the next Node and asks it to improve the answer, one Node at a time. More calls, more cost, better on questions whose answer is assembled from several places. tree_summarize summarises Nodes in pairs up a tree until one summary remains, which is the mode for "summarise everything".

Pick the mode from the question's shape. A lookup question wants compact. A whole-corpus question wants tree_summarize and a much higher top_k. Getting this wrong is a common cause of "the summary only mentioned two sections".

Streaming

Waiting eight seconds in silence feels broken even when it is not. Stream instead:

query_engine = index.as_query_engine(streaming=True, similarity_top_k=5)
streaming_response = query_engine.query("Summarize the documents.")
streaming_response.print_response_stream()

Multi-turn conversation

chat_engine = index.as_chat_engine()
print(chat_engine.chat("What does the document say about pricing?"))
print(chat_engine.chat("And what about discounts?"))

for token in chat_engine.stream_chat("Summarise that in one sentence.").response_gen:
    print(token, end="", flush=True)

The second question has no subject — "what about discounts" is meaningless alone — and it works because the chat engine carries history. Since 0.13 the default chat engine is CondensePlusContextChatEngine. The name describes the mechanism: it condenses your new message plus the history into a single standalone question, retrieves with that, then puts the retrieved context into the prompt along with the history. Understanding this explains the main failure mode: if the condensed question drops a crucial detail from three turns ago, retrieval goes looking for the wrong thing, and the answer is confidently about the wrong topic.

Your first agent

A query engine looks things up. An agent decides what to do. The difference matters when a question needs arithmetic, an API call, or a choice between two sources.

Agents in LlamaIndex are asynchronous. There is no synchronous agent.chat() — it was removed in 0.13 along with the whole previous generation of agent classes. Every run is await agent.run(...), which means your code needs an async entry point.

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b


def add(a: float, b: float) -> float:
    """Useful for adding two numbers together."""
    return a + b


agent = FunctionAgent(
    tools=[multiply, add],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can do arithmetic.",
)


async def main():
    print(await agent.run("What is 1234 * 4567, then add 100?"))


if __name__ == "__main__":
    asyncio.run(main())

Notice what you did not write: no schema, no JSON, no registration. You passed plain Python functions and LlamaIndex derived the tool definitions. The rules are worth stating explicitly because they are the whole interface:

• The function name becomes the tool name.
• The docstring becomes the description, which is what the model reads when deciding whether to call it. A vague docstring is the single most common cause of an agent ignoring a perfectly good tool.
• Type hints become the parameter schema. Untyped parameters give the model nothing to go on. You can add per-parameter descriptions with Annotated[str, "the city name"].

For more control, wrap the function yourself: FunctionTool.from_defaults(fn, async_fn=..., name=..., description=..., return_direct=...). Setting return_direct=True ends the agent loop immediately and returns the tool's output as the final answer, which is useful when the tool already produces exactly what the user asked for and you do not want to pay for the model to paraphrase it.

Giving the agent your documents

The interesting move is combining the two halves of this guide: wrap a query engine as a tool, and the agent can search your documents when — and only when — it decides it needs to.

import asyncio
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.tools import QueryEngineTool
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

docs_tool = QueryEngineTool.from_defaults(
    query_engine=index.as_query_engine(similarity_top_k=5),
    name="company_handbook",
    description=(
        "Answers questions about the company handbook: leave policy, "
        "working hours, expenses and travel. Input is a natural-language question."
    ),
)

agent = FunctionAgent(
    tools=[docs_tool],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="Answer from the handbook tool. If it has no answer, say so plainly.",
)


async def main():
    print(await agent.run("How many days of annual leave do I get?"))


if __name__ == "__main__":
    asyncio.run(main())

Write that description as if it were documentation for a colleague who can only see the description, because that is exactly the situation the model is in. "Searches documents" tells it nothing. Naming the topics and saying what the input looks like is what makes routing work, and with several tools it is the difference between an agent that picks the right source and one that guesses.

Memory across runs

Each agent.run() is independent by default. To carry a conversation, create a Context and pass it every time.

import asyncio
from llama_index.core.workflow import Context
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

agent = FunctionAgent(tools=[], llm=OpenAI(model="gpt-4o-mini"))


async def main():
    ctx = Context(agent)
    print(await agent.run("My name is Logan.", ctx=ctx))
    print(await agent.run("What is my name?", ctx=ctx))


if __name__ == "__main__":
    asyncio.run(main())

For anything longer-lived, Memory.from_defaults(session_id="user-42") gives you a proper memory object with a token budget (30,000 tokens by default, in-memory SQLite storage) that you pass as memory= to run(). The older ChatMemoryBuffer class is documented as deprecated in favour of Memory, though agents still construct one internally when you pass no memory at all. Use Memory in new code.

Two agent limits will eventually interrupt you. The iteration cap is 20: exceed it and you get WorkflowRuntimeError: Max iterations of 20 reached! Either something went wrong, or you can increase the max iterations with .run(.., max_iterations=...) or use early_stopping_method='generate' to generate a final response instead. When you see that, suspect a loop — a tool that keeps failing and a model that keeps retrying it — before you raise the cap. Second, if your model does not support streaming, pass streaming=False, since agents stream by default.

ReActAgent, imported from the same module, is the alternative when your model has no native tool-calling API: it prompts for Thought/Action/Observation text and parses it. CodeActAgent writes and executes code and requires you to supply a code_execute_fn — it raises ValueError: code_execute_fn must be provided for CodeActAgent otherwise. Do not hand it a bare exec; running model-written code needs a sandbox, which is a Senior-level topic for a good reason. If multi-agent orchestration is where you are heading, LangGraph covers the graph-shaped approach to the same problem.

Reading the errors

LlamaIndex has unusually informative error messages. Learning to read six of them will save you more time than any other half-hour in this guide.

No API key. A long ValueError beginning Could not load OpenAI model. If you intended to use OpenAI, please check your OPENAI_API_KEY. The embedding variant says Could not load OpenAI embedding model. ... Consider using embed_model='local'. The cause is that something resolved to OpenAI because Settings left it unset. The sneaky case is a local setup where you set Settings.llm = Ollama(...) and forgot Settings.embed_model: the LLM is local, the embedder silently is not, and the error appears at index time rather than query time. Set both.

Missing integration package. ImportError: `llama-index-llms-openai` package not found, please run `pip install llama-index-llms-openai`, or a plainer ModuleNotFoundError: No module named 'llama_index.llms.ollama'. The naming rule makes this self-solving: a package called llama-index-{type}-{name} imports as llama_index.{type}.{name}. Read the failed import backwards and you have the pip command.

Legacy imports from old tutorials. ImportError: cannot import name 'FunctionCallingAgent' from 'llama_index.core.agent' or ModuleNotFoundError: No module named 'llama_index.core.query_pipeline', or an AttributeError on ReActAgent.from_tools or agent.chat. All of these mean the tutorial predates 0.13, which removed the old agent classes and QueryPipeline entirely. Modern code imports FunctionAgent and ReActAgent from llama_index.core.agent.workflow and uses await agent.run(...). There is no compatibility shim, so a tutorial that hits this needs replacing, not patching.

Embedding dimension mismatch. ValueError: shapes (384,) and (1536,) not aligned: 384 (dim 0) != 1536 (dim 0). You built the index with one embedding model and queried it with another — commonly a 384-dimension local model against an index built with a 1536-dimension OpenAI one. Vectors from different models are not comparable. Rebuild the index, or query with the model you built it with. Hosted vector databases raise their own version of this, usually at collection-creation time.

Prompt too big. ValueError: Calculated available context size -123 was not non-negative. The arithmetic is context window minus prompt minus reserved output, and it went negative. Either the model's context window is unknown — so the framework used its 3900-token fallback — or your local model genuinely has a small one. Fix it by setting a correct context_window and num_output on the LLM or in Settings, or by lowering similarity_top_k or chunk_size. A 2048-token chunk times top_k=10 is 20,000 tokens of context; the arithmetic is not mysterious once you do it.

Async misuse. RuntimeError: asyncio.run() cannot be called from a running event loop appears when you call asyncio.run() inside a Jupyter notebook or a FastAPI handler, both of which already run a loop. In a notebook, just await agent.run(...) directly. In a script, have exactly one asyncio.run(main()). The related SyntaxError: 'await' outside function means you pasted agent code into a script without wrapping it in async def main().

Putting it all together

One small project that uses everything above: a command-line assistant over a folder of documents that builds its index once, caches it, retrieves with a similarity floor, and runs as an agent so it can refuse out-of-scope questions instead of inventing answers.

import asyncio
import os
import sys

from llama_index.core import (Settings, SimpleDirectoryReader, StorageContext,
                              VectorStoreIndex, load_index_from_storage)
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import SimilarityPostprocessor
from llama_index.core.tools import QueryEngineTool
from llama_index.core.workflow import Context
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

PERSIST_DIR = "storage"
DATA_DIR = "data"

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.text_splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)


def build_or_load_index() -> VectorStoreIndex:
    if os.path.exists(PERSIST_DIR):
        print(f"loading index from {PERSIST_DIR}")
        return load_index_from_storage(
            StorageContext.from_defaults(persist_dir=PERSIST_DIR)
        )
    print(f"building index from {DATA_DIR}")
    documents = SimpleDirectoryReader(
        DATA_DIR, recursive=True, required_exts=[".pdf", ".md", ".txt"]
    ).load_data()
    if not documents:
        sys.exit(f"no documents found in {DATA_DIR}")
    print(f"loaded {len(documents)} documents")
    index = VectorStoreIndex.from_documents(documents, show_progress=True)
    index.storage_context.persist(PERSIST_DIR)
    return index


def build_agent(index: VectorStoreIndex) -> FunctionAgent:
    query_engine = index.as_query_engine(
        similarity_top_k=5,
        response_mode="compact",
        node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)],
    )
    tool = QueryEngineTool.from_defaults(
        query_engine=query_engine,
        name="document_search",
        description=(
            "Searches the local document collection and answers questions "
            "about its contents. Input is a full natural-language question."
        ),
    )
    return FunctionAgent(
        tools=[tool],
        system_prompt=(
            "You answer questions using the document_search tool only. "
            "If the tool returns nothing relevant, say you could not find it "
            "in the documents. Never guess."
        ),
    )


async def main() -> None:
    agent = build_agent(build_or_load_index())
    ctx = Context(agent)
    print("ask a question, or an empty line to quit")
    while True:
        try:
            question = input("\n> ").strip()
        except (EOFError, KeyboardInterrupt):
            break
        if not question:
            break
        print(await agent.run(question, ctx=ctx))


if __name__ == "__main__":
    asyncio.run(main())

Read it once against the six nouns. SimpleDirectoryReader produces Documents. SentenceSplitter, configured on Settings, turns them into Nodes. VectorStoreIndex.from_documents embeds and indexes them. persist and load_index_from_storage make that work survive a restart. as_query_engine assembles a retriever, a postprocessor and a synthesiser. QueryEngineTool presents the whole thing to the agent as one capability, and Context keeps the conversation coherent across turns.

Four exercises turn this from a demo into something you understand. First, ask a question you know is out of scope and verify that the system prompt plus the similarity cutoff actually produce a refusal — if it still invents an answer, raise the cutoff and look at the scores to decide where the real threshold is. Second, add a second tool, a plain Python function, and watch the agent choose. Third, delete storage, change chunk_size to 1024, rebuild, and compare answers on the same five questions. Fourth, swap both models for local ones using the Ollama snippet from the install section and confirm that nothing else in the file has to change — that substitutability is the payoff for learning the vocabulary.

What you can now do, and what comes next

You can install LlamaIndex without falling into the readers-file trap, load a folder of mixed file types, control how it is chunked, build and persist a vector index, query it with a sensible top_k and a similarity floor, hold a multi-turn conversation over it, inspect exactly which chunks produced an answer, wrap the whole thing as a tool for an agent, give that agent your own Python functions, and read the framework's six most common errors without reaching for a search engine. You also know which tutorials to throw away: anything with ServiceContext, QueryPipeline, FunctionCallingAgent, agent.chat() or llamaindex-cli is from before 0.13 and will not run.

What you have deliberately not done is anything that survives contact with scale. The in-memory vector store loads entirely into RAM and runs in one process. Re-ingesting a changed folder re-embeds everything, including the 99% that did not change. Nothing measures whether answers are actually correct. There is no tracing, so when quality drops in production you have no record of what was retrieved. And the chunking is uniform, which is the wrong strategy for documents full of tables.

Mid-level picks up precisely there: IngestionPipeline with a docstore and a cache so unchanged documents are skipped, real vector stores and metadata filters, retrieval tuning and reranking, memory blocks, multi-agent workflows with AgentWorkflow, evaluation, and the async patterns that let one process serve many users. Senior goes on to custom Workflows, durable execution and human-in-the-loop, multi-tenancy, cost control and the security model — which matters because LlamaIndex is a library with no built-in authentication, authorisation, tenancy or sandboxing, and every one of those is yours to supply.

Three directions from here, depending on where you are going. For retrieval quality, pair this with a real vector database — Qdrant, Chroma or pgvector — and learn metadata filtering properly. For trustworthiness, add tracing with Langfuse and scoring with RAGAS; an untraced, unevaluated RAG system is one you cannot improve, because you cannot tell whether a change helped. For serving, put the whole thing behind FastAPI, which is the natural fit for an async-first framework. And if you want to compare approaches before committing, read the LangChain and LangGraph guides; knowing why you chose one is worth more in an interview than knowing one deeply.

One last habit for the long run. Pin your versions — llama-index-core>=0.14.25,<0.15 — because the minor number is the breaking-change boundary in this project, and every integration package declares that same <0.15 bound. When 0.15 lands, all of them will be re-released together, and the upgrade will be a deliberate afternoon rather than a surprise on a Monday morning. Read the changelog before you take it.

Sources

• LlamaIndex Python framework documentation
• Installation and setup
• Starter tutorial
• Starter tutorial with local models
• High-level concepts
• Using async Python with LlamaIndex
• Settings
• SimpleDirectoryReader
• VectorStoreIndex
• Storing data
• Persisting and loading data
• Node postprocessors
• Chat engines
• Agents
• Agent tools
• Agent memory
• Building a basic agent
• Workflows documentation
• LlamaIndex CHANGELOG
• llama-index-core on PyPI

Interviewers rarely ask you to recite the LlamaIndex API. What they probe is whether you understand why a retrieval system is built the way it is, and whether you have actually run one or only read about it. The giveaway is almost always the same: a candidate who has run one talks about what went wrong, and a candidate who has not talks only about what the library offers.

This file is the compressed version of the detailed guide, arranged the way a conversation goes.

The thirty-second answer

LlamaIndex is a Python framework for building LLM applications over your own data. It handles the whole retrieval pipeline — loading documents, splitting them into chunks, embedding and indexing them, retrieving the relevant ones for a question and synthesising an answer — and it provides an orchestration layer called Workflows, on which its agents are built.

If they want one more sentence, add the reason it exists: a language model has never seen your private documents, and you cannot paste a thousand pages into a prompt, so you need something that finds the relevant few paragraphs at query time and sends only those. That is retrieval-augmented generation, and LlamaIndex is the machinery for it.

Resist the urge to list integrations. "It supports 300 vector stores" tells an interviewer nothing about you.

The version facts to have ready

Being specific about versions signals that you have used the tool recently rather than read a tutorial from two years ago.

• The current line is llama-index-core 0.14.25, released 21 September 2026. The 0.14 line opened in September 2025.
• Python 3.10 to 3.14. Python 3.9 support was dropped during the 0.14 line.
• The minor number is the breaking-change boundary — 0.13 to 0.14 was breaking, 0.14.x patches are not. Every integration package pins llama-index-core<0.15.0, so 0.15 will require all of them to be re-released.
• llama-index-workflows versions independently; it is at 2.25.0.
• There is no supported CLI. llama-index-cli is deprecated, unmaintained, dropped from the bundle in 0.14.20, and broken on current core.

The model, in the words you should use

Six nouns carry the data layer, and naming them in order is the cleanest possible answer to "walk me through how it works".

Document — a whole source: a file, a page, a database row. It has text, metadata and an id. Node — a chunk of a Document, the thing that is actually embedded and retrieved; node.ref_doc_id points back to its parent. Reader — turns a source into Documents; SimpleDirectoryReader is the built-in local one. Node parser (a text splitter, and more generally a transformation) — turns Documents into Nodes; the default is SentenceSplitter. Index — a structure over Nodes that serves retrieval; VectorStoreIndex is the default. Query engine — retriever plus postprocessors plus synthesiser, obtained with index.as_query_engine().

Three supporting terms finish it. A retriever is the retrieval half alone, returning NodeWithScore objects. A node postprocessor filters or reranks retrieved Nodes before synthesis. A response synthesizer writes the answer; its modes are compact (the default), refine and tree_summarize.

The sentence that shows you understand rather than memorised: you load Documents, but you retrieve Nodes. Everything about cost, latency and answer quality is decided at the step where twenty files become two thousand chunks.

The agent half

Agents sit on Workflows, which is why they are async. An agent is an LLM plus memory plus tools, looping: send the message and the tool schemas, the model answers or requests tool calls, you execute them, append results, repeat. The classes are FunctionAgent (native provider tool-calling), ReActAgent (Thought/Action/Observation prompting, for models without tool-calling), CodeActAgent (writes and runs code, requires your own code_execute_fn) and AgentWorkflow (several agents with hand-offs).

A plain Python function becomes a tool: the name becomes the tool name, the docstring becomes the description, and the type hints become the parameter schema.

The commands and calls to have ready

Install, and the one thing people get wrong:

pip install llama-index llama-index-readers-file

Five-line RAG:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents, show_progress=True)
query_engine = index.as_query_engine(similarity_top_k=5)
print(query_engine.query("Summarise the documents."))

Persist and reload:

index.storage_context.persist("storage")

from llama_index.core import StorageContext, load_index_from_storage
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))

Configure the models, which is the first thing you do in any real project:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

A first agent, async as all agents are:

import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

agent = FunctionAgent(tools=[multiply], llm=OpenAI(model="gpt-4o-mini"))

async def main():
    print(await agent.run("What is 1234 * 4567?"))

asyncio.run(main())

Audit an answer, which is the habit interviewers notice:

for node in response.source_nodes:
    print(node.score, node.metadata.get("file_name"), node.text[:200])

The defaults worth memorising

Interviewers like defaults because they are checkable and they reveal whether you have read output or only code.

The classic traps, and why they happen

The umbrella package no longer installs the file readers. Since 0.14.18, llama-index does not pull in llama-index-readers-file, and the installation page has not caught up. Without it, SimpleDirectoryReader logs a warning about missing file readers and reads PDFs as plain text — garbage in the index, no exception raised. Install both packages. This is a strong answer because it shows you have debugged a real install rather than copied a snippet.

Default similarity_top_k is 2. Fine for a narrow lookup, useless for "summarise the document", where the model will summarise two paragraphs as though they were the whole corpus. The question's shape should pick both the top_k and the response mode: compact and a low top_k for lookups, tree_summarize with a high top_k for whole-corpus questions.

Retrieval always returns something. There is no built-in "nothing matched". The vector store returns the nearest neighbours however far away they are, so an out-of-scope question still yields two chunks and often a confident answer. You add the floor yourself with SimilarityPostprocessor(similarity_cutoff=...) plus a system prompt that permits refusal.

Forgetting one of the two models. Set Settings.llm to a local Ollama model and leave Settings.embed_model unset, and the embedder silently resolves to OpenAI. You then get Could not load OpenAI embedding model... at index time, not at query time, which is why people hunt in the wrong place. Set both, always.

Changing the embedding model without re-indexing. Vectors from different models live in different spaces and often different dimensions, giving ValueError: shapes (384,) and (1536,) not aligned. Any change to the embedding model or the chunking means a rebuild.

Tutorials from before 0.13. The 0.13 release removed the entire previous generation of agents — FunctionCallingAgent, the old ReActAgent.from_tools style, AgentRunner, OpenAIAgent — and removed QueryPipeline outright. It also changed as_chat_engine() to return condense-plus-context. If code uses agent.chat(), ServiceContext, QueryPipeline or llamaindex-cli, it predates that and will not run. There is no compatibility shim.

asyncio.run() inside a notebook. Agents are async-only now. In Jupyter or FastAPI there is already a running loop, so asyncio.run() raises RuntimeError: asyncio.run() cannot be called from a running event loop. In a notebook, just await. In a script, one asyncio.run(main()).

Common interview questions

"What is RAG, and why not just fine-tune?"

RAG looks the answer up at query time and puts the relevant passages in the prompt. Fine-tuning changes the model's weights. Retrieval wins for factual knowledge over changing documents for three reasons: a document edit takes effect on the next ingestion run rather than the next training run, you can show which passages produced the answer, and it costs embedding tokens rather than training compute. Fine-tuning is for changing behaviour — tone, output format, a domain's idiom — not for installing facts. In practice teams that need both do both, and they start with retrieval because it is cheaper to get working and much cheaper to get wrong.

"Walk me through what happens when I call query_engine.query("...")."

The question is embedded with the same embedding model that built the index. The vector store returns the similarity_top_k nearest Nodes as NodeWithScore objects. Any node postprocessors run — a similarity cutoff, a reranker. The response synthesizer takes the surviving Nodes plus the question and builds a prompt; in compact mode it packs as many Nodes as fit into one LLM call. The LLM's answer comes back on response.response, and the Nodes that produced it are on response.source_nodes with their scores.

That last clause is the one that matters. Volunteer it, because it leads straight to the next question.

"The answer is wrong. How do you debug it?"

Split the problem in two before changing anything. Print response.source_nodes. If the text that contains the answer is not in that list, retrieval failed: raise similarity_top_k, reduce chunk_size so matches are more precise, check the documents actually loaded as readable text, or add metadata filtering. If the right text is in the list and the answer is still wrong, synthesis failed: the top_k may be so high that the useful chunk is buried, the response mode may be wrong for the question, or the model is too weak. Guessing which half failed is the most common waste of time in this field, and one print settles it.

"Why are LlamaIndex agents asynchronous?"

Because agents are implemented as Workflows, and Workflows are an event-driven async runtime. The synchronous agent.chat() and agent.query() were removed in 0.13 along with the old agent classes. It is also the right default for the workload: an agent run is almost entirely waiting on network calls, so async lets one process serve many conversations without threads.

"How does the agent know when to use a tool?"

It reads the tool's metadata — name, description, parameter schema — which for a plain Python function come from the function name, the docstring and the type hints. The model decides from the description alone, so a vague docstring is the most common cause of an agent ignoring a perfectly good tool. Writing tool descriptions is prompt engineering: name the topics the tool covers and say what the input looks like, because the model sees nothing else.

"Chunk size: how would you choose one?"

By measuring, with reasoning to narrow the starting point. Small chunks of 256 to 512 tokens are precise, because the embedding describes one topic, so a match is meaningful and you can retrieve several. Large chunks carry more context but dilute the vector — a 2000-token chunk spanning five topics averages all five and matches none strongly. Overlap exists so a sentence on a boundary survives whole in one chunk, and it must stay well below the chunk size or you get ValueError: Got a larger chunk overlap (200) than chunk size (100), should be smaller. I would start at 512 with 50 overlap for prose, build an evaluation set of real questions, and compare.

"What is the difference between a query engine and a chat engine?"

A query engine is single-turn and stateless: one question in, one answer out. A chat engine carries history, so a follow-up like "and what about discounts?" resolves. Since 0.13 the default is CondensePlusContextChatEngine, which condenses the new message plus the history into one standalone question, retrieves with that, then puts the retrieved context and the history into the prompt. Its failure mode follows from the mechanism: if the condense step drops a detail from three turns ago, retrieval looks for the wrong thing and the answer is confidently off-topic.

"When would you not use LlamaIndex?"

When the data fits in the prompt — if your whole corpus is a ten-page handbook, retrieval is machinery you do not need. When the task is not knowledge retrieval at all but classification or extraction over text you already have. When you need a single narrow production path and the framework's generality is cost without benefit, in which case an embedding call plus a vector-store query of your own is a hundred lines you fully control. Being willing to say this is a point in your favour; interviewers are wary of candidates who reach for a framework reflexively.

"How would you run this if the documents are not allowed to leave the country?"

This comes up constantly with employers in the Gulf and in Egypt, and it is a question about architecture, not about the framework. Every component in LlamaIndex is behind an interface, so the answer is that application code does not change: swap Settings.llm for an Ollama-served model and Settings.embed_model for a HuggingFace model, and point the vector store at something you host. That is two lines plus the install. What does change is the operational side — you now own GPU capacity, model updates and the storage — and the embedding model is usually the harder choice, because quality on Arabic text varies a lot more between open models than it does on English. The strong version of this answer names the trade-off rather than claiming local is free.

"What is persisted when you call storage_context.persist(), and what is not?"

A StorageContext bundles a docstore for the Nodes, an index store for the index metadata, a vector store for the embeddings, and optionally a property graph store and chat stores. With the defaults, persist() writes three JSON files into ./storage: docstore.json, index_store.json and default__vector_store.json. They are plain JSON and worth opening once. What is not persisted is your configuration — the models and the splitter live in Settings, so the loading process must configure the same embedding model or queries will not match. And when you move to a hosted vector database, most of them store text alongside the vectors, so persist() stops being necessary; FAISS is the exception, because it holds vectors only and still needs a docstore beside it.

"What does LlamaIndex not give you?"

It is a library, not a platform. No authentication, no authorisation, no multi-tenancy, no sandbox. Tools execute arbitrary Python with your process's privileges, and CodeActAgent runs model-written code through a function you supply, so that function needs a real sandbox. Retrieved documents and tool outputs are untrusted input and a prompt-injection surface. The in-memory default stores are prototype-only: they load fully into RAM and run in one process.

Sixty-second self-test

Answer out loud, then check yourself against the sections above.

1. What is the difference between a Document and a Node, and which one gets embedded?
2. Name the six core nouns in data-flow order.
3. What is the default similarity_top_k, and when is it wrong?
4. Which two Settings attributes must you always set, and what happens if you set only one?
5. Why does pip install llama-index alone break PDF loading, and how do you tell?
6. What are the three response modes, and which suits "summarise the whole corpus"?
7. Your RAG system answers an out-of-scope question confidently. Why, and what two changes fix it?
8. What tells an agent whether to call your Python function?
9. Why must agent calls be awaited, and what breaks in a notebook?
10. Which version number in this project signals a breaking change, and what will 0.15 require?

Three or more that you could not answer cleanly means a re-read, not a guess. The five-line starter plus one deliberate error of each kind from the detailed guide is an hour's work and it closes every one of these gaps.

Most of the time people lose with LlamaIndex goes to a handful of mistakes, and almost all of them share one property: nothing raises an exception. The index builds, the query returns a paragraph, the paragraph is wrong, and there is no stack trace to read. This file is the list of those failures with the symptom, the cause and the fix, followed by the habits that prevent them.

The mistakes beginners actually make

Installing only the umbrella package

Symptom. You index a folder of PDFs. The build succeeds. Every answer is nonsense, or the model says it cannot find anything. Somewhere in the scroll-back is a line you did not read: `llama-index-readers-file` package not found, some file readers will not be available if not provided by the `file_extractor` parameter.

Cause. The llama-index bundle dropped llama-index-readers-file in 0.14.18. The official installation page still says the bundle includes it. Without that package, SimpleDirectoryReader falls back to reading PDFs, DOCX and PowerPoint files as plain text, so your Nodes contain %PDF-1.4 and compression artefacts.

Fix. pip install llama-index llama-index-readers-file. Then verify rather than trusting: print documents[0].text[:200] after every load of a new dataset and confirm it is readable prose.

Leaving the models at their defaults

Symptom. It works, and it is worse than you expected, and it costs more than you expected.

Cause. With nothing set, Settings.llm resolves to OpenAI() whose default model is gpt-3.5-turbo, and Settings.embed_model to OpenAIEmbedding() whose default is text-embedding-ada-002. Both are old. You chose neither.

Fix. Set both explicitly in a config.py you import first in every script. Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1) and Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small"). Low temperature for factual question answering; you want repeatable, not creative.

Setting one model and not the other

Symptom. You configured a local Ollama model carefully, and you get ValueError: ... Could not load OpenAI embedding model. ... Consider using embed_model='local'. — at index time, which feels unrelated to anything you just changed.

Cause. Settings.llm and Settings.embed_model are independent. Set one and the other still falls back to OpenAI. The reverse happens too: a local embedding model with the LLM left unset fails only at query time, which is even more confusing.

Fix. Treat them as a pair. Any time you change one, check the other in the same edit. For a fully local setup you need llama-index-llms-ollama and llama-index-embeddings-huggingface, and both lines of Settings.

Accepting similarity_top_k=2

Symptom. Narrow factual questions work beautifully. "Summarise the main points" produces a tidy summary of one section and ignores the rest of the document.

Cause. The default is 2. Two chunks is all the model ever saw, and it summarised them faithfully.

Fix. Choose top_k from the question's shape, and the response mode with it. A lookup: similarity_top_k=3 with response_mode="compact". A whole-corpus question: similarity_top_k=10 or more with response_mode="tree_summarize". There is no single right value, which is exactly why leaving the default is a decision rather than a non-decision.

Assuming retrieval can return nothing

Symptom. You ask something the documents definitely do not cover and get a confident, plausible, entirely invented answer.

Cause. A vector search returns nearest neighbours, not matches. There is no threshold. The two least-unrelated chunks come back whatever their scores, and the model does its best with them.

Fix. Two changes, and you need both. Add node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)] so weak matches are dropped, and write a system prompt that permits refusal — "if the tool returns nothing relevant, say you could not find it in the documents; never guess". Pick the cutoff by looking at real scores from response.source_nodes on questions you know are in scope and out of scope; the right number depends on your embedding model.

Debugging by guessing

Symptom. An answer is wrong. You change the chunk size. Then the model. Then the prompt. An hour passes.

Cause. You have not established whether retrieval or synthesis failed, and they have disjoint fixes.

Fix. Print the sources before touching anything.

for node in response.source_nodes:
    print(f"{node.score:.3f} {node.metadata.get('file_name')}")
    print(node.text[:300].replace("\n", " "))

If the answer's text is not in that list, it is a retrieval problem: raise top_k, lower chunk_size, check the documents loaded properly, add metadata filtering. If it is there, it is a synthesis problem: the right chunk may be buried among nine irrelevant ones, the response mode may be wrong, or the model is too weak.

Rebuilding the index on every run

Symptom. Every run of your script waits thirty seconds and spends money embedding text that has not changed.

Cause. VectorStoreIndex.from_documents builds in memory and the process exits.

Fix. The build-or-load pattern, written once:

import os
from llama_index.core import (VectorStoreIndex, SimpleDirectoryReader,
                              StorageContext, load_index_from_storage)

if os.path.exists("storage"):
    index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
else:
    index = VectorStoreIndex.from_documents(SimpleDirectoryReader("data").load_data())
    index.storage_context.persist("storage")

Forgetting that SimpleDirectoryReader is not recursive

Symptom. Your index is much smaller than your folder. No warning, no error.

Cause. recursive=False is the default. Subfolders are skipped silently.

Fix. Pass recursive=True, and print len(documents) after loading so a wrong number is visible immediately. Pair it with required_exts=[...] so you do not accidentally index your .DS_Store files or a stray binary.

Changing the embedding model without rebuilding

Symptom. ValueError: shapes (384,) and (1536,) not aligned: 384 (dim 0) != 1536 (dim 0), or a hosted vector database's own dimension error.

Cause. Vectors from different embedding models are not comparable, and usually not even the same length. A persisted index carries vectors from the model that built it.

Fix. Delete the persist directory and rebuild whenever the embedding model or the chunking changes. Make it a rule rather than a judgement call. It is also worth writing the model name into a small JSON file beside storage/ so a mismatch is obvious to the next person.

Following a pre-0.13 tutorial

Symptom. ImportError: cannot import name 'FunctionCallingAgent' from 'llama_index.core.agent', ModuleNotFoundError: No module named 'llama_index.core.query_pipeline', an AttributeError on ReActAgent.from_tools or agent.chat, or ValueError: ServiceContext is deprecated.

Cause. The 0.13 release removed the entire previous generation of agents and removed QueryPipeline outright. ServiceContext was replaced by Settings. There is no compatibility layer.

Fix. Abandon the tutorial rather than patching it; if it is that old, the rest of its advice is stale too. Modern code imports FunctionAgent and ReActAgent from llama_index.core.agent.workflow and uses await agent.run(...). Configuration goes through Settings or local arguments. And do not use llamaindex-cli, download_llama_pack or download_loader: the CLI package is deprecated, unmaintained and broken on current core.

Calling asyncio.run() in a notebook

Symptom. RuntimeError: asyncio.run() cannot be called from a running event loop, or SyntaxError: 'await' outside function when you paste notebook code into a script.

Cause. Agents are async-only. Jupyter and FastAPI already run an event loop, so you cannot start another. A bare script has no loop, so a top-level await is a syntax error.

Fix. In a notebook, await agent.run(...) directly. In a script, wrap your code in async def main() and have exactly one asyncio.run(main()) under if __name__ == "__main__":.

Writing a vague tool description

Symptom. You give an agent a perfectly good QueryEngineTool and it answers from the model's own knowledge instead, or picks the wrong tool when there are two.

Cause. The model chooses from the description alone. For a plain function that is the docstring; for a wrapped tool it is whatever you passed as description. "Searches documents" says nothing.

Fix. Write it for a colleague who can see nothing else: name the topics the tool covers and say what the input looks like. "Answers questions about the company handbook: leave policy, working hours, expenses and travel. Input is a natural-language question." If two tools overlap, the fix is sharper boundaries in the descriptions, not a cleverer model.

Attaching too much metadata

Symptom. ValueError: Metadata length (N) is longer than chunk size (M). Consider increasing the chunk size or decreasing the size of your metadata to avoid this. Or a working index whose answers are oddly diluted.

Cause. Metadata is prepended to the Node's text before embedding and before the LLM sees it, so it eats the chunk budget and influences matching.

Fix. Keep metadata short. When you want a field for filtering but not for matching, exclude it from embedding or from the prompt with excluded_embed_metadata_keys and excluded_llm_metadata_keys on the Document.

Ignoring the agent iteration cap

Symptom. WorkflowRuntimeError: Max iterations of 20 reached! with a suggestion to raise max_iterations or use early_stopping_method='generate'.

Cause. Usually a loop: a tool keeps failing or returning something unusable and the model keeps retrying it.

Fix. Read the tool outputs before raising the cap. Stream the run and watch ToolCallResult events, or log every tool call. Raising the limit on a genuine loop just makes the bill larger. Also note that Workflow has a 45-second default timeout, which is short for LLM steps — agents themselves default to no timeout.

Letting the chunk budget go negative

Symptom. ValueError: Calculated available context size -123 was not non-negative.

Cause. The arithmetic is the model's context window minus the prompt minus the reserved output, and it went below zero. Either the framework does not recognise your model and fell back to its 3900-token assumption, or your local model genuinely has a small window and you are sending it ten 2048-token chunks.

Fix. Do the multiplication before you blame the library: chunk_size times similarity_top_k is roughly what you are sending. Then either set a correct context_window and num_output on the LLM or in Settings, or lower similarity_top_k or chunk_size. With a local model, setting context_window explicitly on the constructor is almost always the right move, since the framework cannot know it.

Treating retrieved text as trustworthy

Symptom. Nothing, for a long time. Then an agent does something it was never asked to do, because a document in the corpus contained instructions.

Cause. Retrieved document text and tool outputs go into the prompt. If an attacker can get a file into your indexed folder, they can write instructions in it. The framework has no defence against this and does not claim to.

Fix. Treat everything retrieved as untrusted input. Keep the toolset narrow, so the worst case is bounded. Control who can write to the ingested source. Never give an agent a tool that mutates data without a human step in between at this stage of your learning, and never hand CodeActAgent a bare exec — it requires a code_execute_fn precisely so that sandboxing is your explicit decision.

Habits worth adopting on day one

Put configuration in one importable module. A config.py that sets Settings.llm, Settings.embed_model and Settings.text_splitter, imported first in every script. It removes a whole class of "it behaved differently in the other file" confusion, and it is the file you edit when you move from a hosted model to a local one.

Print the sources, always, while developing. Four lines, and it makes the system transparent. The moment you stop being able to see which chunks produced an answer, you are tuning blind.

Keep a fixed question set from the very first day. Five to ten real questions with known answers, plus two you know are out of scope, in a text file. Re-run them after every change. This is the cheapest possible evaluation and it is the difference between improving the system and merely changing it. When the prototype graduates, formalise it with a tool like RAGAS — but the text file comes first, and most people never write it.

Pin your versions. llama-index-core>=0.14.25,<0.15 and an explicit pin on each integration. The minor number is this project's breaking-change boundary, every integration declares the same <0.15 bound, and they will all be re-released together when 0.15 lands. A pinned project upgrades on an afternoon you chose.

Make the out-of-scope case a first-class requirement. Decide early what your system does when it does not know, and test it as deliberately as you test the happy path. For an internal tool at a bank or a ministry, a confident wrong answer is far more damaging than a refusal, and this is usually what a reviewer asks about first.

Keep secrets in the environment. OPENAI_API_KEY as an environment variable or injected from a secret manager, never api_key="sk-..." in a file. Some official examples hard-code keys; do not copy that part. The same applies to Memory(async_database_uri=...), which embeds database credentials.

Learn the package naming rule instead of memorising imports. A package called llama-index-{type}-{name} imports as llama_index.{type}.{name}. llama-index-vector-stores-qdrant gives from llama_index.vector_stores.qdrant import QdrantVectorStore. Read any failed import backwards and you have the pip install command, which makes every missing-integration error self-solving.

Prefer local arguments to global Settings for one-off changes. index.as_query_engine(llm=..., similarity_top_k=5) rather than mutating the singleton. Settings is process-wide, so in a web server mutating it per request is a race condition.

Open the persisted JSON once. Look inside storage/docstore.json and read a few Nodes. Seeing your own chunk boundaries and overlap, in the actual data, teaches more about chunking than any explanation.

Expect to re-ingest. Treat the index as derived data you can throw away, and keep the build script runnable from nothing with one command. Every embedding or chunking change forces a rebuild, so make rebuilding boring.

The checklist

Before you call a beginner-level LlamaIndex project finished, walk this list.

• Both llama-index and llama-index-readers-file installed, and documents[0].text confirmed readable.
• Settings.llm and Settings.embed_model both set explicitly, in one config module.
• similarity_top_k and response_mode chosen deliberately for the kind of question asked.
• A SimilarityPostprocessor cutoff, with the threshold picked from observed scores.
• A system prompt that permits "I could not find that in the documents".
• Index persisted, with a build-or-load path so nothing re-embeds unnecessarily.
• recursive and required_exts set on the reader, and the document count printed.
• A fixed question set, including out-of-scope questions, re-run after every change.
• No secrets in source. No ServiceContext, QueryPipeline, agent.chat() or llamaindex-cli anywhere.
• Versions pinned below 0.15 and the changelog read before any upgrade.
• Agent tool descriptions a stranger could route on correctly.
• One asyncio.run(main()) entry point, and no top-level await in scripts.

When all twelve hold, the next limits you meet are real ones — re-embedding unchanged documents, an in-memory store that will not scale past one process, and no tracing when quality drops. Those are where the mid-level guide starts, and a good next step either way is pairing this with a real vector store such as Qdrant or pgvector and adding tracing with Langfuse, so that you can see what your system retrieved for a question a user asked last Tuesday.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Where you already know the call, we go straight to what it does underneath, because at this level the question stops being "which function do I call?" and becomes "why did it answer that, and what will it do under load?"

Everything here was checked against llama-index-core 0.14.25 and llama-index-workflows 2.25.0.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a real project in a few minutes.

The query engine, stage by stage

index.as_query_engine().query("…") is one line that hides five stages. You cannot debug a bad answer until you know which stage produced it, because the fixes are completely different: a retrieval miss and a synthesis miss look identical from the outside.

The stages, in order:

Retrieval embeds the query with Settings.embed_model and asks the vector store for the nearest similarity_top_k nodes. The default is 2. That default is the single most common cause of "it didn't find the answer that is obviously in the document": two chunks of roughly 1024 tokens is a very small window onto a corpus.

Postprocessing receives NodeWithScore objects and may drop or reorder them. SimilarityPostprocessor(similarity_cutoff=0.75) drops anything below a score; rerankers such as LLMRerank or CohereRerank re-score with a stronger model.

Synthesis turns the surviving nodes plus the query into prose. The default mode for query engines is compact: stuff as many nodes as fit into one prompt, then refine over any leftovers. tree_summarize summarises hierarchically and is better for "summarise everything" questions. refine walks node by node, which is slower and more expensive but survives nodes that contradict each other.

The diagnostic habit that makes all of this tractable: always look at source_nodes before you blame the model.

response = query_engine.query("What is the notice period in the Dubai contract?")
print(response)
for sn in response.source_nodes:
    print(f"{sn.score:.3f}  {sn.node.metadata.get('file_name')}  {sn.node.text[:120]!r}")

If the right text is not in that list, no amount of prompt engineering will help; you have a retrieval problem. If the right text is in the list and the answer is still wrong, you have a synthesis problem, and the fixes are a different response_mode, a stronger LLM for that call, or a custom prompt template.

Each stage takes its knobs at construction:

from llama_index.core.postprocessor import SimilarityPostprocessor

query_engine = index.as_query_engine(
    similarity_top_k=5,
    response_mode="tree_summarize",
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.75)],
    streaming=True,
)

A query engine is single-turn. Its stateful sibling is the chat engine, and since 0.13 index.as_chat_engine() returns a CondensePlusContextChatEngine: it condenses the conversation into a standalone query, retrieves against that, and puts the result in the system prompt. The old agent-based default was removed. The docstring for ChatMode.BEST still claims it "uses an agent", which is stale — the code maps it to condense-plus-context.

Chunking: the decision that sets your ceiling

Chunk size is where most RAG quality is won or lost, and it is settled before a single query runs. The reason it matters is a conflict: embedding similarity works best on small, focused passages, while an LLM answers best when it can see surrounding context. One chunk size has to serve both jobs, and it cannot serve both well.

The verified defaults in 0.14.25 are Settings.chunk_size = 1024 and Settings.chunk_overlap = 200 tokens, measured with tiktoken's cl100k encoding. Note that constants.DEFAULT_CHUNK_OVERLAP is 20, which is not what Settings.chunk_overlap returns — the Settings documentation page uses 20 only as an example. Trust the runtime value.

from llama_index.core import Settings
from llama_index.core.node_parser import SentenceSplitter

Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

SentenceSplitter respects sentence boundaries, which is why it is the default and the right first choice for prose. The alternatives earn their place on specific inputs: TokenTextSplitter splits purely on tokens and is predictable but blunt; SemanticSplitterNodeParser breaks where the embedding of consecutive sentences diverges, which handles documents with abrupt topic changes; CodeSplitter respects syntax trees and gained a token-based mode in 0.14.13; MarkdownElementNodeParser and HTMLNodeParser split on document structure, which is usually what you want for a documentation site or a scraped knowledge base.

The production-RAG guidance in the docs is to stop trying to find one good chunk size and instead decouple the chunk you retrieve from the chunk you synthesise over. Two patterns do this. Sentence-window indexes single sentences for precise matching but hands the LLM a window of sentences around each hit. Small-to-big indexes small chunks and expands each hit to its larger parent. Both let you have precise retrieval and generous context at once, at the cost of a little more plumbing.

Two chunking errors you will hit, and what they really mean:

ValueError: Got a larger chunk overlap (200) than chunk size (100), should be smaller.

That is the default overlap meeting a small chunk size you set yourself. Set both together or neither.

ValueError: Metadata length (612) is longer than chunk size (512). Consider increasing the
chunk size or decreasing the size of your metadata to avoid this.

This one is instructive, because it reveals that metadata is prepended to the node text before embedding. A SimpleDirectoryReader node carries file_path, file_name, file_type, file_size, creation_date, last_modified_date and last_accessed_date. Add a TitleExtractor and a long document title and you can spend most of a small chunk on housekeeping. There is also a printed warning when fewer than 50 tokens of real content remain, which is worth grepping your ingestion logs for.

The fix is to control what each consumer sees:

for node in nodes:
    node.excluded_embed_metadata_keys = [
        "file_size", "creation_date", "last_modified_date", "last_accessed_date",
    ]
    node.excluded_llm_metadata_keys = ["file_size", "file_type"]

Keep file_name visible to the LLM — citations are the main reason anyone trusts a RAG answer — and keep volatile timestamps out of the embedding, because they add noise and change the vector every time a file is touched.

Ingestion pipelines that you can re-run safely

At beginner level you rebuilt the index each time. In production you re-run ingestion on a schedule over a corpus that mostly has not changed, and re-embedding unchanged documents is the easiest large bill in this whole stack to run up by accident. IngestionPipeline exists to make re-runs cheap and idempotent.

from llama_index.core.ingestion import IngestionPipeline, IngestionCache
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import TitleExtractor
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.storage.kvstore.redis import RedisKVStore

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=50),
        TitleExtractor(),
        OpenAIEmbedding(model="text-embedding-3-small", embed_batch_size=100),
    ],
    vector_store=vector_store,
    docstore=SimpleDocumentStore(),
    cache=IngestionCache(
        cache=RedisKVStore.from_host_and_port("127.0.0.1", 6379),
        collection="ingest-cache",
    ),
)

nodes = pipeline.run(documents=docs, num_workers=4)

Two independent mechanisms are at work here, and conflating them is a common source of confusion.

The cache is keyed on the pair (node, transformation). If the same node has already been through the same transformation, the stored output is reused. This is what stops you paying twice for an embedding. IngestionCache defaults to memory; RedisCache, MongoDBCache and FirestoreCache make it survive restarts and let several workers share it, which is what you want in CI or in a Kubernetes job.

The docstore is keyed on doc_id plus a content hash, and it gives you upsert semantics. With both a docstore and a vector store attached, a document whose hash is unchanged is skipped entirely, and one whose hash has changed is re-processed and upserted, replacing its old nodes. With no vector store attached, the pipeline only de-duplicates its inputs — it cannot delete anything it does not own.

pipeline.persist("./pipeline_storage")
pipeline.load("./pipeline_storage")

Two details that bite. embed_batch_size on the embedding model sets how many texts go in one API request, which is the difference between an ingestion that takes four minutes and one that takes forty. And num_workers uses multiprocessing.Pool: it needs the if __name__ == "__main__": guard, and the docs warn that Windows users "may see less or no performance gains". For I/O-bound work prefer await pipeline.arun(documents=docs), which gives you concurrency inside one process.

For documents that arrive already indexed, the index itself exposes the lifecycle directly: index.insert(doc), index.update_ref_doc(doc), index.refresh_ref_docs(docs), and index.delete_ref_doc(ref_doc_id, delete_from_docstore=True). That last flag was only honoured from 0.14.19 onward, so if you are on an older patch and your docstore keeps growing after deletions, that is why. These are the calls behind a right-to-erasure request, which matters if you are serving users in a jurisdiction with a data-protection law — and the UAE, Saudi Arabia and Egypt all have one now.

Storage: what persists, and where

StorageContext bundles four stores and most confusion about persistence comes from not knowing which of them a given setup actually uses.

With all four at their defaults, your entire index is a Python dict that persist() writes to JSON in ./storage. That is perfect for a prototype and wrong for production: it loads fully into RAM, it is single-process, and the JSON format is versioned with the library, so you must test loading after every upgrade.

Point it at a real vector store and the picture changes:

from qdrant_client import QdrantClient
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore

client = QdrantClient(url="https://qdrant.internal:6333", api_key=QDRANT_KEY)
vector_store = QdrantVectorStore(client=client, collection_name="docs")

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(docs, storage_context=storage_context)

Most hosted stores — Qdrant, Chroma, Pinecone, Weaviate, Milvus, Postgres with pgvector, Redis, OpenSearch, Azure AI Search, LanceDB — keep the text alongside the vectors. That makes the docstore redundant for retrieval, and it makes persist() unnecessary. The reconnect in your service is then one line, and this is the single most important production habit in the whole guide:

index = VectorStoreIndex.from_vector_store(vector_store)

That reads nothing and embeds nothing; it just binds a query path onto a collection that already exists. A service that calls from_documents at startup re-ingests the corpus on every pod restart, which costs money, takes minutes, and turns a rolling deploy into an outage.

FAISS is the exception worth remembering: it stores vectors only, not text, so a FAISS index still needs a docstore persisted next to it or your retrieved nodes come back with empty content. If you are choosing between stores, the guides for Qdrant, pgvector, Chroma, Pinecone, Weaviate, Milvus and FAISS go into the trade-offs properly.

Persistence to the default stores goes through fsspec, so S3 and R2 work as targets. One caution: an official storage example sets ACL: "public-read" on the S3 object. Do not copy that. Your docstore contains the full text of every document you indexed.

Loading back has three error messages worth recognising:

FileNotFoundError: [Errno 2] No such file or directory: './storage/docstore.json'
ValueError: No index in storage context, check if you specified the right persist_dir.
ValueError: Expected to load a single index, but got 3 instead. Please specify index_id.

The third is the interesting one. If you keep several indexes in one directory, name them on the way in and ask for them by name on the way out:

index.set_index_id("vector_index")
index.storage_context.persist("storage")

from llama_index.core import StorageContext, load_index_from_storage
sc = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(sc, index_id="vector_index")

Retrieval tuning and metadata filters

Raising similarity_top_k is the first lever and it has a cost: more nodes means a longer prompt, more tokens, and more chance that an irrelevant chunk distracts the model. The grown-up version of the same idea is retrieve wide, then narrow: fetch 20 candidates cheaply by vector similarity, then let a reranker choose the best five.

from llama_index.core.postprocessor import LLMRerank

query_engine = index.as_query_engine(
    similarity_top_k=20,
    node_postprocessors=[LLMRerank(top_n=5)],
    response_mode="compact",
)

That buys precision for the price of one extra LLM call, which is usually a good trade when the alternative is a wrong answer. 0.14.16 added a multimodal LLMRerank if your nodes carry images.

Metadata filters are the other half of retrieval, and they are the half that matters for correctness rather than quality. A filter is applied by the vector store itself, before similarity ranking, so it is a hard constraint rather than a hint.

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

retriever = index.as_retriever(
    similarity_top_k=5,
    filters=MetadataFilters(filters=[
        MetadataFilter(key="tenant_id", value="acme"),
        MetadataFilter(key="department", value="hr"),
    ]),
)

MetadataFilters.condition defaults to AND when it is None, a behaviour that was made explicit in 0.14.24. Both filters above must match.

Filters only work if the metadata is there, which means ingestion has to put it there. SimpleDirectoryReader takes a file_metadata callable for exactly this:

from pathlib import Path
from llama_index.core import SimpleDirectoryReader

def tag(path: str) -> dict:
    p = Path(path)
    return {"department": p.parent.name, "tenant_id": p.parts[1], "source": "sharepoint"}

docs = SimpleDirectoryReader(
    input_dir="corpus",
    recursive=True,
    required_exts=[".pdf", ".docx", ".md"],
    exclude=["**/drafts/*"],
    file_metadata=tag,
).load_data(num_workers=4)

Note recursive=True: the reader is top level only by default, which quietly explains a lot of "half my documents are missing". And if PDFs come back as garbled text with this log line, the umbrella package no longer ships the file readers:

`llama-index-readers-file` package not found, some file readers will not be available
if not provided by the `file_extractor` parameter.

pip install llama-index-readers-file. The installation page still lists it as part of the bundle; it was dropped in 0.14.18 and the docs have not caught up.

Workflows: the execution model underneath everything

The most important structural fact about modern LlamaIndex is that agents are Workflows. FunctionAgent, ReActAgent, CodeActAgent and AgentWorkflow all live in llama_index.core.agent.workflow and all run on the llama-index-workflows engine. Once you understand workflows you can debug agents, and you can replace an agent with something deterministic when an agent is the wrong tool.

This also explains the big break in the library's history. 0.13.0 removed the entire previous generation: FunctionCallingAgent, the old ReActAgent.from_tools style, AgentRunner, every step worker, OpenAIAgent, and QueryPipeline. Tutorials written before mid-2025 fail with errors like these, and recognising them saves hours:

ImportError: cannot import name 'FunctionCallingAgent' from 'llama_index.core.agent'
ModuleNotFoundError: No module named 'llama_index.core.query_pipeline'

ReActAgent is particularly confusing because the name still imports — it just resolves to the new workflow class, so ReActAgent.from_tools(...) fails on an attribute rather than an import.

A workflow is a set of @step methods. Each step takes an event and returns an event, and the return type annotations define the graph. There is no separate wiring step and no DSL.

from workflows import Workflow, Context, step
from workflows.events import Event, StartEvent, StopEvent

class Retrieved(Event):
    nodes: list

class RagFlow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent) -> Retrieved:
        nodes = await retriever.aretrieve(ev.query)
        await ctx.store.set("query", ev.query)
        return Retrieved(nodes=nodes)

    @step
    async def synthesize(self, ctx: Context, ev: Retrieved) -> StopEvent:
        query = await ctx.store.get("query")
        answer = await synthesizer.asynthesize(query, nodes=ev.nodes)
        return StopEvent(result=str(answer))

flow = RagFlow(timeout=120)
result = await flow.run(query="What changed in the policy?")

Three mechanics to internalise. The kwargs you pass to run() become fields on the StartEvent, reachable as ev.query or ev.get("query"). Branching is a union return type, -> Approved | Rejected; fan-out returns list[E] and fan-in accepts list[E], which has been supported since workflows 2.22. And validation runs before the workflow starts, which is why a typo in an annotation fails loudly rather than hanging:

WorkflowValidationError: The following events are produced but never consumed: Retrieved
WorkflowConfigurationError: At least one Event of type StopEvent must be returned by any step.

Graph checks also report unreachable steps and dead ends, with the hint that "Only StopEvent and InputRequiredEvent may be terminal". If a step genuinely sends events dynamically through ctx.send_event and the checker cannot see it, opt out narrowly with @step(skip_graph_checks=["reachability"]) rather than disabling validation wholesale.

State lives on ctx.store, and the API changed in 0.14. The old ctx.get / ctx.set / ctx.data are gone:

AttributeError: 'Context' object has no attribute 'get'

Use await ctx.store.get(key, default=…), await ctx.store.set(key, value), and for read-modify-write async with ctx.store.edit_state() as state: which takes a lock. Keep LLM calls out of that block — holding a lock across a ten-second network call is how a workflow deadlocks itself under concurrency. Since 2.23 state is keyed by run id plus namespace, so concurrent runs of the same workflow do not collide.

The timeout is the first thing everyone trips over:

workflows.errors.WorkflowTimeoutError: Operation timed out after 45.0 seconds.
Currently active steps: synthesize

A plain Workflow defaults to 45 seconds, which one slow LLM call will exceed. Agents default to no timeout. Set Workflow(timeout=300) or timeout=None deliberately. Note the 2.25.0 change: the timeout now measures accumulated alive time across resumes rather than resetting each time, so a long-running suspended workflow that resumes repeatedly can now time out where it previously would not.

Concurrency has two knobs: @step(num_workers=4) for how many instances of one step run at once, and Workflow(num_concurrent_runs=N) for whole runs. The default runtime is in-process, so horizontal scale needs either a durable runtime or your own queue — a senior-level topic, but worth knowing the boundary exists before you design around it.

Agents: tools, events, and the loop's limits

An agent is a loop: send the message, the history and the tool schemas to the LLM; the LLM either answers or asks for tool calls; execute them, append the results, repeat. Everything that goes wrong with agents goes wrong somewhere in that loop, and the two places you can actually influence are the tool schemas and the stopping conditions.

The tool schema is written by your function signature and docstring. This is not a convenience, it is the entire interface the model sees:

from typing import Annotated
from llama_index.core.tools import FunctionTool

async def search_contracts(
    query: Annotated[str, "Natural-language description of the clause to find"],
    year: Annotated[int, "Four-digit contract year, e.g. 2026"],
) -> str:
    """Search signed customer contracts for a clause. Use for questions about
    contractual terms. Do not use for invoices or purchase orders."""
    ...

tool = FunctionTool.from_defaults(search_contracts, return_direct=False)
print(tool.metadata.get_parameters_dict())

That last line is the debugging move most people never learn. If an agent keeps calling a tool with nonsense arguments, print the schema; nine times out of ten a parameter has no description, or the docstring says what the function does but not when to use it. Say what the tool is not for — negative guidance cuts misrouting more than any amount of positive description.

QueryEngineTool.from_defaults(query_engine, name=…, description=…) wraps a whole RAG pipeline as a tool, which is how you give one agent several corpora. ToolSpecs bundle a service, e.g. GmailToolSpec().to_tool_list(). And return_direct=True ends the loop immediately with the tool's output as the answer, which is right for a tool whose output is already the answer and which also caps your blast radius: the model gets no second turn to act on what came back.

Streaming is how you see inside a run. agent.run() returns a handler, not a coroutine result:

from llama_index.core.agent.workflow import AgentStream, ToolCall, ToolCallResult

handler = agent.run(user_msg="Summarise the Riyadh contract renewals", ctx=ctx)
async for ev in handler.stream_events():
    if isinstance(ev, AgentStream):
        print(ev.delta, end="", flush=True)
    elif isinstance(ev, ToolCall):
        print(f"\n→ {ev.tool_name}({ev.tool_kwargs})")
    elif isinstance(ev, ToolCallResult):
        print(f"\n← {ev.tool_output}")
result = await handler

The event classes are AgentInput, AgentSetup, AgentStream, AgentOutput, ToolCall, ToolCallResult and AgentStreamStructuredOutput. AgentStream also carries thinking_delta for reasoning models, added in 0.14.1.

Stopping conditions. The iteration cap is 20 by default, and hitting it tells you exactly what to do:

WorkflowRuntimeError: Max iterations of 20 reached! Either something went wrong, or you can
increase the max iterations with `.run(.., max_iterations=...)` or use
`early_stopping_method='generate'` to generate a final response instead.

Resist raising the cap as a reflex. Twenty iterations is a lot of thinking; a loop that long is nearly always a tool returning something the model cannot use, or two tools whose descriptions overlap enough that it oscillates between them. early_stopping_method defaults to "force", which raises. "generate" asks the model for a best-effort final answer instead, which is kinder in a user-facing product and was added in 0.14.13. As of 0.14.25, a tool that raises is no longer retried, which stops one broken tool from burning the whole budget.

Memory is the other half of a conversational agent. Memory.from_defaults gives short-term FIFO history backed by SQLite in memory, plus optional long-term blocks:

from llama_index.core.memory import Memory, StaticMemoryBlock, FactExtractionMemoryBlock

memory = Memory.from_defaults(
    session_id=f"tenant-{tenant_id}-user-{user_id}",
    token_limit=40000,
    memory_blocks=[
        StaticMemoryBlock(name="core", static_content="The user works in Cairo.", priority=0),
        FactExtractionMemoryBlock(name="facts", llm=llm, max_facts=50, priority=1),
    ],
    insert_method="system",
    async_database_uri="postgresql+asyncpg://user:pass@host:5432/chat",
)
response = await agent.run("What did we decide last week?", memory=memory)

Defaults are token_limit=30000, chat_history_token_ratio=0.7 and token_flush_size=3000. Block priority=0 is never truncated; higher numbers are dropped first. The session_id is your isolation boundary for conversation history, so build it from authenticated identity, never from anything the user can set.

Two notes. ChatMemoryBuffer, ChatSummaryMemoryBuffer, VectorMemory and SimpleComposableMemory are documented as deprecated in favour of Memory, but in 0.14.25 agents still construct a ChatMemoryBuffer by default when you pass no memory, and it emits no warning — so "I didn't see a deprecation warning" does not mean you are on the new path. And a custom Memory object is not serializable, so it must be passed at runtime rather than stored in a context snapshot; for human-in-the-loop you pass both ctx and memory.

Multi-agent, and the cheaper patterns that usually win

AgentWorkflow gives you hand-offs between named agents:

from llama_index.core.agent.workflow import AgentWorkflow

workflow = AgentWorkflow(
    agents=[research, write, review],
    root_agent=research.name,
    initial_state={"research_notes": {}, "report_content": "Not written yet."},
)
result = await workflow.run(user_msg="Draft a one-page brief on the new tax rules")

Each agent declares can_handoff_to, and shared state is edited the same way as any workflow state:

async with ctx.store.edit_state() as state:
    state["research_notes"]["tax"] = summary

The constructor validates hard, and the messages are self-explanatory once you have seen them: At least one agent must be provided, All agents must have a name in a multi-agent workflow, Exactly one root agent must be provided, Root agent X not found in provided agents, Must provide either user_msg or chat_history, Handoff prompt must contain {agent_info}, State prompt must contain {state} and {msg}.

The honest guidance is that hand-off swarms are the most expensive and least predictable of the three documented patterns, and they are the one people reach for first. Every hand-off is extra LLM calls, and control flow decided by a model is control flow you cannot test. The alternatives:

Pattern two sits in between: one orchestrator agent with the sub-agents exposed as tools. You keep a single loop and a single memory, and you get the routing behaviour without the hand-off protocol.

Structured output is often what people actually want from "multi-agent", and it is cheaper. Pass output_cls=MyPydanticModel (honoured inside AgentWorkflow since 0.14.24) or structured_output_fn, or go one level down with llm.structured_predict(Model, prompt) — which since 0.14.19 raises ValueError on failure rather than returning something half-parsed. A deterministic flow over structured outputs beats a swarm over prose for most business tasks.

If you are doing serious multi-agent orchestration, read the LangGraph guide alongside this one: its explicit graph model makes a different trade-off, and knowing both makes the choice an argument rather than a habit.

Routing, and the questions one index cannot answer

There is a class of question a single VectorStoreIndex will always answer badly, and recognising it stops you tuning chunk size forever in search of a fix that does not exist. "What does the leave policy say about unpaid leave?" is a lookup, and vector similarity is the right tool. "Summarise the themes across all forty incident reports from last quarter" is not a lookup at all — the answer is not in any two chunks, so retrieving the nearest k of them cannot produce it. No value of similarity_top_k makes that work, because top-k retrieval is the wrong shape for a question whose answer is the whole corpus.

This is why VectorStoreIndex is only the default rather than the only index. SummaryIndex keeps every node and synthesises over all of them, which is exactly right for exhaustive summarisation over a small, bounded set. DocumentSummaryIndex stores an LLM-written summary per document and retrieves at document granularity, so the first hop narrows forty reports to four before any chunk is read. PropertyGraphIndex serves questions about relationships between entities rather than similarity between passages. KeywordTableIndex and TreeIndex cover the remaining shapes.

The production-RAG guidance in the docs names the same idea from the other direction: for a large corpus, use structured retrieval — metadata filters, or document summaries — rather than one flat similarity search, and route dynamically by task rather than sending everything down one path. Flat search over a hundred thousand chunks is not a search problem, it is a filtering problem wearing a search problem's clothes.

The simplest way to route is to make each index a tool and let one agent choose:

from llama_index.core.tools import QueryEngineTool
from llama_index.core.agent.workflow import FunctionAgent

lookup = QueryEngineTool.from_defaults(
    vector_index.as_query_engine(similarity_top_k=5),
    name="policy_lookup",
    description=(
        "Answers specific factual questions about a single HR policy clause: "
        "notice periods, entitlements, named rules. Not for comparisons across documents."
    ),
)
overview = QueryEngineTool.from_defaults(
    summary_index.as_query_engine(response_mode="tree_summarize"),
    name="policy_overview",
    description=(
        "Summarises or compares across the whole policy set. Use for 'what are the themes', "
        "'how do these differ', 'give me an overview'. Slower and more expensive."
    ),
)

router = FunctionAgent(
    tools=[lookup, overview],
    llm=llm,
    system_prompt="Pick exactly one tool. Prefer policy_lookup unless the question spans documents.",
)

Everything that makes this work or fail is in those two descriptions. The model has no other information about what the tools do, so each description must say what the tool is for, what it is not for, and what it costs. Write them as if for a new colleague who will be fired for guessing, then watch the ToolCall events on real questions and rewrite whichever description lost the argument.

Routing has a second payoff that is easy to miss: it is a cost lever. The expensive path runs only when it is asked for, instead of on every question. Pair it with per-tool model choices — a cheap model behind the lookup, a stronger one behind the summary — and you have spent your budget where it changes answers.

Configuration, Settings, and the multi-tenant trap

Settings is a global singleton holding llm, embed_model, node_parser / text_splitter, chunk_size, chunk_overlap, transformations, tokenizer, callback_manager, context_window and num_output. Local arguments always override it.

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small", embed_batch_size=100)
Settings.chunk_size = 512
Settings.chunk_overlap = 50

Set it once at startup, from configuration rather than inline literals, and set both the LLM and the embedding model. Setting only one is the top source of this error:

ValueError:
******
Could not load OpenAI model. If you intended to use OpenAI, please check your OPENAI_API_KEY.

Anything unset falls back to OpenAI — OpenAI() resolving to gpt-3.5-turbo, and OpenAIEmbedding() to text-embedding-ada-002. So a service configured for Ollama but with no embed model set will still try to reach OpenAI the first time it embeds, and will fail in whatever environment has no key. The embedding variant of the message even suggests embed_model='local'.

Two sharp edges. ServiceContext still imports but raises on use: ValueError: ServiceContext is deprecated. Use llama_index.settings.Settings instead…. And the Settings documentation page has a typo, Settings.tokenzier; the real attribute is Settings.tokenizer, a callable from str to a list.

There is also no supported CLI any more. llama-index-cli is deprecated and unmaintained, and on core 0.14.25 llamaindex-cli --help prints a DeprecationWarning and then crashes with ModuleNotFoundError: No module named 'llama_index.core.download'. Anything built on download_llama_pack or download_loader is dead code. The one CLI worth knowing is the workflow server launcher.

Pinning. Every integration pins llama-index-core<0.15.0, so 0.15 is the next breaking boundary and it will require every integration to be re-released. Pin accordingly and upgrade minors deliberately:

llama-index-core>=0.14.25,<0.15
llama-index-workflows>=2.25.0,<3
llama-index-llms-openai>=0.8.2,<0.9
llama-index-embeddings-openai>=0.6.0,<0.7
llama-index-readers-file>=0.5
llama-index-vector-stores-qdrant>=0.8

Patch releases inside 0.14.x are meant to be non-breaking, and they carry security fixes — RestrictedUnpickler for unsafe pickle deserialization in 0.14.16, an nltk vulnerability in 0.14.20, and a broad sweep in 0.14.25. Take them. Minors of llama-index-workflows do change behaviour, and the changelog is where you find out: 2.23.1 made every retry delay one step shorter by fixing an off-by-one, and 2.25.0 changed timeout semantics.

Observability: spans, events, and OpenTelemetry

A RAG answer is produced by an embedding call, a vector search, zero or more rerank calls and one or more LLM calls. When it is slow or wrong you need to see which. Print statements do not scale to that, and the framework has a proper answer.

The current mechanism is the instrumentation module — Dispatcher, Event, EventHandler, Span, SpanHandler — which superseded the legacy CallbackManager and set_global_handler from 0.10.20. The legacy path still works and many partner integrations still use it, so you will meet both.

The pragmatic production setup is OpenTelemetry, because it lands in the tracing backend you already run:

from llama_index.observability.otel import LlamaIndexOpenTelemetry
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

otel = LlamaIndexOpenTelemetry(
    span_exporter=OTLPSpanExporter("http://otel-collector:4318/v1/traces"),
    service_name_or_resource="rag-api",
)
otel.start_registering()

That captures workflow steps, LLM calls and custom events as spans. From there the OpenTelemetry and Grafana guides cover the collector and the dashboards.

If you want LLM-specific views — prompt and completion side by side, token costs, evaluation scores attached to traces — the one-line handlers remain the fastest route, and Langfuse and Arize Phoenix both have guides here:

from llama_index.core import set_global_handler
set_global_handler("langfuse")

For your own spans and events, get a dispatcher and decorate:

import llama_index.core.instrumentation as instrument

dispatcher = instrument.get_dispatcher(__name__)

@dispatcher.span
def rerank_and_answer(query: str) -> str:
    ...

Subclass BaseEventHandler with a handle(event) method and register it with dispatcher.add_event_handler(...); LLM lifecycle events include LLMChatStartEvent, LLMChatInProgressEvent and LLMChatEndEvent. For BaseSpanHandler the hooks are new_span, prepare_to_exit_span and prepare_to_drop_span — that last one is where you record failures. In a standalone workflows app with no core installed, import get_dispatcher from llama_index_instrumentation.

Workflow-level debugging has its own tools. Workflow(verbose=True) prints each step as it runs. More useful in anger:

handler = flow.run(query="…")
async for ev in handler.stream_events(expose_internal=True):
    print(type(ev).__name__, getattr(ev, "name", ""), getattr(ev, "step_state", ""))

expose_internal=True yields StepStateChanged events carrying name, worker_id, input_event_name, output_event_name and a StepState of PREPARING, RUNNING or NOT_RUNNING. That is how you find the step that is actually slow, as opposed to the step you assumed was slow.

Testing and cost control before production

LLM code has a reputation for being untestable, which is mostly a failure to separate the deterministic parts from the model. Three layers of test cover most of what breaks.

Deterministic unit tests, no model involved. Chunking, metadata, tool schemas and workflow graphs are all pure logic:

def test_chunks_fit_budget():
    nodes = SentenceSplitter(chunk_size=512, chunk_overlap=50).get_nodes_from_documents(docs)
    assert all(len(n.get_content(metadata_mode="embed")) < 4000 for n in nodes)
    assert all(n.metadata["tenant_id"] for n in nodes)

def test_tool_schema_documents_every_arg():
    params = search_contracts_tool.metadata.get_parameters_dict()
    for name, spec in params["properties"].items():
        assert spec.get("description"), f"{name} has no description"

def test_workflow_graph_is_valid():
    RagFlow().validate()

That last line is free insurance in CI: workflow validation catches a broken annotation before a deploy does.

Mocked LLM tests. MockLLM and MockEmbedding let you exercise the full path without a network call, and MockFunctionCallingLLM (added in the 0.14.10–0.14.12 window) does the same for agent tool-calling:

from llama_index.core.llms import MockLLM
from llama_index.core import MockEmbedding

index = VectorStoreIndex.from_documents(
    docs, embed_model=MockEmbedding(embed_dim=1536)
)
answer = index.as_query_engine(llm=MockLLM(max_tokens=64)).query("anything")

These also give you a cost dry run. Pair them with token counting and you can price an ingestion before you pay for it:

import tiktoken
from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler

counter = TokenCountingHandler(tokenizer=tiktoken.encoding_for_model("gpt-4o-mini").encode)
Settings.callback_manager = CallbackManager([counter])
# ... run your pipeline ...
print(counter.total_embedding_token_count, counter.total_llm_token_count)
print(counter.prompt_llm_token_count, counter.completion_llm_token_count)

TokenCountingHandler is a legacy callback and still the most practical token counter in the library.

Evaluation, which is the only test that measures quality. The framework ships response evaluators — Correctness, SemanticSimilarity, Faithfulness, ContextRelevancy, AnswerRelevancy and Guideline — plus retrieval metrics like MRR, hit-rate and precision, and synthetic question generation to build a dataset from your own corpus. Faithfulness and ContextRelevancy are the two to start with, because they separate the failure modes: ContextRelevancy says whether retrieval found the right text, Faithfulness says whether the answer stuck to it. Run them on a fixed set of 50 real questions, in CI, as a nightly job rather than on every pull request, since they cost tokens. Community integrations include RAGAS and DeepEval, both of which have guides here.

Rate limits and spend caps belong in the same conversation, because the fastest way to break production is to succeed at ingestion:

from llama_index.core.rate_limiter import TokenBucketRateLimiter

limiter = TokenBucketRateLimiter(requests_per_minute=30, tokens_per_minute=100_000)
llm = OpenAI(model="gpt-4o-mini", rate_limiter=limiter)
embed = OpenAIEmbedding(model="text-embedding-3-small", rate_limiter=limiter)

Both objects share one limiter, so one global budget is enforced across LLM and embedding traffic. SlidingWindowRateLimiter is the other shape. Added in 0.14.16, these replace a lot of hand-rolled retry code — though workflow retry policies with exponential jitter are still the right answer for transient provider failures.

Deploying: FastAPI, WorkflowServer, and CI

The production shape the docs describe is a split, and it is worth adopting even for a small system because it separates two very different cost and latency profiles.

A minimal FastAPI service that gets the important parts right:

from contextlib import asynccontextmanager
from fastapi import FastAPI, Depends
from llama_index.core import VectorStoreIndex
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

state = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    state["index"] = VectorStoreIndex.from_vector_store(make_vector_store())
    state["llm"] = OpenAI(model="gpt-4o-mini", rate_limiter=limiter)
    yield

app = FastAPI(lifespan=lifespan)

@app.post("/ask")
async def ask(question: str, tenant: str = Depends(tenant_from_token)):
    engine = state["index"].as_query_engine(
        llm=state["llm"],
        similarity_top_k=5,
        filters=MetadataFilters(filters=[MetadataFilter(key="tenant_id", value=tenant)]),
    )
    response = await engine.aquery(question)
    return {
        "answer": str(response),
        "sources": [sn.node.metadata.get("file_name") for sn in response.source_nodes],
    }

Four things to notice. The index is built once in lifespan, not per request. The query engine is constructed per request, because it carries the tenant filter. The tenant comes from the token, not the body. And it uses aquery, because everything in modern LlamaIndex is async-first and a synchronous query() inside an async handler blocks the event loop for the entire LLM call — which is how a service with plenty of CPU headroom still times out under ten concurrent users.

The async-first design produces one error so often it deserves naming:

RuntimeError: asyncio.run() cannot be called from a running event loop

In a notebook, just await. In a script, have exactly one asyncio.run(main()) entry point. In FastAPI, use async def handlers and await.

The alternative to writing the HTTP layer yourself is llama-agents-server:

from llama_agents.server import WorkflowServer, SqliteWorkflowStore

server = WorkflowServer(workflow_store=SqliteWorkflowStore(db_path="workflows.db"))
server.add_workflow("rag", RagFlow(timeout=300))

Run it with python -m llama_agents.server my_server.py, which binds 0.0.0.0:8080 by default and takes WORKFLOWS_PY_SERVER_HOST and WORKFLOWS_PY_SERVER_PORT. You get POST /workflows/{name}/run, POST /workflows/{name}/run-nowait returning a handler_id, GET /handlers/{handler_id} (202 while running, 200 when done, 500 on failure), GET /results/{handler_id}, an SSE stream at GET /events/{handler_id}?sse=true, POST /events/{handler_id} to send an event into a running workflow, cancellation, and a debugger UI at /. Three caveats: the default MemoryWorkflowStore loses everything on restart, so use SqliteWorkflowStore or better; there is one reader per run's event stream and streamed events cannot be replayed; and there is no authentication, so it must sit behind your own gateway.

For containers, the usual Python image rules apply (Docker covers them properly), with two LlamaIndex-specific notes. llama-index-core pre-bundles its NLTK and tiktoken data specifically to avoid downloads at runtime, so a locked-down container does not need network access for tokenisation. And if you use HuggingFaceEmbedding, bake the model into the image or mount a cache volume, or every pod downloads it on first request and your readiness probe lies to you.

In CI, the cheap and high-value checks are: install with your lockfile, import the integrations you depend on, run workflow.validate() on every workflow, run the mocked tests, and run the evaluation suite on a schedule rather than per commit. GitHub Actions and Argo CD cover the pipeline and the deploy.

One regional note that genuinely matters. A RAG system sends document text to a model provider at query time, and that is a cross-border data transfer. If you are serving a bank or a government entity in the Gulf, check which region your LLM and embedding endpoints actually resolve to before you design the system, not after. Azure OpenAI in UAE North, Bedrock in Bahrain, and a self-hosted model via vLLM or Ollama are all reachable from LlamaIndex through swapping two lines in Settings, which is exactly the kind of decision this abstraction is for — provided you keep provider choice in configuration rather than scattered through the code.

Putting it all together

Here is a small system that uses most of what this level covered: an ingestion job you can re-run safely, and a query workflow with filters, reranking and an explicit timeout.

import asyncio
from pathlib import Path
from llama_index.core import SimpleDirectoryReader
from llama_index.core.ingestion import IngestionPipeline, IngestionCache
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.rate_limiter import TokenBucketRateLimiter

limiter = TokenBucketRateLimiter(requests_per_minute=60, tokens_per_minute=200_000)

def tag(path: str) -> dict:
    p = Path(path)
    return {"tenant_id": p.parts[1], "department": p.parent.name}

async def main() -> None:
    docs = SimpleDirectoryReader(
        input_dir="corpus", recursive=True,
        required_exts=[".pdf", ".md", ".docx"], file_metadata=tag,
    ).load_data()

    for d in docs:
        d.excluded_embed_metadata_keys = [
            "file_size", "creation_date", "last_modified_date", "last_accessed_date",
        ]

    pipeline = IngestionPipeline(
        transformations=[
            SentenceSplitter(chunk_size=512, chunk_overlap=50),
            OpenAIEmbedding(model="text-embedding-3-small",
                            embed_batch_size=100, rate_limiter=limiter),
        ],
        vector_store=make_vector_store(),
        docstore=SimpleDocumentStore(),
        cache=IngestionCache(),
    )
    nodes = await pipeline.arun(documents=docs)
    pipeline.persist("./pipeline_storage")
    print(f"{len(nodes)} nodes upserted")

if __name__ == "__main__":
    asyncio.run(main())

from workflows import Workflow, Context, step
from workflows.events import Event, StartEvent, StopEvent
from llama_index.core import VectorStoreIndex
from llama_index.core.postprocessor import LLMRerank
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

class Retrieved(Event):
    nodes: list

class AnswerFlow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent) -> Retrieved:
        index = VectorStoreIndex.from_vector_store(make_vector_store())
        retriever = index.as_retriever(
            similarity_top_k=20,
            filters=MetadataFilters(filters=[
                MetadataFilter(key="tenant_id", value=ev.tenant),
            ]),
        )
        nodes = await retriever.aretrieve(ev.query)
        await ctx.store.set("query", ev.query)
        ctx.write_event_to_stream(Retrieved(nodes=nodes))
        return Retrieved(nodes=nodes)

    @step
    async def answer(self, ctx: Context, ev: Retrieved) -> StopEvent:
        query = await ctx.store.get("query")
        kept = LLMRerank(top_n=5).postprocess_nodes(ev.nodes, query_str=query)
        answer = await synthesizer.asynthesize(query, nodes=kept)
        return StopEvent(result={
            "answer": str(answer),
            "sources": [n.node.metadata.get("file_name") for n in kept],
        })

flow = AnswerFlow(timeout=180)

Read what this buys you. Re-running ingest.py costs almost nothing if nothing changed, because the cache and docstore make it idempotent. The query path never rebuilds the index, the tenant filter is a hard constraint built from a server-side argument rather than a prompt, reranking buys precision, the sources come back so a user can check the answer, and the timeout is set deliberately rather than inherited from a 45-second default. Every one of those is a decision this level taught you to make on purpose. One detail worth noting: postprocess_nodes takes query_str or query_bundle, never both.

Where you are now

You can predict what a LlamaIndex query will do, which is the real difference between the beginner and mid levels. Given a bad answer you know whether to look at retrieval or synthesis, and you have a one-line diagnostic for the question. You know chunk size sets a ceiling on quality, that metadata eats into the chunk budget, and that retrieval chunks and synthesis chunks do not have to be the same chunks. You can write an ingestion job that is safe to re-run, and you know which of the four stores a hosted vector database makes redundant.

You know that agents are workflows, which turns agent debugging from guesswork into reading an event stream. You can read a validation error, a timeout and an iteration cap and know what each is telling you. You know the tool schema is your real interface to the model, and how to print it. You know why Settings must not be mutated per request, and why a multi-tenant filter must never be a tool argument.

What is left for senior level is ownership rather than usage: failure modes and which component gives way first under load; durability, context snapshots and at-least-once resume semantics; the sandbox a CodeActAgent needs before it is allowed near production; prompt injection through retrieved documents and tool output; per-tenant cost attribution; running the 0.15 upgrade for several teams at once; and the question of where LlamaIndex stops being the right tool. Before that, the practice worth doing is at mid-tips in this guide: wire the observability, write the evaluation set, and read your own traces for a week.

Sources

• LlamaIndex Python framework documentation
• Installation
• High-level concepts
• Using async with LlamaIndex
• Settings
• SimpleDirectoryReader
• Ingestion pipeline
• Vector store index
• Storing
• Persisting and loading data
• Vector stores
• Node postprocessors
• Agents
• Tools
• Memory
• Chat engines
• Multi-agent systems
• Evaluating
• Observability
• Instrumentation
• Production RAG
• Workflows
• Managing state in workflows
• Streaming events
• Retrying steps
• Workflow deployment
• llama_index CHANGELOG
• llama-index-core on PyPI
• llama-index-workflows on PyPI

At mid level, interviewers stop asking what LlamaIndex is. They describe a system that is behaving badly and watch how you narrow it down. The questions below are the ones that actually come up, with the reasoning a strong answer shows. Everything is current for llama-index-core 0.14.25 and llama-index-workflows 2.25.0.

Scenario: "The RAG answers are wrong. Where do you look first?"

The answer that lands: before anything else, print the retrieved nodes. One diagnostic splits the whole problem space in two.

response = query_engine.query(question)
for sn in response.source_nodes:
    print(f"{sn.score:.3f}  {sn.node.metadata.get('file_name')}  {sn.node.text[:150]!r}")

If the text that contains the answer is not in that list, it is a retrieval problem, and the candidate fixes are similarity_top_k (the default is 2, which is tiny), chunk size, the embedding model, a metadata filter that is excluding too much, or a similarity_cutoff that is throwing away good nodes. If the right text is there and the answer is still wrong, it is a synthesis problem, and the fixes are a different response_mode, a stronger LLM on that call, or a custom prompt.

What a weak answer does: jumps straight to prompt engineering, or offers "use a better model" without establishing which half is broken. Spending a day rewriting prompts to fix a retrieval miss is the single most common waste of time in this field, and interviewers ask this question specifically to find out whether you do it.

Mention one extra thing to show production experience: a similarity cutoff is not portable between embedding models. Scores from different embedding families are not on the same scale, so a cutoff tuned for one will silently filter everything for another, and the symptom is a system that confidently answers "I don't know" after a model swap.

Scenario: "Ingestion takes forty minutes every night, and nothing changed"

Two mechanisms fix this and they are different, so say both.

The IngestionCache is keyed on (node, transformation). If the same node has already been through the same transformation, the stored output is reused and you do not pay for the embedding twice. Default is in-memory, which is useless across runs, so point it at Redis, Mongo or Firestore so a nightly job or a CI container shares it.

The docstore is keyed on doc_id plus a content hash, and with a vector store also attached it gives you upsert: unchanged documents are skipped, changed ones are re-processed and their old nodes replaced. Without a vector store attached, the pipeline can only de-duplicate its inputs, because it does not own anything it could delete.

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=512, chunk_overlap=50), OpenAIEmbedding()],
    vector_store=vector_store,
    docstore=SimpleDocumentStore(),
    cache=IngestionCache(cache=RedisKVStore.from_host_and_port("127.0.0.1", 6379)),
)

Then the throughput levers: embed_batch_size on the embedding model, which decides how many texts go per API request, and await pipeline.arun(...) for I/O-bound work. run(num_workers=N) uses multiprocessing.Pool, needs the if __name__ == "__main__": guard, and the docs warn that Windows sees little or no gain.

What a weak answer misses: that an embedding model must be in transformations explicitly when a vector store is attached. It is not implicit the way it is in from_documents, and leaving it out gives you a pipeline that writes nothing.

Scenario: "A pod restart takes four minutes and costs money"

This is almost always VectorStoreIndex.from_documents(...) running at service startup. Rebuilding the index re-embeds the entire corpus on every deploy and every crash loop.

index = VectorStoreIndex.from_vector_store(vector_store)

That binds a query path onto a collection that already exists: no reads, no embedding calls, effectively instant. Say why it works — most hosted stores (Qdrant, pgvector, Pinecone, Weaviate, Milvus, Redis, OpenSearch) keep the text alongside the vectors, so the docstore is redundant for retrieval and persist() is unnecessary.

Name the exception, because it shows you have actually used more than one store: FAISS stores vectors only, not text, so a FAISS setup still needs a docstore persisted next to it or your retrieved nodes come back with empty content.

Finish with where the index should be built: once, in FastAPI's lifespan, never per request and never at module import. The per-request object is the query engine, because that is what carries the tenant filter.

Scenario: "The agent loops twenty times and then throws"

The error tells you what it is:

WorkflowRuntimeError: Max iterations of 20 reached! Either something went wrong, or you can
increase the max iterations with `.run(.., max_iterations=...)` or use
`early_stopping_method='generate'` to generate a final response instead.

A strong answer refuses to raise the cap as the first move. Twenty iterations is a great deal of thinking for most tasks, so the loop is nearly always a symptom: a tool returning something the model cannot use, an error string the model keeps retrying, or two tools whose descriptions overlap enough that it oscillates between them.

The way to find out is the event stream:

handler = agent.run(user_msg=question)
async for ev in handler.stream_events():
    if isinstance(ev, ToolCall):
        print(ev.tool_name, ev.tool_kwargs)
    elif isinstance(ev, ToolCallResult):
        print(ev.tool_output)
result = await handler

Read the tool calls. If the arguments are nonsense, print tool.metadata.get_parameters_dict() — nine times out of ten a parameter has no description, or the docstring says what the function does but not when to use it. Say what the tool is not for; negative guidance cuts misrouting more than any amount of positive description.

Then the two legitimate configuration answers. early_stopping_method defaults to "force", which raises; "generate" asks the model for a best-effort final answer, which is kinder in a user-facing product. And as of 0.14.25 a tool that raises is no longer retried, so one broken tool no longer burns the whole budget.

Mention the stream's one rule, because it catches everyone: it is consumable exactly once. Trying to iterate it twice — once for a websocket, once for a log — gives you WorkflowRuntimeError: All the streamed events have already been consumed. Consume once and fan out yourself.

Scenario: "Tenant A can see tenant B's documents"

Say the hard part first: LlamaIndex has no built-in tenancy, authentication or authorisation. It is a library. Isolation is entirely yours to build, and there are three places it has to hold.

Retrieval is enforced with a metadata filter applied by the vector store before ranking, so it is a hard constraint rather than a hint:

retriever = index.as_retriever(
    similarity_top_k=5,
    filters=MetadataFilters(filters=[MetadataFilter(key="tenant_id", value=tenant)]),
)

MetadataFilters.condition defaults to AND when it is None. The alternative, a separate collection or namespace per tenant, isolates harder and costs more to operate.

Conversation history is isolated with Memory(session_id=...) built from authenticated identity. Workflow state is per-run Context, and since workflows 2.23 state is keyed by run id plus namespace.

The answer that distinguishes a senior-leaning candidate: never let the model choose the tenant filter. If you give an agent a retrieval tool with a tenant argument, your isolation boundary is a prompt, and prompts can be talked around by anything in the conversation or in a retrieved document. Build the filter server-side from the session and keep the field out of the tool schema entirely. Add that retrieved documents and tool outputs are untrusted input generally, which is the prompt-injection point.

Scenario: "Users say it can't summarise across documents"

This is the question that catches candidates who only know one index. A strong answer separates two shapes of question before suggesting any fix.

"What does the leave policy say about unpaid leave?" is a lookup, and top-k vector similarity is exactly right. "Summarise the themes across all forty incident reports from last quarter" is not a lookup at all — the answer is not contained in any two chunks, so retrieving the nearest k of them cannot produce it. No value of similarity_top_k fixes this, and saying so is the point of the answer. Candidates who propose raising top-k to 50 have missed that the retrieval shape is wrong, not the retrieval size.

The options, roughly in order of cost: SummaryIndex keeps every node and synthesises over all of them, which is right for exhaustive summarisation over a bounded set; DocumentSummaryIndex stores an LLM-written summary per document and retrieves at document granularity, so a first hop narrows forty reports to four before any chunk is read; PropertyGraphIndex handles questions about relationships between entities rather than similarity between passages.

Then say how you would ship it, which is routing: wrap each index as a QueryEngineTool and let one agent pick, with each description stating what the tool is for, what it is not for, and that the summary path is slower and more expensive. That turns routing into a cost lever too — the expensive path runs only when it is asked for.

What a weak answer misses: that VectorStoreIndex is a default, not the only index, and that the docs' production-RAG guidance explicitly recommends structured retrieval and dynamic routing for large corpora rather than one flat similarity search.

Scenario: "How would you test any of this?"

Interviewers ask this because LLM code has a reputation for being untestable, and they want to see whether you accept that excuse. Describe three layers.

Deterministic tests, no model involved. Chunking, metadata, tool schemas and workflow graphs are pure logic. Assert that chunks fit a token budget, that every node carries the metadata keys your filters depend on, that every tool parameter has a non-empty description, and run MyFlow().validate() — free insurance that catches a broken annotation before a deploy does.

Mocked-model tests. MockLLM and MockEmbedding exercise the full path with no network call, and MockFunctionCallingLLM does the same for agent tool-calling. These double as a cost dry run: pair them with a TokenCountingHandler and you can price an ingestion before you pay for it, reading total_embedding_token_count and total_llm_token_count.

Evaluation, which is the only layer that measures quality. The framework ships Correctness, SemanticSimilarity, Faithfulness, ContextRelevancy, AnswerRelevancy and Guideline evaluators, plus retrieval metrics like MRR and hit-rate, and synthetic question generation to build a dataset from your own corpus. Name Faithfulness and ContextRelevancy specifically and say why: ContextRelevancy tells you whether retrieval found the right text, Faithfulness whether the answer stuck to it — the same split as the first question in this file, now automated. Run them nightly on a fixed set of real questions rather than per commit, because they cost tokens.

Finish with the operational point that makes this credible: the evaluation set is how you tell a retrieval regression from a synthesis regression after someone changes a chunk size, and without it every model or embedding swap is a guess.

Trade-off: multi-agent swarm or a workflow you wrote?

Interviewers ask this to see whether you reach for the most impressive option or the right one.

AgentWorkflow with hand-offs is the most expensive and least predictable of the three documented patterns. Every hand-off is extra LLM calls, and control flow decided by a model is control flow you cannot unit-test. The other two are an orchestrator agent with sub-agents exposed as tools — one loop, one memory, you still get routing — and a custom Workflow where you encode the sequence because you already know it, with one agent per step.

The test to state out loud: if you can write the steps down, write them down. Put the LLM where it is genuinely better than code — wording, extraction, judgement, classification — and keep sequencing in code, where branches are union return types you can test. Swarms earn their cost when the task shape genuinely varies per request.

Add that structured output is often what people actually want from "multi-agent" and is far cheaper: output_cls on an agent, honoured inside AgentWorkflow since 0.14.24, or llm.structured_predict(Model, prompt) one level down, which raises ValueError on failure since 0.14.19.

Trade-off: how do you pick a chunk size?

A strong answer refuses the premise. Small chunks embed precisely; large chunks give the LLM context. One size cannot do both, so the production answer is to decouple the chunk you retrieve from the chunk you synthesise over — sentence-window, or small-to-big — rather than hunting for a compromise.

Then show you know the numbers and the trap. Defaults are 1024 tokens with 200 overlap, measured with tiktoken cl100k. Metadata is prepended to node text before embedding, which is why you can hit ValueError: Metadata length (612) is longer than chunk size (512), and why excluded_embed_metadata_keys and excluded_llm_metadata_keys matter: keep volatile timestamps out of the embedding, keep file_name visible to the LLM so answers can cite sources.

Close with the operational consequence: changing chunking or the embedding model means re-indexing. Query an index with a different embedding model and you get ValueError: shapes (384,) and (1536,) not aligned, or the hosted store's own dimension error.

Trade-off: Settings or local arguments?

Settings is a process-wide singleton. Set it once at startup from configuration, and set both llm and embed_model, because anything unset falls back to OpenAI — gpt-3.5-turbo and text-embedding-ada-002 — and produces ValueError: Could not load OpenAI model… in whatever environment has no key. A local LLM with no embed model set is the classic version of this bug.

Never mutate Settings per request. In an async server, request A changing Settings.llm while request B is mid-retrieval gives B the wrong model, and the bug is intermittent and tenant-dependent. Pass models locally: as_query_engine(llm=…), from_documents(docs, embed_model=…), FunctionAgent(..., llm=…).

The questions behind the questions

Three framing points worth landing whenever they fit.

Agents are Workflows. FunctionAgent, ReActAgent, CodeActAgent and AgentWorkflow all live in llama_index.core.agent.workflow and run on the workflows engine. This is why agent debugging is reading an event stream, and why you can replace an agent with something deterministic. It also explains the big break: 0.13 removed FunctionCallingAgent, the old ReActAgent.from_tools style, AgentRunner, OpenAIAgent and QueryPipeline outright, which is why pre-2025 tutorials fail with ImportError and ModuleNotFoundError: No module named 'llama_index.core.query_pipeline'.

Know the two timeouts. A plain Workflow defaults to 45 seconds, which one slow LLM call will exceed, giving WorkflowTimeoutError: Operation timed out after 45.0 seconds. Currently active steps: …. Agents default to no timeout. Since workflows 2.25.0 the timeout measures accumulated alive time across resumes rather than resetting.

Know what pinning means here. Every integration pins llama-index-core<0.15.0, so 0.15 is the next breaking boundary and will require every integration to be re-released. Patch releases inside 0.14.x carry security fixes and should be taken. And there is no supported CLI: llama-index-cli is deprecated and crashes on 0.14.25, so anything built on download_llama_pack or download_loader is dead.

Sixty-second self-test

Answer out loud. Any hesitation is a page to re-read.

1. What is the default similarity_top_k, and why is it a problem?
2. Which stage of a query engine does response_mode configure, and which does similarity_top_k?
3. What two stores does an IngestionPipeline need before a re-run becomes cheap, and what is each keyed on?
4. Why is from_vector_store the right call in a service, and which vector store breaks that rule?
5. Where does a tool's description come from, and how do you print the schema the model sees?
6. Why must a tenant filter never be a tool argument?
7. What is the default Workflow timeout, and what does an agent default to?
8. What does AttributeError: 'Context' object has no attribute 'get' tell you, and what replaced it?
9. Name two reasons never to mutate Settings inside a request handler.
10. You changed the embedding model. What must you do, and what error appears if you do not?

These are the errors, habits and recipes that separate a LlamaIndex service that works on your laptop from one that works on a Tuesday afternoon with real users. Verified against llama-index-core 0.14.25 and llama-index-workflows 2.25.0.

The errors you will actually hit

ValueError: Could not load OpenAI model. If you intended to use OpenAI, please check your OPENAI_API_KEY. Symptom: a service configured for a local model still tries to reach OpenAI. Cause: anything unset in Settings falls back to OpenAI — OpenAI() resolves to gpt-3.5-turbo, OpenAIEmbedding() to text-embedding-ada-002. The usual version is a local LLM set but no embed model, or the reverse. Fix: set both Settings.llm and Settings.embed_model, or pass both locally. The embedding variant of the message even suggests embed_model='local'.

`llama-index-readers-file` package not found… Symptom: PDFs and DOCX files produce garbage or empty text. Cause: the umbrella llama-index package dropped llama-index-readers-file in 0.14.18, and the installation page has not caught up. Fix: pip install llama-index-readers-file. This one is nasty because it only logs — ingestion "succeeds" and your index is full of binary noise.

ImportError: cannot import name 'FunctionCallingAgent' from 'llama_index.core.agent' and ModuleNotFoundError: No module named 'llama_index.core.query_pipeline' Cause: you are following a tutorial written before 0.13, which removed the entire previous agent generation and QueryPipeline. Fix: FunctionAgent / ReActAgent from llama_index.core.agent.workflow, with await agent.run(...). Watch for the trap: ReActAgent still imports but resolves to the new class, so ReActAgent.from_tools(...) fails on an attribute rather than an import, which looks like a different bug.

AttributeError: 'Context' object has no attribute 'get' Cause: the 0.14 state API. Fix: await ctx.store.get(key, default=…), await ctx.store.set(key, value), and async with ctx.store.edit_state() as state: for read-modify-write.

WorkflowTimeoutError: Operation timed out after 45.0 seconds. Currently active steps: synthesize Cause: a plain Workflow defaults to 45 seconds. Agents default to no timeout. Fix: set it deliberately, Workflow(timeout=300) or timeout=None. Since workflows 2.25.0 the timeout counts accumulated alive time across resumes, so a long-suspended run that resumes repeatedly can now time out where it previously would not.

WorkflowValidationError: The following events are produced but never consumed: Retrieved Cause: a missing or wrong return annotation, or events sent only through ctx.send_event. Validation runs before the workflow starts, which is a feature. Related messages cover events consumed but never produced, unreachable steps, dead ends, and WorkflowConfigurationError: At least one Event of type StopEvent must be returned by any step. Fix: annotate returns; for genuinely dynamic steps opt out narrowly with @step(skip_graph_checks=["reachability"]) rather than disabling validation.

WorkflowRuntimeError: All the streamed events have already been consumed. Cause: iterating handler.stream_events() twice — typically once to feed a websocket and once to log. Fix: consume once and fan out yourself.

ValueError: shapes (384,) and (1536,) not aligned: 384 (dim 0) != 1536 (dim 0) Cause: you changed the embedding model without re-indexing. Fix: query with the model the index was built with, or rebuild the collection. Hosted stores raise their own dimension errors.

ValueError: Metadata length (612) is longer than chunk size (512). Cause: metadata is prepended to node text before embedding, and SimpleDirectoryReader adds seven keys before you add any. Fix: raise the chunk size, or trim with excluded_embed_metadata_keys. There is also a printed warning when fewer than 50 tokens of real content remain — worth grepping ingestion logs for.

ValueError: Calculated available context size -123 was not non-negative. Cause: context_window minus prompt minus num_output went below zero, usually an unknown model or a local LLM with a small window. Fix: set a correct context_window and num_output on the LLM or in Settings, or lower similarity_top_k or chunk_size.

RuntimeError: asyncio.run() cannot be called from a running event loop Cause: the library is async-first and you are in a notebook or an async framework. Fix: await directly in notebooks, exactly one asyncio.run(main()) in scripts, async def handlers in FastAPI.

ValueError: Expected to load a single index, but got 3 instead. Please specify index_id. Fix: index.set_index_id("vector_index") before persisting, then load_index_from_storage(sc, index_id="vector_index"). Its siblings are FileNotFoundError: … docstore.json for a wrong directory and ValueError: No index in storage context for an empty one.

ValueError: ServiceContext is deprecated. It still imports but raises on use. Use Settings or local arguments.

The practices that pay off most, in order of leverage

1. Reconnect, never rebuild. Put VectorStoreIndex.from_vector_store(vector_store) in FastAPI's lifespan and nowhere else. A service that calls from_documents at startup re-embeds the corpus on every deploy and every crash loop. This is the highest-value single line in the whole stack.

2. Print source_nodes before touching a prompt. It splits every quality problem into retrieval or synthesis, and the fixes have nothing in common. Make it a logged field in production, not a debugging habit: log the retrieved file names and scores with every answer and you can diagnose complaints from the logs alone.

3. Raise similarity_top_k off 2, then rerank. The default of two chunks is a very small window. Retrieve 20 and rerank to 5 with LLMRerank(top_n=5): one extra LLM call buys precision, and precision is usually what the user is complaining about.

4. Attach a docstore and a remote cache to every pipeline. It turns re-ingestion from a bill into a no-op, and it gives you upsert so a changed document replaces its old nodes rather than duplicating them.

5. Pass models locally, configure Settings once. Settings is a process-wide singleton. Mutating it per request is a genuine cross-request bug in an async server, and it is intermittent, which makes it expensive to find.

6. Build tenant filters server-side. From the authenticated session, as a MetadataFilters on a per-request retriever, with the tenant field absent from every tool schema. Isolation that depends on a prompt is not isolation.

7. Write tool descriptions for a reader who will be fired for guessing. Say what the tool is for, what it is not for, and what it costs. Then verify with tool.metadata.get_parameters_dict(). Most agent misbehaviour is a documentation bug.

8. Set every timeout and iteration cap explicitly. 45 seconds and 20 iterations are defaults, not decisions. Write them down in code so the next person knows they were chosen.

9. Share one rate limiter across LLM and embeddings. TokenBucketRateLimiter(requests_per_minute=30, tokens_per_minute=100_000) passed to both objects enforces one global budget, which is what your provider quota actually is.

10. Keep provider choice in configuration. Swapping Azure OpenAI in UAE North, Bedrock in Bahrain, or a self-hosted model for a residency requirement should be two lines in startup config, not a refactor. That portability is most of why you chose a framework.

Habits that stop problems rather than fixing them

Log the sources, always. An answer without its retrieved file names is an answer you cannot investigate. Put the names and scores in the same structured log line as the question, and user complaints become a query rather than a reproduction attempt. It also happens to be the thing that makes users trust the system, because the citation is right there.

Treat retrieved text as untrusted input. A document in your corpus is something a user uploaded, and its content reaches the model inside a prompt. That is the prompt-injection surface, and it is why a narrow toolset and return_direct=True on tools whose output is already the answer are security controls and not just tidiness — return_direct ends the loop, so the model gets no second turn to act on what came back.

Decide the chunking contract once, and write it down. Chunk size, overlap, splitter, which metadata keys are excluded from embeddings, and which embedding model. All five together define your index, and changing any of them means re-indexing. A comment above the splitter saying so saves a future colleague a confusing afternoon.

Prefer async everywhere. aquery, aretrieve, pipeline.arun, await agent.run. The library is async-first by design since the 0.13 agent rewrite, and mixing in synchronous calls inside an event loop produces latency problems that look like provider problems.

Keep one entry point for event loops. Exactly one asyncio.run(main()) in a script, plain await in notebooks, async def handlers in FastAPI. Everything else produces RuntimeError: asyncio.run() cannot be called from a running event loop eventually, usually in the environment you did not test.

Read the workflows changelog, not just the core one. Core patch releases are meant to be non-breaking. llama-index-workflows minors are where behaviour quietly changes, and no error message announces it.

Practice cards

Operational recipes

Reconnect-only startup.

from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app):
    state["index"] = VectorStoreIndex.from_vector_store(make_vector_store())
    state["llm"] = OpenAI(model="gpt-4o-mini", rate_limiter=limiter)
    yield

A per-request, tenant-scoped query engine.

engine = state["index"].as_query_engine(
    llm=state["llm"],
    similarity_top_k=20,
    node_postprocessors=[LLMRerank(top_n=5)],
    filters=MetadataFilters(filters=[MetadataFilter(key="tenant_id", value=tenant)]),
)
response = await engine.aquery(question)

Use aquery, not query. A synchronous call inside an async handler blocks the event loop for the whole LLM call, which is how a service with idle CPU still times out under ten concurrent users.

Metadata at ingestion time, because filters need it.

def tag(path: str) -> dict:
    p = Path(path)
    return {"tenant_id": p.parts[1], "department": p.parent.name}

docs = SimpleDirectoryReader(
    input_dir="corpus", recursive=True,
    required_exts=[".pdf", ".md", ".docx"], file_metadata=tag,
).load_data()

recursive=True matters: the reader is top-level only by default, which quietly explains most "half my documents are missing" reports.

Cost measurement you can put in CI.

import tiktoken
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler

counter = TokenCountingHandler(tokenizer=tiktoken.encoding_for_model("gpt-4o-mini").encode)
Settings.callback_manager = CallbackManager([counter])
# ... run the path you want to price ...
print(counter.total_embedding_token_count, counter.total_llm_token_count)

Three CI checks worth more than their runtime. Validate every workflow graph with MyFlow().validate(); assert every tool parameter has a description via get_parameters_dict(); assert every node carries the metadata keys your filters depend on. All three are free, deterministic and catch real deploys.

Pins that survive the next minor.

llama-index-core>=0.14.25,<0.15
llama-index-workflows>=2.25.0,<3
llama-index-llms-openai>=0.8.2,<0.9
llama-index-embeddings-openai>=0.6.0,<0.7
llama-index-readers-file>=0.5

Every integration pins llama-index-core<0.15.0, so 0.15 is the next breaking boundary and will need every integration re-released. Read both changelogs before a minor bump: the workflows one carries behaviour changes that no error message will warn you about, such as the 2.23.1 off-by-one that made every retry delay one step shorter, and the 2.25.0 timeout semantics.

Two deployment shapes, and which to pick. Writing the HTTP layer yourself with FastAPI gives you authentication, tenancy, and control over what happens per request, which is why most teams end up there. llama-agents-server is the shortcut: python -m llama_agents.server my_server.py binds 0.0.0.0:8080 (configurable with WORKFLOWS_PY_SERVER_HOST and WORKFLOWS_PY_SERVER_PORT) and gives you run, run-nowait with a handler_id, handler status, results, an SSE event stream, send-event, cancel, and a debugger UI at /. Three things to know before you put it anywhere real: the default MemoryWorkflowStore loses everything on restart, so use SqliteWorkflowStore(db_path=…) at minimum; a run's event stream allows exactly one reader and cannot be replayed; and there is no authentication, so it must sit behind your own gateway.

Container notes that save a confusing first deploy. llama-index-core pre-bundles its NLTK and tiktoken data specifically to avoid runtime downloads, so a network-restricted container is fine for tokenisation. HuggingFaceEmbedding is not: bake the model into the image or mount a cache volume, or every pod downloads it on first request and your readiness probe will happily report healthy while the first user waits.

A note on data residency, because it is a design decision and not a deployment detail. A RAG query sends document text to a model provider, which is a cross-border transfer. If you are serving a bank, an insurer or a government entity in the Gulf, establish which region your LLM and embedding endpoints resolve to before you design the system. Azure OpenAI in UAE North, Bedrock in Bahrain, and a self-hosted model served through vLLM or Ollama are all reachable by changing two lines of startup configuration — provided provider choice lives in configuration rather than scattered through your code, which is the practical reason the tenth practice above matters.

The checklist before you call it production

Run through this on a pull request that touches retrieval, ingestion or an agent.

• The index is reconnected, not rebuilt, and the reconnect is in startup rather than a handler.
• Settings.llm and Settings.embed_model are both set, from configuration, once.
• No Settings. assignment exists inside any request handler.
• Every query engine logs the retrieved file names and scores alongside the answer.
• similarity_top_k is a chosen number, and any similarity_cutoff was measured against the embedding model in use.
• Ingestion has a docstore and a cache that outlives the process, and an embedding stage in transformations.
• excluded_embed_metadata_keys excludes volatile timestamps; file_name stays visible to the LLM for citations.
• Tenant scope comes from the authenticated session and appears in no tool schema.
• Memory(session_id=...) is derived from authenticated identity, never from a request body.
• Every Workflow has an explicit timeout; every agent has an explicit max_iterations and a chosen early_stopping_method.
• One rate limiter is shared by the LLM and the embedding model.
• No exec is passed to a CodeActAgent, and code_execute_fn runs in a sandbox. (ValueError: code_execute_fn must be provided for CodeActAgent is the reminder that there is no default — which is deliberate.)
• No API key is hard-coded, including in examples copied from the docs, which do hard-code them. No storage example sets ACL: public-read; your docstore holds the full text of every indexed document.
• Dependencies are pinned with upper bounds, and patch releases of 0.14.x are being taken for the security fixes.
• An evaluation set of real questions exists and runs on a schedule, with Faithfulness and ContextRelevancy at minimum, so you can tell a retrieval regression from a synthesis one.

Where to go next

Observability is the habit that compounds fastest: wire llama-index-observability-otel to a collector, or set_global_handler("langfuse") for prompt-level views, and spend a week reading your own traces. The Langfuse, Arize Phoenix and OpenTelemetry guides cover the backends. For measuring quality rather than latency, RAGAS and DeepEval both integrate here. If the vector store is the part you are least sure about, Qdrant and pgvector are the two worth knowing properly before you commit. And if the orchestration questions are the ones keeping you up, read LangGraph as a deliberate contrast — knowing both models turns your architecture choice into an argument you can win rather than a habit you defend.

This is part three of three. The earlier levels were about making LlamaIndex work: loading documents, building a VectorStoreIndex, tuning retrieval, writing an agent, wiring an IngestionPipeline. This level is about owning it. When three product teams build on a retrieval platform you maintain, the questions change shape. What happens when a workflow run dies halfway through a ten-minute agent loop. Who can see which tenant's documents. What the monthly embedding bill is made of, and which line of code decides it. What breaks on the day llama-index-core 0.15 lands and every one of the three hundred integration packages needs re-releasing.

Everything here is verified against llama-index-core 0.14.25 (released 21 September 2026) and llama-index-workflows 2.25.0. Both numbers matter more than usual, because LlamaIndex moved its agent and orchestration story twice in eighteen months, and a surprising amount of production code and blog guidance still targets the removed APIs.

Where this picks up

I start with the architecture, because every security, cost and scaling decision below follows from where you drew the line between offline and online.

The architecture you actually own

LlamaIndex is a library, not a service. There is no daemon, no scheduler, no control plane. That means the architecture is entirely yours, and the single most consequential decision is the split between ingestion and serving. Teams that skip this decision end up with a FastAPI process that calls VectorStoreIndex.from_documents(SimpleDirectoryReader("data").load_data()) at startup, which works for four documents and then quietly becomes a five-minute cold start, a duplicated embedding bill on every deploy, and an outage whenever the document bucket is slow.

The shape the official production guidance points at has three tiers.

The serving side reconnects rather than rebuilds:

from llama_index.core import VectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding

# One client and one index object per process, created at startup.
vector_store = QdrantVectorStore(client=qdrant_client, collection_name="docs_v3")
index = VectorStoreIndex.from_vector_store(
    vector_store,
    embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
)

Two details in that snippet are load-bearing. The collection name carries a version suffix, because a re-index is a new collection and a pointer swap, not an in-place migration. And embed_model is passed locally rather than set on Settings, for reasons that become a security issue in the multi-tenancy section.

Most hosted vector stores keep the node text alongside the vector, so the serving tier needs no docstore and no persist() at all. FAISS is the exception a lot of teams discover painfully: it stores vectors only, so it needs a separate docstore to return text, and that docstore is one more thing to deploy and keep consistent. If you are choosing today and want one less moving part, the Qdrant or pgvector paths are the ones with fewer surprises at this tier.

The third tier is orchestration, and whether you need it is a real decision rather than a default. A single-turn query engine behind FastAPI needs nothing. An agent that calls six tools, pauses for a human approval, and may run for twenty minutes needs durability, and that is where DBOSRuntime or your own context snapshots come in.

Where the framework ends and you begin

It is worth being blunt about what LlamaIndex does not give you, because every item on this list is something a senior owner is expected to have an answer for, and none of them has a framework default.

There is no authentication and no authorization. Nothing in the library knows who is asking. There is no tenancy primitive: a VectorStoreIndex has no notion of an owner, and a retriever will happily return any node in the collection. There is no sandbox: a tool is a Python callable executed in your process with your process's privileges, and CodeActAgent runs model-written code through a code_execute_fn that you supply. There is no scheduler and no queue: IngestionPipeline.run() runs when you call it. There is no built-in checkpointer any more, because the deprecated one was removed in Workflows 2.0, and the replacement is Context.to_dict() snapshots that you store and reload yourself. And the WorkflowServer from llama-agents-server exposes run, send-event and cancel endpoints with no authentication at all, so it belongs behind your gateway and never on a public address.

None of this is a criticism. It is the normal shape of a library that wants to compose with whatever stack you already run. But it does mean the platform you are building is mostly the code around LlamaIndex, and a design review that only examines the retrieval config has looked at the smaller half of the system.

The practical consequence for a platform team is that you should ship an opinionated internal wrapper rather than the raw library. Not a fork, and not an abstraction that hides the framework, but a thin package that supplies the defaults product teams should not be choosing individually: the embedding model and chunk size, the vector store client with credentials from your secret manager, the rate limiter, the instrumentation dispatcher, and a build_retriever(tenant_id) function that applies the tenant metadata filter server-side. Everything else stays plain LlamaIndex.

Failure modes of the workflow runtime

Agents in LlamaIndex are workflows. FunctionAgent, ReActAgent, CodeActAgent and AgentWorkflow are all built on the Workflows engine, which means every agent inherits the workflow failure modes, and understanding them once covers both. The legacy agent classes that were not workflows (FunctionCallingAgent, AgentRunner, the old ReActAgent.from_tools style, OpenAIAgent) were removed in 0.13, along with QueryPipeline. If you inherited a codebase that still imports them, the migration is not optional and the error messages are unambiguous: ImportError: cannot import name 'FunctionCallingAgent' from 'llama_index.core.agent', and ModuleNotFoundError: No module named 'llama_index.core.query_pipeline'.

The first failure mode is the timeout, and it bites almost everyone once. A plain Workflow defaults to timeout=45.0 seconds. One LLM call on a reasoning model can exceed that on its own.

workflows.errors.WorkflowTimeoutError: Operation timed out after 45.0 seconds.
Currently active steps: synthesize

The message names the active steps, which is genuinely useful during an incident. The fix is to set the timeout deliberately per workflow, not to disable it everywhere. Note the asymmetry: the agent classes default to timeout=None, so an agent has no deadline unless you give it one, while a custom workflow has an aggressive one. Both defaults are wrong for production, in opposite directions.

Workflows 2.25.0 changed what the timeout measures. It now counts accumulated alive time across resumes rather than restarting the clock each time a suspended run is resumed. For a human-in-the-loop workflow that sits waiting for an approval over three days and resumes a dozen times, this is the difference between a run that never times out and one that does. If you upgrade through 2.25 and a long-lived HITL workflow starts failing with a timeout it never hit before, this is why.

The second failure mode is graph validation, which runs before a workflow starts. It is a static check over the return annotations of your steps, and the errors are precise:

WorkflowConfigurationError: At least one Event of type StopEvent must be returned by any step.
WorkflowValidationError: The following events are produced but never consumed: EnrichedEvent
WorkflowValidationError: The following events are consumed but never produced: ReviewEvent

Since 2.17 the checks also report Unreachable steps, Dead-end steps, and a hint that only StopEvent and InputRequiredEvent may be terminal. The usual cause of a false positive is a step that dispatches dynamically with ctx.send_event(...) instead of returning the event, so the analyser cannot see the edge. Disable the check narrowly rather than globally: @step(skip_graph_checks=["reachability"]) on the one dynamic step, not disable_validation=True on the workflow. You can also call workflow.validate(validate_resources=True) in a unit test, which is a cheap way to catch a broken graph in CI rather than at the first production run.

The third is the iteration cap on agents, which defaults to 20:

WorkflowRuntimeError: Max iterations of 20 reached! Either something went wrong,
or you can increase the max iterations with `.run(.., max_iterations=...)`
or use `early_stopping_method='generate'` to generate a final response instead.

Treat a hit on this cap as a signal, not a nuisance. Twenty tool-calling rounds almost always means a tool whose description is ambiguous enough that the model keeps retrying it, or a tool that returns an error string the model reads as "try again". Raising the cap raises your bill and your latency without fixing either cause. The one legitimate production use of early_stopping_method="generate" is to give the user a partial answer instead of a 500, and 0.14.25 helps here too: failed function tools are no longer retried automatically, which removes one common source of loops.

The fourth is the stream being consumable once. handler.stream_events() can be iterated by exactly one consumer:

WorkflowRuntimeError: All the streamed events have already been consumed.

This surfaces in production as "the websocket works but the audit log is empty", because two parts of the request handler both tried to read the stream. Consume once, in one place, and fan out yourself to the websocket, the tracer and the audit sink. The same constraint applies through WorkflowServer: one reader per run's event stream, and streamed events cannot be recovered after the fact.

Which component breaks first

Scaling questions are easier to answer when you know the order in which things give way. For a LlamaIndex platform, the order is fairly consistent.

The in-memory stores break first, and earliest. SimpleVectorStore and SimpleDocumentStore are JSON files loaded fully into RAM in a single process. They are excellent for a prototype and a liability the moment you have two replicas, because each replica has its own copy and nothing reconciles them. The symptom is not an error; it is two users getting different answers to the same question. Move to a real vector store before you move to two replicas, not after.

The ingestion tier breaks second, on throughput. The knobs, in the order worth reaching for: embed_batch_size on the embedding model, which decides how many chunks go in one API call; insert_batch_size on VectorStoreIndex, which defaults to 2048 nodes per insert and is the knob to lower when a remote vector DB rejects large writes or when memory is tight; await pipeline.arun(documents=docs) for I/O-bound work, which is most ingestion; and pipeline.run(documents=docs, num_workers=4), which uses a multiprocessing.Pool. The docs are candid that LlamaIndex typically avoids multiprocessing and that asyncio gives you concurrency rather than parallelism, so reach for num_workers only when profiling shows CPU-bound parsing, and expect little or nothing from it on Windows. For genuinely large corpora, RayIngestionPipeline (added in 0.14.13) distributes the work across a Ray cluster.

Provider rate limits break third, and they break during ingestion more often than during serving, because ingestion is the part that embeds a hundred thousand chunks as fast as it can. Since 0.14.16 there is a first-class answer:

from llama_index.core.rate_limiter import TokenBucketRateLimiter
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# One instance shared across both clients enforces ONE global budget.
limiter = TokenBucketRateLimiter(requests_per_minute=30, tokens_per_minute=100_000)

llm = OpenAI(model="gpt-4o-mini", rate_limiter=limiter)
embed_model = OpenAIEmbedding(model="text-embedding-3-small", rate_limiter=limiter)

Sharing a single limiter across the LLM and the embedding client is the detail that makes this useful: provider quotas are usually per account, not per client object, so two independently throttled clients will still collectively exceed the quota. SlidingWindowRateLimiter is the alternative shape when your provider enforces a rolling window rather than a bucket. Pair either with a workflow retry policy using exponential jitter so that the requests you do shed come back spread out.

The workflow runtime breaks fourth, on horizontal scale. The default runtime is in-process: a WorkflowHandler lives in the memory of one Python process, and if that process dies the run dies with it. Per-step concurrency is @step(num_workers=N), default 4; whole-workflow concurrency is Workflow(num_concurrent_runs=N). Both are within-process knobs. Crossing the process boundary requires either a durable runtime or your own queue in front.

The vector store breaks last, which is why it is worth choosing a managed one. When it does break, it breaks on recall quality rather than on availability: a collection that has grown past the point where a flat similarity_top_k=5 over the whole corpus finds the right chunks. That is a retrieval-architecture problem, not a capacity problem, and the fixes are the production-RAG ones: decouple the chunk you retrieve from the chunk you synthesize over, add metadata filters or document summaries so retrieval is structured rather than flat, route by task, and rerank for precision.

Durability, resume and exactly-what-once

Long-running agent and workflow runs need to survive a deploy. There are three levels of answer, and choosing among them is a real architectural decision.

Level one: do not need durability. Keep runs short, make them idempotent, and retry the whole thing on failure. For a single-turn RAG query this is correct and everything below is overhead.

Level two: snapshot the context yourself. Context is serializable, and the pattern is explicit:

import json
from workflows import Context

handler = workflow.run(query="quarterly summary")
# ... later, or on a StepStateChanged event with NOT_RUNNING:
snapshot = json.dumps(handler.ctx.to_dict())
await store.put(run_id, snapshot)

# After a restart:
ctx = Context.from_dict(workflow, json.loads(await store.get(run_id)))
result = await workflow.run(ctx=ctx)

Two properties of this pattern decide whether it is safe for your workload. First, resume is at-least-once: steps that had completed are not re-run, but steps that were in flight when the snapshot was taken are. If an in-flight step charged a card or sent an email, it will do so again. The discipline is the same as for any at-least-once system: make side-effecting steps idempotent, with an idempotency key derived from the run id.

Second, to_dict() is all-or-nothing. A single unserializable value anywhere in state or in the in-flight events makes the whole snapshot fail, and the errors come from the ContextSerdeError family. The cure is to keep clients, connections and byte buffers out of state entirely and inject them as resources instead:

from typing import Annotated
from workflows import Workflow, step, Context
from workflows.resources import Resource

def get_qdrant():
    return QdrantVectorStore(client=qdrant_client, collection_name="docs_v3")

class RagFlow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent,
                       store: Annotated[QdrantVectorStore, Resource(get_qdrant)]) -> Retrieved:
        ...

A Resource is created once, cached, never serialized, and re-created on resume. That is exactly the lifecycle a database client wants. Where state genuinely must hold something exotic, Workflow(serializer=JsonSerializer(allowed_types=[...])) lets you allow-list types rather than loosening everything.

Level three: a durable runtime. llama-agents-dbos provides DBOSRuntime, which persists run state in Postgres so that a run survives the process and can be picked up by another replica.

from dbos import DBOS
from llama_agents.dbos import DBOSRuntime

DBOS(config={
    "name": "rag-platform",
    "run_admin_server": False,
    "system_database_url": "postgresql://...",   # from your secret manager, not source
    "executor_id": "replica-1",                  # must be unique per replica
})
runtime = DBOSRuntime()
workflow = RagFlow(runtime=runtime)
await runtime.launch()
result = await workflow.run(run_id="order-8821")   # same run_id resumes

The executor_id is the detail people get wrong: two replicas sharing one id will fight over runs. idle_timeout releases idle human-in-the-loop runs from memory, which matters when approvals take days. And note the error you get if you mix the levels up: BasicRuntime cannot restore durable state store '...'. Use the matching durable runtime or pass an in-memory context snapshot.

Human-in-the-loop deserves one specific warning. There are two patterns: return an InputRequiredEvent and consume a HumanResponseEvent in a separate step, or call await ctx.wait_for_event(HumanResponseEvent, ...) inline. The docs recommend the two-step pattern, and the reason is that wait_for_event replays the code before it on resume. Any non-idempotent work above that line runs twice.

The trust model

There are three distinct trust boundaries in a LlamaIndex application, and conflating them is the most common security mistake at this level.

Boundary one: retrieved text is untrusted input. A RAG system's entire purpose is to put third-party text into a prompt. If any document in your corpus can be authored or influenced by someone outside your trust circle (a support ticket, a scraped page, a user-uploaded PDF, a wiki anyone can edit), then that text can contain instructions, and the model may follow them. There is no setting that fixes this. What reduces blast radius is keeping the toolset narrow, so that a successfully injected instruction has nothing dangerous to call; using return_direct=True on tools whose output should end the loop rather than feed back into the model's reasoning; and never letting the model choose a security-relevant parameter — which brings us to the tenancy filter, below.

Boundary two: tools are code execution. A FunctionTool is a Python callable invoked with model-chosen arguments, in your process, with your process's credentials. A tool that takes a SQL string is a SQL injection endpoint with extra steps. A tool that takes a file path is an arbitrary-read primitive. Write tools that take narrow, typed, validated parameters, and do the authorization inside the tool against the caller's identity rather than against anything the model supplied. CodeActAgent is the extreme case: it requires a code_execute_fn, raising ValueError: code_execute_fn must be provided for CodeActAgent if you forget, and whatever you pass there will run model-authored code. Passing raw exec is the single worst line you can write with this framework. Run it in a container or a restricted interpreter with no network and no credentials, which is where the Docker and Kubernetes parts of your platform earn their keep.

Boundary three: persisted artefacts are code. Loading a persisted index or a workflow context is deserialization, and deserialization of untrusted data is a vulnerability class. The framework has been hardening this: 0.14.16 added a RestrictedUnpickler to SimpleObjectNodeMapping (CWE-502), and Workflows 2.21 added JsonSerializer(allowed_types=..., allow_unknown_types=...) so that context deserialization can be restricted to a declared set of types. Use the allow-list, and treat the storage holding your ./storage directories and your context snapshots with the same access control you would give a code artifact repository. Only load from storage you control.

On secrets: every provider integration reads an environment variable (OPENAI_API_KEY, LLAMA_CLOUD_API_KEY, LANGFUSE_SECRET_KEY and friends) or accepts api_key= on the constructor. Load them from your secret manager into the environment at startup — Vault is the usual choice — and remember the two non-obvious credential carriers: Memory(async_database_uri="postgresql+asyncpg://user:pass@host/db") and DBOS's system_database_url both embed database credentials in a connection string that will end up in a log or a stack trace if you are careless. The official docs hard-code api_key="sk-..." in examples and one storage example sets ACL: public-read on S3; both are teaching shortcuts, and both should fail your code review.

Multi-tenancy without framework support

There is no tenancy primitive in LlamaIndex, so tenancy is a pattern you implement with four independent decisions.

Isolation of the corpus. The strongest option is a separate collection, namespace or database per tenant in the vector store. It gives you hard isolation, per-tenant deletion that is a drop rather than a scan, and per-tenant capacity accounting. It costs you a collection count that grows linearly, which is fine for dozens of tenants and awkward for tens of thousands. The alternative is one shared collection with a tenant key in node metadata and a server-side filter on every retrieval:

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

def build_retriever(index, tenant_id: str, top_k: int = 5):
    """The ONLY sanctioned way to get a retriever in this codebase."""
    return index.as_retriever(
        similarity_top_k=top_k,
        filters=MetadataFilters(filters=[
            MetadataFilter(key="tenant_id", value=tenant_id),
        ]),
    )

MetadataFilters.condition defaults to AND when it is None (since 0.14.24), so adding further filters narrows rather than widens. Make this function the only path to a retriever, and enforce that with a lint rule or a code-review checklist item, because the failure mode of a forgotten filter is silent and severe.

Isolation of conversation state. Memory.from_defaults(session_id=...) keys short-term history by session, and the SQL-backed chat store means two tenants' histories live in different rows rather than in one process's memory. Choose the session key so that it cannot collide across tenants — tenant id plus user id, not user id alone.

Isolation of run state. Use a fresh Context per run rather than sharing one. Workflow state has been keyed by run id plus namespace since Workflows 2.23, which removes a class of cross-run leakage that earlier versions allowed if you reused a context.

And the one that catches people: do not mutate Settings per request. Settings is a process-wide singleton. In a multi-tenant async server, setting Settings.llm = tenant_model at the top of a request handler means that while an await is suspended, another tenant's request can read your value, or overwrite it before your synthesis step runs. The result is a tenant's data being sent to another tenant's configured model endpoint. Pass models locally instead: index.as_query_engine(llm=tenant_llm), VectorStoreIndex.from_documents(docs, embed_model=...), agent.run(..., memory=tenant_memory). Treat Settings as boot-time configuration only.

For teams in the Gulf and Egypt this intersects with data residency in a concrete way. If a tenant's contract requires that their documents never leave a specific jurisdiction, the tenancy boundary has to extend to the model endpoint and the vector store region, not just to the metadata filter. In practice that means per-tenant model configuration (an Azure OpenAI deployment in UAE North, or Bedrock in a permitted region) resolved from the tenant record at request time and passed locally. LlamaParse, if you use the hosted parser, offers managed SaaS in North America or the EU, single-tenant SaaS, bring-your-own-cloud on AWS, Azure or GCP, and self-hosting via Helm on Kubernetes — the last two are the options to look at when a contract names a region the managed offering does not have.

Cost control as a platform concern

An LLM platform's bill has three components, and only one of them is obvious.

The obvious one is synthesis tokens: the prompt you send at query time, which is roughly similarity_top_k multiplied by chunk_size, plus the question and the system prompt, plus the completion. The default similarity_top_k is 2 and the default chunk_size is 1024 tokens, so a default query engine sends on the order of two thousand tokens of context. A team that raises similarity_top_k to 20 to "improve recall" has multiplied the per-query cost by roughly ten, usually without measuring whether answers improved. response_mode matters too: compact (the query-engine default) packs as much context into as few calls as it can, refine makes one call per node, and tree_summarize makes a logarithmic number. Switching from compact to refine on a top_k=10 query engine turns one LLM call into ten.

The less obvious one is embedding tokens, which are paid twice: once per chunk at ingestion, and once per query for the query string. Ingestion is where the volume is, and it is where the IngestionPipeline cache is worth real money. Every node-and-transformation pair is hashed and cached, so a re-run over an unchanged corpus is almost free. Put the cache somewhere shared:

from llama_index.core.ingestion import IngestionPipeline, IngestionCache
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.storage.kvstore.redis import RedisKVStore

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=512, chunk_overlap=50),
        OpenAIEmbedding(model="text-embedding-3-small", embed_batch_size=100),
    ],
    vector_store=vector_store,        # embeddings must be a stage when a store is attached
    docstore=SimpleDocumentStore(),   # enables doc_id + hash dedup and upsert
    cache=IngestionCache(
        cache=RedisKVStore.from_host_and_port("127.0.0.1", 6379),
        collection="ingest_cache",
    ),
)
nodes = await pipeline.arun(documents=docs)

With both a docstore and a vector store attached, a document whose hash changed is re-processed and upserted, and one whose hash is unchanged is skipped. With no vector store attached, the pipeline only de-duplicates its inputs, which is a much weaker guarantee than people assume. RedisCache, MongoDBCache and FirestoreCache are the remote cache options.

The component nobody budgets for is hidden LLM calls. Several things that look like data processing are model calls: TitleExtractor and the other metadata extractors call the LLM once per node during ingestion; LLMRerank calls it once per retrieved batch per query; FactExtractionMemoryBlock calls it as conversations grow; and the LLM-based evaluators call it once per evaluation. A TitleExtractor over a hundred thousand chunks is a hundred thousand LLM calls, and it will dwarf the embedding cost. Keep them, if they earn their keep — titles genuinely improve retrieval on some corpora — but measure first.

Measuring is straightforward. TokenCountingHandler still works through the legacy callback manager and exposes total_llm_token_count, total_embedding_token_count, prompt_llm_token_count and completion_llm_token_count. For forecasting before you spend anything, MockLLM and MockEmbedding let you run the full pipeline and count what it would have cost; MockFunctionCallingLLM (0.14.10 onward) does the same for agent tests. On a platform, wire the counts into per-tenant attribution from day one, because the question "which team is spending this" arrives about a month after the first invoice.

The levers, roughly in order of return: the ingestion cache, a smaller similarity_top_k with a reranker to recover precision, a cheaper model for the components that do not need the expensive one (routing, query rewriting and metadata extraction rarely do), max_iterations on agents, and local models through Ollama with HuggingFace embeddings where latency and RAM allow.

Observability you can be paged by

LlamaIndex has two generations of observability, and knowing which one you are in saves confusion. The legacy CallbackManager and set_global_handler mechanism still works and many partner integrations still use it. The current mechanism is the instrumentation module, built on a dispatcher with events and spans.

import llama_index.core.instrumentation as instrument
from llama_index.core.instrumentation.event_handlers import BaseEventHandler

dispatcher = instrument.get_dispatcher(__name__)

class AuditHandler(BaseEventHandler):
    def handle(self, event, **kwargs) -> None:
        # LLMChatStartEvent, LLMChatInProgressEvent, LLMChatEndEvent, and more
        log.info("llm_event", type=type(event).__name__, span_id=event.span_id)

dispatcher.add_event_handler(AuditHandler())

For standalone workflows without core installed, the dispatcher comes from llama_index_instrumentation instead. Spans are added with @dispatcher.span on your own functions and a BaseSpanHandler subclass implementing new_span, prepare_to_exit_span and prepare_to_drop_span.

For a platform, the thing to standardise on is OpenTelemetry, because it puts LLM spans in the same trace as your HTTP and database spans:

from llama_index.observability.otel import LlamaIndexOpenTelemetry
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

instrumentor = LlamaIndexOpenTelemetry(
    span_exporter=OTLPSpanExporter("http://collector:4318/v1/traces"),
    service_name_or_resource="rag-api",
)
instrumentor.start_registering()

That captures workflow steps, LLM calls and custom events as OTel spans. On top of it, a trace-centric LLM tool gives you the per-run view that generic tracing does not: Langfuse and Arize Phoenix both have one-click handlers, and MLflow is documented too. The set_global_handler(name, **kwargs) shortcut supports "arize_phoenix", "langfuse", "openinference", "honeyhive", "promptlayer", "agentops" and "simple", several of which are the legacy callback-based path.

At the workflow level there are two further sources. stream_events(expose_internal=True) yields StepStateChanged events carrying the step name, worker id, input and output event names, and a StepState of PREPARING, RUNNING or NOT_RUNNING — which is both a progress feed and the natural trigger for taking a durability snapshot. And Workflow(verbose=True) prints step transitions, which is a development tool rather than a production one.

The four signals worth alerting on, in my experience: the rate of WorkflowTimeoutError, the rate of max-iteration hits, retrieval returning zero nodes above the similarity cutoff (a silent quality failure that no exception reports), and token spend per tenant per hour against a budget. Export them to Prometheus from your own counters; the framework will not do it for you.

Upgrades, pinning and the 0.15 boundary

LlamaIndex's version scheme makes the minor number the breaking boundary: 0.13 → 0.14 was breaking, and 0.14.x patches are meant not to be. Core ships roughly every two to six weeks, and the 0.14 line has had twenty-five patch releases since September 2025. Every integration package declares llama-index-core<0.15.0, which tells you two useful things: a 0.14.x bump will not break your integrations, and 0.15 will require every integration you depend on to be re-released before you can move. At each minor bump the project re-releases all packages to accept the new core, but that is a window of days to weeks during which a long-tail integration you depend on may not be ready.

The pinning policy that works:

llama-index-core>=0.14.25,<0.15
llama-index-workflows>=2.25.0,<3
llama-index-readers-file>=0.5
llama-index-llms-openai>=0.8.2,<0.9
llama-index-vector-stores-qdrant>=0.8

Note llama-index-readers-file as an explicit dependency. The umbrella llama-index package was slimmed down through early 2026 and its current dependency set is only llama-index-core, llama-index-embeddings-openai, llama-index-llms-openai and nltk. It dropped llama-index-readers-file in 0.14.18, the LlamaCloud index and LlamaParse readers in 0.14.19, and llama-index-cli in 0.14.20. The installation docs have not caught up. The consequence is specific and easy to miss: without llama-index-readers-file, SimpleDirectoryReader logs a warning and falls back to reading PDFs and DOCX files as plain text, which produces garbage chunks rather than an error. If your corpus quality dropped after an upgrade, check this first.

`llama-index-readers-file` package not found, some file readers will not be
available if not provided by the `file_extractor` parameter.

Two more upgrade rules. Re-index when the embedding model or the chunking changes — a dimension change gives you ValueError: shapes (384,) and (1536,) not aligned, but a same-dimension model swap gives you silently worse retrieval, which is worse. Do it as a new collection plus a pointer swap, so rollback is a pointer swap too. And test loading your persisted artefacts after every upgrade: the JSON layout of SimpleVectorStore and the docstore is versioned with the library, and so is the context snapshot format.

Read both changelogs, not just core's. The workflows changelog is where the behaviour changes hide: 2.23.1 fixed an off-by-one so that every retry delay is now one step shorter than it was, and 2.25.0 changed timeout accounting across resumes. Neither is a major version, and both will change how your production system behaves.

Finally, a list of things not to carry forward. ServiceContext raises ValueError: ServiceContext is deprecated. Use llama_index.settings.Settings instead on any use. The llamaindex-cli tool is deprecated and actually broken on 0.14.25, crashing with ModuleNotFoundError: No module named 'llama_index.core.download', which also means download_llama_pack and download_loader are gone and llamaindex-cli upgrade cannot help you migrate. The old ctx.get() / ctx.set() state API is replaced by await ctx.store.get(key, default=...), await ctx.store.set(key, value) and async with ctx.store.edit_state() as state:. ChatMemoryBuffer and friends are documented as deprecated in favour of Memory, though in 0.14.25 agents still construct a ChatMemoryBuffer by default when you pass no memory — so new code should pass Memory.from_defaults(...) explicitly rather than rely on the default changing. And ConstantDelayRetryPolicy / ExponentialBackoffRetryPolicy are superseded by the composable retry_policy(retry=..., wait=..., stop=...).

Incident playbooks

Four incidents account for most of the pages on a RAG platform. Each is worth having written down before it happens.

"The answers went bad" with no error and no deploy. This is a retrieval failure, and the diagnostic order is: first confirm the index is the one you think (collection name, document count, a known document present); second, retrieve without synthesis for a query you know the answer to, by calling index.as_retriever(...).retrieve(q) and inspecting the NodeWithScore scores and sources; third, compare the embedding model in the serving config against the one the collection was built with. If retrieval returns the right nodes and the answer is still wrong, it is a synthesis problem (prompt, response_mode, model change on the provider's side), not a retrieval one. The reason to separate these two steps is that they have disjoint fixes and people routinely tune the wrong one for a day.

Workflow runs piling up and never completing. Check the timeout first, because None on agents means runs can hang indefinitely on a stalled provider call. Then check the iteration counter. Then check whether something is consuming the event stream — a run whose stream has a dead reader can stall waiting on backpressure. If you are on a durable runtime, check for duplicate executor_id values across replicas, which causes replicas to contend over the same runs.

Ingestion cost spike. Almost always one of three things: a cache that is not actually shared (each worker got its own in-memory cache, so nothing hits), a document whose hash changes on every run even though its content does not (a timestamp in metadata is the classic cause, since metadata participates in the hash), or a newly added metadata extractor making one LLM call per node. Compare total_embedding_token_count between two consecutive runs over an unchanged corpus; on a working cache the second run is near zero.

A tenant reports seeing another tenant's content. Treat as a security incident. Capture the exact query and the retrieved node ids before anything else, because the node metadata will tell you which tenant the content belonged to. Then check, in order: whether the retriever was built through your sanctioned filter function, whether anything in the request path mutated Settings, whether the session key for Memory can collide across tenants, and whether a tool received a tenant identifier from model output rather than from the request context. Rotate nothing and change nothing until you have the retrieved node ids; they are the only evidence that distinguishes a filter bug from a mis-ingested document that was written into the wrong tenant's partition in the first place.

Where LlamaIndex stops

A senior owner's most valuable contribution is sometimes to say that the tool in front of them is the wrong one.

LlamaIndex is the wrong choice when your orchestration is the hard part and retrieval is trivial. The Workflows engine is good, but if you have no corpus and your problem is a complex multi-step graph with checkpointing and a visual debugger, LangGraph is more specialised at exactly that, and you will not be paying for a data framework you do not use. Conversely LlamaIndex is the right choice when the corpus is the problem: when the interesting work is parsing messy PDFs, structuring retrieval over a large heterogeneous corpus, and keeping an ingestion pipeline incremental and cheap.

It is the wrong choice when you need a managed, supported, warranty-bearing RAG product. It is a library under active and fast-moving development, with two breaking migrations in recent memory. If your organisation cannot absorb a quarterly dependency review, a managed service from your cloud provider will cost more money and less of your time.

It is the wrong choice for pure prompt-and-evaluate work with no retrieval, where a thinner client library or a dedicated evaluation tool fits better. And when the question is specifically "how good is my RAG system", LlamaIndex's built-in evaluators (Correctness, Faithfulness, SemanticSimilarity, ContextRelevancy, AnswerRelevancy, Guideline, plus retrieval metrics like MRR and hit-rate with synthetic question generation) are a reasonable start, but RAGAS and DeepEval are built for it and integrate cleanly.

Finally, it stops at the boundary of your own infrastructure. Serving, scheduling, authentication, tenancy, sandboxing and quota are yours. If you were hoping the framework would supply those, the honest answer is that it never intended to, and the sooner you plan for building them the better your platform will be.

Running it as a platform for other teams

The difference between "we use LlamaIndex" and "we run a retrieval platform" is a small number of deliberate choices, made once, centrally.

Publish an internal package, not a document. A shared yourco-rag package that exports get_llm(), get_embed_model(), get_vector_store(), build_retriever(tenant_id, top_k) and instrument() is worth more than any amount of guidance, because it makes the right thing the default thing. Pin LlamaIndex inside that package so that a core upgrade is one merge request in one repository rather than a migration across six.

Own the embedding and chunking contract. These are platform decisions, not product decisions, because changing them requires a re-index that costs money and coordination. Version them explicitly: the collection name carries the contract version, and changing the contract means a new collection, a backfill and a pointer swap.

Expose workflows over a stable interface. If you offer llama-agents-server, teams get a predictable HTTP surface: GET /health, GET /workflows, POST /workflows/{name}/run, POST /workflows/{name}/run-nowait returning a handler_id, GET /handlers/{handler_id} (202 while running, 500 on failure, 200 when done), GET /results/{handler_id}, GET /events/{handler_id}?sse=true and POST /handlers/{handler_id}/cancel?purge=true. Two operational facts to publish alongside it: the default MemoryWorkflowStore loses everything on restart, so use SqliteWorkflowStore or something durable; and there is one reader per run's event stream, with no replay. Run it behind your gateway, because none of those endpoints authenticate.

Make quota and attribution a platform feature. A shared rate limiter per tenant, token counting wired into your metrics, and a monthly report per team. Teams optimise what they can see.

Set a dependency rhythm. A standing monthly review of the core and workflows changelogs, a staging environment that upgrades first, and a test that loads a persisted index and a context snapshot from the previous version. This is dull and it is the single highest-leverage habit on this list.

Write the golden-set test. Twenty to fifty questions with known correct answers and known source documents, run in CI against the real index. It catches the class of regression that no unit test catches: a chunking change, a model change on the provider's side, a reranker that was quietly dropped.

The review checklist

Run this over any pull request that touches retrieval, agents or ingestion.

1. No legacy imports. Nothing from llama_index.core.query_pipeline, no FunctionCallingAgent, no AgentRunner, no ReActAgent.from_tools, no ServiceContext, no agent.chat(). Agents are async and run with await agent.run(...).
2. No Settings mutation outside startup. Models and embed models passed locally in request paths.
3. Every retriever goes through the sanctioned tenant-filtered factory. No raw index.as_retriever() in product code.
4. Timeouts set deliberately. Not the 45-second workflow default, not the agent's None.
5. max_iterations and early_stopping_method chosen, with a reason in the description.
6. No secrets in source. No api_key= literals, no credentials in a async_database_uri or system_database_url string.
7. Tools are narrow and typed, authorize against the request identity, and nothing takes a free-form SQL string, path or URL.
8. code_execute_fn, if present, is sandboxed. No exec, no network, no credentials.
9. Workflow state holds no clients or connections. Those are Resources. to_dict() is exercised in a test.
10. Side-effecting steps are idempotent, keyed by run id, because resume is at-least-once.
11. The ingestion pipeline has a shared remote cache and a docstore, and the embedding stage is present when a vector store is attached.
12. Embedding model and chunk size unchanged, or the PR includes a re-index plan and a new collection name.
13. Instrumentation is registered, and the change adds whatever counter or span its new failure mode needs.
14. Dependency pins are explicit, including llama-index-readers-file if any file reading happens.
15. The golden-set test passes, and the PR says what its score was before and after.

The complete picture

Stand back from the API surface and the system is simple to describe. There is a write path that turns sources into embedded nodes in a store, run offline, made cheap by hashing and caching, and versioned by a contract covering the embedding model and the chunk shape. There is a read path that reconnects to that store, retrieves under a filter derived from the authenticated caller, narrows with postprocessors, and synthesizes an answer. And there is an orchestration layer — event-driven steps, which agents are a special case of — that sequences the read path's components when one call is not enough, and which needs a durability story the moment a run outlives a process.

Every senior concern lands on one of those three. Security is mostly about the read path's inputs being untrusted and the orchestration layer's tools being code. Cost is mostly about how many tokens the read path sends and how often the write path pays twice for the same chunk. Scale is about the in-memory defaults on the write path and the in-process runtime on the orchestration layer. Multi-tenancy is a filter on the read path, a session key on memory, and a prohibition on touching a global singleton. Upgrades are about a pinned contract on the write path and a changelog habit everywhere.

What makes LlamaIndex pleasant to own, once you have internalised that, is that it does not hide any of it. There is no magic layer to fight. The downside is the same fact from the other side: nothing is provided for you either, and a team that treats the five-line starter example as an architecture will build something that works beautifully until the second tenant arrives.

Where the series leaves you

Across the three levels you have gone from VectorStoreIndex.from_documents to owning a retrieval platform: the ingestion and serving split, the workflow runtime's failure modes and its durability levels, the three trust boundaries, tenancy without a tenancy primitive, where the money goes, an observability stack you can be paged by, a pinning policy aimed at the 0.15 boundary, incident playbooks, and a review checklist.

The natural next steps depend on which part you own. If it is retrieval quality, go to RAGAS and build the golden set properly. If it is the data layer, Qdrant or pgvector for the store you now depend on. If it is operations, Langfuse or Arize Phoenix for per-run visibility and OpenTelemetry for the trace that spans your whole request. If it is tool surface area, MCP, which LlamaIndex supports both as a consumer of MCP server tools and as a provider of workflows-as-MCP-servers. And if you are weighing the orchestration layer against alternatives, read LangGraph with the comparison from "Where LlamaIndex stops" in mind.

Sources

• LlamaIndex Python framework documentation
• Core concepts
• Installation
• Settings
• Ingestion pipeline
• Storing and persistence
• Vector stores
• Node postprocessors
• Agents
• Agent memory
• Multi-agent workflows
• Production RAG
• Observability
• Instrumentation
• Evaluation
• MCP integration
• Workflows
• Managing workflow state
• Workflow streaming
• Human in the loop
• Retrying steps
• Durable workflows
• DBOS runtime
• Workflow deployment
• LlamaParse enterprise readiness
• LlamaParse self-hosting
• llama_index CHANGELOG
• llama-agents repository, workflows CHANGELOG
• llama-index-core on PyPI
• llama-index-workflows on PyPI

At this level, interviews stop asking whether you can build a RAG pipeline. They assume you can. What they probe instead is whether you have owned one when it was expensive, wrong, or shared by teams who did not consult you. The questions below are the ones that actually separate candidates, with the reasoning a strong answer shows and the specific thing a weak answer leaves out.

Everything here is grounded in llama-index-core 0.14.25 and llama-index-workflows 2.25.0. Being precise about versions is itself a signal: the framework removed its entire legacy agent stack in 0.13 and reshaped orchestration in 0.14, so a candidate who talks confidently about AgentRunner and QueryPipeline is describing a system that no longer exists.

Part one: system design prompts

"Design a retrieval platform three product teams will build on."

Start by refusing to answer it as one system. The first thing to say is that there are two deployables with different shapes: an offline ingestion service and an online query service. Ingestion is batch, restartable, throughput-bound and triggered by a schedule or an event. Serving is request-scoped, latency-bound and stateless. The reason to separate them is not tidiness; it is that a serving process which calls VectorStoreIndex.from_documents(...) at startup pays the full embedding bill on every deploy and every replica scale-up, and its cold start grows with the corpus. The serving tier reconnects with VectorStoreIndex.from_vector_store(vector_store) instead.

Then name the contract between them, because this is the part interviewers are listening for: the embedding model and the chunking configuration. Nothing in the framework enforces that both sides agree. If they diverge by dimension you get ValueError: shapes (384,) and (1536,) not aligned; if they diverge without changing dimension you get quietly worse answers and no error at all. The way to make the contract visible is to version it in the collection name — docs_v3 — so that a change is a new collection, a backfill and a pointer swap, which also gives you rollback for free.

Third, say what tenancy means here, because with three teams it will be asked. There is no tenancy primitive in LlamaIndex. Either a collection per tenant, which buys hard isolation and cheap deletion at the cost of linear collection growth, or one collection with a tenant_id in node metadata and a MetadataFilters filter applied on every retrieval. If the shared-collection route, the filter is constructed server-side from the authenticated request, through one sanctioned factory function, and nothing the model produces ever reaches it.

Fourth, the platform deliverable is an internal package, not a wiki page. It exports the LLM, the embedding model, the vector store client and build_retriever(tenant_id, top_k), and it pins LlamaIndex in one place so that an upgrade is one merge request rather than six migrations.

What a weak answer misses: the offline/online split entirely, or the embedding contract, or it answers tenancy with "we'll filter in the application" without saying where the filter value comes from. A weak answer also often puts Settings.llm = ... in the request handler, which is the next question.

"Why is mutating Settings per request a bug?"

Because Settings is a process-wide singleton and your server is async. Setting Settings.llm = tenant_model at the top of a handler means that while your coroutine is suspended at an await, another request can read your value or overwrite it before your synthesis step runs. In a multi-tenant service that is one tenant's content being sent to another tenant's configured model endpoint — a data-residency and confidentiality incident, not a flaky test.

The fix is to treat Settings as boot-time configuration only and pass models locally: index.as_query_engine(llm=tenant_llm), VectorStoreIndex.from_documents(docs, embed_model=...), await agent.run(..., memory=tenant_memory). Local arguments override Settings by design, so nothing is lost.

A strong answer adds the related points: per-tenant Memory(session_id=...) so histories cannot collide, and a fresh Context per run rather than a shared one — workflow state has been keyed by run id plus namespace since Workflows 2.23, which closed a class of cross-run leakage.

"An agent run needs to survive a deploy. How?"

Lay out three levels and say which you would pick.

Level one is not needing durability: keep runs short and idempotent, retry the whole run. For single-turn RAG this is the right answer and everything else is overhead.

Level two is snapshotting the context yourself. Context is serializable: json.dumps(handler.ctx.to_dict()) to save, Context.from_dict(workflow, d) then workflow.run(ctx=ctx) to resume. The natural trigger is a StepStateChanged event with state NOT_RUNNING, which you get from handler.stream_events(expose_internal=True). Two caveats to state explicitly, because they are what the question is really testing. Resume is at-least-once: completed steps are not re-run, but steps that were in flight when you snapshotted are, so any step with a side effect needs an idempotency key derived from the run id. And to_dict() is all-or-nothing: one unserializable value in state or in the in-flight events fails the whole snapshot, which is why clients and connections belong in Resources — created once, cached, never serialized, re-created on resume — rather than in state. Where exotic types are unavoidable, Workflow(serializer=JsonSerializer(allowed_types=[...])) allow-lists them.

Level three is a durable runtime: DBOSRuntime from llama-agents-dbos, backed by Postgres, where re-running with the same run_id resumes. The operational detail worth mentioning is that executor_id must be unique per replica, or replicas contend over the same runs.

What a weak answer misses: claiming there is a built-in checkpointer. There was, and it was removed in Workflows 2.0 along with sub-workflows and stepwise execution. Saying so is a strong signal that you have read the changelog rather than an old tutorial.

"Design the human approval step in an agentic workflow."

Two patterns exist and they are not equivalent. The documented recommendation is the two-step one: a step returns InputRequiredEvent(prefix=...), the run suspends, the caller does await handler.send_event(HumanResponseEvent(response=...)), and a separate step consumes it. The alternative is await ctx.wait_for_event(HumanResponseEvent, waiter_event=..., waiter_id=..., requirements={...}) inline, which is more compact but replays the code before it on resume — so anything non-idempotent above that line runs twice.

Then raise the thing most candidates miss: since Workflows 2.25.0, a workflow's timeout measures accumulated alive time across resumes rather than resetting on each resume. An approval workflow that sits for three days and resumes a dozen times can now time out where it previously did not. If a HITL flow starts failing after an upgrade, that is the first thing to check.

Part two: incident response

"Answer quality dropped. No deploy, no errors. Go."

The discipline is to separate retrieval from synthesis before touching either, because they have disjoint fixes and tuning the wrong one wastes a day.

Confirm the index first: collection name, document count, and a known document retrievable by a query you have an expected answer for. Then retrieve without synthesis — index.as_retriever(similarity_top_k=5).retrieve(q) — and read the NodeWithScore scores and sources. If the right nodes come back and the answer is still wrong, it is synthesis: the prompt, the response_mode, or the provider changing a model behind a floating alias. If the right nodes do not come back, it is retrieval, and the next check is whether the embedding model in the serving config is the one the collection was built with.

The specific cause worth naming, because it produces exactly this symptom with no error, is a missing llama-index-readers-file. The umbrella llama-index package dropped it in 0.14.18 and the install docs still list it. Without it, SimpleDirectoryReader logs a warning and reads PDFs and DOCX files as plain text, producing garbage chunks. A corpus re-ingested on a fresh machine after that version is the classic trigger.

"Workflow runs are piling up and never completing."

Check timeouts first, and know that the defaults are asymmetric: a plain Workflow defaults to timeout=45.0 seconds, while the agent classes default to timeout=None. So custom workflows fail too eagerly and agents hang forever. A WorkflowTimeoutError names the active steps in its message, which is genuinely useful during an incident.

Second, the iteration cap: agents default to max_iterations=20 and the error tells you to raise it or switch to early_stopping_method="generate". Both are the wrong first move. Twenty tool rounds almost always means a tool whose description is ambiguous enough that the model keeps retrying it, or a tool returning an error string the model reads as "try again". Raising the cap raises cost and latency without addressing either.

Third, the event stream. It is consumable once — WorkflowRuntimeError: All the streamed events have already been consumed. — and a run whose stream has a dead or contending reader can stall. Through WorkflowServer the same rule applies: one reader per run, with no replay.

Fourth, on a durable runtime, duplicate executor_id across replicas.

"The embedding bill tripled overnight."

Three causes account for nearly all of these. The IngestionPipeline cache is not actually shared, so each worker built its own in-memory cache and nothing hits; the fix is a RedisKVStore, MongoDBCache or FirestoreCache behind IngestionCache. Or a document's hash changes on every run even though its content did not, usually because metadata participates in the hash and something injects a timestamp. Or someone added a metadata extractor: TitleExtractor makes one LLM call per node, so over a hundred thousand chunks it dwarfs the embedding cost entirely.

The measurement that settles it in five minutes: run the pipeline twice over an unchanged corpus with a TokenCountingHandler and compare total_embedding_token_count. On a working cache the second run is near zero.

"A tenant reports seeing another tenant's content."

Treat it as a security incident and capture evidence first: the exact query and the retrieved node ids, before changing anything. The node metadata tells you which tenant the content actually belonged to, and that single fact distinguishes a filter bug from a document that was ingested into the wrong partition in the first place — two incidents with completely different remediation.

Then check, in order: whether the retriever was built through the sanctioned tenant-filtered factory or a raw index.as_retriever(); whether anything in the request path mutated Settings; whether the Memory session key can collide across tenants; and whether any tool received a tenant identifier from model output rather than from the authenticated request. That last one is the injection path: if the tenant id can come from something the LLM produced, then a malicious document in the corpus is a cross-tenant read.

Part three: arguing trade-offs to leadership

"Convince me we need a second service just for ingestion."

Lead with the number, not the architecture. Measure the embedding tokens your current startup spends — MockEmbedding plus TokenCountingHandler gives you this without spending anything — multiply by deploys per week and by replica count, and put a currency figure on it. Then add the second cost, which is availability: a serving process that rebuilds at startup cannot start when the document source is slow, so your document bucket is now in the critical path of your API's readiness.

The counter-argument you should pre-empt is operational overhead: two deployables, two pipelines. Concede it, and bound it. The ingestion service is a batch job on a schedule, not a second always-on API, and it shares the internal package with the serving tier so there is no duplicated configuration.

"Why can't we just upgrade to the latest LlamaIndex every sprint?"

Explain the version model honestly, because it is favourable to you. The minor number is the breaking boundary: 0.13 → 0.14 was breaking, 0.14.x patches are intended not to be, and every integration package pins llama-index-core<0.15.0. So routine patch upgrades inside 0.14 are low-risk and should be frequent, and that is the policy you want. What you are protecting against is the 0.15 bump, which will require every integration you depend on to be re-released — a window of days to weeks where a long-tail package may not be ready.

Then name the two things that are expensive regardless of version discipline. Re-indexing, which is required whenever the embedding model or chunking changes and costs real money and coordination. And persisted-artefact compatibility: the JSON layout of SimpleVectorStore, the docstore and context snapshots is versioned with the library, so a load test against the previous version's artefacts belongs in CI.

Close with the ask, which is cheap: a standing monthly changelog review of both core and llama-index-workflows. The workflows changelog is where behaviour changes hide without a major version — 2.23.1 shortened every retry delay by one step via an off-by-one fix, and 2.25.0 changed timeout accounting across resumes.

"Should we use LlamaIndex or LangGraph?"

Answer on where the difficulty is, not on features. If the corpus is the hard part — messy PDFs, a large heterogeneous collection, incremental and cheap ingestion — LlamaIndex, because that is what the data layer is built for and the Workflows engine is good enough for the orchestration such a system needs. If retrieval is trivial and the orchestration graph is the hard part, with checkpointing and a visual debugger as first-class requirements, LangGraph is more specialised at exactly that.

A strong answer names a third option: neither, if the organisation cannot absorb a quarterly dependency review. A managed service costs more money and less of your time, and recommending it when it is right is a senior behaviour rather than a concession.

"Why do we need a sandbox for the code agent? It's our own code."

It is not. CodeActAgent executes code the model wrote, through a code_execute_fn you supply — the framework refuses to construct one without it (ValueError: code_execute_fn must be provided for CodeActAgent). Passing exec runs model-authored code in your process with your process's credentials. And because retrieved documents are, in any realistic corpus, partly authored outside your trust circle, the model's instructions are partly attacker-influenced. The chain from "a support ticket contains text" to "arbitrary code runs with your database credentials" is three steps long.

The ask is a container with no network and no credentials, which your platform already knows how to run. Frame the cost as the latency of a container start per execution, and the alternative as unbounded.

Common interview questions

• "What replaced QueryPipeline?" Workflows. The module was removed in 0.13 and import llama_index.core.query_pipeline raises ModuleNotFoundError.
• "What is the default similarity_top_k and why does it matter?" Two. It matters because it, times chunk_size (default 1024 tokens), is roughly your per-query prompt size and therefore your per-query cost.
• "Where does IngestionPipeline deduplicate?" With a docstore attached it hashes doc_id plus content: a changed hash re-processes and upserts, an unchanged one is skipped. With a vector store attached but no docstore, it only removes duplicate inputs — a much weaker guarantee than people assume.
• "What does return_direct=True do, and when is it a security control?" It ends the agent loop with the tool's output as the answer. It is a security control because the output never re-enters the model's reasoning, so injected text in that output cannot steer subsequent tool calls.
• "Why does FAISS need extra work?" It stores vectors only, not text, so it needs a separate docstore. Most hosted stores keep text alongside vectors, so persist() is unnecessary with them.
• "How do you read workflow state?" await ctx.store.get(key, default=...), await ctx.store.set(key, value), and async with ctx.store.edit_state() as state: for atomic read-modify-write. The old ctx.get() raises AttributeError. Keep LLM calls outside the edit_state lock.
• "What is a Resource?" A dependency injected into a step via Annotated[T, Resource(factory)]: created once, cached, never serialized, re-created on resume. It is how you keep clients out of snapshots.
• "How would you alert on retrieval quality?" Count queries where no node clears the similarity cutoff. It raises no exception, so nothing else will tell you.

Final self-test

Answer these aloud in sixty seconds each. If you hesitate, that is the section to reread.

1. Name the contract between the ingestion tier and the serving tier, and the two ways it breaks.
2. Why is Settings.llm = x in a request handler a confidentiality bug?
3. What exactly is re-run when a workflow context snapshot resumes, and what does that imply for a step that charges a card?
4. Which three trust boundaries exist in a LlamaIndex application, and which one is the reason a tenant filter must never come from model output?
5. Two workflows hit a timeout: one a custom Workflow, one a FunctionAgent. What were the defaults, and which surprise belongs to Workflows 2.25.0?
6. The embedding bill tripled. Name three causes and the one measurement that distinguishes them.
7. What will the 0.15 upgrade require that a 0.14.x upgrade does not?
8. Why does a missing llama-index-readers-file degrade answer quality without raising an error?
9. How many readers can consume a run's event stream, and what breaks when two try?
10. Give one situation where the right recommendation is not LlamaIndex, and say why.

These are the things that only show up once a LlamaIndex deployment is shared, large, or old. Each one has cost somebody a weekend. Verified against llama-index-core 0.14.25 and llama-index-workflows 2.25.0.

Traps that only appear at scale

The in-memory store that worked until the second replica. SimpleVectorStore and SimpleDocumentStore are JSON loaded fully into one process's RAM. With two replicas you get two divergent copies and no error — just two users receiving different answers to the same question. Move to a real vector store before you scale out, not after you are debugging a ghost.

Settings mutated inside a request handler. It is a process-wide singleton and your handler is async. Between your assignment and your synthesis call there is an await, and another request can read or overwrite your value in that gap. In a multi-tenant service this sends one tenant's content to another tenant's model endpoint. Pass models locally — index.as_query_engine(llm=...), agent.run(..., memory=...) — and treat Settings as boot-time only.

The embedding contract drifting between tiers. Ingestion and serving are deployed separately but joined by the embedding model and chunk configuration, and nothing enforces it. A dimension mismatch at least errors: ValueError: shapes (384,) and (1536,) not aligned. A same-dimension model swap does not error, it just answers worse. Version the contract in the collection name so a change is a new collection and a pointer swap.

The umbrella package quietly shrinking. llama-index dropped llama-index-readers-file in 0.14.18, the LlamaCloud index and LlamaParse readers in 0.14.19, and llama-index-cli in 0.14.20. The installation docs have not caught up. Without readers-file, SimpleDirectoryReader logs a warning and reads PDFs and DOCX as plain text, producing garbage chunks and no exception. Depend on it explicitly.

A metadata timestamp defeating the ingestion cache. Metadata participates in the hash, so anything that changes per run — a reprocessing timestamp, a build id — makes every node a cache miss and you pay the full embedding bill every time. Symptom: two consecutive runs over an unchanged corpus report similar total_embedding_token_count. On a working cache the second run is near zero.

Hidden LLM calls in what looks like data processing. TitleExtractor and the other metadata extractors call the LLM once per node at ingestion. LLMRerank calls it per query. FactExtractionMemoryBlock calls it as conversations grow. The LLM-based evaluators call it per evaluation. A TitleExtractor over a hundred thousand chunks is a hundred thousand calls, and it will exceed your embedding spend.

The 45-second default on custom workflows, and None on agents. The defaults are wrong in opposite directions: a plain Workflow times out faster than one reasoning-model call, while FunctionAgent has no deadline at all and will hang on a stalled provider. Set both deliberately.

The timeout semantics change in Workflows 2.25.0. A workflow's timeout now measures accumulated alive time across resumes rather than restarting the clock on each resume. Long-suspended human-in-the-loop runs that resume repeatedly can now time out where they previously did not. If an approval flow starts failing after an upgrade, this is why.

The event stream being consumable once. WorkflowRuntimeError: All the streamed events have already been consumed. Two parts of a request handler both reading handler.stream_events() means one of them silently gets nothing — usually the audit log, which nobody checks. Consume once, fan out yourself. The same single-reader rule applies to WorkflowServer's /events/{handler_id}, with no replay.

to_dict() failing because one value is exotic. Context snapshots are all-or-nothing: a client, a socket or a byte buffer anywhere in state or in-flight events fails the whole snapshot with a ContextSerdeError. Keep those in Resources.

At-least-once resume re-running side effects. On resume, completed steps are skipped but in-flight steps re-run. A step that sent an email will send it twice. Idempotency keys derived from the run id, on every side-effecting step.

wait_for_event replaying the code above it. await ctx.wait_for_event(HumanResponseEvent, ...) replays everything before it on resume. The two-step InputRequiredEvent / HumanResponseEvent pattern is the documented recommendation for exactly this reason.

Duplicate executor_id on a durable runtime. Two DBOS replicas sharing an executor_id contend over the same runs. One id per replica, derived from the pod name.

max_iterations=20 treated as a tuning knob. Hitting it means a tool description the model keeps misreading, or a tool returning an error string the model treats as "retry". Raising the cap buys cost and latency, not correctness. 0.14.25 helps by no longer retrying failed function tools automatically.

WorkflowServer on a reachable address. Run, send-event and cancel endpoints are unauthenticated and a debugger UI is mounted at /. Private network and your own gateway, always. The default MemoryWorkflowStore also loses everything on restart; use SqliteWorkflowStore or something durable.

Platform-ownership habits

Ship a package, not a document. An internal yourco-rag exporting get_llm(), get_embed_model(), get_vector_store(), build_retriever(tenant_id, top_k) and instrument() makes the correct configuration the default configuration. It also pins LlamaIndex in one repository, so a core upgrade is one merge request rather than six migrations.

Own the embedding and chunking decision centrally. It is a platform decision because changing it costs a re-index. Product teams should not be able to change it accidentally, and the collection name should make the current version visible from a dashboard.

One sanctioned retriever factory, enforced. Make raw index.as_retriever() in product code a review failure or a lint error. The failure mode of a forgotten tenant filter is silent and severe, so it needs a mechanical control rather than a cultural one.

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

def build_retriever(index, tenant_id: str, top_k: int = 5):
    if not tenant_id:
        raise ValueError("tenant_id is required; it must come from the authenticated request")
    return index.as_retriever(
        similarity_top_k=top_k,
        filters=MetadataFilters(filters=[MetadataFilter(key="tenant_id", value=tenant_id)]),
    )

One shared rate limiter per budget. Provider quotas are per account, not per client object, so two independently throttled clients still collectively exceed them.

from llama_index.core.rate_limiter import TokenBucketRateLimiter

limiter = TokenBucketRateLimiter(requests_per_minute=30, tokens_per_minute=100_000)
llm = OpenAI(model="gpt-4o-mini", rate_limiter=limiter)
embed_model = OpenAIEmbedding(model="text-embedding-3-small", rate_limiter=limiter)

SlidingWindowRateLimiter is the alternative when the provider enforces a rolling window.

A golden set in CI. Twenty to fifty questions with known answers and known source documents, run against the real index. It is the only thing that catches a chunking regression, a reranker silently dropped from a config, or a provider changing a model behind a floating alias.

Per-tenant token attribution from day one. TokenCountingHandler exposes total_llm_token_count, total_embedding_token_count, prompt_llm_token_count and completion_llm_token_count. Wire them to a metric labelled by tenant before the first invoice arrives, not after.

A monthly changelog review of both repositories. Core's changelog and llama-index-workflows's. The workflows one is where behaviour changes hide without a major version: 2.23.1 shortened every retry delay by one step through an off-by-one fix, 2.25.0 changed timeout accounting.

A persisted-artefact load test. The JSON layout of SimpleVectorStore, the docstore and context snapshots is versioned with the library. A CI test that loads last version's artefacts under the new version turns a production surprise into a red build.

Validate workflow graphs in tests. workflow.validate(validate_resources=True) in a unit test catches a broken graph before the first production run. Disable graph checks narrowly when a step dispatches dynamically with ctx.send_event — @step(skip_graph_checks=["reachability"]) on that one step — never with disable_validation=True on the workflow.

Operational recipes

Reconnect, never rebuild.

index = VectorStoreIndex.from_vector_store(
    QdrantVectorStore(client=client, collection_name="docs_v3"),
    embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
)

Forecast a cost before spending it. Run the real pipeline with MockLLM and MockEmbedding plus a TokenCountingHandler against a 10% sample, multiply out, and you have a number for the design review. MockFunctionCallingLLM does the same for agent tests.

Snapshot on a step boundary.

import json

handler = workflow.run(query=q)
async for ev in handler.stream_events(expose_internal=True):
    if type(ev).__name__ == "StepStateChanged" and ev.step_state.name == "NOT_RUNNING":
        await store.put(run_id, json.dumps(handler.ctx.to_dict()))
result = await handler

Keep clients out of state.

from typing import Annotated
from workflows.resources import Resource

class Flow(Workflow):
    @step
    async def retrieve(self, ctx: Context, ev: StartEvent,
                       store: Annotated[QdrantVectorStore, Resource(get_store)]) -> Retrieved:
        ...

Separate retrieval from synthesis when debugging quality. Call .retrieve(q) on the retriever alone and read the scores and source filenames. Right nodes plus wrong answer is a synthesis problem; wrong nodes is a retrieval problem. These have disjoint fixes, and the single most common waste of a day is tuning the wrong one.

Re-index as a new collection. Build docs_v4 alongside docs_v3, run the golden set against both, swap the pointer, keep the old collection for a week. Rollback is then a config change rather than another backfill.

Atomic state edits without holding a lock over an LLM call.

async with ctx.store.edit_state() as state:
    state["state"]["reviewed_count"] += 1     # fast, no I/O
# the LLM call goes out here, outside the lock

The pull-request checklist

1. No legacy imports: nothing from llama_index.core.query_pipeline, no FunctionCallingAgent, no AgentRunner, no ReActAgent.from_tools, no ServiceContext, no agent.chat(). Agents are async: await agent.run(...).
2. No Settings mutation outside startup.
3. Every retriever built through the tenant-filtered factory.
4. Timeouts set explicitly on both workflows and agents.
5. max_iterations and early_stopping_method chosen, with a stated reason.
6. No secrets in source: no api_key= literals, no credentials inside async_database_uri or system_database_url.
7. Tools narrow and typed, authorizing against the request identity; no free-form SQL string, file path or URL parameters.
8. code_execute_fn, if present, is sandboxed — no exec, no network, no credentials.
9. Workflow state holds no clients or connections; to_dict() is exercised in a test.
10. Side-effecting steps idempotent, keyed by run id.
11. Ingestion pipeline has a shared remote cache and a docstore, with the embedding stage present when a vector store is attached.
12. Embedding model and chunk size unchanged, or a re-index plan and a new collection name are in the PR.
13. Instrumentation registered, and any new failure mode has a counter or span.
14. Pins explicit, including llama-index-readers-file wherever files are read.
15. Golden-set score reported before and after.

The design-document checklist

1. Two tiers or one, and why. If one, state the embedding cost per deploy and per replica and show that someone accepted it.
2. The embedding and chunking contract, named, versioned, and visible in the collection name.
3. The tenancy model. Collection-per-tenant or filtered-shared, and specifically where the tenant identifier comes from. If any answer involves model output, the design is rejected.
4. Data residency, if any tenant contract names a jurisdiction. The boundary has to extend to the model endpoint and the vector store region, not just the filter — which usually means per-tenant model configuration resolved at request time. For LlamaParse, the options are managed SaaS in North America or the EU, single-tenant SaaS, bring-your-own-cloud on AWS, Azure or GCP, or self-hosting via Helm on Kubernetes.
5. The durability level, chosen from the three: none with idempotent retries, self-managed context snapshots, or DBOSRuntime on Postgres. Say which and why.
6. The serving surface. FastAPI or WorkflowServer, and if the latter, where the authentication layer sits, since none of its endpoints authenticate.
7. Rate limiting and quota, shared across LLM and embedding clients.
8. Observability, with the four signals named: timeout rate, max-iteration rate, zero-result retrieval rate, token spend per tenant per hour.
9. Cost model, broken into synthesis tokens, embedding tokens, hidden LLM calls and vector store cost, with the lever identified for each.
10. Pins and the upgrade path, explicitly mentioning that 0.15 will require every integration to be re-released.
11. The re-index plan, including who pays for it and how long the old collection is kept.
12. The golden set: how many questions, who owns them, where they run.
13. What is explicitly out of scope — the honest list usually includes authentication, tenancy enforcement outside the retriever, sandboxing and scheduling, because none of them come from the framework.

Incident one-pagers worth pinning

Quality dropped, no error, no deploy. Confirm the index identity first: collection name, document count, one known document retrievable. Then retrieve without synthesis and read the scores and source filenames. Then compare the serving embedding model against the one that built the collection. If the right nodes come back and the answer is still wrong, stop looking at retrieval. The specific silent cause to rule out early is a missing llama-index-readers-file on a freshly built machine, which turns PDFs into plain-text garbage without raising anything.

Runs piling up. Timeout, then iteration cap, then stream reader, then executor_id collisions on a durable runtime. The WorkflowTimeoutError message names the currently active steps, which usually identifies the culprit in one line. If runs are stuck in a state that is neither running nor finished, check whether a step is waiting on a HumanResponseEvent that nothing will ever send — an abandoned approval looks identical to a hang.

Cost spike. Two consecutive runs over an unchanged corpus with a TokenCountingHandler, comparing total_embedding_token_count. Near zero on the second run means the cache is healthy and the spike is elsewhere — most likely a newly added extractor or reranker making LLM calls. A similar number on both runs means the cache is not being hit, and the causes are a per-process in-memory cache or a metadata field that changes every run.

Suspected cross-tenant exposure. Capture the query and the retrieved node ids before touching anything; the node metadata says which tenant the content belonged to, and that distinguishes a filter bug from a mis-ingested document. Then walk the four candidates in order: a raw as_retriever() call bypassing the factory, a Settings mutation in the request path, a colliding Memory session key, and a tenant identifier that reached a tool from model output rather than from the authenticated request.

A workflow refuses to start after an upgrade. Read the validation error literally — it names the events produced but never consumed, consumed but never produced, or the unreachable and dead-end steps. The usual cause is a step whose return annotation no longer matches what it sends, or dynamic dispatch the analyser cannot see.

Cost and latency levers, in order of return

1. The ingestion cache, shared and remote. Biggest single win on any corpus that is re-ingested.
2. similarity_top_k down, reranker in. Fewer, better nodes beat more nodes, and the prompt shrinks.
3. A cheaper model per component. Routing, query rewriting and metadata extraction rarely need the expensive model; pass it locally rather than globally.
4. response_mode. compact is the query-engine default and packs context into as few calls as possible. refine makes one call per node — switching to it on a top_k=10 engine turns one call into ten.
5. embed_batch_size and insert_batch_size. Throughput levers. The insert batch defaults to 2048 nodes; lower it when a remote store rejects large writes or memory is tight.
6. await pipeline.arun(...) over pipeline.run(num_workers=N) for the I/O-bound case, which most ingestion is. Reach for num_workers (a multiprocessing.Pool) only when profiling shows CPU-bound parsing, and expect little from it on Windows. RayIngestionPipeline for genuinely distributed runs.
7. Agent max_iterations, as a cost ceiling rather than a tuning knob.
8. Local models through Ollama with HuggingFace embeddings, where RAM and latency allow — removes API cost, adds hardware cost.

Things to stop carrying forward

ServiceContext raises on any use; use Settings or local kwargs. llamaindex-cli is deprecated and broken on 0.14.25, which also means download_llama_pack, download_loader and llamaindex-cli upgrade are not available — migrate by hand. ctx.get() and ctx.set() are gone in favour of ctx.store. ChatMemoryBuffer and the other legacy memory classes are documented as deprecated in favour of Memory.from_defaults(...); note that agents in 0.14.25 still build a ChatMemoryBuffer when you pass no memory, so pass one explicitly rather than waiting for the default to change. ConstantDelayRetryPolicy and ExponentialBackoffRetryPolicy are superseded by the composable retry_policy(retry=..., wait=..., stop=...). The legacy CallbackManager and set_global_handler path still works and many partner integrations use it, but new instrumentation belongs on the instrumentation dispatcher.