The Complete TruLens Guide

This is part one of three. It covers everything you need to do real work with TruLens, not a teaser. By the end you can take a small question-answering app that calls a language model, record every run of it, score each answer for relevance and for hallucination, open a dashboard that compares two versions of the app, and read the scores well enough to decide which version to ship. Mid-level and Senior take the same topics further; nothing here is thrown away.

This guide is written against TruLens 2.14.0, the release current in September 2026. TruLens changes quickly, and a lot of the blog posts and tutorials you will find online describe older APIs. Where an older pattern is still common on the internet, this guide names it and shows the current form, so you can recognise a stale tutorial when you meet one.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas stick only after you have seen a score appear for an answer you produced yourself.

Why evaluating an LLM app is hard, and what TruLens does about it

Ordinary software has a comforting property: give it the same input and you get the same output, so you can write a test that says "this input must produce exactly that output". An application built on a large language model (LLM, a model that generates text) does not behave that way. Ask it the same question twice and the wording differs. Change one sentence in the prompt, swap the model for a cheaper one, or re-index your documents, and the answers shift in ways that no assert answer == "..." line can capture. Worse, the failures are not crashes. The app returns a fluent, confident, well-formatted paragraph that happens to be wrong, or that answers a different question, or that quietly invents a fact that appears in none of your documents. The word for that last failure is hallucination.

Teams used to handle this by reading outputs by hand. Someone pastes twenty questions into the app, eyeballs the answers, says "looks fine", and ships. That works exactly once. The second time you change something, you have no record of what the twenty answers looked like before, no way to tell whether the new version is better or merely different, and no way to read two hundred answers a day once real users arrive. You need two things that manual reading cannot give you: a record of what the app did on every call, and a score for each call that you can average, compare, and watch over time.

TruLens is an open-source Python library that provides exactly those two things. It does three jobs, and it helps to keep them apart in your head:

1. Tracing. It watches your app run and records each step as a structured event: the question that came in, the documents that were retrieved, the prompt that was sent to the model, the answer that came back, how long each step took, and what it cost.
2. Evaluation. It runs metrics over those recordings. A metric is a small function that looks at part of a recording and returns a score between 0 and 1. Many of TruLens's metrics use a second language model as a judge: the judge reads the question and the answer and says how relevant the answer is.
3. Storage and display. It saves the recordings and scores in a database and gives you a dashboard, so that you can compare version A of your app with version B on the same numbers.

The loop that results is the heart of the tool, and the rest of this guide fills in each step of it:

A little history explains some of the names you will meet. The project began at a company called TruEra, which Snowflake later acquired; Snowflake now maintains it, and the code is MIT-licensed and lives at github.com/truera/trulens. The early library was called trulens_eval, and its central object was called Tru. In version 1.0 (2024) the library was split into separate packages under a single trulens namespace, Tru became TruSession, and the old trulens_eval package became an empty shim that exists only to print a deprecation notice. In 2.3.0 (August 2025) the tracing machinery switched to OpenTelemetry by default, and in 2.7.0 (February 2026) the word "feedback function" gave way to "metric". You will still see all the old words in search results. The current words are the ones in this guide.

What TruLens is not matters too. It is not a framework for building the app; you build the app with plain Python, LangChain, LlamaIndex, or whatever you like, and TruLens watches it. It is not a hosted service: the core library runs inside your own Python process and writes to a file or database you control. And it is not a magic truth detector. Its scores come from other language models or from simple code, and they are evidence to reason about, not verdicts. A later section returns to that point, because treating a judge's score as ground truth is the most common conceptual mistake beginners make.

The places TruLens sits next to other tools are worth knowing. Langfuse and LangSmith are tracing and evaluation platforms with a similar purpose and different emphasis; RAGAS and DeepEval are libraries that focus on scoring. This catalogue has guides for them, and you can read them afterwards to compare how each approaches the same problem. TruLens's particular stance is that evaluation should be built from a small set of named, explainable metrics, and that every score should be traceable back to the exact spans of the exact run that produced it.

The mental model: five nouns

TruLens has a long API, but a beginner needs only five nouns. Learn them now, because every error message and every doc page assumes you know them.

Session. The TruSession is the object that owns everything stateful: the connection to the database, the machinery that exports recordings, and the background workers that run metrics. There is only one per Python process. Calling TruSession() a second time returns the same object rather than creating a new one, which has a consequence we will meet in the errors section: you cannot change the database after the first call.

App. Your application, wrapped. You hand TruLens your Python object and give it two labels, an app_name and an app_version. The wrapper is called a recorder. For a plain Python class the wrapper is TruApp; for LangChain it is TruChain, for LangGraph TruGraph, for LlamaIndex TruLlama. The app_name plus app_version pair is what the dashboard groups by, so "RAG / v1" and "RAG / v2" are two entries you can compare. Choosing good version labels is a real skill, and the habits section of this guide and the tips file come back to it.

Record (and span). One run of your app is a record. Inside the record, each step is a span: one retrieval, one model call, one tool call. Spans nest, so the record as a whole is the outermost span, and the retrieval and generation sit inside it. Every span has a span type, which tells TruLens what kind of step it was. The types you need first are RECORD_ROOT (the whole run), RETRIEVAL (fetching documents), and GENERATION (asking the model). The word "record" is slated to be renamed "trace" in future releases, so you will see both.

Metric. A scoring function. You create one with Metric(...), and it needs three things: the function that does the scoring (the implementation), a selector for each of the function's inputs saying which part of the recording to feed it, and a name. Old tutorials call this a "feedback function" and use a class called Feedback. Feedback still works in 2.14.0, but constructing one prints a deprecation warning, and it will eventually be removed.

Provider. The model backend that does the judging. OpenAI is a provider; so are Anthropic, Google, Bedrock, LiteLLM, and a local one for Ollama. A provider object has many ready-made scoring methods, such as relevance and groundedness_measure_with_cot_reasons, and you pass one of those methods as the implementation of a metric.

Put together, the picture looks like this:

Two more ideas complete the model. The first is the selector. A metric such as "is this answer relevant to the question" needs two strings: the question and the answer. But your app does not hand TruLens a tidy pair; it records a tree of spans with attributes. A selector is a pointer into that tree: "the input of the record root", "the output of the record root", "every retrieved chunk". You attach one selector to each parameter of the metric's function, and the parameter names must match, which is the source of the most common beginner error. We will spend a whole section on it.

The second is the RAG Triad. RAG means retrieval-augmented generation: instead of asking the model a question cold, your app first searches a collection of your own documents, then pastes the best matches into the prompt, then asks the model to answer using them. It is the most common shape of LLM app in practice, and it can fail in three separate places. The retrieval can fetch the wrong documents. The model can ignore the documents and make something up. Or the model can answer faithfully from the documents but miss the question. TruLens's headline evaluation has one metric for each failure:

If all three are high, the app retrieved the right material, stuck to it, and answered what was asked. If one is low, it tells you which part to fix, and that diagnostic power is what makes the triad more useful than a single "quality" number.

Installing TruLens and checking the setup

TruLens installs with pip and is split into several packages, so that you only download what you use. You need Python 3.10 to 3.13 for the main trulens package. A virtual environment is strongly recommended, because the library pulls in a fair number of dependencies and you do not want them fighting with your other projects.

On Linux or macOS:

python3 -m venv .venv
source .venv/bin/activate
pip install trulens trulens-providers-openai

On Windows (PowerShell):

py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1
pip install trulens trulens-providers-openai

The two packages do different things. trulens is a meta-package that installs the core library (trulens-core), the built-in metrics (trulens-feedback), the dashboard (trulens-dashboard), and the attribute definitions (trulens-otel-semconv). It deliberately does not install any provider or framework integration. trulens-providers-openai is the provider we will use as our judge. If you would rather judge with another model, install the matching package instead. The common ones are trulens-providers-anthropic, trulens-providers-google, trulens-providers-bedrock, trulens-providers-litellm (one package that reaches over a hundred models), and trulens-providers-ollama for models running on your own machine.

The wrappers for frameworks follow the same pattern. If your app is written with LangChain, add trulens-apps-langchain; for LangGraph, trulens-apps-langgraph; for LlamaIndex, trulens-apps-llamaindex. The wrapper for a plain Python class, TruApp, ships inside trulens-core and needs nothing extra.

The judge needs credentials. For OpenAI, set the key as an environment variable before you run anything. TruLens also reads a .env file in the current directory or a parent, so either approach works:

export OPENAI_API_KEY="sk-..."          # Linux / macOS

$env:OPENAI_API_KEY = "sk-..."          # Windows PowerShell

Never paste a key into a notebook cell or a file you will commit. If you use a .env file, add it to .gitignore first. Other providers read their own variables: ANTHROPIC_API_KEY for Anthropic, AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT for Azure OpenAI, and the usual cloud credentials for Bedrock and Google.

Now verify the install. The first command checks that the packages import and prints the version; the second lists everything TruLens-related that pip installed:

python -c "import trulens.core, trulens.dashboard; from importlib.metadata import version; print(version('trulens-core'))"
pip list | grep -i trulens

The first command should print 2.14.0, or a later version if you are reading this after a new release. On Windows, replace grep -i with findstr. One subtlety worth knowing: all the trulens-* packages are released together and share a version number. If pip list shows trulens-core at one version and a provider at a different one, upgrade them together, because mismatched versions are a classic source of confusing import errors.

Finally, create a session:

from trulens.core import TruSession

session = TruSession()

The first time, TruLens prints a line that begins with a squid emoji and says Initialized with db url sqlite:///default.sqlite. That line tells you where your data lives: a file called default.sqlite in whatever directory you ran Python from. Two beginner surprises follow from that. The first is that if you run the same script from a different directory, you get a different, empty database, and wonder where your results went. The second is that the file keeps growing as you run things; we will see how to reset it in a moment.

If you are offline or do not want to spend money on a hosted judge while learning, you can run a model locally with Ollama and use the trulens-providers-ollama package. Small local models give noisier scores than large hosted ones, but the workflow is identical, and for learners in regions where sending prompts to a foreign API is a data-residency concern, a local judge is a legitimate default. The mid and senior guides return to that choice.

Your first project, step by step

We will build a tiny RAG app, record it, score it, and look at the results. To keep the moving parts few, the "document collection" is a Python list and the retriever is a keyword match; in a real project this is where a vector database goes, and the instrumentation you learn here does not change when you swap it. The generator calls OpenAI through its normal client, which the provider package installs for you.

Create a file called app.py. We build it in four pieces, then run it.

Step 1: the app, with no TruLens yet

from openai import OpenAI as OpenAIClient

DOCS = [
    "Employees receive 25 days of paid annual leave per calendar year.",
    "Unused leave can be carried over, up to a maximum of 5 days.",
    "The office is open Sunday to Thursday, from 9 am to 5 pm.",
    "Remote work is allowed up to two days per week with manager approval.",
]

client = OpenAIClient()


class HRBot:
    def retrieve(self, query: str) -> list[str]:
        words = {w.lower().strip("?.,") for w in query.split()}
        scored = [(len(words & set(d.lower().replace(".", "").split())), d) for d in DOCS]
        scored.sort(reverse=True)
        return [d for score, d in scored[:2] if score > 0]

    def generate_completion(self, query: str, context_list: list[str]) -> str:
        context = "\n".join(context_list)
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Answer using only the context. If it is not there, say you do not know."},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
            ],
        )
        return response.choices[0].message.content

    def query(self, query: str) -> str:
        context_list = self.retrieve(query)
        return self.generate_completion(query, context_list)

Read it as three steps: retrieve finds up to two documents that share words with the question, generate_completion asks the model to answer from them, and query is the public entry point that chains the two. This is ordinary Python. It is worth running HRBot().query("How many days of annual leave do I get?") once by itself, before adding any TruLens, so you know the app works and your key is valid. Debugging an instrumented app whose underlying app is broken is a miserable way to spend an afternoon.

Step 2: mark the steps with @instrument

TruLens cannot watch code it has not been told about. You tell it by adding the @instrument decorator to the methods that are steps, and you say what type of step each is and which values to capture. Add these imports at the top of the file and decorate the three methods:

from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes


class HRBot:
    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list[str]:
        ...  # same body as before

    @instrument(span_type=SpanAttributes.SpanType.GENERATION)
    def generate_completion(self, query: str, context_list: list[str]) -> str:
        ...  # same body as before

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def query(self, query: str) -> str:
        ...  # same body as before

The ... lines stand for the bodies you already wrote; keep them. The attributes dictionaries are the important part. The keys are the names TruLens uses for well-known facts, from SpanAttributes. The values are strings that say where to find the fact in the function call: the name of one of its parameters, or the special word "return" for the returned value. So RETRIEVAL.QUERY_TEXT: "query" says "the retrieval's query text is the argument called query", and RETRIEVAL.RETRIEVED_CONTEXTS: "return" says "the retrieved contexts are whatever this method returns". On the record root, INPUT is the question that came in and OUTPUT is the final answer.

Why all this ceremony? Because the metrics need to find these values later. The groundedness metric needs "the retrieved contexts", and it locates them by looking for a span of type RETRIEVAL with the RETRIEVED_CONTEXTS attribute. If you forget to mark the retrieval, nothing crashes; the metric simply finds no context and produces no score. That is the quietest and most common failure in the whole tool, and this paragraph is the cure.

Step 3: define the metrics

Now create the judge and the three RAG Triad metrics. Put this below the class:

import numpy as np
from trulens.core import Metric, Selector, TruSession
from trulens.apps.app import TruApp
from trulens.providers.openai import OpenAI

session = TruSession()
session.reset_database()          # start from an empty database while learning

provider = OpenAI(model_engine="gpt-4o-mini")

f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
        "question": Selector.select_record_input(),
    },
)

f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

Take it slowly, because this block holds most of the vocabulary. provider = OpenAI(model_engine="gpt-4o-mini") creates the judge and pins its model. Always name the model explicitly; the class has a default, but defaults change between releases, and an unpinned judge means your scores can shift when you upgrade the library rather than when you change your app.

Each Metric takes an implementation, which here is a method of the provider, a name that appears in the dashboard, and a dictionary of selectors. The dictionary's keys must match the parameter names of the implementation. The relevance method is declared as relevance(prompt, response), so the keys are "prompt" and "response". Context relevance is context_relevance(question, context), and groundedness takes source and statement and optionally question. Get a key wrong and the metric fails or silently scores nothing. The names ending in _with_cot_reasons return a score plus a written explanation (CoT stands for chain of thought: the judge reasons step by step and TruLens keeps the reasoning), and that explanation is what you will read in the dashboard to understand a low score.

The selectors deserve a second look. Selector.select_record_input() and select_record_output() point at the question and answer captured on the record root. Selector.select_context(collect_list=True) points at the retrieved contexts. The collect_list flag decides how many judge calls happen. With True, all the chunks are passed together as one list and the judge scores them in one call; that is right for groundedness, where the question is "is the answer supported by the documents as a whole". With False, each chunk is scored separately and the scores are combined with the agg function, here np.mean; that is right for context relevance, where you want to know how relevant each chunk is on average. The default aggregation is also the mean, but writing it out helps you remember it is there.

Step 4: wrap the app and record some runs

bot = HRBot()

tru_bot = TruApp(
    bot,
    app_name="HR Bot",
    app_version="v1",
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)

questions = [
    "How many days of annual leave do I get?",
    "Can I carry unused leave into next year?",
    "What are the office hours?",
    "Do you offer a company car?",
]

for q in questions:
    with tru_bot as recording:
        bot.query(q)

print(session.get_leaderboard())

TruApp(...) wraps your object and registers it with the name HR Bot and the version v1. The argument that takes the metrics is still spelled feedbacks=, even though the objects are now Metrics; that is a leftover of the old name and it is not a bug in your code. The with tru_bot as recording: block is the recording context: everything your app does inside it is captured as one record. You call the original object, bot.query(q), not the wrapper, and TruLens observes from the side.

The last question is deliberate. The documents say nothing about company cars, so a good model should say it does not know. That is a useful test case, because it is exactly where groundedness matters: an answer of "Yes, all staff get a car" would be a hallucination, and the groundedness score should drop.

Run the file with python app.py. The metrics are computed in the background after each record is ingested, so the leaderboard printed at the very end may be empty or partial on a first run. That is not a failure; the next section explains why, and how to wait for the scores.

Waiting for scores and reading the leaderboard

Because evaluation is asynchronous, you need a way to say "hold on until the scores exist". TruLens gives you one on the recording object. Replace the final loop of your script with this:

for q in questions:
    with tru_bot as recording:
        bot.query(q)
    results = recording.retrieve_feedback_results(timeout=180)
    print(q)
    print(results.T)

session.force_flush()
print(session.get_leaderboard())

recording.retrieve_feedback_results(timeout=180) blocks until the metrics for that recording finish, up to 180 seconds, and returns a pandas DataFrame with one column per metric. session.force_flush() at the end pushes any spans still queued out to the database before the script exits. Use both habits in scripts; in a notebook you can usually rely on the cell finishing slowly enough, but a script cannot.

Do not use wait_for_feedback_results() on the app object. That method existed in older versions and in older tutorials, and under the current tracing it raises an error telling you to use retrieve_feedback_results instead.

Now read the output. Each score is a float between 0 and 1, where higher is better for every metric in the RAG Triad. A typical result for the first question might look like this:

Groundedness        1.0
Answer Relevance    1.0
Context Relevance   0.67

How do you read that? Groundedness of 1.0 says every claim in the answer was supported by the retrieved text. Answer relevance of 1.0 says the answer addressed the question. Context relevance of about 0.67 says that, averaged over the two retrieved chunks, one was a strong match and the other was only partly relevant, which matches our toy retriever, which returns the top two keyword matches whether or not the second is any good. That is a real signal: it tells you retrieval, not generation, is the weakest link, and it points at the exact knob (how many chunks, or how they are ranked) to turn.

Where do the scores come from? The judge model is prompted to rate on a small integer scale, by default 0 to 3, and TruLens divides by the maximum to land in 0 to 1. So your scores cluster at values like 0, 0.33, 0.67 and 1, which are not measurements of a continuous quantity but the judge's coarse opinion. Averages over many records smooth this out; single scores on single records are noisy, and this is the place to start being sceptical.

session.get_leaderboard() aggregates across all the records of each app version and returns a DataFrame: one row per app version, with the mean of each metric, plus average latency and total cost. With only four questions it is a smoke test rather than a verdict, but its shape is the shape you will use all the time, for example to compare v1 and v2 of the bot on the same questions.

You can also fetch the raw records, which is how you look at individual runs from code:

records, feedback_columns = session.get_records_and_feedback(app_name="HR Bot")
print(feedback_columns)
print(records[["input", "output"] + feedback_columns].head())

get_records_and_feedback returns a DataFrame of records and a list naming the metric columns. Filtering by app_name and, if you like, app_version keeps it focused.

The dashboard: seeing runs and comparing versions

Numbers in a terminal are fine for a smoke test, but the point of TruLens is to explore. The dashboard is a local web page, built with Streamlit, that reads the same database your app wrote to. Start it from code, in the same folder:

from trulens.core import TruSession
from trulens.dashboard import run_dashboard

session = TruSession()
run_dashboard(session)

Or, since 2.14.0, from the shell, with no Python file at all:

trulens-dashboard

The command looks in the current directory for default.sqlite (or another SQLite file containing TruLens tables) and serves it. If it cannot find one, it says No TruLens database found and tells you to run your app first or pass --database-url. A few flags are worth knowing from day one: --port 8502 picks the port, --address 127.0.0.1 restricts it to your own machine, --force stops a dashboard that is already running, and --find prints which database it would serve and exits.

The dashboard prints a local URL, usually http://localhost:8501 or the port you chose. Open it. There are four pages.

Leaderboard lists your app versions with the mean of each metric, latency and cost. This is where you compare "HR Bot v1" with "HR Bot v1-bad" at a glance. Colours highlight the metrics so that weak spots stand out.

Records is where you learn the most. It lists every individual run with its input, output and scores. Click one and you get the trace: the tree of spans, showing the retrieval, the generation, and the timing of each, with the inputs and outputs of every step. Below it are the metric results with the judge's written reasons. When a score surprises you, this is where you find out why, by reading the reason the judge gave.

Compare puts two app versions side by side, record against record, so you can see how the answer to the same question changed.

Trends (added in 2.13) plots metrics, latency and cost over time, with confidence intervals. It becomes useful once you have a stream of production traffic rather than a handful of test questions.

The habit to build is this: never trust an aggregate number without opening a few of the records behind it. If the leaderboard says Groundedness is 0.55, open the lowest-scoring records, read the answer, read the retrieved context, read the judge's reason, and decide whether you agree. Sometimes the app is wrong. Sometimes the judge is. Learning to tell which is the whole craft of LLM evaluation, and the dashboard is where you practise it.

If the page says the dashboard is already running, you have a previous instance holding the port. Pass force=True (or --force on the command line) and it stops the old one first.

Selectors and instrumentation in more depth

Most of your debugging in the first week will be about getting the right data into the right metric, so it is worth understanding the two halves of that pipeline properly: how data gets onto spans (instrumentation) and how metrics pull it off (selectors).

Putting data on spans

The @instrument decorator wraps a method. Every time the method runs inside a recording, TruLens opens a span, runs the method, and closes the span. Used with no arguments it still records something useful: each argument is stored as ai.observability.call.kwargs.<name> and the return value as ai.observability.call.return. The attributes argument lets you give those values standard names such as RETRIEVAL.QUERY_TEXT. The span_type argument tags the span so that selectors can find spans by kind.

There are three ways to decide what a span records:

• a string naming a parameter, like "query", or the word "return";
• a function that receives the return value, any exception, and the call arguments, and returns a dictionary of attributes, for when you need to transform a value first;
• nothing, in which case you get the default capture above.

When the method you want to watch lives in a library you cannot edit, you do not decorate it; instead you patch it from outside with instrument_method(cls=..., method_name=..., span_type=..., attributes=...). You will not need it for your first app, but it is the answer when someone asks "how do I trace code I do not own?"

If your app uses LangChain, LangGraph or LlamaIndex, the framework wrappers (TruChain, TruGraph, TruLlama) instrument the framework's own steps for you, so you may not need any decorators at all. The concepts are identical; the wrapper just does the marking.

Pulling data off spans

A selector answers "which value on which span". The shortcuts cover nearly everything a beginner needs:

When a shortcut is not enough you build a selector from parts, for example by span type and attribute:

Selector(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    span_attribute=SpanAttributes.RETRIEVAL.QUERY_TEXT,
)

or by function name, when you want a value from one specific function of your own code:

Selector(function_name="app.HRBot.generate_completion", function_attribute="return")

Two failure patterns are worth memorising. If a metric never produces a score, first suspect that the span it needs does not exist or lacks the attribute: check that your retrieval method has the RETRIEVAL span type and the RETRIEVED_CONTEXTS attribute. If a metric raises an error about a missing argument, suspect a key in selectors that does not match a parameter name of the implementation. Between them, those two causes explain the large majority of "my metric does nothing" questions.

You can also write your own metric. The implementation can be any Python function that returns a number between 0 and 1. A classic first one is a length check:

f_short = Metric(
    name="Short Answer",
    implementation=lambda text: 1.0 if len(text.split()) <= 60 else 0.0,
    selectors={"text": Selector.select_record_output()},
)

It costs nothing and calls no model, which makes it a good way to experiment with selectors without spending tokens. Plain-code metrics like this are also the most trustworthy, because they are deterministic: the same answer always gets the same score.

The provider methods you will reach for

The judge's built-in methods are grouped by what you want to measure. You do not need to memorise them, but knowing the families lets you choose without hunting through the reference. Each of these returns a float, and most have a _with_cot_reasons twin that returns the score plus an explanation.

Two practical notes. First, the safety metrics score the presence of the bad property, so for some of them a high score is bad; check each method's docstring and, in any guardrail, mark such metrics as lower-is-better. Second, names change. In the past qs_relevance was renamed to relevance, and the agent metrics briefly carried a trajectory_ prefix that was later dropped, so an old tutorial's method name may simply not exist anymore. When a method is missing, look at the current provider reference rather than guessing.

The same provider object can drive any of them, so the pattern is always the same: pick a method, work out which parameters it takes, map a selector to each, give it a name, and add it to the recorder. Nothing else about your app changes.

Configuration and common errors

TruLens has very little configuration a beginner must touch, and most of what goes wrong is one of a dozen recurring errors. This section covers both, with the messages as they actually appear.

The settings you will actually use

Where data is stored. The default is the SQLite file default.sqlite. To use another file or database, pass a URL once, at the start of your program, before anything else creates the session:

from trulens.core import TruSession

session = TruSession(database_url="sqlite:///my_experiments.sqlite")

SQLite is ideal for learning and for a single developer, and unsuitable once several processes or machines write at the same time. The mid and senior guides cover PostgreSQL for that case.

Starting clean. session.reset_database() wipes all tables. It is convenient in tutorials and dangerous anywhere else, because it deletes your history without asking. Keep it out of any script you run against data you care about.

Redacting secrets. TruSession(database_redact_keys=True) masks API keys that might otherwise be captured inside stored objects. It is off by default; turn it on as soon as you share a database with anyone.

Turning tracing off. Setting the environment variable TRULENS_OTEL_TRACING=0 disables the OpenTelemetry tracing that everything in this guide depends on. You will rarely want to, and TruLens warns once if you do; it matters because a colleague's stray environment variable is a common reason for "no spans are recorded".

Errors, and how to read them

Key OPENAI_API_KEY needs to be set; please provide it in one of these ways... The judge has no credentials. Set the variable in the same terminal session you run Python from, or put it in a .env file. The message lists the options, so read it to the end.

These packages are required for using <X>: <package>. You should be able to install them with pip: pip install "<package>" You used an integration whose package is not installed, typically trulens-apps-langchain or a provider. Run the pip install line it prints.

Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing! You copied an older tutorial that passes feedback_mode=FeedbackMode.DEFERRED to the recorder. Delete the argument. Current TruLens computes metrics on a background thread attached to the app, and other modes are not available under the default tracing.

Already recording with a context manager, cannot nest! You opened a with tru_bot as recording: block inside another one. Use one recording context at a time and call your app once per block.

Cannot find TruLens context. The instrumented method ran on a different thread or asyncio task from the recording, so TruLens lost track of which record it belonged to. The fix is to run the work with TruLens's own thread helpers, or to keep it on the same thread. The doc page named in the message lists the options.

Database schema is behind the expected revision... You upgraded TruLens and pointed it at a database created by an older version. Back the file up, then run TruSession().migrate_database(). The mirror message, Database schema is ahead of the expected revision, means an older TruLens is reading a database a newer one already upgraded; the fix is to upgrade the older client.

TruSession was already initialized. Cannot change database configuration after initialization. You called TruSession(database_url=...) after a session already existed in the same process. The session is a singleton; configure it once, at the very top, and everything else reuses it.

DeprecationWarning: Feedback is deprecated... Use Metric instead Not an error, a signal. You, or a tutorial, used the old class. Replace Feedback(...) with Metric(implementation=..., selectors={...}) when convenient.

RuntimeError: Dashboard is already running An earlier dashboard still holds the port. Use force=True, or stop the old process.

Failures that print nothing

The most frustrating problems are the silent ones, and they share a short list of causes.

• No scores, no errors. The selectors found nothing. Check the span type and attributes on retrieval, and the selector keys.
• Scores missing for the last record. The script ended before the background thread finished. Use retrieve_feedback_results and force_flush.
• An empty dashboard. You are in a different folder from the one that holds default.sqlite, or the dashboard is reading another file. Run trulens-dashboard --find to see which database it picked.
• Scores look random. Your judge is too small or too noisy, or your questions are too few. Try a stronger judge and more examples before you change your app.

Two common mistakes about what the scores mean

Having the scores is not the same as understanding them, and two misunderstandings cause more bad decisions than any API slip.

The first is treating the judge as truth. Every LLM-judged metric is a language model's opinion, produced by a prompt. It can be inconsistent, it can be fooled by confident phrasing, and it has biases, such as favouring longer answers. A score of 0.67 is not a measurement of the answer's quality the way a thermometer reading is a measurement of temperature. It is one judge's coarse rating. The remedy is not to distrust everything but to calibrate: read a sample of the judge's reasons, compare them with your own judgement on a few dozen examples, and use scores mostly to compare versions, where consistent bias cancels out, rather than as absolute grades. The mid-level guide covers how to check a judge against human labels.

The second is drawing conclusions from tiny samples. Four questions can show you that the machinery works. They cannot tell you whether v2 is better than v1. A difference of 0.05 in an average over ten records is well inside the noise. Build an evaluation set of dozens to hundreds of realistic questions, including the nasty ones (questions the documents cannot answer, questions with typos, questions that try to get the bot to misbehave), and run every version against the same set. If you keep only one habit from this guide, keep that one: fixed question set, fixed judge, one change at a time.

Putting it all together

Let us finish with one small project that uses everything: two versions of the HR bot, evaluated on the same fixed question set, compared in the dashboard. The change between versions is one line, the system prompt, so any difference in scores has one explanation.

import numpy as np
from openai import OpenAI as OpenAIClient
from trulens.apps.app import TruApp
from trulens.core import Metric, Selector, TruSession
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
from trulens.providers.openai import OpenAI

DOCS = [
    "Employees receive 25 days of paid annual leave per calendar year.",
    "Unused leave can be carried over, up to a maximum of 5 days.",
    "The office is open Sunday to Thursday, from 9 am to 5 pm.",
    "Remote work is allowed up to two days per week with manager approval.",
]
QUESTIONS = [
    "How many days of annual leave do I get?",
    "Can I carry unused leave into next year?",
    "What are the office hours?",
    "How many days can I work remotely?",
    "Do you offer a company car?",
    "What is the dental insurance limit?",
]

client = OpenAIClient()


class HRBot:
    def __init__(self, system_prompt: str):
        self.system_prompt = system_prompt

    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list[str]:
        words = {w.lower().strip("?.,") for w in query.split()}
        scored = [(len(words & set(d.lower().replace(".", "").split())), d) for d in DOCS]
        scored.sort(reverse=True)
        return [d for score, d in scored[:2] if score > 0]

    @instrument(span_type=SpanAttributes.SpanType.GENERATION)
    def generate_completion(self, query: str, context_list: list[str]) -> str:
        context = "\n".join(context_list)
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
            ],
        )
        return response.choices[0].message.content

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def query(self, query: str) -> str:
        return self.generate_completion(query, self.retrieve(query))


session = TruSession()
provider = OpenAI(model_engine="gpt-4o-mini")

metrics = [
    Metric(
        implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
        name="Groundedness",
        selectors={
            "source": Selector.select_context(collect_list=True),
            "statement": Selector.select_record_output(),
            "question": Selector.select_record_input(),
        },
    ),
    Metric(
        implementation=provider.relevance_with_cot_reasons,
        name="Answer Relevance",
        selectors={
            "prompt": Selector.select_record_input(),
            "response": Selector.select_record_output(),
        },
    ),
    Metric(
        implementation=provider.context_relevance_with_cot_reasons,
        name="Context Relevance",
        selectors={
            "question": Selector.select_record_input(),
            "context": Selector.select_context(collect_list=False),
        },
        agg=np.mean,
    ),
]

VERSIONS = {
    "strict": "Answer using only the context. If the answer is not in it, say you do not know.",
    "loose": "You are a helpful HR assistant. Answer the question as best you can.",
}

for version, prompt in VERSIONS.items():
    bot = HRBot(prompt)
    tru_bot = TruApp(bot, app_name="HR Bot", app_version=version, feedbacks=metrics)
    for q in QUESTIONS:
        with tru_bot as recording:
            bot.query(q)
        recording.retrieve_feedback_results(timeout=180)

session.force_flush()
print(session.get_leaderboard())

Run it with python compare.py, then start trulens-dashboard in the same folder. On the leaderboard you should see two rows, strict and loose. The instructive rows are the two questions the documents cannot answer, the company car and the dental limit. The strict prompt should say it does not know, which keeps groundedness high. The loose prompt invites the model to improvise, and any invented figure will lower its groundedness. Open the Records page for those two questions under each version, read the answers side by side on the Compare page, and read the judge's reasons. You have now done, in miniature, the complete loop: instrument, record, evaluate, compare, iterate.

A note on cost and honesty. Each metric per record is at least one judge call, and context relevance with collect_list=False is one call per chunk. Six questions, three metrics, two versions is a few dozen calls, which costs cents. The same script over a thousand questions is a real bill, and later guides show how to sample. Also, your exact numbers will differ from any written here, because models vary from run to run. Judge the pattern, not the digits.

What you can now do, and what comes next

You can now explain what TruLens is for and how its five nouns fit together: session, app, record and span, metric, provider. You can install the current packages, pin a judge, instrument a small app with @instrument, define the three RAG Triad metrics with selectors, record runs, wait for and read the scores, and open the dashboard to compare two versions and inspect a single trace. You can recognise the usual errors by their messages, you know the silent failures and where to look for them, and you know not to treat a judge's score as truth or a four-question sample as proof.

What comes next is depth. The Mid-level guide shows how the tracing and the background evaluator actually work, how to move storage to PostgreSQL, how to evaluate a whole dataset offline with BatchEvaluator and Runs, how to write guardrails that block bad inputs and outputs at runtime, how to evaluate conversations and agents, and how to check a judge against human labels. The Senior guide treats TruLens as a platform: sampling and cost budgets for online evaluation, exporting to an OpenTelemetry backend, security of the dashboard and of captured content, upgrades and database migrations, multi-team isolation, and when to choose a different tool.

If you want to look sideways first, read the guides for Langfuse (/student-guides/langfuse) and RAGAS (/student-guides/ragas) to see how other tools approach tracing and scoring, and the guide for LangGraph (/student-guides/langgraph) if your app is an agent. Comparing the same idea in two tools is the fastest way to learn which parts are essential and which are one tool's accent.

Sources

• TruLens home: https://www.trulens.org/
• Installation: https://www.trulens.org/getting_started/install/
• Quickstart: https://www.trulens.org/getting_started/quickstarts/quickstart/
• Core concepts: https://www.trulens.org/getting_started/core_concepts/
• The RAG Triad: https://www.trulens.org/getting_started/core_concepts/rag_triad/
• Dashboard: https://www.trulens.org/getting_started/dashboard/
• Trends page: https://www.trulens.org/getting_started/dashboard/trends/
• OpenTelemetry tracing: https://www.trulens.org/otel/
• Semantic conventions: https://www.trulens.org/otel/semantic_conventions/
• Instrumentation guide: https://www.trulens.org/component_guides/instrumentation/
• Migrating from Feedback to Metric: https://www.trulens.org/component_guides/evaluation/metric_migration/
• Feedback providers: https://www.trulens.org/component_guides/evaluation/feedback_providers/
• No-context warning: https://www.trulens.org/component_guides/other/no_context_warning/
• Migrating from trulens_eval: https://www.trulens.org/component_guides/other/trulens_eval_migration/
• Release history: https://www.trulens.org/contributing/release_history/
• GitHub releases: https://github.com/truera/trulens/releases
• PyPI: https://pypi.org/project/trulens/

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

TruLens is an open-source Python library for tracing and evaluating LLM applications. You instrument your app so each run is recorded as a trace of spans, then attach metrics, which are scoring functions that often use a second model as a judge, to rate every run between 0 and 1. The results go into a database and a dashboard, so you can compare versions of the app on the same numbers instead of reading answers by hand.

Then the line that shows you have used it: "for a RAG app I score the RAG Triad, context relevance, groundedness and answer relevance, because each one points at a different failure, so a low score tells me whether to fix retrieval, generation or the prompt."

The model

Three facts make the model stick. First, TruLens is a library inside your process, not a server; the dashboard is a separate Streamlit process that reads the same database. Second, evaluation is asynchronous: metrics run on a background thread after the record is ingested, which is why you wait for results explicitly. Third, tracing is built on OpenTelemetry and has been on by default since version 2.3.0, so every current example uses @instrument from trulens.core.otel.instrument.

The commands and code to have ready

You are rarely asked to type these from memory, but you should be able to say them fluently, and to write the shape of the code on a whiteboard.

pip install trulens trulens-providers-openai        # core + dashboard + an OpenAI judge
python -c "import trulens.core; from importlib.metadata import version; print(version('trulens-core'))"
trulens-dashboard                                   # serve default.sqlite in this folder (2.14+)
trulens-dashboard --find                            # which database would it serve?

from trulens.core import TruSession, Metric, Selector
from trulens.apps.app import TruApp
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
from trulens.providers.openai import OpenAI

session = TruSession()                        # sqlite:///default.sqlite
provider = OpenAI(model_engine="gpt-4o-mini") # always pin the judge

relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={"prompt": Selector.select_record_input(),
               "response": Selector.select_record_output()},
)

tru_app = TruApp(app, app_name="HR Bot", app_version="v1", feedbacks=[relevance])
with tru_app as recording:
    app.query("How many days of leave do I get?")
recording.retrieve_feedback_results(timeout=180)
session.get_leaderboard()

Be ready to explain each line in one sentence. The two lines interviewers probe most are the selectors dictionary, whose keys must equal the parameter names of the implementation, and feedbacks=, which is still the keyword name for metrics even though the class is now Metric.

The classic traps, and why they happen

Nothing is scored, and nothing errors. The metric's selectors found no data. The usual cause is that the retrieval method was not instrumented with the RETRIEVAL span type and the RETRIEVED_CONTEXTS attribute, so select_context() has nothing to select. It is silent because a selector that matches nothing is not an error in itself.

The last record has no scores. Evaluation happens on a background thread. If the process exits right after the last call, the pending work is lost. The fix is recording.retrieve_feedback_results(timeout=...) for one recording and session.force_flush() before exit.

The dashboard is empty. SQLite is a file in the current directory, so a different working directory means a different, empty database. trulens-dashboard --find tells you which file it would serve.

Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing! The code came from an older tutorial. Under default OpenTelemetry tracing only one mode exists, so the argument must be deleted.

A Feedback is deprecated warning. Feedback is now an alias of Metric and will be removed in a future release. Use Metric(implementation=..., selectors={...}).

Scores that drift when nothing changed in the app. The judge model was left on its default, and the default changed with a release, or the provider silently updated the model. Pin the model explicitly and record its name.

TruSession was already initialized. The session is a singleton, so a second call with a different database is ignored. Configure it once at startup.

Trusting a single number. A judge's score is an opinion from a language model, scaled from a small integer range onto 0 to 1. A difference of a few hundredths over a handful of records is noise.

Common interview questions

What problem does TruLens solve, and what did people do before it?

LLM apps fail without crashing: fluent answers that are wrong, off-topic, or invented. Before tools like TruLens, teams read outputs by hand, which does not scale, leaves no record, and cannot tell you whether a change helped. TruLens gives you a structured record of every run and a numeric score per run, so you can compare versions and watch quality over time. A strong answer also says what it does not do: it does not build the app, and its scores are judgements to be calibrated, not ground truth.

Explain the RAG Triad and what each metric catches.

RAG means the app retrieves documents and feeds them to the model. The triad has three metrics. Context relevance compares the question with each retrieved chunk and catches bad retrieval. Groundedness compares the answer with the retrieved chunks and catches hallucination, meaning claims the sources do not support. Answer relevance compares the question with the answer and catches an answer that is fluent but off-topic. The value is diagnostic: which metric is low tells you which component to fix.

What is the difference between a record, a span and a trace?

A span is one unit of work with a type, such as a retrieval or a generation. A record is one invocation of the app's top-level method, made of the tree of spans under a root span of type RECORD_ROOT. In OpenTelemetry vocabulary a trace is that same tree, and the TruLens glossary says the word record will be renamed trace. So in practice the answer is: spans make up a record, and a record is a trace.

What is a selector and why do metrics need them?

A metric's implementation is a function with named parameters, such as relevance(prompt, response). The data it needs lives as attributes on spans. A selector maps each parameter to a place on the trace: Selector.select_record_input() for the user's question, select_record_output() for the answer, select_context() for retrieved chunks. The keys in the selectors dictionary must match the parameter names exactly. That is what lets one metric work on any app as long as the app marks its data.

What does collect_list do?

It controls how many judge calls happen. With True, all matching values are passed as one list and scored in one call, which suits groundedness, where the question is whether the answer is supported by the context as a whole. With False, each value is scored separately and the results are combined with the agg function, which defaults to the mean, which suits per-chunk context relevance. False costs more calls, which matters when you pay per call.

What changed between Feedback and Metric?

Version 2.7.0 (February 2026) introduced one unified Metric API replacing Feedback and MetricConfig. Feedback still works but is a deprecated subclass of Metric, and constructing one emits a DeprecationWarning. The argument imp= became implementation=, and the new style prefers a selectors dictionary over the chained .on_input_output() methods. The recorder argument is still spelled feedbacks=, which surprises people.

What are the _with_cot_reasons methods?

They return the score together with a written explanation, because the judge is asked to reason step by step first. The explanation appears in the dashboard and is how you understand a low score. One rule to remember: guardrails need a function that returns only a float, so you use the plain variant there, not the chain-of-thought variant.

How do you know your LLM judge is right?

You do not assume it. You read a sample of its reasons, compare its scores with your own judgement or with human labels on a few dozen examples, and treat scores mostly as relative signals between versions rather than absolute grades. You pin the judge model and keep it fixed while comparing. At this level it is enough to show you understand that a judge is fallible; Mid-level goes into measuring agreement with humans.

Where does TruLens store data, and what are the limits?

By default in a SQLite file called default.sqlite in the working directory, created when TruSession() is first called. SQLite suits one developer; for several processes, containers or teammates you move to PostgreSQL, which has been supported since 2.6, or Snowflake. The dashboard has no authentication, so you do not expose it publicly.

How would you compare two versions of a prompt?

Keep everything else fixed: the same question set of dozens of realistic items including unanswerable and adversarial ones, the same pinned judge, and one changed variable. Record both under different app_version labels, wait for the scores, and compare on the leaderboard. Then open the lowest-scoring records of each version and read the traces and judge reasons, because an average can hide one systematic failure.

A metric returns nothing. What do you check?

Three things in order. Whether the needed span exists and has the right type and attribute, for instance a RETRIEVAL span with RETRIEVED_CONTEXTS. Whether the selector keys match the implementation's parameter names. And whether the process waited for background evaluation to finish. Then I look at the dashboard trace to see what was actually recorded.

Does TruLens send my data anywhere?

The library runs in your process and writes to your database. The only outbound calls are the ones your app makes and the judge calls to whichever provider you configure, and those contain your prompts, documents and answers. Where data residency matters, which is a real concern for employers in the Gulf and Egypt, you can use a judge you host yourself, through the Ollama provider or an OpenAI-compatible endpoint set with base_url.

What is TruSession, and why can you not change the database later?

It is the process-wide singleton that owns the connector, exporter and evaluator threads. Calling it again returns the same object, so a second call with a different database_url logs a warning and is ignored. The practical rule is to configure it once at the top of your program.

How does it compare with Langfuse or RAGAS?

All three help evaluate LLM apps, with different emphasis. Langfuse is a tracing and prompt-management platform you typically run as a service. RAGAS is a library focused on scoring RAG outputs. TruLens combines in-process tracing with a named set of metrics and a local dashboard, and it ties every score back to specific spans. I would say which I prefer depends on whether the team needs a shared server, and I would avoid claiming one is better in general. There are guides for both in this catalogue at /student-guides/langfuse and /student-guides/ragas.

What strong answers have that weak ones miss

Interviewers for this topic are listening for a few things. They want to hear that you separate retrieval quality from generation quality, because that separation is the point of the triad. They want an unprompted caveat that a judge is a model and can be wrong. They want to hear about the asynchronous nature of evaluation, since people who have only pasted a quickstart often think scores exist the moment the call returns. And they want to hear that you pin versions and models, because the library releases fast and scores that shift for no reason are a real operational problem.

Weak answers describe TruLens as "a dashboard", or list metric names without saying what each catches, or claim the scores measure truth. Another weak sign is teaching trulens_eval, Tru(), or Feedback as current, which shows the candidate learned from an old source. If you are caught out on a detail, say what you would check in the docs; admitting a gap calmly is better than a confident wrong guess.

If the conversation moves to code, expect to be asked to write a metric from scratch. A plain function with a selector is enough:

Metric(
    name="Short Answer",
    implementation=lambda text: 1.0 if len(text.split()) <= 60 else 0.0,
    selectors={"text": Selector.select_record_output()},
)

Say why it is useful: it is deterministic and free, so it is a good first metric and a good way to test that selectors work before spending tokens on a judge.

Questions to ask back

Turning the interview around shows experience. Good questions include: how does the team decide a version is good enough to ship, and on what question set; who reads the lowest-scoring records; which judge model do they use and how was it checked against people; and where do the evaluation results live and who can see them. These are the same questions the later levels of this guide answer, and asking them signals that you think of evaluation as a process rather than a library call.

Mini scenarios you may be handed

Some interviewers skip definitions and describe a situation. Practise answering these in four or five sentences, in the order symptom, hypothesis, check, fix.

"Our bot's groundedness dropped from 0.9 to 0.6 after we changed the embedding model. Why?" A new embedding model changes which chunks are retrieved. If the new chunks are less relevant, the model has weaker material, and either refuses or improvises. I would first look at context relevance for the same versions: if it also dropped, retrieval is the cause and I would examine the chunks returned for a few failing questions. If context relevance held steady, I would suspect the prompt or the judge. I would confirm by opening the lowest-scoring records under both versions and reading the retrieved text and the judge's reasons.

"The score is 1.0 on everything. Is the bot perfect?" Almost certainly not. Either the question set is too easy, the judge is lenient, or the metric is scoring the wrong data, for example because a selector points at the question instead of the answer. I would add hard and unanswerable questions, read a few reasons, and try a second judge. A metric that can never go low tells you nothing.

"A teammate wants to run the dashboard on a public server so everyone can see it." I would say no to exposing it directly. It has no authentication and displays prompts and documents that may contain personal data. Bind it to localhost and put single sign-on or a reverse proxy in front, or share exported results instead.

"How do you keep evaluation costs under control while learning?" Use a small question set for iteration, start with plain-Python metrics, choose a cheaper or local judge, and set collect_list=True where one call per record is enough. Sampling and budgets come later, at Senior level.

Practising these aloud is worth more than rereading. The pattern, symptom then hypothesis then check then fix, works for almost every question an interviewer can ask about a tool like this one, and it shows you think about the tool as a debugging instrument rather than a list of functions.

Sixty-second self-test

Answer out loud, without looking back.

1. In one sentence, what does TruLens do?
2. Name the three jobs: tracing, evaluation, and what else?
3. What do app_name and app_version let you do?
4. Which decorator marks a step, and which span type marks the retrieval?
5. What are the three RAG Triad metrics and what does each catch?
6. Why must the keys of a selectors dictionary match the implementation's parameters?
7. What does collect_list=False change?
8. Why might the last record of a script have no scores, and what fixes it?
9. What happens if you pass feedback_mode=FeedbackMode.DEFERRED to a recorder today?
10. Where is data stored by default, and why is the dashboard empty if you run it from another folder?
11. Name two reasons not to trust a single judge score.
12. Which command starts the dashboard from the shell, and from which version?

If you hesitated on more than three, reread the matching section of the detailed guide and rerun the compare.py project. The answers sit in it: the project records two versions, waits for scores, and forces you to read the traces.

Part one of three. The mistakes almost everyone makes in their first weeks with TruLens, the habits that prevent them, and a checklist to keep beside you. Each item states the symptom you will see, the cause, and the fix, so you can search this page by what is on your screen.

Common errors at this level

Copying a tutorial that uses the old API

Symptom. ImportError or ModuleNotFoundError for trulens_eval, Tru is not found, or a method such as qs_relevance does not exist. Or the code runs but prints a wall of deprecation warnings.

Cause. TruLens was reorganised in version 1.0, reworked around OpenTelemetry in 2.3.0, and moved from Feedback to Metric in 2.7.0. Blog posts from before those dates are still the top search results.

Fix. Check the date and the imports before you trust a tutorial. The current ones come from trulens.core, trulens.apps.*, trulens.providers.* and trulens.dashboard. If you see from trulens_eval import, close the tab. When in doubt, compare with the quickstart at https://www.trulens.org/getting_started/quickstarts/quickstart/, which is kept current.

Installing only trulens and wondering where the provider is

Symptom. ModuleNotFoundError: No module named 'trulens.providers', or the message These packages are required for using ... followed by a pip install line.

Cause. The meta-package installs the core, the built-in metrics and the dashboard, but no judge and no framework integration. Each provider and each framework wrapper is its own package.

Fix. Install what you use: pip install trulens-providers-openai, or trulens-providers-anthropic, trulens-providers-ollama and so on, and for frameworks trulens-apps-langchain, trulens-apps-langgraph, trulens-apps-llamaindex. The error message prints the exact command; run it.

Forgetting the API key, or putting it in the wrong place

Symptom. Key OPENAI_API_KEY needs to be set; please provide it in one of these ways...

Cause. The variable is not set in the terminal or notebook kernel you are running from. A very common version is setting it in one terminal tab and running Python from another.

Fix. Export it in the same shell, or add it to a .env file in the project folder, which TruLens reads. Then add .env to .gitignore before your next commit, so the key never reaches a repository.

Metrics that score nothing, with no error

Symptom. The dashboard shows records but the metric columns are empty, or a metric is missing from the leaderboard.

Cause. The selectors matched nothing. Usually one of three things: the retrieval method is not instrumented with span_type=SpanAttributes.SpanType.RETRIEVAL and the RETRIEVED_CONTEXTS attribute; the record root is missing its INPUT and OUTPUT attributes; or a key in the selectors dictionary does not match the implementation's parameter name.

Fix. Open one record in the dashboard and read its trace. Does it show a retrieval span with the contexts in it? Does the root show the question and the answer? Then compare your selector keys with the signature, for example relevance(prompt, response) and context_relevance(question, context). Fix the gap, rerun, and the scores appear.

Reading the leaderboard before the scores exist

Symptom. The leaderboard is empty, or partly filled, right after your script ran, and appears complete a minute later.

Cause. Metrics are computed asynchronously on a background thread. Your code returned before the judge finished.

Fix. After each recording call recording.retrieve_feedback_results(timeout=180), and at the end of a script call session.force_flush(). Do not use wait_for_feedback_results; it is unsupported under the current tracing and raises an error pointing you to retrieve_feedback_results.

Passing arguments from an older recording style

Symptom. ValueError: Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing! or RuntimeError: Not supported with OTel tracing enabled!.

Cause. Older pages, including one official page that still describes the legacy path, show feedback_mode=FeedbackMode.DEFERRED, start_evaluator, update_record and similar. Under the default tracing they are not available.

Fix. Remove the argument and use the current pattern: a recording context, then retrieve_feedback_results. When the code and a doc page disagree, follow the code, meaning the error message you actually received.

Nesting recording contexts

Symptom. RuntimeError: Already recording with a context manager, cannot nest!

Cause. A with tru_app as recording: block was opened inside another one, often because a helper function opens its own recording and is called from within a loop that already has one.

Fix. Have exactly one recording context open at a time, and record at the outermost call you want as one record.

The dashboard is empty or shows the wrong data

Symptom. The dashboard opens, but you see no apps. Or you see yesterday's experiments mixed in.

Cause. default.sqlite lives in the working directory. Running from another folder means a different file. Mixed experiments mean you never cleaned or separated the database.

Fix. Run trulens-dashboard --find to see which database it would serve. Launch scripts and the dashboard from the same project folder. For separate experiments, use separate database files by passing TruSession(database_url="sqlite:///experiment2.sqlite") once, at the start.

A second TruSession that changes nothing

Symptom. A warning: TruSession was already initialized. Cannot change database configuration after initialization. Your data still goes to the old file.

Cause. The session is a per-process singleton. Some library, notebook cell or helper created it first with the default database.

Fix. Create the session once at the very top of your entry point, with the settings you want. In a notebook, restart the kernel if you need to change it.

Nothing recorded because tracing is off

Symptom. No spans at all, and a warning about OTel tracing being disabled.

Cause. The environment variable TRULENS_OTEL_TRACING is set to 0 or false, perhaps by a colleague's shell profile or a leftover from an experiment.

Fix. Unset it: unset TRULENS_OTEL_TRACING. The warning exists so you notice; read it.

Mismatched package versions

Symptom. Strange ImportErrors from inside TruLens after an upgrade, or a database error about revisions.

Cause. The trulens-* packages release in lockstep, and mixing versions breaks them. Separately, upgrading TruLens against a database made by an older version raises Database schema is behind the expected revision.

Fix. Upgrade everything together with one pip install -U covering all the trulens-* packages you use, then back up the database and run TruSession().migrate_database(). Check with pip list | grep -i trulens that the versions agree.

The habits that pay off most

Pin the judge model. Write OpenAI(model_engine="gpt-4o-mini"), never a bare OpenAI(). Defaults move between releases, and an unpinned judge means a score change can come from an upgrade rather than from your app. Record the judge model in your notes next to the results.

Pin TruLens too. In requirements.txt write trulens==2.14.0 and the matching provider packages. The project releases every few weeks, and reproducible scores need a reproducible toolchain.

Choose version labels that mean something. app_version="v1" teaches you nothing in a month. Use labels that say what changed: strict-prompt, chunk-300, gpt-4o-mini. The leaderboard groups by this label, so a good one is documentation you get for free.

Change one thing at a time. If you change the prompt, the model and the chunk size together and the scores move, you know nothing. One variable per version, run on the same questions.

Build a fixed question set early. Write thirty to fifty questions in a file: easy ones, hard ones, ones the documents cannot answer, ones with typos, and a few that try to make the bot misbehave. Run every version against the same set. This matters more than any single metric.

Open the records, not just the averages. After every run, sort by lowest score, open the bottom five, and read the trace and the judge's reason. Most of what you learn comes from here.

Start with free metrics. A plain-Python metric, such as an answer-length check, costs nothing and is deterministic. Use it to prove your selectors work before you spend money on judge calls.

Keep the dashboard private. It has no login and shows prompts and documents. Start it with --address 127.0.0.1 when you only need it locally.

Keep secrets out of the repo. Keys in environment variables or an ignored .env file. If the data might hold personal information, set database_redact_keys=True and do not commit the SQLite file.

Write down what a score means. For each metric you use, keep a one-line definition in your README. "Groundedness is the judge's rating of whether every claim in the answer is supported by the retrieved text." It stops arguments later.

Treat cost as a variable. Each LLM-judged metric is at least one model call per record, and per-chunk context relevance is one per chunk. Count calls before you run a large set, and keep a small set for quick iterations.

Practice cards

Five short exercises. Do one when you have ten minutes.

Reading a low score without panicking

A low number is a question, not a verdict. Work through it in a fixed order and you will rarely be misled.

First, find the record. On the Records page, filter to the lowest scores for that metric and open one. Second, read the trace from the top. What was the question? What did retrieval return? Did the generation use it? Third, read the judge's reason. A reason that quotes the answer and the context and explains a real gap means the app is at fault. A reason that contradicts what you can see with your own eyes means the judge is at fault, or the metric was given the wrong data. Fourth, decide which component to change. Low context relevance points at retrieval: the chunking, the number of chunks, the embedding or the search. Low groundedness with good context points at generation: the prompt or the model. Low answer relevance with good groundedness means the model answered faithfully from the sources but missed what the user asked, which often means the retrieved material was on topic but not quite the right page.

A worked example. Suppose groundedness on the company-car question is 0.0. You open the record and see that retrieval returned only the office-hours document, and the answer says "Yes, all staff receive a company car after one year." The judge's reason says the claim appears nowhere in the context. The app is at fault: the prompt lets the model improvise. The fix is a stricter system prompt, recorded as a new version such as strict-prompt, rerun on the same questions. If instead the answer had said "I do not know" and the score was still 0.0, you would suspect the metric variant: some groundedness variants treat "I do not know" differently, which is why the answerability-aware variant exists and why reading one record is worth more than staring at the average.

Finally, resist the temptation to tune until the number is high. Scores are a tool for finding problems, and a version that has been tuned against ten questions will look wonderful on those ten and mediocre on the eleventh. Keep a few questions out of your tuning set and check them at the end.

When to stop tuning and ship

A common beginner trap is the endless loop: change something, rerun, look at the average, change something else. Decide in advance what "good enough" means, in writing, before the first run. For example: groundedness averages at least 0.85 over the fixed set, no record under 0.5 on the safety metrics, and no answer to an unanswerable question that invents a fact. Write the rule next to the question set. When a version meets it on the fixed set, and a small held-out set you did not tune on, you have a defensible reason to move on.

Two cautions go with this. First, scores from a judge are coarse, so a target such as "0.92 rather than 0.90" is false precision. Prefer rules you can check by reading: "none of the unanswerable questions gets an invented answer" can be verified by a person in five minutes. Second, your app will meet questions you never imagined. Plan to add every real failure you find in use to the question set, so that the set grows toward reality. That habit, a question set that grows from real mistakes, is the single most valuable thing you can carry from this level into the next.

Finally, keep notes. A plain text file with one line per experiment, the version label, the change, the judge, and what you concluded, will save you from repeating yourself and will make your reasoning visible to teammates and interviewers alike.

Small things to do from day one

• Create a virtual environment per project and install TruLens into it.
• Put trulens==<version> and your provider packages in requirements.txt.
• Keep OPENAI_API_KEY or your provider's key in .env, and list .env and *.sqlite in .gitignore.
• Run the app once with no TruLens at all, to confirm the app itself works.
• Decorate the retrieval method with the RETRIEVAL span type and both attributes, and the top-level method with RECORD_ROOT.
• Pin the judge model by name.
• Use retrieve_feedback_results after each recording and force_flush before exit.
• Open the dashboard after the first run and read one full trace.
• Save your question set in a file under version control.
• Note, in the README, which judge and which TruLens version produced each set of results.

A starter layout to copy

A folder like this keeps you out of trouble and scales to the later levels without restructuring:

hr-bot/
  .env                # keys, ignored by git
  .gitignore          # .env, *.sqlite, .venv/
  requirements.txt    # trulens==2.14.0, trulens-providers-openai==2.14.0
  app.py              # the app, with @instrument
  metrics.py          # the Metric definitions and the provider
  questions.json      # the fixed question set
  run_eval.py         # loops over versions and questions
  results/            # exported leaderboards, one file per run

The separation is the lesson. The app does not know about metrics, so you can change your metrics without touching it. The question set is data, so it is reviewed like code. The runner is small enough to read in one screen. Keeping these apart makes the later guides, which add databases, datasets and CI gates, into additions rather than rewrites.

The checklist

Before you trust a run, tick these.

• Python 3.10 to 3.13, in a virtual environment.
• trulens and exactly one provider package installed, all at the same version.
• API key set in this shell or in an ignored .env file.
• Judge model pinned with model_engine=.
• TruSession() created once, at the top of the entry point.
• Retrieval instrumented with the RETRIEVAL span type, QUERY_TEXT and RETRIEVED_CONTEXTS.
• Top-level method instrumented with RECORD_ROOT, INPUT and OUTPUT.
• Selector keys match the implementation's parameter names.
• Metrics passed through feedbacks=; no Feedback and no feedback_mode.
• retrieve_feedback_results called and force_flush before exit.
• A fixed question set that includes unanswerable and adversarial questions.
• app_version says what changed, and only one thing changed.
• Lowest-scoring records opened and read, not just averages.
• Dashboard bound to 127.0.0.1; database and .env not committed.

When the checklist is second nature, continue to the Mid-level guide, which adds storage on PostgreSQL, offline batch evaluation, guardrails, and calibrating the judge against human labels.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. If you already run TruLens on a real application, this is the level where you stop treating scores as magic numbers and start treating the evaluation setup as engineering: something with a cost, a failure mode, a storage layer, and a test suite of its own.

The stakes are practical. An evaluation setup that is wrong does not crash. It returns a number, the number goes on a dashboard, and someone decides to ship a prompt change because a metric went from 0.71 to 0.78. If the selector silently matched nothing, or the judge model drifted under you, or the database was a SQLite file on one engineer's laptop, that decision was made on noise. By the end of this guide you will be able to predict what a metric will see before you run it, move storage to PostgreSQL safely, control what online evaluation costs, gate a pull request on evaluation results, and debug the silent failures that account for most TruLens confusion in practice. Everything here is verified against TruLens 2.14.0.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can do on a real project in a few minutes. Examples assume Python 3.10 to 3.13 and the packages trulens, trulens-providers-openai and, where named, the framework packages. Install any of them at the same version, because the family releases in lockstep.

How TruLens works under the hood

Most confusing TruLens behaviour disappears once you hold the right picture of the architecture. TruLens is a library, not a server. There is no TruLens daemon. Your application process imports it, and three things happen inside that process.

First, instrumentation creates spans. Since version 2.3.0, OpenTelemetry tracing is the default, so the @instrument decorator from trulens.core.otel.instrument wraps your functions and emits OpenTelemetry spans through the standard SDK. Each span carries a span type and attributes in the ai.observability.* namespace, defined in the trulens-otel-semconv package. A record is simply the set of spans under one RECORD_ROOT span, which is the top-level call of your app.

Second, an exporter writes those spans somewhere. By default the destination is the database connector that belongs to the TruSession. The export is asynchronous: your function returns, and the spans travel to the database on a background path. This one fact explains a large share of "my results are empty" questions, and we return to it in the section on getting results.

Third, an evaluator runs metrics on the recorded data. The evaluator is a set of threads in your app process. It polls for new records belonging to apps that have metrics attached, runs each metric, and writes the results back as EVAL_ROOT and EVAL spans. A metric is at least one LLM call per record, which is why evaluation is the cost and throughput limit of the whole system.

Separately, the dashboard is a Streamlit subprocess that reads the same database. It does not talk to your app. The database is the integration point, and that has consequences: many app replicas can write to it, many readers can read from it, and all of them must agree on the schema version.

The vocabulary you need precisely. The TruSession is the process-wide singleton that owns the connector, the exporter, the evaluator threads and the query APIs. Calling TruSession() twice returns the same object. The connector is the storage backend: DefaultDBConnector for any SQLAlchemy database, SnowflakeConnector for Snowflake. An app recorder (TruApp, TruChain, TruGraph, TruLlama, TruBasicApp, TruVirtual) registers your application under an app_name and app_version. That pair is the unit of comparison in the leaderboard, in Trends and in Compare, so it is worth choosing version names that mean something, such as a git short SHA or a prompt version.

A metric is what older material calls a feedback function. Since 2.7.0 the unified Metric class replaces both Feedback and MetricConfig. Feedback still works, because it is now a subclass alias of Metric, but constructing one emits a DeprecationWarning, and the project states it will be removed in a future release. Write Metric in all new code. One oddity survives: the recorder constructor argument is still named feedbacks=, even though you pass Metric objects to it.

The official evaluation loop, in one line, is instrument, record, evaluate, compare or monitor, iterate. The rest of this guide is about making each arrow in that loop dependable.

Instrumentation in depth

Beginner taught you to decorate three methods. The problem at this level is that real applications have code you cannot edit, helper functions that matter, and data that does not look like a clean string. Instrumentation is how you tell TruLens what the structure of your app is, and every metric you write later depends on it.

Span types and what they are for

Each span has a type from SpanAttributes.SpanType: RECORD_ROOT, RETRIEVAL, GENERATION, AGENT, TOOL, MCP, GRAPH_TASK, GRAPH_NODE, WORKFLOW, RERANKING, GUARDRAIL, EVAL_ROOT, EVAL and UNKNOWN. The default is UNKNOWN. The type is not decoration. Selectors such as Selector.select_context() look for RETRIEVAL spans that carry a RETRIEVED_CONTEXTS attribute. If your retrieval function is decorated with no span type, it is recorded faithfully and is invisible to the context selector, and the context metrics then find nothing to score.

Each type has its own attribute namespace. For retrieval, SpanAttributes.RETRIEVAL.QUERY_TEXT and SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS. For the record root, SpanAttributes.RECORD_ROOT.INPUT and SpanAttributes.RECORD_ROOT.OUTPUT. The attribute mapping is a dictionary from attribute to a string naming either an argument of the decorated function or the special word "return":

from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes

class Pipeline:
    @instrument(
        span_type=SpanAttributes.SpanType.RETRIEVAL,
        attributes={
            SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
            SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
        },
    )
    def retrieve(self, query: str) -> list[str]:
        return self.index.search(query, k=5)

    @instrument(span_type=SpanAttributes.SpanType.GENERATION)
    def generate(self, query: str, context_list: list[str]) -> str:
        return self.llm(query, context_list)

    @instrument(
        span_type=SpanAttributes.SpanType.RECORD_ROOT,
        attributes={
            SpanAttributes.RECORD_ROOT.INPUT: "query",
            SpanAttributes.RECORD_ROOT.OUTPUT: "return",
        },
    )
    def answer(self, query: str) -> str:
        return self.generate(query, self.retrieve(query))

Without an explicit mapping, TruLens still logs every argument as ai.observability.call.kwargs.<name> and the return value as ai.observability.call.return. That default is why a bare @instrument() is a useful first step when you are exploring an unfamiliar code path: you get a span for free and can inspect its attributes in the dashboard before deciding how to map them.

Computing attributes with a callable

Sometimes the value you want is not an argument. The retrieval function returns objects with a .text field, and the metric wants strings. The attributes parameter also accepts a function that receives the return value, any exception, and the call's arguments, and returns a dictionary:

def retrieval_attrs(ret, exception, *args, **kwargs):
    return {
        SpanAttributes.RETRIEVAL.QUERY_TEXT: kwargs.get("query") or args[1],
        SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: [d.text for d in ret],
    }

class Pipeline:
    @instrument(span_type=SpanAttributes.SpanType.RETRIEVAL, attributes=retrieval_attrs)
    def retrieve(self, query: str):
        return self.index.search(query, k=5)

Note how the function receives self as the first positional argument for a method, so a positional query is args[1]. Handling both the keyword and positional case, as above, prevents a mismatch that only appears when a caller switches calling style.

Code you cannot edit

You cannot add a decorator to a vendor SDK class. instrument_method patches it from the outside:

from trulens.core.otel.instrument import instrument_method
from trulens.otel.semconv.trace import SpanAttributes
from vendor_sdk import CustomRetriever

instrument_method(
    cls=CustomRetriever,
    method_name="retrieve",
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    attributes={
        SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
        SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
    },
)

Since 2.9.0 this call is idempotent, so importing the module twice no longer stacks two wrappers on the same method. Before that, a notebook that re-ran its setup cell produced duplicate spans, which quietly doubled your retrieval counts.

The main method

TruApp needs to know which method is the entry point. When your class has a RECORD_ROOT decorated method, that is enough. When it does not, pass main_method=, and remember that it must be bound to the same instance you pass as the app:

app = Pipeline()
tru_app = TruApp(app, app_name="support-bot", app_version="v12", main_method=app.answer)

Passing Pipeline().answer while wrapping a different Pipeline() instance raises main_method <name> must be bound to the provided app instance. The message is exact, and so is the fix: create the instance once, keep it in a variable, and pass both from that variable.

Span groups

Agent and multi-hop apps call the same function many times in one record: a retriever three times, a tool five times. A metric scored once per record hides which call was bad. Span groups tag spans so that a metric runs once per group:

from trulens.core.otel.instrument import instrument, span_group

def run(question):
    with span_group("hop1"):
        first = retriever.retrieve(question)
    with span_group("hop2"):
        second = retriever.retrieve(rewrite(question, first))

You can also tag a function declaratively with @instrument(attributes={SpanAttributes.SPAN_GROUPS: "idx"}). Nested span_group contexts merge their labels, so an inner group keeps the outer one. Use groups when the question you want to answer is "which hop lost the answer", and skip them otherwise, because each group multiplies the number of metric evaluations and therefore the judge cost.

Selectors: how a metric finds its data

A metric is a function plus instructions for where to get each argument. The instructions are selectors. Most "the metric returned nothing" bugs are selector bugs, so this section is worth reading slowly.

Metric(implementation=..., name=..., selectors={...}, agg=...) takes a dictionary whose keys are the parameter names of the implementation. For provider.relevance, the parameters are prompt and response. For context_relevance they are question and context. For the groundedness family they are source, statement and, for the answerability variant, question. A key that does not match a parameter name is an error at best and a silently missing input at worst, so when you are unsure, print the signature: help(provider.context_relevance_with_cot_reasons).

The values are Selector objects. The shortcuts cover nearly everything in a RAG app:

When the shortcuts are not enough, build a selector from the pieces: span_type, span_name, function_name, span_attribute, function_attribute, trace_level, conversation_level, collect_list, ignore_none_values and match_only_if_no_ancestor_matched. For example, to read the retrieval query regardless of the function name:

from trulens.core import Selector
from trulens.otel.semconv.trace import SpanAttributes

query_selector = Selector(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    span_attribute=SpanAttributes.RETRIEVAL.QUERY_TEXT,
)

return_value_selector = Selector(
    function_name="my_module.my_function",
    function_attribute="return",
)

collect_list is a cost and a meaning decision

collect_list decides whether all matching values are passed to the implementation as one list (True, the default) or evaluated one at a time (False). The two choices answer different questions and cost different amounts.

Context relevance asks "is this chunk relevant to the question". Five chunks means five independent judgments, each a separate LLM call, and then agg (default np.mean) combines the five scores into one. So you want collect_list=False. Groundedness asks "is the answer supported by the context as a whole". That is one judgment over the joined context, so you want collect_list=True, which is also one call instead of five.

import numpy as np
from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o-mini")

context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons_consider_answerability,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
        "question": Selector.select_record_input(),
    },
)

The aggregation matters as much as the list setting. A mean over chunks hides one excellent chunk among four useless ones, and np.max answers "did we retrieve anything relevant at all". Neither is wrong. Pick the one that matches the failure you are trying to detect, and put the choice in the metric name if it is not the default, so the dashboard does not mislead the next reader.

Custom metrics are ordinary functions

Any callable that returns a float, or a (float, dict) pair of score and reasons, is a metric. A deterministic check costs nothing and never flakes, so use them wherever a rule is enough:

word_count = Metric(
    name="word_count",
    implementation=lambda text: len(text.split()),
    selectors={"text": Selector.select_record_output()},
)

def cites_source(answer: str, source: list) -> tuple[float, dict]:
    cited = any(f"[{i + 1}]" in answer for i in range(len(source)))
    return (1.0 if cited else 0.0), {"reason": "citation marker present" if cited else "no citation marker"}

Returning the (score, {"reason": ...}) shape makes your metric show its reasoning in the dashboard exactly like the built-in *_with_cot_reasons methods. Metric also accepts criteria, additional_instructions, examples, min_score_val, max_score_val (defaults 0 and 3), temperature (default 0.0) and description, which let you adjust a built-in judge without rewriting it. The judge is prompted on the min to max scale and TruLens normalises the result to the 0 to 1 range.

The newer form of the old chained methods

The chained .on_input(), .on_output(), .on_context() and .aggregate() methods still work on Metric, but the documentation recommends the selectors={} dictionary plus agg= for new code. The dictionary is explicit about parameter names, which makes a wrong key visible in code review. Also, the imp= argument is deprecated. Use implementation=.

Getting results: asynchronous export and the evaluator

Here is the problem this section solves. You write a script, run the app, immediately call session.get_leaderboard(), and see an empty table. Nothing is broken. You asked before the work finished.

Under OTel tracing, records are exported asynchronously and metrics are computed afterwards by the evaluator threads. The exit of the with block means your function returned, not that the spans are in the database and the metrics are scored. The supported way to wait is retrieve_feedback_results:

with tru_app as recording:
    app.answer("What is our refund window?")

df = recording.retrieve_feedback_results(timeout=180)
print(df[["Answer Relevance", "Context Relevance", "Groundedness"]])

The same call is available on the recorder, with explicit record IDs, as tru_app.retrieve_feedback_results(record_ids=..., timeout=180). On the session there are two lower-level waiters, session.wait_for_records(record_ids, timeout=10) for ingestion and session.wait_for_feedback_results(record_ids, feedback_names, timeout=60) for scoring. The recording object gives you recording.get() for the single record and recording.records and recording.record_ids for several.

Two rules follow from the asynchronous design. First, call session.force_flush() before a short-lived process exits. A batch script, a test, or a serverless function can reach its last line while spans are still in an export buffer, and the process exit discards them. Second, recording contexts cannot be nested. A with tru_app: inside another raises Already recording with a context manager, cannot nest!. Use one context per top-level call.

Why feedback_mode no longer applies

Older documentation describes four feedback modes, WITH_APP, WITH_APP_THREAD, DEFERRED and NONE, and a start_evaluator() call. Under OTel tracing, which is the default, only WITH_APP_THREAD exists, and the recorder raises ValueError("Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing!") for anything else. The legacy page "Running with your app" still lists all four modes because it describes the legacy path. Follow the code, not that page. Likewise, wait_for_feedback_results() on the app is not supported under OTel and points you to retrieve_feedback_results, and update_record() raises RuntimeError("Not supported with OTel tracing enabled!").

If you want evaluation decoupled from the request path, the OTel-era tools are three: sampled online evaluation through configure_online_eval, explicit recomputation through compute_feedbacks, and batch Runs. Each is covered below.

Recomputing on demand

When you add a metric to an app version after the records already exist, recompute instead of re-running the app:

tru_app.compute_feedbacks(
    raise_error_on_no_feedbacks_computed=True,
    metric_scope="record",   # "all", "record" or "conversation"
)

compute_feedbacks is OTel-only, and it raises This method is only supported for OTEL Tracing if you set TRULENS_OTEL_TRACING=0. A note from the source: it fetches all events for an app version at once, so on a large history pass events= explicitly or use a Run, both covered below.

Threads, async code and the "Cannot find TruLens context" error

The next trap arrives as soon as your app does anything concurrent. The message is Cannot find TruLens context. See https://www.trulens.org/component_guides/other/no_context_warning for more information. and the cause is always the same: an instrumented method ran in a different thread or task than the recording, so TruLens could not see which recording it belonged to.

TruLens tracks the active recording with Python context variables. A context variable is carried into a new asyncio task automatically on Python 3.11 and later, but a plain threading.Thread or ThreadPoolExecutor does not copy it. So an app that fans retrieval out to a thread pool quietly loses the context, the instrumented function runs un-recorded, and the warning appears.

The fixes, in order of preference:

1. Use the context-aware classes from TruLens: from trulens.core.utils.threading import Thread, ThreadPoolExecutor. They propagate the context. Importing threading after TruLens also works.
2. Use TP().submit for a submit-style API.
3. Wrap the target explicitly with contextvars.copy_context().run.
4. For asyncio, keep the default context copying and do not run coroutines on a separate event loop in another thread.

from trulens.core.utils.threading import ThreadPoolExecutor

class Pipeline:
    @instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT,
                attributes={SpanAttributes.RECORD_ROOT.INPUT: "query",
                            SpanAttributes.RECORD_ROOT.OUTPUT: "return"})
    def answer(self, query: str) -> str:
        with ThreadPoolExecutor(max_workers=4) as pool:
            chunks = list(pool.map(lambda shard: shard.search(query), self.shards))
        return self.generate(query, [c for shard in chunks for c in shard])

The same mechanism causes a second symptom in web frameworks. A request handler that hands work to a background worker process loses the recording entirely, because a process boundary cannot carry a context variable. Record inside the process that does the work.

Storage: from SQLite to PostgreSQL

By default, TruSession() creates default.sqlite in the current working directory and prints Initialized with db url sqlite:///default.sqlite. That is perfect for a laptop and wrong for anything else, for a concrete reason: SQLite is a file with a single writer. Two service replicas, a dashboard and a notebook will contend for it, and a container with an ephemeral filesystem loses it on restart. PostgreSQL support has existed since 2.6.0, and it is the right default for a team.

import os
from trulens.core import TruSession

session = TruSession(
    database_url=os.environ["DATABASE_URL"],   # postgresql+psycopg2://user:pass@host:5432/trulens
    database_redact_keys=True,
)

Install a synchronous driver: psycopg2-binary, psycopg2 or psycopg (version 3). An asyncpg URL (postgresql+asyncpg://) is not supported and surfaces as SQLAlchemy's confusing MissingGreenlet error. Supported URL forms are postgresql://, postgresql+psycopg2://, postgresql+psycopg://, and they accept ?sslmode=require. If you need full control of the engine, for example TLS settings or pool size, build it yourself and hand it over:

import sqlalchemy
from trulens.core import TruSession

engine = sqlalchemy.create_engine(
    os.environ["DATABASE_URL"],
    connect_args={"sslmode": "require"},
    pool_pre_ping=True,
)
session = TruSession(database_engine=engine)

The connector keyword arguments to know are database_url, database_engine, database_redact_keys (default False), database_prefix (default trulens_), database_args and database_check_revision (default True). Passing connector= together with any of them raises ValueError("Cannot provide both connector and connector argument(s) database_url.").

The singleton bites

TruSession is a singleton, so the first call wins. A second TruSession(database_url=...) anywhere later in the process logs TruSession was already initialized. Cannot change database configuration after initialization. and is ignored. The classic version of this bug: a library module calls TruSession() at import time with defaults, your main module later configures PostgreSQL, and all your traces land in a SQLite file you did not know existed. Configure the session once, at process start, before importing anything that might create it, and treat the startup banner as a required log line to check. For PostgreSQL the banner redacts the password, so it reads postgresql://trulensuser:***@localhost:5432/trulens.

An app constructed with a connector that differs from the session's raises Already created TruSession with different connector!, which is the same mistake reached from the other side.

Keys and secrets

database_redact_keys=True redacts keys in stored objects, and the project recommends it in production. Keep the database URL in the environment or a secret manager and build the engine in code. Do not commit the URL into a config file that the dashboard process also reads.

Schema versions and migrations

TruLens manages the schema with Alembic, and every client checks the revision at startup. You will meet three exact errors:

The second error is the operational one. When a shared PostgreSQL is used by several services and a dashboard, the first process to start on a new version migrates the database, and every service still on the old version then refuses to start. So upgrade in lockstep: pin exact versions of all trulens-* packages in one constraints file shared by all services, and roll them together. database_check_revision=False skips the check, and you should not use it, because it trades a clear startup error for silent schema mismatches.

pg_dump "$DATABASE_URL" --format=custom --file=trulens-before-upgrade.dump
pip install --upgrade "trulens==2.14.0" "trulens-providers-openai==2.14.0"
python -c "from trulens.core import TruSession; TruSession(database_url='$DATABASE_URL').migrate_database()"

Online evaluation: sampling, throttling and a cost budget

The problem: attaching three LLM-judged metrics to a production app means every user request triggers at least three extra LLM calls, and context relevance with collect_list=False multiplies that by the number of chunks. On a busy service the evaluation bill can exceed the application bill, and the evaluator threads can fall behind the traffic. Since 2.11.0, configure_online_eval puts dials on this.

session.configure_online_eval(
    sample_rate={"prod_rag": 0.05, "staging_rag": 1.0},
    throttle=100,
    cost_budget=10.0,
    feedbacks=[context_relevance, groundedness, answer_relevance],
)

sample_rate is either a single float for every app or a dictionary by app_name. Apps missing from the dictionary are evaluated at 100 percent, which is an easy way to get a surprise bill when a new app ships. throttle caps the evaluation rate. cost_budget is a daily limit in US dollars that resets at UTC midnight. A second call to configure_online_eval replaces the whole configuration, so keep it in one place in your startup code rather than scattering partial calls.

Sampling is deterministic: TruLens hashes the record_id with SHA-256, salted with the app name, so the same record gets the same decision on every replica and on every restart. That matters for reproducibility, and it means you cannot "retry until it is sampled". The resulting DataFrame gains three columns, sampled, sample_rate and eval_decision_reason, and the reason is one of evaluated, not_sampled, throttled or over_budget. The counts are also available as session.sampling_controller.counters. The leaderboard shows the observed sample rate, so a reader can tell that a score is based on five percent of traffic.

Explicit calls are never gated. compute_now() and compute_metrics() ignore the sampling controller, which gives you a backfill path: sample at five percent in production, then when a customer complains about a specific conversation, evaluate exactly those records.

skipped = df[df["eval_decision_reason"] == "not_sampled"]["record_id"].tolist()
tru_app._evaluator.compute_now(record_ids=skipped[:20])

Note the leading underscore on _evaluator: it is an internal attribute that the project documents for this purpose but that can change. Wrap it in one helper function in your codebase so a future rename is a one-line fix.

The other cost levers, in rough order of impact: use collect_list=True where one judgment over the joined context is enough; choose a cheaper judge model, verified against human labels as described later; avoid Jury except where judge disagreement is the problem; and turn on trace compression through the metric's enable_trace_compression option for trace-level agent metrics, where the full span tree can be large.

Evaluating without a running app: BatchEvaluator and Runs

Online evaluation scores what real users send. You also need evaluation on a fixed dataset, before a release, so that two prompt versions are compared on the same questions. TruLens gives you two tools with different guarantees, and choosing the wrong one is a common mistake.

BatchEvaluator (2.13.0) runs metrics over a DataFrame or a list of dictionaries in memory. Nothing is persisted and no instrumentation is needed. It is the right tool for notebooks, unit tests and for scoring outputs that came from somewhere else, such as a CSV of answers exported from a vendor system.

import pandas as pd
from trulens.core import BatchEvaluator, Metric, Selector

df = pd.DataFrame({
    "question": ["What is the refund window?"],
    "answer": ["Refunds are accepted within 30 days."],
    "contexts": [["Our policy allows refunds within 30 days of purchase."]],
})

groundedness = Metric(
    name="Groundedness",
    implementation=provider.groundedness_measure_with_cot_reasons,
    selectors={
        "source": Selector.from_column("contexts", collect_list=True),
        "statement": Selector.from_column("answer"),
    },
)

scored = BatchEvaluator(metrics=[groundedness], max_workers=8).evaluate(df)
print(scored["Groundedness"])

The result is a DataFrame with a column per metric. A list of dictionaries works too, with column_map={"user_question": "query"} to rename fields. max_workers controls parallel judge calls, so set it to match your provider's rate limit rather than to the number of CPU cores, because the work is waiting on the network.

Runs are the persisted version. A Run evaluates an app over a dataset and stores the results in the database so the Compare page can put two versions side by side. A local TruSession wires in a SQLite-backed run store, so Runs work without Snowflake.

from trulens.core.run import RunConfig, RunStatus

run = tru_app.add_run(
    run_config=RunConfig(
        run_name="eval_2026_10_01",
        dataset_name="support_questions",
        source_type="DATAFRAME",
        dataset_spec={"input": "question", "ground_truth_output": "expected"},
        description="Regression set, prompt v12",
    )
)

run.start()                       # returns before ingestion completes
print(run.get_status())           # poll until invocation is complete
run.compute_metrics([groundedness, answer_relevance])

run.start() returns before ingestion is complete, and calling compute_metrics too early is one of the silent failures in the source notes. Poll run.get_status() until the status reaches INVOCATION_COMPLETED. The full status vocabulary is CREATED, INVOCATION_IN_PROGRESS, INVOCATION_COMPLETED, COMPUTATION_IN_PROGRESS, COMPLETED, PARTIALLY_COMPLETED, FAILED and CANCELLED. PARTIALLY_COMPLETED deserves respect: it means some metrics or records failed while others finished, so a dashboard showing averages from a partial run is averaging a biased subset.

A run has two modes: APP_INVOCATION, the default, which calls your app on each dataset input, and LOG_INGESTION, which builds spans from data you already logged. Related methods are tru_app.get_run(name), list_runs(), delete() and delete_version(). Errors such as Run <name> not found. and run_name is required when app is not None are API misuse, so check the name you passed.

Guardrails: metrics as runtime gates

Everything so far observes. A guardrail uses the same metric on the request path and changes what happens. The decorators are block_input, block_output and context_filter, all from trulens.core.guardrails.base.

from trulens.core import Metric
from trulens.core.guardrails.base import block_input, block_output, context_filter

harm = Metric(implementation=provider.criminality, higher_is_better=False)
relevance = Metric(implementation=provider.context_relevance)

class SafeApp:
    @block_output(feedback=harm, threshold=0.9,
                  return_value="I can't help with that request.")
    def respond(self, question: str) -> str:
        return self.llm(question)

    @context_filter(relevance, 0.5, keyword_for_prompt="query")
    def retrieve(self, query: str) -> list[str]:
        return self.index.search(query, k=8)

Two details cause most guardrail bugs. First, a guardrail metric must return a bare float. The documentation says a feedback function used as a guardrail must only return a float score, so a *_with_cot_reasons variant, which returns a tuple, breaks it. Use the float-only method. Second, direction: a harmfulness metric is bad when high, so higher_is_better=False tells the guard to block above the threshold rather than below. Forgetting it inverts the guard, which is dangerous because it blocks your good answers and passes your bad ones.

The cost consideration is unique: a guardrail is synchronous and adds an LLM call to user-visible latency. context_filter on eight chunks runs eight relevance judgments before generation starts. Where that is too slow, use a small local classifier such as a Hugging Face provider method, or a deterministic check, and keep the LLM judge for offline evaluation. For frameworks there are ready wrappers: WithFeedbackFilterDocuments.of_retriever(retriever=..., feedback=..., threshold=0.5) for LangChain and WithFeedbackFilterNodes(query_engine, feedback=..., threshold=0.5) for LlamaIndex. Since 2.8.1, SpanType.GUARDRAIL lets the guard's own activity show as a distinct span type.

Conversations, agents and trace-level metrics

A single-turn RAG record is self-contained. A chatbot is not: turn five makes sense only with turns one to four, and a per-turn metric cannot tell you whether the conversation drifted. Since 2.12.0, TruLens groups records by a conversation ID and supports conversation-scoped metrics.

with tru_chatbot(conversation_id="conv-001") as recording:
    chatbot.reply("I need to change my flight")
with tru_chatbot(conversation_id="conv-001") as recording:
    chatbot.reply("Make it Thursday instead")

coherence = Metric(
    implementation=provider.coherence_across_turns,
    name="Coherence Across Turns",
).on_conversation()

.on_conversation() makes the metric run once per conversation, and the selectors Selector.select_conversation(), select_conversation_input() and select_conversation_output() read the combined content. Retrieve them with session.get_conversations() and session.get_records_by_conversation(...). The 2.14.0 dashboard labels the conversation thread columns "latest" instead of "avg". Conversation metrics are not supported on SnowflakeConnector, which matters if a Snowflake-first team is evaluating TruLens for chatbots.

For agents, the metrics are trace-level: they read the whole tree of a record rather than one input and output. The names follow the GPA taxonomy, and each has a _with_cot_reasons form: logical_consistency, execution_efficiency, plan_adherence, plan_quality, tool_selection, tool_calling and tool_quality. You select the whole trace with Selector(trace_level=True).

from trulens.core import Metric, Selector

plan_quality = Metric(
    implementation=provider.plan_quality_with_cot_reasons,
    name="Plan Quality",
    selectors={"trace": Selector(trace_level=True)},
)

The parameter name in selectors has to match the real signature of the method, so print it with help() as you did earlier rather than trusting this example's key. Trace-level metrics are the most expensive kind, since the judge reads the entire span tree. That is exactly where sampling and trace compression pay off. For LangGraph, use TruGraph from trulens-apps-langgraph, and for LlamaIndex Workflows the 2.4.0 tracing is part of TruLlama.

Trusting the judge

The least-discussed fact about LLM-judged metrics is that they are measurements made with an instrument you have not calibrated. Every 0.83 in your dashboard is a judge model's opinion, scaled. If the judge is biased towards long answers, or disagrees with your domain experts, the leaderboard is precise and wrong.

Start with cheap hygiene. Pin the judge model explicitly: OpenAI(model_engine="gpt-4o-mini") rather than relying on the class default, because defaults changed in 2.8.1 for several providers and will change again. The 2.14.0 defaults are gpt-4o-mini (OpenAI and LiteLLM), claude-sonnet-4-6 (Anthropic), gemini-2.5-flash (Google), amazon.nova-lite-v1:0 (Bedrock), llama3.3-70b (Cortex) and llama3.2 (Ollama). The judge temperature defaults to 0.0, which reduces but does not eliminate variation between runs.

Then measure. The trulens-benchmark package and trulens.feedback provide tools for exactly this, added in 2.9 through 2.12:

A practical workflow for a team with no labels yet: sample fifty traces with GoldenSetGenerator using the uncertainty strategy, have a domain expert label them as pass or fail in an afternoon, run AlignmentReport, and only then argue about thresholds. If the judge agrees with the expert at, say, 85 percent, you know what a score difference of 0.05 is worth, and if it agrees at 60 percent you have saved yourself months of optimising a noisy number. CrossModelAlignment then answers the cost question with evidence: if a small model agrees with a large one, downgrade the judge and cut the bill.

The Jury tool aggregates with mean, median, trimmed mean, majority vote or weighted mean. It costs one call per juror, so spend it on the metrics where disagreement is large, not on all of them.

The dashboard, Trends and the CLI

The dashboard has four pages. The Leaderboard shows aggregate metric means, latency and cost per app version, plus the observed online-evaluation sample rate. Records shows individual traces, evaluation details and conversation threads. Compare puts app versions side by side. Trends, new in 2.13.0, plots metrics with 95 percent confidence intervals, average, P90 and P99 latency, app cost and evaluation cost, in daily or weekly buckets, and offers "Investigate records" to jump from a bad bucket to the traces in it.

The shape of the work follows from the pages. Use Compare before a release to choose between versions. Use Trends after a release to detect regression: a metric that drifts down over two weeks while latency holds is usually a data or upstream change rather than a code change. And use Records to read the reasons when a number moves, because a score without its reason text is an invitation to guess.

From Python:

from trulens.dashboard import run_dashboard

run_dashboard(session, port=8502, address="127.0.0.1", force=True)

force=True stops any existing dashboards first, and without it a second call raises RuntimeError: Dashboard is already running or returns the existing process. The methods session.run_dashboard(), start_dashboard() and stop_dashboard() are deprecated aliases, so use the functions from trulens.dashboard, with stop_dashboard in trulens.dashboard.run. The dashboard also refuses to start on a SnowflakeConnector: from 2.7.2 it raises RuntimeError and tells you to use the Evaluations UI in Snowsight.

Since 2.14.0 there is also a separate command that does not need your app code at all, which is the right way to give colleagues a read-only view:

trulens-dashboard --find                                   # print which database would be served
trulens-dashboard --dir ./runs
trulens-dashboard --database-url postgresql+psycopg://user:pass@host/db --database-prefix trulens_
trulens-dashboard --port 8502 --address 127.0.0.1 --force

The database URL is resolved in a fixed order: --database-url, then the TRULENS_DATABASE_URL environment variable, then discovery of default.sqlite or another SQLite file with TruLens tables in the directory. If it finds none it prints No TruLens database found in <dir>, which usually means you ran it from the wrong directory. Because the dashboard is a separate process, it must run the same TruLens version as the writers, or it hits the schema-mismatch errors described earlier.

Exporting to an OpenTelemetry backend

Because TruLens emits standard OpenTelemetry spans, teams often ask whether traces can go to Jaeger, Grafana Tempo, Datadog or Honeycomb. Since 2.14.0 they can, through OTLP export:

session = TruSession(otel_exporter="otlp", otlp_endpoint="http://otel-collector:4317")

Read the constraints before adopting it, because three of them surprise people. The transport is gRPC only in 2.14.0, so use port 4317, not the HTTP port 4318. HTTP with protobuf has been merged to the development branch after 2.14.0 and is not available in the release you install. OTLP mode replaces the database exporter and does not fan out, so the TruLens dashboard and any evaluation that reads the database will not see those spans. You choose between the TruLens database and OTLP for a session, and cannot have both from one. If you omit the endpoint, the standard OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS variables apply. The third surprise is dependency resolution: the extra trulens[otlp] adds the gRPC exporter, which pins the OpenTelemetry SDK range, and if your application pins a different SDK version pip will fail. Install a matching pair or isolate the environment.

The exported metrics are gen_ai.client.token.usage, with gen_ai.token.type of input or output, and gen_ai.client.operation.duration in seconds, and 2.14.0 added time-to-first-token and throughput for streaming. From 2.9.0 TruLens dual-emits the official gen_ai.* attributes next to its own, which is what lets a generic backend render them meaningfully.

Prompt and response content is not exported as span events by default. The opt-in is TRULENS_OTEL_CAPTURE_CONTENT=true or the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Because traces then contain user text, treat that switch as a data-protection decision, not a debugging convenience. For Gulf and Egyptian employers that handle regulated customer data, it is also a residency question: check where the collector and its storage physically live before enabling content capture.

The reverse direction also works: TruLens can consume spans produced by other OpenTelemetry instrumentation, such as opentelemetry-instrumentation-openai, and you can write selectors against their attributes. If your organisation already runs Grafana or a vendor APM, this is the cheapest way to get LLM spans into it. If your team wants a dedicated LLM observability UI with prompt management and tracing, compare it with Langfuse and Arize Phoenix, which take different positions on where evaluation lives.

Testing and CI gates

An evaluation is a test that has a fuzzy oracle. You can still wire it into pull requests, and the project ships official examples for GitHub Actions, Azure Pipelines and a pre-commit smoke test. The design question is what to gate on, because LLM-judged scores are noisy and a gate that flakes will be disabled within a week.

A workable design has three layers. The first is deterministic checks on every pull request: does the app import, does a single recorded call produce a RECORD_ROOT span, do the selectors match, is the output valid against a schema. Since 2.8.0, SchemaValidation provides a non-LLM JSON or Pydantic check, and it is free. The second is a small LLM-judged regression set on pull requests that touch prompts or retrieval, using BatchEvaluator or a Run over perhaps thirty to fifty questions, and failing only on a drop larger than your measured noise. The third is a nightly larger run with Trends watching the slope.

import sys
import pandas as pd
from trulens.core import BatchEvaluator

THRESHOLDS = {"Groundedness": 0.80, "Answer Relevance": 0.85}

def main() -> int:
    rows = pd.read_json("eval/regression.jsonl", lines=True)
    answers = rows.assign(answer=rows["question"].map(app.answer))
    scored = BatchEvaluator(metrics=METRICS, max_workers=4).evaluate(answers)
    means = scored[list(THRESHOLDS)].mean()
    print(means.to_string())
    failed = {k: round(v, 3) for k, v in means.items() if v < THRESHOLDS[k]}
    if failed:
        print(f"FAILED: {failed}")
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

name: llm-eval
on:
  pull_request:
    paths: ["prompts/**", "src/retrieval/**", "eval/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.lock
      - run: python eval/ci_eval.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Three practices keep a gate honest. Use the mean over the set with a margin, not per-record thresholds, because a single borderline record would flap. Fix the judge model, temperature and criteria, since a judge change looks exactly like an app regression. And remember a BatchEvaluator result is in memory only, so the job should print or upload the table as an artifact, or the evidence vanishes with the runner. NaN values deserve a special mention: as of 2.14.0, IR metrics such as ndcg_at_k and recall_at_k return nan rather than 0.0 when the result is undefined, for example for an empty relevance list, so a mean that once quietly included zeros now has to decide what to do with missing values. Use skipna deliberately and count how many rows were dropped.

This pairs naturally with a tool built specifically for test-style evaluation. DeepEval uses a pytest model, and promptfoo is configuration-driven, so if your team wants assertions in a test runner rather than a dashboard, compare them with the approach here.

Integrating with frameworks, agents and coding tools

TruLens wraps the popular frameworks with dedicated classes, each from its own package: TruChain for LangChain (trulens-apps-langchain), TruGraph for LangGraph, TruLlama for LlamaIndex, TruRails for NeMo Guardrails (Python below 3.13), and TruBasicApp for any text-to-text function. They auto-instrument the framework's components, so you often need no decorators. The metrics and selectors are the same.

The trap is the missing optional package. Importing trulens.apps.langchain without trulens-apps-langchain raises These packages are required for using <X>: <pkg> ... pip install "<pkg spec>", which is a clear message, but a version mismatch yields the other variant, There was a problem using these packages, which asks you to install the exact packages. Both come from keeping the family out of lockstep.

Version 2.14.0 added an unusual integration: coding agents. The trulens-client-hooks command installs hooks that send traces from Claude Code, Cursor or OpenCode to a destination of your choice.

pip install trulens-core trulens-apps-claude
trulens-client-hooks install claude-code --dry-run
trulens-client-hooks install claude-code
trulens-client-hooks status cursor
trulens-client-hooks flush

The destination is TRULENS_DESTINATION: database (local SQLite by default, TRULENS_DATABASE_URL to change it), snowflake, or otlp. The privacy design is conservative: only lifecycle metadata is captured by default, and content, tool payloads, diffs and file paths are each separate opt-ins (TRULENS_CAPTURE_CONTENT, TRULENS_CAPTURE_TOOL_PAYLOADS, TRULENS_CAPTURE_DIFFS, TRULENS_CAPTURE_PATHS), redacted and size-bounded by TRULENS_MAX_FIELD_BYTES, which defaults to 16384. Hooks fail open, which means a broken hook never blocks the coding client. That is the right property for developer tooling, and it also means a misconfigured destination fails silently, so run trulens-client-hooks validate after installing.

The 2.14.0 release also added session.curate_records_to_dataset(...), which turns selected production traces into a dataset. The IDs are content-addressed, so curating the same rows twice is idempotent. This closes a loop worth having: sample production traces, label the interesting ones, curate them into a dataset, and run them as the regression set above, so your gate reflects real failures.

Security, data residency and multi-team use

Traces are a copy of your application's data. They contain prompts, retrieved documents and model outputs, and those can contain personal data. The judge call is a second copy: your content is sent to the judge provider. A mid-level engineer needs to know where each copy lives.

The controls available in 2.14.0 are database_redact_keys=True, the opt-in content capture switches described above, and the choice of provider. For regulated data, use a local judge: Ollama through trulens-providers-ollama, HuggingfaceLocal, or any OpenAI-compatible server through OpenAI(model_engine=..., api_key=..., base_url="http://localhost:8000/v1"), which covers vLLM and similar. LiteLLM(model_engine="ollama/llama3.1:8b", api_base="http://host:11434") reaches an internal Ollama. AzureOpenAI(deployment_name=...) lets you keep the judge inside an Azure region you chose, and Bedrock runs inside your AWS account and region, which is how many teams in the Gulf keep prompts in-region. Always confirm that the model and region you select actually exist for your tenant, and pin them.

There is no tenant or role model in the dashboard. If two teams share one database they see each other's traces. Your isolation choices are separate databases or schemas, a database_prefix per tenant inside one database, app_name conventions, and Snowflake roles when using SnowflakeConnector. Separate databases are the honest answer when the data must not mix, because prefixes isolate tables but not access.

Secrets: provider keys come from the environment or a .env file read through python-dotenv, from the current directory or its parents. A missing key raises ApiKeyError with a message that begins Key OPENAI_API_KEY needs to be set. Do not bake keys into images, and do not log the database URL. Also keep in mind that a .env in a parent directory is silently picked up, which is convenient on a laptop and surprising in a shared build environment.

Debugging beyond the basics

Most TruLens problems at this level are silent, so the debugging method is to walk the pipeline from the front and ask at each stage whether data arrived. This checklist follows the architecture.

1. Are spans created at all? Check TRULENS_OTEL_TRACING. If it is 0 or false, tracing is off, and since 2.14.0 a warning is logged once per process. Also confirm you imported the OTel instrument from trulens.core.otel.instrument, not the legacy one from trulens.apps.custom.
2. Are they reaching the database? Read the startup banner for the database URL. Then call session.force_flush() and query session.get_records_and_feedback(app_name=...). An empty result with OTLP mode on is by design.
3. Did the selectors match? Open the record in the dashboard and read the attribute names on the span. A missing RETRIEVAL type or missing RETRIEVED_CONTEXTS attribute means select_context() finds nothing.
4. Did the evaluator run? Look for the eval_decision_reason column. not_sampled, throttled and over_budget are expected outcomes, not errors.
5. Did the judge answer? Check the API key, rate limits, and whether the model name exists for that provider.
6. Is the version consistent? Compare every package version across the app, the dashboard and the database migration state.

A few other messages to recognise quickly. Invalid TruLens OTEL Tracing syntax. means you mixed the legacy recording syntax with OTel mode. Not supported with OTel tracing enabled! means a legacy helper such as update_record or with_record was called. This method is only supported for OTEL Tracing. Please enable OTEL tracing in the environment! means the opposite: a new API was called with tracing disabled. Finally, address argument cannot be used in colab is what you get if you pass address= to run_dashboard in Colab.

Putting it all together

Here is one realistic setup that uses the pieces in order. A support assistant runs as a FastAPI service with three replicas. The goal is to evaluate it cheaply in production, gate releases on a regression set, and keep the data in-region.

The service configures the session exactly once in its startup module, using PostgreSQL from DATABASE_URL with database_redact_keys=True, before any other import could create a default session. The app class maps a RETRIEVAL span with query text and retrieved contexts, and a RECORD_ROOT span with input and output. Retrieval fans out with the TruLens ThreadPoolExecutor so the context propagates. The judge is an explicitly pinned model on a provider deployed in the region the company has chosen.

The metrics are the triad, with context relevance per chunk and groundedness over the whole list, plus one deterministic cites_source check and a schema validation. configure_online_eval samples five percent of production with a throttle, and a budget is set only because the provider reports costs. A separate staging app name runs at 100 percent. The dashboard runs as trulens-dashboard behind SSO, bound to localhost, with the same pinned version as the writers.

In CI, the thirty-question regression set runs through BatchEvaluator on every pull request that touches prompts or retrieval, gated by a margin measured from repeated runs. Each week, a person curates the worst production traces with curate_records_to_dataset, labels them, and adds them to the regression set. Trends is reviewed after each release for slope, and upgrades happen for every trulens-* package in one constraints-file change, after a pg_dump.

service/
  startup.py            # TruSession configured once, before anything else
  app.py                # Pipeline with RETRIEVAL + RECORD_ROOT spans
  metrics.py            # triad + cites_source + SchemaValidation
  online_eval.py        # configure_online_eval in one place
eval/
  regression.jsonl      # curated from real failures
  ci_eval.py            # BatchEvaluator + thresholds with margin
constraints.txt         # every trulens-* package pinned to 2.14.0

Note what this design does not do. It does not attach an LLM-judged guardrail to the request path, because of the latency. It does not share one database with another team. And it does not trust a score without an agreement measurement against human labels for the two metrics the release decision depends on.

Where you are now

You can now predict what a TruLens metric will see by reading the span tree and the selector, rather than hoping. You know that export is asynchronous, that the session is a singleton and the first configuration wins, that SQLite does not scale past one process, and that every client must run the same version as the database. You can control online evaluation cost with sampling, throttling and a budget, and you know which providers the budget actually works for. You can evaluate offline with BatchEvaluator or persisted Runs, gate a pull request on the result, use a guardrail without breaking the return type, evaluate conversations and agent traces, and export to an OpenTelemetry backend knowing it replaces the database exporter.

Most importantly, you treat the judge as an instrument that needs calibration, and you treat the dashboard as something that needs access control. The Senior level takes this from one service to a platform: what breaks first as volume grows, multi-team isolation, upgrade and migration planning, governance, and when to pick a different tool. If you are choosing between tools rather than learning this one, the catalogue also has Langfuse, Arize Phoenix, DeepEval and promptfoo guides to compare against.

Sources

• TruLens documentation home
• Installation
• Core concepts
• The RAG Triad
• Dashboard
• Trends dashboard page
• OpenTelemetry tracing in TruLens
• Semantic conventions
• Instrumentation guide
• Span groups
• Conversation evaluation
• Client hooks for coding agents (source at tag trulens-2.14.0)
• Metric migration guide
• Feedback providers
• Running feedback functions with your app
• Batch evaluation
• Agentic evaluations
• LLM judge alignment
• Where to log
• Logging in PostgreSQL
• Logging in Snowflake
• Guardrails
• Inline evals
• The "Cannot find TruLens context" warning
• Release history
• TruLens 2.14.0 release notes
• Deprecation policy and list
• trulens on PyPI

This is the mid-level interview guide for TruLens. At this level an interviewer is not asking whether you can name the RAG Triad. They want to know whether you have run evaluation against real traffic and been burned by it: whether you can explain why a metric column is empty, what an evaluation costs, where the data lives, and how you would decide whether a score is trustworthy. The answers below are written the way a strong candidate would speak them, with the reasoning shown, and each question ends with what separates a strong answer from a weak one. Everything is checked against TruLens 2.14.0.

Where this picks up

The beginner interview guide covered the thirty-second answer, the RAG Triad definition and the first commands. This guide assumes you can already say that TruLens traces LLM apps as OpenTelemetry spans, scores them with metrics, and stores results in a SQL database you view in a Streamlit dashboard. From here the questions are scenarios and trade-offs. If a term below is unfamiliar, read the mid-level detailed guide first, because the interview answers lean on its mechanisms: the asynchronous exporter, the singleton session, selectors, and the evaluator threads.

A useful frame for every answer is the pipeline: instrument, record, evaluate, compare or monitor, iterate. When something goes wrong in an interview scenario, say which stage you would inspect first and why. Interviewers reward a candidate who debugs in pipeline order rather than guessing.

Scenario questions

The leaderboard is empty, but the app is clearly being called. What do you check?

Start with the most common cause and work forward. First, did the spans get created. Check the TRULENS_OTEL_TRACING environment variable, because setting it to 0 or false turns OTel tracing off, and recent versions log a warning once per process when that happens. Confirm the code imports instrument from trulens.core.otel.instrument rather than the legacy module.

Second, are they reaching the database. Look at the startup banner for the database URL, because a singleton session may have been created earlier by a library with default settings, sending everything to a default.sqlite you did not expect. If the script ends right after the call, spans may not have flushed, so call session.force_flush(). Also recall that export is asynchronous, so reading the leaderboard immediately after the call can show nothing. Third, are metrics being computed. A blank score column usually means the selector matched no span, for instance because the retrieval function has no RETRIEVAL span type or no RETRIEVED_CONTEXTS attribute. Fourth, check whether OTLP export is enabled, since that replaces the database exporter by design.

What a strong answer adds: it walks the pipeline in order, names the asynchronous export and the singleton as real causes, and mentions that a missing selector match is silent rather than an exception. What a weak answer does: reinstalls the package or blames the API key first.

Context relevance scores are fine but groundedness is always low. How do you investigate?

These measure different things, so a split tells you where the failure is. Context relevance asks whether retrieved chunks relate to the question, and groundedness asks whether the answer is supported by the context. If retrieval is good and groundedness is low, either the generator is adding claims from its own knowledge, or the groundedness metric is not seeing the context you think it is. So inspect the metric's configuration before the model.

I would open a few low-scoring records in the Records page and read the reason text from the *_with_cot_reasons variant. I would check the selector: groundedness should use Selector.select_context(collect_list=True) so the judge sees all the context together. If it were collect_list=False, each chunk would be judged alone against the whole answer, and a correct answer that draws on several chunks would score low against each one. Then I would check whether the prompt encourages the model to use outside knowledge, and whether the consider_answerability variant is appropriate, because it avoids penalising a correct "I don't know" when the context lacks the answer.

Strong: separates metric configuration from model behaviour, knows the collect_list meaning, reads the reasons. Weak: raises the threshold or swaps the judge before looking at a single record.

The evaluation bill is larger than the app's own LLM bill. What do you do?

First find where the calls come from. Every metric is at least one judge call per record, and a per-chunk metric such as context relevance with collect_list=False makes one call per retrieved chunk. Trace-level agent metrics read the whole span tree and are the most expensive. So I would count calls per metric on a sample of records.

Then apply levers in order of leverage. Use configure_online_eval to sample, for example five percent of production, with a throttle, and a cost_budget in dollars per day. Be careful here: the budget is enforceable only for providers that report costs, which in 2.14.0 means OpenAI, LiteLLM, Google and Cortex, and not Anthropic, so I would verify rather than assume. Next, convert metrics to a single call where the question allows it, replace LLM metrics with deterministic checks where a rule is enough, enable trace compression for trace-level metrics, and move to a cheaper judge. I would not just pick a cheaper judge blindly. I would run CrossModelAlignment or compare it against a human-labelled set, to make sure the cheaper one agrees.

Strong: quantifies first, knows the reports_costs limitation, and validates a cheaper judge. Weak: turns evaluation off in production entirely, losing the signal.

A deploy fails at startup with "Database schema is ahead of the expected revision". Explain and fix.

The shared database was migrated by a newer TruLens client, and this service still runs an older version. It is common when several services and a dashboard use one PostgreSQL and they are not upgraded together. The first process to start on the new version migrates the schema, and everything on the old version then refuses to start.

The fix is to upgrade the stale client to the same version as the rest. The prevention is process: pin exact versions of every trulens-* package in one shared constraints file, roll all writers and readers together, and take a backup with pg_dump before calling TruSession().migrate_database(). I would explicitly not use database_check_revision=False, which only hides the problem and risks silent schema mismatches. The sibling errors are "behind", which needs migrate_database(), and "reconfigured", which happens when database_prefix changed and needs migrate_database(prior_prefix=...).

Strong: identifies a coordination problem rather than a bug, and proposes lockstep pinning. Weak: disables the revision check.

Your retrieval runs in a thread pool and the logs say "Cannot find TruLens context". Why?

TruLens finds the active recording through Python context variables. Those propagate into asyncio tasks on Python 3.11 and later, but a plain threading.Thread or concurrent.futures.ThreadPoolExecutor does not copy the context, so the instrumented method runs outside the recording and the spans are lost. I would switch to Thread or ThreadPoolExecutor from trulens.core.utils.threading, or use TP().submit, or wrap the function with contextvars.copy_context().run. If work crosses a process boundary, for example a background worker, no context variable can cross it, so I would record inside the process doing the work.

Strong: explains context variables, gives at least two fixes, and mentions the process boundary. Weak: suggests disabling the warning.

A teammate's script says feedback_mode=FeedbackMode.DEFERRED and raises a ValueError. What happened?

The script was copied from older documentation. Under OTel tracing, which has been the default since 2.3.0, only WITH_APP_THREAD is allowed, and any other mode raises Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing!. The documentation page for running with your app still lists four modes because it describes the legacy path. The code is the authority. To decouple evaluation from the request, use sampled online evaluation through configure_online_eval, explicit compute_feedbacks(), or batch Runs, and use retrieve_feedback_results when a script needs to wait for scores.

Strong: knows docs and code can disagree and names the OTel-era replacements. Weak: removes the argument without understanding why evaluation still runs.

Two prompt versions score 0.81 and 0.83. Is the newer one better?

I would not claim it from that evidence alone. Several things can make a two-point gap meaningless: judge noise even at temperature 0.0, a small evaluation set, a different sample of questions, or a judge that is biased. I would check the confidence intervals in Trends, hold the judge model, criteria and dataset fixed, run both versions on the same regression set through a Run so Compare can put them side by side, and repeat the run to measure the natural variation. Then I would look at individual records where the two differ, reading the reasons. If the judge's agreement with human labels is unknown, I would measure it with a small labelled set, because a two-point difference from a judge that agrees with experts only 65 percent of the time is not informative.

Strong: raises noise, sample, and judge validity, and uses confidence intervals and record-level review. Weak: ships the higher number.

How would you gate a pull request on an LLM evaluation without it flaking?

Use layers. On every pull request, run deterministic checks, which are free and stable: the app imports, one recorded call produces the expected spans, the selectors match, and the output validates against a schema with SchemaValidation. On pull requests touching prompts or retrieval, run a small regression set of perhaps thirty to fifty questions through BatchEvaluator or a Run, and compare the mean of each metric to a threshold set below the noise floor I measured from repeated runs, not at the value I hope for. Fix the judge model, temperature and criteria so a judge change cannot look like an app regression. Run a larger set nightly and watch Trends. Upload the result table as a CI artifact, since BatchEvaluator output is in memory only.

Strong: separates deterministic from judged checks, bases thresholds on measured noise, and keeps the evidence. Weak: a single hard threshold on a single record.

The security team asks where customer text goes when you use TruLens. Answer.

It goes to two places. Traces in your database hold prompts, retrieved documents and outputs, which can include personal data. And each judge call sends your content to the judge provider. I would list both. For the database, I would use PostgreSQL in our region with access control, set database_redact_keys=True, and keep GenAI content capture as span events off, which is the default. For the judge, I would use a provider deployment inside the approved region, such as Azure OpenAI or Bedrock in the chosen region, or a local judge through Ollama or a vLLM endpoint via base_url when data must not leave our network. I would also tell them that the dashboard has no authentication of its own, so it needs to sit behind SSO or a reverse proxy, bound to localhost.

Strong: identifies both copies, mentions the dashboard's lack of auth, and proposes concrete mitigations. Weak: says "it's open source so it stays with us".

The dashboard shows data from your app, but a colleague sees nothing in theirs. Why?

They are almost certainly reading a different database. The default is a default.sqlite in the working directory, so two people running from different directories have different stores. The trulens-dashboard --find command prints which database would be served. The shared answer is a PostgreSQL URL supplied through TRULENS_DATABASE_URL or --database-url, and a check of the startup banner.

Strong: knows the discovery order and uses --find. Weak: restarts the dashboard.

A metric that used to give 0.0 for empty retrieval now gives NaN after an upgrade. Is that a bug?

No, it is a documented behaviour change in 2.14.0. The information-retrieval ground-truth metrics such as ndcg_at_k and recall_at_k now return nan instead of 0.0 when the result is undefined, for example for an empty relevance list. A zero was misleading, because it said "we scored badly" when the truth was "there was nothing to score". The consequence is for aggregation: a mean that used to include those zeros now has to decide what to do with missing values. I would use an explicit skip-missing mean, count how many rows were dropped, and report that count next to the score so a silent shrinking sample does not flatter the number.

Strong: knows it is intentional, and handles the aggregation consequence. Weak: replaces NaN with zero to restore the old dashboard.

Trade-off questions

SQLite or PostgreSQL?

SQLite is a file with a single writer, which is fine for a laptop, a notebook or a single-process test. PostgreSQL, supported since 2.6.0, is required once more than one process writes, such as replicated services, containers with ephemeral disks, or a shared dashboard. The trade-off is operational: PostgreSQL needs a synchronous driver (psycopg2 or psycopg version 3, never asyncpg), backups, and lockstep version upgrades because of Alembic migrations. I would start on SQLite for exploration and move to PostgreSQL the moment a second writer or reader appears.

Online evaluation, batch evaluation, or both?

They answer different questions. Online evaluation measures real traffic and catches distribution shift, but it costs money per request and cannot compare two versions on the same inputs. Batch evaluation with a fixed dataset gives controlled comparison and a release gate, but only covers the questions you thought of. You want both: a sampled online evaluation for monitoring, and a regression set for gating, fed from production through curate_records_to_dataset so the set reflects real failures.

BatchEvaluator or Run?

BatchEvaluator runs in memory with no instrumentation and persists nothing, so it suits tests, notebooks and scoring outputs from another system. A Run invokes your app over a dataset, persists the results and appears in Compare and Trends, so it suits release decisions and history, at the price of status polling and a database. Choose by whether you need the result to outlive the process.

LLM-judged metric or deterministic check?

A deterministic check such as a schema validation, a citation-marker check or a word count is free, instant, and never flaky, but it only sees what you can express as a rule. An LLM judge handles fuzzy qualities such as relevance and groundedness, but costs money, varies, and needs calibration. The right mix uses deterministic checks for everything a rule can decide and the judge for the remainder, then validates the judge against human labels.

A big judge, a small judge, or a jury?

A larger judge is typically more consistent and more expensive. A small one is cheaper and may be good enough for a narrow criterion. A Jury combines several judges by mean, median, trimmed mean, majority vote or weighted mean, which reduces single-model bias at several times the cost. I would measure agreement with human labels and across models before deciding, and reserve the jury for the metrics where disagreement actually hurts decisions.

Hosted judge or local judge?

Hosted judges are easy and strong but send your data to a provider. Local judges through Ollama, a local Hugging Face model, or a vLLM endpoint keep data inside your network, at the cost of hardware, operations and often lower judging quality. For Gulf and Egyptian employers with residency requirements, the practical middle path is a hosted judge deployed in an approved region, and local judges only for the most sensitive datasets.

Export to an OTLP backend or keep the TruLens database?

OTLP export, available from 2.14.0, is gRPC-only, lets you reuse Grafana, Jaeger, Datadog or Honeycomb, and replaces the database exporter rather than fanning out. You then lose the TruLens dashboard and any database-reading evaluation on that session. Keep the database if evaluation is the goal, use OTLP if the goal is to fold LLM spans into an existing APM, and do not plan on having both from one session.

Synchronous guardrails or asynchronous evaluation?

A guardrail changes the response, so it runs on the request path and adds an LLM call to user-visible latency. Asynchronous evaluation observes without blocking. Use guardrails only for outcomes that must not reach the user, use the cheapest reliable check you can, such as a small classifier, and keep expensive judging in the background. Guardrail metrics must also return a bare float, so the reasons variants do not work.

Use TruLens or another tool?

TruLens is strong when you want metric-centred evaluation, the RAG Triad, agent trace metrics and a built-in dashboard in a Python library. Teams that want a hosted platform with prompt management and collaborative review often look at Langfuse. Teams that want OpenTelemetry-native tracing with an evaluation UI look at Arize Phoenix. Teams that want assertions inside a test runner look at DeepEval or promptfoo. An honest comparison names what each is best at, and notes that TruLens's dashboard has no authentication or tenant model.

What a strong answer covers that a weak one misses

Across these questions, the pattern is consistent. Strong candidates share several habits.

• They debug in pipeline order. They name the stage where data could be lost, rather than jumping to the last stage.
• They know the asynchronous and singleton behaviours. Export is asynchronous, the session is a singleton, and the first configuration wins.
• They treat the judge as an instrument. They ask how it was validated against humans, and keep its model, criteria and temperature fixed across comparisons.
• They quantify cost. They can say what makes a metric expensive and which levers reduce it, and they know the budget's provider limitation.
• They know where data lives. They can list the database, the judge provider, the optional OTLP backend, and the dashboard's lack of authentication.
• They distinguish docs from code. They know some pages describe legacy behaviour, and that old blog posts use trulens_eval, Tru() and Feedback.
• They respect versions. They pin the whole trulens-* family and mention that release cadence is fast.

Weak answers tend to do the opposite: they blame the model first, propose disabling checks, quote a single score as truth, and cannot say where a trace is stored.

Questions to ask back

Mid-level interviews are two-way, and good questions signal experience. Ask which database the team uses for TruLens and whether every service is on the same version. Ask what fraction of traffic is evaluated online and who owns the evaluation budget. Ask whether the judge has ever been compared against human labels. Ask whether there is a regression dataset and how it is refreshed from production failures. Ask how access to the dashboard is controlled. The answers tell you whether the team has an evaluation practice or an evaluation dashboard.

Quick-fire answers

These short answers are worth having memorised for follow-ups.

• What replaced Feedback? Metric, since 2.7.0. Feedback still works as an alias and emits a DeprecationWarning.
• When did OTel become the default? Version 2.3.0.
• How do you run the dashboard from the command line? trulens-dashboard, added in 2.14.0, with --find, --port, --address, --database-url and --force.
• How do you wait for scores in a script? recording.retrieve_feedback_results(timeout=180).
• What do you call before a script exits? session.force_flush().
• Which driver for PostgreSQL? A synchronous one, psycopg2 or psycopg, never asyncpg.
• What does collect_list=False do? Evaluates each matching value separately and aggregates with agg, which defaults to np.mean.
• Which span type makes select_context work? RETRIEVAL, with the RETRIEVED_CONTEXTS attribute.
• Which variable turns tracing off? TRULENS_OTEL_TRACING=0.
• What do conversation metrics need? A conversation_id on the recording and .on_conversation() on the metric, and they are not supported on Snowflake.
• What is the OTLP transport in 2.14.0? gRPC only.
• What happens when the CoT variant is used in a guardrail? It returns a tuple, and the guardrail needs a float.

A worked mini-whiteboard

Sometimes an interviewer asks you to sketch. A good sketch for TruLens has four boxes in a line: the app process containing your code, the instrumentation and the evaluator threads; the database; the dashboard process; and the judge provider as an external box with an arrow from the evaluator. Draw a second arrow from the app process to an optional OTLP collector and label it "replaces the database exporter". Then annotate the failure points on the diagram: the export buffer between process and database that needs force_flush, the singleton at the session that the first configuration wins, the schema version shared by every box that touches the database, and the trust boundary at the judge provider. A candidate who can draw that and explain each annotation has shown the whole mid-level model in two minutes.

Sixty-second self-test

Answer each in a sentence, out loud, without looking.

1. Why can a leaderboard be empty right after a successful call?
2. What does a blank metric column most often mean?
3. Why should groundedness use collect_list=True and context relevance False?
4. What error do you get when an old client meets a migrated database, and how do you prevent it?
5. Why does a thread pool lose the recording, and what replaces it?
6. Which providers enforce cost_budget in 2.14.0?
7. What is the difference between BatchEvaluator and a Run?
8. Why is a two-point score difference not proof?
9. What does OTLP mode do to the dashboard?
10. Where does customer text go when you evaluate?

If you can answer all ten in under a minute, you are ready for the mid-level round. If you stumble on one, reread the matching section of the detailed guide, because each question maps to a mechanism rather than a fact to memorise.

This is the practical companion to the mid-level TruLens guide. The detailed guide explains the mechanisms. This one is the list of things that go wrong at work, what they look like, and what to do about them, followed by the habits that pay off most and a set of recipes you can paste into a real project. Everything is checked against TruLens 2.14.0.

Where this picks up

The beginner tips covered first-contact mistakes: a missing API key, an unset selector key, a forgotten provider package, a dashboard that would not start. Here the mistakes are subtler, because the code runs and produces numbers. The typical mid-level failure is an evaluation that is quietly wrong, quietly expensive, or quietly stored in the wrong place. So this guide is organised as symptom, cause and fix, then habits ranked by leverage, then cards and recipes.

Errors at this level

The leaderboard is empty right after the call

Symptom. You run the app, call session.get_leaderboard(), and see nothing or only some rows.

Cause. Under OTel tracing, export and evaluation are asynchronous. Leaving the with block means your function returned, not that the spans reached the database or that the evaluator finished.

Fix. Wait for the scores explicitly, and flush before exit.

with tru_app as recording:
    app.answer("What is the refund window?")

df = recording.retrieve_feedback_results(timeout=180)
session.force_flush()

Do not reach for wait_for_feedback_results() on the app. Under OTel it raises an error that points you to retrieve_feedback_results.

A metric column is blank

Symptom. The record exists, other metrics have scores, one metric has none, and there is no exception.

Cause. The selector matched no span. The usual reasons are a retrieval function without span_type=RETRIEVAL, a missing RETRIEVED_CONTEXTS attribute mapping, or a selector dictionary key that does not match the implementation's parameter name.

Fix. Open the record in the dashboard and read the attribute names on the spans. Print the signature with help(provider.context_relevance_with_cot_reasons) and compare the parameter names with your selector keys. Then fix the span type or mapping.

Everything landed in a SQLite file you did not expect

Symptom. You configured PostgreSQL, but the dashboard shows nothing, and a default.sqlite has appeared in the working directory.

Cause. TruSession is a singleton and the first configuration wins. Something created it earlier with defaults, often a library imported before your startup code. Your later TruSession(database_url=...) logs TruSession was already initialized. Cannot change database configuration after initialization. and is ignored.

Fix. Configure the session once, in the first module that runs, before importing anything else that touches TruLens. Check the startup banner in every environment.

"Database schema is ahead of the expected revision"

Symptom. A service will not start and names a database schema that is ahead of its own version.

Cause. Another TruLens client migrated the shared database to a newer schema. Your service is on an older version.

Fix. Upgrade this service to the same version. Prevent a recurrence by pinning every trulens-* package in one constraints file and rolling all writers and readers together. The companion errors are "behind" (run TruSession().migrate_database() after a backup) and "reconfigured" (run migrate_database(prior_prefix="<old>"), because database_prefix changed).

"Cannot use feedback_mode other than WITH_APP_THREAD"

Symptom. A ValueError when constructing a recorder.

Cause. Copied code from the legacy docs passes feedback_mode=FeedbackMode.DEFERRED or another mode. Only WITH_APP_THREAD is valid under OTel tracing, the default since 2.3.0.

Fix. Remove the argument. If you wanted evaluation away from the request path, use configure_online_eval, compute_feedbacks(), or a Run.

"Cannot find TruLens context"

Symptom. A warning naming the no-context page, and missing spans for a part of your pipeline.

Cause. An instrumented method ran in another thread, outside the recording's context variable. Plain threads and thread pools do not copy context.

Fix. Use Thread and ThreadPoolExecutor from trulens.core.utils.threading, or TP().submit, or contextvars.copy_context().run. Across processes, record inside the worker process.

"Already recording with a context manager, cannot nest!"

Symptom. A RuntimeError when one recorded function calls another recorded app.

Cause. Recording contexts cannot be nested.

Fix. Use one with tru_app: per top-level call. Nested TruApp record roots are linked correctly since 2.9, so you can instrument an inner app without opening a second context.

A guardrail explodes or passes bad answers

Symptom. Either a type error when the guard runs, or harmful outputs pass.

Cause. Two separate mistakes. A *_with_cot_reasons method returns a tuple, but a guardrail needs a float. And for a metric where high is bad, such as criminality, leaving out higher_is_better=False inverts the direction.

Fix. Use the float-only method, and set direction explicitly.

harm = Metric(implementation=provider.criminality, higher_is_better=False)

@block_output(feedback=harm, threshold=0.9, return_value="I can't help with that.")
def respond(self, question: str) -> str:
    return self.llm(question)

Nothing shows in the dashboard after turning on OTLP export

Symptom. Spans appear in Jaeger or Tempo, but the TruLens dashboard is empty for the new records.

Cause. otel_exporter="otlp" replaces the database exporter and does not fan out. This is by design.

Fix. Decide which destination this session is for. Use OTLP to fold spans into your APM, or the database for evaluation, and do not expect both from one session. Also remember the transport is gRPC only in 2.14.0, so use port 4317.

pip cannot resolve the OTLP extra

Symptom. An installation of trulens[otlp] fails against your application's OpenTelemetry SDK version.

Cause. The gRPC exporter pins a specific SDK range, and your app pins a different one.

Fix. Install a matching exporter and SDK pair, or run TruLens in an isolated environment.

PostgreSQL raises MissingGreenlet

Symptom. A SQLAlchemy MissingGreenlet error when the session starts.

Cause. The URL uses postgresql+asyncpg://, which TruLens does not support.

Fix. Install psycopg2-binary or psycopg and use postgresql+psycopg2:// or postgresql+psycopg://.

A run computes no metrics

Symptom. You call run.compute_metrics(...) and nothing is computed or results are partial.

Cause. run.start() returns before ingestion completes. The status was not yet INVOCATION_COMPLETED.

Fix. Poll run.get_status() first. Treat PARTIALLY_COMPLETED as a warning rather than a pass, because the averages then describe a subset.

Scores jump after an upgrade with no code change

Symptom. The leaderboard shifts the day after a dependency update.

Cause. The judge changed. Provider defaults change between releases, for example the updated defaults for Bedrock, Cortex and LiteLLM in 2.8.1, and a class-default model is whatever the installed version says.

Fix. Pin the judge model explicitly, and record it in the app version or run description.

A deprecation warning floods the logs

Symptom. Feedback is deprecated and will be removed in a future version. Use Metric instead.

Cause. Old code still constructs Feedback, which is now a subclass alias of Metric.

Fix. Migrate to Metric(implementation=..., selectors={...}) and replace isinstance(x, Feedback) checks with isinstance(x, Metric). Also replace the imp= argument with implementation=, the deprecated qs_relevance with relevance, and TruCustomApp with TruApp.

The dashboard is reachable by everyone on the network

Symptom. A security review finds the Streamlit port open to the office network.

Cause. The dashboard has no authentication and binds wherever you told it to.

Fix. Bind to localhost with address="127.0.0.1" or --address 127.0.0.1, and put SSO or a reverse proxy in front for shared use. Remember that everyone who reaches it sees the whole database.

The practices that pay off most, in order of leverage

1. Put the session configuration in one place, first

One startup module creates the session, sets the database, and calls configure_online_eval. Everything else imports from it. This single habit prevents the wrong-database problem, the double-configuration problem, and the surprise evaluation bill, because the three things that decide where data goes and what it costs are all visible in one file.

2. Pin the whole family and the judge

Pin exact versions for every trulens-* package in one constraints file, and pin the judge model with model_engine=. Releases are frequent, the packages move in lockstep, the schema is migrated by whichever client starts first, and judge defaults change. Four problems in this guide become one line in a lock file.

3. Measure the judge before arguing about thresholds

Sample fifty real traces, label them, and run an alignment report. You then know what a score difference is worth. A metric the judge and your experts disagree on cannot be fixed by tuning thresholds.

4. Instrument for the metrics you need

Add span types and attribute mappings only where a selector needs them, and check the span tree in the Records page after every instrumentation change. A correct selector on the wrong span type is the most common silent failure.

5. Sample production, gate with a regression set

Use configure_online_eval with a low sample rate and a throttle for monitoring. Use a thirty-to-fifty question regression set for release gates. Feed the regression set from real failures with curate_records_to_dataset.

6. Prefer deterministic checks wherever a rule decides

A schema validation, a citation marker, a length check and a regular expression are free and stable. Every question you can move from a judge to a rule saves money and removes noise.

7. Put cost and sample rate on the dashboard

The leaderboard shows the observed online-evaluation sample rate and total evaluation cost. Review both with the scores. A number from five percent of traffic deserves different trust from one from all of it.

8. Keep judging and serving failures separate

A judge failure must not become a user failure. Keep LLM-judged metrics off the request path unless they are guardrails, and give guardrails a cheap, float-returning check.

9. Treat traces as data

Decide who can read them, how long they live, and whether they leave the region. Use database_redact_keys=True, leave content capture off unless you need it, and choose a judge deployment in an approved region.

10. Change one thing per experiment

Fix the dataset, judge, temperature and criteria, change one variable, and use the same app name with a new app version. Compare in Compare, and confirm with Trends over time.

Practice cards

Card 1: the empty leaderboard drill

1. Is TRULENS_OTEL_TRACING set to 0 or false? If so, unset it.
2. Read the startup banner. Is it the database you expect?
3. Call session.force_flush(), then query session.get_records_and_feedback(app_name="...").
4. Open a record and read the span attribute names.
5. Check eval_decision_reason for not_sampled, throttled or over_budget.

Card 2: add a new metric to old records

1. Define the Metric with selectors that match the parameter names.
2. Attach it to a new recorder version, or use compute_feedbacks() on the existing app version.
3. Wait with retrieve_feedback_results.
4. Compare the new column across versions in Compare.

Card 3: the judge audit

1. Sample thirty records with GoldenSetGenerator.
2. Label them pass or fail.
3. Produce an AlignmentReport.
4. Repeat with a cheaper judge and run CrossModelAlignment.
5. Write the agreement figure next to the metric in your README.

Card 4: the cost audit

1. List every metric and whether it is per record or per chunk.
2. Count the judge calls for one record.
3. Multiply by daily traffic and the sample rate.
4. Decide whether the provider enforces cost_budget.
5. Change collect_list, the sample rate or the judge, and recompute.

Card 5: upgrade day

1. Back up the database with pg_dump.
2. Update the single constraints file for every trulens-* package.
3. Roll all writers and the dashboard at once.
4. Run TruSession().migrate_database() once.
5. Verify the banner, one record, and one metric.

Operational recipes

Recipe: a safe startup module

import os
from trulens.core import TruSession

def init_session() -> TruSession:
    session = TruSession(
        database_url=os.environ["DATABASE_URL"],
        database_redact_keys=True,
    )
    return session

session = init_session()

Import startup before anything else in the entry point. If DATABASE_URL is missing, the KeyError is a feature: the service fails loudly instead of silently writing to SQLite.

Recipe: pin the family

trulens==2.14.0
trulens-core==2.14.0
trulens-feedback==2.14.0
trulens-dashboard==2.14.0
trulens-otel-semconv==2.14.0
trulens-providers-openai==2.14.0
trulens-apps-langchain==2.14.0

Install with pip install -c constraints.txt -r requirements.txt in every service, in the dashboard image and in CI.

Recipe: check versions before starting

python - <<'PY'
from importlib.metadata import distributions
found = {d.metadata["Name"]: d.version for d in distributions() if d.metadata["Name"].startswith("trulens")}
versions = set(found.values())
print(found)
assert len(versions) == 1, f"trulens packages out of lockstep: {versions}"
PY

Run it in the container entrypoint or CI. A mixed-version family fails before it touches the database.

Recipe: a sampled online evaluation block

session.configure_online_eval(
    sample_rate={"support-bot-prod": 0.05, "support-bot-staging": 1.0},
    throttle=100,
    cost_budget=10.0,
    feedbacks=[context_relevance, groundedness, answer_relevance],
)

Remember that a second call replaces the whole configuration, that apps missing from the dictionary run at 100 percent, and that the budget works only for providers reporting costs.

Recipe: explain why a record was skipped

records, cols = session.get_records_and_feedback(app_name="support-bot-prod", limit=200)
print(records["eval_decision_reason"].value_counts())

The four reasons are evaluated, not_sampled, throttled and over_budget. A high throttled share means the evaluator is behind and the throttle or worker capacity needs attention.

Recipe: evaluate exactly the records a customer complained about

ids = ["rec-123", "rec-456"]
tru_app._evaluator.compute_now(record_ids=ids)
df = tru_app.retrieve_feedback_results(record_ids=ids, timeout=180)

Explicit computation is never gated by the sampling controller. Keep this behind a small helper, since the leading underscore marks an internal attribute.

Recipe: share a read-only dashboard

export TRULENS_DATABASE_URL="postgresql+psycopg://reader:${PGPASSWORD}@db.internal/trulens"
trulens-dashboard --address 127.0.0.1 --port 8502 --force

Put this behind your SSO proxy. Use a database role with read-only privileges for the dashboard if your operations allow it, and keep the dashboard package at the same version as the writers.

Recipe: a local judge for sensitive data

from trulens.providers.openai import OpenAI

provider = OpenAI(
    model_engine="qwen2.5:14b-instruct",
    api_key="not-needed",
    base_url="http://vllm.internal:8000/v1",
)

Replace the model name with whatever your server actually serves. This pattern works for vLLM and other OpenAI-compatible servers, and for Ollama you can also use the trulens-providers-ollama package or LiteLLM(model_engine="ollama/llama3.1:8b", api_base="http://host:11434"). Always compare its scores with a hosted judge on a small set before switching.

Recipe: a CI gate that keeps its evidence

import sys
from trulens.core import BatchEvaluator

scored = BatchEvaluator(metrics=METRICS, max_workers=4).evaluate(rows)
scored.to_csv("eval-results.csv", index=False)

THRESHOLDS = {"Groundedness": 0.80, "Answer Relevance": 0.85}
means = scored[list(THRESHOLDS)].mean()
print(means.to_string())
sys.exit(0 if all(means[name] >= floor for name, floor in THRESHOLDS.items()) else 1)

Upload eval-results.csv as a pipeline artifact, because the BatchEvaluator result is in memory only. Choose thresholds from repeated local runs, below the noise floor.

Recipe: group a multi-hop app

from trulens.core.otel.instrument import span_group

with span_group("hop1"):
    first = retriever.retrieve(question)
with span_group("hop2"):
    second = retriever.retrieve(rewrite(question, first))

Use this only when you need a score per hop, since each group multiplies judge calls.

Recipe: turn production failures into test cases

session.curate_records_to_dataset(
    dataset_name="support_regression",
    records_df=bad_records_df,
    on_error="raise",
)

Curating the same rows twice is idempotent because the IDs are content-addressed. Review the rows for personal data before you commit the dataset anywhere shared.

A weekly evaluation routine

A small team can keep this healthy with a fifteen-minute weekly routine. Open Trends and look at each metric's slope with its confidence interval and at latency and cost. For any bucket that dipped, use Investigate records and read five reasons. Check the sample rate and the eval_decision_reason counts. Pick the five worst production records, curate them, label them, and add them to the regression set. Check the release notes for the TruLens family before anyone upgrades, and schedule the upgrade as a single change. Once a quarter, redo the judge audit, because the judge, the app and the data all move.

Habits to avoid

• Quoting a score without its sample. Say how many records, which judge, which dataset.
• Comparing runs with different judges. The judge change looks like an app change.
• Using default.sqlite as a shared store. It is a local file with one writer.
• Copying code from old blogs. Anything with trulens_eval, Tru(), Feedback(...).on_input_output() or trajectory_* names predates the current API.
• Turning on content capture casually. Prompts and responses in traces are customer data.
• Learning about the budget from the invoice. Confirm the provider enforces it.
• Ignoring NaN. From 2.14.0, undefined IR metrics return nan, so decide how to aggregate them.

Checklist

Before you call an evaluation setup production-ready, check the following.

• The session is configured once, in the first module, from an environment variable.
• The database is PostgreSQL with a synchronous driver, or Snowflake, and not a shared SQLite file.
• The startup banner is checked in every environment.
• Every trulens-* package is pinned to one version, and so is the judge model.
• Retrieval spans have span_type=RETRIEVAL and the context attribute mapping.
• Selector keys match the implementation's parameter names.
• Scripts wait with retrieve_feedback_results and end with force_flush().
• Threaded code uses the TruLens thread classes.
• Online evaluation has a sample rate, a throttle and a known budget behaviour.
• Guardrails use float-only metrics with the right higher_is_better.
• A regression set exists, fed from production failures, and CI gates on a margin.
• The judge has been compared with human labels for the metrics that drive decisions.
• The dashboard is bound to localhost behind SSO, at the same version as the writers.
• Content capture is off unless there is a documented reason.
• You know where every copy of customer text goes, including the judge provider's region.

For the next level, where the questions become scale, multi-team isolation and governance, continue with the senior guide. If you are still deciding between tools, compare with Langfuse, Arize Phoenix and DeepEval.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, and adds the work you own when TruLens is not your tool but a platform that other teams depend on to decide whether their LLM applications are good enough to ship.

Up to this point the questions have been about making evaluation work for one application. From here they change: what happens when forty replicas write traces into one database, who is allowed to see the prompts inside those traces, what a judge model costs you in a month when nobody is watching, how you upgrade a library that ships a minor release every few weeks, and how you explain to a product lead why a score moved. This guide is written against TruLens 2.14.0, the current release at the time of writing. Version 2.15.0 has a bump commit on the main branch but is not on PyPI, so nothing in this guide depends on it.

Where this picks up

I start with the architecture, because nearly every production problem with TruLens is a consequence of one fact that is easy to miss: there is no TruLens server.

Architecture and what runs where

TruLens is a library. When you import it and construct a TruSession, your application process gains an OpenTelemetry tracer, an exporter that writes spans to storage, and an evaluator that runs on threads inside that same process. There is no collector service to deploy, no ingestion API to scale, and no control plane to secure. That is a large part of why it is pleasant to adopt, and it is also the source of the properties a platform owner must plan around.

Follow one request through the system. A user calls your application. The instrumented top-level method opens a RECORD_ROOT span, and each retrieval, generation or tool call beneath it becomes a child span with a span type and ai.observability.* attributes. When the record closes, the span processor hands the spans to the exporter, which writes them to the database. The exporter is asynchronous, which is why a script that exits right after the recording block can lose its last spans unless it calls session.force_flush().

The evaluator then notices the new record, resolves each metric's selectors against the spans, calls the judge provider, and writes the scores back as EVAL_ROOT and EVAL spans under the same trace. Under OTel tracing, which has been the default since 2.3.0, the recorder's feedback mode is fixed at WITH_APP_THREAD. Passing any other mode raises a ValueError. The older documentation page that lists four modes and a separate start_evaluator describes the legacy path, and the code wins over that page.

The dashboard is the third part. run_dashboard launches streamlit run as a subprocess that reads the same database the application writes to. It is a reader, it holds no state of its own, and it can be started and stopped without touching the applications. It also has no login, which is the subject of a later section.

Three architectural consequences follow and you should be able to state them without looking them up.

First, evaluation competes with serving for the same process resources. The evaluator threads run beside your request handlers, so a slow or rate-limited judge can consume threads, memory and outbound connections in the process that is supposed to be serving users. Second, there is no central rate limiter. Each replica samples, throttles and budgets on its own, so a fleet-wide limit is something you must arrange through configuration rather than something the library enforces globally. Third, there is no central version authority. Every client checks the schema revision itself, and a mismatch is discovered by whichever process starts first.

The failure modes of the pipeline

An observability library is unusual in that its failures are often silent. A broken web endpoint returns an error. A broken evaluation pipeline returns a dashboard that looks healthy and contains no new data, or contains data that is subtly wrong. Learn the failure modes in the order you are likely to meet them.

Spans never arrive. The causes are a short list. OTel tracing was disabled with TRULENS_OTEL_TRACING=0 (since 2.14 TruLens logs a warning once per process when that is explicit, so search the logs for it). The process exited before the exporter flushed. The application was started with OTLP export configured, and the spans went to a collector rather than the database, so the dashboard shows nothing by design. Or the instrumented method ran in a thread or asyncio task that does not carry the recording context, and TruLens printed Cannot find TruLens context.

Spans arrive but metrics are never computed. This is the classic selector failure. A metric with Selector.select_context() looks for spans of type RETRIEVAL carrying the RETRIEVED_CONTEXTS attribute. If the retrieval method was instrumented without that type or attribute, the selector matches nothing, and the metric has no input to evaluate. There is no loud error, because from the evaluator's point of view nothing matched. This is the single most common support request a platform team receives, and the cure is a pre-merge check that records one trace in CI and asserts that every metric produced a score.

Metrics are computed but the score is wrong. The judge saw the wrong data, because collect_list was set to the wrong value and a per-chunk metric received one concatenated list, or the selector key did not match the implementation's parameter name, or the judge model was changed underneath you. A default judge model is a moving target: OpenAI defaults to gpt-4o-mini, Anthropic to claude-sonnet-4-6, Google to gemini-2.5-flash, and those defaults belong to the library version. Upgrading TruLens without pinning the judge can change every score you have without changing a line of your application.

The evaluator falls behind. Each metric costs at least one judge call per record, or one per chunk when collect_list=False. If records arrive faster than the judge can answer, evaluations queue. The symptom is a leaderboard whose scores lag the traffic by minutes or hours. The fix is not more threads. It is a lower sample rate, a throttle, a cheaper judge, or the decision that this metric does not need to run online.

The judge fails. Judge providers rate-limit, time out and return malformed output. A failed evaluation leaves a record without a score, and your aggregates silently exclude it. If a provider outage removes forty percent of your scores, the mean of the remainder may look perfectly normal. Always report the count of scored records next to the mean.

The schema drifts. Two classes of clients talking to one database will eventually disagree. A client older than the database raises Database schema is ahead of the expected revision. A client newer than the database raises the behind error and asks you to call migrate_database(). Either way the process that hit it does not start, and in a fleet that means a rollout stalls on the first pod.

Build one synthetic canary. A scheduled job records a fixed question through a dedicated app name, waits with retrieve_feedback_results(timeout=180), and checks that every expected metric column is present and numeric. When the canary fails, you learn about a broken judge key, a schema mismatch or a selector regression before a team asks why their leaderboard is empty.

Scaling: which component breaks first

Ask of any system, in order, what saturates first. For TruLens the honest ranking is the judge, then the database, then the dashboard, and only rarely the instrumentation.

The judge is the real bottleneck and the real bill. Span creation is cheap. A metric is a network call to a language model. A RAG application with the triad of three metrics and ten retrieved chunks evaluated per chunk makes roughly thirty-three judge calls for every user request, and each call has latency, a rate limit and a price. At ten requests per second that is over three hundred judge calls per second, which no provider account will grant you without a conversation. Everything in the next section, about sampling and budgets, exists because of this ratio.

The database comes next. SQLite is a single file with a single writer and belongs on a laptop. The moment you have two processes, two containers or two teams, move to PostgreSQL, which TruLens has supported since 2.6, or to Snowflake. On Postgres, three things matter. Use a synchronous driver such as psycopg2-binary or psycopg version 3, because an asyncpg URL produces a MissingGreenlet error. Put a connection pooler in front if you run many replicas, since each replica holds its own engine and the connection count multiplies with your replica count. And watch table growth, because every span is a row and a busy application generates many spans per request.

The table growth deserves numbers. Suppose a record produces twelve spans of application work and another six spans per metric for three metrics. You are at thirty spans for one user request. At a modest million requests a month, that is thirty million span rows, each holding attribute payloads that include prompts and retrieved documents. The database becomes a content store, and its size is dominated by payload text. Plan retention from the start: a scheduled job that deletes records older than your policy, or partitions you can drop, is far cheaper than discovering a half-terabyte table.

-- Illustrative only: table and column names depend on your database_prefix
-- and schema revision. Inspect the real schema first, take a backup, and
-- test on a copy. Do not run this against production unchanged.
-- \dt trulens_*
-- \d trulens_events

I am deliberately showing a sketch rather than a deletion statement. TruLens manages its own schema through Alembic, the table layout has changed between releases, and a retention job that hard-codes column names will break on the next migration. Inspect the schema at your exact version, and test the job on a restored copy of the database.

The leaderboard query used to be the dashboard's weak point. Since 2.8 the aggregation happens in SQL rather than by pulling every record into Python, and get_records_and_feedback accepts limit and offset. Use them in any script that reads history. The source carries a comment that compute_feedbacks() fetches all events for an app version at once, so on a large history pass events= explicitly, or use Runs, rather than recomputing a year of traffic in one call.

The dashboard scales with its viewers, not with your traffic. It is one Streamlit process. A handful of engineers are fine. A company-wide rollout where two hundred people open the Trends page will make it slow, and the answer is to run several dashboard instances behind a proxy, all reading the same database, or to export aggregates to a tool built for broad access.

Offline batch evaluation has its own ceiling. BatchEvaluator(metrics=[...], max_workers=N) runs in memory, with nothing persisted and no instrumentation, and it is the right tool for a nightly regression over a few thousand rows. It is bounded by judge throughput and by memory, since the DataFrame lives in the process. For persisted, resumable work use a Run, and remember that the server-side computation path is a Snowflake feature.

Controlling the cost of online evaluation

configure_online_eval, added in 2.11, is the platform owner's main control. It turns evaluation from "every metric on every record" into a policy you can state in one line and review.

from trulens.core import TruSession

session = TruSession()

session.configure_online_eval(
    sample_rate={"prod_rag": 0.05, "staging_rag": 1.0},
    throttle=100,
    cost_budget=10.0,
)

Read each argument as a separate promise. sample_rate accepts a single float or a dictionary keyed by app name, and any app missing from the dictionary is evaluated at 100 percent, which is a trap worth remembering: adding a new application does not inherit the conservative default, it inherits full evaluation. throttle caps the evaluations per period. cost_budget is a daily ceiling in US dollars that resets at UTC midnight. A second call replaces the whole configuration rather than merging with it, so keep the policy in one place in code and never spread it across modules.

Sampling is deterministic. TruLens hashes the record identifier with the app name using SHA-256, so the same record always gets the same decision and a replica restart does not reshuffle which traffic is evaluated. That matters for reproducibility, and it means the sampled fraction is stable enough to reason about. The decision is recorded: the results DataFrame gains sampled, sample_rate and eval_decision_reason columns, and the reason is one of evaluated, not_sampled, throttled or over_budget. The leaderboard shows the observed sample rate, so a reader can see that a score is an estimate from five percent of traffic rather than a census.

Now the part that catches teams. The budget is only enforceable for providers that report their costs. In 2.14.0 the reports_costs flag is true for OpenAI, LiteLLM, Google and Cortex. Anthropic tracks per-model cost but does not override that flag, so a cost_budget combined with the Anthropic provider is not enforced, and the library logs a warning at configuration time. If your judge is Anthropic, your real controls are the sample rate and the throttle, and you should alert on spend from the provider's own billing instead of trusting the budget.

Explicit calls are never gated. compute_now() and compute_metrics() ignore the sample rate, the throttle and the budget. That is a feature, and it gives you a backfill path. When an incident makes you want scores for the traffic you skipped, you can request exactly those records.

# record_ids gathered from the results DataFrame where
# eval_decision_reason is "not_sampled" for the window in question
app._evaluator.compute_now(record_ids=skipped_ids)

Note that this uses an attribute with a leading underscore, which signals that it is not a documented stable interface. Wrap it in one function in your platform code so that a future change has one place to fix.

The other cost levers are structural. Use collect_list=True so that context relevance runs once over the list rather than once per chunk, when the judge can handle the combined context and you accept a coarser signal. Choose a cheaper judge for high-volume metrics and keep the strong judge for the few metrics that gate releases. Use Jury only where the decision justifies several model calls. Use trace compression through enable_trace_compression on metrics that read large traces. And keep the expensive agent metrics, the trace-level GPA family, for offline runs rather than every production request.

Judge quality as a governed asset

A score from an LLM judge is a measurement, and a measurement has an instrument. If the instrument drifts, every number downstream drifts with it, and nobody will notice, because the application did not change. A platform team therefore owns the judges the way a metrology lab owns its scales.

Start with the simple, unglamorous rule: pin the judge explicitly. Write OpenAI(model_engine="gpt-5-nano") or the equivalent, never rely on the class default, and treat a judge change as a versioned event. When you change the judge, old and new scores are not comparable, so record the change in the app version or in metadata and draw a line on the trend chart.

Then measure the judge against people. The 2.9 and 2.10 releases added tooling for this, and it is the most valuable part of the library for a platform owner.

The workflow is a loop. Sample production records with the generator, because stratified or uncertainty sampling finds the hard cases that random sampling misses. Have two humans label them, and measure the agreement between those humans first, since a judge cannot be more reliable than the people it is checked against. Run the alignment report to see where the judge disagrees. Then change one thing at a time: the criteria text, the few-shot examples, the model. A cheaper model that agrees well enough is pure savings, and CrossModelAlignment exists to show that without a leap of faith.

Two subtleties deserve attention. First, the default score scale runs from 0 to 3 and is normalised to 0 to 1, with temperature 0.0, so scores are quantised into a few levels. A judge that returns the same two values for everything gives a flat leaderboard that cannot discriminate between versions, and the distribution analyzer will show that. Second, judges have biases. A judge that favours longer answers, or answers from its own model family, will reward the wrong thing. The defence is human-labelled data, not intuition.

Data residency enters here. Every judge call sends the prompt, the retrieved context and the answer to the judge provider. For a Gulf bank or an Egyptian government workload where that data may not leave a region, you have three routes: a provider deployed in an approved region, a local judge through Ollama, HuggingfaceLocal or a vLLM server addressed with OpenAI(..., base_url="http://localhost:8000/v1"), or a Snowflake deployment with Cortex running inside your account. Decide this before adoption, because retrofitting a judge after teams have built dashboards on scores from another model resets every comparison.

Security: the trust model

Write down who trusts what before you write a single control. For TruLens the trust model has four boundaries, and each has a default you must consciously change.

The dashboard has no authentication. It is a Streamlit application. Anyone who can reach its port sees every record in the database, including prompts, retrieved documents and model outputs, and can browse every app version. The default run_dashboard call picks an unused port and, unless you pass address, does not restrict who can connect in any way TruLens enforces. Bind it with address="127.0.0.1" or --address 127.0.0.1, and publish it only behind an identity-aware proxy or an SSO-fronted ingress. The address argument cannot be used in Colab, which raises an error.

Traces contain whatever your users typed. A support chatbot records customer names, account numbers and complaints, and a RAG system records the confidential documents it retrieved. The database is therefore as sensitive as the most sensitive prompt that ever passes through, and its access control, encryption at rest and backup handling must match. Set database_redact_keys=True, which the documentation recommends for production, so that API keys held in objects the library serialises are redacted. That option protects keys, not personal data: it does not scrub a user's name from a prompt. If you need prompt scrubbing, do it in your application before the text reaches an instrumented method.

Content capture is opt-in in the newer paths. Since 2.13, capturing GenAI prompt and response content as OTel span events is off by default, enabled with TRULENS_OTEL_CAPTURE_CONTENT=true or the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Leave it off in any environment exporting to a third-party backend until a review says otherwise. The coding-agent hooks follow the same philosophy: by default they capture lifecycle metadata only, and content, tool payloads, diffs and file paths are each separate opt-ins, bounded by TRULENS_MAX_FIELD_BYTES, which defaults to 16384. The hooks fail open, so they never block the coding client, which is the right default for developer tools and the wrong one if you were counting on them as an audit trail.

The judge provider is a data recipient. Covered above, but restate it in the threat model: the judge sees what the application saw.

Secrets

Provider keys come from the environment or a .env file in the working directory or a parent, read through python-dotenv. The second source is where accidents happen. A .env committed to a repository, or baked into a container image layer, publishes the key to everyone with pull access. In production inject keys from a secret store as environment variables at runtime, and keep .env for laptops and out of version control.

For the database, never put the password in source. Read DATABASE_URL from a secret store and construct the engine in code, as in the example below. TruLens redacts the password in its startup banner, printing postgresql://trulensuser:***@localhost:5432/trulens, but redaction of the banner does nothing for your shell history or CI logs, so avoid passing the URL on a command line.

import os
import sqlalchemy
from trulens.core import TruSession

engine = sqlalchemy.create_engine(
    os.environ["TRULENS_DB_URL"],          # injected from a secret store
    connect_args={"sslmode": "require"},
    pool_pre_ping=True,
)
session = TruSession(database_engine=engine, database_redact_keys=True)

Give the application's database role only what it needs. The writer role needs to insert and read; only a migration job needs to alter tables. Splitting them means a compromised application cannot drop the schema, and it forces the migration step to be explicit. For Snowflake, prefer key-pair authentication, OAuth or SSO over passwords, and remember that ingestion tasks need CREATE TASK on the schema and EXECUTE TASK on the account.

The supply chain

The install surface is wide. pip install trulens pulls in trulens-core, trulens-feedback, trulens-dashboard[full] and trulens-otel-semconv, plus the trulens_eval compatibility shim until the deprecation period ends. Provider and framework packages are separate. The project releases frequently: versions 2.9 through 2.14 shipped between late July and early September 2026. Fast releases are good for features and demanding for dependency review.

Pin exact versions of every trulens-* package, use hashes in a lock file, and mirror packages through an internal index so a compromised upload cannot reach a build before you have reviewed it. Keep all the packages on the same version, since they release in lockstep, and scan the lock file like any other. The HuggingFace provider imports torch and transformers lazily since 2.13.1, which reduces the footprint when you do not use it, and the otlp extra pins an OpenTelemetry exporter whose SDK requirement can conflict with your application's existing pin, which is both an operational and a supply-chain decision. Isolate the environment if the pins disagree.

The license is MIT, and Snowflake maintains the project after acquiring TruEra. For a procurement review, note that the code is open, the repository is github.com/truera/trulens, and there is no hosted TruLens service involved when you run it yourself.

Multi-tenancy without a tenant model

TruLens has no concept of a tenant, a workspace or a role. There is no built-in RBAC. Every dashboard user sees the whole database. If your platform serves five teams with different confidentiality requirements, you must build the isolation from the parts the library does give you.

There are four real options, and they sit on a spectrum of isolation against operating cost.

The prefix approach deserves care. The connector accepts database_prefix, default trulens_, and changing it after data exists produces the Database has been reconfigured error until you run migrate_database(prior_prefix="..."). Do not treat the prefix as a label you can rename casually. Decide it once, at onboarding.

For sensitive teams I recommend a database per team with its own role, its own dashboard instance bound to localhost behind the proxy, and its own retention schedule. It costs more to run and gives you an isolation story that a security reviewer accepts without a long conversation. The operating cost is migration choreography, since every team's database needs the same upgrade, and you handle that with automation rather than by hand.

There is a second form of multi-tenancy, about shared metrics. If fifteen teams each define their own "groundedness", the numbers mean different things and cannot be compared. Publish a platform package of vetted metric definitions, each with a pinned judge, a documented selector mapping, a stated scale and a link to its alignment report, and let teams import them. The value of a platform is that "0.82 groundedness" means the same thing on two dashboards.

The app_name and app_version pair is the unit of comparison in the leaderboard, Trends and Compare views. Standardise what goes in them. A version string of "final2" tells the next person nothing; a version derived from the git commit or the model and prompt identifiers does. The Compare page is only as good as the version discipline behind it.

Upgrades and migrations

The release cadence makes upgrades a standing activity rather than an occasional project. Between 2.0 and 2.14 the project added PostgreSQL, replaced Feedback with Metric, deprecated the Snowflake dashboard path, and introduced OTLP export. Plan for that cadence.

The schema is the coupling

TruLens manages the schema with Alembic. A client that starts against a database at a different revision raises one of three errors: behind, ahead, or reconfigured. The operational rule follows directly. Upgrade writers and readers together, and take a backup first.

In a fleet this has a specific choreography.

1. Announce the window and freeze deploys of applications that embed TruLens.
2. Back up the database.
3. Run a single migration job with the new version and the privileged role: TruSession(database_url=...).migrate_database().
4. Roll the applications to the new version. Until they roll, the old clients will raise the ahead error when they next initialise, so restarts of old pods during this window will fail.
5. Upgrade the dashboard last, or alongside.
6. Run the canary and compare the record counts.

The awkward part is step four. A rolling deploy briefly mixes old and new clients. Old pods that are already running keep working against the old in-memory assumption, but any pod that restarts in the window will hit the error. Keep the window short, or use a maintenance flag that drains traffic.

"""Run once per upgrade, with the migration-capable database role."""
import os
from trulens.core import TruSession

session = TruSession(database_url=os.environ["TRULENS_DB_URL"])
session.migrate_database()
print("migrated")

Do not set database_check_revision=False to get around the error. It skips the check and lets a client write against a schema it does not understand, which converts a loud, safe failure into quiet corruption.

The API migrations

Feedback to Metric is the one every fleet has. Feedback still works, but it is now a subclass alias of Metric and emits a DeprecationWarning whose text begins Feedback is deprecated and will be removed in a future version. The official guidance is to replace isinstance(x, Feedback) checks with isinstance(x, Metric), because Feedback will be removed in a future release. MetricConfig and SnowflakeFeedback are deprecated as well. The imp= argument is deprecated in favour of implementation=. The chained .on_input(), .on_output() and .aggregate() methods still work, but new code should use a selectors dictionary and agg=.

Do this as a measured migration. Turn deprecation warnings into test failures in CI for the platform package first, because that is where the warnings originate for most teams. Then provide a codemod or a documented before-and-after, and give teams a date. Until they migrate, run with -W error::DeprecationWarning in your own pipeline only, to avoid breaking their builds while still finding your own leftovers.

trulens_eval to trulens is older, and the trulens_eval package on PyPI is only a shim now. If any service still imports it, there is a Grit pattern, grit apply trulens_eval_migration, that automates the move. Never teach or approve new code that imports from trulens_eval.

Dashboard launching changed. TruSession().run_dashboard() and its siblings are deprecated aliases. The current form is from trulens.dashboard import run_dashboard and run_dashboard(session), or the trulens-dashboard CLI from 2.14. The SiS dashboard path, run_dashboard_sis and sis_compatibility_mode=True, is deprecated, and as of 2.7.2 the dashboard refuses to start on a SnowflakeConnector with a message that points to the Evaluations UI in Snowsight.

Legacy recording is unavailable under OTel. update_record, with_record, deferred feedback modes and wait_for_feedback_results either raise or are deprecated. The replacement for waiting is retrieve_feedback_results. Any test that relied on synchronous evaluation after a recording needs the timeout-based retrieval.

Behavioural changes to watch for

Two changes altered numbers without altering code. In 2.14.0 the information-retrieval metrics, ndcg_at_k, recall_at_k and their relatives, return nan instead of 0.0 when the result is undefined, for example for an empty relevance list. Any dashboard or gate that averaged those will change, and nan will propagate through a naive mean. And in 2.2.0 the agent evaluations dropped the trajectory_ prefix and adopted the GPA names, so older blog posts and older code use names that no longer match.

Exporting to an OpenTelemetry backend

If your organisation already runs Tempo, Jaeger, Datadog or Honeycomb, the natural question is whether TruLens traces should flow there. From 2.14 they can, and the trade-off is sharper than the feature suggests.

from trulens.core import TruSession

session = TruSession(
    otel_exporter="otlp",
    otlp_endpoint="http://otel-collector.observability:4317",
)

Install the extra first, with pip install "trulens[otlp]". Then keep four facts in mind.

The transport is gRPC only in 2.14.0. Point at port 4317, not the HTTP port 4318. An http/protobuf exporter has been merged to the main branch but is not in the released version, so if your collector only accepts HTTP you cannot use this yet. The value of otel_exporter must be None or "otlp", and any other string raises a ValueError.

OTLP mode replaces the database exporter; it does not fan out. This is the decision. With OTLP on, the spans go to the collector and do not go to the TruLens database. The dashboard will show nothing, and any evaluation that reads from the database will find nothing to evaluate. You cannot have both from one session. If you want online evaluation and the TruLens dashboard, stay on the database. If you want TruLens spans in your existing tracing backend and do your quality analysis elsewhere, use OTLP. Some teams run two separate processes or environments for the two purposes.

It exports metrics too. The GenAI metrics gen_ai.client.token.usage, with a gen_ai.token.type attribute of input or output, and gen_ai.client.operation.duration in seconds, flow alongside the spans. Streaming calls additionally report time to first token and throughput as of 2.14. These are standard OTel semantic conventions, so your existing dashboards and alert rules can consume them.

Dependency pins bite. The otlp extra brings opentelemetry-exporter-otlp-proto-grpc, and the lock file pins 1.44.0, which needs opentelemetry-sdk at 1.44.x. If your application already pins a different SDK, the pip resolver fails. Install a matching exporter and SDK pair, or isolate the environment.

If endpoint and headers are omitted in code, the standard OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS variables apply, which is how you should supply a collector token: through the environment from a secret store, not in source.

There is also the question of what belongs in the spans. Prompt and response content is off by default in the span events, and I recommend leaving it off when exporting to a third-party SaaS backend, because your tracing vendor then becomes another recipient of user data.

TruLens can also consume spans that other instrumentation produced, such as those from opentelemetry-instrumentation-openai, and use their attributes in selectors, which lets you evaluate without rewriting instrumentation. The spans carry both the ai.observability.* namespace and, since 2.9, dual-emitted gen_ai.* attributes, with correct span kinds since 2.10.

Governance: datasets, prompts and release gates

Evaluation only governs anything if it can say no. The platform question is how a score becomes a decision, and who is allowed to change the threshold.

Regression sets come from production. TruLens 2.14 added session.curate_records_to_dataset(...), which turns selected records into a named dataset with optional mapping, an expected-response function and provenance. Ground-truth identifiers are content-addressed, so curating the same rows twice is idempotent, which means a nightly job can run without creating duplicates. The loop this enables is the heart of a mature evaluation practice: users hit a failure in production, a person flags the record, the record is curated into the regression set with the correct expected answer, and every future release runs against it. That converts a one-time incident into a permanent test.

Prompts get versions and labels. The session now has create_prompt, create_prompt_version, set_prompt_label, get_prompt, get_prompts, get_prompt_versions, get_prompt_labels and get_prompt_label_history. A label such as production pointing at a version gives you rollback and an audit trail of who moved it. Use it if you have no other prompt registry. If you already use another system, such as the Langfuse prompt store covered in the Langfuse guide, do not run two sources of truth.

Release gates run offline. A gate compares the candidate app version with the baseline on the frozen regression set. Use the BatchEvaluator for a quick in-memory check, or a Run when you need persistence. Official examples exist for GitHub Actions and Azure Pipelines, and a pre-commit smoke test was added in 2.9. A gate should check three things together: the mean score against a floor, the score against the baseline with a tolerance, and the count of scored rows, because a gate that passes on an empty result is worse than no gate.

import sys
from trulens.core import BatchEvaluator

FLOORS = {"Groundedness": 0.80, "Answer Relevance": 0.85}
MIN_ROWS = 100

def gate(df, metrics):
    results = BatchEvaluator(metrics=metrics, max_workers=8).evaluate(df)
    if len(results) < MIN_ROWS:
        sys.exit(f"only {len(results)} rows scored, need {MIN_ROWS}")
    for name, floor in FLOORS.items():
        scored = results[name].dropna()
        if len(scored) < 0.95 * len(results):
            sys.exit(f"{name}: too many unscored rows")
        if scored.mean() < floor:
            sys.exit(f"{name}: {scored.mean():.3f} below {floor}")
    print("gate passed")

The Trends page, added in 2.13, supports the monitoring half of governance. It plots metrics with 95 percent confidence intervals, latency at average, P90 and P99, app cost and evaluation cost, in daily or weekly buckets, and its "Investigate records" action takes you from a dip straight to the responsible records. Use the intervals. A drop from 0.86 to 0.84 on two hundred sampled records is noise, and the interval says so; a platform owner who reacts to every wiggle trains teams to ignore the dashboard.

Finally, write down the ownership rules, because governance is mostly agreements. Who may change a threshold? Who approves a new judge? Who can delete a dataset? Keep thresholds in version-controlled files reviewed by the owning team and by the platform team, never in a notebook.

Incident playbooks

Write these before you need them. Each follows the same shape: the symptom, the first three checks in order, and the fix.

Playbook one: the leaderboard is empty or stale

1. Is the application writing at all? Check the database for recent rows, and check the application logs for the one-time warning that OTel tracing is disabled.
2. Is the exporter pointed where you think? If OTLP mode is on, the database is not being written.
3. Did the process flush? Short-lived jobs must call session.force_flush() before exit.

Fix by correcting the flag or the exporter, and add the canary so this pages someone next time.

Playbook two: scores collapsed to zero or vanished

1. Did a deploy change a selector, a span attribute, or the retrieval method's instrumentation? Diff the instrumentation, not only the metrics.
2. Did the judge change? Compare the provider and model on the metric against the last known good version, and check whether the library upgrade changed the default.
3. Is the judge failing? Query the evaluated-to-ingested ratio and the eval_decision_reason values. A surge in over_budget or throttled means policy, not a bug.

Fix by pinning the judge, restoring the instrumentation, and backfilling with compute_now for the affected window.

Playbook three: the database is full or slow

1. Which tables hold the bytes? Span payloads dominate, so the largest table is the events table.
2. Is a retention job running, and did a migration change a table name under it?
3. Are dashboards running unbounded queries? Streamlit sessions left open on the Records page can be heavy.

Fix by deleting or partitioning by age, not by truncating blindly. Take a backup, test the job on a copy, and never delete rows that evaluations still reference.

Playbook four: the fleet will not start after an upgrade

The error text tells you which case you are in.

The ahead case is the dangerous one. There is no supported downgrade of a migrated schema, so the rollback path for a bad release is to restore the database from the backup you took, and to accept losing the records written since. That is why the backup step is not optional, and why you should rehearse the restore.

Playbook five: a cost spike

Check the provider's billing first, since that is the authoritative number. Then check whether a new app was added without an entry in the sample_rate dictionary, which defaults to full evaluation. Check whether a deploy flipped collect_list to False on a metric, multiplying calls by the number of chunks. Check whether a retrieval change raised the number of chunks per query. Fix by lowering the rate, restoring the selector, and, if the provider does not report costs, by putting a hard cap at the account level.

Playbook six: traces containing data they should not

Stop the export first, by disabling content capture or pointing the exporter away. Identify the window and the apps. Then remove the data from the database, which is a destructive operation: take a backup, delete by record identifier, and record the deletion in your incident log. Afterwards add scrubbing in the application before instrumentation, and review who had dashboard access during the window, since the dashboard itself has no access log.

Where TruLens stops

An honest platform owner knows what the tool is not. TruLens is an evaluation and tracing library with a dashboard. It stops in several places.

It is not a general production observability platform. It has no alerting engine, no on-call integration, no service map, and its dashboard is single-purpose. For operational monitoring of latency, errors and saturation, use your existing stack and let TruLens feed it through OTLP. The two concerns are different: is the service up, and is the answer good.

It is not a multi-tenant SaaS. There is no user model, no RBAC and no per-tenant quota. If you need strict isolation with self-service for many teams, you will assemble it from databases, proxies and conventions, or you will pick a tool that has those features built in. Langfuse is the natural comparison: it runs as a server with projects and access control, whereas TruLens runs inside your process.

It is not a prompt-management or experimentation product. The 2.14 prompt functions give you versions and labels, not a review workflow, approvals or a playground. If prompt workflow is the main need, use a tool built around it.

Its judge-based metrics are estimates. A score from a language model has bias and variance, and no amount of dashboard polish changes that. If the decision is high-stakes, such as a medical or legal answer, you need human review in the loop, and the metrics are a triage aid rather than a verdict.

The Snowflake experience is split. With a SnowflakeConnector the OSS dashboard refuses to start and you use Snowflake AI Observability in Snowsight, which has no Trends view and does not support conversations. If Snowflake is your platform, evaluate that product on its merits rather than assuming parity with the OSS one.

Some metric families are not its strength. For retrieval evaluation with rich ground-truth datasets and synthetic test-set generation, compare RAGAS. For unit-test-style evaluation in pytest, compare DeepEval. For red-teaming and prompt regression in CI, compare Promptfoo. TruLens's distinguishing strengths are the RAG triad, the agent GPA metrics, the span-level selector model, and judge alignment tooling, and you should choose it for those.

Choose something else when any of these is the binding constraint: a hosted multi-tenant service with access control, alerting and paging, a pytest-first workflow, or a need for guardrail-grade latency with no judge call at all. Runtime guardrails are supported through block_input, block_output and context_filter, but each one adds a synchronous judge call to the request path and requires a float-only metric, so a *_with_cot_reasons variant will not work there. For hard blocking policies, use a dedicated guardrail system with deterministic rules and keep TruLens for measurement.

Running it as a platform for other teams

The technical material above is necessary and not sufficient. A platform succeeds when other teams adopt it without you in the room. Five habits do most of the work.

Ship a paved road. A template repository with a correctly instrumented sample app, the platform metrics package, a CI gate and a canary. A team that starts from it gets correct instrumentation for free, and a team that does not will invent twelve incompatible ways to record a retrieval span.

Write the contract, not just the code. State what the platform guarantees and what it does not. A reasonable contract: scores are estimates from a pinned judge; traces are retained for a stated number of days; the dashboard is read-only and is not a system of record; thresholds are owned by the application team and reviewed by the platform team; upgrades happen on a published schedule with a named window.

Make the cost visible to the teams that cause it. Break evaluation cost down by app name and show each team its own number, from the leaderboard cost columns and the Trends page. A team that can see that their 100 percent sample rate costs more than their inference will change it without being asked.

Hold an upgrade cadence. Adopt a rule such as "within two minor versions of current, exact pins, staging first". It stops the fleet drifting across a dozen versions, which makes the single shared schema impossible to migrate.

Keep the on-ramp short and the off-ramp possible. Because TruLens is open source and emits OpenTelemetry spans, teams can leave. Do not build your platform so that leaving is impossible. Keep datasets exportable, keep metric definitions in plain Python, and keep prompts in a system you can move. A platform people cannot leave is one they will resent.

Finally, publish a small number of numbers about the platform itself. The number of apps onboarded, the evaluated-to-ingested ratio, the median time from a flagged production failure to a regression test, the monthly judge spend, and the age of the oldest pinned version. Those tell you whether the platform is healthy, and they give you something to show leadership that is about outcomes rather than activity.

The review checklist

Use this for any design document or pull request that adds or changes TruLens usage. It is a list of questions, because each one is a place where a past incident started.

Instrumentation

• Is every retrieval method typed as RETRIEVAL with the query and retrieved-context attributes mapped?
• Is the top-level method the RECORD_ROOT with input and output mapped?
• Do threads and asyncio tasks carry the recording context, using TruLens's threading helpers or contextvars.copy_context?
• Does a CI test record one trace and assert every metric returned a number?

Metrics and judges

• Is the judge model pinned explicitly, and is that version recorded?
• Do the selector keys match the implementation's parameter names?
• Is collect_list set deliberately for each metric?
• Is there a human-labelled golden set and an alignment measurement for this judge?
• Are guardrail metrics float-only?

Cost

• Is there a per-app sample rate, a throttle and, where the provider reports costs, a budget?
• Does a new app without an entry fall back to 100 percent, and is that acceptable?
• Is there a hard cap at the provider account?

Storage and security

• Is the database PostgreSQL or Snowflake with a synchronous driver, SSL, and a pooler where needed?
• Are credentials injected from a secret store, with separate writer and migration roles?
• Is database_redact_keys=True set?
• Is the dashboard bound to localhost behind SSO, and is content capture off for external exports?
• Is there a retention policy that was tested on a restored copy?

Upgrades and operations

• Are all trulens-* packages pinned to one exact version?
• Is the migration a separate job with a backup before it?
• Are deprecation warnings Feedback, imp= and MetricConfig gone from the code?
• Is there a canary, an alert on evaluated-to-ingested ratio, and a rehearsed restore?

The complete picture

Beginner taught you to instrument an app, attach the RAG triad, and read the leaderboard. Mid showed you how storage, selectors, conversations and batch evaluation behave under real use. This level added the layer above the code: TruLens is a library whose state lives in a database you operate, whose evaluators run inside processes you deploy, and whose numbers are measurements made by a judge you must govern.

The through-line is that every convenient property is a responsibility in disguise. No server means no central control, so you supply configuration discipline. A shared database means one schema, so you supply upgrade choreography. A dashboard without authentication means you supply the access layer. A judge call per metric means you supply the budget. Open source and OpenTelemetry mean you can leave, so you supply the exit path by keeping your definitions portable.

The habits that keep it healthy are few. Pin versions and the judge. Sample deliberately and watch the evaluated ratio. Measure the judge against people. Curate production failures into a regression set. Gate releases on it. Keep the dashboard private and the database classified. Rehearse restore and upgrade. Everything else is detail that follows from those.

Where the series leaves you

You can now own TruLens for a team. You can explain why a leaderboard goes quiet and find the cause in order. You can set an evaluation policy that holds a spending limit, and you know which providers it cannot hold it for. You can tell a product lead whether a score change is the application or the instrument. You can design tenancy on top of a library with no tenant model, run a fleet-wide upgrade without stranding pods, and say plainly when another tool is the better choice.

From here the natural neighbours in this catalogue are Langfuse for a server-based tracing platform with access control, RAGAS for retrieval-focused evaluation and synthetic test sets, Kubernetes for running the application fleet and the dashboard, and Terraform for provisioning the database, secrets and network paths as code. Read the release notes for each new version as it arrives, because this project moves quickly and the details in this guide are a snapshot of 2.14.0.

Sources

• TruLens documentation home
• Installation
• Core concepts
• The RAG triad
• Dashboard
• Trends dashboard page
• OpenTelemetry in TruLens
• Semantic conventions
• Instrumentation guide
• Span groups
• Conversation evaluation
• Metric migration guide
• Feedback providers
• Batch evaluation
• Agentic evaluations
• LLM judge alignment
• Where to log
• Logging in PostgreSQL
• Logging in Snowflake
• Guardrails
• Inline evals
• The missing-context warning
• Migrating from trulens_eval
• Release history
• GitHub releases
• Release trulens-2.14.0
• DEPRECATION.md at 2.14.0
• Client hooks guide at 2.14.0
• trulens on PyPI

Part three of three. A senior interview about TruLens is rarely about whether you can write a metric. It tests whether you can run evaluation as shared infrastructure: design it, keep it honest, cost it, secure it, and defend the decisions to people who do not care about the library. This file is organised as system-design prompts, incident questions, leadership arguments and a rapid-fire round. Every answer is written the way you would say it, then annotated with what a strong answer adds. The version under discussion is TruLens 2.14.0.

The thirty-second answer

If you have half a minute, this is the answer to "what is TruLens and how would you run it for a company".

TruLens is an open-source Python library that traces LLM applications as OpenTelemetry spans, scores those traces with metrics that usually call a judge model, and stores the results in a SQL database you read through a Streamlit dashboard. It has no server of its own: instrumentation, exporters and evaluators run inside each application process, and the shared database is the integration point. Running it as a platform means owning that database, the judge models, the evaluation budget and the upgrade path, because the library gives you none of multi-tenancy, authentication or central rate limiting.

Then add the sentence that marks you as senior: "The scores are measurements from an instrument, so I would pin the judge, calibrate it against human labels, and treat any judge change as a versioned event."

System-design prompts

Design one: evaluation for a company with thirty LLM applications

Prompt: "Thirty teams are shipping RAG and agent features. Design the evaluation platform using TruLens."

Start by asking questions, because the answer depends on them. How sensitive is the data, and does it have to stay in a region? Are the teams on one cloud? What is the request volume of the largest application? Is there an existing tracing backend? Who owns release decisions? Say you would ask these before drawing anything.

Then draw the architecture in layers.

Instrumentation layer. A shared internal package that wraps the official pattern: @instrument on retrieval with the query and retrieved-context attributes mapped, a generation span, and a RECORD_ROOT with input and output. Teams import it rather than writing their own, because inconsistent span typing is the top cause of silent selector failures.

Metrics layer. A versioned package of vetted metrics: the RAG triad, plus the agent GPA metrics where relevant. Each metric has a pinned judge, a stated scale and a link to its alignment report. Teams import the function that builds them. This is what makes "groundedness 0.82" mean the same thing across thirty teams.

Storage layer. PostgreSQL with a synchronous driver, behind a pooler, because every replica of every app holds its own engine. The question of one database or many comes from the tenancy answer: TruLens has no tenant model, so isolation is by database, schema or database_prefix. For sensitive teams I would give each its own database and role. For low-sensitivity teams a shared database with prefixes is acceptable. I would state openly that app_name alone is only a label.

Evaluation policy. configure_online_eval with a per-app sample rate dictionary, a throttle, and a daily budget where the provider reports costs. I would call out the trap that an app missing from the dictionary runs at 100 percent, and that Anthropic as a judge does not enforce the budget in 2.14.0, so I would add a provider-side spending cap.

Access layer. The dashboard has no authentication. I would run one instance per isolation domain, bound to localhost, behind an identity-aware proxy.

Release gates. Offline regression on a frozen, human-labelled golden set, using BatchEvaluator in CI, with a floor, a baseline comparison and a minimum row count.

Operations. A canary app that records a fixed query every few minutes and alerts on missing or NaN scores; alerts on the evaluated-to-ingested ratio; a retention job; a rehearsed restore; an upgrade runbook.

Close the answer with what you would not build. You would not build a custom tracing backend, and you would not make every team run the dashboard.

Design two: evaluating a customer-support agent in production

Prompt: "A support agent calls tools, retrieves policy documents and talks to customers over several turns. Design how you measure whether it is good."

Break the agent into what can be measured separately.

The retrieval step: context relevance on each retrieved chunk, scored with collect_list=False so each chunk gets its own score and the mean is taken. The answer: groundedness against the retrieved policy text and answer relevance against the question. The tool use: the trace-level agent metrics, tool_selection_with_cot_reasons, tool_calling_with_cot_reasons and plan_adherence_with_cot_reasons, selected with Selector(trace_level=True). The conversation: since 2.12, a conversation_id groups turns and conversation-scoped metrics such as coherence_across_turns, topic_adherence and agent_goal_accuracy produce one score per conversation.

Then cost it. Agent metrics read whole traces and are expensive. I would run the cheap triad online at a low sample rate and the trace-level agent metrics offline on a stratified sample, with trace compression enabled. I would mention span_group for metrics that must run once per hop or per repeated tool call.

Then mention limits honestly. Conversation metrics are not supported on the Snowflake connector. LLM-judged helpfulness is noisy, so I would calibrate against human-labelled conversations using the alignment tooling. And for the dangerous actions, such as issuing a refund, I would not rely on an after-the-fact score at all: I would use deterministic checks, plus human approval, at the point of action. A score tells you a refund was probably wrong; it does not stop it.

What a strong answer adds: the feedback loop. Flagged conversations are curated with curate_records_to_dataset into the regression set, so each incident becomes a permanent test.

Design three: a regulated bank in the Gulf

Prompt: "A bank in the Gulf wants TruLens for an internal assistant. Customer data may not leave the country."

The constraint reshapes three components.

The judge is the first. Each metric call sends the prompt, context and answer to a model provider. Options are a provider deployed in an approved in-country region, a local judge through Ollama, HuggingfaceLocal or a vLLM endpoint addressed with OpenAI(base_url=...), or a Snowflake account in an approved region using Cortex. I would pick based on judge quality, since small local models are weaker graders, and I would measure that with an alignment report before committing.

The storage is the second. Traces contain prompts and retrieved documents, so the database is classified at the level of the most sensitive input, encrypted at rest, with backups in the same jurisdiction, and with database_redact_keys=True for keys.

The export is the third. If the bank uses a SaaS tracing vendor, OTLP export sends spans out of country. Content capture is off by default in the newer paths, and I would keep it off, and export only metadata and metrics if export is needed at all.

Add the dashboard: no authentication, so it sits behind the bank's identity provider with localhost binding and audit logging at the proxy, because the dashboard itself keeps no access log.

What a strong answer adds: the statement that redaction of keys is not redaction of personal data, and that PII scrubbing has to happen in the application before instrumented code sees the text.

Design four: moving 200 million traces a month without the database collapsing

Prompt: "Volume grew ten times. The dashboard is slow and the database is huge. What do you do?"

Diagnose before prescribing. Find which tables hold the bytes: the span payloads dominate. Find the query pattern: Streamlit sessions and notebooks that pull everything. Then the levers, in order.

Sample harder, since you do not need an evaluation on every record, and the leaderboard shows the observed sample rate so readers know it is an estimate. Set retention by age, deleting or partitioning by time, tested on a restored copy because the schema is managed by Alembic and table layouts change between versions. Reduce what is stored by trimming the attributes you map and by avoiding content capture where not needed. Use SQL-level aggregation, which the leaderboard has had since 2.8, and use limit and offset in every script. Separate read load: point heavy analysis at a replica. Put the pooler in front so replica counts do not multiply connections.

If volume keeps growing, question the premise. You may be storing traces that nobody reads. Move operational tracing to your existing OTLP backend, which is built for that volume, and keep the TruLens database for the sampled, evaluated subset. The OTLP mode replaces the database exporter and does not fan out, so this means two environments or two sessions by purpose.

Incident-response questions

"The leaderboard has not updated since this morning"

Order the checks by likelihood and cost. First, is the application writing rows at all? Second, was OTel tracing turned off, which TruLens warns about once per process since 2.14? Third, is the session in OTLP mode, which sends spans away from the database by design? Fourth, did the process exit before flushing? Fifth, is it the evaluator rather than the ingestion: are records arriving with no scores? Then the eval_decision_reason values tell you whether it is policy (throttled, over_budget) or failure.

Say what you would do to prevent repeat: a canary and an alert on evaluated-to-ingested ratio.

"Groundedness dropped from 0.88 to 0.61 after a deploy"

Do not assume the application regressed. There are four candidate causes and they need different fixes. The application changed, for instance the retriever now returns worse chunks. The instrumentation changed, so the selector now feeds the metric the wrong span. The judge changed, because a library upgrade moved a default model or because someone edited the criteria. The data changed, because a new traffic segment is harder.

Separate them by replaying a frozen golden set through the old and new application with the same judge, then through one application with old and new judges. If the golden set score is unchanged and production dropped, suspect instrumentation or traffic. If the golden set dropped, it is the application or the judge.

Also look at the count of scored rows, and at the confidence interval on the Trends page. A drop on forty samples may be noise.

"Our LLM bill tripled"

Get the authoritative number from the provider's billing. Then attribute it: split by app name using the leaderboard cost columns. Then look for the usual causes: a new app added without an entry in the sample_rate dictionary (defaults to 100 percent), a metric flipped to collect_list=False so it runs per chunk, a retrieval change that increased chunk counts, or a judge upgrade to a larger model. Check whether cost_budget was set with a provider that does not report costs, in which case it was never enforced.

Contain first: lower the rate through a single configure_online_eval call, remembering a second call replaces the whole configuration. Then fix the cause and add a per-app cost alert.

"The pods crash-loop after the upgrade"

Read the message. "Schema is behind the expected revision" means you deployed new clients before migrating: back up, run a single migration job, and the pods recover. "Schema is ahead of the expected revision" means an old client met a migrated database: upgrade that client. There is no supported downgrade of a migrated schema, so if the release is bad, the rollback is a restore from backup and the loss of newer records. That is why the backup and a rehearsed restore come before the upgrade.

Add what you would change in process: a dedicated migration job with its own role, a short window, and exact version pins on every trulens-* package.

"We found customer data in the traces"

Contain, then remediate. Stop the source: disable content capture, or stop the exporter. Scope it: which apps, which time range, which readers had dashboard access. The dashboard has no access log, so the proxy log is your evidence. Remove the data by record identifier from a backed-up database, record the deletion, and verify backups and replicas. Then prevent: scrub in the application before instrumentation, shorten retention, and review who can reach the dashboard.

A strong answer mentions that database_redact_keys=True protects keys and not user text, since people often assume otherwise.

"Developers say the metric returns nothing"

This is almost always a selector mismatch. Check that the span type and attributes exist on the trace, that the selector keys match the implementation's parameter names, and that the method ran inside the recording context. Check for the Cannot find TruLens context message, which means the instrumented method ran on a thread or task that did not inherit the context. The fix is TruLens's thread helpers, contextvars.copy_context, or importing threading after TruLens. A guardrail that "returns a tuple" is the _with_cot_reasons variant being used where a float is required.

How to argue a trade-off to leadership

Leadership questions test whether you can turn a technical property into a decision. The pattern is the same: state the decision, the options, what each costs, your recommendation, and what would change your mind.

"Why not just evaluate every request?"

Because the marginal information is small and the bill is not. A sample of five percent, chosen deterministically, gives a tight estimate of the mean for a large application, and the Trends page shows the confidence interval so you can see it. Evaluating everything multiplies judge calls by twenty. Spend the saved budget on better judges for the metrics that gate releases, and on human labelling, which improves the measurement itself. The exception is a low-volume, high-stakes application, where I would evaluate all of it.

"Can we trust an LLM to grade our LLM?"

Partly, and the honest answer is that trust is something you measure. Judges have bias toward length and toward their own family. So we keep a human-labelled golden set, measure agreement between humans first, then between judge and humans using the alignment report, and only adopt a judge that meets an agreed threshold. For high-stakes decisions the score is a triage signal and a person decides. This answer tends to earn trust because it does not oversell.

"Why pay to self-host when we could buy a hosted tool?"

Put it as a comparison of what you own. TruLens costs no licence and keeps data in your boundary, which matters for residency. What you take on is the database, upgrades, access control and the operational runbooks. A hosted product moves those to a vendor and adds a data-processing agreement, per-seat or per-event pricing, and less control. I would recommend self-hosting when data cannot leave, when we already run Postgres well and have platform engineers, and a hosted service when the team is small and the data is low-sensitivity. And I would state the exit: because TruLens emits OpenTelemetry spans and stores data in SQL, leaving costs a migration rather than a rewrite.

"How long will the migration from the old API take?"

Give a range with assumptions. The Feedback to Metric change is mostly mechanical, since Feedback is now a subclass alias that still works and warns. The risky part is code that checks isinstance(x, Feedback), code using imp=, and code using legacy recording such as with_record. I would inventory with a warnings-as-errors test run, migrate the shared metrics package first so most teams inherit the change, and give teams a date before removal. Emphasise that nothing breaks today, but the library says Feedback will be removed in a future release, so the cost of delay is a forced migration at a time we do not choose.

"What is the risk of a fast-moving open-source dependency?"

The project shipped six minor releases in about six weeks. Those releases added features but also changed defaults, default judge models and numerical behaviour, for example IR metrics returning nan rather than 0.0 in 2.14.0. The mitigation is process: exact pins, a staging environment with a golden-set regression, release-note review and a scheduled cadence. The benefit of the pace is that the library tracks frameworks, so I would not choose a slower tool just to avoid upgrades.

Trade-off questions with short answers

Database or OTLP export? Database gives you the dashboard and online evaluation. OTLP gives you spans in an existing backend but replaces the database exporter, so no dashboard and no database-reading evaluations. Choose by purpose, not by fashion.

One shared database or one per team? Shared is cheaper and gives no real isolation, since every dashboard user sees everything. One per team costs more in migrations and gives isolation you can defend. Use per-team for sensitive data.

One strong judge or a jury? A jury costs several calls per metric. Use it only where the decision justifies the spend, and confirm with alignment data that it outperforms the single judge.

Per-chunk context relevance or one call over the list? Per chunk (collect_list=False) gives a finer signal and costs a call per chunk. One call is cheaper and coarser, and long contexts may degrade the judge.

Online evaluation or offline runs? Online finds drift in real traffic and costs continuously. Offline is reproducible and cheap to gate releases. A mature setup does both, with online sampled and offline on a frozen set.

Guardrails in TruLens or a dedicated system? TruLens guardrails add a synchronous judge call to the request path and need float-only metrics. A dedicated system gives deterministic, low-latency blocking. Use TruLens to measure and a dedicated tool to enforce.

SQLite, Postgres or Snowflake? SQLite for one developer. Postgres for shared teams. Snowflake if the company is on it, accepting that the OSS dashboard will not start on it and that Snowsight's Evaluations UI has no Trends view and no conversation support.

Questions about the library's edges

What happens if two apps share a session with different databases? TruSession is a singleton. A second construction with a different database logs that it was already initialised and cannot change the configuration. Configure the session once at process start.

Can you nest recording contexts? No. It raises Already recording with a context manager, cannot nest!.

Why can't I use FeedbackMode.DEFERRED? Under OTel tracing, which is the default since 2.3.0, only WITH_APP_THREAD is allowed. Older docs describing other modes refer to the legacy path. Use configure_online_eval and explicit compute calls.

What is compute_now and why do I care? It is the explicit evaluation call that bypasses sampling, throttle and budget, so it is the backfill tool. It lives on an underscore attribute, so wrap it.

How do you get synchronous results in a test? Records export asynchronously, so use recording.retrieve_feedback_results(timeout=...) and session.force_flush(). wait_for_feedback_results on the app raises under OTel.

Which Python versions? The meta-package needs 3.10 or later, and NeMo needs a version below 3.13, so teach 3.10 through 3.13.

Why did my Postgres connection fail with MissingGreenlet? The URL used asyncpg. Use psycopg2 or psycopg version 3.

What a strong answer covers that a weak one misses

The pattern across all of these is consistent, and you can use it as a self-check on any answer you give.

• It names the bottleneck. Weak answers say "scale the database". Strong ones say the judge is the limit and give the arithmetic.
• It distinguishes the measurement from the thing measured. Weak answers blame the application for every score change. Strong ones check the judge, the selector and the sample first.
• It says what is not enforced. Weak answers trust cost_budget. Strong ones know it is unenforceable for providers that do not report costs.
• It treats the dashboard as a security boundary. Weak answers forget it has no authentication.
• It plans the upgrade as choreography. Weak answers say "run pip install -U". Strong ones give backup, single migration job, then rollout, and know there is no schema downgrade.
• It closes the loop. Weak answers stop at monitoring. Strong ones turn production failures into regression tests.
• It is honest about limits. It says where TruLens stops and another tool, such as Langfuse for a server with access control or RAGAS for synthetic test sets, is the better answer.

Sixty-second self-test

Answer each out loud without looking back.

1. Where does the evaluator run, and what does that imply for capacity?
2. Name the three schema errors and the action for each.
3. Why does a cost_budget not protect you when the judge is Anthropic?
4. What does OTLP mode do to the dashboard, and why?
5. Which default makes a new app run at full evaluation cost?
6. What does database_redact_keys=True not protect?
7. How do you tell whether a score dropped because of the app or the judge?
8. Why must a guardrail metric avoid the _with_cot_reasons variants?
9. What is the rollback path after a schema migration?
10. Give two situations where you would recommend a different tool.

The answers are in order: in threads inside each app process, so evaluation load scales with the fleet; behind, ahead and reconfigured, meaning migrate after backup, upgrade the client, and migrate with prior_prefix; because Anthropic does not override reports_costs in 2.14.0; it replaces the database exporter so the dashboard sees nothing; an app absent from the sample_rate dictionary; personal data in prompts; replay a frozen golden set with old and new application and with old and new judge; because guardrails need a float and those variants return a tuple; restore from backup; a server with access control and projects, or a pytest-first or synthetic test-set workflow.

If you can answer eight of ten without hesitation, you are ready to defend the platform in an interview or in a design review.

Part three of three. This file is the working notes of someone who has been paged about an evaluation platform: production traps that only appear at scale, the habits that keep a shared TruLens deployment healthy, and review checklists you can paste into pull requests and design documents. Everything is written against TruLens 2.14.0. Where a trap depends on a version, the version is named.

Production traps that only appear at scale

Each trap below gives the symptom, the cause and the fix. They are ordered roughly by how often they cost teams real time.

Trap 1: the healthy average over a dying pipeline

Symptom. The groundedness average is steady at 0.84 for a week. Nobody worries. Then someone notices the evaluated-record count has been a fifth of the ingested count for days.

Cause. The judge provider was rate-limiting, or a budget was being hit, and most evaluations never produced a score. The mean of the survivors looks normal, and a mean carries no information about how many values went into it.

Fix. Report the count beside every mean, and alert on the ratio of evaluated to ingested records. Read the eval_decision_reason column: evaluated, not_sampled, throttled and over_budget separate expected policy from failure. A rising throttled or over_budget share means your policy is too tight for your traffic, and missing rows with none of those reasons means the judge is failing.

Trap 2: new apps inherit 100 percent evaluation

Symptom. A team launches a new service, and two days later the judge bill spikes.

Cause. configure_online_eval takes a dictionary of sample rates by app name, and an app not listed in it is evaluated at 100 percent. The conservative default you set for existing apps does not apply to names you did not list.

Fix. Have the platform package supply a default rate for unknown apps, or make onboarding a required pull request that adds the app to the policy file. Remember that a second call to configure_online_eval replaces the entire configuration, so keep it in one module and build the dictionary from a reviewed file instead of calling it from several places.

Trap 3: a budget that cannot be enforced

Symptom. You set cost_budget=10.0 and still spent three times that in a day.

Cause. The daily budget only works for providers whose reports_costs is true. In 2.14.0 those are OpenAI, LiteLLM, Google and Cortex. Anthropic tracks per-model cost but does not override that flag, so the budget is not enforced, and a warning is logged once at configuration time that nobody reads.

Fix. Grep the startup logs for that warning in your deployment pipeline and fail the build if it appears for a production config. Put a hard spending cap at the provider account as the real backstop. Use sample rate and throttle as your controls for providers that do not report costs.

Trap 4: the default judge moved under you

Symptom. Every score shifts a few points after a library upgrade, with no application change.

Cause. The judge model was not pinned, so the provider class default applied. Defaults belong to the library version. They are gpt-4o-mini for OpenAI and LiteLLM, claude-sonnet-4-6 for Anthropic, gemini-2.5-flash for Google, amazon.nova-lite-v1:0 for Bedrock, llama3.3-70b for Cortex and llama3.2 for Ollama in 2.14.0, and release 2.8.1 already updated several of them.

Fix. Always write model_engine= explicitly, and store the judge identifier in the app version or metadata. A judge change is a versioned event with its own line on the trend chart.

from trulens.providers.openai import OpenAI

JUDGE_MODEL = "gpt-5-nano"            # pinned; changing it is a reviewed, announced change
provider = OpenAI(model_engine=JUDGE_MODEL)

Trap 5: the selector that matches nothing

Symptom. A metric column is empty or missing, with no error anywhere.

Cause. Selector.select_context() looks for retrieval spans with the retrieved-contexts attribute. If the method was decorated without span_type=RETRIEVAL or without mapping RETRIEVED_CONTEXTS, there is nothing to match and the evaluator quietly has no input. The other cause is a selector key that does not match the implementation's parameter name, for instance question where prompt is expected.

Fix. A CI test that records one trace and asserts each metric returned a number. Run it against every app, on every pull request that touches instrumentation. No test, no merge.

Trap 6: the context that does not follow the thread

Symptom. Cannot find TruLens context in the logs and missing spans in the trace.

Cause. Instrumented code ran in a thread or asyncio task that did not inherit the recording context. Common offenders are a bare ThreadPoolExecutor from the standard library and background tasks created outside the request.

Fix. Use TruLens's threading helpers, trulens.core.utils.threading.Thread or its ThreadPoolExecutor, or submit through TP().submit, or wrap the target in contextvars.copy_context().run. Importing threading after TruLens also works. On asyncio, keep the default context copying of Python 3.11 and later.

Trap 7: OTLP mode quietly empties the dashboard

Symptom. After enabling OTLP export, the dashboard shows no new data and database-reading evaluations never run.

Cause. TruSession(otel_exporter="otlp") replaces the database exporter rather than adding to it. There is no fan-out. The spans went to your collector, as designed.

Fix. Decide by purpose. Keep the database path for online evaluation and the TruLens dashboard. Use OTLP in a separate environment or session for operational tracing. Remember the transport is gRPC only in 2.14.0, so the endpoint is port 4317, not 4318; HTTP export exists only on the main branch.

Trap 8: the exporter and SDK pins fight

Symptom. pip install "trulens[otlp]" fails to resolve, or the application breaks after adding the extra.

Cause. The extra pins opentelemetry-exporter-otlp-proto-grpc 1.44.0, which needs an opentelemetry-sdk in the 1.44 series. If your application already pins a different SDK, the resolver cannot satisfy both.

Fix. Install a matching exporter and SDK pair, or isolate TruLens in its own environment or container. Do this in a scratch environment first and commit the resulting lock file.

Trap 9: asyncpg and the MissingGreenlet error

Symptom. A MissingGreenlet error from SQLAlchemy on startup.

Cause. The database URL uses postgresql+asyncpg://. TruLens uses synchronous SQLAlchemy and does not support asyncpg.

Fix. Use psycopg2-binary, psycopg2 or psycopg version 3. The URL forms postgresql://, postgresql+psycopg2:// and postgresql+psycopg:// are supported, and ?sslmode=require works.

Trap 10: connection count grows with the fleet

Symptom. Postgres runs out of connections after scaling the application horizontally.

Cause. Every replica is a TruLens writer and an evaluator, holding its own engine and its own pool. Connection count equals replicas multiplied by the pool size.

Fix. Place a pooler in front of the database, pass an explicit engine with sane pool_size and pool_pre_ping=True, and size the pooler for the largest planned replica count. Count the evaluator threads as well as the request handlers when estimating.

import os
import sqlalchemy
from trulens.core import TruSession

engine = sqlalchemy.create_engine(
    os.environ["TRULENS_DB_URL"],
    pool_size=5,
    max_overflow=5,
    pool_pre_ping=True,
    connect_args={"sslmode": "require"},
)
session = TruSession(database_engine=engine, database_redact_keys=True)

Trap 11: the schema is one shared fact

Symptom. Pods crash-loop with Database schema is ahead of the expected revision during a partial rollout.

Cause. One service was upgraded, migrated the shared database, and other services still run the older client. Nothing stops the first service from migrating.

Fix. Make migration a separate job with its own privileged role, run by the platform team, and give application roles no permission to alter tables. Then a service upgrade cannot migrate the database by accident. Upgrade all clients in one window, and take a backup first, because there is no supported downgrade.

Trap 12: averages over NaN

Symptom. After the 2.14.0 upgrade, a retrieval-quality chart shows gaps or a release gate behaves strangely.

Cause. The information-retrieval ground-truth metrics, ndcg_at_k, recall_at_k and others, now return nan instead of 0.0 when the result is undefined, for example for an empty relevance list. A plain mean over a list containing nan is nan.

Fix. Use nan-aware aggregation, dropna() or numpy.nanmean, and count how many were undefined, because that count is itself a finding about your retrieval.

Trap 13: the dashboard is open to anyone who can reach it

Symptom. A security review finds prompts visible to anyone on the office network.

Cause. The dashboard is a Streamlit app with no authentication. Anyone who reaches the port reads the entire database.

Fix. Bind to localhost with address="127.0.0.1" or --address 127.0.0.1, and publish through an identity-aware proxy. One instance per isolation domain. Log access at the proxy, since the dashboard keeps no access log of its own.

Trap 14: legacy recording patterns that raise under OTel

Symptom. Not supported with OTel tracing enabled! or Cannot use feedback_mode other than WITH_APP_THREAD with OTel tracing!.

Cause. Code or copied documentation using the legacy path: update_record, with_record, FeedbackMode.DEFERRED, or wait_for_feedback_results on the app. Some documentation pages still describe that path.

Fix. Remove the feedback-mode argument. Use configure_online_eval for policy and retrieve_feedback_results when a test needs the results. When the docs and the exception disagree, the exception is right.

Trap 15: the singleton ignores your second configuration

Symptom. A second TruSession(database_url=...) seems to do nothing, and records land in the wrong database.

Cause. TruSession is a process-wide singleton. A later call with a different database only logs that it was already initialised and cannot change configuration.

Fix. Configure the session exactly once at process start, in one module, and import the instance everywhere else. Never construct a session in library code.

Platform-ownership habits

Traps are what break. Habits are what stop you having to think about the traps. These are the practices that paid off most for teams that run TruLens for others, in rough order of leverage.

Publish a metrics package, not a wiki page. Teams copy code. If the vetted Metric definitions, with pinned judge, correct selectors and a stated scale, live in one versioned internal package, fifteen teams get the same definition of groundedness and inherit every fix. A wiki page produces fifteen subtly different versions. Version the package with semantic versioning, and put in the changelog which scores are expected to shift when a judge changes.

Keep a frozen, human-labelled golden set. It is the unit test for the measuring instrument. Before any judge, criteria or model change ships, run it against the golden set. Use GoldenSetGenerator with stratified or uncertainty sampling to pick records worth labelling, label them twice, and measure agreement between the two people first. A judge cannot be reliably better than the humans it is compared with.

Run a canary. One synthetic app, one fixed question, recorded every few minutes, with retrieve_feedback_results(timeout=180) and an assertion that every expected metric is present and numeric. The canary catches an expired judge key, a schema mismatch, a selector regression and an exporter outage before any team does.

Treat versions as data. app_name and app_version drive the leaderboard, Trends and Compare pages. Adopt a naming rule that a newcomer can apply: a version derived from the commit hash, plus the model and prompt identifiers in metadata. "Final" and "v2-new" tell the next person nothing.

Pin everything, upgrade on a schedule. Exact versions for every trulens-* package, with hashes in the lock file, and a rule such as staying within two minor versions of current. Upgrade in staging first with the canary and golden set running. The project shipped 2.9 through 2.14 in about six weeks, so a skipped-quarter upgrade spans a dozen changes. Read the release notes for each version you pass.

Make deprecation warnings a build signal. Feedback, MetricConfig, SnowflakeFeedback, imp=, the old dashboard launch methods and TruCustomApp are all deprecated. Run the platform's own test suite with deprecation warnings as errors, so its code is clean, and give consuming teams a dated migration plan.

Show teams their own cost. Break evaluation spend down by app name from the leaderboard and Trends cost columns, and send each team its number monthly. People change sampling rates when they see the figure. They do not when a central team holds the bill.

Write retention down and test it. A retention job that hard-codes table names breaks when a migration changes them. Test on a restored copy, schedule it, and alert when the database grows faster than the forecast.

Rehearse the restore. The only rollback for a migrated schema is a restore, so measure how long a restore takes, in a scratch instance, before you need it.

Keep the exit open. Metrics as plain Python, datasets exportable, prompts in a system you can move, traces available over OpenTelemetry. A platform teams can leave is one they choose to stay on.

Close the loop from production to tests. Give every team a one-step way to flag a bad record and a scheduled job that runs curate_records_to_dataset on flagged records. The ground-truth identifiers are content-addressed, so re-running the job is idempotent. Every production failure becomes a permanent regression case.

Watch the statistical basics. Use the confidence intervals the Trends page draws. A move from 0.86 to 0.84 on two hundred sampled records is inside the noise. Teams that react to every wiggle learn to ignore the dashboard, and teams that never react miss the real regression. Set a rule for what counts as a real change, such as a non-overlapping interval on two consecutive days, and write it down.

Separate measurement from enforcement. Use TruLens to measure. Where you must block something, such as a dangerous tool call or a banned output, use deterministic rules or a dedicated guardrail system. TruLens's guardrails decorators, block_input, block_output and context_filter, put a synchronous judge call in the request path and require float-only metrics, so use them for soft, low-stakes filtering.

Mind the data residency from day one. Decide the judge location, the database region and the export destination before onboarding teams. For Gulf and Egyptian employers with in-country requirements, a local judge through Ollama, HuggingfaceLocal or vLLM, or a provider deployed in an approved region, is cheaper to choose at the start than to retrofit after dashboards exist.

Review checklists

Copy these into your pull request template or design-document template. They are phrased as questions, so each can be answered yes, no or not applicable.

Pull request checklist: a change to an instrumented application

Instrumentation

• Is the retrieval method typed as RETRIEVAL, with QUERY_TEXT and RETRIEVED_CONTEXTS mapped?
• Is the entry point a RECORD_ROOT with input and output attributes mapped?
• Do background threads and tasks carry the context through TruLens's helpers or copy_context?
• Does a test record one trace and assert that every metric returned a number?
• Does the change avoid nested with app: recording blocks?

Metrics

• Is the judge model pinned with model_engine=?
• Do selector keys match the implementation's parameter names?
• Is collect_list chosen deliberately, and is per-chunk cost acknowledged?
• Does a guardrail metric use the float-only variant?
• Is a new metric covered by an alignment check against labelled data?

Versioning

• Does app_version follow the naming rule, and does it change when the model, prompt or retriever changes?
• Is the judge identifier recorded in metadata?

Deprecations

• Are there new uses of Feedback, imp=, MetricConfig, TruCustomApp, run_dashboard_sis or the TruSession dashboard methods?
• Does the code use the OTel instrument import from trulens.core.otel.instrument?

Design-document checklist: adopting or extending TruLens

Architecture

• Which database, which driver, and is a pooler in place?
• What is the isolation model: prefix, schema, database or Snowflake roles?
• Is the dashboard bound to localhost behind SSO, one instance per isolation domain?
• If OTLP export is planned, is the database-versus-OTLP trade-off written down?

Capacity and cost

• How many judge calls does one user request cause, and what is the provider rate limit?
• What are the sample rates per app, the throttle and the budget, and does the provider report costs?
• Is there a provider-side hard cap?
• What is the retention period, and has the deletion job been tested on a restored copy?

Quality of measurement

• Which judge, pinned to which model, and where is its alignment report?
• Is there a frozen, human-labelled golden set, and who owns it?
• What is the rule for a real change in a score?

Security and compliance

• Where does judge data go, and does that satisfy residency requirements?
• Are provider keys and database credentials injected from a secret store, with separate migration and application roles?
• Is database_redact_keys=True set, and is personal-data scrubbing done in the application?
• Is content capture off for external exports?
• Is there a runbook for removing sensitive data from traces?

Operations

• Are all trulens-* packages pinned to one exact version?
• Is migration a separate job with a backup before it, and is the restore rehearsed?
• Is there a canary, and an alert on the evaluated-to-ingested ratio and on spend?
• Is there an on-call owner and a written incident playbook?

Exit and scope

• What would make us choose another tool, and what would leaving cost?

Operational recipes

Short, copyable procedures for the situations that come up repeatedly.

Recipe: a safe upgrade

1. Read the GitHub release notes for every version between yours and the target.
2. In staging, create a fresh virtual environment, install exact pins for all trulens-* packages, and run the test suite with deprecation warnings visible.
3. Back up the production database.
4. Run the migration job once:

import os
from trulens.core import TruSession

TruSession(database_url=os.environ["TRULENS_DB_URL"]).migrate_database()

5. Roll the application fleet to the new pins during the agreed window, then the dashboard.
6. Run the canary and compare evaluated-to-ingested ratios with the previous day.
7. Keep the backup until the next successful upgrade.

Recipe: check the judge still agrees with people

1. Sample fifty records with GoldenSetGenerator(session, seed=42).sample(...), using the stratified strategy.
2. Have two people label them independently, then compute their agreement.
3. Run the alignment report for the current judge against the labels.
4. If a cheaper candidate is proposed, run CrossModelAlignment and adopt it only if agreement stays within the agreed margin.
5. Record the result next to the metric definition in the platform package.

Recipe: a cost triage in ten minutes

1. Open the provider's billing view for the last seven days.
2. Open the leaderboard cost columns by app name and find the outlier.
3. Check that app's entry in the sample-rate dictionary, and whether any metric has collect_list=False.
4. Count chunks per query from a few recent records.
5. Lower the rate with one configure_online_eval call, knowing it replaces the whole configuration, and confirm with the eval_decision_reason counts.

Recipe: serve a read-only dashboard from a snapshot

trulens-dashboard --database-url "$TRULENS_DATABASE_URL" --address 127.0.0.1 --port 8502

Point TRULENS_DATABASE_URL at a read replica, bind to localhost, and publish through your proxy. The URL is resolved from --database-url, then the environment variable, then discovery in the directory. Use --find to print which database it would serve without starting anything.

Recipe: capture coding-agent sessions safely

1. Install the package for your client, for example pip install trulens-core trulens-apps-claude.
2. Preview the change with trulens-client-hooks install claude-code --dry-run.
3. Choose the destination with TRULENS_DESTINATION: database, snowflake or otlp.
4. Leave content, tool payloads, diffs and paths off unless a review approves each, because they are separate opt-ins.
5. Check with trulens-client-hooks status and validate, and remember that the hooks fail open, so they are telemetry and not an audit control.

Recipe: prove the canary works

Break it on purpose in staging. Revoke the judge key and confirm the alert fires. Point at a database with a different schema revision and confirm the alert fires. Remove the retrieval attribute and confirm the assertion on the metric fails. A canary that has never failed has never been shown to work.

A weekly platform routine

A platform does not run itself. A short, regular routine catches drift before users do.

• Monday. Read the canary history and the evaluated-to-ingested ratio for the past week. Investigate any day below the agreed floor.
• Tuesday. Review spend by app against the provider bill. Send each team its number. Adjust sample rates where a team is far over.
• Wednesday. Check database growth against the forecast and confirm the retention job ran and deleted what it should.
• Thursday. Check the project's releases. Read the notes for any new version, and decide whether it enters staging this cycle.
• Friday. Review flagged production records, curate the confirmed failures into the regression set, and run the golden set and the regression set against the current judge and application.

Once a quarter, rehearse a restore, review dashboard access at the proxy, rotate provider keys and the database credentials, and re-measure judge alignment on a fresh sample, since both the traffic and the models move.

The short version

Most of what goes wrong with a shared TruLens deployment is one of three things: a number nobody can trust because the judge moved or the denominator is missing, a cost nobody is watching because the budget was not enforceable or the default rate applied, or a change nobody could roll back because the schema is shared and has no downgrade. Pin the judge, publish the denominator, add the canary, keep the dashboard private, migrate as a separate job, and rehearse the restore. Do those, and the rest is ordinary engineering. For neighbouring tools, see Langfuse for a server with projects and access control and RAGAS for retrieval-focused evaluation and synthetic test sets.