The Complete LangSmith Guide

This is part one of three. It covers everything you need to do real work with LangSmith, not a teaser. By the end you can send traces from a Python or TypeScript program to LangSmith, open one and read what your application actually did, group a conversation into a thread, attach scores to runs, build a small dataset, run an evaluation over it, and version a prompt. Mid-level and Senior take the same topics further; nothing here is thrown away.

LangSmith is a hosted product from LangChain, so this guide has two jobs. The first is to teach the concepts, which are few and which transfer to every other observability tool you will meet. The second is to be precise about a product that moves quickly. LangSmith has no single version number. The Python SDK, the JavaScript SDK, the command-line tool and the self-hosted Helm chart each have their own, and this guide was checked against Python langsmith 0.14.2, JavaScript langsmith 0.10.5 and langsmith-cli v0.2.60. The SDK ships several releases a week, so run pip install -U langsmith instead of memorising a number.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and tracing only makes sense once you have watched your own program appear in the interface.

What LangSmith is, and the problem it solves

A program that calls a large language model is hard to debug for a reason that has nothing to do with the model being clever. An ordinary function gives the same output for the same input, so a failing test points at a line. An LLM application is a chain of steps: a prompt is assembled, a model is called, the answer is parsed, a tool may be invoked, a retrieval may fetch documents, and the model is called again. Any step can be the one that went wrong, the model is not deterministic, and the user only ever sees the final sentence. When the answer is bad you are left guessing which step produced it.

LangSmith is a platform that records what happened at every step and lets you look at it afterwards. It calls this observability, and it builds two more things on top of the recording: evaluation, which is running your application over a fixed set of examples and scoring the results, and prompt management, which is storing and versioning the prompt templates your application uses.

The diagram is the whole product in one line. Your code is instrumented, a small client library sends what happened to LangSmith without slowing the program down, and you read it there.

Before tools like this existed, people debugged LLM programs with print statements and log files. That works for one request on your laptop. It stops working when a colleague says "the bot gave a strange answer yesterday afternoon", because the prints are gone, the log lines from one request are interleaved with a hundred others, and nobody kept the exact prompt that was sent. A trace is a structured record that keeps the inputs, the outputs, the timings, the token counts and the errors of every step together, tied to one request, and searchable.

You do not need to use LangChain to use LangSmith. The name is confusing because the two come from the same company, and LangChain and LangGraph do trace into LangSmith with no code changes, but LangSmith works with plain Python, plain TypeScript, the OpenAI, Anthropic and Gemini client libraries, and many other frameworks. If you want the framework side, the LangGraph guide is the natural companion to this one, and LangSmith has open-source alternatives such as Langfuse, covered in the Langfuse guide. Comparing the two is a good way to learn what is essential and what is product-specific.

A note on names, because older tutorials will confuse you. The interface and the docs have been reorganised. The "tenant" of old material is now called a workspace, although the API still uses the words tenant and X-Tenant-Id. Prompt "repos" are now just prompts. The documentation moved from docs.smith.langchain.com, which now redirects, to docs.langchain.com/langsmith. When a tutorial and this guide disagree, trust the current docs.

The vocabulary: runs, traces, projects and threads

LangSmith has a handful of nouns, and every screen and every API call is built from them. Learn them now and the rest of the guide is easy to follow.

A run is one unit of work: a single model call, a single tool call, a single retrieval, a single formatting step. If you know OpenTelemetry, a run is the same thing as a span. Each run records its inputs, its outputs, its start and end time, any error, and a run type. The run type is one of llm, chain, tool, retriever, embedding, prompt or parser. The type matters because LangSmith shows each one differently: llm runs show token counts and cost, retriever runs show the documents returned. A function you decorate yourself defaults to chain.

A trace is all the runs that belong to one operation, shaped as a tree. When a user asks a question and your function calls a retriever and then a model, the function is the root run and the retriever and the model are its children. All of them share one trace ID. A single trace can hold at most 25,000 runs; beyond that the extras are rejected, which only ever matters for very long-running agents.

A project is the container that holds the traces of one application or service. You choose it with the LANGSMITH_PROJECT setting, and it is created automatically the first time something is sent to it. If you never set it, traces go to a project named default. One wrinkle to remember: in the API a project is called a session, so when you see session_id in a method signature it means the project's UUID, not a chat session.

A thread is a sequence of traces that form one multi-turn conversation. One turn of a chatbot is one trace, and the whole conversation is a thread. LangSmith groups traces into a thread when they carry a metadata key called session_id or thread_id with the same value. You set it yourself, and it must be on every run including the child runs.

Around these four, a few more nouns appear later in the guide. Feedback is a score attached to a run, for example "correctness: 1" or "user_thumbs: 0", with an optional comment. Tags are free-form strings on a run, and metadata is a set of key-value pairs on a run; both are searchable, and you will use them constantly. A dataset is a collection of examples, where each example has inputs and usually a reference output. An experiment is the result of running one version of your application over a dataset. An evaluator is a function or a judge model that turns an output into a score.

Finally, the organisation of your account. An organization contains one or more workspaces, and the workspaces contain your projects, prompts and datasets. Billing lives at the organization level. On a personal account you will have one organization and one workspace and can ignore all of this. It starts to matter when a key can see more than one workspace, which the configuration section covers.

Creating an account and an API key

Everything that follows needs an account and a key. Sign up at https://smith.langchain.com with Google, GitHub or an email address. The Developer plan costs nothing, includes one seat and 5,000 base traces per month, and is plenty for learning. Without a payment card on the account, the free tier stops accepting traces past 5,000 in a month; adding a card raises the hourly limits and lets you go past the included traces on pay-as-you-go pricing. Pricing changes, so read the pricing page when you decide, and do not rely on any number quoted in a blog post.

Next, create an API key. In the interface open Settings, then API Keys, and create one. The key is shown once; copy it straight into a password manager or a .env file. There are two kinds, and their prefixes tell you which you hold:

Keys never expire unless you set an expiry when creating them. Very old keys that start with ls__ were switched off on 22 October 2024; if a tutorial shows one, create a new key instead.

The last account decision is the region, and it matters more than it looks. LangSmith Cloud runs in separate regions, your organization lives in exactly one of them, and the API address differs per region. A key created in one region does not work against another region's address.

The web interface follows the same pattern: smith.langchain.com, eu.smith.langchain.com and so on. An organization cannot be moved between regions afterwards, so choose deliberately. If you work for an employer in the Gulf or Egypt with rules about where data may live, check which region the company's data-handling policy allows before you send real prompts, since traces contain the actual text your users typed. There is no Middle East region at the time of writing, and there is no EU contracting entity, so this is a question for your legal or security team rather than a setting you can ignore. Self-hosting, which keeps everything in your own cloud account, is an Enterprise option and is covered in the Senior guide.

Installing the SDK and checking the setup

The SDK is a normal library. You need Python 3.10 or newer. Always use a virtual environment, so that the install does not disturb anything else on your machine.

python3 -m venv .venv && source .venv/bin/activate
pip install -U langsmith
python -c "import langsmith; print(langsmith.__version__)"

The last line prints the installed version, which for this guide is 0.14.2 or newer. On Windows PowerShell the equivalent is:

py -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -U langsmith
python -c "import langsmith; print(langsmith.__version__)"

If you prefer uv, the equivalent of the install line is uv add langsmith. Optional extras exist for specific integrations, for example pip install -U "langsmith[pytest]" for the pytest plugin, but you do not need any of them yet.

Configuration is done with environment variables, which is a good design: nothing secret lives in your source files. Set three of them.

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_pt_your_key_here"
export LANGSMITH_PROJECT="my-first-app"

LANGSMITH_TRACING=true is the master switch. Without it the SDK records nothing, which is convenient for switching tracing off in tests. LANGSMITH_API_KEY is your key. LANGSMITH_PROJECT is optional, and names the project that will receive the traces. If your organization is outside the US region, also set LANGSMITH_ENDPOINT as described above.

On Windows, $env:LANGSMITH_TRACING="true" sets the variable for the current PowerShell window only. setx LANGSMITH_API_KEY "lsv2_pt_..." stores it permanently, but only new terminals see it.

You will see older material that uses LANGCHAIN_TRACING_V2, LANGCHAIN_API_KEY and LANGCHAIN_PROJECT. The SDK reads the LANGSMITH_ name first and falls back to the LANGCHAIN_ name, so the old names still work, but write the new ones in anything you create today.

Now check that everything is connected, with the smallest possible traced function.

from langsmith import Client, traceable


@traceable
def hello(text: str) -> str:
    return text.upper()


print(hello("hi from langsmith"))
Client().flush()

Run it with python hello_trace.py. It prints HI FROM LANGSMITH, and within a few seconds a run named hello appears in the project. The last line, Client().flush(), matters: the SDK sends traces from a background thread in batches, so a short script can exit before the batch is sent. Flushing waits until it is. A long-running web server never has this problem, but every script, test and serverless function does.

If you installed the command-line tool you can also check from the terminal. The installers are below; pick the one for your system.

# macOS or Linux
curl -fsSL https://cli.langsmith.com/install.sh | sh
# macOS with Homebrew
brew install langchain-ai/tap/langsmith-cli
# then log in in the browser and list your projects
langsmith auth login
langsmith project list

On Windows use irm https://cli.langsmith.com/install.ps1 | iex in PowerShell, or scoop install langsmith-cli. If a browser is not available, for example over SSH, add --no-browser to the login command. The CLI is a convenience and everything in this guide can be done without it.

Your first real trace: a small question-answering app

The hello example proves the plumbing. Now build something with the shape of a real application: a function that retrieves context, calls a model and returns an answer. You will need the OpenAI library and a key for it, but the same approach works for the Anthropic and Gemini clients.

pip install -U openai
export OPENAI_API_KEY="sk-..."

from openai import OpenAI
from langsmith import Client, traceable
from langsmith.wrappers import wrap_openai

# wrap_openai makes every call on this client show up as an llm run
openai_client = wrap_openai(OpenAI())

DOCS = {
    "refunds": "Refunds are issued within 14 days of purchase.",
    "shipping": "Standard shipping takes 3 to 5 working days.",
}


@traceable(run_type="retriever")
def get_context(question: str) -> list[dict]:
    # a toy retriever: return documents whose key appears in the question
    hits = [t for k, t in DOCS.items() if k in question.lower()]
    return [{"page_content": t} for t in hits]


@traceable  # run_type defaults to "chain"
def answer_question(question: str) -> str:
    context = get_context(question)
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Answer using only this context: {context}"},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    print(answer_question("How long do refunds take?"))
    Client().flush()

Read the code for what it does, not for what it says about OpenAI. There are exactly two LangSmith ideas in it.

The first is @traceable. Put it above a function and every call to that function becomes a run, with the arguments as inputs and the return value as outputs. When one traced function calls another, the inner call is recorded as a child of the outer one automatically, because the SDK tracks the current run for you. That is how get_context ends up nested under answer_question without any wiring. The decorator works on both normal and async functions.

The second is wrap_openai. It wraps the OpenAI client so that each call to chat.completions.create is recorded as an llm run, with the messages, the response, the model name and the token counts. Equivalent wrappers exist for other providers: wrap_anthropic and wrap_gemini are in the same module. Wrapping is what gives you the cost figure, because cost is computed from the model name and the token counts the provider returned.

Run it and open the trace. You should see answer_question at the top, with get_context and the model call nested inside. Click the model call and read the actual messages sent, the answer returned, the latency, and the token counts. That panel is the thing you will look at most often in this product.

Both decorator and wrapper are opt-in and cost nothing when LANGSMITH_TRACING is unset, so you can leave them in production code and control tracing with the environment variable.

Reading a trace in the interface

Tracing is only useful if you can read the result quickly, so it is worth a section of its own. Open your project from Tracing in the sidebar. You see a table with one row per trace, and columns for the name, the input, the output, the start time, the latency, the token count and the cost. Click a row and a panel opens with the run tree on the left and the details of the selected run on the right.

Start with the tree. Each node is a run, indented under its parent, with its duration next to it. The slowest node is usually obvious, and the waterfall view shows the durations as bars against a timeline so you can see whether steps ran one after another or in parallel. A red marker means the run raised an error, and errors propagate upward, so if a child failed you will see the parent marked too. The deepest red node is normally the real culprit.

Then read the details of one run at a time. Input and Output are the arguments and the return value. For an llm run they appear as chat messages, so you read the conversation as the model saw it. Metadata shows the model name, the provider and anything you added yourself. Feedback shows scores attached to the run. The token and cost figures sit in the header.

The table is also a search box. The filter bar accepts conditions on name, status, latency, tags, metadata and more. The same condition language is available from the SDK and the CLI, written as functions, for example and(eq(run_type, "llm"), gt(latency, 5)), meaning "model calls slower than five seconds". You will meet it again in the commands section.

Three habits make the interface much faster. Filter before you scroll: filtering to errors or slow runs turns a thousand rows into ten. Open the failing run's parent as well as the run itself, since the cause is often in what the parent sent. And copy the trace link to share it: a colleague with access to the workspace can open the same tree, which is worth more than any screenshot. The link to a run comes from the interface; do not build it by hand.

Tracing options: names, tags, metadata and run types

Once traces arrive, the next question is how to make them findable. A project of a thousand anonymous runs is a haystack; the same thousand runs with good tags and metadata is a database you can query. The decorator takes everything you need.

from langsmith import traceable


@traceable(
    name="support-answer",
    run_type="chain",
    tags=["support", "v2"],
    metadata={"app_version": "2.1.0", "customer_tier": "free"},
)
def answer(question: str) -> str:
    return f"You asked: {question}"


# values that only exist at call time go in langsmith_extra
answer(
    "where is my order?",
    langsmith_extra={"metadata": {"user_id": "u-1042"}, "tags": ["beta"]},
)

The two words to keep apart are tags and metadata. A tag is a label with no value: support, v2, beta. Use tags for categories you filter by. Metadata is a key with a value: app_version: 2.1.0, user_id: u-1042. Use metadata for facts you want to group or search by, such as a version, a customer tier or a model setting. Values known when you write the code go in the decorator; values known only at call time, such as the user's ID, go in langsmith_extra, which the decorator strips out before calling your function, so your function signature does not change.

The decorator accepts several more keyword arguments. client lets you supply a specific Client object. project_name sends this function's traces to a different project than the default, which is handy in a monorepo. process_inputs and process_outputs let you rewrite the payload before it leaves your machine, which the configuration section uses for redaction. enabled switches tracing on or off for one function. reduce_fn tells the SDK how to combine the chunks of a streamed output into a single stored output.

Choosing a run type is worth a moment. If you leave it, a function is a chain. If your function calls a model through a library that is not wrapped, set run_type="llm" so that the interface treats it as a model call. For a custom model call to show cost, put the provider and model name in metadata as ls_provider and ls_model_name and report token counts in usage_metadata. For a retrieval function, run_type="retriever" makes the returned documents display as documents. For a function that calls an external API, run_type="tool". The type is only a label, but it decides how the run is drawn and what is computed for it.

If you need to trace a block of code instead of a whole function, use the trace context manager, which records a run for the code inside the with block:

import langsmith as ls


def handle(question: str) -> str:
    with ls.trace(name="post-process", run_type="chain", inputs={"q": question}) as run:
        result = question.strip().lower()
        run.end(outputs={"result": result})
    return result

To switch tracing off for a region of code, even when the environment variable says it is on, use tracing_context:

import langsmith as ls

with ls.tracing_context(enabled=False):
    # nothing inside this block is sent to LangSmith
    print("private work")

The precedence, from strongest to weakest, is the tracing_context block, then a setting on the client or the decorator, then the environment variable. That ordering is why you can leave LANGSMITH_TRACING=true set for a whole service and still carve out an exception.

Tracing from LangChain, LangGraph and TypeScript

You can reach LangSmith from several places besides a decorated Python function. The right choice depends on what you already use.

If your application is written with LangChain or LangGraph, you do not change any code. Set the same three environment variables and every chain, model call, tool call and graph node is traced automatically, nested in the order they ran. This is the reason many people meet LangSmith first: the integration is invisible. The LangGraph guide shows what the graphs look like, and each node of a graph appears as a run in the trace.

If you use TypeScript or JavaScript, install the package and use the same environment variables:

npm install langsmith openai
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_pt_..."
export LANGSMITH_PROJECT="my-first-app"

import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";

const openai = wrapOpenAI(new OpenAI());

const answerQuestion = traceable(
  async (question: string) => {
    const res = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: question }],
    });
    return res.choices[0].message.content;
  },
  { name: "answer_question", tags: ["support"] }
);

console.log(await answerQuestion("How long do refunds take?"));

In TypeScript, traceable is a function that wraps your function rather than a decorator, and the options object comes second. In a serverless environment such as a function that returns as soon as it has a response, the batch may not be sent before the runtime freezes. Set LANGSMITH_TRACING_BACKGROUND=false so that traces are sent inline, or call await client.awaitPendingTraceBatches() on a Client before returning.

If you use a different framework, check the integrations list in the docs first. There are integrations for the OpenAI Agents SDK, the Vercel AI SDK, CrewAI, Pydantic AI, Google ADK, LiteLLM and more, plus OpenTelemetry for anything else that can emit spans. Reaching for a documented integration is nearly always less work than wrapping things by hand.

Most real applications mix the two: the framework or wrapper traces the model calls, and @traceable covers your own code so the tree tells the whole story.

Threads: following a conversation

A chatbot trace shows one turn. The bug you are chasing is usually a whole conversation: the assistant forgot what the user said three messages ago, or the second answer contradicts the first. To see the conversation, LangSmith needs to know which traces belong together, and you tell it with a thread identifier.

The rule is simple. Give every trace in one conversation the same value under the metadata key session_id, thread_id or conversation_id, and put it on the runs themselves. The docs recommend a UUID, version 7 being the preferred form. Keep it stable for the whole conversation and generate a new one for the next.

import uuid
from langsmith import Client, traceable

thread_id = str(uuid.uuid4())  # one per conversation


@traceable(run_type="chain", name="chat-turn")
def chat_turn(message: str, history: list[dict]) -> str:
    # call your model here; this echo keeps the example runnable
    return f"You said: {message}"


history: list[dict] = []
for text in ["Hello", "What is your refund policy?", "Thanks"]:
    reply = chat_turn(
        text,
        history,
        langsmith_extra={"metadata": {"thread_id": thread_id}},
    )
    history += [{"role": "user", "content": text}, {"role": "assistant", "content": reply}]

Client().flush()

After it runs, open the Threads tab in your project. The three traces appear as one thread, and opening it shows the whole exchange as a conversation. The key has to be on the runs LangSmith looks at, so if you trace child runs yourself, pass the same metadata to those as well.

Threads are what make conversation-level questions possible: how many turns did users need before giving up, in which thread did the model start repeating itself. The Mid guide builds evaluators that work on whole threads. At this level, the skill to learn is simply to set the identifier from the first turn, because you cannot add it retroactively to traces that have already been stored without it.

One product note that catches people using old tutorials: the Turns view in the thread detail page is deprecated and is scheduled to be removed from Cloud on 31 October 2026. Use the Messages or Details view instead.

Feedback: attaching scores to runs

A trace tells you what happened. Feedback tells you whether it was good. A feedback item is attached to a specific run and has a key (the name of the quality, such as correctness or user_score), a score (a number) or a value (a category), and an optional comment. The source can be a person clicking in the interface, a thumbs-up button in your own product, an automatic evaluator, or a reviewer in an annotation queue.

From code, you attach feedback to a run by its ID. For that you need the run ID, which you can set yourself before calling the function, through langsmith_extra:

import uuid
from langsmith import Client, traceable

client = Client()


@traceable(name="support-answer")
def answer(question: str) -> str:
    return "Refunds take 14 days."


run_id = uuid.uuid4()
answer("How long do refunds take?", langsmith_extra={"run_id": run_id})
client.flush()

# later, when the user clicks thumbs-up in your product
project = client.read_project(project_name="my-first-app")
client.create_feedback(
    run_id,
    key="user_score",
    score=1.0,
    comment="Helpful",
    session_id=project.id,
)

Note session_id, which is the project's UUID, found by reading the project by name. Without it the SDK warns that creating feedback for a run without a session ID is deprecated and will stop working, and on the newer datastore that LangSmith is migrating to it raises a ValueError saying the deployment cannot locate the run. Passing it is cheap, so pass it.

The feedback then appears on the run in the interface, and you can filter and chart by it. A typical use is to record a user's thumbs-up or thumbs-down and then open only the thumbs-down traces to find what went wrong.

You can also give feedback directly in the interface: open a run, use the feedback controls, and enter a score. That is how you start building a sense of quality before you have any automation.

Finding traces from code and from the command line

Sooner or later you will want traces outside the browser: to count errors, to pull the last ten failures into a notebook, or to give an AI coding assistant access. There are two routes, the SDK and the CLI.

There is a change here that older tutorials do not mention. LangSmith introduced a new trace datastore called SmithDB in 2026, and the older query methods are deprecated in favour of new ones. The deprecated client.list_runs() and client.read_run() still exist in SDK 0.14.2 but emit a warning, "<method>() is deprecated and will be removed after Jan 31, 2027.", and they stop working when the Cloud backend removes them. The new methods live in client.runs, client.traces and client.threads, and need Python langsmith 0.10.15 or newer.

from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()
project = client.read_project(project_name="my-first-app")

for run in client.runs.query(
    project_ids=[project.id],
    min_start_time=datetime.now(timezone.utc) - timedelta(days=7),
    selects=["NAME", "STATUS", "INPUTS"],
    filter='eq(run_type, "llm")',
    page_size=20,
):
    print(run)

Four details decide whether this works. project_ids takes project UUIDs; the old project_name argument was removed, which is why the example reads the project first. min_start_time defaults to one day ago, so a query without it silently covers only the last 24 hours and looks as if your older traces are missing. selects lists the fields you want, written in uppercase, and the default is to return only the id, so forgetting it gives you a list of bare IDs. And the run_type value in a filter is written in the filter language shown above. Whether the loop is sync or async depends on the client; in async code use async for run in client.runs.query(...).

The command-line tool is often quicker for looking around:

langsmith project list
langsmith trace list --project my-first-app --limit 5
langsmith trace get <trace-id> --project my-first-app --full
langsmith run list --project my-first-app --run-type llm
langsmith thread list --project my-first-app
langsmith --format json trace list --project my-first-app --limit 5 -o traces.json

trace list defaults to the last seven days and 20 rows, and run list defaults to 50 rows, so a small number of results is not necessarily a problem. thread list requires --project. The --format json flag and the -o file option make the output easy to hand to other tools. Useful filters include --error to show only failures, --min-latency for slow runs and --tags to select by tag.

If a CLI command fails with an authentication error, run langsmith auth login again, or make sure LANGSMITH_API_KEY is exported. Note that an exported LANGSMITH_ENDPOINT overrides the address stored in a CLI profile, which can silently send a command to the wrong place.

Datasets and your first evaluation

Tracing shows what your application did. Evaluation answers a different question: is this version better than the last one? You cannot answer it by eyeballing a few traces, because improving one answer often breaks another. The fix is the same as in ordinary software: a fixed set of test cases that you run every time.

A dataset is that set. It is a collection of examples, and each example has inputs, which are a dictionary your application accepts, an optional reference outputs dictionary holding the answer you expect, and optional metadata. An experiment is what you get when you run your application over a dataset: the outputs, the scores and the traces for every example, all saved and comparable. An evaluator is the function that produces the scores.

Here is the smallest complete evaluation. The target is a plain function, so this runs without any model key and you can see the machinery clearly.

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="refund-faq",
    description="Questions about refunds and shipping",
)

client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {"inputs": {"question": "How long do refunds take?"},
         "outputs": {"answer": "14 days"}},
        {"inputs": {"question": "How long does shipping take?"},
         "outputs": {"answer": "3 to 5 working days"}},
    ],
)


def target(inputs: dict) -> dict:
    # replace this with a call to your real application
    q = inputs["question"].lower()
    if "refund" in q:
        return {"answer": "Refunds are issued within 14 days."}
    return {"answer": "Standard shipping takes 3 to 5 working days."}


def contains_reference(outputs: dict, reference_outputs: dict) -> dict:
    ok = reference_outputs["answer"].lower() in outputs["answer"].lower()
    return {"key": "contains_reference", "score": 1 if ok else 0}


results = client.evaluate(
    target,
    data="refund-faq",
    evaluators=[contains_reference],
    experiment_prefix="first-eval",
    max_concurrency=2,
)

Run it once. If you run it a second time, create_dataset will complain that the dataset already exists, so for repeated runs delete the dataset in the interface or comment out the creation lines.

There are four ideas to take from it. The target function takes the example's inputs dictionary and returns a dictionary of outputs; LangSmith calls it once per example. An evaluator is an ordinary function whose parameters are chosen by name: it may ask for inputs, outputs, reference_outputs, run or example, and LangSmith passes what you ask for. It returns a boolean, a number, or a dictionary with a key and a score (or a categorical value) and an optional comment. experiment_prefix gives the experiment a readable name, and LangSmith adds a unique suffix. And max_concurrency controls how many examples run at once; keep it low until you know your model provider's rate limits.

Open Datasets & Experiments, choose refund-faq, and open the experiment. You see one row per example with the output, the contains_reference score and a link to the full trace of that example. Every evaluation run is itself traced, so a low score is always one click from the exact steps that produced it.

Checking whether a string contains another is a weak evaluator for free-form answers. A more capable kind is an LLM-as-judge: a second model reads the question, the answer and the reference and returns a score. The separate openevals package provides ready-made ones such as create_llm_as_judge, and the Mid guide covers them and their pitfalls. For now, remember the taxonomy: human, code and LLM-judge evaluators exist; offline evaluation runs on a dataset before you deploy; online evaluation scores real production traces as they arrive, without reference answers.

Prompts and the Playground

Prompt text is the part of an LLM application that changes most often, and the part least suited to living inside source code, because a wording tweak then requires a code release. LangSmith lets you keep prompts as versioned objects and pull them at run time.

A prompt is a template with variables, either a chat template with system and user messages or a plain completion template. Every time you save a change it creates a commit, an immutable version identified by a hash. A commit tag is a movable label on a commit; staging and production are reserved names used for environments, so you can promote a version by moving the tag rather than changing code.

To create and use a prompt from code you need the langchain-core package for the template class, and the langsmith client to push and pull.

pip install -U langchain-core

from langchain_core.prompts import ChatPromptTemplate
from langsmith import Client

client = Client()

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a support agent. Answer in one sentence."),
    ("user", "{question}"),
])

# creates the prompt the first time, a new commit afterwards
url = client.push_prompt("support-answer-prompt", object=prompt)
print(url)

latest = client.pull_prompt("support-answer-prompt")
print(latest.invoke({"question": "How long do refunds take?"}))

You address a prompt by name, optionally followed by a colon and either a commit hash or a tag: support-answer-prompt, support-answer-prompt:12344e88, support-answer-prompt:production. A prompt you do not own and that has been shared publicly is addressed as owner/name. Pulling the tag production is how an application gets "the approved version" while a colleague experiments with newer commits in the interface.

Pulled prompts are cached in your process, on by default, with a five-minute time to live and a background refresh every minute. That is usually what you want, but it explains a puzzle for beginners: you edit a prompt in the interface and your running program seems to ignore it for a few minutes. Wait, restart, or pass skip_cache=True to pull_prompt while testing.

The Playground is the interface for trying a prompt without writing code. You open a prompt, choose a model, fill in variables, run it, and change the text and run again. It can also run your prompt over a dataset, which gives you an evaluation without a script. You supply the model provider's API key in the workspace settings as a secret, so that calls are made on your behalf. The Playground is the best place for a non-engineer, such as a product manager or a domain expert, to improve wording while you keep the evaluation rigour.

Configuration and privacy: what leaves your machine

Every trace carries the real inputs and outputs of your program. For a chatbot that means the user's messages; for a healthcare or HR assistant it may mean personal data. Decide what you send before you send it, not after.

The environment variables you will use most:

The simplest privacy control is hiding everything: with LANGSMITH_HIDE_INPUTS=true and LANGSMITH_HIDE_OUTPUTS=true, the structure, timings and errors are still recorded but the content is not. That is a good default while you work out a better policy, though it makes traces far less useful for debugging. The better middle path is redacting only sensitive fields, with a function applied per traced function:

import re
from langsmith import traceable


def scrub(inputs: dict) -> dict:
    text = inputs.get("message", "")
    text = re.sub(r"[\w.+-]+@[\w-]+\.[\w.]+", "[EMAIL]", text)
    return {**inputs, "message": text}


@traceable(process_inputs=scrub)
def handle(message: str) -> str:
    return "ok"

Two behaviours are worth knowing. A function-level process_inputs or process_outputs takes precedence over client-level settings. And since SDK 0.12 these redactors fail closed: if your redactor raises an error, the payload is replaced by a message such as Processing failed; inputs dropped (process_inputs: KeyError) instead of sending the unredacted data. That is the safe behaviour, and a trace showing it means your redaction function has a bug to fix, not a setting to disable.

For volume, sampling keeps only a fraction of traces. Setting LANGSMITH_TRACING_SAMPLING_RATE=0.25 records about a quarter of requests. Since SDK 0.12 sampling is deterministic. It is a cost lever for busy production services and unnecessary while you are learning.

If your key spans several workspaces, for instance an organization-scoped service key, LangSmith cannot guess which workspace you mean. Set LANGSMITH_WORKSPACE_ID to the workspace's UUID, or the requests fail with a 403.

For notebooks there is one more trap. The Python SDK caches environment variable reads for speed. If you change LANGSMITH_PROJECT or the key in a running Jupyter kernel and then see traces in the wrong place, clear the cache and reload your .env:

from dotenv import load_dotenv
from langsmith import utils

utils.get_env_var.cache_clear()
load_dotenv(override=True)

Common errors and how to read them

Most LangSmith failures are silent by design. Tracing runs in a background thread and is built never to crash your application, so when something is wrong you usually see a warning in the logs and no traces in the interface. Learn the handful of messages below and you will diagnose nearly everything.

LangSmithMissingAPIKeyWarning: API key must be provided when using hosted LangSmith API. The key is not set in the process that is running. Check echo $LANGSMITH_API_KEY in the same terminal, and in a notebook clear the environment cache as shown earlier.

LangSmithAuthError: Authentication failed for /runs/multipart, or a log line Failed to multipart ingest runs. An HTTP 401. The causes, in order of likelihood: the endpoint belongs to a different region than your organization, there is a trailing slash on LANGSMITH_ENDPOINT, the key was deleted or deactivated, or it is an old ls__ key. Deactivation takes up to a minute to apply because authentication is cached.

LangSmithUserError: This API key is org-scoped and requires workspace specification. You used an organization-wide service key. Set LANGSMITH_WORKSPACE_ID or pass Client(workspace_id=...).

LangSmithRateLimitError: Rate limit exceeded. An HTTP 429. The SDK retries with back-off, so isolated ones are harmless. Persistent ones mean you hit an hourly event or data limit, or the free tier's monthly cap of 5,000 traces. Sample your traces or raise the plan.

LangSmithConnectionError: Connection error caused failure to POST /runs. The SDK could not reach the server. Check the endpoint, your proxy and your network. If your employer inspects HTTPS traffic with its own certificate, trust that certificate authority in your environment instead of turning verification off. A variant says the content length exceeds the maximum size limit, which means a single payload is too large; send less data per run.

The traces never show up, but no error appears. The usual reason is that the process exited before the background batch was sent. Call Client().flush() at the end of scripts and tests. In LangChain code, call wait_for_all_tracers() from langchain_core.tracers.langchain. In serverless TypeScript, set LANGSMITH_TRACING_BACKGROUND=false.

Traces appear, but in the wrong project. Either LANGSMITH_PROJECT was not set where you think it was, or the notebook cache explains it. A project_name on a decorator also overrides the environment variable for that function.

A query returns only one day of data, or only IDs. Those are the defaults of client.runs.query, as described earlier.

A deprecation warning saying a method "will be removed after Jan 31, 2027". You used a legacy query method. It works for now. Plan to move to the runs, traces and threads methods.

Retention, limits and cost: the parts that surprise people

A tracing tool that costs nothing on day one can cost something by month three, so understand the levers early.

Traces have a retention period. On LangSmith Cloud, base traces are kept for 14 days, and extended traces for 180 days. The extended period was 400 days until 14 September 2026, when it was reduced for new traces. A trace is upgraded from base to extended in three situations: an online evaluator scores it, an automation rule that extends retention matches it, or your code sends feedback with extend_trace_retention=True. Upgrades cost more, and on new evaluators and rules the option is switched on by default, so read those settings when you create them. Experiments are created with extended retention.

Datasets are not affected by any of this: they are kept indefinitely. So the practice is to treat traces as temporary and promote the ones you care about into a dataset.

The Developer plan includes 5,000 base traces a month; Plus costs $39 per seat per month with 10,000 included. Beyond that you pay per trace. The documentation says to read per-trace prices from the pricing page, and this guide does not repeat them because they change. There are also hourly limits on the number of events and the volume of data, which differ per plan. You can set monthly usage limits per workspace, project or user, so a bug that loops cannot produce a surprise invoice.

Two technical limits matter to beginners. One trace can contain at most 25,000 runs. And the ingestion API has per-minute rate limits per key, which the SDK respects by batching up to 100 runs per call.

For a new project, three habits keep cost and risk low: use a separate project per application, so that you can see and delete traces by application; sample production traffic once it is high enough that storing everything is wasteful; and do not send data you would not want stored. The Senior guide treats cost governance as a platform topic.

Putting it all together: a small support bot, traced and tested

Now combine the pieces in one project you could show a colleague. The goal is a tiny support assistant that answers from a few documents, traces every step with tags and metadata, groups a conversation into a thread, records user feedback, and has a dataset and an evaluation that run before each change.

Set up the project layout:

mkdir support-bot && cd support-bot
python3 -m venv .venv && source .venv/bin/activate
pip install -U langsmith openai python-dotenv
printf 'LANGSMITH_TRACING=true\nLANGSMITH_PROJECT=support-bot\n' > .env
echo '.env' > .gitignore

Add your keys to .env by hand, so they never enter the shell history: LANGSMITH_API_KEY and OPENAI_API_KEY. Then the application:

import uuid
from dotenv import load_dotenv

load_dotenv()

from openai import OpenAI
from langsmith import Client, traceable
from langsmith.wrappers import wrap_openai

llm = wrap_openai(OpenAI())
ls_client = Client()

DOCS = {
    "refund": "Refunds are issued within 14 days of purchase.",
    "shipping": "Standard shipping takes 3 to 5 working days.",
    "warranty": "All devices have a one-year warranty.",
}


@traceable(run_type="retriever", name="lookup-docs")
def lookup(question: str) -> list[str]:
    return [t for k, t in DOCS.items() if k in question.lower()]


@traceable(name="support-answer", tags=["support"], metadata={"app_version": "1.0"})
def answer(inputs: dict) -> dict:
    question = inputs["question"]
    context = lookup(question)
    res = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system",
             "content": "Answer in one sentence using only this context: " + " ".join(context)},
            {"role": "user", "content": question},
        ],
    )
    return {"answer": res.choices[0].message.content}


def chat(questions: list[str]) -> None:
    thread_id = str(uuid.uuid4())
    for q in questions:
        out = answer(
            {"question": q},
            langsmith_extra={"metadata": {"thread_id": thread_id}},
        )
        print(q, "->", out["answer"])
    ls_client.flush()


if __name__ == "__main__":
    chat(["How long do refunds take?", "And the warranty?"])

And a separate script that turns the same function into a tested one:

from dotenv import load_dotenv

load_dotenv()

from langsmith import Client
from bot import answer

client = Client()
NAME = "support-bot-eval"

if not client.has_dataset(dataset_name=NAME):
    ds = client.create_dataset(dataset_name=NAME)
    client.create_examples(
        dataset_id=ds.id,
        examples=[
            {"inputs": {"question": "How long do refunds take?"},
             "outputs": {"answer": "14 days"}},
            {"inputs": {"question": "How long is the warranty?"},
             "outputs": {"answer": "one year"}},
        ],
    )


def mentions_reference(outputs: dict, reference_outputs: dict) -> dict:
    hit = reference_outputs["answer"].lower() in outputs["answer"].lower()
    return {"key": "mentions_reference", "score": int(hit)}


client.evaluate(
    answer,
    data=NAME,
    evaluators=[mentions_reference],
    experiment_prefix="v1",
    max_concurrency=2,
)

The workflow is then a loop you will repeat for the rest of your career with this tool. Run python bot.py and read the traces: does the retriever return the right documents, does the prompt contain them, is the answer faithful? Fix what you find, for example by making lookup smarter. Run python evaluate_bot.py and compare the new experiment with the previous one in the interface. When a real user reports a bad answer, open that trace, add it to the dataset, and the bug has become a permanent test. Change the app version in the metadata with each release, so that every trace records which code produced it.

What you can now do, and what comes next

You now hold the working vocabulary and the core loop of LangSmith. You can create an account and a key, choose the right regional endpoint, install the SDK, and send traces from Python or TypeScript using @traceable, traceable, the provider wrappers or a framework integration. You can read a trace, find the slow or failing step, filter with tags and metadata, group a conversation into a thread, and attach feedback. You can query traces from the SDK and the CLI, taking care of the defaults of the new query method. You can build a dataset, run an experiment and read the scores, version a prompt, and control what data leaves your machine. You know the error messages that matter and the limits that cost money.

Three things to carry forward. Tracing is cheap to add and expensive to lack, so instrument from the first day, not after the first incident. Metadata is what makes traces useful later, so decide your keys early. And evaluation is a loop, not an event: production failures feed the dataset, and the dataset guards the next release.

Here is what the next levels add. The Mid guide covers writing good evaluators including LLM judges, running evaluation in pytest and CI, online evaluation and automation rules, annotation queues, feedback from end users at scale, cost tracking for custom models, prompt caching, and tracing across services with OpenTelemetry and distributed tracing. The Senior guide covers running LangSmith as a platform for several teams: workspaces and access control, the SmithDB migration at scale, self-hosting on Kubernetes with Helm, security and data residency, cost governance and upgrade policy. Meanwhile, compare what you learned here with the Langfuse guide and the evaluation-metrics approach in the RAGAS guide, which together show which ideas are universal across LLM observability and evaluation tools.

Sources

• LangSmith documentation home
• Observability quickstart
• Observability concepts
• Create an account and API key
• Regions FAQ
• Threads
• Conditional tracing
• Sample traces
• Mask inputs and outputs
• Serverless environments
• Troubleshooting variable caching
• Evaluation concepts
• Evaluation quickstart
• Evaluate an LLM application
• Prompt engineering concepts
• Manage prompts programmatically
• LangSmith CLI
• SmithDB SDK migration
• Usage and billing
• LangSmith changelog
• Python SDK on PyPI
• JavaScript SDK on npm
• LangSmith SDK releases

This is part two of three. It prepares you to talk about LangSmith in an interview or a design review: what it is, how to explain it in half a minute, which commands and snippets to have ready, the traps that catch people out and why they happen, and the questions you are most likely to be asked with model answers. Read the detailed guide first, because an interview answer is only convincing when you have actually sent a trace and read it.

One honest framing helps throughout. LangSmith is a fast-moving commercial product, and interviewers know that. Nobody expects you to recite version numbers. They expect you to explain the concepts precisely, to show that you have used the tool, and to be straight about what you would check in the current documentation instead of bluffing. Saying "the query API changed in 2026 and I would check the migration page" is a stronger answer than a confident wrong one.

The thirty-second answer

LangSmith is a platform for observing, evaluating and managing LLM applications. You instrument your code with a decorator, a wrapper or a framework integration, and every step of a request, each model call, tool call and retrieval, is recorded as a run inside a trace, with inputs, outputs, timings, token counts and errors. On top of that you can score runs with feedback, save examples into datasets, run experiments that compare versions of your application, and version your prompts.

If you have a second breath, add the reason it matters: LLM applications are chains of non-deterministic steps, so when an answer is bad the question is always which step caused it, and a trace answers that in one click where logs would not.

Practise saying it out loud without the word "observability" once, because interviewers sometimes ask you to explain it to a non-technical colleague. "It is a flight recorder for an AI feature" is a fair version.

The model in one page

Have the nouns ready, with one sentence each. You should be able to produce this list without notes.

• Run: one unit of work, such as a model call or a tool call. The same thing as an OpenTelemetry span. It has a run type: llm, chain, tool, retriever, embedding, prompt or parser.
• Trace: all the runs of one operation, arranged as a tree with one root run. At most 25,000 runs per trace.
• Project: the container for the traces of one application, chosen with LANGSMITH_PROJECT, created automatically, default if unset. In the API it is called a session.
• Thread: a multi-turn conversation, made of several traces that share a thread_id or session_id metadata value.
• Feedback: a score, with a key, a number or category and an optional comment, attached to a run.
• Dataset, example, experiment, evaluator: the evaluation kit. Examples have inputs and reference outputs; an experiment is one version of your app run over a dataset; an evaluator turns an output into a score.
• Workspace and organization: the organization holds billing, and workspaces hold resources and keys. "Tenant" is the old name for a workspace.

The sentence that shows understanding is the relationship between them: a project holds traces, a trace is a tree of runs, feedback attaches to runs, and a dataset is built from examples that often come from traces.

The commands and snippets to have ready

An interviewer may hand you a laptop or a whiteboard and ask you to "add tracing to this function". These are the pieces you should be able to write from memory. First the setup:

pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_pt_..."
export LANGSMITH_PROJECT="my-app"
export LANGSMITH_ENDPOINT="https://eu.api.smith.langchain.com"   # only outside the US region

Then the instrumentation, which is two lines of idea:

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())


@traceable(run_type="chain", tags=["support"], metadata={"app_version": "1.0"})
def answer(question: str) -> str:
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return res.choices[0].message.content

For a call-time value: answer("hi", langsmith_extra={"metadata": {"thread_id": tid}}). For a short script: Client().flush() at the end. For a LangChain or LangGraph application: nothing, only the environment variables.

The evaluation skeleton, which you should be able to explain line by line:

from langsmith import Client

client = Client()
client.evaluate(
    target,                      # function: inputs dict -> outputs dict
    data="my-dataset",           # dataset name
    evaluators=[my_evaluator],   # functions returning a score
    experiment_prefix="v2",
    max_concurrency=2,
)

And the terminal commands, because showing that you can inspect from the command line is a sign of real use:

langsmith auth login
langsmith project list
langsmith trace list --project my-app --limit 5
langsmith trace get <trace-id> --project my-app --full
langsmith run list --project my-app --run-type llm

Know the defaults, since they are the classic source of confusion: trace list looks at the last seven days and returns 20 rows, and run list returns 50.

The classic traps and why they happen

These are the failures interviewers love, because each one reveals whether you understand the machinery or only copied a tutorial. For each, know the symptom, the cause and the reason the cause exists.

Traces are missing from a short script. The SDK sends runs from a background thread in batches, because blocking your application on a network call for every step would slow it down. A script that finishes before the batch is sent loses it. The fix is Client().flush(); in LangChain code wait_for_all_tracers(); in serverless TypeScript LANGSMITH_TRACING_BACKGROUND=false or awaitPendingTraceBatches(). Being able to say why the design exists earns more than the fix.

Authentication fails although the key is correct. Organizations live in one region and each region has its own API address. The default is the US address. A key from an EU organization sent to the US address is rejected. A trailing slash on the endpoint causes the same error. The reason is data residency: regions are isolated by design.

Everything is in the wrong project, or a change of settings has no effect in a notebook. Environment variable reads are cached by the SDK. In a notebook the cache survives until the kernel restarts, unless you clear it and reload the file.

The key is org-scoped and requests return 403. A service key scoped to a whole organization does not say which workspace you mean. Set LANGSMITH_WORKSPACE_ID.

Old traces seem to vanish. Base traces are kept for 14 days, extended ones for 180 days. The change to 180 days, from 400, took effect on 14 September 2026 for new traces. Datasets do not expire, so anything worth keeping should be copied into one.

A query returns only yesterday's data, or only IDs. The new runs.query defaults to a start time of one day ago and to selecting only id. You pass min_start_time and uppercase selects.

The bill is higher than expected. Extended retention is triggered by online evaluators, by automation rules that extend retention, and by feedback sent with extend_trace_retention=True. On new evaluators and rules the option is on by default.

A redactor that stops working and the payload says "Processing failed". Since SDK 0.12 per-function redactors fail closed, so an exception in your redaction code drops the payload instead of leaking it.

Common interview questions

What problem does LangSmith solve that logging does not? Logs are lines of text that you must correlate yourself. A trace is a structured tree for one request, with every step's inputs, outputs, timing, cost and error tied together, filterable by tags and metadata and linkable to a colleague. It also connects to evaluation: a failing trace can become a test case with one click.

Do you need LangChain to use LangSmith? No. LangChain and LangGraph trace with only environment variables, but plain Python and TypeScript work with @traceable and the provider wrappers, and there are integrations for many frameworks, plus OpenTelemetry for the rest.

What is the difference between a run and a trace? A run is one step. A trace is all the runs of one request, arranged as a tree that shares an ID. The root run is the top-level call and the others are its children.

How do you group a conversation? Put the same thread_id (or session_id) in the metadata of every run of every turn. Then the Threads view shows them together. I would generate a UUID at the start of a conversation and pass it through langsmith_extra.

What is the difference between tags and metadata? Tags are labels without a value; metadata are key-value pairs. Both are filterable. I use tags for categories such as support and metadata for facts such as app_version or user_id.

How do you add tracing to an existing OpenAI call? Wrap the client with wrap_openai, and decorate the functions around it with @traceable. The wrapper produces the llm run with token counts and cost and the decorator produces the parent runs. Tracing is activated by LANGSMITH_TRACING=true.

What is an evaluation, and how does it differ from monitoring? Monitoring watches production. Offline evaluation runs a fixed dataset through a new version of the application before release and scores it. Online evaluation scores live traces without reference answers. You want both: offline to prevent regressions and online to catch the things your dataset did not anticipate.

What goes in a dataset example? An inputs dictionary that matches what your application takes, usually a reference outputs dictionary, and optional metadata. Good examples often come from real production failures.

What kinds of evaluator are there? Human review, code functions, LLM-as-judge and pairwise comparison. A code evaluator is a function that returns a boolean, a number or a dictionary with a key and a score. LLM judges are better for free text, but they need calibrating against human judgement.

How do you manage prompts? Keep them as versioned prompts in LangSmith. Each save is an immutable commit, and commit tags such as production are movable labels. The application pulls name:production, so changing the live prompt is moving a tag, and a rollback is moving it back. Pulls are cached for five minutes by default, which is why an edit may not show up instantly.

How would you handle sensitive data? Decide at the source: redact in process_inputs, or set LANGSMITH_HIDE_INPUTS and LANGSMITH_HIDE_OUTPUTS to stop content leaving the process, choose a region that fits policy, sample, and keep keys out of code. For strict cases, self-host.

How do you use LangSmith from a script or an agent? The CLI for interactive work, for example langsmith trace list --project my-app --limit 5, and the SDK's client.runs.query(...) for code. I know the older list_runs is deprecated, to be removed after 31 January 2027 on Cloud.

What would you do first if a user reports a wrong answer? Find the trace, using the user ID in the metadata or the thread ID. Open the tree and find the failing step. Read the actual prompt the model received: nine times in ten either the retrieval was empty or the prompt was wrong. Then add the example to a dataset, fix it, and run the evaluation.

What would you not use LangSmith for? It is not a place to store application data, it is not a replacement for ordinary logs and metrics of your infrastructure, and it is a paid hosted service. If an organization cannot send data to a third party, self-hosting or an open-source alternative such as Langfuse enters the discussion.

Questions to ask back

An interview is a two-way conversation, and asking good questions shows that you think beyond the tutorial. Useful ones: which region and plan does the team use, and is there a data-handling policy for traces? Is tracing on in production, and at what sampling rate? How does the team turn production failures into dataset examples? Who owns the evaluators, and how are they validated against human judgement? How are prompts promoted from experiment to production? You are not testing them; you are showing the concerns a careful engineer has.

Sixty-second self-test

Cover the answers and try to answer each question aloud. If you hesitate on more than two, reread the matching section of the detailed guide.

1. Name the four nouns of tracing and say what each contains.
2. Which environment variable switches tracing on, and what is the default project name?
3. Why might a script lose its last traces, and what is the fix?
4. What would be wrong with an EU organization using the default endpoint?
5. What is the difference between tags and metadata?
6. How do you group three traces into one thread?
7. Which part of a run tells you how much a model call cost, and which wrapper produces it?
8. What do you pass to client.evaluate, and what does a target function receive?
9. How long are base and extended traces kept, and which three things extend a trace?
10. What are the two defaults that make a new runs.query call return surprising results?
11. How do you address the production version of a prompt?
12. Name two ways to stop personal data from being stored.

The answers, in order. A run is one step, a trace a tree of runs, a project a container of traces, and a thread a conversation of traces. LANGSMITH_TRACING=true, and default. The process exits before the background batch is sent; call Client().flush(). The key is rejected because the endpoint does not match the region; set LANGSMITH_ENDPOINT without a trailing slash. Tags are labels; metadata are key-values. Give every run the same thread_id in metadata. The llm run created by a wrapper such as wrap_openai, using the model name and token counts. A target function, a dataset name and evaluators; the target receives the example's inputs dictionary. 14 days and 180 days; online evaluators, retention-extending automation rules and feedback with extend_trace_retention=True. min_start_time defaults to one day ago and selects defaults to only id. name:production. Redact with process_inputs and hide inputs and outputs with the LANGSMITH_HIDE_* variables, or sample and self-host.

If you can give those twelve answers without looking, you can hold a sensible conversation about LangSmith with any interviewer. The next file, Tips and Tricks, collects the mistakes that beginners actually make in their first weeks and the habits that avoid them. After that, the Mid-level guide takes evaluation and production use further.

A worked example to rehearse

Interviewers often turn a question into a small scenario. Practise this one aloud, because it touches every concept above. The scenario: "Our support bot answered a customer that refunds take 30 days. The policy says 14. How do you find out what happened and stop it happening again?"

A strong answer has five moves. First, locate the evidence: search the project for the customer's thread by the thread_id or user_id in metadata, and open the exact trace. Second, walk the tree from the root downwards. Look at the retriever run: did it return the refund document? If the list is empty, the model improvised, and the fault is retrieval. If the document was returned and says 14 days, open the llm run and read the system prompt the model actually received, because the fault may be a prompt that let the model ignore its context. Third, form a hypothesis and change one thing, either the retriever or the prompt, and test it in the Playground or a script. Fourth, protect the fix: add this input, with the correct reference answer, to a dataset, and run an experiment comparing the old and new versions, so you know the fix did not break shipping questions. Fifth, prevent the next occurrence in production: add a feedback signal or an online evaluator that flags answers contradicting policy, sample the traffic if volume is high, and set the metadata app_version so the next regression can be tied to a release.

What makes the answer strong is the order and the restraint. You start from evidence instead of guessing, you change one variable at a time, and you finish with a test and a monitor rather than a one-off patch. What makes it weak is jumping straight to "I would change the prompt" or "I would use a better model" without opening the trace. If the interviewer pushes on cost, mention that you would not turn on extended retention for everything, only for the traces you deliberately want to keep.

A second scenario to rehearse is the privacy version: "Legal says we cannot store customers' phone numbers outside our country. Can we still use LangSmith?" The honest answer has layers. Redact the numbers before they leave the process with a process_inputs function, verify the redaction by reading real traces, and remember that redactors fail closed. Hide whole inputs and outputs where redaction is not reliable. Confirm the region, since there is no Middle East region at the time of writing and an organization cannot move between regions. And if the policy forbids any external processing, raise self-hosting as an Enterprise option or evaluate an open-source alternative, instead of pretending a setting solves it. Interviewers reward candidates who say where a tool's limits are.

This is part three of three. The detailed guide taught you how LangSmith works and the interview guide taught you how to talk about it. This file is the shortest route to avoiding the mistakes that cost beginners an afternoon. Each mistake is written the same way: the symptom you will see, the cause underneath, and the fix. After the mistakes come the habits worth adopting on day one, a few practical recipes, and a checklist you can paste into your own notes.

Most LangSmith problems are quiet. Tracing is built never to crash your application, so when it breaks you rarely get an exception. You get a program that works perfectly and an interface that shows nothing. That is why a list of symptoms is more useful here than in most tools: the skill is recognising the quiet failure quickly.

Mistakes that make traces disappear

Mistake: the script ends and the last traces are missing. Symptom: your program ran, printed its answer, and the project shows nothing or only the first few traces. Cause: the SDK sends runs from a background thread in batches. A short-lived process can exit before the final batch leaves. Fix: call Client().flush() at the end of scripts, tests and notebooks cells that finish quickly. For LangChain callbacks use wait_for_all_tracers() from langchain_core.tracers.langchain. In TypeScript on a serverless platform, set LANGSMITH_TRACING_BACKGROUND=false or call await client.awaitPendingTraceBatches() before returning.

Mistake: forgetting the master switch. Symptom: the key is right, the code is decorated, and nothing arrives. No warning either. Cause: LANGSMITH_TRACING is not true in the process that runs the code. A variable exported in one terminal is not visible in another, an IDE run configuration or a Docker container. Fix: print it from inside the program: import os; print(os.getenv("LANGSMITH_TRACING")). If it prints None, the environment did not reach the process. For containers pass it with -e or an env file.

Mistake: using the wrong region endpoint. Symptom: a log line such as Failed to multipart ingest runs or a LangSmithAuthError with a 401, although the key is freshly created. Cause: your organization is in the EU, APAC or AWS region and the SDK is talking to the default US address. A trailing slash on the endpoint produces the same failure. Fix: set LANGSMITH_ENDPOINT to the exact regional address from the docs, with no trailing slash, and check which host your browser uses: eu.smith.langchain.com goes with https://eu.api.smith.langchain.com.

Mistake: changing settings in a notebook and seeing no effect. Symptom: you change LANGSMITH_PROJECT or the key, re-run the cell, and traces still go to the old place. Cause: the SDK caches environment reads, and the kernel keeps the cache for its whole life. Fix: run from langsmith import utils; utils.get_env_var.cache_clear() and load_dotenv(override=True), or restart the kernel.

Mistake: a key that works in the browser but not in code, or the reverse. Symptom: a 403 saying the key is org-scoped and requires workspace specification. Cause: an organization-wide service key cannot guess the workspace. Fix: set LANGSMITH_WORKSPACE_ID to the workspace UUID, or use a personal access token for your own scripts.

Mistakes that make traces useless

Mistake: anonymous runs. Symptom: the trace list is full of runs called run, wrapper or RunnableSequence, and you cannot tell which is which. Cause: the run name defaults to the function name, and lambdas and generic helpers have unhelpful names. Fix: name things on purpose with @traceable(name="support-answer"), and use descriptive function names for the ones that matter.

Mistake: tracing too little. Symptom: the trace shows one model call and nothing else, so you cannot tell whether a bad answer came from retrieval, the prompt or the model. Cause: only the provider client is wrapped, and your own functions are invisible. Fix: decorate the functions that make decisions: the retriever with run_type="retriever", tools with run_type="tool", and the top-level handler. Children nest automatically.

Mistake: tracing too much. Symptom: a trace with thousands of tiny runs, a cluttered interface and a bill that grows faster than traffic. Cause: decorating every helper, or tracing inside a hot loop. Fix: trace at the level of meaningful steps. A rule of thumb: if you would not want to see the run in the tree when debugging, do not decorate it. Remember the cap of 25,000 runs per trace.

Mistake: no metadata, then a question you cannot answer. Symptom: someone asks "did the error rate change after last Tuesday's release?" and the traces carry no version. Cause: metadata is added later than it is needed, and stored traces cannot be edited retroactively in bulk. Fix: from the first day, put app_version, environment and, where allowed, a pseudonymous user_id in metadata. Use langsmith_extra for values known only at call time.

Mistake: conversations that never group into threads. Symptom: a chatbot produces hundreds of separate traces and the Threads tab is empty. Cause: no thread_id or session_id metadata, or it was put on the parent run only. Fix: generate one identifier per conversation, and set it on every run including child runs, using the same key.

Mistake: metadata versus tags mixed up. Symptom: you cannot filter by the thing you care about because you stored a value as a tag. Cause: a tag has no value, so user_id as a tag means a different tag for each user. Fix: a category is a tag and a fact with a value is metadata.

Mistakes with privacy and keys

Mistake: an API key in the source file. Symptom: the key appears in Git history, a screenshot or a shared notebook. Cause: pasting a key for a quick test. Fix: keys live in the environment or a .env file in .gitignore. If one leaks, delete it in Settings immediately, since a key can read your traces, and create another. Prefer service keys with an expiry for applications.

Mistake: personal data in traces. Symptom: a colleague opens a trace and sees a real customer's email address or national ID. Cause: tracing records whatever your function received. Fix: redact at the source. A process_inputs function on the traced function can replace patterns, and LANGSMITH_HIDE_INPUTS=true and LANGSMITH_HIDE_OUTPUTS=true remove content entirely. Decide before launch, and for employers in the Gulf or Egypt check the data-residency policy before choosing the region.

Mistake: a redactor that raises an exception. Symptom: payloads read Processing failed; inputs dropped (process_inputs: KeyError). Cause: since SDK 0.12 redactors fail closed. A KeyError inside your function drops the data rather than sending it raw. Fix: make the redactor tolerant, for example with inputs.get("message", ""), and test it with awkward inputs. Do not enable LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS unless you accept that raw data may be sent.

Mistake: disabling certificate checks behind a company proxy. Symptom: LangSmithConnectionError or SSL errors on a corporate laptop, and a search result telling you to set verification to false. Cause: your network inspects HTTPS traffic with its own certificate authority, which Python does not trust by default. Fix: give Python the company's root certificate through SSL_CERT_FILE or REQUESTS_CA_BUNDLE. Never turn verification off; you would be sending prompts over an unverified connection.

Mistakes with queries, feedback and cost

Mistake: a query that seems to lose old data. Symptom: client.runs.query(...) returns only today's traces, or a list of bare IDs. Cause: min_start_time defaults to one day ago and selects defaults to only id. Fix: pass min_start_time explicitly, pass selects=["NAME", "STATUS", "INPUTS"] in uppercase, and pass project_ids, since project_name was removed.

Mistake: learning from outdated tutorials. Symptom: DeprecationWarning-style messages saying a method will be removed after Jan 31, 2027, or code using LangChainTracer, ls__ keys or langchainhub. Cause: the product and SDK have changed. The legacy list_runs and read_run are deprecated, ls__ keys stopped working in October 2024, and langchainhub is deprecated in favour of the langsmith package. Fix: check the date of a tutorial, and trust the current docs and this guide's version notes over a blog post.

Mistake: feedback without a project ID. Symptom: a warning that creating feedback for a run without session_id is deprecated, or later a ValueError. Cause: the newer datastore cannot find a run without knowing its project. Fix: read the project once with client.read_project(project_name=...) and pass session_id=project.id to create_feedback.

Mistake: an evaluator that scores everything 1. Symptom: your experiment shows perfect scores and the application is obviously imperfect. Cause: a check so loose it always passes, for example testing whether the answer is non-empty. Or a dataset with only easy examples. Fix: add a failing example on purpose and check that the evaluator catches it. Read a few traces by hand against their scores before trusting the numbers.

Mistake: experiments that cannot be compared. Symptom: two experiments with different scores, and no way to say which code produced which. Cause: no experiment prefix and no metadata. Fix: use experiment_prefix="v2-shorter-prompt" and pass metadata={"git_sha": "..."} to evaluate.

Mistake: an edited prompt that seems to be ignored. Symptom: you saved a new prompt version and your running app still behaves as before. Cause: the prompt cache, on by default, keeps pulled prompts for five minutes and refreshes every minute. Fix: wait, restart, or pass skip_cache=True to pull_prompt while testing.

Mistake: a surprise bill. Symptom: usage climbs faster than traffic. Cause: online evaluators or automation rules that extend retention, which costs more per trace and is on by default for new rules, or feedback sent with extend_trace_retention=True on every call. Fix: open the settings of each evaluator and rule and untick extension unless you need it, set a monthly usage limit on the workspace, and sample production traffic.

Mistake: expecting old traces to stay. Symptom: a bug report refers to a trace from last month and it is gone. Cause: base retention is 14 days. Fix: add interesting traces to a dataset, which never expires, or deliberately extend retention for them.

Habits worth adopting on day one

Give every application its own project. The project is the unit you filter, sample, delete and reason about. One shared default project for five applications is a mess you will spend a week untangling.

Separate environments with metadata or projects. Put environment: dev, staging or prod in metadata or use distinct projects. Developers' experiments should never drown production traces.

Pin your versions. The SDK ships several times a week. In requirements.txt write a pinned version such as langsmith==0.14.2 and upgrade deliberately with a changelog read. Check the version in use with python -c "import langsmith; print(langsmith.__version__)".

Turn failures into tests. Every time a user reports a bad answer, add the trace to a dataset. Within a month you have a regression suite drawn from reality instead of imagination.

Read traces by hand before building automation. Spend an hour reading twenty real traces. You will learn more about your application's failure modes than any dashboard shows, and you will know what evaluators to write.

Keep prompts out of code. Once a prompt changes more than twice, move it to a versioned prompt and pull name:production. Code releases and wording changes then have separate lifecycles.

Turn tracing off in unit tests. Leave LANGSMITH_TRACING unset in ordinary tests so they do not send data or depend on the network. Use the pytest integration deliberately when you want tests to record results.

Watch the changelog. LangSmith has a public changelog, and Cloud changes weekly. Deprecated endpoints carry Deprecation: true and Sunset headers, and the SDK warns. Deprecations on Cloud are supported for at least six months, so you have time, but only if you notice.

Practical recipes

Send traces from a Docker container. Pass the variables at run time instead of baking them into the image:

docker run --rm \
  -e LANGSMITH_TRACING=true \
  -e LANGSMITH_API_KEY \
  -e LANGSMITH_PROJECT=support-bot \
  my-image:latest

Passing -e LANGSMITH_API_KEY with no value forwards the host's variable, so the key never appears in your shell history or the image layers. The Docker guide explains why that matters.

Switch tracing off for one block. When a section of code handles data you do not want stored:

import langsmith as ls

with ls.tracing_context(enabled=False):
    sensitive_work()

Sample a busy service. Keep a quarter of requests with no code change:

export LANGSMITH_TRACING_SAMPLING_RATE=0.25

Look at the last failures from the terminal. Quick triage without opening the browser:

langsmith run list --project support-bot --error --limit 10
langsmith trace get <trace-id> --project support-bot --full

Pull a prompt by tag. The application code that separates wording from releases:

from langsmith import Client

prompt = Client().pull_prompt("support-answer-prompt:production")

Name experiments so they sort well. Use a short convention such as v3-fewer-examples so the experiment list reads like a changelog, and always record the Git commit in metadata.

A short checklist

Paste this into your project's README and tick it before you call an LLM feature done.

• Keys are in the environment or an ignored .env, never in code or Git.
• LANGSMITH_ENDPOINT matches the organization's region, with no trailing slash.
• Each application has its own project, named for the application.
• Traced functions have meaningful names and sensible run types.
• Every trace carries app_version and environment metadata.
• Conversations carry a thread_id on every run.
• Personal data is redacted or hidden before it leaves the process.
• Scripts and tests call Client().flush(), and serverless code sends traces before returning.
• The SDK version is pinned and the changelog is read before upgrades.
• There is a dataset with at least ten real examples and an evaluation that runs before releases.
• Evaluators and rules have been checked for retention extension, and a monthly usage limit is set.
• Prompts that change often live in versioned prompts, pulled by tag.
• Queries pass min_start_time, selects and project_ids, and avoid deprecated methods.

Do these thirteen things and you are ahead of most teams using the tool. The Mid-level guide starts from here: writing trustworthy evaluators, running evaluation in CI, scoring live traffic, and wiring tracing across services. If something in this file surprised you, go back to the matching section of the detailed guide and run the Try it task again; the failures are far easier to recognise once you have caused each of them yourself.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. You already send traces, so the questions now are the ones a team asks a few months in: why is a trace missing, why is the bill higher than the forecast, how do we stop regressions before they ship, and what do we do before a deprecated API disappears.

One point before we start, because it shapes how you read everything else. LangSmith has no single product version. There is a Python SDK (langsmith 0.14.2 at the time of writing), a JavaScript SDK (0.10.5), a command-line tool (langsmith-cli v0.2.60), and, if your employer runs it themselves, a Helm chart (stable v16, with a v17 preview channel). The SDK ships several releases a week. Pin what you depend on, upgrade deliberately, and read release notes rather than trusting a number in a tutorial, including this one.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a real project in a few minutes. Code is Python unless stated, and the TypeScript equivalents are noted where they differ in a way that matters.

The data model, precisely

Beginner gave you the nouns. To predict behaviour you need their exact definitions, because most surprises at work come from one of them meaning something slightly different from what you assumed.

A run is one unit of work: an LLM call, a tool call, a retrieval, a prompt format. It is the same idea as an OpenTelemetry span. Every run has a run_type, which is one of llm, chain, tool, retriever, embedding, prompt or parser. The default for @traceable is chain. The type is not decoration. The UI renders each type differently, cost and token tracking apply to llm runs, and filters such as "all LLM runs slower than five seconds" depend on it being set honestly.

A trace is every run that belongs to one operation, joined by a trace ID and shaped as a tree. The root run is the entry point you decorated or the chain you invoked, and its children are everything it called. There is a hard cap of 25,000 runs per trace. Runs past that are rejected. A long-running agent that loops for hours inside one root run will hit it, and the fix is structural rather than a setting: end the trace at a natural boundary and group the pieces as a thread.

A thread is a sequence of traces that form one multi-turn conversation. One turn is one trace. You group traces into a thread by putting a metadata key, either session_id or thread_id, on the runs, and the docs recommend a UUID v7 as the value. The detail that catches people: the key must be on all runs, including child runs, not just the root. A thread ID on the root alone is a common cause of "my conversation shows up as six unrelated traces".

A project, called a tracing project in the docs, is the container for the traces of one application or service. You choose it with LANGSMITH_PROJECT, and it is created on first ingestion. If you set nothing, traces go to a project literally named default. Here is the naming trap you will meet as soon as you read API code: in the REST API and in method parameters a project is called a session. So session_id in a feedback call means the project UUID, not a chat session, and it is unrelated to the thread key that happens to share the word.

Feedback is a score attached to a run. It has a key (the name of the metric, such as correctness), a numeric score or a categorical value, and an optional comment. It can come from your code, from a person clicking in the UI, from an annotation queue, or from an evaluator. Tags are strings on a run and metadata is a dictionary on a run. Both are filterable, and the difference in practice is that tags are for coarse labels (prod, experiment-7) while metadata carries identifiers (user_id, git_sha, thread_id).

Finally the organisational hierarchy, which matters as soon as two teams share a tenant. An organisation contains workspaces. You will see the older word "tenant" in API paths (/api/tenants) and in the X-Tenant-Id header, and it means workspace. Resources such as projects, prompts, datasets and annotation queues live in a workspace. API keys, secrets, automation rules and feedback configuration are workspace-scoped. Billing is organisation-scoped. Applications, a newer grouping, are built on resource tags and let you slice resources within a workspace.

How a trace leaves your process

The problem this section solves: tracing is supposed to be free for the application, yet runs go missing in scripts and serverless functions, and latency sometimes changes after turning tracing on. Both come from the same design, and it is worth understanding once.

The Python SDK does not send each run to the network as it happens. When a traced function starts, the SDK records a post operation for the run. When it ends, it records an update. These operations go into a queue, and a background thread drains the queue, batches the operations, compresses them, and sends them to the ingestion endpoint (/runs/multipart). A batch carries up to 100 runs. Your application code never waits on the network, which is the whole point, and it is also the source of the classic failure: if the process exits before the background thread has sent what is queued, the runs are lost without an error.

your function ── post/patch ops ──▶ in-memory queue ──▶ background thread ──▶ /runs/multipart
                                          │                   │
                                     batched, up to       zstd-compressed,
                                     100 runs per call    retried on 429/5xx

Long-lived services drain the queue continuously, so you rarely notice it. The cases that do lose data are short-lived processes: a one-off script, a notebook cell that finishes, a cron job, an AWS Lambda or other serverless function that freezes the process the moment it returns. The remedies are explicit flushes:

from langsmith import Client, traceable

client = Client()

@traceable
def run_job(payload: dict) -> dict:
    return {"ok": True, "items": len(payload["items"])}

try:
    run_job({"items": [1, 2, 3]})
finally:
    client.flush()        # block until queued runs are sent

For LangChain and LangGraph code the equivalent is from langchain_core.tracers.langchain import wait_for_all_tracers; wait_for_all_tracers(). In JavaScript, call await client.awaitPendingTraceBatches() before the handler returns, or set LANGSMITH_TRACING_BACKGROUND=false so tracing completes inline in serverless environments. For extremely high concurrency in JS you can set manualFlushMode: true on the client and call await client.flush() yourself.

Two behaviours of the batching pipeline changed recently and are worth knowing. Since Python SDK 0.11.0, the SDK no longer re-sends a run's inputs on the final update. LANGSMITH_EXCLUDE_INPUTS_ON_PATCH defaults to true, so if your code sets inputs after the initial post, for example by mutating a run tree late, those late inputs are not uploaded. If you see a run with empty inputs after upgrading across 0.11, this is the first thing to check. Compression is zstd with one worker thread by default, which is a small CPU cost traded for much smaller payloads.

What if ingestion fails? The SDK logs the failure (for example Failed to multipart ingest runs: ...) and retries transient errors, but it does not raise into your application, because tracing must never break the request. That is a good default and a debugging hazard: a wrong key or endpoint produces log lines, not exceptions. You can pass tracing_error_callback to the client to handle these in code, and you should watch for the log lines in any new environment.

One more caching trap sits here. The SDK reads environment variables through a cached function. In a notebook, if you change LANGSMITH_PROJECT or the API key after the first traced call, nothing changes, which looks exactly like "traces go to the wrong project". Clear the cache:

from dotenv import load_dotenv
from langsmith import utils

load_dotenv(override=True)
utils.get_env_var.cache_clear()

Configuration, precedence and the client

The problem: the same code runs on a laptop, in CI and in production, and each needs different destinations, sampling and privacy settings. You want the code unchanged and the environment to decide.

The SDK reads LANGSMITH_<NAME> first and falls back to the legacy LANGCHAIN_<NAME>. Teach and write the LANGSMITH_ form; the old prefix still works, which is why an inherited codebase with LANGCHAIN_TRACING_V2=true keeps tracing. The variables you will use at this level:

Regions deserve a paragraph because they cause a large share of authentication failures. LangSmith Cloud runs in separate regions, each with its own API host: https://api.smith.langchain.com (GCP US, the default), https://eu.api.smith.langchain.com, https://apac.api.smith.langchain.com, and https://aws.api.smith.langchain.com. An organisation lives in exactly one region and cannot be moved or span regions. A key created on the EU site does not work against the US endpoint, and the error you get is an authentication failure, not "wrong region". For employers in the Gulf and Egypt, note what this means for data residency: the regions offered are in the US, EU and Asia-Pacific, and there is no Middle East region, so a residency requirement that insists on in-country storage is a reason to ask about self-hosting or BYOC, both Enterprise options, rather than something you can configure.

Beyond environment variables, the code-level controls have a precedence order you should memorise. The tracing_context context manager wins over a client or decorator setting, which wins over the environment variable. That is what makes per-request overrides safe:

import langsmith as ls
from langsmith import Client, traceable

@traceable
def answer(question: str) -> str:
    return "..."

# Turn tracing off for one code path, even though LANGSMITH_TRACING=true
with ls.tracing_context(enabled=False):
    answer("health check ping")

# Send one path to a different project with a dedicated client
other = Client(tracing_sampling_rate=0.25)
with ls.tracing_context(client=other, project_name="canary", enabled=True):
    answer("a question")

# Process-wide defaults once, at startup
ls.configure(project_name="my-app", tags=["prod"], metadata={"service": "qa-api"})

Sampling has one property worth knowing: since Python SDK 0.12.0, trace sampling is deterministic. The decision is derived from the trace rather than a fresh coin flip per run, so a sampled trace is kept whole instead of arriving with random holes. That release was marked breaking for this reason, and it means a sampling rate of 0.1 yields roughly one trace in ten rather than ten percent of the runs of every trace.

The Client constructor is the place for anything that should not be global. In 0.14.2 the parameters include api_url, api_key, workspace_id, timeout_ms, retry_config, auto_batch_tracing, anonymizer, hide_inputs, hide_outputs, hide_metadata, tracing_sampling_rate, tracing_mode, tracing_error_callback, disable_prompt_cache and headers. Two arguments are deprecated and you will meet them in older code: otel_enabled (use tracing_mode) and cache (use configure_global_prompt_cache() or disable_prompt_cache=True).

Instrumentation patterns that stay useful

The problem: auto-tracing gets you a first trace, but a tree of anonymous chain runs is hard to query. The work at this level is deciding what you will want to filter by in six months and putting it on the runs now.

Start with @traceable and its arguments. In 0.14.2 the keyword arguments are run_type, name, metadata, tags, client, reduce_fn, project_name, process_inputs, process_outputs, process_chunk, enabled, exceptions_to_handle and dangerously_allow_filesystem. At call time you can add per-invocation data through langsmith_extra, which accepts metadata, tags, project_name, run_id and parent. The split is deliberate: static facts about a function go on the decorator, facts about this particular call go in langsmith_extra.

import uuid
import langsmith as ls
from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI

llm = wrap_openai(OpenAI())

@traceable(run_type="retriever", name="search-docs")
def retrieve(query: str) -> list[dict]:
    return [{"page_content": "LangSmith traces are trees of runs.", "type": "Document"}]

@traceable(name="qa-pipeline", tags=["qa"])
def answer(question: str) -> str:
    docs = retrieve(question)
    context = "\n".join(d["page_content"] for d in docs)
    resp = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Use only this context:\n{context}"},
            {"role": "user", "content": question},
        ],
    )
    return resp.choices[0].message.content

run_id = uuid.uuid4()
answer(
    "What is a trace?",
    langsmith_extra={
        "run_id": run_id,
        "metadata": {"user_id": "u-42", "git_sha": "a1b2c3d", "thread_id": "t-7"},
    },
)
print(run_id)   # keep this if you will attach feedback later

Three habits in that example carry most of the value. Name your runs so the tree reads like your architecture, not like function addresses. Set run_type honestly, because retrievers and tools get specialised rendering and because it is a filter dimension. Choose your own run_id when you will need to refer to the run afterwards, for example to attach user feedback from a later HTTP request, since it saves a lookup.

Wrappers (wrap_openai, wrap_anthropic, wrap_gemini) trace model calls as llm runs automatically and capture token usage. They compose with @traceable: the wrapped call appears as a child of whatever traced function is active. In TypeScript the equivalents are traceable from langsmith/traceable and wrapOpenAI from langsmith/wrappers. LangChain and LangGraph need no code at all, only the environment variables, and they tag runs by component automatically.

Streaming needs one argument. When a traced function returns a generator, LangSmith records the output once the stream ends; reduce_fn tells it how to combine the chunks into the final recorded output, and process_chunk can reshape each chunk before it is aggregated. Without a reducer you get a list of raw chunks, which is hard to read and to evaluate.

A second pattern is the trace context manager for code that is not a function you can decorate, such as a block inside a larger function, and the RunTree API for full manual control. Reach for them when a third-party library blocks decoration or when you need to set outputs explicitly. They produce the same runs; only the way you open and close them differs.

The metadata vocabulary is your real design decision. A set that works for most teams: user_id or a hashed user identifier, thread_id, git_sha or app_version, environment, tenant if you serve several customers, and feature_flag values that changed behaviour. With these, "show me errors from version a1b2c3d for one customer" is a filter, not an investigation. Keep personal data out of metadata, because redaction of metadata is separate from inputs and outputs (next section).

Redaction and what leaves your network

The problem: traces contain the prompt and the answer, which for many applications means customer names, account numbers and free-text messages. At work, someone will ask what LangSmith is storing, and "everything you sent" is not an acceptable answer. Redaction happens in your process, before the data is sent, which is the only place it is reliable.

There are three layers, from coarse to fine.

The coarsest is the environment: LANGSMITH_HIDE_INPUTS=true, LANGSMITH_HIDE_OUTPUTS=true and LANGSMITH_HIDE_METADATA=true replace the whole payload. You keep the structure, timing, token counts and errors, which is often exactly the observability you wanted for a sensitive workload, and you lose the ability to debug by reading prompts.

The middle layer is client-level: a callable passed as hide_inputs or hide_outputs, or a rule-based anonymizer built with create_anonymizer. Anonymizers match patterns and replace them:

import re
from langsmith import Client
from langsmith.anonymizer import create_anonymizer

anonymizer = create_anonymizer([
    {"pattern": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "replace": "<email>"},
    {"pattern": re.compile(r"\b\d{10,16}\b"), "replace": "<number>"},
])

client = Client(anonymizer=anonymizer)

The finest layer is per function: @traceable(process_inputs=..., process_outputs=...). These let you keep the question and drop an attached document, or hash a user ID. They take precedence over the client-level settings, and this is where a recent behaviour change bites. Since Python SDK 0.12.0, per-function redactors fail closed. If your redaction function raises an exception, the payload is not sent raw; it is replaced with a message of the form Processing failed; inputs dropped (process_inputs: KeyError). Before 0.12 a broken redactor silently sent the unredacted data, which is the dangerous direction to fail. So a trace full of those messages means your redactor has a bug, and the fix is the redactor. The environment variable LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS restores the old behaviour, and you should treat it as a last resort that needs a written justification.

from langsmith import traceable

def scrub_inputs(inputs: dict) -> dict:
    cleaned = dict(inputs)
    cleaned.pop("customer_document", None)       # never upload the attachment
    if "user_email" in cleaned:
        cleaned["user_email"] = "<redacted>"
    return cleaned

@traceable(process_inputs=scrub_inputs)
def review(customer_document: str, user_email: str, question: str) -> str:
    return "..."

Test redactors like any other code, because a silent gap is a compliance problem and a loud crash is a debugging problem. The docs also show recipes that use Microsoft Presidio or Amazon Comprehend to detect personal data, which are worth a look when regular expressions stop being enough, but they add latency inside your request path, so measure it.

Querying runs, and the SmithDB migration

The problem: you export runs to analyse failures, build a dashboard, or copy production examples into a dataset. The methods most tutorials use for this are deprecated, and they stop working. This is the single most important dated fact in the guide.

In May 2026 LangSmith released SmithDB, a new columnar datastore that replaces ClickHouse as the query engine for runs, traces and threads. The legacy query methods are deprecated on all Cloud regions from the end of July 2026 and are removed on 31 January 2027. On self-hosted the deprecation starts in v0.16 and the removal is in v0.18. The Python SDK emits the warning "<method>() is deprecated and will be removed after Jan 31, 2027." when you call one. The methods affected are read_run, read_thread, list_runs, list_threads, get_run_url, share_run, unshare_run, read_run_shared_link, read_shared_run and list_shared_runs. The new methods need Python langsmith>=0.10.15 or TypeScript langsmith>=0.8.9, and, on self-hosted, version 0.16 or later.

The mapping you will use most:

It is not a rename. The parameters changed in ways that return wrong data silently if you only change the method name. Learn these as a list:

• project_name is gone. Pass project_ids=[uuid]. Resolve the UUID with client.read_project(project_name=...).
• start_time became min_start_time, and it defaults to one day ago. A query with no time bound quietly covers only the last 24 hours. max_start_time defaults to now.
• select became selects, with uppercase names such as "NAME", "STATUS" and "INPUTS". The default is to return only the id.
• run_type values are uppercase: "LLM", "CHAIN", "TOOL", "RETRIEVER", "EMBEDDING", "PROMPT", "PARSER".
• error became has_error, run_ids became ids, reference_example_id became reference_examples (a list).
• limit was removed. Use page_size (default 100, maximum 1000) with cursors. query and parent_run_id were removed with no equivalent. The filter, trace_filter, tree_filter and is_root parameters are unchanged.

from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()
project = client.read_project(project_name="my-app")
since = datetime.now(timezone.utc) - timedelta(days=7)

rows = []
for run in client.runs.query(
    project_ids=[project.id],
    min_start_time=since,
    selects=["NAME", "STATUS", "INPUTS"],
    filter='and(eq(run_type, "llm"), gt(latency, 5))',
    page_size=500,
):
    rows.append(run)
print(len(rows), "slow LLM runs in the last 7 days")

The filter language itself did not change. It is a small functional expression syntax with eq, neq, gt, gte, lt, lte, has, search, and and or, plus trace_filter (select runs whose trace matches) and tree_filter (select runs whose tree contains a match). That lets you say things like "root runs that received negative feedback and contain a slow retriever", which the UI filter builder compiles into the same syntax. A useful trick is to build a filter in the UI, then copy the generated expression into code.

Two related changes land in the same migration. Creating feedback on a run should now pass session_id, the project UUID; without it the SDK warns Creating feedback for a run without session_id is deprecated and will stop working in a future release., and on SmithDB-only deployments it raises ValueError: session_id must be provided when creating feedback for a run: this deployment cannot locate the run without it. And add_runs_to_annotation_queue should receive runs=[{"run_id", "session_id", "start_time"}] rather than bare run_ids. Python also gains a new exception hierarchy for the new methods: LangsmithError (note the casing), AuthenticationError, RateLimitError, NotFoundError, ConflictError, APITimeoutError, APIConnectionError and InternalServerError, among others. The old LangSmith... classes in langsmith.utils are still raised by the old methods, so catch both while you migrate.

The REST side moves too: POST /api/v1/runs/query becomes POST /api/v2/runs/query, with /api/v2/traces/query and /api/v2/threads/query alongside. The command-line tool supports the new methods from v0.2.44.

How do you find what to migrate? Search your repositories for the deprecated names and run your test suite with warnings as errors for the langsmith module:

grep -rnE "\.(list_runs|read_run|list_threads|read_thread|get_run_url|share_run|unshare_run|get_experiment_results)\(" --include="*.py" .
python -W "error::DeprecationWarning" -m pytest tests/ -x

Datasets as a maintained asset

The problem: a prompt change feels better on the three examples you tried. A dataset replaces the feeling with a number, but only if it resembles real traffic and someone keeps it current.

A dataset is a collection of examples. An example has inputs (a dictionary), optional reference outputs (a dictionary) and optional metadata. Examples can belong to splits, named subsets such as train, test or hard, so that you can run a fast smoke evaluation on a small split and the full one nightly. Datasets never expire, which matters because traces do: base traces are kept for 14 days and extended traces for 180 days, so a dataset is the only durable copy of a production failure. Copy interesting traces into a dataset while they still exist.

from langsmith import Client

client = Client()
ds = client.create_dataset(
    dataset_name="support-qa",
    description="Questions from support chats, with reference answers",
)
client.create_examples(
    dataset_id=ds.id,
    examples=[
        {
            "inputs": {"question": "How do I reset my password?"},
            "outputs": {"answer": "Use the Forgot password link on the sign-in page."},
            "metadata": {"source": "prod-trace", "topic": "account"},
        },
        {
            "inputs": {"question": "Do you support SSO?"},
            "outputs": {"answer": "Yes, SAML and OIDC on the Enterprise plan."},
            "metadata": {"source": "hand-written", "topic": "security"},
        },
    ],
)

The same operations exist on the command line, which is convenient in scripts and in CI: langsmith dataset list, langsmith dataset create --name support-qa, langsmith dataset export support-qa ./support-qa.json, langsmith dataset upload f.json --name support-qa, and langsmith example list --dataset support-qa --split test. Exporting a dataset to a JSON file and committing it is a reasonable way to keep an audit trail of what you evaluated against.

Where do good examples come from? In order of value: real failures from production traces, real successes you want to protect, hand-written edge cases for behaviour the product must never get wrong, and synthetic examples to fill gaps. Automation rules (covered later) can copy sampled or filtered production runs into a dataset continuously, and annotation queues let a person correct the reference output before the example is trusted.

Keep the metadata honest. A source field tells you later which examples came from production and which you invented, which you will want when a score moves and you ask whether the dataset or the application changed. Keep inputs in the shape your target function receives, because the evaluation runner passes inputs straight to it.

A caution on dataset size. Bigger is not better until the examples are right. Fifty carefully labelled examples that cover your failure modes find more regressions than five thousand scraped ones with noisy references, and every example costs a model call per run and per judge. Grow the set when a production incident reveals a category the dataset lacked, and tag it so you can see why it is there.

Running evaluations properly

The problem: evaluate() is easy to run once and easy to get wrong in ways that produce a confident number that means little. This section covers the parts of the function and the evaluator shapes that determine whether the number is trustworthy.

An experiment is the result of running one version of your application over a dataset: outputs, evaluator scores and a trace for every example. It is created by evaluate() (or aevaluate() for async code). Experiments are created at extended retention, so they persist for 180 days rather than 14.

from langsmith import Client

client = Client()

def target(inputs: dict) -> dict:
    return {"answer": answer(inputs["question"])}     # your application

def exact_enough(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"].lower() in outputs["answer"].lower()

def concise(outputs: dict) -> dict:
    n = len(outputs["answer"].split())
    return {"key": "concise", "score": 1.0 if n <= 60 else 0.0, "comment": f"{n} words"}

results = client.evaluate(
    target,
    data="support-qa",
    evaluators=[exact_enough, concise],
    experiment_prefix="prompt-v3",
    metadata={"git_sha": "a1b2c3d", "model": "gpt-4o-mini"},
    max_concurrency=4,
    num_repetitions=3,
)

Read the arguments as decisions. data accepts a dataset name, an ID, or a list of examples, and it can be narrowed to a split. max_concurrency defaults to 0, meaning no parallelism, so a large dataset is slow until you raise it, and too high a value meets your model provider's rate limits. num_repetitions (default 1) runs each example several times, which matters for non-deterministic applications: one run per example measures one sample of a distribution, while three or five give you a spread. metadata goes on the experiment so you can later ask which commit and model produced a score. blocking (default true) makes the call wait for results. upload_results=False runs without recording an experiment, which is useful for a dry run on your laptop. Since SDK 0.13.0, disable_evaluator_tracing lets you stop the evaluators themselves from generating traces, which cuts noise and cost when a judge runs thousands of times. In TypeScript the call is await evaluate(target, { data, evaluators, experimentPrefix }) from langsmith/evaluation.

Evaluator functions may take any of inputs, outputs, reference_outputs, run and example, and LangSmith passes what you name. They can return a boolean, a number, a dictionary {"key": ..., "score": ... or "value": ..., "comment": ...}, or a list of such dictionaries to emit several metrics at once. A boolean becomes a score of 0 or 1 under the function's name. The comment is underrated: it is what a human reads when they open a failing row, so write the reason there.

There are several kinds of evaluator, and the choice is about cost and trust. Code evaluators are deterministic, cheap and should handle everything that can be checked mechanically: valid JSON, a required field, a citation present, latency under a budget. LLM-as-judge evaluators grade qualities that code cannot, such as whether an answer is faithful to its sources. The separate openevals package provides prebuilt judges (from openevals.llm import create_llm_as_judge, with prompts such as CORRECTNESS_PROMPT in openevals.prompts). Human evaluation goes through annotation queues. Pairwise evaluation compares two experiments directly; passing a tuple of two experiments as the target runs comparative evaluators, which is more reliable than two absolute scores when differences are subtle. Summary evaluators (summary_evaluators=) compute one number over the whole experiment, such as precision and recall, rather than one per row.

Evaluators also divide into reference-free (they look only at inputs and outputs, so they can also run on production traffic) and reference-based (they compare against reference_outputs, so they need a labelled dataset). This distinction is what decides whether a check can later be reused as an online evaluator.

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

correctness = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    feedback_key="correctness",
    model="openai:gpt-4o-mini",
)

An LLM judge is itself an application that can be wrong, so calibrate it: take 30 examples, grade them yourself, and check that the judge agrees on most of them before you trust it on two hundred. Judges prefer longer answers and answers that resemble their own style, so pin the judge model, keep its prompt in version control, and change it as rarely as you change the dataset, because either change moves scores without any change in your application.

Evaluation in CI with pytest

The problem: a notebook evaluation happens when someone remembers. A regression gate has to run on every pull request, fail loudly, and stay cheap enough that nobody skips it. LangSmith's pytest integration makes each test case an example in a tracked experiment, so a failing build links to traces.

Install the extra, then mark tests:

pip install -U "langsmith[pytest]"

import pytest
from langsmith import testing as t

from myapp.agent import generate_sql, is_valid_sql

CASES = [
    ("top 5 customers by revenue", "SELECT"),
    ("orders placed last week", "WHERE"),
]

@pytest.mark.langsmith
@pytest.mark.parametrize("question,must_contain", CASES)
def test_generates_valid_sql(question: str, must_contain: str):
    t.log_inputs({"question": question})
    t.log_reference_outputs({"must_contain": must_contain})

    sql = generate_sql(question)
    t.log_outputs({"sql": sql})

    t.log_feedback(key="valid_sql", score=int(is_valid_sql(sql)))
    t.log_feedback(key="has_keyword", score=int(must_contain in sql.upper()))

    assert is_valid_sql(sql)
    assert must_contain in sql.upper()

LANGSMITH_TEST_SUITE="SQL agent tests" pytest tests/ --langsmith-output

LANGSMITH_TEST_SUITE names the dataset that collects the test cases, and each run of the suite becomes an experiment against it, so you get history for free. --langsmith-output prints a readable results table instead of the usual dots. The t.log_* helpers are what turn a plain assertion into recorded inputs, outputs and feedback; without them the test still passes or fails, but you lose the data. The marker accepts options such as output_keys=[...] and cached_hosts=[...].

Two environment variables keep CI practical. LANGSMITH_TEST_CACHE=tests/cassettes records the HTTP calls your code makes (to a model provider, say) and replays them on later runs, so a pull request check costs nothing and does not flake on provider outages. Commit the cassettes, and delete them when you intentionally change a prompt. LANGSMITH_TEST_TRACKING=false runs the tests without sending anything to LangSmith, which is what you want for a developer's quick local loop or for forks that have no secrets. Vitest and Jest have an equivalent integration for TypeScript projects.

A sensible split of responsibilities is to keep deterministic, cheap assertions in pytest on every pull request, and to run the larger evaluate() experiment with a judge nightly or on merge. The pull request gate answers "did I break something obvious", the nightly run answers "is quality drifting". Both write to LangSmith, so you can see them in one place.

name: llm-tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -U -e ".[dev]" "langsmith[pytest]"
      - run: pytest tests/ --langsmith-output
        env:
          LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
          LANGSMITH_TEST_SUITE: "SQL agent tests"
          LANGSMITH_TEST_CACHE: tests/cassettes

Prompts as versioned artifacts

The problem: prompts edited in a string literal are invisible to reviewers and unrecoverable after a bad edit; prompts edited in a UI can change production behaviour without a deployment. The hub gives prompts history and a promotion path, provided you use it as one.

A prompt is a versioned template, chat or completion, in f-string or mustache format. Every save creates a commit, an immutable hash. A commit tag is a movable label pointing at a commit. The names staging and production are reserved for prompt Environments, which is how you promote: test a commit, then move the production tag to it. You address a prompt as name (latest), name:commit_hash (exact), name:tag (labelled), or owner/name for a public prompt.

from langchain_core.prompts import ChatPromptTemplate
from langsmith import Client

client = Client()

prompt = ChatPromptTemplate.from_messages([
    ("system", "You answer support questions using only the provided context."),
    ("user", "Context:\n{context}\n\nQuestion: {question}"),
])

url = client.push_prompt("support-answer", object=prompt, description="v2: stricter grounding")
print(url)

latest = client.pull_prompt("support-answer")
pinned = client.pull_prompt("support-answer:12344e88")
live = client.pull_prompt("support-answer:production")
with_model = client.pull_prompt("support-answer-with-model", include_model=True)

push_prompt creates the prompt or adds a commit, and accepts parent_commit_hash, is_public, description, readme, tags, commit_tags and commit_description. The langchainhub package is deprecated; push and pull now live in langsmith. In TypeScript you pull through langchain/hub.

The decision that matters is what your application pulls at runtime. Pulling name (latest) means every edit in the UI reaches production immediately, which is fast and dangerous. Pulling name:production means a person must move the tag, which is a deliberate act and is recorded. Pulling name:<commit_hash> means the prompt changes only when code changes, which is the most conservative and keeps the prompt reviewable in the same pull request as the code that parses its output. Many teams choose the tag for prompts that a non-engineer iterates on and the hash for anything whose output format code depends on.

The prompt cache shapes behaviour here. It is on by default: up to 100 entries, a 300 second time to live, and a 60 second background refresh, with stale-while-revalidate semantics. Practical consequences: a request is not delayed by a network pull on every call, a tag move takes up to a few minutes to reach a running service rather than instantly, and a brief LangSmith outage does not stop an already-warm service. You can tune it with configure_global_prompt_cache(ttl_seconds=None) for an offline, never-expiring cache, disable it for one client with Client(disable_prompt_cache=True), and bypass it for one pull with skip_cache. The older cache= client argument is deprecated.

Pulling a prompt that embeds a model configuration or secrets has its own parameters: include_model=True returns the prompt chained with its configured model, and secrets or secrets_from_env supply provider keys. Pulling someone else's public prompt requires dangerously_pull_public_prompt=True, and the name is a warning: a prompt is text your application will send to a model, so an edited public prompt is a prompt-injection vector into your system. Pin a commit hash for any public prompt you depend on.

The Playground is where most iteration happens, and the discipline is to connect it to evaluation. Open a prompt, select a dataset, and run it across the examples, then compare runs side by side. When a change is good, save the commit, run the same dataset through evaluate() from code, and only then move the staging or production tag.

Feedback, online evaluation and automation rules

The problem: offline evaluation tells you how a version behaves on examples you chose. Production tells you what users actually do, and you need a way to turn that into signals, datasets and alerts without reading every trace.

Feedback from your application is the simplest loop. A thumbs-up in your UI becomes a feedback record on the run that produced the answer:

from langsmith import Client

client = Client()

client.create_feedback(
    run_id=run_id,                     # the ID you chose when you called the function
    key="user_score",
    score=1.0,
    comment="Helpful",
    session_id=project.id,             # the project UUID, required on SmithDB deployments
)

If the feedback call comes from a browser or a mobile app, you should not ship an API key there. Create a presigned feedback token on the server with client.create_presigned_feedback_token(run_id, feedback_key) and hand the resulting URL to the client, which can post one piece of feedback to it without credentials. That is the supported way to collect user ratings from untrusted code.

One consequence to know before you turn feedback on widely: passing extend_trace_retention=True upgrades the whole trace to extended retention, at a higher price. Use it for feedback that marks a trace worth keeping, such as a thumbs-down you will investigate, and not for every rating.

Automation rules apply a filter, a sampling rate and an action to a project. The filter is the same expression language as queries, the sampling rate controls volume and cost, and the actions include adding runs to a dataset, adding them to an annotation queue, running an evaluator, calling a webhook, or extending retention. A rule's item type is either runs or threads. Typical rules at work:

• Send every trace with has_error true and a negative user score to an annotation queue, so a person reviews real failures.
• Sample five percent of successful root runs into a dataset named prod-sample, which grows your test set with current traffic.
• Post to a webhook when a trace exceeds a cost or latency threshold. Webhook payloads include a trace_url, which makes the message actionable.

Online evaluators are evaluators that run on production runs or threads automatically, without references, which is why only reference-free checks can be online. They score a sampled fraction of live traffic: groundedness against retrieved context, toxicity, whether an answer refused unnecessarily. They write feedback, and the feedback feeds dashboards and alerts. The offline and online halves complement each other, and the cleanest design is to write one reference-free evaluator and use it in both places, so the number in a pull request check and the number on the production dashboard mean the same thing.

The annotation queue is the human half. It is a structured review workflow, either single-run or pairwise, with multiple reviewers and reservations so two people do not grade the same item. Reviewers can correct outputs and the corrected example can go straight into a dataset. A few hundred human-reviewed examples are what make your judges trustworthy, because they are the ground truth you calibrate against.

Now the bill. Base traces are kept for 14 days and extended traces for 180 days; since 14 September 2026 the extended tier is 180 days, where it was previously 400, and that change applies to new traces. A trace is upgraded automatically to extended retention by an online evaluator, by an automation rule with retention extension enabled, and by feedback sent with extend_trace_retention=True. Extension is on by default for new evaluators and rules. A rule that samples fifty percent of traffic with extension on quietly puts half your volume in the more expensive tier. Review every rule and evaluator for this flag, and set monthly usage limits at workspace, project or user level as a backstop; hitting the extended-trace limit blocks the rules and evaluators that would extend retention.

Cost, tokens and limits

The problem: there are two different costs, what your application spends on model calls and what LangSmith charges to store traces, and both need attention before someone asks.

Application cost inside traces. LangSmith calculates cost automatically for known models when token counts are present on an llm run. The wrappers and LangChain integrations supply them. For a custom model call that the SDK cannot see into, set the run type to llm, set ls_provider and ls_model_name in metadata so the price lookup can match, and supply usage_metadata:

from langsmith import traceable
from langsmith.run_helpers import get_current_run_tree

@traceable(
    run_type="llm",
    metadata={"ls_provider": "my-gateway", "ls_model_name": "llama-3.1-70b"},
)
def call_model(prompt: str) -> str:
    text, in_tok, out_tok = my_gateway_call(prompt)
    run = get_current_run_tree()
    run.set(usage_metadata={"input_tokens": in_tok, "output_tokens": out_tok,
                            "total_tokens": in_tok + out_tok})
    return text

Prices are configured per workspace, and cost is resolved from the most specific token type to the least specific: cached input tokens are priced before plain input tokens, and so on. If you use a model the price table does not know, add it, or cost shows as missing rather than zero. Cost and latency are then available as columns, filters (gt(total_cost, 0.05) style filters in the UI), dashboards and alerts.

LangSmith's own cost. The plans at the time of writing are Developer (free, one seat, 5,000 base traces a month), Plus ($39 per seat per month, 10,000 base traces included) and Enterprise (custom). Beyond the included traces usage is pay-as-you-go; quote per-trace prices only from the pricing page, because they change. The billable meters are base traces and extended data retention upgrades, and newer products are billed in separate units.

The levers, in order of effect:

1. Keep projects at base retention and justify every extension.
2. Sample. A rate of 0.1 on a high-volume, low-risk path cuts ingestion by ninety percent with deterministic whole-trace sampling.
3. Do not trace what you do not need. Health checks and polling loops belong under tracing_context(enabled=False).
4. Avoid oversized payloads. Large documents in inputs inflate storage and approach per-request size limits; redact or reference them, or use attachments for binary data.
5. Use usage limits per workspace, project or user, and evaluator spend limits, so an errant loop hits a ceiling rather than an invoice.

Rate limits are the other constraint. On Cloud the per-key, per-minute limits are 5,000 for POST and PATCH on runs, 5,000 for feedback posts, 30 for GET /runs/:id and for DELETE /sessions, and 2,000 for everything else. The SDK batches up to 100 runs per call, so ordinary tracing rarely approaches them. Hourly event limits are 50,000 on Developer (250,000 with a card), 500,000 on Plus, and custom on Enterprise, with matching hourly data limits. A Developer account without a card is capped at 5,000 traces a month. When you hit a limit you receive LangSmithRateLimitError and HTTP 429; the SDK backs off and retries, and persistent 429s mean you should sample or raise the tier. Reading the API in a loop is the other way to trip a limit: GET /runs/:id at 30 a minute is tiny, so use a query and pages, not one request per run.

Dashboards, alerts and bulk export

The problem: traces are for investigating one request. Operating a service needs aggregates and notifications, and sometimes a copy of the data in your own warehouse.

Each project has prebuilt dashboards for trace volume, latency percentiles, error rate, token usage and cost, and you can build custom charts from filters and group them by metadata or tags. This is where your metadata vocabulary pays off: a chart grouped by git_sha shows whether the last deployment moved latency, and grouped by tenant it shows which customer drives cost.

Alerts fire on run count, cost, errors (as a count or a percentage), feedback score and latency, evaluated over 5 or 15 minute windows, and notify PagerDuty or a webhook. A workable starter set: error percentage above a threshold for 15 minutes, p95 latency above your budget, and average user feedback score dropping below a floor. Alert on feedback score only if you have enough volume for the window to be meaningful; a score computed from three ratings will page you for nothing.

Bulk data export writes traces to S3 or an S3-compatible bucket (GCS and MinIO work) as Parquet, compressed with zstandard by default on self-hosted 0.16. Use it when you need joins against business data, a long archive beyond retention, or analysis in a warehouse. Remember that exported data is outside LangSmith's retention and access control, so apply your own. Insights reports cluster traces to surface common usage patterns and failure modes, which is a quick way to find categories worth adding to your dataset.

The command line is useful for ad hoc operations and scripts. langsmith trace list --project my-app --limit 5 defaults to the last 7 days and 20 rows; langsmith run list --project my-app --run-type llm defaults to 50 rows; langsmith thread list --project my-chatbot requires --project. Filter flags include --last-n-minutes, --since, --error, --min-latency, --min-tokens, --tags, and a --filter expression, and --include-io, --include-feedback and --show-hierarchy shape the output. Use --format json and -o file to feed other tools. For anything the subcommands do not cover, langsmith api calls the REST API directly, for example langsmith api sessions?limit=5.

langsmith --format json trace list --project my-app --error --last-n-minutes 60 -o errors.json
langsmith --format json run list --project my-app --run-type llm --min-latency 5 --include-io

Install the CLI with brew install langchain-ai/tap/langsmith-cli on macOS, curl -fsSL https://cli.langsmith.com/install.sh | sh on Linux, or the PowerShell installer on Windows, and upgrade with langsmith self-update. Log in with langsmith auth login, which opens a browser OAuth flow and stores tokens under a profile in ~/.langsmith/config.json. One trap: the LANGSMITH_ENDPOINT environment variable overrides a profile's api_url, so a shell with that variable exported will send CLI calls to the wrong instance even with a self-hosted profile selected. Unset it or pass --api-url.

Distributed tracing and OpenTelemetry

The problem: one user request crosses services. Your API calls a worker, which calls a model gateway. Without propagation you get three disconnected traces, and with it you get one tree.

LangSmith propagates context through headers. The caller takes its current run tree and sends the headers it produces, langsmith-trace and baggage. The receiving service either adds the tracing middleware, which does this for every request, or attaches the headers explicitly with tracing_context(parent=headers):

# caller
import httpx
from langsmith import traceable
from langsmith.run_helpers import get_current_run_tree

@traceable(name="api-handler")
def handle(question: str) -> str:
    headers = get_current_run_tree().to_headers()
    r = httpx.post("http://worker.internal/answer", json={"q": question}, headers=headers)
    return r.json()["answer"]

# receiver (FastAPI)
from fastapi import FastAPI
from langsmith import traceable
from langsmith.middleware import TracingMiddleware

app = FastAPI()
app.add_middleware(TracingMiddleware)

@traceable(name="worker-answer")
def compute(q: str) -> str:
    return "..."

@app.post("/answer")
def answer(body: dict):
    return {"answer": compute(body["q"])}

Treat those headers as trusted input only from your own services. A service that accepts langsmith-trace from the public internet lets a stranger attach runs to your traces and choose where they land. Strip the headers at your edge and accept them only on internal routes.

OpenTelemetry is the other integration route, and the right one when your company already runs an OTel pipeline or your services are not Python or TypeScript. There are two patterns. For LangChain applications, set LANGSMITH_TRACING_MODE=otel (send OTel only) or hybrid (send both), which replaces the older LANGSMITH_OTEL_ENABLED flags that still work. For any OTLP exporter, point it at the LangSmith endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.smith.langchain.com/otel"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=$LANGSMITH_API_KEY,Langsmith-Project=my-app"

The trap here is specific and common: the endpoint is a base URL. Do not append /v1/traces, because the exporter adds the path itself, and a doubled path means spans that never arrive and no obvious error. Self-hosted instances use https://<host>/api/v1/otel. LangSmith maps the GenAI semantic conventions and several popular attribute schemas, including OpenInference, Traceloop and Logfire, onto its own run model. An invalid LANGSMITH_TRACING_MODE fails loudly with Invalid LANGSMITH_TRACING_MODE='x'. Must be one of: hybrid, langsmith, otel. If your organisation wants traces in both LangSmith and another backend, send OTLP to a collector and fan out from there; that also gives you one place to apply redaction before data leaves the network.

Workspaces, keys and access

The problem: a tool that holds every prompt your company sends to a model needs the same care as a database. Mid-level engineers are usually the ones creating keys and arranging access, so the model needs to be in your head.

The roles split in two levels. Organisation roles are Admin, Operator, User and Viewer (a Viewer cannot create personal access tokens). Workspace roles are Admin, Editor and Viewer. Fine-grained workspace RBAC, custom roles, attribute-based access policies built on resource tags, and audit logs are Enterprise features; on lower plans you have the simpler role set.

Keys come in two kinds. A personal access token (lsv2_pt_...) inherits the permissions of the person who created it, which is right for your own scripts and wrong for anything that must outlive you. A service key (lsv2_sk_...) is created by an administrator, scoped to one or more workspaces or the whole organisation, and belongs to no individual. Applications use service keys. Keys do not expire by default, so set an expiry when you create them. Deactivating a PAT takes effect within about a minute because authentication is cached. Service keys can only be deleted, not deactivated, and an organisation administrator can change a service key's role without rotating it. PAT creation can be disabled organisation-wide if policy demands. Legacy keys with the ls__ prefix stopped working on 22 October 2024.

A service key scoped to the whole organisation spans workspaces, and the API then needs to know which one you mean. The error reads LangSmithUserError: This API key is org-scoped and requires workspace specification. Please provide 'workspace_id' parameter, or set LANGSMITH_WORKSPACE_ID environment variable. with a 403 and the code org_scoped_key_requires_workspace. Fix it with LANGSMITH_WORKSPACE_ID, Client(workspace_id=...), or the X-Tenant-Id header on raw REST calls. A workspace-scoped key needs none of this, which is one more reason to scope keys narrowly.

How many workspaces should you have? The guidance is one workspace per team, because resources cannot be shared across workspaces. For environments use resource tags such as Environment: dev, staging and prod rather than separate workspaces, so a prompt or dataset can be used by all of them, and use prompt tags or Environments for promotion. The access model then says who may edit production resources by tag rather than by workspace sprawl.

For single sign-on, LangSmith supports SAML and OIDC with just-in-time provisioning, SCIM, and group sync; removing a user from all the mapped groups revokes SSO-granted access at their next sign-in, not instantly, so a leaver process should also deactivate PATs. Workspace secrets hold provider keys used by the Playground and by evaluators, so an engineer never needs to paste a model key into the UI. Compliance documentation covers SOC 2 Type 2, HIPAA and GDPR, and the docs note that there is no EU contracting entity, which can matter for procurement in regulated sectors.

Running LangSmith yourself

The problem: some employers cannot send prompts to a third-party SaaS, including for residency reasons common in Gulf and Egyptian public-sector and banking work. Self-hosting is the answer, it is an Enterprise option (as is BYOC, where LangChain operates the control plane and your data stays in your VPC), and even if you are not the platform owner you will be asked to debug it.

The architecture in brief. The frontend (Nginx plus the UI) is the only component you must expose. The backend handles CRUD, business logic and the hub API. The platform-backend handles authentication and run ingestion, the high-volume path. The queue and ingest-queue services persist asynchronously with retries. The playground proxies calls to model providers, and the ACE backend runs sandboxed code for code evaluators. Behind them are ClickHouse for traces and feedback, PostgreSQL 14 or later (with the extensions btree_gin, btree_gist, pgcrypto, citext, ltree and pg_trgm) for operational data, Redis 6.2 or later (or Valkey 8) for queues and caching, and optional blob storage (S3, GCS or Azure Blob). By default everything is bundled in the cluster. For production use external managed stores.

Installation is a Helm chart:

helm repo add langchain https://langchain-ai.github.io/helm
helm search repo langchain/langsmith --versions
helm upgrade -i langsmith langchain/langsmith \
  --values langsmith_config.yaml \
  --version <chart-version> -n langsmith --create-namespace --wait --debug

config:
  langsmithLicenseKey: "<license>"
  apiKeySalt: "<random, never rotate>"
  authType: mixed
  basicAuth:
    enabled: true
    initialOrgAdminEmail: "admin@example.com"
    initialOrgAdminPassword: "<at least 12 characters>"
    jwtSecret: "<random>"
insights:
  encryptionKey: "<openssl rand -hex 32>"
polly:
  encryptionKey: "<openssl rand -hex 32>"

Override only what you need and never copy the whole default values.yaml, because a copied default is a pinned default that hides changes in later chart versions. Never rotate apiKeySalt: it invalidates every API key. Keep secrets out of files with config.existingSecretName, pointing at a Kubernetes Secret. The instance needs egress to https://beacon.langchain.com for license checks and usage reporting unless it runs in offline mode. This cluster-side work is covered in depth in the Helm and Kubernetes guides.

Releases follow a policy worth internalising. There are two channels: stable receives weekly patches with critical and security fixes only, and preview publishes the next major as release candidates. A new stable major arrives about every six weeks; versions N and N minus one are supported, anything older is end of life. Chart version and application version advance independently, so "v16" means chart 0.16.Y. New or removed services, breaking changes and data migrations land only in a new major.

Upgrade one major at a time, never skipping (0.15 to 0.16 to 0.17), and know there is no downgrade path. The routine is: read the self-hosted changelog, back up Postgres, ClickHouse and blob storage, run helm repo update, run helm upgrade with the next version and your values file, then verify with helm status and confirm that the langsmith-pg-migrations-* and langsmith-ch-migrations-* jobs show Completed. Rehearse on the preview channel in staging. Version 0.16 brought several changes you must handle: bulk exports default to zstandard compression, the go-backend, playground and host-backend images were folded into smith-backend (remove overrides of the old images from your values), and a new backfillCheck job blocks the upgrade until required checks complete. Images now ship with Cosign signatures and signed SBOM attestations, which your admission policy can verify.

SmithDB on self-hosted is opt-in. It adds its own services, a Postgres metastore, object storage and a disk cache. ClickHouse is still required on 0.16 and remains the default on 0.17, so do not disable it below 0.17, and remember the deprecated query methods stop working once ClickHouse is off. That is why migrating your code (the earlier section) comes before migrating the datastore.

The failure messages you will see most often, with their causes:

bash get_k8s_debugging_info.sh --namespace langsmith from the Helm repository's charts/langsmith/scripts/ collects what support will ask for. To check a fresh install, curl <frontend-host>/api/tenants should return a JSON array of workspaces, and the SDK variables become LANGSMITH_ENDPOINT=https://<host>/api/v1 with a key created in that instance. A corporate TLS-inspecting proxy in front of the instance will break SDK connections with a connection error; the fix is to trust the company CA in the client environment, not to disable verification.

Debugging beyond the basics

The problem: something is wrong and the symptom is indirect. A systematic order saves more time than cleverness, and most LangSmith problems fall in a handful of classes.

A trace is missing. Work outward from the process. Was tracing enabled in that process (LANGSMITH_TRACING or a tracing_context override)? Did the process exit before the flush? Is the sampling rate below 1? Did the env cache keep an old project? Is the endpoint for the right region, without a trailing slash? Is the key valid for that workspace? Then look at the log: a LangSmithAuthError: Authentication failed for /runs/multipart with a 401 is the wrong key, a deactivated key, a legacy ls__ key or the wrong region. A missing-key warning, LangSmithMissingAPIKeyWarning: API key must be provided when using hosted LangSmith API, means the variable was never set in that process. A 429 means limits.

A trace is there but wrong. Inputs empty after an upgrade: the 0.11 PATCH change. Everything replaced with Processing failed; inputs dropped: your redactor raised. A conversation split into many traces: the thread key is missing on child runs. Cost is zero or missing: the run is not llm, lacks the provider and model metadata, or has no token counts. A child run appears as its own root: context was lost across a thread or async boundary. In Python, plain threads and some executors do not carry the tracing context automatically, so wrap the submit with the context or use the helpers in langsmith.run_helpers; in async code, make sure the awaited function is decorated rather than fired as a bare task.

A query returns too little. The defaults: one day and only ids. Then check project_ids is the right UUID and that run_type is uppercase.

An evaluation gives a suspicious number. Open the experiment rows, not the average. Check that the evaluator receives what you think it does (print its arguments once), that the reference outputs are right, that repetitions are not hiding noise, and that the judge was not changed since the last comparison. Compare two experiments per example before concluding.

Something is slow. Tracing overhead is normally small because of background batching, but a redactor with heavy regular expressions or a model call inside process_inputs sits in your request path. Profile with tracing disabled through tracing_context(enabled=False) to see the difference. Very deep or wide traces approach the 25,000-run cap and make the UI slow; split them.

Putting it all together

Here is a realistic arc that uses the whole level, written as the sequence a team follows for a support question-answering service.

1. Instrument. Name the runs, set run types, put thread_id, git_sha and environment in metadata, redact emails and account numbers with an anonymizer, and choose a project per service. Verify with a three-line function and langsmith trace list.
2. Configure by environment. Use service keys with expiry, the correct regional endpoint, and sampling at 1.0 in staging and lower in production for high-volume paths. Turn tracing off around health checks.
3. Build the dataset. Seed with real failures, add hand-written must-not-fail cases, record a source for each, and split a small smoke subset.
4. Gate pull requests. Mark the deterministic tests with @pytest.mark.langsmith, cache HTTP calls with LANGSMITH_TEST_CACHE, and run them in CI with a service key.
5. Evaluate on merge. Run evaluate() with num_repetitions=3, code evaluators plus one calibrated judge, experiment metadata including the commit, and compare to the previous experiment per example.
6. Promote prompts deliberately. Pull support-answer:production at runtime, test new commits against the dataset, and move the tag only when the comparison is acceptable.
7. Watch production. Run a reference-free online evaluator on a sample, with retention extension consciously off or on, alert on error percentage and latency, and send negative-feedback traces to an annotation queue.
8. Close the loop. Reviewed failures go into the dataset, which grows on the categories that hurt, and the next evaluation includes them.
9. Stay ahead of deprecations. Grep for the legacy query methods, migrate to runs.query() with explicit project, time window and selects, and test row counts, well before 31 January 2027.

A compact skeleton for the core of it:

import langsmith as ls
from langsmith import Client
from langsmith.anonymizer import create_anonymizer
import re

anonymizer = create_anonymizer([
    {"pattern": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "replace": "<email>"},
])

client = Client(anonymizer=anonymizer)

def setup(version: str, environment: str) -> None:
    ls.configure(
        client=client,
        project_name=f"support-qa-{environment}",
        tags=[environment],
        metadata={"git_sha": version, "service": "support-qa"},
    )

This is where the neighbouring guides connect. Container builds and CI belong to Docker, the self-hosted chart to Helm and Kubernetes, the infrastructure around it to Terraform (there is an official LangSmith Terraform provider, langchain-ai/langsmith, for workspaces, roles, evaluators, run rules and alert rules), and open-source observability alternatives to Langfuse. For metrics on retrieval quality in particular, see RAGAS, and for agent orchestration LangGraph.

Where you are now

You can explain why a run goes missing and how the background queue decides, configure tracing by environment with the right precedence and the right regional endpoint, and design an instrumentation vocabulary that makes traces queryable. You can redact in-process and know that redactors now fail closed, and you can query runs with the new SmithDB-backed methods without being caught by the defaults on time windows and selected fields. You can maintain a dataset as a test asset, run experiments with repetitions and calibrated evaluators, put pytest evaluation in CI, promote prompts through commit tags with an eye on the cache, feed production signals back with feedback, rules and annotation queues, and keep the bill predictable by controlling retention extension. You can carry context across services with headers or OpenTelemetry, reason about keys and workspaces, and read a self-hosted deployment well enough to plan an upgrade.

The next level, Senior, treats LangSmith as a platform for a team: SmithDB and ClickHouse architecture and their failure modes, scaling profiles, multi-tenant designs, security review, upgrade and migration programmes, and when to choose something else. Before you go, do the one concrete job that has a date on it: find and migrate your legacy query calls.

Sources

• LangSmith documentation home
• Observability concepts
• Observability quickstart
• Create an account and API key
• Regions FAQ
• Conditional tracing
• Sample traces
• Mask inputs and outputs
• Threads
• Serverless environments
• Troubleshooting variable caching
• SmithDB SDK migration
• Migrating query runs
• Migrating feedback
• Endpoint deprecation policy
• Evaluation concepts
• Evaluate an LLM application
• Test with pytest
• Test with Vitest and Jest
• Prompt engineering concepts
• Manage prompts programmatically
• Cost tracking
• Alerts
• Data export
• Distributed tracing
• Trace with OpenTelemetry
• LangSmith CLI
• Administration overview
• Role-based access control
• Usage and billing
• Self-hosted overview
• Self-hosted on Kubernetes
• Self-hosted upgrades
• Release versions
• Self-hosted changelog
• Troubleshooting
• Manage LangSmith with Terraform
• langsmith on PyPI
• langsmith on npm
• LangSmith pricing

This is part two of three. At this level an interviewer is not asking what a trace is. They are giving you a situation, such as a bill that doubled or a dashboard that went quiet, and watching how you narrow it down. The answers below are written the way a strong candidate would say them aloud: the first thing checked, the reason, and the trade-off accepted. Where a number or name appears, it is the current one, because stale specifics are the quickest way to lose credibility.

Where this picks up

Beginner gave you the vocabulary: runs, traces, projects, feedback, datasets, experiments, prompts. This level assumes you can use them and asks about the decisions behind them.

Keep the two dated facts of this year in your head, because interviewers who follow the product love them. First, the legacy query methods (list_runs, read_run, list_threads and their relatives) are deprecated in favour of SmithDB-backed client.runs.query() and friends, and they are removed from Cloud on 31 January 2027. Second, extended trace retention is 180 days since 14 September 2026, down from 400, while base retention stays at 14 days.

The model in one screen

If you are asked to explain LangSmith in a sentence and then defend it, say this. It is an observability and evaluation platform for LLM applications. Your code emits runs, which are OpenTelemetry-span-like units of work with a type (llm, chain, tool, retriever, embedding, prompt, parser). Runs form a trace, a tree capped at 25,000 runs. Traces from one conversation form a thread, joined by a session_id or thread_id metadata key that must be on every run. A project holds the traces of one service, and in the API a project is confusingly called a session. Quality work happens with datasets of examples, experiments that run an application version over a dataset, and evaluators that return feedback. Prompts are versioned with immutable commit hashes and movable tags. Organisation contains workspaces, and workspaces contain resources.

Three facts to have ready because they show you have operated it: the SDK sends runs from a background thread in batches of up to 100, so short-lived processes must flush; queries on the new API default to one day of data and only the id field; and the langsmith package has no fixed version to quote, since the SDK ships several releases a week, so you pin and upgrade deliberately.

Commands and code to have ready

You will be asked to sketch these from memory, so be able to write them without looking.

pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_..."
export LANGSMITH_PROJECT="my-app"
export LANGSMITH_ENDPOINT="https://eu.api.smith.langchain.com"   # only for a non-US organisation

langsmith auth login
langsmith project list
langsmith trace list --project my-app --limit 5
langsmith run list --project my-app --run-type llm --min-latency 5
langsmith dataset export support-qa ./support-qa.json

import langsmith as ls
from langsmith import Client, traceable

@traceable(run_type="chain", name="qa", tags=["qa"])
def qa(question: str) -> str: ...

client = Client()
client.flush()                                           # short-lived processes

with ls.tracing_context(enabled=False):                  # conditional tracing
    qa("ping")

results = client.evaluate(
    target, data="support-qa", evaluators=[correctness],
    experiment_prefix="prompt-v3", max_concurrency=4, num_repetitions=3,
)

project = client.read_project(project_name="my-app")
for run in client.runs.query(
    project_ids=[project.id], min_start_time=since,
    selects=["ID", "NAME", "STATUS"], filter='eq(run_type, "llm")',
):
    ...

The habit interviewers notice: you reach for selects and min_start_time in the same breath as runs.query, because you have been bitten by the defaults.

Scenario questions

1. Traces from the production service are missing. Walk me through it.First decide whether it is all traces or some. All missing: check the process has LANGSMITH_TRACING true, the key is valid for that workspace, and the endpoint is the right region with no trailing slash. A wrong-region key fails as an authentication error, not a region message. Then read the application logs for Failed to multipart ingest runs or LangSmithAuthError, because ingestion failures never raise into the request. Some missing: a sampling rate below one, a tracing_context(enabled=False) override, a short-lived process exiting before the background thread flushed, or a notebook whose cached environment variables point at an old project. A strong answer separates configuration from code by running a three-line traced function from the same environment and checking it with langsmith trace list.
2. The serverless function traces locally but never in the cloud.The runtime freezes the process when the handler returns, so the background queue never drains. Flush before returning with client.flush(), or in JavaScript await client.awaitPendingTraceBatches(), or set LANGSMITH_TRACING_BACKGROUND=false so tracing completes inline. The trade-off is a little added latency per invocation in exchange for not losing data.
3. Our bill doubled but traffic did not. What do you look at?Retention. Base traces keep for 14 days and extended for 180, and a trace is upgraded automatically by online evaluators, by automation rules with retention extension enabled, and by feedback sent with extend_trace_retention=True. Extension is on by default for new rules and evaluators. So list every rule and evaluator, check that flag and the sampling rate, and look for a rule that samples heavily and extends. Then look at volume sources that should not be traced at all, such as health checks, and at oversized payloads. The fix is to extend only for signals that justify it, sample the rest, and set usage limits as a backstop.
4. A conversation shows up as eight separate traces instead of one thread.The thread key is on the root run but not on the children, or it differs between turns. The docs require session_id or thread_id metadata on all runs including child runs, with a UUID v7 recommended. Verify by inspecting a child run's metadata in the UI, then fix by passing it through langsmith_extra or a shared decorator so it is set once per turn.
5. An old script that lists runs now prints warnings, and a teammate says to ignore them.Do not ignore them. The warning says the method is deprecated and will be removed after 31 January 2027. Migrate to client.runs.query() with project_ids (not the project name), an explicit min_start_time, uppercase selects and uppercase run types. Then compare old and new counts for the same window before removing the old call, because forgetting the time bound silently returns only the last day and the default selects returns only ids.
6. Users rate answers in the browser. How do you record that without exposing an API key?Create a presigned feedback token server-side with client.create_presigned_feedback_token(run_id, feedback_key) and give the URL to the browser, which can post one piece of feedback without credentials. Keep the run ID by choosing it yourself via langsmith_extra={"run_id": ...}, and pass the project UUID as session_id where you create feedback server-side. Mention that extend_trace_retention=True costs money and should be reserved for ratings you will investigate.
7. The evaluation score went up but users say quality is worse.Suspect the measurement before the product. Was the dataset or the judge changed since the last comparison? Are references correct? Is the dataset representative of current traffic, or frozen on last quarter's questions? One repetition per example hides variance, so use num_repetitions. Look at the per-example comparison, not the mean, because an unchanged average can hide ten fixes and ten breakages. And check whether the judge prefers longer answers. The remedy is to calibrate the judge against a set of human-graded examples and to feed new production failures into the dataset.
8. Someone changed a prompt in the UI and production behaviour changed within minutes. Stop this.The application pulled the latest version. Change the pull to a tag such as support-answer:production, so promotion is a deliberate tag move after tests pass, or to a commit hash for prompts whose output format code depends on. Explain the prompt cache: default 300 second time to live with a 60 second refresh and stale-while-revalidate, so a tag move reaches running services within a few minutes, and a brief outage does not break a warm service.
9. Customer emails appear in traces and legal has noticed.Redact in process, before data is sent. Choose the layer by need: environment variables such as LANGSMITH_HIDE_INPUTS for whole payloads, a client anonymizer for pattern-based masking, or per-function process_inputs and process_outputs for precise control. Since SDK 0.12 redactors fail closed, so a bug yields Processing failed; inputs dropped rather than a leak. Explain that data already sent before the fix stays under normal retention, so also delete affected traces and verify by viewing a real trace afterwards.
10. Two services call each other and the traces do not join.Context is not propagating. The caller sends langsmith-trace and baggage headers from its run tree's to_headers(), and the receiver needs TracingMiddleware or tracing_context(parent=headers). If the stack uses OpenTelemetry, check both sides use the same propagation story. Mention the security point unprompted: accept those headers only from trusted internal callers.
11. OpenTelemetry spans are not showing up.Check the endpoint. It is a base URL, https://api.smith.langchain.com/otel, and appending /v1/traces doubles the path because the exporter adds it. Check the headers carry x-api-key and Langsmith-Project, that the region matches, and that the self-hosted form is https://<host>/api/v1/otel. For LangChain apps the simpler option is LANGSMITH_TRACING_MODE=otel or hybrid.
12. The CLI keeps hitting Cloud even though my profile points at the company instance.The LANGSMITH_ENDPOINT environment variable overrides a profile's api_url. Unset it or pass --api-url. It is a good example of an answer that shows you read the precedence rules rather than guessing.
13. A 403 says the key is org-scoped and requires workspace specification.A service key scoped to the whole organisation spans workspaces, so the request must name one. Set LANGSMITH_WORKSPACE_ID, pass workspace_id to the client, or send X-Tenant-Id on raw REST calls. The better long-term fix is a service key scoped to a single workspace.
14. The self-hosted ClickHouse pod reports NOT_ENOUGH_SPACE and ingestion stalls.The disk is full. Expand the data-langsmith-clickhouse-0 volume claim, which requires a storage class with allowVolumeExpansion: true, then delete the StatefulSet with --cascade=orphan and run helm upgrade with the larger size so the template matches. Then address the cause: a TTL that is too long, a missing blob-storage offload, or unexpected volume from a runaway application.
15. You are asked to upgrade a self-hosted instance two majors ahead.You cannot skip. Upgrade one major at a time, read the changelog for each, back up Postgres, ClickHouse and blob storage first because downgrades are not supported, rehearse on the preview channel in staging, and verify that the langsmith-pg-migrations-* and langsmith-ch-migrations-* jobs complete. Mention the v0.16 specifics: consolidated images, a backfillCheck that blocks the upgrade until checks finish, and zstandard exports by default.
16. Can we turn ClickHouse off now that SmithDB exists?Not below 0.17. ClickHouse is required on 0.16 and stays the default on 0.17, and the deprecated query methods stop working once it is disabled. So migrate callers to the new methods first, then plan the datastore change as its own project.

Trade-off questions

1. LangSmith Cloud, BYOC or self-hosted?Cloud is managed, continuously deployed and cheapest to operate, but the regions are US, EU and APAC, with none in the Middle East. BYOC keeps your data in your VPC while LangChain runs the control plane. Self-hosted gives full control and is the route for strict residency, at the price of upgrades, scaling and on-call for ClickHouse, Postgres and Redis. Both BYOC and self-hosted are Enterprise. A good answer names the driver (residency, security review, team capacity) rather than a preference.
2. Sample traces or keep them all?Keep everything where volume is modest and debugging value is high. Sample where traffic is high and paths are repetitive. Sampling is deterministic per trace since SDK 0.12, so a kept trace is whole. Keep errors and low-feedback traces regardless by combining sampling with rules that extend or route them, and never sample the paths you are evaluating online.
3. Code evaluators or an LLM judge?Code evaluators are cheap, deterministic and debuggable, so they take everything that can be checked mechanically. A judge handles qualities code cannot express but costs a model call per example, can be biased toward longer answers, and needs calibration against human grades. Use both, and keep the judge and its prompt pinned.
4. Pull the latest prompt, a tag or a commit hash?Latest is fastest to iterate and unreviewed. A tag gives a deliberate, recorded promotion and easy rollback. A hash ties the prompt to a code release and keeps it reviewable next to the parser that depends on it. Choose by who edits the prompt and what code depends on its format.
5. Personal access token or service key?A personal token inherits one person's permissions and breaks when they leave. A service key belongs to no individual and can be scoped to a workspace, so applications and CI use it, with an expiry. The cost is that an administrator must create it.
6. One workspace per environment, or per team?Per team. Resources cannot be shared across workspaces, so splitting by environment makes prompts and datasets awkward to reuse. Separate environments with resource tags such as Environment: staging, and promote prompts with tags.
7. LangSmith SDK tracing or OpenTelemetry?The SDK is the richest path, with run types, cost tracking and evaluation integrations. OpenTelemetry fits organisations with an existing pipeline, polyglot services and a need to fan out to several backends, at the cost of a mapping layer and some features. Hybrid mode sends both while you decide.
8. Offline evaluation or online evaluation?Offline runs before deployment on labelled examples and can use references, so it gates releases. Online runs on live traffic without references, so it detects drift and unknown failure modes. Reference-free evaluators can serve both, which gives one metric that means the same thing in CI and on the dashboard.
9. Build the evaluation harness yourself or use evaluate()?The harness gives you tracing, concurrency, repetitions and comparison views for free and stores results next to traces. A custom loop gives you control but rebuilds all of that. Build your own only for a constraint the platform cannot meet, and even then keep writing results to an experiment.

What a strong answer covers that a weak one misses

The distinction is almost always the same: a strong answer names the mechanism, the default, and the failure mode, and a weak one names the feature.

Other signals that separate candidates:

• They mention the deprecation without prompting. A candidate who has maintained a LangSmith integration knows about the 31 January 2027 removal and the new parameter names.
• They talk about who owns each resource. Keys, datasets and prompts have owners, expiries and review, not just names.
• They treat evaluators as code. Judges are versioned, calibrated and pinned; changing one is a measurement change, recorded as such.
• They reason about cost on both sides. Model spend inside traces and LangSmith spend on storage are different problems with different levers.
• They know what they would not verify from memory. Plan prices and rate limits change, and the right answer is to read the pricing page and the docs.

Common interview questions

1. What is the difference between a run, a trace and a thread?A run is one unit of work. A trace is the tree of runs for one operation, capped at 25,000 runs. A thread is a sequence of traces forming one conversation, joined by a thread_id or session_id metadata key that must be on every run.
2. Why does the API call projects "sessions"?Historical naming. In API parameters session_id is the project UUID, which is why feedback creation takes it, and it should not be confused with the thread key of the same name.
3. How does the SDK send traces, and what does that imply?A background thread batches runs, up to 100 per call, compresses them and posts them to the multipart ingestion endpoint. It never blocks your code and never raises on ingestion failure, so short-lived processes must flush and failures show in logs.
4. What changed in SDK 0.12 that affects privacy?Per-function anonymizers fail closed: if process_inputs raises, the payload becomes Processing failed; inputs dropped (...) instead of being sent raw. Sampling also became deterministic.
5. What is the precedence between environment variables, the client and tracing_context?tracing_context overrides client and decorator settings, which override the environment.
6. How do you keep traces out of a health check?Wrap the call in ls.tracing_context(enabled=False), or pass enabled=False to the decorator for that function.
7. What does num_repetitions do and why use it?It runs each example several times. For non-deterministic applications one run per example is one sample, so repetitions reveal variance and stop you celebrating noise.
8. What shapes can an evaluator return?A boolean, a number, a dictionary with key, score or value, and comment, or a list of dictionaries for several metrics. It can declare any of inputs, outputs, reference_outputs, run and example as parameters.
9. What is a summary evaluator?One that computes a single metric over the whole experiment, such as precision and recall, rather than one score per row.
10. When would you run pairwise evaluation?When two versions differ subtly and absolute scores cannot separate them. You pass two experiments and use comparative evaluators, which tend to be more reliable than comparing two independent scores.
11. How do you make LLM tests cheap and stable in CI?Use pytest with @pytest.mark.langsmith, cache HTTP calls with LANGSMITH_TEST_CACHE, and set LANGSMITH_TEST_TRACKING=false for local loops or forks without secrets. Keep the expensive judge-based evaluation for merge or nightly runs.
12. What is the difference between a commit and a commit tag?A commit is an immutable hash of a prompt version. A tag is a movable label pointing at a commit. staging and production are reserved for Environments.
13. How does the prompt cache behave?On by default with 100 entries, a 300 second time to live and a 60 second refresh, stale-while-revalidate. It can be tuned with configure_global_prompt_cache, disabled per client with disable_prompt_cache=True, or bypassed with skip_cache.
14. Why is pulling a public prompt flagged as dangerous?Because it is text your application will send to a model, so an edited public prompt is an injection path. The parameter is named dangerously_pull_public_prompt, and you should pin a commit hash.
15. How do you get cost for a custom model call?Set the run type to llm, put ls_provider and ls_model_name in metadata, and supply usage_metadata with token counts. Cost is computed from workspace model prices, most specific token type first.
16. What are the two kinds of API key, and which prefix does each have?Personal access tokens lsv2_pt_... inherit a user's permissions. Service keys lsv2_sk_... are admin-created and scoped to workspaces or the organisation. Legacy ls__ keys stopped working on 22 October 2024.
17. How do you export traces to a warehouse?Bulk data export to S3 or an S3-compatible store in Parquet, compressed with zstandard by default on self-hosted 0.16. Remember that exported data leaves LangSmith's access control and retention.
18. What must be exposed in a self-hosted deployment?Only the frontend, which is Nginx and the UI. The rest talk internally, and production should use managed external Postgres, Redis and blob storage instead of the bundled ones.
19. Why should you never rotate apiKeySalt on self-hosted?It invalidates every API key issued by the instance.
20. What is the rule for upgrading self-hosted versions?One major at a time, with no downgrades, a new major roughly every six weeks, and support for the active major and the one before it.
21. How do you tell whether a self-hosted install is healthy?helm status for the release, the two migration job families showing Completed, and curl <frontend-host>/api/tenants returning a JSON array of workspaces.

Questions to ask the interviewer

Good candidates finish by asking questions that show operating experience. Choose two or three that fit.

• Which region is the organisation in, and was that a residency decision or a default?
• Do applications use service keys with expiry, or personal tokens?
• Do any automation rules or online evaluators extend retention, and who reviews them?
• Are the legacy query methods still used anywhere, and is there a migration plan before the January 2027 removal?
• How are prompts promoted, and what do services pull at runtime?
• Is evaluation part of pull request checks, or a notebook exercise?
• For self-hosted, which version and channel are you on, and how often do you rehearse upgrades?

Final self-test

• Describe how a trace gets from your function to the dashboard, and name two ways runs are lost.
• State the precedence of tracing_context, client settings and environment variables.
• List the four parameter traps when moving from list_runs to runs.query.
• Explain which features upgrade a trace to extended retention, and what the two retention periods are.
• Say what fails closed in SDK 0.12 and what you would see when a redactor breaks.
• Compare pulling latest, a tag and a hash, and choose one for a parser-coupled prompt.
• Explain why a mean evaluation score can rise while quality falls.
• Describe the safe self-hosted upgrade routine and the rule about ClickHouse below version 0.17.
• Name two things you would check before quoting a plan price or a rate limit.

Part two of three. At this level tracing works and dashboards fill up, so the problems change character. They are no longer "why is nothing showing" but "why did the bill change", "why did this query shrink", and "why does the score say we improved". Start with the error table, then the practices and practice cards underneath it, and finish with recipes you can paste into a real project.

Versions for reference while reading: Python SDK langsmith 0.14.2, JavaScript SDK 0.10.5, langsmith-cli v0.2.60, self-hosted Helm chart stable v16. These move weekly, so treat them as a snapshot and check the release pages before you rely on a behaviour.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In order of leverage. The first two prevent the largest number of incidents.

Practice cards

1. Lose some runs on purposeWrite a script that calls a traced function and exits. Run it ten times and count arrivals, then add a finally: client.flush() and repeat. You will not forget the flush again.
2. Migrate one queryPort a list_runs call to runs.query. Run both for the same week and compare counts. Every difference is a default you did not know about.
3. Split a thread and rejoin itTrace a two-turn chat with the thread key on the root only, and look at the threads view. Add it to every run and look again.
4. Break a redactorMake a process_inputs function raise on an unexpected key and read the fail-closed message. Then fix it with a safe default.
5. Measure varianceRun the same evaluation with num_repetitions=1 and then 5. Compare the spread in scores. If it is wide, your earlier comparisons were partly noise.
6. Calibrate a judgeHand-grade thirty examples, run the judge on them, and compute agreement. Below your threshold, rewrite the judge prompt and repeat before using it anywhere.
7. Time a tag movePull name:staging in a running process, move the tag, and measure how long the new text takes to arrive with the default cache. Then shorten the TTL and measure again.
8. Audit retentionList every online evaluator and automation rule, record its sampling rate and whether it extends retention, and estimate what share of traffic is in the extended tier.
9. Join two servicesSend to_headers() from a caller to a FastAPI receiver with TracingMiddleware, confirm one tree, then remove the headers and confirm two traces.
10. Rehearse an upgradeOn a scratch cluster, upgrade one chart version at a time, verify the migration jobs, and write down what you would need to restore if it failed.

Operational recipes: verifying a setup in one minute

The first recipe settles the most common argument, whether the problem is configuration or code. It should run from the same environment as the failing application, meaning the same container, shell or CI job.

python - <<'PY'
import os
import langsmith as ls
from langsmith import Client, traceable

print("sdk:", ls.__version__)
print("tracing:", os.getenv("LANGSMITH_TRACING"))
print("endpoint:", os.getenv("LANGSMITH_ENDPOINT", "(default US)"))
print("project:", os.getenv("LANGSMITH_PROJECT", "default"))
print("workspace:", os.getenv("LANGSMITH_WORKSPACE_ID", "(none)"))

@traceable(name="smoke-test")
def ping(x: str) -> str:
    return x.upper()

ping("hello")
Client().flush()
print("sent; now check: langsmith trace list --project", os.getenv("LANGSMITH_PROJECT", "default"), "--limit 3")
PY
langsmith trace list --project "${LANGSMITH_PROJECT:-default}" --limit 3

If the smoke test arrives and the application's traces do not, the bug is in how the application is instrumented or in a code path that disables tracing. If the smoke test does not arrive either, the bug is in the key, region, workspace or network, and the log output from the SDK will say which.

A second check looks for leftovers of the legacy prefix, because mixed prefixes cause confusing precedence:

env | grep -E '^(LANGCHAIN|LANGSMITH)_' | sed -E 's/(API_KEY=).*/\1***redacted***/'

Operational recipes: migrating queries safely

The migration fails quietly when it fails, so make it testable. First find the call sites, then port each one with explicit arguments, then assert on the result.

grep -rnE "\.(list_runs|read_run|list_threads|read_thread|get_run_url|share_run|unshare_run|read_run_shared_link|read_shared_run|list_shared_runs|get_experiment_results)\(" \
  --include="*.py" --include="*.ipynb" .

from datetime import datetime, timedelta, timezone

from langsmith import Client


def slow_llm_runs(project_name: str, days: int = 7, min_seconds: float = 5.0) -> list:
    client = Client()
    project = client.read_project(project_name=project_name)
    since = datetime.now(timezone.utc) - timedelta(days=days)

    rows = list(
        client.runs.query(
            project_ids=[project.id],
            min_start_time=since,
            selects=["NAME", "STATUS", "INPUTS"],
            filter=f'and(eq(run_type, "llm"), gt(latency, {min_seconds}))',
            page_size=500,
        )
    )
    return rows


if __name__ == "__main__":
    rows = slow_llm_runs("my-app")
    print(f"{len(rows)} slow LLM runs")
    assert rows, "query returned nothing: check project, window and run_type casing"

The assertion at the end is the point. A query that returns zero rows for a busy project is almost always a default, not an empty week. To make the warnings loud in CI, run the suite with deprecation warnings escalated:

python -W "error::DeprecationWarning" -m pytest tests/ -x -q

If a third-party integration calls a deprecated method internally, you will see it too. Check whether a newer release of that package has migrated, and pin it rather than silencing the warning.

The feedback half of the migration is small but easy to miss. Wherever you create feedback on a run, pass the project UUID:

from langsmith import Client

client = Client()
project = client.read_project(project_name="my-app")

client.create_feedback(
    run_id=run_id,
    key="user_score",
    score=1.0,
    session_id=project.id,
)

Operational recipes: a cached pytest gate

This recipe gives you a pull request check that costs nothing on reruns. The cassette directory is committed, so a teammate's run and CI's run replay the same HTTP exchanges.

import os

import pytest


@pytest.fixture(autouse=True, scope="session")
def _default_test_suite():
    os.environ.setdefault("LANGSMITH_TEST_SUITE", "support-qa tests")
    yield

import pytest
from langsmith import testing as t

from myapp.qa import answer

CASES = [
    {"question": "How do I reset my password?", "must_mention": "Forgot password"},
    {"question": "Do you support SSO?", "must_mention": "SAML"},
]


@pytest.mark.langsmith
@pytest.mark.parametrize("case", CASES, ids=lambda c: c["question"][:30])
def test_answer_mentions_key_fact(case: dict):
    t.log_inputs({"question": case["question"]})
    t.log_reference_outputs({"must_mention": case["must_mention"]})

    out = answer(case["question"])
    t.log_outputs({"answer": out})

    ok = case["must_mention"].lower() in out.lower()
    t.log_feedback(key="mentions_key_fact", score=int(ok))
    assert ok, f"expected '{case['must_mention']}' in: {out}"

# Record cassettes once, locally
LANGSMITH_TEST_CACHE=tests/cassettes pytest tests/ --langsmith-output

# CI: replay them, no provider cost
LANGSMITH_TEST_CACHE=tests/cassettes LANGSMITH_TEST_SUITE="support-qa tests" pytest tests/ --langsmith-output

# Fast local loop: do not send anything to LangSmith
LANGSMITH_TEST_TRACKING=false pytest tests/ -q

When you deliberately change a prompt, delete the affected cassettes so the next run records the new behaviour. Otherwise the cache will replay the old answer and your test will pass against a prompt that no longer exists.

Operational recipes: evaluations that can be compared

An experiment is only useful if you can later say what produced it. Put identity in the metadata, and keep the judge stable.

import os
import subprocess

from langsmith import Client
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

from myapp.qa import answer

client = Client()
git_sha = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"], text=True).strip()

judge = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    feedback_key="correctness",
    model="openai:gpt-4o-mini",          # pin the judge, change it rarely and on purpose
)


def target(inputs: dict) -> dict:
    return {"answer": answer(inputs["question"])}


def has_citation(outputs: dict) -> dict:
    ok = "[source:" in outputs["answer"]
    return {"key": "has_citation", "score": int(ok), "comment": "" if ok else "no [source: ...] marker"}


results = client.evaluate(
    target,
    data="support-qa",
    evaluators=[judge, has_citation],
    experiment_prefix=f"qa-{git_sha}",
    metadata={"git_sha": git_sha, "judge": "gpt-4o-mini", "prompt": os.getenv("PROMPT_REF", "support-answer:production")},
    max_concurrency=4,
    num_repetitions=3,
    disable_evaluator_tracing=True,
)
print(results.experiment_name)

Three details make this recipe hold up. The experiment prefix and metadata carry the commit and the judge, so a comparison a month later is interpretable. disable_evaluator_tracing removes thousands of evaluator traces that you would otherwise pay to store and never read. Repetitions make a noisy application visible.

To compare, open the dataset's experiments tab, select two experiments, and inspect the examples that changed in each direction. If you want the regressions in code, query the experiment's runs with client.datasets.experiment_runs.query(), which replaces the deprecated get_experiment_results().

A guard against the most common self-deception: before celebrating a score, open the experiment, sort by the lowest feedback value, and read the five worst rows next to their reference outputs. The habit of reading raw examples beside the scores is what catches a broken judge or a wrong reference before anyone quotes a percentage.

Operational recipes: redaction you can test

A redactor is code that protects data, so it needs tests that fail. Write the redaction as a plain function, test it without LangSmith, and attach it afterwards.

import re

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
LONG_NUMBER = re.compile(r"\b\d{10,16}\b")


def scrub_text(text: str) -> str:
    text = EMAIL.sub("<email>", text)
    return LONG_NUMBER.sub("<number>", text)


def scrub_inputs(inputs: dict) -> dict:
    cleaned = {}
    for key, value in inputs.items():
        if key == "customer_document":
            continue                                  # never upload attachments
        cleaned[key] = scrub_text(value) if isinstance(value, str) else value
    return cleaned

from myapp.redact import scrub_inputs


def test_email_and_number_masked():
    out = scrub_inputs({"q": "mail me at a.b@example.com, card 4111111111111111"})
    assert "example.com" not in out["q"]
    assert "4111111111111111" not in out["q"]


def test_attachment_dropped():
    assert "customer_document" not in scrub_inputs({"customer_document": "secret", "q": "hi"})


def test_non_string_values_survive():
    assert scrub_inputs({"n": 3})["n"] == 3

from langsmith import traceable

from myapp.redact import scrub_inputs


@traceable(name="qa", process_inputs=scrub_inputs)
def answer(question: str) -> str:
    ...

Because per-function redactors fail closed, a bug here shows up as a dropped payload in the trace, which is safe but unhelpful. The unit tests catch the bug before it gets that far. Do not set LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS to hide a failing redactor.

Operational recipes: prompt promotion

The promotion routine in five steps, with the commands you will use.

1. Edit and save in the Playground. That creates a commit.
2. Run the dataset through the Playground and then through evaluate() from code, pointing the experiment metadata at the commit hash.
3. Compare to the experiment of the prompt currently tagged production, per example.
4. If acceptable, move the production tag to the new commit, in the UI or by pushing with commit_tags.
5. Watch the online evaluator and the error rate for one cache interval plus a margin, and move the tag back if they regress.

from langsmith import Client, configure_global_prompt_cache

client = Client()

# Pin exactly what an evaluation used
candidate = client.pull_prompt("support-answer:12344e88")
current = client.pull_prompt("support-answer:production", skip_cache=True)

# A service that must survive a LangSmith outage
configure_global_prompt_cache(ttl_seconds=None)   # never expires; refreshes only on restart or skip_cache

Pick the TTL deliberately. The default 300 second time to live with a 60 second refresh balances freshness and resilience. A shorter TTL makes promotion and rollback faster but means more pulls. A never-expiring cache makes a service independent of LangSmith at runtime, at the price of needing a restart or an explicit refresh to pick up a change.

Operational recipes: auditing cost and retention

Run this audit monthly, or whenever someone says the bill looks wrong.

1. List every online evaluator and automation rule, with its project, filter, sampling rate and the retention-extension flag. Anything that extends retention on a broad filter is the prime suspect.
2. Check what feeds extended retention by other routes: code that passes extend_trace_retention=True to create_feedback, and experiments, which are created at extended retention.
3. Find volume you never read. Health checks, polling loops, internal batch jobs. Wrap them in tracing_context(enabled=False) or lower the sampling rate for that path.
4. Look for oversized traces and payloads. A full document in every input inflates storage. Redact, truncate or reference.
5. Set usage limits at workspace, project or user level, and evaluator spend limits, so a loop meets a ceiling rather than an invoice. Remember that hitting the extended-trace limit blocks the rules and evaluators that extend retention.
6. Re-read the pricing page and the usage page. Do not quote a per-trace price from memory, and do not assume the extended-retention period is what it was last year.

import langsmith as ls
from langsmith import traceable

@traceable
def handle_request(payload: dict) -> dict:
    ...

def health() -> dict:
    with ls.tracing_context(enabled=False):          # never trace the load balancer probe
        return {"ok": True}

Operational recipes: self-hosted checks

If your employer runs LangSmith on Kubernetes, these are the commands that answer most questions. They assume the release is named langsmith in a namespace of the same name.

helm list -n langsmith
helm status langsmith -n langsmith
helm search repo langchain/langsmith --versions | head -5

kubectl get pods -n langsmith
kubectl get jobs -n langsmith | grep -E 'pg-migrations|ch-migrations'
kubectl get pvc -n langsmith
kubectl logs -n langsmith deploy/langsmith-backend --tail=100

curl -s https://langsmith.example.internal/api/tenants | head -c 300

Read the output this way. helm status should report deployed. The migration jobs should be Completed; a failed one blocks the rest, and a message such as Dirty database version ... fix and force version means a ClickHouse migration needs a forced version from the backend pod before rerunning the upgrade. The persistent volume claims should show free space, and NOT_ENOUGH_SPACE in ClickHouse logs means it is full. The /api/tenants request should return a JSON array of workspaces; anything else points at the frontend, ingress or TLS.

For a support case, run bash get_k8s_debugging_info.sh --namespace langsmith from the Helm repository's charts/langsmith/scripts/ directory and attach the output. For a browser-only problem capture a HAR file.

The upgrade runbook, short form:

helm repo update
helm get values langsmith -n langsmith > values.before.yaml      # keep what is deployed
# back up Postgres, ClickHouse and blob storage first: there is no downgrade
helm upgrade langsmith langchain/langsmith \
  --version <next-major-chart> --values langsmith_config.yaml \
  -n langsmith --wait --debug
helm status langsmith -n langsmith
kubectl get jobs -n langsmith | grep migrations

Upgrade one major at a time and never skip. Rehearse the preview channel in staging. When you move to 0.16, remove overrides of the old go-backend, playground and host-backend images from your values, because they are now part of smith-backend. See the Helm and Kubernetes guides for the chart and cluster mechanics.

Habits worth adopting now

A short checklist

• Tracing is verified in each environment with a smoke test, and failures are visible in logs.
• Short-lived code paths flush, and serverless handlers await pending batches.
• Keys are service keys with expiry, scoped to one workspace, and the regional endpoint has no trailing slash.
• Runs are named, typed honestly, and carry user, thread, version and environment metadata on every run.
• Redaction is tested, and broken redactors are fixed rather than bypassed.
• Every call to a deprecated query method is migrated with explicit project, window and fields, and covered by a row-count assertion.
• Datasets have a source on each example, a small smoke split, and new production failures added after incidents.
• Evaluations use repetitions, pinned and calibrated judges, and experiment metadata with the commit.
• Pull requests run cached pytest checks with a service key.
• Production pulls a prompt by tag or hash, and promotion and rollback are written down.
• Every rule and online evaluator has a deliberate retention-extension setting and a sampling rate.
• Self-hosted upgrades go one major at a time, with backups, a staging rehearsal and migration jobs verified.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, where each one gains a scale, security or ownership dimension, and it adds the work you own when LangSmith is a platform that other teams depend on rather than a tool you use yourself.

Up to this point the questions were about getting traces in and making evaluations useful. From here they change: what happens to your traces when the ingestion path is saturated, who can read a prompt that contains a customer's national ID number, which component of a self-hosted install falls over first, what a retention setting quietly does to next month's invoice, and how you upgrade a system that has no downgrade path while forty engineers are using it.

Where this picks up

I am starting with the architecture, because every scaling, security and upgrade decision in this track depends on knowing which process does what.

What the platform is made of

LangSmith Cloud hides its structure from you, but the self-hosted and bring-your-own-cloud variants do not, and a senior engineer should be able to draw the self-hosted picture from memory even when the team runs on Cloud. The reason is that Cloud behaves the way the architecture implies: its rate limits, its ingestion semantics and its retention tiers all follow from the same design.

The frontend is an Nginx container that serves the UI and routes API calls, and it is the only component that has to be reachable from outside the cluster. The backend handles CRUD operations and the prompt hub API. The platform-backend handles authentication and run ingestion, which is the traffic that scales with your users rather than with your engineers. The queue and ingest-queue services persist work asynchronously with retries, so a slow database write does not become a failed request to your application. The playground proxies calls from the UI to model providers, and the ACE backend runs the code evaluators your teams upload inside a sandbox, which is worth remembering whenever someone asks whether an evaluator can import an arbitrary package.

The stores are where operations concentrate. ClickHouse holds traces and feedback, and it is the component that determines how much data you can keep and how fast you can query it. PostgreSQL (version 14 or later, with the extensions btree_gin, btree_gist, pgcrypto, citext, ltree and pg_trgm) holds the operational data: workspaces, datasets, examples, prompts and rules. Redis (6.2 or later) or Valkey (8 or later) runs queues and caches. Blob storage is optional but recommended, because large inputs and outputs do not belong in a columnar store. Everything is bundled in the cluster by default, which is convenient for a proof of concept and wrong for production: use managed external stores so their backups, failover and patching are somebody's existing job.

The versions in the catalogue are worth stating precisely because LangSmith has no single product version. The Python SDK is langsmith 0.14.2 and the JavaScript SDK is 0.10.5, the command-line tool is langsmith-cli v0.2.60, and self-hosted installs follow Helm chart releases: stable is v16 (chart 0.16.36 at the time of writing) and the next major is in preview as v17 release candidates. Cloud has no version at all, because it is deployed continuously. When you write a runbook, name the component you mean.

There are three deployment shapes, and the choice is mostly a question of who holds the data and who holds the pager. Cloud is managed and upgrades itself, in GCP US (the default), GCP EU, GCP APAC and AWS US regions. Bring-your-own-cloud has LangChain run the control plane while the data stays in your VPC. Self-hosted runs entirely in your infrastructure, and you scale and upgrade it. Both BYOC and self-hosted require the Enterprise plan and a license key. For employers in the Gulf and Egypt the honest summary is that Cloud has no region in the Middle East, so a data-residency requirement that names a country pushes you toward self-hosted or BYOC, and a requirement that only names "the EU" or "Asia-Pacific" may be met by a Cloud region. Check the contract before the architecture: the docs list no EU contracting entity.

An organization lives in exactly one region and cannot be moved, which means the region decision is permanent in the way a database engine choice is permanent. Make it once, with legal in the room.

The ingestion path and what it does when things go wrong

A tracing system has an unusual property: it must never make the application it observes worse. Every design choice in the SDK follows from that, and understanding those choices is what lets you answer the two questions a platform owner gets most: "why is my trace missing" and "will tracing slow down production".

The Python SDK does not send each run as it finishes. It places run operations on a buffer and a background thread batches them, up to 100 runs per call, and posts them to the multipart ingestion endpoint. The application thread only pays for serialising the run and putting it on the queue. The consequences are all visible in the failure modes. If the process exits before the background thread has flushed, the last batch is lost, which is why short scripts need client.flush(), LangChain callbacks need wait_for_all_tracers(), and JavaScript needs await client.awaitPendingTraceBatches(). In serverless environments the process may be frozen after the handler returns, so the JavaScript setting LANGSMITH_TRACING_BACKGROUND=false makes tracing synchronous at the cost of latency.

When the backend is unavailable or slow, the SDK retries with backoff and then logs a warning and drops. It does not raise into your application. That is the correct behaviour for an observability client and it means that a LangSmith outage does not become your outage, but it also means trace loss is silent unless you look at the logs. A platform team should therefore decide whether loss is acceptable per application. For a chatbot it usually is. For an audit trail of regulated decisions it is not, and that application needs a different design, such as writing its own record of the decision and treating LangSmith as a convenience view. The client accepts a tracing_error_callback, which lets you count failed uploads in your own metrics rather than discovering them during an incident.

Rate limits are the other half. On Cloud the per-key limits per minute are 5,000 for POST and PATCH on runs, 5,000 for feedback posts, 30 for GET /runs/:id, 30 for DELETE /sessions, and 2,000 for everything else. Hourly event limits are 50,000 on Developer (250,000 with a card on file), 500,000 on Plus, and custom on Enterprise, with hourly data limits of 500 MB, 2.5 GB and 5 GB respectively. A Developer account without a card stops at 5,000 traces per month. A 429 surfaces in the SDK as a rate-limit error and the client retries, but a sustained overrun drops data. Two levers exist: sample (LANGSMITH_TRACING_SAMPLING_RATE, a float between 0 and 1) and reduce payload size.

Sampling deserves precision because the behaviour changed. From Python 0.12.0 trace sampling is deterministic, so the decision is a function of the trace rather than a fresh coin flip per run, which keeps a trace whole instead of keeping half its spans. That is what you want. It also means that if you sample at a rate of 0.1 and then wonder why the same failing conversation never appears, the answer is that it was consistently excluded.

Another default changed quietly. From 0.11.0 the SDK no longer re-sends inputs on the run PATCH (LANGSMITH_EXCLUDE_INPUTS_ON_PATCH defaults to true), which reduces bandwidth for large prompts but means inputs set after the initial post() are not uploaded. If a custom integration builds runs by hand with the RunTree API and assigns inputs late, they will be missing, and the fix is to set inputs at creation.

Each trace can hold at most 25,000 runs; beyond that, runs are rejected. A long-lived agent that loops for hours inside a single trace will hit this, and the design answer is to make each user turn a trace and group turns into a thread with a thread_id or session_id metadata key on every run, including children. A thread is the unit you reason about for conversations and the trace is the unit you reason about for a single piece of work.

import os
from langsmith import Client, traceable

failed_uploads = 0

def on_tracing_error(error: Exception) -> None:
    # Replace with a real metrics call (Prometheus counter, StatsD, etc.)
    global failed_uploads
    failed_uploads += 1

client = Client(
    api_url=os.environ.get("LANGSMITH_ENDPOINT"),
    tracing_sampling_rate=0.25,
    tracing_error_callback=on_tracing_error,
)

@traceable(client=client, run_type="chain", name="answer")
def answer(question: str) -> str:
    return question.upper()

answer("is tracing healthy?")
client.flush()   # short-lived processes must flush before exit

The same idea applies to the network. A corporate proxy or TLS-inspecting gateway that re-signs certificates produces a connection error from the SDK, and the correct fix is to trust the organization's root certificate in the runtime, never to disable certificate verification. Write that into the onboarding document once, because it is the most common first-week failure and the tempting shortcut is the wrong one.

Scaling: which component breaks first

Capacity planning for LangSmith is easier than for most systems, because the documentation publishes profiles and because the load has two independent axes: how many traces per second you write, and how many people query them. The scale guide describes three reference profiles. The default profile handles about 10 traces per second with about 5 concurrent UI users. A low-read, high-write profile targets 1,000 traces per second with about 5 users, and calls for roughly 20 platform-backend replicas, 24 ingest-queue replicas, 5 backend replicas, a 26 Gi external Redis, ClickHouse with 10 CPUs and 32 Gi of memory, and blob storage turned on. A high-read, high-write profile also targets 1,000 traces per second but with about 50 UI users, raises the backend to 50 replicas and moves ClickHouse to three nodes.

The pattern in those numbers is the lesson. Writes scale the ingestion tier (platform-backend, the queues, Redis) and ClickHouse insert capacity. Reads scale the backend and ClickHouse query capacity. A team that adds fifty analysts who filter across a month of traces is a different workload from a team that adds fifty services that emit traces, and the same hardware does not serve both. The recommended baseline cluster is at least 16 vCPU and 64 GB, in-cluster ClickHouse needs a node with 4 vCPU and 16 GB allocatable, and storage should be SSD class (the docs recommend 7,000 IOPS and 1,000 MiB/s) with volume expansion enabled. That last point is not decoration, as the incident section shows.

Autoscaling is available through HPAs and KEDA for the queues. The default HPA ranges include 2 to 6 for the backend and 3 to 10 for platform-backend and ingest-queue. Autoscaling the stateless tier is easy; the stateful tier is where the pager goes off.

In practice the order in which things break is usually this. First, payload size: a retrieval-augmented application that traces a 40-page context on every call produces more bytes than spans, and bytes, not run counts, exhaust disks and hit hourly data limits. Second, the ingest queue: when downstream writes slow, queue depth grows, and the symptom is traces that appear minutes late rather than not at all. Third, ClickHouse disk, which fails loudly with DB::Exception: Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert. (NOT_ENOUGH_SPACE). Fourth, ClickHouse CPU and memory under wide queries. Only then does the stateless tier become the constraint.

The disk-full incident has a specific and slightly surprising fix. You expand the persistent volume claim (data-langsmith-clickhouse-0), which requires a storage class with allowVolumeExpansion: true. Kubernetes does not let you change a StatefulSet's volume claim template, so you delete the StatefulSet while orphaning its pods with kubectl delete statefulset langsmith-clickhouse --cascade=orphan and then run helm upgrade with the new size in your values file, which recreates it around the existing volume. If the storage class does not allow expansion, discovering that during the incident is the worst time, which is why the review checklist asks about it up front.

The other failure that looks like a capacity problem but is not: 413 - request entity too large. That is an Nginx body limit, and the fix is frontend.maxBodySize: "100M" in your values file, not more replicas. Pair it with a conversation about why a single request is that large, because the better answer is usually to shrink payloads, move large blobs to attachments, or redact them.

Payload economics are the quiet driver of everything above, so build a view for them. The cost of a trace is a function of how many runs it has and how many bytes each carries. Two habits keep both small. First, trace the decision points rather than every helper function; a trace with six meaningful runs is more useful to a human than one with six hundred. Second, apply process_inputs and process_outputs to the functions that handle large objects (document lists, embeddings, images) so the trace records a summary such as the count and identifiers rather than the content. That serves scale, cost and privacy at once.

Finally, the query side. The fastest way to hurt a shared instance is an unbounded query from a script. runs/query has per-tenant limits of its own, and the new SmithDB-backed client.runs.query() defaults to a one-day window and returns only id unless you ask for more fields, both of which protect you. Teach people to pass an explicit time range and the fields they need, and to use cursors with page_size (default 100, maximum 1,000) rather than pulling everything.

The SmithDB migration

This is the change a senior owner must understand best, because it has a deadline and because it breaks code that works today. SmithDB is a new columnar observability datastore released in May 2026. It keeps durable data in object storage with a per-pod disk cache and serves new /v2/* query endpoints. It replaces ClickHouse as the query engine for runs, traces and threads, and the old query surface is being retired.

The deprecated Python methods are read_run, read_thread, list_runs, list_threads, get_run_url, share_run, unshare_run, read_run_shared_link, read_shared_run and list_shared_runs. In SDK 0.14.2 each emits a warning of the form "<method>() is deprecated and will be removed after Jan 31, 2027.". The replacements require Python langsmith>=0.10.15 or JavaScript langsmith>=0.8.9, CLI v0.2.44 or later, and, on self-hosted, version 0.16 or later.

The mapping is mostly mechanical, and the platform owner's job is to find every call site across every team before the deadline, not to rewrite each one personally.

The mapping table is the easy part. The parameter changes are where silent bugs live, and every one of these has caught a real team.

• project_name is gone. Pass project_ids=[uuid] and resolve the UUID with client.read_project(project_name=...).
• The time window defaults to one day. start_time became min_start_time and defaults to 24 hours ago. A query without a time bound does not fail; it quietly covers only yesterday, and a dashboard built on it shows a plausible but wrong number.
• selects defaults to id only. Field names are uppercase ("NAME", "STATUS", "INPUTS"), and so are run_type values ("LLM", "CHAIN", "TOOL", "RETRIEVER", "EMBEDDING", "PROMPT", "PARSER").
• Renames: error became has_error, run_ids became ids, and reference_example_id became reference_examples, a list.
• Paging: limit was removed in favour of page_size and cursor. query and parent_run_id were removed without a replacement, while filter, trace_filter, tree_filter and is_root are unchanged.

import asyncio
from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

async def slow_llm_runs(project: str):
    # Legacy: client.list_runs(project_name=project, run_type="llm", error=False)
    project_id = (await client.aread_project(project_name=project)).id
    async for run in client.runs.query(
        project_ids=[project_id],
        min_start_time=datetime.now(timezone.utc) - timedelta(days=7),
        selects=["NAME", "STATUS", "INPUTS"],
        filter='and(eq(run_type, "llm"), gt(latency, 5))',
        page_size=500,
    ):
        print(run)

asyncio.run(slow_llm_runs("my-app"))

The filter language itself is unchanged. It uses functions such as eq, neq, gt, gte, lt, lte, has, search, and and or, plus trace_filter and tree_filter for conditions on the root run or sibling runs. That continuity is a gift: your saved filters keep working, and only the calling code changes.

Feedback has its own migration. Creating feedback for a run should now pass session_id, the project UUID. Without it the SDK warns that the behaviour "is deprecated and will stop working in a future release", and on a SmithDB-only deployment it raises ValueError: session_id must be provided when creating feedback for a run: this deployment cannot locate the run without it. The reason is architectural: a columnar store partitioned by project cannot cheaply find a run from its ID alone. Similarly, annotation-queue helpers want runs=[{"run_id", "session_id", "start_time"}] rather than a bare list of run IDs. The new SDK methods also raise a new exception hierarchy (LangsmithError with AuthenticationError, RateLimitError, NotFoundError, APIConnectionError and others, all importable from langsmith). Note the casing change from LangSmithError, and note that code which catches the old classes around a migrated call will stop catching anything.

On self-hosted there is an additional trap with a clear rule. SmithDB is opt-in on self-hosted. ClickHouse is required on 0.16 and remains the default on 0.17, and you must not disable ClickHouse below 0.17. The deprecated methods stop working once ClickHouse is disabled, so the order of operations is: migrate every caller first, confirm no deprecation warnings remain, then consider the infrastructure change. SmithDB adds its own services (ingestion, query, compaction workers, compaction, cluster manager), its own Postgres metastore, object storage and a disk cache, so it is a different capacity and cost profile and not merely a swap.

# Find every call site of the deprecated query surface across checked-out repos.
grep -rnE "\.(read_run|read_thread|list_runs|list_threads|get_run_url|share_run|unshare_run|read_run_shared_link|read_shared_run|list_shared_runs)\(" \
  --include="*.py" . 

# JavaScript / TypeScript equivalents
grep -rnE "\.(readRun|listRuns|getRunUrl|listThreads)\(" --include="*.ts" --include="*.js" .

# Fail the test run on the deprecation text itself
python -W "error:.*is deprecated and will be removed after Jan 31, 2027" -m pytest -q

Security: the trust model

Start with what LangSmith stores, because every security question is an answer to that. A trace contains the inputs, outputs and metadata of every instrumented call. For an LLM application that means prompts, which contain user messages, which contain whatever users type. It also contains retrieved documents, tool arguments and tool results. LangSmith is therefore, by construction, a copy of the most sensitive text flowing through your AI system, searchable and retained. Treat the project as a data store with the classification of its most sensitive contents, not as a log.

There are three trust boundaries to reason about: from your application to LangSmith, between your users and each other inside LangSmith, and between LangSmith and the model providers it calls for the playground and evaluators.

Application to LangSmith. The credential is an API key, and there are two kinds. Personal access tokens (prefix lsv2_pt_) inherit the permissions of the person who created them and belong in personal scripts. Service keys (prefix lsv2_sk_) are created by administrators, scoped to one or more workspaces or the whole organization, and belong in applications. The old ls__ keys stopped working on 22 October 2024, which matters only because old blog posts still use them. Keys do not expire by default, so set an expiry through policy. An administrator can change a service key's role without rotating it, PATs can be deactivated and reactivated while service keys can only be deleted, and deactivation takes effect within about a minute because authentication is cached. Organizations can disable PAT creation altogether, which is the right setting when service keys are the sanctioned path.

An organization-scoped service key spans workspaces, and it therefore requires you to say which one you mean. The symptom of forgetting is a 403 with the code org_scoped_key_requires_workspace, and the fix is LANGSMITH_WORKSPACE_ID, Client(workspace_id=...) or the X-Tenant-Id header on raw REST calls. The header name is a historical artifact: "tenant" is now called workspace in the product, while the API still says tenant.

Redaction. The SDK gives you four layers, and a platform team should pick one as the default and make it hard to bypass. The blunt layer is LANGSMITH_HIDE_INPUTS, LANGSMITH_HIDE_OUTPUTS and LANGSMITH_HIDE_METADATA, which remove the whole payload; it is right for applications where you want shape and latency but no content. The rule layer is create_anonymizer from langsmith.anonymizer, passed as Client(anonymizer=...), which rewrites matching strings with patterns. The callable layer is hide_inputs and hide_outputs on the client, where you write a function that returns a safe dictionary. The function layer is @traceable(process_inputs=..., process_outputs=...), which is the most precise because the function knows its own payload, and it takes precedence over the client-level settings. The docs show recipes using Presidio and Amazon Comprehend for named-entity detection.

The important security property changed in Python 0.12.0. Per-function redactors now fail closed. If your redaction function raises, the payload is replaced with "Processing failed; inputs dropped (process_inputs: <ExceptionType>)" rather than being sent unredacted. That is the behaviour you want from a privacy control, and it also means a bug in a redactor shows up as traces full of that message instead of as a leak. You can restore the old fail-open behaviour with LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS, and a platform owner should treat that variable as one requiring a written exception.

import re
from langsmith import Client, traceable
from langsmith.anonymizer import create_anonymizer

# Rule-based anonymizer: applied to every payload this client sends.
anonymizer = create_anonymizer([
    {"pattern": re.compile(r"\b\d{10,14}\b"), "replace": "<ID-NUMBER>"},
    {"pattern": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "replace": "<EMAIL>"},
])
client = Client(anonymizer=anonymizer)

def summarise_docs(inputs: dict) -> dict:
    # Keep identifiers and counts, drop the raw text.
    docs = inputs.get("docs", [])
    return {"doc_count": len(docs), "doc_ids": [d["id"] for d in docs]}

@traceable(client=client, process_inputs=summarise_docs)
def answer(docs: list[dict], question: str) -> str:
    return "..."

Test redaction the way you test any security control: with a corpus of synthetic sensitive strings and an assertion that none survive in what the client would send. Redaction that has never been tested is a hope.

Distributed tracing is a trust decision. To join a trace across services, the caller sends the langsmith-trace and baggage headers produced by run_tree.to_headers(), and the receiver continues the trace using TracingMiddleware (for Starlette or FastAPI) or tracing_context(parent=headers). Those headers tell the receiver which trace and project to attach to. A service that accepts them from the public internet lets any caller inject runs into your traces and influence their metadata. Accept them only from services you control, and strip them at the edge.

Between users. Organization roles are Admin, Operator, User and Viewer, and a Viewer cannot create PATs. Inside a workspace the roles are Admin, Editor and Viewer. Workspace RBAC with custom roles, attribute-based access policies on resource tags, and audit logs are Enterprise features, so if a compliance requirement says "we must be able to show who read this trace", that is a plan decision before it is a configuration one. SSO through SAML or OIDC supports just-in-time provisioning, SCIM and group sync, and removing a user from all mapped groups revokes their SSO-granted access at next sign-in; it is not instantaneous, and you should account for that window in offboarding. Custom-role permissions were renamed: repos:* became prompts:* and run-rules:* became rules:*, so an old role definition in Terraform may need updating.

Between LangSmith and providers. The playground and the evaluators call models using keys stored as workspace-scoped secrets. Those keys are a real credential with a real budget. Give each workspace its own provider key, set spend limits at the provider, and remember that an evaluator running an LLM judge over a large dataset is a job that spends money without anyone deploying anything.

On self-hosted, the supply chain has been strengthened: from v0.16 the images ship with Cosign signatures and signed SBOM attestations. Verify them in admission control rather than assuming, mirror the images to your own registry if your cluster cannot reach the public one, and note that a license check calls out to https://beacon.langchain.com unless you run in offline mode. A firewall rule that blocks it will turn a healthy install into a licence failure, and some features, such as Engine, cannot run air-gapped at all. The platform also holds compliance attestations (SOC 2 Type 2, HIPAA and GDPR are listed), which is useful evidence for a vendor review, but it is evidence about LangChain's controls, not about yours.

Multi-tenancy: organizations, workspaces and applications

When many teams share LangSmith, the question is what isolates them and what they can still see of each other. The hierarchy is Organization, then Workspace, then Application (built on resource tags), then the resources themselves: projects, prompts, datasets, deployments and annotation queues. Some things are scoped higher than you would guess. API keys, secrets, rules and feedback configuration are workspace-scoped, and billing is organization-scoped. The word "tenant" in older material and in the API headers means workspace.

The docs describe three models for dividing work, and they are worth naming because platform teams tend to reinvent one of them badly. A team-centric model gives each team a workspace, which is the simple default and the right one when teams do not need to see each other's prompts and datasets. A collaborative model uses a shared workspace with access controlled by roles and tags, which suits a small company where everyone reads everything. A project-isolated model puts sensitive projects in their own workspaces regardless of team, which suits regulated work.

The decision that surprises people is environments. Do not use one workspace per environment. Resources cannot be shared across workspaces, so a prompt authored in a "dev" workspace cannot be promoted into a "prod" workspace; you would copy it, and the copy has no lineage. Instead use resource tags such as Environment: dev, staging and prod inside a workspace, and use prompt commit tags or Environments (the reserved staging and production labels) to promote versions. Attribute-based access policies on those tags let you say that only release engineers can modify anything tagged production, which is the control a change-management auditor wants to see.

Isolation is also a keys question. A service key scoped to one workspace cannot read another's traces, so issue keys per application rather than per team, and prefer workspace-scoped over organization-scoped wherever you can. Organization-scoped keys are for automation that genuinely spans teams, such as the Terraform provider managing workspaces and roles, which needs an organization-scoped service key with Organization Admin permissions. Every such key is a high-value credential, so keep the set small and keep it in a secret manager.

Naming and tagging discipline is what makes multi-tenancy usable in practice. Agree a convention before the tenth team arrives: project names that encode application and environment, required tags (team, environment, data-classification), and required metadata on traces (application version, deploy SHA, model name). Filters and alerts work on tags and metadata, so the conventions are what turn a pile of traces into something a platform owner can ask questions of. Enforce them in a shared wrapper library rather than in a wiki page, so that the default path is the compliant path.

A shared library is, in fact, the most effective lever a platform owner has. A thin internal package that configures the Client (endpoint, sampling, redaction, error callback), sets default tags and metadata, and exposes a traced decorator means teams get correct behaviour by importing it. When the SmithDB deadline or a redaction policy changes, you change one package.

Cost and retention governance

LangSmith bills for traces, and the pricing model has one feature that catches nearly every team once: two retention tiers. Base traces are kept 14 days. Extended traces are kept 180 days, a figure that changed on 14 September 2026 when the tier was shortened from 400 days, and the change applies to new traces only. Extended retention costs more. The plan prices at the time of writing are Developer at $0 with one seat and 5,000 base traces a month, Plus at $39 per seat per month with 10,000 base traces and one free small Serverless deployment, and Enterprise at custom pricing; per-trace rates should be read from the pricing page on the day, not from a guide. Newer products are billed in LangSmith Compute Units ($1.50) and LangSmith Standard Units ($1.00).

The trap is that retention extension happens automatically. Online evaluators, automation rules with retention extension enabled, and SDK feedback sent with extend_trace_retention=True each upgrade the whole trace to extended retention, and extension is on by default for newly created evaluators and rules. Experiments are also created at extended retention. A well-meaning engineer who adds an online evaluator running over ten percent of production traffic has just moved ten percent of the company's traces to the expensive tier, for the lifetime of the evaluator, without a deployment, a pull request or a cost review.

The cost levers, in order of effectiveness, are these. Keep projects at base retention by default and extend deliberately, for example by sending only failures or low-scoring traces to a dataset (which never expires) instead of extending retention for everything. Sample: a rate of 0.1 on a healthy high-volume endpoint loses little information and cuts volume by 90 percent, while a rule that always captures errors keeps the interesting tail. Set usage limits per workspace, per project or per user, so a runaway loop has a ceiling. Set spend limits on evaluators. And reduce payload size, which affects hourly data limits even when the trace count is fine.

Datasets are the retention escape hatch and the governance tool in one. They never expire, so a trace that matters should be copied into a dataset as an example rather than retained as an extended trace. That also produces something useful: a regression suite built from real failures. Establish the habit that an incident's trace goes into a dataset before anyone closes the ticket.

Cost attribution across teams needs the tags from the previous section. Model cost is computed automatically for known models when token counts are present; for custom model runs you set run_type="llm", add ls_provider and ls_model_name metadata, and supply usage_metadata. Model prices are configurable in the workspace, and cost is computed from the most specific token type to the least specific, so cached-input tokens are priced before plain input tokens. A platform team that wants showback should export data rather than screen-scrape the UI. Bulk data export writes Parquet to S3 or an S3-compatible store (GCS and MinIO are supported), zstd-compressed by default since 0.16, and that is the input for a cost report in your warehouse.

Alerts close the loop. LangSmith alerts can fire on run count, cost, error count or rate, feedback score and latency, over 5 or 15 minute windows, and notify PagerDuty or a webhook. A cost alert per workspace is cheap insurance and is the answer to the question "how will we find out a rule went wrong".

Evaluation as release infrastructure

At the mid level evaluation was a practice: datasets, evaluators, experiments. At this level it is infrastructure with owners, and the failure modes are organizational. The dataset rots, the judge drifts, the threshold is loosened until it stops blocking anything, and six months later the green check means nothing.

Start with ownership. Every dataset that gates a release needs a named owner and a change process, because adding or editing examples changes what "passing" means. Version the dataset (it supports versions and splits) and record the dataset version with each experiment, so a score drop can be attributed to the code, the model or the data. Evaluators are workspace-level resources, and the platform question is who may change the one that gates production. An LLM-as-judge prompt is a piece of code with a behaviour; change it and every historical comparison becomes suspect. Pin the judge model, keep judge prompts in the prompt hub with commit tags, and run a small calibration set of human-labelled examples through the judge periodically to measure its agreement with people.

Offline evaluation runs on a dataset before deployment and can use reference outputs. Online evaluation runs on live runs or threads without references and answers a different question: not "is the new version better" but "is production healthy right now". Use both, for different decisions. A gate built only on offline scores tells you about the dataset; a gate built only on online scores tells you after the fact.

The CI integration is the pytest plugin (pip install -U "langsmith[pytest]") and the Vitest or Jest equivalent. Tests marked @pytest.mark.langsmith log inputs, reference outputs and feedback through langsmith.testing, and run with LANGSMITH_TEST_SUITE naming the suite. LANGSMITH_TEST_CACHE caches HTTP calls to cassettes so a test run in CI does not re-spend model tokens, and LANGSMITH_TEST_TRACKING=false turns off the upload for local runs. Nondeterminism is the senior problem here: a model-backed test that fails one run in twenty trains the team to rerun until green. Use num_repetitions in evaluate(), assert on aggregate scores with a tolerance, and separate cheap deterministic checks (format, schema, latency) that block a merge from expensive judged checks that block a release.

from langsmith import Client

client = Client()

def target(inputs: dict) -> dict:
    return {"answer": "..."}  # call the application under test here

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    ok = outputs["answer"].strip().lower() == reference_outputs["answer"].strip().lower()
    return {"key": "correctness", "score": 1.0 if ok else 0.0}

results = client.evaluate(
    target,
    data="release-gate-v3",
    evaluators=[correctness],
    experiment_prefix="pr-check",
    num_repetitions=3,
    max_concurrency=4,
    metadata={"git_sha": "abc1234", "model": "pinned-model-id"},
    disable_evaluator_tracing=True,   # added in 0.13: keeps judge traces out of the project
)

scores = [r["evaluation_results"]["results"][0].score for r in results]
mean = sum(scores) / len(scores)
assert mean >= 0.85, f"release gate failed: {mean:.2f} < 0.85"

Two details in that example are senior-level. disable_evaluator_tracing (added in 0.13.0) keeps the evaluators' own traces out of your project so a large experiment does not flood it with judge calls. And the gate threshold lives in version control, next to the code, so loosening it is a reviewable change rather than a setting someone edits in a UI. The composite-evaluator pattern replaces the old feedback formulas, which were discontinued and return 410 Gone: write a code evaluator that combines scores and attach it with a run rule.

Prompts as a deployable artifact

A prompt that lives in a hub is an artifact with a lifecycle, and the platform owner decides how it is promoted. Each change creates an immutable commit identified by a hash; tags are movable labels on commits, and staging and production are reserved for environments. Applications pull by name (latest), by name:commit_hash (pinned) or by name:tag (floating). The choice is the risk decision. Pulling name:production means a prompt change ships without a code deploy, which is the feature for fast iteration and the hazard for change control. Pulling a pinned commit means every prompt change needs a code change, which is slower and fully auditable.

Most teams land on a hybrid: floating tags in development and staging, and a process in which promotion to production is a gated action (an evaluation run against the release dataset passes, then a person with the right role moves the tag). Restrict who can move that tag with the access policies described earlier, and record who did it; audit logs on Enterprise cover this.

The SDK caches pulled prompts, on by default since Python 0.7.0 and JavaScript 0.5.0, with a 100-entry maximum, a 300-second time to live, a 60-second background refresh and stale-while-revalidate behaviour. That has two consequences a platform owner must know. First, a tag move takes up to a few minutes to reach running instances, so a rollback is not instant and incident runbooks should say so. Second, if LangSmith is unreachable, cached prompts keep serving, which is resilience you want. Tune it with configure_global_prompt_cache(ttl_seconds=None) for an offline-friendly cache that never expires, or construct Client(disable_prompt_cache=True) when freshness matters more than availability. Older code may use Client(cache=...), which is deprecated, and the langchainhub package is deprecated too since prompt push and pull live in langsmith.

Pulling a public prompt from another owner (owner/name) executes someone else's template in your application. The dangerously_pull_public_prompt option is named that way on purpose. Treat it like a dependency: pin a commit, review it, and prefer copying into your own workspace.

Upgrades and migrations

For Cloud, upgrades are invisible and the work is watching the changelog, which ships weekly. For self-hosted, upgrades are a procedure, and the release policy determines it. There are two channels: stable, which receives weekly patches containing critical fixes and security patches only, and preview, the next major, published as 0.X.0-rc.N. A new stable major arrives about every six weeks. Supported versions are N (active) and N minus one (critical fixes only); anything older is end of life. New or removed services, breaking changes and data migrations arrive only in a new major.

That cadence is faster than most organizations' change windows, and it is the operational reality of owning the install. If you do not plan for it, you fall two majors behind and discover that there is no direct path, because you must upgrade one major at a time (0.13 to 0.14 to 0.15, never 0.13 to 0.15), and downgrades are not supported. A failed upgrade is therefore recovered by restoring from backup, which makes backups of PostgreSQL, ClickHouse and blob storage the precondition of the upgrade and not a separate project.

set -euo pipefail
NS=langsmith
REL=langsmith
TARGET="0.16.36"    # chart version, always pinned explicitly

# 1. Read the changelog for this major: breaking changes land only in majors.
# 2. Back up PostgreSQL, ClickHouse and blob storage. There is no downgrade.

helm repo update
helm upgrade "$REL" langchain/langsmith \
  --version "$TARGET" \
  --values langsmith_config.yaml \
  -n "$NS" --wait --debug

helm status "$REL" -n "$NS"
kubectl get jobs -n "$NS" | grep -E 'pg-migrations|ch-migrations'
# Both migration jobs must show Completions 1/1 before you call it done.

The 0.16 major is a useful case study because it carried several things at once. The go-backend, playground and host-backend images were consolidated into smith-backend, so any values.yaml override of those images must be removed or it will fight the chart. Bulk exports now default to zstandard compression, which matters to any downstream reader. The agent-bootstrap script was deleted and Fleet (formerly Agent Builder) must move to a standalone deployment. A new backfillCheck job blocks the upgrade until required checks complete, and installs using IAM authentication may need to add matching annotations and labels. None of those is hard, but all are invisible unless someone read the release notes, which is the point of step one.

A failed ClickHouse migration can leave error: Dirty database version 'version'. fix and force version. The documented recovery is to run the migrate tool with force <dirty-1> from the backend pod and rerun the upgrade, and you should only do that after confirming from the migration job's logs which version failed. Rehearse it: run every upgrade in a staging install first, and run the preview release candidates there too, because that is how you learn about a breaking change before it reaches you as stable.

Migrations between deployment shapes are a different class of work. Moving from Cloud to self-hosted or the reverse is not a one-click operation, and because an organization cannot change region, a region change means a new organization. Datasets, prompts and evaluators can be exported and recreated through the SDK and CLI (langsmith dataset export, langsmith dataset upload); historical traces are a separate question, answered by bulk export to Parquet. Decide up front what you are prepared to lose, and say it in writing to the teams affected.

The same discipline applies to the endpoint deprecation policy. Deprecated endpoints are marked with Deprecation: true and Sunset: <date> response headers and an SDK warning, with a minimum window of six months on Cloud and at least one major release on self-hosted. A cheap control is a proxy or gateway log query for those headers, which finds callers that no grep will: scripts on someone's laptop, a notebook, a vendor integration.

Governance and incident playbooks

Governance for a tracing platform is mostly about three questions: who may see what, what is kept for how long, and who may change what gates a release. The earlier sections gave the mechanisms. What is left is to write them down as policy and to rehearse the incidents, because the first time you run an incident playbook should not be during the incident.

A short policy set is enough, and each item should be one page. A data classification policy says which applications may send content to LangSmith, which must redact, and which may send only metadata with LANGSMITH_HIDE_INPUTS and LANGSMITH_HIDE_OUTPUTS. A retention policy says that projects default to base retention, names who may enable extension and for what, and says that evidence worth keeping goes into datasets. An access policy says who holds organization roles, who may create service keys, the required expiry, and that PAT creation is disabled if service keys are the sanctioned path. A change policy covers prompt promotion, evaluator changes and gate thresholds. Publishing these is what lets you say no to a request with a reference rather than an opinion.

The playbooks below each follow the same shape: symptom, first check, likely cause, fix, and what you change afterwards.

Playbook one: traces are missing. Symptom: a team reports an empty project or gaps. First check which of four causes applies, in order of frequency. Did the process flush before exit? Are the key and region right (look for LangSmithAuthError: Authentication failed for /runs/multipart and the log line Failed to multipart ingest runs, and check for a trailing slash on LANGSMITH_ENDPOINT)? Is sampling set lower than the team believes? Is the key organization-scoped without a workspace (org_scoped_key_requires_workspace)? In notebooks, cached environment variables send traces to the wrong project, and the fix is utils.get_env_var.cache_clear() followed by reloading .env with override. Afterwards: add the error callback metric so the next occurrence is visible to the team before it is visible to you.

Playbook two: ingestion is slow or rejected. Symptom: traces arrive minutes late, 429 responses, or 413. First check the ingest-queue depth and platform-backend saturation; for 413 check frontend.maxBodySize; for 429 on Cloud identify which limit (per-minute key limit, hourly events, hourly bytes, the Developer monthly cap or a usage limit you set yourself). Fix with sampling and payload reduction before capacity, because adding replicas to absorb oversized payloads moves the bottleneck to ClickHouse. Afterwards: a dashboard for queue depth and for bytes per trace.

Playbook three: ClickHouse disk full. Symptom: NOT_ENOUGH_SPACE and failing inserts. Expand the data-langsmith-clickhouse-0 volume, orphan-delete the StatefulSet, and run helm upgrade with the larger size, as described in the scaling section. While you work, the queue absorbs writes, but only for so long, so communicate that recent traces may be delayed. Afterwards: a disk-usage alert at 70 percent, a TTL policy that matches your retention policy (self-hosted TTLs are configurable and a cleanup job exists), and verification that the storage class permits expansion.

Playbook four: a failed upgrade. Symptom: a migration job fails, a pod crash-loops, or the backfillCheck job blocks. First read the migration job logs to learn which step failed. For a dirty ClickHouse migration use the documented force procedure from the backend pod, and for a blocked upgrade satisfy the required check. Do not attempt a downgrade; there is no supported path, so the fallback is restore from the pre-upgrade backup. Afterwards: a rehearsal in staging, which would have found the problem.

Playbook five: a secret or personal data was traced. Symptom: someone finds a credential or a national ID number in a trace. This is an incident, not a cleanup task. Contain first: rotate the credential, because deleting the trace does not un-expose it. Then delete the affected traces on demand (the manage-trace documentation covers this, and self-hosted has a delete script), check the datasets, since examples copied from traces survive deletion of the trace, check exports and any shared links, and check whether prompts in the hub embed the secret. Fix the cause in the redaction layer and add the string to your synthetic redaction tests. Notify under your own breach policy; the platform's compliance attestations do not discharge your obligations.

Playbook six: a bad prompt reached production. Symptom: quality or safety metrics drop after a tag move. Move the production tag back to the last known-good commit, remembering that clients with a warm cache may serve the old prompt for up to the cache interval. Preserve the bad commit and the traces that show the failure by copying them into a dataset as a regression case. Afterwards: add that case to the release gate.

Playbook seven: the SDK or API changed under you. Symptom: a new warning, a 410 Gone, or an unexpected behaviour change after an SDK bump. Common cases from this year include the fail-closed redactors in 0.12, the input-on-patch default in 0.11, the feedback formula endpoints returning 410, and POST /feedback/eager being removed in favour of POST /feedback. Read the release notes for the versions between the last good and the current one, pin the SDK in lockfiles, and upgrade in a branch with the test suite running.

Where LangSmith stops

An honest platform owner knows when to recommend something else, and saying so early builds more trust than defending a tool past its fit. LangSmith is strongest when the application is built with LangChain or LangGraph, since those trace with environment variables alone, and it has wrappers and integrations for the OpenAI, Anthropic and Gemini clients and many agent frameworks. Its evaluation, prompt and annotation features live in one place next to the traces, which is a real advantage over stitching tools together.

It is a weaker fit in several situations. If your hard requirement is a fully open-source stack you can run and modify without an enterprise license, note that self-hosting requires the Enterprise plan and a license key; open-source observability tools such as Langfuse (see the Langfuse guide) and Arize Phoenix (see the Phoenix guide) have different licensing and self-hosting stories and are the natural comparison. If the requirement is data residency in a country without a Cloud region, your options are self-hosted or bring-your-own-cloud, at Enterprise pricing, or a different tool. If your organization already runs an OpenTelemetry-based observability stack and wants LLM traces in the same backend as everything else, LangSmith accepts OpenTelemetry (OTEL_EXPORTER_OTLP_ENDPOINT=https://api.smith.langchain.com/otel, with headers x-api-key=<key>,Langsmith-Project=<project>), and LANGSMITH_TRACING_MODE=hybrid sends to both, but it does not replace your general-purpose APM, and the mapping of GenAI, OpenInference, Traceloop and Logfire attributes is an interpretation layer to validate against your own data.

LangSmith is not your metrics system. Dashboards and alerts exist, but service-level objectives for a product should live in the monitoring stack your on-call already watches, with LangSmith as the place you go to explain why a number moved. It is not your data warehouse either: for analysis beyond the built-in views, export to Parquet and query it where your analysts already work. It is not a substitute for application-level audit records in regulated flows. And for evaluation-specific work, libraries such as RAGAS or DeepEval can run independently of the platform and be logged into it, so evaluation logic need not be locked to one vendor.

Finally, the exit strategy. Design it before you need it. Tracing through OpenTelemetry keeps your instrumentation portable, datasets and prompts can be exported through the SDK and CLI, and periodic bulk exports preserve history. A platform you can leave is one you can negotiate with.

Running LangSmith as a platform for other teams

The technical material above matters less than how you run the service. A platform is a product with customers, and its customers are engineers who will route around you if you are slow.

Make the default path the easy path. Provide the internal wrapper package, a project template with tracing, redaction and an evaluation gate preconfigured, and a single onboarding document that ends with a first trace in the UI. The measure of success is time to first trace for a new team, and it should be under an hour.

Automate the provisioning. The Terraform provider (langchain-ai/langsmith, documented with a ~> 0.0.6 pin) manages workspaces, custom roles, members, evaluators, run rules and alert rules, taking api_url, api_key, workspace_id and profile as arguments. Organization-level resources need an organization-scoped service key with Organization Admin permissions. Putting workspace creation, roles, rules and alerts in Terraform gives you review, history and repeatability, and it turns "please add me to the workspace" into a pull request. The provider is young, so pin its version and read its changelog.

terraform {
  required_providers {
    langsmith = {
      source  = "langchain-ai/langsmith"
      version = "~> 0.0.6"
    }
  }
}

provider "langsmith" {
  api_url = "https://api.smith.langchain.com"
  api_key = var.langsmith_admin_key   # organization-scoped service key, from a secret manager
}

Script the rest with the CLI and API. The langsmith CLI (v0.2.60 at the time of writing) supports profiles (langsmith profile create dev --workspace-id <id> --set-current), browser-based login with langsmith auth login, and a raw API passthrough (langsmith api sessions?limit=5, langsmith api ls --tag datasets) that reads the cached OpenAPI spec. On self-hosted, OAuth login needs CLI v0.2.46 or later, LangSmith 0.16 or later and a signing JWKS configured in the chart. One trap worth a line in the runbook: the LANGSMITH_ENDPOINT environment variable overrides a profile's api_url, so a CLI pointed at self-hosted through a profile will silently talk to Cloud if the variable is set in the shell.

Publish the service's own SLOs. Decide and publish what you promise: ingestion latency, availability of the UI and API, the upgrade cadence and notice period, and the support path. For self-hosted, queue workers emit Prometheus metrics and SmithDB has its own metrics; wire them into the same on-call tooling as your other services. Offer a debugging bundle for support requests: the helm repo provides get_k8s_debugging_info.sh --namespace langsmith, and HAR files help for browser problems.

Run a cadence. A weekly review of the changelog, a monthly review of usage and cost by workspace, a quarterly review of service keys and roles, and an upgrade every major inside the support window. Each is small; skipping any of them is how a platform decays.

Teach. The fastest way to reduce incidents is to train the teams. A one-page guide to "why is my trace missing" resolves most tickets at the source. A short guide on writing evaluators that do not flake reduces pager noise from gates. Link them from the onboarding document and from the error messages your wrapper produces.

Connect the neighbours. LangSmith rarely stands alone. Agent frameworks trace into it (see LangGraph and LangChain), Langfuse is the common open-source comparison, LiteLLM can sit as a gateway in front of providers, and Kubernetes and Helm are the substrate when you self-host. Knowing those edges is part of owning the platform.

The review checklist

Use this for design documents and for periodic reviews of the install. Each question has a failure it is designed to catch.

Architecture and deployment

• Are production stores external and managed, with backups whose restore has been tested?
• Does the ClickHouse storage class allow volume expansion, and is there a disk alert?
• Is the region chosen with legal sign-off, knowing an organization cannot move?
• Is egress to the license endpoint allowed, or is offline mode configured deliberately?

Ingestion and scale

• Is there a measured traces-per-second and bytes-per-trace figure, and is it matched to a scale profile?
• Is tracing_error_callback wired to a metric, and do short-lived processes flush?
• Are large payloads summarised, offloaded or sampled?

Security

• Are applications using workspace-scoped service keys with expiry, and is PAT creation disabled or justified?
• Is apiKeySalt stored in a secret manager with a "never rotate" note?
• Is redaction tested with synthetic sensitive data, and is LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS unset?
• Are langsmith-trace headers accepted only from trusted services?
• Are image signatures and SBOM attestations verified in admission?

Tenancy and cost

• Is each team in its own workspace, with environments as tags rather than workspaces?
• Does every rule and online evaluator have a deliberate retention-extension setting?
• Are usage limits, spend limits and cost alerts set per workspace?
• Are incident traces copied to datasets rather than kept at extended retention?

Evaluation and prompts

• Does every gating dataset have an owner, a version and real production cases?
• Are judge models pinned and calibrated against human labels?
• Is the gate threshold in version control, with repetitions to absorb nondeterminism?
• Is promotion of a prompt to production gated and audited, and is rollback timed including cache?

Migration and lifecycle

• Has every call site of the deprecated query methods been found and scheduled, ahead of 31 January 2027 on Cloud and v0.18 on self-hosted?
• Is ClickHouse left enabled below 0.17?
• Is the install within N or N minus one, with a rehearsed upgrade in staging and a pre-upgrade backup?
• Is there an exit path: exports, portable instrumentation, evaluators in code?

The complete picture

Across the three levels, the subject was a system for seeing, judging and improving LLM applications. Beginner gave you the nouns: a run is one unit of work, a trace is a tree of runs, a thread is a conversation, and a project holds them. You learned to instrument with @traceable and the wrappers, to set the environment variables, and to read a trace. Mid gave you the practice: datasets and examples, evaluators and experiments, prompts in a hub, feedback, online evaluation and the integration with tests and CI. Senior gives you ownership.

The connecting idea is that the same traces serve four audiences who want different things: engineers debugging a failure, evaluators scoring quality, finance watching cost and security watching exposure. A platform owner's job is to make one stream of data safe and useful for all four. Redaction and sampling shape what enters. Retention and datasets shape what is kept. Roles and tags shape who sees it. Gates and alerts shape what it is allowed to change.

The recurring traps of the series all come from defaults that were reasonable for a single user and wrong for a platform: keys that never expire, retention extension that is on by default, a one-day query window, a fail-open anonymizer in older SDKs, in-cluster databases, and floating prompt tags. Most of a senior engineer's value is knowing which defaults to change before the first team arrives.

Where the series leaves you

You can now draw the architecture, name which component fails first under load, and explain why. You can plan and rehearse an upgrade under a six-week release cadence, and you know the dates that matter: the SmithDB legacy methods disappear from Cloud on 31 January 2027 and from self-hosted in v0.18. You can design tenancy with workspaces and tags, write redaction that fails closed and is tested, control cost by governing retention extension, and run evaluation as a gate that resists flakiness. You can also argue, with specifics, when LangSmith is and is not the right tool.

What comes next depends on your setting. If you operate a self-hosted install, the next skills are in the substrate: Kubernetes for the cluster, Helm for the chart mechanics, and Terraform for the infrastructure and the LangSmith provider. If you are building the evaluation culture, read RAGAS for retrieval-specific metrics and compare approaches in Langfuse. If you are building agents, go deeper on LangGraph. And keep the habit that matters most with a platform whose surface moves weekly: read the changelog, pin your versions, and treat every deprecation notice as a dated task rather than background noise.

The interview guide and the tips file in this level turn this material into practice questions and a production-trap checklist.

Sources

• LangSmith documentation (the old docs.smith.langchain.com address redirects here)
• Self-hosted overview
• Kubernetes installation
• Self-hosted scale guide
• Self-hosted dependency versions
• Self-hosted SmithDB
• Self-hosted upgrades
• Release versions policy
• Self-hosted changelog
• Cloud changelog
• Endpoint deprecation policy
• SmithDB SDK migration
• SmithDB migration: querying runs
• SmithDB migration: feedback
• Administration overview
• Role-based access control
• Audit logs
• Data storage and privacy
• Regions FAQ
• Usage and billing
• Pricing
• Mask inputs and outputs
• Distributed tracing
• Trace with OpenTelemetry
• Sample traces
• Cost tracking
• Alerts
• Data export
• Manage with Terraform
• LangSmith CLI
• Evaluation concepts
• pytest integration
• Manage prompts programmatically
• Troubleshooting
• langsmith on PyPI
• langsmith-sdk releases
• LangSmith Helm chart releases

Part three of three. A senior interview is less about recalling commands and more about showing that you can design, operate and defend a platform. This file has three parts: system-design prompts built around LangSmith, incident-response questions, and the conversations where you argue a trade-off to people who control budget. Read each prompt, answer it aloud before reading the model answer, and then check what a strong answer includes that a weak one leaves out. About fifty minutes.

The thirty-second answer

If you have half a minute, say this. LangSmith is a platform for tracing, evaluating and managing LLM applications. A run is one unit of work, a trace is the tree of runs for one operation, a thread groups traces into a conversation, and a project holds them. Datasets and experiments make quality measurable before release, online evaluators and rules make it measurable in production, and the prompt hub versions the prompts. It runs as managed Cloud in four regions, or self-hosted and bring-your-own-cloud on the Enterprise plan. As an owner, the things I watch are ingestion capacity, retention cost, data exposure in traces, and the upgrade cadence, including the SmithDB migration that removes the legacy query methods from Cloud on 31 January 2027.

What makes that a senior answer is the last sentence: it names operational concerns and a dated risk rather than features.

System-design prompts

Design LangSmith observability for forty teams

The prompt: a company of several hundred engineers is building LLM features in forty teams. Design the LangSmith setup.

A strong answer starts by asking questions before drawing anything. What data do the applications handle, and is there a residency or sector constraint? Cloud or self-hosted is decided by that answer: a country-specific residency requirement in the Gulf or Egypt rules out Cloud, since the regions are GCP US, EU, APAC and AWS US, and leads to self-hosted or bring-your-own-cloud on the Enterprise plan. What is the expected volume in traces per second and bytes per trace? How many people will query it?

Then the tenancy design. One organization, because it is one region and billing is organization-scoped. One workspace per team for isolation of prompts, datasets, keys and rules, with the option of separate workspaces for regulated projects. Environments as resource tags, not workspaces, because resources cannot be shared across workspaces and a workspace per environment destroys prompt lineage. Service keys per application, workspace-scoped, with expiry. Personal access tokens disabled if possible. Single sign-on through SAML or OIDC with group sync, and role-based access using custom roles, which is an Enterprise feature.

Then guardrails that scale to forty teams: a shared internal library that configures the client once, with sampling, redaction that fails closed, default tags for team, environment and data classification, and a tracing error callback wired to metrics. Usage limits per workspace and a cost alert per workspace. A default of base retention with deliberate extension.

Then provisioning: the Terraform provider for workspaces, roles and alert rules, so requests are pull requests.

Then the operating model: who owns the platform, what the service promises, the upgrade cadence and how teams are told about deprecations.

What a weak answer misses: it lists features and never asks about residency, never mentions retention extension as a cost driver, proposes a workspace per environment, and has no account of who operates it after launch.

Size and deploy a self-hosted install

The prompt: you must run LangSmith on your own Kubernetes cluster for about 800 traces per second at peak and thirty concurrent users.

Start from the scale guide's reference profiles rather than inventing numbers. The default profile is about 10 traces per second and 5 users. The high-write profiles target 1,000 traces per second, with roughly 20 platform-backend replicas, 24 ingest-queue replicas, 5 backend replicas, a 26 Gi external Redis and ClickHouse with about 10 CPUs and 32 Gi, with blob storage on. Thirty concurrent users is between the low-read and high-read profile, so I would start from the high-write configuration, size the backend up for reads, and expect to move ClickHouse to multiple nodes if query latency suffers. I would state that these are starting points to validate by load testing with realistic payload sizes.

Stores: managed PostgreSQL 14 or later with the required extensions, external Redis or Valkey, ClickHouse on a node with at least 4 vCPU and 16 GB, blob storage for large payloads, SSD storage classes with volume expansion. The cluster should have at least 16 vCPU and 64 GB as a baseline. Autoscale the queue tier with HPA or KEDA. Secrets come from an existing Kubernetes secret, referenced by config.existingSecretName, with the API key salt never rotated. Egress to https://beacon.langchain.com is allowed for the license check. Images are signature-verified.

Finally the operating plan: backups of Postgres, ClickHouse and blob storage, because there is no downgrade; a staging install that runs the preview release candidates; one major at a time within the N and N minus one support window.

What a strong answer adds: it identifies payload size as the first thing to break, proposes summarising large payloads, and raises the SmithDB decision, noting that ClickHouse must stay enabled below 0.17.

Design the release gate for a prompt change

The prompt: prompt changes are shipping without evaluation. Design the process.

Keep prompts in the prompt hub, with immutable commits and movable tags. Give each application a release dataset with an owner, made of real production cases, including regressions copied in from past incidents. A change to a prompt opens a pull request or a hub commit that triggers an evaluation in CI using the pytest or Vitest integration or evaluate(). Deterministic checks (schema validity, format, latency budget) block a merge because they are cheap and stable. Judged checks run with pinned judge models and repetitions, because model-backed scores are noisy, and a threshold that lives in version control. The gate compares against the current production baseline rather than against an absolute number.

Promotion to production is the tag move, done by a role-restricted person or pipeline after the gate passes. Rollback is moving the tag back, and the runbook says how long it takes given the client-side prompt cache, which defaults to a 300-second time to live. After release, online evaluators sample production traffic with retention extension turned off unless justified, and alerts watch the feedback score.

What a weak answer misses: the noise in model-backed scores, the cache delay on rollback, the ownership of the dataset, and the idea that a judge is itself a versioned artifact.

Migrate off the legacy query methods

The prompt: dozens of internal scripts and dashboards call list_runs. What is your plan?

First, establish the facts and the deadline: the methods are deprecated and will be removed from all Cloud regions on 31 January 2027, and from self-hosted in v0.18. The replacements need Python langsmith>=0.10.15 or JavaScript langsmith>=0.8.9, and self-hosted 0.16 or later. Then inventory: search every repository for the ten deprecated method names, and query gateway logs for Sunset headers to find callers that live outside source control. Classify callers by owner and risk.

Then provide a migration kit rather than asking forty people to read the docs: a mapping table, a helper that resolves a project name to its UUID, and a small wrapper with safe defaults. Brief the team on the silent traps: the default time window is one day, selects default to only the id, field names and run types are uppercase, project_name is gone, and paging uses cursors. Make CI fail on the deprecation warning text so regressions cannot return.

Sequence it: migrate read-only dashboards first, then automation, then anything that writes feedback, which also needs session_id. Only after every warning is gone would you consider the infrastructure change to SmithDB, and never disable ClickHouse below 0.17.

Incident-response questions

Traces from a production service stopped appearing

State the approach first: confirm the scope, then walk the causes in order of likelihood. Is it one service or all? One service suggests configuration; all suggests the platform or the network. Check for authentication errors in the application logs, such as Failed to multipart ingest runs or an authentication failure on /runs/multipart. The usual causes are an expired or deactivated key, a wrong regional endpoint, a trailing slash on the endpoint value, and an organization-scoped key without a workspace. Check whether a deploy changed sampling or disabled tracing. Check whether the process exits before flushing, which is common in serverless. If everything is down, check the platform's status and, on self-hosted, the ingest-queue depth and ClickHouse health. The fix follows the cause, and the follow-up is to add the tracing error callback metric so this becomes an alert.

A strong answer separates "is the platform down" from "is my configuration wrong" in the first minute and does not restart things blindly.

ClickHouse reports NOT_ENOUGH_SPACE

Recognise the message: DB::Exception: Cannot reserve 1.00 MiB, not enough space: While executing WaitForAsyncInsert. Inserts are failing, and the queue is absorbing writes for now, so time matters. Expand the persistent volume claim data-langsmith-clickhouse-0, which requires a storage class with allowVolumeExpansion: true. Because a StatefulSet's claim template is immutable, delete the StatefulSet with --cascade=orphan so its pods keep running, then run helm upgrade with the new size. Communicate that recent traces may be delayed. Afterwards add a disk alert, align the TTL with your retention policy, and review whether payload size is the underlying driver.

A developer found a customer's ID number in a trace

This is a data incident. Contain before cleaning up: if a credential is involved, rotate it immediately, because deletion does not retract exposure. Identify the scope: which project, which time range, which users could read it. Delete the affected traces on demand, then check the places a copy can survive: datasets built from those traces, bulk exports, shared links and webhook payloads. Find the cause in the instrumentation, typically a missing process_inputs or anonymizer rule, fix it, and add the pattern to a synthetic test corpus. Involve your privacy and security contacts under your own incident policy, since the platform's attestations do not cover your obligations. A strong answer mentions that since Python 0.12.0 a failing redactor fails closed, and asks whether LANGSMITH_ALLOW_UNPROCESSED_PAYLOADS was set.

The upgrade failed halfway

Do not improvise a rollback, because downgrades are not supported. Read the migration job logs to find the failing step: the langsmith-pg-migrations and langsmith-ch-migrations jobs. For a dirty ClickHouse state, error: Dirty database version 'version'. fix and force version, the documented route is to force the migration version to one below the dirty one from the backend pod and rerun the upgrade, but only after understanding what the failed migration did. If the state cannot be repaired, restore from the pre-upgrade backup, which is why the backup is a precondition. A strong answer says what they would do differently: rehearse in staging, run preview candidates there, and never skip a major.

The monthly bill doubled

Start with what changed, not with blame. Break the bill into base trace volume and extended retention upgrades. A doubling with flat traffic points to retention extension: a new online evaluator or automation rule, which extends retention by default, or feedback sent with extend_trace_retention=True. Find the workspace and project with the largest growth using usage data. Then fix: disable extension on the rules that do not need it, sample, add usage limits and a cost alert. Mention that evaluators also spend model tokens, and that extended retention is 180 days since 14 September 2026, shortened from 400 days, for new traces only.

Queries from a dashboard return only yesterday's data

The dashboard was migrated to client.runs.query(), whose min_start_time defaults to one day ago. Nothing failed; the window silently narrowed. The fix is an explicit min_start_time and, if needed, max_start_time, plus explicit selects, because the default returns only ids. The lesson for the interviewer is that the candidate recognises a default-changing migration and proposes a test that pins the expected count.

Arguing a trade-off to leadership

Senior interviews include a question where the listener is a director or a finance partner. The skill is translating engineering choices into cost, risk and time, without hiding uncertainty.

Cloud versus self-hosted. The honest framing is that self-hosted buys data control and residency at the price of operating a multi-service system with a six-week release cadence, plus Enterprise licensing. Cloud buys speed and zero operations, and moves the cost to the pricing model and to the data-residency limits. A good argument names the deciding factor rather than listing pros and cons: if a regulator or contract requires data to stay in-country, self-hosting or bring-your-own-cloud is not a preference but a requirement; if not, Cloud lets two engineers spend their time on evaluation instead of upgrades. Put a number on the operations side: how many engineers, how many upgrade windows a quarter.

Buy or build. A team proposing to build its own tracing will underestimate ingestion, retention, query and UI work. Argue from the platform's time to first trace and from the cost of owning a columnar store. Also be fair: if the organization already runs OpenTelemetry end to end, building a thin layer that exports spans to a backend may be cheaper than a new vendor. LangSmith accepts OpenTelemetry, so the decision does not have to be permanent.

Spending on evaluation. Leaders ask why an LLM judge over every production trace is worth paying for. A good answer proposes sampling, separates cheap deterministic checks from expensive judged ones, and ties evaluation to a business risk such as a regulated answer or a support deflection rate. It also shows the cost of not evaluating, with an example of a prompt regression that reached customers.

Saying no to a request. When a team wants ninety-day extended retention on all traces, the reply is not no but a counter-offer: extend selectively on failures, copy important cases into datasets, which never expire, and use the workspace usage limit as the guardrail. Always pair a refusal with a path to yes.

Reporting risk. The SmithDB deadline is a good example of how to brief leadership: what breaks (legacy query methods), when (31 January 2027 on Cloud, v0.18 on self-hosted), how many call sites exist, the effort per call site, and the date by which you will report completion. Precision beats alarm.

What to avoid. Do not quote per-trace prices from memory, since they change and the pricing page is authoritative. Do not promise a recovery time without a rehearsed restore. Do not describe compliance attestations as if they covered your own controls.

Common interview questions

What is the difference between a trace and a thread? A trace is the tree of runs for one operation, with a maximum of 25,000 runs. A thread is a sequence of traces forming one multi-turn conversation, linked by a thread_id or session_id metadata key on all runs, including children.

Why would you not create one workspace per environment? Resources cannot be shared across workspaces, so promotion would mean copying and lineage would be lost. Use resource tags for environments and prompt commit tags for promotion.

What does extending retention do and how does it happen by accident? It keeps a trace 180 days instead of 14, at higher cost. Online evaluators, rules with extension enabled, and feedback sent with extend_trace_retention=True upgrade a trace automatically, and extension is on by default for new evaluators and rules.

Which key type would you give an application and why? A workspace-scoped service key, because it is created by an administrator, is not tied to an employee, can have an expiry, and has a role an administrator can change without rotation. A personal access token inherits a person's permissions and disappears when they leave.

What is special about the API key salt in self-hosted? Rotating it invalidates every API key, so it must be stored securely and never rotated casually.

How do you upgrade a self-hosted install? Read the changelog, back up all stores, update the repo, upgrade one major at a time with a pinned chart version, verify the migration jobs completed, and rehearse in staging first. There is no downgrade.

What changed in the Python SDK that could affect a platform? From 0.11.0 inputs are not re-sent on the run patch by default; from 0.12.0 sampling is deterministic and per-function redactors fail closed; 0.13.0 added disable_evaluator_tracing; and the tracing mode setting replaces the older OpenTelemetry flags.

How would you propagate a trace across services safely? Send run_tree.to_headers() from the caller and use TracingMiddleware or tracing_context(parent=headers) on the receiver, and accept those headers only from services you control.

How do you send LangSmith traces from a non-LangChain service? Through the OpenTelemetry endpoint: set the exporter endpoint to the base .../otel URL, without appending /v1/traces, with the x-api-key and Langsmith-Project headers.

When would you not use LangSmith? When a fully open-source self-hosted stack is required without an Enterprise license, when residency demands a region Cloud does not offer and Enterprise is not affordable, or when the need is general-purpose application monitoring rather than LLM-specific tracing.

Sixty-second self-test

Answer each in one sentence, out loud, then check against this file.

1. Name the four places a missing trace usually comes from.
2. Which component of a self-hosted install fails first under large payloads, and what is the error?
3. What is the deadline for the legacy query methods on Cloud, and on self-hosted?
4. Give two silent defaults in runs.query.
5. What three things extend trace retention without anyone deploying code?
6. Why must you never rotate the API key salt?
7. What does a fail-closed redactor do when it raises?
8. How many majors can you skip when upgrading?
9. Where do environments live: workspaces or tags?
10. Name one situation in which you would recommend another tool.

If you hesitate on more than two, reread the matching section of the detailed guide, and continue with the tips file for the production traps.

Part three of three. At this level the problems are not failed imports but incidents and invoices: a bill that doubled without a deploy, a dashboard that quietly shows one day of data, a redactor that erased every input, an upgrade with no way back. Start with the error table, then the platform habits, the review checklists, and the operational recipes.

Production traps that appear at scale

Cumulative: everything from Beginner and Mid still applies. These rarely fail a build. They cost money, hide data or cause outages.

Each of these is a default doing something reasonable for one person and something surprising for a platform. Learn the defaults, and you will recognise most of the table before you read the cause.

The traps in more detail

Retention extension is the silent bill

Base traces are kept 14 days and extended traces 180 days (shortened from 400 on 14 September 2026, for new traces). Three things promote a whole trace to extended retention: online evaluators, automation rules with extension enabled, and feedback sent with extend_trace_retention=True. New evaluators and rules have extension on by default, and experiments are created at extended retention. The trap is that nobody deploys anything. A rule that samples ten percent of a busy endpoint is a pricing change, made in a UI, invisible to code review.

The defence is procedural and technical together. Procedurally: any new rule or online evaluator requires a line in a ticket stating the retention setting and the expected volume. Technically: usage limits per workspace, a cost alert, and a monthly report of which rules extended how many traces. When the evidence is worth keeping, put it in a dataset, which never expires, instead of extending retention.

A default window is a data-quality bug

The new runs.query does not fail when you omit a time range; it covers the last day. That makes it the most dangerous kind of migration bug, one that returns plausible numbers. A weekly quality report built on it will report that quality is fine because the bad runs were last Tuesday. Write a test that asserts a query over a known period returns a known count, and review every migrated call for the three defaults: time window, selected fields and page size.

Fail-closed redaction needs an alert

Since Python 0.12.0 a failing process_inputs or process_outputs replaces the payload with Processing failed; inputs dropped (process_inputs: <ExceptionType>). That is correct for privacy, and it converts a redactor bug into an observability outage: every trace loses its content and the team stops being able to debug. Build a saved filter that counts traces containing that text, alert on it, and unit-test redactors with malformed and empty inputs, which are what raise.

The rollout of a prompt is slower than you think

Teams believe a rollback is instant because it is one tag move. The client caches prompts, on by default, with a 300-second time to live and a 60-second refresh, so running instances may serve the old version for minutes. That is resilience when LangSmith is down and latency when you need to undo a mistake. Measure it once, write it in the runbook, and for the highest-risk prompts pin a commit in code or disable the cache.

Judges are code

An LLM-as-judge evaluator is a prompt plus a model. Change either and historical scores are no longer comparable. Pin the model identifier, keep the judge prompt in the hub with a tag, record the judge version in experiment metadata, and calibrate quarterly against a small human-labelled set. A gate whose judge drifts is worse than no gate, because it produces confidence.

Trace headers are an input

The langsmith-trace and baggage headers decide which trace a request joins. A public-facing service that honours them lets a stranger write into your traces. Strip them at the edge and accept them only on internal hops.

Platform-ownership habits

These are ordered by leverage: the first pays back the most per hour spent.

1. Ship a wrapper, not a wiki page. A small internal package that configures the client (endpoint, sampling, redaction, error callback, default tags and metadata) makes the compliant path the default. When a deprecation or policy changes, you change one place.
2. Put the platform in code. Workspaces, custom roles, run rules and alert rules belong in the Terraform provider (langchain-ai/langsmith, pinned), so changes are reviewed and reversible. Keep the organization-scoped admin key in a secret manager and use it only from that pipeline.
3. Pin everything that moves weekly. The Python SDK ships several releases a week. Lock langsmith in every lockfile, upgrade in a branch with the test suite, and read the release notes across the gap. Do the same with the chart version and the CLI.
4. Run a cadence. Weekly: read the changelog and the Friday stable patch notes. Monthly: usage and cost by workspace, rules with extension enabled. Quarterly: service keys, roles, expiry, and a restore test. Each major: a rehearsal in staging, then the upgrade within the N and N minus one window.
5. Measure the platform. Time to first trace for a new team, ingestion delay, tracing error counts, cost per workspace, and count of deprecation warnings. What you do not measure, you discover from a complaint.
6. Write the exit path down. Keep instrumentation behind your wrapper, evaluators in version control, datasets exported on a schedule and bulk traces exported to Parquet. Portability is also your negotiating position.
7. Turn every incident into a dataset row and an alert. The trace goes into a dataset before the ticket closes, and the missing signal becomes a dashboard or alert. The platform gets harder to break each time.
8. Say no with a counter-offer. Requests for blanket extended retention, organization-wide keys or unreviewed gate thresholds should be answered with the nearest safe alternative.
9. Teach the failure modes. A one-page "why is my trace missing" guide and a short note on evaluator flakiness reduce tickets more than any tooling.

Review checklists

For a pull request that touches tracing or evaluation

• Does the change use the shared wrapper, or does it construct its own client?
• Are inputs that may contain personal data passed through a redactor, and is there a test with synthetic sensitive strings?
• Are large payloads summarised with process_inputs rather than traced whole?
• Are required tags and metadata set: team, environment, application version?
• If it calls runs.query or another SmithDB-backed method: explicit min_start_time, explicit selects in uppercase, project_ids rather than a name, cursor paging?
• If it creates feedback: is session_id passed?
• If it adds an evaluator or rule: is the retention-extension setting deliberate, and is sampling set?
• If it adds a judge: is the model pinned and the prompt versioned?
• Does a test pin the expected result of any query that feeds a dashboard?
• Does any new code accept langsmith-trace headers from an untrusted caller?

For a design document proposing a new LLM application

• What data classification does the application handle, and what reaches LangSmith as a result?
• Which workspace, which project naming, which environment tags?
• What is the expected trace volume and payload size, and what sampling rate follows from it?
• What is the release gate: dataset, owner, evaluators, threshold, repetitions?
• How are prompts promoted, who may move the production tag, and what is the rollback time including cache?
• What happens to the application if tracing fails, and is loss acceptable?
• What is the retention plan, and which incidents become dataset rows?
• Is there an exit path if the platform is replaced?

For a review of the self-hosted install

• Are PostgreSQL, ClickHouse, Redis and blob storage external and backed up, and when was the last restore test?
• Is the storage class set with allowVolumeExpansion: true, and is there a disk alert?
• Is the chart version pinned, and is the install within the supported window?
• Has a rehearsal run in staging, including the preview channel?
• Is apiKeySalt in a secret manager, and is the note "never rotate" in the runbook?
• Are images signature-verified, and is egress to the license endpoint allowed?
• Is ClickHouse still enabled, and is it at least version 0.17 before anyone proposes disabling it?
• Are queue metrics scraped and alerted on?

How to read a platform incident

Most incidents on this platform look alike from a distance, so train the reflex of classifying before acting. Ask which lane is affected. If only one application is missing traces, the fault is almost always in that application's configuration: a key, a region, a sampling value, a flush. If many applications are affected at once, suspect the ingestion tier, the queue or the stores, and look at queue depth and disk before anything else. If the UI is slow but ingestion is healthy, the read path is overloaded, usually by wide queries against a month of data, and the answer is query discipline and backend or ClickHouse capacity rather than more ingestion replicas. If everything worked until an upgrade, read the migration jobs first, because breaking changes and data migrations arrive only with a new major.

Then ask what changed. The platform's weekly cadence means something changed recently: an SDK release in a team's lockfile, a chart patch, a rule someone created, a key that reached its expiry, a network policy. Keep a change log that teams and platform owners can both read, because the question "what changed last Thursday" is answered faster by a list than by memory.

Finally, decide whether the incident is about availability, correctness or exposure, since each has a different clock. Availability incidents, such as delayed traces, are urgent but usually recoverable, because the SDK retries and the queue absorbs writes. Correctness incidents, such as a dashboard quietly showing one day of data or a judge drifting, are slow and corrosive: no alarm fires, and decisions get made on wrong numbers, so they need tests that pin expected values. Exposure incidents, where sensitive text reached a trace, are the ones where rotation and containment come before anything else. Naming the category in the first minute tells you who to involve and how fast to move.

Practice cards

Operational recipes

Find callers of the legacy query methods

grep -rnE "\.(read_run|read_thread|list_runs|list_threads|get_run_url|share_run|unshare_run|read_run_shared_link|read_shared_run|list_shared_runs)\(" \
  --include="*.py" .
python -W "error:.*is deprecated and will be removed after Jan 31, 2027" -m pytest -q

The first command finds source call sites; the second makes the test suite fail on the deprecation text so a regression cannot slip back in.

A safe wrapper for querying runs

from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

async def query_runs(project: str, days: int, flt: str, selects: list[str]):
    """Always sets an explicit window and fields; resolves the project UUID."""
    project_id = (await client.aread_project(project_name=project)).id
    now = datetime.now(timezone.utc)
    rows = []
    async for run in client.runs.query(
        project_ids=[project_id],
        min_start_time=now - timedelta(days=days),
        max_start_time=now,
        selects=selects,
        filter=flt,
        page_size=500,
    ):
        rows.append(run)
    return rows

Upgrade one major at a time

helm repo update
helm search repo langchain/langsmith --versions | head -5
helm upgrade langsmith langchain/langsmith \
  --version 0.16.36 \
  --values langsmith_config.yaml \
  -n langsmith --wait --debug
helm status langsmith -n langsmith
kubectl get jobs -n langsmith | grep -E "pg-migrations|ch-migrations"

Back up PostgreSQL, ClickHouse and blob storage first, and never skip a major.

Expand a full ClickHouse volume

kubectl get storageclass -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion
kubectl patch pvc data-langsmith-clickhouse-0 -n langsmith \
  -p '{"spec":{"resources":{"requests":{"storage":"500Gi"}}}}'
kubectl delete statefulset langsmith-clickhouse -n langsmith --cascade=orphan
# Set the new size in langsmith_config.yaml, then:
helm upgrade langsmith langchain/langsmith --values langsmith_config.yaml -n langsmith --wait

Collect a debugging bundle for support

bash get_k8s_debugging_info.sh --namespace langsmith

The script lives in the Helm repository under charts/langsmith/scripts/. For browser problems, attach a HAR file.

Keep the CLI pointed at the right place

langsmith profile create selfhosted --workspace-id <workspace-uuid> --set-current
langsmith auth login --api-url https://langsmith.example.internal --profile self-hosted
unset LANGSMITH_ENDPOINT        # otherwise it overrides the profile's api_url
langsmith --profile selfhosted project list

Closing checklist

Before you call the platform healthy, confirm you can answer yes to each.

• Production stores are external, backed up, and a restore has been tested.
• The chart and SDK versions are pinned, and the install is within N or N minus one.
• Every call site of the deprecated query methods is found and scheduled before 31 January 2027.
• No service key lacks an expiry without a written reason, and apiKeySalt is protected.
• Redaction is tested, fails closed, and is alerted on.
• Every rule and online evaluator has a deliberate retention setting.
• Environments are tags, and the production prompt tag is role-restricted.
• Release gates are in version control, with repetitions and a pinned judge.
• There is a cost alert and usage limit per workspace.
• There is a written exit path, and you have exported a dataset to prove it.

For the full argument behind each item, go back to the detailed guide; for practice under pressure, use the interview file.