Part one of three: Build and read your first Langfuse trace

Start here if Langfuse is new to you. You need basic Python and one application that calls an LLM. By the end, you will capture a request, read its trace, compare a prompt change, and protect sensitive data.

1. Understand a trace before writing code

Langfuse is an observability and evaluation system for applications that call language models. A trace represents one user request or workflow. Inside it, observations describe work: a span for retrieval, a generation for a model call, a tool for an external action, or an event for a small point-in-time fact.

Do not treat every model call as a separate story. A support request might retrieve policy documents, assemble a prompt, call a model, and format an answer. The trace keeps those observations together so you can ask, “why was this answer slow or wrong?”

A useful trace tells a causal story. The root names the user-visible operation. Children name the work that contributed to it, and their nesting explains dependency. If retrieval finishes before generation starts, the tree should show the retrieval span and generation as siblings under the same request, with timestamps that reveal the sequence.

support-answer                         TRACE: one user-visible request
|-- validate-input                     SPAN: application work
|-- retrieve-policy                    SPAN: search and ranking
|   |-- vector-search                  SPAN: database call
|   `-- rerank                         SPAN: ranking work
|-- compose-answer                     GENERATION: model input/output/usage
`-- format-response                    SPAN: final application work

Total request time is the root duration.
Model time is the generation duration, not the sum of every child.

The tree answers several questions without reading every payload. A long vector-search child points to retrieval latency. A fast generation with the wrong prompt version points to release configuration. A failed format-response after a successful generation explains why the user saw an error even though the model call completed.

Choose boundaries that match ownership. If the search team owns retrieval, give retrieval a stable span. If one helper function only converts a list to a tuple, tracing it adds noise. Instrument work that can fail, consume material time or cost, cross a service boundary, or answer a recurring debugging question.

2. Capture your first request with Python

Install the SDK and configure credentials outside source control. The v4 API uses one observation API instead of separate span and generation constructors. The @observe decorator is the quickest safe starting point; it records timing, input, output, and errors around a function.

pip install langfuse
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"

from langfuse import observe, get_client

langfuse = get_client()

@observe(name="support-answer")
def answer_question(question: str) -> str:
    documents = retrieve_policy(question)
    return call_model(question, documents)

@observe(name="retrieve-policy")
def retrieve_policy(question: str) -> list[str]:
    return ["Refunds are available within 30 days."]

@observe(name="call-chat-model", as_type="generation")
def call_model(question: str, documents: list[str]) -> str:
    return "The refund window is 30 days."

The decorator creates a root observation when the function is first entered. Nested decorated functions become children through the active OpenTelemetry context. Use start_as_current_observation when the boundary is clearer in a block.

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(
    as_type="span", name="support-answer", input={"question": question}
) as root:
    with langfuse.start_as_current_observation(
        as_type="generation", name="answer-model", model="gpt-4.1"
    ) as generation:
        response = run_model(question)
        generation.update(output=response)
    root.update(output={"answer": response})

3. Choose the right observation type

Use span for application work and generation for model calls. Use tool for an external action when the integration supports it. Set a useful status message and error level when a child fails; do not hide failure inside a successful root output.

4. Find traces by user, session, and release

A trace is useful only when you can locate the user, session, release, and environment behind it. Use propagate_attributes near the request boundary so every child observation receives the same correlating fields. Keep metadata small, stable, and non-sensitive.

from langfuse import observe, propagate_attributes

@observe(name="support-answer")
def handle_request(user_id: str, session_id: str, question: str):
    # 2. Shared fields make the trace searchable without changing node names.
    with propagate_attributes(
        user_id=user_id,
        session_id=session_id,
        metadata={"channel": "web"},
        tags=["support"],
        version="2026.09",
        environment="production",
        trace_name="support-answer",
    ):
        return answer_question(question)

Correlation fields form an index over your traces. user_id groups a person's requests, session_id reconstructs a conversation, release separates deployments, and environment keeps test traffic out of production analysis. These values should describe the request, not duplicate the full request body.

HTTP request
  |  user_id=u_42, session_id=s_9, release=2026.09.12
  v
support-answer trace
  |-- retrieve-policy      inherits the same safe attributes
  |-- compose-answer       inherits the same safe attributes
  `-- format-response      inherits the same safe attributes

Search "release=2026.09.12 AND environment=production"
then group failures by observation name or user-safe segment.

Use opaque internal identifiers rather than email addresses or customer names. A searchable field spreads across indexes, exports, dashboards, and retained telemetry. If an operator needs to resolve an opaque ID to a person, keep that mapping in the application system with its existing access controls.

5. Capture failures you can debug

Let exceptions remain visible to the observation boundary. Add a short status message for expected failures, but do not replace the exception with a vague “failed”. Include a stable error category in metadata, never a secret or full customer payload.

6. Track the prompt and model behind an answer

Hard-coded prompts make changes difficult to compare. Prompt Management stores a named template with versions and labels. Fetch the intentional production label, compile variables, and link the prompt version to the generation that used it.

from langfuse import get_client

langfuse = get_client()
prompt = langfuse.get_prompt("support-answer", label="production")
compiled = prompt.compile(
    company_policy="Refunds are available within 30 days.",
    question="Can I return this item?",
)

with langfuse.start_as_current_observation(
    as_type="generation", name="answer-model", model="gpt-4.1",
    input={"prompt_name": "support-answer", "prompt_version": prompt.version},
) as generation:
    output = run_model(compiled)
    generation.update(output=output)

Prompt caching keeps retrieval off the critical path after the first fetch. Still design a fallback for a cold instance and decide which last-known version is safe.

7. Add a score that measures one outcome

A trace tells you what happened; a score tells you whether the result was useful. Use numeric scores for a measurable range, categorical scores for a named class, and boolean scores for a yes/no outcome. Keep the score name and definition stable.

from langfuse import get_client

langfuse = get_client()

langfuse.create_score(
    trace_id=trace_id,
    name="correctness",
    value=1.0,
    data_type="NUMERIC",
    comment="Matches the policy answer",
)

langfuse.create_score(
    trace_id=trace_id,
    name="route",
    value="refunds",
    data_type="CATEGORICAL",
)

Start with one score that answers one decision. “Quality” is too broad; “policy-correct” has an observable meaning. Human reviewers can add scores through annotation workflows, while automated evaluators can apply the same named score to many traces.

Scores need a subject and a definition. A trace-level score can describe the final task outcome. An observation-level score can target retrieval or one generation. Put the score at the narrowest level that still matches the claim; otherwise an engineer opens a poor trace and cannot tell which stage earned the result.

Keep the scoring rule beside its name. quality=0.8 means little without a scale, rubric version, evaluator, and target. Prefer a name such as policy_groundedness_v2, document that 1.0 means every factual claim has policy support, and attach evaluator metadata. Human labels and automated scores may share a concept, but they should retain their source so you can compare agreement.

8. Compare a change on a small test dataset

A dataset is a named collection of test cases. Each item should include an input and, when possible, an expected output or evaluation criterion. Run the same task against fixed cases so prompt and model changes are comparable.

from langfuse import get_client, Evaluation

langfuse = get_client()

def task(*, item, **kwargs):
    return answer_question(item.input)

def exact_match(*, input, output, expected_output, **kwargs):
    value = float(output.strip().lower() == expected_output.strip().lower())
    return Evaluation(name="exact-match", value=value)

result = langfuse.run_experiment(
    name="support-baseline",
    data=[
        {"input": "Refund window?", "expected_output": "30 days"},
    ],
    task=task,
    evaluators=[exact_match],
    max_concurrency=2,
)
print(result.format())

Keep evaluation data redacted and versioned. A production trace can become a regression case after an incident, but only after an owner removes unnecessary identifiers and writes a clear expected behavior.

9. Inspect latency, tokens, and estimated cost

Generations can record model and usage details. A token count without the model, route, and release is difficult to price or explain. Capture provider usage when available, then reconcile estimates against the provider invoice.

with langfuse.start_as_current_observation(
    as_type="generation", name="answer-model", model="gpt-4.1",
) as generation:
    response = run_model(question)
    generation.update(
        output=response.text,
        usage_details={
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
        },
    )

Use the UI first: filter by release, model, environment, and score. Add a custom report only when the built-in view cannot answer a recurring operational question. A useful first dashboard has request volume, p95 latency, error rate, cost estimate, and quality score by release.

10. Protect keys, customer data, and trace delivery

Use separate projects or environments for development and production. Store the secret key in the deployment secret manager. Public and secret keys must never appear in browser code or committed examples. Langfuse queues data in the background, so short-lived jobs must flush before exit.

from langfuse import Langfuse

def mask(value):
    if isinstance(value, str):
        return value.replace("alice@example.com", "[EMAIL]")
    return value

langfuse = Langfuse(mask=mask)

try:
    run_job()
finally:
    langfuse.flush()

For larger systems, add retention, access review, sampling, alert ownership, and a documented self-hosting decision. Do not sample away every error trace. A dropped trace is an operational tradeoff, not a free optimization.

11. Separate development from production

Use a development project for experiments and a production project for customer traffic. Keep environment names explicit and rotate credentials through the secret manager. Never solve a local setup problem by copying production keys into a notebook.

Delivery is asynchronous in many SDK paths. Your request code records observations in memory, the client batches them, and a background worker sends them to Langfuse. This keeps telemetry latency away from the user path, but a short-lived process can exit before the batch leaves.

application thread -> SDK buffer -> background batch -> Langfuse ingest -> UI/index
       |                 |               |                    |
   never block       bounded memory   retry policy       searchable trace
       |
       +-> on shutdown: flush with a bounded timeout

Treat telemetry as a secondary path. An ingestion outage should not make the support API unavailable. Log or measure dropped batches, expose queue health, and flush during worker or CLI shutdown. For a long-running web process, let batching work normally; flushing after every request removes the performance benefit and can amplify an outage.

12. Debug a trace in a repeatable order

Start at the root input and output, then inspect the slowest or failed child, then compare a known-good trace. This order keeps debugging focused. A dashboard tells you where to look; the trace tells you what changed.

Beginner checkpoint: can you do these tasks?

Before moving on, you should be able to instrument one request, identify its model generation, find it by session, compare a prompt version, attach a defined score, run a fixed case, inspect usage, and flush a short process.

13. Complete annotated example: trace one support answer

The following combines the beginner path: one root, propagated context, a child generation, explicit usage, and a final flush.

from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()

# 1. One root represents the user-visible request.
@observe(name="support-answer")
def answer(*, user_id: str, session_id: str, question: str) -> str:
    # 2. Shared fields make the trace searchable without changing node names.
    with propagate_attributes(
        user_id=user_id,
        session_id=session_id,
        version="2026.09",
        environment="staging",
        metadata={"channel": "web"},
    ):
        # 3. Record the model call as a generation with model and input context.
        with langfuse.start_as_current_observation(
            as_type="generation", name="answer-model", model="gpt-4.1",
            input={"question": question},
        ) as generation:
            output = "Refunds are available within 30 days."
            # 4. Usage and output explain cost and behavior for this call.
            generation.update(
                output=output,
                usage_details={"input_tokens": 18, "output_tokens": 9},
            )
        return output

try:
    answer(user_id="user-42", session_id="session-7", question="Refund window?")
# 5. Short-lived processes must drain buffered telemetry before exit.
finally:
    langfuse.flush()

Next: continue to Part two: reliable production tracing for contracts, experiment gates, cost reconciliation, privacy controls, and rollout ownership.

Beginner interview review: Langfuse essentials

Use this review after the beginner lesson. Practise short answers first, then explain one trace from request to score in your own words.

Fast review by topic

Python SDK patterns to remember

with langfuse.start_as_current_observation(as_type="generation", name="model") as generation:
    result = call_model(prompt)
    generation.update(output=result)

How prompts, scores, and experiments connect

Beginner operations checklist

Common beginner interview questions

How would you debug a bad answer?

Open the trace, inspect the root input and output, then walk retrieval, prompt, and generation children. Compare the prompt version, model, release, and score with a known-good trace.

Why not create one trace per model call?

That loses the request story. One trace lets you see retrieval, tool calls, retries, and model generations together.

How do you compare two prompts?

Use the same redacted dataset, version both prompts, record model and release, run an experiment, and compare a defined score.

How do you protect customer data?

Minimize capture, mask known identifiers, restrict project access, separate environments, and define retention. Never place secret keys in frontend code.

What happens when the process is short-lived?

The SDK buffers asynchronously, so call flush() or shutdown() before exit.

60-second self-test

Next: Mid-level Interview Review adds integration boundaries, retries, evaluation design, and production ownership.

Beginner scenario: debug one bad answer

Beginner tips and practice lab

Use this lab while instrumenting your first application. Each exercise targets a mistake that makes traces hard to find, read, or trust.

Common errors

Practice cards

1. Give observations stable, readable names

Name observations by domain boundary: support-answer, retrieve-policy, and answer-model. Do not put random request IDs in names; IDs belong in context. Stable names make filters and comparisons useful after a function is renamed.

1. Root the requestUse one name for the user-visible operation.
2. Separate workMake retrieval and model calls visible children.
3. Explain exceptionsUse status and error information when work fails.

2. Capture only the data you need

Inputs and outputs are powerful but can contain secrets and personal data. Decide what each observation needs before enabling broad capture. Use metadata for labels, not as a second database.

3. Repeat one prompt comparison

Fix the input, prompt label, model, release, and score definition. Change one variable. If all five change together, the trace may be detailed while the experiment remains uninterpretable.

Next: Mid-level Tips & Tricks adds failure isolation, safe integrations, and release gates.

Beginner practice checkpoint

Keep a one-page runbook with the request boundary, required environment variables, masking rule, score definition, and flush behavior. Review it whenever the integration or model provider changes.

Part two of three: Make Langfuse reliable in production

Choose this level after you can create and read a useful trace. You will make telemetry consistent across services, evaluate changes in CI, control prompt rollouts, and operate the data path under production traffic.

Where this picks up

Part one taught you to create, find, and debug a trace. Part two keeps every earlier topic and adds the controls a team needs in production.

Production observability has two connected paths. The request path serves the customer and must remain healthy if telemetry fails. The telemetry path exports a bounded, privacy-filtered description of that request. Correlation IDs connect Langfuse to application logs without copying all log data into every observation.

USER -> API -> retrieval -> model -> response
         |         |          |
         +---------+----------+---- safe observations
                              |
                              v
                    SDK buffer / exporter
                              |
                         Langfuse ingest
                              |
               traces + scores + cost dashboards

An ingest failure alerts the telemetry owner; it does not block USER -> response.

The trace contract defines what crosses that boundary: stable names, required attributes, allowed payload fields, observation types, and error semantics. Framework integrations may create extra children, but they cannot replace the logical root or remove fields that dashboards and incident queries depend on.

1. Define a trace contract that survives refactors

The Beginner tree is a mental model. A team needs a contract: which observation names are allowed, which fields are required, and which operation owns each child. Keep the logical request stable when providers retry. A provider attempt can be a child event or generation; it should not create a new user-facing trace for every attempt.

2. Keep a readable tree across framework integrations

Use native integrations where they capture model inputs, outputs, usage, and errors consistently. LangChain's callback handler belongs at the chain boundary; it no longer accepts the removed update_trace parameter. Keep one explicit application root around the integration.

from langfuse import get_client, propagate_attributes
from langfuse.langchain import CallbackHandler

langfuse = get_client()

def answer(question: str):
    handler = CallbackHandler()
    with langfuse.start_as_current_observation(
        as_type="span", name="support-answer", input={"question": question}
    ):
        with propagate_attributes(version="2026.09.3", environment="production"):
            return chain.invoke({"question": question}, config={"callbacks": [handler]})

Test the emitted tree with a fixture. Assert semantic shape and required fields, not every internal callback node. Integration upgrades can add or rename implementation observations.

3. Preserve request identity across services

When a request crosses services, propagate only the values needed to correlate work. as_baggage=True places attributes in outbound headers, so never use it for secrets or raw customer content. Define a tenant boundary in your application and enforce it in project access, not only in a tag.

from langfuse import propagate_attributes

with propagate_attributes(
    user_id="user-42",
    session_id="session-7",
    metadata={"tenant": "acme", "route": "support"},
    as_baggage=True,
):
    call_retrieval_service()

Cross-service propagation carries identity, not payload. Service A creates or receives trace context, sends the supported carrier to service B, and B creates a child or linked observation under the same logical request. If a queue delays work, preserve the relationship while allowing the consumer to record its own processing duration.

client -> API service [trace root]
              |-- retrieval span
              |      `-- search service [remote child]
              |-- generation
              `-- enqueue job ---- trace context ----> worker
                                                     `-- notify-user span

application logs in each service include the same safe trace_id.
message payload carries trace context, never secret keys or full prompts by default.

Validate propagation with an integration test. Start one request, cross the real HTTP or queue boundary, and assert that the UI shows one coherent story rather than two unrelated roots. Also test a malformed or untrusted carrier; the receiving service should reject unsafe baggage fields and start a controlled trace rather than accepting arbitrary metadata.

4. Roll out and roll back prompts safely

Fetching a prompt by production label makes the selection intentional. The SDK caches prompts client-side, but a cold instance still needs a policy. Decide whether the last known version is safe, whether startup may fail closed, and how a rollback changes the label.

from langfuse import get_client

langfuse = get_client()
prompt = langfuse.get_prompt("support-answer", label="production")
compiled = prompt.compile(question="Refund window?", policy="30 days")

generation = langfuse.start_observation(
    name="answer-model", as_type="generation",
    model="gpt-4.1", input={"prompt_version": prompt.version},
)
try:
    generation.update(output=run_model(compiled))
finally:
    generation.end()

5. Calibrate automated and human evaluation

Human feedback and automated evaluation answer different questions. An annotation queue gives reviewers a controlled set of traces and a consistent schema. An LLM-as-a-judge can scale a rubric, but it needs a calibrated reference set and a versioned judge prompt. Keep human review for disagreement and policy-critical decisions.

6. Test prompt and model changes in CI

The current experiment runner supports local data and hosted datasets. Use run_experiment for repeatable task execution, evaluators, concurrency limits, and trace links. Promote a score threshold to a release gate only after the dataset is stable and owned.

from langfuse import get_client, Evaluation

langfuse = get_client()

def task(*, item, **kwargs):
    return answer_question(item.input)

def policy_match(*, output, expected_output, **kwargs):
    return Evaluation(
        name="policy-match",
        value=float(expected_output.lower() in output.lower()),
    )

result = langfuse.run_experiment(
    name="support-release-candidate",
    data=redacted_cases,
    task=task,
    evaluators=[policy_match],
    max_concurrency=5,
    metadata={"release": "2026.09.3"},
)
if result.summary["policy-match"] < 0.95:
    raise SystemExit("evaluation gate failed")

A release evaluation connects four immutable identities: dataset version, candidate prompt or model version, evaluator version, and code revision. Without all four, a score difference cannot be reproduced. The CI job should publish row evidence and a compact decision, while restricted payloads remain in the evaluation store.

fixed dataset v12
      |---------------------|
      v                     v
prompt baseline v7     prompt candidate v8
      |                     |
      +---- same model -----+
              |
        scored by evaluator v3
              |
      paired rows by item ID
              |
   pass / review / fail + rollback label

Read disagreements before enforcing the mean. A candidate can improve ten easy rows and break one account-security case. Slice gates and must-pass items protect high-impact behavior; a human review state handles borderline evaluator disagreement. Store the production prompt label before promotion so rollback changes one controlled label rather than requiring a code deployment.

7. Reconcile cost, usage, and latency

Token fields support estimates, not invoices. Join usage to model, provider, route, release, and pricing-version metadata. Compare Langfuse aggregates with provider billing. Define p95 latency and error budgets by route rather than using one global average.

generation.update(
    model="gpt-4.1",
    usage_details={"input_tokens": 1200, "output_tokens": 180},
    metadata={"route": "support", "pricing_version": "2026-09"},
)

8. Set sampling, retention, and access rules

Sampling reduces ingestion volume but can hide rare failures. Preserve errors and a deliberate baseline sample, then document the rate. Retention and RBAC should follow policy and project boundaries. A production project needs an owner, access review, and an incident runbook.

9. Handle async work and manual observation lifecycles

Manual observations are useful for work that begins and ends in different callbacks. They do not become the active context automatically, so create children from the parent when nesting matters and always call .end() in a finally block. Concurrency limits protect providers and the ingestion path.

10. Correlate Langfuse traces with application logs

Use deterministic trace IDs when an external ticket or workflow ID must correlate with telemetry. Do not put that identifier in every observation name. W3C trace context connects services; a trace ID is not an authorization token.

trace_id = langfuse.create_trace_id(seed="ticket-54321")
with langfuse.start_as_current_observation(
    as_type="span", name="process-ticket",
    trace_context={"trace_id": trace_id},
):
    process_ticket()

The observability control loop starts with a signal and ends with an owned change. A dashboard that has no threshold, runbook, or owner reports history but does not operate the system. Define each chart in terms of a decision someone can make.

For example, p95 trace latency by release can trigger rollback when a new version adds 800 ms. Generation cost per successful task can reveal a prompt that uses more tokens without improving scores. Ingestion delay belongs to the platform owner because it affects visibility, while low groundedness belongs to the product team because it describes application behavior.

11. Build metrics teams can act on

Use built-in views for common questions. When a custom report is required, query dimensions that map to the trace contract: name, model, release, environment, score, and time. Keep queries bounded and cache reports rather than making every request scan raw observations.

12. Support retention and deletion requests

Retention is a product and policy decision. Classify raw inputs, outputs, scores, and aggregate metrics separately. Define how a deletion request is located, who approves it, and how backups or exports are handled. Do not keep raw content merely because storage is cheap.

Intermediate checkpoint: production readiness

You are ready for Senior when you can explain a trace contract, integration fixture, retry boundary, baggage risk, prompt rollback, evaluator calibration, CI gate, cost variance, sampling policy, and incident owner.

13. Complete annotated example: run a controlled release evaluation

This example joins the Mid-level topics in one release check: a fixed redacted dataset, a named evaluator, bounded concurrency, release metadata, and an owned threshold.

from langfuse import Evaluation, get_client

langfuse = get_client()

# 1. Keep test cases redacted, versioned, and unchanged between candidates.
release_cases = load_redacted_cases("support-cases-v7.jsonl")

def task(*, item, **kwargs):
    # 2. The task records the same trace contract used by production traffic.
    return answer_question(item.input, prompt_label="candidate")

def policy_match(*, output, expected_output, **kwargs):
    # 3. A named deterministic rule is easier to audit than a vague quality score.
    passed = expected_output.lower() in output.lower()
    return Evaluation(name="policy-match", value=float(passed))

result = langfuse.run_experiment(
    name="support-2026.09.3",
    data=release_cases,
    task=task,
    evaluators=[policy_match],
    max_concurrency=5,                 # 4. Bound provider and ingestion load.
    metadata={
        "release": "2026.09.3",
        "dataset": "support-cases-v7",
        "owner": "support-quality",  # 5. Name the team that handles failure.
    },
)

if result.summary["policy-match"] < 0.95:
    raise SystemExit("release blocked; review failed cases and the exception path")

langfuse.flush()                       # 6. Preserve the final experiment traces.

The dataset owner controls case changes, the evaluation owner controls the rule, and the service owner responds when the gate fails. Save the experiment link and versions with the release record.

Next: continue to Part three: Langfuse platform governance for organization contracts, tenant isolation, capacity planning, audit evidence, and incident ownership.

Mid-level interview review: reliable production Langfuse

Use this review after the production lesson. Interviewers at this level expect tradeoffs, failure modes, and a clear reason for each design choice.

Fast cumulative review

Integration and failure-mode questions

How should retries appear?

Keep one logical trace. Represent provider attempts as children or events with bounded retry metadata. Do not create a new user-facing trace per attempt or aggregate latency and cost will be misleading.

Why test tree shape instead of node count?

Integrations add internal nodes across versions. The contract should assert the root, important child boundaries, required fields, and error behavior, not brittle implementation details.

When is baggage unsafe?

Whenever values could be sensitive or reach untrusted downstream systems. Baggage is sent in outbound headers; allowlist non-sensitive correlation fields.

Evaluation and release questions

What makes an LLM judge credible?

A named rubric, reference examples, evaluator version, fixed dataset, and reviewed human disagreement. A judge is an aid to measurement, not an objective oracle.

When should a score block deployment?

Only after the dataset, metric, threshold, and exception owner are stable. Keep exploratory scores separate from release gates.

How do you compare prompts?

Hold dataset, model, release context, and evaluator constant. Change the prompt version, run the experiment, inspect representative traces, and document the decision.

Production operations review

Common questions to rehearse

60-second self-test

Next: Senior Interview Review adds platform governance, capacity, compatibility, and organizational decision-making.

Intermediate scenario: repair unreliable telemetry

Mid-level tips and production practice lab

Use these drills when a working integration starts serving real traffic. Keep the Beginner habits, then add stable trace contracts, safe context propagation, controlled prompt changes, and delivery under failure.

Common errors

Practice cards

Cumulative practice map

1. Test the boundary your framework hides

Native wrappers are convenient, but the application still owns the root and its context. Put the handler inside a named operation and test one representative trace after dependency upgrades.

1. PinRecord SDK and integration versions.
2. FixtureRun one deterministic request.
3. CompareInspect tree shape, fields, delivery, and latency.

2. Pass only safe, stable request context

Use as_baggage=True only for a small, non-sensitive allowlist. Tenant authorization belongs in service policy and project access. Add a test that attempts cross-tenant lookup and export.

3. Release prompt versions like code

A label promotion needs a candidate, fixed evaluation set, approval, rollback version, and owner. Prompt caching improves latency; it does not replace rollout policy.

4. Separate exploratory scores from release gates

Exploratory scores answer questions. Release scores block or permit change. Use different names and dashboards so an experimental judge does not silently become a deployment dependency.

5. Test queues, flushes, and ingestion failure

Pin a sampling policy, keep error visibility, test masking before ingestion, and run an access review. When traces disappear, check destination, keys, SDK execution, filter, network, and flush in that order.

Next: Senior Tips & Tricks adds platform-level controls and failure drills.

Intermediate practice checkpoint

Every production route should have a trace contract, fixture trace, retry policy, prompt rollback, evaluator owner, cost dimensions, sampling rule, and incident responder.

Part three of three: Design and govern Langfuse as a platform

Choose this level when several teams depend on Langfuse or you own the platform. You will define organization-wide contracts, privacy and evaluation policy, rollout controls, capacity plans, SLOs, and incident ownership.

Where this picks up

Parts one and two established the complete application and team workflow. Part three keeps that spine and turns each topic into an organization-level policy with a security boundary, scale limit, and named owner.

At platform scale, Langfuse sits between application teams, security policy, evaluation standards, and infrastructure. The platform contract should let teams instrument independently while keeping traces comparable enough for shared dashboards, incident response, and cost allocation.

APPLICATION TEAMS                PLATFORM TEAM
stable logical spans ---------> ingest + SDK support
prompt/model metadata --------> schema compatibility
safe user/session IDs --------> access and retention enforcement
          |                              |
          +---------- traces ------------+
                         |
                 EVALUATION OWNERS
             score definitions + datasets
                         |
                    release evidence
                         |
               SECURITY / GOVERNANCE
             policy, audit, deletion, review

Ownership should follow failure modes. An application team owns a missing retrieval span. The platform team owns ingestion lag and an incompatible SDK rollout. Evaluation owners approve score definitions. Security owns the rules for tenant access and sensitive fields. Put escalation paths and service objectives beside the contract so teams know where to route a broken trace.

1. Publish an organization-wide trace contract

At senior scale, application teams should not invent incompatible telemetry schemas. Publish allowed observation types and names, required release/model fields, metadata limits, redaction rules, and ownership. Keep a compatibility window when changing names; dashboards and evaluators are consumers.

support-answer [root]
  retrieve-policy [span, owner=retrieval]
  answer-model [generation, model+usage required]
  answer-policy-score [score, owner=quality]

2. Roll out SDK and schema changes safely

Pin the SDK and integration versions. In v4, the default exporter filters many non-LLM spans, which can change trace trees during migration. Use a disposable project, compare fixed fixture traces, roll out to one service or tenant, and keep a rollback version. Debug mode helps inspect dropped scopes.

from langfuse import Langfuse
from langfuse.span_filter import is_default_export_span

def export_policy(span):
    return is_default_export_span(span) or (
        span.instrumentation_scope is not None
        and span.instrumentation_scope.name.startswith("company.agent")
    )

langfuse = Langfuse(
    debug=True,
    should_export_span=export_policy,
)

3. Enforce tenant isolation and privacy policy

Projects and access controls are boundaries; tags are dimensions. Decide whether tenants share a project, how exports are authorized, and how deletion requests map to retained observations. Metadata values are not a place for raw prompts or personal records.

import re
from langfuse import Langfuse

EMAIL = re.compile(r"\b[\w.+-]+@[\w.-]+\.\w+\b")

def mask(value):
    if isinstance(value, str):
        return EMAIL.sub("[EMAIL]", value)
    if isinstance(value, dict):
        return {key: mask(item) for key, item in value.items()}
    if isinstance(value, list):
        return [mask(item) for item in value]
    return value

langfuse = Langfuse(mask=mask)

Tenant isolation needs controls at collection, storage, query, export, and deletion. Filtering the UI by tenant after loading mixed data is not an isolation boundary. Bind tenant identity at ingestion, authorize every query against it, and test that exports and evaluator jobs cannot cross the boundary.

request identity -> ingest policy -> tenant project/store -> authorized query
       |                 |                  |                 |
   authenticated    redact/allowlist    encryption/ACL    audited export
       |                 |                  |                 |
       +---------------- deletion index ---------------------+
                              |
                    retention or legal hold

Use synthetic canary records to test isolation. Place a unique marker in tenant A, query as tenant B through UI, API, export, and evaluation paths, and require zero matches. Run the same test after SDK, server, role, or data-pipeline changes. A successful UI test alone leaves background jobs and exports untested.

4. Govern the prompt supply chain and rollback path

Treat prompts as deployable dependencies. A production label should point to a reviewed version with an owner, evaluation evidence, model assumptions, and rollback instructions. Cache behavior must be part of the availability design: cold instances, stale values, and unavailable Langfuse need explicit outcomes.

5. Own evaluation standards and score policy

An organization needs a score taxonomy, evaluator versioning, calibration sets, annotation ownership, and a rule for disagreements. Separate human feedback, deterministic code evaluators, LLM judges, and release gates. Keep the evaluator prompt and rubric beside the result.

from langfuse import Evaluation

def grounded(*, output, expected_output, **kwargs):
    return Evaluation(
        name="groundedness-v2",
        value=float(expected_output.lower() in output.lower()),
        comment="Deterministic baseline; human review required for exceptions",
    )

6. Turn experiments into auditable release evidence

Use the experiment runner for repeatable tasks, item evaluators, run evaluators, concurrency controls, and trace links. A senior platform should publish dataset provenance, expected criteria, evaluator versions, and threshold changes. A failing experiment should identify whether code, prompt, model, data, or evaluator changed.

result = langfuse.run_experiment(
    name="support-release-2026-09",
    data=versioned_redacted_dataset,
    task=task,
    evaluators=[grounded],
    run_evaluators=[aggregate_quality],
    max_concurrency=10,
    metadata={"candidate": "2026.09.3", "dataset": "support-v4"},
)

Promotion should consume evidence rather than a verbal claim. The release record joins the candidate, dataset, evaluator policy, score artifact, approver, production label change, and rollback target. This chain lets an incident commander reconstruct what changed without searching chat history.

candidate prompt v18
       |
       +-> dataset support-golden@12
       +-> evaluator policy groundedness@3
       +-> row results + critical-slice decision
       +-> reviewer approval
       v
production label: v17 -> v18
rollback label:   v17 retained
       |
deployment monitor -> trace/score/cost evidence

Require separation of duties for high-impact changes. The author can create the candidate and run the experiment; a designated owner reviews the evidence and promotes the label. Emergency rollback should need fewer steps than promotion, but it must still write an audit event and open follow-up work.

7. Plan capacity and control LLM observability cost

At scale, usage is a budget signal and a capacity signal. Aggregate input/output tokens by model, provider, route, release, tenant, and pricing version. Reconcile against invoices. Watch concurrency, queue latency, ingestion failures, and ClickHouse growth if self-hosted. Assign a responder for unexpected variance.

8. Set SLOs, retention rules, and incident ownership

Define SLOs for ingestion delay, trace availability, dashboard freshness, and evaluation completion. Retention should reflect legal, product, and debugging needs. Self-hosted deployments add responsibility for Postgres, ClickHouse, Redis or Valkey, blob storage, web, worker, encryption keys, backups, and upgrades.

request -> SDK queue -> ingestion -> ClickHouse / blob storage
                       |             |
                       +-> worker ---+-> dashboards and evaluations

Operate telemetry with explicit SLOs. Availability measures whether accepted events become queryable. Freshness measures ingestion delay. Completeness compares expected and observed traces. Query latency protects incident response. Each SLO needs an error budget and an action when the budget burns too quickly.

SDK health -> ingest queue -> worker -> analytical store -> query/API
    |            |            |            |              |
 dropped      queue age    failure rate   merge lag     latency/errors
    \____________|____________|____________|______________/
                              |
                         SLO dashboard
                              |
              alert -> incident owner -> mitigate
                              |
                 replay / rollback / capacity change

Preserve the application path during telemetry incidents. Apply bounded buffers, sampling, or dropping according to policy rather than exhausting application memory. If completeness falls below the release-evidence requirement, pause automated decisions that depend on traces even while the product continues serving traffic.

9. Maintain an SDK and server compatibility policy

Separate SDK compatibility from server compatibility. Pin both where possible, record the tested matrix, and use legacy API namespaces only when a self-hosted server version requires them. A migration note should name removed methods, changed fields, and rollback conditions.

10. Decide what your self-hosted team owns

A self-hosted installation adds Postgres for application state, ClickHouse for analytical observations, Redis or Valkey for queues and coordination, and S3-compatible storage for blobs. The web service serves the UI and API; the worker processes background ingestion and jobs. Operators own backups, encryption secrets, upgrades, and capacity.

web/API -> Postgres       configuration and users
       -> ClickHouse      observations and analytics
       -> Redis/Valkey     queue and coordination
       -> object storage  media and blobs
worker -> ingestion, jobs, exports, evaluations

11. Design for capacity limits and backpressure

Estimate event volume from request rate, observations per request, payload size, and retention. Watch queue depth, worker latency, ClickHouse merges, object storage growth, and API response time. Apply sampling and payload limits deliberately; do not let ingestion failure silently impact the application path.

Self-hosted capacity begins with trace shape. Estimate requests per second × observations per request × bytes per observation, then apply peak factor and retention. Large prompts or media may move cost from analytical storage to object storage, while high-cardinality metadata increases query and index pressure.

SDKs -> load balancer -> web/API -> queue/Redis -> workers -> ClickHouse
                            |                         |          |
                            +-> Postgres config       |       analytics
                                                      +-> object storage

pressure signals: API latency, queue age, worker saturation,
ClickHouse merge backlog, object growth, and query latency.

Choose a degradation order before an incident. Drop debug observations before release evidence, cap oversized payloads before rejecting whole traces, and preserve error traces according to security policy. If the queue reaches its hard limit, clients must fail telemetry calls safely and increment a local drop counter. They must not retry without bounds or block customer requests.

12. Keep evidence for changes, incidents, and audits

A governed platform keeps before/after traces, evaluator results, access reviews, retention decisions, incident timelines, and rollback evidence. Store the decision next to the release or policy it explains. Dashboards without provenance are difficult to defend during an incident or audit.

Advanced checkpoint: can you defend the platform?

Senior practice connects every object to a decision: trace contracts protect consumers, instrumentation exposes work, context protects correlation, prompts control behavior, scores measure outcomes, datasets test change, cost informs budgets, and operations protects the data path.

13. Complete annotated example: enforce the platform boundary

from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()

# 1. The organization contract gives every service a stable logical root.
@observe(name="support-answer")
def serve(request):
    # 2. The service supplies approved identity, release, route, and tenant fields.
    with propagate_attributes(
        user_id=request.user_id,
        session_id=request.session_id,
        version=request.release,
        environment="production",
        metadata={"route": "support", "tenant": request.tenant},
    ):
        # 3. The model gateway owns model choice and usage reporting.
        with langfuse.start_as_current_observation(
            as_type="generation", name="answer-model",
            model=request.model, input={"prompt_version": request.prompt_version},
        ) as generation:
            answer = model_gateway(request)
            # 4. The generation records only content allowed by the masking policy.
            generation.update(output=answer.text, usage_details=answer.usage)
        return answer.text

# 5. Application shutdown owns delivery; platform SLOs monitor ingestion.
try:
    serve(request)
finally:
    langfuse.flush()

The platform contract makes the ownership visible: deployment supplies release and model context, the application supplies the logical root, the mask policy controls capture, evaluation owns scores, and shutdown owns delivery.

Next: use all three parts as one operating model. Return to Part one for application onboarding and Part two for service reliability; use this part for platform policy, incidents, and ownership reviews.

Senior interview review: Langfuse platform governance

Use this review for staff, platform, and architecture interviews. Build each answer from fundamentals, then state the policy, owner, failure mode, and rollback path.

Fast cumulative review

Platform architecture questions

What belongs in a trace contract?

Stable root and child names, observation types, required model/release fields, safe metadata, redaction rules, ownership, and migration policy. Dashboards and evaluators consume it.

What changed in the Python v4 data model?

Observations are primary. start_observation replaces separate span/generation constructors, and propagate_attributes applies correlating fields to current and child observations. Export filtering can change non-LLM span visibility.

How do you roll out an SDK upgrade?

Pin, capture a fixture, compare tree shape and fields in a disposable project, stage to one service or tenant, monitor delivery and dashboards, then widen with rollback ready.

Governance and evaluation questions

How do you make an LLM judge trustworthy?

Use a versioned rubric, calibration references, fixed redacted data, evaluator metadata, human disagreement review, and a named owner. Keep judge output separate from release policy until validated.

When is an experiment a release gate?

When dataset provenance, metric definition, threshold, evaluator version, exception path, and owner are stable. Otherwise it is useful exploratory evidence but not a blocking policy.

How do you protect multi-tenant telemetry?

Enforce project and export authorization, minimize and mask content, allowlist baggage, review roles, and test cross-tenant access. Tags help filtering; they do not enforce isolation.

Reliability and incident questions

Common questions to rehearse

60-second self-test

Next: Senior Tips & Tricks turns the model into drills and operating habits.

Advanced scenario: govern a multi-team rollout

Senior tips and platform governance lab

Use these drills when teams depend on Langfuse for release decisions, debugging, or compliance evidence. Test the policy and failure path, not only the happy-path SDK call.

Common errors

Practice cards

Cumulative practice map

1. Enforce trace contracts in CI

Check required root names, observation types, release fields, redaction behavior, and error traces in CI. Ignore incidental nodes that integrations may add. Store the contract beside the service ownership metadata.

2. Canary SDK and schema upgrades

Capture before/after traces, compare delivery and dashboard freshness, and stage by service or tenant. Keep debug logging available during the canary, then remove broad verbosity after the decision.

3. Test privacy and tenant boundaries

Test nested inputs, metadata, baggage, exports, access roles, retention deletion, and logs. A masking callback that only handles one string shape is not a privacy program.

4. Assign owners to evaluation policy

Keep evaluator, rubric, dataset, prompt, model, and release versions together. Separate exploratory metrics from gates. Every exception needs a person, a reason, and an expiry.

5. Review cost beside quality and latency

Pair cost with quality and latency. A cheaper model that increases retries or support failures may increase total cost. Reconcile telemetry estimates with provider billing and document pricing changes.

6. Rehearse the incident path

For missing traces: destination, credentials, SDK execution, filter, network, queue, and flush. For quality drift: dataset, prompt, model, evaluator, release. For cost drift: route, model, retries, usage, pricing.

Next: return to Beginner when a new teammate needs the model; return to Mid-level when a service needs reliability; return to Senior when the organization needs a policy.

Advanced platform checkpoint

Review platform health as a quarterly ownership exercise: compatibility matrix, capacity forecast, access review, retention evidence, evaluator calibration, cost variance, and rollback rehearsal.