The Complete Arize Phoenix Guide

This is part one of three. It covers everything you need to do real work with Arize Phoenix, not a teaser. By the end you can start a Phoenix server on your own laptop, send traces from a Python LLM application into it, read those traces in the web interface, label them by hand, score them automatically with a second model acting as a judge, save interesting cases into a dataset, and run an experiment against that dataset. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas stick only once you have watched your own application appear, span by span, in a browser tab. The examples use Python and the OpenAI client because they are the most common starting point, but the ideas carry over to every provider and framework Phoenix supports.

One honest warning first. Phoenix ships a new minor release every few days, and this guide was checked against version 20.16.0, released on 23 September 2026. The version you install tomorrow will be newer. Most of what you read here is stable because it rests on OpenTelemetry, but the Phoenix-specific APIs did change a lot between versions 12 and 14, and many tutorials on the internet still show the old forms. Where that matters, this guide says so and shows the current form.

What Phoenix is, and the problem it solves

Arize Phoenix is an open-source platform for seeing inside applications that use large language models, and for measuring whether they behave well. You run it as a small server. Your application sends it a record of everything it did on each request: which prompt went to which model, what came back, how many tokens it cost, how long each step took, and which tool or database lookup happened in between. Phoenix stores those records, draws them as a tree in a web interface, and gives you tools to label them, score them, and turn them into test cases.

To see why this is needed, think about what a normal web service gives you when something goes wrong. You have logs, a stack trace, and a status code. An LLM application is different in a way that breaks all three. The call succeeds, the status code is 200, no exception is raised, and the answer is still wrong. The model invented a policy that does not exist, ignored the document you retrieved, or called the wrong tool with plausible-looking arguments. Nothing in a conventional log line tells you that. To find out, you have to read the actual prompt and the actual reply, and you have to read them in the context of the steps around them.

Applications also stopped being a single call. A modern assistant may rewrite the user's question, search a vector database, pass the top five passages to a model, let the model call a calculator tool, feed the result back, and finally produce an answer. That is six or seven operations, several of them non-deterministic, and a bad answer could be caused by any of them. If the retrieval step returned the wrong passages, rewriting the final prompt will not help. Without a record of every step you are guessing.

Phoenix solves this in two halves that feed each other.

Observability is the first half: a trace of each request, captured automatically, so you can open any single interaction and see exactly what happened. This is how you debug.

Evaluation is the second half: a way to attach judgements to those interactions, whether a person clicked "thumbs down", a rule checked a format, or another model scored the answer for faithfulness. Judgements let you move from "this one looks wrong" to "eleven percent of answers are unfaithful, and the rate went up after Tuesday's change". This is how you improve.

The loop the Phoenix documentation teaches is the same one this guide follows: trace, then annotate or evaluate, then curate datasets, then run experiments, then iterate on your prompts, then trace again. Each arrow in that loop is a section below.

What came before

Teams first reached for print statements and log files. Prompts and completions were written to a log line, and someone searched with grep. That works for ten requests and fails for ten thousand, because a flat log has no structure: you cannot ask it to show the slowest step, or all requests where the tool call failed, or the tree of operations behind one particular answer.

Next came general application performance monitoring tools. They do understand requests as trees of timed operations, but they were built for latency and errors, not for the content of a prompt. They show that a call took 2.1 seconds. They do not show the text, do not count tokens by model, and have no concept of scoring an answer.

Then came a wave of LLM-specific platforms, a category usually called LLM observability or LLMOps tooling. Phoenix belongs there, next to Langfuse, LangSmith and others. Its particular choice is to be built directly on OpenTelemetry, the open standard that the rest of the observability world uses for traces. That choice has a practical consequence: the data your application emits is not locked to Phoenix. You can send the same traces to Phoenix and to another backend at once, and you can keep your instrumentation if you ever switch.

Phoenix also used to be a tool for classic machine-learning model monitoring, with embedding visualisations and drift analysis. That functionality was removed in version 13. If an older course or blog post shows px.Inferences, embedding point clouds or a Schema object, it is teaching a version of Phoenix that no longer exists. This guide teaches only the LLM and agent side, which is what Phoenix is today.

Three products with similar names

Newcomers often confuse the offerings, so separate them now.

• Phoenix is the open-source project this guide is about. You install it yourself and run it wherever you like. It is licensed under the Elastic License 2.0, which is free to use and to self-host but is not an Apache or MIT licence, so read it if your employer has a policy on licences.
• Phoenix Cloud is the same Phoenix hosted for you at app.phoenix.arize.com. You sign up, receive an endpoint that looks like https://app.phoenix.arize.com/s/<your-space>, and skip the server setup.
• Arize AX is Arize's commercial enterprise platform, with multi-team organisations and managed hosting. It is a separate product.

For a student or a first project, running Phoenix on your own laptop is the right choice: it costs nothing and needs no account. For employers in the Gulf or Egypt with data-residency rules, the self-hosted route has a real advantage too. Prompts and completions often contain customer data, and when you run Phoenix yourself, that data stays in the database you chose, in the cloud region you chose, rather than leaving for a hosted service.

The mental model: four nouns

Phoenix has a large surface, but the core vocabulary is small. Learn these four nouns well and every screen will make sense.

Trace and span

A span is one unit of work. Calling the model is a span. Running a search is a span. A Python function you decorated is a span. Each span records a name, a start time and an end time, a status (OK, ERROR or UNSET), a pointer to its parent span, and a bag of key-value attributes describing what it did.

A trace is the complete record of one request moving through your application. Technically it is the set of all spans that share the same trace_id, arranged as a tree. The first span has no parent and is called the root span. Everything it triggered hangs beneath it.

If you have used a browser's network tab or a profiler's flame graph, you already know the shape. The tree tells you what called what, and the time bars tell you where the seconds went.

Attributes and span kind

Attributes are where the LLM-specific information lives. For a model call, Phoenix expects attributes such as llm.model_name, input.value, output.value, and the token counts llm.token_count.prompt and its completion counterpart. You rarely write these by hand. The instrumentation library fills them in.

The set of names comes from OpenInference, a set of conventions that Arize maintains on top of OpenTelemetry. One of those conventions is the span kind, stored in the attribute openinference.span.kind. It tells Phoenix what sort of work the span represents, so the interface can show a retriever's documents differently from a model's messages. The kinds are written in capitals: LLM, CHAIN, AGENT, TOOL, RETRIEVER, EMBEDDING, RERANKER, GUARDRAIL, EVALUATOR, PROMPT and UNKNOWN. A span with no kind set is UNKNOWN.

Project

A project is a container of traces, normally one per application or environment. When you open Phoenix you first see a list of projects. The default project is literally named default, and you should avoid using it for anything real, because traces from every unnamed experiment pile up there. You choose the project from your code with project_name, which Phoenix writes onto your data as a resource attribute called openinference.project.name.

Instrumentation

Instrumentation is the code that creates spans. There are two styles. Automatic instrumentation uses a library that knows how to wrap a framework: install the OpenAI instrumentor and every OpenAI call becomes an LLM span without changing your application code. Manual instrumentation is where you mark your own functions, using decorators that Phoenix provides or the standard OpenTelemetry tracer. Real applications use both: automatic for the model and framework calls, manual for your own business logic.

Sessions, annotations, datasets, experiments

Four more nouns appear later in the guide, and it helps to meet them briefly now.

• A session groups several traces that belong to one conversation, using a shared session.id. The interface shows it as a chat timeline.
• An annotation is a label, a score, or a short explanation attached to a span or a trace. Its source, the annotator_kind, is HUMAN, LLM or CODE. You will see older material call these "evaluations"; since version 14 the word is annotations everywhere.
• A dataset is a versioned collection of examples, each with an input and usually an expected output.
• An experiment runs a function, called the task, over every example in a dataset and scores the results with evaluators.

Installing the server and checking it works

Phoenix is an ordinary Python package, so you install it the way you install any Python tool. It supports Python 3.10 through 3.14 and runs on Linux, macOS and Windows. Always use a virtual environment, a private folder of packages for one project, so that Phoenix's many dependencies cannot collide with anything else on your machine.

On Linux or macOS:

python3 -m venv .venv && source .venv/bin/activate
pip install arize-phoenix
phoenix serve

On Windows, in PowerShell:

py -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install arize-phoenix
phoenix serve

The first two commands create and enter the virtual environment, pip install downloads the server and its web interface, and phoenix serve starts it. The terminal will print a startup log and stay busy. That is correct: a server is a program that keeps running until you stop it with Ctrl+C. Leave this terminal open and use a second one for everything else.

If you would rather install nothing permanently, uvx arize-phoenix serve runs the server through uv, a fast Python package runner, without a separate install step. The documentation calls it the quickest local option.

Notice the order of the words. Flags go after serve. Before version 14 the command line was arranged differently, and old tutorials show phoenix --port 6006 serve, which now fails. The current form is:

phoenix serve --host 0.0.0.0 --port 6006

What the server listens on

Phoenix opens two ports that matter now.

Why two? OpenTelemetry defines two transports for sending traces, HTTP with protobuf encoding and gRPC, and Phoenix accepts both. Which one your application uses will matter in a later section, because mixing them up is the most common first-day failure.

Where your data lives

With no database configured, Phoenix stores everything in a SQLite file inside a working directory, which defaults to .phoenix in your home folder. That means your traces survive restarting the server, and you can delete the folder to start completely fresh. SQLite is perfect for learning and for development. A shared or production Phoenix should use PostgreSQL instead; Mid-level and Senior cover that.

If you work in a Jupyter notebook you may also see import phoenix as px; px.launch_app(). That starts a server inside the notebook process, and it does not keep data by default. It is handy for a demo and a poor habit for real work. Prefer the separate phoenix serve process.

Checking that it is alive

Open http://localhost:6006 in a browser. You should see the Phoenix interface with an empty list of projects. From a second terminal you can also ask the server directly:

curl http://localhost:6006/healthz
curl http://localhost:6006/readyz
curl http://localhost:6006/arize_phoenix_version
pip show arize-phoenix

/healthz answers whether the process is alive. /readyz answers whether it is ready to do work, which includes being able to reach its database. The third returns the running server's version, and pip show reports the version of the package installed in your environment. If those two numbers differ, you are talking to a different server than the one you think you installed, a mix-up that costs people real time later.

Running it in Docker instead

If you already use Docker, the official image runs the same server:

docker run -p 6006:6006 -p 4317:4317 -i -t arizephoenix/phoenix:20.16.0

The -p flags publish the two ports so your host can reach them. Pin the version number rather than using latest; the project's documentation warns against latest for anything you care about. Data written inside this container disappears when the container is removed, unless you mount a volume and set PHOENIX_WORKING_DIR to point at it. If you do not know Docker yet, our Docker guide is the natural companion, and our Kubernetes guide covers the eventual production home.

Port already in use

If the server exits at startup with "Address already in use" (the error number differs by operating system: [Errno 48] on macOS, [Errno 98] on Linux, [WinError 10048] on Windows), something else is holding port 6006 or 4317, often an earlier copy of Phoenix you forgot about. Either stop that process or move Phoenix:

phoenix serve --port 6007 --grpc-port 4318

Remember the new ports when you point your application at the server.

Your first traced application

Now the main event. We will write a tiny program that asks an OpenAI model a question, then watch it appear in Phoenix. You need an OpenAI API key for this exact example, exported as OPENAI_API_KEY. Any other provider works the same way with its own instrumentor; we use OpenAI because it is the most widely documented.

Install the tracing helper and the OpenAI instrumentor into the same virtual environment as your application:

pip install arize-phoenix-otel openinference-instrumentation-openai openai

arize-phoenix-otel is a thin wrapper that configures OpenTelemetry for Phoenix with sensible defaults. openinference-instrumentation-openai is the piece that knows how to wrap the OpenAI client and turn each call into an LLM span. One version note: from release 0.17.2, arize-phoenix-otel needs Python 3.11 or newer, and on Python 3.10 pip quietly installs the previous release instead, which still works.

Create a file named first_trace.py:

from phoenix.otel import register
from openai import OpenAI

tracer_provider = register(
    project_name="my-first-app",
    auto_instrument=True,
    protocol="http/protobuf",
    endpoint="http://localhost:6006/v1/traces",
)

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain what a span is in one sentence."}],
)
print(response.choices[0].message.content)

Run it with python first_trace.py. Before the answer prints, register() writes a short banner describing what it configured: the project name, the span processor, the collector endpoint and the transport. Read that banner. It is the quickest diagnostic you will ever get, and later sections rely on it.

What each line did

register() is the one function you call at startup. It builds an OpenTelemetry TracerProvider, the object that creates spans and hands them to an exporter, and points it at Phoenix. Its arguments are keyword-only. The ones you will use first are:

With auto_instrument=True, register() looks at what is installed and instruments it. If you prefer to be explicit, you can leave it off and call the instrumentor yourself:

from openinference.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Both routes give the same result. Explicit is clearer when you read the code a year later; automatic is shorter for a demo.

Why we passed the protocol and endpoint

We spelled out protocol="http/protobuf" and a full /v1/traces URL on purpose. If you pass a bare base URL such as http://localhost:6006 and say nothing about the protocol, the Python SDK infers gRPC on port 4317, not HTTP on 6006. That works when you started Phoenix locally and both ports are open. It fails silently when you later point at a server behind a load balancer that exposes only HTTPS. Being explicit from day one avoids learning this the hard way. The errors section returns to it.

Seeing the result

Switch to the browser and refresh. A project named my-first-app should now appear. Open it, and you will see a single row in the traces table. Click the row.

Reading a trace in the interface

A trace you cannot read is decoration. Spend this section learning the screen, because you will spend most of your working life in it.

Open your project. The default view is a table of traces, newest first, with columns for the start time, the input and output of the root span, the latency, the token counts and often a cost figure. Phoenix calculates cost per span from a built-in table of model prices, which it updates almost every release, so the numbers are estimates and you should check them against your provider's invoice before quoting them.

Click any row and a panel opens with the tree on the left and the details of the selected span on the right. Learn to read four areas.

The tree. Each indented row is a span, labelled with its name and a small icon for its kind. A long bar beside it shows how much of the total time it consumed. Bars that start late show work waiting on something earlier. In a real application, the tree is where you notice that retrieval took three seconds and the model call took one, a fact no amount of staring at the final answer would reveal.

The Info tab. For an LLM span this shows the input messages and output messages as a readable chat, including the system prompt, exactly as the model saw them after templating. This is the single most valuable view. Most "why did it say that" questions are answered by reading the assembled prompt and discovering it contained something you did not intend.

The Attributes tab. The raw key-value list behind the nice display. When something is missing from the pretty view, it is usually here in raw form. It also teaches you the attribute names, which you need when you write filters.

The status and events. A span that raised an exception has status ERROR, and the exception message and stack trace are recorded as an event on the span. A failing tool call in an agent is found by looking for the red span, not by reading a log.

Filtering the table

As traces multiply you will not scroll. The search bar above the table accepts a filter expression, a small Python-like language over span fields. A few you will use immediately:

span_kind == 'LLM'
'refund' in input.value
status_code == 'ERROR'
parent_id is None

The first keeps only model calls, the second finds spans whose input contains a word, the third shows failures, and the last keeps only root spans, which gives you one row per request. Later sections add filters on annotations and metadata. The same expressions work in code, so the skill transfers.

Tracing your own code: decorators and span kinds

Automatic instrumentation captures calls to libraries it recognises. It knows nothing about your function look_up_policy() or the little routine that decides which department a ticket belongs to. Yet those are often exactly the steps you need to see. Manual instrumentation fills the gap.

The tracer you get from register() provides decorators. You create a tracer once, then mark functions:

from phoenix.otel import register
from openai import OpenAI

tracer_provider = register(
    project_name="support-bot",
    auto_instrument=True,
    protocol="http/protobuf",
    endpoint="http://localhost:6006/v1/traces",
)
tracer = tracer_provider.get_tracer(__name__)
client = OpenAI()

POLICIES = {
    "refund": "Refunds are available within 14 days of purchase with a receipt.",
    "shipping": "Standard shipping to the Gulf region takes 5 to 8 working days.",
}


@tracer.tool
def look_up_policy(topic: str) -> str:
    """Return the policy text for a topic, or an empty string."""
    return POLICIES.get(topic, "")


@tracer.chain
def answer_question(question: str) -> str:
    topic = "refund" if "refund" in question.lower() else "shipping"
    policy = look_up_policy(topic)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Answer using only this policy: {policy}"},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    print(answer_question("Can I get a refund after two weeks?"))

Run it. In Phoenix you now see a single trace with a tree: a CHAIN span named answer_question at the root, a TOOL span for look_up_policy beneath it, and the automatic LLM span for the OpenAI call beside it. The decorator records the function's arguments as the span's input and its return value as the output, which is why the trace table shows a readable question and answer on the root row.

The decorators mirror the span kinds: @tracer.chain for a pipeline step, @tracer.tool for something an agent would call, @tracer.agent for an autonomous loop and @tracer.llm for a model call you wrap yourself. Pick the kind that matches what the function does, because the interface and the evaluators use the kind to decide how to display and treat the span. A retrieval function that returns documents is a RETRIEVER, and Phoenix will show the returned documents as a list with their scores, which is far more useful than a raw dump.

If you need full control, the standard OpenTelemetry tracer API also works, and you may use a with tracer.start_as_current_span("name") as span: block, then set attributes on the span yourself. Decorators are enough for the vast majority of beginner work.

Why the tree matters more than the flat list

Consider what you now gain. If the answer says "refunds take 30 days", you open the trace and see in the TOOL span that the policy text returned was empty, because the topic guess was wrong. The model did its best with nothing. The bug is in your keyword routing and not in the prompt, and you found that in under a minute. A flat log of only the model call would have sent you off rewriting the prompt for an afternoon.

Pausing and stopping tracing

Sometimes you do not want spans. A health check that calls a model every ten seconds would fill your project with noise. Wrap the code in a context manager to suppress tracing inside it:

from phoenix.trace import suppress_tracing

with suppress_tracing():
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "ping"}],
    )

To stop an instrumentor for good, call its uninstrument() method, for example OpenAIInstrumentor().uninstrument().

Streaming and token counts

A detail that catches people: when you stream a response from OpenAI, the provider does not include token usage in the stream unless you ask for it. The trace then shows no token counts and no cost. Pass stream_options={"include_usage": True} to the call and the numbers return.

Sessions, users, metadata and tags

One trace is one request. Real products have conversations, users and versions, and you will want to slice your data by them. OpenInference provides context managers, wrappers that attach extra attributes to every span created inside them, so you write the identifier once at the boundary instead of passing it through every function.

from openinference.instrumentation import using_attributes

def handle_message(conversation_id: str, user_id: str, text: str) -> str:
    with using_attributes(
        session_id=conversation_id,
        user_id=user_id,
        metadata={"app_version": "1.4.0", "channel": "web"},
        tags=["beta"],
    ):
        return answer_question(text)

Inside that block every span, including those created automatically by the OpenAI instrumentor, carries session.id, user.id, metadata and tag.tags. You can also use the smaller managers individually: using_session(session_id=...), using_user(user_id=...), using_metadata({...}) and using_tags([...]).

What sessions give you

Open your project and switch to the Sessions view. Phoenix groups traces that share a session.id and shows them as a chat timeline, turn by turn. That is how you judge a conversation rather than an isolated answer. A reply that looks fine alone can be plainly wrong in context, for instance when the bot forgets what the user said two turns earlier, and only the session view shows it.

Metadata and tags for slicing

Metadata is a dictionary of anything you like, and tags are a list of short labels. Their real value comes later, when you filter:

metadata['channel'] == 'web'
metadata['app_version'] == '1.4.0'

When you release version 1.5.0 and complaints rise, you can compare the two versions directly. If you never recorded the version, you can only guess. The habit to adopt on day one is to stamp every trace with the version of your code and of your prompt.

A small caution on naming: be consistent. app_version in one place and appVersion in another produce two separate fields, and your filter will silently match only half the data.

Annotations: attaching judgements to traces

Seeing a trace tells you what happened. An annotation records what you think about it. An annotation is a named judgement attached to a span, a trace, a document retrieved inside a span, or a whole session. It can hold a label such as correct or incorrect, a numeric score, a free-text explanation, or any combination. It also records who made the judgement, in the annotator_kind: HUMAN for a person, LLM for a model acting as judge, or CODE for a rule.

The vocabulary changed in version 14. Older tutorials call annotations "evaluations" and use px.log_evaluations and SpanEvaluations. Those are removed. Everything is now annotation, and the old REST endpoints under /v1/evaluations no longer exist; the current ones are /v1/span_annotations, /v1/trace_annotations and /v1/document_annotations.

Annotating by hand in the interface

Open a trace, select a span, and use the annotation panel to add a label, a score or a note. A person reviewing twenty traces for ten minutes and marking each good or bad with a short reason produces surprisingly valuable data. It tells you what "good" means for your product, in a form you can later teach to an automatic judge.

To make annotation consistent, an admin can define an annotation config: a named, typed annotation that appears as a ready-made control. It is either categorical (a fixed set of labels), continuous (a number within a range) or freeform. Using a config means two reviewers cannot invent correct and Correct and right for the same idea.

There is also a special annotation called a note, free text attached to a span or a trace. The name note is reserved for these. Phoenix rejects an ordinary annotation named note with an error that tells you to use the notes endpoint.

Annotating from code

When feedback arrives from your own product, such as a thumbs-down button, you send it with the Python client. Install it and create a client:

pip install arize-phoenix-client

from phoenix.client import Client

client = Client(base_url="http://localhost:6006")

client.spans.add_span_annotation(
    span_id="PASTE_A_SPAN_ID_HERE",
    annotation_name="user_feedback",
    annotator_kind="HUMAN",
    label="thumbs_down",
    score=0.0,
    explanation="The bot quoted the wrong refund window.",
)

Two points deserve attention. First, import from phoenix.client. The old px.Client() and phoenix.session.client.Client were removed in version 14 and raise ModuleNotFoundError. Second, the parameter is base_url, which was called endpoint in the old client. The span id comes from the span panel in the interface, or from your own code if you capture it when the span is created.

Why annotations are the hinge of the whole loop

Everything later depends on them. Filter for annotations['user_feedback'].label == 'thumbs_down' and you have a worklist of problem cases. Put those cases into a dataset and you have a regression test. Ask an LLM judge to reproduce your human labels and you can measure the judge before you trust it. Annotation is not paperwork; it is the currency the rest of Phoenix spends.

Evaluations: letting a model score the answers

Reading traces by hand does not scale past a few dozen. To score hundreds or thousands you need an automatic judge. Phoenix's evaluation library, arize-phoenix-evals, provides evaluators: objects that take some input and return a score with a name, a label, a numeric value and an explanation. There are two families.

Code evaluators are plain functions with deterministic rules. Did the answer contain valid JSON? Did it match the expected string exactly? Is it shorter than 200 words? They are cheap, fast and reliable, and you should reach for them first wherever a rule can express the check.

LLM-as-a-judge evaluators ask a second model to grade the first. You write a prompt describing the criterion, for example "Is this answer fully supported by the provided context?", and constrain the judge to a short list of labels. They handle fuzzy qualities like faithfulness, relevance and tone, at the cost of money, latency and the judge's own mistakes.

Installing and a first judge

pip install arize-phoenix-evals openai pandas

The current evals API, introduced in version 2 and cleaned up in version 3.0 of the package, is built around an LLM object and a ClassificationEvaluator. The old names llm_classify, OpenAIModel and run_evals are gone. If you meet them in a tutorial, the tutorial is out of date.

import pandas as pd
from phoenix.evals import LLM, ClassificationEvaluator, evaluate_dataframe

llm = LLM(provider="openai", model="gpt-4o-mini")

TEMPLATE = """You are checking a customer support answer.

Policy: {policy}
Question: {question}
Answer: {answer}

Is the answer fully consistent with the policy? Reply with exactly one word:
consistent or inconsistent."""

faithful = ClassificationEvaluator(
    name="policy_consistency",
    prompt_template=TEMPLATE,
    llm=llm,
    choices={"consistent": 1.0, "inconsistent": 0.0},
)

df = pd.DataFrame(
    {
        "policy": ["Refunds are available within 14 days of purchase with a receipt."] * 2,
        "question": ["Can I get a refund after two weeks?"] * 2,
        "answer": [
            "Refunds are only available within 14 days, so two weeks is the limit.",
            "Yes, refunds are available for up to 60 days.",
        ],
    }
)

results = evaluate_dataframe(dataframe=df, evaluators=[faithful])
print(results)

Read the template carefully. The placeholders in curly braces ({policy}, {question}, {answer}) must match the column names of the dataframe, and the last line tells the judge to reply with exactly one of the labels in choices. The choices dictionary maps each label to a numeric score. The first row should come back consistent with score 1.0 and the second inconsistent with 0.0. The exact layout of the results table depends on the library version, so print it and read it rather than assuming a column name.

Built-in evaluators

You do not always need to write templates. The library includes ready-made metrics under phoenix.evals.metrics, such as FaithfulnessEvaluator, CorrectnessEvaluator, CompletenessEvaluator, ConcisenessEvaluator, RefusalEvaluator, ToxicityEvaluator and RetrievalRelevanceEvaluator, plus the deterministic exact_match. Two deprecations to know: HallucinationEvaluator is deprecated in favour of FaithfulnessEvaluator, and the older document-relevance evaluators are deprecated in favour of RetrievalRelevanceEvaluator.

Evaluating real traces

The value appears when you score the traces already in Phoenix. The pattern is: fetch spans as a dataframe, run the evaluators over it, and log the results back as annotations so they appear in the interface.

from phoenix.client import Client
from phoenix.evals.utils import to_annotation_dataframe

client = Client(base_url="http://localhost:6006")

spans = client.spans.get_spans_dataframe(project_identifier="support-bot", limit=100)
# ... run your evaluators over `spans`, producing `results` ...
client.spans.log_span_annotations_dataframe(
    dataframe=to_annotation_dataframe(results),
    annotation_name="policy_consistency",
    annotator_kind="LLM",
)

After this runs, each scored span shows a policy_consistency annotation in the interface, and you can filter on it. The middle step is elided because it depends on how your span columns map to your template placeholders; evaluators take an input_mapping for exactly that purpose, and Mid-level covers it in detail.

Trust, but verify the judge

An LLM judge is a model, and models are wrong sometimes. Before you believe a judge, compare it against the hand labels you made in the previous section. If you labelled thirty traces and the judge agrees on twenty-eight, it is useful. If it agrees on seventeen, fix the template before you chart anything. This is the most skipped step in evaluation work and the one that separates dashboards you can act on from dashboards that only look confident.

Datasets and experiments: turning problems into tests

You now can find bad answers. The next question is how to stop them coming back. The answer is a dataset and an experiment.

A dataset is a named collection of examples. Each example has an input, which is a dictionary of whatever your application takes, an optional expected output, which is also a dictionary, and optional metadata. Phoenix versions datasets: every insert, update or delete creates a new version, so you can always say exactly which examples an experiment ran against. Datasets are the same idea as a test suite, except the expected results are sometimes fuzzy.

There are three common sources. You can build one by hand from cases you care about, you can export rows from a dataframe, or, most usefully, you can select traces in the interface, such as all the ones marked thumbs_down, and add them to a dataset with a button. The second route in code looks like this:

from phoenix.client import Client

client = Client(base_url="http://localhost:6006")

dataset = client.datasets.create_dataset(
    name="refund-questions",
    inputs=[
        {"question": "Can I get a refund after two weeks?"},
        {"question": "Do I need a receipt to get my money back?"},
        {"question": "How long does shipping to Riyadh take?"},
    ],
    outputs=[
        {"answer": "No. Refunds are available only within 14 days."},
        {"answer": "Yes, a receipt is required."},
        {"answer": "Between 5 and 8 working days."},
    ],
)
print(dataset)

Each position in inputs pairs with the same position in outputs, so keep the lists the same length. In the interface, the Datasets page now lists refund-questions with three examples.

Experiments

An experiment runs a task, which is an ordinary Python function from an example to an answer, across every example in a dataset, then scores the outputs with evaluators. The task is usually your application's entry point, so the experiment measures the real behaviour of your code on a fixed set of cases.

from phoenix.client import Client
from phoenix.client.experiments import run_experiment

from support_bot import answer_question

client = Client(base_url="http://localhost:6006")
dataset = client.datasets.get_dataset(dataset="refund-questions")


def task(input):
    return answer_question(input["question"])


def mentions_14_days(output, expected):
    # a simple code evaluator: did the answer contain the key fact?
    return "14" in output if "14" in expected["answer"] else True


experiment = run_experiment(
    dataset=dataset,
    task=task,
    evaluators=[mentions_14_days],
)

The task function may declare any of the parameters input, expected, metadata or example, and Phoenix passes the matching pieces. Notice that experiments live in phoenix.client.experiments; the old phoenix.experiments module was removed in version 14, and the arguments are now keyword-only. After the run finishes, open the Experiments tab on the dataset. You will see one row per example with the output, the evaluator results, and a link to the full trace of that run. That link is the point: a failing example in an experiment is one click from the spans that explain it.

A useful extra is repetitions. Language models are not deterministic, so a case that passes once may fail on the next run. Passing repetitions=3 to run_experiment runs each example three times and exposes the flakiness.

The improvement loop in practice

Here is how these pieces combine into a routine a team can follow.

1. You change your prompt or your retrieval logic.
2. You run the experiment against the same dataset as last time.
3. You compare the two experiment runs side by side. Did the pass rate rise? Did any example that used to pass now fail?
4. If it improved and nothing regressed, you ship. If not, you open the failing example's trace and read the prompt.

This is the difference between "I think the new prompt is better" and "the new prompt passes 41 of 45 cases, up from 36, and breaks none". It also makes change safe. Nobody is afraid to modify the prompt when a fast check stands behind it.

Prompts and the Playground

Until now the prompt lived inside your Python file. Phoenix also offers prompt management: prompts stored in Phoenix as named, versioned templates, together with the model settings to run them. Each save creates a new version, and you can attach tags such as production to a version, so the application can ask for "the prompt tagged production" instead of a hard-coded string. Changing the live prompt then becomes a tag move, with history, rather than a code deployment.

The Playground is the interface for editing and running a prompt. You pick a provider and model, write or paste a prompt with variables, and run it. Its powerful feature is that you can run the prompt over an entire dataset at once and see all the outputs in a grid, which makes it the quickest way to eyeball a change before you write any code. It can also start from a real span: open a trace, choose to replay that LLM call in the Playground, and you land in an editor holding the exact prompt that produced the bad answer, ready to tweak.

Playground needs a provider key. You can paste your key into the interface's provider settings, or administrators can store provider secrets under Settings → Secrets, where they are write-only. A provider that does not appear in the dropdown may simply be filtered by the PHOENIX_ALLOWED_PROVIDERS setting, which limits what the Playground shows.

For a first project you can ignore the programmatic side of prompts. Try the loop in the interface first: replay a bad span, edit the prompt, run it, and save a new version. Mid-level covers fetching a prompt version from code and formatting it for a provider call.

Getting your data back out: client and command line

Sooner or later you want the data outside the browser, to chart it, to share it or to script a check. Phoenix gives you three routes.

The Python client

You have already met Client. A few calls cover most needs:

from phoenix.client import Client

client = Client(base_url="http://localhost:6006")

df = client.spans.get_spans_dataframe(project_identifier="support-bot", limit=200)
print(df[["name", "start_time", "status_code"]].head())

print(client.projects.list())
print(client.datasets.list())

get_spans_dataframe returns a pandas dataframe with one row per span and columns such as name, span_kind, start_time and status_code, with attributes as attributes.* columns. Treat the column names as something to print and check, because the exact set depends on what your spans contain. You can also pass a query to narrow it, for instance only root spans:

from phoenix.client.types.spans import SpanQuery

root_spans = client.spans.get_spans_dataframe(
    project_identifier="support-bot",
    query=SpanQuery().where("parent_id is None"),
)

The old helpers get_spans_dataframe on the top-level client and query_spans were replaced in version 14; if you see them, use the forms here.

The px command line

Phoenix also has a command line tool, published on npm as @arizeai/phoenix-cli. It needs Node.js 22 or newer. It installs a program named px:

npm install -g @arizeai/phoenix-cli
px setup
px trace list --limit 5
px span list --span-kind LLM --status-code ERROR
px dataset list

px setup is an interactive helper that writes a small file called .env.phoenix holding your endpoint and project, then waits for a trace to arrive so you know the connection works. px trace list prints recent traces in your terminal, which is handy over SSH or inside a coding agent where a browser is awkward. Configuration resolves in a fixed order: command-line flags win, then the process environment, then the active profile, then .env.phoenix, then the defaults. Knowing that order solves a whole class of "why is it talking to the wrong server" mysteries.

The assistant, briefly

Since version 17 Phoenix includes PXI, a built-in assistant you can chat with inside the interface, and in a terminal with the command pxi (which needs a server on version 20 or newer). It can help you explore traces and draft filters. It is optional, and an administrator can switch it off. Treat it as a convenience on top of the skills in this guide, not a substitute for learning to read a trace yourself.

Configuration: environment variables and the files behind them

Hard-coding endpoint="http://localhost:6006/v1/traces" in every script is fine for learning and awkward afterwards. Phoenix is designed to be configured with environment variables, named settings that live outside your code. You set them once per project, and the same code then works on your laptop and on a team server.

Client-side settings

These are the ones your application and tools read:

Two of these are easy to confuse. PHOENIX_COLLECTOR_ENDPOINT is for trace export. PHOENIX_ENDPOINT is for everything else that talks to the server's API. If you set only the second, your Python application's spans will go to the default location and you will wonder why the project you configured stays empty.

The .env.phoenix file

The SDKs and the command line automatically look for a file named .env.phoenix, walking up from the current directory. It holds the same variables:

PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
PHOENIX_PROJECT=support-bot

Three rules. The name is .env.phoenix, not .env. The real process environment always beats the file. And if you ever put an API key in it, add the file to your .gitignore before the first commit, because a key pushed to a public repository is compromised the moment it lands. You can turn off file discovery with PHOENIX_DISCOVER_CONFIG=false.

The second rule bites people. If an old export PHOENIX_PROJECT=demo lives in your shell profile, it silently overrides every .env.phoenix you ever write. When traces land in the wrong project, run echo $PHOENIX_PROJECT and echo $PHOENIX_ENDPOINT first.

Server-side settings you will meet early

For PostgreSQL you need the extra driver: pip install 'arize-phoenix[pg]'. The server supports PostgreSQL 14 or newer. Use it for anything shared; SQLite is for development.

Turning on authentication

When you move beyond your own laptop, enable auth. Generate a secret and pass it in the environment:

export PHOENIX_ENABLE_AUTH=true
export PHOENIX_SECRET="$(openssl rand -hex 32)"
phoenix serve

The first login is admin@localhost with password admin. Change it immediately. After that, each application needs an API key, which you create in the interface under Settings, and then export as PHOENIX_API_KEY where your application runs. Enabling auth on a server that applications already use stops their traces until they hold keys, so plan the switch. Note also that since version 19 an API key can no longer create other API keys; keys are created by a person signed in to the interface.

If the server refuses to start after you turn auth on, read the message. It will say the secret must be set, or that it must be at least 32 characters long with a digit and a lowercase letter. The fix is a longer, better secret.

Data privacy and air-gapped machines

Phoenix stores prompts and completions, so think about what you send. Two settings are worth knowing. PHOENIX_TELEMETRY_ENABLED=false turns off anonymous web analytics from the user interface, and the documentation states that trace data is never collected. On a machine without internet access, the UI may wait many seconds trying to fetch web fonts; set PHOENIX_ALLOW_EXTERNAL_RESOURCES=false to stop that.

Reading common errors

Most failures in the first week are one of a handful of patterns. The skill is reading the symptom and knowing where to look.

No traces appear, and no error

This is the most common problem by a wide margin, and the frustrating part is that nothing fails loudly. Work through it in order.

First, read the register() banner. It shows the endpoint and the transport. If it says gRPC and port 4317 while you started Phoenix in Docker without -p 4317:4317, nothing is listening, and spans are dropped. If you are behind an HTTPS load balancer that exposes only port 443, gRPC on 4317 will never arrive either.

Second, force the transport. Pass protocol="http/protobuf" and a full .../v1/traces endpoint, which is what this guide's examples do. There is one specific trap the documentation calls out: if an environment variable holds a URL ending in /v1/traces and you let Python infer the protocol, it rewrites the port to 4317 and keeps the path, and no spans arrive.

Third, check that you set PHOENIX_COLLECTOR_ENDPOINT, not only PHOENIX_ENDPOINT, and that no stale variable from your shell profile is overriding your file.

Fourth, if you use a short script, make sure it does not exit before spans are sent. With the default simple processor each span is sent as it finishes, so this is rare. With batch=True spans are buffered, and a script that ends within a moment may need to let the provider flush.

Warnings you will see

Errors from the server

Errors from old tutorials

A very common source of confusion is code copied from a pre-2026 tutorial. These all point to the same cause: the API changed in version 13 or 14.

Permission denied on a mounted volume

If you run the -nonroot Docker image variant and mount a folder, the container user may lack write permission, and Phoenix reports "Permission denied" writing to disk. Give the volume the right ownership with chown, or use the regular image while you learn.

Putting it all together

Let us finish with one small project that uses everything above, end to end. The goal: a support bot for a fictional online shop, traced, reviewed, scored, tested and improved. It should take about an hour. Keep the server from the first section running.

Step one, set up. Create a fresh folder, a virtual environment, and install the packages:

python3 -m venv .venv && source .venv/bin/activate
pip install arize-phoenix arize-phoenix-otel arize-phoenix-client arize-phoenix-evals \
  openinference-instrumentation-openai openai pandas
phoenix serve

Run phoenix serve in one terminal, and do the rest in another with the same environment activated.

Step two, instrument. Reuse support_bot.py from earlier. Add using_attributes around a handle_message function with a session id and prompt_version set to "v1" in the metadata. Run it twelve times with a mix of refund, shipping and out-of-scope questions, using two or three session ids.

Step three, read. Open the project. Look at the tree of three or four traces. Open the Sessions view. Find a trace where the tool returned nothing. Write one sentence on why.

Step four, annotate. Hand-label at least eight traces as good or bad with a short explanation, using a name like quality.

Step five, build a dataset from the bad ones. Select the bad traces in the interface and add them to a dataset called support-regressions. Add a few good ones so the set is balanced.

Step six, evaluate. Write a judge like judge.py with a template suited to your shop. Run it over the dataset's inputs and outputs, and compare its labels against your hand labels. If it disagrees on several, adjust the template until it agrees on most.

Step seven, experiment. Run an experiment over support-regressions with your current code. Record the pass rate. Now improve answer_question so that an out-of-scope question gets an honest "I do not know" instead of a policy guess, change prompt_version to "v2", and run the experiment again.

Step eight, compare. Open the two experiment runs. Did the pass rate rise? Did any previously passing example fail? Open one of the failures and read the prompt.

Step nine, tidy. Move your endpoint and project into a .env.phoenix file, add it to .gitignore, set batch=True in register(), and write a two-paragraph README explaining how to start the server and run the experiment.

When you finish you have used every idea in this guide: the trace tree, manual and automatic instrumentation, sessions and metadata, human and automatic annotation, a dataset, an experiment and configuration by environment. The same loop is what professionals run on production systems, with larger data and more care.

What you can now do, and what comes next

You can start and check a Phoenix server, and you know which port carries what. You can trace an application automatically with register(auto_instrument=True) and explicitly with decorators, and you can read the resulting tree to find which step caused a bad answer. You can group traces into sessions, stamp them with users, versions and tags, and filter them. You can annotate by hand and from code, score traces with an LLM judge, and know why you must check the judge against human labels. You can turn failures into a dataset and use an experiment to compare two versions of your application. You can configure everything with environment variables and a .env.phoenix file, and you recognise the typical errors, especially the silent no-traces failure and the outdated-tutorial failures.

Just as important is what you now know to avoid: the old phoenix --port serve ordering, the removed px.Client, the removed llm_classify family, and the obsolete embedding and drift features. When a tutorial disagrees with this guide, check its date.

Here is where to go next.

• Mid-level covers how tracing works under the hood, including processors, batching and sampling, richer context attributes, the evals library in depth with input mappings and the built-in metrics, the experiment API with repetitions, prompt management from code, and the px command line in full.
• Senior covers running Phoenix as a platform: PostgreSQL, scaling, authentication and single sign-on, retention, upgrades, multi-team isolation, security and the point at which a different tool is the better choice.
• For neighbouring tools in this catalogue, compare the approach with Langfuse, learn a model-judging library in RAGAS, and review Docker and Kubernetes for deploying the server itself.

If you keep one habit from this guide, make it this: before you change a prompt, open a trace and read what the model was actually given. Everything else is tooling around that simple act of looking.

Sources

• Arize Phoenix documentation
• Get started
• Self-hosting and architecture
• Docker deployment
• Configuration reference
• Authentication
• Tracing overview
• What are traces
• Span kinds
• Context managers for sessions, users and metadata
• Sessions
• Filter expressions
• Evaluation quickstart (Python)
• Pre-built evaluation metrics
• Datasets concepts
• Running experiments
• Prompt engineering overview
• Python client reference
• Phoenix CLI reference
• What is my Phoenix endpoint
• Migration guide
• Release notes
• OpenInference repository
• arize-phoenix on PyPI

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. Every answer below is written the way you would say it aloud: short enough to finish in a minute, specific enough to prove you have used the tool.

The thirty-second answer

Arize Phoenix is an open-source observability and evaluation platform for LLM applications. Your app sends it traces, which are trees of timed spans recording each model call, retrieval and tool use, using OpenTelemetry and the OpenInference conventions, and Phoenix stores them, shows them in a web interface, and lets you annotate, score and turn them into datasets and experiments.

Then the line that shows you have used it: "a wrong LLM answer still returns HTTP 200, so you cannot debug it from status codes; you open the trace and read the exact prompt the model received."

The model

The core loop to recite: trace, annotate or evaluate, curate a dataset, run an experiment, iterate on the prompt, trace again. If you can explain why each arrow exists, you have understood the tool.

The commands and calls to have ready

Interviewers often ask you to "show me how you would" do something. These are the ones to have memorised.

Start the server. Flags come after serve:

pip install arize-phoenix
phoenix serve --port 6006 --grpc-port 4317

Check it:

curl http://localhost:6006/healthz
curl http://localhost:6006/readyz
curl http://localhost:6006/arize_phoenix_version

Trace an app in Python:

from phoenix.otel import register

tracer_provider = register(
    project_name="my-app",
    auto_instrument=True,
    batch=True,
    protocol="http/protobuf",
    endpoint="http://localhost:6006/v1/traces",
)

Mark your own functions:

tracer = tracer_provider.get_tracer(__name__)

@tracer.tool
def look_up_policy(topic: str) -> str: ...

Add context to every span in a block:

from openinference.instrumentation import using_attributes

with using_attributes(session_id="conv-42", user_id="u-7", metadata={"prompt_version": "v2"}):
    answer_question("...")

Read spans back and annotate:

from phoenix.client import Client

client = Client(base_url="http://localhost:6006")
df = client.spans.get_spans_dataframe(project_identifier="my-app", limit=100)
client.spans.add_span_annotation(
    span_id="...", annotation_name="quality", annotator_kind="HUMAN", label="bad"
)

And the ports: 6006 for the interface, REST, GraphQL and OTLP over HTTP at /v1/traces; 4317 for OTLP over gRPC.

Two transports and the silent failure

This is the trap interviewers love because it separates people who have run the tool from people who have read about it.

If you call register() and give it only a base URL such as http://localhost:6006 (or set PHOENIX_COLLECTOR_ENDPOINT to one) and do not say which protocol, the Python SDK infers gRPC on port 4317. That works against a local server with both ports open. It does nothing useful when the server runs in Docker with only port 6006 published, or sits behind an HTTPS load balancer that exposes only port 443. There is no crash and no error in your application. Spans simply never arrive.

The fix is to be explicit: pass protocol="http/protobuf", or pass a full endpoint ending in /v1/traces. The documentation also warns about a related combination: a /v1/traces URL in an environment variable plus an inferred protocol causes Python to rewrite the port to 4317 while keeping the path, and again nothing arrives.

The diagnostic is to read the banner that register() prints at startup. It lists the project, the span processor, the endpoint and the transport. That banner answers the question in five seconds.

Old tutorials, new APIs

Phoenix changed a great deal between versions 12 and 14, and search results are full of out-of-date code. Interviewers sometimes show a snippet and ask what is wrong with it. Know these substitutions.

The reason behind these changes is a good thing to mention. Phoenix narrowed its focus to LLM and agent applications, renamed "evaluations" to "annotations" because humans, models and code all produce them, and moved to a standalone client package so that the application-side code is small and independent of the server.

Common interview questions

What problem does Phoenix solve that normal logging does not?

LLM applications fail without failing. The request returns 200, there is no exception, and the content is wrong: a hallucinated policy, an ignored document, a misused tool. To understand that you need the actual prompt and reply, and you need them in the context of the surrounding steps, because modern applications are pipelines of retrieval, prompting and tool calls. Phoenix records each request as a trace with every step as a span, stores the model inputs and outputs, and then adds a way to score the results. Logs are flat text; traces are structured trees you can filter and query. The evaluation half matters as much: it lets you move from one suspicious answer to a measured rate across hundreds of answers.

What is a trace, and what is a span?

A span is one unit of work: a model call, a retrieval, a function. It carries a name, start and end times, a status, a parent span id and attributes. A trace is all the spans that share a trace id, arranged as a tree, representing one request through the application. The first span has no parent and is the root. If I am asked for an example, a question answering request has a root CHAIN span, a RETRIEVER child that fetched documents and an LLM child that produced the answer.

How does Phoenix relate to OpenTelemetry?

Phoenix is built on OpenTelemetry. The server is a collector that accepts OTLP, the OpenTelemetry protocol, over HTTP and gRPC. On top of the generic span format, OpenInference defines LLM-specific attribute names and span kinds, plus instrumentation packages for frameworks such as OpenAI, LangChain and LlamaIndex. The practical benefit is portability: the data is standard OpenTelemetry, so I can send it to Phoenix and to another backend, and my instrumentation does not lock me in.

What are the span kinds and why do they matter?

They describe what sort of work a span does: LLM, CHAIN, AGENT, TOOL, RETRIEVER, EMBEDDING, RERANKER, GUARDRAIL, EVALUATOR, PROMPT and UNKNOWN, the default. They matter because the interface and the evaluators treat them differently. A RETRIEVER span shows its documents with scores, and a retrieval-relevance evaluator can score each document. An LLM span shows messages and token counts. Choosing the right kind means getting the right display and the right analysis.

How do you instrument an application?

Two ways, usually combined. Automatic: install the OpenInference instrumentor for my library, for example openinference-instrumentation-openai, and call register(auto_instrument=True), or call the instrumentor's instrument(tracer_provider=...) explicitly. Every call to that library becomes a span without changing application code. Manual: create a tracer from the provider and decorate my own functions with @tracer.chain, @tracer.tool, @tracer.agent or @tracer.llm, so my business logic appears in the tree too.

What does register() do?

It builds an OpenTelemetry tracer provider configured for Phoenix and sets it as the global provider. Its main arguments are project_name, auto_instrument, endpoint, protocol, batch, headers and api_key. It reads defaults from environment variables such as PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_PROJECT. It prints a banner with the effective settings, which I use as the first debugging step.

Why use batch=True?

The default span processor exports each span synchronously as it ends, which adds network delay to the request path and is noisy under load. The batch processor buffers spans and sends them in groups in the background. The banner explicitly warns that a batch processor is advised in production. The small cost is that a short script might exit before the buffer flushes, so scripts should shut down the provider or wait.

What is a project, and what should I avoid?

A project is a container for traces, normally one per application or environment, set with project_name and stored as the resource attribute openinference.project.name. I avoid the default project for anything real because everything unnamed lands there, and I avoid mixing environments in one project because it makes comparisons meaningless.

What is a session, and how do you create one?

A session groups traces that belong to one conversation, using a shared session.id. I set it with the using_session context manager, or with using_attributes, which can also set user_id, metadata and tags together. The Sessions view then shows the conversation turn by turn, which is how I catch failures like forgetting earlier context that no single trace reveals.

What is an annotation, and how is it different from an evaluation?

An annotation is a named judgement attached to a span, trace, retrieved document or session, holding a label, a score and an explanation, with an annotator kind of HUMAN, LLM or CODE. "Evaluation" is the older name for the same idea and is removed from the current API; an evaluator is the thing that produces a score, and its output is logged as an annotation. The reason for the rename is that the judgement might come from a person clicking a button, not only from an automatic eval.

What is LLM-as-a-judge and what are its risks?

A second model grades the first against a criterion I write in a prompt, constrained to a short list of labels. It scales to thousands of examples and handles fuzzy qualities like faithfulness. The risks: the judge can be wrong or biased, it costs money and latency, and its prompt is itself something to test. So I validate a judge against a sample of human labels before trusting it, and I use deterministic code evaluators wherever a simple rule can do the job.

How would you evaluate a RAG application with Phoenix?

I would trace it so the retriever and the model call are separate spans, then run evaluators at each stage. For retrieval, a relevance evaluator scores whether each retrieved document is useful for the question. For generation, a faithfulness evaluator checks whether the answer is supported by the retrieved text. I would log the results as annotations so I can filter for low-scoring traces, and I would put the failing cases into a dataset to use in experiments. Separating the two stages tells me whether a bad answer is a retrieval problem or a generation problem.

What is an experiment?

An experiment runs a task, usually my application's entry point, across every example in a dataset, optionally several times with repetitions, and scores the outputs with evaluators. Each result links to its trace. I compare runs on the same dataset version to see whether a change helped and whether any previously passing example regressed. It is regression testing for non-deterministic systems.

Why are datasets versioned?

Every insert, update or delete creates a new version, so an experiment can state exactly which examples it ran against. Without versioning, a score going up might mean the code improved or merely that easy examples were added, and you could not tell which.

How do you find the cause of a bad answer?

I open the trace and read the LLM span's input messages first, since that is what the model actually saw. Then I look up the tree: did the retriever return the right documents, did a tool return an empty or error result, did the routing step pick the wrong branch? Only after that do I consider changing the prompt, because most bad answers are caused by bad inputs to the model, not bad instructions.

Where does Phoenix store data and what should you use in production?

By default in a SQLite file under the working directory, which is .phoenix in the home folder unless PHOENIX_WORKING_DIR is set. That is for development. Production uses PostgreSQL 14 or newer, configured with PHOENIX_SQL_DATABASE_URL and the arize-phoenix[pg] extra. Notebook launch_app() is in-process and does not persist by default.

How do you secure Phoenix?

The bare server has authentication off, which is safe only on localhost. To enable it I set PHOENIX_ENABLE_AUTH=true and a PHOENIX_SECRET of at least 32 characters with a digit and a lowercase letter, change the default admin password, and give applications API keys in PHOENIX_API_KEY. Roles are admin, member and viewer. Since version 19 an API key cannot create other keys. I also remember that traces contain prompts and completions, so access to Phoenix is access to user data.

What is the difference between Phoenix, Phoenix Cloud and Arize AX?

Phoenix is the open-source project I self-host, under the Elastic License 2.0. Phoenix Cloud is the same software hosted by Arize at app.phoenix.arize.com. Arize AX is the separate commercial enterprise platform. For data-residency-sensitive employers, self-hosting keeps prompts and completions inside their own cloud region.

Which environment variable sets the project, and which wins?

PHOENIX_PROJECT is the canonical name. PHOENIX_PROJECT_NAME is an alias, and if both are set PHOENIX_PROJECT wins. A project_name argument in code is explicit and sets the project directly. If traces land in the wrong project, I check for a stale variable exported in my shell profile, because the process environment beats the .env.phoenix file.

What is the difference between PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_ENDPOINT?

PHOENIX_COLLECTOR_ENDPOINT is where traces are exported. PHOENIX_ENDPOINT is the base URL for the client, the px command line and MCP. Python trace export does not read PHOENIX_ENDPOINT, which is a classic reason a project stays empty.

What would you do if traces show no token counts when streaming?

With OpenAI streaming the provider omits usage unless requested, so I pass stream_options={"include_usage": True}. Then tokens, and the cost Phoenix derives from its price table, appear.

What is PXI?

PXI is Phoenix's built-in assistant, introduced in version 17, available in the interface and in the terminal through the pxi command. Conversations are stored on the server since version 20. It is optional and can be disabled by an administrator.

What a strong answer covers that a weak one misses

Whiteboard: sketch a trace

A common exercise is to draw the trace for a small application. Practise this one. A user asks a refund question. Draw the root CHAIN span answer_question. Under it, a RETRIEVER span that returns two policy documents, and an LLM span that receives a prompt containing those documents and returns the answer. Annotate the RETRIEVER with "relevance evaluator scores each document" and the LLM with "faithfulness evaluator scores the answer against the documents". Then explain that if faithfulness is low but relevance is high, the model is ignoring good context, and if relevance is low, fixing the prompt is pointless.

Sixty-second self-test

Answer each aloud without looking back.

1. Why can you not debug an LLM application from HTTP status codes alone?
2. What is the difference between a trace and a span, and which span is the root?
3. Which two ports does Phoenix use, and what does each carry?
4. Where do flags go in phoenix serve, and what happens in the old order?
5. A script runs without error but no traces appear. Give your first three checks.
6. What is the difference between PHOENIX_COLLECTOR_ENDPOINT and PHOENIX_ENDPOINT?
7. What was px.log_evaluations replaced by, and why the rename?
8. Why must you check an LLM judge against human labels?
9. What does an experiment do, and why are datasets versioned?
10. What database do you use for development, and what for production?

If you hesitated on more than two, reread the matching section of the detailed guide, then try the exercise again tomorrow. Next in the series: the Mid-level track, which opens the hood on how tracing works and how to run evaluations at scale. To place Phoenix next to its neighbours, read the Langfuse guide and the RAGAS guide.

Part one of three. Most beginner problems with Phoenix come from four misunderstandings: the application and the server are two separate processes that must be told where each other is, traces contain real user text, an old tutorial is teaching removed APIs, and a score from a model is not the same as the truth. Start with the table, then the habits, the practice cards and the checklist.

Common errors at this level

Each row gives the symptom you will see, the real cause, and the fix. The longer stories behind the commonest ones follow the table.

The silent no-traces failure

This single problem accounts for more lost first evenings than everything else combined. The symptom: your script runs, prints the model's answer, and Phoenix shows nothing. The cause is almost always transport. A base URL makes the Python SDK choose gRPC on 4317, and your server either does not publish that port or sits behind something that only passes HTTPS. The fix costs one argument.

register(
    project_name="my-app",
    protocol="http/protobuf",
    endpoint="http://localhost:6006/v1/traces",
    auto_instrument=True,
)

And the diagnosis costs one glance: read the banner register() prints. Note the endpoint and the transport listed there, and compare them with how you started the server. If you used Docker, check that your command included both -p 6006:6006 and -p 4317:4317.

The shadowing variable

The second most common problem is an environment variable you forgot you set. Perhaps months ago you exported PHOENIX_ENDPOINT in ~/.zshrc for another experiment. Now every tool on the machine reads it, and it overrides the .env.phoenix file in the project you are working on. The symptom looks random: traces arrive at a different server, or the px command lists somebody else's project. The cure is to run env | grep PHOENIX whenever behaviour surprises you, and to prefer per-project .env.phoenix files over global exports.

The tutorial from last year

Because Phoenix moved quickly, the internet is full of code that no longer runs. Treat the date of any tutorial as a warning. If a snippet uses px.Client(), px.log_evaluations, llm_classify, phoenix.experiments, px.Inferences or /v1/evaluations, it belongs to a version older than 14. Do not try to patch it line by line. Find the equivalent in the current documentation and start from that.

The habits that pay off most

Keep the application and the server separate in your head

Phoenix is a server. Your application is a client that sends it data. They are two processes, often on two machines, and every problem in the first week is some form of "the client does not know where the server is" or "the server does not accept what the client sent". Hold that picture, and most errors explain themselves.

Use the decorator that matches the job

The span kind is not decoration. A function that fetches documents should be a RETRIEVER so that Phoenix can show the documents and so that a relevance evaluator can score them. A function an agent may call is a TOOL. Leaving everything as generic spans loses display and loses the ability to target evaluators by kind.

Turn failures into tests immediately

Whenever a user or you notice a bad answer, add that trace to a dataset before you fix anything. The fix then has something to be measured against, and the bad case can never silently return. After a few weeks you will have a regression suite built from real failures, which no amount of imagination could have written.

Check the judge before trusting it

An LLM judge is a model. Take a handful of traces you have labelled yourself, run the judge on the same ones, and compare. If they disagree often, the template is wrong, the label names are ambiguous, or the criterion is too vague. Fix that first. Charts built on an unchecked judge look authoritative and mean nothing.

Keep sensitive data out of spans

Traces store prompts and completions in readable form. Before you instrument a function, ask what its arguments contain. National ID numbers, phone numbers, health details and card numbers should be removed or masked before they reach a traced function. For teams handling customer data under regional data-protection rules, this habit is worth more than any setting.

Expect the UI to be a starting point

The interface is excellent for exploring and poor for repeating. As soon as you do a task twice, script it with the client, which gives you a dataframe, or the px command. The skill of moving from "I clicked this" to "I wrote this" is what turns a debugging session into a pipeline.

Practice cards

Short, self-contained exercises. Each takes a few minutes and ends in a result you can check on screen.

1. The three checks. Start a server, then run curl against /healthz, /readyz and /arize_phoenix_version. Compare the version with pip show arize-phoenix. Result: you can prove which server you are talking to.
2. Break it on purpose. Point register() at the wrong port, run a traced script, and read the banner and the error. Fix it. Result: you recognise the symptom next time.
3. Two projects. Run the same script with PHOENIX_PROJECT=alpha and again with beta. Result: two projects in the UI, and proof that the environment variable selects the target.
4. Shadow the file. Write a .env.phoenix with one project name, then export a different one in your shell and run. Result: the environment wins, and you understand precedence.
5. A decorated tree. Write a pipeline of three functions, decorated as chain, tool and retriever style steps, and call it once. Result: a three-level tree with distinct kinds.
6. A conversation. Send five messages under one session id using using_session. Result: a single session with five turns in the Sessions view.
7. Hand-label ten. Annotate ten traces good or bad, then filter for the bad ones with an annotations[...] expression. Result: a filtered worklist of your own judgements.
8. Judge the judge. Run an LLM evaluator over the same ten and count disagreements with your labels. Result: a number that tells you whether to trust it.
9. Regression dataset. Turn the bad traces into a dataset and run an experiment over it before and after a prompt change. Result: two runs you can compare row by row.
10. Turn on auth. Restart the server with PHOENIX_ENABLE_AUTH=true and a generated secret, log in, change the admin password and create an API key. Result: your applications now need the key to send traces.

Questions people ask in their first week

Do I need a paid account?

No. The open-source server is free to run yourself, and you need only the API keys of whichever model provider your application calls. A hosted option exists, called Phoenix Cloud, if you prefer not to run anything.

Does it slow down my application?

Tracing adds a small amount of work per span. The default processor sends each span as it finishes. For anything with real traffic, use batch=True so that export happens in the background. Measure on your own workload rather than trusting a general claim.

Can I use it with something other than OpenAI?

Yes. OpenInference provides instrumentors for many providers and frameworks, including Anthropic, Amazon Bedrock, Google GenAI, LangChain, LlamaIndex and more. Install the matching openinference-instrumentation-* package. Anything that speaks OpenTelemetry can also send traces.

Can my traces go to two places?

Because the data is standard OpenTelemetry, yes. A common arrangement is an OpenTelemetry Collector that forwards traces to Phoenix and to another backend. That is a Mid-level topic, but it is the reason the standard matters.

Is my data sent to Arize?

When you self-host, your traces stay in your own database. The documentation states that an optional setting, PHOENIX_TELEMETRY_ENABLED=false, turns off anonymous interface analytics and that trace data is never collected. If your employer has data-residency rules, discuss self-hosting in your approved cloud region rather than a hosted service.

How long are traces kept?

Forever, by default. Set PHOENIX_DEFAULT_RETENTION_POLICY_DAYS or a per-project retention policy before the database grows large. Deleting is permanent.

Which version should I install?

For learning, the current release. For anything shared, pin an exact version, because the project ships a minor release every few days. This guide was checked against 20.16.0.

Where do I go when the docs and a blog post disagree?

The documentation and the project's MIGRATION.md file are authoritative. When a tutorial disagrees with them, the tutorial is older. Check the date and the version number in the code's imports.

A worked mini-diagnosis

Practise reading a failure the way a professional does. Suppose a colleague says, "Phoenix is broken, my traces are gone." Here is the questioning sequence that resolves most cases in minutes.

First ask whether the server is alive: curl http://localhost:6006/healthz. If that fails, the problem is the server, and the startup log in its terminal will say why, perhaps a port conflict or a failed configuration. If it succeeds, go on.

Next ask whether the application can reach it from where it runs. A script inside a Docker container cannot use localhost to reach a server on the host; localhost there means the container itself. A server on your laptop is reachable from a container through the host's address, not through localhost. This one catches many people the first time they containerise an application.

Then read the banner and compare its endpoint with the ports the server actually exposes. Then check for shadowing variables with env | grep PHOENIX. Then confirm that an instrumentor is installed and that the library is being called after register(), not before. Order matters: if you create your OpenAI client before instrumenting, some setups miss it, so register first and build clients after.

If traces arrive but look empty, check that the function you decorated is really being called, and that an exception is not being swallowed before the span ends. If traces arrive in the wrong project, check the variable precedence again.

Notice what this routine never does: it never starts by rewriting the prompt, reinstalling Phoenix or restarting the machine. It narrows the problem by asking a fixed sequence of cheap questions. That habit is worth more than any specific command.

The day-one checklist

Copy this into your project notes and tick each line.

• The virtual environment is active and pip show arize-phoenix shows the version I expect.
• phoenix serve is running in its own terminal, with flags after serve.
• /healthz, /readyz and /arize_phoenix_version all respond.
• register() has a project_name, protocol="http/protobuf" and a full /v1/traces endpoint, or the same values come from .env.phoenix.
• I read the startup banner and it matches my intention.
• The right openinference-instrumentation-* package is installed for each library I call.
• My own functions use the decorator that matches their job.
• Sessions, users and a prompt version are attached through using_attributes or the smaller managers.
• .env.phoenix is in .gitignore, and no key is in the repository.
• No sensitive personal data passes through a traced function.
• I labelled traces by hand before building any automatic judge, and I compared the judge against those labels.
• Bad answers are going into a dataset, and experiments compare runs on the same dataset version.
• Authentication is enabled on any server that is not on localhost, and the default admin password is changed.
• A retention period is set, agreed with whoever owns the data rules.
• Versions are pinned, and I have read MIGRATION.md before any upgrade.

When every box is ticked you are operating Phoenix the way its maintainers expect. The Mid-level track takes each of these habits further, into batching and sampling, richer evaluation, production deployment and team workflows. For a neighbouring view, compare this workflow with Langfuse, and for container basics revisit Docker.

This is part two of three. The beginner level got you to a running Phoenix server, a traced application and a first look at a trace in the UI. This level is for the moment after that, when Phoenix is no longer a demo: a team depends on it, the traces carry real customer text, somebody asks why last Tuesday's answers got worse, and you need an answer that is backed by data rather than by a hunch. By the end you will know exactly how a span travels from your code into the database, how to decide what gets recorded, how to query and score it, how to turn production failures into regression tests, and how to run the server so that it survives an upgrade and an auditor.

A note on versions before anything else. This guide is written against Arize Phoenix 20.16.0, released on 2026-09-23. The server ships a minor release every few days, so whatever you install tomorrow will be newer. The sub-packages (arize-phoenix-otel, arize-phoenix-client, arize-phoenix-evals) are versioned independently, and the OTLP wire format and OpenInference conventions are backward compatible, so a recent tracing helper works against an older or newer server. Pin exact versions in anything you commit, and read the migration notes before you move across a major version.

Where this picks up

The table below lists what you already use from the beginner level and what this level adds to each topic.

Each section opens with the problem it solves and closes with a Try it you can run against a real project.

How a span travels from your code to the database

When a trace is missing or a number is wrong, the fault sits at one specific hop. Knowing the hops turns debugging from guessing into bisecting, so we start with the whole pipeline.

Phoenix is built on OpenTelemetry (OTel), the vendor-neutral tracing standard, plus OpenInference, a set of semantic conventions (agreed attribute names such as llm.model_name) and instrumentation libraries layered on top. That design choice explains a lot. Your application never talks to a Phoenix-specific protocol; it emits standard OTLP (the OpenTelemetry wire protocol), and the Phoenix server is simply a collector that happens to understand OpenInference attributes well enough to render prompts, tool calls and token counts nicely.

Walk the hops in order.

Instrumentation creates spans. An instrumentor such as OpenAIInstrumentor wraps the client library so that every call produces a span with the OpenInference attributes filled in: input messages, output messages, model name, token counts. Manual instrumentation (decorators, or the OTel tracer API) produces spans for your own functions. Either way, a span is created in memory and handed to the tracer provider.

The tracer provider stamps a resource. The provider carries the resource attributes that describe the whole process. The one Phoenix cares about is openinference.project.name. That is how a span finds its project: the project is not a property of the endpoint, it is an attribute on every batch of spans. register(project_name="checkout-agent") sets it. If you omit the argument, register() reads PHOENIX_PROJECT, then the alias PHOENIX_PROJECT_NAME (when both are set, PHOENIX_PROJECT wins), and finally falls back to default.

The span processor decides when to export. The default SimpleSpanProcessor exports each span synchronously, as it ends. That is perfect for a notebook because you see traces instantly, and poor for a service because every span costs a network round trip inside your request path. BatchSpanProcessor buffers spans and ships them in the background. You turn it on with register(batch=True). When you forget, register() prints a banner line saying it is strongly advised to use a batch span processor in production. Take that line at face value.

The exporter sends OTLP. Two transports exist: HTTP with protobuf bodies (http/protobuf) on port 6006 at /v1/traces, and gRPC on port 4317. Which one the Python SDK picks depends on the endpoint you give it, and that decision is the single most common reason that traces "just don't arrive". We look at it closely in the next section.

Phoenix ingests through a bounded queue. The server accepts the request, places spans on an in-memory queue and writes them to SQL in the background. The queue has a ceiling, PHOENIX_MAX_SPANS_QUEUE_SIZE, which defaults to 20,000. Beyond it the server answers HTTP 503 with Server is at capacity and cannot process more requests. This is a deliberate back-pressure valve, and it tells you something precise: the database is not keeping up with the ingest rate. Raising the limit only moves the problem into memory. The real fixes are client-side batching, sampling, a faster database, or more replicas behind a load balancer.

SQL is the only state. Everything else in the server is stateless: the UI, the REST and GraphQL APIs, the OTLP collector, the MCP endpoint and the background jobs all live in one process. SQLite is the default and is fine for a laptop; PostgreSQL 14 or newer is what you run in production. Migrations run automatically at startup through Alembic, which matters later when we discuss upgrades.

Two practical consequences follow from this picture. First, a missing trace has only five places to hide (your instrumentation, the provider's resource, the processor, the transport, the server), and each has a cheap test. Second, because the project is an attribute on the span and not a URL, you can point five services at one server and keep them apart simply by naming their projects.

Endpoints, protocols and configuration that actually wins

Beginner introduced the endpoint. At this level you need the full rules, because the defaults are clever and cleverness fails quietly.

register() accepts keyword-only arguments: endpoint, project_name, batch, set_global_tracer_provider, headers, protocol, verbose, auto_instrument and api_key. The endpoint logic works like this:

• If you pass a full URL ending in /v1/traces, such as http://localhost:6006/v1/traces, the SDK treats it as HTTP.
• If you pass nothing, it reads PHOENIX_COLLECTOR_ENDPOINT and, failing that, uses http://localhost:6006.
• Given only a base URL, the Python SDK uses gRPC on the same host at port 4317 (overridable with PHOENIX_GRPC_PORT), unless you pass protocol="http/protobuf", in which case it appends /v1/traces and uses port 6006.
• Phoenix Cloud URLs on app.phoenix.arize.com are special-cased to /s/<space>/v1/traces.
• The TypeScript SDK always appends /v1/traces and uses OTLP over HTTP.

That is why a remote Phoenix behind an HTTPS load balancer that only publishes 443 needs an explicit protocol. The SDK sees a base URL, assumes gRPC on 4317, and your load balancer has nothing listening there. The docs call out one nastier combination: if PHOENIX_COLLECTOR_ENDPOINT already contains /v1/traces and you let Python infer the transport, it rewrites the port to 4317 while keeping the path, and no spans arrive. When inference cannot decide, you will see the warning Could not infer collector endpoint protocol, defaulting to HTTP., which is your cue to pass protocol= yourself.

A robust production registration is explicit about everything that has a default:

import os
from phoenix.otel import register

tracer_provider = register(
    project_name=os.environ.get("PHOENIX_PROJECT", "checkout-agent"),
    endpoint="https://phoenix.example.internal/v1/traces",
    protocol="http/protobuf",
    batch=True,
    auto_instrument=True,
    verbose=True,          # keep the startup banner in your logs
)

With auto_instrument=True, the helper instruments every installed openinference-instrumentation-* package. If none are installed it prints No OpenInference instrumentors found and skips instrumentation; nothing crashes, nothing is traced, and that is the giveaway. Install the package for your framework (for example openinference-instrumentation-openai), or instrument explicitly with OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) when you want control over exactly what is wrapped.

Precedence across the tools

Configuration comes from several places, and knowing the order saves an afternoon. For the px CLI the order is: command-line flags (--endpoint, --project, --api-key), then the process environment, then the active profile, then .env.phoenix, then defaults. The Python and TypeScript SDKs and the CLI also auto-discover a file named .env.phoenix, found by walking up from the current directory. The process environment always beats that file. Two traps sit in this design:

1. A variable exported globally, in a shell profile or a tool's settings file, silently shadows the project's .env.phoenix. If px talks to the wrong server or traces land in the wrong project, run echo $PHOENIX_ENDPOINT and echo $PHOENIX_PROJECT before anything else.
2. The filename is .env.phoenix, not .env. Put it in your ignore rules before you write an API key into it.

Setting PHOENIX_DISCOVER_CONFIG=false turns the discovery off, which is what you want in CI where configuration should come only from the environment you deliberately set.

Finally, when auth is enabled the SDK sends PHOENIX_API_KEY as a bearer token, and register() picks it up automatically. If you export over gRPC and set headers yourself, the header name must be lowercase (authorization), a gRPC requirement that surprises people who copy an HTTP example.

Deciding what gets recorded: span kinds, attributes and manual spans

Auto-instrumentation captures your LLM client calls. It cannot know which of your functions is a retrieval step, which is a tool, or where one "request" begins. The quality of the trace tree, and therefore of everything you can do with it afterwards, depends on how deliberately you shape it.

Span kinds

Every span carries openinference.span.kind, written in capitals. The values are LLM, CHAIN, AGENT, TOOL, RETRIEVER, EMBEDDING, RERANKER, GUARDRAIL, EVALUATOR, PROMPT and UNKNOWN. UNKNOWN is what you get when nobody set a kind, and a trace full of UNKNOWN spans is a sign that manual instrumentation is missing. The kind is not decoration. It drives how the UI renders a span (an LLM span shows messages and token counts, a RETRIEVER span shows documents with scores), and it is a filter target: span_kind == 'LLM' is how you find every model call across a project.

Decorators and manual spans

Phoenix provides decorators on the tracer so you can mark your own functions with the right kind:

from phoenix.otel import register

tracer_provider = register(project_name="support-bot", batch=True, auto_instrument=True)
tracer = tracer_provider.get_tracer(__name__)


@tracer.tool
def lookup_order(order_id: str) -> dict:
    """Fetch an order from the internal API."""
    return {"order_id": order_id, "status": "shipped"}


@tracer.chain
def build_context(question: str) -> str:
    order = lookup_order("A-1042")
    return f"Question: {question}\nOrder: {order}"


@tracer.agent
def answer(question: str) -> str:
    context = build_context(question)
    # an auto-instrumented LLM call here becomes a child LLM span
    return f"(model answer for) {context}"

The decorators record the function's arguments as input.value and its return value as output.value, and they nest: lookup_order becomes a child of build_context, which becomes a child of answer. The auto-instrumented LLM call inside joins the same tree because OTel propagates the active span through context. @tracer.llm exists for wrapping an LLM call you make through a library with no instrumentor. When you need more control, use the standard OTel tracer API and set attributes yourself; the OpenInference packages define the attribute names in openinference-semantic-conventions.

The attributes worth setting

Some attributes are set for you, some you must add. The ones that pay for themselves:

• input.value and output.value on the root span, so the list view of your project shows the question and the answer without opening anything.
• llm.model_name and the token counts (llm.token_count.prompt and friends), which power cost tracking.
• metadata, a dictionary for anything you will want to filter by: tenant, feature flag, release version, experiment arm.
• session.id, user.id and tag.tags, which have their own section next.

Stopping tracing on purpose

Not every call should be traced. A health probe that calls the model every ten seconds, or a step that handles data you are not allowed to store, belongs outside the trace store. Wrap it in with suppress_tracing(): (importable from phoenix.trace per the docs, or from openinference.instrumentation). To turn a library off for the whole process, call that instrumentor's uninstrument(). Suppression applies to spans created inside the block, which makes it a precise tool where redaction is a blunt one.

Sessions, users, tags and metadata: making traces queryable

A trace answers "what happened in this request". Production questions are wider: "what happened to this customer across a conversation", "which release broke refund questions", "how do internal testers differ from real users". For those you attach context attributes, and OpenInference gives you context managers that apply them to every span created inside a block.

from openinference.instrumentation import using_attributes


def handle_turn(conversation_id: str, user_id: str, message: str, release: str) -> str:
    with using_attributes(
        session_id=conversation_id,
        user_id=user_id,
        metadata={"release": release, "channel": "web"},
        tags=["prod", "support"],
    ):
        return answer(message)   # the agent from the previous section

The individual managers exist too: using_session(session_id=...), using_user(user_id=...), using_metadata({...}), using_tags([...]) and using_prompt_template(template=..., variables=..., version=...). using_attributes combines them. There is also dangerously_using_project(project_name=...), which routes spans to a different project from inside a block; the docs mark it for notebooks and scripts only, and the name is a warning. In a long-running service the project belongs to the tracer provider, set once at startup.

A session is a group of traces that share a session.id. The UI renders it as a chat-like timeline, which is far easier to review than forty disconnected traces. The one rule for session ids is that they must be stable across turns of the same conversation and different between conversations. Generating a fresh id per request defeats the feature, and reusing one id for every user merges unrelated chats into one unreadable blob. Take the id from your own conversation or thread identifier.

Sessions also have their own filter expressions in the UI, so you can find conversations rather than single requests. They can reference things like num_traces_with_error and total_cost, and use any(...) comprehensions over the session's spans. Phoenix tracks LLM token cost per span and per session from a built-in price table, which the project updates almost every release. You can override prices for models it does not know or where you have a negotiated rate.

Privacy deserves a moment here. user.id should be an opaque identifier, not an email or a phone number. Metadata and message content end up in a database you administer, which for employers in the Gulf and Egypt often means data-residency rules. Self-hosting Phoenix in a regional cloud location keeps trace content in-region, and that is one of the practical reasons teams choose it. Decide what may be stored before you turn tracing on, not after the first audit.

Querying what you recorded: the filter DSL and the Python client

The UI is for looking; the filter language and the client are for asking questions repeatedly. Phoenix's filter expressions are a small Python-like language, available in the UI search box and in code through SpanQuery.where.

Useful forms, all valid in the UI:

span_kind == 'LLM'
parent_id is None
'refund' in input.value
metadata['release'] == '2026-09-30'
annotations['correctness'].label == 'incorrect'
annotations['hallucination'].score > 0.5

parent_id is None selects root spans, meaning one row per trace, which is usually what you want when sampling requests. The older evals['name'] spelling still works, but annotations[...] is current, because "evaluations" were renamed "annotations" across Phoenix in v14. Since 20.10 the DSL can also filter on span-level cost.

In code, use the Python client, not anything you find in an old tutorial. The legacy px.Client() and phoenix.session.client.Client were removed in v14; importing them raises ModuleNotFoundError. The current client lives in its own package, arize-phoenix-client:

import pandas as pd
from phoenix.client import Client
from phoenix.client.types.spans import SpanQuery

client = Client(base_url="http://localhost:6006")  # api_key=... when auth is on

query = SpanQuery().where("parent_id is None and metadata['release'] == '2026-09-30'")
df: pd.DataFrame = client.spans.get_spans_dataframe(
    project_identifier="support-bot",
    query=query,
    limit=500,
)
print(df.columns.tolist()[:8])
print(len(df), "root spans")

The endpoint parameter of the old client was renamed base_url. The DataFrame comes back with columns named after the attributes (attributes.input.value, attributes.output.value and so on), which you then slice with ordinary pandas. The same client handles the rest of the API surface. For migrations from older code the mapping is: get_spans_dataframe became client.spans.get_spans_dataframe, query_spans(q) became client.spans.get_spans_dataframe(query=q), log_evaluations became client.spans.log_span_annotations_dataframe, upload_dataset became client.datasets.create_dataset, and log_traces became client.spans.log_spans. The helper functions get_retrieved_documents, get_qa_with_reference and get_called_tools were deleted; write a SpanQuery instead.

The REST API underneath uses Authorization: Bearer <key>, and two changes are worth knowing because old scripts call the dead routes. /v1/evaluations (GET and POST) was removed in v14. Its replacements are POST /v1/span_annotations, /v1/trace_annotations and /v1/document_annotations. If a script of yours gets a 404 there, that is the reason.

Annotations: humans, code and LLMs writing judgments back

A trace records what happened. An annotation records a judgment about it: a label, a numeric score and an explanation, attached to a span, a trace, a document (a retrieved chunk inside a retriever span) or a session. Every annotation has an annotator_kind: HUMAN, LLM or CODE. That distinction is how you later separate "what the judge model thought" from "what a reviewer decided", which matters the day you need to measure how well the judge agrees with people.

An annotation config defines a named, typed annotation so that every reviewer uses the same vocabulary: categorical (a fixed set of labels), continuous (a numeric range) or freeform. Without configs, one reviewer writes good and another writes OK, and your dashboard shows two categories that mean the same thing. Notes are free-text annotations, and the name note is reserved for them. If you try to create an annotation called note through the annotation routes, the server answers 400 with The name 'note' is reserved for trace and span notes. Use POST /v1/trace_notes instead.

Logging from code:

from phoenix.client import Client

client = Client(base_url="http://localhost:6006")

client.spans.add_span_annotation(
    span_id="6f1c9a2b8d3e4f50",          # take this from the UI or a DataFrame
    annotation_name="resolution",
    annotator_kind="HUMAN",
    label="resolved",
    score=1.0,
    explanation="Customer confirmed the refund arrived.",
)

For bulk work, log a DataFrame: client.spans.log_span_annotations_dataframe(dataframe=df, annotation_name="correctness", annotator_kind="LLM"). Trace-level and document-level equivalents exist (client.traces.log_trace_annotations_dataframe, client.spans.log_document_annotations_dataframe). Document annotations are how you record "retrieved chunk 3 was irrelevant", and they are what retrieval evaluation builds on.

Two design habits separate useful annotation data from noise.

Keep names stable and meaningful. The annotation name becomes a column and a filter key. correctness, faithfulness and resolution are fine; test2 is not. Renaming later splits your history in two.

Annotate the smallest thing that holds the truth. If the retrieval was bad but the answer happened to be right, a span-level annotation on the retriever is accurate, whereas a trace-level "bad" hides where the fault lies. When in doubt, annotate at span level and roll up later.

The UI supports human annotation directly: reviewers click through a queue of traces and record labels that follow the annotation config. That is the cheapest way to collect the ground truth you need for everything in the next two sections.

Evaluating with phoenix.evals 3.x

Human labels do not scale to every production trace. Evaluators do the first pass: an LLM-as-judge (a model asked to grade an output against criteria) or plain code. The library is arize-phoenix-evals, and the most important thing to know is which API is current, because the internet is full of the previous one.

The 2.0 release (September 2025) introduced the current API, and 3.0.0 (2026-04-07) deprecated "evals 1.0" and removed the legacy experiments module. Names from older tutorials, such as llm_classify, OpenAIModel and run_evals, are gone. The vocabulary you use now is LLM, ClassificationEvaluator, create_classifier, create_evaluator, evaluate_dataframe and async_evaluate_dataframe. An evaluator returns a Score, which has a name, a kind (llm, code or human), a direction (whether higher or lower is better), and optionally a score, a label, an explanation and metadata.

A classification evaluator maps the judge's label to a number:

import asyncio
from phoenix.client import Client
from phoenix.client.types.spans import SpanQuery
from phoenix.evals import ClassificationEvaluator, LLM, async_evaluate_dataframe
from phoenix.evals.utils import to_annotation_dataframe

TEMPLATE = """You are grading a customer-support answer.

Question: {input}
Answer: {output}

Reply with exactly one word: "correct" if the answer resolves the question
without inventing facts, or "incorrect" otherwise."""

llm = LLM(provider="openai", model="gpt-4o")
correctness = ClassificationEvaluator(
    name="correctness",
    prompt_template=TEMPLATE,
    llm=llm,
    choices={"correct": 1.0, "incorrect": 0.0},
)

client = Client(base_url="http://localhost:6006")
df = client.spans.get_spans_dataframe(
    project_identifier="support-bot",
    query=SpanQuery().where("parent_id is None"),
    limit=200,
)
df = df.rename(columns={"attributes.input.value": "input", "attributes.output.value": "output"})


async def main() -> None:
    results = await async_evaluate_dataframe(
        dataframe=df, evaluators=[correctness], concurrency=10
    )
    client.spans.log_span_annotations_dataframe(
        dataframe=to_annotation_dataframe(results),
        annotation_name="correctness",
        annotator_kind="LLM",
    )


asyncio.run(main())

Read the flow: pull spans, shape the columns your template expects, run the judge with bounded concurrency, and write the scores back as annotations so they show up in the UI next to the traces. Evaluators also accept an input_mapping, which maps or transforms fields, and templates can reference span columns directly, for example {attributes.llm.input_messages}. Use the mapping so the same evaluator works on a production DataFrame and on an experiment's output without renaming columns in both places.

Built-in metrics and what to write yourself

The package ships prebuilt evaluators in phoenix.evals.metrics: FaithfulnessEvaluator (is the answer supported by the provided context), CorrectnessEvaluator, CompletenessEvaluator, ConcisenessEvaluator, RefusalEvaluator, ToxicityEvaluator, UserFrictionEvaluator, RetrievalRelevanceEvaluator, a session-level PiiDetectionEvaluator, tool evaluators (ToolSelectionEvaluator, ToolInvocationEvaluator, ToolResponseHandlingEvaluator), and code-based ones such as exact_match, MatchesRegex and PrecisionRecallFScore. Two deprecations to honour: HallucinationEvaluator is deprecated in favour of FaithfulnessEvaluator, and the document-relevance evaluators were replaced by RetrievalRelevanceEvaluator. The phoenix.evals.templating module is also deprecated in favour of phoenix.evals.llm.prompts, with removal planned for version 4.

For metrics that encode your business rules, such as "the answer must cite a policy id" or "the response must be under 120 words", write a code evaluator. It is cheaper, deterministic and explainable. Reach for an LLM judge only when the criterion is genuinely fuzzy. A good mix in practice is a handful of code checks that catch hard failures and two or three LLM judges for the fuzzy qualities.

Making the judge trustworthy

An unvalidated judge is an opinion generator. Three practices keep it honest:

1. Validate against humans. Hand-label fifty to a hundred traces, run the judge on the same ones, and compare. If agreement is poor, fix the template before trusting any dashboard.
2. Expect NOT_PARSABLE. When the judge's output does not match one of your choices, the result is NOT_PARSABLE. The two usual causes are truncated output (finish_reason="length", so raise max_tokens) and a template that does not force one exact label. Tell the model to reply with exactly one of the choice labels and nothing else.
3. Use a stronger or different model as judge than the model under test when you can, so a model is not grading its own habits.

Every judge call is an LLM call that you pay for, in money and in latency. Evaluating every production trace is rarely necessary; sample, or evaluate only the traces that matter (errors, low-confidence, flagged by users).

Datasets: turning production failures into regression tests

The moment Phoenix pays off most is when a failure in production becomes a test that runs before the next release. The object that makes that possible is the dataset: a versioned collection of examples. Each example has an input (a dictionary), an optional expected output (also called the reference or expected value, also a dictionary) and metadata. Every insert, update or delete creates a new dataset version, so an experiment can say precisely which snapshot it ran against. Splits tag subsets of a dataset, for example smoke for twenty fast examples and full for the lot.

Build a dataset from the spans you just queried:

from phoenix.client import Client
from phoenix.client.types.spans import SpanQuery

client = Client(base_url="http://localhost:6006")

bad = client.spans.get_spans_dataframe(
    project_identifier="support-bot",
    query=SpanQuery().where(
        "parent_id is None and annotations['correctness'].label == 'incorrect'"
    ),
    limit=100,
)
bad = bad.rename(
    columns={"attributes.input.value": "question", "attributes.output.value": "bad_answer"}
)

dataset = client.datasets.create_dataset(
    name="support-regressions",
    dataframe=bad,
    input_keys=["question"],
    output_keys=[],              # no gold answer yet
    metadata_keys=["bad_answer"],
)

You can also pass inputs=[...], outputs=[...] and metadata=[...] as lists. Notice the workflow: the dataset is made of real inputs that really failed, which is a far better test set than anything you would invent. The expected output is empty here because you do not yet have a gold answer. Add one by reviewing the examples, or use evaluators that do not need a reference (such as a faithfulness or rubric check) in the experiment.

Three habits keep datasets valuable:

• Curate rather than dump. A hundred mixed-quality failures produce noisy experiments. Twenty well-understood ones, each with a reason it is in the set, produce signal. Put the reason in metadata.
• Include passes. A regression set of only failures cannot tell you that a change broke something that used to work. Add a slice of examples that were correct.
• Watch for sensitive data. A dataset copies prompt text out of traces. If the traces hold personal data, the dataset does too, with its own access and retention story.

To reread a dataset, use client.datasets.get_dataset(dataset="support-regressions") and call .to_dataframe() on the result. client.datasets.list() and client.datasets.get_dataset_versions(...) show what exists and what changed.

Experiments: measuring a change before you ship it

An experiment runs a task (a function that takes one example and produces an output) over a dataset, then scores the outputs with evaluators. It answers "is prompt B better than prompt A on the examples that matter". Results are linked to traces, so you can click from a low score straight into the execution that produced it.

from openai import OpenAI
from phoenix.client import Client
from phoenix.client.experiments import run_experiment

client = Client(base_url="http://localhost:6006")
oai = OpenAI()
dataset = client.datasets.get_dataset(dataset="support-regressions")


def task(input):                      # parameters are matched by name
    resp = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer briefly. Never invent policies."},
            {"role": "user", "content": input["question"]},
        ],
    )
    return resp.choices[0].message.content


def under_120_words(output):
    return len(str(output).split()) <= 120


experiment = run_experiment(
    dataset=dataset,
    task=task,
    evaluators=[under_120_words],
    experiment_name="brief-system-prompt",
    repetitions=3,
)

Task functions can declare any of these parameters, and Phoenix passes the matching piece: input, expected (with reference as an alias), metadata and example. Declare only what you use. The module moved in v14: use phoenix.client.experiments (or client.experiments.run_experiment(...)), and note that arguments are keyword-only. Code that imports phoenix.experiments is from before the move.

Repetitions run every example N times. LLM outputs vary, so a single run per example mixes real signal with sampling noise; three or five repetitions give you a spread. If a change moves the average by less than the spread between repeats, you have not shown an improvement.

Evaluators can also be attached after the fact with evaluate_experiment(experiment=..., evaluators=[...]). That matters because a judge is slow and costs money: you run the task once, then iterate on the evaluators, or add a new metric later, without regenerating outputs. You can build evaluators for an experiment from the same classes as in the previous section, so the judge you validated on production traces is the same one scoring the experiment. Consistency between "how we grade in production" and "how we grade before shipping" is what makes the numbers comparable.

Reading results honestly

The UI compares experiments side by side per example. Look at the examples that changed, not only the mean. A mean that rises while three previously correct examples turn incorrect is a trade you should choose deliberately. Also remember that an experiment on a small dataset is evidence, not proof. Its job is to catch regressions and to rank candidate changes, which it does well at twenty to fifty curated examples.

Experiments in CI

Because the client is plain Python (and TypeScript for JS shops), an experiment can run in a pipeline. The docs cover pytest, vitest and jest integrations for running evals in CI. A reasonable pattern is to run the small smoke split on every pull request, fail the build on a hard threshold for code-based checks, and run the full dataset nightly with judge evaluators and a report. Keep judge-based gates advisory at first; teams that make a noisy metric a hard gate learn to ignore or bypass it. If you already run Docker in your pipeline, the Docker guide covers how to keep that stage cached and fast.

Prompts and the Playground: versioning what you iterate on

Prompts change more often than code and break things more quietly. Prompt Management in Phoenix gives a prompt a name, a version history and model configuration, and lets versions carry tags. The tag production is the useful convention: your application asks for "the production version of the support prompt" and a human moves the tag when a new version is ready. Rollback then means moving a tag, not redeploying.

The Playground is the UI for running a prompt against a provider, against a single input or against a whole dataset. It is where you iterate. When a trace looks wrong, you can open it in the Playground with the exact messages, edit the prompt and rerun. The providers it offers can be limited with PHOENIX_ALLOWED_PROVIDERS (for example OPENAI,ANTHROPIC), but note that this is a UI filter and not enforcement at the API level, and an unknown name causes a startup error. Provider credentials can live in Settings under Secrets, which is admin-only and write-only.

From code, create and fetch prompts with the client:

from openai import OpenAI
from phoenix.client import Client
from phoenix.client.types import PromptVersion

client = Client(base_url="http://localhost:6006")

client.prompts.create(
    name="support-answer",
    version=PromptVersion(
        [
            {"role": "system", "content": "You are a support agent. Be brief and never invent policy."},
            {"role": "user", "content": "{{question}}"},
        ],
        model_name="gpt-4o-mini",
    ),
    prompt_description="Customer support answer prompt",
)

prompt = client.prompts.get(prompt_identifier="support-answer")
kwargs = prompt.format(variables={"question": "Where is my refund?"})
reply = OpenAI().chat.completions.create(**kwargs)

prompt.format(...) returns keyword arguments in the shape the provider library expects, which you pass straight to the client call. Keeping the prompt out of your source tree has a real cost as well as a benefit, so decide consciously. The benefit is that prompt engineers iterate without a deploy and every change is versioned and tied to experiments. The cost is that application behaviour now depends on state outside git. Mitigate that by fetching a pinned version or tag in production, caching the result with a sensible refresh, and recording the prompt version in span metadata with using_prompt_template(template=..., variables=..., version=...) so a bad trace tells you which prompt produced it.

Since 20.5 prompt versions can also be created over REST, which helps when a CI job promotes a prompt after an experiment passes. That closes the loop: change the prompt, run the experiment on the dataset, and if it clears the bar, create the version and move the production tag.

Running the server for real: PostgreSQL, containers and Kubernetes

Everything so far works against the SQLite default on your laptop. A shared server needs three upgrades: a real database, a pinned and reproducible deployment, and a plan for change.

PostgreSQL

SQLite is for development only. For anything shared, use PostgreSQL 14 or newer, reached through the asyncpg driver, which is the only PostgreSQL driver since v14 (the older psycopg was removed). Install the extra with pip install 'arize-phoenix[pg]', or use the container, and set the URL:

export PHOENIX_SQL_DATABASE_URL='postgresql://phoenix:change-me@db.internal:5432/phoenix'
export PHOENIX_PORT=6006 PHOENIX_GRPC_PORT=4317
phoenix serve

You can instead provide PHOENIX_POSTGRES_HOST, _PORT, _USER, _PASSWORD and _DB separately. Since v12 the host variable may no longer include a port; PHOENIX_POSTGRES_HOST=db:5432 fails at startup, and the port goes in PHOENIX_POSTGRES_PORT. The URL form accepts TLS parameters such as sslmode, and sslmode=verify-full is the right setting for a database across a network. PHOENIX_SQL_DATABASE_SCHEMA selects a PostgreSQL schema and creates it if it is missing, which is one way to run several Phoenix instances against one database server. On AWS or Azure, prefer IAM authentication (PHOENIX_POSTGRES_USE_AWS_IAM_AUTH) or managed identity (PHOENIX_POSTGRES_USE_AZURE_MANAGED_IDENTITY) over static passwords. An optional read replica is configured with PHOENIX_SQL_DATABASE_READ_REPLICA_URL and serves read queries.

Back the database up, and test the restore. Because the database is the only stateful component, it is the only thing that matters in a disaster.

Docker

Pin the tag. The docs warn against latest, and with a release every few days latest is a moving target that can carry a migration you did not plan for.

services:
  phoenix:
    image: arizephoenix/phoenix:20.16.0
    depends_on:
      db:
        condition: service_healthy
    ports:
      - "6006:6006"   # UI, REST, GraphQL, OTLP/HTTP
      - "4317:4317"   # OTLP/gRPC
    environment:
      PHOENIX_SQL_DATABASE_URL: postgresql://phoenix:${DB_PASSWORD}@db:5432/phoenix
      PHOENIX_ENABLE_AUTH: "true"
      PHOENIX_SECRET: ${PHOENIX_SECRET}
      PHOENIX_DEFAULT_RETENTION_POLICY_DAYS: "30"
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: phoenix
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: phoenix
    volumes:
      - database_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U phoenix"]
      interval: 5s
      retries: 10
volumes:
  database_data:

The -nonroot image variants exist for hardened clusters. They need write permission on mounted volumes, and a Permission denied writing to disk is the symptom when they do not have it; fix it with fsGroup or chown on the volume. To persist SQLite in a container, mount a volume and set PHOENIX_WORKING_DIR to it. For background on health checks, depends_on and image tags, see the Docker guide.

Kubernetes with Helm

The official chart lives at oci://registry-1.docker.io/arizephoenix/phoenix-helm. Its defaults differ from the bare server in ways that matter: auth is on, access tokens last 60 minutes rather than 10, the refresh token lasts 30 days rather than 7, a bundled PostgreSQL is deployed, and ingress is enabled. The chart's default image also lags the server release (chart 12.0.15 defaults to the 20.14.0 nonroot image), so always set image.tag yourself. Persistence strategies are mutually exclusive: bundled PostgreSQL, SQLite on a PVC, or an external database through database.url.

image:
  tag: version-20.16.0-nonroot
auth:
  enableAuth: true
  createSecret: false        # bring your own Secret
postgresql:
  enabled: false
persistence:
  enabled: false
database:
  url: postgresql://phoenix@pg.internal:5432/phoenix   # password comes from a Secret
  defaultRetentionPolicyDays: 30
server:
  enablePrometheus: true

helm upgrade --install phoenix oci://registry-1.docker.io/arizephoenix/phoenix-helm \
  --version 12.0.15 -f values.yaml --namespace observability

The chart wires /healthz as the liveness probe and /readyz as the readiness probe. Readiness includes database connectivity, which is exactly the signal you want for deciding whether to send traffic to a replica. Uninstalling the release keeps the PostgreSQL volume claim, which is kind and also means a leftover claim can surprise you. The Helm guide and the Kubernetes guide cover the chart mechanics and cluster side of this in depth.

Scaling out

Phoenix scales horizontally: run several replicas behind a load balancer against one PostgreSQL database. Remember that each replica has its own ingest queue and its own 20,000-span ceiling. The scaling signals to watch, in the order they usually bite, are database write latency, server memory (it grows with ingest volume, attribute cardinality and retention), and the 503 rate on /v1/traces. Arize positions its commercial AX product for very high volume and OLAP workloads; Phoenix publishes no numeric throughput ceiling, so measure your own.

Upgrades and migrations without losing a weekend

With a release every few days, upgrading is routine work, and most of it is uneventful. The dangerous cases are few and knowable.

Migrations run at startup through Alembic, and new instances block until they finish. Read the MIGRATION.md in the Phoenix repository for every major version you cross. The major versions since 12 contain the changes you must plan around: v12 forbade a port inside PHOENIX_POSTGRES_HOST; v13 removed inferences, embeddings and the UMAP point cloud (Phoenix stopped being an ML drift tool) and changed Azure OpenAI to the v1 API; v14 made the CLI subcommand-first, removed the legacy client, and removed the /v1/evaluations endpoints; v16 added code evaluators and sandboxes; v17 shipped PXI; v18 restructured indexes; v19 made Phoenix an OAuth2 authorization server and added the /mcp endpoint; v20 persisted PXI sessions.

A safe routine:

1. Back up the database and confirm you could restore it.
2. Read the migration notes for every major version between yours and the target.
3. Upgrade a staging instance first, against a copy of production data if size allows, and time the migration.
4. On PostgreSQL with rolling deploys, set PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true so indexes are built with CREATE INDEX CONCURRENTLY. It runs two to three times slower but avoids locking the table. For very large spans tables the v13 notes describe pre-creating a no-op index (ix_spans_session_id) before upgrading.
5. Run migrations as their own step with phoenix db migrate (for instance in a Kubernetes init job) rather than letting N replicas race at boot.
6. Do not interrupt a long migration. The v9 annotation migration is the documented example where a restart mid-flight can leave the database needing manual repair.

If a v18 migration fails and mentions an INVALID index, an earlier concurrent build failed; run DROP INDEX CONCURRENTLY IF EXISTS <name> and rerun.

The v20 upgrade needs no manual steps, but it adds three tables (agent_sessions, agent_session_messages, agent_session_snapshots) for the PXI assistant and has a server-first rule: the pxi terminal command and the /v1/agent_sessions routes need a server on 20.0.0 or newer, so upgrade the server before the CLI. The old POST /agents/server/sessions/{session_id}/chat route is deprecated in favour of /v1/agent_sessions.

Authentication, API keys and the proxy in front

The bare server defaults to auth off, which is right for localhost and wrong for anything reachable by another person. With auth off, every API and the /mcp endpoint are open. Never expose an unauthenticated Phoenix publicly.

Turn auth on with PHOENIX_ENABLE_AUTH=true and a PHOENIX_SECRET. The secret must be at least 32 characters with at least one digit and one lowercase letter, and it signs tokens and derives the encryption key for stored secrets. Generate it with openssl rand -hex 32. PHOENIX_ADMIN_SECRET (same rules, but different from PHOENIX_SECRET) acts as a bearer token for the first system user and is only valid when PHOENIX_SECRET is set. The error messages are explicit, for instance PHOENIX_SECRET must be set when auth is enabled with PHOENIX_ENABLE_AUTH, and each fix is exactly what the message says. The first login is admin@localhost with the password admin (or whatever PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD was at first creation); change it immediately.

Roles and keys

There are three roles. ADMIN manages users, system keys, secrets, sandbox configuration and retention assignment. MEMBER can write (traces, annotations, datasets, experiments, prompts). VIEWER is read-only but can still manage their own user keys. A 403 on a POST, PUT or DELETE to /v1/... almost always means the caller has the VIEWER role.

There are two kinds of API key. A system key belongs to the instance, is created by an admin and survives that admin leaving; use these for applications and pipelines. A user key acts as its user and is deleted with them; use these for people's scripts. A v19 change bites automation: API keys can no longer mint API keys, through GraphQL (createUserApiKey, createSystemApiKey) or REST. User keys come from a human session, and system keys come from an admin session or from PHOENIX_ADMIN_SECRET. A provisioning script that used one key to create the next gets Unauthorized; restructure it to use the admin secret.

For people, use single sign-on instead of passwords. Phoenix can act as an OIDC client for Google, AWS Cognito, Microsoft Entra ID, Okta, Keycloak, Auth0 or any OIDC provider with discovery, configured with PHOENIX_OAUTH2_<IDP>_CLIENT_ID, _CLIENT_SECRET and _OIDC_CONFIG_URL. LDAP is supported too. Without role mapping, new SSO users arrive as VIEWER, so map groups to roles (_ROLE_MAPPING, _ALLOWED_GROUPS). A frequent surprise is a user landing as VIEWER or denied sign-in because the group claim is only in the access token; Phoenix reads claims from the ID token or UserInfo, so configure the identity provider to put them there and request the right scope.

Two meanings of "OAuth"

Keep these apart. In v19 Phoenix also became an OAuth2 authorization server for its own clients: px auth login and MCP clients such as Claude Code and Cursor use it (authorization code with PKCE and dynamic client registration). It is on by default when auth is on. PHOENIX_ENABLE_OAUTH2_AUTHORIZATION_SERVER=false disables it, after which /oauth2/* returns 404 and the CLI says This Phoenix server does not support OAuth login; use an API key.

Behind a reverse proxy

The v19 release put real requirements on the proxy in front of Phoenix:

• Set PHOENIX_ROOT_URL to the external URL, and PHOENIX_HOST_ROOT_PATH if you serve under a subpath.
• Forward /.well-known/* at the host root unmodified.
• Do not put proxy authentication in front of /mcp, /oauth2/register, /oauth2/token, /oauth2/revoke or /.well-known/*.
• Let OPTIONS requests through, and pass the WWW-Authenticate, Mcp-Session-Id and Access-Control-* headers unmodified.
• Do not buffer or time out the /mcp event stream (in nginx, proxy_buffering off).

When MCP or OAuth clients fail behind a proxy with CORS errors or a redirect to your SSO page, the proxy is gating or buffering one of those anonymous endpoints. Also set PHOENIX_CSRF_TRUSTED_ORIGINS (CSRF protection is off until you do) and PHOENIX_USE_SECURE_COOKIES=true when serving over HTTPS. A 401 with Token is not valid for this resource usually means PHOENIX_ROOT_URL does not match the URL clients actually use.

Retention, capacity, cost and watching Phoenix itself

A trace store grows without limit unless you stop it. By default Phoenix keeps traces forever: the Default retention policy is 0 days, which means keep everything. Decide your policy before the disk decides for you.

Set an instance-wide default with PHOENIX_DEFAULT_RETENTION_POLICY_DAYS (or the Helm value database.defaultRetentionPolicyDays), or create per-project policies in the UI with a maximum age and/or a maximum trace count plus a cron schedule (the documented default schedule is weekly, Sunday midnight). Assigning a policy needs the admin role. Deletion is irreversible, so treat a policy as a data-governance decision: keep production traces for as long as your debugging and compliance needs require, and keep curated datasets, which copy what they need out of the traces, as the long-term record. Empty sessions are cleaned up on an hourly sweep. PXI chats have their own limits (30 idle days, 30 chats per user, temporary chats deleted after 24 hours), adjustable by admins under Settings, then Assistant.

Disk guardrails protect you from the worst case. PHOENIX_DATABASE_ALLOCATED_STORAGE_CAPACITY_GIBIBYTES, PHOENIX_DATABASE_USAGE_EMAIL_WARNING_THRESHOLD_PERCENTAGE and PHOENIX_DATABASE_USAGE_INSERTION_BLOCKING_THRESHOLD_PERCENTAGE make the server email admins as usage climbs and eventually refuse inserts. The email part needs SMTP configured (PHOENIX_SMTP_HOSTNAME, _PORT, _USERNAME, _PASSWORD, _MAIL_FROM).

What does the storage cost depend on? Payload size and cardinality more than span count. A RAG application that puts ten retrieved documents of two thousand tokens each into every span stores a lot of text. Two levers help without losing the ability to debug: trace fewer things (suppress health checks, evaluate a sample) and record less per span. OpenInference supports configuration to hide inputs, outputs or images for privacy and size; check the instrumentation docs for the exact setting names in your version before relying on them.

Observing the server

Turn on Prometheus metrics with PHOENIX_ENABLE_PROMETHEUS=true; they are served on port 9090, and Helm's server.enablePrometheus does the same. Metrics include request timing (starlette_requests_processing_time_seconds_summary), exceptions (starlette_exceptions_total), memory_usage_bytes, cpu_usage_percent, bulk-loader insertion and exception counters, rate_limiter_throttles_total, jwt_store_api_keys_active, and the disk family (database_disk_usage_bytes, database_disk_usage_ratio, database_insertions_blocked). A small set of alerts covers most of the risk:

• database_insertions_blocked above zero: ingestion has stopped.
• database_disk_usage_ratio climbing toward your blocking threshold.
• A sustained rate of 503 responses on /v1/traces.
• Readiness failures on /readyz.
• Growth in starlette_exceptions_total.

Phoenix can also trace itself: set PHOENIX_SERVER_INSTRUMENTATION_OTLP_TRACE_COLLECTOR_HTTP_ENDPOINT (or the gRPC variant) and the server emits its own spans to a collector, which is useful when an endpoint is slow and you need to know whether the time is in the database or in Python. Log level and format come from PHOENIX_LOGGING_LEVEL and PHOENIX_LOGGING_MODE, with PHOENIX_LOG_SQL available when you need to see the queries. For a fuller treatment of dashboards and alerting, the Langfuse guide and the LangSmith guide show how the neighbouring tools approach the same operational questions, which is useful when choosing between them.

The px CLI, MCP and PXI

Three newer surfaces let you and your tools reach Phoenix without writing code: a command-line client, a built-in MCP server, and an in-app assistant.

The px CLI

The CLI is an npm package, @arizeai/phoenix-cli, which needs Node 22 or newer and installs the binaries px, phoenix-cli and pxi. Its value is scripting and quick inspection:

npm install -g @arizeai/phoenix-cli
px setup                                   # writes .env.phoenix, waits for a trace
px trace list --limit 5 --last-n-minutes 60 --include-annotations --format json
px span list --span-kind LLM --status-code ERROR --limit 20
px dataset list
px experiment list --dataset support-regressions

px setup is the fastest way to configure a project: it writes .env.phoenix and waits for a trace to arrive, exiting with code 6 if none did (other codes: 1 error, 2 cancelled, 3 bad flags, 4 auth, 5 unreachable). Profiles store several servers: px profile create prod --endpoint ... --project ... --api-key ... --activate, then px profile use. px auth login signs in through the OAuth flow described earlier, so you do not paste a key into a terminal. px api graphql '<query>' runs read-only GraphQL (mutations and subscriptions are rejected), and --curl prints the equivalent request.

Deletion commands such as px trace-annotations delete are disabled unless PHOENIX_CLI_DANGEROUSLY_ENABLE_DELETES=true. Keep that variable out of your shell profile and set it for a single command when you mean it.

The MCP endpoint

Since v19 the server mounts an MCP (Model Context Protocol) server at /mcp over Streamable HTTP, controlled by PHOENIX_ENABLE_MCP_SERVER (default true). By default it exposes "code mode" tools: search, get_schema, tags, list_tools, and an execute tool that runs model-written Python in a sandbox. Set PHOENIX_ENABLE_MCP_CODE_MODE=false for a plain tool list gated by group. px setup mcp --agent claude|codex|gemini|cursor|opencode|vscode writes the client configuration for you. This lets a coding agent inspect your traces while it debugs the code that produced them, which is a genuinely useful loop. See the MCP guide for the protocol itself.

The security point is blunt: with auth disabled, /mcp needs no credentials. With auth enabled it follows roles, but code mode executes code, so on a shared instance decide whether you want it, and restrict with PHOENIX_ENABLE_MCP_CODE_MODE, PHOENIX_ALLOWED_SANDBOX_PROVIDERS and the PHOENIX_AGENTS_DISABLE_* switches.

PXI

PXI is the in-app assistant, shipped in v17 and persisted server-side since v20. You open it in the UI or with pxi in a terminal, ask about your data ("which tool call failed most this week"), and it answers using the same APIs. It uses a model provider you configure, so its calls cost money and its prompts leave your environment through whichever provider you choose. PHOENIX_DISABLE_AGENT_ASSISTANT=true turns it off entirely, which is the right default for locked-down environments until you have a policy.

Debugging beyond "no traces appear"

The beginner level taught you to check the endpoint. These are the next layers, in the order that usually finds the fault fastest.

Bisect the pipeline. Use the five hops from earlier. Is a span created (add a print or a console exporter)? Does it carry the right project resource? Did the processor flush (try batch=False briefly)? Did the exporter reach the server (the OTel SDK logs lines like Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in ...)? Did the server accept it (look for 401, 415, 503 in its logs)? Each has one test.

Read the status codes at the ingest door. 401 means credentials are missing, wrong or expired. 415 (Unsupported content type or Request body is invalid ExportTraceServiceRequest) means JSON OTLP or a wrong encoding reached the protobuf endpoint; use the http/protobuf exporter or gRPC. 503 is the queue ceiling. 403 is a role problem. 400 with the reserved-note message is a naming problem. A table of status code to cause, kept next to the runbook, pays for itself.

Wrong project, wrong server. Check PHOENIX_PROJECT, PHOENIX_ENDPOINT and PHOENIX_COLLECTOR_ENDPOINT in the actual process environment, not in the file you think is used. Remember that the process environment beats .env.phoenix.

Spans arrive but the tree is broken. Orphaned spans that should be children usually mean context was lost across a boundary: a thread pool, an asyncio task created without context, or a background job. OTel propagates context through contextvars, which survive await but not every thread handoff. Either pass the context explicitly or start a new span at the boundary and link it by attributes such as session.id.

Missing token counts or cost. Check streaming (include_usage), check that the model name is one in the price table, and override prices for custom names.

Evals returning NOT_PARSABLE. Raise max_tokens, tighten the template to demand exactly one label, and look at a raw judge response rather than the parsed result.

Slow UI. In an air-gapped install a 10 to 20 second delay loading the UI is usually the page waiting on Google Fonts. Set PHOENIX_ALLOW_EXTERNAL_RESOURCES=false. Large projects with deep filters are slow for a database reason: check PostgreSQL itself, and consider a read replica.

Port in use. Address already in use ([Errno 48] on macOS, [Errno 98] on Linux, [WinError 10048] on Windows) means another process holds 6006 or 4317. Use phoenix serve --port 6007 --grpc-port 4318 and update the exporter.

Putting it all together

Here is the loop you now have, end to end, on one realistic project: a support agent that answers order questions.

1. Trace deliberately. The service registers once at startup with an explicit endpoint, protocol="http/protobuf", batch=True, and a project per environment. Tools, retrieval and the agent entry point are decorated so the tree has real span kinds. Each request runs inside using_attributes with the conversation id, an opaque user id, and the release in metadata.
2. Watch it. Prometheus scrapes the server; alerts cover blocked inserts, disk ratio, 503s and readiness. Retention is set to thirty days for production traces.
3. Score it. A scheduled job pulls yesterday's root spans with SpanQuery, samples two hundred, runs a validated correctness judge plus a code check for policy ids, and logs both back as annotations. Reviewers label a weekly sample by hand so the judge can be re-validated.
4. Harvest failures. Incorrect or user-flagged traces flow into a support-regressions dataset with a reason in metadata, alongside a slice of known-good examples.
5. Change safely. Prompt edits are made as new prompt versions. Each candidate runs as an experiment with three repetitions; results are compared per example, and CI runs the small split on every pull request.
6. Promote. When a version clears the bar, the production tag moves, and the application, which fetches that tag and records the version in span metadata, picks it up. If quality dips, the same metadata tells you which version to roll back to.
7. Operate. The server runs as a pinned image against PostgreSQL behind an authenticated proxy, upgrades go staging first with a backup and a separate migration step, and keys are system keys held in a secret store.

# the skeleton of the weekly routine
px trace list --last-n-minutes 10080 --include-annotations --format json > week.json
python score_yesterday.py            # pulls spans, runs judges, logs annotations
python harvest_failures.py           # appends to the support-regressions dataset
python run_experiments.py --split smoke

None of these steps is large. The leverage comes from their being connected: a trace becomes an annotation, an annotation selects a dataset example, a dataset example gates a prompt change, and the prompt version shows up in the next trace.

Where you are now

You can explain how a span gets from your code into the database and bisect the pipeline when it does not. You can shape traces with span kinds, decorators and context attributes, and you know which configuration wins when several disagree. You can query your data with the filter language and the client, record human, code and LLM judgments as annotations, and evaluate with the current phoenix.evals API instead of the removed one. You can turn production failures into versioned datasets, measure changes with repeated experiments, and manage prompts with tags. You can deploy Phoenix on PostgreSQL with Docker or Helm, secure it with roles, keys and a correct proxy, bound its storage with retention, and watch it with Prometheus.

The senior level picks up the platform side: architecture and failure modes, which component breaks first at scale, the trust model behind sandboxes and MCP, running one Phoenix as a service for several teams when a single instance is a single tenant, governance, and the honest question of when another tool fits better. If you are comparing tools now, the Langfuse and LangSmith guides show the same ideas from different angles, and the RAGAS guide covers retrieval-focused evaluation you can feed into Phoenix annotations.

Sources

• Arize Phoenix documentation
• Self-hosting: architecture
• Self-hosting: configuration
• Deployment with Docker
• Deployment with Kubernetes and Helm
• Authentication
• Network security
• Upgrades and migrations
• Production guide
• Data retention
• Secrets
• API keys
• Access control (RBAC)
• PXI
• Remote MCP
• Sessions
• Projects
• How tracing works
• Span kinds
• Context managers
• Filter expressions
• Cost tracking
• Evaluators
• Pre-built metrics
• Datasets
• Running experiments
• Repetitions
• Prompt management
• Python client reference
• Phoenix CLI reference
• Release notes
• MIGRATION.md
• OpenInference

Part two of three. This is a cumulative review of the beginner and mid-level material on Arize Phoenix, organised by the kind of question an interviewer asks someone who already runs it at work: scenarios, trade-offs, and "what would you check first". It takes about thirty-five minutes. The answers are written the way you would say them aloud, and after several of them there is a short note on what separates a strong answer from a weak one. Senior reviews all three levels.

Where this picks up

The foundations in one screen

Phoenix is an open-source observability and evaluation platform for LLM applications, built on OpenTelemetry and the OpenInference conventions. One server collects traces, stores them in SQL, renders them in a UI, and supports the loop of tracing, annotating, curating datasets, running experiments and iterating on prompts.

Have these facts ready, because interviewers probe them: the current server line is 20.x and releases arrive every few days; the license is Elastic License 2.0, not Apache; the default ports are 6006 (UI, REST, GraphQL, OTLP over HTTP) and 4317 (OTLP over gRPC); SQLite is for development and PostgreSQL 14 or newer is for production; and since v14 "evaluations" are called annotations.

Scenario questions

"Traces from the new service are not showing up. Walk me through it."

I bisect the pipeline rather than guess. A span is created by instrumentation, stamped with a project by the tracer provider, handed to a span processor, sent by an exporter and accepted by the server. So I ask five things in order. Is the instrumentation actually active? If auto_instrument=True printed "No OpenInference instrumentors found", no instrumentor package is installed and nothing is traced. Is the project what I expect? It is a resource attribute set by register(project_name=...) or PHOENIX_PROJECT, so traces may be in a different project than the one I am looking at. Did the processor flush? A short script can exit before a batch processor ships anything. Did the exporter reach the server? Then I look at the transport: the Python SDK, given a bare base URL, assumes gRPC on port 4317, and if that port is not published in Docker or the load balancer only serves 443, nothing arrives and often nothing errors. The fix is protocol="http/protobuf" or a full /v1/traces endpoint. Finally, did the server accept it? I read its status codes: 401 is credentials, 415 is JSON or wrong encoding sent to the protobuf endpoint, 503 is the ingest queue being full.

What a strong answer covers that a weak one misses: it names the register() banner as the first evidence, distinguishes PHOENIX_COLLECTOR_ENDPOINT from PHOENIX_ENDPOINT, and knows about the silent gRPC inference. A weak answer says "check the URL" and stops.

"The server starts returning 503 on trace ingest. What is happening and what do you do?"

The response text is Server is at capacity and cannot process more requests. Phoenix ingests through a bounded in-memory queue, PHOENIX_MAX_SPANS_QUEUE_SIZE, defaulting to 20,000, and it returns 503 when spans arrive faster than the database can absorb them. So the real question is why the database is slow to write. I check PostgreSQL write latency, disk usage and locks first; I look at whether a migration or a retention purge is running; and I check whether ingest volume jumped because someone turned on tracing for a chatty path. The mitigations are on both sides. On the client, use a batch span processor and gRPC or HTTP compression, sample, and suppress low-value spans such as health probes. On the server, scale the database, run more replicas behind a load balancer (each has its own queue), and only as a last resort raise the queue limit, because that just moves the backlog into memory and risks an out-of-memory kill.

A strong answer refuses to treat the limit as the fix and mentions back-pressure as a deliberate design. A weak answer says "increase the queue size".

"Quality dropped after last Tuesday's release. How do you find out why?"

If I set it up properly, the release is in span metadata, so I filter root spans with metadata['release'] == '<new>' and compare the annotation distribution against the previous release. That tells me whether the drop is real and which annotation moved: correctness, faithfulness, refusals, latency. Then I look at examples, because an aggregate points but does not explain. I filter annotations['correctness'].label == 'incorrect' for the new release, read a dozen, and look for a pattern such as a retriever returning irrelevant documents, a tool erroring, or a prompt change. If the prompt changed, the prompt version recorded on the span tells me which one. Finally I turn the failures into a dataset and run an experiment against the previous prompt version to confirm the cause before I roll back.

A strong answer depends on instrumentation decisions made earlier (release in metadata, prompt version recorded) and says so. A weak one starts reading random traces.

"Someone pasted a customer's phone number into a chat and it is in Phoenix. What now?"

First, contain: decide whether the record must be deleted, and do so through the supported deletion paths and retention controls, remembering that deletion is irreversible and that datasets and annotations may have copied the text. Second, find the exposure: traces, sessions, datasets built from them, and any exports. Third, fix the source so it does not recur: redact or mask at the application before it reaches the span, use opaque user.id values, and use suppress_tracing() around paths that handle regulated data. Fourth, set a retention policy so the window of exposure is bounded. I would also mention data residency: for employers in the Gulf or Egypt, self-hosting in a regional cloud location keeps trace content in-region, but only if the LLM provider calls made by evals and the Playground are also considered.

A strong answer talks about prevention and scope of copies, not only deletion.

"Your LLM judge says answers are ninety-five percent correct, but support says otherwise. What is wrong?"

An unvalidated judge. I would hand-label fifty to a hundred traces, run the judge on the same ones, and measure agreement. I expect one of a few causes: the template is too lenient or ambiguous; the judge sees only the final answer and not the retrieved context, so it cannot spot invented facts; the judge shares the same blind spots as the model being graded; or many results are NOT_PARSABLE and were dropped, biasing the sample toward easy cases. I would check the raw judge outputs, raise max_tokens if they are truncated, tighten the template so it must reply with exactly one label, and use a different or stronger model as the judge. Then I would keep the human-labelled set as a standing calibration check.

"A teammate's experiment shows prompt B beats prompt A by two points. Do you ship B?"

Not on that alone. I ask how many examples, how many repetitions, and what the spread between repeats is. If repeating the same prompt moves the score by two points, there is no signal. I look at per-example changes: did B fix some and break others that used to pass? I check that the dataset includes passing examples as well as failures, because a set of only failures cannot show a regression. I check that the evaluator was validated against human labels. If it holds up, I run B on a fresh slice it has not been tuned on, then ship it behind the production tag so rollback is moving a tag.

A strong answer talks about noise and overfitting to the dataset. A weak one compares two averages.

"We enabled auth and every service stopped sending traces."

Expected. With auth on, every request needs credentials, including OTLP ingest, so services without an API key get 401. The correct order is to create a system API key, put it in each service's environment as PHOENIX_API_KEY (from a secret store), confirm they work, and then enable auth in a maintenance window. To recover now, either provision the keys quickly or temporarily revert. I would also remember that keys cannot create keys since v19, so provisioning has to come from an admin session or PHOENIX_ADMIN_SECRET, and that a VIEWER-role key gets 403 on writes.

"A coding agent cannot connect to our Phoenix /mcp endpoint behind nginx."

The v19 release added an MCP server at /mcp and made Phoenix an OAuth2 authorization server, and both need the proxy to behave. I check that PHOENIX_ROOT_URL matches the external URL; that /.well-known/* at the host root is forwarded unmodified; that there is no proxy authentication in front of /mcp, /oauth2/register, /oauth2/token, /oauth2/revoke or /.well-known/*; that OPTIONS requests pass; that WWW-Authenticate, Mcp-Session-Id and Access-Control-* headers are not stripped; and that the event stream is not buffered (proxy_buffering off). A redirect to an SSO page or CORS errors point to the proxy gating or buffering an anonymous endpoint.

Trade-off questions

"SQLite or PostgreSQL?"

SQLite for a laptop, a demo, or a single-developer instance: zero setup and the data is one file under PHOENIX_WORKING_DIR. PostgreSQL the moment more than one person or more than one replica is involved, because it supports horizontal scaling behind a load balancer, concurrent writers, backups with point-in-time recovery, a read replica, and cloud IAM authentication. The cost of PostgreSQL is operating it, and it is the only stateful component, so its backup and restore process is the thing to test.

"Simple or batch span processor?"

Simple exports each span synchronously: immediate visibility, at the price of a network round trip in the request path and lost control if the collector is slow. Batch buffers and exports in the background: lower overhead and better compression, at the price of spans possibly being unflushed at shutdown and a small delay in visibility. Notebooks and short scripts: simple. Services: batch, with a flush on shutdown.

"LLM judge or code evaluator?"

Code when the criterion is objective: format, length, presence of an identifier, exact match. It is cheap, deterministic and explainable. A judge when the criterion is fuzzy: helpfulness, faithfulness to context, tone. The judge costs money and latency, varies between runs, and needs validating. A healthy suite is mostly code checks plus a few validated judges, and a human-labelled calibration set behind them.

"Evaluate everything or sample?"

Sample for cost unless the criterion is cheap. Evaluate all errors and all traces users flagged, and a random slice of the rest. Increase the rate temporarily after a release. The trade-off is statistical power against spend; a few hundred judged traces per day is usually enough to detect a large regression.

"Keep prompts in Phoenix or in git?"

In Phoenix, prompt engineers iterate without a deploy, every version is tracked and tied to experiments, and rollback is moving a tag. In git, behaviour changes are reviewed and reproducible with the code. A middle path: manage prompts in Phoenix, fetch a pinned version or tag in production with caching, record the version on each span, and promote from CI after an experiment passes. Whichever you choose, the failure to avoid is an application whose behaviour depends on an unversioned external state you cannot reconstruct.

"One Phoenix for every team, or one per team?"

One Phoenix instance is one tenant: all projects are visible to every user, subject to role, and per-project permissions were not available at the time of writing. If teams must not see each other's data, run separate instances or separate PostgreSQL schemas (PHOENIX_SQL_DATABASE_SCHEMA). One shared instance is cheaper to run and easier to compare across projects. The deciding factor is data sensitivity, not convenience.

"Self-host Phoenix, use Phoenix Cloud, or use Arize AX?"

Self-hosting gives control of data location and cost, and you operate it. Phoenix Cloud is the same software hosted for you, at app.phoenix.arize.com. AX is the commercial enterprise product with multi-tenancy and a different storage engine, aimed at very high volume. The decision turns on residency rules, team operations capacity and scale.

What the strong answers have in common

Across these questions, the same habits separate answers that get offers from answers that do not.

• They bisect instead of guess. A strong candidate names the stages of the pipeline and the one cheap test for each.
• They know the defaults and where they surprise you. gRPC inferred from a base URL, auth off on the bare server but on in the Helm chart, retention forever, latest being a moving target.
• They separate signal from noise. Repetitions, validated judges, per-example review.
• They name the current API. phoenix.client.Client with base_url, phoenix.evals.LLM and ClassificationEvaluator, phoenix serve --port, annotations rather than evaluations. Quoting removed APIs is a tell that someone learned from an old tutorial.
• They think about data. What is stored, for how long, who can see it, where it lives.
• They close the loop. Trace, annotate, dataset, experiment, prompt version, next trace.

Common interview questions

What is the difference between a trace, a span, a session and an annotation? A trace is one request as a tree of spans sharing a trace id. A span is one unit of work with attributes. A session groups traces that share a session.id, usually a conversation. An annotation is a label, score or explanation attached to a span, trace, document or session, recorded by a human, an LLM or code.

Why are spans written in OpenInference rather than plain OTel? OpenInference adds semantic conventions for LLM work (messages, token counts, span kinds such as LLM, RETRIEVER and TOOL), so the UI can render and filter them. It stays wire-compatible with OTel, so any OTel collector can carry the data.

How do you keep environments apart? One project per environment and application, set by project_name or PHOENIX_PROJECT at startup, or separate instances if access must differ.

What changed in v14 that breaks old code? The server CLI became subcommand-first (phoenix serve --port 6006); the legacy px.Client was removed in favour of phoenix.client.Client(base_url=...); psycopg was dropped so asyncpg is the only PostgreSQL driver; the /v1/evaluations endpoints were removed in favour of the /v1/*_annotations routes; and experiments moved to phoenix.client.experiments.

What changed in v19 that affects automation? API keys can no longer create API keys; Phoenix became an OAuth2 authorization server; and an MCP server was mounted at /mcp. Each has reverse-proxy implications.

How do you stop a retention problem before it starts? Set PHOENIX_DEFAULT_RETENTION_POLICY_DAYS or a per-project policy, configure storage capacity thresholds so admins are warned and inserts are eventually blocked, and keep curated datasets as the long-term record.

How do you evaluate retrieval separately from generation? Use document annotations and RetrievalRelevanceEvaluator on the retriever span to judge whether each retrieved chunk is relevant, and a faithfulness evaluator on the generation to check the answer against the context. Separating them tells you which stage to fix.

How would you run evals in CI? Keep a small curated smoke split, run the experiment on each pull request with code-based evaluators as hard checks, run judge evaluators as advisory, and run the full dataset nightly. Fail only on metrics you have shown to be stable.

What would you monitor on the Phoenix server itself? Readiness on /readyz, 503 rate on trace ingest, database_disk_usage_ratio and database_insertions_blocked from Prometheus, memory, and exceptions. Prometheus must be enabled with PHOENIX_ENABLE_PROMETHEUS=true.

What is PXI and should it be on? It is the in-app assistant shipped in v17, with persisted sessions since v20. It uses a model provider you configure, can run code in sandboxes, and sends prompts to that provider. Enable it by policy; PHOENIX_DISABLE_AGENT_ASSISTANT=true turns it off.

Questions to ask back

Good candidates ask questions that show operational thinking. You can ask which database the team uses and how its backups are tested, how traces are scoped to projects, whether evaluators have been validated against human labels, what the retention policy is and who approved it, how prompts reach production, and what happens to traces that contain personal data. Interviewers read these questions as evidence that you have run the tool rather than read about it.

Sixty-second self-test

Answer aloud without looking back.

1. Name the five hops a span takes before it is stored, and one test for each.
2. Given only http://phoenix:6006, which transport does the Python SDK infer, and what do you pass to change it?
3. What does a 503 from /v1/traces mean, and what are two fixes that are not "raise the limit"?
4. Give the current spelling of: starting the server on port 7000, creating the client, and running an experiment.
5. Why should HallucinationEvaluator not be in new code?
6. What three pieces of context would you attach to every production span?
7. What does parent_id is None select?
8. Which database and driver do you run in production, and what does PHOENIX_MIGRATE_INDEX_CONCURRENTLY protect against?
9. Why does enabling auth on a live server stop tracing, and how do you avoid it?
10. What is the difference between a system key and a user key, and which do services use?

If you stumbled on more than two, reread the matching section of the detailed guide and run its Try it again. For related tools, compare with Langfuse and LangSmith, and for retrieval metrics see RAGAS.

Part two of three. At this level Phoenix is running and traces are arriving, so the problems change character. They are no longer "why is the UI empty" but "why did ingest start returning 503", "why does the judge disagree with our reviewers", and "why did everything stop when we turned auth on". Start with the error table, then the practices in order of leverage, the practice cards, and the operational recipes at the end.

Common errors at this level

Cumulative: the beginner errors still apply, and these are the ones that appear once things mostly work.

The practices that pay off most

Ranked by leverage, the same list in prose. The first four cost almost nothing and prevent the most common failures.

1. Explicit endpoint, protocol and batch. This removes the largest class of "nothing arrives" and "tracing slows my API" problems.
2. Attach context once, at the request boundary. Session, user, release and tags in a single using_attributes block make every later query possible.
3. Retention before disk pressure. A default of keep-forever is a decision you did not make. Set one.
4. Keys and roles before exposure. Turn auth on before anyone else can reach the server, and give applications system keys.
5. Validated judges over many judges. Two trusted evaluators beat ten unexamined ones.
6. A regression dataset that grows. Each incident adds examples; over a quarter it becomes your most valuable test asset.
7. Prompts as versions with tags. Rollback becomes a tag move.
8. Monitor the monitor. Prometheus on, alerts on blocked inserts and disk ratio.

Practice cards

1. Reproduce the silent gRPC failureRun Phoenix in Docker publishing only 6006, register with a base URL and no protocol, and confirm traces vanish. Add protocol="http/protobuf" and confirm they return. You will now recognise it in seconds.
2. Feel the processor differenceTrace a loop of twenty model calls with batch=False and again with batch=True, timing the loop. Then exit immediately after the last call and see what a missing flush looks like.
3. Shape one real trace treeDecorate the retrieval, tool and agent entry point of one request path, run it, and eliminate every UNKNOWN span kind you can.
4. Build a queryable sessionWrap a handler in using_attributes, send two conversations, and confirm two sessions and a working metadata['release'] filter.
5. Pull failures into pandasUse SpanQuery().where("parent_id is None") with the client to get yesterday's root spans, and print the five slowest and the five with errors.
6. Calibrate a judgeHand-label twenty traces, run a ClassificationEvaluator, compute agreement, edit the template, and rerun until it improves.
7. Harvest a datasetCreate a dataset from incorrect spans plus some passes, add a reason to each example's metadata, then add one example and list the versions.
8. Measure noiseRun the same prompt as an experiment twice with three repetitions each. The difference between those two runs is your noise floor.
9. Promote a prompt by tagCreate a prompt, create a second version, move the production tag, and confirm the application fetches the new one and records its version on the span.
10. Survive enabling authOn a scratch instance, enable auth with a generated secret, watch an unkeyed service get 401, provision a system key, and watch tracing resume.
11. Rehearse a migrationTake a database backup, upgrade a staging copy one minor version with a separate phoenix db migrate step, time it, and write down the rollback.
12. Break it three waysWrong port, missing key, JSON exporter. Record the exact symptom of each in a runbook table.

Operational recipes

Generate safe secrets

The server refuses weak secrets and gives a precise message. Generate two distinct ones, at least 32 characters each, with a digit and a lowercase letter:

export PHOENIX_SECRET="$(openssl rand -hex 32)"
export PHOENIX_ADMIN_SECRET="$(openssl rand -hex 32)"

Store them in your secret manager rather than in a shell profile. Losing PHOENIX_SECRET later means losing the ability to decrypt stored provider credentials, because the encryption key is derived from it. Without it stored secrets fall back to a deterministic key that gives no real confidentiality.

Check a server in ten seconds

curl -s  http://localhost:6006/healthz                 # liveness
curl -s  http://localhost:6006/readyz                  # readiness, includes the database
curl -s  http://localhost:6006/arize_phoenix_version   # which server is this

Read the three together: liveness up and readiness down is a database problem, not a crashed server.

Verify a pipeline end to end

px setup --no-input --endpoint http://localhost:6006 --project smoke-test
px trace list --limit 5 --no-progress --format json

px setup exits with code 6 when no trace arrived, 5 when the server was unreachable and 4 for auth, which makes it a usable smoke test in a pipeline.

Pin and upgrade deliberately

Record the server tag in one place, such as a values.yaml or Compose file, and upgrade by editing that one line in a pull request that also links the release notes. Check MIGRATION.md for every major version crossed, take a backup, run phoenix db migrate as a separate step, and on PostgreSQL with rolling deploys set PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true. Upgrade the server before the px CLI when crossing 20.0.0.

Give an application a least-privilege identity

Create a system API key as an admin, store it in the secret manager, and inject it as PHOENIX_API_KEY. Do not use an admin user's key for a service, since user keys vanish with the user. Rotate on a schedule by creating a new key, rolling it out, and deleting the old one. Keep the human-facing px access on OAuth (px auth login) so no one pastes keys into terminals.

Cap storage growth

export PHOENIX_DEFAULT_RETENTION_POLICY_DAYS=30
export PHOENIX_DATABASE_ALLOCATED_STORAGE_CAPACITY_GIBIBYTES=200
export PHOENIX_DATABASE_USAGE_EMAIL_WARNING_THRESHOLD_PERCENTAGE=70
export PHOENIX_DATABASE_USAGE_INSERTION_BLOCKING_THRESHOLD_PERCENTAGE=95

Email warnings need SMTP configured. Insertion blocking is a last line of defence that stops new traces to protect the database, so alert well before it.

Lock down a shared instance

Decide these explicitly rather than inheriting defaults: auth on; PHOENIX_CSRF_TRUSTED_ORIGINS set; PHOENIX_USE_SECURE_COOKIES=true behind HTTPS; PHOENIX_ENABLE_MCP_CODE_MODE=false if you do not want model-written code executing; PHOENIX_DISABLE_AGENT_ASSISTANT=true until a policy exists; and network policies that let the pod reach only its database, DNS and approved provider domains.

Keep evals honest in CI

Run a small split on each pull request. Use code evaluators as hard gates and judges as advisory. Name each experiment with the commit SHA. Store the judge model and template version alongside results, so a score change can be traced to a prompt change, a model change or a judge change.

A weekly review ritual

Fifteen minutes on a fixed day: open the sessions with errors, read five, annotate by hand, add the worst to the regression dataset, and check that the judge still agrees with your labels. The ritual matters more than the tooling, because it keeps humans in the loop that evaluators only approximate.

Reading the errors: which layer is at fault

Most of the table above collapses into a short diagnostic habit. When something is wrong, decide first which layer it lives in, because the layers have different owners and different tools.

If the symptom is "nothing arrives", it is on the client or the network, and the evidence is the register() banner and the exporter's own log lines. If the symptom is an HTTP status from the server, read the number: 401 and 403 are identity and role, 415 is encoding, 503 is capacity, and 404 on an old route is a removed API. If the symptom is "the data is there but wrong", it is an instrumentation problem: the wrong span kind, a lost context, a missing attribute, a stable-session mistake. If the symptom is "the numbers are there but I do not trust them", it is an evaluation problem: an unvalidated judge, too few repetitions, a dataset with no passing examples. And if the symptom appears only after a deploy or an upgrade, it is an operations problem: an unpinned tag, a chart lagging the server, a proxy rule from v19, or a migration that was interrupted.

Write that sorting step into your runbook as four lines. It saves the most time on the nights when the person on call did not build the system.

Habits worth keeping once it works

Three habits compound over months. The first is to treat every incident as a dataset contribution. When a user complains and you fix the cause, add the failing input to the regression dataset before you close the ticket, with a line of metadata saying what went wrong. The second is to keep a short list of queries you actually run, the filter expressions and the px commands, in the repository next to the code, so the knowledge outlives the person who discovered it. The third is to re-validate on a schedule. Judges drift when the model behind them changes, prompts drift when someone edits them, and datasets go stale when the product changes. A quarterly hour spent comparing the judge with fresh human labels is cheap insurance against dashboards that have quietly stopped meaning anything.

Finally, be wary of tips that reach you from old tutorials. Phoenix has changed its CLI order, client, evaluation library and endpoints within the last year, and a great deal of search-engine advice describes the previous generation. When a snippet does not run, check its age before you check your setup.

A short checklist

• Endpoint and protocol are explicit, and batch=True in services.
• Every production span carries session, user and release context.
• Span kinds are set for retrieval, tools and the agent entry point.
• PostgreSQL 14 or newer, backed up, with a tested restore.
• Server image pinned, including in Helm via image.tag.
• Auth on, secrets in a secret store, applications use system keys.
• Reverse proxy follows the v19 rules, and PHOENIX_ROOT_URL is correct.
• Retention policy set and storage thresholds configured.
• Prometheus scraped, with alerts on blocked inserts, disk ratio, 503s and readiness.
• Judges validated against human labels and sampled, not run on everything.
• A regression dataset that includes passing examples and reasons.
• Experiments use repetitions and versioned names.
• Prompts versioned with a production tag, and the version recorded on spans.
• No code imports removed APIs (px.Client, llm_classify, phoenix.experiments).

Next in the series, the senior guide covers architecture, failure modes and running Phoenix as a platform for several teams. For neighbouring tools see Langfuse and LangSmith, and for containerising the server see Docker.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, so that each now has a capacity, security or ownership dimension, and it adds the work you take on when Phoenix stops being your tool and becomes a platform other teams depend on.

Up to this point the questions were about getting traces to appear and making sense of them. From here they change. What happens to the database when forty services start exporting at once? Who can read the prompts that customers typed into someone else's chatbot? What does an upgrade from 19 to 20 do to a table with four hundred million spans in it? And when a team says "Phoenix lost our traces", how do you prove where they went? An owner answers those questions before the incident, not during it.

Where this picks up

The version this guide is written against is Arize Phoenix 20.16.0 (released 2026-09-23). The server ships a minor release every few days, so whatever you install will be newer. Everything below is stated as a mechanism and a configuration key, not as a promise about a particular build, and the release notes remain the final word.

I am starting with the architecture, because every operational decision in this track depends on knowing which parts of Phoenix hold state and which do not.

Architecture and failure modes

A Phoenix deployment is deliberately small. There is one server process and one SQL database. The process does everything: it receives traces over OTLP (HTTP at /v1/traces on port 6006, gRPC on port 4317), serves the web UI, answers the REST API under /v1 and the GraphQL API at /graphql, hosts an in-process MCP server at /mcp, and runs background jobs such as the retention sweep and the hourly cleanup of expired assistant chats. The database is either SQLite (a single file under PHOENIX_WORKING_DIR, which defaults to ~/.phoenix/) or PostgreSQL 14 or newer.

The design has one important consequence: the server is stateless and the database is the platform. You can kill and replace a Phoenix pod at any time and lose nothing that was already committed. You cannot do the same to the database. Backups, point-in-time recovery, storage growth, vacuum behaviour and failover are the things that decide whether your platform survives, and they are all PostgreSQL concerns rather than Phoenix concerns. A senior owner spends most of their time on the database side of the diagram.

Where state actually lives

It helps to list what is in that database, because each kind of data has a different value and a different growth rate.

• Spans and traces. The bulk by volume. They grow with request rate, with the number of spans per request (an agent with ten tool calls produces many more than a single chat completion) and with payload size, because input.value, output.value and the message attributes carry full prompts and completions.
• Annotations. Labels, scores and explanations attached to spans, traces, documents and sessions. Small per row, but they are the output of expensive human and LLM-judge work.
• Datasets, examples and experiments. Versioned, and growing only when people curate them. Low volume, high value, and the hardest data to regenerate.
• Prompts and their versions, with tags such as production. Tiny, but a wrong edit here changes application behaviour.
• Users, API keys, secrets and settings. Tiny, and the most sensitive.
• Agent sessions (v20 and later). Conversations with the in-app assistant PXI, stored server-side in agent_sessions, agent_session_messages and agent_session_snapshots.

The practical rule that follows is that traces are cheap to lose and everything else is not. A sensible platform treats them differently: aggressive retention on spans, careful backup of datasets, prompts, annotation configs and users. If the database ever has to be restored, you will care far more about the second group than the first.

The failure modes, in the order you will meet them

Every component has a characteristic way of failing. Knowing them in advance turns an incident into a lookup.

The database is slow or full. Ingest and UI reads compete for the same connection pool and the same disk. The first symptoms are slow UI pages (especially the sessions and spans tables with broad time ranges), then a growing in-memory queue of spans waiting to be written. When storage is full, PostgreSQL stops accepting writes and Phoenix cannot persist anything. The documented guardrails (PHOENIX_DATABASE_ALLOCATED_STORAGE_CAPACITY_GIBIBYTES with the warning and insertion-blocking thresholds) let Phoenix itself email admins and then refuse inserts before the disk is actually exhausted. That is better than a crash, but only if you set them.

The span queue fills. Incoming spans are held in a queue before they are written. Its size is PHOENIX_MAX_SPANS_QUEUE_SIZE, default 20,000. Past that, the ingest endpoint returns HTTP 503 with the message Server is at capacity and cannot process more requests. This is the platform's back-pressure signal, and it is the most important thing to understand about ingest: the server would rather reject than fall over.

A client-side exporter drops data silently. OpenTelemetry exporters retry a few times and then give up. A 503 that lasts longer than the exporter's retry budget means those spans are gone, and the application does not crash or slow down. Nothing on the server records spans it never received, so you only find out by comparing counts. This is why sections below insist on measuring both ends.

The process runs out of memory. Memory grows with ingest volume, attribute cardinality and the amount of data a UI query or an evaluation job pulls into the process. A container with a hard memory limit that is set too low will be OOM-killed under load, and the kill looks like a random restart. Because the process is stateless, Kubernetes brings it back, but any spans in the in-memory queue at that moment are lost.

A migration runs at the wrong moment. Schema migrations run at startup through Alembic, and a new instance blocks until they finish. On a large spans table, an index build can hold locks for minutes. If a readiness probe has a short deadline, the orchestrator may kill the pod mid-migration and start over. This is the single most common self-inflicted outage, and the upgrade section gives the defence.

An upstream dependency you forgot about. Phoenix makes outbound calls to LLM providers for the Playground, for evaluations that run inside the server, and for PXI. A provider outage or a revoked key breaks those features, not ingest. It is worth remembering when a team reports "Phoenix is down" and what they mean is "the Playground returns an error".

Reading the probes

Phoenix exposes /healthz for liveness and /readyz for readiness, and the readiness check includes database connectivity. Use them for what they say. Liveness answers "is this process wedged", so a failure restarts the container. Readiness answers "should traffic be sent here", so a failure removes the pod from the load balancer without killing it. The mistake to avoid is setting a liveness probe so aggressively that it fires during a long startup migration. Give the startup phase its own generous startupProbe, and keep the liveness probe gentle.

startupProbe:
  httpGet: { path: /healthz, port: 6006 }
  periodSeconds: 10
  failureThreshold: 90        # up to 15 minutes for a large migration
livenessProbe:
  httpGet: { path: /healthz, port: 6006 }
  periodSeconds: 20
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 6006 }
  periodSeconds: 10
  failureThreshold: 3

The numbers are an example and not a recommendation from the docs: size the startup budget by timing a real migration on a copy of your data.

Scaling: which component breaks first

"Does Phoenix scale?" is the wrong question, because it has no numeric answer. Arize publishes no throughput ceiling for Phoenix, and it states openly that very high volume and analytical workloads are where it steers people to the commercial Arize AX product. A better question is "which part of this system reaches its limit first, and what do I do when it does?"

In practice the order is predictable.

1. The client, before anything else. A default register() call uses a synchronous span processor, which exports each span while the request is waiting. Under load this adds latency to your application. register(batch=True) switches to a batch processor, which is what the startup banner tells you to do in production. Fix this first, because it costs nothing.
2. The database write path and storage. Each span is a row with large text attributes. Write throughput is bounded by PostgreSQL, and storage grows linearly with retention.
3. The ingest queue. When writes cannot keep up, the queue fills and ingest returns 503. This is the honest signal that you have outrun the database.
4. Server memory and CPU. Payload parsing, protobuf decoding and query results all happen in the Python process.
5. The UI and query path. Broad time ranges over large tables get slow well before ingest fails. This is often what teams complain about first, and it is fixed by retention, by narrower default time ranges and by the read replica.

Horizontal scale

You scale ingest and UI capacity by running more replicas behind a load balancer, all pointed at one PostgreSQL database. There is no sharding and no leader election to configure. That makes scaling the application tier trivial and makes the database the ceiling. Two features help you push that ceiling:

• PHOENIX_SQL_DATABASE_READ_REPLICA_URL (added in 14.0.0) sends read queries to a replica so that heavy UI and API reads do not compete with ingest on the primary.
• PHOENIX_SQL_DATABASE_SCHEMA lets several Phoenix instances share one PostgreSQL server while keeping their data in separate schemas. It is an isolation tool as much as a scaling one.

replicaCount: 3
image:
  tag: "version-20.16.0-nonroot"      # the chart's default appVersion lags the server
postgresql:
  enabled: false                       # external database
persistence:
  enabled: false
database:
  url: ""                              # injected from a Secret, not committed
server:
  enablePrometheus: true

The keys above come from the published chart's values file (replicaCount defaults to 1; the persistence strategies are mutually exclusive as described in the fact sheet, so exactly one of bundled PostgreSQL, a SQLite PVC, or an external database.url is chosen). Check each key against the chart version you install with helm show values, because the chart and the server are versioned separately and the chart lags.

Shaping the load before it arrives

The cheapest capacity you will ever buy is the span you never send. Four levers, from most to least effective:

Sample at the source. Standard OpenTelemetry samplers work because Phoenix is just an OTLP endpoint. A parent-based ratio sampler keeps whole traces together, which matters: a half-sampled trace with missing children is worse than no trace. Keep 100 percent of errors and of low-volume high-value routes, and sample the noisy ones.

Put an OTel Collector in front. A collector with a batch processor absorbs bursts, retries on your behalf and lets you apply tail sampling (decide after seeing the whole trace, for example "keep anything with an error or a latency over ten seconds"). It also gives you one place to fan traces out to Phoenix and to another backend such as Tempo or Jaeger. The price is another component to run and another place for data to be lost.

Trim the payloads. Span attributes that hold entire documents or base64 images dominate storage. OpenInference has configuration to hide inputs, outputs or images at instrumentation time. Decide per application whether the full prompt is worth its storage cost, and whether it should be stored at all (see the security section).

Prefer gRPC where you can. The Python SDK infers gRPC on port 4317 when given a bare base URL, and gRPC compresses better. Behind an HTTPS load balancer that exposes only 443, pass protocol="http/protobuf" explicitly.

from phoenix.otel import register

tracer_provider = register(
    project_name="checkout-assistant",
    endpoint="https://phoenix.internal.example.com/v1/traces",
    protocol="http/protobuf",
    batch=True,
    auto_instrument=True,
)

Capacity planning you can actually do

You cannot look up a throughput number, so measure your own. Take a representative service, run it at a known request rate for an hour, and read three figures from your database: rows added, bytes added, and the write latency on the primary. Multiply by your real request rate and your retention window. Then add the multiplier nobody estimates: evaluation and annotation data, which can be as large as the traces when an LLM judge scores every span.

The ingest path and its back-pressure

Because ingest failures are silent on the client, it is worth understanding the path in detail.

An instrumented application creates spans in memory. The span processor hands them to an exporter. With batch=True spans accumulate in a bounded buffer and are sent in batches on a timer or when the batch fills. The exporter serialises them as OTLP protobuf and sends them either over gRPC to port 4317 or as an HTTP POST to /v1/traces. The Phoenix server decodes the request, validates it, and places the spans on the internal queue. A background task drains the queue into the database.

Each hop has a different failure signature.

Two details catch experienced people. First, gRPC needs the authorization header in lowercase, because gRPC metadata keys are lowercase. Second, once a 503 starts, raising PHOENIX_MAX_SPANS_QUEUE_SIZE only buys time and costs memory. It is a buffer, not a capacity. The real fix is a faster database, more replicas draining concurrently, or less data.

#!/usr/bin/env bash
# Probe the ingest endpoint the way an exporter would, and show the status code.
set -euo pipefail
PHOENIX=${PHOENIX_ENDPOINT:-http://localhost:6006}
curl -s -o /dev/null -w "readyz=%{http_code}\n" "$PHOENIX/readyz"
curl -s -o /dev/null -w "traces (empty body, expect 4xx not 5xx)=%{http_code}\n" \
  -X POST "$PHOENIX/v1/traces" \
  -H "content-type: application/x-protobuf" \
  -H "authorization: Bearer ${PHOENIX_API_KEY:-none}"

An empty protobuf body is a deliberately harmless probe: a 4xx means the path, protocol and auth layers are doing their jobs, and a 5xx or a timeout means something upstream is broken. Do not use it as a substitute for a real end-to-end trace count.

The security model

Phoenix collects the most sensitive data in an LLM system: the literal prompts and completions, including whatever users typed and whatever your retrieval layer pulled from internal documents. An owner has to hold that in mind. The rest of this section is the trust model, from the outside in.

Authentication is off by default

The bare server runs with PHOENIX_ENABLE_AUTH unset, which means no authentication at all. Every UI page, every REST and GraphQL call, every ingest request and the /mcp endpoint are open to anyone who can reach the port. The Helm chart flips this default to on, which is the right default, but it means the same product behaves very differently depending on how you installed it. Never expose an unauthenticated Phoenix beyond a network you fully control.

Turning authentication on has three prerequisites and one big side effect.

export PHOENIX_ENABLE_AUTH=true
export PHOENIX_SECRET="$(openssl rand -hex 32)"         # signs JWTs, derives the encryption key
export PHOENIX_ADMIN_SECRET="$(openssl rand -hex 32)"   # must differ from PHOENIX_SECRET
phoenix serve

PHOENIX_SECRET must be at least 32 characters with at least one digit and one lowercase letter. PHOENIX_ADMIN_SECRET follows the same rules, must differ from the main secret, and is only valid when PHOENIX_SECRET is set. The side effect is the one that catches teams: the moment auth is enabled on a live deployment, every unauthenticated exporter starts getting 401s and trace collection stops. Plan it as a change with a window, issue keys first, roll the keys into applications, and only then flip the switch, or accept a gap.

What the secret does

PHOENIX_SECRET is more than a login setting. Phoenix uses it to sign its JSON Web Tokens and to derive the key that encrypts stored secrets such as LLM provider credentials (Fernet encryption, per the docs). Passwords are hashed with PBKDF2-HMAC-SHA256. Two operational rules follow. First, rotating PHOENIX_SECRET invalidates sessions and affects the decryption of stored secrets, so treat rotation as a planned migration, not a routine toggle, and test it on a copy. Second, if you run without PHOENIX_SECRET, stored secrets fall back to a deterministic key, which gives no real confidentiality. Do not store provider keys in Phoenix on an instance without a real secret.

The initial login is admin@localhost with the password admin, unless you set PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD. That value is applied only when the default admin is first created. Change the password on first login, or better, set the initial password through a secret and create real admin accounts with PHOENIX_ADMINS.

Roles, and what a role cannot do

There are three roles. ADMIN manages users, system API keys, secrets, sandbox configuration and retention-policy assignment. MEMBER can write: create datasets, run experiments, annotate, edit prompts. VIEWER is read-only, although it can still manage its own user keys. A 403 on a POST, PUT or DELETE under /v1 almost always means the key belongs to a viewer.

The limit an owner must state plainly to other teams is this: roles are global across the instance. They are not per project. A member can write to every project, and anyone who can log in can read every project's prompts and completions, subject only to viewer versus member versus admin. Phoenix has no per-project permissions today; the project's maintainers have described resource tags as planned for later in 2026 (see the open issue #10504). Until that ships, a project is a convenience for grouping data, not an access-control boundary.

API keys: system versus user

A user key acts as its user and is deleted when the user is. A system key belongs to the instance, is created by an administrator and survives that administrator leaving. Applications that export traces should use system keys, because a service that stops working when an engineer leaves the company is a service you built on the wrong identity. Interactive use (the CLI, MCP clients, a person's notebook) should use OAuth through px auth login, or a personal user key.

Since version 19 an API key can no longer create API keys, through either GraphQL (createUserApiKey, createSystemApiKey) or REST. User keys come from a human session; system keys come from an admin session or from PHOENIX_ADMIN_SECRET calling POST /v1/system/api_keys. This closes a privilege-escalation path (a leaked low-privilege key could previously mint more keys) and it means key issuance automation needs the admin secret, which you must therefore protect like a root credential.

# Run from a trusted automation host. PHOENIX_ADMIN_SECRET acts as a bearer token for the system user.
curl -sS -X POST "$PHOENIX_ENDPOINT/v1/system/api_keys" \
  -H "authorization: Bearer $PHOENIX_ADMIN_SECRET" \
  -H "content-type: application/json" \
  -d '{"key": {"name": "checkout-assistant-exporter"}}'

The request body shape above is an illustration. Confirm the exact schema against the OpenAPI document your server serves before scripting it, since the field names are the server's to define.

Identity: SSO, LDAP and role mapping

Password login is for bootstrap. For anything real, put Phoenix behind your identity provider. Phoenix supports OAuth2 and OIDC with Google, AWS Cognito, Microsoft Entra ID, Okta, Keycloak, Auth0 and any provider with OIDC discovery, and it supports LDAP. Each OAuth2 provider is configured with a family of variables named PHOENIX_OAUTH2_<IDP>_..., at minimum _CLIENT_ID, _CLIENT_SECRET and _OIDC_CONFIG_URL.

Three details decide whether your SSO rollout is safe.

• Role mapping. Without mapping, every new OAuth user becomes a VIEWER. Use _ROLE_ATTRIBUTE_PATH and _ROLE_MAPPING (for example platform-admins:ADMIN,ml-engineers:MEMBER) so that role follows group membership, and leave _ROLE_RESYNC at its default of true so that removing someone from a group actually removes the access at next login.
• Group gating. _ALLOWED_GROUPS refuses sign-in for people outside the listed groups. This is the closest thing to "only this organisation's data scientists" that the platform offers.
• Where claims must live. Phoenix reads the group and role claims from the ID token or from UserInfo, not from access-token claims. The classic symptom of getting this wrong is that everybody lands as a viewer, or sign-in is denied, even though the identity provider's own test shows the right groups. Configure the provider to put the claim in the ID token and request the scope that carries it (for example groups).

When SSO is live, set PHOENIX_DISABLE_BASIC_AUTH=true so the password form is not a side door, but only after you have confirmed that at least one SSO admin exists. Locking yourself out of your own platform is an entirely avoidable outage.

Transport, cookies and CSRF

Terminate TLS at the load balancer or ingress, use Phoenix's native TLS (PHOENIX_TLS_ENABLED, PHOENIX_TLS_CERT_FILE, PHOENIX_TLS_KEY_FILE, with per-protocol switches for HTTP and gRPC), or use a service mesh. For client certificate authentication add PHOENIX_TLS_CA_FILE with PHOENIX_TLS_VERIFY_CLIENT. To the database, use sslmode=verify-full in the connection URL, so that a network attacker cannot present a false database.

Behind HTTPS set PHOENIX_USE_SECURE_COOKIES=true. Then set PHOENIX_CSRF_TRUSTED_ORIGINS, because CSRF protection is off unless that variable is set, and set PHOENIX_ALLOWED_ORIGINS only if you genuinely need cross-origin browser access, because CORS is off when it is unset and that is the safer state. Brute-force protection is on by default (five failed attempts lock an account for five minutes), and PHOENIX_ENABLE_STRONG_PASSWORD_POLICY=true raises the password rule from the default four-character minimum to twelve characters with mixed classes. Both matter only while password login is enabled.

Network exposure, SSRF and agent surfaces

Phoenix is not only a place data flows into. It also makes outbound requests and, since recent majors, it can run code. Both change the threat model.

Outbound calls and server-side request forgery

The Playground, server-side evaluations and PXI all call LLM providers from inside the Phoenix process, using endpoints and keys that users can configure. Anyone with write access who can point a custom provider at an arbitrary URL is asking your server to make a request on their behalf. That is the definition of server-side request forgery (SSRF), and in a cloud the prize is the instance metadata service at 169.254.169.254, which hands out cloud credentials.

The official guidance is to constrain the network, not to trust the application. On Kubernetes that means a NetworkPolicy, enforced by a CNI such as Cilium or Calico, that allows egress only to the database, to DNS and to the approved provider domains, and denies the link-local range 169.254.0.0/16 (plus 169.254.169.252/32 on GKE). If your CNI cannot filter by domain, route egress through an HTTP(S)_PROXY with a domain allowlist instead. PHOENIX_ALLOWED_PROVIDERS narrows which providers the Playground offers, but it is a UI filter, not enforcement, and an unknown provider name fails startup. Use it for tidiness and never as a control.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: phoenix-egress
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: phoenix
  egress:
    - toEndpoints:
        - matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s-app": kube-dns }
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
    - toFQDNs:
        - matchName: api.openai.com
        - matchName: api.anthropic.com
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]
    - toCIDR: ["10.20.0.0/24"]        # the PostgreSQL subnet

The label selectors and CIDR here are placeholders for your cluster. The shape is what matters: DNS, named providers, the database, and nothing else. Test it by exec-ing into the pod and trying to reach the metadata address. A control you have never seen refuse anything is still untested.

The remote MCP endpoint and the OAuth2 authorization server

From version 19, Phoenix mounts an in-process MCP server at /mcp and, when auth is enabled, acts as its own OAuth2 authorization server with authorization code plus PKCE and dynamic client registration. That is how px auth login and MCP clients such as Claude Code and Cursor sign in without copying a key. Both are on by default (PHOENIX_ENABLE_MCP_SERVER, PHOENIX_ENABLE_OAUTH2_AUTHORIZATION_SERVER).

As an owner you decide whether they should be. The questions are concrete:

• Who can register a client? PHOENIX_OAUTH2_DYNAMIC_CLIENT_REGISTRATION takes enabled (default), local_only or disabled, and PHOENIX_OAUTH2_DCR_RATE_LIMIT_PER_HOUR defaults to 10. A locked-down platform may prefer local_only or disabled.
• Where can a grant redirect? PHOENIX_OAUTH2_ALLOWED_REDIRECT_HOSTS narrows it, and PHOENIX_OAUTH2_CONSENT_ORIGIN_CHECK should stay strict.
• How long does a grant live? PHOENIX_OAUTH2_GRANT_EXPIRY_DAYS defaults to 90.
• What can a model do through MCP? By default the endpoint presents code-mode tools, including an execute tool that runs model-written Python in a pydantic-monty sandbox. Set PHOENIX_ENABLE_MCP_CODE_MODE=false for the plain, group-gated tool list if you do not want that.

The sharpest edge is an old one restated: with auth disabled, /mcp needs no credentials, so an unauthenticated Phoenix exposes a programmatic query interface to all trace data. The other agent surface is PXI, the built-in assistant. You can disable it (PHOENIX_DISABLE_AGENT_ASSISTANT=true) or restrict what it may do with PHOENIX_AGENTS_DISABLE_BASH, PHOENIX_AGENTS_DISABLE_WEB_ACCESS and PHOENIX_AGENTS_DISABLE_GITHUB, and code-execution sandboxes are admin-configured with PHOENIX_ALLOWED_SANDBOX_PROVIDERS (a comma-separated list of WASM, E2B, DAYTONA, VERCEL, DENO, MODAL or NONE).

Reverse-proxy requirements

Since v19 a proxy in front of Phoenix must respect a specific list, and most "the MCP client cannot connect" tickets are a violation of it:

• Set PHOENIX_ROOT_URL to the full external URL, and PHOENIX_HOST_ROOT_PATH as well if Phoenix lives under a subpath.
• Forward host-root /.well-known/* unmodified.
• Do not put proxy authentication (an SSO gateway, basic auth) in front of /mcp, /oauth2/register, /oauth2/token, /oauth2/revoke or /.well-known/*. Those endpoints do their own auth, and a gateway that redirects them to a login page breaks the flow.
• Let OPTIONS requests through for CORS preflight.
• Do not buffer or time out the /mcp event stream; in nginx, proxy_buffering off.
• Pass WWW-Authenticate, Mcp-Session-Id and Access-Control-* headers unmodified, and let 404 responses stay 404.

location /mcp {
    proxy_pass http://phoenix:6006;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_read_timeout 1h;
}
location /.well-known/ {
    proxy_pass http://phoenix:6006;
}

Secrets, data sensitivity and the supply chain

What is in a trace

Start from the data. A span for a chat completion carries input.value, output.value, the full message list, and in retrieval pipelines the text of every retrieved document. If your application handles personal data, health data, financial data or customer support conversations, that content is now in a second database with a different access model from the one the application uses. For Gulf and Egyptian employers this is often the real conversation: data-residency rules and sector regulators may require that such content stays in-country, which is one of the strongest reasons to self-host Phoenix in a regional cloud region instead of using the hosted Phoenix Cloud. Ask legal where the line is before the first trace is exported, not after.

You have three tools to keep sensitive content out, in order of preference.

1. Do not record it. OpenInference instrumentation can be configured to hide inputs, outputs, message content and images. For a service that handles raw personal data, record structure (model, latency, token counts, tool names) and omit text.
2. Redact before export. Scrub in the application or in an OTel Collector processor before spans leave your network. The collector is the better place when many services are involved, because one policy covers all of them.
3. Limit who can see it, and for how long. Roles are global, so a short retention window is the most effective access control you have. Data that has been purged cannot leak.

Secrets inside Phoenix

Phoenix can store LLM provider keys under Settings → Secrets. The page is admin-only and write-only: you can set or replace a value but never read it back, and PUT /v1/secrets with value: null deletes one. Those values are encrypted with a key derived from PHOENIX_SECRET. That is convenient and it concentrates risk, so apply a rule: store in Phoenix only the keys that Phoenix itself must use (for the Playground and server-side evals), scope those provider keys to the minimum model access and a spending cap, and keep application keys out of it entirely.

Everything Phoenix needs to boot belongs in your secret store, not in a values file: PHOENIX_SECRET, PHOENIX_ADMIN_SECRET, the database password, SMTP credentials and identity-provider client secrets. In Helm use valueFrom.secretKeyRef. Prefer managed identity for the database where you can: PHOENIX_POSTGRES_USE_AWS_IAM_AUTH for Amazon Aurora and RDS, or PHOENIX_POSTGRES_USE_AZURE_MANAGED_IDENTITY for Azure, so that there is no long-lived password to leak.

The exporters' keys live in the applications' environments, and there is a quiet trap in it: Phoenix's SDKs and CLI discover a .env.phoenix file by walking up from the current directory. It is convenient on a laptop, and a hazard when the file holds a real key and is not in .gitignore. The process environment always wins over the file, which also means a stale PHOENIX_ENDPOINT exported in a shell profile silently redirects traffic. When a developer's traces land in the wrong project or server, run echo $PHOENIX_ENDPOINT before anything else.

The supply chain

Phoenix is software you run with powerful access, so apply the same scrutiny as to any platform dependency.

• The license. The server is licensed under the Elastic License 2.0 (ELv2), not Apache or MIT. You can self-host it freely, including inside a company, but ELv2 restricts offering it to third parties as a managed service. If your plan is to resell a hosted Phoenix, or to embed it in a product that customers run, read the license page and ask legal. The client libraries and OpenInference instrumentors are separate packages with their own licenses, so check each.
• Pin everything. Use an exact image tag such as arizephoenix/phoenix:20.16.0 or its -nonroot variant, never latest, and pin the Python and npm packages the same way. The sub-packages are versioned independently (for example arize-phoenix-otel, arize-phoenix-client, arize-phoenix-evals, @arizeai/phoenix-cli), so your lockfile is where compatibility is recorded.
• Prefer the nonroot image in hardened clusters, and solve the write-permission problem it creates (a volume the unprivileged user cannot write to) with fsGroup or a correct chown, instead of falling back to root.
• Mirror and scan. Copy the image into your own registry, scan it, and deploy the digest you scanned. Do the same for the Helm chart, which is published as an OCI artifact.
• Mind the Python environment. Phoenix and its instrumentors pull in many transitive packages. A private index or a vetted lockfile is ordinary supply-chain hygiene and worth doing here.
• Pin the client side too. One subtlety from the fact sheet: arize-phoenix-otel 0.17.2 requires Python 3.11 or newer, and on 3.10 pip quietly resolves the older 0.17.1. Two services that look identical can be running different tracing code, which is why you should record resolved versions, not just declared ones.

Multi-tenancy: one instance is one tenant

This is the most consequential fact about Phoenix as a shared platform, and it deserves its own section because teams assume the opposite.

One Phoenix instance is one tenant. Projects group traces, datasets group examples, and prompts carry names, but none of them is an authorisation boundary. Every user with an account can see every project, subject to the three global roles. There are no per-project grants, no per-team API key scoping and no row-level security. The maintainers have indicated resource tags for later in 2026, and until they ship you cannot rely on them.

That leaves you a few honest options, and the choice is an architecture decision you should make deliberately.

A workable pattern for most organisations is a tiered split: one shared instance for non-sensitive internal tools and experimentation, where everyone may see everything, and dedicated instances for any application that handles regulated or customer data. Promote a team between tiers by exporting what matters (datasets and prompts through the client) rather than trying to move rows in the database.

The instance-per-team option only stays affordable if deployment is a template. Keep one Helm values file per tenant that overrides a small, reviewed set of keys (name, hostname, database URL, secrets, retention), and generate them from a single source. The platform work then becomes maintaining the template, not hand-tending fifteen installs.

# One file per tenant; everything else inherits from the platform baseline.
fullnameOverride: phoenix-fraud
image:
  tag: "version-20.16.0-nonroot"
auth:
  enableAuth: true
  createSecret: false                      # the Secret below must already exist
  name: phoenix-secret                     # name of that pre-existing Secret (chart default)
database:
  url: ""                                  # from the tenant's own database credentials
  defaultRetentionPolicyDays: 30
ingress:
  host: phoenix-fraud.internal.example.com

Those keys are illustrative of the Helm chart's documented structure (auth.enableAuth, database.defaultRetentionPolicyDays and the ingress block exist per the chart defaults). ingress.host and auth.createSecret are in the chart's values file. With createSecret: false and auth enabled, the chart expects a pre-existing Secret (named by auth.name) holding PHOENIX_SECRET and PHOENIX_ADMIN_SECRET; individual entries can also use valueFrom.secretKeyRef under auth.secret. Verify these with helm show values for your chart version before you template anything.

Cost: where the money goes

Phoenix is free to self-host, so the bill comes from three places, and only one of them is obvious.

Compute is modest and predictable: a few replicas of the server, sized for ingest and query load.

Storage is the big one and it is driven entirely by retention, by payload size and by attribute cardinality. Traces are kept forever by default. PHOENIX_DEFAULT_RETENTION_POLICY_DAYS=0 means exactly that, and so does the Helm value database.defaultRetentionPolicyDays: 0. A platform that never set a policy has a database that only grows, and it will eventually find the disk guardrail the hard way. Set a global default on day one, then create per-project policies where the business needs something different: a policy can limit by maximum days, by maximum trace count, or both, with a cron schedule (the documented default is weekly, Sunday midnight). Assigning a policy requires the admin role, and deletion is irreversible, so a policy that is too aggressive is a data-loss event. Start long, shorten by measurement.

LLM spend is the quiet one. Phoenix runs LLM calls on your behalf for evaluations (an LLM judge scoring every production trace can cost as much as the application), for the Playground and for PXI. Budget for it explicitly, cap the provider keys that Phoenix holds, and choose what to evaluate: a sampled evaluation of ten percent of traffic often gives the same signal as scoring all of it.

There is also the cost Phoenix reports: it tracks LLM token cost per span and per session from a built-in model price table that is updated nearly every release, lets you override prices, and, from 20.10, supports filtering spans by cost in the filter DSL. That is a genuinely useful governance tool, and it carries one caution: the table is a snapshot of list prices. If your company has negotiated rates or runs a private deployment, override the prices, or the dashboard will confidently show the wrong numbers.

total_cost > 0.25                      # (sessions) conversations that were expensive
num_traces_with_error > 0              # (sessions) conversations that contained a failure

The session filter names above are taken from the documentation of session filter expressions; span-level cost filtering arrived in 20.10, and the exact span attribute names you can filter on are listed in the cost-tracking page. Verify the syntax in the UI's filter box, which validates as you type.

Upgrades and migrations

Phoenix's release pace is the defining fact of its operation. The server reached 20.0.0 on 2026-08-11 and 20.16.0 six weeks later, and the majors arrive every few weeks or months. You cannot skip upgrades for a year and expect a quiet life, and you cannot chase every minor either. You need a policy.

What a major upgrade changes

The MIGRATION.md file in the repository is required reading for every major. From the recent history, the pattern is clear:

• v20 persisted PXI chats in three new tables, with default retention (30 idle days, at most 30 chats per user, temporary chats deleted after 24 hours), and deprecated an older chat route in favour of /v1/agent_sessions. No manual steps, but saved filters on the assistant project must be rebuilt, because the session.id format changed.
• v19 made Phoenix an OAuth2 authorization server, mounted /mcp, stopped API keys from minting keys, and added the reverse-proxy requirements.
• v18 restructured indexes for the sessions time-range filter, honouring PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true.
• v14 was the big cleanup: a subcommand-first CLI (phoenix serve --port 6006, not the old flag order), asyncpg as the only PostgreSQL driver, the removal of the legacy px.Client, the rename of evaluations to annotations, and the removal of the /v1/evaluations endpoints.
• v13 removed inferences, embeddings and the UMAP point cloud. Phoenix stopped being an ML-drift tool, which matters if a team still plans around it.

The lesson for an owner is that breaking changes are mostly about your users' code and your proxy, not about your data. The migrations are automatic. The pain is a service still calling a removed endpoint, or a proxy that gates a path that must be open. Your upgrade process should therefore include a compatibility pass on the clients.

The safe sequence

# 1. Read MIGRATION.md for every major between your version and the target.
# 2. Take a backup and confirm it restores (do this part on a copy, not in prod).
pg_dump --format=custom --file=phoenix-$(date +%F).dump "$DATABASE_URL"

# 3. Rehearse the migration on a restored copy and time it.
phoenix db migrate --database-url "$COPY_DATABASE_URL"

# 4. In production, run the migration once, as its own job, not inside a pod start.
#    Set PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true for rolling deploys on PostgreSQL.
PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true phoenix db migrate --database-url "$DATABASE_URL"

# 5. Roll the new image tag through the replicas, then verify.
curl -s "$PHOENIX_ENDPOINT/arize_phoenix_version"

Several points deserve emphasis.

Rehearse on a copy. The time a migration takes depends on the size of your tables, and you only learn it by running it on comparable data. A rehearsal also discovers failures, such as the documented v18 case in which an earlier concurrent index build had left an INVALID index that must be dropped with DROP INDEX CONCURRENTLY IF EXISTS <name> before re-running.

Run migrations as a job. Because new instances block until migrations finish, running them inside the normal pod start means your readiness and liveness probes race the schema change. Running phoenix db migrate in an init job or a pre-upgrade hook separates the two, and lets you keep the probes tight.

Use concurrent index builds on live systems. PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true builds indexes with CREATE INDEX CONCURRENTLY, which avoids the table lock at the price of running roughly two to three times slower. For a very large spans table the v13 guide also describes pre-creating a no-op index (ix_spans_session_id) before the upgrade.

Never interrupt a long migration. The documented example is the v9 annotation migration, a long boot-time data migration that can leave the database needing manual repair if the process is restarted mid-way. Raise probe budgets and disable autoscaling restarts during upgrades rather than hoping.

Pin the image, and know the chart lags. The Helm chart's default appVersion trails the server release (chart 12.0.15 defaulted to 20.14.0 when the server was at 20.16.0), so set image.tag explicitly and record the chart version you deployed.

Order matters for the CLI. phoenix-cli pxi and the /v1/agent_sessions routes need a server on 20.0.0 or newer. Upgrade servers before you roll out a new CLI to developers.

Client compatibility

The good news is that OTLP and the OpenInference conventions are backward compatible, and the arize-phoenix-otel documentation states that any recent version works with any Phoenix server version. That lets you upgrade the server without a synchronised client rollout. The bad news is that code written to older APIs fails at import time, so have a short "old to new" table in your platform docs.

Observing Phoenix itself

A platform that watches other systems needs its own eyes. Phoenix gives you four sources, and a good deployment uses all of them.

Prometheus metrics. Set PHOENIX_ENABLE_PROMETHEUS=true (or the Helm value server.enablePrometheus) and the server exposes metrics on port 9090. The documented series include starlette_requests_processing_time_seconds_summary and starlette_exceptions_total for the request path; memory_usage_bytes and cpu_usage_percent for the process; bulk_loader_evaluation_insertions_total and bulk_loader_exceptions_total for the write path; rate_limiter_throttles_total; jwt_store_api_keys_active; and the disk series database_disk_usage_bytes, database_disk_usage_ratio and database_insertions_blocked. The source also has a counter for spans rejected when the queue is full. Check its exact name against /metrics on your version, because it is the most important one.

Health endpoints. /healthz and /readyz, used for probes and for a synthetic check from outside the cluster.

Self-tracing. The server can trace itself if you set PHOENIX_SERVER_INSTRUMENTATION_OTLP_TRACE_COLLECTOR_HTTP_ENDPOINT or the gRPC equivalent. Point it at a different backend than Phoenix itself, otherwise an outage of Phoenix hides its own evidence.

Logs. PHOENIX_LOGGING_MODE and PHOENIX_LOGGING_LEVEL control the application log, PHOENIX_DB_LOGGING_LEVEL and PHOENIX_LOG_SQL the database layer, and PHOENIX_LOG_MIGRATIONS shows migration progress, which you want on during upgrades. PHOENIX_MASK_INTERNAL_SERVER_ERRORS defaults to true so that clients do not see internals; your logs still do.

From these, a short alert set covers most incidents:

groups:
  - name: phoenix
    rules:
      - alert: PhoenixDiskNearGuardrail
        expr: database_disk_usage_ratio > 0.8
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Phoenix database storage above 80 percent of allocation"
      - alert: PhoenixInsertionsBlocked
        expr: database_insertions_blocked == 1
        for: 1m
        labels: { severity: page }
        annotations:
          summary: "Phoenix is refusing new spans because of the storage guardrail"

The thresholds are examples. Set the warning below the value you configured for the email warning, so a human is paged before Phoenix starts emailing itself.

Governing what Phoenix holds: evaluation, prompts and datasets

So far this guide has treated Phoenix as infrastructure. Its users treat it as a place where their quality decisions live, and that adds a governance dimension an owner should take seriously.

Evaluations are measurements with error bars

An LLM-as-a-judge evaluator is a model reading a rubric, so its score is an estimate. The evals library (arize-phoenix-evals 3.x) gives teams LLM, ClassificationEvaluator, create_classifier and a set of built-ins such as FaithfulnessEvaluator, CorrectnessEvaluator and RetrievalRelevanceEvaluator. Older names (llm_classify, OpenAIModel, HallucinationEvaluator) are removed or deprecated, and a platform that tolerates them is tolerating code that will break.

What you can do as an owner is make evaluation trustworthy across teams:

• Require human-labelled anchors. An evaluator is only as good as its agreement with people. Ask teams to keep a small labelled dataset, run the judge against it, and record the agreement before using the score in a release gate. Human annotations (annotator_kind="HUMAN") live alongside LLM and CODE ones, so anchors and judge outputs can be compared in the same place.
• Pin the judge. Changing the judge model changes the scores. Record the model and the prompt version used for each evaluation, so a trend line does not silently mix two judges.
• Watch NOT_PARSABLE. When a judge's answer is truncated or does not match the allowed labels, the result is NOT_PARSABLE. A rising rate signals a broken template or too small a max_tokens, not bad application quality.
• Define annotation configs centrally. An annotation config gives a name, a type (categorical, continuous or freeform) and its allowed values. Platform-owned configs for shared concepts such as correctness or user satisfaction stop every team from inventing its own slightly different label set. The name note is reserved for free-text notes.

Prompts are production code

Prompt management stores named, versioned templates with model configuration, and versions can carry tags such as production. If an application fetches the version tagged production at runtime, moving that tag is a production deployment, performed from a web UI by anyone with the member role. That is either a feature or a hole, depending on whether you have a process.

A reasonable minimum: applications pin to a specific version or to a tag you control; changing the production tag requires a review tracked outside Phoenix and an experiment run against a fixed dataset showing no regression; and the Playground is for exploration, not for editing live prompts. Because roles are global, you cannot stop a member from moving a tag with permissions alone, so the control is procedural and audited.

Datasets are the expensive, slow-to-rebuild asset

Datasets are versioned: every insert, update or delete creates a new version, and experiments can be tied to a dataset version. This is what makes comparisons meaningful, and it is why dataset loss is the failure you most want to prevent. Include datasets and annotation configs in your restore tests, and encourage teams to export their golden datasets to their own repositories. The client makes that a few lines:

from phoenix.client import Client

client = Client(base_url="https://phoenix.internal.example.com")
dataset = client.datasets.get_dataset(dataset="checkout-golden")
dataset.to_dataframe().to_parquet("checkout-golden.parquet")

Evaluation in CI

The documentation covers running evals from pytest, vitest and jest. As a platform owner, offer a shared template that runs an experiment against a named dataset on every pull request and fails when a chosen metric drops by more than an agreed margin. Expect flakiness from LLM judges: use repetitions (repetitions=N on an experiment runs each example N times), set a margin wider than the judge's noise, and alert on the noise itself. A gate that fails randomly is switched off within a month.

Incident playbooks

Incidents on a Phoenix platform are rarely dramatic. They are quiet losses and quiet exposures. The playbooks below follow the same order every time: stabilise, find the layer, fix, and then prevent.

"Our traces are missing"

1. Is it all teams or one? All teams means the server or database; one team means that client.
2. Check the server first. curl -i $PHOENIX/readyz, then the ingest 503 alert and the disk metrics. A full queue or blocked insertions explains "all teams".
3. Then the client. Read the register() banner in the application logs: it prints the project, the span processor, the collector endpoint and the transport. Most single-team cases are one of the causes below.
4. Confirm with a canary. Send one synthetic trace from the same network path and look for it.

"The UI is slow, or the database is struggling"

Stabilise by shortening the default time range users query and pausing heavy evaluation jobs. Check the database first: connections, long-running queries, autovacuum and disk. If reads dominate, confirm the read replica is configured. If writes dominate, look for a single service sending an unusual volume (a retry storm, or a loop creating spans) and ask that team to sample or pause. Then fix retention so the table stops growing.

"We are out of disk"

Do not delete rows by hand first. Raise the volume if you can, because PostgreSQL needs headroom to vacuum. Then apply a retention policy, let the purge run, and reclaim space with the normal PostgreSQL mechanisms. If Phoenix has already blocked insertions through the guardrail, raise PHOENIX_DATABASE_ALLOCATED_STORAGE_CAPACITY_GIBIBYTES only after you have added real capacity, never as the fix itself. Afterwards, add the disk alert you were missing.

"A migration failed or hung"

Read the log with PHOENIX_LOG_MIGRATIONS on. If the failure mentions an INVALID index from an earlier concurrent build, drop it with DROP INDEX CONCURRENTLY IF EXISTS <name> and re-run. If the process was interrupted during a long data migration, do not keep restarting. Restore the pre-upgrade backup into a clean database and retry with a longer window. This is the moment the rehearsal and the backup pay for themselves.

"A key or the admin secret leaked"

1. Contain. Revoke the user or system key in the UI. If it was PHOENIX_ADMIN_SECRET, rotate it, because it acts as a system bearer token.
2. Scope. Keys are global within the instance, so assume the holder could read every project. Use your proxy access logs to see what that key touched.
3. Rotate dependants. Provider keys stored in Phoenix secrets may have been used through the Playground or evals; rotate them at the provider too.
4. If it was PHOENIX_SECRET, plan a full rotation: sessions and stored secrets depend on it. Test on a copy before doing it in production.
5. Prevent. Add secret scanning to repositories, move the credential to the secret manager, and check that .env.phoenix is ignored.

"Sensitive data is in traces that should not have it"

This one is a privacy incident and needs legal involvement. Stop the exposure at the source first (disable or redact in instrumentation), then purge. Deleting specific data can mean shortening a retention policy and letting the purge run, or deleting the affected project, which is irreversible. Record what you deleted and why. Afterwards, add a review step to the onboarding of any application that handles personal data.

Where Phoenix stops

An owner has to know when the tool is the wrong one, and say so before a team discovers it the hard way.

It is not a multi-tenant product. If you need per-team access control inside one deployment, hard isolation between customers, or organisations and spaces, the answer today is multiple instances or the commercial Arize AX. Do not stretch a single instance to cover what it was not designed for.

It is not an analytics warehouse. Phoenix is a transactional SQL store with a UI. If you want to run arbitrary, large-scale analytical queries over billions of spans, export to a warehouse or an OLAP system and query there. Arize itself positions AX for very high volume and OLAP workloads.

It is not a general observability stack. Phoenix understands LLM concepts (spans of kind LLM, TOOL, RETRIEVER, sessions, token cost, evaluations), not your infrastructure. Keep metrics, logs and ordinary distributed traces in your existing stack (Prometheus, Grafana, Tempo, Jaeger) and let an OTel Collector send LLM spans to both.

It is not an ML drift tool any more. Inferences, embeddings and the UMAP point cloud were removed in v13. If a team wants tabular or embedding drift monitoring, point them at a tool built for it, such as Evidently (see /student-guides/evidently).

It is not an API gateway or a guardrail. Phoenix observes and evaluates; it does not sit in the request path to block a response. Real-time safety checks belong in your application or a dedicated guardrail layer.

Licensing can rule it out. If your business model involves offering a hosted observability service to third parties, the Elastic License 2.0 restricts that. Read it before you build on it.

The alternatives are real. Langfuse (see /student-guides/langfuse) and LangSmith are the tools most often compared with Phoenix, and the Phoenix documentation has comparison pages of its own. Choose on your constraints: data residency, licence, the frameworks you use, and how much you want to operate. Because Phoenix speaks OpenTelemetry, instrumentation you write for it is more portable than most, which lowers the risk of choosing it and then changing your mind.

Running Phoenix as a platform for other teams

Everything above is technical. The difference between a tool and a platform is the work around it.

Define the service

Write a one-page service definition before you onboard anyone. It says what the platform provides (tracing, evaluation, datasets, prompt management), what it does not (per-project privacy, real-time blocking, long retention), the retention window, the availability target, the support channel, and the tier rules from the multi-tenancy section. Teams will otherwise assume the most generous version of every promise.

Make adoption a paved road

Publish a template that gives a new service working tracing in ten minutes: a pinned arize-phoenix-otel and the right openinference-instrumentation-* package, a register() call with batch=True, project naming rules, and environment variables set from the platform's configuration. A naming convention such as <team>-<service>-<environment> for projects keeps a shared instance navigable.

import os
from phoenix.otel import register

def setup_tracing(service: str, env: str):
    """The one function every service calls. Owned and versioned by the platform team."""
    return register(
        project_name=f"{os.environ['TEAM']}-{service}-{env}",
        protocol="http/protobuf",
        batch=True,
        auto_instrument=True,
    )

Wrapping register() in a shared library gives you a single place to change the endpoint, add a sampler, hide sensitive attributes or move to a collector, with one library upgrade instead of forty pull requests. That is the highest-leverage decision in this whole section.

Onboarding and offboarding

Onboarding gives a team a system key, a project, a role mapping through their identity group, and a link to the template. Offboarding matters as much: revoke keys, remove group mappings, and apply retention so abandoned projects do not sit in the database indefinitely. Review the list of system keys quarterly; a key with no recent use and no owner is a liability.

Support and communication

Publish upgrade dates ahead of time, with the client-facing changes listed (for example "the px CLI requires Node 22 or newer", or "pxi needs the server on 20 or newer"). Keep a status note for incidents, and a changelog of the platform's own configuration. Users forgive downtime they were warned about far more readily than surprises.

Measure the platform

Track a small set of indicators that say whether the platform is working: the share of production services that are traced, the canary loss rate, p95 query latency, the proportion of traces with at least one annotation, the age of the running Phoenix version, and the number of teams on outdated client libraries. The first and fourth are adoption measures, and they tell you whether you are running a platform people use or a dashboard nobody opens.

Decide what you do not support

You will be asked for things outside the boundary: per-project privacy, a custom retention rule for one team, a fork with local patches. Decide in advance how those are handled. Saying "that needs its own instance" is a legitimate answer if you have made instances cheap to create.

The review checklist

Use this for pull requests that touch the platform, for design documents that propose a new Phoenix deployment, and as an audit of an existing one.

Architecture and data

• Is the database PostgreSQL 14 or newer, with backups, point-in-time recovery, and a restore tested this quarter?
• Are datasets, prompts, annotation configs and users included in the restore test, not just spans?
• Is there a read replica, and is PHOENIX_SQL_DATABASE_READ_REPLICA_URL set where read load justifies it?
• Are the storage guardrail variables set, and is the alert threshold below them?

Ingest

• Do all services use batch=True, an explicit protocol, and a full endpoint?
• Is there sampling or a collector where volume warrants it?
• Is there a canary comparing spans sent with spans received?
• Is the 503 rate alerted?

Security

• Is PHOENIX_ENABLE_AUTH true, with PHOENIX_SECRET and PHOENIX_ADMIN_SECRET distinct, strong and stored in a secret manager?
• Is the default admin password changed, and is basic auth disabled once SSO works?
• Are role mapping and group gating configured, and are claims in the ID token?
• Are PHOENIX_CSRF_TRUSTED_ORIGINS and secure cookies set?
• Do applications use system keys, and are key owners recorded?
• Is egress restricted with a NetworkPolicy or a proxy allowlist, including the metadata range?
• Has a decision been made and written down on MCP, the OAuth2 authorization server, PXI and sandboxes?
• Does the reverse proxy follow the v19 list, and is PHOENIX_ROOT_URL correct?

Tenancy and data

• Does every team know that a project is not a privacy boundary?
• Are sensitive applications on dedicated instances, with content hidden or redacted at source?
• Has legal confirmed where data may be stored, particularly for regulated Gulf and Egyptian workloads?

Upgrades and supply chain

• Is the image pinned to an exact tag, mirrored and scanned, with the nonroot variant where possible?
• Was MIGRATION.md read, the migration rehearsed on a copy, and run as a separate job?
• Is PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true set for live PostgreSQL upgrades?
• Is the Helm chart's lag handled with an explicit image.tag?

Governance

• Are human-labelled anchors used before an LLM judge gates a release?
• Are the judge model and prompt version recorded with each evaluation?
• Is the production prompt tag protected by a review process?
• Are retention policies set globally and per project, and agreed with the data owners?

The complete picture

Put the pieces together and the platform has a clear shape.

Applications call one shared function that sets up tracing. A collector absorbs bursts and removes what should never leave the network. Phoenix runs as several replicas with authentication, SSO role mapping, TLS at the edge, restricted egress and a pinned image. PostgreSQL holds the state, with a replica for reads, tested backups and a retention policy. Around that sit the things that make it a service: alerts on the ingest path and on storage, a canary that measures loss, an upgrade process that rehearses on a copy, and a governance layer for evaluations, prompts and datasets.

Each section in this guide is one of those boxes or arrows. The order they became urgent in is also a sensible order to build them:

1. Move off SQLite to PostgreSQL, with a backup that restores.
2. Turn on authentication, SSO and role mapping, and issue system keys.
3. Fix the client: a shared wrapper, batch=True, an explicit protocol.
4. Set retention and the storage guardrail.
5. Add the alerts and the canary.
6. Restrict egress and decide the agent-surface policy.
7. Write the upgrade runbook and rehearse it.
8. Publish the service definition and the tenancy tiers.

You will notice that nothing on this list is about Phoenix's features. The features are good, and they are the reason people come. The list is about making it safe to rely on them.

Where the series leaves you

You began this series by starting a server and seeing your first trace. You learned what a span, a project and a session are, how auto-instrumentation creates spans, and how to read a trace tree. At the middle level you learned context attributes, annotations, datasets, experiments, the clients and the px CLI, and how to run evaluations and use the filter language. Here you took on the work that sits beneath all of that: architecture and failure modes, scaling limits, the trust model, multi-tenancy, cost, upgrades, incidents and governance.

What makes a senior owner is not memorising environment variables. It is holding the whole system in mind at once: knowing that the database is the platform, that an exporter can fail silently, that a project is a filter and not a wall, that a prompt tag is a deployment, and that a fast-moving release train needs a rehearsed upgrade. It is also being honest with other teams about what the platform does not do.

Where to go next depends on your gap. If your evaluation practice is thin, read /student-guides/ragas and /student-guides/deepeval, and compare with /student-guides/langfuse to see a different answer to the same problem. If you will run this on Kubernetes, /student-guides/kubernetes and /student-guides/helm cover the deployment layer, and /student-guides/argo-cd covers how to keep the platform's configuration declarative. Above all, rehearse: restore a backup, run a migration on a copy, break an exporter on purpose. The skills in this guide become reliable only after you have done them once when nothing was on fire.

Sources

• Phoenix documentation home
• Self-hosting overview
• Architecture
• Configuration reference
• Docker deployment
• Kubernetes with Helm
• Authentication
• Network security
• Encryption
• Migrations and upgrades
• Production guide
• Data retention
• Secrets
• API keys
• Access control (RBAC)
• Remote MCP
• PXI
• Cost tracking
• Sandbox runtimes
• Self-hosting license
• Release notes
• MIGRATION.md
• Server configuration source (config.py)
• Helm chart values
• Evals package changelog
• arize-phoenix on PyPI
• arizephoenix/phoenix on Docker Hub
• OpenInference

Part three of three. A senior interview about Arize Phoenix is rarely about whether you can make a trace appear. It is about whether you can run the platform that thousands of traces flow into, defend the decisions you made, and explain them to people who do not care how OpenTelemetry works. About fifty minutes of reading. Start with the framing, then the system-design prompts, then incidents, then the leadership conversations.

How senior interviews differ

At the beginner level the interviewer checks vocabulary. At mid level they check that you have used the tool at work. At senior level they check judgement under constraints: what you would build, what you would refuse to build, what you would measure, and what you would say when someone pushes back. The best answers share four habits.

• They start from the requirement, not the tool. "How much data, how sensitive, who reads it, how long do we keep it" comes before any configuration key.
• They name the failure mode. Anyone can describe the happy path. A senior answer says what breaks first and how you would notice.
• They state the boundary. Saying "Phoenix is one tenant per instance, so I would not promise project-level privacy" earns more trust than pretending the tool does everything.
• They commit to a number or a rule. "It depends" is true and useless. "I would start with thirty days of retention and shorten it by measurement" is an answer.

Facts you should have ready, because they come up in almost every senior conversation: Phoenix is a single stateless server plus one SQL database; SQLite is for development and PostgreSQL 14 or newer is for production; authentication is off by default on the bare server and on by default in the Helm chart; roles are global (admin, member, viewer); one instance is one tenant; the ingest queue returns HTTP 503 when it passes PHOENIX_MAX_SPANS_QUEUE_SIZE; migrations run at startup; and the license is Elastic License 2.0. The version this guide was written against is 20.16.0, and because the server ships a minor release every few days you should say "at the time of writing" when quoting details.

The thirty-second platform answer

If an interviewer says "tell me how you would run Phoenix for the company", this is the shape of a good opening.

I would run it as a small service with a clear boundary: a few stateless Phoenix replicas behind a load balancer, one managed PostgreSQL with a read replica and tested point-in-time recovery, authentication and SSO turned on from day one with roles mapped from identity groups, and applications exporting through a shared tracing wrapper that sets batching, sampling and the endpoint in one place. I would set retention on day one, alert on the ingest 503 rate and on disk, and treat one instance as one tenant, so sensitive applications get their own instance.

Then stop and let them pick the thread. Every clause in that answer is something you can be asked to expand, which is the point: it shows the map before you walk the territory.

System-design prompts

Design 1: observability for forty services at a bank

The prompt. A regional bank is adding LLM features to forty services across six teams. Some handle customer conversations that include account details. Regulators in the region require that customer data stay in-country. Design the tracing and evaluation platform.

A strong answer, in order.

1. Clarify. Ask for request volume per service, the payload sizes, the retention the regulator requires, whether prompts may contain account numbers, and who needs to read traces (developers, risk, auditors).
2. Decide the hosting question first. Customer content cannot go to a hosted service outside the country, which rules out the hosted Phoenix Cloud and points to self-hosting in an in-country cloud region or on premises. Mention this unprompted, because it is the real constraint.
3. Decide tenancy. One Phoenix instance is one tenant and roles are global, so a project is not a privacy boundary. I would separate by sensitivity: one instance for non-sensitive internal tooling, and dedicated instances for the customer-facing services, each with its own PostgreSQL database or schema. Teams that handle account data get an instance that only their group can sign in to.
4. Minimise what is recorded. Hide or redact message text at instrumentation time for the most sensitive services, record structure (model, latency, token counts, tool names), and scrub in an OTel Collector for the rest. The best protection against a leak is data that was never stored.
5. Identity. SSO through the bank's identity provider, _ROLE_MAPPING from groups to roles, _ALLOWED_GROUPS to restrict sign-in, and password login disabled after confirming an SSO admin exists. Claims must be in the ID token.
6. Network. TLS at the edge, sslmode=verify-full to the database, and a NetworkPolicy that permits egress only to the database, DNS and the approved model providers, with the metadata range blocked to contain server-side request forgery.
7. Capacity and retention. Measure bytes per request on a representative service, multiply by volume and the retention window, and set the storage guardrail below the disk. Set a global retention default and shorter policies where the regulator allows it.
8. Operations. Managed PostgreSQL with point-in-time recovery, a restore test on a schedule, alerts on 503s and disk, a canary comparing spans sent with spans received, and an upgrade process that rehearses migrations on a copy.
9. Adoption. A shared library that wraps register() so there is one place to change endpoint, sampling and redaction.

What a weak answer does. It says "deploy Phoenix with Docker" and stops. It forgets residency, assumes projects isolate teams, leaves authentication off, and never mentions who can read the prompts.

Follow-up you should expect. "Auditors want to see every conversation for seven years." The honest response is that Phoenix is an observability store with a retention policy, not an immutable archive. For a regulatory record, export to an append-only store designed for it, and keep Phoenix retention short. Saying so is a mark of seniority.

Design 2: surviving ten times the traffic

The prompt. Traffic on your tracing platform is about to grow tenfold because a product launch moves LLM features into the main checkout flow. You have one PostgreSQL primary and three Phoenix replicas. What do you do in the next two weeks?

A strong answer.

• Find the current headroom first. Load-test a representative service, record bytes and rows per request, and read the write latency on the primary. Phoenix publishes no numeric throughput ceiling, so you must measure your own.
• Reduce what you send. Enforce batch=True everywhere, sample (a parent-based ratio sampler so whole traces stay together), keep all errors and slow traces with tail sampling in a collector, and trim payloads that hold entire documents or images.
• Scale the application tier, then the database. More replicas drain the queue faster, but they all write to one database, so the primary is the ceiling. Add the read replica with PHOENIX_SQL_DATABASE_READ_REPLICA_URL so that UI reads do not compete with ingest, and size the primary's storage and IOPS.
• Protect yourself. Keep PHOENIX_MAX_SPANS_QUEUE_SIZE as a back-pressure signal rather than raising it to hide the problem, because a bigger queue costs memory and only delays the 503s. Alert on the 503 rate.
• Shorten retention for the busy project so the table does not triple in size.
• Know the exit. If measurement shows you will outgrow a single PostgreSQL primary, the right move may be to export to a warehouse for analysis, or to move to the commercial Arize AX, which is built for that scale. Saying this openly is better than promising that Phoenix will scale indefinitely.

What a weak answer does. "Add more pods." More replicas do nothing once the database is the bottleneck, and the interviewer is waiting to see whether you know it.

Design 3: a multi-team evaluation platform

The prompt. Six teams each build their own evaluation scripts. Scores are incomparable, nobody trusts them, and a prompt change last month shipped a regression that the scores did not catch. Design the evaluation layer on Phoenix.

A strong answer.

1. Separate the concepts. Datasets are versioned collections of examples, experiments run a task over a dataset and score it with evaluators, and annotations are labels attached to spans and traces. Teams were probably mixing these.
2. Platform-owned annotation configs for shared concepts, so "correctness" means one thing.
3. Human anchors. Every LLM judge is validated against a small human-labelled set before it gates anything. Record the agreement. Without this, a judge is an opinion with a number on it.
4. Pin and record the judge. The judge model and its prompt version go into annotation metadata, so a trend never silently mixes two judges.
5. Use the current evals API. LLM, ClassificationEvaluator, create_classifier and the built-ins. The legacy llm_classify and OpenAIModel names are gone in evals 3.0, so code using them should be migrated rather than supported.
6. CI gate with noise handling. Run an experiment against a golden dataset on each pull request with repetitions, and set the failure margin wider than the observed noise of the judge. Report the noise.
7. Prompts as code. Applications pin a prompt version, and moving the production tag needs a review and a passing experiment, because that tag move is a deployment made from a web UI.
8. Governance on cost. Judge calls cost money. Sample production evaluation, cap the provider keys that Phoenix holds, and track spend.

The follow-up. "The scores still did not catch the regression." The answer is that the golden dataset did not contain the failing case. The fix is a feedback loop: production failures found by annotation or user report are added to the dataset as new versions, so the next regression is caught. A dataset that never grows is a dataset that decays.

Design 4: migrating off a competitor, or to Phoenix

The prompt. A team wants to move from a hosted tracing product to Phoenix. What do you check?

A strong answer.

• Instrumentation portability. Phoenix accepts OpenTelemetry, so OpenInference instrumentors or any OTel source work. Check how the current traces are produced, because vendor-specific SDKs need replacing.
• What must be migrated. Usually it is datasets and prompts, not historical traces. Export those and recreate them through the client. Historical spans are rarely worth moving.
• Gaps. Compare what the team actually uses: alerting, per-project permissions, dashboards. Phoenix has no per-project permissions, and alerting is something you build around Prometheus and your own tools.
• Dual-running. Send traces to both for a period using a collector, and compare counts before cutting over.
• Licence and data residency. Confirm the ELv2 terms fit, and whether self-hosting is needed to meet residency.
• An exit plan. Because the data path is OTel, leaving Phoenix later is also feasible. Mention it, because it reduces the risk of the decision.

Incident-response questions

These are usually asked as "it is three in the morning and...". Answer in a fixed order: stabilise, locate the layer, fix, prevent. Say the order aloud.

"Traces from one team stopped appearing."

Start by asking whether it is one team or all. One team points at that client. Read the startup banner from register(), which shows the project, the processor, the endpoint and the transport. The usual causes, in order of likelihood: the Python SDK inferred gRPC on port 4317 while only 443 or 6006 is reachable; a /v1/traces URL combined with an inferred protocol, which rewrites the port to 4317 and keeps the path; only PHOENIX_ENDPOINT was set while trace export reads PHOENIX_COLLECTOR_ENDPOINT; authentication was enabled and the service has no key, giving 401; or no OpenInference instrumentor is installed. Fix with an explicit protocol="http/protobuf" and a full endpoint. Confirm with a canary trace. Prevent with the shared wrapper.

"Everything is returning 503."

That message is Server is at capacity and cannot process more requests, and it means the span queue passed PHOENIX_MAX_SPANS_QUEUE_SIZE. It is back-pressure, not a bug. Check the database: slow writes, a long-running query, a full disk or a blocked insertion flag are the usual reasons the queue stops draining. Stabilise by reducing load (ask the heaviest service to sample), then fix the database. Say clearly that raising the queue size is a temporary measure that costs memory and risks an out-of-memory kill. Then add the alert you did not have, and mention that exporters that gave up have lost those spans, so the gap in data is real and should be communicated.

"The upgrade is stuck and pods keep restarting."

Migrations run at startup and new instances block until they finish. If a liveness probe kills the pod during a long index build, the migration restarts forever. Stop the restarts, give the startup phase a generous probe budget, and run phoenix db migrate as a dedicated job. Check the log with PHOENIX_LOG_MIGRATIONS. If the log mentions an INVALID index from an earlier concurrent build, drop it with DROP INDEX CONCURRENTLY IF EXISTS. If a long data migration was interrupted, stop restarting and restore the pre-upgrade backup into a clean database. The preventive answer is the interviewer's real target: rehearse on a copy, take a backup, use PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true for live systems, and time it beforehand.

"An engineer left and their key is in a public repository."

Contain first: revoke the key. Then scope: roles are global, so assume the key could read every project that role allows. Check proxy logs for what it touched. If it was a user key, it dies with the user, which is a reason applications should use system keys. If it was PHOENIX_ADMIN_SECRET, rotate it, because it acts as a bearer token for the system user. Note that since version 19 an API key cannot create other keys, which limits the damage from a leaked low-privilege key. Finish with prevention: secret scanning, a secret manager, .env.phoenix ignored by git.

"Legal says customer data is in traces."

This is a privacy incident. Stop the flow at the source by hiding or redacting in instrumentation or in the collector. Then remove the data, which may mean shortening retention and letting the purge run, or deleting a project, both irreversible, so confirm the backup position first. Document what was removed. The deeper answer is the review gate: applications that handle personal data do not get onboarded without a decision about what is recorded. If asked about regional rules, say you would involve legal on residency rather than guess.

"The UI takes a minute to load."

Separate the layers. Is the database slow (long queries, missing index after a migration, vacuum behind)? Are users querying a huge time range? Is the read replica configured? Is it an air-gapped install waiting on external fonts, which has a known fix in PHOENIX_ALLOW_EXTERNAL_RESOURCES=false? A senior answer lists candidates by likelihood and says which metric distinguishes them.

"The MCP client cannot connect through our proxy."

This is a version 19 reverse-proxy problem. Check that PHOENIX_ROOT_URL matches the URL clients use, that host-root /.well-known/* is forwarded unmodified, that no SSO gateway sits in front of /mcp or the /oauth2/* endpoints, that OPTIONS passes, and that the event stream is not buffered (proxy_buffering off in nginx). Symptoms are CORS errors or a redirect to a login page.

Trade-off questions

One shared instance or one per team? A shared instance is cheap and easy to browse, and everyone can read everyone's prompts. Separate instances give real isolation and independent upgrades, at the cost of operational overhead. The answer is tiered: shared for low-sensitivity work, dedicated for regulated data, with instances made cheap through templates.

SQLite or PostgreSQL? SQLite for a laptop. PostgreSQL for anything shared, because SQLite has a single writer, no replica and no point-in-time recovery, and you cannot run multiple server replicas against it.

Sample or keep everything? Keeping everything is simple and expensive. Sampling loses information but makes cost predictable. Use tail sampling to keep errors and slow traces, sample routine traffic, and never sample in a way that splits a trace.

Direct export or an OTel Collector? Direct export is fewer parts. A collector adds batching, retry, redaction, tail sampling and fan-out to other backends, at the price of another component that can lose data. Use a collector when you have many services or need redaction.

Store prompts or hide them? Storing full text makes debugging and evaluation far better. It also puts sensitive content in a second store. The compromise is per-application: full content for low-risk services, structure only for high-risk ones.

Self-host or use a hosted Phoenix? Self-hosting gives data control and residency at the cost of operating it. The hosted Phoenix Cloud removes operations but sends data to a third party. For Gulf and Egyptian employers with residency rules, self-hosting in-region is often the only compliant option.

LLM judge or human labels? Humans are accurate and slow. Judges are fast and noisy. Use humans to validate a judge, then use the judge for scale, with periodic re-checks.

Phoenix or Langfuse or LangSmith? Decide on constraints, not preference: licence (Phoenix is ELv2), self-hosting, frameworks, residency and how much you want to operate. Phoenix's OpenTelemetry basis makes the instrumentation portable, which lowers switching cost. See /student-guides/langfuse for the main open-source alternative.

Chase every minor release or upgrade quarterly? The server releases often. Pinning and a quarterly cadence with a rehearsal is the stable answer, plus fast-tracking security fixes. Never track latest.

Arguing a trade-off to leadership

Leadership questions are about money, risk and time. The technique is to translate each technical fact into one of those three and to offer a decision, not a lecture.

Asking for budget for PostgreSQL and a read replica. Do not say "SQLite does not scale". Say: "Today our evaluation datasets and prompts live in a single file on one disk. If that disk fails we lose months of curated work. A managed database with point-in-time recovery costs roughly the price of one engineer-week per quarter and removes that risk." Quantify the loss, then the price.

Explaining why one instance cannot be private per team. "Projects are labels, not walls. Anyone who can sign in can read every project. We have two choices: accept that for low-sensitivity work, or pay for separate instances for the sensitive services. I recommend the tiered split and have costed it." You are presenting a decision with a recommendation.

Justifying retention. "Every extra month of traces costs storage and widens what a breach exposes. I propose thirty days by default, ninety for the two projects where investigations need it, and I will review the numbers after a quarter of measured use." Leadership hears risk and cost, and a review date.

Saying no to a custom request. "We can give that team a private instance in about two days, because the deployment is templated. We cannot give them a private project on the shared one, because the product does not support it." A no with an alternative is received far better than a no.

Explaining an outage honestly. State what happened, what data was lost, how you know, and what changes. "For forty minutes the ingest queue was full and exporters dropped spans. We lost about that window of traces for three services. We found it from a canary alert. We have added a disk alert and will rehearse the migration that triggered it." Precision builds credibility, and vagueness destroys it.

Arguing for an investment in evaluation. Tie it to an incident: "A prompt change regressed answer quality and nobody noticed for nine days. A golden dataset with a CI gate would have caught it in minutes. The build cost is about two engineer-weeks plus the judge spend, which I estimate at this figure per month."

Arguing against a rewrite. If someone proposes replacing Phoenix with a home-built tracing store, say what it would take: a collector, storage, a UI, evaluation, prompt management, authentication, upgrades and an on-call rota. Compare that with the licence terms and the operating cost of Phoenix, and recommend building only the thin layer that is genuinely unique to you.

Scenario questions with model answers

"How would you know you are losing spans?" Compare a count from the source (the application or collector) with a count in Phoenix for a canary service over the same window, and alert on the ratio. Also alert on 503s. Nothing else detects silent drops, because the client does not fail.

"Why are all our new OAuth users viewers?" Without role mapping every new OAuth user is a viewer. Configure _ROLE_ATTRIBUTE_PATH and _ROLE_MAPPING, and make sure the claim is in the ID token or UserInfo, because Phoenix does not read access-token claims.

"We turned on authentication and everything broke." Enabling auth on a live system makes unauthenticated exporters fail with 401 and stops collection until they have keys. The right sequence issues system keys and rolls them out before flipping the switch, or accepts a planned gap.

"Can a viewer create an API key?" A viewer can manage their own user keys, but cannot write through the API, so writes return 403. This is a useful detail because it shows you know the difference between managing your own credentials and having write permission.

"What is the risk of the Playground?" It makes server-side calls to model providers with configurable endpoints and stored keys, so it is a server-side request forgery surface. The controls are network policy on egress, the metadata range blocked, fixed-endpoint custom providers, and scoped provider keys. PHOENIX_ALLOWED_PROVIDERS only filters the UI.

"What changed in version 14 that still bites people?" The server CLI became subcommand-first (phoenix serve --port 6006), asyncpg became the only PostgreSQL driver, the legacy px.Client was removed in favour of phoenix.client.Client, evaluations became annotations, and the /v1/evaluations endpoints were removed. Old tutorials fail on all of these.

"What changed in version 13?" Inferences, dimensions, embeddings and the UMAP point cloud were removed, so Phoenix stopped being an ML drift tool.

"How do you upgrade with no downtime?" Be honest: back up, rehearse, run migrations as a job with PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true to avoid table locks, then roll replicas with readiness probes. Note that migrations can block new instances, so the old version should keep serving while the job runs, and some migrations are long and must not be interrupted.

"What would you never do?" Expose an unauthenticated Phoenix to a network you do not control. Track latest in production. Promise project-level privacy. Interrupt a running migration. Put a proxy login page in front of /mcp or the OAuth endpoints.

What a strong answer covers that a weak one misses

Sixty-second self-test

Answer each aloud, without notes, in about ten seconds.

1. Which component is stateful, and what does that imply for backups?
2. What status code do you see when the ingest queue is full, and what is the variable?
3. Why is a project not an access-control boundary?
4. What does enabling authentication do to existing exporters?
5. Name the two secrets you must set, and the rule that relates them.
6. Who can mint a system API key since version 19?
7. What are the four documented reverse-proxy mistakes that break MCP?
8. Which variable makes migrations avoid table locks, and what does it cost?
9. Why is moving the production prompt tag a deployment?
10. Name three things Phoenix is not.

Part three of three. At this level the problems are operational and they appear with scale: a database that quietly filled, exporters that dropped spans without a sound, a prompt tag someone moved on a Friday, an upgrade that held a lock for twenty minutes. Start with the error table, then the practices, the recipes, and the review checklists.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These cause incidents and quiet losses, not failed first runs.

Why these happen: four shared causes

Most rows in that table reduce to four ideas. Learning them is faster than memorising the list.

The client fails silently. An OpenTelemetry exporter retries and then gives up without affecting the application. Every "missing traces" incident shares this shape, which means the server cannot tell you what it never received. You need a measurement at both ends, not a hope.

The database is the platform. Phoenix is stateless, so everything that hurts (growth, locks, slow queries, loss) is a database property. Retention, backups, replicas and migration behaviour are all database decisions.

Scope is global. Roles, keys and project visibility are instance-wide. Anything that sounds like "per team" is a convention, not a control, until resource-level permissions exist.

The release train is fast and breaks old code. The server ships a minor release every few days and majors regularly. The data is migrated for you. Your clients, proxies and scripts are not.

The practices that pay off most

Platform-ownership habits

These are not commands. They are the routines that keep a shared platform healthy, in order of leverage.

1. Publish a service definition. One page: what the platform does, what it does not, retention, support, upgrade cadence, tenancy tiers. It prevents the most expensive misunderstanding, which is teams assuming privacy or long retention you never promised.
2. Make the paved road the easy road. A template with tracing, dataset loading, an experiment and a CI job. People follow the path that is already built.
3. Review the key inventory quarterly. List system keys, find those unused for ninety days or with no owner, revoke them. Include who holds PHOENIX_ADMIN_SECRET.
4. Keep a written decision on agent access. MCP, the OAuth2 authorization server, PXI and sandboxes. Each major release added one. Put the answer and the variables that enforce it in version control.
5. Watch adoption and loss, not only uptime. Share of production services traced, canary loss rate, and proportion of traces annotated. A healthy server with no users is a failed platform.
6. Announce changes before they happen. Upgrade dates, client requirements (the px CLI needs Node 22 or newer, pxi needs a server on 20 or newer), deprecations. Warned downtime is forgiven.
7. Keep the old-to-new table current. When a release removes an API, add a row. It turns a ticket into a link.
8. Write the postmortem for near-misses. A guardrail that fired and blocked writes is a near-miss worth an hour of analysis.

Practice cards

1. Watch the queue refuseStart a local Phoenix with a tiny PHOENIX_MAX_SPANS_QUEUE_SIZE, send a burst, and read the 503 and the exporter's behaviour. Seeing it once makes the alert meaningful.
2. Break an exporter on purposeUse a /v1/traces URL with an inferred protocol and diagnose it from the startup banner alone. Then fix it with an explicit protocol.
3. Prove the viewer cannot writeCreate a viewer user, try a POST to /v1/span_annotations with their key, and confirm 403. Repeat after each upgrade.
4. Time a migration on a copyRestore a backup, run phoenix db migrate, then again with PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true. Record both numbers in your runbook.
5. Try to reach the metadata addressFrom inside the Phoenix pod, request 169.254.169.254. It must fail. If it does not, your egress policy is decorative.
6. Restore and look for datasetsAfter a restore to a scratch database, list datasets and prompts through the client. Spans are the least valuable thing to recover.
7. Measure bytes per requestSend a known number of traced requests, read the database growth, and compute the monthly figure for your real volume.

Operational recipes

Generate and set the two secrets

Two distinct strong values, stored in a secret manager, never in a values file.

openssl rand -hex 32   # PHOENIX_SECRET
openssl rand -hex 32   # PHOENIX_ADMIN_SECRET (must differ)

PHOENIX_SECRET needs at least 32 characters with a digit and a lowercase letter, which openssl rand -hex 32 satisfies in practice. Keep the admin secret set only together with the main secret.

Verify a deployment from the outside

#!/usr/bin/env bash
set -euo pipefail
P=${PHOENIX_ENDPOINT:?set PHOENIX_ENDPOINT}
echo "version:   $(curl -s "$P/arize_phoenix_version")"
echo "readyz:    $(curl -s -o /dev/null -w '%{http_code}' "$P/readyz")"
echo "no key:    $(curl -s -o /dev/null -w '%{http_code}' "$P/v1/projects")  (expect 401 when auth is on)"
echo "with key:  $(curl -s -o /dev/null -w '%{http_code}' -H "authorization: Bearer ${PHOENIX_API_KEY:?}" "$P/v1/projects")  (expect 200)"

Run it after every deployment. If the "no key" line returns 200 on a platform you believe is protected, stop and find out why before doing anything else.

Run the migration as its own job

# Run once, before rolling replicas. Build indexes without locking the table.
PHOENIX_MIGRATE_INDEX_CONCURRENTLY=true \
PHOENIX_LOG_MIGRATIONS=true \
phoenix db migrate --database-url "$PHOENIX_SQL_DATABASE_URL"

Keep a backup taken immediately before it, and never interrupt it once it has begun a long data migration.

Pin the Helm deployment explicitly

The chart's default image lags the server release, so be explicit about both versions.

helm upgrade --install phoenix oci://registry-1.docker.io/arizephoenix/phoenix-helm \
  --version 12.0.15 \
  --set image.tag=version-20.16.0-nonroot \
  -f values.yaml

Confirm the chart version you pass exists with helm show chart before relying on it, since chart versions advance independently of the server. Uninstalling the release keeps the PostgreSQL volume claim, so a deleted release is not a deleted database.

Count spans at both ends

# Spans Phoenix holds for the canary project in the last hour.
px span list --project canary --last-n-minutes 60 --limit 5000 --format json --no-progress | jq 'length'

Compare this number with the count the canary application reports having emitted. A persistent gap is silent loss. The default px span list limit is 100, so always pass --limit when counting.

Keep sensitive text out of traces

# Hide message text for a high-risk service. Check the option names in the
# OpenInference TraceConfig documentation for the instrumentor you use.
from openinference.instrumentation import TraceConfig

config = TraceConfig(hide_inputs=True, hide_outputs=True)

Pass the config to the instrumentor when you call instrument. Structure (model, latency, token counts) stays useful while the text never leaves the process.

Restrict the Playground's providers

export PHOENIX_ALLOWED_PROVIDERS="OPENAI,ANTHROPIC"

This only filters the UI. Pair it with the egress policy, because it does not stop a determined user from configuring an endpoint in some other way.

Turn off agent surfaces you have not approved

export PHOENIX_ENABLE_MCP_CODE_MODE=false
export PHOENIX_DISABLE_AGENT_ASSISTANT=true
export PHOENIX_ALLOWED_SANDBOX_PROVIDERS=NONE

Write the decision in the repository next to these lines, so a future reader knows it was intended.

Review checklist for pull requests

Use it when a change touches the platform's configuration, a shared library, or how a service exports traces.

• Does the change pin versions for images, charts and client packages?
• Does any new service call register() with batch=True, an explicit protocol and a full endpoint, or use the shared wrapper?
• Does a new environment variable come from a secret store and not from a committed file?
• Does the change add or alter a project name in a way that breaks the naming convention?
• If it changes auth, roles or keys, is there a test for 401, 200 and 403?
• If it touches the proxy, does it still forward /.well-known/*, leave /mcp and /oauth2/* ungated, and avoid buffering the event stream?
• If it changes retention, who agreed, and is there a record of what is deleted?
• If it enables an agent feature, is there a written decision?
• Does the change include a rollback, and was it tested?
• Does any code use a removed API (px.Client, llm_classify, phoenix.experiments, /v1/evaluations)?

Review checklist for design documents

Use it when someone proposes a new Phoenix deployment or a new integration.

• Requirements. Are volume, payload size, sensitivity, retention and readers stated as numbers?
• Hosting. Is data residency addressed, and is self-hosting justified or ruled out with reasons?
• Tenancy. Does it state that one instance is one tenant, and choose an isolation tier per sensitivity class?
• Database. PostgreSQL 14 or newer, a read replica decision, a backup and restore plan, growth and guardrail numbers.
• Ingest. Sampling, batching, a collector decision, and how loss will be detected.
• Security. Secrets, system keys, SSO and role mapping, TLS, CSRF, egress policy, agent-surface decision.
• Upgrades. A cadence, a rehearsal process, pinned versions, and a plan for the chart lagging the server.
• Governance. Who owns annotation configs, prompt tags and golden datasets, and how evaluation cost is capped.
• Operations. Alerts, a canary, dashboards, on-call, and a service definition.
• Exit. What it would take to leave Phoenix, and whether the OpenTelemetry data path keeps that cheap.
• Non-goals. Does the document say what Phoenix will not do here, such as per-project privacy or acting as an archive?

Habits worth adopting this month

Read the release notes on a schedule. The server ships often and the notes tell you about new surfaces before they surprise you. A fifteen-minute monthly read beats an unplanned incident.

Keep a one-page runbook per failure mode. Missing traces, 503s, disk, failed migration, leaked key, sensitive data. The playbook you wrote calmly is better than the one you improvise at night.

Test the controls by trying to break them. Try the metadata address, the viewer write, the unauthenticated call, the restore. A control you have never seen refuse is untested.

Say what is unverified. When you quote a number or a behaviour from memory, say so, and check the current documentation. Details such as chart defaults and exact metric names change between releases.

Teach the old-to-new table. Most avoidable tickets are people following a tutorial that predates version 14. Link them to your table.

Short checklist

• Database is PostgreSQL, backups restore, datasets and prompts included.
• Auth on, two distinct strong secrets, default admin password changed.
• SSO with role mapping, basic auth off once SSO works.
• Applications use system keys through one shared tracing wrapper.
• batch=True, explicit protocol, full endpoint, sampling where needed.
• Retention set, storage guardrail set, alert below it.
• 503 alert, disk alert, readiness alert, and a span-count canary.
• Egress restricted and proven to refuse the metadata address.
• Agent surfaces decided and written down.
• Images and packages pinned; upgrades rehearsed on a copy; migrations run as a job.
• Service definition published; tenancy tiers explained; key inventory reviewed.

Reading the first ten minutes of an incident

When a page arrives saying "Phoenix is broken", resist the urge to restart anything. Spend the first ten minutes narrowing the layer, because a restart of a stateless server rarely fixes a database problem and can discard spans that were waiting in memory.

First, ask who is affected. One team points at a client configuration; every team points at the server, the database or the network in front of them. Second, ask what changed. Phoenix incidents correlate strongly with a deployment, an upgrade, a key rotation, a retention purge or a new high-volume service, so the change log is usually the fastest diagnostic you have. Third, look at the three cheapest signals: the readiness endpoint, the rate of 503 responses on ingest, and the database disk and connection figures. Fourth, send one canary trace and see whether it arrives, which separates "the server is down" from "one client is misconfigured".

Only then choose an action, and prefer the reversible ones. Asking the heaviest service to sample is reversible. Pausing an evaluation job is reversible. Raising a volume is usually safe. Deleting data, rotating PHOENIX_SECRET or restarting a pod in the middle of a migration is not, and each deserves a second pair of eyes. Write down timestamps as you go, because the postmortem depends on them and memory at four in the morning is unreliable. Finish by confirming recovery with a fresh canary trace rather than with the absence of alerts, since a silent exporter looks identical to a healthy one.