The Complete OpenTelemetry Guide

This is part one of three. It covers everything you need to start doing real work with OpenTelemetry, not a teaser. By the end you can explain what OpenTelemetry is and is not, run a Collector on your laptop, read a config file for it without fear, add tracing to a small Python service without touching its code, add your own spans and metrics by hand, and diagnose the handful of mistakes that stop nearly every beginner's first attempt. Mid-level and Senior take the same topics further; nothing here is thrown away.

A note on versions before we begin. OpenTelemetry is not one program with one version number. It is a specification, a wire protocol, a set of conventions, a program called the Collector, and a separate software development kit (SDK) for every programming language, each released on its own schedule. When this guide gives a version it means the Collector v0.162.0, released at the end of September 2026, alongside version 1.61.0 of the specification and version 1.45.0 of the Python SDK. The Collector is still a 0.x program, which means any release may rename or remove a setting. That is why you will see a few notes below of the form "older tutorials show X, current releases want Y". Trust the current form.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and telemetry only becomes real when you watch your own request turn into a tree of timed spans on your own screen.

What OpenTelemetry is, and the problem it solves

When a program runs on your laptop you understand it by reading the code and adding print statements. When it runs on a server you cannot attach to, serving thousands of users, calling a database, a cache, a payment provider and three other services, print statements stop working. You need the running system to tell you what it is doing. That ability is called observability: the degree to which you can work out what is happening inside a system from the data it emits.

The data comes in a few shapes that we will meet one by one: traces (the story of one request), metrics (numbers measured over time), and logs (timestamped text records). Collectively these are called telemetry. OpenTelemetry, usually shortened to OTel, is the open standard and toolkit for generating, collecting and exporting that telemetry. It belongs to the Cloud Native Computing Foundation, the same foundation that hosts Kubernetes and Prometheus.

The most important sentence in this guide is the next one. OpenTelemetry is not a backend. It does not store your data and it does not draw you a dashboard. It produces telemetry in a standard shape and delivers it to whatever you choose to store and visualise it: Jaeger, Grafana Tempo, Prometheus, Loki, Elasticsearch, or a commercial vendor. Beginners who install OpenTelemetry and then ask "where do I look at my traces?" have missed this. You look at them in a backend you pick separately.

The diagram is the whole architecture. Everything in the next eight thousand words is a detail of one of those four boxes.

What came before

Before OpenTelemetry, every monitoring product shipped its own library. If you wanted Datadog you installed the Datadog agent and its tracing library; if you wanted New Relic you ripped that out and installed theirs; if you preferred open source you used OpenTracing for traces and OpenCensus for metrics, two competing projects that solved overlapping halves of the problem. The consequence was lock-in. Your application code was littered with calls to one vendor's API, so changing vendor meant re-instrumenting every service. Worse, each library invented its own names: one called the HTTP status code http.status, another statusCode, another response_code, which made it impossible to compare data between tools.

OpenTelemetry was formed by merging OpenTracing and OpenCensus. It answers the lock-in problem with two separations. First, it separates how you produce telemetry from where you send it: you instrument your code once against the OpenTelemetry API, and you change the destination by changing configuration, not code. Second, it standardises the names of things. A set of rules called semantic conventions says that an HTTP status code is always http.response.status_code, a database system is always db.system.name, and so on, so data from a Python service and a Java service line up.

The project has a few consequences you should notice now.

Instrumentation and backend are decoupled. You can send the same data to two backends at once, or move from a self-hosted tool to a vendor and back, without a code change. For a team in the Gulf or in Egypt weighing data-residency rules, this matters in a practical way: you can keep telemetry inside a regional cloud or on your own servers today, and change your mind later without touching application code.

There is a lot of it, and it moves quickly. The Collector releases roughly every two weeks. Tutorials from a year ago may show component names that are now deprecated. This guide uses the current names and tells you when an old one is still accepted.

Not everything is equally mature. Traces and metrics are stable in the major languages. Logs are stable in some languages and still in development in others, including Python and JavaScript at the time of writing. Profiles, a fourth signal, are alpha. We will be honest about which parts are solid as we go.

Traces, metrics and logs: the three signals

A signal is a category of telemetry. OpenTelemetry has three you will use daily, plus a newer fourth. Understanding what each is good at tells you which to reach for.

Traces

A trace is the path of a single request through your system. Suppose a user taps "Pay" in a mobile app. The app calls an API gateway, which calls an orders service, which calls an inventory service and a payments service, which each talk to a database. A trace records that whole journey, and because it follows one request it can tell you something no average can: this request was slow, and it was slow here.

A trace is made of spans. A span is one timed operation: "handle POST /pay", "SELECT from orders", "call the payment provider". Every span in a trace shares the same 16-byte trace ID. Each span also has its own 8-byte span ID and, except for the first, the ID of its parent. Follow the parent links and you get a tree. The span with no parent is the root span, usually the first service to receive the request.

POST /pay                                  [=================================] 420 ms
  validate cart                            [==]                                 30 ms
  inventory.reserve                           [=======]                         90 ms
    SELECT stock                                [===]                           40 ms
  payments.charge                                    [========================] 260 ms
    POST https://psp.example/charge                    [====================]   230 ms

Read the picture the way you would read a waterfall in your browser's network tab. Time runs left to right, indentation shows who called whom, and the widest bar near the bottom is the place to look. Here the payment provider took 230 of the 420 milliseconds. Nobody had to guess.

What does a span carry? A name, start and end timestamps, a kind (SERVER when it handles an incoming request, CLIENT when it makes an outgoing one, PRODUCER and CONSUMER for messaging, INTERNAL for work inside one process), a status (Unset, Ok or Error), and three kinds of extra detail. Attributes are key-value pairs that describe the operation, such as http.response.status_code = 200 or order.id = "123". Events are timestamped notes inside the span, such as "cache miss" or an exception with its stack trace. Links point to spans in other traces, for cases like a batch job that processes messages from many different requests.

Metrics

A metric is a measurement aggregated over time: requests per second, the 95th-percentile latency, the number of items in a queue, memory in use. Where a trace answers "what happened to this request?", a metric answers "how is the system doing overall?". Metrics are cheap, because a million requests still become a handful of numbers per minute, which is why alerts and dashboards are built on them.

You create metrics through instruments, and choosing the right instrument is the one piece of metrics vocabulary worth learning now. A counter only goes up: orders processed, bytes sent. An up-down counter can go up and down: items currently in a queue. A histogram records a distribution of values, such as request durations, so you can later ask for percentiles. A gauge records the latest value of something, such as a temperature. Counters, up-down counters and gauges also have asynchronous forms, where instead of calling the instrument when something happens you register a callback that the SDK invokes when it collects data, which suits things like "current memory use" that you read on demand.

Logs as the third signal

A log record is a timestamped message, usually with a severity such as INFO or ERROR, a body, and attributes. Logs are the oldest signal and every application already produces them. OpenTelemetry's contribution is not a new way to write logs. It connects the logging you already have to the rest of your telemetry. Through a bridge or appender attached to your existing logging library (Python's logging, Java's Logback, and so on), log records are emitted in the OpenTelemetry format, and, crucially, each one is stamped with the trace ID and span ID of the request that was running when the line was written. Click from a trace to the log lines from that exact request, or the reverse. That correlation is the payoff.

The fourth signal and the odd one out

Profiles, continuous records of where a program spends CPU time, are the newest signal. They are in alpha, so this guide mentions them and moves on. Baggage is sometimes mistaken for a signal, but it is a mechanism: key-value pairs that travel along with a request from service to service so that, say, a customer tier set at the front door is visible deep in the call chain. Baggage is not added to your spans automatically. You have to copy it into attributes yourself, a trap we return to later.

The mental model: resource, scope, context

Three more nouns give you most of the model. Learn them properly now and the rest of the guide is easy to read.

Resource: who is speaking

A resource is a set of attributes describing the thing that produced the telemetry: the service, the host, the container, the Kubernetes pod, the cloud region. Every span, metric point and log record from one process carries the same resource. The most important attribute in all of OpenTelemetry is service.name. It is how your backend groups data: without it, everything arrives labelled unknown_service (or unknown_service:java, unknown_service:python), and the first thing you notice in a dashboard is a service with that useless name. Other commonly set attributes are service.version, service.namespace, service.instance.id and deployment.environment.name (the current name; the older deployment.environment is replaced).

You set them with two environment variables that every language honours:

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=staging

The second variable takes comma-separated key=value pairs. Resource detectors can also fill in host, container, process and cloud attributes automatically, which is why you rarely type them by hand in production.

Instrumentation scope: which code spoke

The instrumentation scope is the name, and optionally the version, of the library or module that created the telemetry. When your code asks for a tracer you pass a name such as checkout.orders, and every span made through that tracer carries it. This lets you tell a span created by the Flask instrumentation library from one created by your own business code, even though both belong to the same service.

Context and propagation: how a trace survives the network

Inside one process, the context is an immutable carrier that knows which span is currently active. When you start a span inside another, the SDK uses the context to set the parent automatically. You do not pass anything around by hand.

Across a network boundary the context cannot travel by itself, so it is written into the request and read back out on the other side. This is context propagation, performed by a propagator that injects context into a carrier (the HTTP headers of an outgoing request) and extracts it on the receiving end. The default standard is W3C Trace Context, which uses a header called traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

The four dash-separated fields are the version (00), the 32-hex-digit trace ID, the 16-hex-digit ID of the calling span, and two hex digits of flags, where the last bit says "this trace is being sampled". When service B receives that header it creates its spans with the same trace ID and the caller's span as parent. That is the entire mechanism that stitches many services into one trace. If any service in the chain drops the header, the trace splits into two unrelated traces, which is the most common cause of "my trace is broken" questions.

The pieces: API, SDK, OTLP, Collector

The phrase "OpenTelemetry" covers several distinct parts, and confusing them is the next most common beginner stumble. Here they are in the order data flows.

The API is the set of calls your code makes: "start a span", "add one to this counter", "emit this log record". The API is deliberately a no-op until something installs an implementation: without an SDK, the calls do nothing and cost almost nothing. That design is what lets a library author, say the maintainer of an HTTP client, add OpenTelemetry calls without forcing any overhead or dependency on users who do not care. The rule is that libraries depend only on the API.

The SDK is the implementation of the API that your application installs and configures. It decides what happens to a span once it ends: whether it is sampled at all, how it is batched, and which exporter sends it. The SDK is entered through providers (TracerProvider, MeterProvider, LoggerProvider), normally registered globally once at startup.

An exporter serialises telemetry and sends it somewhere. The standard destination is another process speaking OTLP.

OTLP, the OpenTelemetry Protocol, is the wire format: protobuf messages carried over gRPC on port 4317 or over HTTP on port 4318. The HTTP flavour uses the paths /v1/traces, /v1/metrics and /v1/logs. Commit this pair to memory: 4317 is gRPC, 4318 is HTTP. Sending one protocol to the other's port is the number-one first-day bug, and we will meet it again in the errors section.

The Collector is a standalone program that receives telemetry, processes it and exports it onward. It is optional, since an SDK can send straight to a backend, but almost every real setup includes one, for reasons we unpack in the next section. Semantic conventions are not software at all: they are the agreed attribute and metric names that keep data comparable.

And then there is instrumentation, the act of adding telemetry. It comes in three forms worth distinguishing:

• Code-based (manual) instrumentation: you call the API yourself, to create a span around your own business logic.
• Instrumentation libraries: packages that wrap popular frameworks (Flask, Express, a JDBC driver) and emit standard spans for you.
• Zero-code (automatic) instrumentation: an agent that attaches to your program at startup and applies the instrumentation libraries without you editing source. Python has opentelemetry-instrument, Java has a -javaagent JAR, and Node.js has a register hook.

Start with zero-code, add manual spans where the automatic ones do not describe your business, and you will have good coverage with little work.

Why a Collector, and where it sits

If an SDK can send OTLP directly to a backend, why run an extra program? Four reasons, all practical.

First, offloading. The SDK inside your application should finish quickly and return to work. Handing data to a nearby Collector on the same machine is faster than negotiating with a distant backend, and the Collector absorbs the retries, backoff and buffering so that your application does not.

Second, one place for credentials and policy. If every service holds a vendor API key, rotating it means redeploying everything. With a Collector, only the Collector holds the key. The same goes for scrubbing personal data, dropping noisy health-check spans, and adding Kubernetes labels: do it once, centrally.

Third, fan-out and format conversion. One Collector can receive OTLP and send traces to Jaeger, metrics to Prometheus and everything to a vendor at the same time. It can also receive other formats, such as Prometheus scrape targets, Zipkin or Jaeger spans, and convert them.

Fourth, changing your mind later costs a config edit in one place, not a change in every application.

There are three common shapes. With no Collector, the SDK exports directly to the backend: simplest for a demo, weakest for production. As an agent, a Collector runs next to each application, as a sidecar, a Kubernetes DaemonSet or a service on the host. As a gateway, a central pool of Collectors sits behind a load balancer. Many production setups use both, agents close to the applications and a gateway in front of the backends. As a beginner you run one Collector on your laptop and that is precisely the right size.

The Collector's own model

Inside the Collector there are five kinds of component, and once you know them you can read any config in the world.

• A receiver gets data in. Some receivers listen for pushes (the otlp receiver), and some pull or scrape (the prometheus and host_metrics receivers).
• A processor changes data on its way through: batching it, limiting memory, dropping things, adding attributes. Order matters.
• An exporter sends data out.
• A connector is an exporter of one pipeline and a receiver of another, and can change the signal type, for example turning traces into metrics. You will not need one yet.
• An extension provides capabilities that are not part of the data path, such as a health-check endpoint.

Components are wired together in pipelines. There is one pipeline type per signal (traces, metrics, logs), and each pipeline is a list of receivers, then processors, then exporters.

Two rules explain more confusing behaviour than anything else. First, defining a component is not enabling it: a receiver, processor or exporter that appears at the top of the file but is not listed under service.pipelines simply does nothing. Second, each component has an ID of the form type or type/name, for example otlp_grpc/tempo, so that you can have two exporters of the same type pointing at different places.

Installing the Collector and checking it works

There are two official builds you need to know about. The core distribution (otelcol) contains a small set of components. The contrib distribution (otelcol-contrib) contains nearly everything the community maintains, including the file_log, host_metrics, prometheus and k8s_attributes components. For learning, use contrib, so that a component you find in a tutorial is actually present. For production you will eventually build a slimmer custom distribution, which is a Mid-level and Senior topic.

All artifacts live on the collector-releases GitHub page for the version. Replace the version below if a newer one exists, and always pin an exact version; never use latest, because the Collector is 0.x and releases can break configs.

With Docker (any operating system)

This is the quickest route, and the one used for the rest of the guide. Create a file called config.yaml (we write its content in the next section), then run:

docker run --rm \
  -p 127.0.0.1:4317:4317 -p 127.0.0.1:4318:4318 -p 127.0.0.1:13133:13133 \
  -v "$(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.162.0

The -p flags publish the OTLP ports to your machine; the -v flag mounts your file over the image's default config path. The image runs the Collector as its main process, so the terminal is held by its log output. If Docker itself is new to you, the Docker guide covers the flags above.

On Linux, macOS or Windows without Docker

On Debian or Ubuntu, download the .deb package and install it, which also registers a systemd service:

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.162.0/otelcol-contrib_0.162.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.162.0_linux_amd64.deb

The service reads /etc/otelcol-contrib/config.yaml. After you edit that file, run sudo systemctl restart otelcol-contrib, because the Collector does not reload its config from disk on its own, and read its logs with sudo journalctl -u otelcol-contrib. On Red Hat family systems use the .rpm. On macOS there is no official Homebrew formula for the Collector, so download the tarball for your chip and run it directly:

curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.162.0/otelcol-contrib_0.162.0_darwin_arm64.tar.gz
tar -xvf otelcol-contrib_0.162.0_darwin_arm64.tar.gz
./otelcol-contrib --config=config.yaml

If macOS refuses to run the unsigned binary, remove the quarantine flag with xattr -d com.apple.quarantine ./otelcol-contrib. On Windows there are MSI installers that create a Windows service, plus a plain .tar.gz you can unpack and run as .\otelcol-contrib.exe --config=config.yaml.

Checking the installation

The Collector gives you four quick checks before you ever send it data. Use the binary name that matches what you installed.

otelcol-contrib --version
otelcol-contrib components
otelcol-contrib validate --config=config.yaml

--version confirms which build you have. components lists every receiver, processor, exporter, connector and extension compiled into this binary, along with its stability level; it is how you answer "is file_log in this build?" in five seconds. validate parses your config and checks it, exiting non-zero with the reason if it is wrong; run it in the habit of a compiler run, before you start the program. With Docker you can run the same commands by appending them to the image name, for example docker run --rm otel/opentelemetry-collector-contrib:0.162.0 components.

When you finally start the Collector with a valid config, look for this line near the end of the startup log:

Everything is ready. Begin running and processing data.

If you see it, the receivers are listening. On shutdown (Ctrl+C) you will see Shutdown complete.

Your first Collector configuration

The Collector is configured by one YAML file. The structure is a mirror of the model you just learned: a top-level section for each kind of component, then a service section that switches some of them on.

Here is a complete, working first config. Save it as config.yaml.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch: {}

exporters:
  debug:
    verbosity: detailed

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

Read it top to bottom, because every line answers a question.

The otlp receiver opens two listeners, one for gRPC on 4317 and one for HTTP on 4318, bound to 0.0.0.0 so that they are reachable from outside a container. A single receiver handles all three signals on both protocols, which is why the three pipelines below can share it.

The memory_limiter processor is a safety valve. It checks memory use every second, and when the Collector climbs past a soft limit (80 percent of available memory minus the 25 percent spike allowance) it starts refusing new data rather than being killed by the operating system. Two beginner traps live here. Its check_interval has no useful default, so you must set it, and you must set either limit_mib or limit_percentage; leaving them out produces startup errors such as 'check_interval' must be greater than zero. Place it first in every pipeline so that it can protect everything after it.

The batch processor collects data into groups before export, which is far cheaper than sending each span alone. batch: {} means "use the defaults": a batch is sent when it reaches 8192 items or after 200 milliseconds, whichever comes first. Put it last among the processors. One forward-looking note: the project is moving batching into the exporters themselves (a sending_queue with a batch block), and a newer processor called queue_batch exists in development. The batch processor remains supported in this release and is the right thing to learn first.

The debug exporter prints telemetry to the Collector's own log. It is how you see that data arrived. The verbosity levels are basic, normal and detailed, and detailed dumps every attribute. You may find old tutorials using an exporter called logging. That was removed, and using it stops the Collector with the message the logging exporter has been deprecated, use the debug exporter instead.

The health_check extension serves a small HTTP endpoint you can curl, or point a Kubernetes probe at, to ask whether the Collector is up. It is an extension because it is not part of any data pipeline.

Finally service turns things on. The extensions list enables health_check. Each pipeline lists receivers, then processors in execution order, then exporters. If you deleted the service block, nothing above would run.

Start the Collector with this config, then, from another terminal, check it is alive:

curl http://localhost:13133/

A small JSON status document comes back. You now have an idle, healthy Collector. The next step is to give it something to receive.

Sending fake telemetry

The project ships a tool called telemetrygen that generates synthetic data. It is a Go program, so you need Go installed to fetch it:

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
telemetrygen traces --otlp-insecure --traces 3

--otlp-insecure tells it to use plaintext gRPC, matching the plain 4317 listener. In the Collector's terminal you will see the debug exporter print three traces in detail, each with spans, attributes such as network.peer.address, and the resource with the service name telemetrygen. The first time you see a wall of output appear because of a command you typed, the whole idea clicks: a producer sent OTLP to a receiver, a pipeline handled it, an exporter emitted it.

Your first instrumented app, with no code changes

Now the producing side. We will instrument a tiny Python web service using zero-code instrumentation, which means we do not edit the service at all.

Create a folder with a virtual environment and a file called app.py:

from flask import Flask

app = Flask(__name__)


@app.route("/hello/<name>")
def hello(name):
    return {"greeting": f"Hello, {name}"}


@app.route("/slow")
def slow():
    import time
    time.sleep(0.4)
    return {"status": "finally"}


if __name__ == "__main__":
    app.run(port=8080)

Install Flask and the OpenTelemetry tooling:

pip install flask opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

The first package, opentelemetry-distro, is a convenience bundle: it brings the SDK and the opentelemetry-instrument launcher and sets sensible defaults. The second, opentelemetry-exporter-otlp, is the exporter that speaks OTLP. The third command, opentelemetry-bootstrap, is the clever one. It inspects the packages installed in your environment and installs the matching instrumentation library for each: it sees Flask and installs the Flask instrumentation. If you use uv instead of pip, plain -a install misbehaves; use uv run opentelemetry-bootstrap -a requirements | uv add --requirement -.

Now run the service through the launcher, with the environment variables that say who it is and where to send data:

export OTEL_SERVICE_NAME=hello-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
opentelemetry-instrument python app.py

Nothing about app.py changed. opentelemetry-instrument starts Python, loads the SDK and instrumentation before your code runs, and wraps Flask so that every incoming request creates a span. Call it:

curl http://localhost:8080/hello/aya
curl http://localhost:8080/slow

In the Collector's terminal, after a few seconds (the SDK batches and sends on a timer), spans appear. A trimmed version of what the detailed output looks like:

Resource attributes:
     -> service.name: Str(hello-api)
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
ScopeSpans #0
InstrumentationScope opentelemetry.instrumentation.flask 0.66b0
Span #0
    Trace ID       : 4bf92f3577b34da6a3ce929d0e0e4736
    Parent ID      :
    ID             : 00f067aa0ba902b7
    Name           : GET /slow
    Kind           : Server
    Attributes:
         -> http.request.method: Str(GET)
         -> http.response.status_code: Int(200)
         -> url.path: Str(/slow)

Read it with the vocabulary you now own. The resource shows service.name: hello-api, which came from the environment variable. The instrumentation scope names the library that created the span, here the Flask instrumentation. The span has a name (GET /slow), a kind of Server, and attributes using standard semantic-convention keys. The parent ID is empty, so this is the root span of its trace. The exact IDs and version strings will differ on your machine. The structure will not.

What about Java and Node.js?

The pattern is the same; only the launcher differs. For Java you download the agent JAR once and attach it with a flag, leaving your application untouched:

java -javaagent:./opentelemetry-javaagent.jar -Dotel.service.name=hello-api -jar app.jar

For Node.js you install @opentelemetry/api and @opentelemetry/auto-instrumentations-node, then preload the register hook so it loads before your libraries:

npm install --save @opentelemetry/api @opentelemetry/auto-instrumentations-node
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
node app.js

Loading order matters in Node: if your application imports Express before the instrumentation is installed, Express will not be patched and you will see no spans. That is why the flag preloads it.

Adding your own spans and attributes

Automatic instrumentation knows about HTTP requests and database calls. It knows nothing about your business: "validate the cart", "score this loan application", "run this model". To see inside your own code you create spans by hand. This is manual instrumentation, and it layers on top of the automatic kind.

The zero-code launcher has already set up the SDK, so your code needs only the API. A span is created with a tracer, obtained from the global provider, and used as a context manager so that it ends automatically:

from flask import Flask
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

app = Flask(__name__)
tracer = trace.get_tracer("hello.orders", "1.0.0")


def price_for(item):
    prices = {"book": 12, "pen": 2}
    if item not in prices:
        raise KeyError(item)
    return prices[item]


@app.route("/order/<item>")
def order(item):
    with tracer.start_as_current_span("price-order") as span:
        span.set_attribute("order.item", item)
        span.add_event("lookup started")
        try:
            price = price_for(item)
        except KeyError as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "unknown item"))
            return {"error": "unknown item"}, 404
        span.set_attribute("order.price", price)
        return {"item": item, "price": price}

Run it again with opentelemetry-instrument python app.py and call curl localhost:8080/order/book and curl localhost:8080/order/ghost.

Several things are going on, and each one is a habit.

trace.get_tracer("hello.orders", "1.0.0") asks the global provider for a tracer and names the instrumentation scope. Use a stable, dotted name per module or component. start_as_current_span does three jobs at once: it creates the span, makes it the active span in the context, and, because it is a with block, ends it when the block exits, even if an exception flies out. Because the automatic Flask span is already active when order() runs, your new span becomes its child. You never set a parent. The context does it.

set_attribute adds a key-value pair. Prefer the standard semantic-convention names when one exists (http.request.method, db.system.name) and use your own dotted names (order.item) for business data. add_event records a timestamped note inside the span. record_exception attaches an exception, including its type, message and stack trace, as an event, and set_status marks the span as an ERROR so that backends can highlight and filter failed traces. Note that recording an exception does not by itself set the error status, so do both.

A warning about cardinality: an attribute with unbounded distinct values, such as a user ID or full URL with an ID in the path, is fine on a span but dangerous on a metric, where every distinct value creates a new time series. We return to that in the metrics section.

Metrics and logs from your own code

Traces explain single requests. To count things over time, add a meter. As with spans, the API is what your code calls, and the SDK, started by the launcher, does the work.

from opentelemetry import metrics

meter = metrics.get_meter("hello.orders", "1.0.0")

orders = meter.create_counter(
    "orders.processed", unit="{order}", description="Orders processed"
)
latency = meter.create_histogram(
    "orders.duration", unit="s", description="Time to price an order"
)

def record_order(item, seconds):
    orders.add(1, {"order.item": item})
    latency.record(seconds, {"order.item": item})

Each call to orders.add(1, {...}) adds one to the counter for that attribute combination. The unit uses the UCUM convention: s for seconds, {order} for a plain count of things. The histogram stores the distribution of durations, so a backend can compute percentiles.

The SDK does not send every measurement. A periodic metric reader collects the aggregated values and pushes them on an interval. The default is every 60 seconds, set by OTEL_METRIC_EXPORT_INTERVAL in milliseconds. During a demo that feels like nothing is happening, so shorten it:

export OTEL_METRIC_EXPORT_INTERVAL=5000

Two more ideas belong in your vocabulary. Temporality describes whether each exported number is cumulative (the total since the process started, the Prometheus style) or delta (the change since the last export). Cardinality is the number of distinct attribute combinations a metric has. A counter with attribute order.item and ten items has ten series. The same counter with user.id and a million users has a million, which can overwhelm a backend and your bill. Keep metric attributes to small, bounded sets such as item category, HTTP method or status class. The SDK's Views can rename, filter or drop attributes of a metric when a library exposes too many, which Mid-level covers.

Logs from your own code

For logs, the goal is correlation, not a new logging style. Keep using your language's standard logging. A bridge attaches OpenTelemetry to it so each record carries the active trace and span IDs. Logs are the least mature signal in Python and JavaScript at the time of writing: the Python logs module is still marked unstable in its import path (opentelemetry.sdk._logs), and the old LoggingHandler in the SDK is deprecated in favour of the opentelemetry-instrumentation-logging package. For a beginner the correct advice is modest: get traces and metrics working first, then add log correlation as a second step, and expect some API movement. In Java, Go and .NET the logs signal is stable or close to it.

Context propagation and sampling in practice

You met propagation in theory. In practice you rarely write code for it, because the instrumentation libraries do it: an instrumented HTTP client injects traceparent into outgoing requests, and an instrumented server extracts it from incoming ones. What you must do is check that each hop is instrumented and that nothing strips the headers.

You can prove propagation works with two small services. Write a second Flask app on port 8081 whose route calls http://localhost:8080/hello/aya using the requests library (install the matching opentelemetry-instrumentation-requests through opentelemetry-bootstrap, then run both under opentelemetry-instrument with different OTEL_SERVICE_NAME values). Call the second service and inspect the Collector output. The spans from both services carry the same trace ID, and the span of service A's outgoing call is the parent of service B's incoming span. Two processes, one trace.

curl -v http://localhost:8080/hello/aya 2>&1 | grep -i traceparent

Run that against a service that calls you and log incoming headers, and you will see the traceparent value the client attached. The default propagator list is tracecontext,baggage, set via OTEL_PROPAGATORS. Other formats such as b3 exist for systems that predate W3C Trace Context. The Jaeger and OT Trace propagators were deprecated in the specification, so prefer W3C for anything new.

Sampling: keeping only some traces

Recording every trace of a busy system is often too expensive. Sampling decides which traces to keep. There are two places to decide.

Head sampling happens in the SDK when a span starts, before anything is known about how the request will turn out. It is cheap and predictable. The default sampler is parentbased_always_on: record everything, but always follow the parent's decision, so a trace is either entirely kept or entirely dropped across services. To keep roughly ten percent of new traces:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

The parentbased_ prefix is important. It means "if a parent already decided, obey it; otherwise apply the ratio at the root". Without it, every service would roll its own dice and your traces would come back with holes.

Tail sampling happens later, in the Collector, after the whole trace has been seen, so it can decide "keep every trace that contains an error or took longer than two seconds". That is powerful, and it is also stateful and memory-hungry, so it is a Senior topic. For now the rule is simple: head sampling is easy and blind, tail sampling is smart and costly, and on your laptop you need neither.

Configuring the SDK with environment variables

Every OpenTelemetry SDK is configured by the same set of environment variables, defined in the specification, which is why you can move between Python, Java and Node.js and still know what to type. You have met several. Here is the working set a beginner needs, grouped by purpose.

Two details deserve attention. First, the difference between the base endpoint and the per-signal endpoint. With the HTTP protocol, OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 is a base: the SDK appends /v1/traces, /v1/metrics and /v1/logs itself. If instead you set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, that value is used exactly as written, so it must include /v1/traces. Forgetting this gives a 404 from the Collector.

Second, the console exporter is the cheapest way to debug an SDK in isolation. Setting OTEL_TRACES_EXPORTER=console prints spans to the application's own standard output with no Collector involved, which tells you immediately whether the instrumentation is producing spans at all. If the console shows spans but the Collector shows nothing, the problem is the connection. If neither does, the problem is in the instrumentation.

export OTEL_TRACES_EXPORTER=console
opentelemetry-instrument python app.py

Java turns every variable into a system property by lowercasing it and swapping underscores for dots, so OTEL_SERVICE_NAME becomes -Dotel.service.name. You will also meet a newer approach called declarative configuration, a YAML file pointed at by OTEL_CONFIG_FILE. When it is set the SDK ignores the other OTEL_* variables except ones you reference explicitly inside the file. Support differs by language and it is beyond this level, but be aware that the old name OTEL_EXPERIMENTAL_CONFIG_FILE is deprecated.

Everyday Collector commands and configuration

You now know enough to work with the Collector day to day. This section collects the commands and config habits grouped by what you are trying to do.

Start, check and inspect

otelcol-contrib --config=config.yaml
otelcol-contrib validate --config=config.yaml
otelcol-contrib components
otelcol-contrib print-config --config=config.yaml
otelcol-contrib featuregate

print-config shows the final configuration after all merging and environment expansion, which is invaluable when config comes from several places. By default it hides secrets (--mode=redacted); --mode=unredacted shows them, so be careful where you paste the output. Older tutorials and even some docs pages tell you to add --feature-gates=otelcol.printInitialConfig. That gate was removed, and passing it now fails with no such feature gate "otelcol.printInitialConfig". Just run print-config. featuregate lists feature gates, the switches that control experimental and migrating behaviour.

Multiple configs and overrides

--config may be given more than once, and later files are merged over earlier ones. It accepts several URI schemes: file: (or a bare path), env:, yaml:, http: and https:. A classic pattern is one base file plus a small environment-specific override:

otelcol-contrib --config=base.yaml --config=prod-overrides.yaml
otelcol-contrib --config=config.yaml --set exporters::debug::verbosity=basic

--set overrides a single key after merging, using :: to separate nesting levels. A trap: if a later file contains an empty section such as a bare processors:, it is a null value and can wipe out what an earlier file defined. Write processors: {} or omit the key.

Environment variables in config

Inside the YAML you can read environment variables with ${env:NAME} and give a default with ${env:NAME:-fallback}:

exporters:
  otlp_http/vendor:
    endpoint: ${env:BACKEND_URL:-https://otlp.example.com}
    headers:
      authorization: Bearer ${env:BACKEND_TOKEN}

Never paste a real token into the file. Read it from the environment and let your orchestrator supply it. Note that the bare $NAME form is no longer expanded, and a literal dollar sign must be written $$.

Adding a real destination

The otlp_grpc exporter sends OTLP over gRPC to any OTLP-speaking backend, and otlp_http does the same over HTTP. These are the current names. Earlier versions called them otlp and otlphttp; those still work as deprecated aliases and the Collector logs a warning like "otlp" alias is deprecated; use "otlp_grpc" instead at startup. The trap to remember is that the receiver is still called otlp, while only the exporter was renamed.

exporters:
  otlp_grpc/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

Jaeger accepts OTLP natively on 4317 and 4318, so an otlp_grpc exporter pointed at it is all you need. The old dedicated Jaeger exporter is gone. tls.insecure: true disables TLS for a plaintext local connection, which is right for a demo on your laptop and wrong across the internet. Add the exporter to the traces pipeline's exporters list ([debug, otlp_grpc/jaeger]) to fan out to both.

For metrics the prometheus exporter exposes an endpoint that a Prometheus server can scrape:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

Publish port 8889 from your container, add prometheus to the metrics pipeline, and scraping http://localhost:8889/metrics returns your counters and histograms in Prometheus format. The Prometheus guide covers what to do with them next.

Watching the Collector itself

The Collector reports on itself. Its own metrics are served on port 8888 in Prometheus format at http://localhost:8888/metrics, bound to localhost by default. The ones a beginner should learn to spot are otelcol_receiver_accepted_spans (data arrived), otelcol_exporter_sent_spans (data left), and the failure counters otelcol_receiver_refused_* and otelcol_exporter_send_failed_*. If accepted is rising and sent is not, the problem is on the export side.

Sending traces to a real viewer

Reading spans in a terminal teaches the model, but you will want a picture. The lowest-friction way is to run a local Jaeger, an open-source trace viewer, and add it as a second exporter. Start it in Docker, on a shared network with your Collector so that the name jaeger resolves:

docker network create otel-net
docker run -d --rm --name jaeger --network otel-net \
  -p 16686:16686 jaegertracing/all-in-one

Then restart the Collector on the same network with the Jaeger exporter from the previous section in the traces pipeline:

docker run --rm --network otel-net \
  -p 127.0.0.1:4317:4317 -p 127.0.0.1:4318:4318 \
  -v "$(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.162.0

Open http://localhost:16686, choose the service hello-api, and click "Find Traces". You see the waterfall diagram from the start of this guide, now drawn for your own service, with the /slow request's 400 milliseconds as the long bar. Click a span to see its attributes. Nothing in your application changed to make this appear. You added an exporter to the Collector's config. That is the decoupling in action.

The same Collector could send the same spans to Grafana Tempo, to Elasticsearch, or to a commercial vendor by adding another exporter. The Langfuse guide and the Arize Phoenix guide are good next reads if your telemetry comes from LLM applications, since both can ingest OTLP.

Common errors and how to read them

Nearly every first-week problem falls into a small set. Learn to read these messages and you will fix most of them in a minute. Collector errors arrive with a prefix: failed to get config: means the file could not be parsed or resolved, and invalid configuration: means it parsed but does not make sense.

A reference to something undefined. service::pipelines::traces: references exporter "otlp_grpc/tempo" which is not configured. The pipeline names an ID that has no definition at the top of the file. Usually it is a typo in type/name, or you deleted the definition but not the reference. The same message exists for receivers, processors and extensions.

An unknown component type. unknown type: "file_log" for id: "file_log" (valid values: [...]). The component is not compiled into your binary, or its name is misspelled. The usual cause is using the core distribution for something that only exists in contrib. Run components and use the contrib build.

Invalid keys. decoding failed due to the following error(s): ... has invalid keys: flush. A key is misspelled, in the wrong place, or was removed in a newer release. The Collector is 0.x, so tutorials go stale. Check the component's README for the version you run, and check your indentation, which is also the usual cause.

Incomplete memory_limiter. 'check_interval' must be greater than zero or 'limit_mib' or 'limit_percentage' must be greater than zero. The limiter has no usable defaults. Set both, as in the config above.

Nothing pipeline-shaped. service must have at least one pipeline, must have at least one receiver, or must have at least one exporter. Your service block is empty or a pipeline is missing a list.

The removed logging exporter. the logging exporter has been deprecated, use the debug exporter instead. Replace the name.

Address already in use. listen tcp 0.0.0.0:4317: bind: address already in use. Another Collector, a Jaeger container, or an old process holds the port. Find and stop it, or change the endpoint.

A deprecation warning. "otlp" alias is deprecated; use "otlp_grpc" instead. This is a warning, not an error. Your config works. Rename it when convenient. You will see the same pattern for other components renamed to snake_case, such as filelog to file_log and k8sattributes to k8s_attributes.

Connection refused, silently. The Collector looks healthy, your app cannot connect, and nothing is logged. The receiver is bound to localhost inside a container. Set 0.0.0.0:4317.

Protocol and port mismatch. The application logs Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1.00s, or the Collector sees HTTP/2 framing errors, or transport is closing. Sending gRPC to 4318 or HTTP to 4317 fails. Match the protocol to the port, and remember the Python distro defaults to gRPC.

TLS errors on export. tls: first record does not look like a TLS handshake means the exporter is trying TLS against a plaintext server: set tls: { insecure: true } for a local plaintext backend. x509: certificate signed by unknown authority is the reverse, and needs tls.ca_file pointing at the right certificate authority.

A 404 over HTTP. The per-signal endpoint lacks /v1/traces, or /v1/traces was appended to a gRPC endpoint. See the endpoint rule above.

The service name unknown_service. OTEL_SERVICE_NAME is not set in the process that actually runs your app. Check the environment of the process, not your shell.

No Python spans under a pre-fork server. With gunicorn --workers 2 the SDK's background threads do not survive the fork, so telemetry silently vanishes. Initialise the SDK in a post-fork hook. Similarly, Flask's debug=True reloader breaks instrumentation; use use_reloader=False. On slim Docker images, pip builds for instrumentation packages may fail for lack of a compiler; install gcc or build-essential and Python headers.

A Node service with no spans. The SDK loaded after the libraries. Preload it with --require or --import, and do not combine both mechanisms.

Putting it all together

Let us finish with one small end-to-end project that uses everything above. You will build an "orders" service, instrument it both automatically and by hand, run a Collector with a real viewer and a Prometheus endpoint, and confirm the whole path. Think of it as a checklist you can redo in fifteen minutes on any laptop.

1. Make a project folder with app.py (the Flask service with the manual price-order span and the orders.processed counter), config.yaml (the Collector config), and a virtual environment with flask, opentelemetry-distro and opentelemetry-exporter-otlp installed, followed by opentelemetry-bootstrap -a install.

2. Write the Collector config with the otlp receiver on both protocols at 0.0.0.0, the memory_limiter and batch processors, three exporters (debug, otlp_grpc/jaeger, prometheus), the health_check extension, and pipelines that send traces to [debug, otlp_grpc/jaeger] and metrics to [debug, prometheus].

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch: {}

exporters:
  debug: { verbosity: basic }
  otlp_grpc/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889

extensions:
  health_check: { endpoint: 0.0.0.0:13133 }

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug, otlp_grpc/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug, prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

3. Validate, then run Jaeger and the Collector on a shared Docker network, publishing ports 4317, 4318, 8889 and 13133:

otelcol-contrib validate --config=config.yaml
docker network create otel-net
docker run -d --rm --name jaeger --network otel-net -p 16686:16686 jaegertracing/all-in-one
docker run -d --rm --name otelcol --network otel-net \
  -p 127.0.0.1:4317:4317 -p 127.0.0.1:4318:4318 \
  -p 127.0.0.1:8889:8889 -p 127.0.0.1:13133:13133 \
  -v "$(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.162.0
curl http://localhost:13133/

4. Run the service with its identity and a short metric interval:

export OTEL_SERVICE_NAME=orders
export OTEL_RESOURCE_ATTRIBUTES=service.version=0.1.0,deployment.environment.name=dev
export OTEL_METRIC_EXPORT_INTERVAL=5000
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
opentelemetry-instrument python app.py

5. Generate traffic, including a failure:

for i in 1 2 3; do curl -s localhost:8080/order/book; done
curl -s localhost:8080/order/ghost
curl -s localhost:8080/slow

6. Check each stage of the path. Read docker logs otelcol for the debug summaries (data arrived). Open http://localhost:16686 and find the failed orders trace with its exception event (traces flow). Curl http://localhost:8889/metrics and find orders_processed (metrics flow; the Prometheus exporter rewrites the dotted name into Prometheus style). If a stage is empty, you now know which link of the chain to inspect, because you can test each one separately: console exporter for the SDK, debug exporter for the Collector, viewer for the backend.

The exact rewritten metric name and any suffixes depend on the Prometheus exporter's naming rules for your version, so read the /metrics output rather than expecting a particular string.

What you can now do, and what comes next

You can now explain, without notes, that OpenTelemetry is a standard and a toolkit and not a backend; that the data flows from your instrumented code through an SDK and OTLP to a Collector and on to one or more backends; and that traces, metrics and logs are three views of one system joined by trace IDs and resource attributes. You can install and validate a Collector, read its config as receivers, processors and exporters wired into pipelines, instrument a Python service with no code changes, add your own spans with attributes, events and error status, add a counter and a histogram, set the SDK up with environment variables, and read the dozen error messages that account for most first-week pain.

Just as valuable are the habits you have picked up: set OTEL_SERVICE_NAME first, match protocol to port, keep the debug exporter close, put memory_limiter first and batch last, pin the Collector version, never paste secrets into config, and keep metric attributes low-cardinality.

What the Mid-level guide adds is the machinery behind these behaviours and the patterns that hold up in production: how processors, queues and retries actually behave, running the Collector as an agent and gateway in Kubernetes, enriching telemetry with Kubernetes metadata, filtering and transforming with the OpenTelemetry Transformation Language, controlling cardinality with Views, and testing your instrumentation. The Senior guide treats the Collector as a platform: tail sampling, scaling stateful components, multi-tenancy, security, upgrades, and when OpenTelemetry is not the right tool.

Natural neighbours in this catalogue are the Prometheus guide for the metrics side, the Kubernetes guide and the Helm guide for deploying Collectors, and the Langfuse guide and Arize Phoenix guide when your telemetry comes from LLM applications. One useful piece of news for that audience is that the gen_ai.* semantic conventions for LLM calls now live in their own repository, semantic-conventions-genai, separate from the main conventions, so look there rather than in older material.

Sources

• OpenTelemetry documentation
• Concepts
• Traces
• Metrics
• Logs
• Baggage
• Context propagation
• Resources
• Sampling
• Semantic conventions
• Collector
• Collector quick start
• Collector installation with Docker
• Collector installation on Linux
• Collector installation on macOS
• Collector installation on Windows
• Collector configuration
• Collector internal telemetry
• Collector troubleshooting
• Zero-code instrumentation for Python
• Zero-code instrumentation for Java
• Zero-code instrumentation for JavaScript
• SDK general configuration
• OTLP exporter configuration
• Specification environment variables
• OTLP exporter specification
• Semantic conventions specification
• Collector v0.162.0 release notes
• Collector distributions v0.162.0
• Collector core changelog
• Collector contrib changelog
• GenAI semantic conventions repository

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. The aim is not to recite definitions. It is to show that you have run the tool, broken it, and know why.

The thirty-second answer

OpenTelemetry is a vendor-neutral standard and toolkit for generating, collecting and exporting telemetry: traces, metrics and logs. Applications are instrumented once against its API and SDK, send data in a standard protocol called OTLP, usually to a Collector that processes it, and the Collector forwards it to whichever backend you choose. It is not a backend: it stores and shows nothing.

Then the line that shows you have used it: "the most common bugs are a missing service name, a protocol that does not match the port, and a Collector endpoint bound to localhost inside a container."

The model

Two sentences to have ready. First: "Libraries depend only on the API, applications install the SDK." Second: "Defining a Collector component does not enable it. It has to appear in a pipeline under service."

The commands and settings to have ready

You will rarely be asked to type these, but you will be asked what they do. Know them cold.

otelcol-contrib --version
otelcol-contrib components                        # what is compiled into this binary
otelcol-contrib validate --config=config.yaml     # parse and check, exit non-zero if wrong
otelcol-contrib print-config --config=config.yaml # the final merged config, secrets redacted
otelcol-contrib --config=a.yaml --config=b.yaml   # later files merge over earlier ones
otelcol-contrib --config=a.yaml --set exporters::debug::verbosity=basic

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=staging
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc      # or http/protobuf, which uses 4318
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
opentelemetry-instrument python app.py

And a minimal Collector config you can sketch on a whiteboard in under a minute:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
  batch: {}
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [debug] }

Explain it by reading it aloud: one receiver that accepts both protocols, two processors in a deliberate order, one exporter, and a service block that switches it all on. Interviewers like a candidate who states the order rule unprompted: memory_limiter first, batch last.

The classic traps and why they happen

Traps are where interviewers separate people who have read about the tool from people who have run it. For each, know the symptom, the cause and the fix.

The remaining traps, briefly. A broken trace is almost always a dropped traceparent header at some hop, for example a proxy, a queue, or an uninstrumented service, and it fails silently. A Node.js service with no spans loaded the SDK after its libraries, so use --require or --import to load it first. A per-signal endpoint variable such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is used verbatim, so it must include /v1/traces, while the base variable has the path appended for you. Python metrics under pre-fork servers such as gunicorn with several workers vanish because background threads do not survive fork. And sampling too early hides exactly the rare failures you wanted to see.

Common interview questions

What is OpenTelemetry, and what problem does it solve?

It is an open standard and toolkit, under the Cloud Native Computing Foundation, for producing, collecting and exporting telemetry. Before it, each monitoring vendor shipped its own instrumentation library with its own attribute names, so switching vendors meant re-instrumenting every service, and data from different tools could not be compared. OpenTelemetry separates producing telemetry from consuming it. You instrument once against a stable API, you standardise names through semantic conventions, and you choose or change the destination in configuration. A strong answer names the lock-in problem and mentions that it came from merging two earlier projects, OpenTracing and OpenCensus.

Is OpenTelemetry a monitoring tool? Where do I see my data?

No. It generates and ships data, and does not store or visualise it. You send it to a backend such as Jaeger or Grafana Tempo for traces, Prometheus for metrics, Loki or Elasticsearch for logs, or a commercial platform. If a candidate says "I'd open the OpenTelemetry dashboard", that is the tell that they have not used it. A good follow-up is to name your choice and say why: for a local demo you would run Jaeger in Docker.

Explain the three signals and when you would use each.

Traces follow one request across services and show where time went, so you use them to debug a specific slow or failed request. Metrics are cheap aggregated numbers over time, so you use them for dashboards and alerts such as request rate, error rate and latency percentiles. Logs are timestamped records, valuable for detail and context; OpenTelemetry's contribution is to stamp each with trace and span IDs so you can jump between a log line and its trace. Say that they complement each other. Add, if you can, that profiles are a fourth signal still in alpha, and that baggage is a propagation mechanism rather than a signal.

What is a span? What does it contain?

A span is one timed operation in a trace. It has a name, a trace ID shared with its trace, its own span ID, a parent span ID (absent for the root), start and end times, a kind (SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL), a status (Unset, Ok, Error), attributes, events and links. Mention that the immutable part that travels over the network is the span context: trace ID, span ID, flags and trace state.

What is context propagation, and how does a trace cross services?

The calling service injects the trace context into the outgoing request, by default as a W3C traceparent header, and the receiving service extracts it and creates its spans as children with the same trace ID. The header has four parts: version, trace ID, parent span ID and flags, the last carrying the sampled bit. If any hop drops the header, the trace splits. A precise answer also says the propagators are configurable with OTEL_PROPAGATORS, that the default is tracecontext,baggage, and that Jaeger and OT Trace propagators are deprecated.

What is the difference between the API and the SDK?

The API is the interface code calls to create spans, record metrics and emit logs, and it does nothing on its own. The SDK is the implementation that samples, processes, batches and exports. Library authors depend on the API only, so that a library adds no cost or dependency for users who do not use OpenTelemetry, and the application owner decides whether and how to install an SDK. This question is a favourite because it tests design understanding rather than memory.

What is OTLP, and what are ports 4317 and 4318?

OTLP is the OpenTelemetry Protocol, protobuf messages carried over gRPC or HTTP. gRPC listens on 4317. HTTP listens on 4318 with paths /v1/traces, /v1/metrics and /v1/logs, and supports binary protobuf or JSON. The practical point is that the exporter's protocol must match the port.

What is the Collector, and do I need one?

The Collector is a vendor-agnostic program that receives, processes and exports telemetry. You do not strictly need one, because SDKs can export directly to a backend. You want one for offloading work from the application, centralising credentials and policy such as dropping health-check spans or scrubbing personal data, fanning out to multiple backends, converting formats, and changing backends without touching applications. For a small demo, or when a vendor's SDK exporter is all you need, skipping it is acceptable, and saying so shows judgement.

Walk me through a Collector config.

Top-level sections define components: receivers (data in), processors (modify in transit), exporters (data out), connectors (an exporter of one pipeline and a receiver of another), and extensions (capabilities outside the data path, such as a health check). The service section enables them: it lists extensions, and defines pipelines per signal, each a list of receivers, processors and exporters. Processors run in the listed order. Components are identified as type or type/name. Add the trap: defining is not enabling.

In what order should processors go?

memory_limiter first so that it can refuse data before anything expensive happens, then any sampling or filtering so you drop early, then processors that need request context, then enrichment or transformation, and batch last, because batching discards the request context that earlier processors may need. Bonus: the project is moving batching into exporters through sending_queue with a batch block, and a newer processor queue_batch is in development. The batch processor is still supported.

What is the difference between the core and contrib distributions?

Core (otelcol) contains a small, stable set of components. Contrib (otelcol-contrib) contains nearly everything the community maintains. If a config uses a component that is not in your build, you get an unknown type error listing valid values, and otelcol components shows what you have. For production the answer is a custom distribution built with the OpenTelemetry Collector Builder, to include only what you use.

What is zero-code instrumentation?

It attaches instrumentation at startup without editing the application. Python uses opentelemetry-instrument, Java a -javaagent JAR, Node.js a preloaded register module. Each applies instrumentation libraries for frameworks it detects. The limits: it knows about frameworks and I/O, not your business logic, so you add manual spans where you need business detail. Mention OBI, the eBPF-based instrumentation that works at the kernel level for many languages on Linux, as the newest option.

How do you add a custom span in Python?

Get a tracer with trace.get_tracer("my.module", "1.0.0"), then use with tracer.start_as_current_span("name") as span:. Inside, call set_attribute, add_event, and on failure record_exception together with set_status(Status(StatusCode.ERROR, ...)). The span becomes a child of whatever span is active, because the context supplies the parent. Note that recording an exception does not set the error status by itself.

What is service.name, and what happens if you forget it?

It is the resource attribute that names your service, and backends group data by it. If unset, the SDK reports unknown_service (or with a process name suffix). You set it with OTEL_SERVICE_NAME, or in OTEL_RESOURCE_ATTRIBUTES. Mention service.version and deployment.environment.name as the next two to set.

What is sampling? What is the difference between head and tail sampling?

Sampling keeps only some traces to control cost. Head sampling decides at the start of a trace, in the SDK, so it is cheap but blind. Tail sampling decides after the whole trace is seen, in the Collector, so it can keep every error or slow trace, but it needs memory and all spans of a trace on the same instance. The SDK default is parentbased_always_on, and parentbased_traceidratio with an argument samples a fraction while letting downstream services follow the decision. Explain why parentbased matters: without it each service decides independently and traces come back with holes.

What is cardinality, and why do metrics care?

Cardinality is the number of distinct attribute combinations of a metric, and each combination is a separate time series. Putting a user ID on a counter makes a series per user, which can overwhelm the backend and inflate cost. Keep metric attributes bounded, use Views to drop or rename attributes, and keep high-cardinality values on spans where they are cheap.

How would you debug "I see no data in my backend"?

Split the path. First the SDK: set OTEL_TRACES_EXPORTER=console and confirm spans are produced. Then the connection: check the endpoint, the protocol against the port, TLS, and that the Collector is reachable from where the app runs. Then the Collector: read its log, use the debug exporter, and check otelcol_receiver_accepted_spans against otelcol_exporter_sent_spans on the internal metrics port 8888. Then the backend: authentication headers, tenant, and time range. A methodical split beats guessing, and saying so is most of the answer.

What has changed recently that could bite someone following an old tutorial?

Several things, all worth a sentence. The exporter names otlp and otlphttp became otlp_grpc and otlp_http, and many other components moved to snake_case, with old names accepted as deprecated aliases. The logging exporter is gone in favour of debug. Printing the final config is now otelcol print-config, without a feature gate. Endpoints default to localhost. The environment variable for declarative config is OTEL_CONFIG_FILE. And the GenAI semantic conventions moved to their own repository. The Collector is 0.x and releases about every two weeks, so pin versions and read the changelogs.

How does OpenTelemetry relate to Prometheus and Jaeger?

They are backends, or formats it interoperates with, not competitors to the standard. Jaeger accepts OTLP natively, so you send to it with an otlp_grpc exporter. The dedicated Jaeger exporter is gone. Prometheus can scrape the Collector's prometheus exporter, or the Collector's prometheus receiver can scrape targets and convert them to OTLP. OpenTelemetry standardises production and transport, and these tools store and query.

What a strong answer covers that a weak one misses

A weak answer defines terms. A strong answer attaches each term to something that breaks. Asked about context propagation, the weak answer says "it passes IDs between services". The strong answer names traceparent, says what each field is, and says what a broken trace looks like and where to look. Asked about the Collector, the weak answer says "it collects data". The strong answer describes receivers, processors, exporters and pipelines, states that defined is not enabled, and gives the processor order.

Three other habits impress. State the stability honestly: traces and metrics are stable in the major languages, logs are still in development in Python and JavaScript, profiles are alpha. Admit what you would check: "I'd confirm the protocol and port first" is better than a confident guess. Know the current names: an interviewer who sees otlp_grpc and print-config in your answer knows your knowledge is recent.

Sixty-second self-test

Answer aloud, without looking back. Each has a one-sentence answer.

1. Is OpenTelemetry a backend? What do you send its data to?
2. Name the four boxes from code to storage.
3. Which port is gRPC and which is HTTP?
4. What is the one attribute you must always set, and with which variable?
5. What does a traceparent header carry?
6. What is the difference between the API and the SDK, and which do libraries depend on?
7. Name the five kinds of Collector component.
8. What must be true for a component defined in the YAML to actually run?
9. In what position should memory_limiter and batch go in a pipeline?
10. Why does a receiver on localhost fail inside Docker?
11. What is the exporter's current name for OTLP over gRPC, and what was it called before?
12. Name one metric-related cost risk and how you control it.
13. Head or tail sampling: which runs in the SDK, and which can keep all errors?
14. Name the command that shows the final merged Collector config.
15. What three checks split "no data in my backend" into pieces?

If you hesitated on more than three, reread the matching sections of the Detailed track and redo its Try it tasks. Then continue to the Tips and Tricks tab for the day-one habits that prevent these problems in the first place.

Part one of three. Three mistakes produce most of the errors below: you forget that OpenTelemetry is a pipeline with several stages and debug the wrong one, you let the protocol, port and endpoint drift out of agreement, or you trust a tutorial that is a year old in a project that changes every two weeks. Start with the table, then the habits and practice cards.

Common errors at this level

The mistakes that cost the most time

The table is a lookup. Four of those mistakes deserve a longer look, because each one wastes hours when you do not recognise it.

Debugging the wrong stage

Telemetry travels through at least four stages: the application's SDK, the network, the Collector and the backend. When "nothing shows up in the viewer" you have no idea which stage failed, and beginners guess, usually by changing things in the Collector config, which is often the innocent party.

The fix is to test stages separately, from the source outward. First set OTEL_TRACES_EXPORTER=console so the SDK prints spans to the application's own output. If you see spans there, instrumentation works and your problem lies further along. If you see nothing, the trouble is in the application: the instrumentation is not loaded, the SDK is disabled with OTEL_SDK_DISABLED, or the code path did not run. Then switch back to otlp and look at the Collector's log with the debug exporter enabled. If spans show there, the network and receiver are fine. Finally look at the backend: authentication headers, the right tenant, the right time window, and whether the viewer filters by service name. Four stages, four tests, about five minutes, in place of an afternoon of guessing.

Protocol, port and endpoint disagreeing

There are three separate values that must agree, and any one of them can drift. The protocol (grpc or http/protobuf) decides how bytes are framed. The port (4317 or 4318) decides which listener answers. The endpoint form decides whether a path is appended. gRPC to 4317 and HTTP to 4318 are the matched pairs. A mismatch produces confusing errors, such as UNAVAILABLE, transport is closing, or an HTTP/2 framing complaint, that do not say "wrong protocol".

What makes this trap persistent is that language defaults differ. The Python distro defaults to gRPC, whereas the specification default and the Java agent and JavaScript SDK use http/protobuf. A team with a Python service and a Java service will find that the same OTEL_EXPORTER_OTLP_ENDPOINT works for one and not for the other. Be explicit: always set OTEL_EXPORTER_OTLP_PROTOCOL alongside the endpoint, and write both in the same file so they change together. And remember the per-signal variables are used as written, so OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://collector:4318/v1/traces needs the path, while the base variable does not.

Trusting a stale tutorial

The Collector is 0.x software that releases about every two weeks, and its component names have been in flux: exporters otlp and otlphttp became otlp_grpc and otlp_http, and dozens of others moved to snake_case, for example filelog to file_log, hostmetrics to host_metrics and k8sattributes to k8s_attributes. Old names still work as deprecated aliases in most cases, and the Collector logs a warning. One component, dynamic_sampling, became adaptive_tail_sampling with no alias at all, so the old name simply stops working. Some settings were removed outright. Even the official documentation lags GitHub by a release and still shows a feature-gate flag for print-config that no longer exists.

Treat any config you find online as a draft. Run it through validate before anything else, then read the warnings. Look at the date of the tutorial. If it predates this year, expect at least one renamed component. When two sources disagree, the release notes and changelogs of the version you actually run are the referee.

Forgetting the SDK is separate from the Collector

A surprising number of beginners install and configure a Collector, see no data, and wonder why. The Collector does not generate telemetry about your application. It only receives what your application's SDK sends. Without instrumentation nothing arrives. The reverse also happens: an application is instrumented, but the endpoint still points at a default that nothing listens on. Keep the two halves in your head as a pair, a producer and a consumer, each with its own settings, and confirm each half on its own.

The habits that pay off most

Start with zero-code, add manual spans where the business is

Automatic instrumentation gives you HTTP servers and clients, database calls and message queues for free, which covers much of the shape of a request. Add manual spans only where a human would want a name for a step: "score the application", "render the invoice", "call the model". A dozen well-named spans make a better trace than two hundred tiny ones. If every function gets a span, the trace becomes noise and the overhead grows.

Name things the standard way

When a semantic convention exists for what you are describing, use it. http.request.method, http.response.status_code, db.system.name and error.type are understood by every backend and every dashboard template, while method and status are understood by nobody. For your own business data, use a dotted namespace with your domain in front, such as order.item or loan.product. Keep span names low-cardinality, like price-order, and put the variable parts in attributes.

Set error status and record exceptions, both

Two calls, two jobs. record_exception adds the exception as an event with its type, message and stack trace. set_status with an error code is what lets a backend highlight and filter failing traces. Doing only one gives you either a red span with no explanation, or an explanation on a span that looks healthy. Most instrumentation libraries do both for you on unhandled exceptions, but your own try and except blocks swallow the exception, so you must do it yourself.

Leave sampling off until you have a reason

On a laptop, in staging, and in early production, record everything. Learn what your volume actually is, then sample deliberately, and when you do, use a parentbased_ sampler so that all services agree on each trace. Sampling is a cost lever. It is a mistake to pull it before you have a cost.

Read the Collector's own metrics

The Collector reports on itself at http://localhost:8888/metrics in Prometheus format. Three numbers tell you most of what you need: otelcol_receiver_accepted_spans (data arrived), otelcol_exporter_sent_spans (data left), and otelcol_exporter_send_failed_spans (the backend is rejecting or unreachable). If accepted grows and sent does not, look at the exporter and the backend. If accepted does not grow, look upstream.

Protect the Collector's memory from day one

Always include memory_limiter as the first processor, with a real check_interval and a limit, and in containers give the process a memory limit to match. A Collector with no limiter that is hit with a burst will be killed by the operating system, and everything in its in-memory queue disappears with it. On a laptop it is invisible. In production it is an outage.

Practice cards

Five small exercises, each targeted at one habit. Do them in a scratch folder, in the order given.

Habits around the Collector config

A few practices are specific to keeping the YAML healthy as it grows.

Keep one pipeline per signal and name your components. When you have two OTLP exporters, give them names: otlp_grpc/tempo and otlp_grpc/vendor. Unnamed duplicates collide. Names also make the error messages meaningful, since every message quotes the ID.

Comment why, not what. The YAML already says which component is used. The reader three months from now needs to know why memory_limiter is set to 80 percent, or why the filter processor drops /healthz spans. Put the reason in a comment.

Split configuration by concern once it grows. The Collector merges several --config files, so you can keep a base file for receivers and processors and small overlays per environment. Watch for the null trap: a bare processors: key in an overlay is an empty value and can erase the base definition. Write processors: {} or leave the key out.

Keep the stable components and avoid the experimental ones. otelcol-contrib components lists each component's stability. The OTLP receiver and the otlp_grpc and otlp_http exporters are stable for traces, metrics and logs. The batch processor is beta. Some components are in development and can change or disappear. As a beginner, favour the stable ones, and read the stability column before you adopt something new.

Avoid internet-facing receivers. Binding to 0.0.0.0 is necessary in containers, and harmless on a laptop, but the same file on a public host exposes an unauthenticated endpoint that accepts telemetry from anyone. Receivers can be protected by TLS and authentication extensions, which the Mid-level and Senior tracks cover. For now: know which ports your Collector opens and who can reach them.

A note for teams in the region

If you run telemetry for an employer in the Gulf or in Egypt, your first question may be where the data lives. Spans and logs can carry personal or regulated information: usernames, email addresses, account numbers, sometimes the full text of a database query. OpenTelemetry helps because you choose the backend and can run it inside a regional cloud or in your own data centre, and because a Collector can redact or drop sensitive attributes before anything leaves the network. As a beginner, the discipline is small: never put secrets, national ID numbers or full query parameters into attributes on purpose; be aware that some instrumentation libraries capture URLs and query text automatically; and look at real exported spans in the debug output early, so you know what your services are actually sending. The Mid-level and Senior tracks go deeper on scrubbing.

Day-one checklist

Tick these off on the first day of any new OpenTelemetry project.

• I know which version of the Collector I run and I have pinned it, not used latest.
• I installed the contrib distribution for learning, and checked components for what I need.
• Every service sets OTEL_SERVICE_NAME, and ideally service.version and deployment.environment.name.
• The protocol and port agree: gRPC with 4317, HTTP with 4318, and I set both explicitly.
• Collector receiver endpoints use 0.0.0.0 inside containers.
• The pipeline starts with memory_limiter (with a real check_interval) and ends with batch.
• Every component I defined is listed in a pipeline, and validate passes.
• The debug exporter is on, so I can see data reaching the Collector.
• No secret appears in any file: tokens come from ${env:NAME}.
• Metric attributes are bounded, and identifiers live on spans.
• Custom span names are low-cardinality, and variable parts are attributes.
• Sampling is off until I have a measured reason.
• I read at least one exported span in debug output to check what it contains.
• I know where I look when nothing shows up: SDK console, Collector debug, then the backend.

When the list is green you are ready for the Mid-level track, which covers the machinery behind these behaviours: how queues and retries work, the Collector as agent and gateway in Kubernetes, enrichment with Kubernetes metadata, transforming and filtering with OTTL, and testing your instrumentation. The matching guides in this catalogue are the Prometheus guide for the metrics side and the Kubernetes guide for running Collectors in a cluster.

This is part two of three. It assumes you can already get telemetry out of an application and into a Collector, and it takes you to the point where you can predict what the whole pipeline will do before you run it. That prediction is what separates an engineer who "has OpenTelemetry" from one who can be trusted with it in production. The things that go wrong at this level are quiet: traces that break in half at a queue, a metrics bill that triples after one new label, a Collector that is killed by the kernel at three in the morning, a sampling policy that keeps every boring trace and drops every interesting one.

A note on versions before anything else. OpenTelemetry has no single version number. It is a specification plus many separately released implementations. This guide is written against OpenTelemetry Collector v0.162.0 (released 29 September 2026), specification v1.61.0, OTLP v1.11.1 and semantic conventions v1.44.0. The Collector is still a 0.x project, so every minor release, roughly every two weeks, is allowed to break configuration. That single fact shapes a large part of how you run it, and we will come back to it.

Where this picks up

Every section starts with the problem it solves and ends with a Try it you can run against a real service in a few minutes. You will need one Collector you can edit (the contrib distribution, otel/opentelemetry-collector-contrib:0.162.0, is the easiest) and one service in any language that you can restart with environment variables.

The path telemetry takes, end to end

When something is missing from your backend, the first skill is knowing which hop lost it. There are more hops than most people draw on the whiteboard, and each one has its own queue, its own timeout and its own failure mode.

Start at the left. Instrumentation code calls the API. The API is deliberately a no-op when no SDK is installed, which is why a library can depend on it safely: if the application never configures OpenTelemetry, the library's calls cost almost nothing and produce nothing. This is also the reason that "I added tracing to my library and nothing shows up" is the correct behaviour until the application installs an SDK.

The SDK is what the application installs. It owns the providers (a TracerProvider, a MeterProvider and a LoggerProvider), and each provider holds the configuration for one signal. For traces, the provider holds a sampler, which decides at span start whether a span is recorded and exported; a chain of span processors; and a resource, the immutable description of the service. A span processor sits between the span ending and the exporter. There are two kinds that matter. The simple processor exports synchronously on every span end and is only appropriate for debugging. The batch processor queues finished spans and exports them on a timer or when a batch fills, and is what every production setup uses.

The batch processor inside the SDK has four knobs, all of which can be set with environment variables that work in every language:

The log equivalents are the OTEL_BLRP_* variables. Notice what is absent: there is no persistent queue and no unbounded buffer. If the Collector is unreachable and your service produces more than 2,048 spans before the exporter recovers, the SDK drops the excess, and by design it does so without blocking your request path. That is the correct trade for an observability library, because telemetry must never take the application down, but it means the SDK side of the pipeline is lossy under sustained failure. Durable buffering is a job for the Collector, not for the application.

Metrics follow a different shape. A MetricReader collects measurements from the SDK. The usual reader is periodic: it wakes every OTEL_METRIC_EXPORT_INTERVAL (60 seconds by default), gathers every instrument's current aggregation and exports the result. Because it is periodic, a metric you increment once and never touch again is still re-exported every interval, and a spike that lasts five seconds is visible only as whatever that spike did to the aggregation. A Prometheus reader is the pull-based alternative: nothing is pushed, and the aggregation is read when Prometheus scrapes.

Then there is the wire. OTLP is protobuf over gRPC on port 4317 or over HTTP on port 4318, with HTTP paths /v1/traces, /v1/metrics and /v1/logs. The single most common cause of "nothing arrives" remains a mismatch between protocol and port, and it is worth knowing which defaults disagree. The specification default protocol is http/protobuf, and the Java agent 2.x and the JavaScript SDK follow it. The Python opentelemetry-distro sets OTEL_EXPORTER_OTLP_PROTOCOL=grpc. Two services in one cluster, with no configuration differences at all, can therefore disagree about which port to talk to.

Two endpoint rules are worth memorising because they produce silent 404s. With the HTTP protocol, OTEL_EXPORTER_OTLP_ENDPOINT is a base URL and the SDK appends /v1/traces and friends. The per-signal variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is used as written, so it must include the /v1/traces path itself. In code, passing endpoint= to a Python HTTP exporter has the same as-written behaviour.

Finally the Collector, which adds its own receivers, processors, exporters, queues and retry loops, and the backend, which adds ingestion limits, rate limits and its own drop rules. A debugging session is nothing more than walking this chain from left to right and asking, at each hop, "did it arrive here?". Section after section below gives you a way to ask that question precisely.

Context propagation and the traces that break

A trace is only as good as its weakest link, and the links are made by context propagation. Every trace problem at this level, a trace that splits into two, a child that appears as its own root, a missing service in the middle, is a propagation problem until proven otherwise.

Two terms first. The span context is the immutable, propagatable part of a span: the 16-byte trace ID, the 8-byte span ID, the trace flags (which include the sampled bit) and the trace state. The context is the in-process carrier that holds the currently active span and baggage. Inside a process, the context flows implicitly along with the call stack, using language machinery: thread-local storage in Java, contextvars in Python, AsyncLocalStorage in Node.js, an explicit context.Context in Go. Across a process boundary nothing is implicit. A propagator must serialise the span context into a carrier, typically HTTP headers or message metadata, on the client side, and extract it again on the server side.

The default propagator is W3C Trace Context, and its header is worth being able to read by eye:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^
             |  trace id (32 hex)                 parent span id   flags
             version                                               01 = sampled

The default value of OTEL_PROPAGATORS is tracecontext,baggage. Both the caller and the callee must agree on the propagator. If one service still emits B3 headers, which was the norm for Zipkin-era code, and the next expects traceparent, the second service starts a brand-new trace and you will see two unconnected traces where you expected one. The fix is to configure the same list on both sides, for instance OTEL_PROPAGATORS=tracecontext,baggage,b3 while you migrate. The Jaeger and OT Trace propagators were deprecated in specification v1.54.0 and are now optional for SDKs, so prefer W3C for anything new.

Where does propagation fail in practice? There are five repeating patterns.

1. A hop that does not forward headers. A proxy, a gateway or a hand-written HTTP client that copies only a chosen set of headers drops traceparent. Instrumented HTTP client libraries inject it for you; raw sockets and some message brokers do not.
2. Asynchronous boundaries. A queue, a job system or a thread pool that runs work later loses the implicit in-process context. The fix is to capture the span context when you enqueue, put it into the message headers, and extract it in the consumer. Instrumentation libraries for common brokers do this, but a custom task queue needs you to do it by hand.
3. Uninstrumented frameworks. Auto-instrumentation only covers the libraries it knows. A service built on an unsupported HTTP framework appears as a gap.
4. Sampling disagreement. The parent decided "not sampled", the child respects that, and the trace seems to be missing half its services. Section on sampling explains why this is usually correct behaviour.
5. Initialisation order. The SDK must be set up before the libraries it instruments are imported. In Node.js this is the classic failure: load the instrumentation with node --import ./instrumentation.mjs app.js (ESM) or --require (CommonJS), never after the application has already imported http.

Span kind, links, status and errors

A span carries more than a name and a duration, and several of the extra fields are decisions you make when writing manual instrumentation. The span kind is one of INTERNAL, SERVER, CLIENT, PRODUCER or CONSUMER. It is not decorative: backends use it to draw service maps and to decide which span in a pair is the "caller". An outbound HTTP call should be CLIENT, the handler for the inbound call SERVER, and work inside one process INTERNAL. A message being published is PRODUCER, the processing of that message CONSUMER.

A link connects a span to a span in a different trace. The main use is batch processing. A consumer that handles fifty messages in one go cannot have fifty parents, so it starts its own span and adds a link to each producer span. The parent-child relationship expresses "this ran inside that"; a link expresses "this is causally related to that".

Status has three values, Unset, Ok and Error, and the rules are stricter than people expect. Leave a span Unset when it succeeded; instrumentation libraries do this. Set Error when the operation failed. Setting Ok explicitly is a statement that you, the application author, have decided the outcome is final, and it prevents later code from overriding it. On the server side of HTTP, a 4xx response is not an error for the server span, because the server did its job correctly; it is an error for the client span. Get this wrong and your error-rate dashboards will light up every time a user types a bad password.

Recording an exception is a separate action from setting status. In Python:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout.orders", "1.0.0")

def charge(order):
    with tracer.start_as_current_span("charge-card") as span:
        span.set_attribute("order.id", order.id)
        try:
            return gateway.charge(order)
        except gateway.Declined as exc:
            span.record_exception(exc)              # adds an event with exception.* attributes
            span.set_status(Status(StatusCode.ERROR, "card declined"))
            raise

Note that the with block already ends the span for you, and in most SDKs the context manager records an unhandled exception and sets the error status automatically. Doing it explicitly is for cases where you catch the exception and do something else with it, such as returning a fallback value, and still want the failed attempt visible.

Baggage is not attributes

Baggage is a set of key-value pairs that travels with the context, in its own baggage header, across every service in the request. It is useful for passing a tenant ID or a feature-flag cohort deep into a call chain without changing every function signature. The trap is expecting it to show up in your telemetry. It does not. Baggage is not copied onto spans, metrics or logs automatically; you must do that yourself, typically with a baggage span processor that copies chosen keys to span attributes on start. Remember also that baggage is sent in plain text to every downstream service, including third-party ones. Never put a secret or personal data in it.

Metrics that do not hurt you

Metrics have the steepest cost curve of the three signals, because a single careless attribute can multiply the number of time series by ten thousand. The design decisions in this section, instrument choice, temporality and cardinality, determine whether your bill and your query latency are stable.

Choosing the instrument

The instrument you choose defines what the data means, and mistakes cannot be fixed downstream. A Counter only goes up and is for things that happen: requests served, bytes sent. An UpDownCounter goes up and down and is for things that are in a state: queue depth, active connections. A Histogram records a distribution and is the right tool for latency and sizes; you get percentiles, which an average can never give you. A synchronous Gauge records the last value you told it about. Each of these has an asynchronous (observable) counterpart, which takes a callback that the SDK invokes once per collection. Use the asynchronous form when the value is something you read rather than something you record, such as the current size of a cache or the temperature from a sensor. Recording a value on every read would be wasteful and, worse, would make the exported value depend on how often something happened to call your code.

The unit and the name are part of the contract. Use UCUM-style units (s, By, {request}) and follow semantic convention names where they exist, for example http.server.request.duration, which is a histogram in seconds. Counting units in braces, such as {order}, is how you say "a count of things" without inventing a fake unit.

Temporality and aggregation

Temporality is whether each exported data point is cumulative (the total since the process started, as Prometheus expects) or delta (only the change since the last export). Choosing wrong is a classic source of graphs that are off by a factor or reset to zero. Most backends that speak Prometheus want cumulative. Some commercial backends prefer delta because it makes the SDK stateless between exports. The Collector can convert between the two, with cumulative_to_delta and delta_to_cumulative, but the conversion is stateful, which has consequences for scaling that we meet later.

The aggregation chooses how measurements are combined. The default for a histogram is explicit buckets, and the default bucket boundaries are not tuned to your latency. If all your requests fall between 5 ms and 200 ms, the default buckets give you coarse percentiles at exactly the place you care about. You have two remedies: set explicit boundaries appropriate to your service in a view, or use the exponential histogram aggregation, which picks resolution automatically and is much cheaper to get right, provided your backend understands it.

Views and cardinality

A View is SDK configuration that changes how a metric stream is produced: rename it, change its aggregation or buckets, or choose which attributes to keep. Views are your primary defence for cardinality, the number of distinct attribute combinations and therefore distinct time series.

Here is the arithmetic. A request counter with attributes http.request.method (5 values), http.response.status_code (10 values) and http.route (50 routes) produces at most 2,500 series. That is fine. Now somebody adds user.id, with a million users. The series count is 2.5 billion, at which point some backend will reject your data, charge you heavily or fall over. Cardinality explosions rarely come from a single large decision; they come from one extra attribute added during a debugging session that nobody removed.

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation

latency_view = View(
    instrument_name="http.server.request.duration",
    attribute_keys={"http.request.method", "http.route", "http.response.status_code"},
    aggregation=ExplicitBucketHistogramAggregation(
        boundaries=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5]
    ),
)
provider = MeterProvider(views=[latency_view])   # add metric_readers=[...] in real code

The view does two things at once: it keeps only three attributes, so user.id can never reach the exporter even if an instrumentation adds it, and it sets bucket boundaries that resolve the 5 ms to 5 s range. The same idea exists in every language SDK, with different syntax.

The rule of thumb is that metric attributes must be bounded and low-cardinality: methods, routes (the template /orders/{id}, never the raw path /orders/8841), status classes, regions. Identifiers belong on spans and logs, not on metrics. If you need per-user analysis, use traces or logs, which are priced per event rather than per distinct series.

Exemplars connect metrics to traces

An exemplar is a sample measurement stored with a metric data point, carrying the trace ID and span ID that were active when it was recorded. On a latency histogram, the exemplar for the 99th-percentile bucket is a link to an actual slow trace. That is the most useful bridge in observability: you see the spike on a dashboard, click the exemplar, and land in the trace that caused it. Exemplars require that the SDK records them, that they survive the pipeline (the span_metrics connector can produce them with exemplars.enabled), and that the backend and the dashboard tool support them.

Logs and correlation

Logs are the signal you already have. Nobody sets out to replace a working logging setup, so OpenTelemetry's approach to logs is to meet you where you are. The Logs Bridge API exists for appenders, small adapters that connect an existing logging library (log4j, Logback, Python's logging) to OpenTelemetry. As an application developer you normally do not call it directly; you keep writing logger.info(...) and an appender converts each record into an OpenTelemetry log record with a timestamp, an observed timestamp, a severity, a body, attributes and, crucially, the trace ID and span ID of the active span.

That last part is the payoff. When a log line carries the trace ID, your backend can jump from a slow trace to the exact log lines emitted while it ran, and back. Without correlation, logs and traces are two piles of data in the same building.

There are two ways to get correlation, and they serve different needs.

In-process bridging. The SDK's log exporter sends log records over OTLP, with the trace context attached automatically. This is the cleanest route and gives structured attributes end to end. Be aware of maturity: in the language table, Java, .NET, C++ and PHP have stable logs, Go is at release candidate, and Python and JavaScript are still in development for logs. In Python, the SDK's LoggingHandler is deprecated in favour of the opentelemetry-instrumentation-logging package, and the logs module is still spelt opentelemetry.sdk._logs, where the leading underscore is the warning. Treat the Python and Node log APIs as usable but subject to change, and pin versions.

Out-of-process collection. The application writes JSON lines to stdout, the container runtime puts them in files, and a Collector tails them with the file_log receiver. For this to correlate, the application must put the trace and span IDs in each line; the OpenTelemetry logging instrumentation for Python, for example, can inject otelTraceID and otelSpanID into the log format. The Collector then parses them out with an operator and maps them onto the log record.

receivers:
  file_log:
    include: [/var/log/pods/*/*/*.log]
    start_at: end
    operators:
      - type: container          # parses the container runtime format (CRI-O, containerd, Docker)
      - type: json_parser
        parse_from: body
        trace:
          trace_id:
            parse_from: attributes.trace_id
          span_id:
            parse_from: attributes.span_id

Two cautions with this receiver. First, file_log was formerly named filelog; the old name still works as a deprecated alias but logs a warning on startup, and all new configuration should use file_log. The same renaming wave hit most multi-word components, which we return to in the configuration section. Second, log tailing belongs on a DaemonSet in Kubernetes, one Collector per node. Run it as a Deployment with several replicas and each replica reads the same files, so you get duplicated logs.

OpenTelemetry also removed the separate Events API from the specification: an event is now just a log record with an event_name. If you read older material about a span-event API or an events API, new code should emit events through the Logs API instead.

Resources and semantic conventions

A trace with no service name is a trace nobody can use. The resource is the immutable set of attributes that describes the entity producing telemetry, attached once per SDK and shared by every span, metric and log it emits. service.name is by far the most important. If you do not set it the SDK falls back to unknown_service, or unknown_service:java (the process name) in some SDKs, and every service in your organisation collapses into one indistinguishable bucket.

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="service.version=2.4.1,service.namespace=shop,deployment.environment.name=production"

The attributes worth setting on every service are service.name, service.version (what you deployed, which makes "did this start with the release?" answerable), service.namespace (a grouping for related services), service.instance.id (distinguishes replicas) and deployment.environment.name. The last of these is worth a note. An older attribute called deployment.environment was replaced by deployment.environment.name in semantic conventions v1.41, so older tutorials use the wrong key, and dashboards built on the old key will stop matching once SDKs or backends move. During a migration, set both.

Resource detectors fill in attributes automatically. The SDK-side detectors cover host, process, container and cloud. In the Collector, the resource_detection processor does the same with detectors named env, system, docker, ec2, eks, gcp, azure, aks and k8snode, and its override: false default means a value your SDK set explicitly wins over a detected one. Keep that default unless you have a reason. The processor is another that was renamed: it was resourcedetection and the old form is a deprecated alias.

Semantic conventions are the shared vocabulary

Semantic conventions (semconv) define the standard names of attributes, spans and metrics, for example http.request.method, http.response.status_code, url.full, server.address, db.system.name, db.query.text, error.type and the exception.* family. The value of using the standard names is that a dashboard, an alert or a service map written once works for any service in any language. The cost of inventing your own (httpMethod, status, db_type) is that every consumer needs translation.

Stability matters here because names have changed. HTTP conventions became stable in v1.23, the core database conventions for PostgreSQL, MySQL, MariaDB and SQL Server in v1.33, and service.instance.id and service.namespace in v1.40. Where an instrumentation still supports both old and new names, OTEL_SEMCONV_STABILITY_OPT_IN controls what it emits: values such as http and database emit only the new names, while http/dup and database/dup emit both, which is what you want for a migration so that old dashboards keep working while new ones are built.

# Emit old and new HTTP and database names during a migration window
export OTEL_SEMCONV_STABILITY_OPT_IN="http/dup,database/dup"

A schema URL on the telemetry records which version of the conventions the data follows, so that backends can, in principle, translate between versions.

One change matters to the MLOps audience in particular. The GenAI conventions (gen_ai.*, including the OpenAI and MCP ones) moved out of the main semantic-conventions repository into a separate repository, open-telemetry/semantic-conventions-genai, in semconv v1.42.0, and are deprecated in the main one. If you instrument LLM calls and your code or tooling refers to the main-repo version, check which repository a given attribute now lives in. The neighbouring tools in this catalogue, such as Langfuse, Arize Phoenix and Opik, ingest OTLP, so well-named GenAI spans are portable between them.

Sampling: deciding what to keep

Keeping every trace from a busy system is usually unaffordable and, more importantly, unnecessary. Ninety-nine percent of requests are fast and successful and look alike. Sampling is the set of techniques for keeping the traces that matter and a statistically honest slice of the rest. There are two families, and in production you normally use both.

Head sampling in the SDK

Head sampling decides at the start of the trace, before anything is known about how it will end. The sampler sees the trace ID, the span name and its initial attributes, and returns record-and-sample, record-only or drop. The built-in samplers are always_on, always_off, traceidratio and the parentbased_* wrappers. The default is parentbased_always_on.

The parent-based wrappers solve the most important problem in distributed sampling: consistency. If the root service keeps a trace and its downstream services make independent random decisions, most traces will be kept in some services and dropped in others. A parent-based sampler says "if my caller made a decision, follow it (the sampled bit in traceparent); if I am the root, apply my own rule". That is why the standard production setting is:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1     # keep 10% of traces that start here

Head sampling is cheap, because a dropped span costs almost nothing, and it is the only mechanism that reduces load at the source. Its weakness is that it is blind. A 10% ratio keeps 10% of the errors too. If an error happens in one request in a thousand, a 10% sample gives you one trace of every ten errors. For the traces you care about most, that is unacceptable.

One caveat on naming. traceidratio remains a valid value, and the specification restored it with a deprecation timeline in v1.50.0 while new composable, consistent-probability samplers are designed. Use it today, and expect to revisit it.

Tail sampling in the Collector

Tail sampling decides after the whole trace has been seen, so it can apply rules such as "keep anything with an error, keep anything slower than two seconds, and keep 5% of the rest". It is implemented by the tail_sampling processor in the Collector.

processors:
  tail_sampling:
    decision_wait: 30s          # how long to hold a trace's spans before deciding
    num_traces: 50000           # traces held in memory at once
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

The price is state. The processor must buffer every span of a trace until decision_wait expires, so memory is roughly num_traces times the average trace size, and the window times your new-trace rate must fit inside num_traces or it evicts undecided traces. More importantly, it only works if every span of a trace reaches the same Collector instance. With several gateway replicas behind a round-robin load balancer, the spans of one trace are scattered across them, each replica sees a fragment, and the decisions are wrong: fragments of error traces are dropped. The fix is a two-tier architecture. A first tier uses the load_balancing exporter with routing_key: traceID to send all spans of a trace to one member of the second tier, where tail_sampling runs. We build this in the senior level; at this level you need to know that the requirement exists, and that a single-replica tail sampler is a perfectly valid starting point.

Metrics before sampling

A subtle trap: if you derive request counts and latencies from the traces that survive sampling, your numbers are wrong by the sampling factor. The span_metrics connector generates RED metrics (rate, errors, duration) from spans, and because it is a connector it can sit in the traces pipeline before the sampler, so every span is counted even though most are later discarded. Generating metrics from complete data and storing a fraction of the traces is the standard way to keep dashboards accurate and the trace bill low.

The Collector's pipeline model, precisely

The Collector is a single Go binary whose behaviour is defined entirely by one configuration document. You know the five top-level sections. What turns that knowledge into prediction is a handful of rules about how they combine, which the quick start glosses over.

Defined is not enabled. A receiver, processor, exporter, connector or extension is only created if the service section references it. You can define a perfectly valid otlp_grpc/backup exporter and, if no pipeline lists it, it does nothing and the Collector says nothing about it. When a component "does not work", check that it is referenced before you check what it does. The reverse error is loud: a pipeline that names an ID with no definition fails validation with service::pipelines::traces: references exporter "otlp_grpc/tempo" which is not configured, which almost always means a typo in the type/name form.

Receivers and exporters are shared; processors are per pipeline. If two pipelines both list the otlp receiver, there is one receiver instance that fans out to both. If two pipelines both list batch, each gets its own instance, with its own buffer and its own state. That has consequences. Two pipelines that both reference a stateful processor do not share its state, and listing the same processor twice in one pipeline is rejected as references processor "batch" multiple times. To process the same data two ways, use two pipelines.

Pipelines are per signal. A pipeline is named traces, metrics or logs (profiles are alpha), optionally with a suffix: traces/errors, traces/default. Each carries a list of receivers, an ordered list of processors and a list of exporters. The suffix lets you have several pipelines for one signal, which is how you fork traffic.

Connectors bridge pipelines. A connector is an exporter in one pipeline and a receiver in another, and it may change signal type. That is what makes span_metrics possible: it consumes spans as an exporter of the traces pipeline and emits metrics as a receiver of a metrics pipeline.

A configuration that wires a connector looks like this, and it is worth reading slowly because the same component appears in two sections:

connectors:
  span_metrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    dimensions:
      - name: http.request.method
      - name: http.response.status_code
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc/tempo, span_metrics]     # connector as an exporter
    metrics/spanmetrics:
      receivers: [span_metrics]                      # the same connector as a receiver
      processors: [batch]
      exporters: [prometheus_remote_write]

Component names are changing

The second fact you must absorb is that the Collector is mid-way through renaming components to snake_case, tracked in an upstream issue. Tutorials from 2024 and 2025 show the old names, and they mostly still work because the old name is kept as a deprecated alias: the Collector starts and logs a warning of the form "otlp" alias is deprecated; use "otlp_grpc" instead. The renames you will meet most often:

Not everything was renamed. The receiver is still otlp, and debug, batch, memory_limiter, tail_sampling, probabilistic_sampler, health_check, transform, filter, attributes, resource, prometheus, redaction, zpages and file_storage are unchanged. The trap this sets is that the receiver is otlp while the exporter is otlp_grpc, so you cannot rename by search and replace. Two renames have no alias and stop working outright: the dynamic_sampling component became adaptive_tail_sampling in contrib v0.160.0, and queuebatch became queue_batch in core v0.162.0. Finally, only the configuration type changed. Package paths and README directories in the repositories keep the old names, such as processor/k8sattributesprocessor, which confuses anyone searching source code.

Distributions decide which components exist

The binary you run determines which component names are valid. The core distribution, otelcol, is small. The contrib distribution, otelcol-contrib, compiles in nearly everything, including file_log, host_metrics, k8s_attributes, transform, tail_sampling and prometheus. There are also focused builds: otelcol-k8s, otelcol-otlp, otelcol-prometheus and otelcol-ebpf-profiler. If you use a component that is not in your build, the error is unknown type: "file_log" for id: "file_log" (valid values: [...]), and the fix is to use the contrib image or to build a custom distribution with the OpenTelemetry Collector Builder (ocb). List what a binary contains, with stability levels, using:

otelcol-contrib components

The stability column (development, alpha, beta, stable, deprecated, unmaintained) tells you what to expect from upgrades. The OTLP receiver and the OTLP exporters are stable for traces, metrics and logs. batch is beta. tail_sampling and transform are not stable. Do not assume "it ships in contrib" means "it is finished".

Processors: order, memory and batching

Processors run in the order you list them, and the order is not cosmetic. Each processor receives the output of the previous one, and some depend on state that later processors destroy. The core project's processor guidance gives a recommended order, and it is worth understanding each step rather than copying it:

1. memory_limiter first. It must be the first processor so that it can refuse data before anything spends memory on it.
2. Sampling and filtering next. Dropping data early means later processors do less work.
3. Processors that need request context, such as k8s_attributes, before batching. Batching merges data from different requests and discards the per-request context (for example connection metadata) that these processors use.
4. Enrichment and transformation: attributes, resource, transform.
5. batch last, or better, batching in the exporter.

The memory limiter

An overloaded Collector does not slow down gracefully; it buffers until the kernel kills it. The memory_limiter processor is the mechanism that turns that cliff into a controlled refusal. It samples memory use every check_interval and compares it with two thresholds. Above the soft limit, which is the limit minus the spike allowance, it starts refusing incoming data. Above the hard limit it also forces garbage collection.

processors:
  memory_limiter:
    check_interval: 1s            # default is 0s, which is invalid: you must set it
    limit_percentage: 80          # of the available memory (container limit if set)
    spike_limit_percentage: 25    # soft limit is 80 - 25 = 55 percent

Remember that the defaults for these fields are zero, and a limiter without them is a config error ('check_interval' must be greater than zero, 'limit_mib' or 'limit_percentage' must be greater than zero). Pair the processor with the GOMEMLIMIT environment variable set to roughly 80 percent of the hard limit, so that Go's own garbage collector becomes aggressive before the limiter needs to act. The old memory_ballast extension no longer exists; do not follow tutorials that use it.

What does refusal look like? Clients receive a retryable error (gRPC UNAVAILABLE, HTTP 503), the log contains data refused due to high memory usage, and the otelcol_receiver_refused_* metrics rise. Well-behaved SDKs and upstream Collectors retry, which is the point: memory pressure becomes backpressure on the sender rather than an out-of-memory kill of the receiver. Note that the processor's own metrics were renamed otelcol_processor_memory_limiter_* in v0.155.0, and that a separate memory_limiter extension, beta since v0.158.0, can refuse at the network layer before decoding.

Batching, today and tomorrow

Sending one item per request is wasteful: each request has fixed overhead in serialisation, compression and network round trips. Batching groups items. The long-standing way is the batch processor, which is still shipped and still beta:

processors:
  batch:
    send_batch_size: 8192      # flush when this many items are waiting
    timeout: 200ms             # or when this much time has passed
    send_batch_max_size: 0     # 0 means no upper bound on a batch

The direction of travel is to batch inside the exporter, in its sending_queue, because that places batching next to retry and persistence and avoids losing data between two separately buffered stages. Setting sending_queue: { batch: {} } on an exporter gives the same defaults as the old processor. A newer queue_batch processor (renamed from queuebatch in v0.162.0) is in the development stability tier, and the feature gate pkg.exporterhelper.queueBatchEnabled switches the default queue configuration to batch-enabled. The core documentation now says to prefer exporter batching where available. The batch processor is not deprecated yet, so you will see both in the wild, and both are correct; choose one per pipeline and do not stack them.

A multi-tenant caveat: the processor can batch per tenant with metadata_keys, which only works if the receiver is configured with include_metadata: true so that request headers are visible to it.

The exporter helper: queues and retries

Every exporter built on the shared exporter helper exposes the same three mechanisms, and understanding their defaults explains most data loss.

Read the defaults as a story. The backend stalls. Senders fail, retry with backoff and keep the failed requests. Meanwhile new data arrives and fills the in-memory queue, 1,000 requests deep. When the queue is full and block_on_overflow is false, new data is dropped and you see sending queue is full. If the outage outlasts five minutes, requests that have been retrying that long are dropped too. And if the Collector is killed, everything in the in-memory queue disappears with it. None of that is a bug; it is a bounded-memory design. To survive longer outages, you add a persistent queue using the file_storage extension and point the exporter's sending_queue.storage at it, at the cost of disk and latency.

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
exporters:
  otlp_grpc/tempo:
    endpoint: tempo.observability.svc:4317
    tls: { insecure: true }
    retry_on_failure:
      enabled: true
      max_elapsed_time: 600s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      storage: file_storage
      batch: {}
service:
  extensions: [file_storage]

Sizing the queue is a memory decision. A large queue_size times a large average batch can exceed the container's memory and cause the very out-of-memory kill the limiter exists to prevent, so the limiter percentage and the container limit must be chosen together.

Transforming data with OTTL

Real pipelines need to change telemetry in flight: remove a health-check endpoint that floods your backend, drop a sensitive attribute, rename something a legacy service spells wrongly, extract a field from a JSON log. OTTL, the OpenTelemetry Transformation Language, is the expression language used by the transform, filter, routing and other components for exactly this. A statement is a function call that reads and writes paths into the telemetry, optionally followed by a where condition.

The path prefix chooses the context: resource, scope, span, spanevent, metric, datapoint, log and so on. In the current transform processor the context is inferred from the path prefix, so statements can be written as a flat list:

processors:
  transform:
    error_mode: ignore
    trace_statements:
      - set(span.attributes["env"], "prod") where resource.attributes["k8s.namespace.name"] == "prod"
      - delete_key(span.attributes, "http.request.header.authorization")
      - truncate_all(span.attributes, 256)
    log_statements:
      - merge_maps(log.cache, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\{")
      - set(log.attributes["user.hash"], SHA256(log.attributes["user.email"]))
      - delete_key(log.attributes, "user.email")

The functions you will use most are set, delete_key, keep_keys, replace_pattern, truncate_all, limit, merge_maps, ParseJSON, convert_sum_to_gauge, Decode, SHA256 and IsMatch. Note one removal: the Base64Decode converter is gone as of v0.161.0; use Decode(x, "base64").

Error handling and filtering

OTTL statements can fail at runtime, for instance when a function receives a value of the wrong type. error_mode decides what happens: propagate fails the whole batch, ignore logs and continues, silent continues without logging. As of v0.157.0 the default for the transform and filter processors is ignore, and the old feature gates that controlled it have since been removed. That is a safe default for production, where one malformed record should not stop the pipeline, but it means a typo can fail quietly: after writing statements, look at the output, do not assume they ran.

The filter processor drops items for which any condition is true. Its current form uses flat condition lists per signal:

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.name == "/healthz"
      - span.attributes["http.route"] == "/metrics"
    log_conditions:
      - log.severity_number < SEVERITY_NUMBER_WARN

The older nested form, traces: span: [...], logs: log_record: [...], is deprecated in favour of trace_conditions, metric_conditions, log_conditions and profile_conditions. Because the semantics are "drop if true", the classic mistake is writing the condition for the data you want to keep. Read each line aloud as "throw away anything where ...".

Put the filtering where it is cheapest

Dropping at the SDK is cheaper than dropping at an agent, which is cheaper than dropping at a gateway, which is cheaper than paying a vendor to ingest it and then ignoring it. That does not mean the Collector is the wrong place: you often cannot change the application, and a central rule covers every service. A reasonable division is for SDKs to set the sampler and the metric views, for the node-level agent to drop obvious noise such as health checks, and for the gateway to hold policies that need a global view, such as tail sampling.

Deployment patterns: agents, gateways and Kubernetes

Where you run Collectors is an architecture decision with real trade-offs. There are three shapes. With no Collector, SDKs export straight to the backend: simple, and fine for a prototype, but every application then holds vendor credentials, every application carries retry and buffering behaviour, and changing a backend means redeploying applications. An agent is a Collector next to the workload: a DaemonSet, a sidecar or a host service. A gateway is a central tier of Collectors behind a load balancer. Most production estates use an agent and a gateway together, and the division of labour is consistent.

Applications export to a node-local address, so that a slow or down backend never sits on the request path. In Kubernetes the usual trick is the downward API, which lets a pod learn its node's IP:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://$(NODE_IP):4318
  - name: OTEL_SERVICE_NAME
    value: checkout

For that to work, the agent must listen on the node address. This leads to one of the most common "it works on my laptop" failures. Every server endpoint in the Collector, the OTLP ports 4317 and 4318, health_check on 13133, zpages, defaults to localhost, not 0.0.0.0. Inside a container that means reachable only from within the container itself. There is no error: clients just report "connection refused". The official distribution configs set 0.0.0.0 explicitly, and so should yours when you must accept traffic from outside the pod, combined with a network policy to keep it from being open to the world. The internal metrics endpoint on 8888 also defaults to 127.0.0.1.

Installing with Helm

The Helm chart is the most direct way to run Collectors on Kubernetes. The mode value is required, and is one of daemonset, deployment or statefulset.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install otel-agent open-telemetry/opentelemetry-collector \
  --set mode=daemonset \
  --set image.repository=otel/opentelemetry-collector-k8s \
  --values agent-values.yaml

The chart offers presets that switch on commonly needed components together with the RBAC they need: presets.logsCollection.enabled (tails container logs, which requires the DaemonSet mode), presets.kubernetesAttributes.enabled, presets.hostMetrics.enabled, presets.kubeletMetrics.enabled, presets.clusterMetrics.enabled and presets.kubernetesEvents.enabled. Use a preset when its behaviour is what you want, and read the rendered ConfigMap with helm template before you rely on it. See Helm for the templating and release mechanics this builds on.

Attributes from Kubernetes

Telemetry from a pod is far more useful once it carries the namespace, pod, deployment and node it came from. The k8s_attributes processor looks these up through the Kubernetes API and attaches them as resource attributes. It needs a service account with permission to get, watch and list pods, namespaces and replica sets, and it must sit before batching, as discussed above.

mode: daemonset
config:
  processors:
    k8s_attributes:
      auth_type: serviceAccount
      passthrough: false
      extract:
        metadata:
          - k8s.namespace.name
          - k8s.pod.name
          - k8s.deployment.name
          - k8s.node.name
      pod_association:
        - sources:
            - from: resource_attribute
              name: k8s.pod.ip
            - from: connection
  service:
    pipelines:
      traces:
        processors: [memory_limiter, k8s_attributes, batch]

The pod_association rules say how the processor works out which pod a piece of telemetry came from. The fallback from: connection uses the source IP of the incoming connection, which only works when the agent receives traffic directly from the pod. Put a gateway or a service mesh in front and the source IP is no longer the pod's, so you must have the SDK or agent set k8s.pod.ip explicitly. Associating wrongly is worse than not associating: the telemetry gets another pod's labels.

The Operator

The OpenTelemetry Operator manages Collectors and instrumentation with Kubernetes custom resources. It needs cert-manager for its admission webhooks, unless you install it with the Helm chart and let the chart generate certificates. The main resources are OpenTelemetryCollector (API version opentelemetry.io/v1beta1, whose spec.config is an ordinary YAML object, and spec.mode is deployment, daemonset, statefulset or sidecar) and Instrumentation (opentelemetry.io/v1alpha1).

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: gateway
  namespace: observability
spec:
  mode: deployment
  replicas: 2
  config:
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
    processors:
      memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
      batch: {}
    exporters:
      otlp_grpc/tempo:
        endpoint: tempo.observability.svc:4317
        tls: { insecure: true }
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp_grpc/tempo]

The Instrumentation resource is how you get zero-code instrumentation without touching application images: it defines the exporter endpoint, propagators and sampler, and pods opt in with an annotation such as instrumentation.opentelemetry.io/inject-python: "true" (or inject-java, inject-nodejs, inject-dotnet; inject-go uses eBPF and needs extra privileges). The annotation value can be "true", an instrumentation name, or "<namespace>/<name>".

The most common failure is placement. The annotation belongs on the pod template, spec.template.metadata.annotations, not on the Deployment's own metadata. Forget that, or create the Instrumentation in a different namespace, or fail to restart the pods after creating it (injection happens at pod creation), and nothing is injected and nothing says so.

For a platform that already uses GitOps, these resources are just manifests, and they belong in the same repository and review flow as the rest: see Argo CD and Kubernetes.

Managing configuration through a fast release cycle

A configuration that worked yesterday can be invalid today, because the Collector releases about every two weeks and every minor release may remove or rename things. Managing that is a process problem, not a typing problem.

Layering

The --config flag can be repeated, and later configs are merged over earlier ones. It accepts more than file paths, through URI schemes: file:, env:, yaml:, http: and https:. A bare path means file:. --set overrides individual keys after merging, using :: for nesting.

otelcol-contrib \
  --config=file:base.yaml \
  --config=file:env/production.yaml \
  --set "exporters::otlp_grpc/tempo::endpoint=tempo-prod:4317"

This gives you the natural structure: a base file with components shared by every environment, and a small overlay per environment that changes endpoints and adds exporters. A trap lives in merging. An empty processors: key in an overlay is a null map and wipes what the base file defined; write processors: {} or omit the key. Lists are replaced, not merged, so an overlay that sets service::pipelines::traces::exporters replaces the whole list, it does not append.

Values can come from the environment with ${env:VAR}, with a default as ${env:VAR:-default}. A literal dollar sign is $$, and the bare $VAR form is no longer expanded. This matters in the Prometheus receiver's relabel rules, where $1 looks like a variable: write $$1. Another provider, ${file:path}, embeds the contents of a file, which suits certificates and tokens mounted as files.

Secrets never go inline. Write headers: { authorization: "Bearer ${env:BACKEND_TOKEN}" } and deliver the variable from a Kubernetes Secret. Fields that hold secrets are typed as opaque in the Collector and are redacted in print-config output and in logs.

Validating before you deploy

Two commands belong in every pipeline that changes Collector configuration.

otelcol-contrib validate --config=file:base.yaml --config=file:env/production.yaml
otelcol-contrib print-config --config=file:base.yaml --config=file:env/production.yaml > resolved.yaml

validate exits non-zero on error. print-config shows the fully resolved configuration after merging and environment expansion; by default with secrets redacted (--mode=redacted). This command replaced the older print-initial-config, which was a feature-gated flag. Do not copy the old instruction to pass --feature-gates=otelcol.printInitialConfig: that gate was removed in v0.155.0, and passing it now fails with no such feature gate "otelcol.printInitialConfig". Just run print-config.

Running validate in CI against the exact image version you will deploy catches the deprecated-and-removed class of error before production. Run it in the same image, not on a laptop binary of a different version, for the reason that valid keys differ between releases. The errors you will see are the wrapper failed to get config: ... for parse and resolution failures and invalid configuration: ... for semantic validation, and a message such as '<path>' has invalid keys: <key> points at a key that was removed, misspelled or mis-indented.

Upgrading on purpose

A safe routine for a two-week cadence looks like this.

1. Pin an exact version. Use 0.162.0, never latest.
2. Read both changelogs (core and contrib) for every version you skip, focusing on the "Breaking changes" sections, removed feature gates and renamed components.
3. Run validate and print-config in CI with the new image against every environment's overlay.
4. Run the new version in a non-production environment with real traffic and compare the Collector's own metrics before and after. Metric names change too: units became singular ({request} instead of {requests}) in v0.148.0, the batch-size histograms moved in v0.159.0, and the :8888 endpoint emits underscore labels such as service_name, so a dashboard can break without the Collector itself misbehaving.
5. Roll out gradually, one tier or one cluster at a time.

If you build a custom distribution, all component versions in the builder manifest must equal the core version, and recent contrib needs Go 1.26 or newer.

Feature gates are the Collector's mechanism for staging behaviour changes. List them with otelcol-contrib featuregate, toggle them with --feature-gates=some.gate,-other.gate, and treat an alpha gate as an experiment, not a dependency. A gate that has reached stable cannot be disabled, and a later release deletes it, at which point any command line still mentioning it fails.

Observing the Collector itself

A Collector is itself a production service that can be overloaded, misconfigured or starved, and it is frequently the component nobody monitors until data goes missing. It emits its own metrics, logs and optionally traces, and these are your primary debugging instrument.

Internal telemetry

Configure it in the service::telemetry section. This section uses the SDK's declarative configuration schema, so its keys are the SDK's, and one detail trips people up: its exporters are called otlp with a protocol: key, not the Collector's otlp_grpc or otlp_http.

service:
  telemetry:
    logs:
      level: INFO
      encoding: json
    metrics:
      level: normal          # none | basic | normal | detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Metrics are exposed in Prometheus format on port 8888, and scraping that endpoint is the most common setup. Alternatively, a periodic reader pushes them over OTLP to a backend; sending the Collector's own health to the pipeline it is monitoring is circular, so prefer a separate path. The older single metrics::address key was replaced by readers. Internal traces are experimental.

The signals to watch

Two items from that table deserve emphasis. First, the decision to scale: if the exporter queue is full because the backend is the bottleneck, adding Collectors does nothing, and may worsen the backend's load. Second, after any upgrade re-check these names, because they have changed more than once.

Extensions for live inspection

Three extensions turn the Collector into something you can interrogate:

• health_check (port 13133) serves an HTTP endpoint for Kubernetes liveness and readiness probes. A newer healthcheckv2 also exists.
• zpages (port 55679) serves live pages such as /debug/tracez for span statistics and /debug/pipelinez for the pipeline structure. It is the fastest way to see what is actually running.
• pprof (port 1777) exposes Go profiling data for CPU and memory investigations. Do not expose it beyond localhost.

A debugging playbook, hop by hop

When telemetry is missing, resist the urge to read configuration and guess. Follow the path, and at each hop ask one question that has a definite answer.

Hop 1, the SDK: is it producing anything? Switch an exporter to console (set OTEL_TRACES_EXPORTER=console), or turn up SDK logging with OTEL_LOG_LEVEL. If nothing prints, the problem is before the export: the SDK is disabled (OTEL_SDK_DISABLED=true), the sampler is dropping (always_off, a ratio of zero, or a parent that decided not to sample), the instrumentation loaded too late, or the service name is wrong and you are looking for the wrong thing. For the Java agent, -Dotel.javaagent.debug=true is verbose but decisive.

Hop 2, the connection: does the export reach the Collector? The SDK logs its own failures, and they are readable once you know the shapes. Python logs Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1.00s, which means the Collector is not running, unreachable or the wrong protocol. UNIMPLEMENTED usually means gRPC was sent to an HTTP-only endpoint. Test the path independently of your application: curl -i http://collector:4318/v1/traces returning a response, even a 405, proves the port is reachable, and telemetrygen isolates the Collector from your code.

Hop 3, the receiver: did the Collector accept it? Check otelcol_receiver_accepted_* and otelcol_receiver_refused_*. Accepted and rising means the problem lies downstream of the receiver. Refused means the memory limiter. Flat at zero means nothing arrived, so go back to hop 2. A listen tcp 0.0.0.0:4317: bind: address already in use at startup means another process, often a second Collector or an all-in-one Jaeger container, already holds the port.

Hop 4, the processors: did something drop or rewrite it? Add a second debug exporter and, temporarily, place a copy of the pipeline without the suspect processor. Compare what the debug exporter prints on each. A filter with an over-broad condition, a tail_sampling policy that never matches, and a transform whose where clause excludes everything all look identical from outside: data went in, none came out. The processor's otelcol_processor_incoming_items and outgoing_items metrics show exactly where the count falls.

Hop 5, the exporter: did it leave? otelcol_exporter_sent_* against send_failed_* and enqueue_failed_*, and the log line Exporting failed. Will retry the request after interval. with the underlying error. TLS messages are the common ones: tls: first record does not look like a TLS handshake means a TLS client talking to a plaintext server, so set tls: { insecure: true } for plaintext or configure the CA; x509: certificate signed by unknown authority is the reverse and needs tls.ca_file. An HTTP 404 from otlp_http means a wrong path, typically a per-signal endpoint missing /v1/traces.

Hop 6, the backend: did it keep it? Many backends drop silently for reasons of their own: rate limits, maximum attribute lengths, out-of-order timestamps, unknown tenants, or a field validation failure. The Collector sees a successful send and the data is still absent. The backend's own ingestion logs and metrics are the only witnesses, which is why checking the destination's rejection counters belongs in your runbook.

The debug exporter is the tool for hops 3 to 5. At verbosity: basic it prints counts, at normal a one-line summary per item, and at detailed every attribute. Turn it on temporarily, in its own pipeline position, and remove it afterwards: at detailed it writes large volumes of logs and can itself cause load.

Integrating with the rest of the stack

OpenTelemetry generates and moves telemetry; it does not store or display it. Every real deployment connects it to backends, and the integrations are mostly a matter of knowing which component speaks which protocol.

Prometheus and metrics. There are three integration directions. The prometheus receiver scrapes Prometheus endpoints using ordinary scrape_configs under config:, which lets the Collector replace a scraper. The prometheus exporter exposes metrics on an endpoint (8889 by convention) for Prometheus to scrape. The prometheus_remote_write exporter pushes to a remote-write receiver such as Mimir, Thanos or Cortex. Prometheus 3.x can also ingest OTLP directly at /api/v1/otlp/v1/metrics when its OTLP receiver is enabled. Whichever you choose, names are translated and dots in attribute names become underscores in label names unless the backend supports UTF-8 names, so a dashboard query written against http.request.method may need http_request_method. See Prometheus and Grafana for the query side.

A scraping receiver has a scaling caveat that the stateless components do not: you cannot simply run two replicas, because both scrape the same targets and you get duplicates. On Kubernetes the Operator's Target Allocator, running with a StatefulSet collector, distributes scrape targets between replicas.

Traces. Jaeger accepts OTLP natively, so send it otlp_grpc and forget the long-gone Jaeger exporter. Grafana Tempo also takes OTLP. Zipkin-era clients can be received with the zipkin receiver. Only the jaeger receiver remains in the Collector, for legacy clients.

Logs. Loki has a native OTLP endpoint under /otlp, reached with an otlp_http exporter. The Elasticsearch exporter and the ELK stack are another common destination, covered in ELK Stack. If you already have Fluent Bit or Fluentd, a fluentforward receiver accepts their output.

Vendor backends and distributions. Because OTLP is a standard, switching or duplicating to a vendor is a matter of adding an exporter with endpoint and headers, and there are vendor-built distributions such as Grafana Alloy or the AWS distribution if you want a supported build. The discipline that makes switching cheap is exporting OTLP from applications and keeping backend-specific knowledge in the Collector.

LLM applications. For a team that runs models in production, OpenTelemetry is increasingly the transport for LLM observability. Tools such as Langfuse, Arize Phoenix and Opik ingest OTLP, and instrumentation packages exist in the language contrib repositories for OpenAI and other client libraries. Record prompt and completion text only with care: those are exactly the personal and confidential payloads the redaction section warns about, and large payloads are expensive to store. For the GenAI attribute names, see the separate semantic-conventions-genai repository mentioned earlier.

Data residency. For teams in the Gulf and Egypt, where regulation or customer contracts often require that data stay in-region, the Collector is where you enforce it: run the gateway in the regional cluster, export only to in-region backends, and use a routing connector or separate pipelines to keep tenants' telemetry apart. Sending raw OTLP from every service to a vendor endpoint in another region is the easiest way to violate a residency promise without noticing.

Testing your instrumentation

Instrumentation is code, and code that has no tests rots. The failure mode is specific: telemetry silently stops being emitted, or changes shape, and nobody notices until an incident when the graph is empty. There are three levels of test, and a team needs at least the first two.

Unit tests with an in-memory exporter. Every SDK ships a testing exporter that keeps finished spans in memory so a test can assert on them. You configure a tracer provider with a simple processor and the in-memory exporter, run the code, and inspect the spans. This is the right place to assert that a span exists, has the correct name and kind, carries the required attributes, records an exception on failure and does not contain a forbidden attribute.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import StatusCode

def test_declined_card_is_an_error_span():
    exporter = InMemorySpanExporter()
    provider = TracerProvider()
    provider.add_span_processor(SimpleSpanProcessor(exporter))
    # inject provider.get_tracer(...) into the code under test

    try:
        checkout.charge(declined_order, tracer=provider.get_tracer("test"))
    except Exception:
        pass

    spans = exporter.get_finished_spans()
    charge = next(s for s in spans if s.name == "charge-card")
    assert charge.status.status_code == StatusCode.ERROR
    assert charge.attributes["order.id"] == declined_order.id
    assert "card.number" not in charge.attributes

Make the tracer injectable, as above, or configure a global provider once in a test fixture. Setting the global provider repeatedly within one process is ignored by most SDKs, which produces tests that pass alone and fail together.

Pipeline tests with a real Collector. Start the Collector in the test environment with a file or debug exporter, send known data, and assert on the output. This is where you test configuration: that the filter drops health checks, that the transform removes the sensitive attribute, that the redaction pattern matches. Because the Collector takes configuration from the command line, running the same pinned image you deploy, with the production base file and a test overlay, tests the real thing. telemetrygen generates synthetic traces, metrics and logs for this purpose.

Contract tests in staging. A synthetic request through a staging environment, followed by a query for its trace in the backend, verifies the whole path including propagation across services. One or two such tests, run on every deploy, catch the regression where a new proxy strips the traceparent header.

Security, privacy and cost

Telemetry is data about your systems and your users, and it leaves your network. Two concerns follow, and they conveniently share a tool set: what must not be sent, and what is not worth sending.

Keeping sensitive data out

Instrumentations capture more than you might expect: URL query strings, HTTP headers, SQL text in db.query.text, exception messages that quote user input. The semantic conventions say to sanitise query parameters, but it is your responsibility to check. Defence is layered.

1. At the source, configure instrumentations to capture nothing you do not need and never put personal data in span or log attributes.
2. In the Collector, strip what slipped through. The redaction processor can allow only listed attribute keys, block values matching regular expressions (card numbers, bearer tokens) and emit a summary of what it touched. The attributes processor can delete or hash, and transform can replace_pattern or hash with SHA256. The filter processor can drop whole items.
3. At the edge of the network, give receivers TLS and authentication, and set max_recv_msg_size_mib and memory limits to resist abuse.

processors:
  redaction:
    allow_all_keys: false
    allowed_keys: [http.request.method, http.response.status_code, http.route, service.name, order.id]
    blocked_values:
      - "4[0-9]{12}(?:[0-9]{3})?"      # card-number-like values
      - "Bearer\\s+[A-Za-z0-9._-]+"
    summary: info

An allow-list is stronger than a block-list, because a block-list only stops what you thought of. Its cost is that a new legitimate attribute is dropped until you add it, which is the failure you want.

Secure the network surface too. Bind endpoints to specific interfaces, use TLS or mutual TLS on receivers, add an authentication extension such as bearertokenauth or oidc for anything crossing a trust boundary, run as a non-root user and give Kubernetes receivers least-privilege RBAC. Since TLS defaults tightened in v0.154.0, insecure cipher suites are off unless you opt in with include_insecure_cipher_suites, and you should not.

Making telemetry affordable

The main cost levers, in approximately descending order of effect, are these. First, sampling traces in the SDK and, at a gateway, tail sampling. Second, dropping noise with filter: health checks, debug-level logs, chatty internal spans. Third, cardinality control for metrics with views, keep_keys or delete_key statements, and a bounded dimensions list on span_metrics. Fourth, compression: the gRPC exporter defaults to gzip and zstd is available. Fifth, the Collector's own metrics level, since level: detailed produces many more series than basic. Measure before you act: the Collector's own accepted-item counters, per pipeline, tell you which signal dominates, and within a signal, a backend's usage by service tells you which service is responsible.

Putting it all together

Here is one coherent design that uses everything above. A checkout service and an orders service run on Kubernetes. The goal: traces that stay whole, accurate RED metrics, an affordable trace volume, no personal data leaving the cluster, and a pipeline that tells you when it is sick.

In each service, set the resource and the protocol explicitly, use parent-based head sampling, and export to the node-local agent:

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES="service.version=2.4.1,service.namespace=shop,deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://$(NODE_IP):4318
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.25
export OTEL_PROPAGATORS=tracecontext,baggage

In the agent (a DaemonSet), enrich and forward, with the limiter first and batching last:

receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
  k8s_attributes:
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  filter:
    error_mode: ignore
    trace_conditions: ['span.name == "/healthz"']
  batch: {}
exporters:
  otlp_grpc/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls: { insecure: true }
extensions:
  health_check: { endpoint: 0.0.0.0:13133 }
service:
  extensions: [health_check]
  telemetry:
    metrics:
      level: normal
      readers:
        - pull: { exporter: { prometheus: { host: 0.0.0.0, port: 8888 } } }
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes, filter, batch]
      exporters: [otlp_grpc/gateway]

In the gateway, generate metrics first, redact, sample, and export with a persistent queue. A single replica keeps tail sampling correct until you introduce trace-aware routing:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 25 }
  batch: {}
  redaction:
    allow_all_keys: true
    blocked_values: ["Bearer\\s+[A-Za-z0-9._-]+"]
  tail_sampling:
    decision_wait: 20s
    num_traces: 50000
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow, type: latency, latency: { threshold_ms: 1500 } }
      - { name: rest, type: probabilistic, probabilistic: { sampling_percentage: 10 } }
extensions:
  file_storage: { directory: /var/lib/otelcol/storage }
exporters:
  otlp_grpc/tempo:
    endpoint: tempo.observability.svc:4317
    tls: { insecure: true }
    sending_queue: { enabled: true, queue_size: 5000, storage: file_storage, batch: {} }
  prometheus_remote_write:
    endpoint: http://mimir.observability.svc/api/v1/push

The pipelines are where the design lives. Tail sampling is a processor, so to give one stream two treatments, counted in full for metrics and sampled for storage, you use two trace pipelines joined by the forward connector, plus span_metrics to feed a metrics pipeline:

connectors:
  span_metrics:
    dimensions:
      - name: http.request.method
      - name: http.route
  forward: {}
service:
  extensions: [file_storage]
  pipelines:
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [span_metrics, forward]       # every span is counted, and every span is forwarded
    traces/sampled:
      receivers: [forward]
      processors: [tail_sampling, batch]
      exporters: [otlp_grpc/tempo]
    metrics/spanmetrics:
      receivers: [span_metrics]
      processors: [batch]
      exporters: [prometheus_remote_write]

Trace the data through it. Every span is received, limited, redacted and then handed to two connectors: span_metrics counts every span, before any sampling, and forward passes the same spans to a second pipeline where tail_sampling keeps errors, slow traces and a tenth of the rest. Metrics are therefore accurate, traces are affordable, and credentials are scrubbed before either leaves. The persistent queue on the Tempo exporter carries the gateway through a backend outage.

Around all of it, validate and print-config run in CI with the pinned 0.162.0 image, the agent's and gateway's :8888 metrics feed alerts on refused data and queue depth, and a unit test asserts that no authorisation header reaches a span.

Where you are now

You can now reason about telemetry as a pipeline with named hops, and predict how each one fails. You know that context propagation is the thing that makes traces whole, and that asynchronous boundaries and mismatched propagators are where it breaks. You can keep metric cardinality bounded with views, choose between head and tail sampling and explain why accurate metrics must be generated before sampling, and you understand that the Collector's pipelines instantiate processors per pipeline, that defined is not enabled, and that the receiver otlp and the exporter otlp_grpc are not the same name. You can size the memory limiter and the queue together, keep configuration valid across a two-week release cycle with pinned versions and CI validation, watch the Collector with its own metrics, walk a debugging path from SDK to backend, and keep sensitive data and cost under control.

What comes next is the senior level: running the Collector as a platform for other teams. That covers two-tier trace-aware routing for tail sampling, scaling stateful components, multi-tenancy, custom distributions with the builder, the trust model and upgrade governance. Before you move on, it is worth going back to the beginner guide if any term in this one felt unfamiliar, and reading the neighbouring guides for the systems OpenTelemetry feeds: Prometheus, Grafana and Kubernetes.

Sources

• OpenTelemetry documentation
• Traces concepts
• Metrics concepts
• Logs concepts
• Baggage
• Context propagation
• Resources
• Sampling
• Semantic conventions
• Collector documentation
• Collector configuration
• Collector internal telemetry
• Collector troubleshooting
• Collector scaling
• Collector resiliency
• Collector deployment patterns
• Collector distributions
• Collector install on Kubernetes
• Security: configuration best practices
• Security: handling sensitive data
• SDK configuration: general
• SDK configuration: OTLP exporter
• Zero-code instrumentation for Python
• Kubernetes Operator
• Operator automatic instrumentation
• Operator Target Allocator
• Helm chart for the Collector
• Environment variable specification
• Collector v0.162.0 release notes
• Collector contrib v0.162.0 release notes
• Collector changelog
• Collector contrib changelog
• Processor ordering guidance
• Memory limiter processor
• Exporter helper
• Transform processor
• Filter processor
• Tail sampling processor
• GenAI semantic conventions repository

Part two of three. A cumulative review of Beginner and Mid-level material for someone who runs OpenTelemetry at work. The questions here are not "what is a span". They are the ones an interviewer asks to find out whether you have been paged because of telemetry, and whether you understood why. Expect scenarios ("traces are broken, what do you check?"), trade-offs ("head or tail sampling?") and the occasional trap built on an old tutorial. Senior adds platform ownership on top.

The facts in this file are checked against OpenTelemetry Collector v0.162.0 (29 September 2026), specification v1.61.0 and semantic conventions v1.44.0. OpenTelemetry has no single version number, and saying so early is itself a good sign in an interview.

Where this picks up

The two-minute model

If you can say this cleanly, the rest of the interview goes easier.

OpenTelemetry is a vendor-neutral standard for generating, collecting and exporting traces, metrics and logs. It is an API and SDK per language, a wire protocol (OTLP), naming conventions, and a Collector that receives, processes and exports telemetry. It is not a backend: it stores nothing and draws nothing. You send the data to Tempo, Jaeger, Prometheus, Loki or a vendor.

Three sentences to add when they let you keep talking. The API is a no-op until an SDK is installed, so libraries may depend on it safely. The Collector is optional but is where you put credentials, redaction, sampling and routing so that applications stay simple. And the Collector is a 0.x project that can break configuration every two weeks, so you pin versions and validate in CI.

Scenario questions

"Traces from service B show up as separate traces instead of children of service A's. What do you check?"

A strong answer walks through context propagation, because a split trace is nearly always a propagation failure, and names the candidates in order of likelihood.

First, are the two services using the same propagator? The default, OTEL_PROPAGATORS=tracecontext,baggage, sends a traceparent header. If one service uses B3 or an older Jaeger header and the other expects W3C, the second starts a fresh trace. Second, does anything between them drop headers: a proxy, a gateway, a hand-rolled HTTP client that copies only selected headers? Capture a request on the wire or at the receiving service and look for traceparent. Third, is the boundary asynchronous? A message queue, a background job or a thread pool loses the in-process context, so the producer must put the span context into the message headers and the consumer must extract it. Fourth, is the receiving side instrumented at all, and did the SDK initialise before the instrumented library was imported? That is the usual Node.js failure, fixed by loading instrumentation with node --import or --require. Fifth, sampling: if the first service decided not to sample and the second does not honour the parent's decision, you can get fragments.

What a weak answer misses: it jumps to "restart the Collector" or "check the backend", neither of which can split a trace, or it says "add more instrumentation" without identifying the boundary. It also usually forgets asynchronous boundaries, which are the most common cause in real systems.

"Nothing arrives in the backend after you deploy a new service. Walk me through it."

The strong answer is a bisection, and it names a check at each hop rather than a guess.

1. SDK: is it producing? Switch to the console exporter or raise SDK log level. If nothing prints: is OTEL_SDK_DISABLED set, is the sampler dropping everything, is instrumentation loaded too late, is OTEL_SERVICE_NAME missing so the data is filed under unknown_service?
2. Connection: does the export reach the Collector? Check protocol and port together: gRPC is 4317, HTTP is 4318, and the Python distro defaults to gRPC while the spec default, Java and JavaScript use http/protobuf. Look for StatusCode.UNAVAILABLE or UNIMPLEMENTED in the SDK logs. Test with telemetrygen to take the application out of the picture.
3. Receiver: otelcol_receiver_accepted_* rising? otelcol_receiver_refused_* would mean the memory limiter. In a container, is the endpoint 0.0.0.0:4317 rather than the default localhost?
4. Processors: a filter or a transform where clause that matches everything removes everything. Compare a debug exporter before and after.
5. Exporter: otelcol_exporter_sent_* against send_failed_*, plus TLS and path errors.
6. Backend: rate limits, tenant headers, and validation drops that the Collector cannot see.

What separates the answers: naming the localhost default and the protocol-port mismatch shows you have lived through both. Saying "I would bisect with a debug exporter in two positions" shows a method. A weak answer reads the whole config top to bottom hoping to spot the problem.

"The Collector pod keeps getting OOMKilled. What is going on, and what do you change?"

A Collector without a memory limiter buffers without bound when something downstream is slow, and the kernel eventually kills it. The strong answer has three parts: diagnose, fix, prevent.

Diagnose. Look at otelcol_process_memory_rss, the exporter queue metrics and whether memory_limiter is present at all. Check the queue configuration: queue_size multiplied by average batch size can exceed the container limit by itself. Check whether a stateful processor such as tail_sampling is holding num_traces traces in memory, which is roughly num_traces times the average trace size.

Fix. Put memory_limiter first in every pipeline with check_interval: 1s (its default is zero, which is invalid) and a percentage limit relative to the container limit, with a spike allowance so the soft limit sits below the hard one. Set GOMEMLIMIT to roughly 80 percent of the hard limit so the garbage collector works harder before the limiter has to act. Right-size the queue. The old memory_ballast extension is gone; if the interviewer mentions it, say so.

Prevent. Refusal makes clients see UNAVAILABLE or HTTP 503 and retry, which is the desired backpressure. Alert on otelcol_receiver_refused_* and on queue utilisation above 60 to 70 percent, and scale out only if the bottleneck is the Collector, not the backend.

"Your metrics bill tripled last week. How do you find out why and stop it?"

Cardinality. The answer should include the arithmetic: series count is the product of the distinct values of each attribute, so a counter with method (5), status code (10) and route (50) makes at most 2,500 series, and one added user.id with a million values makes billions. Find it by looking at which metric and which attribute grew: the backend's per-metric series count, or a debug exporter on a metrics pipeline to see data points and their attributes.

The fix has layers. In the SDK, a view restricts the attribute keys of the metric, so a bad attribute cannot be exported even if an instrumentation adds it. In the Collector, a transform statement with delete_key or keep_keys does the same for services you cannot change. On span_metrics, a bounded dimensions list. And the policy: metric attributes must be bounded, such as route templates rather than raw paths, and identifiers belong on spans and logs. A strong candidate also mentions an upstream process: a review of any new metric attribute.

What a weak answer does: suggests sampling. Sampling applies to traces and logs; it does nothing for metric cardinality.

"A pod has the annotation for auto-instrumentation but nothing is injected."

This is an Operator question, and the answer is a short checklist: the annotation must be on the pod template (spec.template.metadata.annotations), not the Deployment's own metadata; an Instrumentation resource must exist in that namespace (or be referenced as namespace/name); the pods must have been recreated after it was created, because injection happens at admission; the annotation must match the language (inject-python, inject-java, inject-nodejs, inject-dotnet); and the Operator's webhook must be working, which requires cert-manager unless the Helm chart generates certificates. The telltale error for a broken webhook is failed calling webhook "mopentelemetrycollectorbeta.kb.io" with an x509 message.

"Your config worked on the last release and now the Collector will not start."

Expect this one at any company running the Collector, because it is the most likely real problem. A strong answer knows the project is 0.x with releases about every two weeks and that minor releases may break things. The diagnosis is to read the exact error: '<path>' has invalid keys: <key> points at a key that was removed or misspelled, unknown type: "..." at a component that was renamed or is absent from the distribution, and a no such feature gate error at a gate that was stabilised and deleted.

Name real examples. print-initial-config was replaced by otelcol print-config, and the gate otelcol.printInitialConfig was removed in v0.155.0. The exporters otlp and otlphttp are now otlp_grpc and otlp_http, and many other components moved to snake_case with deprecated aliases that warn on startup. dynamic_sampling became adaptive_tail_sampling with no alias. The logging exporter is long gone and debug replaced it. Kafka auth options were removed in v0.160.0.

Then the process answer, which is what the interviewer is really after: pin an exact version, read both the core and contrib changelogs for every version skipped, run validate and print-config in CI with the exact image, roll out gradually and compare the Collector's own metrics before and after.

"We need every error trace and every slow trace, but only a small slice of the rest. How?"

Tail sampling, and the answer must include the constraint that makes it hard. The tail_sampling processor buffers a trace for decision_wait and then applies policies, for example keep if status is error, keep if latency exceeds two seconds, otherwise probabilistic at 5 percent. The constraint is that all spans of a trace must reach the same Collector instance. Behind a round-robin balancer, a trace's spans scatter over replicas and each makes a decision on a fragment. The solution is a two-tier design: a first tier with the load_balancing exporter using routing_key: traceID, sending each trace to one member of a second tier that runs tail_sampling. Mention memory: roughly num_traces times the average trace size.

A strong candidate then says what they would also do: parent-based head sampling in the SDK to shed bulk cheaply, and span_metrics before the sampler so request rates stay accurate. Without that, rates derived from sampled traces are wrong by the sampling factor.

Trade-off questions

Head sampling versus tail sampling

The mature answer is "both": head sampling reduces the load everywhere, tail sampling protects the interesting traces. The parent-based wrapper, parentbased_traceidratio, is essential because it makes downstream services follow the root's decision.

Direct export versus a Collector

Direct export is simpler with fewer moving parts and is fine for a prototype. A Collector adds a hop and an operational burden, and buys you: credentials in one place, redaction and sampling without redeploying applications, buffering and retry that survive a backend outage, enrichment with node and Kubernetes metadata, and the freedom to change or duplicate backends. The answer should lean on the fact that SDK-side buffering is small and lossy by design (a 2,048-span queue, drop rather than block), so anything requiring durability belongs in the Collector. A fair counterpoint: for a single small service, a Collector is overhead, and saying so shows judgement.

Agent, gateway or both

An agent (DaemonSet, sidecar, host service) is close to the data: it enriches with metadata, tails logs, collects host metrics and offloads the application quickly. A gateway is central: authentication, redaction, tail sampling, routing and backend credentials. Most production estates use both. The cost of the two-tier design is latency, more components to upgrade and a second place for data to be lost, which is why resilience settings go on each network hop. A strong answer notes where each thing belongs: log tailing on the agent in DaemonSet mode (a Deployment with replicas duplicates logs), tail sampling on the gateway.

Batching in the processor versus in the exporter

The batch processor is beta and still shipped. The project's direction is exporter-side batching through sending_queue::batch, because it keeps batching beside retry and persistence. A newer queue_batch processor, renamed from queuebatch in v0.162.0, is still in development. The right answer is pragmatic: both work today, pick one per pipeline and never stack them, put batch last if you use the processor, and expect to migrate. Mentioning that batch is not deprecated yet, despite common claims, is a detail that interviewers notice.

Cumulative versus delta temporality

Cumulative reports the total since process start and suits Prometheus-style backends, which compute rates on query. Delta reports the change since the last export, which makes the SDK stateless between exports and suits some vendors. Wrong temporality shows up as graphs that reset or are off by a factor. The Collector can convert with cumulative_to_delta and delta_to_cumulative, but the conversions are stateful, so they need all data for a series on one instance. A good answer connects this to scaling.

Zero-code versus manual instrumentation

Zero-code (the Java agent, opentelemetry-instrument for Python, the Node register hook, .NET auto-instrumentation, the Operator's injection, or eBPF with OBI) gives breadth fast and needs no code change, but it only knows frameworks it recognises and cannot see your business logic. Manual instrumentation gives you meaning: spans for your domain operations, attributes like order.id, correct span kinds and status. The standard answer is to start with zero-code and add manual spans where the business matters. Mention the failure modes of each: auto-instrumentation has initialisation-order and unsupported-framework gaps, and manual instrumentation drifts without tests.

What a strong answer covers that a weak one misses

Quick-fire facts to have ready

• Ports: 4317 OTLP gRPC, 4318 OTLP HTTP, 8888 the Collector's own metrics, 13133 health_check, 55679 zpages, 1777 pprof, 8889 the conventional prometheus exporter port.
• HTTP paths: /v1/traces, /v1/metrics, /v1/logs.
• Endpoint variables: the general one is a base URL that the SDK extends with /v1/traces for HTTP. The per-signal one is used verbatim.
• Default sampler: parentbased_always_on.
• Default propagators: tracecontext,baggage.
• SDK batch defaults: 5,000 ms delay, queue 2,048, batch 512, export timeout 30,000 ms.
• Metric export interval: 60,000 ms.
• Exporter helper defaults: timeout 5s, queue 1,000 requests with 10 consumers, retries up to 300s.
• memory_limiter: check_interval and one of the limits are mandatory.
• Collector commands: validate, components, print-config, featuregate.
• Env expansion: ${env:VAR} and ${env:VAR:-default}; bare $VAR is no longer expanded; $$ is a literal dollar.
• Declarative SDK config: OTEL_CONFIG_FILE (the older OTEL_EXPERIMENTAL_CONFIG_FILE is deprecated), and with a file set, other OTEL_* variables are ignored except those referenced inside the file.
• Environment attribute: deployment.environment.name (earlier deployment.environment).
• GenAI conventions live in a separate repository now.

Questions you will be asked, with model answers

Why does the Collector's OTLP receiver work from my laptop but not from another pod? The default endpoint is localhost, so the receiver is reachable only from inside its own network namespace. Set endpoint: 0.0.0.0:4317 (or the pod IP) in containers, and protect it with a network policy. There is no error message, only "connection refused" from clients.

What is the difference between a processor and a connector? A processor modifies data within a pipeline and keeps its signal type. A connector is an exporter in one pipeline and a receiver in another and can change the signal type, which is how span_metrics turns traces into metrics. forward is the simplest, joining two pipelines of the same signal.

If two pipelines list batch, do they share it? No. Each pipeline gets its own instance of a processor. Receivers and exporters are shared between pipelines. Listing the same processor twice in one pipeline is a validation error.

What does it mean that a component is "defined but not enabled"? It appears in a top-level section but no pipeline or service::extensions references it, so it is never created. Nothing warns you.

How do you keep a secret out of the Collector config? Reference an environment variable, ${env:BACKEND_TOKEN}, populated from a Kubernetes Secret, or use a file provider. Secret-typed fields are redacted in print-config and logs. For SDKs, prefer a local Collector holding the vendor key over putting it in every application.

How would you migrate semantic conventions without breaking dashboards? Set OTEL_SEMCONV_STABILITY_OPT_IN to a /dup value such as http/dup or database/dup so the instrumentation emits old and new names together, build the new dashboards, then switch to the new names only.

Why might an exemplar not appear on my histogram? Either the SDK is not recording them, the pipeline dropped them (for derived metrics, exemplars.enabled on span_metrics), or the backend or dashboard does not support them. Exemplars link a sample to a trace ID, so the trace must also have been kept, which ties back to sampling.

What happens to metrics under gunicorn with several workers? Background export threads do not survive fork, so metrics break. Initialise the SDK in a post_fork hook or use another multiprocess approach. The same family of problem: Flask's debug=True reloader breaks instrumentation, so use use_reloader=False.

Can you rely on Python and JavaScript logs being stable? Not yet. For logs, both are listed as in development, Go is at release candidate, and Java, .NET, C++ and PHP are stable. Python's LoggingHandler in the SDK is deprecated in favour of the opentelemetry-instrumentation-logging package. Pin versions and expect change.

Should I use the Jaeger exporter? No. The Jaeger exporter was deprecated in 2023; Jaeger accepts OTLP natively, so use otlp_grpc. The Collector still has a jaeger receiver for legacy clients, and the Jaeger propagator was deprecated in spec v1.54.0.

Sixty-second self-test

Answer each aloud in a sentence or two. If you hesitate, reread the matching section of the detailed guide.

1. Name the five hops between an application and a backend, and one failure signal for each.
2. Why does the traceparent header split a trace if the propagators differ?
3. A queue sits between two services. How do you keep the trace whole?
4. What is the arithmetic of metric cardinality, and which three tools contain it?
5. Why must memory_limiter be first, and why must k8s_attributes come before batch?
6. What does a pipeline do with a processor referenced in two pipelines?
7. State the two-tier design required for tail_sampling and why.
8. Which is the receiver and which is the exporter: otlp or otlp_grpc?
9. What are the defaults of the exporter queue, and when does data get dropped?
10. What replaced print-initial-config, and why should you not use the old gate?
11. How do you generate accurate request rates and still sample traces?
12. Where does a health check span belong: dropped at the SDK, the agent or the gateway, and why?

Part two of three. At this level your telemetry flows and your dashboards draw, so the problems change character. They are no longer "why does nothing arrive" but "why do half my traces have no children", "why did the Collector restart overnight", "why is this metric costing more than the service it describes" and "why did an upgrade break a config that hadn't changed". Start with the error table, then the practices in order of leverage, then the cards and recipes you can keep open in a terminal.

Everything here is checked against OpenTelemetry Collector v0.162.0 (29 September 2026), specification v1.61.0 and semantic conventions v1.44.0. OpenTelemetry moves fast, and many tutorials from 2024 and 2025 show component names and flags that have since been renamed or removed, so when in doubt, run otelcol-contrib components and validate on the version you actually have.

Common errors at this level

Cumulative: Beginner's errors still apply. These are the ones that appear once things work.

Each of these is covered in the detailed guide. The pattern across the table is worth seeing: the quiet failures, defined-but-unused components, localhost binding and ignored OTTL errors, are more damaging than the loud ones, because nothing tells you to look.

The practices that pay off most, in order of leverage

The list is ordered by how much pain each removes per hour spent. If you can only do three, do the first three.

1. Pin the Collector version and validate in CI

The Collector is 0.x and releases about every two weeks, and any minor release may break configuration. Pin an exact version (otel/opentelemetry-collector-contrib:0.162.0) and never use latest. In CI, run validate and print-config for every environment's configuration with the exact image you will deploy. This single habit removes the largest class of self-inflicted outages.

docker run --rm -v "$PWD/config:/cfg" otel/opentelemetry-collector-contrib:0.162.0 \
  validate --config=file:/cfg/base.yaml --config=file:/cfg/production.yaml

2. Always put memory_limiter first and set GOMEMLIMIT

A Collector without a limiter dies the first time a backend stalls. The limiter turns memory exhaustion into refusals that senders retry. Give it check_interval: 1s, a percentage limit relative to the container's limit, a spike allowance, and set GOMEMLIMIT to about 80 percent of the hard limit.

3. Set the resource on every service

service.name, service.version, service.namespace and deployment.environment.name on every service, from environment variables in the deployment manifest. Without them, nothing can be filtered, grouped or compared, and the first incident will be spent working out which service a trace belongs to.

4. Use parent-based sampling and sample in the right place

OTEL_TRACES_SAMPLER=parentbased_traceidratio with an argument makes downstream services follow the root decision, so traces are whole or absent. Add tail sampling at a gateway only when you need to keep every error and slow trace, and remember that it needs all spans of a trace on one instance.

5. Generate RED metrics before sampling

Put span_metrics in the traces pipeline ahead of the sampler, so rates and latencies count every request while you store a fraction of traces. Derive request counts from sampled traces and they will be wrong by the sampling factor.

6. Bound every metric attribute

Views in the SDK, delete_key or keep_keys in a Collector transform, and a finite dimensions list on span_metrics. Route templates, not raw paths. Identifiers on spans and logs, never on metrics.

7. Monitor the Collector as a service

Scrape :8888/metrics and alert on otelcol_receiver_refused_*, otelcol_exporter_send_failed_*, otelcol_exporter_enqueue_failed_*, and queue utilisation above 60 to 70 percent. Add health_check for probes and zpages for live inspection.

8. Put durability on the network hops

The default queue is 1,000 requests in memory with retries for up to five minutes. For a gateway, add a persistent queue with the file_storage extension, or put Kafka between tiers. Resilience belongs on agent to gateway and gateway to backend.

9. Redact with an allow-list, and test for absence

Decide what may leave, use the redaction processor with allowed_keys on the gateway, and write tests asserting that authorisation headers, emails and card numbers never appear on spans, logs or metrics.

10. Move to current component names as you touch a config

otlp_grpc, otlp_http, k8s_attributes, file_log, host_metrics, resource_detection, span_metrics, prometheus_remote_write, load_balancing. Do it in the same change where you see the alias warning, because an alias is a removal in waiting.

Practice cards

Operational recipes

These are short, runnable procedures. Each one solves a problem you will meet in the first year of running OpenTelemetry.

Recipe: prove the Collector is alive and receiving

curl -fsS http://localhost:13133/                                   # health_check extension
curl -s http://localhost:8888/metrics | grep -E 'otelcol_receiver_(accepted|refused)'
telemetrygen traces --otlp-insecure --traces 5                      # synthetic data, bypasses your app
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_spans

The accepted counter should rise by five. If it does not, the problem is the receiver or the network. If it does, the problem is in your application's export or further down the pipeline.

Recipe: see what a pipeline actually does to your data

exporters:
  debug/before: { verbosity: detailed }
  debug/after:  { verbosity: detailed }
service:
  pipelines:
    traces/tap:                       # receiver only, for comparison
      receivers: [otlp]
      exporters: [debug/before]
    traces:                           # the real pipeline
      receivers: [otlp]
      processors: [memory_limiter, filter, transform, batch]
      exporters: [otlp_grpc/tempo, debug/after]

An item in debug/before and not in debug/after was removed or rewritten by a processor. Remove the tap afterwards: detailed verbosity is noisy and costs CPU.

Recipe: find out which component names your binary accepts

otelcol-contrib components | head -60
otelcol-contrib featuregate                     # list feature gates and their stages
otelcol-contrib print-config --config=file:config.yaml      # fully resolved, secrets redacted

components shows each component's stability. Do not assume that a shipped component is stable: batch is beta, and queue_batch is in development.

Recipe: upgrade the Collector safely

1. Read the core and contrib changelogs for every version between your current and target. Search for "breaking", "removed", "deprecated" and the components you use.
2. Change the image tag in a branch and run validate and print-config for every overlay.
3. Deploy to a non-production environment with real traffic for a day.
4. Compare otelcol_receiver_accepted_*, otelcol_exporter_sent_*, queue size and memory against the previous version. Note that metric names and units have changed in past releases.
5. Roll out gradually. Keep the previous image tag ready.

Recipe: a persistent queue on an exporter

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage     # mount a volume here in Kubernetes
exporters:
  otlp_grpc/backend:
    endpoint: backend.example.internal:4317
    sending_queue:
      enabled: true
      queue_size: 5000
      storage: file_storage
    retry_on_failure:
      max_elapsed_time: 900s
service:
  extensions: [file_storage]

Check the extension is listed under service::extensions, and that the volume survives pod restarts. Without that, a persistent queue on an emptyDir is no more durable than the default, though it does survive a container restart within the same pod.

Recipe: stop a cardinality explosion in the SDK

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View

keep = View(
    instrument_name="http.server.request.duration",
    attribute_keys={"http.request.method", "http.route", "http.response.status_code"},
)
provider = MeterProvider(views=[keep])       # add metric_readers=[...] in real code

The view makes the unwanted attributes unreachable, however they were added. Verify with the debug exporter that the data points lost the extra attribute.

Recipe: drop noise in the Collector

processors:
  filter/noise:
    error_mode: ignore
    trace_conditions:
      - span.name == "/healthz"
      - span.name == "/readyz"
    log_conditions:
      - log.severity_number < SEVERITY_NUMBER_WARN

Read each condition as "drop anything where this is true". If it drops more than you expect, you wrote a condition for what you wanted to keep. Use the flat condition lists, because the nested traces: span: form is deprecated.

Recipe: point SDKs at a node-local agent in Kubernetes

env:
  - name: NODE_IP
    valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://$(NODE_IP):4318
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
  - name: OTEL_SERVICE_NAME
    value: checkout

The agent must bind 0.0.0.0:4318 (or the node IP) and be reachable on the host port. Set the protocol explicitly: a Python service using the distro would otherwise talk gRPC to an HTTP port.

Recipe: emit old and new semantic convention names together

export OTEL_SEMCONV_STABILITY_OPT_IN="http/dup,database/dup"

Build dashboards on the new names while the old ones keep working, then drop /dup. Do this per service, and confirm with the debug exporter which attribute names are emitted.

Recipe: diagnose a protocol and port mismatch quickly

curl -i -X POST http://collector:4318/v1/traces -H 'Content-Type: application/json' -d '{}'
# any HTTP response from the Collector proves the HTTP receiver is reachable on 4318
nc -vz collector 4317                      # is the gRPC port open at all?

If the first call is refused, the port is closed or bound to localhost. If it answers and your SDK still fails, your SDK is using the wrong protocol.

Habits that keep a pipeline healthy

Review telemetry changes like code. A new attribute on a metric, a new log field or a new transform statement changes cost and privacy exposure. Put Collector configuration in the same repository and review flow as the services, in the same way as your Kubernetes manifests (see Kubernetes and Argo CD), and require a second pair of eyes on anything touching filter, redaction or sampling policies.

Write the runbook while the incident is fresh. After the first time the exporter queue fills, write down which metric told you, what you changed, and how you confirmed recovery. A runbook with real metric names is worth more than a diagram.

Keep a synthetic trace in every environment. A scheduled request that exercises several services, followed by a query for its trace, catches a stripped header or a dead Collector within minutes instead of at the next incident.

Treat startup warnings as failures. A deprecation warning is a scheduled break. Make CI fail on the string alias is deprecated in the Collector's startup output for your configuration.

Document sampling policy where engineers will find it. When someone cannot find the trace for a support ticket, the first question is "was it sampled?". If the answer is buried in a Collector config nobody reads, you will field that question forever.

Mind where your data lives. For teams serving customers in the Gulf and Egypt, regulation and contracts often require telemetry to stay in-region. Keep the gateway in the regional cluster, export only to in-region backends, and do not let individual services send OTLP straight to a vendor endpoint elsewhere. The Collector is where you enforce this, and the place to demonstrate it to an auditor.

Mistakes that look like good ideas

Some errors at this level come from doing something sensible in the wrong place, and they survive code review because each step seems reasonable.

Sampling at every layer independently. Someone sets a ten percent ratio in the SDK, another engineer adds a five percent probabilistic policy at the gateway, and a third samples again in the backend. The compounded rate is a fraction of a percent, nobody knows the real number, and the traces for a specific support ticket are almost never there. Decide on one owner for each stage, write the rates down together, and make the gateway policy keep errors and slow traces regardless of the ratio.

Adding a second Collector to fix a slow backend. When the exporter queue fills, the instinct is to scale out. If the backend is the bottleneck, more Collectors send more concurrent requests to the same struggling service and make things worse. Look at otelcol_exporter_send_failed_* and the backend's own latency first, and scale the Collector only when its CPU or memory is what is saturated.

Copying the debug exporter into production. It is invaluable during an investigation and expensive afterwards: at detailed verbosity it writes every attribute of every item to the logs, which costs CPU, inflates your log bill and puts whatever you were trying to keep private into a second system. Use it temporarily, in a tap pipeline, and remove it.

Making one giant pipeline for all tenants and services. It is easy to start with a single traces pipeline and pile processors onto it. Later, one service's transform rule breaks another's data and nobody can say why. Split pipelines by concern as soon as the rules diverge, using suffixed names such as traces/default and traces/sensitive, and route between them with a connector.

Trusting ignore error mode. The default error_mode for transform and filter is ignore, which is the right production choice, but it means a statement that fails at runtime does so quietly. After every change, send a sample item and read the output. If a rule matters for privacy, test that it ran.

A checklist before you call a service instrumented

• service.name, service.version and deployment.environment.name are set from the manifest.
• Protocol and endpoint are explicit, and match the port.
• The propagators are the same across every service in a request path.
• Asynchronous boundaries carry context in message headers.
• Span kinds are correct, and status is Error only for genuine failures (not 4xx on a server span).
• Errors are recorded with record_exception, and business identifiers are on the failing span.
• Metric attributes are bounded, with a view as a backstop.
• Logs carry trace and span IDs.
• Sensitive attributes are absent, and a test proves it.
• A unit test with an in-memory exporter covers at least one span and one error path.
• The Collector has memory_limiter first, a pinned version, validate in CI, health_check, and its own metrics scraped.
• Sampling is parent-based at the SDK, and RED metrics are generated before any tail sampler.
• You know which components in your pipeline are beta or alpha.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, because each one now has a scale, security, cost or ownership dimension, and it adds the work you own when OpenTelemetry is a platform that other teams depend on rather than a library you configured for one service.

The stakes change at this level. When you instrumented your own service, a mistake cost you a confusing trace. When you run the telemetry pipeline for forty teams, a mistake costs you an outage of the very system everyone uses to diagnose outages, a bill that doubles in a month, a customer's email address stored in a searchable backend for a year, or a silent gap in data that nobody notices until the incident where they needed it. Telemetry is unusual infrastructure: nobody is paged when it is merely wrong, only when it is completely absent, and by then the evidence you wanted is already gone.

This guide is written against OpenTelemetry Collector v0.162.0 (released 29 September 2026), with the specification at v1.61.0, OTLP at v1.11.1 and semantic conventions at v1.44.0. The Collector is still a 0.x project, so every claim about component names and configuration keys is tied to that release, and the upgrade section explains why that matters more than anything else in this document.

Where this picks up

I start with the architecture, because every failure mode in the rest of the document is a property of where the pieces sit and what each one holds in memory.

The architecture you own

OpenTelemetry is often described as "the thing that sends telemetry", and at small scale that is enough. At platform scale it helps to see it as four separate things with four separate owners and four separate failure domains.

The ownership column is the important one. The SDK lives in a service team's process, configured by environment variables that somebody on that team set eighteen months ago and nobody remembers. You cannot roll those out, but they determine how much data arrives, in what shape, with which attributes. The agent and gateway tiers are the opposite: fully yours, changeable in a deploy. A large part of platform design is pushing decisions out of the SDK tier, where you have no control, and into the collector tiers, where you do.

That principle explains several choices that otherwise look arbitrary. Vendor credentials live only in the gateway, so a leaked application environment does not leak a backend token. Redaction runs in the gateway, because you cannot audit forty SDK configurations. Tail sampling runs in a collector tier, because the SDK cannot know whether a trace will turn out to be interesting. Metric aggregation is centralised where possible, because a cardinality mistake in one service's SDK should be containable at a gateway rather than after the bill.

There is a cost, and it is worth stating plainly: every tier you add is another process to size, upgrade, secure and watch, and another hop where data can be lost. The default position should be the fewest tiers that satisfy your requirements, added one at a time when a specific requirement cannot be met without them.

Inside a pipeline: what the Collector does under pressure

Mid-level taught the order of processors. This section explains why that order exists, because at scale you will be reasoning about behaviour the documentation does not spell out, and only a correct internal model lets you predict it.

A Collector pipeline is a chain of in-process function calls. A receiver decodes a request into the internal pdata representation, then calls the first processor directly. That processor does its work and calls the next, and so on down to the exporters. There is no queue between a receiver and a processor, and none between processors. The call is synchronous: the receiver's network handler is still waiting while every processor in the chain runs. The first place where the chain becomes asynchronous is the exporter's sending_queue, or a batch processor, which also decouples by holding data until a size or time threshold.

Three consequences follow, and all three show up in incidents.

A slow processor slows ingestion. An expensive OTTL expression, a k8s_attributes lookup that misses its cache, or a tail_sampling processor contending on a lock all make the receiver's handler take longer. Clients see higher latency, then timeouts, then retry, which adds load. The symptom looks like a network problem, but the cause is CPU inside the chain.

Backpressure propagates backwards only as far as the first queue. If a backend is slow and the exporter's queue has room, nothing upstream notices; the exporter simply enqueues and returns. Once the queue is full, what happens depends on block_on_overflow. With the default false, the enqueue fails immediately, the failure is returned up the call chain, and the receiver tells the client the data was refused (a retryable error). With true, the exporter blocks, which blocks the processors, which blocks the receiver's handler, and pressure reaches the client as latency rather than as an error. Neither is wrong, but they are different contracts, and you should choose deliberately. Refusing quickly protects the Collector and pushes the decision to the client's retry logic. Blocking preserves data but ties up threads and memory while it waits.

Fan-out multiplies failure. When a pipeline has two exporters, the data goes to both. If one exporter's queue is full and its enqueue fails, the pipeline reports failure for that call even though the other exporter succeeded, and a client may retry data that one backend already holds. This is why a pipeline that mixes a critical exporter with a best-effort one, such as a production backend and a debugging sink, is a hazard. Give each destination its own pipeline and forward into them with the forward connector, so failure in one does not contaminate the other.

connectors:
  forward/critical: {}
  forward/debug: {}

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter, k8s_attributes]
      exporters: [forward/critical, forward/debug]
    traces/to-backend:
      receivers: [forward/critical]
      processors: [batch]
      exporters: [otlp_grpc/tempo]
    traces/to-debug:
      receivers: [forward/debug]
      processors: [filter/debug-sample]
      exporters: [debug]

Here the connector pair sits at the boundary, each downstream pipeline owns its own processors and queue, and the debug pipeline can be filtered aggressively without touching the critical path. Remember that each pipeline gets its own instance of every processor, which is usually what you want here.

The memory story, end to end

memory_limiter is the most misunderstood component, because people read it as a cap when it is a shedding valve. It checks memory every check_interval. When usage exceeds the soft limit (the limit minus the spike allowance), it starts refusing incoming data; the refusals are what otelcol_processor_refused_* counts, and the receiver returns a retryable error. When usage exceeds the hard limit, it also forces a garbage collection. It does not evict data already held in queues. So the real memory of a Collector is the sum of several things: the decoded request in flight, whatever the batch processor is accumulating, every exporter's sending_queue, tail_sampling's buffered traces, and Go's own overhead. The limiter sees the total but can only relieve it by refusing new data, so anything you have configured to hold a large amount of data can push the process over the edge faster than the limiter reacts.

This is how the classic OOMKill happens: a queue_size that looked modest, multiplied by a batch size that looked modest, multiplied by the number of exporters and pipelines, exceeded the container limit. Size it on paper before you deploy it.

worst-case queue memory  ≈  queue_size × average_request_bytes × number_of_exporters
tail-sampling memory     ≈  num_traces × average_trace_bytes
container limit          ≥  both of the above  +  batch buffers  +  ~30% headroom for Go

Then configure the runtime to cooperate. Set GOMEMLIMIT to roughly 80 percent of the container's memory limit so the Go garbage collector works harder before the kernel's OOM killer acts, and set the limiter's percentage relative to the same container limit. The removed memory_ballast extension is a relic of an earlier era; if you see it in an old configuration, delete it.

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

        env:
          - name: GOMEMLIMIT
            value: 1600MiB          # about 80% of a 2Gi container limit
        resources:
          requests: { cpu: "1", memory: 2Gi }
          limits:   { memory: 2Gi }

Retries, queues and what "at least once" really means

The exporter helper gives every exporter the same machinery, and you should be able to recite its default behaviour: a request that fails is retried with exponential backoff starting at five seconds, capped at thirty seconds between attempts, and abandoned after max_elapsed_time of 300 seconds. Meanwhile new data accumulates in a queue of 1000 requests with ten consumers. So the default survives roughly five minutes of backend outage, provided the queue does not fill first, and only if the process does not restart, because the queue is in memory.

Two details bite at scale. First, the default sizer counts requests, not bytes or items, so a queue of 1000 can hold 1000 tiny requests or 1000 enormous ones, and the memory difference is several orders of magnitude. Setting sizer: bytes or sizer: items makes the capacity mean something you can budget. Second, retries are at least once, not exactly once. If the backend received and stored a request but the response was lost, the Collector retries, and the backend sees the data twice. For traces this is usually harmless because span IDs deduplicate; for metrics and logs, backends vary, and you should find out which behaviour yours has rather than assume.

A note on the direction of travel, because it affects what you build today. Batching is moving out of the standalone batch processor and into the exporter's sending_queue::batch. The batch processor remains shipped and beta, but the core documentation now says to prefer the exporter's batching when it is available, a new queue_batch processor exists in development (renamed from queuebatch in v0.162.0), and the feature gate pkg.exporterhelper.queueBatchEnabled turns on batching in the default queue configuration. For new configuration at v0.162.0, exporter-side batching is the forward-looking choice:

exporters:
  otlp_grpc/tempo:
    endpoint: tempo-distributor.observability:4317
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
      sizer: items
      batch:
        flush_timeout: 200ms
        min_size: 8192
        max_size: 16384

Setting sending_queue: { batch: {} } gives you the defaults, which are equivalent to the old batch processor defaults. Whichever you choose, put the choice in your platform's standard configuration and test it, because the metrics that describe batch sizes also moved: from v0.159.0 the otelcol_exporter_queue_batch_send_size* metrics are recorded after batching, and only when sending_queue::batch is configured. A dashboard built before that upgrade may go quietly empty.

Designing the two tiers

The agent and gateway are not two copies of one configuration. Each exists because a specific class of work can only be done in one place, and the discipline is to put each job where it can be done and nowhere else.

The agent runs close to the application, one per node as a DaemonSet or one per pod as a sidecar, and its strengths are all about proximity. It can read the node's log files with file_log, scrape the kubelet with kubelet_stats, collect host metrics, and enrich telemetry with pod and node metadata via k8s_attributes, because it can identify the source by connection and by the pod IP. It offers applications a fast local endpoint, so an SDK's export call returns in microseconds and the application is insulated from the network. Its job is deliberately small and stateless. Keep the agent lean: the more it does, the more a bad agent release threatens every node at once.

The gateway is a pooled Deployment behind a load balancer, and its strengths are all about centrality. It is the single place to authenticate senders, redact sensitive fields, apply tail sampling, route to backends and tenants, and hold the vendor credentials. Because it is stateless in its simplest form, it scales by replica count, and because it is central, a policy change there reaches everyone without touching a single application.

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  file_log:
    include: [/var/log/pods/*/*/*.log]
    start_at: end
  host_metrics:
    collection_interval: 30s
    scrapers: { cpu: {}, memory: {}, filesystem: {}, network: {} }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  k8s_attributes:
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  resource_detection:
    detectors: [env, system]
    override: false

exporters:
  otlp_grpc/gateway:
    endpoint: otel-gateway.observability:4317
    tls: { ca_file: /etc/otel/ca.crt }
    sending_queue: { batch: {} }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, k8s_attributes, resource_detection], exporters: [otlp_grpc/gateway] }
    metrics: { receivers: [otlp, host_metrics], processors: [memory_limiter, k8s_attributes, resource_detection], exporters: [otlp_grpc/gateway] }
    logs:    { receivers: [otlp, file_log], processors: [memory_limiter, k8s_attributes, resource_detection], exporters: [otlp_grpc/gateway] }

Notice what is absent: no sampling, no redaction policy, no backend credentials, no routing. The agent adds facts (who sent this, from where) and forwards. The pairing of the two tiers is also why the k8s_attributes processor must run before batching: batching merges requests from many sources and discards the connection context that k8s_attributes uses to identify the pod.

What each tier's outage costs

Do this reasoning out loud in design reviews, because it is the real content of an availability requirement.

If an agent dies, the application's exports to it fail, and the SDK's own batch processor retries into a bounded queue and then drops. You lose one node's telemetry for the duration, including logs that were not yet read from files (files persist, and file_log can resume with a checkpoint via the file_storage extension, so logs are the most recoverable signal here). The blast radius is one node.

If the gateway degrades, every agent's exporter queue fills at once. The size of that queue is the time you have. With in-memory queues you have minutes; with a file-backed queue you have as long as the disk allows. The blast radius is everything, which is why the gateway gets the persistent queue, the autoscaler and the careful rollouts, and why you never ship a new gateway version to all replicas at once.

If a backend degrades, the gateway's exporter queues absorb it, and the same arithmetic applies. The design question is whether you would rather drop data (accepting loss, keeping the platform healthy) or hold it (risking memory and disk). Write the decision down for each signal. Most teams choose to protect traces and metrics from loss more strongly than logs or the reverse, depending on what their incident process relies on, and either is defensible if deliberate.

Scaling: which component breaks first

The official scaling guidance classifies components into three groups, and the classification is the most useful single idea in this section because each group scales differently and fails differently.

Stateless components break first on CPU and on the load balancer. The usual trap is gRPC itself. gRPC multiplexes many requests over one long-lived HTTP/2 connection, so a plain layer-4 load balancer assigns each client connection to one backend and keeps it there. You scale from three gateway pods to ten and the new ones receive nothing, because every existing connection is pinned. The fixes are a layer-7 balancer that understands gRPC and balances per request, client-side balancing in the exporter (balancer_name), or using a headless service with the load_balancing exporter as the agent's client. Verify a scale-out by looking at otelcol_receiver_accepted_spans per pod, not by trusting the replica count.

The next failure is the backend, and it is worth saying bluntly: when the backend is the bottleneck, adding collectors makes things worse, because more senders pressure the same slow destination. The signal is that exporter queues are full across all gateways while their CPU is idle. Scale the backend, sample earlier, or reduce what you send.

Scrapers break through duplication. If you run the prometheus receiver in three replicas, each scrapes every target and you store three copies of every series. The Kubernetes Operator's Target Allocator solves this by running the service discovery itself and handing each collector replica a disjoint slice of targets, in a StatefulSet-mode collector, with an allocation strategy such as consistent-hashing, per-node or least-weighted. With prometheusCR.enabled it also discovers ServiceMonitor and PodMonitor resources, which lets teams keep the scrape declarations they already have.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: metrics-scraper
  namespace: observability
spec:
  mode: statefulset
  replicas: 3
  targetAllocator:
    enabled: true
    allocationStrategy: consistent-hashing
    prometheusCR:
      enabled: true
  config:
    receivers:
      prometheus:
        config:
          scrape_configs: []
    exporters:
      otlp_grpc/gateway:
        endpoint: otel-gateway.observability:4317
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [otlp_grpc/gateway]

Stateful processors break through memory and through routing. This is the hard case, and it deserves its own sections: tail sampling and metric generation from spans both require that a particular key always arrive at the same instance. The general solution is a two-tier gateway: a first tier of stateless collectors that uses the load_balancing exporter with a routing key, and a second tier of stateful collectors that receives the partitioned data.

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: { ca_file: /etc/otel/ca.crt }
        timeout: 5s
    resolver:
      dns:
        hostname: otel-tail-headless.observability.svc.cluster.local
        port: 4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [load_balancing]

The resolver is where operational subtlety lives. A dns resolver pointed at a headless Service returns every pod IP and re-resolves periodically, so a scale event changes the ring; the k8s resolver (service: name.namespace) watches endpoint slices and reacts faster but needs RBAC to read them; a static list never changes. Whichever you pick, remember what happens when the membership of tier 2 changes: the consistent-hash ring moves, and traces that were mid-flight have their later spans routed to a different instance than their earlier ones. The result is a fraction of traces that arrive incomplete. You cannot eliminate this, only make it rare by scaling tier 2 rarely and in small steps, and by setting decision_wait with the churn in mind.

Sampling as a policy, not a setting

At the start of this series sampling was a dial in the SDK: keep ten percent of traces. At platform scale it becomes a policy with owners, consequences and trade-offs, because what you keep determines what you can ever investigate.

Head sampling decides at the start of a trace, in the SDK, before anything is known about how it will end. It is cheap, it needs no state, and it reduces load on every hop downstream, including the network. It also decides blindly: a one-percent head sample keeps one percent of the failing requests and one percent of the slow ones, so the traces you most want are the ones most likely to be discarded. The default sampler is parentbased_always_on, which records everything unless a parent said otherwise, and the usual production change is parentbased_traceidratio with an argument such as 0.1. The parentbased prefix is the part that matters: it makes a service honour its caller's decision, so a trace is kept or dropped as a whole rather than shredded by each service flipping its own coin.

Tail sampling decides after seeing the whole trace, in the gateway, and so can keep every error, every slow trace and a fraction of the rest. The price is state: the processor must hold every span of every in-flight trace until decision_wait (30 seconds by default) expires, which is why it needs the routing tier in front of it and why its memory is predictable from two numbers, the trace rate and the decision window.

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 200000
    expected_new_traces_per_sec: 2000
    decision_cache:
      sampled_cache_size: 500000
      non_sampled_cache_size: 500000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1500 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Policies are evaluated together and a trace is kept if any policy matches, so this configuration reads as "all errors, all traces over 1.5 seconds, and five percent of everything else". The policy list is the place where an organisation's values get written down, which makes it worth reviewing like code: the question "what do we keep" has an owner and a pull request history.

Do the memory arithmetic before you deploy. With 2000 new traces per second and a 30-second window, roughly 60,000 traces are in flight at once. If num_traces is lower than that, the processor evicts traces to make room and decides on incomplete data, which silently degrades every policy. If an average trace is 50 KB, 200,000 slots is 10 GB. Numbers like these are why tail sampling is the component that most often dominates a gateway's cost, and why it gets its own tier.

connectors:
  span_metrics:
    histogram:
      explicit:
        buckets: [50ms, 100ms, 250ms, 500ms, 1s, 2500ms, 5s]
    dimensions:
      - name: http.request.method
      - name: http.response.status_code
    exemplars:
      enabled: true
    metrics_flush_interval: 30s

service:
  pipelines:
    traces/ingest:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [span_metrics, load_balancing]
    metrics/derived:
      receivers: [span_metrics]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc/mimir]

A subtlety: span_metrics is itself stateful, because it aggregates per dimension set over an interval. If you run several instances behind a stateless balancer, each computes partial aggregates and the backend receives several series that must be summed. Either route by service before it, or make sure the resulting metrics carry an instance identity and the query layer adds them. The dimensions list is also a cardinality decision: every dimension multiplies series, and putting a high-cardinality attribute such as a user ID there is one of the fastest ways to exhaust a metrics backend.

Sampling also interacts with consistency in a way that shows up in practice. If the SDK head-samples at ten percent and the gateway tail-samples what arrives, you have multiplied the rates, and an error that occurs in ninety percent of unsampled traces was never sent. Decide which tier owns the decision. A common, defensible arrangement is no head sampling for services below a volume threshold, letting the tail sampler do the work, and head sampling only for the few services whose raw volume would overwhelm the pipeline, with the sampled fraction recorded so counts can be scaled back up.

The composable samplers that the specification is developing will eventually replace traceidratio, which is retained with a deprecation timeline rather than removed, so traceidratio remains a valid environment value today. Teach teams to treat the sampler string as platform configuration that may change, not a constant.

Reliability: choosing what you can afford to lose

"Do not lose telemetry" is not a requirement; it is a wish. A requirement names a failure and a duration: "survive a one-hour backend outage without losing traces", or "tolerate a gateway rollout without dropping logs". Each such sentence selects a mechanism, and each mechanism has a failure it does not cover.

The persistent queue is the pragmatic first step for a gateway. You declare a storage extension, give the exporter's queue a reference to it, and add the extension to service:

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    timeout: 1s

exporters:
  otlp_grpc/tempo:
    endpoint: tempo-distributor.observability:4317
    sending_queue:
      enabled: true
      queue_size: 20000
      sizer: items
      storage: file_storage/queue
    retry_on_failure:
      enabled: true
      max_elapsed_time: 3600s

service:
  extensions: [file_storage/queue]

Every piece of that is a decision. The directory must be a real persistent volume, not the container's ephemeral layer, or the queue disappears with the pod, which defeats its purpose. In Kubernetes that means a StatefulSet with a volumeClaimTemplate, and it means thinking about what happens when a pod is rescheduled to a node where its volume is not. Raising max_elapsed_time turns a five-minute retry window into an hour, which only helps if the queue is big enough to hold an hour of traffic; otherwise you have a queue that fills and refuses while still nominally retrying. And a disk queue is slower than a memory queue, so benchmark the throughput you need before adopting it.

Kafka earns its place when you need durability that does not depend on a single Collector's disk, when bursts regularly exceed what a gateway can absorb, or when several consumers need the same stream. The Collector has kafka exporter and receiver components to put a topic between two tiers. It is also a whole stateful system with its own failure modes, partitions, retention and upgrade story; adding it to "make telemetry reliable" without a Kafka operations team trades one reliability problem for a larger one. Recent releases have also been removing long-deprecated Kafka client options (for instance the old auth.tls and auth.plain_text settings were removed in v0.160.0), so a Kafka block copied from an older blog post needs checking against the component README for your version.

The part that most organisations skip is the alert on loss itself. A pipeline that can drop data must also be able to tell you that it did, in a place that does not depend on the pipeline. Section "Watching the pipeline itself" covers which counters to alert on; the principle here is that accepted loss is only acceptable when it is measured.

The trust model and security

A Collector is an unusual security object. It is a network service that accepts data from many sources, holds credentials to write to expensive backends, often runs with broad read access (logs from every pod, metrics from every node, the Kubernetes API), and sits on the path of data that may contain personal information. It deserves the threat modelling you would give an API gateway.

Start by drawing the trust boundaries, because each one prescribes a control.

Network exposure

Since recent releases every server endpoint in the Collector defaults to localhost, including the OTLP receivers, health_check, zpages and the internal metrics endpoint at 8888, which binds 127.0.0.1. This is the safe default and the most common cause of "connection refused" confusion, because in a container a localhost-bound receiver is unreachable from outside it. The official distribution configurations set 0.0.0.0 explicitly where they intend to be reached. When you do open an endpoint, you are making a deliberate exposure decision, so pair it with a NetworkPolicy that names who may connect, and avoid exposing pprof and zpages beyond a loopback or an admin network at all, since they reveal internals and, in the case of pprof, offer a cheap way to burn CPU.

Authentication and transport

Terminate TLS on every receiver that is reachable across a network, and verify client certificates where you can. An authenticator extension turns "anyone who can reach the port" into "anyone holding a credential":

extensions:
  oidc:
    issuer_url: https://login.example.com/realms/platform
    audience: otel-gateway
  bearertokenauth/backend:
    token: ${env:BACKEND_API_TOKEN}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        include_metadata: true
        tls:
          cert_file: /etc/otel/tls.crt
          key_file: /etc/otel/tls.key
          client_ca_file: /etc/otel/clients-ca.crt
        auth:
          authenticator: oidc

exporters:
  otlp_http/vendor:
    endpoint: https://otlp.vendor.example.com
    auth:
      authenticator: bearertokenauth/backend

service:
  extensions: [oidc, bearertokenauth/backend]

Three details deserve emphasis. max_recv_msg_size_mib is a denial-of-service control as much as a tuning knob, since an unbounded message is an invitation. include_metadata: true is what makes request headers available to later components for routing and per-tenant batching, and you will need it again in the multi-tenancy section. And the token is read from an environment variable with ${env:...} rather than written into the file, which leads to secrets.

Also note what changed recently: from v0.154.0 insecure TLS cipher suites are disabled by default in the shared TLS configuration, and you must opt in with include_insecure_cipher_suites. If you carry an old configuration for a legacy client that needs them, prefer upgrading the client to re-enabling weak ciphers.

Secrets

Never put a token in a configuration file that lives in version control or a ConfigMap. Reference it with ${env:NAME} and supply it from a Kubernetes Secret as an environment variable or a mounted file (${file:/path} embeds a file's contents). Fields declared as sensitive by the component, such as headers and passwords, are masked in otelcol print-config, which defaults to --mode=redacted, and in logs. That is a safety net, not permission to be casual: --mode=unredacted exists and will print everything, so never run it in a place whose output is captured. On the application side, keep vendor credentials out of SDKs entirely by pointing every application at the local agent and letting the gateway hold the key. The fewer places a credential exists, the fewer places can leak it.

Remember one piece of syntax that catches people in security-sensitive configuration: a literal dollar sign must be written $$. In a prometheus receiver's relabel rules, $1 is interpreted as an environment variable and silently expands to an empty string, which can turn a restrictive relabel into a permissive one. Write $$1.

Sensitive data in the telemetry itself

The most common real-world security finding in an OpenTelemetry deployment is not an attack on the Collector; it is personal data sitting in telemetry. Instrumentation libraries are built to be useful, so by default they may record full URLs including query strings, database statements (db.query.text), some request headers, and exception messages that embed user input. A user's email in a query string becomes a searchable attribute that is retained for as long as your trace backend retains anything.

For organisations in the Gulf and Egypt this is not an abstract concern. Data-protection regimes in the region, and the residency expectations of regulated sectors such as banking and government, mean that telemetry containing personal data may be subject to the same rules as the primary database, including where it is stored. A tracing backend hosted in another region, or a vendor whose ingestion endpoint is overseas, can be a compliance decision hiding inside an observability decision. Raise it with whoever owns data protection before you choose a backend, not after the first audit.

There are three layers of defence, and you want all three.

1. Prevent capture at the source. Configure instrumentations not to record what you never need: disable query-text capture where the statement may contain literals, and avoid capturing arbitrary headers. This is the cheapest control and the hardest to enforce, because it lives in the SDK tier.
2. Scrub at the gateway. The redaction processor can enforce an allow-list of attribute keys and mask values that match patterns, and transform with OTTL can rewrite or hash specific fields. An allow-list is the stronger posture, because it fails closed: a new attribute nobody reviewed is removed rather than shipped.
3. Test the control. Send known-bad data through the gateway and assert that it comes out clean.

processors:
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.request.method
      - http.response.status_code
      - http.route
      - url.path
      - server.address
      - db.system.name
      - service.name
      - k8s.namespace.name
      - error.type
    blocked_values:
      - "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"
      - "\\b(?:\\d[ -]*?){13,19}\\b"
    summary: info
  transform/scrub:
    error_mode: ignore
    trace_statements:
      - replace_pattern(span.attributes["url.full"], "\\?.*$", "?REDACTED")
      - delete_key(span.attributes, "db.query.text")

Two honest caveats. Regular expressions find patterns, not meaning, so a card-number regex catches numbers and misses a name; the allow-list is the control that does the real work and the patterns are a backstop. And a scrubbing step that errors on a malformed attribute must not silently pass the original through: the transform processor's error_mode now defaults permanently to ignore, meaning a statement that fails is skipped and the data continues unmodified. For a security-critical scrubber, consider propagate so that a failed statement surfaces loudly, and test with malformed inputs to see which you have.

The supply chain of the pipeline

The Collector is a Go binary assembled from hundreds of modules, and the contrib distribution includes well over a hundred receivers, processors and exporters, most of which your platform will never use. Every compiled-in component is code that runs in your environment with your credentials, and code you do not run cannot be exploited. That is the security argument for building your own distribution with the OpenTelemetry Collector Builder, ocb.

dist:
  name: acme-otelcol
  description: Acme platform Collector
  output_path: ./build

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.162.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.162.0
processors:
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.162.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.162.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor v0.162.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.162.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.162.0
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.162.0
extensions:
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.162.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.162.0

ocb --config builder-config.yaml
./build/acme-otelcol components

Note the asymmetry that trips people up here: in the Go module paths the directories keep their long historical names (filelogreceiver, k8sattributesprocessor), while the configuration type names you write in YAML are the new snake_case ones (file_log, k8s_attributes). The rename only changed the configuration type. Also, every component's version must match the core version, and current contrib requires Go 1.26 or newer at build time. I have not shown a full set of the modules your deployment needs, so run components on the result and compare it against the component IDs your configuration uses; a missing component fails at startup with unknown type.

Treat the resulting binary like any artifact you ship: build it in CI with pinned inputs, produce an SBOM, scan it, sign the image, and let your admission policy require the signature. Pin the tag or, better, the digest. The project also publishes CVEs and has a security response process, so subscribe to it; because the Collector is built on Go and gRPC dependencies, a dependency advisory can require you to rebuild even when no OpenTelemetry change occurred, which a scheduled rebuild handles and a frozen image does not.

Multi-tenancy

Once several teams share the pipeline, you need an answer to four questions that have nothing to do with tracing: whose data is this, who may see it, who pays for it, and how do I stop one team's mistake from starving the others? OpenTelemetry does not have a tenant object, so multi-tenancy is something you build from three primitives: a tenant identity attached to the data, per-tenant handling in the pipeline, and isolation in the backends.

Identity. The tenant must come from something the sender cannot forge. Deriving it from a resource attribute that the application sets, such as service.namespace, is convenient and trustworthy only if you control the applications. A stronger source is the authenticated identity: the authenticator extension on the receiver knows which credential connected, and with include_metadata: true the request's headers and auth context are available downstream. In Kubernetes, the pod's namespace as enriched by k8s_attributes in the agent is a good identity, because the agent derives it from the connection rather than from anything the application claims.

Per-tenant handling. The Collector has three levers. The first is batching by tenant, so one tenant's slow or oversized data does not sit in the same batch as everyone else's and one request to a per-tenant backend never mixes tenants. The batch processor has metadata_keys for this, with metadata_cardinality_limit as the ceiling, and the exporter-side equivalent lives under sending_queue::batch::partition::metadata_keys. The second is routing with the routing connector, which evaluates OTTL conditions against resource attributes or request metadata and sends matching data to a named pipeline. The third is forwarding the tenant identity to the backend, for example as the X-Scope-OrgID header that Tempo, Loki and Mimir use for their own multi-tenancy, which is how backend-level isolation gets connected to the tenant you identified.

connectors:
  routing/tenant:
    default_pipelines: [traces/shared]
    error_mode: ignore
    table:
      - context: resource
        condition: attributes["k8s.namespace.name"] == "payments"
        pipelines: [traces/payments]

exporters:
  otlp_grpc/tempo-payments:
    endpoint: tempo-distributor.observability:4317
    headers:
      x-scope-orgid: payments
  otlp_grpc/tempo-shared:
    endpoint: tempo-distributor.observability:4317
    headers:
      x-scope-orgid: shared

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [routing/tenant]
    traces/payments:
      receivers: [routing/tenant]
      processors: [batch]
      exporters: [otlp_grpc/tempo-payments]
    traces/shared:
      receivers: [routing/tenant]
      processors: [batch]
      exporters: [otlp_grpc/tempo-shared]

The static header approach is the simplest to reason about, at the cost of one exporter and pipeline per tenant, which stops being pleasant around a few dozen tenants. Beyond that you want the tenant header set dynamically from the incoming request's metadata, using a headers-setter extension, and you want configuration generated from a tenant registry rather than edited by hand. The general rule is that hand-written per-tenant configuration is fine for three tenants and is a source of incidents at thirty.

Isolation. Routing only divides data; it does not protect the pipeline from a noisy tenant. The controls that do are per-tenant limits, and they live in several places. At the gateway, max_recv_msg_size_mib and memory limits cap the damage one sender can do to a pod. At the backend, per-tenant ingestion limits (every serious backend has them) stop one tenant's cardinality explosion from degrading everyone else's queries. For strict isolation, a separate pipeline or even a separate gateway deployment for a high-value or high-volume tenant is the reliable answer, because it gives that tenant its own queues and its own failure domain. Watch the partition cache limits when batching per tenant: the partition cache_size defaults to 10,000 and idle_timeout to 90 seconds in v0.162.0, and a tenant identity with unbounded cardinality, such as a user ID mistakenly used as a key, exhausts it.

Cost and cardinality

Observability spend grows with volume, with cardinality, and with retention, and the three respond to different controls. Most organisations discover their bill by receiving it, so the platform's job is to make the spend visible per team early and to give teams levers that reduce it without hurting their diagnosis.

Volume is the simplest to reduce. The levers, roughly in order of how much they buy: drop what is never useful (health-check spans and debug-level logs, removed with the filter processor), sample what is useful but repetitive (tail sampling that keeps errors and slow traces and a fraction of the rest), and compress what you send (gzip is the default for the gRPC exporter, and zstd is available). The filter processor's current form uses flat condition lists, and the old nested traces: span: form is deprecated:

processors:
  filter/noise:
    error_mode: ignore
    trace_conditions:
      - 'span.name == "/healthz"'
      - 'span.name == "/readyz"'
    log_conditions:
      - 'log.severity_number < SEVERITY_NUMBER_WARN'

Cardinality is where costs become nonlinear and surprising. A metric's series count is the product of the distinct values of its attributes. One metric with an http.route attribute of twenty values, a status code with five, and a user.id with two million produces two hundred million series, and the backend charges by series or falls over trying to hold them. Traces are far more forgiving of high-cardinality attributes than metrics are, which is why the discipline differs by signal: put identifiers on spans, where they are searchable at no series cost, and keep them off metrics.

There are several places to enforce this, and a platform should use more than one.

• In the SDK, with Views. A View can drop an attribute from a metric stream, rename a metric, or change histogram buckets. It is the earliest and cheapest place to act, but it lives in application code you do not control, so you use it for guidance and for your own services.
• In the gateway, with OTTL. transform with keep_keys or delete_key on the datapoint context removes attributes from metrics from every source, which is the control you can actually enforce.
• In connectors. The dimensions list in span_metrics decides which span attributes become metric labels. Keep it short and review additions.
• In the backend. Per-tenant series limits convert a mistake into a rejected write for that tenant alone, rather than a degraded backend for everyone.

processors:
  transform/metric-labels:
    error_mode: ignore
    metric_statements:
      - keep_keys(datapoint.attributes, ["http.request.method", "http.route", "http.response.status_code", "service.name"]) where metric.name == "http.server.request.duration"

One design choice is worth making once, explicitly: histograms. Explicit-bucket histograms need you to choose boundaries that suit each metric, and boundaries chosen wrongly give useless percentiles. Exponential histograms adapt their resolution to the data and cost less to configure, but your backend must understand them. Check support in the store you operate before you standardise on either, and standardise, because mixed bucket schemes across services make dashboards that combine them misleading.

Attribution closes the loop: a cost you cannot assign to a team is a cost nobody reduces. Count what each tenant sends with the count connector or with the Collector's own per-pipeline metrics, publish a per-team report, and let the numbers start the conversation. Teams that can see that one chatty debug log line costs them a measurable amount fix it without being asked.

Finally, watch the telemetry about the telemetry. The Collector's internal metrics at level: normal are useful and moderately sized; detailed adds considerable volume and should be a temporary setting while debugging. A long-running detailed level on every agent in a large cluster is itself a source of cost.

Upgrades and migrations

The Collector is a 0.x project, and that number is a promise about change. A release arrives roughly every two weeks, core and contrib together, and any minor version may contain breaking changes. A platform team's first upgrade policy is therefore not "how often" but "how do we know a release is safe for us". Everything else in this section flows from that.

Pin exact versions. otel/opentelemetry-collector-contrib:0.162.0, never latest, and never a floating minor. In an ocb build, every component's version must be the same. The image tag in a Helm values file or an OpenTelemetryCollector resource is part of your change-controlled configuration.

Read both changelogs, for every version you skip. The core repository and the contrib repository each have a changelog with a "Breaking changes" section. Upgrading from 0.158 to 0.162 means reading five releases of each. The reading takes an hour and is cheaper than one incident. Focus on four categories: component renames, removed configuration keys, removed feature gates, and changed defaults.

Gate upgrades in CI. Two commands make this mechanical. otelcol validate --config=... exits non-zero on configuration that does not parse or fails validation, and otelcol print-config --config=... shows the merged, resolved configuration after every --config layer, environment expansion and --set override. Run both against every tier's configuration with the candidate version before it reaches a cluster. Note that print-config no longer needs a feature gate: it replaced the old print-initial-config flag in v0.138.0, and the gate that enabled it was removed in v0.155.0. The OpenTelemetry website's page still shows --feature-gates=otelcol.printInitialConfig; copying it from there produces no such feature gate "otelcol.printInitialConfig". Run otelcol print-config directly.

#!/usr/bin/env bash
# ci/check-collector-config.sh: runs in CI for each tier, on the candidate version
set -euo pipefail
IMAGE="otel/opentelemetry-collector-contrib:0.162.0"
for tier in agent gateway tail-sampler; do
  docker run --rm -v "$PWD/config/$tier.yaml:/cfg.yaml:ro" "$IMAGE" \
    validate --config=/cfg.yaml
  docker run --rm -v "$PWD/config/$tier.yaml:/cfg.yaml:ro" "$IMAGE" \
    print-config --config=/cfg.yaml --mode=redacted > "rendered-$tier.yaml"
done
echo "all tiers validate"

A validated configuration can still behave differently, so after validation run a canary: one gateway replica on the new version taking a slice of traffic, compared on the same dashboard as the old version. Watch accepted and refused counts, exporter send failures, queue fill and memory. Promote only when they match.

The rename wave

The largest ongoing migration in the Collector is the move of component type names to snake_case, tracked as a single project-wide effort. Old names keep working as deprecated aliases (with a startup warning of the form "otlp" alias is deprecated; use "otlp_grpc" instead) with the exceptions listed below, so you can migrate at your own pace, but you should not stay on aliases: an alias is a countdown, not a permanent feature. The renames your platform will most likely touch:

Two lines in that table are the dangerous ones. Where there is no alias, the old name simply stops working at the release that renames it, and the Collector fails to start. And the receiver/exporter asymmetry catches everyone once: the receiver is still called otlp, but the exporter is otlp_grpc. A search-and-replace of otlp: across a configuration will break the receiver.

Names that were not renamed include otlp (receiver), debug, batch, memory_limiter, tail_sampling, probabilistic_sampler, health_check, file_storage, k8s_cluster, transform, filter, attributes, resource, prometheus (receiver and exporter), redaction, groupbyattrs, zpages, pprof, bearertokenauth, basicauth, opamp and forward. Do not rename those.

Do the rename as its own pull request, with no other change, and verify with print-config that the rendered configuration differs only in type names. A pull request that mixes a rename with a behavioural change is one that nobody can review.

Removed and changed behaviour

Beyond renames, a short list of changes that has bitten real deployments and that your upgrade review should know by name:

• Removed feature gates. The gates stabilised and removed in v0.155.0 include confighttp.framedSnappy, exporter.PersistRequestContext and pdata.enableRefCounting. A --feature-gates flag naming a removed gate now fails at startup with no such feature gate, and naming a stable gate with a minus (to disable it) fails with feature gate "x" is stable, can not be disabled. Audit your deployment manifests for old gate flags.
• HTTP keepalive. From v0.160.0 the flat fields idle_conn_timeout, max_idle_conns, max_idle_conns_per_host and disable_keep_alives are deprecated in favour of a keepalive: section, and v0.162.0 enables the new keepalive configuration by default. The gate pkg.confighttp.PrioritizeNewKeepalive restores the old behaviour if you need time.
• Error handling defaults. The transform and filter processors' error_mode default is now ignore; routing defaults to ignore from v0.157.0, and signal_to_metrics from v0.162.0. If you relied on a failure surfacing, set propagate explicitly.
• Internal metric changes. Metric units became singular in v0.148.0 ({requests} to {request}), and the :8888 endpoint emits service_name, service_instance_id and service_version labels with underscores. The memory_limiter processor's metrics were renamed to otelcol_processor_memory_limiter_* in v0.155.0, and the batch-size metrics moved in v0.159.0. Dashboards and alerts built on the old names go quiet rather than erroring, which is the dangerous failure mode.
• Contrib removals. Components and options are removed after deprecation: the wavefront receiver (v0.162.0), the mezmo exporter (v0.161.0), old Kafka client authentication options (v0.160.0), the Elasticsearch exporter's flush and num_workers (v0.162.0) and the Base64Decode OTTL converter (use Decode(x, "base64")).

The failure pattern in that list is consistent: changed monitoring names produce silence, changed defaults produce different behaviour with no error, and removed keys produce a startup failure, which is the good case, because it is loud. Your upgrade process should be designed around catching the silent ones, by comparing the metric names your alerts reference against what the new version actually emits.

Migrating the SDK side, and semantic conventions

Application-side migrations are slower because you do not control the release cadence. Two are worth planning now.

Semantic convention changes. The attribute names that dashboards and alerts query (http.request.method, db.system.name, deployment.environment.name) evolved through stable releases, and some names changed. deployment.environment.name replaced deployment.environment in semconv v1.41, for example, and in v1.44.0 some Kubernetes memory metrics were renamed (k8s.pod.memory.paging.faults became k8s.pod.paging.faults). Instrumentations that support a migration window honour OTEL_SEMCONV_STABILITY_OPT_IN, where a value such as http/dup or database/dup makes the library emit both the old and new names during the transition. The safe sequence is: turn on dup everywhere, migrate dashboards and alerts to the new names, then turn dup off. Skipping the dual-emission step means every dashboard breaks on the day a service upgrades its library.

Also worth knowing for an LLM-focused audience: the generative-AI semantic conventions (gen_ai.*, plus the OpenAI and MCP specifics) moved out of the main semantic conventions repository into open-telemetry/semantic-conventions-genai in v1.42.0 and are deprecated in the main one. Tools that ingest OpenTelemetry for LLM observability, such as Langfuse and Arize Phoenix, depend on those attribute names, so read where your chosen instrumentation libraries sit before building dashboards on them.

Declarative SDK configuration. Instead of a thicket of environment variables, SDKs can read one YAML file named by OTEL_CONFIG_FILE (the older OTEL_EXPERIMENTAL_CONFIG_FILE is deprecated). The schema is versioned (file_format: "1.2"), experimental keys carry a /development suffix, and, crucially, when a configuration file is set the SDK ignores all other OTEL_* environment variables, except those referenced inside the file as ${VAR}. That last rule is the trap: a platform that injects OTEL_EXPORTER_OTLP_ENDPOINT into every pod and then a team that adopts a config file will silently stop honouring the injected endpoint unless the file references it. Language support varies, with the Java agent the most complete, so adopt it per language rather than as a blanket policy.

file_format: "1.2"
resource:
  attributes:
    - name: service.name
      value: ${OTEL_SERVICE_NAME}
propagator:
  composite: [tracecontext, baggage]
tracer_provider:
  sampler:
    parent_based:
      root:
        always_on: {}
  processors:
    - batch:
        exporter:
          otlp_http:
            endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://localhost:4318}/v1/traces

Watching the pipeline itself

A telemetry pipeline that nobody watches fails in the worst possible way: invisibly, and exactly when you need it. The Collector exposes its own telemetry, and your platform should alert on it from a path that does not depend on the pipeline it is measuring. If the only place to see that the gateway is dropping data is the gateway's own backend, then a total failure makes the evidence disappear with it. Send the internal metrics to a separate, simpler system, or at least a separate tenant and path.

The internal metrics are served on port 8888 (bound to 127.0.0.1 by default), or pushed through OTLP. The service::telemetry section follows the declarative-configuration schema, which has a consequence for naming that surprises people: its exporter keys are otlp with a protocol: field, not the Collector's otlp_grpc and otlp_http, because it is configuring an SDK rather than a Collector component.

service:
  telemetry:
    logs:
      level: INFO
      encoding: json
    metrics:
      level: normal
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Four families of metrics answer four questions, and each supports an alert.

The last row is the important one, and it is the one worth paging on. Enqueue failures mean data is being dropped right now. Queue fill above the mid-sixties means you will soon be dropping it. Everything before that is a warning, and everything after is an incident. Also track the process metrics (otelcol_process_memory_rss, otelcol_process_cpu_seconds, otelcol_process_uptime): a low uptime on a gateway that should run for weeks means it is being restarted, usually by the OOM killer, and the restart itself loses in-memory queues.

groups:
  - name: otel-collector
    rules:
      - alert: OtelCollectorDroppingData
        expr: sum by (pod) (rate(otelcol_exporter_enqueue_failed_spans_total[5m])) > 0
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Collector {{ $labels.pod }} is dropping spans at enqueue"
      - alert: OtelCollectorQueueFilling
        expr: otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.65
        for: 10m
        labels: { severity: ticket }
      - alert: OtelCollectorRestarting
        expr: changes(otelcol_process_uptime_seconds[30m]) > 1
        labels: { severity: ticket }

The exact exposed names depend on the Prometheus translation, which appends unit and type suffixes such as _total; confirm them against your own :8888/metrics output after every upgrade, since the renames described earlier changed several. For quick investigation, enable the health_check extension for liveness and readiness probes and zpages for live views (/debug/tracez on 55679 shows recent spans and errors, and /debug/pipelinez shows the pipeline wiring), and bind both to loopback or an admin network.

One more signal is easy to forget and cheap: a synthetic heartbeat. Emit a known trace and a known metric every minute from a canary, and alert if they fail to appear in the backend. It is the only test that covers the entire path from SDK to storage, and it catches failures that every internal metric misses: a routing rule sending a tenant to the wrong place, a redaction rule deleting something it should not, a backend silently rejecting writes.

Governance: conventions that make data usable

A pipeline that delivers everything faithfully can still be useless if no two services describe themselves the same way. You cannot answer "what is the error rate of the payments namespace" if half the services call it payments, a quarter call it Payments-Service, and the rest do not set service.name at all. Governance is the work of making the data comparable, and it is cheaper to impose early than to retrofit.

Resource attributes are the contract. Decide and publish the required set. A practical minimum is service.name (the most important attribute of all, which SDKs fill with unknown_service when unset), service.namespace, service.version, deployment.environment.name and service.instance.id. Several of those are stable in the semantic conventions: service.instance.id and service.namespace since v1.40, deployment.environment.name since v1.41, which replaced deployment.environment. Enforce the contract in two places: by injecting defaults with the Operator's Instrumentation resource or a platform-provided base configuration, and by checking at the gateway that required attributes are present, with a counter or a routing rule that sends non-conforming data to a quarantine pipeline and tells its owner.

processors:
  resource/defaults:
    attributes:
      - key: deployment.environment.name
        value: ${env:CLUSTER_ENVIRONMENT}
        action: insert
connectors:
  routing/conformance:
    default_pipelines: [traces/quarantine]
    error_mode: ignore
    table:
      - context: resource
        condition: attributes["service.name"] != nil and not IsMatch(attributes["service.name"], "^unknown_service")
        pipelines: [traces/main]

Naming rules for the things teams create. Span names should describe an operation at low cardinality (GET /orders/{id}, not GET /orders/12345), which matters because span names feed span_metrics and therefore metric series. Custom metric names should follow the semantic convention style, with units expressed in curly-brace form for counts ({order}), and custom attributes should live in a team namespace to avoid colliding with future conventions. Publish these rules as a short document with examples, and back them with a review step rather than hope.

Use the semantic conventions, and pin to a version. Telemetry carries a schema URL that records which version of the conventions it follows, which lets tooling translate between versions. The practical guidance is to adopt the stable areas (HTTP, the core database conventions, code.*, selected Kubernetes attributes) without modification, to treat development-status areas as subject to change, and to record which semconv version your platform's own instrumentation targets so that you can plan the migrations described earlier.

Stability levels apply to components too. otelcol components prints each compiled-in component with its stability: development, alpha, beta, stable, deprecated or unmaintained. Put a policy on it. A reasonable one is that production pipelines may use stable and beta components freely, alpha and development components only with an explicit exception and a named owner, and never components marked unmaintained. The OTLP receiver and the otlp_grpc and otlp_http exporters are stable for traces, metrics and logs and alpha for profiles. The batch processor is beta, which is part of why it is still safe to use while the migration proceeds.

Signal maturity matters for promises. The tracing and metrics APIs are stable, the logs API and data model are stable, baggage is stable, and profiles are alpha. Language SDKs lag: at the time of writing Python and JavaScript logs are still in development, Go logs are a release candidate, and Rust is beta across the board. When you promise a team "logs through OpenTelemetry", check the SDK for their language before you make the promise, because the answer differs from language to language and is the commonest surprise in a platform rollout. Baggage deserves a governance note too: it propagates arbitrary key-value pairs to every downstream service, so anything placed in it is visible across trust boundaries. It is also not added to spans automatically.

The Operator and fleet management

On Kubernetes, the OpenTelemetry Operator turns the pieces above into declarative resources, and at platform scale it is the mechanism that lets you change many workloads without asking each team to edit anything.

The Operator needs cert-manager for its admission webhooks unless you use the Helm chart's option to generate certificates. Its main resources are OpenTelemetryCollector (API version opentelemetry.io/v1beta1, where spec.config is a YAML object rather than a string and spec.mode is one of deployment, daemonset, statefulset or sidecar), Instrumentation (opentelemetry.io/v1alpha1), TargetAllocator and OpAMPBridge. The version mismatch between the two API groups is a trap for copy-paste: a manifest that uses the old v1alpha1 form of the collector resource carries a string config, while v1beta1 does not.

The Instrumentation resource is the platform's lever over the SDK tier that I described at the start as out of your hands. It holds the exporter endpoint, the propagators, the sampler and per-language environment, and workloads opt in with a pod annotation:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: platform-default
  namespace: observability
spec:
  exporter:
    endpoint: http://otel-agent.observability:4317
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://otel-agent.observability:4318

A workload opts in by annotating the pod template, not the Deployment's own metadata, with instrumentation.opentelemetry.io/inject-java: "observability/platform-default", and the same pattern exists for inject-python, inject-nodejs, inject-dotnet, inject-go (eBPF-based and needing extra privileges) and inject-sdk, which injects only environment variables. The value can be "true", a name, or "<namespace>/<name>". Injection happens when a pod is created, so existing pods need a restart, and this is the most common "it doesn't work" report.

Note that Python's distribution defaults to gRPC on 4317 while many other agents default to http/protobuf on 4318, which is why the example above sets the Python endpoint to the HTTP port explicitly. The mismatch between protocol and port is still the single most frequent cause of "no data", and a platform default that hides it is worth the effort.

The platform-level benefit is that changing the sampler, the endpoint or the propagators for a whole namespace becomes one edit plus a rollout. The platform-level risk is symmetric: a bad Instrumentation change reaches everything that uses it, so review it like a deployment-wide configuration change and canary it on one namespace first. The zero-code agents also have a cost: they add startup time and memory and can occasionally interfere with an application, so measure an opted-in service against a control before declaring the agent free.

For fleet management beyond Kubernetes, the Open Agent Management Protocol (OpAMP) lets a server report on and configure many agents. The opamp extension reports status, and the separately distributed opampsupervisor can receive remote configuration and restart the Collector. If you run collectors on hosts outside Kubernetes, it is the standard route to seeing versions and configuration drift across them. Its capability flags have changed: accepts_remote_config now implies reporting remote config, reports_remote_config was deprecated in v0.161.0, and accepts_packages implies reporting package status. For eBPF-based zero-code instrumentation of any language on Linux, the OpenTelemetry eBPF Instrumentation (OBI), formerly Grafana Beyla, is available, including as a Helm chart, but it needs elevated privileges such as CAP_SYS_ADMIN or a more specific set, which makes it a security decision for your trust model rather than a convenience toggle.

Incident playbooks

Telemetry incidents are awkward because the tool you would normally use to investigate is the thing that is broken. Each playbook below starts with how you would notice, then the first check, then the usual causes. Keep them next to your runbooks, and keep the commands that work without the pipeline, such as kubectl logs, curl against :8888/metrics and zpages, at the top.

Data stopped arriving

Everything downstream looks empty. Establish where the flow stops before theorising, by walking the path from the backend toward the source and asking at each hop whether accepted counts are healthy.

1. Check the backend's own ingestion metrics and any rejection reasons. A backend rejecting writes (limits, authentication, a full disk) looks identical to silence from far away.
2. Check the gateway's otelcol_exporter_sent_* and otelcol_exporter_send_failed_*. If sends are failing, read the exporter's log lines: Exporting failed. Will retry the request after interval. with a gRPC error tells you whether it is TLS (authentication handshake failed for plaintext versus TLS mismatch, x509: certificate signed by unknown authority for trust), reachability, or authentication.
3. Check otelcol_receiver_accepted_* on the gateway. If accepted is flat while the agents are fine, suspect the path between them: a NetworkPolicy, an expired certificate, a load balancer that lost its targets.
4. Check the agents. kubectl logs for Everything is ready. Begin running and processing data. confirms a clean start; its absence after a restart means the configuration failed.
5. Check the applications last, because platform-wide silence is almost never the applications. If only one service is silent, read its SDK's own log for Transient error StatusCode.UNAVAILABLE encountered while exporting traces and verify the protocol against the port.

The single fastest discriminator is the synthetic heartbeat from the previous section. If it still arrives, the pipeline is fine and the silence is a specific service. If it does not, the failure is the pipeline itself, and the walk above applies.

The gateway is being OOMKilled

Pods restart with OOMKilled and you are losing data every time, because in-memory queues vanish with the process.

First, determine whether it is steady growth or a spike. Memory that climbs with traffic and never falls points to queues holding data for a slow backend or a tail sampler with too many traces. Memory that spikes points to a burst or a large message. Then check the arithmetic from the pipeline section: queue_size times average request size times the number of exporters, plus num_traces times average trace size, against the container limit. Look for data refused due to high memory usage in the logs: its presence means the limiter is working but cannot keep up, and its absence before the kill means the growth outran the check interval or lived somewhere the limiter cannot shed.

Mitigate in this order: set GOMEMLIMIT if it is missing, since it is the cheapest fix; reduce queue and trace budgets to fit; bound message size with max_recv_msg_size_mib; scale out; and only then raise the container limit. Raising the limit first just moves the point of failure.

Queues are full and the backend looks healthy

otelcol_exporter_queue_size is at capacity, enqueue failures are climbing, and the backend reports no errors. This pattern usually has a simple cause: the exporter cannot push as fast as data arrives. Check that num_consumers is high enough for your latency (at ten consumers and 100 ms per request, you top out near a hundred requests a second), check per-request latency to the backend, and check whether batches are too small, since tiny requests waste round trips. Increase consumers and batch size, compress, and confirm the backend can actually accept more. If exporter CPU is idle and consumers are not the limit, the backend or the network is the ceiling and adding Collector replicas will not help.

Traces are incomplete or fragmented

Users report traces that stop partway or split into several. There are four distinct causes, and telling them apart is the skill.

• Broken propagation. A service in the middle does not forward traceparent, so a new trace starts. Symptom: a clean break at one service, every time. Check that the service's HTTP client is instrumented and that propagators match across services: a mix of W3C Trace Context and the deprecated B3 or Jaeger formats produces silent breaks.
• Inconsistent head sampling. Services sampling independently rather than honouring the parent produce fragments. Symptom: random fragments. Check for parentbased_ on every sampler.
• Tail-sampling routing. Spans of one trace reached different tier-2 instances, so each decided on partial data. Symptom: incomplete traces that correlate with scaling events or with a wrong routing_key. Check the load_balancing resolver and recent tier-2 membership changes.
• Decision window too short. Long traces outlive decision_wait, and late spans arrive after the decision was made. Symptom: long-running operations are always truncated. Raise the window, knowing it raises memory proportionally.

A sensitive value was found in telemetry

Someone finds personal data, a token or a payload in a trace or log. Treat it like any secret exposure, in this order. First stop the flow: add a targeted transform or redaction rule at the gateway and deploy it, because that fixes every source at once. Second, rotate any credential that appeared, since backends are searched and copied, and deleting it afterwards does not un-expose it. Third, remove it from the backend using whatever deletion tooling the backend has, and check retention and backups. Fourth, find the source: which instrumentation captured it, and change the SDK configuration so it is not captured. Finally, write the test that would have caught it and put it in CI, and consider whether the residency and notification obligations of your regulator apply, which for regulated employers in the region is a conversation for your data-protection owner, not for the platform team alone.

A Collector release broke something

Roll back to the previous pinned version, which is why pinning exact versions is non-negotiable. Then diff print-config output between the versions' validated configurations, read the changelog entries for the versions you crossed, and look specifically at changed defaults (error_mode, keepalive, the localhost default for endpoints) and renamed metrics. Add a canary step to your process if you did not have one, and add the failing behaviour to a regression test.

The bill doubled

Start from attribution, not from the pipeline. Which tenant and which signal grew? If it is metrics, look for a new high-cardinality attribute (a release that started setting a per-request ID on a metric), compare series counts by metric name, and apply a keep_keys statement at the gateway as a stop-gap. If it is traces, check whether a sampler was removed or whether a new high-volume service arrived without a policy. If it is logs, look for a debug-level flood from a deployment that changed its log level. The remedy is the same order as the cost section: drop, sample, compress, then negotiate retention.

Where OpenTelemetry stops

A platform owner should be able to say "that is not a job for OpenTelemetry" as confidently as "yes". The project is a standard and a toolkit for producing and moving telemetry. It is deliberately not several other things, and confusing it for them causes expensive detours.

It is not a backend. OpenTelemetry does not store, index, query or visualise anything. Someone still has to run or buy a trace store, a metrics database and a log store, and their cost, retention, query performance and availability are separate engineering problems. Choosing OpenTelemetry settles your instrumentation and transport; it leaves the storage decision entirely open, which is its strength and also why "we adopted OpenTelemetry" is an incomplete observability strategy. See Prometheus and Grafana for the most common open-source destinations.

It is not an alerting or incident system. Turning data into pages, routing them, deduplicating and managing on-call belongs to your alerting stack.

It is not free of cost, even for the parts you do not pay for. The zero-code agents add startup time and memory. The SDK's batch queues hold memory in every process. A high-volume service's tracing overhead is real, and the answer is to measure it, not to assume it is negligible.

It is not the best choice for every kind of data. For infrastructure metrics from a single system with a mature exporter, a native Prometheus exporter and scrape is simpler and cheaper than routing the same data through an OTLP pipeline. For a team with a working vendor agent and no plan to change vendors, replacing it buys portability you may never use and costs real migration effort. For very high-rate, low-level profiling and performance work, purpose-built tools go deeper than the alpha profiles signal does today. And for product analytics or business events that need precise counting rather than sampled operational telemetry, an event pipeline is the right tool.

Some pieces are still moving. The Collector is 0.x. Profiles are alpha. Several language SDKs have logs in development. Declarative configuration is stable in the specification but unevenly implemented. Kubernetes Instrumentation is v1alpha1. None of these is a reason to avoid the project, but they are reasons to pin versions, to write the maturity of each dependency into your design, and to budget engineering time for upgrades.

Running OpenTelemetry as a platform

Everything so far describes a system. This section is about the team that runs it, and the thing to internalise is that the product is not the Collector; it is the experience of a service team getting good telemetry with little effort. A platform that is technically flawless and takes a week of tickets to adopt will be routed around by teams who send data straight to a vendor.

A paved road. Offer one documented, supported way to get a service instrumented: a base image or an Operator annotation that injects the agent, a default Instrumentation resource, a library of shared dashboards and alert templates that work out of the box for any service that follows the conventions, and a template repository. Teams that follow the road get telemetry in an afternoon, and the conventions come with it for free. Teams with unusual needs can leave the road, at the cost of owning their own configuration.

Versioned, tested configuration. Keep every tier's configuration in a repository, generated where possible, with CI that runs validate and print-config, rendering tests that feed known data through the configuration and assert on the output (the redaction test above is an example), and a canary stage. The configuration is code and should be reviewed as code.

A support model. Decide what the platform guarantees and what it does not. A reasonable service-level objective is about the path, such as "ninety-nine percent of telemetry from conforming services is queryable within sixty seconds", not about any individual backend. State what you do not promise: delivery during a multi-hour backend outage, retention of data beyond the agreed window, and support for components outside the supported set.

An upgrade cadence. With a release every two weeks you will not track every one, and you should not try. A sensible pattern is to upgrade on a fixed schedule, such as monthly, read every skipped changelog, canary one tier at a time starting with the least critical, and keep a short list of the versions you have run in production. Always know the last-known-good version. Security advisories interrupt the schedule.

A deprecation process of your own. When a component you recommended is deprecated, or a team's SDK is two major versions old, you need a way to tell people and a date. Publish a short compatibility matrix: which SDK versions, which protocol and port, which semconv version the platform supports. Hold office hours. Measure adoption of the current convention set and the conformance percentage.

Ownership boundaries in writing. Who owns the gateway's availability? Who owns a team's dashboards? Who decides a sampling policy change? Who approves a new attribute on a metric? Write down the answers, because the incident where they are unclear is the incident where data is lost and each side assumes the other was responsible.

The review checklist

Use this for a design review of a new pipeline and for a pull request that changes one. Each question names the failure it prevents.

Architecture and reliability

• Is each processor in the tier where the information it needs exists, and is every secret held only at the gateway?
• For each hop, can you state in one sentence what happens to data if the next hop is down for ten minutes and for one hour?
• Is every exporter's sending_queue sized in a unit that means something (items or bytes), and is worst-case memory within the container limit with headroom?
• Is GOMEMLIMIT set, and is the memory_limiter the first processor in every pipeline with a non-zero check_interval?
• Does each destination have its own pipeline, so a failing best-effort exporter cannot fail a critical one?
• If there is a persistent queue, is its directory on a persistent volume, and has the failover been tested?

Scaling and state

• Is every stateful component (tail sampling, span metrics, service graph) behind a routing tier, and what happens to in-flight traces when the routing tier's membership changes?
• Does the load balancer balance gRPC per request, and has a scale-out been verified with per-pod accepted counts?
• Are scrapers sharded with a Target Allocator rather than replicated?
• Are num_traces and decision_wait derived from measured trace rates?

Sampling and metrics

• Are RED metrics computed before sampling, and is every span_metrics dimension reviewed for cardinality?
• Is there exactly one tier that owns the sampling decision per service, and do all head samplers use parentbased_?

Security and data

• Are all network endpoints bound deliberately, with TLS, authentication and NetworkPolicy where exposed, and are pprof and zpages kept off shared networks?
• Does the redaction pipeline use an allow-list, and is there an automated test proving that known-bad input is removed?
• Are credentials referenced as ${env:...} or mounted files, with no secrets in version control or ConfigMaps?
• Is the binary a minimal, pinned, scanned and signed build, and is the RBAC per receiver rather than shared?
• Has the data's storage location been checked against the residency rules that apply to the data?

Operations and change

• Are exact versions pinned, and do CI checks run validate and print-config on every tier with the candidate version?
• Are there no deprecated type names, removed gates or old keys left in the configuration, and is there a canary step?
• Do alerts cover enqueue failures, queue fill, refusals and restarts, and do they reference metric names confirmed against the current version?
• Is there a synthetic heartbeat through the full path?
• Are required resource attributes enforced, with conformance measured?
• Is each team's volume and series count attributed and visible?

The complete picture

Across the whole series the same telemetry path has been examined at increasing depth. It helps to see it once, with the question each level asks of each stage.

A few ideas recur so often that they are worth stating as principles. Put decisions where you have control: the collector tiers, not the SDK. Make loss visible: any design that can drop data must be able to report it independently. State is the thing that scales badly: find every stateful component and give it a routing tier, a sizing calculation and a failure plan. Silence is the failure mode: changed names and changed defaults produce quiet outages, so monitor the monitors, and keep a heartbeat. Every default is a decision someone else made: localhost binding, queue sizes, error modes, batching, the sampler. Read them, and write down where you overrode them and why.

For the neighbouring tools, Prometheus and Grafana are the natural metric and dashboard partners, Kubernetes and Helm are where most of these tiers run and are packaged, and Argo CD is how the configuration reaches the cluster without anyone editing it by hand.

Where the series leaves you

Across the three levels you have gone from a first span printed to the console to owning the telemetry path for an organisation. The same topics carried through, each time one layer deeper:

You should now be able to look at a Collector configuration and see not only what it does but what it risks: where its queues are and how big they can get, which processors hold state, whether a secret or a piece of personal data can leave through it, what the next upgrade will break, and what it will cost.

Carry that review instinct out of here, because, as with any shared infrastructure, most telemetry incidents are cheaper to prevent in a pull request than to diagnose at three in the morning with half your evidence missing.

Sources

• OpenTelemetry documentation
• Concepts
• Sampling
• Resources
• Context propagation
• Baggage
• Semantic conventions
• Collector
• Collector configuration
• Collector internal telemetry
• Collector troubleshooting
• Scaling the Collector
• Collector resiliency
• Collector deployment patterns
• Collector distributions
• Building a custom Collector with ocb
• Configuration security best practices
• Hosting best practices
• Handling sensitive data
• SDK declarative configuration
• SDK environment variables
• OTLP exporter configuration
• Kubernetes Operator
• Operator automatic instrumentation
• Target Allocator
• Helm chart for the Collector
• eBPF instrumentation (OBI)
• Specification
• Semantic conventions specification
• Collector v0.162.0 release notes (core)
• Collector v0.162.0 release notes (contrib)
• Collector distributions v0.162.0
• Collector changelog
• Collector contrib changelog
• Batching migration RFC
• Processor README
• memory_limiter processor README
• Exporter helper README
• tail_sampling processor README
• load_balancing exporter README
• transform processor README
• filter processor README
• Specification changelog
• Semantic conventions changelog
• GenAI semantic conventions
• Declarative configuration v1.2.0
• Kubernetes Operator v0.160.0

Part three of three. At this level an interviewer is not checking whether you know what a span is. They want to see whether you can design a telemetry platform, reason about what breaks first, run the incident when the pipeline itself fails, and argue the trade-offs to people who pay the bill. This file is organised as a cumulative review, then system-design prompts, incident questions, leadership arguments and a self-test. Allow about fifty minutes.

The thirty-second answer

OpenTelemetry is a vendor-neutral standard and toolkit for generating, collecting and exporting traces, metrics, logs and, in alpha, profiles. It defines an API, an SDK, the OTLP wire protocol and semantic conventions, plus a Collector that receives, processes and exports data. It is not a backend: it stores and shows nothing. As a platform owner, I run the Collector tiers, set sampling and redaction policy, control cost and cardinality, and manage upgrades, because the Collector is still a 0.x project that releases about every two weeks.

Say the last sentence unprompted. It tells the interviewer you have operated it.

Part one: the model in one screen

The vocabulary you must use precisely. A signal is a category of telemetry. A resource describes the entity producing it, and service.name is the most important attribute. A receiver accepts data, a processor changes it, an exporter sends it, a connector is an exporter in one pipeline and a receiver in another and can change signal type, and an extension does work outside the pipeline, such as health checks and authentication. A pipeline wires them together per signal.

The default ports and protocols. OTLP is protobuf over gRPC on 4317 or HTTP on 4318, with HTTP paths /v1/traces, /v1/metrics and /v1/logs. The Collector's own metrics are on 8888. health_check is 13133 and zPages is 55679. A protocol and port mismatch is still the most common cause of no data.

The names that changed. The receiver is otlp, but the exporter is now otlp_grpc (with otlp_http for HTTP). Old names like k8sattributes, filelog, hostmetrics and spanmetrics are deprecated aliases for k8s_attributes, file_log, host_metrics and span_metrics. Knowing this is a cheap signal that your knowledge is current.

Two defaults interviewers love. Every Collector server endpoint now defaults to localhost, so in a container you must bind 0.0.0.0 explicitly. And memory_limiter's check_interval defaults to zero, which is invalid, so you must set it.

The commands to have ready

Note that print-config needs no feature gate any more. If you say to pass --feature-gates=otelcol.printInitialConfig, you are quoting an outdated page and it will fail.

Part two: system-design prompts

For each prompt, the structure that works is the same. Clarify requirements and numbers, sketch tiers, then walk the failure of each hop, then state what you would not build. Interviewers mark you down for naming components without explaining what each does under failure.

Prompt 1: Design the telemetry pipeline for forty teams on one Kubernetes platform

Clarifying questions. How many services and which languages? What volume of spans per second, metrics series and log lines? Which backends are already chosen? What are the retention and residency requirements? Who is on call for the pipeline? Is there a budget?

A strong answer. Two tiers. A DaemonSet agent per node takes OTLP from pods on the node IP, enriches with k8s_attributes and resource_detection, tails pod logs with file_log, collects kubelet and host metrics, and forwards to a gateway. The agent is deliberately small and stateless because a bad agent release reaches every node. A gateway Deployment behind an L7 or client-balanced endpoint authenticates senders with mTLS plus an authenticator extension, applies redaction with an allow-list, routes per tenant, holds backend credentials, and exports with a persistent queue. Inject the SDK with the Operator's Instrumentation resource so service teams opt in with an annotation, which gives the platform control over endpoint, propagators and sampler.

Then the parts that separate seniors from mid-level candidates. Tail sampling needs a second tier: stateless collectors use the load_balancing exporter routing by trace ID, through a headless Service, into a stateful tier that runs tail_sampling. Span-derived metrics are computed before sampling. Prometheus scraping uses the Target Allocator so targets are sharded rather than duplicated. Per-tenant identity comes from the authenticated connection or from agent-side namespace enrichment, never from a client-set attribute. The pipeline's own metrics go to a separate path, with a synthetic heartbeat.

What to say you would not build at first. Kafka between tiers, a separate gateway per tenant, exemplars everywhere. Each has a trigger: Kafka when outages must exceed what a disk queue can hold or bursts exceed what a gateway absorbs, per-tenant gateways when a tenant needs isolation, and so on. Naming the trigger shows you are not gold-plating.

The follow-up they will ask. "Which component breaks first?" Answer: at modest scale, the load balancer if it is L4 (gRPC connection pinning); then the stateful tier's memory; and throughout, the backend, because adding collectors does not help when the backend is the limit.

Prompt 2: Cut the telemetry bill by half without hurting incident response

Start with attribution, not with tools. Which signal, which tenant, which attribute? The answer is usually one of four things, and you should be able to name the lever for each.

Then protect incident response. RED metrics are generated from the full stream before sampling, so dashboards stay accurate. Errors and slow traces are kept by policy, so the traces you need during an incident still exist. Keep the sampling policy in version control so it has owners and review. Finish with attribution: publish per-team volume so the savings persist, because a one-time cleanup without visibility regrows within a quarter.

A trap to avoid. Do not propose a blanket head-sampling percentage. It cuts errors at the same rate as successes and is the answer that loses the traces you most want.

Prompt 3: Design redaction for a regulated employer in the Gulf

Clarify which data classes and which regulations apply, and where the backend is hosted. Residency can be a constraint on the tracing backend's region, so ask about it before choosing a vendor.

Three layers. Prevent capture at the source: configure instrumentation not to record query text with literals or arbitrary headers. Scrub centrally: the redaction processor with allow_all_keys: false and an explicit allowed_keys list, which fails closed so that a new unreviewed attribute is removed, plus blocked_values patterns as a backstop, plus transform for specific rewrites such as stripping query strings from url.full. Test the control: a CI job that sends known-bad spans through the gateway configuration and fails if anything sensitive comes out.

Two honest limits. Pattern matching finds patterns, not meaning, so the allow-list is the real control. And the transform processor's default error_mode is ignore, so a statement that fails is skipped and the original data passes through; for a scrubber you would consider propagate, and you would test with malformed input.

Prompt 4: Tail sampling is losing traces. Redesign it

Ask how it is deployed. If a single instance, scaling changes the picture; if several behind a plain balancer, spans of one trace are splitting across instances, so each decides on partial data. The fix is trace-ID-aware routing in front: the load_balancing exporter with routing_key: traceID and a DNS resolver on a headless Service, or the k8s resolver with endpoint-slice RBAC.

Then the sizing arithmetic, which is the detail that shows experience. In-flight traces are roughly new traces per second times decision_wait. If num_traces is below that, the processor evicts traces early and decides on incomplete data. Memory is num_traces times average trace size, which is frequently the dominant cost of a gateway. Finally the operational hazard: when the stateful tier scales, the hash ring moves and in-flight traces fragment, so scale it rarely and in small steps, and alert on the rate of incomplete traces.

Prompt 5: Roll out OpenTelemetry to a fleet that already uses a vendor agent

The honest answer begins with "do I have to?" Replacing a working agent buys portability you may never use. Adopt where something is changing anyway: new services, a backend migration, or a goal to stop paying per-host agent fees.

A migration plan: introduce the Collector as a gateway that accepts both the vendor's existing protocol and OTLP, run both agents in parallel for a window on a pilot set, compare the data for equivalence, move service by service behind the Operator annotation, and keep the old path until dashboards and alerts have been migrated and verified on the new attribute names. Use OTEL_SEMCONV_STABILITY_OPT_IN with a dup value so the old and new attribute names are both emitted during the transition where the instrumentation supports it. State the exit criteria, not just the steps.

Part three: incident-response questions

The format the interviewer wants is: how do you notice, what do you check first, what are the usual causes, what do you do now, and what do you change afterwards.

"Traces stopped arriving in the backend an hour ago"

Walk the path from the backend toward the source. Is the backend rejecting writes? Are the gateway's otelcol_exporter_send_failed_* rising, and what do the exporter log lines say (Exporting failed. Will retry the request after interval. with a gRPC error distinguishes TLS mismatch, trust, reachability and authentication)? Is otelcol_receiver_accepted_* flat on the gateway while agents look healthy, pointing to the network, a NetworkPolicy, an expired certificate or a lost load-balancer target? Ask what changed in the last day: a Collector upgrade can rename a metric or change a default and produce silence rather than an error.

What a weak answer does. It starts at the application and debugs one service's SDK, when the silence was platform-wide. The synthetic heartbeat is the fast discriminator: if it still arrives, the pipeline is fine.

"The gateway keeps getting OOMKilled"

Distinguish steady growth from spikes. Steady growth points at queues holding data for a slow backend or a tail sampler with too many traces. Then the arithmetic: queue_size times request size times exporters, plus num_traces times trace size, against the container limit. Check whether GOMEMLIMIT is set. Look for data refused due to high memory usage, because the limiter refusing is working while a kill with no refusals means growth outran the one-second check or lived where the limiter cannot shed. Mitigate by setting GOMEMLIMIT, shrinking budgets, bounding message size with max_recv_msg_size_mib, scaling out, and only then raising the limit.

"We scaled the gateway from three pods to ten and nothing improved"

gRPC connections are long-lived, and an L4 balancer pins each client connection to one backend. Verify with per-pod otelcol_receiver_accepted_spans: the new pods are idle. Fix with an L7 gRPC-aware balancer, client-side balancing with balancer_name, or agents that use the load_balancing exporter against a headless Service. If instead the CPU is idle everywhere and exporter queues are full, the backend is the limit and more replicas will make it worse.

"Someone found customer emails in our traces"

Stop the flow first with a gateway rule, since it fixes every source at once. Rotate any credential that appeared. Delete from the backend and check retention and backups. Find and fix the capturing instrumentation. Write the CI test that would have caught it. Then involve the data-protection owner, because for regulated employers the notification and residency questions are not the platform team's to settle alone. Note the ordering principle: contain, rotate, clean, then root-cause.

"After the upgrade, every alert went quiet"

Changed metric names and units produce silence. Between 0.147 and 0.148, units became singular, the :8888 labels changed to underscore forms, memory_limiter metrics were renamed in v0.155.0, and the batch-size metrics moved in v0.159.0. Compare the series your alerts reference against what :8888/metrics emits on the new version, fix the expressions, and add a canary step that compares dashboards between versions before promotion. Also reinforce the structural lesson: alert on absence as well as on thresholds.

"Traces are fragmented, and only some of them"

Four causes, each with a distinct signature. A clean break at one service every time is broken propagation, often mismatched propagators. Random fragments are inconsistent head sampling, so check that every sampler is parentbased_. Fragments correlated with scaling events are tail-sampling routing. Truncated long operations mean decision_wait is shorter than the trace. Naming the signature, rather than listing causes, is what the interviewer wants.

"The exporter queue is full but the backend says it is healthy"

The exporter cannot push as fast as data arrives. Check num_consumers against per-request latency (ten consumers at 100 ms is about a hundred requests per second), batch size, compression, and whether the sizer is counting requests rather than items. If otelcol_exporter_enqueue_failed_* is non-zero, data is being dropped now and that is a page.

Part four: arguing trade-offs to leadership

These questions test communication as much as knowledge. The format is: state the decision in one sentence, give the options with costs in terms the audience cares about, recommend one, and say what would change your mind.

"Why OpenTelemetry instead of our vendor's agent?"

The business case is optionality and leverage, and you should say the costs too. Instrumentation is the expensive thing to change, because it lives in every service; standardising it on an open protocol means a change of backend is a configuration change rather than a re-instrumentation project. It also means one set of conventions across languages, and negotiating leverage with vendors. The costs: the Collector is 0.x with frequent breaking changes, you take on operating the tiers, and some signals and languages are less mature (logs in Python and JavaScript are still in development). Recommend adopting it at the instrumentation edge now, and moving the backend only when there is a reason. What would change your mind: a vendor agent that is cheaper to operate and a credible plan never to change vendors.

"Can we keep all the traces? Storage is cheap."

Storage is rarely the cost that matters; ingestion, indexing and query are. Show the arithmetic: spans per second times average size times retention. Then offer the option that keeps what matters: tail sampling that retains every error and every slow trace, with metrics computed before sampling so counts and rates remain exact. Quantify what you lose (the repetitive, healthy traces that nobody reads) and what you keep (everything that explains an incident). Offer a measurable check after rollout: the percentage of incident investigations that were blocked by a missing trace.

"Why is the platform team asking for a month to upgrade a patch-level version?"

It is not patch-level: the Collector is 0.x, and a minor version can carry breaking changes, removed gates and renamed metrics. The month is reading two changelogs per skipped version, validating and rendering every tier's configuration, and canarying, which is how you avoid silent alert loss. Offer a faster path for security advisories and a schedule for the rest. The cost of skipping it is the incident where monitoring goes quiet during an unrelated outage.

"Do we need Kafka in front of the gateway?"

Name what it buys (durability across long outages, burst absorption, replay, multiple consumers) and what it costs (a stateful system to operate and upgrade, another hop where data can be lost or delayed). Start with a persistent queue on the gateway and an explicit requirement: "survive a one-hour backend outage". If the disk queue meets it, stop. If the requirement is a day, or the same stream must feed several consumers, Kafka earns its place.

"Should each team run its own Collector?"

The trade is autonomy against consistency and cost. Per-team collectors let a team tune its own pipeline but duplicate operation, multiply upgrade work, scatter credentials, and make central policy (redaction, sampling, attribution) impossible to enforce. The recommended compromise is a shared platform pipeline with a paved road, and an escape hatch for unusual needs, where the team owning a custom collector also owns its upgrades and its compliance with the redaction policy.

"How do we know the telemetry itself is trustworthy?"

Three mechanisms: a synthetic heartbeat that proves the path end to end, alerts on the pipeline's own drop and queue metrics from a path that does not depend on the pipeline, and a conformance percentage showing which services send the required attributes. Report these as a platform service-level indicator, such as the share of telemetry queryable within a minute of emission.

Common interview questions

Why does order matter in the processors list? memory_limiter goes first so it can refuse data before anything expensive happens. Dropping and sampling come early so later stages work on less. k8s_attributes must run before batching because batching merges requests and discards the connection context it uses. Batching goes last, or is replaced by exporter-side batching.

What is the difference between head and tail sampling? Head decides at the start of a trace in the SDK without knowing the outcome, cheaply and statelessly. Tail decides after seeing the whole trace in a collector, so it can keep all errors and slow traces, at the cost of holding every in-flight trace in memory and requiring that all spans of a trace reach one instance.

How do you scale a Collector? By component class. Stateless components scale by replicas behind a balancer that understands gRPC. Scrapers are sharded with the Target Allocator. Stateful components need routing so each key lands on one instance. Scale on refused counts, queue fill and CPU, and do not scale if the backend is the bottleneck.

What does the batch processor's status mean for new designs? It is beta and still shipped, but the documentation now prefers exporter-side batching through sending_queue::batch, and a queue_batch processor is in development. For new configuration prefer the exporter's batching, and keep it in the standard platform configuration.

How do you handle secrets in a Collector configuration? Reference them with ${env:NAME} or mounted files, never inline. Sensitive fields are redacted in print-config and logs by default. Keep backend credentials only at the gateway so applications hold none.

What do you do about high-cardinality metrics? Budget them as a product of distinct attribute values. Put identifiers on spans, not metrics. Enforce at the gateway with keep_keys or delete_key, keep the span_metrics dimensions list short, and set per-tenant series limits in the backend.

What is OBI? OpenTelemetry eBPF Instrumentation, formerly Grafana Beyla: Linux kernel-level auto-instrumentation for many languages with no code change. It needs elevated privileges, so it is a security decision under your trust model as well as a convenience.

What is OpAMP for? Managing fleets of agents remotely: reporting status through the opamp extension, and, with the separate supervisor, receiving remote configuration and restarting the Collector.

What changed about Collector configuration recently that a senior should know? Component names moving to snake_case with deprecated aliases, the otlp exporter becoming otlp_grpc, batching moving into exporters, print-config replacing the gated print-initial-config, endpoints defaulting to localhost, and declarative SDK configuration via OTEL_CONFIG_FILE, which makes the SDK ignore other OTEL_* variables unless referenced in the file.

Final self-test

Answer aloud, without notes, in under a minute each.

1. Where do you put redaction, and why not in the SDK?
2. Why can a memory_limiter set to 75 percent still allow an OOM kill?
3. What does block_on_overflow change?
4. Why does a scaled-out gRPC gateway sometimes stay idle?
5. Draw the two-tier path for tail sampling and say what goes wrong when the second tier scales.
6. Why must RED metrics be computed before sampling?
7. Choose between a file queue, Kafka and accepting loss for a one-hour backend outage.
8. Name the three layers of defence for personal data in telemetry.
9. What is the fail-closed redaction configuration, and what is its limit?
10. Name four things you check in a pull request that upgrades the Collector.
11. Which kind of change causes silent alert loss?
12. Name two situations in which you would advise against adopting OpenTelemetry.

If you stumble on more than two, reread the matching section of the detailed guide: the gateway, scaling, sampling, security and upgrade sections cover these in that order.

Part three of three. At this level the problems are operational: a gateway that falls over at the traffic peak, a bill that doubled after one release, a customer's email address sitting in a trace backend, and a Collector upgrade that left every alert silent. Start with the error table, then the practices, verification habits, recipes and the checklists for pull requests and design documents.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These rarely fail a first test. They fail at ten times the traffic, or three releases later.

Three of those rows share a pattern: silence rather than errors. A renamed metric, a changed default and a scrubber that skips a failing statement all continue without a message. They are the category that platform owners must design monitoring around, because a loud failure gets fixed the same day and a quiet one is found during an incident.

The practices that pay off most

In order of leverage. The earlier ones prevent whole classes of incident.

Why pinning and gating come first

Both cost almost nothing and remove the largest single source of platform incidents. With a two-week release cadence and explicit permission to break on any minor, an unpinned Collector is a change you did not review arriving at a time you did not choose. Pinning makes the change a pull request. The CI gate makes the pull request honest, because a configuration that fails validate never reaches a cluster, and the print-config output gives a reviewer a rendered diff rather than a pile of layered overrides.

Why the tiers carry the policy

Every policy you put in an SDK must be rolled out across every team's code, and will drift. Every policy you put in the gateway is one deployment. The practical test is to ask, for each control, "if this were wrong tomorrow, how long would it take to fix everywhere?" For the SDK the answer is weeks. For the gateway it is minutes.

Decide what you drop, and write it down

Whether the platform favours protecting traces, metrics or logs during an overload is a policy choice, and it should be a written one. In the incident you will not have time to decide. A sentence in the runbook such as "under sustained overload we refuse logs before metrics, and metrics before traces" turns a judgment made under pressure into a configuration you made in advance.

Practice cards

1. Fill a queue on purposePoint an exporter at an address where nothing listens with queue_size: 10, send telemetrygen traffic, and watch otelcol_exporter_queue_size and enqueue_failed. Then set block_on_overflow: true and compare what the sender sees.
2. Make the limiter shedSet a very low limit_mib and push traffic until you see data refused due to high memory usage. Find otelcol_processor_refused_* in the metrics and explain what a client receives.
3. Watch a load balancer pin connectionsDeploy three gateway replicas behind a plain Service, then scale to six and compare otelcol_receiver_accepted_spans per pod before and after clients reconnect.
4. Lose a trace to routingRun two tail-sampling instances behind a round-robin balancer and count incomplete traces. Put the load_balancing exporter by traceID in front and count again.
5. Break and fix a scrubberWrite a redaction rule, send a span with an email in url.full, and confirm it is removed. Then feed a malformed attribute that makes a transform statement fail and see whether the original passes through.
6. Run the renameTake an old configuration, run print-config, rename every deprecated type to its current name, run it again, and diff the two. Read the startup warnings from the original.
7. Kill the gateway on purposeStop a gateway for ten minutes with an in-memory queue, then with a file-backed one on a volume, and compare what arrived afterwards.
8. Audit one service's seriesList a service's metrics with their attributes and distinct values, multiply, and find the attribute responsible for most of the series.

The configuration every gateway should start from

A small, opinionated baseline saves every team from rediscovering the same defaults. This is a starting point for a gateway, not a finished design, and every value should be checked against your traffic.

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage/queue:
    directory: /var/lib/otelcol/queue

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
        include_metadata: true
        tls:
          cert_file: /etc/otel/tls.crt
          key_file: /etc/otel/tls.key
          client_ca_file: /etc/otel/clients-ca.crt

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.request.method
      - http.response.status_code
      - http.route
      - url.path
      - service.name
      - service.namespace
      - service.version
      - deployment.environment.name
      - k8s.namespace.name
      - error.type
    summary: info

exporters:
  otlp_grpc/backend:
    endpoint: ${env:BACKEND_ENDPOINT}
    headers:
      authorization: ${env:BACKEND_TOKEN}
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 20000
      sizer: items
      storage: file_storage/queue
      batch: {}
    retry_on_failure:
      enabled: true
      max_elapsed_time: 1800s

service:
  extensions: [health_check, file_storage/queue]
  telemetry:
    logs: { level: INFO, encoding: json }
    metrics:
      level: normal
      readers:
        - pull: { exporter: { prometheus: { host: 0.0.0.0, port: 8888 } } }
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, redaction], exporters: [otlp_grpc/backend] }
    metrics: { receivers: [otlp], processors: [memory_limiter, redaction], exporters: [otlp_grpc/backend] }
    logs:    { receivers: [otlp], processors: [memory_limiter], exporters: [otlp_grpc/backend] }

Read it the way a reviewer would. The limiter is first. Redaction is an allow-list. The token comes from the environment, and the queue has a unit that can be budgeted and a place on disk. Batching is in the exporter. There is no logging exporter (removed; use debug), and the exporter type is otlp_grpc, not the deprecated otlp. Note also what the baseline omits on purpose: no tail sampling and no routing, because those belong to a tier you add when a requirement demands it. The redaction on the logs pipeline is also omitted here because log bodies need their own rules, and a copy-pasted trace allow-list would drop every attribute a log record carries.

Verifying, not assuming

A control you have never seen refuse anything is untested. These checks are cheap and belong in CI or in a regular game day.

# Does the configuration parse and validate on the pinned version?
docker run --rm -v "$PWD/gateway.yaml:/cfg.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.162.0 validate --config=/cfg.yaml

# What does it actually resolve to after every layer and override?
docker run --rm -v "$PWD/gateway.yaml:/cfg.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.162.0 print-config --config=/cfg.yaml

# Which components are compiled in, and how stable are they?
docker run --rm otel/opentelemetry-collector-contrib:0.162.0 components

# Is it alive, and is it dropping anything?
curl -s localhost:13133/
curl -s localhost:8888/metrics | grep -E 'enqueue_failed|send_failed|refused|queue_size'

# Does a synthetic trace make it through the whole path?
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 3

Three habits turn those commands into reliability. Run the first two on every tier in CI. Keep the grep in the last command as a saved query on your dashboard so nobody has to remember it during an incident. And keep the synthetic trace as a scheduled job that alerts when it does not arrive at the backend, since it is the only check that covers the whole path.

For the redaction control, the test is a pipeline run in CI that sends a known-bad span and asserts on the output:

# Send a span carrying an email and an unlisted attribute through the gateway config,
# capture the debug exporter output, and fail if either survives.
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 1 \
  --telemetry-attributes 'url.full="https://x.example/?email=alice@example.com"' \
  --telemetry-attributes 'customer.national_id="1234567890"'
docker logs gateway-under-test 2>&1 | tee out.log
! grep -E 'alice@example.com|1234567890' out.log

The final line fails the job if the sensitive strings appear. Adapt the flags to the version of telemetrygen you install, and confirm with telemetrygen traces --help.

Keeping the platform fresh

The Collector, the SDKs and the conventions all move, and a platform that stands still slides into the situation where every upgrade is a migration. Build a steady rhythm.

A fixed upgrade cadence. Monthly is reasonable. Read the core and contrib changelogs for each version skipped, focusing on breaking changes, removed gates and renames. Canary one tier at a time, starting with the least critical, and record the last-known-good version so a rollback is one edit.

A rename backlog. After each upgrade, start a canary and read the deprecation warnings the Collector logs at startup, each of the form "<old>" alias is deprecated; use "<new>" instead. Convert them in a dedicated pull request with nothing else changed, and verify with print-config that only type names differ. Remember the two renames with no alias: the queuebatch processor became queue_batch and dynamic_sampling became adaptive_tail_sampling, and leaving the old name means a startup failure.

A semconv rhythm. Semantic conventions evolve through stable releases. Turn on dual emission with OTEL_SEMCONV_STABILITY_OPT_IN (values such as http/dup and database/dup), migrate dashboards and alerts, then turn it off. Doing it in the other order breaks every dashboard on the day a service upgrades its library.

A rebuild schedule for custom distributions. A binary you built with ocb does not pick up dependency fixes by itself. Rebuild on a schedule, scan and sign, and subscribe to the project's CVE announcements.

A compatibility matrix. Publish which SDK versions, protocol and port, and semconv version the platform supports, and when support for an old one ends. Teams cannot comply with a policy nobody wrote down.

Incident playbooks

Short versions of the playbooks in the detailed guide, for the moment when you want the steps and not the explanation. Every step that depends on the pipeline is avoided here, in favour of commands that work without it.

Data stopped arriving

1. Is the synthetic heartbeat still arriving? If yes, the silence is one service. If no, continue.
2. curl -s gateway:8888/metrics | grep -E 'exporter_sent|send_failed'. Failing sends mean the backend path; read the exporter log lines for TLS, trust, reachability or authentication.
3. grep accepted on the gateway. Flat while agents are healthy means the network between tiers: NetworkPolicy, certificate expiry, balancer targets.
4. Ask what changed in the last day, and roll back the most recent change before forensics.

Gateway OOMKilled

1. Steady growth or a spike? Steady points to queues or tail-sampling buffers.
2. Check the arithmetic against the container limit, and whether GOMEMLIMIT is set.
3. Mitigate in order: set GOMEMLIMIT, shrink budgets, bound max_recv_msg_size_mib, scale out, then raise the limit.

Sensitive data found

1. Stop the flow with a gateway rule and deploy it.
2. Rotate any credential that appeared.
3. Delete from the backend and check retention and backups.
4. Fix the capturing instrumentation, add the CI test, and inform the data-protection owner.

Bill doubled

1. Attribute by tenant and signal.
2. Metrics: find the new attribute and apply keep_keys at the gateway. Traces: check for a removed sampler or a new high-volume service. Logs: look for a level change.
3. Fix at the source, set a backend limit, and publish the per-team numbers.

Cost and performance levers worth knowing

One performance point worth stating plainly: the largest throughput win on a busy gateway is usually in the exporter, not the processors. Raising num_consumers and batch sizes, and compressing, often buys more than tuning any processor. Measure exporter latency before you change anything else.

Machine-learning and LLM workloads specifically

The audience for this site runs model-serving and LLM applications, and they have a few extra considerations.

• Prompts and completions are sensitive data. Instrumentation for LLM calls can record prompt and response text, which may contain personal or confidential content and can be very large. Decide deliberately whether to capture content, and scrub it at the gateway. Large attributes also inflate cost; max_recv_msg_size_mib and truncation with transform help.
• The GenAI semantic conventions moved. The gen_ai.* attributes now live in open-telemetry/semantic-conventions-genai, and are deprecated in the main semantic conventions repository as of v1.42.0. Check which version your instrumentation libraries follow before building dashboards on those names.
• GPU hosts are still hosts. Run the agent and host_metrics on them like any other node, and accept that accelerator metrics come from vendor exporters or dedicated receivers rather than from the core.
• Latency distributions matter more than averages. Inference latency is long-tailed. Use histograms with buckets suited to the model's real range, and exponential histograms if your backend supports them.
• Keep trace and eval systems connected. Tools such as Langfuse accept OTLP, so the same pipeline can feed them as well as your general traces, provided the gateway routes by tenant and does not scrub the attributes they need.

Review checklists

For a pull request that changes a Collector configuration

• Does validate pass, and is the rendered print-config output attached or diffed?
• Is the version pinned, and are all type names current, with no deprecated alias newly introduced?
• Is every new component referenced under service, and does it exist in the distribution (components)?
• Is memory_limiter first, with a non-zero check_interval, and is GOMEMLIMIT set on the deployment?
• Are queue sizes in a meaningful unit, and does worst-case memory fit the container limit?
• Does any new endpoint bind a non-local address, and if so, where are TLS, authentication and NetworkPolicy?
• Are credentials referenced by ${env:...} and not written into the file?
• Does a new processor that changes data have a test that feeds input and asserts on output?
• For a transform or filter, is error_mode chosen deliberately?
• If it changes a metric name, label or unit, which alerts and dashboards reference it?

For a design document

• What are the availability and loss requirements, stated as a failure and a duration?
• Where does each processor run, and why that tier?
• Which components are stateful, and how is their routing and sizing handled?
• What is the sampling policy, who owns it, and are metrics computed before it?
• What is the trust model: who may send, what is exposed, what holds credentials?
• What personal data can enter the pipeline, where is it scrubbed, and where is the data stored?
• How is each tenant identified and isolated, and what limits exist?
• How will the pipeline itself be monitored, from a path that does not depend on it?
• What is the upgrade and rollback plan, and who owns it?
• What is the expected cost, and how is it attributed?

The checklist to run before shipping

• Versions pinned, configuration validated and rendered in CI, canary planned.
• Memory arithmetic done, GOMEMLIMIT set, limiter first.
• Redaction allow-list in place and tested with known-bad input.
• Enqueue-failure, queue-fill, refusal and restart alerts live, and a heartbeat running.
• Required resource attributes enforced, conformance measured.
• Per-team volume visible.
• A written answer to "what do we drop under overload".
• A runbook whose first steps work without the telemetry pipeline.

Carry these habits out of the series: the most valuable thing a platform owner can do for a telemetry system is to make its failures loud, its changes reviewable and its costs visible.