The Complete Prometheus Guide

This is part one of three. It covers everything you need to do real work with Prometheus, not a teaser. By the end you can install a server, point it at a machine and at your own application, read the numbers it collects with its query language, turn a useful query into a saved rule, and write an alert that fires when something is genuinely wrong. Mid-level and Senior take the same topics further; nothing you learn here is thrown away.

The guide targets Prometheus 3.15, released in September 2026. Version 3.13 is the long-term-support line and is supported until July 2027, so if you work somewhere conservative you will meet it instead. Everything here works on both, and where a setting changed between 2.x and 3.x the text says so, because the internet is full of older tutorials that no longer match the tool. There is no supported 2.x release any more, so this guide teaches 3.x only.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and monitoring concepts only stick once you have watched a graph move because of something you did. You need a terminal, a laptop that can run a small program on port 9090, and curiosity. Nothing here needs a cloud account or a cluster.

What Prometheus is, and the problem it solves

Prometheus is an open-source monitoring and alerting toolkit. It regularly asks your programs "how are you doing right now?", writes the answers into a database built for numbers that change over time, lets you ask questions about that history, and tells a human when the answers look bad. It is a graduated project of the Cloud Native Computing Foundation, the same foundation that hosts Kubernetes, and it has become the default metrics system in the container world.

To see why it exists, picture running a web service without it. A customer writes in: "your site is slow." You log in to the server and run top. The CPU looks fine right now, but the slowness was twenty minutes ago and the evidence is gone. You add a log line, redeploy, and wait for it to happen again. Monitoring is the practice of recording numbers continuously, before you know what you will need to ask, so that when someone says "it was slow at 14:20" you can open a graph and look at 14:20.

Before Prometheus, teams typically had two styles of tool. Check-based systems such as Nagios ran a script on a schedule and showed a green or red light: disk is fine, disk is full. That answers "is it broken?" but not "how fast is it filling?" or "what did the last week look like?". Push-based metric systems such as StatsD and Graphite received numbers sent by your applications, but they made it hard to know whether silence meant "nothing happened" or "the application died". Prometheus combined the best ideas: store numeric time series like Graphite, but pull them so that a missing answer is itself a fact the system records.

The diagram is the whole architecture at beginner level. The word for the pulling step is scraping: every few seconds, Prometheus makes an ordinary HTTP request to an address such as http://myapp:8000/metrics and reads the plain text it gets back. That text is a list of metric names and their current values. Anything that can serve text over HTTP can be monitored, and that is why the ecosystem grew so quickly.

Three design decisions explain most of how Prometheus behaves, so notice them now.

Pull instead of push. Because Prometheus initiates every scrape, it always knows what it should be monitoring and can tell when a target stops answering. It records a synthetic metric called up with value 1 for a healthy scrape and 0 for a failed one. With a push system, a dead application simply stops sending and nobody notices. With Prometheus, death is a number you can alert on.

Every server stands alone. A Prometheus server keeps its data on its own local disk and does not depend on a cluster, a distributed database, or a message queue. When the network falls apart, the monitoring still works for everything it can reach. The price is that one server is one machine's worth of capacity and is not replicated by itself; the Senior guide covers how teams handle that.

Numbers, not events. Prometheus stores metrics: counts, rates, sizes, durations. It does not store individual log lines or individual request traces. It can tell you that 3 percent of requests failed in the last five minutes, but not which customer's request failed. For that you pair it with logs and traces, which other tools in this catalogue cover. It is also a poor fit where every single event must be counted exactly, such as billing, because a scrape sees a sampled picture at fixed intervals.

What people use it for:

The mental model: metrics, labels and time series

Prometheus has a small vocabulary, and once it is clear the rest of the tool is unsurprising. There are four nouns to hold: the metric, the label, the sample, and the time series they combine into.

A metric name says what is being measured: http_requests_total, node_memory_MemAvailable_bytes, process_cpu_seconds_total. By convention names use lowercase words joined by underscores, end in a unit when there is one (_seconds, _bytes), and counters end in _total. Since Prometheus 3.0 names may contain any UTF-8 characters, but the classic shape [a-zA-Z_:][a-zA-Z0-9_:]* is what nearly everything you meet will use. Colons are reserved for rules that you write yourself, which we meet later.

A label is a key and value pair that adds a dimension to a metric. The request counter by itself would tell you the total for the whole application. With labels it can tell you the total per method, per status code and per URL path. Labels are written inside curly braces:

http_requests_total{method="GET", handler="/api/items", status="200"}

A sample is one measurement: a number together with the time it was taken, with millisecond precision. Values are 64-bit floating-point numbers, so even a count is stored as a float.

A time series is the stream of samples that share the same metric name and the same complete set of labels. This is the most important sentence in the guide. The metric name together with every label defines identity. http_requests_total{method="GET", status="200"} and http_requests_total{method="GET", status="500"} are two different time series, each with its own history, even though they share a name.

Internally the metric name is itself stored as a label called __name__. You will occasionally see it in queries such as {__name__="up"}, which is the same as writing up. Label names that begin with two underscores are reserved for Prometheus's own use, and an empty label value is treated as if the label did not exist at all.

Why labels need restraint

Labels are powerful, and that power has one sharp edge you should learn on day one. Every unique combination of label values is a separate time series, and each series costs memory and disk. The number of distinct series is called cardinality, and it is the main thing that makes a Prometheus server slow or expensive.

Suppose you label a request counter with method (about 5 values), status (about 10) and handler (about 50). That is at most 2,500 series, which is fine. Now someone adds a user_id label because it seems useful. With a million users the same metric becomes billions of potential series and the server runs out of memory. The rule is simple: label values must come from a small, bounded set. Never put user IDs, email addresses, request IDs, session tokens or raw URLs with query strings into labels. Those belong in logs.

Jobs and instances

Two labels appear on almost everything you collect, and Prometheus adds them for you. An instance is one single thing you scrape, identified by its host:port address. A job is a group of instances that do the same thing, such as all five copies of your API. So job="api", instance="10.0.0.12:8000" says "this number came from that copy of the API". Queries then ask either about one instance or about the whole job by summing across instances, and that is how you get from "one pod is slow" to "the service is slow".

For every scrape Prometheus also writes a few metrics of its own about the scrape itself. The one to remember is up: it is 1 when the last scrape succeeded and 0 when it failed. It is the simplest health check you will ever use, and the first query almost everyone runs.

The four metric types

Your applications describe their numbers using four types. The type is a convention used by client libraries and documentation, and it tells you which query to use. Prometheus's database itself only stores floats (and, newer, native histograms), but picking the right query for the type is the difference between a correct graph and a nonsense one.

A counter is a number that only goes up, and resets to zero when the program restarts. Requests served, errors seen, bytes sent. A raw counter is almost useless to look at, because "41,872,390 requests since the last restart" tells you nothing. What you want is the rate of change: requests per second over the last five minutes. So the rule for counters is: never graph a counter directly; always wrap it in rate() or increase(). Both handle restarts correctly by noticing when the value drops and treating it as a reset.

A gauge is a number that can go up and down: memory in use, temperature, the number of items in a queue, the number of requests currently in flight. You can graph a gauge directly, and use functions such as avg_over_time or max_over_time to summarise it over a window.

A histogram measures how values are distributed, most often request durations. Instead of storing every duration, it counts how many observations fell into each of a set of buckets such as "under 0.1 seconds", "under 0.5 seconds", "under 1 second". A classic histogram called http_request_duration_seconds exposes three families of series: http_request_duration_seconds_bucket with an le label (less than or equal to) for each bucket boundary, http_request_duration_seconds_sum with the total of all observed values, and http_request_duration_seconds_count with the number of observations. The buckets are cumulative, and the function histogram_quantile() turns them into statements like "the 95th percentile latency is 240 milliseconds". Histograms can be combined across instances, which is why they are preferred.

A summary also measures distributions but calculates the percentiles inside the application and exposes them as quantile labels. It looks convenient and has a serious drawback: percentiles calculated separately in each instance cannot be averaged or combined into a correct overall percentile. For that reason new instrumentation nearly always uses histograms.

Since version 3.9 there is also a stable feature called native histograms, which store a whole distribution in a single series with automatically sized buckets. They are cheaper and more precise than classic histograms, and you switch them on with the configuration option scrape_native_histograms: true. You do not need them to get started, and the Mid-level guide returns to them. The older command-line flag --enable-feature=native-histograms no longer does anything and only prints a warning, so ignore tutorials that still tell you to use it.

Installing Prometheus and checking the setup

Prometheus is a single statically linked program written in Go, so installing it means getting one file and running it. That is also why the binary download is the best way to learn: nothing is hidden. Pick the path that matches your machine, then run the same verification steps.

Linux

Download the release archive from the official GitHub releases page, check its checksum, and unpack it. Use linux-arm64 instead of linux-amd64 on an ARM machine.

VER=3.15.0
curl -LO https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz
curl -LO https://github.com/prometheus/prometheus/releases/download/v${VER}/sha256sums.txt
sha256sum --check --ignore-missing sha256sums.txt
tar xzf prometheus-${VER}.linux-amd64.tar.gz
cd prometheus-${VER}.linux-amd64
ls

Inside you will find the prometheus server, a helper called promtool that we use for checking files, an example prometheus.yml, and the licence files. The checksum step matters: it confirms the file you downloaded is the file the project published. sha256sum should print prometheus-3.15.0.linux-amd64.tar.gz: OK.

macOS

The simplest route is Homebrew, which installs both the server and promtool:

brew install prometheus
prometheus --version

Homebrew places the example config at $(brew --prefix)/etc/prometheus.yml. You can run it in the foreground while learning with prometheus --config.file=$(brew --prefix)/etc/prometheus.yml, or as a background service with brew services start prometheus. If you prefer the tarball, download prometheus-3.15.0.darwin-arm64.tar.gz on Apple Silicon or the darwin-amd64 build on Intel, and if macOS refuses to open the unsigned binaries, remove the quarantine flag with xattr -d com.apple.quarantine ./prometheus ./promtool.

Windows

Download prometheus-3.15.0.windows-amd64.zip from the releases page, extract it, and run it from PowerShell inside the extracted folder with .\prometheus.exe --config.file=prometheus.yml. Allow port 9090 through Windows Defender Firewall if it asks. The project does not ship a Windows service wrapper, so for anything permanent people use Docker, WSL2, or a third-party wrapper such as NSSM.

Docker, on any system

If you already have Docker (see the Docker guide), one command starts a server:

docker run -d --name prometheus -p 9090:9090 \
  -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml \
  -v prometheus-data:/prometheus \
  prom/prometheus:v3.15.0

The first volume mounts your config file into the place the image expects it. The second is a named volume for the database, and you should never skip it: without it the collected data vanishes with the container. One detail catches people out. The image starts with default arguments that point at /etc/prometheus/prometheus.yml and /prometheus. If you pass any extra arguments, they replace those defaults, so you must repeat --config.file and --storage.tsdb.path yourself.

Kubernetes, in one paragraph

On Kubernetes nobody installs the binary by hand. The usual route is the Prometheus Operator and the kube-prometheus-stack Helm chart from the prometheus-community repository, which installs a server, Alertmanager, node metrics and ready-made dashboards together. That belongs to the later guides. Learn on a laptop first, because the concepts are identical and the moving parts are fewer. The Helm guide is the natural follow-up when you get there.

Verify the install

Whatever route you took, verify it the same way. Start the server with the example configuration that came with it:

./prometheus --config.file=prometheus.yml

The server runs in the foreground and prints log lines. The two you want are msg="Completed loading of configuration file" and msg="Server is ready to receive web requests.". Open a second terminal and ask it three things:

./prometheus --version
curl -s localhost:9090/-/healthy
curl -s localhost:9090/-/ready

--version prints the version, which should say 3.15.0. The /-/healthy endpoint answers "Prometheus Server is Healthy." as soon as the process is alive, and /-/ready answers "Prometheus Server is Ready." once it has finished starting and can serve queries. During a slow start the ready endpoint returns 503 "Service Unavailable", which is normal and is how orchestrators know when to send traffic. Finally, open http://localhost:9090 in a browser. You should see the query page of the Prometheus interface.

The configuration file, line by line

Prometheus is configured with one YAML file, usually called prometheus.yml, whose path you give with --config.file. Nearly every question of the form "how do I make Prometheus monitor X" is answered by editing this file, so it is worth reading slowly once. Here is the example that ships with the project, which is also the smallest useful configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

rule_files:
  # - "first_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

The file has four blocks that matter at this level.

global holds defaults for everything else. scrape_interval is how often Prometheus scrapes each target, and evaluation_interval is how often it evaluates your rules. One detail trips people up: the built-in default for both is one minute, not fifteen seconds. The example sets 15 seconds, which is a common and sensible choice because it gives a graph a fresh point four times a minute without loading the targets. There is also scrape_timeout, the time Prometheus waits for one answer, which defaults to 10 seconds and must never be larger than the interval.

scrape_configs is the heart of the file: a list of jobs. Each entry has a job_name (which becomes the job label) and a list of targets, here hard-coded under static_configs. A target is just host:port. By default Prometheus fetches the path /metrics over HTTP, and you can change that with metrics_path and scheme if your application differs.

rule_files lists files of recording and alerting rules, which come later in this guide. alerting says where Alertmanager lives. Both are empty in the example, and the commented lines show the shape to fill in.

There are many more keys in the full reference, for authentication, TLS, relabeling, limits and dozens of ways to discover targets automatically. You will meet a few in later sections, and the rest when a real need appears. The important skill now is not memorising keys but checking a file before you use it.

Validate before you run

Prometheus ships with promtool, a checker that understands the configuration syntax. Make it a reflex to run it after every edit:

./promtool check config prometheus.yml

On success it prints SUCCESS: prometheus.yml is valid prometheus config file syntax. On a mistake it tells you the line and the problem, for example an unknown field name or a bad indentation. This matters more than it might seem, because a server that is told to reload a broken file keeps running the old configuration and logs an error, so a typo can silently leave you with stale settings.

Changing the configuration while running

After editing the file you do not need to stop the server. There are three ways to tell it to re-read the file. You can send the process a hang-up signal with kill -HUP <pid>, which always works. You can enable the lifecycle endpoints by starting with --web.enable-lifecycle and then call curl -X POST http://localhost:9090/-/reload. Or you can start with --config.auto-reload, a stable option since version 3.12, and Prometheus will check the file every 30 seconds by itself. Whichever you choose, confirm the result afterwards by querying prometheus_config_last_reload_successful: it is 1 when the last reload worked and 0 when the new file was rejected.

The lifecycle flag deserves a caution. It is off by default because it lets anyone who can reach the port reload or shut down your server. On a laptop that is fine, and on a shared network it is not.

A tour of the web interface

The interface at port 9090 is where you will spend the first hours, and it is deliberately simple. Version 3.0 replaced the old interface with a new one, so screenshots in older tutorials will look different, but the same ideas are there.

The Query page is the main one. You type an expression into the box and press Execute. The result appears as a Table, showing the current value of each matching series, or as a Graph, showing how each value changed over a time window you choose. Start with the table, because it shows you exactly which series match and which labels they carry. Then switch to the graph when you want to see change over time. As you type, the box suggests metric names, which is the best way to discover what a server actually holds.

The Status menu has the pages you use for diagnosis. Target health lists every scrape target, with its state (UP or DOWN), its labels, how long ago it was last scraped, how long the scrape took, and the last error message if there was one. This is where you look whenever a graph is empty. Service discovery shows what each discovery mechanism found, and what labels the targets had before and after relabeling. Runtime and build information shows the version and start time. Configuration shows the configuration that is actually loaded, which is useful when you are not sure that your edit was picked up. TSDB status lists the metrics and labels with the most series, which is how you find a cardinality problem. And Flags shows the command-line settings the server started with.

The Alerts page lists every alerting rule and its state. You will use it once you write rules. Together, these pages make Prometheus unusually transparent: when something looks wrong, there is almost always a page that shows the state directly, before you have to read logs.

Monitoring a real machine with node_exporter

Watching Prometheus watch itself proves the plumbing, but it is not monitoring anything you care about. Most software does not speak the Prometheus format natively. Linux does not expose a /metrics page; it exposes files under /proc and /sys. The bridge is an exporter: a small separate program that reads the statistics of some other system and serves them in the Prometheus text format. The exporter for Linux and other Unix-like machines is node_exporter, and it is the first one nearly everyone installs. (On Windows the equivalent is windows_exporter, which listens on port 9182.)

The pattern generalises: there are exporters for databases such as MySQL, PostgreSQL and Redis, for message queues, for network devices, for GPUs, and for probing websites from the outside (the blackbox exporter). The shape is always the same. Run the exporter next to the thing it reports on, then add it to scrape_configs as one more job.

Install node_exporter from the Prometheus download page, or with your package manager. The version at the time of writing is 1.12. Running it needs no configuration:

tar xzf node_exporter-1.12.1.linux-amd64.tar.gz
cd node_exporter-1.12.1.linux-amd64
./node_exporter

It starts listening on port 9100. Before wiring it into Prometheus, look at what it serves, because reading the exposition format once removes all mystery from it:

curl -s localhost:9100/metrics | head -n 20

The output looks like this:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 184523.48
node_cpu_seconds_total{cpu="0",mode="user"} 3921.17
# HELP node_memory_MemAvailable_bytes Memory information field MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 5.123584e+09

Lines starting with # HELP describe the metric in words, lines starting with # TYPE give its type, and the other lines are the samples: a name, optional labels in braces, and a value. That is the entire format. Anything that can print text like this can be monitored, which is why a beginner can write a working exporter in twenty lines of any language.

Now add a second job to the configuration file, next to the existing one:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]
        labels:
          environment: "laptop"

The labels block under a target adds extra labels to every series from those targets, which is a simple way to mark an environment or a team. Validate the file, reload the server, and open Status then Target health. Both jobs should now show UP. Notice that you never listed a port for the metrics path or the scheme: the defaults of /metrics and http applied.

PromQL part one: selecting data

The data is now flowing. The skill that turns it into answers is PromQL, the Prometheus Query Language. It is a language designed for one job: selecting time series and calculating on them. It is not SQL and it does not look like it, so give yourself an hour to get used to the shape. The good news is that a small set of ideas covers most real use.

Type queries into the box on the Query page. The simplest query is just a metric name, which returns the current value of every series with that name:

node_memory_MemAvailable_bytes

This is called an instant vector: one value per matching series, all at the same moment. Most queries start here. To narrow the result you add label matchers in braces. There are four operators, and they are worth memorising:

Regular expressions in Prometheus are fully anchored, which means the pattern must match the whole label value, as if it had a caret at the start and a dollar sign at the end. So status=~"5.." matches 500 and 503 but not 2500. You do not write the anchors yourself. Combining matchers narrows the result further:

node_cpu_seconds_total{job="node", mode!="idle"}

One rule surprises beginners: a selector must contain at least one matcher that cannot match an empty string. The expression {job=~".*"} is refused because it could match everything, while {job=~".+"} is allowed. The easy way to stay clear of the error is to start every selector with a metric name.

Range vectors and the time window

An instant vector is one point per series. To look at a stretch of history you add a time range in square brackets, which turns it into a range vector:

node_cpu_seconds_total{mode="idle"}[5m]

This returns, for each series, all the samples from the last five minutes. Durations use the units ms, s, m, h, d, w and y. You rarely want to look at a range vector on its own, because it is a list of raw samples. Its job is to be handed to a function such as rate() that reduces it back to one number per series. That pairing, range vector in and instant vector out, is the pattern behind most useful queries.

You can also ask about the past directly with the offset modifier. node_memory_MemAvailable_bytes offset 1h returns the value one hour ago, which is handy for comparing "now" with "this time yesterday" using offset 1d. Since version 3.14 durations can even be small expressions, such as [5m * 2], although you will rarely need that at the start.

What the four types of value are

Every PromQL expression returns one of four kinds of value, and error messages talk about them, so learn the names now. An instant vector is a set of series with one sample each. A range vector is a set of series with a window of samples each. A scalar is a single plain number such as 0.95. A string is rarely used. The most common beginner error message, expected type range vector in call to function "rate", got instant vector, simply means you forgot the square brackets.

PromQL part two: rates, aggregation and arithmetic

Selecting raw series only gets you so far. The real power arrives with three operations: rates for counters, aggregation across series, and arithmetic between them.

Rates for counters

Recall that counters only go up. The question you actually ask is "how fast is it going up?". The function rate() answers it: given a range vector, it returns the per-second average increase over that window, and it corrects for counter resets. To see how much CPU time your machine's CPUs spent per second in each mode:

rate(node_cpu_seconds_total[5m])

The five minutes in brackets is the window that rate() averages over. A good rule is to pick a window at least four times your scrape interval, so with 15-second scrapes five minutes is comfortable and one minute is the practical minimum. A window that is too short gives jumpy graphs and can return nothing when a scrape is missed. A window that is too long smooths away real spikes.

Two relatives are worth knowing. increase(x[1h]) gives the total increase over the window, which reads more naturally for questions such as "how many errors in the last hour". It is rate() multiplied by the window length. irate() uses only the last two samples and so is very twitchy; it is useful for zoomed-in graphs of fast-changing values and a poor choice for alerts. When in doubt, use rate().

Aggregation across series

The CPU query above returned a row for every CPU core and every mode, which is too much. Aggregation operators collapse many series into fewer. The main ones are sum, avg, min, max, count, and topk (the top N by value). You choose which labels survive with by, or which to discard with without:

sum by (mode) (rate(node_cpu_seconds_total[5m]))

This keeps only the mode label and sums across cores, giving one number per mode. The same idea scales from one laptop to a thousand pods: to get the request rate of a whole service you sum across its instances, and to see the busiest five you use topk(5, ...). Notice how the order reads from the inside out. The innermost part selects and applies rate(), and the outer part aggregates the result.

Arithmetic and comparison

You can use ordinary operators, + - * / % ^, between a series and a number, or between two sets of series. Between two sets, Prometheus matches series that have identical labels and calculates pair by pair. That is how you turn two raw gauges into a percentage:

100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

This reads: the fraction of memory that is available, subtracted from one, expressed as a percentage, which is the percentage of memory in use. The two metrics carry the same labels, so they pair up automatically. A classic CPU utilisation query builds on the idle mode:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

Read it inside out: the per-second idle time of each core over five minutes, averaged over the cores of each machine, turned into "how much of the time was not idle", as a percentage. Comparison operators such as >, < and == work as filters. node_filesystem_avail_bytes < 10 * 1024 * 1024 * 1024 keeps only the series below 10 GiB and drops the rest. That filtering behaviour is exactly what an alert needs: it fires for every series that survives the comparison, and stays silent when the result is empty.

When the labels on two sides do not match, you can tell Prometheus which labels to compare with on(...) or which to ignore with ignoring(...). The Mid-level guide covers matching across different label sets in depth. At this stage it is enough to know the error that signals it: many-to-many matching not allowed means that the two sides have more than one series per match group, and you need to aggregate or choose labels more carefully.

A short list of queries worth memorising

Two of these deserve a word. absent() returns 1 only when the series does not exist, so it catches the case that up == 0 cannot: a target that was never configured or has disappeared from discovery, leaving no up series to be zero. histogram_quantile() takes the bucket series of a classic histogram, and you must keep the le label when aggregating, because the buckets are the information it works from. The predict_linear() function fits a straight line through the window and extrapolates, which turns "the disk is 80 percent full" into the far more useful "the disk will be full before morning".

Recording rules: saving a query as a metric

Some queries are expensive, or are used in many places. A dashboard with twenty panels that each compute the same error ratio across thousands of series will make the server do the same work twenty times every refresh. A recording rule solves this by evaluating the expression on a schedule and saving the result as a new time series, which later queries read cheaply.

Rules live in separate files listed under rule_files. A rule file groups rules, and the rules in a group are evaluated in order, at the group's interval:

groups:
  - name: node_rules
    rules:
      - record: instance:node_cpu_utilisation:ratio_rate5m
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
      - record: instance:node_memory_utilisation:ratio
        expr: 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

Each rule has a record (the name of the new metric) and an expr (the query). The naming style level:metric:operations is a convention worth following, and colons in a name are reserved for exactly this purpose. The first part says which labels are kept (here instance), the middle says what is measured, and the last says which operations were applied. It looks fussy, but a name like that tells a colleague what a series is without opening the rule file.

Hook the file into the configuration, then check both files before reloading:

rule_files:
  - "rules.yml"

./promtool check rules rules.yml
./promtool check config prometheus.yml

After a reload, wait one evaluation interval and query instance:node_cpu_utilisation:ratio_rate5m. It now behaves like any other metric. The Status menu also shows a Rule health page listing each group, how long its last evaluation took and whether it failed. Keep an eye on that duration: if a group takes longer than its interval, evaluations are skipped.

Alerting: from a query to a notification

A dashboard only helps when someone is looking at it. Alerting rules watch a query for you and raise an alert when it is true. Prometheus itself only decides that an alert is firing. Deciding who to tell and how is the job of a second program, Alertmanager, which keeps the two concerns cleanly apart.

Writing an alerting rule

Alerting rules sit in the same kind of file as recording rules. Add a new group to rules.yml:

groups:
  - name: node_alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} target {{ $labels.instance }} is down"
          description: "The scrape has failed for more than 2 minutes."

      - alert: DiskWillFillSoon
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Disk {{ $labels.mountpoint }} on {{ $labels.instance }} will fill within 4 hours"

An alerting rule has four parts. The alert field is its name. The expr is a query, and the alert fires for every series the expression returns, which is why comparisons such as up == 0 (a filter) fit so well. labels are extra labels attached to the alert, and the severity label is a convention that Alertmanager can use for routing. annotations carry human-readable text, and they can include templates such as {{ $labels.instance }}, which inserts the label value of the series that triggered it, and {{ $value }}, which inserts the number.

The pending state and for

The for clause is what keeps you from being woken by a blip. An alert moves through three states. It is inactive when the expression returns nothing. When the expression first returns something it becomes pending, and Prometheus starts a timer. Only when the expression has stayed true for the whole for duration does the alert become firing and get sent to Alertmanager. If the condition clears during the pending phase, the alert quietly goes back to inactive and nobody hears about it.

Choosing for is a judgement call between speed and noise. Two minutes is reasonable for "a target is down", since a single failed scrape is usually a hiccup. Thirty minutes suits slow-moving problems like disk fill. Without a for value the alert fires on the first evaluation that is true, which is almost always too jumpy for paging a human.

Alertmanager: deduplicate, group, route

Alertmanager is a separate program, version 0.34 at the time of writing, that listens on port 9093. Prometheus sends it the firing alerts, and Alertmanager does the things you actually want from a notification system. It deduplicates repeated alerts, groups related ones so that fifty failing pods produce one message instead of fifty, routes alerts to different receivers based on their labels, lets you silence alerts during maintenance, and inhibits lower-priority alerts when a higher-priority one already explains them.

Its configuration is a routing tree. This minimal one groups by alert name and sends everything to a webhook:

route:
  receiver: team-webhook
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: team-webhook
    webhook_configs:
      - url: "http://localhost:5001/"

group_wait is how long Alertmanager waits after the first alert of a group before sending, so that related alerts arrive together. group_interval is how long before it sends updates about a changed group, and repeat_interval is how often it reminds you about a problem that is still open. Real setups replace the webhook with Slack, email, PagerDuty or Microsoft Teams receivers, and use child routes to send severity: page alerts somewhere louder than severity: ticket ones.

Start it, then tell Prometheus where it is by filling in the alerting block. Since Prometheus 3.0 only version 2 of the Alertmanager API is supported, and it is the default:

./alertmanager --config.file=alertmanager.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

Test a rule before trusting it

You do not have to break a production system to see whether an alert works. On your laptop, stop node_exporter and watch the Alerts page: InstanceDown becomes pending, and after two minutes firing. Open http://localhost:9093 and the alert appears in Alertmanager. For rules you need to be sure about, promtool test rules runs a rule against synthetic input series, so you can assert in a file that "with these values, this alert fires at minute five". That belongs in continuous integration, and the Mid-level guide shows it.

Instrumenting your own application

Exporters cover other people's software. For your own services, you add instrumentation: a few lines of code using a client library that keeps counters and gauges in memory and serves them on /metrics. Official libraries exist for Go, Java, Python and Ruby, and the community maintains many more. The Python one, prometheus-client, is the quickest to try and the one most used in ML serving code. Install it with pip install prometheus-client.

The example below is a tiny program that pretends to serve requests. It exposes a counter of requests by outcome, a histogram of how long each one takes, and a gauge of requests currently being handled:

import random
import time

from prometheus_client import Counter, Gauge, Histogram, start_http_server

REQUESTS = Counter(
    "demo_requests_total", "Requests handled, by outcome.", ["outcome"]
)
LATENCY = Histogram(
    "demo_request_duration_seconds", "Time spent handling a request."
)
IN_FLIGHT = Gauge("demo_requests_in_flight", "Requests being handled now.")


def handle_request() -> None:
    with IN_FLIGHT.track_inprogress(), LATENCY.time():
        time.sleep(random.uniform(0.01, 0.4))
        outcome = "error" if random.random() < 0.05 else "ok"
        REQUESTS.labels(outcome=outcome).inc()


if __name__ == "__main__":
    start_http_server(8000)   # serves /metrics on port 8000
    while True:
        handle_request()

Run it with python app.py, then curl -s localhost:8000/metrics | grep demo_. You will see demo_requests_total{outcome="ok"} and demo_requests_total{outcome="error"} climbing, and the demo_request_duration_seconds_bucket family with its le labels. Notice how little code this took: declaring each metric once at module level, and updating it where the work happens. A few habits make instrumentation age well.

Declare metrics once, at the top. Create them as module-level objects, not inside the function that handles requests. Creating the same metric twice raises an error, and it would also reset the values.

Follow the naming conventions. Use a prefix for your application, end counters in _total, and put the unit in the name in base units: seconds rather than milliseconds, bytes rather than megabytes. Dashboards and alerts written by other people assume this.

Keep label values bounded. The outcome label has two values, which is ideal. Never label with anything that grows with users or requests, for the reasons in the cardinality section.

Measure the things users feel. The three most useful signals for a service are the rate of requests, the errors among them, and the duration of each. Many teams call this the RED method, and this single example already provides all three.

Then add the application as a scrape job and watch its numbers appear:

scrape_configs:
  - job_name: "demo-app"
    static_configs:
      - targets: ["localhost:8000"]

Now the queries from earlier apply to your own code. The request rate is sum(rate(demo_requests_total[5m])), the error ratio is sum(rate(demo_requests_total{outcome="error"}[5m])) / sum(rate(demo_requests_total[5m])), and the 95th percentile latency is histogram_quantile(0.95, sum by (le) (rate(demo_request_duration_seconds_bucket[5m]))). This is also the pattern for model servers: wrap the prediction call in a histogram timer and count outcomes, and you have latency and error dashboards for your model from day one. Many serving frameworks, among them BentoML, KServe and vLLM, expose such metrics already, so check before writing your own.

How Prometheus stores data, and how long it keeps it

Everything Prometheus collects goes into its own time-series database (TSDB), stored in the directory set by --storage.tsdb.path, which defaults to data/ next to the program. You do not need to manage it day to day, but a simple picture of how it works helps you answer "where did my disk go" and "what happens if it crashes".

New samples first land in an in-memory area called the head block, and to survive a crash each one is also appended to a write-ahead log, the wal directory, which is replayed when the server restarts. About every two hours the head is written out to disk as an immutable block, a directory holding the chunks of samples, an index and a small meta.json. Later, small blocks are merged into larger ones by compaction, which keeps queries efficient. Samples are compressed well: on average a sample takes one to two bytes on disk. That is why a modest server can keep months of history.

Retention

By default Prometheus keeps 15 days of data and then deletes the oldest blocks. You change this in the configuration file, which is the current way since version 3.8:

storage:
  tsdb:
    retention:
      time: 30d
      size: 50GB

If you set both, whichever limit is reached first wins. You will still find tutorials that use the flags --storage.tsdb.retention.time and --storage.tsdb.retention.size. They still work but are deprecated, and the file setting takes precedence and can be reloaded without a restart. Expired blocks are cleaned up periodically, so do not expect the disk to shrink the second you lower the number.

To plan disk, use this estimate: the retention in seconds, times the number of samples ingested per second, times about two bytes. You can see the ingest rate with rate(prometheus_tsdb_head_samples_appended_total[5m]). A server scraping 100,000 series every 15 seconds ingests around 6,700 samples per second and needs on the order of a gigabyte a day.

What local storage is, and is not

The local database is fast and simple, but it is a single copy on one disk. It is not replicated, and it is not designed for network file systems such as NFS, which do not give the guarantees it needs. Put it on a local SSD or a block volume, and treat long-term retention and high availability as separate problems with separate answers: remote write to a long-term store, running two identical servers, or both. Those are Senior topics. One final rule for containers: two Prometheus processes must never share a data directory. The second one fails at startup with opening storage failed: lock DB directory: resource temporarily unavailable, because the first holds a lock file.

Day-to-day operation and the HTTP API

Once a server is running you mostly talk to it through four things: the web interface, promtool, signals, and the HTTP API. The first three you have met. The API matters because everything the interface does, and everything Grafana does, goes through it.

Every query you can type in the box can be sent over HTTP. An instant query uses /api/v1/query, and a range query with a start, end and step uses /api/v1/query_range:

curl -s 'localhost:9090/api/v1/query' --data-urlencode 'query=up'

curl -s 'localhost:9090/api/v1/query_range' \
  --data-urlencode 'query=rate(node_cpu_seconds_total{mode="idle"}[5m])' \
  --data-urlencode "start=$(date -d '1 hour ago' +%s)" \
  --data-urlencode "end=$(date +%s)" \
  --data-urlencode 'step=60'

(On macOS, replace the date -d '1 hour ago' part with date -v-1H.) The answer is JSON with a status of success or error, and the data inside it. The same API offers /api/v1/targets for scrape targets, /api/v1/rules and /api/v1/alerts for rules and their state, /api/v1/labels and /api/v1/series for exploring what exists, and /api/v1/status/buildinfo for the version. Wrapping curl around these endpoints is how scripts and other tools integrate with Prometheus. The promtool command can do the same from the terminal:

./promtool query instant http://localhost:9090 'up'
./promtool check healthy --url=http://localhost:9090

Reading the logs

Logs go to standard error, in a format of time=... level=INFO source=... msg=... since version 3.0, which uses the key=value style of the Go slog package. Older tutorials show a different style starting with ts= and caller=. You can switch to JSON with --log.format=json. To see more detail, the log level is info by default. The command-line flag --log.level still works but has been deprecated since 3.15, and the configuration file now carries the setting, which can be reloaded without restarting:

runtime:
  log_level: debug

Watching Prometheus itself

Because Prometheus is scraping itself, you can build a small list of self-checks from day one, and they are the seed of the alerts you will want in production. Whether the last reload worked: prometheus_config_last_reload_successful. How many series are in memory: prometheus_tsdb_head_series. Whether any rule evaluations fail: increase(prometheus_rule_evaluation_failures_total[1h]). And whether notifications reach Alertmanager: increase(prometheus_notifications_errors_total[1h]). A monitoring system that nobody is watching is the classic irony, so sooner or later you will add a rule that alerts when Prometheus itself is unhealthy.

Common errors and how to read them

Most Prometheus problems announce themselves clearly, in the logs, on the Target health page, or in a PromQL error box. The skill is knowing where to look first. This section lists the messages you are most likely to meet as a beginner, with the cause and the fix.

Error loading config followed by a parsing message. The configuration file is not valid YAML or has an unknown field. Run promtool check config prometheus.yml and fix the reported line, most often indentation or a tab character.

scrape timeout greater than scrape interval for scrape config with job name "X". The timeout must not exceed the interval. Lower scrape_timeout or raise the interval.

found multiple scrape configs with job name "X". Two jobs share a name. Names must be unique across the whole configuration.

Target DOWN with connection refused. Nothing is listening at that address. Start the exporter, or check the port. If it runs in a container, check that the target address is reachable from the Prometheus container, because localhost inside a container means the container itself, not your laptop.

Target DOWN with context deadline exceeded. The target did not answer within the scrape timeout. The cause is usually a firewall that silently drops packets, or a target that is too slow to generate its output.

Target DOWN with server returned HTTP status 404. The path is wrong, often because the application serves metrics somewhere other than /metrics. Set metrics_path in the scrape config.

received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified. Since 3.0 Prometheus is strict about the content type of a scrape response. You pointed it at a web page instead of a metrics endpoint, or an exporter sends no proper type. Fix the target, or as a last resort set fallback_scrape_protocol: PrometheusText0.0.4.

listen tcp 0.0.0.0:9090: bind: address already in use. Another program, often a previous Prometheus you forgot to stop, holds the port. Stop it, or start this one with --web.listen-address=:9091.

opening storage failed: lock DB directory. Two servers point at the same data directory. Stop one. With Docker, this also happens when you restart a container while the old one is still shutting down.

opening storage failed: ... permission denied. The user running Prometheus cannot write the data directory. In Docker this appears after switching between image variants, because the default image runs as user nobody and the -distroless image runs as user 65532. Change the ownership of the volume to match.

A PromQL parse error such as 1:1: parse error. The box shows the position. Typical culprits are a missing bracket, a missing quote, or rate(x) without a range, which gives expected type range vector in call to function "rate", got instant vector. Another is vector selector must contain at least one non-empty matcher, which means a selector like {job=~".*"} that could match anything.

query processing would load too many samples into memory in query execution. The query touches more than the limit of 50 million samples. Narrow the selector or the time range, or aggregate earlier.

A graph that is empty though the target is UP. Check that the metric name is spelled the way the exporter spells it, which you can do with curl against the target. Check the label values exactly. Also check that you are not in a time range that predates the data.

After a reload, nothing changed. The new file may have been rejected, in which case the old configuration keeps running. Look at the logs, query prometheus_config_last_reload_successful, and run promtool check config.

Putting it all together

Here is one small project that uses everything above, and it is worth building end to end once. Goal: monitor a machine and a small application, record a useful ratio, and alert when the application's error rate climbs or a target stops answering. Allow about an hour.

1. Make a working folderCreate a directory with the Prometheus files, node_exporter, and the app.py from earlier, plus empty prometheus.yml, rules.yml and alertmanager.yml.
2. Write the configurationOne file with a 15-second interval, rule files, the Alertmanager address, and three jobs: prometheus, node, demo-app.
3. Write the rulesTwo recording rules for node utilisation, a recorded error ratio for the app, and two alerts.
4. Validate everythingRun promtool check config and promtool check rules until both say SUCCESS.
5. Start and verifyStart the four processes and confirm all three targets are UP.
6. Break itIncrease the error probability in app.py and watch the alert go pending, then firing, then reach Alertmanager.

The configuration file brings it together:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

rule_files:
  - "rules.yml"

storage:
  tsdb:
    retention:
      time: 15d

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "demo-app"
    static_configs:
      - targets: ["localhost:8000"]

And the rules, combining a recording rule for the error ratio with the alerts it feeds:

groups:
  - name: demo
    rules:
      - record: job:demo_errors:ratio_rate5m
        expr: >
          sum(rate(demo_requests_total{outcome="error"}[5m]))
          / sum(rate(demo_requests_total[5m]))

      - alert: DemoHighErrorRatio
        expr: job:demo_errors:ratio_rate5m > 0.10
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Demo app error ratio is {{ $value | humanizePercentage }}"

      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} is down"

Start the pieces in separate terminals, in the order that feels natural: node_exporter, the app, Alertmanager, then Prometheus. Open the Target health page and confirm three UP rows. Graph job:demo_errors:ratio_rate5m. Now change 0.05 in app.py to 0.30, restart it, and follow the alert through its life: pending for two minutes, firing, and then visible in Alertmanager at port 9093. Restore the value and watch it resolve. You have now completed the full loop that production monitoring follows: expose, scrape, store, query, record, alert, notify.

What you can now do, and what comes next

You started without knowing what a scrape was. You can now explain the pull model and why it makes a dead service visible. You can read the notation metric{label="value"}, know why a label must have bounded values, and choose between counters, gauges, histograms and summaries. You can install Prometheus on your machine, validate its configuration with promtool, reload it without a restart, and read the Target health page when something is wrong. You can monitor a host with node_exporter, instrument a Python program, and write the PromQL queries for the request rate, error ratio and latency percentile. You can turn a query into a recording rule, write an alert with a sensible for duration, and route it through Alertmanager. You can also decode most of the error messages you will meet in your first month.

Each of those is a real skill that appears in job descriptions for platform, DevOps and MLOps roles across the region, where teams run Prometheus on their own Kubernetes clusters or through managed services from the large cloud providers.

What comes next. The Mid-level guide goes under the surface. It covers service discovery so that targets appear automatically, relabeling to reshape what you collect, the trickier parts of PromQL such as vector matching and subqueries, how to unit-test your rules, how to secure a server, and the integrations with Grafana, Kubernetes and OpenTelemetry. The Senior guide treats Prometheus as a platform: high availability, long-term storage, cardinality control at scale, upgrades, and multi-team governance. Two neighbours are worth reading in parallel. Grafana, the dashboard tool most teams put in front of Prometheus, has its own guide at /student-guides/grafana. And if you run on Kubernetes, the Kubernetes guide is the place to learn what Prometheus will be watching there. For an MLOps engineer, the natural next exercise is to serve a model behind a small API, instrument it with a histogram and a counter as you did here, and build the latency and error alerts for it.

Sources

• Prometheus overview
• Release cycle and LTS policy
• Download page
• Prometheus 3.15.0 release
• Prometheus 3.15 changelog
• Migration guide to Prometheus 3.0
• Installation
• Getting started
• Command-line flags
• promtool reference
• Configuration reference
• Recording rules
• Alerting rules
• Storage
• Querying basics
• Querying operators
• Querying functions
• HTTP API
• Management API
• Data model
• Metric types
• Jobs and instances
• Metric and label naming
• Recording rule naming practices
• Alertmanager overview
• Security model
• Docker image Dockerfile, v3.15.0
• Example configuration, v3.15.0

This is part two of three, the interview companion to the detailed guide. It assumes you have read that guide or worked with Prometheus a little. Interviewers for platform, DevOps, SRE and MLOps roles use Prometheus questions to test one thing above all: do you understand how the data gets from an application to a graph and to an alert, and can you reason about it when it breaks? Memorised definitions help for ten seconds; a clear mental model helps for the whole hour. The guide is organised so you can read it top to bottom once and then return to single sections before an interview.

The thirty-second answer

Prometheus is an open-source monitoring and alerting system that periodically pulls numeric metrics from HTTP endpoints, stores them as labelled time series in its own local database, and lets you query them with a language called PromQL. It evaluates rules on those queries, and sends alerts to a separate program, Alertmanager, which groups and routes the notifications.

If you are given one more breath, add why it matters: because it pulls, it knows what it is supposed to monitor and records a health metric called up, so a dead service shows up as a number and not as silence. And because every server is standalone and stores data locally, it keeps working when other parts of your infrastructure do not, which is exactly when you need monitoring.

Practise saying that answer aloud, slowly, without reading. It should take less than thirty seconds and contain the words pull, time series, PromQL and Alertmanager.

The model

Interviewers love to ask you to draw the architecture on a whiteboard. Draw it in this order and explain each arrow.

Note the direction of the first arrow. It points from Prometheus to the target, because Prometheus initiates the request. Candidates who draw it the other way reveal that they think it is a push system. A separate line from Prometheus to Grafana, which queries it over the HTTP API to draw dashboards, completes the picture.

The data model in one sentence. A time series is identified by a metric name plus a complete set of key-value labels, and holds a stream of timestamped float values. Change any label value and you have a different series. Be ready to write one: http_requests_total{method="GET", status="200"}.

The four metric types. A counter only goes up and resets on restart, so you query it with rate(). A gauge goes up and down and you read it directly. A histogram counts observations in cumulative buckets, exposing _bucket (with an le label), _sum and _count, and you turn it into percentiles with histogram_quantile(). A summary computes quantiles inside the application and cannot be aggregated across instances. A good extra sentence is that native histograms, stable since version 3.9, store a distribution in one series and must be switched on with scrape_native_histograms: true.

Jobs and instances. An instance is one scrape endpoint; a job is a set of instances with the same purpose. Prometheus adds both as labels.

Scrape, evaluate, alert. Scraping happens on scrape_interval (default one minute, commonly set to 15 seconds). Rules are evaluated on evaluation_interval. An alerting rule goes inactive, pending, firing, and only firing alerts are sent to Alertmanager.

Where the data lives. In a local time-series database: a write-ahead log and in-memory head for recent data, persisted about every two hours into immutable blocks that are later compacted. Default retention is 15 days.

Pull versus push

This is the single most common conceptual question, and a strong answer is balanced, not a sales pitch.

In a pull model the monitoring server decides when to collect and from whom. Advantages: Prometheus knows what should exist, so it detects dead targets through up == 0. You can run a second Prometheus against the same targets to test changes. You can open a target's /metrics page in a browser to see exactly what it reports. And the targets need to know nothing about the monitoring system's address. Disadvantages: the server must be able to reach the targets through the network, which can be awkward behind firewalls and NAT, and short-lived jobs may end before a scrape.

In a push model the application sends metrics to a collector. It suits short-lived jobs and devices that cannot be reached from outside. The drawback is that silence is ambiguous: a service that stops sending might be dead or might be idle.

Prometheus does offer a bridge for batch jobs, the Pushgateway, which accepts pushes and is itself scraped. The mature answer is that it is for batch jobs only, because it never forgets a series unless you delete it, and it becomes a single point of failure and a place where stale data hides. You can also mention that since Prometheus 3.x it can receive data through remote write and OTLP endpoints when you enable them, but that is an integration path, not a replacement for scraping.

The commands to have ready

Interviews for operational roles often include "show me how you would…". Keep these at your fingertips.

# Start a server and check it
prometheus --config.file=prometheus.yml
prometheus --version
curl -s localhost:9090/-/healthy       # is the process alive
curl -s localhost:9090/-/ready         # is it ready to serve queries

# Validate before reloading
promtool check config prometheus.yml
promtool check rules rules.yml

# Reload without a restart
kill -HUP <pid>
curl -X POST localhost:9090/-/reload   # needs --web.enable-lifecycle

# Query from the terminal
curl -s 'localhost:9090/api/v1/query' --data-urlencode 'query=up == 0'
promtool query instant http://localhost:9090 'up'

And the queries, which matter at least as much as the shell commands:

up == 0
sum by (job) (rate(http_requests_total[5m]))
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0

Be able to read each one aloud in plain words. For example, the third: "the per-second rate of requests with a 5xx status, summed over all series, divided by the per-second rate of all requests, which is the error ratio over the last five minutes."

Labels, cardinality and why they matter

Expect a question such as "a developer wants to add a user_id label to a request counter. What do you say?" The answer has three parts. First, every distinct combination of label values is a separate time series, and the number of series, called cardinality, drives memory use and cost. Second, user_id has unbounded, growing values, so it would multiply the series count by the number of users and could take the server down. Third, the alternative: keep the per-user detail in logs or traces, and keep metrics aggregated by bounded dimensions like route, method and status class.

If you can add one practical detail, mention how to find offenders: the TSDB status page in the interface, the /api/v1/status/tsdb endpoint, or promtool tsdb analyze on a data directory. It shows you have dealt with the problem and not only read about it.

Counters, rate and the reset problem

The follow-up to "what is a counter" is always "why do you use rate()", and the model answer is this. A counter is cumulative since the process started, so its raw value depends on uptime and is meaningless on its own. rate() computes the per-second average increase over a window, and it handles counter resets: when the value drops because the program restarted, it treats the drop as a reset to zero and does not report a huge negative number. increase() gives the total over the window, and irate() uses only the last two samples and is better for zoomed-in graphs than for alerts.

Another detail that shows understanding is the window size. The range in rate(x[5m]) needs to contain at least a couple of samples, so the usual rule is at least four times the scrape interval. Too short a window makes the graph jumpy and can return nothing if a scrape is missed.

Alerting questions

A standard question: "explain how an alert goes from a metric to a message on someone's phone." Walk the chain. A rule file contains an alert with an expr. Prometheus evaluates the expression at every evaluation_interval. When it returns any series, the alert becomes pending; if it stays true for the for duration, it becomes firing, and Prometheus sends it to every Alertmanager it is configured with. Alertmanager deduplicates identical alerts, groups related ones by labels such as alertname and cluster, waits group_wait to collect them together, applies routing rules to select a receiver, honours silences and inhibition rules, and sends the notification. It repeats it every repeat_interval while the alert remains firing, and sends a resolved message when it stops.

Why the for clause? To avoid noise from transient spikes. A condition that is true for a single evaluation is often a blip, and waiting is cheap compared with waking someone for nothing.

Why are Prometheus and Alertmanager separate programs? Because several Prometheus servers, for instance two replicas for high availability, can send to one Alertmanager cluster, which then deduplicates. Separating them also lets notification logic change without touching metric collection.

What makes a good alert? It is based on a symptom users feel (error ratio, latency, availability) more than a cause (CPU), it is actionable, it has a runbook link in an annotation, and it has a for duration tuned to the problem. Be ready to say what you would not alert on: a busy CPU by itself.

Exporters, jobs and service discovery

"How do you monitor a Linux server?" Answer: run node_exporter on it, which reads /proc and /sys and serves the Prometheus text format on port 9100, then add it as a target in scrape_configs. Mention windows_exporter for Windows and the blackbox exporter for probing endpoints from outside, and that the general pattern is one exporter per system you cannot instrument directly.

"How do you monitor something dynamic, such as pods that come and go?" Static lists do not work, so you use service discovery: Prometheus asks Kubernetes (or Consul, AWS, a file, an HTTP endpoint and about thirty other sources) which targets exist, and uses relabeling to keep the ones you want and turn discovered metadata into labels. You do not need detail at beginner level, but naming the concepts shows you know where the road goes.

Storage and retention

"What happens to data when Prometheus restarts?" It is on disk, so it survives. Recent samples are also in the write-ahead log, which is replayed at startup, and that replay can take a while on big servers. In a container without a volume, the data is lost with the container.

"How long does it keep data?" Fifteen days by default. You set it in the configuration file with storage.tsdb.retention.time and optionally a size limit in storage.tsdb.retention.size, and whichever is hit first applies. The older command-line flags for the same settings are deprecated since version 3.8, which is a detail worth mentioning because many candidates only know the flags.

"Can Prometheus store data for years?" Not by itself in a sensible way. Local storage is one copy on one disk. For long-term retention and a global view you send data with remote write to a system built for it, such as Thanos, Grafana Mimir, or a cloud managed service. That is a good place to say that you understand the limit of the tool.

What changed in Prometheus 3

If an interviewer wants to know whether your knowledge is current, they will ask about version 3. The key points are these. The user interface was rebuilt. Several feature flags became defaults or were replaced by proper flags: agent mode is --agent, and the remote write receiver is --web.enable-remote-write-receiver. Only version 2 of the Alertmanager API is supported. Range selectors became left-open and right-closed. Label values for le and quantile are normalised to floats, so le="1" is now le="1.0". A scrape whose response has an invalid or missing content type fails unless fallback_scrape_protocol is set. UTF-8 names are allowed for metrics and labels. In the later 3.x releases, retention settings moved into the configuration file, native histograms became stable, and in 3.15 the --log.level flag was deprecated in favour of runtime.log_level.

You are not expected to recite that list. Two or three of these, mentioned naturally, prove you have used a recent version.

The traps, and why they share one cause

Most wrong answers in a Prometheus interview come from the same root: treating it like something else. People who think of it as a push system say that a dead app "stops sending". People who think of it as a log store want to put user IDs in labels. People who think of it as a database for exact records want to use it for billing. People who think of it as a dashboard tool forget that Grafana draws the pictures and Prometheus supplies the numbers. Keep the correct picture in your head: a pull-based store of bounded-cardinality numeric series, with a query language and a rule engine.

The specific traps, each with its reason:

Common interview questions

What is the difference between a job and an instance? An instance is a single scrape endpoint identified by host:port. A job is a group of instances that perform the same function, for example all replicas of one service. Both become labels on every series Prometheus scrapes from it.

What does the up metric tell you? It is a synthetic metric Prometheus writes on every scrape: 1 if the scrape succeeded, 0 if it failed. It is the basis of "is this target alive?" alerts. It cannot cover a target that was never configured or has vanished from discovery; absent() handles that case.

Explain counter versus gauge with examples. A counter such as http_requests_total only increases and is read with rate(). A gauge such as node_memory_MemAvailable_bytes moves up and down and is read directly.

Why can't you average a summary's quantiles? Because the 95th percentile of the combined traffic is not the average of each instance's 95th percentile. Histograms expose bucket counts, which can be summed across instances and then converted into a quantile with histogram_quantile().

How do you calculate an error rate? Divide the rate of failing requests by the rate of all requests: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])).

What is a recording rule and why use one? A rule that evaluates an expression on a schedule and stores the result as a new series. It saves repeated heavy computation in dashboards and alerts, and gives a stable name to a useful calculation. The convention for naming is level:metric:operations.

What does for do in an alerting rule? It requires the expression to stay true for that long before the alert becomes firing. In the meantime the alert is pending. It filters out short spikes.

What does Alertmanager do that Prometheus does not? Deduplication, grouping, routing by labels, silencing, inhibition, and delivery to receivers such as Slack, email and PagerDuty. Prometheus only evaluates the rules and sends firing alerts to it.

How would you find out why a dashboard panel is empty? Work from the source: is the target UP on the Target health page? Does the metric name exist, and are the label matchers exactly right, which you can check in the table view? Is the time range correct? Is the range in rate() long enough? And if the target is DOWN, read the error text.

How do you reload the configuration safely? Validate with promtool check config, then send SIGHUP, call /-/reload if the lifecycle flag is on, or use --config.auto-reload. Then confirm with prometheus_config_last_reload_successful, since a bad file is rejected and the old configuration keeps running.

How would you monitor a machine learning model server? Instrument or scrape the request counter, a latency histogram and an in-flight gauge, watch error ratio and the 95th percentile, and add a GPU exporter such as NVIDIA's DCGM exporter for utilisation and memory. Many serving frameworks already expose a /metrics endpoint. Alert on symptoms: high error ratio, high latency, target down.

What would you not use Prometheus for? Logs, traces, and any case needing exact per-event accuracy such as billing. For the first two you pair it with a logging tool and a tracing tool, and for the third you use a transactional system.

Where does Prometheus store its data, and what is the limit? In a local TSDB on its own disk, with no built-in replication. The limit is one machine's capacity and durability, which is why high availability and long-term retention use two replicas or remote write to another backend.

What is cardinality and how would you control it? The number of unique series. Control it by keeping label values bounded, dropping unneeded labels or metrics with metric_relabel_configs, setting sample_limit on jobs as a safety net, and using the TSDB status page to find the worst metrics.

A scrape shows context deadline exceeded. What does it mean? The target did not respond within scrape_timeout. Check connectivity and firewalls first, then whether the exporter is slow or returns too much data.

How is Prometheus different from Grafana? Prometheus collects, stores and queries metrics and evaluates alerts. Grafana is a visualisation layer that uses Prometheus as one of many data sources. They are complementary; the Grafana guide covers the other half.

Questions to ask back

Good candidates ask questions too, and in an observability role, these show maturity. Ask how the team keeps alert noise down. Ask whether they run one Prometheus per cluster or a central one and how they retain data long term. Ask how they control cardinality, and whether teams can add labels freely. Ask who owns the alert rules and whether they are kept in Git with checks in CI. You are not testing them; you are showing that you already think like the person they want to hire.

Sixty-second self-test

Answer each one out loud, without looking. If you hesitate on more than two, reread the matching section of the detailed guide.

1. In two sentences, what is Prometheus, and in which direction does the data move?
2. What are the four metric types, and which query habit goes with each?
3. What identifies a time series?
4. Why is a user_id label a problem?
5. What does up mean, and what does it not cover?
6. Why rate() and not the raw counter, and why sum(rate(...)) and not rate(sum(...))?
7. What are the three alert states, and what does for change?
8. Name three things Alertmanager does that Prometheus does not.
9. How long does Prometheus keep data by default, and where do you change it in version 3.8 and later?
10. Your graph is empty. What are the first three things you check?

If you can answer all ten fluently, you are ready for a beginner-level Prometheus conversation, and for the harder scenarios that the Mid-level guide prepares you for.

This is part three of three for the beginner level: the mistakes people actually make in their first weeks with Prometheus, the habits that prevent them, and a checklist to keep. Each mistake is written the same way, with the symptom you see, the cause underneath and the fix. Skim the headings now, and come back when something misbehaves; most problems in the first month are in this list.

Common errors at this level

The graph is empty but the target is UP

Symptom. Status then Target health shows green, yet the query returns "Empty query result" or a graph with no lines.

Cause. Almost always one of three things. The metric name is not spelled the way the exporter spells it (node_memory_available_bytes instead of node_memory_MemAvailable_bytes). A label matcher is slightly off, such as job="node-exporter" when the job is called node. Or the time range on the graph starts before the data existed.

Fix. Fetch the truth from the source with curl -s localhost:9100/metrics | grep -i memavailable. Then in the query box type only the metric name and look at the Table view, which shows every label that exists. Build the selector from what you see, adding one matcher at a time.

A counter graph that is a useless ramp

Symptom. You graph http_requests_total and get a line that climbs forever, with sudden drops to zero.

Cause. A counter is cumulative since the process started, and it resets on restart. The raw value says nothing about how busy the service is right now.

Fix. Wrap counters in rate() for a per-second figure or increase() for a total over a window: rate(http_requests_total[5m]).

rate() returns nothing, or the graph is full of gaps

Symptom. rate(x[1m]) is empty or spiky although the series exists.

Cause. The range window holds fewer than two samples. With a 15-second scrape interval, one minute holds four samples, and a single slow scrape can leave too few. If you rely on the built-in default of a one-minute scrape_interval, a one-minute window holds about one sample and can never produce a rate.

Fix. Use a window of at least four times the scrape interval. Five minutes is the usual choice. If you need faster reaction, scrape more often instead of shortening the window.

Adding a label that grows without limit

Symptom. Memory on the server climbs steadily, queries slow down, and eventually the process is killed for using too much memory. The TSDB status page shows one metric with an enormous number of series.

Cause. A label whose values come from users or requests, such as user_id, email, request_id or a full URL with query strings. Each new value creates a new time series, and the server keeps the recent ones in memory.

Fix. Remove the label in the application. If you cannot change the code immediately, drop it at scrape time with metric_relabel_configs and the labeldrop action, or drop the whole metric. Put per-user detail in logs, and keep metric labels to small, fixed sets such as method, route template and status class.

Editing the config and nothing changes

Symptom. You changed prometheus.yml, but the behaviour is the same.

Cause. You did not reload the server, or you did and the new file was rejected. A rejected file is logged as an error and the old configuration keeps running, so nothing looks broken.

Fix. Always run promtool check config prometheus.yml first. Then reload with kill -HUP <pid>, or curl -X POST localhost:9090/-/reload if you started with --web.enable-lifecycle, or start with --config.auto-reload. Afterwards, query prometheus_config_last_reload_successful: it must be 1. The Configuration page in the Status menu shows what is really loaded.

The reload endpoint refuses the request

Symptom. curl -X POST localhost:9090/-/reload fails with an error saying the lifecycle API is not enabled.

Cause. The reload and quit endpoints are off by default, because they would let anyone who reaches the port shut the server down.

Fix. Start with --web.enable-lifecycle on trusted machines only, or use kill -HUP or --config.auto-reload instead, neither of which needs the flag.

localhost does not mean what you think inside Docker

Symptom. Prometheus runs in a container, and every target at localhost:9100 is DOWN with connection refused, although the exporter works fine when you curl it from your laptop.

Cause. Inside a container, localhost refers to the container itself, not your machine. The exporter is on the host or in another container.

Fix. Put the containers on the same user-defined Docker network and use the container name as the host (node-exporter:9100), which is what Docker Compose does automatically. For something running on the host, use host.docker.internal:9100 on Docker Desktop, or add --add-host=host.docker.internal:host-gateway on Linux. The Docker guide covers networks in detail.

The data disappears when the container is recreated

Symptom. Every time you rebuild your Prometheus container, the history is gone.

Cause. The database lives in /prometheus inside the container. Without a volume, that directory disappears with the container.

Fix. Mount a named volume with -v prometheus-data:/prometheus. Remember also that if you pass extra arguments to docker run, they replace the image's defaults, so repeat --config.file=/etc/prometheus/prometheus.yml and --storage.tsdb.path=/prometheus along with yours.

Copying an old tutorial and meeting unknown flags

Symptom. The server logs Unknown option for --enable-feature, or a query fails with unknown function with name "holt_winters", or an alert never reaches Alertmanager.

Cause. The tutorial was written for Prometheus 2. Version 3.0 removed the feature flags agent and remote-write-receiver (they are now --agent and --web.enable-remote-write-receiver), renamed holt_winters to double_exponential_smoothing, and dropped support for version 1 of the Alertmanager API.

Fix. Check the date of what you are reading. Run prometheus --help on your own installation and trust it over a blog. Use api_version: v2, which is the only one supported, and the default.

Dashboards built on le="1" stop matching after an upgrade

Symptom. After moving to 3.x, a histogram query that filters on a bucket such as le="1" returns nothing.

Cause. Since 3.0, the le and quantile label values are written in float form, so le="1" is stored as le="1.0".

Fix. Update the queries to le="1.0", or better, stop filtering on exact bucket labels and use histogram_quantile(), which does not care.

YAML indentation errors that point at the wrong line

Symptom. promtool reports a parse error on a line that looks fine.

Cause. YAML's structure is its indentation. One mis-indented line makes the next line look wrong, and tab characters are not allowed at all.

Fix. Use two spaces per level and a text editor set to show whitespace. Validate after every small edit instead of after twenty. If you paste from a web page, retype the first indent.

Two jobs or two targets colliding

Symptom. The server logs found multiple scrape configs with job name, or you see warnings about duplicate samples and out-of-order samples, and some data is dropped.

Cause. Two jobs with the same name, or two targets that end up with identical labels, for example two configurations scraping the same endpoint with the same instance value. Prometheus cannot store two different values for one series at one timestamp.

Fix. Give every job a unique job_name, and make sure each target has a distinct instance. If you added a target in two places by accident, remove one.

An alert that never fires, or never stops

Symptom. You expect an alert, and the Alerts page shows it as inactive. Or an alert flaps between firing and resolved every few minutes.

Cause. For a silent alert, the expression returns nothing because of a label typo, or the for duration is longer than the condition lasts. For a flapping alert, the condition hovers around the threshold.

Fix. Paste the expr into the query box and look at its result on a graph; an alert is just a query, so debug it as one. For flapping, lengthen the for, add a margin between the firing level and the healthy level, or use keep_firing_for, which keeps an alert firing for a while after the condition clears.

Alertmanager never receives the alert

Symptom. The alert is firing on the Prometheus Alerts page, but the Alertmanager page at port 9093 is empty and no notification arrives. The log shows Error sending alerts.

Cause. Prometheus cannot reach Alertmanager, because of a wrong address in alerting.alertmanagers, a port or network problem, or an old api_version: v1.

Fix. Run curl localhost:9090/api/v1/alertmanagers, which lists the Alertmanagers Prometheus found. Confirm the address and port and that Alertmanager is running.

A target that is DOWN with a content-type error

Symptom. The target shows received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified.

Cause. You pointed the job at something that is not a metrics endpoint, such as the main page of an application, or the application serves metrics on another path. Version 3 is strict about what a scrape returns.

Fix. Open the URL in a browser: metrics look like lines of name value. Set metrics_path to the right one. Only as a last resort, for an old exporter that sends a wrong header, set fallback_scrape_protocol: PrometheusText0.0.4 on the job.

The habits that pay off most

Validate, then reload, then verify

Every change follows the same three steps: promtool check config (and check rules), then reload, then confirm that the reload succeeded and the targets are still UP. It takes twenty seconds and removes an entire class of silent failure. Put it in a small script or a Makefile target so you never skip it.

Start from up

When anything is wrong, the first query is up, and the second is up == 0. It tells you whether the question is "why is the data strange?" or "why is there no data?", and these have very different answers. Many engineers save up == 0 as their first alert ever, and it remains the most valuable one.

Read the target's own page

Before you write a query for a metric, look at the raw exposition with curl. You see the exact name, the labels, the # HELP text and the # TYPE. It settles arguments about what a number means, and it is the quickest way to spot that an exporter exposes something different from what a tutorial claims.

Name things well from the start

Use base units in names, _seconds and _bytes, end counters with _total, and follow level:metric:operations for recording rules. It costs nothing on day one and saves confusion for years, because a name is the only documentation many metrics ever get.

Use labels sparingly and think about their values

Before you add a label, ask two questions: how many distinct values can it have, and will I ever filter or group by it? If the first answer is "unbounded" or the second is "no", leave it out. A metric with three well-chosen labels is worth more than one with ten.

Alert on symptoms, and write the next step into the alert

Prefer alerts about what users feel: error ratio, latency, availability. Add a summary that says what is wrong and a description or a runbook link that says what to do first. If an alert cannot tell the person on call what to do, it is not finished.

Keep configuration in Git

prometheus.yml, rule files and alertmanager.yml are code. Commit them, review changes, and run promtool check in CI. When the dashboard goes wrong at 2 a.m., git log answers "what changed" faster than anyone's memory.

Prefer the configuration file to flags

Since Prometheus 3.8, retention settings belong in the config file, and since 3.15 so does the log level (runtime.log_level). The config file can be reloaded without a restart and is reviewable in Git; command-line flags need a restart and hide in service definitions. Use flags for the things that only flags can do, such as the listen address and the data path.

Pin the version

Write the exact version, prom/prometheus:v3.15.0 and not latest, in every Docker command and Compose file. Prometheus ships a new minor release about every six weeks, and a silent change in behaviour on a restart is not the surprise you want. If stability matters more than new features, pick the 3.13 long-term-support line, which is supported until July 2027.

Watch Prometheus with Prometheus

Add a few alerts on the server itself as soon as you have a real workload: the configuration reload failed (prometheus_config_last_reload_successful == 0), rule evaluations are failing, notifications to Alertmanager are failing, and the number of series suddenly jumps. A monitoring system that fails silently is worse than none, because people trust it.

Practice cards

Each card takes ten to fifteen minutes. Do them on your laptop, and check the answer only after you have tried.

A debugging routine for a missing or wrong number

When a number looks wrong, walk the same path data takes, from the source to the screen. Stopping at the first failing step saves hours.

1. Is the target alive?Status then Target health. Read the state and the last error. DOWN means stop here and fix the scrape.
2. Does the source expose it?curl the target's /metrics and search for the name. If it is not there, the problem is in the application or exporter, not Prometheus.
3. Did it arrive?Query the bare metric name in the Table view. If nothing comes back, check metric_relabel_configs that might drop it, and limits such as sample_limit.
4. Do the labels match?Read the labels in the Table, and compare them with your matchers character by character, including quotes and regular expression anchoring.
5. Is the query shaped correctly?Counters need rate(), histograms need histogram_quantile() with le kept, and sum goes outside rate.
6. Is the window sensible?At least four scrape intervals inside rate(), and a graph range that covers when the data exists.

Staleness and why a graph line ends

A behaviour that puzzles beginners: when a target disappears, its line on the graph stops cleanly instead of being drawn flat for five minutes. This is staleness handling. When a series vanishes from a scrape, or a target goes away, Prometheus writes a special stale marker, and queries stop returning that series immediately. Without a marker, a query at any moment looks back up to five minutes (the lookback delta) for the most recent sample.

Two practical consequences follow. First, do not be alarmed by lines that end: they show that the series stopped being reported, which is correct. Second, a query for the latest value may keep returning a sample for up to five minutes after a series stopped if no stale marker was written, for example when the whole Prometheus server was down. So when you test an alert by stopping something, allow for the time it takes for signals to settle before deciding the alert is broken.

Platform gotchas

macOS. The first time you run a downloaded binary, Gatekeeper may block it. Remove the quarantine flag with xattr -d com.apple.quarantine ./prometheus ./promtool. With Homebrew's service, flags live in $(brew --prefix)/etc/prometheus.args, and logs in $(brew --prefix)/var/log/prometheus.err.log, which is the first place to look when brew services start prometheus seems to do nothing. The date -d option used in many query examples is a GNU feature; macOS uses date -v-1H.

Windows. Run prometheus.exe from PowerShell and allow it through the firewall on private networks when prompted. Host metrics come from windows_exporter on port 9182, not node_exporter. For a permanent service, run it in Docker or WSL2 or with a service wrapper, since the project does not provide one. Remember that Windows paths in YAML need forward slashes or quotes.

Linux services. Run Prometheus as a dedicated user, not root, with a systemd unit, and make sure that user owns the data directory. A permission error on startup (opening storage failed ... permission denied) is nearly always the data directory's owner.

Cloud and region. If you run Prometheus on a virtual machine in a cloud region in the Gulf or Egypt, remember that metrics can contain hostnames, paths and sometimes identifiers. If your employer has data residency rules, keep the server and its storage in the approved region, and think twice before sending metrics to a managed service outside it.

Clocks. Prometheus timestamps samples when it scrapes, but alerts and graphs compare times across machines. A badly skewed clock on a server running an exporter that sets its own timestamps causes out-of-order and duplicate sample warnings. Run time synchronisation (chrony or your cloud's time service) everywhere.

Keeping your laptop setup clean

It is easy to leave a mess of half-started processes while learning. Keep a single folder per experiment with the configuration and rules inside, and start the processes from a small script, so that a clean restart is one command. List what is listening with lsof -i :9090 -i :9093 -i :9100 on macOS and Linux, and stop stragglers before starting new copies, because a second Prometheus on the same port or the same data directory will not start. When you finish, delete the data directory (data/ by default) to get a clean slate, or remove the Docker volume with docker volume rm prometheus-data. Do not do that with anything you care about.

Small things to do from day one

• Set scrape_interval and evaluation_interval explicitly in global, instead of inheriting the one-minute default by accident.
• Give every job a clear, unique job_name, and add an environment or team label on targets where it will help later.
• Run promtool check config and promtool check rules before every reload.
• Keep up == 0 as your first alert, with a for of a couple of minutes.
• Learn the Status pages before you learn the log format.
• Use curl against a target's /metrics whenever a metric surprises you.
• Keep label values bounded, and review new labels the way you review new database columns.
• Write a one-line summary and a next-step description on every alert.
• Put config and rules in Git, and pin image and binary versions.
• Read the release notes (the changelog) when you upgrade: a [CHANGE] line is a warning.

A starter setup to copy

Here is a complete, small set of files to copy as the base of any learning project. It monitors itself, a host, and one application, loads one rule file, and sends alerts to a local Alertmanager.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    environment: "learning"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

rule_files:
  - "rules.yml"

storage:
  tsdb:
    retention:
      time: 15d
      size: 10GB

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

  - job_name: "app"
    scrape_timeout: 5s
    static_configs:
      - targets: ["localhost:8000"]

groups:
  - name: basics
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} on {{ $labels.instance }} is down"
          description: "Check the process and the network path. Last scrape failed for 2 minutes."

      - alert: PrometheusReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 1m
        labels:
          severity: ticket
        annotations:
          summary: "Prometheus rejected its last configuration reload"

And a tiny script that enforces the habit of validating first:

#!/usr/bin/env bash
set -euo pipefail

promtool check config prometheus.yml
promtool check rules rules.yml
echo "Config and rules are valid; reloading."
curl -fsS -X POST localhost:9090/-/reload
sleep 2
curl -fsS 'localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_config_last_reload_successful'

Run the script after each change. The last command prints a JSON answer whose value is "1" when the reload worked. Note that it assumes the server was started with --web.enable-lifecycle; if it was not, replace the reload line with kill -HUP "$(pgrep -x prometheus)".

The checklist

Keep this list beside you for the first month.

Before you call a setup finished, confirm these, one by one: every target is UP; promtool check passes on all files; the reload metric reads 1; the data directory is on a volume or a real disk; there is at least one alert for a dead target and it has been tested by actually stopping something; the alert reaches Alertmanager and a receiver; retention and disk size have been estimated; and the files are in version control. When all of these are true, you have a monitoring setup that can be handed to a colleague, and you are ready for the Mid-level guide.

This is part two of three. It picks up where Beginner ended and takes every topic from there further: how a scrape really works, what the storage engine does with your samples, how to write queries and rules that stay correct at 3 a.m., and how to keep the server healthy when the cluster around it grows. Everything here targets Prometheus 3.15, and it flags the places where older tutorials will mislead you, because the 3.0 release changed enough defaults that a large share of the blog posts on the internet are now wrong.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a real server in a few minutes. The examples assume a Prometheus 3.15 instance on localhost:9090 with --web.enable-lifecycle turned on and a node_exporter on localhost:9100. If you only have the server, the queries against prometheus_* metrics work on it alone, because Prometheus scrapes itself.

If you work in a Gulf or Egyptian organisation, one thing from this level matters early: Prometheus stores whatever you put into labels, forever, in plain text. Labels that carry customer identifiers turn a monitoring database into a personal-data store, with all the residency and retention obligations that implies. The cardinality section returns to this, because the cure for a cost problem and the cure for a compliance problem turn out to be the same habit.

The scrape loop, precisely

Beginner showed you that Prometheus pulls. The problem at this level is that "it pulls" does not explain why a target flaps, why a metric vanishes for exactly five minutes, or why two servers disagree about the same target. To predict those, you need the loop itself.

For each target in each job, Prometheus runs an independent goroutine. Every scrape_interval it issues an HTTP GET to scheme://address/metrics_path with the params you configured, waits at most scrape_timeout for the complete body, parses it, applies relabeling and limits, and appends the samples to the head block of the TSDB. The defaults are an interval of 1m and a timeout of 10s, not the 15s you see in the example file; the example overrides them. The timeout must not exceed the interval, and Prometheus refuses to load a config where it does:

scrape timeout greater than scrape interval for scrape config with job name "api"

Scrape start times are spread across the interval by a hash of the target's labels, so a thousand targets do not all arrive in the same millisecond. That spreading is deterministic: the same target keeps the same offset across restarts, which is why a graph of a fast-moving gauge looks identical from run to run.

What gets stored, and with which timestamp

Every sample in a scrape is stored with one timestamp: the moment the scrape started, not whatever the target claims. The exposition format allows a target to attach its own timestamp to a sample, and with the default honor_timestamps: true Prometheus will use it. That sounds like fidelity and is usually a trap. Clock skew between the target and the server produces samples that are older than ones already stored, and the server drops them as out of order. Unless you run something that genuinely has to back-date (a cAdvisor-style exporter is the standard example), treat a target-supplied timestamp as a fault to investigate.

Alongside the samples a target exposes, Prometheus writes synthetic series per scrape. They are the cheapest, most honest health signal you have:

With extra_scrape_metrics: true you also get scrape_timeout_seconds, scrape_sample_limit and scrape_body_size_bytes, which let you alert when a target is approaching its own limits before it falls over. These synthetic series are not affected by metric_relabel_configs, so you cannot accidentally relabel up away.

Staleness: why a series disappears at once

When a target vanishes or stops exposing a series, Prometheus does not wait for the five-minute lookback to run out. On the next scrape where a previously seen series is absent, it writes a stale marker for that series, and queries stop returning it from that moment. The marker is a special NaN value that only the query engine understands. Without staleness, a pod that was deleted would keep appearing in sum() results for five minutes after it died.

The lookback is the other half. An instant query at time t returns, for each series, the most recent sample in the window (t - lookback, t], where the lookback delta is 5m by default (--query.lookback-delta). So if a series simply stops arriving without a stale marker, for instance because the target is alive but your scrape interval is ten minutes, the query returns nothing between samples. The practical rule: keep scrape_interval comfortably under the lookback, and never set it above about two minutes for anything you graph.

Protecting the server from a bad target

A single exporter that suddenly emits a million series can take down a server that served ten thousand targets happily. Prometheus gives you per-job and global fences, and a mid-level engineer should set them on purpose rather than discover them in an incident:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  sample_limit: 0            # 0 = unlimited; override per job below
  external_labels:
    cluster: prod-eu1
    replica: a

scrape_configs:
  - job_name: "payments-api"
    sample_limit: 50000      # the whole scrape FAILS if a target exceeds this
    label_limit: 30
    label_value_length_limit: 200
    body_size_limit: 10MB
    target_limit: 500
    static_configs:
      - targets: ["payments-1:8080", "payments-2:8080"]

The behaviour of sample_limit surprises people: it is all or nothing. If the target returns one sample over the limit, the entire scrape is rejected, up goes to 0, and the log says sample limit exceeded. That is deliberate. A partial ingest would hand you a silently truncated dataset, and a loud failure on one job is cheaper than an out-of-memory kill of the whole server. Pair the limit with an alert on prometheus_target_scrapes_exceeded_sample_limit_total, so the owning team hears about it before the next deploy makes it worse.

honor_labels, and who owns a label

A target can expose a metric that already carries a job or instance label. By default (honor_labels: false) Prometheus keeps its own labels and renames the conflicting scraped ones to exported_job and exported_instance, so the server always wins. Setting honor_labels: true reverses that: the scraped value wins, and Prometheus will not attach its own job or instance when the target supplied one.

You want honor_labels: true in exactly two cases: federation (you are copying series from another Prometheus, and their labels are the truth) and the Pushgateway (the pushed job name is the identity, not the gateway's address). Everywhere else, leave it false. Turning it on for a normal application lets that application impersonate another job.

Authentication and redirects

A scrape config takes exactly one of basic_auth, authorization or oauth2, plus an optional tls_config. Prefer the file variants, password_file, credentials_file and client_secret_file, because a secret pasted into prometheus.yml ends up in version control. One change worth knowing from the 3.13 LTS onward: when a target answers with a redirect to a different host, the HTTP client no longer forwards the Authorization header, basic auth, tokens or configured headers. A scrape that used to follow a redirect through a login gateway and succeed can start returning 401 after an upgrade. The fix is to scrape the final address, not to loosen the client.

Inside the TSDB

The problem this section solves: Prometheus will run for months on a disk that looks fine, then one day it restarts and takes eleven minutes to come back, or fills a volume despite a retention setting of fifteen days. Both behaviours make sense once you know how the storage engine is laid out.

Incoming samples go first to the head block, an in-memory structure holding roughly the last two hours of data. Every append is also written to the write-ahead log (WAL) in the wal/ directory, in 128 MB segments, so that a crash does not lose the head. Samples older than the head's window are cut into an immutable on-disk block every two hours. A block is a directory named by a ULID, containing chunks/ (the compressed samples), an index (label-to-series mapping), a meta.json (time range and compaction history) and tombstones (marks for deleted series). Head chunks that are full are memory-mapped from chunks_head/ rather than kept on the Go heap, and if you enable out-of-order ingestion a second log, wbl/, holds those samples.

data/
├── lock
├── wal/                     # 128MB segments; replayed on start
│   ├── 00000121
│   └── checkpoint.00000120/
├── chunks_head/             # memory-mapped head chunks
├── wbl/                     # only with out-of-order ingestion
├── 01J8ZK3...A9/            # a 2h block
│   ├── chunks/000001
│   ├── index
│   ├── meta.json
│   └── tombstones
└── 01J8ZQ7...B2/            # a compacted block, spans longer

Compaction merges adjacent blocks into larger ones so that queries touch fewer files, up to a block spanning 10% of your retention or 31 days, whichever is smaller. During a compaction the source blocks and the new block exist side by side, so the disk needs headroom beyond your steady-state figure. If you size a volume to 100% of expected data, the first large compaction will fail it.

What this predicts

Restart time is WAL replay. On start, Prometheus replays the WAL into the head to rebuild every active series. Replay time scales with the size of the head, so a server with several million active series can spend minutes in /-/ready returning 503. You can watch progress at /api/v1/status/walreplay. A graceful SIGTERM shutdown lets the server checkpoint, but a SIGKILL or an out-of-memory kill leaves a long WAL to replay. If restart time matters to you, the memory-snapshot-on-shutdown and fast-startup feature flags shorten it, at the cost of relying on experimental behaviour.

Memory is about active series. The head keeps an index entry and a write buffer for each series that received a sample in the current window. The figure that predicts your memory is prometheus_tsdb_head_series, and the figure that predicts your future memory is how fast that number is growing. As a practitioner rule of thumb (community experience rather than an official limit), budget a few kilobytes per active series, and expect a well-sized single server to handle a few million. When that number doubles after a deploy, you have found a cardinality problem, which the next sections cover.

Disk is arithmetic. The documentation gives the formula directly:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

with one to two bytes per sample on disk after compression. You measure the middle term with rate(prometheus_tsdb_head_samples_appended_total[5m]). A server ingesting 200,000 samples per second for 15 days needs about 15 * 86400 * 200000 * 2 = 518 GB at two bytes per sample. Plan for that plus a third for compaction headroom.

Retention lives in the config file now

For years you set retention with --storage.tsdb.retention.time=30d. Since 3.8 those flags, and --storage.tsdb.retention.size, are deprecated. They still work, but the supported place is the storage section of prometheus.yml, where the values are reloadable, so changing retention no longer means a restart. When both a flag and the config file are set, the config file wins.

storage:
  tsdb:
    retention:
      time: 30d          # default is 15d if neither time nor size is set
      size: 500GB        # powers of 2; 0 disables the size limit
    out_of_order_time_window: 0s
    chunk_encoding:
      floats: xor        # xor (default) | xor2

If both time and size are configured, whichever trips first triggers deletion. Two details catch people out. Retention deletes whole blocks, so the oldest data can be retained for up to a couple of hours longer than you asked. And size retention counts the WAL and head chunks toward the total, but can only delete blocks, so a WAL that has grown large can make the limit unsatisfiable in the short run.

Release 3.15 also stabilised XOR2 float chunk encoding, selected with storage.tsdb.chunk_encoding.floats: xor2. The old --enable-feature=xor2-encoding flag is deprecated. The warning deserves more attention than the feature: XOR2 chunks cannot be read by older Prometheus versions or by tools that do not know the encoding, which includes blocks uploaded by a Thanos sidecar to object storage. Downgrading after writing XOR2 blocks means deleting them. Turn it on only after confirming every downstream reader supports it.

Out-of-order samples

Normally a series only accepts a sample newer than its latest. A sample at or before the newest timestamp is rejected, with log lines such as Error on ingesting out-of-order samples. Causes include two servers writing the same series, a pusher with a skewed clock, or a pipeline that batches and retries (remote write receivers and OTLP senders do this routinely). If you operate a receiver that legitimately gets late data, set a window:

storage:
  tsdb:
    out_of_order_time_window: 30m

Samples up to 30 minutes behind the newest one are then accepted. This costs memory and WAL space, so leave it at its default 0s on a server that only scrapes, where an out-of-order sample means something is wrong and you want to hear about it.

Looking at your own TSDB

curl -s localhost:9090/api/v1/status/tsdb | jq '.data.headStats'
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'
promtool tsdb list /var/lib/prometheus
promtool tsdb analyze /var/lib/prometheus

The status/tsdb endpoint returns cardinality statistics for the head, capped at 10,000 entries per list. promtool tsdb analyze goes through a whole data directory and prints the highest-cardinality metrics and labels plus churn, which is the tool you reach for when a server is bigger than it should be. Run it against a copy or a snapshot on a busy production host, because it reads a lot.

Service discovery and relabeling as a pipeline

Static target lists stop working the day your targets are pods, autoscaling instances or services that move. Service discovery (SD) asks an external system, Kubernetes, Consul, a cloud API, DNS, a file, for the current target list, and relabeling turns the raw metadata it provides into the labels and scrape settings you want. The two together are the most powerful and least understood part of Prometheus configuration, and nearly every "why is my target missing" question ends here.

The shape of the pipeline

Each discovered target arrives with a set of labels whose names start with __meta_ (and a few special ones starting __), for example __meta_kubernetes_pod_name. Your relabel_configs run in order against that label set. Their output becomes the target's real labels. The pipeline has four distinct stages, and confusing them is the commonest relabeling mistake:

Labels whose names begin with __ are removed from the target after relabel_configs finish. If you need a scratch label to carry a value between two rules, name it with the __tmp_ prefix, which Prometheus guarantees not to use itself.

Actions, and the anchoring rule

The action field decides what a rule does. The default, replace, writes a computed value into target_label. The rest:

Two rules of the regex engine explain a great many silent failures. The engine is RE2, and every regex is fully anchored: regex: "web" matches only the value web, not web-1. Write web.* if you meant a prefix. And when there are several source_labels, their values are joined with the separator (default ;) before matching, so source_labels: [a, b] is matched against the string avalue;bvalue.

A Kubernetes pod job, annotated

This is the pattern behind most hand-written Kubernetes scrape configs. It scrapes only pods that opt in through an annotation, honours an optional custom path and port, and turns useful pod metadata into labels:

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # 1. only pods annotated prometheus.io/scrape: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # 2. optional custom metrics path
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # 3. rewrite the address to use the annotated port
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      # 4. copy pod labels across
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      # 5. stable, readable labels
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Read rule 3 slowly, because it shows everything at once. Two source labels are joined with ; into something like 10.1.4.7:8080;9102. The regex has two capture groups, the host without any port, and the port from the annotation. The replacement rebuilds host:port and writes it into __address__, which is what Prometheus actually connects to. Note that since 3.0 a target address is not given a default port for you, so a target with no port in __address__ is scraped on the scheme's default rather than 9090 or anything you might expect.

Kubernetes SD needs RBAC: the server's service account must be able to get, list and watch nodes, services, endpoints, endpoint slices, pods and ingresses. A missing permission shows up as an SD error in the logs and an empty target list, not as a scrape error.

Other discovery mechanisms

Kubernetes is one of roughly thirty-five SD mechanisms. The ones you meet in practice after it:

• file_sd_configs reads JSON or YAML target files (targets plus labels) and re-reads them on change or every refresh_interval (5m default). This is the universal escape hatch: any tool that can write a file can feed Prometheus, which makes it the natural integration point for Terraform or Ansible outputs.
• http_sd_configs polls a URL (60s default) that returns the same JSON shape, for when a service registry already exposes an API.
• consul_sd_configs, dns_sd_configs, ec2_sd_configs and the unified aws_sd_configs, plus Azure, GCE and, since 3.14, oci_sd_configs for Oracle Cloud Infrastructure. Managed cloud regions in the Middle East are covered by the same mechanisms; you only point them at the regional API endpoint.

And one new capability from 3.15: a target can be scraped over a Unix domain socket by setting the __unix_socket__ target label, with __scheme__ set to http or https. It lets you expose metrics from a sidecar or local daemon without opening a TCP port.

Debugging a target that is not there

A target missing from the Targets page was either never discovered, or discovered and then dropped by relabeling. Work from the outside in.

1. Status, then Service discovery in the UI shows every discovered target with its raw __meta_* labels and, next to it, the labels after relabeling. A target shown as dropped has a relabel rule that rejected it, and the page shows the pre-relabel labels so you can see what your regex was matched against.
2. From 3.15 the Targets and Service Discovery pages also show the effective configuration of each scrape pool, which settles "is my reload live" at a glance.
3. Reproduce from the command line without touching the running server: promtool check service-discovery prometheus.yml kubernetes-pods expands the SD for one job and prints the result of relabeling.
4. curl -s 'localhost:9090/api/v1/targets?state=dropped' returns dropped targets with their discovered labels, handy in scripts.

PromQL that stays correct

Beginner taught you selectors, rate and aggregation. The problem at this level is queries that return plausible numbers that are wrong: a rate that hides a spike, a join that silently drops half the series, a percentile computed from averages. The fixes come from understanding three things: how range functions behave at the edges, how binary operators match series, and how histograms work.

How rate() really works

rate(x[5m]) takes the samples of each series in the five-minute window, accounts for counter resets, and returns the per-second increase, extrapolated to cover the full window edges. Three consequences follow.

First, the window must contain at least two samples per series, or rate returns nothing for it. At a 30-second scrape interval a [1m] range holds two samples at best, and a single failed scrape leaves one. The guidance is to make the range at least four times the scrape interval, which gives rate room to survive a missed scrape.

Second, rate is a smoothed average, and irate uses only the last two samples. irate makes jagged, responsive graphs and is excellent for looking at a fast-moving counter by eye. It is a poor basis for an alert, because a one-scrape blip can make the alert flap, and a spike that falls between two evaluations is never seen. Use rate in alerts and rules.

Third, rates are per second, so a request counter prints as 0.83 and not 50. When you want a count over a window, use increase(x[1h]), which is rate times the window length, and expect non-integer results because of the extrapolation.

# per-second request rate by route, smoothed over 5 minutes
sum by (route) (rate(http_requests_total[5m]))

# how many 5xx responses in the last hour
sum(increase(http_requests_total{status=~"5.."}[1h]))

# error ratio: take the ratio of rates, never the rate of a ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

Since 3.0, range selectors are left-open and right-closed: a sample exactly on the left boundary of the window is excluded. That is the behaviour that makes a query like foo[1m:1m] return a single point where older versions returned two, so if a subquery that worked on 2.x now returns thin data, widen the range, for instance foo[2m:1m].

Counter resets and what they hide

A counter that restarts at zero is handled automatically: rate treats any decrease as a reset and adds the pre-reset value. That protects you from restarts but not from resets that happen between scrapes twice, or from a counter that legitimately goes backwards because an exporter aggregates across children that come and go. Two defensive habits: use resets(x[1h]) to see how often a counter restarts (a high number on a service that never restarts is a clue to a sidecar or an exporter bug), and never take the rate of a sum. Sum the rates, because summing first mixes the series and converts per-instance resets into spurious drops.

# wrong: a single instance restarting makes the sum fall, which looks like a reset
rate(sum(http_requests_total)[5m:])

# right
sum(rate(http_requests_total[5m]))

Vector matching

When you use a binary operator between two instant vectors, Prometheus pairs elements with identical label sets (ignoring the metric name). If the label sets differ, nothing pairs up and the result is empty, which is the most common reason a ratio query returns "no data".

on(...) restricts matching to listed labels; ignoring(...) matches on everything except them. Matching is one-to-one by default, and if either side has several series with the same matching labels you get this error:

found duplicate series for the match group {job="api"} on the right hand-side of the operation:
[...];many-to-many matching not allowed: matching labels must be unique on one side

The fix depends on whether the many-to-one is intentional. When it is, group_left (the left side has many) or group_right tells Prometheus which side fans out. The classic use is attaching metadata from an info-style metric, whose value is always 1, to a measurement:

# add the 'version' label from build_info to a request rate, per instance
sum by (instance) (rate(http_requests_total[5m]))
  * on (instance) group_left (version)
build_info

Read it aloud: multiply each rate by the matching build_info value, which is 1 and so changes nothing, and copy the version label across from the right-hand side. The multiplication by one is the trick; the join is the point.

The set operators work on whole label sets rather than values: a and b keeps elements of a that have a matching element in b, a or b is the union (elements of a plus elements of b that have no match), and a unless b keeps elements of a with no match in b. unless is how you write "alert on this unless that is also true", such as high latency unless the instance is under a planned drain:

job:latency_seconds:p99 > 0.5
  unless on (instance)
maintenance_mode == 1

Comparisons filter by default: up == 0 returns only the series where it is true. Add the bool modifier to get a 0 or 1 for every series instead, which is what you want when the result feeds arithmetic or a graph of "what fraction of targets are down".

avg(up{job="api"} == bool 0)

Histograms and percentiles

An average latency hides the slow tail, and a percentile is what users feel. A classic histogram is exposed as cumulative bucket counters _bucket{le="0.1"}, _bucket{le="0.5"} and so on, plus _sum and _count. You compute a percentile from the bucket rates:

histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Every part of that shape matters. You take rate of the buckets first, because the raw counters are cumulative since process start. You sum by (le) so that buckets from all instances combine, which is legal because buckets are counters and counters add. The le label must survive the aggregation or histogram_quantile has nothing to work on. The result is an estimate by linear interpolation inside a bucket, so its accuracy is bounded by your bucket layout: with buckets at 0.5 and 1, a true p99 of 0.51 and one of 0.99 both come out near the same interpolated value. Choose bucket boundaries around your SLO threshold, so the answer to "are we under 300 ms" is exact even if the shape above it is fuzzy.

A summary exposes quantiles pre-computed in the client. They are precise for one process and cannot be aggregated: the average of two instances' p99 is not the p99 of the combined traffic. Use histograms for anything you aggregate across replicas, which is nearly everything.

Native histograms (stable since 3.9) replace the forest of _bucket series with a single series using sparse exponential buckets, which is cheaper to store and more precise. Two things to know. Ingestion is off by default and is enabled by configuration rather than by the old feature flag, which is now a no-op:

scrape_configs:
  - job_name: "api"
    scrape_native_histograms: true
    static_configs:
      - targets: ["api-1:8080"]

Setting it true makes Prometheus negotiate the protobuf exposition format, since native histograms travel only in protobuf. And queries change shape, with no _bucket or le:

histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m])))
histogram_fraction(0, 0.3, sum(rate(http_request_duration_seconds[5m])))
histogram_avg(rate(http_request_duration_seconds[5m]))

histogram_fraction(0, 0.3, ...) answers "what share of requests finished in under 300 ms" directly, which is the SLI you usually want.

Subqueries, predict_linear and duration arithmetic

A subquery evaluates an instant-vector expression at many steps and produces a range vector you can feed to an _over_time function. The syntax is expr[range:resolution]:

# the highest 5-minute request rate seen in the last 6 hours, sampled each minute
max_over_time(sum(rate(http_requests_total[5m]))[6h:1m])

Subqueries are convenient and expensive, because the inner expression runs once per step. Anything you will run repeatedly belongs in a recording rule, which computes the inner expression once per evaluation and stores the result so the outer query can take max_over_time of a plain series.

predict_linear fits a line to a range and extrapolates. It is how you alert on a disk that will fill before anyone notices:

predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 4 * 3600) < 0

Reading it: using the last six hours of free space, where will the free space be four hours from now, and is that below zero? Because it is a straight-line fit, it handles steady growth well and sawtooth patterns badly (a log rotation every night will make it cry wolf), so pair it with an absolute threshold such as and node_filesystem_avail_bytes < 10e9.

From 3.14, duration expressions are enabled by default, so arithmetic is allowed inside range and offset values: rate(x[5m * 2]) or offset (1h + 5m). The feature flag is now a no-op.

Finding what is missing

absent(up{job="api"}) returns a single series with value 1 when no up series matches the selector at all, and nothing otherwise. It is the tool for "this job has disappeared from service discovery", which up == 0 can never catch because a target that does not exist has no up series to be zero. absent_over_time(x[10m]) is the windowed variant, useful for metrics that appear only intermittently.

Recording rules and alerting rules that work

The problem: a dashboard that runs a heavy aggregation on every refresh for every viewer, and an alert that fires on noise until everyone mutes it. Rules address both. A recording rule evaluates an expression on a schedule and stores the result as a new series, so readers query a cheap precomputed value. An alerting rule evaluates a condition and, when it holds long enough, turns it into an alert that goes to Alertmanager.

Both live in rule groups inside files listed under rule_files. Rules in a group run sequentially at the group's interval, in the order written, which means a recording rule can safely feed an alert defined below it in the same group, and cannot safely feed one in a different group (the order between groups is not defined).

groups:
  - name: api-recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum by (job) (rate(http_requests_total[5m]))

  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: job:http_errors:ratio5m > 0.05
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "Error ratio is {{ $value | humanizePercentage }} for 10 minutes"
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

Naming and what a recording rule should do

The convention from the documentation is level:metric:operations: the aggregation level (the labels that remain), the underlying metric, then the operations applied, such as job:http_requests:rate5m. The colon is reserved for this purpose in metric names, so a name with colons tells every reader that it came from a rule. Keep the operation suffix honest: a rule named rate5m that actually computes rate1m will eventually be trusted for the wrong window.

Precompute only what you use repeatedly. Good candidates are an aggregation that dashboards and alerts share, anything involving a subquery, and ratios across large series sets. Bad candidates are expressions nobody reads, which cost storage and evaluation time forever. When you aggregate in a recording rule, decide deliberately which labels survive with by or without, because a label you drop cannot be recovered from the recorded series.

The alert lifecycle

An alert moves through three states. While the expression returns nothing, it is inactive. The moment it returns a result, the alert becomes pending, and it stays pending until the expression has held for the for duration, at which point it is firing and is sent to Alertmanager. If the condition clears during the pending window, the alert goes back to inactive and nothing is sent. That is the entire purpose of for: to ride out blips. Without it an alert fires on the first evaluation that crosses the threshold.

keep_firing_for is the mirror image. Once firing, the alert stays firing for that extra time after the condition clears, which stops a flapping signal from resolving and re-firing every few minutes and paging the same person repeatedly.

The labels you attach are the contract with Alertmanager, since routing is decided by them. The annotations are for humans: a summary, a description with the current value through {{ $value }}, and a runbook_url. The runbook link is the highest-value annotation in the file. An alert that cannot tell the on-call engineer what to do next is an alert that wakes someone to read a wiki search.

What for does not protect you from

If the Prometheus server restarts, its for timers restart too, unless it can restore the alert state from its own stored ALERTS_FOR_STATE series. On restart it restores that state after a grace period (--rules.alert.for-grace-period, default 10m) and only for outages shorter than --rules.alert.for-outage-tolerance (default 1h). The practical consequence: an alert with a long for, such as two hours, can be silently reset by a restart, and you should prefer a shorter for with a longer window inside the expression.

Every alert fires again at --rules.alert.resend-delay (default 1m) intervals to Alertmanager. That is normal and is not spam: Alertmanager's own grouping and repeat_interval decide what humans receive.

Evaluation problems you can watch

If a group's evaluation takes longer than its interval, the next evaluation is skipped, and the counter prometheus_rule_group_iterations_missed_total increases. The group's duration is in prometheus_rule_group_last_duration_seconds. An alert that depends on a group that is constantly missing iterations is an alert that evaluates at half frequency with no warning. To cap blast radius, a group-level or rule-level limit sets the maximum number of series or alerts a rule may produce; when a rule exceeds it, all its results are discarded and its alerts cleared, so set it with care.

# which groups are running slow or skipping?
topk(5, prometheus_rule_group_last_duration_seconds)
increase(prometheus_rule_group_iterations_missed_total[1h]) > 0

Alert on symptoms, not causes

The practical design rule that separates a quiet pager from a noisy one is to alert on what users experience (error ratio, latency percentile, saturation of a resource that is about to cause either) and to put cause-level detail on dashboards and in the annotations. A CPU at 90% for one node is not a page. An API error ratio above the threshold is. Every alert you add is a promise to a human that it is worth waking up for.

Alertmanager in practice

Prometheus decides that something is wrong. Alertmanager decides who hears about it, once. It is a separate binary because the problem it solves is different: a hundred instances firing the same alert for the same outage should produce one notification, and a planned maintenance window should produce none. It does that through four mechanisms, and it is worth being exact about each.

• Deduplication. Identical alerts from several Prometheus servers (an HA pair) collapse into one.
• Grouping. Alerts sharing the labels in group_by are batched into a single notification.
• Inhibition. One alert can suppress others; a firing "cluster down" alert mutes the hundred "instance down" alerts it causes.
• Silencing. A time-boxed mute created by a human, matching alerts by labels.

Configuration is a routing tree. Each alert enters at the root route and descends to the first child whose matchers match (or, with continue: true, onward to later siblings too). The matched route names a receiver.

route:
  receiver: default
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
    - matchers: ['team="ml-platform"']
      receiver: ml-slack

inhibit_rules:
  - source_matchers: ['alertname="ClusterDown"']
    target_matchers: ['severity=~"page|warn"']
    equal: ["cluster"]

receivers:
  - name: default
    slack_configs:
      - channel: "#alerts"
        api_url_file: /etc/alertmanager/slack-url
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd-key
  - name: ml-slack
    slack_configs:
      - channel: "#ml-platform"
        api_url_file: /etc/alertmanager/slack-url

The three timers shape behaviour more than anything else. group_wait is how long the first notification for a new group is held, to let related alerts arrive and join it; group_interval is how long to wait before sending an updated notification about a group that already notified; and repeat_interval is how often an unchanged, still-firing group is re-sent. These are Alertmanager's documented defaults of 30 seconds, 5 minutes and 4 hours, and they are worth setting explicitly so that nobody has to guess them later.

group_by is the lever that decides whether you get one message or fifty. Group by alertname and cluster and an outage across a hundred pods becomes one notification per cluster. Group by nothing useful and each alert pages separately. Grouping by a label with very high cardinality, such as pod, defeats the purpose.

Connecting the two binaries is a section of prometheus.yml. Only api_version: v2 exists in 3.x, and Prometheus must list every Alertmanager instance, not a load balancer in front of them, so that each replica receives every alert and can deduplicate through its gossip cluster:

alerting:
  alert_relabel_configs:
    - action: labeldrop
      regex: replica          # so a/b replicas send identical alerts
  alertmanagers:
    - api_version: v2
      static_configs:
        - targets: ["alertmanager-0:9093", "alertmanager-1:9093"]

The labeldrop of the replica external label matters for an HA pair. Two Prometheus servers with different replica labels would otherwise emit alerts that differ in a label, and Alertmanager would treat them as two alerts. Dropping the label before sending makes them identical, so they deduplicate.

Checking and testing the routing

Use amtool, which ships with Alertmanager, to validate a file and to test where an alert would go before you rely on it:

amtool check-config alertmanager.yml
amtool config routes show --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=page team=payments

The last command prints the receiver the labels would reach, which turns "did I break paging" into a command instead of a prayer. Keep it in CI next to promtool.

Cardinality and cost

The problem: a server that was healthy on Friday is out of memory on Monday, and nobody changed the config. Almost always, somebody deployed code that added a label with too many values. Cardinality is the number of unique time series, and because each unique combination of metric name and label values is its own series, it multiplies. A metric with a user_id label and a hundred thousand users is a hundred thousand series, each carrying a full set of histogram buckets if it is a histogram. The per-series cost in memory and in every managed backend's bill is what you are managing.

The rule that prevents most incidents: never put an unbounded value in a label. User IDs, email addresses, request IDs, session tokens, full URLs with query strings, and container IDs that change on each restart are all unbounded. This is also the compliance point from the start of this guide. A label that holds a person's identifier in a monitoring database is personal data in a system with long retention, broad read access and no deletion workflow. The discipline that protects your memory protects you from a data-protection conversation too.

Finding the offenders

Go from the coarse view to the fine one.

# the ten metric names with the most series
topk(10, count by (__name__) ({__name__=~".+"}))

# which jobs contribute the most series
topk(10, count by (job) ({__name__=~".+"}))

# how fast are we adding new series? a high, sustained value means churn
rate(prometheus_tsdb_head_series_created_total[5m])

Run the first of those sparingly on a large server; it touches every series in the head. The status page in the UI (Status, then TSDB Status) and the /api/v1/status/tsdb endpoint answer the same question cheaply, including which label names have the most values. For history and churn across blocks, use promtool tsdb analyze.

Churn is a separate cost from total cardinality. A series that exists for one hour and is replaced by a new one, as happens with every rolling deployment that puts the pod name into a label, still occupies memory while it is in the head and bloats the index on disk. A fleet that deploys often can have modest series at any moment and enormous series over a day.

Reducing it, in order of leverage

1. Fix the source. Remove the label in the code. It is always the cheapest fix and the only one that also removes the load on the application.
2. Drop at ingestion with metric_relabel_configs, when you cannot change the source immediately:

scrape_configs:
  - job_name: "api"
    static_configs:
      - targets: ["api-1:8080"]
    metric_relabel_configs:
      # drop a whole family of metrics nobody uses
      - source_labels: [__name__]
        regex: "go_gc_.*"
        action: drop
      # remove a dangerous label but keep the metric
      - regex: "request_id|user_id"
        action: labeldrop

Mind the order in which relabeling meets your data. Removing a label can make two previously distinct series identical, and then the server logs Error on ingesting samples with different value but same timestamp because two series collapsed into one. Drop labels that do not distinguish anything meaningful, and aggregate the ones that do.

3. Use histograms economically. Every classic histogram bucket is a series per label combination, so ten buckets times a hundred routes times fifty instances is fifty thousand series from one metric. Remove buckets you never query, or move to native histograms, which collapse the buckets into a single series.
4. Lengthen the scrape interval for low-value jobs. Going from 15 seconds to 60 reduces samples fourfold, although it does not reduce active series.
5. Filter before remote write with write_relabel_configs, because managed backends typically charge per active series or per sample ingested.

Then keep it from coming back. Set sample_limit per job so that a cardinality explosion fails one scrape instead of the server, alert when prometheus_tsdb_head_series grows faster than a threshold, and review the top-ten list whenever a service changes how it labels.

http_requests_total{route="/orders/:id", status="200"}

http_requests_total{path="/orders/918273", user="a.hassan@example.com"}

Running Prometheus on Kubernetes

The problem: on Kubernetes the targets are pods that start and die constantly, the configuration belongs to many teams, and a hand-edited prometheus.yml in a ConfigMap becomes a bottleneck. The standard answer is the Prometheus Operator, which turns Prometheus, its scrape targets and its rules into Kubernetes resources that a controller reconciles into a running server.

The Operator's custom resources map directly onto the concepts you already know:

Most teams install the whole stack with the kube-prometheus-stack Helm chart from the prometheus-community repository, which brings the Operator, Prometheus, Alertmanager, Grafana, node_exporter as a DaemonSet, kube-state-metrics, and a large set of default dashboards and rules. Treat the chart version as separate from the Prometheus server version: the chart pins its own default image, and upgrading one does not imply the other. See the Helm guide for the packaging mechanics and the Kubernetes guide for the cluster concepts underneath.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: payments
  labels:
    release: kube-prometheus-stack     # must match the Prometheus' serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: payments-api
  endpoints:
    - port: metrics                    # the NAME of the Service port, not a number
      interval: 30s
      path: /metrics

The two most common reasons a ServiceMonitor "does nothing" are both selector mismatches, and neither produces an error. First, the Prometheus resource has a serviceMonitorSelector (in kube-prometheus-stack it defaults to selecting the chart's own release label), so a ServiceMonitor without that label is ignored. Second, port refers to the name of a port in the Service, so if the Service names it http-metrics and the monitor says metrics, no endpoints match. Debug from the inside out: check the Service has endpoints (kubectl get endpoints), check the labels on the Service match spec.selector, then look at the Prometheus UI's Service Discovery page for the serviceMonitor/payments/payments-api/0 pool.

The Operator generates standard kubernetes_sd_configs and relabeling for you, which is exactly the pipeline from earlier in this guide. Open the generated configuration once (Status, then Configuration) so that the machinery is not a mystery the next time something misbehaves.

Standard exporters on a cluster

Three sources account for most cluster metrics. node_exporter, one per node as a DaemonSet, exposes host CPU, memory, disk and network. kube-state-metrics turns the Kubernetes API's object state (deployments, pod phases, replica counts) into metrics, and answers "how many replicas are desired versus available". cAdvisor, built into the kubelet and scraped at its /metrics/cadvisor path, exposes per-container CPU, memory and network, which is what container_cpu_usage_seconds_total and container_memory_working_set_bytes come from.

For a container that keeps restarting, combine them:

# containers restarted more than 3 times in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3

# memory working set as a fraction of the limit
container_memory_working_set_bytes{container!=""}
  / on (namespace, pod, container)
kube_pod_container_resource_limits{resource="memory"}

Delivery and scaling integrations

Prometheus metrics are the signal that feeds other parts of the platform. Progressive delivery tools such as Argo Rollouts run PromQL analysis queries against Prometheus to decide whether a canary is healthy (see the Argo CD guide for the GitOps side). KEDA's Prometheus scaler and the prometheus-adapter let a Kubernetes HorizontalPodAutoscaler scale on a query result such as requests per second or queue depth. In both, the query you write becomes a control-loop input, so treat it as production code: test it, guard it against a missing series with or vector(0) where an empty result would otherwise mean "no data, do nothing" versus "zero load, scale down", and be deliberate about which.

Scaling out and staying available

A single Prometheus server is deliberately simple: no clustering, no shared storage, no consensus. That is the source of its reliability and also its limits. The problem at this level is what to do when one server is not enough, or when a single server's failure is unacceptable. The answer is to keep each server standalone and compose several of them.

High availability means two identical replicas

There is no Prometheus cluster. To survive a server failure you run two identical servers that scrape the same targets independently, differing only in an external label, for example replica: a and replica: b. Each has its own data, so a dead one leaves the other serving queries and alerts. Both send to the same Alertmanager cluster, which deduplicates. You then query one of them, or a layer such as Thanos or Mimir that merges the two copies.

global:
  external_labels:
    cluster: prod-eu1
    replica: a     # b on the second server

External labels are attached to series when they leave the server (federation, remote write) and to alerts, never to locally stored series. Since 3.0, ${var} environment variable expansion works inside external_labels (and only there), which lets one rendered config file serve both replicas with replica: ${POD_NAME}. Use $$ to write a literal dollar sign.

Two replicas do not produce identical data: their scrape timings differ by milliseconds, so values are close but not equal, and a counter reset can land on different sides of a scrape. Do not try to reconcile them; choose one for dashboards or let a global-view layer deduplicate by the replica label.

Federation, used narrowly

Federation lets one Prometheus scrape selected series from another's /federate endpoint, using match[] selectors. It is meant for aggregated series: a global server that pulls job:http_requests:rate5m from each regional server. It is not a way to copy all the data, which is slow, doubles the storage and cannot keep up, so if you are federating raw series you have reached the point where remote write is the right tool.

scrape_configs:
  - job_name: "federate"
    honor_labels: true          # the source server's labels are the truth
    metrics_path: /federate
    params:
      "match[]":
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets: ["prom-eu1:9090", "prom-eu2:9090"]

Remote write and long-term storage

Local storage is bounded by one disk and one retention. For longer history, a global view, or a managed backend, Prometheus streams samples to another system with remote write: Thanos Receive, Grafana Mimir, Cortex, VictoriaMetrics, or a cloud service (Amazon Managed Service for Prometheus, Google Managed Prometheus, Azure Monitor managed Prometheus, Grafana Cloud). The protocol is the same in all of them; what differs is the authentication block.

remote_write:
  - url: https://mimir.example.com/api/v1/push
    name: mimir
    remote_timeout: 30s
    basic_auth:
      username: metrics
      password_file: /etc/secrets/mimir-password
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop
    queue_config:
      capacity: 10000
      max_shards: 50
      min_shards: 1
      max_samples_per_send: 2000
      batch_send_deadline: 5s

Understand the mechanism to operate it. Remote write reads recently written samples from the WAL and puts them into an in-memory queue feeding a dynamic number of shards, each sending batches in parallel. When the destination is slow, Prometheus adds shards up to max_shards to catch up; when it is still behind, the WAL keeps the data until it is sent or ages out of the WAL's retention, at which point samples are lost. So the two states worth alerting on are lag and failure:

# seconds the remote endpoint is behind the local head
max(prometheus_remote_storage_highest_timestamp_in_seconds)
  - on () group_right
max by (remote_name, url) (prometheus_remote_storage_queue_highest_sent_timestamp_seconds)

# samples that failed permanently
rate(prometheus_remote_storage_samples_failed_total[5m]) > 0

# shards maxed out: the backend cannot keep up
prometheus_remote_storage_shards == prometheus_remote_storage_shards_max

An HTTP 429 from the backend means a tenant limit: reduce series, filter with write_relabel_configs, or raise the limit. Other 4xx responses are non-recoverable and the affected batch is dropped, which is why samples_failed_total matters. Note also that the default for HTTP/2 on remote write has been false since 3.0, which matters if a proxy in the path only speaks HTTP/2 well.

Agent mode

When a server's only job is to scrape and ship, for an edge site or a small cluster feeding a central backend, run it with --agent. Agent mode disables local querying, rules and alerting, keeps only a short WAL (by default between five minutes and four hours, governed by --storage.agent.retention.min-time and .max-time), and uses far less memory and disk. Note the syntax changed in 3.0: the old --enable-feature=agent is gone and the dedicated --agent flag replaced it, with its own --storage.agent.path.

prometheus --agent \
  --config.file=agent.yml \
  --storage.agent.path=/var/lib/prometheus-agent

The official documentation's line about the agent serving a UI on port 9095 does not match the code. The agent uses the same --web.listen-address default as the server, 0.0.0.0:9090, which you can change.

Sharding

When a single server cannot hold all your series, you split the targets. Functional sharding gives each team or job group its own server. Hash-based sharding divides one job across servers with hashmod:

relabel_configs:
  - source_labels: [__address__]
    modulus: 3
    target_label: __tmp_shard
    action: hashmod
  - source_labels: [__tmp_shard]
    regex: "0"            # this server is shard 0 of 3
    action: keep

Sharding fragments the query surface, because no single server holds everything, which is the point where Thanos, Mimir or a managed backend earns its complexity by giving you one global query endpoint.

Securing the server

The problem: Prometheus ships with no authentication and no authorisation, and it serves everything it knows, including your target list, rule files and (in /-/config) a rendering of your configuration. The security model in the documentation is explicit: it is presumed that untrusted users can reach the HTTP endpoint and read the logs, and that only trusted users can change the command line, config and rule files. Everything you add is on top of that stance.

The built-in protection is the web config file. Passed as --web.config.file=web.yml, it provides TLS, mutual TLS and basic authentication with bcrypt hashes. It is still labelled experimental in --help, though it is widely used, and it is re-read on every request, so certificate rotation needs no restart.

tls_server_config:
  cert_file: /etc/prometheus/tls/server.crt
  key_file: /etc/prometheus/tls/server.key
  min_version: TLS12
  # client_auth_type: RequireAndVerifyClientCert
  # client_ca_file: /etc/prometheus/tls/ca.crt
basic_auth_users:
  admin: $2y$10$REPLACE_WITH_A_REAL_BCRYPT_HASH

# generate a bcrypt hash, then validate the file
htpasswd -nBC 10 "" | tr -d ':\n'
promtool check web-config web.yml

Once you turn on basic auth, every client must present it: Grafana's data source, Alertmanager's callbacks, other Prometheus servers federating, and health probes. Kubernetes liveness and readiness probes against /-/healthy will start failing with 401, and the fix is a probe that sends the header or one targeting a different port.

For real users, put a reverse proxy in front. Prometheus has no roles, no per-user access and no audit trail. If you need single sign-on, different permissions for different teams, or logging of who ran what query, run it behind oauth2-proxy, nginx or Envoy, and use --web.external-url (and --web.route-prefix if you serve under a path) so the links and redirects are generated correctly behind the proxy. For per-tenant isolation of queries, a label-enforcing proxy such as prom-label-proxy injects a mandatory label matcher; true multi-tenancy, however, is something Mimir, Cortex and Thanos Receive provide, and Prometheus does not.

Endpoints that should stay off

Three features are disabled by default, and each should stay that way unless you know who can reach the port:

The lifecycle flag is usually needed for reloads, and binding the server to a private interface or protecting it with the web config covers it. Prefer kill -HUP or --config.auto-reload (stable since 3.12, checking every --config.auto-reload-interval, default 30s) and skip the lifecycle endpoints entirely if you can.

Secrets

Fields typed as secret in the configuration, such as passwords, bearer tokens and client secrets, are redacted in logs and in /api/v1/status/config and /-/config. Two habits follow. Put credentials only in those fields, since a password embedded in a URL or a label is not redacted. And prefer the *_file variants (password_file, credentials_file, client_secret_file), mounted from a Kubernetes Secret or from Vault (see the Vault guide), so that prometheus.yml can live in version control. Environment variables are not expanded in general config fields, only in external_labels, so do not expect password: ${PW} to work.

Exporters and neighbours are attack surface too

The blackbox exporter and similar parameterised exporters can be made to probe arbitrary internal URLs by anyone who can reach them. The Pushgateway lets anyone with access overwrite or delete pushed series. Alertmanager lets anyone with access create silences, and it serves permissive CORS headers. None of these have authentication by default, so the network policy and the proxy in front of them carry the whole weight. Keep the server patched, too: 2026 brought several fixes, including a stored XSS in the UI and an Azure AD client_secret being exposed through /-/config, both fixed in 3.11.3, so anything older than 3.11.3 should be upgraded.

Testing rules and configuration before they ship

The problem: a rule file with a typo, a wrong label or a threshold that never triggers fails silently. You find out during the incident it was supposed to catch. Prometheus gives you two tools to move that discovery into CI, and using them is the single highest-leverage habit at this level.

promtool check validates syntax and some semantics without running anything:

promtool check config prometheus.yml      # also validates the rule files it references
promtool check rules rules/*.yml          # syntax and duplicate-rule lint
promtool check config --lint=all --lint-fatal prometheus.yml
promtool check web-config web.yml
promtool promql format 'sum(rate(x[5m]))by(job)'

A successful run prints SUCCESS: prometheus.yml is valid prometheus config file syntax. The --lint-fatal flag turns lint warnings, such as duplicate rules and too-long scrape intervals, into failures, which is what you want in a pipeline.

promtool test rules goes further: it evaluates your rules against synthetic input series at simulated times and compares the output with what you expect. It is unit testing for alerts.

rule_files:
  - ../rules/api.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # 100 requests per minute, 10 of them errors, for 15 minutes
      - series: 'http_requests_total{job="api", status="200"}'
        values: '0+90x15'
      - series: 'http_requests_total{job="api", status="500"}'
        values: '0+10x15'

    promql_expr_test:
      - expr: job:http_errors:ratio5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:http_errors:ratio5m{job="api"}'
            value: 0.1

    alert_rule_test:
      - eval_time: 5m          # still pending: for is 10m
        alertname: HighErrorRate
        exp_alerts: []
      - eval_time: 15m         # past for: firing
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              team: payments
              job: api
            exp_annotations:
              summary: "High error rate on api"
              description: "Error ratio is 10% for 10 minutes"
              runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

The series notation 0+90x15 means "start at 0, add 90 each step, for 15 steps", so each one-minute step increases the counter by 90. Two counters growing at 90 and 10 per minute give an error ratio of exactly 0.1, and the test states that expectation. The second half tests the alert at two times, before and after for has elapsed, which also proves the timer behaves as you designed. Run it with:

promtool test rules tests/api_test.yml

Unit Testing:  tests/api_test.yml
  SUCCESS

Note how the exp_annotations must match the rendered template, including humanizePercentage output. That is a feature: a template that stops rendering is caught here.

Together these produce a pipeline. On every pull request that touches monitoring config, run promtool check config, promtool check rules, promtool test rules, and amtool check-config for the Alertmanager file. A few seconds of CI replace a category of 3 a.m. surprises. See the GitHub Actions guide for wiring that into a workflow.

Configuration management and upgrades

The problem: the server works, and then nobody dares touch it. Configuration belongs in version control, changes go through review, and a bad change must fail safe. Prometheus is unusually good at the last part. An invalid configuration is never applied: a failed reload logs the error, leaves the old configuration running, and sets prometheus_config_last_reload_successful to 0.

Reloading safely

There are three ways to reload, and you should pick one deliberately:

The older feature flag --enable-feature=auto-reload-config still works but logs a deprecation warning telling you to use --config.auto-reload. Whichever you choose, alert on failure: prometheus_config_last_reload_successful == 0 is one of the most valuable alerts you can have, because a rejected reload looks like success from the pipeline that triggered it while the server keeps serving the old configuration for days.

What changed from flags to file

Several settings moved into the config file in the 3.x line, and they are reloadable there, whereas flags are not:

• Retention (storage.tsdb.retention.time and .size), since 3.8; the flags are deprecated.
• Log level (runtime.log_level), since 3.15; --log.level is deprecated. The config value is reloadable, so you can turn on debug logging during an incident without a restart and turn it off afterwards.
• Chunk encoding (storage.tsdb.chunk_encoding.floats).
• Native histogram ingestion (scrape_native_histograms).

runtime:
  log_level: info    # change to debug, reload, investigate, change back
  gogc: 75

Deprecated does not mean broken, and old flags keep working through this major version. But new behaviour lands in the file, and a config that uses the file forms is the one that will keep working.

Splitting configuration across files

A monolithic prometheus.yml that every team edits is a merge-conflict factory. The scrape_config_files key includes scrape configs from separate files, so each team owns its own:

scrape_config_files:
  - /etc/prometheus/scrape.d/*.yml
rule_files:
  - /etc/prometheus/rules.d/*.yml

Each included file holds a scrape_configs: list. Job names must be unique across all of them, and a duplicate produces found multiple scrape configs with job name "X" at load time. File-based service discovery (file_sd_configs) is the other decoupling point: Terraform or Ansible can write the target list, and Prometheus picks it up with no config change at all. See the Terraform guide and Ansible guide for producing those files.

Upgrading

Prometheus releases a minor version every six weeks, and a given minor version stops receiving fixes soon after the next one arrives. There is no supported 2.x line any more. Choose between tracking the latest minor (3.15 at time of writing) and a long-term support release (3.13.x, supported until 2027-07-31) which receives selected bug fixes for a year. A team without time to read a changelog every six weeks should be on LTS.

A safe upgrade has a routine:

1. Read the [CHANGE] lines in the CHANGELOG between your version and the target, and the migration guide if crossing a major version.
2. Run promtool check config and promtool check rules using the new binary against your real files.
3. Look for feature-flag deprecation warnings in the old server's logs, since each one is a future break.
4. Upgrade the replicas of an HA pair one at a time, confirming the first is healthy and ingesting before touching the second.
5. Watch prometheus_config_last_reload_successful, up, and the rule-evaluation failure counter for a day.

If you are coming from 2.x, upgrade to 2.55 first and then to 3.x, because a 3.x data directory can only be read back by 2.55 and later, which is also the floor for any downgrade. Turning on XOR2 encoding or st-storage adds a one-way door of its own, as described earlier.

Debugging beyond the targets page

The problem: something is wrong and the obvious views look fine. You need a method that narrows the search quickly, in the order in which failures usually occur along the pipeline from target to screen.

Start at the source. Is the target producing what you think? Fetch it directly from the Prometheus host, bypassing Prometheus entirely, to separate "the exporter is wrong" from "Prometheus is misreading it":

curl -s http://payments-1:8080/metrics | head -40
curl -s http://payments-1:8080/metrics | promtool check metrics
curl -sI http://payments-1:8080/metrics | grep -i content-type

promtool check metrics lints an exposition for naming problems. The Content-Type check is there because of a 3.0 change: Prometheus now insists on a valid Content-Type, and a target that sends text/html or none fails with received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target. The proper fix is in the exporter. Where that is out of your hands, set fallback_scrape_protocol: PrometheusText0.0.4 on the scrape config.

Then the scrape. The Targets page gives the last error per target, and the common ones read like this:

For context deadline exceeded, compare scrape_duration_seconds for the job to scrape_timeout. A duration hugging the timeout means a genuinely slow target, and a failure with no recorded duration means the network never connected.

Then ingestion. Samples can be rejected after a successful scrape. Watch the log for out-of-order and duplicate-sample warnings, and the counters prometheus_target_scrapes_sample_out_of_order_total, prometheus_target_scrapes_sample_duplicate_timestamp_total and prometheus_target_scrapes_sample_out_of_bounds_total.

Then the query. Parse errors (1:1: parse error: ...) come with a position, and the usual ones are rate(x) where rate(x[5m]) was needed, and a selector with no non-empty matcher ({job=~".*"} is rejected, use {job=~".+"}). Query-engine limits surface as query processing would load too many samples into memory in query execution (over --query.max-samples, default 50 million) and query timed out in expression evaluation (over --query.timeout, default 2 minutes). The right response is to narrow the query or add a recording rule, and not to raise the limit, which just moves the failure to the next person.

The API is your debugger

Everything the UI does is an HTTP call you can script. A few you will use weekly:

# is this expression returning what I think at this instant?
curl -s --data-urlencode 'query=up == 0' localhost:9090/api/v1/query | jq .

# what did a range query return, with execution stats?
curl -s --data-urlencode 'query=rate(http_requests_total[5m])' \
  --data-urlencode "start=$(date -u -v-1H +%s)" --data-urlencode "end=$(date -u +%s)" \
  --data-urlencode 'step=60' --data-urlencode 'stats=all' \
  localhost:9090/api/v1/query_range | jq '.data.stats'

# which series exist for a selector?
curl -s -G localhost:9090/api/v1/series --data-urlencode 'match[]=up{job="api"}' | jq .data

# what are my dropped and active targets?
curl -s 'localhost:9090/api/v1/targets?state=dropped' | jq '.data.droppedTargets | length'

# what flags is the server actually running with?
curl -s localhost:9090/api/v1/status/flags | jq .

The response envelope is always {"status": "success" | "error", "data": ..., "errorType": ..., "error": ..., "warnings": [], "infos": []}. The warnings array matters more than people think: a query can succeed with a warning such as a remote-read partial failure, and the UI may not draw attention to it. One 3.14 note: the stats parameter is deprecated for any value except true and all; other values still work but return a warning. The date -v-1H form above is macOS/BSD syntax; on Linux use date -u -d '1 hour ago' +%s.

Logs, profiling and the query log

When the server itself misbehaves, slow or heavy, go to its own telemetry. Log lines in 3.x use slog format: time=... level=INFO source=... msg=..., which changed from the ts=... caller=... of 2.x and breaks log parsers written for the old form. Set global.query_log_file to write every query, with timing, to a file, which is how you find the dashboard that is crushing the server. Go profiling is exposed at /debug/pprof/, and promtool debug pprof http://localhost:9090 collects a bundle of profiles you can attach to an issue. For CPU time attribution on a busy server, the 3.10 /debug/pprof/fgprof endpoint gives wall-time profiles.

Watching Prometheus itself

The problem: a monitoring system that fails quietly is worse than none, because people trust its silence. You need to know that Prometheus is scraping, evaluating, alerting and storing, and you need something outside Prometheus that notices if it stops.

Prometheus exposes its own metrics at /metrics and scrapes itself in the default config. The alerts worth having on every server, each with the reason:

Rather than writing these from scratch, the project maintains a Prometheus mixin in its repository (documentation/prometheus-mixin), which ships dashboards and alerts such as PrometheusHAGroupNotIngestingSamples. Start from it and trim.

Who watches the watcher

Every alert above runs on the server it monitors, so a dead server is silent. Two patterns close that gap. First, a second Prometheus (often a small one per environment) scrapes the first and alerts on up{job="prometheus"} == 0. Second, a dead-man's-switch: an alert that always fires, conventionally named Watchdog, sent through the normal path to an external service that pages you when the notifications stop arriving. It tests the whole chain, including Alertmanager and the network, in a way no internal alert can.

groups:
  - name: meta
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Always firing. If this stops arriving, the alerting pipeline is broken."

For remote write users, add the lag and failure queries from the scaling section. For the health of the TSDB, /api/v1/status/walreplay and prometheus_tsdb_head_series tell you about restart time and growth.

Observing machine-learning services

The reason this catalogue has a Prometheus guide is that model serving is operational work, and the serving frameworks you meet speak Prometheus. KServe, Seldon, BentoML, Triton, vLLM and Ray Serve all expose request counters and latency histograms at a metrics endpoint, and GPU hosts expose utilisation through NVIDIA's DCGM exporter. Connect them with the machinery from this guide and the ML service is monitored like any other.

What to watch differs from a web service in three ways.

Latency is bimodal and long-tailed. An LLM endpoint's time to first token and its total generation time are different numbers for different reasons, and an average of either hides the tail. Use histograms, and alert on percentiles of each rather than one overall latency. Choose bucket boundaries wide enough that a 30-second generation does not fall into the last +Inf bucket, where the percentile estimate becomes meaningless.

Saturation is a queue, not a CPU. GPU inference saturates by queueing requests, so the useful signals are queue depth, batch size, and GPU utilisation and memory together. A service can be at 100% GPU with a healthy queue and be fine, or at 60% with a growing queue and be failing. Pair a utilisation series with a queue series in the alert.

# p95 time-to-first-token over 5m, per model (metric names vary by framework)
histogram_quantile(0.95,
  sum by (le, model) (rate(llm_time_to_first_token_seconds_bucket[5m])))

# GPU memory pressure from the DCGM exporter
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.95

Metric names differ between frameworks and versions, so take them from the target's own /metrics output rather than from a blog post. The queries show the pattern.

Model quality is not infrastructure. Prometheus is excellent at telling you a model endpoint is slow or erroring, and poor at telling you its predictions have drifted. Per-request quality data belongs in a system built for it; see the Evidently guide for drift and the Grafana guide for putting infrastructure and model signals on one dashboard. Aggregate quality signals that you compute elsewhere can be exposed as gauges, for Prometheus to alert on, which keeps paging in one place.

Finally, be wary of labelling metrics with model inputs. A prompt or customer label is unbounded, may be personal data, and is the fastest route from a working ML platform to a cardinality incident and a compliance review.

Putting it all together

Here is one small but realistic setup that uses most of this level: a payments API with an HA pair of Prometheus servers, file-based discovery, a cardinality fence, recording and alerting rules, tests and meta-monitoring, all checked in CI.

The repository layout:

monitoring/
├── prometheus.yml
├── scrape.d/
│   └── payments.yml
├── targets/
│   └── payments.yml
├── rules.d/
│   ├── payments.yml
│   └── meta.yml
├── tests/
│   └── payments_test.yml
├── alertmanager.yml
└── ci.sh

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    cluster: prod-eu1
    replica: ${POD_NAME}

runtime:
  log_level: info

storage:
  tsdb:
    retention:
      time: 30d
      size: 400GB

rule_files:
  - /etc/prometheus/rules.d/*.yml
scrape_config_files:
  - /etc/prometheus/scrape.d/*.yml

alerting:
  alert_relabel_configs:
    - action: labeldrop
      regex: replica
  alertmanagers:
    - api_version: v2
      static_configs:
        - targets: ["alertmanager-0:9093", "alertmanager-1:9093"]

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

scrape_configs:
  - job_name: "payments-api"
    sample_limit: 50000
    label_limit: 30
    file_sd_configs:
      - files: ["/etc/prometheus/targets/payments.yml"]
        refresh_interval: 1m
    metric_relabel_configs:
      - regex: "request_id|user_id"
        action: labeldrop

- targets: ["payments-1:8080", "payments-2:8080"]
  labels:
    team: payments
    env: prod

The rules and tests are the ones from earlier sections, plus the Watchdog and a pair of meta-alerts. The CI script is the part that makes the rest trustworthy:

#!/usr/bin/env bash
set -euo pipefail

promtool check config --lint=all --lint-fatal prometheus.yml
promtool check rules rules.d/*.yml
promtool test rules tests/*.yml
amtool check-config alertmanager.yml
echo "monitoring config OK"

Walk through what a change to this repository now looks like. A developer edits the error-ratio threshold in rules.d/payments.yml and opens a pull request. CI runs the script, and the unit test, which asserts the alert fires at fifteen minutes and is pending at five, fails if the change broke the intent. After review it merges, the config reaches both replicas, --config.auto-reload applies it within thirty seconds, and prometheus_config_last_reload_successful stays at 1, which a meta-alert would report if it did not. A new payments pod is added to targets/payments.yml by Terraform, and the file-based discovery picks it up with no reload. A bug deploys a user_id label: sample_limit fails that job's scrapes loudly, the exceeded_sample_limit alert fires, and the server stays up. Through all of it, Watchdog delivery to the external service proves the alert path is alive.

Where you are now

You can now predict how Prometheus behaves instead of discovering it. You know what a scrape stores and why a series disappears the moment a target does. You can reason about the head, the WAL, blocks and compaction well enough to size a disk, explain a slow restart and choose retention in the config file. You can build a discovery and relabeling pipeline and debug it from the Service Discovery page. You write PromQL that survives resets, matches the right series and estimates percentiles honestly, and you know which queries to turn into recording rules. Your alerts have for, routes, runbooks and unit tests, and your configuration fails safe and is checked before it ships. You can keep cardinality and cost in check, run on Kubernetes through the Operator, build HA pairs, federate narrowly, stream to long-term storage, secure the server behind TLS and a proxy, and watch Prometheus itself from outside.

The Senior level treats Prometheus as a platform that other teams depend on: how its architecture fails at scale and which component breaks first, the trust model and supply chain, multi-tenancy through Mimir, Thanos and managed services, cost governance, upgrade and migration planning, incident playbooks, and the honest boundaries of where Prometheus stops and something else should take over. Before you go there, be sure you can do the following without looking anything up: state the formula for disk sizing, explain for versus keep_firing_for, write a correct histogram_quantile, find the highest-cardinality metric on a server, and say what happens when a config reload fails.

For the neighbouring tools, continue to the Grafana guide for dashboards on top of this data, the Kubernetes guide for the platform Prometheus most often monitors, and the OpenTelemetry guide for how traces and metrics from instrumented applications meet Prometheus through the OTLP receiver.

Sources

• Prometheus overview
• Release cycle and LTS policy
• Prometheus 3.0 migration guide
• Configuration reference
• Recording rules
• Alerting rules
• Unit testing for rules
• Securing with TLS and basic auth
• Storage
• Querying basics
• Query operators
• Query functions
• HTTP API
• Management API
• Federation
• Agent mode
• Feature flags
• Command-line flags
• promtool reference
• Data model
• Metric types
• Jobs and instances
• Security model
• Naming practices
• Rules practices
• Alertmanager overview
• Prometheus 3.15.0 release
• Changelog, release-3.15 branch

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than level, in about thirty-five minutes. At this level interviewers stop asking what a counter is and start asking what you would check first when something is wrong, and why. Fast review first, scenario and trade-off questions after it, and a self-test at the end. Senior reviews all three levels again with platform-ownership questions on top. Everything here targets Prometheus 3.15, and where an answer changed in 3.x the old answer is flagged, because interviewers who use Prometheus daily enjoy catching a 2.x habit.

Where this picks up

Foundations, in one screen

Prometheus is an open-source monitoring system that periodically pulls metrics over HTTP from instrumented targets, stores them as labelled time series in a local database, evaluates rules on a schedule, and sends alerts to Alertmanager. You query it with PromQL. Each server is standalone: no clustering and no shared storage, which is why it is reliable and also why scaling it means composing servers.

A series is a metric name plus a full label set. Change one label value and you have a different series. An empty label value is the same as no label.

Four metric types. Counter (only goes up, reset on restart, query with rate), gauge (up and down), histogram (cumulative buckets plus _sum and _count, aggregatable, quantiles with histogram_quantile), summary (client-side quantiles that cannot be aggregated). Native histograms, stable since 3.9, replace the bucket series with one sparse series.

Pull, not push. The server decides when to scrape, so a missed scrape is itself a signal (up == 0). The Pushgateway exists for short-lived batch jobs only.

Defaults to have ready. Port 9090. scrape_interval 1m and scrape_timeout 10s (the example config overrides the interval to 15s). Retention 15 days. Lookback 5 minutes. Alertmanager 9093, node_exporter 9100, Pushgateway 9091.

Commands and endpoints to have ready

promtool check config prometheus.yml
promtool check rules rules/*.yml
promtool test rules tests/*.yml
promtool tsdb analyze /var/lib/prometheus
promtool query instant http://localhost:9090 'up == 0'
amtool config routes test --config.file=alertmanager.yml severity=page

curl -s localhost:9090/-/healthy           # liveness, always 200 if the process runs
curl -s localhost:9090/-/ready             # 503 while replaying the WAL
curl -X POST localhost:9090/-/reload       # needs --web.enable-lifecycle
kill -HUP $(pidof prometheus)              # always works
curl -s localhost:9090/api/v1/status/tsdb  # cardinality stats
curl -s 'localhost:9090/api/v1/targets?state=dropped'

The scrape loop and staleness

What does up tell you, and what does it not? up is a synthetic series Prometheus writes for every scrape: 1 if the scrape succeeded, 0 if it failed. It proves the endpoint answered within the timeout. It does not prove the service works. A process whose metrics handler is healthy while its request handlers are deadlocked stays at up 1. Alert on up == 0 for the monitoring layer and on an error ratio or request rate for the service.

How does Prometheus know a series has gone? When a target disappears or stops exposing a series, the next scrape writes a stale marker, and queries stop returning that series immediately. Without it, the five-minute lookback delta would keep a dead pod in your sum() for five minutes. If a series simply stops with no marker, a query keeps returning its last value until the lookback window passes.

What timestamp does a sample get? The time the scrape started, unless the target supplies its own and honor_timestamps is true (the default). Target-supplied timestamps are a classic source of out-of-order rejections when clocks skew.

When would you set honor_labels: true? Two cases: federation, where the source Prometheus's labels are the truth, and the Pushgateway, where the pushed job name is the identity. Everywhere else leave it false. With it on, a scraped application can overwrite job and instance, which lets it impersonate another job.

PromQL

Why must you use rate() on a counter? A raw counter is cumulative since process start, so its absolute value means nothing and resets to zero on restart. rate returns the per-second increase over a window and corrects for resets. increase is rate multiplied by the window length.

rate or irate? rate averages over the whole window. irate uses only the last two samples and draws jagged, responsive lines. Use irate to eyeball a fast counter, and rate in alerts and recording rules. An alert on irate flaps on a single-scrape blip and can miss a spike that falls between two points.

How long should the range be? At least four times the scrape interval. rate needs two samples in the window, and a single failed scrape at a thirty-second interval leaves a one-minute window with one.

# error ratio: ratio of rates, each summed first
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

# p99 from a classic histogram
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# disk full within 4 hours, from a 6-hour trend
predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0

What goes wrong if you take the rate of a sum? Summing counters from several instances first means that one instance restarting makes the sum fall, which looks like a reset and produces a spurious drop. Always sum the rates, not rate the sum.

Explain vector matching. A binary operator pairs elements with identical label sets. on(...) restricts the match to the listed labels and ignoring(...) excludes labels. By default the match is one-to-one, and a many-to-one raises many-to-many matching not allowed: matching labels must be unique on one side. When the fan-out is intentional, group_left or group_right names the many side. The common use is joining an info metric (value 1) to add a label such as version.

What changed in 3.x that breaks old queries? The le and quantile label values are normalised to float form, so le="1" is now le="1.0". The PromQL regex . matches newlines. Range selectors are left-open and right-closed, so foo[1m:1m] may return a single point and you should use foo[2m:1m]. holt_winters became double_exponential_smoothing and sits behind the experimental-functions flag.

Why is a percentile from a summary different from one from a histogram? A summary computes quantiles in the client and exposes them, so they are precise for one process and cannot be combined across instances. A histogram exposes bucket counts, which add across instances, so histogram_quantile over summed buckets gives a fleet-wide estimate, bounded in accuracy by the bucket boundaries.

Storage and capacity

Describe how Prometheus stores data. Samples go to the in-memory head block and to the write-ahead log. Every two hours the head is cut into an immutable on-disk block holding chunks, an index, metadata and tombstones. Blocks are compacted into larger ones up to 10% of retention or 31 days, whichever is smaller. Deletion writes tombstones, and expired blocks are removed whole.

Why does a restart take ten minutes? WAL replay. The server rebuilds the head from the WAL on startup, and the time scales with the number of active series. /-/ready returns 503 during it, and /api/v1/status/walreplay shows progress. A SIGKILL or an out-of-memory kill leaves a longer WAL than a graceful SIGTERM.

How do you size the disk? retention_seconds * ingested_samples_per_second * bytes_per_sample, at one to two bytes per sample. Measure the ingest rate with rate(prometheus_tsdb_head_samples_appended_total[5m]), and add about a third for compaction headroom, since source and target blocks coexist during a compaction.

How do you set retention in 3.x? In the config file, under storage.tsdb.retention.time and .size. The --storage.tsdb.retention.time and .size flags are deprecated since 3.8 but still work, and the file values are reloadable and take precedence. If both time and size are set, whichever trips first applies.

What does memory depend on? Active series in the head, plus churn. Watch prometheus_tsdb_head_series and process_resident_memory_bytes. A practitioner rule of thumb is a few kilobytes per series, but the official documentation does not give a fixed figure, and I would say so.

Can the data directory go on NFS? No. The documentation lists non-POSIX filesystems as unsupported, and the failure is corruption. Use local SSD or block storage.

Cardinality

What is cardinality and why does it matter? The number of unique series. Each combination of metric name and label values is a series, so labels multiply. It drives memory, disk, query time and, on a managed backend, the bill.

Your server went from 2 million to 9 million series after a deploy. What do you do? Confirm with prometheus_tsdb_head_series and its growth rate. Find the metric with topk(10, count by (__name__) ({__name__=~".+"})), or cheaper through /api/v1/status/tsdb and the TSDB Status page. Find the label with the highest value count on that metric. Fix the source by removing the label in the code, and in the meantime drop it with labeldrop in metric_relabel_configs. Put a sample_limit on the job so the next explosion fails one scrape.

Name an unbounded label. user_id, email, request ID, full URL path with IDs, session token, a container ID that changes on each restart. They are also often personal data, which is a compliance problem on top of the cost problem.

Rules and alerting

What is the difference between a recording rule and an alerting rule? A recording rule evaluates an expression on a schedule and stores the result as a new series, named by convention level:metric:operations, so dashboards read a cheap precomputed value. An alerting rule evaluates a condition and produces alerts that go to Alertmanager.

Walk through an alert's states. Inactive, pending while the expression is true but for has not elapsed, firing after it has. If the expression turns false during pending, it returns to inactive and nothing is sent. keep_firing_for holds the firing state for an extra period after the condition clears, which stops flapping.

What happens to an alert's for timer if Prometheus restarts? It would restart, but Prometheus restores the state from the stored ALERTS_FOR_STATE series after a grace period (--rules.alert.for-grace-period, default 10 minutes), and only if the outage was shorter than --rules.alert.for-outage-tolerance (default 1 hour). So very long for durations are fragile across restarts.

Rules in a group: what order? Sequentially in the order written, at the group's interval. So a recording rule can feed an alert below it in the same group. If evaluation overruns the interval the next one is skipped and prometheus_rule_group_iterations_missed_total increases.

How do you test an alert? promtool test rules, with input_series written in expanding notation (0+90x15), promql_expr_test for recorded values and alert_rule_test at chosen eval_time values. Test the pending state and a case where the alert must not fire.

Alertmanager

What does Alertmanager do that Prometheus does not? Deduplication, grouping, inhibition, silencing, and routing to receivers. Prometheus decides that something is wrong, and Alertmanager decides who hears about it, once.

Explain group_wait, group_interval and repeat_interval. The first is how long to hold the first notification for a new group so related alerts join it, the second is how long before sending an update to a group that already notified, and the third is how often an unchanged, still-firing group is re-sent. The documented defaults are 30 seconds, 5 minutes and 4 hours.

How do you make an HA pair of Prometheus servers not double-page? Give them different replica external labels, drop that label with alert_relabel_configs before sending, and point both at all Alertmanager instances, never a load balancer, so the Alertmanager cluster deduplicates identical alerts.

What is the difference between inhibition and a silence? Inhibition is automatic and rule-based: one firing alert suppresses others with matching equal labels. A silence is manual and time-boxed, matching by labels. Both leave Prometheus's view of the alert unchanged.

Service discovery and relabeling

What are the four relabeling stages? relabel_configs on targets before the scrape, metric_relabel_configs on samples after the scrape and before storage, alert_relabel_configs on alerts before Alertmanager, and write_relabel_configs per remote write.

A keep rule drops every target. Why? Almost always the regex. Every regex is RE2 and fully anchored, so regex: "prod" matches only the value prod. Also, multiple source_labels are joined with ; before matching. Compare the raw value on the Service Discovery page against the regex.

How do you debug a missing target? Status, Service Discovery in the UI shows discovered targets with raw __meta_* labels, the relabeled labels, and dropped ones. The API gives targets?state=dropped. And promtool check service-discovery prometheus.yml <job> expands one job from the command line.

Scaling and availability

How do you make Prometheus highly available? Two identical servers scraping the same targets, each with a distinct replica external label, both sending to the same Alertmanager cluster. There is no Prometheus clustering. Their data is close but not identical, because scrape timings differ, so you query one or use a layer that deduplicates.

When do you use federation? To pull a small set of already-aggregated series, such as job: recording rules, from many servers into a global one. Not to copy everything. If you are federating raw series you want remote write.

What is remote write and what do you monitor on it? Streaming samples from the WAL to another system, such as Mimir, Thanos Receive, VictoriaMetrics or a managed cloud service, through a queue of shards. Monitor lag (the gap between the highest timestamp written locally and the highest sent), failed samples, pending samples, and whether shards have hit max_shards.

What is agent mode? --agent runs Prometheus as a lightweight collector that scrapes and remote-writes, with no local querying, rules or alerting and a short WAL. It is the edge-site pattern. In 3.x it is a real flag, not --enable-feature=agent.

Security

How do you secure Prometheus? The project's stance is that untrusted users can reach the endpoint, and only trusted users change configuration. The built-in web config file gives TLS, mutual TLS and bcrypt basic auth. There is no RBAC, so for users and teams you put a reverse proxy or an OAuth2 proxy in front. Keep the lifecycle and admin APIs off unless you control who can reach the port, and use the *_file variants for secrets.

Why is enabling --web.enable-admin-api dangerous? It exposes snapshot, delete_series and clean_tombstones, so anyone who can reach the port can delete your data. The lifecycle flag similarly lets anyone stop the server through /-/quit.

Scenario questions

1. A dashboard shows "no data" for a target you know is healthy. Walk me through it.Start at the source: curl the target's /metrics from the Prometheus host. Then the Targets page for the last error. Then up{job="x"}, because a missing series and a zero series are different problems. Then the query: run it in the expression browser and strip it down, checking label matchers (a le="1" that is now le="1.0", a regex that is anchored), and whether the range is long enough for rate. Last, check the dashboard's own time range and data source.
2. Alerts stopped arriving and nobody noticed. How do you prevent a repeat?Add alerts on the pipeline itself: prometheus_notifications_errors_total and prometheus_notifications_dropped_total, evaluated from somewhere other than the broken path. Add a Watchdog rule that always fires, routed to an external dead-man's-switch that pages when notifications stop arriving. Add prometheus_config_last_reload_successful == 0, because a rejected config reload also fails silently.
3. After a 2.x to 3.x upgrade some dashboards show nothing. What do you check?The 3.0 changes in order: le and quantile label values now in float form, regex . matching newlines, left-open range semantics affecting subqueries, removed functions and renamed ones, and strict Content-Type on scrapes. Also logs have a new format, so any log-based alert changed.
4. Memory keeps climbing and the server is OOM-killed weekly.Compare prometheus_tsdb_head_series over time. Growth that correlates with deploys points at churn, and steady growth at new labels. Run promtool tsdb analyze on a copy or the status endpoint to find the top metrics and labels. Fix the source, drop at ingestion, add sample limits, and consider moving heavy jobs to their own server.
5. A rule group is slow and alerts are late.Check prometheus_rule_group_last_duration_seconds against the group interval and prometheus_rule_group_iterations_missed_total. Identify expensive expressions, particularly subqueries, and turn the shared inner expression into a recording rule. Split the group so slow rules do not block quick alerts, since rules in a group run in sequence.
6. A target fails with context deadline exceeded. What are the causes?Network path or firewall, a target that genuinely takes longer than scrape_timeout, or a very large exposition. Compare scrape_duration_seconds with the timeout. If the duration is near the timeout the target is slow, and if there is none the connection never completed. Raising the timeout is a last resort, because it must stay at or under the interval.
7. Two replicas of the same service produce a "duplicate sample" warning after you add a relabel rule.The rule removed a label that distinguished them, so two series now have identical label sets, and the server logs Error on ingesting samples with different value but same timestamp. Keep a distinguishing label, or aggregate intentionally.
8. Remote write to the managed backend is falling behind.Check the lag expression and whether prometheus_remote_storage_shards equals its maximum. A 429 means tenant rate limits, so reduce series with write_relabel_configs or raise the limit. Raise max_shards only if the backend can take it, otherwise you queue more and lose data when the WAL ages out.
9. You need to delete a metric that leaked personal data.With the admin API enabled for the duration, POST /api/v1/admin/tsdb/delete_series?match[]=... with a time range, then clean_tombstones. Stop the source from emitting it first, or it will be recreated. Remember it may already be in remote storage and in backups, so the deletion is a process, not a call.

Trade-off questions

1. Thanos, Mimir or a managed service for long-term storage?It is a trade between operating effort and control. Thanos keeps Prometheus local and adds object storage and a global query layer, good when you already run Prometheus widely. Mimir and similar remote-write backends centralise ingestion and give real multi-tenancy, at the cost of running a distributed system. A managed service removes the operating burden for a per-series cost, and on any of them cardinality control before remote write decides the bill. Data residency may also decide it, for Gulf or Egyptian employers with in-region requirements.
2. Recording rule or compute at query time?Record when the expression is shared, expensive or used by an alert. Compute at query time for ad-hoc exploration. Every recorded series costs storage and evaluation time forever, and loses any label you aggregated away.
3. Short or long scrape interval?Short gives resolution and faster alerts and multiplies samples. Fifteen to thirty seconds is typical, with low-value jobs at a minute. Staying under the lookback of five minutes, and in practice under two minutes, keeps graphs continuous.
4. Pull or push for a batch job?For a job that lives shorter than a scrape interval, the Pushgateway with honor_labels: true, accepting that it never forgets series unless you delete them. For long-running services, pull, because it gives you up for free.
5. Classic or native histograms?Native is cheaper, more precise and needs one series, and is stable since 3.9, but requires ingestion to be enabled (scrape_native_histograms: true), the protobuf format, and downstream support. Classic works everywhere. Migrating means changing the queries too, as there is no le.
6. One big server or several small ones?One server is simple, with one place to query. Several give isolation and independent failure domains, and cost you a global view, which needs federation or a backend. Shard by team or job first, and by hash only when a single job is too big.
7. Plain Prometheus or the Operator on Kubernetes?The Operator makes monitors and rules per-team Kubernetes objects and avoids editing one central file. It adds CRDs to understand and selectors that can silently mismatch. For a small cluster with one owner, a plain config is fine. For many teams, use the Operator.
8. LTS or the latest minor?A release every six weeks means a team that cannot read a changelog that often belongs on LTS (3.13.x, supported until 2027-07-31). A team that wants new features and will keep up takes the latest. Both must read the [CHANGE] lines.

What a strong answer covers that a weak one misses

Common interview questions

1. What is Prometheus and what is it not good for?A pull-based monitoring and alerting system with a local time-series database and PromQL. It is not for data that must be 100% accurate per request, such as billing, and it is not for logs or traces.
2. Why does Prometheus pull?The server controls the cadence and knows what should exist, so a missing target is visible as up 0, and you can curl a target to see exactly what it serves.
3. What do job and instance mean?An instance is one scrape endpoint, host:port. A job is a set of instances with the same purpose. Prometheus attaches both labels automatically.
4. What is a stale marker?A special value written when a series disappears, so queries stop returning it at once rather than for the five-minute lookback.
5. What does the bool modifier do?It makes a comparison return 0 or 1 for each series instead of filtering, which is useful when the result feeds arithmetic, such as avg(up == bool 0).
6. How would you alert that a job has disappeared?absent(up{job="api"}), because up == 0 cannot catch a target that no longer exists to have a series.
7. What is the rule for naming recording rules?level:metric:operations, for example job:http_requests:rate5m, with colons reserved for rules.
8. How do you back up Prometheus?Enable the admin API, call POST /api/v1/admin/tsdb/snapshot, which creates hard links under the data directory's snapshots folder, and copy that. Or use volume snapshots. Blocks are immutable so incremental copying works.
9. What is an external label for?Identifying this server in anything that leaves it: federation, remote write and alerts. They are not stored on local series. Only there do environment variables expand, as ${var}.
10. What happens when a config reload fails?The new config is not applied, the old one keeps running, an error is logged, and prometheus_config_last_reload_successful becomes 0. Alert on it.
11. How are secrets handled in a scrape config?Secret-typed fields are redacted in logs and in the config endpoints, and the file variants (password_file, credentials_file) let you mount them. Environment variables are not expanded in general fields.
12. How do you monitor Prometheus?It scrapes itself, and you alert on reload failures, rule evaluation failures, missed iterations, notification errors, sample-limit breaches, compaction failures and disk forecasts. A second server or a dead-man's-switch covers the case where the server itself is gone.
13. What is the Pushgateway's main pitfall?It never forgets. A pushed series stays until deleted, so a job that stopped running looks alive, with its last value.
14. How do you expose GPU metrics?NVIDIA's DCGM exporter on each GPU node, scraped like any exporter, with the pairing of utilisation and queue depth for alerts.

Final self-test

• Say what up proves and what it does not, and name two more synthetic series.
• Explain why sample_limit fails the whole scrape.
• State the rate range rule and why irate is wrong in an alert.
• Explain why you sum rates rather than rate a sum.
• Write a correct histogram_quantile and say what limits its accuracy.
• List four 3.0 changes that break old queries.
• Describe head, WAL, blocks and compaction, and what each predicts about restarts and disk.
• Give the disk formula and the compaction headroom rule.
• Say where retention lives in 3.x and which flags are deprecated.
• Give the four relabeling stages and the anchoring rule.
• Walk the alert states, and say what keep_firing_for does.
• Explain how an HA pair avoids double-paging.
• Say when to use federation, remote write and agent mode.
• Name three endpoints or flags that should stay off by default and why.
• State what a failed reload leaves running and which metric reveals it.
• Give the order of a debugging session: source, scrape, ingestion, query.

Part two of three. At this level your targets are scraped and your dashboards draw, so the problems change character. They are no longer "why is my target down" but "why did the server run out of memory on Monday", "why did this alert fire at 3 a.m. and stay silent during the outage", and "why does the ratio query return nothing". Start with the error table, then the practices in order of leverage, the practice cards, and the recipes you will reach for in operations.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In order of leverage. The first three prevent most of the incidents you would otherwise have.

Practice cards

1. Watch a series go staleAdd a job pointing at a dead port, then delete it and reload. Confirm its series stop returning at once, not after five minutes, and explain why.
2. Breach a sample limitSet sample_limit: 5 on the self-scrape job, reload, read the error on the Targets page, and watch up go to 0 for the whole job. Then remove it.
3. Compare rate and irateGraph both over prometheus_http_requests_total while you refresh the UI. State which you would put in an alert and why.
4. Join an info metricUse group_left (version) to add the version label from prometheus_build_info onto another series.
5. Estimate your diskRead the ingest rate and retention, apply the formula with two bytes per sample, and compare with the real size of the data directory.
6. Find the biggest metricUse the TSDB Status page and the status/tsdb endpoint to find the metric name with the most series, then the label with the most values on it.
7. Drop a labelAdd a labeldrop in metric_relabel_configs, reload, and compare scrape_samples_scraped with scrape_samples_post_metric_relabeling.
8. Unit-test an alert both waysWrite a test where it fires after for and a second where it must stay inactive. Break the expression and watch one of them fail.
9. Break and recover the configIntroduce a bad indent with auto-reload on, read the log and the gauge, then fix it and confirm the gauge returns to 1.
10. Kill AlertmanagerStop it for a minute, watch prometheus_notifications_errors_total rise, and check that your Watchdog delivery stops arriving.
11. Test a routeUse amtool config routes test with labels that should reach the pager and labels that should not.
12. Take and restore a snapshotWith the admin API enabled in a test environment, snapshot, copy the snapshot directory to a fresh data path, and start a second server on it.

Reading a server's health in five queries

When you inherit a server, run these before anything else. Together they tell you whether it is healthy, how big it is and how fast it is growing.

# 1. how many series are live, and how fast is it growing?
prometheus_tsdb_head_series
deriv(prometheus_tsdb_head_series[1h])

# 2. how many samples per second is it ingesting?
rate(prometheus_tsdb_head_samples_appended_total[5m])

# 3. are scrapes failing or being limited?
count by (job) (up == 0)
increase(prometheus_target_scrapes_exceeded_sample_limit_total[1h])

# 4. are rules keeping up?
increase(prometheus_rule_group_iterations_missed_total[1h])
increase(prometheus_rule_evaluation_failures_total[1h])

# 5. is the alert path alive?
increase(prometheus_notifications_errors_total[1h])
increase(prometheus_notifications_dropped_total[1h])

Interpret them as a set. A rising series count with a stable ingest rate is churn, typically from pod names in labels. A rising ingest rate with stable series means a scrape interval was shortened or a histogram gained buckets. Missed iterations with a healthy series count mean an expensive rule, not a big server.

Shrinking a server, in order of leverage

The same ladder as with container images: start at the top and measure after each step, instead of doing everything blind.

1. Remove the label at the source. The only fix that also relieves the application.
2. Drop whole metric families nobody queries with metric_relabel_configs and action: drop. Go runtime and process metrics from many exporters are a common first candidate.
3. Trim histogram buckets, or move to native histograms. Buckets times label combinations is where a modest metric becomes fifty thousand series.
4. Lengthen the interval for low-value jobs. It reduces samples, not active series.
5. Turn dashboards into recording rules, so repeated heavy queries run once.
6. Split by job into a second server once one job dominates, before you reach for a distributed backend.

scrape_configs:
  - job_name: "node"
    scrape_interval: 60s          # host metrics rarely need 15s
    static_configs:
      - targets: ["node-1:9100", "node-2:9100"]
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "node_(scrape_collector|textfile)_.*|go_.*"
        action: drop

Writing alerts people do not mute

An alert that fires often and means little trains the team to ignore the pager. These habits keep the signal.

• One condition, one owner. Every alert carries a team label that routes to that team, so nothing lands in a channel nobody reads.
• Choose for from the cost of a false alarm. A minute for a total outage, ten for a ratio that blips. Never zero unless a single evaluation is genuinely enough.
• Give the window a safety margin. A rate(x[5m]) alert at a 30-second scrape is fine, and a [1m] one is not.
• Guard ratios against low traffic. An error ratio of 100% from one failed request at 3 a.m. is noise. Combine it with a minimum rate.

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
              /
            sum by (job) (rate(http_requests_total[5m]))
          ) > 0.05
          and
          sum by (job) (rate(http_requests_total[5m])) > 1
        for: 10m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "{{ $labels.job }} error ratio is {{ $value | humanizePercentage }}"
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

• Page on the burn, ticket on the trend. Route severity page to the pager, and severity ticket to a queue that is read during working hours. A disk forecast that fills in four days is a ticket.
• Route the noisy ones away from humans at night. Inhibition rules let a "cluster down" alert mute the hundred alerts it causes.
• Review the alerts that fired last month. An alert that paged five times and was never acted on is either wrong or should not page.

Operational recipes

Change the log level without a restart

runtime:
  log_level: debug     # back to info after the incident

kill -HUP $(pidof prometheus)
curl -s localhost:9090/api/v1/status/flags | jq '.data["log.level"]'

In 3.15 runtime.log_level is the supported place, and it is reloadable. The --log.level flag still works but is deprecated.

Reload, three ways

kill -HUP $(pidof prometheus)                 # always works
curl -X POST localhost:9090/-/reload          # needs --web.enable-lifecycle
prometheus --config.file=prometheus.yml --config.auto-reload   # polls every 30s

Whichever you use, confirm afterwards with prometheus_config_last_reload_successful and the Completed loading of configuration file log line.

Find what is eating the server

# the top series by metric name, from the running server
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.seriesCountByMetricName[:10]'

# the top labels by value count
curl -s localhost:9090/api/v1/status/tsdb \
  | jq '.data.labelValueCountByLabelName[:10]'

# offline, against a data directory or a snapshot
promtool tsdb analyze /var/lib/prometheus

Run the offline form against a copy on a busy production host, because it reads a lot.

Find the slow queries

global:
  query_log_file: /var/log/prometheus/queries.log

# slowest queries, assuming one JSON object per line with a stats block
jq -r 'select(.stats.timings.evalTotalTime > 1) | [.stats.timings.evalTotalTime, .params.query] | @tsv' \
  /var/log/prometheus/queries.log | sort -rn | head

The expression in the jq line depends on the log's JSON layout, so look at one line first and adjust the path. The aim is to find the dashboard panel that is crushing the server and turn it into a recording rule.

Take an online backup

# needs --web.enable-admin-api
curl -X POST localhost:9090/api/v1/admin/tsdb/snapshot
# {"status":"success","data":{"name":"20261001T120000Z-1a2b3c4d5e6f7a8b"}}

ls /var/lib/prometheus/snapshots/
rsync -a /var/lib/prometheus/snapshots/20261001T120000Z-1a2b3c4d5e6f7a8b/ backup-host:/backups/prom/

A snapshot is made of hard links, so it is cheap and quick, and you copy it off the machine. Pass skip_head=true to leave out the in-memory head if you can live without the last two hours. Turn the admin API back off afterwards, or keep the port private.

Delete a leaked series

curl -X POST -g \
  'localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~"leaky_metric.*"}'
curl -X POST localhost:9090/api/v1/admin/tsdb/clean_tombstones

Both return 204 on success. Stop the source emitting the series first, or it is recreated on the next scrape, and remember that remote storage and backups are separate copies.

Backfill history from recording rules

When you add a new recording rule, dashboards that use it have no history. You can generate it from existing data:

promtool tsdb create-blocks-from rules \
  --start 2026-09-01T00:00:00Z --end 2026-09-30T00:00:00Z \
  --url http://localhost:9090 rules/api.yml

The command writes blocks to an output directory, which you then move into the data directory. Do not backfill the most recent three hours, since those belong to the head.

Test a scrape by hand

curl -s http://payments-1:8080/metrics | promtool check metrics
curl -sI http://payments-1:8080/metrics | grep -i '^content-type'
promtool query instant http://localhost:9090 'up{job="payments-api"}'

A CI script worth copying

#!/usr/bin/env bash
set -euo pipefail

promtool check config --lint=all --lint-fatal prometheus.yml
promtool check rules rules.d/*.yml
promtool test rules tests/*.yml
promtool check web-config web.yml
amtool check-config alertmanager.yml

Kubernetes specifics

• A ServiceMonitor does nothing. Two silent causes: the labels do not match the Prometheus resource's serviceMonitorSelector (kube-prometheus-stack selects on its own release label by default), or port names a Service port that does not exist. The value is the port name.
• Targets appear, then fail with 403. The service account lacks RBAC for discovery. It needs get, list and watch on nodes, services, endpoints, endpoint slices, pods and ingresses.
• Chart and server versions differ. The kube-prometheus-stack chart pins its own Prometheus image, so upgrading the chart can move the server by several minor versions. Read the chart's notes and the server's changelog together.
• Pod names in labels cause churn. Every rollout creates a new set of series. Use the owner (deployment, statefulset) for long-lived aggregation, and keep pod-level series in recording rules for a short window.
• Autoscaling on a query. A KEDA or HPA metric that returns no data can mean "do nothing" or "scale to minimum". Decide which with or vector(0) and test it.

# restarts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3

# CPU throttling: fraction of periods throttled
sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
  /
sum by (namespace, pod) (rate(container_cpu_cfs_periods_total[5m]))

Docker specifics

• Extra arguments replace the default command. The image's CMD supplies --config.file and --storage.tsdb.path. Pass any argument and you must repeat both.
• Data vanishes on restart. No volume on /prometheus. Mount a named volume.
• Changed to the distroless image and got permission denied. It runs as UID 65532, and the default busybox image runs as nobody. Fix the volume ownership with docker run --rm -v prometheus-data:/prometheus alpine chown -R 65532:65532 /prometheus.
• Config edits do not show up. A single-file bind mount can keep pointing at the old inode after an editor replaces the file. Mount the directory instead, then reload.

docker run -d --name prometheus -p 9090:9090 \
  -v "$PWD/config:/etc/prometheus" \
  -v prometheus-data:/prometheus \
  prom/prometheus:v3.15.0 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --config.auto-reload

See the Docker guide for the mount and entrypoint mechanics, and the Kubernetes guide for the cluster side.

ML services

• Measure the right latencies. For generation endpoints, time to first token and total time are different series with different causes. Histogram both, and alert on percentiles.
• Bucket for the long tail. If generation can take thirty seconds, add buckets that reach that far, or the percentile lands in +Inf and means nothing.
• Pair GPU utilisation with queue depth. High utilisation with a short queue is healthy. Moderate utilisation with a growing queue is not.
• Take metric names from the target. They differ between frameworks and versions, so read the /metrics output.
• Never label by prompt or customer. It is unbounded and probably personal data.

For putting model and infrastructure signals on one screen, continue to the Grafana guide, and for drift and quality signals see the Evidently guide.

Habits worth adopting now

1. Read prometheus_tsdb_head_series on every server you own, at least weekly.
2. Put sample_limit on every job, and alert on breaches.
3. Keep configuration in Git, and run the CI script before any change merges.
4. Write the test before the alert, and include the must-not-fire case.
5. Give every alert a team label and a runbook_url.
6. Run a Watchdog alert to something outside the server.
7. Alert on prometheus_config_last_reload_successful == 0.
8. Use the config-file forms: storage.tsdb.retention, runtime.log_level, scrape_native_histograms.
9. Before you upgrade, read the [CHANGE] lines, and check downstream readers before enabling XOR2 encoding.
10. When a query looks wrong, go in order: source, scrape, ingestion, query.

A short checklist

1. Every job has limitsA sample_limit, and ideally a label_limit and body_size_limit.
2. No unbounded labelsNo user IDs, emails, request IDs or full URLs, checked with the TSDB status page.
3. Rules are testedpromtool test rules covers fire, pending and must-not-fire.
4. Alerts are routableA severity and team label, a runbook link, and amtool route tests.
5. The pipeline is watchedReload failures, notification errors, missed iterations, and a Watchdog to an external service.
6. HA is realDistinct replica labels, dropped before Alertmanager, every Alertmanager listed.
7. Storage is plannedDisk sized by the formula with compaction headroom, on local disk and not NFS.
8. The admin surface is closedLifecycle and admin APIs off or private, TLS and a proxy in front, secrets through *_file fields.
9. Upgrades are routineOn LTS or on the latest minor with a changelog habit, never older than 3.11.3.

This is part three of three. It takes every topic from Beginner and Mid one level further, because each one now carries a scale, security, or ownership dimension, and it adds the work you take on when Prometheus stops being a tool you use and becomes a platform other teams depend on.

Up to now the questions were about getting the right number on the screen. From here they are different. What happens to every team's alerts when the server is OOM-killed at 3 a.m.? Which single label on which single metric will double the memory bill next quarter? Who is allowed to read another team's metrics, and what actually stops them? When the data has to outlive one disk, which component do you add, and what does it cost you? A Prometheus that nobody owns works beautifully until the week it does not, and that week tends to coincide with an incident in something else, which is precisely when you need your monitoring most.

The version this guide targets is Prometheus 3.15 (released 2026-09-24), with the 3.13 line as the long-term-support choice for conservative estates; 3.13 receives fixes until 2027-07-31. There is no supported 2.x line any more, so nothing here teaches it.

Where this picks up

I am starting with the architecture, because every decision after it depends on knowing exactly which component owns which piece of state.

Architecture and failure modes

A Prometheus server is a single process that does five jobs at once: it discovers targets, scrapes them, appends the samples to a local time-series database, evaluates rules against that database, and answers queries. There is no cluster and no consensus, which is the design's great strength and also the source of nearly every limit you will hit. Reliability comes from each server being standalone. Scale comes from running more servers, never from making one server distributed.

It helps to hold a single picture of how data moves. A scrape returns text or protobuf. After target relabeling has produced the target's labels and metric relabeling has filtered the samples, each sample is appended to the head, which is the in-memory block covering the most recent data. Every append is also written to the write-ahead log (WAL) under wal/, in 128 MB segments, so that a crash can be replayed. Roughly every two hours the head is cut into an immutable on-disk block: a directory holding chunks/, an index, a meta.json, and tombstones. Older small blocks are later compacted into larger ones, up to the smaller of 10 percent of the retention period or 31 days. The head's recent chunks are memory-mapped from chunks_head/, and an out-of-order WAL (wbl/) exists if you enabled an out-of-order window.

That description predicts the failure modes better than any checklist does.

Memory exhaustion is the most common fatal failure. The head holds every active series, meaning any series that received a sample in the current head window, plus the index that lets you look them up by label. Memory therefore scales with active series count and with churn (series being created and abandoned, as when pods are replaced). The queries you run add transient memory on top. Practitioners often quote a few kilobytes per active series as a planning figure, and a well-provisioned single server commonly handles a few million series; these are community rules of thumb rather than official limits, so measure your own with prometheus_tsdb_head_series and process_resident_memory_bytes before planning capacity around them.

Slow restarts are the failure people forget until it hurts. After a crash or a normal restart, Prometheus replays the WAL to rebuild the head. On a server with a large head, that replay can take minutes, during which the server is not ready: /-/ready returns 503 while /-/healthy still returns 200. You can watch progress at /api/v1/status/walreplay. The flags memory-snapshot-on-shutdown and fast-startup exist to shorten this, and both are feature flags, so test them before relying on them. The operational point is that an OOM loop is self-reinforcing: each restart replays the WAL, which takes memory, which triggers another kill.

Disk failure loses data, because nothing replicates it. Local storage is not clustered and not durable across the loss of the volume. It also must be a real local or block-storage filesystem: NFS and EFS are not supported and have produced corruption. Your durability story is either a second replica, a remote-write copy, or backups, and you need to choose one deliberately.

Rule evaluation and queries compete with ingestion for the same CPU and memory. A dashboard that scans a month of a high-cardinality metric can starve the scrape loops. The server protects itself with --query.max-concurrency (default 20), --query.timeout (default 2m), and --query.max-samples (default 50 million), and those defaults are a judgement about a typical server rather than about yours.

The TSDB under pressure

The previous section named the moving parts. Here is how each behaves when you push on it, because the retention and backup decisions you make as an owner are all consequences of this design.

Retention is now configuration, not a flag. Since 3.8 the settings live in prometheus.yml under storage.tsdb.retention, and they are reloadable, so changing them no longer needs a restart. The old --storage.tsdb.retention.time and --storage.tsdb.retention.size flags still work but are deprecated, and when both exist the config file wins. If you set neither, the default is 15 days. If you set both time and size, whichever limit triggers first removes the oldest blocks.

storage:
  tsdb:
    retention:
      time: 30d
      size: 500GB          # binary units; the size cap counts WAL and head chunks
    out_of_order_time_window: 0s

Two details bite here. Size-based retention counts the WAL and the head chunks, but it can only delete whole blocks, so if your head and WAL alone approach the cap, no deletion can satisfy it. And cleanup of expired blocks is not instantaneous; it can lag by up to two hours. Compaction also needs headroom, because the source blocks and the new merged block briefly coexist. Size your volume with that overlap in mind, not at the exact capacity you calculate.

The disk formula is worth knowing because leadership will ask for it: needed space is retention in seconds, multiplied by ingested samples per second, multiplied by bytes per sample, and Prometheus averages one to two bytes per sample on disk. Get the ingest rate from rate(prometheus_tsdb_head_samples_appended_total[5m]).

Chunk encoding is a one-way door. In 3.15, XOR2 float chunk encoding is stable and selected with storage.tsdb.chunk_encoding.floats: xor2. The old --enable-feature=xor2-encoding is deprecated. The default remains xor. The trap is compatibility: blocks written with XOR2 cannot be read by older Prometheus versions or by tools that do not support it, and the release notes name Thanos sidecar-uploaded blocks as an example. If you need to go back, you must delete the affected blocks. So before enabling it, inventory every reader of your blocks: backup tooling, long-term-storage uploaders, and any analysis scripts.

Out-of-order ingestion is opt-in for a reason. By default, a sample older than the newest one for its series is rejected, and you see warnings such as Error on ingesting out-of-order samples. That is usually a symptom, not the disease: duplicate producers writing the same series, clock skew, or a pusher with unreliable ordering. OTLP pushers and some remote-write senders legitimately deliver out of order, for which you set storage.tsdb.out_of_order_time_window (default 0s). A window costs extra memory and disk work, so turn it on for a known producer, not as a blanket fix that hides a duplicate-target bug.

Backups are cheap because blocks are immutable. With the admin API enabled (--web.enable-admin-api), a POST to /api/v1/admin/tsdb/snapshot creates a snapshot directory under <data-dir>/snapshots/ using hard links, which is nearly instant and nearly free in space. Copy that directory off the machine. Add skip_head=true if you only want completed blocks. Because the admin API also grants deletion, enable it only where you need it and protect the endpoint; a safer variation is a volume-level snapshot from your storage layer, which needs no API at all.

curl -s -X POST 'http://localhost:9090/api/v1/admin/tsdb/snapshot'
# {"status":"success","data":{"name":"20260930T101500Z-3f9c1a2b7d4e5f60"}}
ls /var/lib/prometheus/snapshots/

Deletions are lazy. POST /api/v1/admin/tsdb/delete_series (with match[], start, end) writes tombstones, and the space is only reclaimed when compaction runs or when you call clean_tombstones. This matters when you must remove data that should never have been recorded, for instance a label that leaked an identifier: deleting it is not instant, and it does not remove copies already sent via remote write.

Cardinality: the budget that runs out first

Cardinality is the number of distinct time series, and it is the variable that drives memory, disk, query time, and, with a managed backend, your invoice. Every unique combination of a metric name and a full label set is its own series. A metric with three labels that have 10, 50, and 200 possible values can reach 100,000 series from a single name, and the product grows faster than intuition suggests.

The rule every platform owner ends up enforcing is simple to state: never put an unbounded value in a label. User IDs, email addresses, request IDs, session tokens, full URLs with query strings, and container IDs that change on every deploy all belong in logs or traces, not in metrics. The data model itself tells you why: an empty label value is equivalent to no label, and every other value creates a new stream that Prometheus must keep in memory for as long as it is active.

There are two kinds of cardinality problem and they need different cures. A steady-state explosion means a metric is simply too wide (a histogram with 20 buckets, multiplied by 5 labels, multiplied by 400 pods). A churn explosion means series are created and abandoned quickly, typically because a label such as pod or a version string changes on every deploy. Churn is sneaky: prometheus_tsdb_head_series may look moderate while the rate of series creation is enormous, and each new series costs head memory and index work.

To find offenders you have four tools, and you should know all of them because each answers a slightly different question.

# 1. The running server's own view of its worst metrics and labels
curl -s localhost:9090/api/v1/status/tsdb | head -c 600

# 2. Offline analysis of a data directory: churn and label cardinality
promtool tsdb analyze /var/lib/prometheus

# 3. A PromQL probe, the top ten metric names by series count (expensive, run it off-peak)
#    topk(10, count by (__name__) ({__name__=~".+"}))

# 4. Live growth: how fast are new series appearing?
#    rate(prometheus_tsdb_head_series_created_total[5m])

The /api/v1/status/tsdb endpoint returns cardinality statistics, limited to 10,000 entries. The promtool tsdb analyze output is long but rewards reading: it lists the highest-cardinality labels and the label pairs that appear most often, which tells you whether the problem is one metric or one label spread across many.

Once you find a culprit, the cures, in rising order of cost to the owning team, are these. Drop the label or the whole metric in metric_relabel_configs at the scrape, which takes effect on the next reload. Replace a wide classic histogram with a native histogram, which is a single series with sparse exponential buckets and is stable since 3.9; ingestion must be switched on with scrape_native_histograms: true, and the scraper then negotiates the protobuf format. Aggregate with a recording rule and point dashboards at the aggregate. Lengthen the scrape interval for low-value jobs. Or, last, fix the instrumentation, which is the real fix but needs the owning team.

The protection you build in is limits, set per scrape job so one team's bug fails its own scrape and nobody else's.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  sample_limit: 0            # 0 means unlimited globally; override per job below
scrape_configs:
  - job_name: team-payments
    sample_limit: 50000      # the whole scrape fails (up=0) if exceeded
    label_limit: 30
    label_name_length_limit: 64
    label_value_length_limit: 256
    target_limit: 200
    body_size_limit: 20MB
    static_configs:
      - targets: ["payments-1:8080"]

Read the consequence of sample_limit carefully: exceeding it does not truncate the scrape, it fails the entire scrape and sets up to 0 with the error sample limit exceeded. That is deliberately loud. It protects the server, but it also means a cardinality bug in a service now looks like an outage of that service's metrics, and any alert on up fires. Make sure the team understands that a limit breach is a signal about their instrumentation, and alert on prometheus_target_scrapes_exceeded_sample_limit_total so the cause is obvious rather than mysterious.

Scaling out: which component breaks first

When one server is no longer enough, the useful question is not "how do we scale Prometheus" but "what is actually saturated". The answer differs, and so does the cure.

Sharding is the structural answer for scrape and memory pressure, and there are two ways to do it. Functional sharding splits by job: one server for infrastructure, one for the platform team's services, one per large tenant. It is simple to reason about, it aligns with team ownership, and it is what most estates do first. Hash sharding uses the hashmod relabel action to split the targets of a single job across several servers, so each scrapes a stable slice.

scrape_configs:
  - job_name: pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__address__]
        action: hashmod
        modulus: 3                 # three shards
        target_label: __tmp_shard  # __tmp_ prefix: dropped after relabeling
      - source_labels: [__tmp_shard]
        action: keep
        regex: "1"                 # this server keeps shard 1 of 0..2

Notice what sharding costs. A query that needs to aggregate across shards can no longer be answered by any single server, so you need a global query layer (federation for a few aggregated series, or a remote-write and long-term-storage backend for everything). Alerts that compare series held by different shards stop working. Sharding buys ingest capacity at the price of the global view, and you should say that out loud whenever someone proposes it.

Rule evaluation deserves its own attention because it is where platform owners are surprised. Rules within a group run sequentially at the group interval. If a group takes longer than its interval to evaluate, the next evaluation is skipped and prometheus_rule_group_iterations_missed_total increments, which means your alerts are being evaluated less often than you configured, silently. The concurrent-rule-eval feature flag allows independent rules to evaluate in parallel, bounded by --rules.max-concurrent-evals (default 4), but it only helps rules that do not depend on each other's output. Often the better fix is structural: move expensive aggregation into a recording rule in an earlier group, and keep alerting rules cheap.

The query path is the part the platform owner influences least, because you do not control what people type into a dashboard. The defense is layered: a recording rule for every expensive panel, --query.max-samples and --query.timeout tuned to a number you can justify, and education about range selectors. Also set global.query_log_file so you can see who ran the query that took the server down, not guess.

Agent mode deserves a mention as a scaling tool rather than a storage one. Run with --agent and Prometheus scrapes and remote-writes only: no local querying, no rules, no alerting. It uses --storage.agent.path and trims its local WAL on a short horizon (--storage.agent.retention.min-time default 5m, --storage.agent.retention.max-time default 4h). That makes it the right shape for an edge collector in each cluster feeding a central backend, because the collector needs little memory beyond the head and has no query load. The price is that an agent has no local alerting, so alert evaluation must happen in the backend.

High availability, and what it does not give you

The standard Prometheus HA design is two identical servers scraping the same targets. Each has a distinct external label, typically replica: a and replica: b, and both send alerts to the same Alertmanager cluster. If one replica dies, the other keeps evaluating rules and alerting.

global:
  external_labels:
    cluster: prod-me-central
    replica: a            # 'b' on the second server
alerting:
  alert_relabel_configs:
    - action: labeldrop
      regex: replica      # so both replicas send identical alerts
  alertmanagers:
    - api_version: v2
      static_configs:
        - targets: ["am-0:9093", "am-1:9093", "am-2:9093"]

The labeldrop on replica is the detail that makes the design work. alert_relabel_configs runs after external labels are applied, so dropping the replica label makes the two servers' alerts byte-identical in label terms and Alertmanager deduplicates them into one notification. Without it you get two notifications per alert, and people learn to ignore them.

It is important to be precise about what this buys you, because HA is the most over-sold feature in this area.

• It protects alert delivery, not data. The two replicas each keep their own independent copy of the data, scraped at slightly different moments. They will not be byte-identical: the timestamps differ, and a scrape that failed on one may have succeeded on the other.
• A query sent to a replica sees only that replica's data. After a replica restarts, it has a gap in its history. Grafana pointed at a load balancer in front of both will show a graph that flips between two slightly different histories, including the gap. Point dashboards at one replica, or use a query layer that deduplicates on the replica label.
• It doubles your cost in scrape load, storage, and any remote-write volume unless you handle deduplication downstream.
• It does not protect against a bad rule, a bad config, or a cardinality explosion, because both replicas receive the same configuration and the same targets, so both fail together.

For Alertmanager, the HA model is different: run several instances that form a gossip cluster (--cluster.peer, cluster port 9094), and configure Prometheus with every Alertmanager instance in the alertmanagers list rather than a load balancer in front. Each Prometheus sends each alert to every Alertmanager, and they deduplicate among themselves via gossip. A load balancer would send an alert to one instance only, defeating the redundancy.

There is a last failure that HA does not address at all: the server that dies silently. If Prometheus is the thing that detects outages, who detects Prometheus's own? The answer is a dead-man's-switch. You define an alert that always fires, conventionally called Watchdog, and route it to an external service that pages you if the alert stops arriving. Any external heartbeat monitor works; the point is that the check lives outside the system it watches.

groups:
  - name: meta
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Always firing; absence means the alerting pipeline is down"

Long-term storage and the global view

A single server with local disk answers "what is happening now" well and "what happened last year across all regions" badly. Two separate needs hide inside that sentence, and choosing well means separating them: retention (keeping data longer than one disk or one server's life) and global querying (one query that spans many servers). Different tools emphasize different halves.

There are three mechanisms in Prometheus itself, and then a set of systems that build on them.

Federation lets one Prometheus scrape selected series from another's /federate endpoint using match[] selectors. It is the oldest mechanism and the most misused. It is designed for a hierarchy in which a global server pulls a few pre-aggregated series (the job: recording rules) from many leaf servers. It is not designed for copying everything: a global server that federates all series from ten leaves must hold ten servers' worth of head memory, which defeats the purpose.

Remote write streams every sample (or a filtered subset) to an external system over HTTP as they are scraped. This is the mechanism that connects Prometheus to Thanos Receive, Grafana Mimir, Cortex, VictoriaMetrics, and the managed services from the major clouds and Grafana Cloud. Prometheus acts as a collector and the backend provides durability, long retention, and horizontal query scale.

Remote read lets Prometheus query an external store through POST /api/v1/read, but it has fallen out of favour for new designs, because a query layer that speaks PromQL directly over the backend is simpler than proxying through a Prometheus.

The two architectural families are worth contrasting, because the decision between them is the question an interviewer or a leadership review will ask.

I describe the named backends by category rather than by tuning advice, because their behaviour, sizing, and pricing are documented by their own projects and change faster than this guide. What is stable, and sourced from Prometheus itself, is how you operate the sending side.

remote_write:
  - url: https://mimir.example.com/api/v1/push
    name: long-term
    remote_timeout: 30s
    basic_auth:
      username: collector
      password_file: /etc/secrets/remote-write-password
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "expensive_.*"
        action: drop               # do not pay to store what nobody queries
    queue_config:
      capacity: 10000
      min_shards: 1
      max_shards: 50
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 5s
    metadata_config:
      send: true
      send_interval: 1m

Understanding the queue makes tuning rational. Prometheus reads from the WAL and feeds each remote endpoint through a set of shards, which are parallel senders. It starts at min_shards and scales toward max_shards when it is falling behind, sending batches of up to max_samples_per_send or whatever accumulates within batch_send_deadline. If a send fails with a recoverable error it retries with exponential backoff between min_backoff and max_backoff. A non-recoverable 4xx (the backend rejecting the data, for instance for exceeding a tenant limit) drops those samples, which is the dangerous case because it looks like a small amount of loss that nobody sees.

What you alert on is lag and loss, not merely "remote write is up":

• prometheus_remote_storage_highest_timestamp_in_seconds minus prometheus_remote_storage_queue_highest_sent_timestamp_seconds is the delay, in seconds, between the newest data scraped and the newest data sent.
• prometheus_remote_storage_samples_failed_total counts samples that failed to send.
• prometheus_remote_storage_samples_pending shows the backlog, and prometheus_remote_storage_shards shows how many senders are active, which is pinned at max_shards when you are saturated.

There is one more aspect of the WAL that a platform owner should understand: if the backend is unavailable longer than the WAL retains data, the unsent samples are gone. Prometheus keeps about two hours of WAL, so an outage that exceeds that is permanent loss for that stream, even though local TSDB data is fine. That is a reason to keep local retention at a useful size even when you run remote write, so the local server remains a valid fallback for recent data.

Two newer knobs matter for interoperability. Remote-Write 2.0 is selectable with protobuf_message: io.prometheus.write.v2.Request, and the receiving side must enable it through --web.remote-write-receiver.accepted-protobuf-messages. And send_native_histograms and send_exemplars default to false, so if you adopt native histograms you must also decide to forward them and confirm the backend can store them.

Finally, consider receiving as well as sending. --web.enable-remote-write-receiver makes a Prometheus accept pushes at /api/v1/write, and --web.enable-otlp-receiver accepts OpenTelemetry metrics at /api/v1/otlp/v1/metrics. Both turn a pull-based, scrape-controlled server into something that can be written to by any client with network access, which changes the trust model and the cardinality risk, because now the producer and not your scrape config decides what arrives. Use them deliberately and enable an out-of-order window for OTLP producers, which commonly deliver out of order.

The security and trust model

Prometheus's documented security position is blunter than most people expect, and you should be able to recite it. It is presumed that untrusted users have access to the Prometheus HTTP endpoint and to its logs, and only trusted users can change the command line, the configuration, and the rule files. In other words, the server authenticates nothing by default, authorizes nothing ever, and considers the configuration a privileged input.

That model has consequences you must design around, because the server has no concept of users or roles.

There is no authorization, so there is no row-level or tenant-level access control. Anyone who can reach the query API can read every series the server holds: every team's metrics, every label, including anything sensitive that leaked into one. The built-in web configuration (--web.config.file, still marked experimental in --help despite wide use) provides three things only: TLS, mutual TLS, and bcrypt basic authentication. For real authentication and authorization you put a reverse proxy in front, such as oauth2-proxy, nginx, or Envoy, and for label-level isolation you need something that enforces label matchers on every query.

tls_server_config:
  cert_file: /etc/prometheus/tls/server.crt
  key_file: /etc/prometheus/tls/server.key
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/prometheus/tls/clients-ca.crt
  min_version: TLS12
http_server_config:
  http2: true

promtool check web-config web.yml
prometheus --config.file=/etc/prometheus/prometheus.yml --web.config.file=/etc/prometheus/web.yml

The web config file is re-read on every request, so rotating certificates needs no restart. That is convenient, and it also means a bad file breaks requests immediately, so validate with promtool check web-config in CI before it ships.

Two endpoint groups are dangerous by design, and both are off by default. --web.enable-admin-api exposes snapshot, delete_series, and clean_tombstones; --web.enable-lifecycle exposes POST /-/reload and /-/quit. Anyone who can reach those endpoints can delete your data or stop the server. If you need lifecycle reload for a GitOps flow, the safer patterns are --config.auto-reload (stable since 3.12, checking every --config.auto-reload-interval, default 30s) or sending SIGHUP, neither of which opens an HTTP control path. Enable the admin API only on a server whose network is tightly restricted, and only while you need it.

The configuration is a secret-bearing file. Fields typed as secrets, such as password, credentials, and client_secret, are redacted in logs and in the output of /api/v1/status/config and /-/config. Two habits follow. Prefer the file variants (password_file, credentials_file, client_secret_file), which you mount from a Kubernetes Secret or from a secret manager, rather than inlining values in a config that is committed to Git. And note that environment variables are not expanded in general configuration fields; only external_labels supports ${var} expansion. Do not assume ${TOKEN} will work elsewhere, and do not commit a config with a literal token because "it is redacted in the UI". Redaction protects the display, not the repository.

A reminder of why keeping up with patch releases matters: 2026 has had security fixes affecting the UI (a stored XSS, CVE-2026-40179), the exposure of a client secret for one cloud's service discovery through the config endpoint (CVE-2026-42151), and a remote-write decoding limit. The fact sheet for this guide recommends treating anything older than 3.11.3 as needing an upgrade. The general lesson is that the config endpoint is part of your attack surface, which is one more reason the HTTP port must not face an untrusted network.

Outbound trust matters too. As of 3.13, HTTP clients no longer forward credentials (the Authorization header, basic auth, bearer tokens, OAuth2 tokens, and configured headers) when following a redirect to a different host. That affects scraping, remote read and write, alerting, and service discovery. It closes a real credential-leak path, and it can surface as a sudden 401 after an upgrade if a target or backend relied on a cross-host redirect. The fix is to point the config at the final host, not to work around the protection.

Exporters and the neighbours have their own trust problems. Exporters that take parameters, such as blackbox and SNMP, can be made to probe arbitrary endpoints by anyone who can call them, which is a server-side-request-forgery shape. The Pushgateway and Alertmanager let anyone with network access modify data or create silences, and Alertmanager serves Access-Control-Allow-Origin: *. Put each behind the same proxy discipline as Prometheus itself. Denial of service is explicitly not treated as a security vulnerability by the project, which means your resource limits are your own responsibility and not something a patch will grant.

For containers, prefer the distroless image variant, which runs as UID and GID 65532 and has no VOLUME declaration. The default image is busybox-based and runs as nobody. Moving an existing named volume between the two needs a one-time ownership change:

docker run --rm -v prometheus-data:/prometheus alpine chown -R 65532:65532 /prometheus
docker run -d --name prometheus -p 9090:9090 \
  -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  -v prometheus-data:/prometheus \
  prom/prometheus:v3.15.0-distroless \
  --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus

Note that passing any arguments replaces the image's default command, which is why --config.file and --storage.tsdb.path are repeated. Forgetting that produces a server running on a default path and a confusing "data disappeared" report later. If you run on a Gulf or Egyptian employer's infrastructure with data-residency obligations, this is also the section to revisit: a remote-write target in another region moves your metrics, and labels often carry more identifying detail than teams realize.

Multi-tenancy

Prometheus has no multi-tenancy. No tenant concept, no per-tenant limits, no per-tenant retention, no per-tenant queries. Everything you hear described as "multi-tenant Prometheus" is one of four patterns, each with a different price, and a platform owner's job is to pick one deliberately and be honest with tenants about what it does and does not isolate.

In a Kubernetes estate the practical blend is often this: one platform-owned collector per cluster remote-writes to a multi-tenant backend, each team's workload gets a tenant identity through the ServiceMonitor or scrape configuration that selects it, and the platform enforces per-tenant series limits at the backend. The Prometheus Operator's CRDs (Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager, ScrapeConfig) let each team own its scrape definitions and rules as Kubernetes objects in their own namespace, which is also the natural place to attach quotas and admission policy. Treat chart versions, such as those of kube-prometheus-stack from prometheus-community/helm-charts, as separate from server versions and pin both.

Whichever you choose, external labels are the tenant stamp. A team or tenant label applied consistently at the scrape (through relabeling from namespace metadata) is what makes the enforcement, the billing showback, and the alert routing possible at all. Decide the label scheme first, document it, and make it a required field in your onboarding template, because retrofitting it across a hundred services is far harder than starting with it.

Cost attribution follows the same thread. A cardinality report broken down by the tenant label, run weekly, tells each team what their instrumentation costs, and it is a far more effective lever than a central team pleading in a channel. The mechanics are as simple as count by (team) ({__name__=~".+"}) on a server you can afford to run it on, or the equivalent breakdown your backend exposes.

Alerting as a system you operate

At Mid you wrote alerts. At this level the question is whether the alerting system is trustworthy, meaning: does a real problem reliably turn into a notification a human sees, and does a non-problem reliably not. There are three layers to reason about, and each can fail silently.

Layer one is evaluation. An alerting rule moves through inactive, pending (the expression is true but the for duration has not elapsed), and firing. keep_firing_for holds an alert firing for a period after the condition clears, which damps flapping. On restart, Prometheus restores the for state from its own history, governed by --rules.alert.for-outage-tolerance (default 1h) and --rules.alert.for-grace-period (default 10m), so an alert that was pending before a restart does not always restart its clock from zero. Evaluation failures and overruns are visible only through metrics: prometheus_rule_evaluation_failures_total and prometheus_rule_group_iterations_missed_total. A rule group with a limit set also has a sharp edge: if the number of series or alerts produced exceeds the limit, all of that rule's series are discarded and its alerts cleared, which can turn a real alert off exactly when the condition is worst.

Layer two is delivery from Prometheus to Alertmanager. Alerts are queued (--alertmanager.notification-queue-capacity, default 10000) and sent in batches (--alertmanager.notification-batch-size, default 256). If Alertmanager is unreachable, or the queue overflows, alerts are dropped and the log reads Error sending alerts. The queue is drained on shutdown by default, and the --rules.alert.resend-delay (default 1m) sets how often firing alerts are re-sent. The metrics to watch are prometheus_notifications_dropped_total and prometheus_notifications_errors_total. Note also that only api_version: v2 is supported in 3.x, so old configs that say v1 will fail.

Layer three is Alertmanager itself: grouping, routing, inhibition, silences, and the notification integrations. It is a separate binary with its own state, and several of its properties are worth a platform owner's attention. Inhibition rules suppress notifications for alerts that are explained by a higher-level alert; a badly written inhibition rule can hide a real problem, so inhibition deserves the same review discipline as a firewall rule. Silences are state held in Alertmanager, and anyone with access can create one; treat the Alertmanager UI as a privileged surface. The notification provider, whether email, Slack, PagerDuty, a webhook, or Microsoft Teams, is a dependency that can be down, rate-limited, or silently rejecting payloads. The discipline that catches this is an end-to-end test: the always-firing Watchdog alert routed through the same notification path as your real alerts, so a break anywhere in the chain is noticed.

The worst thing a platform owner can do to alerting is let the volume grow until people stop trusting it. The habits that prevent this are small. Every alert carries a severity label and a runbook link in its annotations. Every alert pages or it does not: a notification that nobody is expected to act on is noise to be deleted. Alert on symptoms users feel more than on causes, and keep cause-level alerts as tickets rather than pages. And review alert firing history monthly: alerts that fired often and were never acted on are candidates for deletion or retuning.

The meta-monitoring set, the alerts on Prometheus itself, is one you should make mandatory for every server you own, and ideally ship as a rule file that is part of your platform distribution:

groups:
  - name: prometheus-self
    rules:
      - alert: PrometheusTargetDown
        expr: up{job="prometheus"} == 0
        for: 5m
        labels: {severity: page}
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m
        labels: {severity: page}
      - alert: PrometheusRuleEvaluationFailing
        expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0
        for: 10m
        labels: {severity: page}
      - alert: PrometheusMissedRuleEvaluations
        expr: increase(prometheus_rule_group_iterations_missed_total[10m]) > 0
        for: 10m
        labels: {severity: ticket}
      - alert: PrometheusNotificationsDropped
        expr: increase(prometheus_notifications_dropped_total[10m]) > 0
        labels: {severity: page}
      - alert: PrometheusSampleLimitExceeded
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        labels: {severity: ticket}
      - alert: PrometheusCompactionsFailing
        expr: increase(prometheus_tsdb_compactions_failed_total[3h]) > 0
        labels: {severity: page}
      - alert: PrometheusWalCorruptions
        expr: increase(prometheus_tsdb_wal_corruptions_total[1h]) > 0
        labels: {severity: page}

Prometheus also ships an official mixin under documentation/prometheus-mixin in its repository, with dashboards and alerts such as PrometheusHAGroupNotIngestingSamples. Start from it rather than from a blank page, and adapt the thresholds to your estate.

Cost, and the cardinality budget again

Cost in a Prometheus estate has four parts: compute and memory for the servers, disk, the network and backend for remote write, and human time. The first three are driven by the same quantity, active series and ingest rate, which is why cardinality shows up in every section of this guide.

The levers, roughly in order of how much each typically saves, are these. First, cut series you never query: the set of metrics exposed by an exporter or framework is usually far larger than what dashboards and alerts use. Audit what is queried (your query_log_file and the usage data from Grafana help) and drop the rest at the scrape with metric_relabel_configs; Go runtime metrics such as go_gc_.* are a common, low-risk example if no one charts them. Second, lengthen the scrape interval for low-value jobs: halving the sampling rate halves the samples, and few infrastructure signals need 15-second resolution. The default interval is 1m, and the official example configuration uses 15s, so a team copying the example is opting in to four times the samples it needs. Third, choose histograms deliberately: a classic histogram costs one series per bucket per label combination, plus sum and count, while a native histogram is one series and is more precise. Adopting native histograms is one of the higher-leverage cost decisions available, though it requires protobuf scraping and backend support. Fourth, filter at remote write as described earlier. Fifth, use recording rules to make dashboards cheap, which reduces query load and lets you run a smaller server.

Managed services bill typically by active series or by samples ingested, so there the cost equation is nearly linear in cardinality, and a single bad deploy can be felt on an invoice within the hour. Put a budget alert on the ingest rate of each remote-write endpoint, exactly as you would on cloud spend, and treat an unexplained jump as an incident.

A useful internal habit is a short cost review with each team that has a large footprint. Bring three numbers: their active series, their top five metrics by series count, and the fraction of those metrics that appear in any dashboard or alert. A team that sees eighty percent of its series are never read tends to fix the problem by itself.

Upgrades and migrations

Prometheus releases a new minor version every six weeks, and a minor normally stops receiving bug fixes once the next cycle begins. That cadence is fast enough that "we upgrade when we have to" quietly becomes "we are two security releases behind". You have two coherent strategies, and the mistake is having neither. Either follow the latest minor (3.15 at the time of writing) with a disciplined rolling upgrade, or stay on the LTS line (3.13, supported until 2027-07-31), taking patch releases only. The LTS line trades new features for a year of fixes, which suits regulated estates that need change control.

The stability promise helps you plan. Within a major version, the configuration file, PromQL, HTTP API v1, rule format, remote write and read, and the TSDB format are stable. Anything marked experimental or gated behind a feature flag may change or vanish without a major bump. That tells you where to spend review effort: the experimental surface.

The lessons of the 2.x to 3.0 migration are still useful, both because some estates have not finished it and because the same kinds of change will occur again. The prerequisites: you cannot jump from an old 2.x directly to 3.x. The path is to upgrade to 2.55 first, then to 3.x, and downgrades are only possible to 2.55 or later, because a v3 TSDB can only be read by 2.55 and up. The behaviour changes that caused real incidents were these.

For recent upgrades the lesson to internalize is watch the deprecations, because they are scheduled removals. In 3.15, the --log.level flag is deprecated in favour of runtime.log_level in the config file, which is also reloadable, so you can change the log level without a restart. The retention flags have been deprecated since 3.8 in favour of config-file settings. --enable-feature=xor2-encoding is deprecated in favour of storage.tsdb.chunk_encoding.floats. --enable-feature=auto-reload-config is replaced by --config.auto-reload. --enable-feature=native-histograms is now a no-op and logs a warning that tells you to use scrape_native_histograms: true. Each of these logs a message at startup. A platform owner should grep the logs of a staged upgrade for the words "deprecated" and "no-op" and treat every hit as a ticket.

A safe upgrade procedure, which you can turn into a runbook, has these steps:

1. Read the CHANGELOG between your version and the target, focusing on [CHANGE] and [SECURITY] lines, and note any experimental feature you use.
2. Run promtool check config and promtool check rules with the new promtool against the existing files, and run promtool test rules for your unit tests.
3. Upgrade one replica of an HA pair, or a non-critical shard, first. Check /-/ready, then the log, then prometheus_config_last_reload_successful.
4. Compare query results and alert behaviour between the upgraded and non-upgraded replicas for a day. This is the real value of running two.
5. Check downstream readers before enabling any new storage encoding. The XOR2 downgrade path requires deleting blocks.
6. Roll the rest, and keep the previous binary or image tag available.

The Kubernetes version of the same discipline pins the image tag (prom/prometheus:v3.15.0, never latest), because a floating tag turns a routine pod restart into an unplanned upgrade. Also remember that latest is still the busybox image that runs as nobody, not the distroless variant, so a tag change can change file ownership expectations on your volume.

Governance: rules and config as code

When many teams write rules and scrape configs, the failure you are preventing is one team's mistake becoming everyone's outage. The mechanism is the same as for any shared infrastructure: configuration lives in Git, changes go through review, and machines check what humans are bad at checking.

The checks Prometheus gives you for free are strong. Every pull request that touches configuration should run, in CI, with the same promtool version as production:

promtool check config prometheus.yml --lint=all --lint-fatal
promtool check rules rules/*.yml
promtool test rules tests/*.yml
promtool check web-config web.yml

promtool check config validates syntax and, with --lint=all, enables lint checks such as duplicate rules and scrape intervals that are suspiciously long; --lint-fatal makes lint findings fail the build. promtool check rules validates rule files, and by default already lints for duplicate rules. The item people skip is promtool test rules, which is the most valuable: it feeds synthetic series into your rule file and asserts what the alerts do.

rule_files:
  - ../rules/errors.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", code="500"}'
        values: '0+60x20'        # 1 error per second, for 20 minutes
      - series: 'http_requests_total{job="api", code="200"}'
        values: '0+60x20'
    alert_rule_test:
      - eval_time: 5m
        alertname: HighErrorRate
        exp_alerts: []           # still pending; "for" has not elapsed
      - eval_time: 15m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels: {severity: page, job: api}

A test that asserts an alert does not fire at five minutes and does fire at fifteen pins down the for duration, which is exactly the parameter people change in review without thinking. Write one test per alert that pages a human.

Beyond syntax, governance is a set of policies, and the ones that pay off are few.

1. Every alert has an owner and a runbook. Enforce it in CI by requiring severity and a runbook_url annotation on any alert with a paging severity.
2. Every scrape job declares limits. A CI check that refuses a job without sample_limit is a cardinality policy you never have to argue for again.
3. Recording rules follow the naming convention level:metric:operations, for example job:http_requests:rate5m, because colons are reserved for recording rules and the convention tells a reader what kind of series it is.
4. Labels follow a schema. Decide the tenant and ownership labels and reject configuration that omits them.
5. Experimental features require a sign-off. Anything behind --enable-feature, and the experimental PromQL functions, goes through an explicit decision because it may change between minors. Keep an inventory.
6. Config is changed only through Git. With --config.auto-reload, a merged change reaches the server within the reload interval, with no HTTP lifecycle endpoint needed. The deployment shows a failed reload as prometheus_config_last_reload_successful == 0, and the pipeline should check it after every rollout and fail loudly.

For estates that use the Prometheus Operator, the same policies become admission policy on the PrometheusRule and ServiceMonitor objects, which is an argument for that model: Kubernetes already has a place to enforce rules on objects. For splitting one large scrape configuration into team-owned files, scrape_config_files lets the main file include many others, so each team owns a file and a CODEOWNERS entry protects it.

Incident playbooks

The playbooks below follow one shape: the symptom you see, the first checks that narrow the cause, and the fix. They are deliberately short enough to be a runbook.

The server is OOM-killed and restarts in a loop. Symptoms: restarts, /-/ready returning 503 for a long time, no data gaps filled. First, do not keep restarting: each start replays the WAL, which needs memory. Check the last successful prometheus_tsdb_head_series before the crash and compare it with the limit. Then find what grew: a new deployment, a new scrape job, a label that changed. If you can, start it with a bigger memory limit to get it stable, open /api/v1/status/tsdb and run promtool tsdb analyze against a copy of the data, and drop the culprit with metric_relabel_configs. Bring memory back down afterwards. The durable fix is the per-job sample_limit that should have contained it.

A target is down, or everything is down at once. up == 0 for one target is a network, a process, or an auth problem; read the error string on the Targets page, because it is specific. connection refused means nothing listens; context deadline exceeded means it is slow or blocked by a firewall, so compare scrape_duration_seconds with scrape_timeout; 401 and 403 mean credentials, and after an upgrade to 3.13 or newer you should think about cross-host redirects no longer forwarding credentials; received unsupported Content-Type means the strict content-type rule. If every target fails at once, suspect the server or its network, not the targets.

Alerts are not firing, or are firing twice. For silence, check that the rule was actually loaded (/api/v1/rules), that the group is not skipping evaluations (prometheus_rule_group_iterations_missed_total), that evaluation is not failing, and that notifications are not being dropped. Then look at Alertmanager: a silence or inhibition rule may be hiding it. For doubles, suspect the missing labeldrop on replica, or an Alertmanager list that is not the same on both replicas.

Alerts stopped, and no one noticed. The Watchdog did not page, or was never wired up. That is a process failure, and the fix is to route Watchdog through the same path as real alerts to an external receiver that alarms on silence.

Remote write is lagging or dropping. Compute the lag from the two timestamp metrics, then check prometheus_remote_storage_samples_failed_total, pending samples, and the shard count. If shards are pinned at max_shards, the sender is saturated: filter series, raise max_shards carefully, or fix the backend. If logs show 429 Too Many Requests, the backend is rate limiting you by tenant, and the answer is fewer series rather than harder retries. Remember the two-hour WAL horizon above: a long outage cannot be replayed.

Queries are timing out, and the server is slow. Identify the expensive query from the query log, not from guesswork. The errors are informative: query processing would load too many samples into memory in query execution means --query.max-samples was hit, and query timed out in expression evaluation means --query.timeout was. Fix the query (narrower selector, shorter range, recording rule) before you raise the limit, because raising it moves the failure to memory.

The disk is full. Check what the data directory holds: blocks, the WAL, and snapshots. Old admin-API snapshots pile up silently because they are not subject to retention, so check data/snapshots/ first, since deleting those is always safe once copied. If it is the WAL or head, you have a cardinality or a remote-write-backlog problem, not a retention one. Reducing retention frees space only on the next block cleanup, which can lag by up to two hours.

The data directory is locked, or the filesystem is wrong. opening storage failed: lock DB directory: resource temporarily unavailable means another process holds data/lock: two servers share a directory, or a stale container was not stopped. Never share a data directory between instances. Corruption on NFS or EFS is a known consequence of unsupported storage, and the fix is to move to local or block storage and restore from a snapshot or the remote copy.

After an upgrade, dashboards show no data. Suspect the 3.0 semantic changes if you crossed that boundary (le="1.0", left-open ranges), and suspect deprecated flags that turned into errors. Read the startup log first.

For post-incident follow-up, every incident should produce one of three artifacts: an alert that would have caught it earlier, a limit that would have contained it, or a rule test or CI check that would have refused the change.

Where Prometheus stops

Knowing where a tool ends is part of owning it. Prometheus is deliberately narrow, and its own documentation says so: it is not suited to per-request data that must be 100 percent accurate, such as billing, and it is not a log store or a tracing system.

• Logs and traces. Metrics are aggregates; they tell you that latency rose, not which request was slow. For that you need a logging system and a tracing backend. The bridge is exemplars, trace IDs attached to samples, which are experimental in storage (--enable-feature=exemplar-storage), so treat them as a convenience.
• Exact accounting. Scrapes are sampled, counters reset, and series can be dropped by limits. Use a real event pipeline for money.
• Long retention and global query at large scale. A single server is not the tool, which is what remote write to Thanos, Mimir, Cortex, VictoriaMetrics, or a managed service is for.
• Real multi-tenancy and access control. As covered, isolation means separate servers or a backend built for it.
• Very high cardinality analytics. If the question is about individual users or request identifiers, you want an analytics or event store.
• Push-first workloads. The Pushgateway exists for short-lived batch jobs, and it is not a general push path: it never forgets a series unless it is deleted. If most of your sources are push-based, the OpenTelemetry pipeline with the OTLP receiver, or a remote-write-native design, fits better than bending the pull model.
• Unified observability. When the platform question becomes "one tool for metrics, logs, and traces", the answer is usually a stack around Prometheus rather than Prometheus alone. The neighbouring guides in this catalogue cover the pieces: Grafana for dashboards and alerting UI, OpenTelemetry for instrumentation and collection, and the ELK stack for logs.

A decision rule that holds up in review: choose Prometheus when you need reliable, cheap, real-time operational metrics with a powerful query language; add a remote backend when you need retention, global view, or tenancy; and choose something else when the question is about individual events rather than aggregates.

Running Prometheus as a platform

The reason this level exists is that you are not just running a server, you are offering a service, and the customers are other engineers. A platform has a product shape: an interface, a contract, documentation, and a support model.

The interface. Decide how a team gets monitored, and make the easy path the secure and cheap one. In Kubernetes, that usually means a ServiceMonitor or PodMonitor and a PrometheusRule in the team's own namespace, with a template that sets the tenant label, a default sample_limit, and a runbook link. Outside Kubernetes it means a file-based service discovery (file_sd_configs) or an HTTP service discovery (http_sd_configs) endpoint that teams register into, rather than editing a central file. A golden path that takes one pull request is worth more than any wiki page.

The contract. Write down and publish what you promise. Examples: scrape interval defaults and the minimum allowed; the sample limit per job and how to request more; retention for local data and for the long-term store; the alerting path and the expected delivery time; the supported Prometheus version and the upgrade window; and what is explicitly not offered, such as cross-team read isolation on a shared server. Publish service-level objectives for the platform itself, for example the fraction of scrapes that succeed, the rule-evaluation lag, and the remote-write lag, and measure them with the meta-monitoring above.

The enablement. Teams misuse metrics because nobody taught them. Provide client-library guidance and instrumentation conventions: the naming practices (units in the name, _total on counters), the rule never to use unbounded labels, guidance on choosing histogram buckets or native histograms, and a short worked example in each language you support. Run office hours. The cheapest capacity you will ever buy is a team that never creates the bad metric.

The support model. Decide who is paged when the platform is down, and what a team can expect when its own instrumentation breaks. Offer a cardinality report per team, a dashboard of each team's own scrape health, and a clear boundary: the platform owns the server and the pipeline, the team owns its metrics and its alerts.

The lifecycle. Metrics and alerts rot. Put a review on the calendar: retire dashboards and alerts that nobody looks at, revisit limits, and prune jobs whose targets have gone. Track the platform's own health over time. For the MLOps estates this community works in, there is one more dimension worth building into the platform: model-serving frameworks such as KServe, Seldon, BentoML, Triton, vLLM, and Ray Serve expose Prometheus metrics, and GPU utilisation arrives through the NVIDIA DCGM exporter. A platform that offers a ready-made scrape template and dashboard for model-serving latency, throughput, and GPU use gives ML teams observability without asking them to learn any of the above. The neighbouring guides Kubernetes, Helm, and Argo CD cover the deployment mechanics that make those templates repeatable.

The final piece is honesty about cost and limits in the conversation with leadership. A platform owner earns trust by reporting the numbers without being asked: active series and their growth, cost per team, the incidents avoided by limits, and the upgrade debt. When someone asks for "monitoring of everything forever", the answer is a menu with prices, not a refusal.

The review checklist

Use this for design documents and for pull requests that touch the monitoring platform. Every item is a question with a concrete answer.

Capacity and cardinality

• What is the expected number of active series, and what is the growth rate? Is there a per-job sample_limit at roughly double today's count?
• Does any label carry an unbounded or fast-churning value? Would it survive a deploy that replaces every pod?
• Are histograms justified, and have native histograms been considered?
• Is there headroom for compaction and WAL replay in both memory and disk?

Reliability

• What happens when this server is gone: is there a replica, a remote copy, or a backup, and has the restore been tested?
• Is /-/ready used for readiness and /-/healthy for liveness?
• Are replicas distinguished by an external label that is dropped from alerts?
• Do all Alertmanager instances appear in the alertmanagers list, with no load balancer between?
• Is there a Watchdog alert going through the real notification path to an external receiver?

Security

• Is the HTTP port reachable only by the intended callers, and is authentication in front of it?
• Are the admin API and lifecycle endpoints off, or restricted and justified?
• Are secrets in *_file fields from a secret store, never inline in Git?
• Is the image pinned to a version, and is it the distroless variant where possible?
• Is the version at or above the latest security fixes for its line?

Rules and alerts

• Does every paging alert have an owner, a severity, a runbook link, and a rule test?
• Does any group risk overrunning its interval? Is prometheus_rule_group_iterations_missed_total alerted on?
• Could a rule limit silently clear an important alert?
• Are expensive expressions behind recording rules with conventional names?

Change and upgrade

• Does CI run promtool check config, check rules, and test rules with the production version?
• Is any experimental feature flag or function in use, and is that recorded and signed off?
• Does the plan account for deprecations and no-op warnings, and does it say how to roll back?
• If enabling XOR2 or a new storage feature, has every reader of the blocks been confirmed compatible?

Cost and tenancy

• Who owns each series, through which label, and does anyone see their own cost?
• Is write_relabel_configs filtering what is sent to a billed backend?
• Is data residency satisfied for where the metrics and their labels end up?

The complete picture

Step back and the whole series fits in one path. At the beginner level you learned to read the model: a pull-based server, targets and jobs, a labelled data model, the rate() function, and the first dashboard. At mid level you learned the machinery well enough to predict it: service discovery and relabeling, recording and alerting rules, the HTTP API, Alertmanager, and the TSDB. At this level the same pieces acquired a second dimension, which is what happens at scale and under ownership.

Each arrow is a place where a platform owner has put a control, and each control has a metric that proves it is working. That is the test of a mature setup: not that nothing breaks, but that every break is bounded, visible, and owned.

Where the series leaves you

You can now reason about Prometheus the way its owner must: from the memory a label costs to the notification a human receives, and through every failure in between. You can size a server, find and remove a cardinality bomb, choose between sharding, federation, and a remote backend, design an HA pair that does what you claim, and describe honestly what it does not protect. You know the trust model well enough to say where authorization has to come from. You can plan a version upgrade from the release notes, run an incident from a playbook, and run a platform with a published contract.

What to do next depends on your path. If you will build the delivery side, read Argo CD and Helm to manage rule files and configuration as code. If dashboards and alert UX are your next problem, go to Grafana. If you are expanding from metrics toward traces and a vendor-neutral pipeline, read OpenTelemetry. And keep the habit that matters most: read the CHANGELOG before every upgrade, because the project tells you what it is about to change, and the estates that stay healthy are the ones that listen.

Sources

• Prometheus overview
• Release cycle and LTS policy
• Prometheus 3.15.0 release notes
• Prometheus 3.15 CHANGELOG
• Prometheus 3.13 CHANGELOG (LTS)
• Migration guide to 3.0
• Command-line flags: prometheus
• Command-line reference: promtool
• Configuration reference
• Recording rules
• Alerting rules
• Unit testing for rules
• Securing with TLS and basic auth (web config)
• Feature flags
• Storage
• HTTP API
• Management API
• Federation
• Agent mode
• Stability guarantees
• Data model
• Metric types
• Security model
• Naming practices
• Recording rule practices
• Alertmanager overview
• Using Prometheus as an OpenTelemetry backend
• Official Dockerfile at v3.15.0

Part three of three. The senior interview is not a quiz on flags. You are asked to design a system, to run an incident while someone watches, and to persuade a person who controls budget. This page rehearses all three, with reasoned answers and a note on what separates a strong answer from a weak one. About fifty minutes.

The pattern that works in every answer below: state the constraint, name the component that breaks first, give the trade-off you are accepting, and say how you would know it is working. Interviewers at this level are listening for that last clause more than any other.

How to answer at this level

Three habits separate senior answers from mid-level ones.

Ask before you design. The first question to any design prompt is about scale and tenancy: how many active series today, how fast is it growing, how many teams, what retention, and what happens if metrics are lost for ten minutes? A design given without those numbers is a guess, and a good interviewer will say so.

Name what you are not promising. Prometheus has sharp edges: there is no authorization, HA does not duplicate data, and sharding loses the global view. A candidate who volunteers these limits before being asked is showing ownership experience, because those are the lessons people learn from incidents.

Connect every decision to a metric. "I would add limits" is weak. "I would add a per-job sample_limit at about twice current scrape_samples_scraped and alert on prometheus_target_scrapes_exceeded_sample_limit_total" is strong, because it is testable.

System-design prompts

Design monitoring for forty teams on a shared Kubernetes platform

Clarify first. Ask for the current and forecast active series, the number of clusters, retention needs (two weeks for debugging, or a year for capacity planning), whether teams must be isolated from one another, and the availability target for alerting.

A strong answer. Separate collection from storage from querying. In each cluster, run a platform-owned collector: a Prometheus (in agent mode if it needs no local alerting) that scrapes via Kubernetes service discovery and remote-writes to a central multi-tenant backend. Teams describe what to scrape with ServiceMonitor and PodMonitor objects in their own namespaces, and their alerts as PrometheusRule objects, so ownership follows the namespace. The platform injects a tenant label from namespace metadata by relabeling, applies a default sample_limit per job, and enforces per-tenant series limits at the backend. Alert evaluation happens either in local non-agent servers (for latency) or in the backend's ruler, and all alerts go to a clustered Alertmanager, with every instance listed in the Prometheus configuration and no load balancer between them.

Trade-offs to state. Running a multi-tenant backend (Mimir, Cortex, Thanos Receive) or paying for a managed one buys real isolation and per-tenant limits, at the cost of a distributed system or a bill. The alternative of one Prometheus per team is simpler and gives strong isolation, but it multiplies patching and loses the cross-team view. Agent mode makes collectors cheap but moves alerting away from the edge, so a cluster that loses its uplink loses alerting until it returns; for critical cluster-level alerts you may keep local evaluation too.

What a weak answer does. Proposes a single large Prometheus for everyone, "with RBAC in Grafana", not realizing that Prometheus itself has no authorization, so any query-API access reads every team's data.

Follow-up you should expect. "How do you stop one team taking it all down?" The answer is layered: scrape limits per job, per-tenant limits at the backend, --query.max-samples and timeouts, and a cost showback by tenant label so teams see their own bill.

A single Prometheus is at its memory limit. What now?

Clarify first. Is the pressure steady-state or from churn? Has it grown slowly or jumped? Did a deployment change a label?

A strong answer, in order. Diagnose before scaling. Look at prometheus_tsdb_head_series against the series creation rate, at /api/v1/status/tsdb for the top metrics and labels, and run promtool tsdb analyze on a copy of the data directory. Nine times out of ten the answer is one label or one histogram. Drop it at the scrape with metric_relabel_configs, and add the per-job limit that would have contained it. Only when the legitimate series count is too large for one box do you scale structurally: functional sharding by job or team first, because it aligns with ownership, then hash sharding of a single giant job with hashmod, then a horizontally scalable backend.

Trade-offs to state. Sharding gives you ingest capacity but removes the ability to answer global queries from any one server, and breaks alerts that compare series held on different shards, so you add a global layer (federation for a few aggregated series, or a remote-write backend). Vertical scaling buys time but leaves the restart problem: a bigger head means a slower WAL replay.

The detail that impresses. Mentioning that restart time scales with the head, so a memory-bound server in an OOM loop should be stabilized with more memory first, because each restart replays the WAL and needs the memory that is missing.

Design long-term retention of a year of metrics for a regulated client

Clarify first. Where must the data live? For Gulf and Egyptian employers with residency obligations, a remote-write target in another region is a compliance question, not a technical one, and labels can carry identifying detail. What resolution is needed for old data? Who queries it?

A strong answer. Keep local retention short and useful (for example 15 to 30 days), because local storage is the fast path for debugging and the fallback if the backend is unavailable. Send data by remote write to a backend in the required region, or use a block-upload model to object storage in that region. Filter with write_relabel_configs so you only store what has a retention requirement. Monitor the pipeline with lag and failure metrics, and alert on both.

Trade-offs to state. Remote write costs a constant stream and a backend; the sidecar model keeps ingest servers stateful and ties you to the block format, so a newer chunk encoding such as XOR2 must be checked against the uploader before enabling it, because blocks written that way cannot be read by older tools and the only rollback is deleting blocks. Downsampling for old data is a property of the backend, not of Prometheus.

What a weak answer does. Sets retention.time: 365d on a single server and calls it done, ignoring disk sizing (retention multiplied by ingest rate multiplied by bytes per sample), the compaction headroom, and the absence of any replica.

Design the alerting path so that "no news" is never mistaken for "good news"

A strong answer. Treat the pipeline as three layers: evaluation, delivery to Alertmanager, and notification. For each, name the signal that shows it failing: rule-evaluation failures and missed iterations, dropped notifications and Error sending alerts, and a Watchdog alert that always fires and is routed through the same notification path to an external receiver that pages when it goes quiet. Add a meta-monitoring rule file to every server, and alert on prometheus_config_last_reload_successful == 0, because a failed reload keeps the old config running and nothing else tells you your change never applied.

Trade-offs to state. An external dead-man's-switch is one more dependency and a small cost, in exchange for removing the one failure that monitoring itself cannot see.

Follow-up you should expect. "What if Alertmanager is down?" A clustered Alertmanager with gossip (--cluster.peer), every instance listed in Prometheus, and the Watchdog path tested regularly.

Onboard a machine-learning platform that serves models on GPUs

A strong answer. Offer a golden path: a scrape template for model servers (KServe, Seldon, BentoML, Triton, vLLM, and Ray Serve all expose Prometheus metrics), the NVIDIA DCGM exporter for GPU utilisation, and a standard dashboard of latency histograms, throughput, errors, and GPU use. Use native histograms for latency if the backend supports them, because latency histograms across many model variants are a classic source of series explosion. Put the model name and version in labels only if the set is bounded, and never a request or user ID. Alert on the symptom (error rate and latency SLO burn), not the cause.

Trade-offs to state. Model version labels are useful and also a churn source when versions are created continuously; cap them with sample_limit and drop stale versions by relabeling.

Incident-response questions

"Prometheus is restarting every few minutes. Walk me through it."

Strong answer. Start with whether it is being killed or crashing. Check the container or service status and exit code; 137 points to the OOM killer. Then look at what the log says before the restart, and whether startup is spending its time replaying the WAL (readiness returning 503). Do not simply keep restarting, because a loop of replay then kill is self-reinforcing. Give it more memory to stabilize, find what grew (a new deployment, a new job, a churning label), contain it by dropping the series at the scrape, then reduce memory again. Finally, add the limit that would have prevented it and an alert on head series growth. If you mention /-/healthy versus /-/ready, and that a liveness probe wired to the wrong one can itself cause the loop, you have shown real experience.

"We upgraded and every dashboard for request latency is empty."

Strong answer. If you crossed from 2.x to 3.x, suspect the label normalization: le="1" is now le="1.0", so selectors filtering on the old form match nothing. Check le values with a quick query. Then check range semantics, since range selectors became left-open and right-closed, so sub-query windows like [1m:1m] can return too little. Fix queries rather than reverting, and mention that downgrade is only possible to 2.55 or later because older servers cannot read a v3 TSDB. Follow-up: how would you have caught it? By running the old and new versions side by side and diffing query results before cutting over, and by testing dashboards' queries.

"Alerts paged twice for every incident after we added a second replica."

Strong answer. The replicas differ by an external label, so their alerts differ by a label and Alertmanager treats them as distinct. Drop the replica label in alert_relabel_configs on both servers, which applies after external labels, so the alerts become identical and are deduplicated. Also verify both Prometheus servers list all Alertmanager instances.

"A target's up has gone to zero and the error says sample limit exceeded."

Strong answer. The limit is working as designed: the whole scrape fails, deliberately, to protect the server. Determine whether it is a legitimate increase or a bug. Look at scrape_samples_scraped over time for the job and at which metric grew, then fix the instrumentation (drop a label, bound a value) with the owning team, and only then consider raising the limit. The point to make is that this alert is about the team's instrumentation, and the platform should have told them before their data went missing.

"The remote-write backend says 429. Do we just retry harder?"

Strong answer. No. A 429 means the tenant is being rate limited, and retrying harder keeps the queue full and adds lag. Compute the lag from the timestamp metrics, look at shard counts pinned at max_shards, and reduce what you send with write_relabel_configs. If the data is legitimately needed, negotiate the limit with whoever runs the backend. Also note that a long enough outage exceeds the WAL horizon of about two hours, after which unsent samples are gone, so the durable fix is reducing volume, not patience.

"We need to delete data that should never have been recorded."

Strong answer. The admin API's delete_series writes tombstones, and space is reclaimed later by compaction or clean_tombstones, so it is not instant. The admin API must be enabled for this, which is a security trade-off, so enable it temporarily and restrict network access. Point out that copies already sent by remote write are not deleted by this, and that the real fix is removing the source of the data and, if it leaked something sensitive, rotating whatever it exposed.

Questions that test depth

Why can't you load-balance two Prometheus replicas? Because each replica has its own independent data, scraped at different instants and with its own gaps after restarts. A load balancer makes every refresh show a different history. Pin to one replica, or deduplicate by the replica label in a query layer built for it.

What does HA actually protect? Alert delivery. It does not protect data identity, it does not protect against bad config or a cardinality explosion, and it doubles cost.

Why is federation usually the wrong tool? Because it pulls series into the global server's head, so copying everything makes the global server as large as all leaves combined. It is right for a few pre-aggregated series and wrong for bulk replication.

How does Prometheus authorize users? It does not. The web config supports TLS, mTLS, and bcrypt basic auth only. Authorization is a proxy, a label-enforcing proxy, or a multi-tenant backend.

What changed about credentials on redirects in 3.13? HTTP clients no longer forward credentials when a redirect goes to a different host, across scraping, remote read and write, alerting, and service discovery. It closes a credential leak and can appear as a 401 after an upgrade.

Why is a rule group limit dangerous for alerting rules? Exceeding it discards all of that rule's series and clears its alerts, so the alert goes quiet when the condition is at its worst.

When would you choose the LTS? When change control matters more than new features, for example in regulated environments; 3.13 is supported until 2027-07-31, and you take patch releases only. When would you not? When you need a feature that arrived later, and when you can afford the six-week cadence.

Arguing trade-offs to leadership

Senior candidates are often asked to make a case to someone who does not care about TSDB internals. The skill is translating a technical constraint into a decision, a cost, and a risk.

"Why can't we just have one big Prometheus for the whole company?" Start with what leadership cares about: blast radius and cost. A single server means one team's bad metric is everyone's outage, and it has no built-in way to stop people reading each other's data. Offer the menu: one server per domain (simple, isolated, more to maintain), or a shared multi-tenant backend (real isolation, higher cost and complexity). Recommend one, say what you would measure to know you were wrong, and give a price.

"Why is the monitoring bill growing faster than traffic?" The honest answer is that cost follows active series, not traffic, and series growth comes from labels and deploy churn. Offer a cost showback by team, a short list of the top series nobody queries, and a plan that typically removes a large fraction of them with relabeling. Give the expected saving as a range, and commit to a date for the first report.

"Do we really need two replicas? It doubles the cost." Explain what it buys (alerting survives one server's loss) and what it does not (it is not a backup). Then offer the cheaper alternative if the risk is acceptable: one replica plus a tested restore and an external Watchdog, with the recovery time stated honestly. Let them choose with the numbers in front of them.

"Can we keep everything for five years?" Not on a single server, and not for free. Show the formula: retention seconds times samples per second times bytes per sample. Propose tiers: short local retention for debugging, filtered long-term retention in the required region for capacity planning, and explicit decisions about which series qualify.

"Should we delay the upgrade? It works fine." Point to the release cadence and the security record: several fixes in 2026 affected the UI and the config endpoint, and minor versions stop getting fixes once the next cycle begins. Offer the LTS line as the low-change option, with a dated support horizon. Frame the cost of not upgrading as accumulated debt with a deadline, not as a risk in the abstract.

"Why did we not see this outage coming?" Do not defend the tooling. Say what you will change: an alert that would have caught it, a limit that would have contained it, and a check in CI that would have refused the change. Bring the timeline and the three artifacts, and a date.

What a strong answer covers that a weak one misses

Final self-test

Answer these aloud, in two minutes each, without notes.

1. A server has 6 million active series and is at 90 percent of its memory limit. Give your diagnosis order and your three cheapest mitigations.
2. Explain precisely what two Prometheus replicas do and do not protect, and how you would make their alerts deduplicate.
3. List the three layers of the alerting pipeline and the metric that shows each failing.
4. Why does a failed configuration reload not cause an outage, and why is that a problem?
5. What are the security boundaries of Prometheus, and what must you add to isolate two teams?
6. You must enable XOR2 encoding. What do you check first, and what is the rollback?
7. Design collection, storage, and alerting for forty teams. Name the component that breaks first.
8. Justify to a finance lead why the monitoring bill rose thirty percent, and what you will do about it.

If you can answer all eight with a constraint, a component, a trade-off, and a metric each time, you are ready to be asked these questions by someone who has operated the tool themselves. The neighbouring guides, Grafana for the dashboard side and OpenTelemetry for the collection side, are the natural follow-up questions a panel will move to next.

Part three of three. At this level the problems are operational: a server that falls over when one team ships a label, an alert pipeline that fails silently, an upgrade that changes the meaning of a query. Start with the error table, then the practices, then the recipes and checklists you can run this week.

Common errors at this level

Cumulative: everything from Beginner and Mid still applies. These cause incidents and quiet data loss rather than failed first scrapes.

The practices that pay off most

Practice cards

1. Blow up cardinality on purposeEmit a counter with a request_id label from a toy service. Watch prometheus_tsdb_head_series and memory, find it with promtool tsdb analyze and the TSDB status endpoint, then remove it with labeldrop.
2. Trip a limit and read the failureSet a sample_limit below a job's sample count. Read the exact error on the Targets page and check which metric counts the event.
3. Overrun a rule groupCreate a group whose expression takes longer than its interval. Find the missed-iterations counter, then fix it with a recording rule in an earlier group.
4. Double your alerts, then deduplicate themRun two replicas and one Alertmanager. Count notifications before and after the labeldrop on replica.
5. Kill the pipeline and find the silencePoint Prometheus at a dead Alertmanager address. See Error sending alerts and the error counter, then confirm your Watchdog path would have caught it.
6. Break a reloadIntroduce a YAML error, send SIGHUP, and confirm the old config still runs and the reload gauge reads 0. Wire an alert to it.
7. Restore from a snapshotTake a snapshot with the admin API, delete the data directory, restore, and measure the gap between the snapshot and the moment you stopped.
8. Translate a deprecated start commandStart with --log.level, the retention flags, and the native-histograms feature flag. Convert each to its config-file form and watch the warnings vanish.
9. Watch a remote write lagPoint remote_write at a receiver you can stop. Stop it, graph the lag and the pending samples, restart, and watch them drain.

Operational recipes

Size a server from measurements

Estimate, then verify. Read three numbers off the live server and compute the disk.

# Active series and ingest rate
curl -s 'localhost:9090/api/v1/query?query=prometheus_tsdb_head_series'
curl -s 'localhost:9090/api/v1/query?query=rate(prometheus_tsdb_head_samples_appended_total%5B5m%5D)'

# Disk = retention seconds x samples/s x bytes/sample (1 to 2 bytes on disk)
python3 - <<'PY'
retention_s = 30 * 24 * 3600
samples_per_s = 120_000          # replace with the measured rate
for bytes_per_sample in (1, 2):
    gib = retention_s * samples_per_s * bytes_per_sample / 2**30
    print(f"{bytes_per_sample} B/sample -> {gib:,.0f} GiB")
PY

Add headroom for compaction, because source and merged blocks coexist for a while, and for the WAL and snapshots. Treat the result as a floor rather than an exact number.

Find the worst cardinality offenders

curl -s localhost:9090/api/v1/status/tsdb | python3 -m json.tool | head -60
promtool tsdb analyze /var/lib/prometheus | head -80

Then the PromQL probe, run off-peak because it touches every series:

topk(10, count by (__name__) ({__name__=~".+"}))

Read the two outputs together. If one metric dominates, it is an instrumentation problem. If the top label values are many and random, look for an identifier in a label.

A safe per-job limit template

scrape_configs:
  - job_name: team-checkout
    scrape_interval: 30s
    sample_limit: 40000          # about 2x current scrape_samples_scraped
    label_limit: 30
    label_value_length_limit: 256
    target_limit: 150
    body_size_limit: 10MB
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*"
        action: drop             # runtime metrics nobody charts
    static_configs:
      - targets: ["checkout-1:8080"]
        labels:
          team: checkout

Find the right number with max_over_time(scrape_samples_scraped{job="team-checkout"}[7d]) and set the limit at roughly double that. Review each quarter.

The HA pair in five lines

global:
  external_labels: {cluster: prod-me-central, replica: a}   # 'b' on the second
alerting:
  alert_relabel_configs:
    - {action: labeldrop, regex: replica}
  alertmanagers:
    - api_version: v2
      static_configs:
        - targets: ["am-0:9093", "am-1:9093", "am-2:9093"]

Every Alertmanager is listed, with no load balancer. Dashboards point at one replica, or at a layer that deduplicates on replica.

Reload safely, and know it worked

promtool check config prometheus.yml --lint=all --lint-fatal
promtool check rules rules/*.yml
promtool test rules tests/*.yml
kill -HUP "$(pidof prometheus)"            # always works; no HTTP endpoint needed
sleep 2
curl -s 'localhost:9090/api/v1/query?query=prometheus_config_last_reload_successful'

If the value is 0, the server kept the previous configuration. Read the log, fix the file, and reload again. With --config.auto-reload the same check applies after the reload interval (default 30s).

Take and restore an online backup

curl -s -X POST 'http://localhost:9090/api/v1/admin/tsdb/snapshot'
# copy data/snapshots/<name>/ somewhere durable

# restore: stop Prometheus, then point a fresh data directory at the snapshot
mkdir -p /var/lib/prometheus-restore
cp -a /var/lib/prometheus/snapshots/<name>/. /var/lib/prometheus-restore/
prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus-restore

This needs --web.enable-admin-api on the server being snapshotted, so enable it deliberately and turn it off afterwards. A volume-level snapshot from your storage layer needs no API at all.

A remote-write health set

# Seconds behind: newest scraped versus newest sent, per endpoint
prometheus_remote_storage_highest_timestamp_in_seconds
  - ignoring(remote_name, url) group_right
    prometheus_remote_storage_queue_highest_sent_timestamp_seconds

# Loss and backlog
rate(prometheus_remote_storage_samples_failed_total[5m])
prometheus_remote_storage_samples_pending
prometheus_remote_storage_shards

The first expression subtracts across two metrics with different label sets, so check it on your server before alerting on it, and adjust the matching labels to your remote_name and url. Shards that sit at max_shards mean the sender is saturated.

Validate the security posture quickly

# From a machine that should NOT have access: each of these should fail or require auth
curl -s -o /dev/null -w "%{http_code}\n" http://prometheus.internal:9090/api/v1/query?query=up
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://prometheus.internal:9090/-/reload
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://prometheus.internal:9090/api/v1/admin/tsdb/snapshot
# Check what the config endpoint reveals
curl -s http://prometheus.internal:9090/api/v1/status/config | head -c 400

A 200 on any of these from the wrong network means the boundary is not where you believe. A 404 or 403 on the lifecycle and admin paths is the good result when those features are off.

Verifying rather than assuming

Every control in this guide has a way to prove it works, and the habit worth building is to run the proof after the change.

• A limit works when you deliberately exceed it on a lab job and watch the scrape fail with sample limit exceeded.
• A dedup works when a test alert produces exactly one notification with two replicas running.
• A Watchdog works when you stop Alertmanager on purpose and the external receiver pages you.
• A reload works when prometheus_config_last_reload_successful is 1 and prometheus_config_last_reload_success_timestamp_seconds is recent.
• A backup works when you have restored it and measured the gap.
• A rule test works when changing for makes it fail.
• An upgrade works when a replica on the new version returns the same query results as one on the old version for a day.
• An isolation boundary works when a user in team A is refused a query that reads team B's label. If you never tried it, you do not know.

Keeping the platform current

Three cadences cover it.

Weekly: read the meta-monitoring dashboard for missed rule iterations, dropped notifications, failed reloads, and head series growth; send each team its cardinality report.

Every six weeks: a new minor release begins, and the previous minor stops receiving bug fixes once the next cycle starts. Read the CHANGELOG [CHANGE] and [SECURITY] lines, stage the upgrade on one replica or shard, compare behaviour, and then roll. If you track the LTS line instead, take the patch releases as they arrive; 3.13 is supported until 2027-07-31, so set the migration date now.

Quarterly: prune. Delete dashboards and alerts nobody uses, review each team's limits, retire scrape jobs whose targets are gone, and recheck retention against actual usage.

When you upgrade, grep the startup log for two words, "deprecated" and "no-op", and treat each hit as a ticket. The current set worth knowing is that --log.level is deprecated in favour of runtime.log_level (which reloads without a restart), the retention flags have been deprecated since 3.8 in favour of storage.tsdb.retention, --enable-feature=xor2-encoding is superseded by storage.tsdb.chunk_encoding.floats, and --enable-feature=native-histograms is a no-op replaced by scrape_native_histograms: true.

Incident playbooks, compressed

For each, end the incident with one of three outputs: an alert, a limit, or a test.

Review checklists

For a pull request that changes scrape or rule configuration

• Does CI pass promtool check config --lint=all --lint-fatal, check rules, and test rules on the production version?
• Does every new scrape job set sample_limit, and does it carry the ownership label?
• Does every new paging alert have a severity, a runbook link, an owner, and a test?
• Is any expression expensive enough to need a recording rule, and is it named level:metric:operations?
• Could a label added here take an unbounded value? Would it survive a full pod replacement?
• Does any rule group set a limit, and what happens to its alerts if it is exceeded?
• Is for long enough to avoid flapping and short enough to meet the response target?
• Are secrets in *_file fields, and not inline?

For a design document that adds or changes a Prometheus deployment

• What are the active series today and at twelve months, and what does the growth assumption rest on?
• What is the failure domain, and what happens when the server or its disk is lost?
• Is HA claimed? If so, does the document say what it protects and what it does not, and how alerts deduplicate?
• Who can reach the HTTP port, and where does authorization come from?
• Is multi-tenancy needed, and which of the four patterns is chosen, and why?
• How is data retained and where does it physically live, given data-residency obligations?
• What is the upgrade policy, the named support horizon, and the rollback?
• Which experimental features are used, and who signed off?
• What are the platform's own service-level indicators, and how are they measured?
• What does it cost, and what is the showback by team?

Platform-ownership habits

Write the contract down and publish it: defaults, limits, retention, alerting path, version policy, and what is not offered. Offer a golden path in which adding a service to monitoring is one pull request. Treat the meta-monitoring rule set as part of the product, and ship it to every server. Run a game day each month. Report numbers to leadership without being asked: series, cost per team, incidents contained by limits, and upgrade debt. Teach instrumentation, because the cheapest capacity you will ever buy is the metric that never gets created. And when you are unsure how a feature behaves on this version, check the feature flag page and the CHANGELOG instead of a blog post, since many older tutorials still show flags and functions that 3.x removed.

For related guides, see Grafana for the dashboard layer, OpenTelemetry for collection, Kubernetes for where most of these servers run, and Helm for packaging the configuration.

The checklist to run before you call it production

1. Memory and disk sized from measured series and ingest rate, with compaction and WAL headroom.
2. Every scrape job has limits, an owner label, and a known current sample count.
3. Liveness on /-/healthy, readiness on /-/ready, and probe timeouts that tolerate WAL replay.
4. A replica or a tested restore, and a written statement of which one you chose and the data you would lose.
5. Replica labels dropped from alerts, and every Alertmanager listed, with no load balancer between them.
6. A Watchdog routed through the real notification path to an external receiver.
7. Meta-monitoring rules on every server, including failed reload, evaluation failures, missed iterations, dropped notifications, and sample-limit breaches.
8. HTTP access restricted and authenticated at a proxy or with mTLS; admin and lifecycle endpoints off; secrets in *_file fields.
9. Image pinned to a version, distroless where possible, and a volume with the correct ownership.
10. Remote write filtered, with lag and failure alerts, and residency confirmed.
11. CI running promtool checks and rule tests with the production version.
12. A version policy, the next upgrade date, and a clean startup log with no deprecation or no-op warnings.