The Complete Grafana Guide

This is part one of three. It covers everything you need to do real work with Grafana, not a teaser. By the end you can install it, connect it to a data source, explore data with queries, build a dashboard with variables, set up an alert that reaches a human, keep your configuration in files instead of clicks, and call the API from a script. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide is written against Grafana 13.2 (the latest patch at the time of writing is 13.2.3, published on 29 September 2026). Grafana moves fast: a new minor version arrives every six to ten weeks and patches roughly every two weeks, so a few commands that older tutorials show no longer work. Wherever that matters, this guide says so and shows the current form.

Each section ends with a Try it task. Do them as you go. Grafana is a visual tool, and the concepts only stick once you have watched your own panel draw a line, your own alert turn red, and your own dashboard survive a restart.

What Grafana is, and the problem it solves

Grafana is a web application that asks your data stores questions and draws the answers. You point it at a place where numbers, logs, or traces already live (Prometheus, a PostgreSQL database, Loki, Elasticsearch, a cloud monitoring service) and it gives you charts, tables, dashboards, and alerts on top.

The most important sentence in this guide is this one: Grafana is not a datastore. It does not collect your metrics and it does not hold your logs. The data stays in the system that produced or stored it. Grafana only keeps its own bookkeeping: your dashboards, users, data source settings, alert rules, and preferences. That bookkeeping lives in a small database that Grafana manages for itself.

To see why this design exists, think about what monitoring looked like before. A typical team had a metrics tool with its own graph page, a log tool with its own search page, a database console for business numbers, and a cloud console for the infrastructure bill. During an incident, an engineer had four browser tabs, four query languages, and four different time pickers, and had to line up timestamps by eye. Each tool drew its own charts, and each had its own idea of users and permissions.

Grafana's answer is to separate storing from looking. The stores stay specialised (Prometheus is very good at metrics, Loki at logs), and one interface sits on top that speaks to all of them. One time picker covers every panel. One dashboard can show a latency graph from Prometheus next to the error logs from Loki next to a business count from PostgreSQL. One alerting system watches all of them.

For an MLOps engineer this matters in concrete ways. A model-serving endpoint produces request latency and error counts (metrics), prediction logs (logs), and request traces. A training pipeline produces GPU utilisation and loss curves. A feature pipeline produces freshness and row-count numbers in a warehouse. None of that belongs in one database, but all of it belongs on one screen when something breaks at two in the morning. Teams in the Gulf and Egypt who run these systems in a regional cloud or an on-premises cluster for data-residency reasons can also run Grafana in the same place, since it is ordinary software you host yourself; nothing about your data needs to leave your network.

What people use Grafana for:

Grafana comes in three forms, and it helps to know the names now. Grafana OSS is the open-source edition (AGPLv3 licence) that you host. Grafana Enterprise is the same software plus licensed extras such as fine-grained role-based access control, SAML single sign-on, and some extra data sources; the Enterprise image runs with all the open-source features and needs no licence for that. Grafana Cloud is the hosted service run by Grafana Labs. This guide works on the first two and notes where a feature is Enterprise or Cloud only.

The mental model: five nouns

Grafana has many features, but almost everything reduces to five nouns. Learn these and the interface stops being a maze.

A data source is a saved connection to a backend: a type (Prometheus, PostgreSQL, Loki), an address, and credentials. Every data source has a UID, a short string identifier. You will see UIDs in URLs, in provisioning files, and in the API. Older tutorials use a numeric ID instead; that form is legacy, and since Grafana 13 the numeric-ID data source APIs are disabled by default, so always work with UIDs.

A query is the question you ask a data source. The language depends on the data source: PromQL for Prometheus, LogQL for Loki, SQL for PostgreSQL. Grafana sends the query, and the data source answers with data frames: tables with typed columns. A time series is just a frame with a time column and one or more value columns. You rarely need to think about frames until you use transformations, but knowing that every answer is a table explains much of Grafana's behaviour.

A panel is one rendered visualisation: a time series line chart, a single big number (called a Stat), a gauge, a table, a log viewer, a heatmap. A panel is made of queries (what to ask), optional transformations (how to reshape the answer), and display options (how to draw it).

A dashboard is a saved page of panels, together with a time range, variables, and links. A dashboard is stored as JSON and identified by a UID. Dashboards live inside folders, and folders are where you assign who may see or edit them.

An alert rule is a saved query plus a condition, evaluated on a schedule. When the condition holds, the rule produces an alert, and Grafana routes that alert to a contact point (Slack, email, a webhook) according to a notification policy.

A few supporting nouns come up constantly. A plugin extends Grafana: a data source plugin adds a new backend type, a panel plugin adds a new visualisation, and an app plugin bundles pages and features. An organisation (org) is a hard boundary inside one Grafana instance: data sources, dashboards, and teams belong to exactly one org. A fresh install has one org called "Main Org." and most people never need a second. A team is a group of users that you grant permissions to in one step. Explore is the scratch pad where you run queries without building a panel. A variable is a dropdown at the top of a dashboard that changes the queries underneath it.

Finally, know where Grafana keeps its own state. By default it uses SQLite, a single file inside the data directory. That is fine for learning and for a single-user setup. For anything shared and important, Grafana supports MySQL 8.0 or later and PostgreSQL 12 or later. You never put your monitoring data in this database; it holds only Grafana's own configuration.

Versions, editions, and what changed recently

Before installing anything, it is worth five minutes on versions, because Grafana's recent history contains a few changes that break older instructions. You will meet blog posts and videos from 2022 or 2023 that describe a different product than the one you are about to install.

The numbering is simple: Grafana 12 was released in 2025, and Grafana 13 arrived in April 2026, followed by 13.1 in July and 13.2 on 18 August 2026. Several maintenance branches receive patches at the same time: as of late September 2026 those are 13.2, 13.1, 13.0, and the last 12.x line, 12.4. Patch releases are frequent and often security-motivated; the 13.2.3 patch contains only security fixes. The practical rule is to pin an exact version tag such as grafana/grafana-enterprise:13.2.3 in anything you keep, and never use a floating tag like latest outside an experiment, because latest will silently change underneath you on the next pull.

The changes from Grafana 13.0 that matter to a beginner:

• One binary, two subcommands. The old stand-alone commands grafana-server and grafana-cli were removed. You now run grafana server and grafana cli. The Linux service is still named grafana-server (the systemd unit name did not change), which confuses everyone at first: the unit keeps its name, the removed thing is the standalone command.
• The Image Renderer plugin is gone. Turning dashboards into PNG images (for scheduled reports and alert screenshots) now needs the separate remote rendering service. You will not need this as a beginner, but old guides that say "install the renderer plugin" are obsolete.
• /api is deprecated in favour of /apis. Grafana is moving to Kubernetes-style resource APIs. The old /api routes still work and are not being switched off, but they no longer receive updates. This guide uses the familiar /api endpoints for a beginner's needs and shows where the newer form appears.
• Dynamic dashboards and Git Sync reached general availability. Dynamic dashboards add rows, tabs, and automatic layouts. Git Sync lets a Git repository and a Grafana folder stay in step. Both come up again near the end.
• Older removals still bite. Angular-based plugins no longer load (removed in Grafana 12), legacy alerting is gone (removed in 11), and API keys were replaced by service accounts.

There is also a warning about upgrades worth memorising early. Grafana 13.0.0 had a migration bug that could lose or revert dashboards and folders when upgrading from 12.x with Git Sync enabled. If you ever upgrade across a major version, back up the database first, use the latest patch of the target line rather than the first release, and try it in a throwaway copy before the real one.

Installing Grafana and checking the setup

There are several ways to install Grafana. They all produce the same program. Pick the one that matches where you will run it, and do not agonise over it: you can move your dashboards later.

Grafana needs little: a minimum of 512 MB of memory and one CPU core for a single-user setup. The sizing guidance for a shared instance is roughly 2 cores and 2 to 4 GB for fewer than 25 concurrent users, rising from there. Supported platforms are Debian and Ubuntu, RHEL and Fedora, SUSE, macOS, and Windows.

Docker (the quickest way to start)

If you have Docker (see the Docker guide), this is the fastest route and leaves nothing behind on your machine.

docker volume create grafana-storage
docker run -d -p 3000:3000 --name=grafana \
  --volume grafana-storage:/var/lib/grafana \
  grafana/grafana-enterprise:13.2.3

Read it line by line. The first command creates a named volume, a piece of storage Docker manages, so that Grafana's database survives when you delete and recreate the container. Without it, removing the container removes your dashboards. -d runs the container in the background. -p 3000:3000 publishes the container's port 3000 on your machine's port 3000, which is Grafana's default. --volume grafana-storage:/var/lib/grafana attaches the volume at the directory where Grafana keeps its data. The image grafana/grafana-enterprise holds every open-source feature and needs no licence for them; if you prefer the pure open-source build, the image grafana/grafana is published too. Note the pinned tag at the end.

You can change settings at start-up with environment variables, which are explained in the configuration section. For example, -e GF_LOG_LEVEL=debug raises the log detail, and -e GF_PLUGINS_PREINSTALL=grafana-clock-panel@1.0.1 installs a plugin at start.

Docker Compose (better for a real project)

When Grafana will sit beside other services such as Prometheus, a Compose file is cleaner than a long docker run line.

services:
  grafana:
    image: grafana/grafana-enterprise:13.2.3
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_storage:/var/lib/grafana

volumes:
  grafana_storage: {}

Start it with docker compose up -d. We extend this file in the first-project section.

Linux packages

On Debian or Ubuntu, add Grafana's package repository and install from it:

sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafana-enterprise
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server

The commands fetch the repository's signing key so the package manager can verify what it installs, register the repository, then install the package and start the service. Install grafana instead of grafana-enterprise if you want the pure open-source package. On RHEL, Fedora, Rocky, or Alma you create /etc/yum.repos.d/grafana.repo pointing at https://rpm.grafana.com, import the key, and run sudo dnf install grafana-enterprise; the exact file is on the official install page listed in Sources. The same systemctl enable --now grafana-server starts it.

macOS and Windows

On macOS, Homebrew is the easiest route:

brew update && brew install grafana
brew services start grafana

Homebrew's formula may lag behind the newest patch, so check what you got with grafana --version. On Apple Silicon the configuration file lives at /opt/homebrew/etc/grafana/grafana.ini. On Windows, download the MSI installer or the ZIP from the download page; with the ZIP, right-click it, open Properties, and choose Unblock before extracting. From Grafana 13 the command to start the server from the extracted folder is bin\grafana.exe server.

Checking that it works

Whichever route you took, verify in three ways. First, ask the health endpoint, which needs no login:

curl -s http://localhost:3000/api/health

{
  "database": "ok",
  "version": "13.2.3",
  "commit": "…"
}

"database": "ok" means Grafana reached its own database. If it says failing, Grafana is running but its storage is broken (a common cause is a permissions problem on the data directory). Second, open http://localhost:3000 in a browser. Third, read the logs: docker logs grafana for a container, journalctl -u grafana-server for a Linux service. You are looking for a line saying the HTTP server is listening on port 3000.

The first login is admin with password admin. Grafana immediately asks you to choose a new password; do it, because that default is the first thing anyone scanning for Grafana instances tries.

A guided tour of the interface

Once you are logged in, spend ten minutes looking around before you build anything. Grafana's interface is organised around a left-hand navigation menu, and knowing what each entry is for saves hours of hunting. The exact wording of menu items shifts slightly between releases (the homepage was redesigned in 13.2, for instance), so if a label differs, search by purpose.

• Home is the landing page. It shows starred and recent dashboards.
• Dashboards lists folders and dashboards, with search, tags, and a button to create new ones or import existing ones.
• Explore is the ad hoc query page. You pick a data source, type a query, and see results immediately, without saving anything.
• Alerting holds alert rules, contact points, notification policies, silences, and mute timings.
• Connections is where data sources live (Connections, then Data sources), and where you can add new ones.
• Administration holds users, teams, service accounts, plugins, settings, and the stats page. Only admins see most of it.

The command palette opens with Ctrl+K (Cmd+K on a Mac) since Grafana 13.2. Type a dashboard name, a page, or an action and jump straight to it. It is the fastest way to move around once you know what you want.

Two controls appear on almost every dashboard and deserve an early introduction. The time range picker at the top right sets the window all panels show: "Last 6 hours", "Last 7 days", or an absolute range. The refresh picker beside it tells the dashboard to re-run its queries every so often (5 seconds, 1 minute, and so on) or never. Both apply to the whole dashboard, which is what makes a dashboard feel like one instrument rather than a pile of charts.

Finally, look at your profile (your avatar) and Administration, then General, then Stats and licensing. The second page shows the exact version of the server you are talking to, which is worth checking whenever behaviour does not match the documentation you are reading.

Your first data source: Prometheus

An empty Grafana draws nothing, so the first real step is to give it something to ask. We will use Prometheus, the most common metrics store in the Grafana world (see the Prometheus guide for the full story). Prometheus regularly pulls numbers from your services, stores them as time series, and answers queries in a language called PromQL. Prometheus can scrape Grafana itself, which gives us real data with nothing else to install.

Extend the Compose file so that Prometheus runs next to Grafana and scrapes both. First the Prometheus configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: grafana
    metrics_path: /metrics
    static_configs:
      - targets: ["grafana:3000"]

A scrape is one HTTP request in which Prometheus fetches the current numbers from a target. scrape_interval: 15s means it does that every fifteen seconds. The targets use service names (prometheus, grafana), not localhost, because inside a Compose network every container has its own idea of localhost. Keep that in mind; it is the most common first error.

Now the Compose file:

services:
  grafana:
    image: grafana/grafana-enterprise:13.2.3
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      GF_METRICS_ENABLED: "true"
    volumes:
      - grafana_storage:/var/lib/grafana

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus

volumes:
  grafana_storage: {}
  prometheus_data: {}

GF_METRICS_ENABLED turns on Grafana's own /metrics endpoint, which is how Prometheus will read Grafana's numbers. (This environment variable naming rule is explained in the configuration section.) In a real project, pin the Prometheus image to an exact tag too. Bring it up with docker compose up -d, then open http://localhost:9090/targets and confirm both targets say UP.

Next, tell Grafana about Prometheus. In the UI choose Connections, then Data sources, then Add data source, pick Prometheus, and fill in one field that matters:

• URL: http://prometheus:9090

Leave the rest at its defaults and press Save and test. A green banner saying the data source is working means Grafana reached Prometheus and got a sensible answer.

The access mode on a data source form used to offer "Server" and "Browser". Server mode, where Grafana's backend makes the request on your behalf, is the default and the only sensible choice; it keeps credentials out of the browser and lets Grafana apply permissions and timeouts.

If the test fails, read the message. "connection refused" means nothing listens at that address from Grafana's point of view. "no such host" means the name does not resolve (a typo, or the containers are on different networks). "401" means the backend demands authentication you have not configured. A slow test that ends in a timeout usually means a firewall between the two.

Explore: asking questions without building anything

Before you design any dashboard, use Explore. It is the place to work out what to ask, and it is where experienced people spend much of their time, because a dashboard is only the frozen answer to a question someone already figured out.

Open Explore, choose your Prometheus data source, and type the simplest possible PromQL query:

up

up is a metric Prometheus creates for every target it scrapes: 1 if the last scrape succeeded, 0 if it failed. Press Run query (or Shift+Enter). You see a line for each target, sitting at 1. Each line is a time series: a metric name plus a set of labels (key and value pairs such as job="grafana") that identify it, and a stream of timestamped values.

Labels are the heart of PromQL. To select only one job, filter by label:

up{job="grafana"}

Now something more useful. Prometheus stores many metrics as counters: numbers that only ever go up, like the total number of requests served. A raw counter is rarely interesting; what you want is how fast it grows. That is what rate does:

rate(prometheus_http_requests_total[5m])

Read it inside out. prometheus_http_requests_total is a counter of requests Prometheus has served. [5m] selects the last five minutes of samples for each series. rate(...) turns that into the average per-second increase over that window. The result is requests per second, per handler.

To collapse the series into one total, aggregate:

sum(rate(prometheus_http_requests_total[5m]))

And to keep a breakdown by one label while summing away the rest:

sum by (handler) (rate(prometheus_http_requests_total[5m]))

Explore has a Builder mode that lets you pick the metric, labels, and functions from dropdowns, and a Code mode where you type PromQL directly. Start in Builder to learn the vocabulary; switch to Code when you know what you want. The toggle sits at the top right of the query editor. Explore also keeps a query history, so a query you ran yesterday is one click away, and it has a split view for comparing two queries (or two data sources) side by side.

There is one more thing worth understanding early, because it confuses beginners: the step. A chart has a limited number of pixels, so Grafana asks Prometheus for a limited number of points across your time range. When you zoom out to seven days, each point covers a much longer period than when you look at the last hour. Grafana provides built-in variables for this, and the most important one is $__rate_interval, which picks a safe window for rate based on the step and the scrape interval. In a dashboard panel you should write:

sum by (handler) (rate(prometheus_http_requests_total[$__rate_interval]))

instead of hard-coding [5m]. A fixed window that is too small for the step produces gaps ("No data" holes) when you zoom out; $__rate_interval adapts.

Building your first dashboard

An investigation in Explore becomes durable once it lives in a dashboard. You will now build one with three panels.

Create a dashboard from Dashboards, then New, then New dashboard, and choose to add a visualisation. Grafana asks which data source to use: pick Prometheus. You land in the panel editor, which has three areas: the preview on top, the query editor below, and the options pane on the right. Everything you do in it falls into those three areas.

Panel one: a time series of request rate. Paste the query from the last section into the query editor:

sum by (handler) (rate(prometheus_http_requests_total[$__rate_interval]))

The preview draws one line per handler. On the right, find the Title field and call it "Prometheus requests per second". Under Standard options, set the Unit to "requests/sec (rps)" so the axis reads properly. Good units are the cheapest upgrade you can give a dashboard: a number with no unit forces everyone to guess.

Panel two: a Stat for target health. Add a new visualisation. Change the visualisation type (top right of the options pane) to Stat, and use this query:

count(up == 1)

up == 1 keeps only the healthy targets and count counts them, so the Stat shows a single big number. Stat panels are the right choice for "what is it right now". Under Thresholds you can make the value go green above a number and red below it; a colour tells a viewer more quickly than a digit does.

Panel three: a gauge or a table. Add a third panel. Try a Table with the query up, and set Format to Table in the query options so the result arrives as rows rather than as a time series. Tables are the right choice when the question is "which one" rather than "how it changes".

Choosing a visualisation comes down to the question you are answering:

When you finish, press Save dashboard, give it a name such as "Monitoring stack overview", and choose a folder. The dashboard is now stored in Grafana's database. Two habits are worth forming immediately: give every panel a title and a unit, and put the most important panel top-left, because that is where eyes land first.

A word on layout. Since Grafana 13, dynamic dashboards are generally available, and every dashboard is rendered by the newer Scenes engine. In practice that means you can group panels into rows (collapsible sections) or tabs, and you can select several panels and group them in one step (a 13.2 feature). Existing dashboards are migrated automatically when opened. One caution: once a dashboard adopts the newer dynamic schema, it cannot be downgraded to an older Grafana. That is fine if you only run 13.x, and worth remembering if you share dashboards with colleagues on older servers.

Dashboards are JSON underneath. Open the dashboard's settings and find JSON Model to see the whole thing: panel positions, queries, options, and variables. You do not need to read it fluently now, but two facts make it valuable. Everything you can click, you can also put in a file (which is how provisioning and Git work later), and when something looks wrong, the JSON shows exactly what Grafana stored.

Panels in depth: options, overrides, and transformations

A panel looks simple, but three layers of control sit inside it, and confusing them is a classic beginner stumble. Understanding the layers also tells you where to look when a chart does not look right.

Layer one is the query. It decides what data arrives. If the data is wrong (missing series, wrong numbers), fix it here. The query editor also has query options: the maximum data points, the minimum interval, the relative time range, and the format (time series, table, or heatmap). You can add several queries to one panel, labelled A, B, C, and each draws its own series.

Layer two is transformations. A transformation reshapes the data after it arrives, on the Grafana side, before it is drawn. The Transform tab in the panel editor lists them. A few are worth knowing on day one:

• Organize fields renames, reorders, and hides columns. It is the tool you reach for when a table has ugly column names.
• Filter data by values drops rows that do not match a condition.
• Group by aggregates rows, like a SQL GROUP BY.
• Join by field merges results from two queries on a shared column, for example a host name.
• Add field from calculation creates a new column from existing ones, such as a ratio.
• Reduce collapses each series to a few statistics (last, max, mean), which is how you build a compact table of time series.

Transformations run in a pipeline. Each one receives the output of the previous one, not the original data, so order matters: hiding a column before you calculate from it breaks the calculation. When a transformed result surprises you, disable transformations one by one from the bottom up and watch where the data changes.

Layer three is the field configuration: the Standard options (unit, decimals, min, max, display name), thresholds, value mappings (turn 0 and 1 into "down" and "up"), and overrides. An override applies a setting to some fields only: "for the series named errors, use red and draw as bars". Overrides are how you make one chart carry two different units or styles without splitting it.

The other decision is when to do work in the query and when to do it in a transformation. If the data source can do it (PromQL aggregation, SQL GROUP BY), do it there: it moves less data across the network and scales better. Use transformations when you need to combine results from different data sources, or when the source cannot express what you need.

A few more panel features pay off quickly. Legends can show values (last, max, mean) in a table beside the graph, which turns a chart into a readable summary. Tooltips can show all series at once or only the one under the cursor. Panel links and data links let a click jump to another dashboard, carrying the time range and variable values with it. Library panels let you define a panel once and reuse it in many dashboards, and editing the library panel updates all of them. Annotations draw vertical markers on time series: a deployment, an incident, a retraining run. Adding a marker by Ctrl-clicking a graph is the quickest way to start, and annotations can also come from a query, so every deploy in your database shows up on every latency graph.

The visualisations you will use most as a beginner, and when:

Time ranges, refresh, and variables

A dashboard that shows one fixed service is useful once. A dashboard that lets a viewer choose the service, the environment, or the model is useful for a whole team, and that is what variables do. Combined with the time range, they turn a dashboard into a reusable tool.

The time range

Every panel shares the dashboard's time range. A few points save confusion. Relative ranges like "Last 6 hours" move with the clock; absolute ranges are fixed and good for reproducing an incident. Dragging across a graph zooms the whole dashboard into that selection. The time zone can be browser local, UTC, or a named zone, and UTC is the safest for teams spread across countries, because everyone reads the same timestamps. Refresh intervals have a floor: the server setting min_refresh_interval (5 seconds by default) stops people from hammering a backend with very fast refresh.

Variables

A variable is defined in the dashboard settings and shows as a dropdown at the top. Its value is substituted into queries wherever you write $name (or ${name}). The kinds you will use first:

• Query variables fill their options by asking a data source, for example "all values of the job label".
• Custom variables have a fixed list you type in, such as dev,staging,prod.
• Interval variables offer step sizes like 1m,5m,1h.
• Data source variables let the viewer switch between data sources of one type.
• Text box and constant variables hold a free-form or fixed value.

Let us add a job variable to the dashboard. In Settings, open Variables and add a Query variable named job, with the data source set to Prometheus and this query:

label_values(up, job)

label_values(up, job) returns every distinct value of the job label on the up metric, which here is prometheus and grafana. Turn on Multi-value and Include All option so viewers can pick several or everything. Now use it in the first panel's query:

sum by (handler) (rate(prometheus_http_requests_total{job=~"$job"}[$__rate_interval]))

Notice =~ instead of =. It is the regular-expression match, and it is required because a multi-value variable expands into a pattern like (prometheus|grafana). Using = with a multi-value variable is a classic error: the query returns nothing, because no label value literally equals the whole pattern.

Variables can depend on each other. A second variable can filter by the first, for example the instances that belong to the chosen job, so the choices always make sense together. The order of variables in the list matters because one can only reference variables above it.

Interpolation has formats. ${job:csv} renders a comma-separated list, ${job:pipe} a pipe-separated list, and ${job:regex} a regex-safe pattern. You can ignore these until a data source expects something other than the default; when a multi-value selection produces a syntax error in SQL, a format such as ${var:singlequote} is usually what fixes it. Grafana also provides built-in variables you did not define: $__interval, $__rate_interval, $__range, $__from, $__to, $__dashboard, $__org, and $__user.

Newer Grafana versions add more ways to filter: group-by and filter controls that let viewers slice by label without writing variables, multi-property variables, and variables scoped to a single row or tab. They are useful, but the classic dashboard variable above is the thing every Grafana engineer knows, so learn it first.

A second kind of data: SQL and logs

Grafana is not only for metrics. Seeing one non-metrics source early stops you from thinking of it as "a Prometheus viewer". The two to know about are SQL databases and Loki for logs.

PostgreSQL and MySQL

For a SQL database, add a data source of type PostgreSQL (or MySQL), and fill in the host and port, the database name, a user, and a password. Create a dedicated read-only database user for Grafana. Anyone who can edit a dashboard can write any query against that connection, so a user that can only SELECT from the tables you chose limits the damage of a mistake or a malicious edit.

A SQL panel query returns rows. For a time series, the query must return a column named time (or use a time macro) and one or more numeric columns:

SELECT
  $__timeGroupAlias(created_at, '1h'),
  count(*) AS predictions
FROM prediction_log
WHERE $__timeFilter(created_at)
GROUP BY 1
ORDER BY 1

The two macros do real work. $__timeFilter(created_at) expands into a condition that restricts rows to the dashboard's current time range, so the query gets cheaper when you zoom in. $__timeGroupAlias(created_at, '1h') buckets timestamps into hours and names the column time. Without them, you would hard-code dates and the panel would not respond to the time picker. This is how an MLOps team can chart "predictions served per hour" or "rows ingested per day" straight from a warehouse table.

Loki and logs

Loki is Grafana Labs' log store. You query it with LogQL, which looks like PromQL with a log-filtering step. A query has a stream selector in curly braces and optional filters:

{job="api"} |= "error"

{job="api"} selects the stream of logs with that label, and |= "error" keeps only lines containing the text. LogQL can also turn logs into numbers: rate({job="api"} |= "error" [5m]) gives errors per second, which you can chart or alert on. Put a logs panel beside a metrics panel on the same dashboard and you can see a spike and read its cause in one glance. The ELK stack guide covers the main alternative for log search, and Grafana can query Elasticsearch directly as well.

The general lesson is that once a data source is configured, the rest of Grafana (panels, variables, alerting, permissions) behaves the same way. You learn one tool and apply it to many stores. There are dozens of data source plugins: CloudWatch, Azure Monitor, Google Cloud Monitoring, InfluxDB, Elasticsearch and OpenSearch, and more.

Folders, users, and permissions

The moment more than one person uses Grafana, the question becomes who can see and change what. Grafana's model has three layers, and a beginner only needs to know them at a high level.

Organisation roles are the coarse layer. Every user has a role in each organisation they belong to: Viewer can look at dashboards and run queries, Editor can also create and change dashboards and alert rules, and Admin can also manage data sources, users, and settings for the organisation. There is also a None role with no access by default. Separate from these, a Grafana server admin flag marks someone who manages the whole instance (all organisations); it is distinct from an organisation Admin, which surprises people.

Folder and dashboard permissions are the fine layer. Folders are the main permission boundary: you can give a team View, Edit, or Admin on a folder, and everything inside inherits it. Organising dashboards into folders per team or per system, then granting permissions per folder, is far easier to maintain than granting per dashboard. Folders can nest.

Teams group users so you grant permissions once to "ml-platform" instead of to each person. Role-based access control (RBAC), with custom roles and data source permissions, is an Enterprise and Cloud feature; in the open-source edition you combine organisation roles with folder permissions.

Two security facts every beginner should hold on to:

How users get in matters too. Self-sign-up is off by default ([users] allow_sign_up = false), so an admin creates users or connects a login provider. Grafana supports logging in through GitHub, GitLab, Google, Azure AD, Okta, a generic OAuth or OIDC provider, LDAP, and others; SAML and automatic team sync are Enterprise features. Grafana has no built-in multi-factor authentication, so if you need it, use an identity provider that provides it. A small team can start with local users; anything exposed beyond a single-user lab should use single sign-on.

For automation you do not use a person's account. You create a service account, a non-human identity with an organisation role, and give it tokens. That topic comes up in the API section.

Alerting: from a query to a notification

A dashboard only helps if someone is looking at it. Alerting makes Grafana look for you, and it is the part of the tool that has the most moving pieces, so take it slowly. The pieces fit together in a chain:

Grafana's alerting is built on the Prometheus Alertmanager, embedded in Grafana, and since version 11 it is the only alerting system; the older "legacy" alerting was removed.

Alert rules

An alert rule has four parts. First, one or more queries: the same kind of query you wrote for panels. Second, expressions that process the query results on the Grafana server; a common pair is a Reduce expression (collapse a time series to one number, such as the last value) and a Threshold expression (is that number above 0.5?). Third, a condition, which says which expression decides the outcome. Fourth, metadata: a folder, an evaluation group, labels, and annotations.

The evaluation group is a set of rules that are evaluated together at the same interval, for example every minute. The shortest interval the server allows is ten seconds by default, and there is rarely a reason to go near it; most rules do fine at one minute. The pending period says how long the condition must stay true before the alert actually fires. If you set it to 5 minutes, a spike that lasts 30 seconds does nothing. That single setting is the most effective way to stop alerts from flapping, and a great deal of alert fatigue comes from leaving it at zero.

An alert rule produces alert instances, one per series the query returns. If your query returns one series per job, then a single rule tracks each job separately. Each instance moves through states:

• Normal: the condition is false.
• Pending: the condition is true but the pending period has not elapsed yet.
• Alerting (also called firing): the condition has held long enough, and a notification is sent.
• Recovering: the condition has cleared but Grafana is holding the alert briefly before resolving it (added in Grafana 12).
• No Data and Error: the query returned nothing, or failed.

The last two deserve attention. By default, when a rule gets no data, Grafana creates a synthetic alert named DatasourceNoData, and on a query failure it creates DatasourceError. Both carry the rule name and the data source's UID as labels. This is deliberate: a monitoring system that stays silent when it cannot read its data is worse than useless. In the rule's settings you can instead choose to treat no data as Normal, as Alerting, or to keep the last state. Choose carefully, since "Normal" means a broken exporter looks like a healthy system.

Labels and annotations

Labels identify and route: severity=critical, team=ml-platform, service=model-api. The notification policy reads labels to decide who is told. Annotations are human-readable text attached to the alert: a summary (one line), a description (what it means), and a runbook_url (what to do). A good annotation answers "what is wrong and what do I do first" for someone who was asleep a minute ago.

Annotations can use templates, which let them include the actual value:

High error rate on {{ $labels.job }}: {{ printf "%.1f" $values.B.Value }} errors per second

The text inside {{ }} is Go template syntax; $labels holds the alert's labels and $values holds the results of the rule's expressions, referenced by the expression's name (here B).

Contact points

A contact point is a destination: email, Slack, Microsoft Teams, PagerDuty, Opsgenie, a generic webhook, Discord, Telegram, and more. Create one under Alerting, then Contact points. Every contact point has a Test button that sends a sample notification; always press it, because it catches wrong tokens and blocked networks immediately, instead of at three in the morning. A contact point can hold several integrations (email and Slack together). Email needs the [smtp] section of the configuration to be filled in, which is why email is often the first thing that does not arrive on a fresh install.

Notification policies

The notification policy tree decides which contact point receives which alerts. It starts with one default policy, which catches everything and sends it to a default contact point. You add nested policies with label matchers: "if severity = critical, send to the on-call pager; if team = ml-platform, send to their Slack channel". Alerts flow down the tree; the first matching child takes them, unless that child is marked to continue matching.

Policies also control timing through three settings with these defaults:

• Group wait (30 seconds): how long to wait for more alerts of the same group before sending the first notification, so a burst arrives as one message.
• Group interval (5 minutes): how long to wait before sending an update about a group that has new alerts.
• Repeat interval (4 hours): how long to wait before reminding you about an alert that is still firing.

The group by setting chooses which labels split alerts into groups; grouping by alertname and job, say, sends one message per alert and job instead of one per instance. Those timings explain a common puzzle: a rule turns red on the dashboard but the Slack message arrives 30 seconds later. That is group wait working as designed, not a fault.

Silences and mute timings

Two tools stop notifications without touching the rule. A silence is a one-off, time-boxed mute matched by labels: use it during planned maintenance. A mute timing is a recurring schedule, such as weekends or a nightly batch window. Neither stops the rule from being evaluated; the alert still exists and still appears in the UI, but nobody is paged.

Building your first alert

Create an alert that fires when a scrape target is down. Open Alerting, then Alert rules, then New alert rule:

1. Give the rule a name such as "Scrape target down".
2. For the query, choose Prometheus and enter up{job="grafana"}.
3. Add a Reduce expression (function Last) and a Threshold expression that fires when the reduced value is below 1. Set the threshold expression as the alert condition.
4. Choose a folder and create an evaluation group with a one-minute interval.
5. Set the pending period to 1 minute.
6. Under labels, add severity=critical. Under annotations, add a summary such as "Grafana target is down".
7. Choose the contact point, or leave routing to the notification policy.

To test it, stop the thing the rule watches. With Compose, stop Grafana's scrape target by editing prometheus.yml to point grafana:3000 at a port that is not open, and reload Prometheus, or simply pause a container that is scraped in your environment. Watch the rule's state change from Normal to Pending to Alerting in the UI, then confirm the notification arrives at your contact point.

One more principle. Alert on symptoms that a person must act on, not on every number that moves. Grafana's own best-practice guidance suggests building alerts from the RED view of a service (Rate, Errors, Duration): how many requests, how many fail, how long they take. A high-CPU alert with no consequence trains people to ignore the pager, and an ignored pager is how real incidents get missed.

Configuration: files, environment variables, and precedence

Everything so far was done in the browser. The server itself, meanwhile, has settings (which port, where data lives, how users sign in, whether email works), and learning to change them is what turns Grafana from a toy into a service.

Grafana reads its settings from an ini file. Where that file is depends on the install: /etc/grafana/grafana.ini for Linux packages, /opt/homebrew/etc/grafana/grafana.ini for Homebrew on Apple Silicon, conf/custom.ini in a Windows or tarball install. Inside the install there is also a defaults.ini that lists every setting with its default; never edit that file, because upgrades overwrite it. Copy the lines you want to change into your own file. Comments start with ; or #, which matters because a line that still begins with ; is silently ignored, which is a classic source of "my change did nothing".

[server]
http_port = 3000
root_url = https://grafana.example.com/

[security]
admin_user = admin
cookie_secure = true

[users]
allow_sign_up = false

[log]
mode = console file
level = info

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
password = $__env{SMTP_PASSWORD}
from_address = grafana@example.com

The file is divided into sections in square brackets, each holding key = value pairs. A few worth knowing on day one: [server] http_port (the port), [server] root_url (the public address, which must be right when you sit behind a proxy or sub-path, otherwise logins and links break), [users] allow_sign_up (off by default), [log] level (raise it to debug when troubleshooting), and [smtp] (required for email alerts).

Environment variables

Every setting can also be set with an environment variable named GF_<SECTION>_<KEY>, in upper case, with dots and dashes turned into underscores. So [server] http_port becomes GF_SERVER_HTTP_PORT, and [auth.google] client_secret becomes GF_AUTH_GOOGLE_CLIENT_SECRET. This is how you configure a container, where editing files is awkward:

docker run -d -p 3000:3000 --name=grafana \
  -e GF_SERVER_ROOT_URL=https://grafana.example.com/ \
  -e GF_USERS_ALLOW_SIGN_UP=false \
  -e GF_LOG_LEVEL=debug \
  --volume grafana-storage:/var/lib/grafana \
  grafana/grafana-enterprise:13.2.3

The precedence is fixed: built-in defaults, then the ini file, then environment variables, then command-line overrides. Later wins. If a setting "does not stick", an environment variable higher up the chain is usually overriding your file.

Keeping secrets out of the file

Never paste passwords into a file you commit. Grafana can read values from elsewhere. $__env{NAME} (used above for the SMTP password) inserts an environment variable. $__file{/path/to/file} inserts the contents of a file with the trailing newline trimmed, which suits Docker and Kubernetes secrets that are mounted as files. A HashiCorp Vault expander also exists, but it is an Enterprise feature. See the Vault guide for that side.

Some settings are worth setting deliberately from the start. [security] secret_key encrypts the secrets Grafana stores in its database (data source passwords, contact point tokens). If you ever lose it, those stored secrets become unreadable, so back it up together with the database. [paths] controls where data, logs, plugins, and provisioning files live. After editing the ini file, restart Grafana; it reads the file at start-up.

Provisioning: configuration as files instead of clicks

Clicking is how you learn, but it is a poor way to run a service. If your only copy of a data source or dashboard is a row in a database that somebody set up by hand, you cannot review it, reproduce it on a second server, or rebuild it after a disaster. Provisioning fixes that: you describe data sources, dashboards, plugins, and alerting resources in files, and Grafana reads them at start-up.

This is the same idea as infrastructure as code (see the Terraform guide), applied to Grafana. The files live in a provisioning directory, which by default is /etc/grafana/provisioning on a package install, and is organised by kind: datasources/, dashboards/, plugins/, and alerting/.

Provisioning a data source

Here is the Prometheus data source from earlier, as a file:

apiVersion: 1

datasources:
  - name: Prometheus
    uid: prom-main
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s

Every field has a reason. apiVersion: 1 is the file format version. uid: prom-main is a stable identifier that you choose. Always set it: if you leave it out, Grafana generates one, and then dashboards that refer to the data source by UID will not match when you recreate the server. access: proxy is the Server mode discussed earlier. isDefault marks the data source that new panels pick automatically. editable: false stops people from changing it in the UI, so the file stays the single source of truth. timeInterval tells Grafana the scrape interval so $__rate_interval can make good decisions. Secrets go under secureJsonData, and you should fill them from environment variables, for example httpHeaderValue1: $PROM_TOKEN, rather than typing them in the file. Environment interpolation works in values, not keys, and a literal dollar sign is written $$.

With Compose, mount the directory into the container:

services:
  grafana:
    image: grafana/grafana-enterprise:13.2.3
    ports:
      - "3000:3000"
    volumes:
      - grafana_storage:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning:ro

volumes:
  grafana_storage: {}

Recreate the container and the data source appears without you clicking anything. Delete the Grafana volume, start again, and it still appears; that is the point.

Provisioning dashboards

Dashboards use a two-part arrangement. A provider file tells Grafana where to look, and the dashboard JSON files sit in that folder.

apiVersion: 1

providers:
  - name: default
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    allowUiUpdates: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

path is the directory with *.json dashboards. Grafana rescans it every updateIntervalSeconds, so changing a file changes the dashboard within about half a minute, with no restart. foldersFromFilesStructure: true turns sub-directories into folders (up to four levels deep). allowUiUpdates: false means people cannot save edits to these dashboards in the UI; they must change the file. That is the safe choice for a shared service, and it explains the error you will meet sooner or later.

How do you get the JSON? Build the dashboard in the UI, open Export (or Settings, then JSON Model), and copy it into a file. Make sure the dashboard has a stable uid and that its panels refer to your data source by the UID you provisioned (prom-main), not by a generated one. That pairing of "stable UIDs on both sides" is what makes a dashboard portable.

Other ways to manage it as code

Provisioning files are the simplest, but not the only, method. The Terraform provider for Grafana (published as grafana/grafana) manages folders, data sources, dashboards, and alert rules as Terraform resources, which suits teams already using Terraform. Git Sync, generally available since Grafana 13, connects a Git repository to a Grafana folder so dashboards stay in step in both directions: edit in the UI and Grafana proposes a commit, or merge a change in Git and Grafana picks it up. Kubernetes users often use the Helm chart's sidecar, which loads dashboards from ConfigMaps, or the Grafana Operator. You do not need any of these yet; just understand that "dashboards in version control" is the goal and there are several roads to it. The official dashboard best-practice guide makes the same point: keep dashboard JSON in version control and avoid drift.

The HTTP API and service accounts

Anything you can do in the interface, a script can do through Grafana's HTTP API. That is how CI pipelines create dashboards, how scripts list what exists, and how other tools talk to Grafana. You do not need it on day one, but you will meet it quickly.

The health endpoint needs no authentication, as you already saw. Everything else needs a credential. There are two kinds.

Basic authentication uses a real user's name and password (-u admin:password). It is fine for a quick local experiment and required for a few server-admin tasks, but it ties a script to a person and is a poor habit in automation.

Service account tokens are the right tool. A service account is a non-human identity with an organisation role (Viewer, Editor, or Admin), and you attach one or more tokens to it. Tokens begin with glsa_, can be given an expiry, and can be revoked individually without touching anything else. They replace the older "API keys", which were migrated to service accounts automatically in earlier versions; if a tutorial says "create an API key", create a service account token instead.

Create one through the UI (Administration, then Users and access, then Service accounts), or from the command line with basic authentication:

curl -s -X POST -H "Content-Type: application/json" -u admin:admin \
  -d '{"name":"ci","role":"Editor"}' \
  http://localhost:3000/api/serviceaccounts

The response includes the new account's numeric id. Use it to mint a token with a thirty-day lifetime (2,592,000 seconds):

curl -s -X POST -H "Content-Type: application/json" -u admin:admin \
  -d '{"name":"ci-token","secondsToLive":2592000}' \
  http://localhost:3000/api/serviceaccounts/2/tokens

The response contains a key field with the token. It is shown once. Copy it into a password manager or a CI secret store immediately; Grafana cannot show it again. Replace the 2 with the id you received.

Now call the API with the token in an Authorization header:

export TOKEN="glsa_xxxxxxxxxxxx"

# search dashboards
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:3000/api/search?query=overview"

# list data sources
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/datasources

# fetch one dashboard by its UID
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/dashboards/uid/<dashboard-uid>

Search returns a JSON list with each dashboard's uid, title, and folderTitle; the last call returns the full dashboard JSON, which is the same document you saw in JSON Model. A 401 means the token is missing, mistyped, or expired. A 403 means the token is valid but its role is too low for the action. A 404 usually means a wrong UID.

The API is in transition, and a beginner should know the shape of it. Grafana 12 introduced a Kubernetes-style dashboard API under /apis/dashboard.grafana.app/..., and Grafana 13 deprecated the older /api routes in favour of /apis. The old routes are not being switched off and remain fully working, but they will no longer receive updates. The new endpoints take a body with metadata (including the UID as name) and spec (the dashboard JSON). For learning, scripts, and quick automation, /api is perfectly fine today; for long-lived tooling that you are building now, check the current API reference in the Sources and prefer /apis where it covers your need. One related change is firm: the old numeric-ID data source endpoints are disabled by default, so always use UIDs.

Plugins and the command line

Grafana ships with the common data sources and panels built in. A plugin adds anything else. There are three kinds: data source plugins (a new backend), panel plugins (a new visualisation), and app plugins (bundled pages and features). Plugins are signed: Grafana Labs signs its own, and community plugins can carry a community signature. By default Grafana refuses to load an unsigned plugin, which protects you from running unknown code inside the server. Note that backend plugins run as processes on the Grafana host, so install only what you trust.

Since Grafana 12, Angular plugins no longer load at all. If an old plugin logs a message about Angular not being supported, it must be rewritten by its author; there is no setting that brings it back.

The command line is the classic way to install plugins. Since Grafana 13 it is a subcommand of the single grafana binary:

grafana cli plugins list-remote
grafana cli plugins install grafana-clock-panel
grafana cli plugins ls
grafana cli plugins update grafana-clock-panel
grafana cli plugins remove grafana-clock-panel

list-remote lists what is available, install downloads a plugin, ls shows what is installed, and update and remove do what they say. Restart Grafana after changing plugins. The command works on the host that runs Grafana, because it reads and writes local files, and usually needs sudo. For a container, there is a neater way: set the environment variable GF_PLUGINS_PREINSTALL to a comma-separated list of id@version items, and Grafana installs them at start.

docker run -d -p 3000:3000 --name=grafana \
  -e GF_PLUGINS_PREINSTALL=grafana-clock-panel@1.0.1 \
  --volume grafana-storage:/var/lib/grafana \
  grafana/grafana-enterprise:13.2.3

The same command line also resets a forgotten admin password, as you saw earlier: grafana cli admin reset-admin-password 'NewPassword'. On a package install you may need to pass --homepath /usr/share/grafana so it finds its files.

Reading errors: the ones you will actually meet

Most early frustration with Grafana comes from a small set of failures. Learn to read them and most problems take a minute. Here are the common ones, grouped by where they appear. Always begin by asking two questions: did the request leave Grafana, and what did the other side say? The second answer is usually in the error text.

Data source errors

• connection refused or dial tcp ... connect: connection refused when testing a data source. The address is wrong from Grafana's point of view. In Docker, use the service name and port, not localhost. Confirm by running docker exec grafana wget -qO- http://prometheus:9090/-/healthy or an equivalent from inside the Grafana container.
• Bad Gateway in a panel. Grafana tried to reach the backend and could not. Same causes as above.
• client_error: client error: 401. The backend wants credentials. Add them in the data source's authentication section.
• 422 ... parse error. Your PromQL (or SQL) has a syntax mistake. The message points at the position.
• context deadline exceeded or Timeout exceeded. The query is too heavy. Shorten the time range, reduce the series, or raise the data source's query timeout.
• No data with no error. The query is valid but returned nothing: check the time range, the label values, and any variable selection.

Login and access errors

• Login failed or Invalid username or password. Wrong credentials, or the admin password from the environment was ignored because the database already existed. Use grafana cli admin reset-admin-password.
• Origin not allowed on login behind a proxy. Grafana compares the request's origin to root_url; set root_url to the public address and make the proxy forward the Host header.
• 401 Unauthorized or Invalid API key from the API. Use a service account token that starts with glsa_.
• 403 Forbidden from the API or the UI. The role is too low for that action.

Start-up and storage errors

• mkdir /var/lib/grafana: permission denied or unable to open database file. Grafana cannot write to its data directory. Fix ownership with chown -R grafana:grafana /var/lib/grafana, or in Docker run with a user that owns the bind mount (--user "$(id -u)"). The container's default user is 472, which matters for Kubernetes fsGroup.
• database is locked. You are using SQLite with concurrent writers. The fix is to move to PostgreSQL or MySQL, and never run two Grafana processes on one SQLite file.
• Dashboards vanish after restart. The data directory was not on a volume, so the container's database was thrown away with the container. Attach a volume at /var/lib/grafana.

Plugin and provisioning errors

• Plugin ... is not signed or an invalid-signature message. Grafana refuses unsigned plugins by default. Install a signed version; for a plugin you wrote yourself, allow it with [plugins] allow_loading_unsigned_plugins = plugin-id and understand that you are accepting the risk.
• Panel plugin not found. The plugin is not installed or did not load; install it and restart.
• YAML errors in provisioning such as yaml: line 7: did not find expected key. Indentation. YAML allows spaces only, never tabs, and a misaligned line shifts everything under it.
• the same UID is used more than once. Make every provisioned dashboard and data source UID unique.

Putting it all together

Now combine everything into one small project that you could hand to a colleague: a monitoring stack with Grafana and Prometheus that starts from nothing with one command, arrives with its data source and a dashboard already in place, and has an alert ready to use. You have already built every piece; this exercise is about seeing them as a whole.

The folder layout is the first design decision. Keep everything in one Git repository:

monitoring/
  docker-compose.yml
  prometheus.yml
  provisioning/
    datasources/
      prometheus.yaml
    dashboards/
      default.yaml
  dashboards/
    overview.json

Follow these steps:

1. Write the Compose file with Grafana (pinned to 13.2.3) and Prometheus (pinned to an exact tag you choose). Give both named volumes. Mount ./prometheus.yml into Prometheus, and mount ./provisioning and ./dashboards into Grafana (the dashboards at /var/lib/grafana/dashboards, matching the provider file).
2. Write prometheus.yml so Prometheus scrapes itself and Grafana, and set GF_METRICS_ENABLED=true on Grafana.
3. Provision the data source with a stable UID (prom-main), access: proxy, and editable: false.
4. Provision the dashboard provider, pointing at the mounted directory.
5. Export your dashboard from the UI, making sure panels use the prom-main UID and the dashboard has its own stable uid. Save it as dashboards/overview.json. It should contain the request-rate time series, the healthy-targets Stat, the table, and the job variable.
6. Take the admin password from the environment with GF_SECURITY_ADMIN_PASSWORD read from a .env file that you do not commit, so no secret sits in the repository.
7. Bring it up with docker compose up -d, then verify: curl -s http://localhost:3000/api/health shows "database": "ok"; the data source test is green; the dashboard opens with data.
8. Add the alert in the UI ("Scrape target down", one-minute pending period, severity=critical) and a webhook contact point, and test both.
9. Destroy and recreate with docker compose down -v followed by docker compose up -d. Everything you provisioned returns; only the alert rule and contact point, which you created by hand, do not.

That last observation is the key lesson, and a good way to finish: whatever you did by clicking is fragile, and whatever you put in a file survives. The natural next step is to provision the alerting resources too (the provisioning/alerting directory takes contact points, policies, and rule groups), or manage them with Terraform. The same instinct applies to the dashboards: the JSON in dashboards/ should be edited, reviewed, and committed like code, instead of being changed in the browser and forgotten.

For a service you would share, add a short checklist before calling it done:

What you can now do, and what comes next

You started without knowing what Grafana was, and you can now do real work with it. You can explain that Grafana queries other systems and stores only its own bookkeeping. You can name the five nouns (data source, query, panel, dashboard, alert rule) and say which layer to look at when a chart is wrong. You can install Grafana in a container or a package, verify it with the health endpoint, and read its logs. You can connect Prometheus and a SQL database, run PromQL and SQL in Explore, and build a dashboard with time series, stat, and table panels. You can add variables that make a dashboard reusable. You can build an alert that goes from rule through notification policy to a contact point, and you know why the message arrives after a short delay. You can configure the server with an ini file and environment variables, keep secrets out of files, put data sources and dashboards in version control with provisioning, call the API with a service account token, and install plugins without the removed commands. And you know the errors you will meet first and where each one shows up.

Some habits to carry forward. Pin versions and upgrade deliberately, reading the "what's new" page each time, since Grafana changes quickly. Put everything you can into files. Give Grafana read-only credentials, because every user of an organisation can query every data source in it. Give every panel a title and a unit. Alert on symptoms someone must act on, with a pending period, and always test the notification.

What the next levels cover:

• Mid-level goes under the hood of queries and dashboards: moving off SQLite to PostgreSQL, single sign-on, folder and team design, recording rules and alert quality, Grafana with Kubernetes and Helm, templated dashboards at scale, and Git Sync and Terraform workflows.
• Senior treats Grafana as a platform: high availability, the alerting cluster, security and secrets, multi-tenancy, upgrades and migrations (including the 12 to 13 checklist), cost control, and knowing when a different tool fits better.

Natural neighbours in this catalogue: the Prometheus guide for the metrics store behind most dashboards, the OpenTelemetry guide for producing the telemetry, the Docker guide and Kubernetes guide for running it all, Helm for installing Grafana on a cluster, and the Evidently guide for model-quality metrics that you can chart in Grafana.

Sources

• Grafana documentation home
• What's new in Grafana
• What's new in Grafana v13.0
• What's new in Grafana v13.1
• What's new in Grafana v13.2
• What's new in Grafana v12.0
• Grafana releases on GitHub
• Install Grafana
• Install on Debian or Ubuntu
• Install on RHEL or Fedora
• Run Grafana Docker image
• Install on macOS
• Install on Windows
• Configure Grafana
• Configure security
• Grafana CLI
• HTTP API reference
• Dashboard HTTP API
• Provision Grafana
• Service accounts
• Roles and permissions
• Grafana Alerting
• Alerting fundamentals
• Alert rule state and health
• Notification policies
• Git Sync
• Dashboard best practices
• Troubleshooting

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. The guide targets Grafana 13.2, so the answers below use current names and commands.

The thirty-second answer

Grafana is a web application that queries data where it already lives (Prometheus, Loki, SQL databases, cloud monitoring) and turns the results into dashboards and alerts. It is not a datastore: it keeps only its own metadata, such as dashboards, users, data source settings, and alert rules, in a small database of its own.

Then the line that shows you have used it: "Grafana separates storing from looking, so one time range and one set of dashboards cover metrics, logs, and SQL together, and if Grafana dies, no monitoring data is lost."

The model

What changed recently (say this and sound current)

Interviewers like candidates who know the tool has moved. The current release is Grafana 13.2 (patch 13.2.3 is the latest at the time of writing). The points that matter:

• Single binary. The stand-alone grafana-server and grafana-cli commands were removed in 13; use grafana server and grafana cli. The systemd unit is still called grafana-server.
• API direction. /api is deprecated in favour of the Kubernetes-style /apis. The old routes still work but will no longer be updated. Numeric-ID data source APIs are disabled by default, so use UIDs.
• API keys are gone. Service accounts and their tokens replaced them.
• Legacy removals. Angular plugins no longer load, legacy alerting is gone, and the Image Renderer plugin was removed in 13 in favour of a separate rendering service.
• Generally available in 13. Dynamic dashboards (rows, tabs, automatic layouts) and Git Sync (a Git repository kept in step with a folder).
• Helm chart repository moved. The Grafana chart now lives at grafana-community.github.io/helm-charts.
• Upgrade warning. A migration bug in 13.0.0 could lose or revert dashboards when upgrading from 12.x with Git Sync enabled; use the latest patch and back up first.

The commands to have ready

# Run it
docker run -d -p 3000:3000 --name=grafana \
  --volume grafana-storage:/var/lib/grafana \
  grafana/grafana-enterprise:13.2.3

# Is it healthy? (no login needed)
curl -s http://localhost:3000/api/health

# Reset a forgotten admin password
grafana cli admin reset-admin-password 'NewPassword'

# Plugins
grafana cli plugins install grafana-clock-panel
grafana cli plugins ls

# API with a service account token
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/search?query=overview
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/datasources

And the PromQL you should be able to write from memory:

up
rate(http_requests_total[$__rate_interval])
sum by (job) (rate(http_requests_total[$__rate_interval]))
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))

The first shows which targets are up. The second turns a counter into a per-second rate. The third totals that rate per job. The fourth estimates the 95th-percentile latency from a histogram; note that you sum the buckets by le before calling the function. The metric names are examples, since yours depend on what your applications export.

The classic traps and why they happen

Common interview questions

1. What is Grafana, and what is it not? Grafana is a visualisation and alerting front end that queries data sources and draws the results. It is not a database for metrics or logs: Prometheus, Loki, and SQL databases store the data. Grafana stores only its own metadata. A strong answer adds the consequence: losing Grafana's database loses dashboards and users but no telemetry, and losing Prometheus loses metrics but leaves dashboards intact.

2. Explain data source, query, panel, and dashboard. A data source is a saved connection to a backend. A query is the question asked of it, in its own language. A panel is one visualisation built from queries, optional transformations, and display options. A dashboard is a saved collection of panels that share a time range and variables. I would mention that every data source and dashboard has a UID, and that automation should use UIDs.

3. How do you install Grafana and check that it works? For a quick start I use Docker with a pinned tag and a named volume at /var/lib/grafana. On Linux I use the official APT or RPM repository and enable the grafana-server service. To check, I call /api/health, which needs no login and returns the database status and version, open port 3000 in a browser, and read the logs. The first login is admin/admin, and it forces a password change.

4. What database does Grafana use? SQLite by default, stored in the data directory, which is fine for learning or a single user. For shared or important installations it supports MySQL 8.0 or later and PostgreSQL 12 or later. SQLite cannot be shared between several Grafana replicas, and concurrent writes produce "database is locked" errors.

5. A data source test fails with "connection refused". What do you do? I remember that the Grafana server makes the request, not the browser, so I check what address is reachable from where Grafana runs. In Docker I use the service name. I confirm from inside the container with a command such as docker exec grafana wget -qO- http://prometheus:9090/-/healthy. I also check that both containers are on the same network and that the port is right. If instead I see a 401, it is an authentication problem, and a timeout suggests a firewall.

6. What is the difference between Explore and a dashboard? Explore is for ad hoc questions: I run a query, look at it, and change it, with no saving. A dashboard is the durable answer to a question I already know how to ask. I work out the query in Explore first, then move it into a panel.

7. Why use $__rate_interval in a rate() query instead of a fixed [5m]? The panel's step changes with the time range. If the window is too small for the step or the scrape interval, rate sees too few samples and the graph gets gaps. $__rate_interval is computed from the step and the data source's scrape interval so the window is always safe. It is one of the small habits that separates a working dashboard from a flaky one.

8. How do variables work, and what goes wrong with them? A variable is defined in dashboard settings and substituted into queries as $name. A query variable fills its options from a data source, for example label_values(up, job). The usual failures are using = where a multi-value variable needs =~, variables that depend on one defined below them, and options that come back empty because no data exists in the selected time range.

9. What are transformations, and when do you use them instead of the query? A transformation reshapes the result on the Grafana side: rename or hide columns, join two queries, group, calculate a field. They run as a pipeline where each step sees the previous step's output, so order matters. If the data source can do the work (a PromQL aggregation, a SQL GROUP BY), do it there, because it moves less data. I use transformations when I need to combine different data sources or the source cannot express the shape I need.

10. Walk me through how an alert reaches a person. A rule evaluates a query and expressions on its group's interval. If the condition holds for the pending period, the instance becomes Alerting. Grafana then runs the alert's labels through the notification policy tree, which picks a contact point, waits the group wait (30 seconds by default) to batch related alerts, sends the message, and repeats it at the repeat interval (4 hours by default) while it still fires. Silences and mute timings suppress the notification but not the evaluation.

11. What happens when an alert query returns no data or errors? By default Grafana creates synthetic alerts called DatasourceNoData and DatasourceError, carrying the rule name and the data source UID, so a broken data source does not look like a healthy system. Per rule you can instead treat it as Normal, as Alerting, or keep the last state. I choose carefully: treating no data as Normal hides failed exporters.

12. How do you stop alerts from flapping? The pending period is the main tool: require the condition to hold for several minutes before firing. Beyond that I alert on symptoms a person must act on, use sensible thresholds, group notifications by label, and use silences for planned maintenance.

13. How do you manage Grafana configuration as code? Provisioning: YAML files under /etc/grafana/provisioning for data sources, dashboard providers, and alerting, plus dashboard JSON files in a directory the provider watches. I always set explicit UIDs so dashboards and data sources line up across environments. Alternatives are the Terraform provider, Git Sync (generally available since 13), the Helm chart sidecars, and the Grafana Operator. Secrets come from environment variables or files, not from the committed YAML.

14. How do you configure Grafana, and which setting wins? An ini file (grafana.ini), with every key also settable through a GF_<SECTION>_<KEY> environment variable. The precedence is defaults, then the ini file, then environment variables, then command-line overrides. The default file should never be edited. Values can be pulled from the environment with $__env{} or from a file with $__file{}.

15. How do you call the API securely? With a service account token in an Authorization: Bearer header. A service account is a non-human identity with an organisation role, and its tokens start with glsa_, can expire, and can be revoked individually. The token is shown once at creation. I avoid basic authentication in automation, and I do not use API keys, which service accounts replaced.

16. What are organisations, folders, and teams? An organisation is a hard tenancy boundary: data sources, dashboards, and service accounts belong to exactly one. A folder is the main permission boundary inside an organisation. A team is a group of users that I grant folder permissions to in one step. Fine-grained role-based access control with custom roles is an Enterprise and Cloud feature.

17. Is it safe to give everyone Viewer access? Viewer stops people from changing things, not from querying. Any user in an organisation can issue queries against any data source in it. So the data source credentials must be least-privilege. Anonymous access exposes every data source to anyone who can reach the URL and should be off except on trusted networks. Grafana has no built-in multi-factor authentication, so that comes from the identity provider.

18. What changed in Grafana 13 that you would warn a colleague about? The removal of the stand-alone grafana-server and grafana-cli commands, the deprecation of /api in favour of /apis, the disabled numeric-ID data source APIs, the removal of the Image Renderer plugin (rendering now needs the separate remote service), and the 13.0.0 migration bug with Git Sync. The safe routine is to back up the database, read the what's-new page, use the latest patch, and test in staging.

19. Where would you use Grafana in an ML system? Serving metrics (latency, error rate, throughput) from Prometheus, GPU and training metrics, prediction or evaluation tables charted from a SQL warehouse, logs from Loki, and data-freshness checks. Model-quality metrics from a tool like Evidently can be exported and charted beside the infrastructure metrics. For teams with data-residency rules in the Gulf or Egypt, Grafana is ordinary software that can be hosted in the same region as the data.

20. When would you not choose Grafana? When the job is storing or collecting data (use Prometheus, Loki, or a warehouse), when the need is deep model-quality analysis (a dedicated tool fits better), or when a team needs a full business-intelligence product with governed semantic models. Grafana shines at operational, time-oriented views.

21. How do plugins work, and how do you install one safely? A plugin is an extension: a data source, a panel, or an app. Plugins are signed, and Grafana refuses unsigned ones unless you allow them by ID, which I would only do for code I wrote. I install with grafana cli plugins install <id> and restart, or in containers I set GF_PLUGINS_PREINSTALL so the list is reproducible. Angular plugins no longer load in current versions, so an old plugin that depends on Angular has to be replaced.

22. What goes in Grafana's data directory, and what must you back up? The database (SQLite by default) holds dashboards, users, data source definitions, and alert rules. I back up that database, the configuration file, and the secret_key setting, because the key encrypts stored secrets and losing it makes them unreadable. If everything is provisioned from files in Git, the backup matters less, which is another argument for configuration as code.

23. Your dashboard is slow. Where do you look first? At the queries, because Grafana itself is rarely the bottleneck. The Query inspector shows how long each query took and what was sent. I look for a time range that is too wide, too many series, a missing aggregation, a refresh interval that is too fast, and too many panels loading at once. Fixes are aggregating in the query, recording rules in the backend, a longer refresh interval, and fewer panels.

24. How would you show a deployment on a latency graph? With an annotation: a vertical marker at the deployment time. Annotations can be added by hand, or come from a query so that every deployment recorded in a database or logged event appears automatically on every time series panel.

What a strong answer adds

Sixty-second self-test

Answer out loud, without looking back.

If you hesitated on more than two, reread the matching section of the detailed guide. For the next step, see the Prometheus guide for the data behind most dashboards and the Docker guide for the container basics used here.

Part one of three. Most beginner trouble with Grafana comes from four confusions: thinking Grafana stores the data, forgetting that the Grafana server (not your browser) makes the requests, treating clicked configuration as permanent, and trusting an old tutorial. Start with the table, then the longer explanations, the habits, and the checklist. Everything here is written for Grafana 13.2.

Common errors at this level

Eight of those deserve the longer explanations that follow.

Mistake 1: using localhost in a data source URL

Symptom. You add a Prometheus data source, type http://localhost:9090, press Save and test, and get connection refused or Bad Gateway, although Prometheus opens fine in your browser.

Cause. The address is used by the Grafana process, not your browser. When both run in containers, each container has its own localhost, and in Grafana's there is nothing on port 9090. The same applies to a Grafana on a server whose database lives on another machine: what matters is what the server can reach.

Fix. Use the name the network gives the other container, for example http://prometheus:9090, and make sure both share a Compose network (Compose puts services in one by default). To prove it, run a request from inside the Grafana container:

docker exec grafana wget -qO- http://prometheus:9090/-/healthy

If that prints a healthy message, the address works. If it fails, the problem is the network, not Grafana.

Mistake 2: losing everything when the container is recreated

Symptom. You upgrade Grafana by removing and recreating the container, and all your dashboards, users, and data sources are gone.

Cause. Grafana keeps its database in /var/lib/grafana. A container's own filesystem disappears when the container is removed, so a database there goes with it.

Fix. Attach a named volume, and never use docker compose down -v casually, because -v deletes the volumes too.

services:
  grafana:
    image: grafana/grafana-enterprise:13.2.3
    volumes:
      - grafana_storage:/var/lib/grafana

volumes:
  grafana_storage: {}

Better still, provision your data sources and dashboards from files, so a lost volume costs you only users and preferences, not your work.

Mistake 3: editing the password setting and expecting a change

Symptom. You set GF_SECURITY_ADMIN_PASSWORD to something new, restart, and the old password still works (or neither does).

Cause. The setting seeds the admin account only when Grafana creates its database for the first time. After that the password is stored in the database, and the setting is ignored.

Fix. Reset it with the command line on the host that runs Grafana:

grafana cli admin reset-admin-password 'a-long-new-password'

For a container, run it with docker exec grafana grafana cli admin reset-admin-password '...'. On a package install you may need to pass --homepath /usr/share/grafana. Do this once, then sign in and never share the admin account: create named users, or use single sign-on.

Mistake 4: multi-value variables and the wrong operator

Symptom. A dashboard works with one value selected in a dropdown and shows "No data" when you select two, or "All".

Cause. Selecting several values expands the variable into a pattern such as (api|worker). With a PromQL label matcher job="$job" the query asks for a label value equal to that whole text, which never exists. The regex operator =~ treats it as a pattern.

Fix.

sum by (handler) (rate(http_requests_total{job=~"$job"}[$__rate_interval]))

For SQL, a multi-value variable needs a list, so use IN ($job) and, if the values are strings, the format ${job:singlequote}. When a variable behaves oddly, open the dashboard settings, find the variable, and read its preview table: if the options are empty, the variable's own query is the problem, and often the cause is a time range that contains no data.

Mistake 5: alerts that nobody can act on

Symptom. The channel fills with alerts, people mute it, and a real incident goes unnoticed. Or an alert fires for ten seconds and recovers, again and again.

Cause. Rules with no pending period fire on every blip. Alerts on causes (CPU at 80 percent) rather than symptoms (users see errors) add noise. Annotations that say only "High CPU" leave the reader guessing. And a broken data source that is treated as "Normal" creates the opposite problem: silence while the system is down.

Fix. Set a pending period of a few minutes on most rules. Start from the symptoms of a service: Rate, Errors, and Duration (the RED method that Grafana's own best-practice guidance points to). Write annotations as if for someone who was just woken:

summary:     Error rate above 5% on {{ $labels.job }}
description: The share of failing requests has been above 5% for 5 minutes.
runbook_url: https://wiki.example.com/runbooks/model-api-errors

Leave "no data" handling at its default (a DatasourceNoData alert) unless you have a reason. Group notifications by alertname so a burst becomes one message. Use a silence during planned maintenance instead of deleting rules.

Mistake 6: following an old tutorial

Symptom. A command is not found, a menu item does not exist, or a plugin refuses to load.

Cause. Grafana changes fast. Recent releases removed or replaced several things that older material still teaches.

Fix. Translate the old step to the current one:

When in doubt, check the version at the bottom of the help menu or on Administration, then General, then Stats and licensing, and read the "what's new" page for that release.

Mistake 7: expecting the provisioned file and the UI to agree

Symptom. You change a provisioned dashboard in the browser, and the change vanishes after a while, or Grafana refuses to save it at all.

Cause. A provisioned dashboard belongs to its file. With allowUiUpdates: false, the UI will not save it, and even with updates allowed, the file is the source of truth and is reapplied. People then "fix" the problem by editing in two places and lose work.

Fix. Decide where the truth lives. For a shared service it is the file in Git: export the JSON from the UI, edit it in the repository, and let Grafana reload it within the provider's update interval. For experiments, use Save as to make an unprovisioned copy in a scratch folder, and only move the result into the repository when it is ready.

Mistake 8: alerting on a shape the rule cannot read

Symptom. A rule shows an error state, and you get a DatasourceError alert, with a message like input data must be a wide series but got type long.

Cause. Alert expressions expect time series. A query that returns a table, such as a SQL query without a time column, arrives in the wrong shape.

Fix. Set the query format to Time series, return a time column and one numeric column, and add a Reduce expression before the Threshold. Test the query in the rule editor's preview before saving, and read the preview's per-series values to confirm the number you expect is the number the rule sees.

Habits worth adopting on day one

These cost almost nothing at the start and are painful to add later.

Pin versions. Write the exact tag in every Compose file and Helm value. When you upgrade, do it on purpose, after reading the release notes and backing up the database. Patches arrive roughly every two weeks and are often security fixes, so "pinned" means "pinned and periodically updated", not "forgotten".

Explore before you build. Write and test a query in Explore, look at the series and labels it returns, and only then put it in a panel. Half of all "my panel is wrong" problems are query problems that Explore would have shown in seconds.

Give everything a title, a unit, and a sensible scale. A panel called "Panel 3" with a bare number helps nobody. Set the unit, set the decimals, and use thresholds so colour carries meaning.

Use $__rate_interval and variables from the start. Hard-coded windows and hard-coded hostnames are what make dashboards stop working when you zoom out or add a second environment.

One dashboard, one purpose. A dashboard that tries to show everything shows nothing. Start with a top-level overview that says healthy or not, and link to detail dashboards. Match the method to the target: RED for services, USE (utilisation, saturation, errors) for machines.

Organise in folders and grant permissions per folder. Per-dashboard permissions do not scale. Name folders after teams or systems, and grant the team access once.

Keep dashboards in files. Export the JSON, give it a stable uid, provision it, and commit it. Then a rebuilt server, a second environment, and a code review all work without effort. Put data sources in provisioning files too, with stable UIDs.

Secrets never go in files you commit. Use $__env{} or $__file{} in the ini file, environment interpolation in provisioning files, and a secret store for tokens. Service account tokens get an expiry and are shown only once, so store them when you create them.

Give Grafana read-only credentials. Every user of an organisation can query every data source in it. A database user that can only SELECT from chosen tables, and a backend token that can only read, limit what a mistake or a curious colleague can do.

Keep anonymous access and self-sign-up off. Both are off by default. Leave them that way unless you understand exactly who can reach the URL.

Test the whole alert path once. From a broken data source, through the pending period and the notification policy, to a message in a real channel. Write down the delay you measured.

Annotate events. Deployments, retraining runs, and incidents drawn as vertical lines on your graphs answer "what changed" faster than any query.

Practice cards

A few diagnostic recipes

When something is wrong, a fixed order saves time. These recipes cover the situations beginners meet most.

A panel is empty. First open the panel's Inspect menu and the Query tab to see the request and the raw response. If the response has data, the problem is on the display side (transformations, field options). If it is empty, open the same query in Explore and widen the time range. If it is still empty, run the query directly against the backend (for Prometheus, its own expression browser) to see whether the data exists at all.

Grafana will not start. Read the last lines of the log: docker logs grafana or journalctl -u grafana-server. The two usual culprits are permissions on /var/lib/grafana (fix ownership, or set the user) and a syntax error in the ini file or a provisioning file (check indentation and that no tabs slipped in). If the health endpoint reports "database": "failing", Grafana is running but cannot reach its database.

A provisioned change does not show. Dashboard files are rescanned every updateIntervalSeconds, so wait that long first. Then check the log for the provider name, confirm the file is valid JSON, and confirm the mounted path matches the provider's path. A file that is not readable by the Grafana user is silently absent from the list.

A login loop or a blank page behind a proxy. Compare [server] root_url with the address in the browser; they must match, including the scheme. If Grafana is served from a sub-path, serve_from_sub_path must be on and root_url must include the path. Make sure the proxy forwards the Host header.

The API returns 401 or 403. A 401 means the credential is missing, wrong, or expired: check for the glsa_ prefix and the Bearer word. A 403 means the credential is valid but the service account's role is too low for that call.

The checklist

Tick these before you call a setup "done":

This is part two of three. It picks up where Beginner ended and assumes you already use Grafana at work: you can open a dashboard, edit a panel, and have probably been handed a pager that fires from an alert somebody else wrote. The goal here is to make you the person who can explain why a panel is slow, why an alert did not fire, and how to keep fifty dashboards from drifting into fifty hand-edited snowflakes. The version throughout is Grafana 13.2 (latest patch 13.2.3, published 29 September 2026), and where 13.x changed something you may have learned on 11 or 12, the change is called out.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run against a development instance in a few minutes. If you do not have one, the Docker command from the beginner level is enough; pin a tag rather than taking the moving default.

How a query travels through Grafana

Most Grafana confusion comes from picturing the browser talking to Prometheus. It does not. When you open a dashboard, your browser sends a request to Grafana's own backend, and the backend talks to the data source. This is the "Server" access mode, and it is why a data source URL such as http://prometheus:9090 only has to be reachable from the Grafana server, never from your laptop. It also explains the classic failure where "Save & test" works and a dashboard still breaks: the test hits one health endpoint, while a real panel sends a real query with a real time range.

The path has five stages, and being able to name them lets you place any symptom.

A query is sent to the data source, and the answer comes back as one or more data frames. A data frame is a columnar table with typed fields: a time field, one or more number or string fields, and labels attached to each value field. A single Prometheus query returning twelve series becomes twelve value fields (the "wide" shape) or one long table with a label column (the "long" shape), depending on the data source and query format. You rarely need to think about frames until something mismatches, and then the mismatch is the whole bug. The alerting error input data must be a wide series but got type long is exactly this: a query returned the long shape and a server-side expression needed the wide one.

Two kinds of post-processing exist and they run in different places. Expressions (Math, Reduce, Resample, Threshold, Classic condition, and SQL expressions, which are generally available from 13.0) run on the Grafana server after the data source responds. Alert rules are built out of them. Transformations run in the browser, on panel data only, and they are the subject of a later section. A common mistake is expecting a transformation to affect an alert. It cannot: the alert evaluator never sees your browser's pipeline.

Reading the query inspector

The fastest way to stop guessing is the panel Query inspector (panel menu, Inspect, Query). It shows the exact request Grafana sent after variable interpolation and interval calculation, along with the raw response and timing. Three tabs matter. Query shows the final expression and the step the data source was asked to use. Data shows the frames after transformations, with an option to view the raw data before them. Panel JSON shows the complete definition, which is what you should copy when you ask a colleague for help.

# Reproduce what the inspector shows: the same query the panel sent,
# but from the Grafana host, to separate "Grafana problem" from "network/data source problem".
curl -sG 'http://prometheus:9090/api/v1/query_range' \
  --data-urlencode 'query=sum by (job) (rate(http_requests_total[5m]))' \
  --data-urlencode 'start=2026-10-01T08:00:00Z' \
  --data-urlencode 'end=2026-10-01T09:00:00Z' \
  --data-urlencode 'step=30' | head -c 400

If this returns data and the panel does not, the fault is in Grafana's configuration of the data source (authentication header, tenant header, URL), in the interpolated query, or in a transformation. If it is slow here, the data source is the problem and no amount of dashboard tuning will fix it.

Making queries behave: intervals, data points and time ranges

A panel showing thirty days of data cannot draw thirty days of fifteen-second samples. That would be 172,800 points per series, and a screen is about 1,500 pixels wide. Grafana therefore decides a step (also called the interval) for every query, and understanding that calculation removes a large class of "my graph changes shape when I zoom" confusion.

The step is roughly the time range divided by the max data points setting, which defaults to the panel's pixel width. It is rounded to a sensible value and never goes below the min interval, which you can set on the data source (for Prometheus, the scrape interval field) or override per panel. The result is exposed to queries as variables you should learn to use instead of hard-coding durations.

The entry to internalise is $__rate_interval. A rate() needs at least two samples inside its window, so a hard-coded [1m] window over a 30-second scrape interval works until someone zooms out and the step grows beyond the window, and the graph develops gaps. $__rate_interval picks a window that stays valid at every zoom level, based on the scrape interval configured on the data source. That makes the scrape interval field on the Prometheus data source a real setting: if it says 15s and the target is actually scraped every minute, the window is too short and rates go missing.

# Hard-coded window: breaks at some zoom levels
rate(http_requests_total{job="api"}[1m])

# Zoom-safe
rate(http_requests_total{job="api"}[$__rate_interval])

# A latency percentile, the standard shape
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[$__rate_interval]))
)

Two traps live in that last query. The sum by (le) must stay inside histogram_quantile, because quantiles of already-averaged series are meaningless. And averaging percentiles across panels or instances afterwards is also meaningless; compute the quantile from the summed buckets. The PromQL depth lives in the Prometheus guide; the point here is that Grafana's job is to give the query the right window.

Query options worth knowing

Open the Query options button above the queries. Max data points lowers the resolution of one expensive panel without touching the others. Min interval stops a panel from asking for a finer step than the data has. Relative time and Time shift let one panel show a different window from the dashboard, which is how you build "now versus the same hour last week" comparisons. In the Mixed data source you can send different queries in one panel to different data sources, for instance a Prometheus metric next to a SQL deployment-events table.

Finally, query caching (Enterprise and Cloud) can absorb identical repeated queries. It helps the "everyone opens the incident dashboard at once" case, but it is a mitigation. The durable fixes are recording rules in Prometheus or Mimir for expensive expressions, fewer panels per dashboard, and a coarser step where precision is not needed.

Variables that make one dashboard serve many

Without variables, teams copy a dashboard for each cluster, each service and each environment, and within a quarter the copies disagree. Template variables replace that copying: one dashboard, a dropdown at the top, and every query interpolates the current choice. You have used simple ones; this section is about the ones that stay correct when the data gets large.

Variable types and when each is right

A query variable asks a data source for its values, for example all values of the namespace label. A custom variable is a fixed list you type in. A constant is hidden and exists so you can change a value in one place. An interval variable offers steps such as 1m,5m,1h. A data source variable lets the viewer switch between data sources of one type, which is how a single dashboard serves dev, staging and production Prometheus instances. A text box takes free input. Ad hoc filters (and the newer group-by control, generally available from 13.1) let viewers add label filters that apply to every query from a supported data source without you writing a variable per label.

Grafana 13.1 made multi-property variables generally available: one variable can carry several fields for each option, for example both a human-readable name and an ID, so a dropdown can show "Payments API" while the query uses svc-4f2a. Before this you built two variables and kept them aligned by hand. Section-level variables, which scope a variable to a row or tab, arrived in public preview with 13.0 and became generally available in 13.1.

Interpolation and formats

A variable is referenced as $name or ${name}. The braces form is required when the variable is followed by text, and it is the only one that accepts a format: ${name:format}. The legacy [[name]] syntax still parses but should not appear in anything you write today.

Formats matter because different data sources quote differently. A multi-value variable holding api and web interpolates in PromQL as a regex alternation by default, which is what you want for =~ matchers, but in SQL you need quoted, comma-separated values.

# PromQL: multi-value becomes (api|web) automatically
sum by (job) (rate(http_requests_total{job=~"$job"}[$__rate_interval]))

# SQL: ask for single quotes explicitly
SELECT ts, value FROM requests WHERE service IN (${service:singlequote})

# Show the variable in a link as a query string
/d/abc123/service-overview?var-service=${service:queryparam}

The formats you will reach for are csv, pipe, regex, singlequote, doublequote, sqlstring, lucene, json, queryparam, text and raw. When a query "works with one value and breaks with two", the format is almost always the cause, and the query inspector shows you the interpolated text so you can see the quoting.

Chained variables and refresh behaviour

A query variable can reference another variable, so a cluster variable feeds a namespace variable feeds a pod variable. The second query uses the first as a filter, for instance label_values(kube_pod_info{cluster="$cluster"}, namespace). The variable's refresh setting decides when it re-queries: "On dashboard load" is cheap and right for slowly changing things, "On time range change" is needed when available values depend on the time window, but it re-runs the variable queries on every zoom. Use the Regex field to trim noisy values, and Sort to keep the dropdown predictable.

Dashboards can also repeat a panel or row for each value of a multi-value variable. A row per namespace is a legitimate use. Repeating thirty panels by fifty pods is how dashboards become unusable, so prefer a table with sparklines. The time-series-to-table transformation, generally available from 13.1, exists for exactly this.

Transformations and field configuration, in the right order

Raw query results are rarely the table you want to show. A service owner wants one row per service with error rate, p95 latency and request count side by side, but those come from three separate queries. Transformations reshape the data in the browser, after the queries and before the visualization. The rule that explains nearly every surprise is that each transformation operates on the output of the one above it, not on the raw data. Reorder them and you get a different result, or none.

A typical "service table" pipeline is worth memorising because the same shape recurs:

1. Join on a shared fieldCombine query A (error rate), B (p95) and C (request count) into one frame, joining on the job field. Use the join by labels or join by field transformation depending on whether your identifying value is a label or a column.
2. Calculate fieldAdd a derived column, such as error rate multiplied by request count for "errors per second".
3. Filter data by valueDrop rows that do not matter, for example services with fewer than one request per second.
4. Organize fieldsRename columns, hide the time field, and reorder. Cosmetic work goes last, so the steps above still see the original names.

If the join produces empty rows, inspect the data between transformations: each step in the Transform tab has a debug toggle that shows that step's output. There is also a performance fact hidden here. Transformations run in the viewer's browser, so a transformation over a hundred thousand rows costs CPU on every refresh for every viewer. If a join is expensive, push it to the data source where possible, with a SQL JOIN, a PromQL on() match, or a recording rule.

Field config: standard options and overrides

Separate from transformations is the field config, which decides how each field is presented. Standard options apply to every field: unit, decimals, min and max, thresholds, value mappings, data links, color scheme and display name. Overrides apply a different setting to specific fields, matched by name, by regex, by field type or by the query that produced them.

Three habits remove most dashboard sloppiness. Always set the unit, because a number with no unit forces every reader to guess whether 0.83 means a fraction, seconds or a percentage. Use thresholds to colour by meaning, and set the base colour deliberately. Use value mappings to turn opaque codes into words, so 1 and 0 become Up and Down. Overrides answer "I need this one series on a right-hand axis in another unit": add a Fields with name override and set the axis placement and unit. The override is stored on the panel, so it survives copying and travels with the JSON.

Dashboard design that survives a team

A dashboard is a product with users, and most production dashboards fail as products: they answer no question, they hold forty unrelated panels, and nobody knows which copy is current. Grafana's own best-practices guidance describes a maturity ladder. At low maturity, dashboards are created ad hoc, copied and never pruned. At medium, template variables and a deliberate hierarchy exist. At high, dashboards are generated or managed as code, sprawl is actively reduced, and a separate environment exists to test changes before they reach people on call.

The standard methods give every dashboard a reason to exist:

• RED for request-driven services: Rate of requests, Errors, and Duration (latency). Grafana's guidance is that alerting should be done on RED dashboards, meaning the signals that describe user-visible behaviour.
• USE for infrastructure: Utilisation, Saturation and Errors for each resource such as CPU, memory, disks and network.
• The four golden signals (latency, traffic, errors, saturation) combine both views for a service.

A hierarchy that matches how people investigate

An investigator starts broad and narrows. Design three layers and link them. An overview dashboard for each team or product shows RED for each service on one screen. A service dashboard, with a service variable, shows the detail behind one overview row. A drill-down layer (logs, traces, profiles) is reached by data links that carry the time range and current variables through the URL. Grafana's Drilldown apps for Metrics, Logs and Traces, generally available since 12.0, provide queryless exploration when you do not yet know what to ask.

Practical rules that keep the hierarchy honest: one question per dashboard, stated in its title and description; the answer at the top, as a row of Stat panels with thresholds; the same variable names across dashboards so $cluster and $namespace carry through links; obsolete dashboards deleted, because on-call engineers find them and trust them; and dashboard JSON kept in version control. Library panels define a panel once for reuse across dashboards. The trade-off is coupling: editing the library panel changes every dashboard that uses it, which is what you want for a corporate standard and a hazard for an experiment.

Dynamic dashboards in 13

Grafana 13.0 made dynamic dashboards generally available. They use a newer dashboard schema with richer layouts: rows, tabs, auto-grid layouts, conditional rendering, and section-level variables. Existing dashboards are migrated automatically when opened. Two consequences matter for operators. First, once a dashboard is adopted into the new schema you cannot downgrade it, so this is a forward-only change that belongs in your upgrade plan. Second, tooling that reads dashboard JSON needs to understand both shapes, which is part of why the newer /apis dashboard endpoints exist.

Configuration: files, environment variables and precedence

Grafana is configured by an ini file, and once you run it in containers you mostly configure it with environment variables. Knowing the rules exactly saves you from the most common production mistake: editing a file that is being ignored.

Packages read /etc/grafana/grafana.ini. The shipped defaults live in conf/defaults.ini inside the install and are never edited. Homebrew on Apple Silicon reads /opt/homebrew/etc/grafana/grafana.ini. Comments begin with ; or #. Precedence, lowest to highest, is: defaults, then the ini file, then environment variables, then command-line cfg: arguments. An environment variable therefore silently beats the same key in your file, which is a real source of "I changed the file and nothing happened".

The environment variable name is GF_ followed by the section and key, uppercased, with dots and dashes turned into underscores. [auth.google] client_secret becomes GF_AUTH_GOOGLE_CLIENT_SECRET, and [server] root_url becomes GF_SERVER_ROOT_URL. Secrets should not sit in plain text in either form, so Grafana supports value expanders inside ini values:

[database]
type = postgres
host = pg.internal:5432
name = grafana
user = grafana
# Read the password from a mounted Kubernetes or Docker secret file.
# A trailing newline in the file is trimmed.
password = $__file{/run/secrets/grafana_db_password}
ssl_mode = require

[security]
admin_user = admin
admin_password = $__file{/run/secrets/grafana_admin_password}
secret_key = $__file{/run/secrets/grafana_secret_key}
cookie_secure = true

[server]
root_url = https://grafana.example.com/

$__env{NAME} reads an environment variable, $__file{/path} reads a file, and $__vault{...} reads HashiCorp Vault but is Enterprise only. The file form is the natural fit for Kubernetes secret volumes; the Vault guide covers the source side.

The keys that matter at this level

Beyond the beginner keys, these are the ones a person running Grafana at work actually edits:

• [server] root_url and serve_from_sub_path. Behind a reverse proxy, root_url must be the public URL, including any sub-path. A wrong value is behind the Origin not allowed login error and redirect loops. enable_gzip defaults to true from 13, so turn compression off at the proxy to avoid compressing twice.
• [database]. type, host, name, user, password, ssl_mode, and the pool settings max_open_conn, max_idle_conn and conn_max_lifetime.
• [auth]. login_maximum_inactive_lifetime_duration (seven days by default), token_rotation_interval_minutes (ten), and disable_login_form once SSO works.
• [users]. allow_sign_up should stay false, and auto_assign_org_role decides what a new login can do.
• [log]. level and mode, plus filters for raising one subsystem, for example filters = sqlstore:debug.
• [metrics] enabled = true, which exposes /metrics for Prometheus to scrape.
• [plugins]. allow_loading_unsigned_plugins, plugin_admin_enabled and preinstall.
• [feature_toggles] enable = a,b. Toggles in experimental or preview stages carry no support guarantee, so prefer generally available ones. Since 13.0, dashboardNewLayouts, sqlExpressions and others are on by default and need no toggle.

The simplest verification that the running process sees what you think it sees is the Administration, General, Settings page in the UI for server admins, which shows the effective merged values, together with the debug-level log output.

A real database, and state you can lose

By default Grafana stores its state in SQLite, a single file under /var/lib/grafana. That is fine for evaluation and a single developer's instance. It is not acceptable for a shared instance, for three reasons that show up in this order. SQLite allows one writer at a time, so provisioning plus several users editing dashboards can produce database is locked. You cannot run two replicas against one SQLite file. And the file is a single point of loss, because in a container with no persistent volume every restart silently returns you to an empty Grafana.

Production therefore uses PostgreSQL 12 or later or MySQL 8.0 or later. The [database] section points there, and Grafana creates and migrates its own schema at startup, which requires a database user with DDL privileges. Everything you configured through the UI (dashboards, users, organizations, data source definitions, alert rules and their state) lives in that database, and so does the encrypted copy of your secrets. Two things must therefore be backed up together: the database, and the [security] secret_key. Without the key, the encrypted secrets in a restored database cannot be decrypted, and every data source password and contact point credential is lost.

The discipline that follows is to treat the database as a cache of your configuration, not as its source of truth. If dashboards, data sources and alerting are all in Git and applied by provisioning, Terraform or Git Sync, the database can be rebuilt. If they exist only in the UI, the database is the only copy, and its backup is your disaster recovery plan. The rest of this guide is largely about moving from the second situation to the first.

Provisioning: Grafana configured by files

Provisioning is Grafana's native configuration-as-code. At startup it reads YAML files from a directory and makes the running instance match them. The default directory is /etc/grafana/provisioning, with one subdirectory per resource kind: datasources, dashboards, plugins and alerting. The problem it solves is repeatability: a new environment, a rebuilt pod or a disaster recovery comes up with the right data sources and dashboards without anyone clicking.

apiVersion: 1
prune: true                     # delete provisioned sources that vanish from this file
datasources:
  - name: Prometheus
    uid: prom-main              # always set a stable uid
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s         # the scrape interval used by $__rate_interval
    secureJsonData:
      httpHeaderValue1: $PROM_TOKEN
    version: 1
deleteDatasources:
  - name: OldPrometheus
    orgId: 1

Every field carries a lesson. uid is the identity everything else refers to: dashboards, alert rules and correlations reference the data source by UID. If you omit it, Grafana generates one, your dashboards break when you rebuild, and since 13.0 the numeric-ID based data source APIs are disabled by default, so UID is the only stable handle. timeInterval is the Prometheus scrape interval field discussed earlier. secureJsonData holds secrets, and the $PROM_TOKEN value is expanded from the environment, so the file itself contains no secret. Environment interpolation applies to values and not to keys, and a literal dollar sign is written $$. editable: false stops someone changing the source in the UI and being surprised by a revert. prune: true makes the file authoritative.

A dashboard provider tells Grafana to load dashboard JSON files from a path and keep them synchronised:

apiVersion: 1
providers:
  - name: team-dashboards
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    allowUiUpdates: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true   # directory names become folders, up to 4 levels

With allowUiUpdates: false, a person who edits a provisioned dashboard can experiment but cannot save over it, and Grafana answers a save attempt with Cannot save provisioned dashboard. That is the correct default, because otherwise the UI edit and the file fight. The intended workflow is to edit in the UI, export the JSON (Export, Export as JSON), commit it, and let provisioning apply it. Dashboards carry their own uid. Two files with the same UID produce the same UID is used more than once or dashboard with the same uid already exists in folder, so never copy a file and forget to change it.

Alert rules, contact points, notification policies, mute timings and message templates can all be provisioned from provisioning/alerting/*.yaml, through the provisioning HTTP API under /api/v1/provisioning, or through Terraform. Choose one mechanism per resource type and stay with it, because mixing provisioning with UI edits produces the same fight as with dashboards.

Dashboards and alerts as code: Terraform, Git Sync and the API

Provisioning files solve the "rebuild the instance" case. Teams then need a way for many people to propose dashboard changes, a way to apply them to several instances, and a way to manage resources that files do not cover well. Grafana offers three routes, and the right choice depends on who edits and where the source of truth lives.

Terraform. The Grafana provider (grafana/grafana) manages folders, data sources, dashboards, alert rule groups, contact points and notification policies as resources such as grafana_folder, grafana_data_source, grafana_dashboard, grafana_rule_group, grafana_contact_point and grafana_notification_policy. It authenticates with a service account token. The benefit is the workflow you already have: plan in a pull request, apply on merge, and drift is detectable. The cost is that dashboard JSON becomes a string inside HCL, so the common pattern is file() reading JSON kept as real files. 13.2 added Terraform provisioning of saved queries. See the Terraform guide for state and module structure.

terraform {
  required_providers {
    grafana = { source = "grafana/grafana" }
  }
}

provider "grafana" {
  url  = "https://grafana.example.com"
  auth = var.grafana_sa_token     # a service account token (glsa_...)
}

resource "grafana_folder" "payments" {
  title = "Payments"
}

resource "grafana_dashboard" "payments_overview" {
  folder      = grafana_folder.payments.uid
  config_json = file("${path.module}/dashboards/payments-overview.json")
}

Git Sync, generally available from 13.0, flips the model: the dashboard JSON lives in a Git repository, and Grafana keeps a folder (and since 13.1, optionally the whole instance root) in two-way sync with it. Edit in the UI and Grafana commits or opens a pull request; merge a change in Git and Grafana picks it up. By default it polls every 60 seconds, and webhooks give near-instant sync. Supported hosts include GitHub, GitHub Enterprise, GitLab, Bitbucket and plain Git. Git Sync depends on Unified Storage, the Kubernetes-style resource layer under the newer /apis API, and that connects to the most important warning in this guide: a migration bug in Grafana 13.0.0 could cause dashboards and folders to be lost or reverted when upgrading from 12.x with Git Sync enabled. Take a database backup before any 12-to-13 upgrade, move to the latest patch of your target line instead of 13.0.0, and rehearse in staging.

Do not run two of these against the same folder. They each believe they are the source of truth.

The HTTP API and service accounts

Automation that is not Terraform, such as a script that annotates a dashboard at deploy time, uses the HTTP API with a service account token. A service account is a non-human identity with an organization role and any number of tokens that start with glsa_. It replaced the old API keys, which were migrated automatically in the 11.x series. Service accounts work inside one organization and cannot perform server-admin operations.

# Create a service account and a token that expires in 30 days (run as an admin).
SA_ID=$(curl -s -X POST -H "Content-Type: application/json" -u admin:"$ADMIN_PASSWORD" \
  -d '{"name":"ci","role":"Editor"}' \
  http://localhost:3000/api/serviceaccounts | jq -r .id)

curl -s -X POST -H "Content-Type: application/json" -u admin:"$ADMIN_PASSWORD" \
  -d '{"name":"ci-token","secondsToLive":2592000}' \
  "http://localhost:3000/api/serviceaccounts/$SA_ID/tokens" | jq -r .key

Give each automation its own service account, at the lowest role that works, with an expiry. Then call the API with the token:

export TOKEN=glsa_xxxxxxxx
curl -s -H "Authorization: Bearer $TOKEN" 'http://localhost:3000/api/search?query=payments' | jq '.[].uid'
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/datasources/uid/prom-main/health

Grafana's direction of travel is the Kubernetes-style /apis API. Since 12 the dashboard resource lives at /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards, with the namespace default for organization 1. The 13.0 release states that the old /api routes remain functional but will no longer be updated.

Alerting in depth: how a rule becomes a notification

Alerting is where Grafana stops being a viewer and becomes an operational dependency. This section explains the machinery precisely enough that you can predict what will happen, because most alerting incidents are a difference between what the author expected and what the pipeline does.

A Grafana-managed alert rule is a set of queries, optional expressions, and a condition. The rule lives in a folder and belongs to an evaluation group, and the group defines the evaluation interval for all its rules. The minimum allowed interval is set by [unified_alerting] min_interval (ten seconds by default). Each evaluation runs the queries, runs the expressions on the server, and produces one alert instance for each series or label set that the query returned.

The state machine

Each instance moves through states. Normal means the condition is false. Pending means the condition is true but has not been true for the rule's pending period yet. Alerting (also shown as Firing) means the condition has held for the whole pending period and notifications are sent. Recovering, added in 12.0, sits between firing and normal so a condition must stay false before the alert resolves. No Data and Error describe a failure to evaluate rather than a verdict. Separately, the rule health reports ok, nodata or error.

The pending period is how you avoid pager storms from blips. A condition "error rate above 5%" with a pending period of five minutes must hold for five minutes before anyone is notified. The trade-off is straightforward and worth stating to the team: each minute of pending period is a minute added to your detection time.

No Data and Error: the two states people forget

If a query returns nothing, or fails, the rule does not quietly pass. Each rule has a policy for both cases. You can set the rule to Alerting, Normal, or Keep Last State, or leave the default, which creates a synthetic alert named DatasourceNoData or DatasourceError, carrying the rule's name, UID and the data source UID as labels. The annotation grafana_state_reason tells you why.

This is the most consequential choice in a rule. Consider an alert "no traffic to the checkout service": the absence of data is the failure, so No Data should be Alerting. Consider "queue depth above 1000" over a metric that legitimately disappears when a job is idle: No Data should be Normal, or you will page for a healthy system. For the Error state, which means the query itself failed, the usual answer is to keep the default, so a broken data source becomes a visible alert rather than silent health. The evaluation timeout is thirty seconds by default, with up to three attempts.

[sse.dataQueryError] failed to execute query [A]: ...           -> the data source query failed (DatasourceError)
[sse.readDataError] [A] got error: input data must be a wide series but got type long
                                                                 -> the query returned a shape an expression cannot use
failed to build query 'A': data source not found                 -> UID mismatch, often a numeric ID left over from older config

Expressions and reducing a series to a verdict

A query returns a time series, but an alert condition needs a single value per instance. A standard rule therefore has three stages. Query A fetches the series. A Reduce expression B turns each series into one number using a function such as Last, Mean or Max. A Threshold expression C compares that number to a limit and produces 1 or 0, and C is set as the condition. Last reacts fast and is noisy; Mean over a window smooths. Choose deliberately. Math expressions combine queries, for example $A / $B * 100, and SQL expressions let you write a SQL query over the results of other queries when a join is needed.

Routing notifications: policies, timings and templates

Alert instances are useless until the right person hears about them once, at the right time. Grafana-managed alerting embeds a Prometheus Alertmanager for each organization, and routing is driven by labels. The rule's labels (and the labels from the query, such as service) decide where an alert goes. Annotations carry the human text: summary, description and a runbook_url. A common early mistake is to put routing information in annotations, where the policy tree cannot see it.

The notification policy tree starts at a root policy with a default contact point. Child policies match on label matchers, for example team = payments or severity = critical, and route matching alerts to their own contact point. By default a matching alert stops at the first matching child. Enabling Continue matching lets one alert also fall through to later siblings, which is how you send the same alert to a team channel and to a paging service. Children inherit the settings of their parent unless they override them. Three timers govern delivery, and their defaults explain behaviour you will otherwise think is a bug:

Group by decides what is bundled into one message. Grouping by alertname and cluster turns fifty pods failing on one cluster into one notification; grouping by nothing gives you fifty. The 30 second group wait is the reason an alert appears in your channel about half a minute after it begins firing.

apiVersion: 1
policies:
  - orgId: 1
    receiver: platform-default
    group_by: ['alertname', 'cluster']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: payments-pager
        object_matchers:
          - ['team', '=', 'payments']
          - ['severity', '=', 'critical']
        continue: true
      - receiver: payments-slack
        object_matchers:
          - ['team', '=', 'payments']

Silences, mute timings and active timings

A silence is an ad hoc, matcher-based, time-boxed block, created during an incident or maintenance. A mute timing is a recurring window, such as weekends or the nightly batch hour, during which matched policies do not notify. An active timing is the inverse: notify only during the window. All of them stop notifications, not evaluation. The rule still evaluates, the instance still fires and is visible in the UI, and it will notify normally once the silence or window ends if it is still firing.

Message templates and the end-to-end check

Contact points (email, Slack, PagerDuty, Microsoft Teams, webhook and others) format messages with Go templates. A shared template keeps messages consistent and puts the runbook where the responder needs it:

{{ define "alert.summary" -}}
[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .Alerts.Firing | len }} firing)
{{ range .Alerts.Firing -}}
- {{ .Labels.service }}: {{ .Annotations.summary }}
  Runbook: {{ .Annotations.runbook_url }}
{{ end }}
{{- end }}

Test a contact point with its Test button before relying on it, and remember what a successful test proves: that Grafana can reach the destination with those credentials. It does not prove that a policy routes real alerts there. When "the alert fired but nobody heard", walk the chain in order. Confirm the instance is Alerting and read its labels. Check which policy those labels match. Check for a matching silence or mute timing. Check the Grafana log for Failed to send notification. Only then suspect the destination. The Slack or Teams side is almost always right; the label that did not match the matcher is almost always wrong.

Access control: organizations, roles, folders and what a Viewer can still do

Access control in Grafana is layered, and the layers do different jobs. Mixing them up is how an instance ends up either locked down so tightly nobody can work, or open in ways nobody intended.

An organization is the hard boundary. Data sources, dashboards, teams, service accounts and alert rules belong to one organization, and a user can be a member of several with a different role in each. A single-team instance uses only the default organization (id 1, "Main Org."). Separate organizations suit separate customers or groups who must not see each other's data sources, but they cost you: nothing is shared across them, so provisioning and Terraform must be applied to each, and only a server admin can manage them. The Grafana Admin flag is an instance-level power, distinct from the Admin role inside an organization. Keep the set of server admins very small.

Inside an organization, the basic roles are None, Viewer, Editor and Admin. On top of those, folders are the main permission boundary: you grant a team Edit on its folder, and everything in it (dashboards, alert rules, library panels, including nested folders) inherits that. Teams keep these grants maintainable, because you add a person to a team rather than to fifty folders. Fine-grained RBAC with custom roles, and permissions on individual data sources, are Enterprise and Cloud features; in OSS you work with the basic roles plus folder and dashboard permissions.

The important caveat about Viewers

It is widely assumed that a Viewer can only see the dashboards they have been given. The real behaviour is weaker. A Viewer can run arbitrary queries against any data source in their organization, for instance through Explore or by crafting an API call, regardless of which dashboards they were shown. Dashboard permissions control what people see by default, not what the data source will answer. The controls that do constrain queries are: data source permissions (Enterprise), label-based access control on supported data sources (Enterprise and Cloud), separate organizations, and, most robustly, least-privilege credentials on the data source itself, for example a read-only database user limited to the needed schemas. Anonymous access ([auth.anonymous] enabled) extends the same exposure to anyone who can reach the URL, so it is justified only for a deliberately public instance.

Authentication

Grafana supports OAuth and OIDC providers (GitHub, GitLab, Google, Azure AD, Okta and a generic one), LDAP, JWT, an authenticating proxy, and basic login. SAML, team sync and SCIM provisioning belong to Enterprise and Cloud. Grafana has no built-in multi-factor authentication, so enforce it in your identity provider.

For OAuth, a handful of keys decide whether login works and whether it stays safe:

[auth.generic_oauth]
enabled = true
name = Company SSO
client_id = grafana
client_secret = $__file{/run/secrets/oauth_client_secret}
scopes = openid email profile groups offline_access
auth_url = https://sso.example.com/authorize
token_url = https://sso.example.com/token
api_url = https://sso.example.com/userinfo
use_pkce = true
use_refresh_token = true
role_attribute_path = contains(groups[*], 'grafana-admins') && 'Admin' || contains(groups[*], 'grafana-editors') && 'Editor' || 'Viewer'

role_attribute_path maps identity-provider claims to a Grafana role using a JMESPath expression, so access follows group membership instead of being assigned by hand. use_refresh_token = true matters for revocation: without refresh tokens, an OAuth session can remain valid for up to 30 days after the identity provider has disabled the user. Errors here have recognisable fingerprints. OAuth: missing saved state means the state cookie was lost, usually from an HTTP versus HTTPS mismatch, a root_url that does not match the address in the browser, or a proxy stripping cookies. user does not belong to a role means the role mapping returned nothing; set a default in the expression, or decide whether role_attribute_strict should refuse such logins on purpose.

Once SSO works, set disable_login_form = true so the local admin path is no longer the soft target, and keep a documented break-glass procedure, namely the grafana cli admin reset-admin-password command, for the day SSO is down.

Service accounts as the machine identity

Everything that is not a person uses a service account: Terraform, CI jobs, the Grafana Image Renderer callback, scripts. Name each after its purpose, give it the lowest role that works, put the token in a secret store, and set an expiry. A token with no expiry and an Admin role is the incident report waiting to be written. Tokens begin with glsa_, which also makes them easy for secret scanners to find in repositories.

Running Grafana on Docker and Kubernetes

Most production Grafana runs in containers. The container itself is simple; the operational details are the parts that bite.

Docker

The current Docker documentation recommends the grafana/grafana-enterprise image, which contains every OSS feature and runs without a licence, and still publishes grafana/grafana for the OSS build. In either case pin an exact tag, such as grafana/grafana:13.2.3, never latest and never a bare minor. Patches are frequent and often security-driven (13.2.3 shipped three CVE fixes), and an unpinned tag means your next restart is also an unreviewed upgrade.

services:
  grafana:
    image: grafana/grafana-enterprise:13.2.3
    restart: unless-stopped
    ports: ['3000:3000']
    environment:
      GF_SERVER_ROOT_URL: http://localhost:3000/
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
      GF_PLUGINS_PREINSTALL: grafana-clock-panel@1.0.1
    volumes:
      - grafana_storage:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning:ro
    secrets:
      - grafana_admin_password
secrets:
  grafana_admin_password:
    file: ./secrets/grafana_admin_password.txt
volumes:
  grafana_storage: {}

Three details deserve attention. GF_PLUGINS_PREINSTALL installs plugins at startup, which replaces the old pattern of baking an install script into a derived image. The __FILE suffix on an environment variable reads its value from a file, which keeps the secret out of docker inspect. And the data volume is what makes state survive, at the path /var/lib/grafana. If you bind-mount a host directory instead, the container's user (uid 472 in the official image) must be able to write to it, which is the classic mkdir /var/lib/grafana: permission denied. Either fix ownership or run the container with --user "$(id -u)".

Kubernetes with Helm

The Grafana chart has moved. Since 30 January 2026 it lives at https://grafana-community.github.io/helm-charts under the repository name grafana-community; the older grafana.github.io chart is deprecated and receives no updates. A CI job or an Argo CD application still pointing at the old URL keeps working until it quietly stops being patched.

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm upgrade --install grafana grafana-community/grafana \
  --namespace monitoring --create-namespace -f values.yaml

image:
  tag: "13.2.3"

persistence:
  enabled: true            # otherwise all state is lost on pod restart
  size: 5Gi

admin:
  existingSecret: grafana-admin     # a Secret you manage, not a chart-generated one

grafana.ini:
  server:
    root_url: https://grafana.example.com/
  database:
    type: postgres
    host: pg.monitoring.svc:5432
    name: grafana
    user: grafana
    ssl_mode: require

envFromSecret: grafana-db          # supplies GF_DATABASE_PASSWORD

sidecar:
  dashboards:
    enabled: true
    label: grafana_dashboard       # ConfigMaps with this label are loaded as dashboards
  datasources:
    enabled: true
    label: grafana_datasource

The sidecars are the idiomatic Kubernetes route to config-as-code. A sidecar container watches for ConfigMaps or Secrets carrying a label, and writes their contents into Grafana's provisioning paths. A team that owns a service can ship its dashboard as a ConfigMap inside its own Helm chart, and it appears in Grafana without touching Grafana's chart. That is how large organisations avoid a central team becoming the bottleneck for every dashboard. Check the chart's value names against the chart version you install with helm show values grafana-community/grafana, since chart keys are the sort of detail that changes between releases. The Helm guide and Argo CD guide cover packaging and deployment of these values, and the Kubernetes guide covers the probes and Secrets.

Health checks use the unauthenticated /api/health endpoint, which returns JSON with "database": "ok" and the version. Use it for liveness and readiness. If it reports "database": "failing", Grafana can still serve static assets while every real request fails, which is an instructive failure: a readiness probe that only checks the port would keep sending traffic to a broken pod.

Plugins, signing and image rendering

Plugins come in three kinds: data source plugins add backends, panel plugins add visualizations, and app plugins bundle pages, dashboards and sometimes their own backends. Backend plugins run as separate processes on the Grafana server and execute code there, so the trust decision for a plugin is similar to installing software on the host.

Every plugin is signed. Grafana-signed and community-signed plugins load normally. An unsigned plugin is blocked unless you name it explicitly:

[plugins]
allow_loading_unsigned_plugins = my-internal-datasource

Do that only for plugins you built or reviewed. The log messages Plugin ... is not signed and invalid signature mean exactly this, and the fix is either an updated plugin or a conscious allow-list entry, never a blanket allowance. Since 12.0, Angular plugins no longer load at all, and there is no toggle to bring them back. A community panel that was Angular-based needs a React replacement.

Installation is done with grafana cli, run on the Grafana host, then a restart:

grafana cli plugins list-remote
grafana cli plugins install grafana-clock-panel
grafana cli plugins ls

For containers and clusters, GF_PLUGINS_PREINSTALL is preferable, because it makes the plugin set part of the declared configuration rather than the state of a running container. Pin plugin versions the same way you pin the image. The format is plugin-id@version, and for a custom zip it is plugin-id@version@https://example.com/plugin.zip. grafana cli also performs stricter compatibility checks against a plugin's grafanaDependency than before, so a plugin pinned years ago may refuse to install on 13.

Image rendering changed in 13

Rendering panels to PNG, which powers alert screenshots and scheduled reports, used to be possible through an Image Renderer plugin that ran a headless browser inside Grafana's host. Grafana 13 fully removed that plugin. Rendering now works only through the standalone remote rendering service, the grafana/grafana-image-renderer container or an equivalent, running separately. Two further changes came with it. Rendering authentication defaults to JWT (the renderAuthJWT toggle is on by default), so you must set [rendering] renderer_token to a non-empty value other than the default -, and the same token in the renderer's configuration. A mismatch surfaces as Failed to send request to remote rendering service or as 401s from the renderer.

[rendering]
server_url = http://renderer:8081/render
callback_url = http://grafana:3000/
renderer_token = $__file{/run/secrets/renderer_token}

If your screenshots in alert notifications stop arriving after a 12-to-13 upgrade, this is the first place to look. On RPM-based renderer images, text missing from PNGs is a fonts problem, fixed by installing the fontconfig and font packages. Because a headless browser is resource hungry, treat the renderer as its own service with its own limits so it cannot starve Grafana.

Connecting Grafana to the rest of the stack

Grafana earns its place by linking signals, so that a person can jump from a metric spike to the logs and traces that explain it. The linking is configuration you set up once per data source, and it is easy to skip. The catalogue has guides for the neighbouring tools; this section covers how Grafana joins them.

The common stack is Prometheus or Mimir for metrics (PromQL; see the Prometheus guide), Loki for logs (LogQL), Tempo for traces (TraceQL), and Pyroscope for profiles. Telemetry is collected by Grafana Alloy, Grafana's distribution of the OpenTelemetry Collector that replaced Grafana Agent, or by a plain OpenTelemetry Collector sending OTLP (the OpenTelemetry guide). Mimir, Loki and Tempo support multi-tenancy through an X-Scope-OrgID header, which you set per data source as a custom HTTP header, so one Grafana can serve several tenants without mixing their data.

The links you should configure:

• Logs to traces (derived fields) on the Loki data source: a regex extracts a trace ID from each log line, and Grafana renders it as a link that opens the trace in Tempo.
• Traces to logs and traces to metrics on the Tempo data source: from a span, jump to the logs of the same service in the same time window, or to its RED metrics.
• Exemplars on Prometheus: a latency histogram carries trace IDs on sample points, and the panel shows them as dots you can click to open the exact slow trace.
• Correlations: a general mechanism that turns a value in one data source's results into a query against another.

When a link goes to the wrong place, the cause is almost always a mismatch of labels: the service name in metrics is payments-api while in logs it is payments. Consistent resource attributes, which OpenTelemetry's semantic conventions give you when everything is instrumented the same way, are what make this work without hand-written mappings.

Alerting from data sources versus from Grafana

You have two places to run alert rules. Grafana-managed rules are evaluated by Grafana against any data source, and their state and history live in Grafana's database. Data source-managed rules are evaluated by the backend itself, such as Prometheus, Mimir or the Loki ruler, and Grafana displays and routes them. For a large Prometheus estate, evaluating rules inside Prometheus or Mimir is cheaper and keeps evaluation working even if Grafana is down. Grafana-managed rules win when an alert needs multiple data sources, SQL expressions, or a single UI for teams that do not own Prometheus. The two can coexist, and recording rules exist in both.

Performance: finding what is actually slow

"Grafana is slow" is a report, not a diagnosis. There are four different slow things, and the fix differs for each: the page load, the panel queries, the alert evaluation, and the Grafana server itself. Start by deciding which one.

Panels slow, page shell fast. This is the most common case, and it is almost always the data source. The query inspector's timing separates Grafana overhead from the data source response. Fixes, in order of leverage: a recording rule for any expression that aggregates many series, a coarser step or lower max data points, fewer panels that refresh at once, a shorter default time range, and a longer refresh interval. Grafana CPU is rarely the bottleneck; the data source's concurrency is. A dashboard that fires sixty queries at once against a small Prometheus is asking for a queue.

Everything slow for everyone. Look at the Grafana database and the host. A database on an undersized volume or with connection-pool exhaustion makes every page slow. The [database] pool settings (max_open_conn, max_idle_conn, conn_max_lifetime) exist for this. Sizing guidance from the docs is about 2 to 4 GB and two cores for fewer than 25 concurrent users, and 4 to 8 cores with 8 to 16 GB for 25 to 200, rising from there. Most installs are under-provisioned on the database, not on Grafana.

Alerts late or missing. Evaluation is a scheduled job with a timeout, and a rule group with hundreds of rules over a slow data source may not finish within its interval. The Grafana metrics for alert evaluation show duration and failures, and failing evaluations become DatasourceError alerts rather than silence, which is the feature working as designed.

High cardinality on the Grafana side. Variables populated from a label with tens of thousands of values, or "Include All" used in a regex matcher, produce queries whose text alone is huge. Filter variable queries to what the dashboard needs, and use the Regex field.

The 13.0 release also changed one performance-adjacent default: [server] enable_gzip is now true. If a reverse proxy in front of Grafana also compresses, you pay CPU twice for no gain; let exactly one layer compress.

Watching Grafana itself

A monitoring system that nobody monitors fails silently. Grafana exposes what you need to watch it, but you have to turn it on.

Set [metrics] enabled = true and Grafana serves Prometheus metrics at /metrics, optionally protected with basic_auth_username and basic_auth_password. Backend plugin metrics are served under /metrics/plugins/<plugin_id>. The request duration metric grafana_http_request_duration_seconds is published as a native histogram, and you can set classic_http_histogram_enabled = false to stop publishing the classic buckets and cut the number of series. Alerting evaluation has its own grafana_alerting_* metrics, and since 13.0 many go_sql_stats_* and grafana_database_conn_* metrics are deprecated in favour of standard Go SQL metrics, so dashboards built on the old names need updating.

scrape_configs:
  - job_name: grafana
    metrics_path: /metrics
    basic_auth:
      username: metrics
      password_file: /etc/prometheus/grafana_metrics_password
    static_configs:
      - targets: ['grafana:3000']

Logs go to the console and/or a file depending on [log] mode. For a cluster, log as JSON and ship to Loki. Traces of Grafana's own HTTP API, with context propagation to data sources, can be sent over OTLP, which is the way to see where a slow request spent its time. Grafana Advisor, generally available in 13.0, runs weekly checks for failing data sources, outdated plugins and misconfigured SSO, which catches the slow rot that nobody notices.

Meta-monitoring: who watches the watcher

The rule that matters most is that Grafana alerting cannot tell you that Grafana is down. If Grafana is the only thing evaluating and routing alerts, a crashed pod produces no alert at all. Give the Grafana service an independent check: a Prometheus rule on up{job="grafana"} == 0 evaluated in Prometheus and routed through a separate Alertmanager, or an external uptime probe against /api/health. The same applies to a dead-man's-switch: an always-firing alert sent to an external service that pages when it stops arriving. That detects both Grafana being down and the notification path being broken.

Debugging beyond the basics

The beginner level taught you to read the red box on a panel. At this level you work through a fixed sequence, from the nearest evidence to the farthest, so you stop bouncing between theories.

1. Read the panel error and the inspectorThe error text names the layer. client_error: client error: 401 is authentication to the data source. 422 ... parse error is a bad query. context deadline exceeded is a timeout. dial tcp ...: connection refused is a network path from the Grafana server, not from you.
2. Test from the Grafana hostRun curl from the same network namespace as Grafana. In Docker use the service name, never localhost, because inside the container that means Grafana itself. In Kubernetes use kubectl exec into the Grafana pod.
3. Read the Grafana log at a higher levelRaise [log] level to debug, or raise one subsystem with filters, reproduce once, and lower it again. Debug logging is voluminous.
4. Check the data source definitionAuthentication headers, tenant headers, the timeout, and the scrape interval. A 401 that appeared after a secret rotation is this.
5. Check what changedDashboards have a version history, and alert rules have a state history. Compare the working version to the broken one.
6. Reproduce in ExploreRun the same query in Explore with the same time range. If Explore works and the dashboard does not, the fault is a variable, a transformation, or an override.

A catalogue of messages and their meaning

For alerts that fire and never arrive, the notification-side checklist comes first: is the instance really Alerting, does a silence or mute timing match, did the policy tree route it to the contact point you expect, and what does the contact point's log say. A duplicate notification in a multi-replica deployment without ha_peers set is the signature of each replica sending its own copy.

Upgrades, version pinning and the move to 13

Grafana ships a new minor version every six to ten weeks and a patch about every two. The current line is 13.2, with maintained patches also on 13.1, 13.0 and 12.4 as of the end of September 2026. Upgrades are therefore routine work, and the skill is making them boring.

The documented principle is that you can upgrade from one supported release to another, and dashboards do not change meaning. The discipline around that principle is a short checklist:

1. Back up the database and the configThe database dump, secret_key and your ini or environment settings. A database migration runs on startup and downgrades are not supported afterwards, so the backup is your rollback.
2. Read the target version's "What's new"Since 12.0 breaking changes are listed there, with no separate page per release. Read the breaking section for every minor you cross.
3. Upgrade staging firstWith a restored copy of production data, so migrations run against realistic rows.
4. Start one replica firstSchema migrations run at startup, and several new-version replicas racing through them is avoidable risk.
5. Verify, then proceedCheck /api/health, one dashboard, one alert evaluation, one notification and one login.

The 12-to-13 checklist

The 13.0 release is the largest change since 11, and a real upgrade touches most of the following:

• Use a patched 13.x. Do not land on 13.0.0. It contained the Unified Storage migration bug that could lose or revert dashboards and folders when Git Sync was enabled.
• Replace grafana-cli and grafana-server commands in scripts, Dockerfiles and runbooks with grafana cli and grafana server.
• Replace the renderer plugin with the remote rendering service, and set renderer_token on both sides.
• Move numeric-ID API calls to UIDs, as described earlier.
• Re-check alerting RBAC. Enforcement tightened for custom roles, Terraform-managed roles and provisioning, especially roles scoped to a particular data source UID.
• Update the Helm repository URL to the grafana-community chart repository.
• Update dashboards and alerts on removed or deprecated metrics, such as the duplicate grafana_caching_items and grafana_caching_size, which were removed, and the deprecated go_sql_stats_* and grafana_database_conn_* series.
• Audit plugins. Angular plugins already stopped working in 12, and React 19 broke at least one panel plugin.
• Note changed defaults: enable_gzip is true, and the audit-log defaults for log_datasource_query_request_body and log_datasource_query_response_body changed.

Putting it all together

Here is a compact, end-to-end setup that uses most of what this guide covered: a pinned Grafana with PostgreSQL, a provisioned Prometheus data source with a stable UID, a dashboard provider, an alert rule with a deliberate No Data state, and a notification policy. It is small enough to run on a laptop.

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: grafana
      POSTGRES_USER: grafana
      POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
    volumes: [pg_data:/var/lib/postgresql/data]
    secrets: [pg_password]

  prometheus:
    image: prom/prometheus   # pin an exact tag in real use, as with Grafana
    volumes: ['./prometheus.yml:/etc/prometheus/prometheus.yml:ro']

  grafana:
    image: grafana/grafana-enterprise:13.2.3
    depends_on: [postgres, prometheus]
    ports: ['3000:3000']
    environment:
      GF_DATABASE_TYPE: postgres
      GF_DATABASE_HOST: postgres:5432
      GF_DATABASE_NAME: grafana
      GF_DATABASE_USER: grafana
      GF_DATABASE_PASSWORD__FILE: /run/secrets/pg_password
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/admin_password
      GF_SECURITY_SECRET_KEY__FILE: /run/secrets/secret_key
      GF_METRICS_ENABLED: "true"
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    secrets: [pg_password, admin_password, secret_key]

secrets:
  pg_password: { file: ./secrets/pg_password }
  admin_password: { file: ./secrets/admin_password }
  secret_key: { file: ./secrets/secret_key }

volumes:
  pg_data: {}

apiVersion: 1
groups:
  - orgId: 1
    name: payments-red
    folder: Payments
    interval: 1m
    rules:
      - uid: payments-high-5xx
        title: Payments 5xx rate above 2%
        condition: C
        for: 5m
        noDataState: Alerting        # silence here would hide a dead service
        execErrState: Error
        labels:
          team: payments
          severity: critical
        annotations:
          summary: "Payments returning more than 2% 5xx"
          runbook_url: https://runbooks.example.com/payments-5xx
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: prom-main
            model:
              expr: sum(rate(http_requests_total{job="payments",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="payments"}[5m])) * 100
              instant: true
          - refId: B
            datasourceUid: __expr__
            model: { type: reduce, expression: A, reducer: last }
          - refId: C
            datasourceUid: __expr__
            model: { type: threshold, expression: B, conditions: [{ evaluator: { type: gt, params: [2] } }] }

The pieces work together as follows. The data source has a fixed uid, so the dashboards and the rule refer to it by a name that survives a rebuild. The alert rule's query, reduction and threshold are the three stages explained earlier, and the No Data state is a decision, not a default. The routing labels team and severity are set on the rule so the policy tree from the earlier section can match them. The secrets are files, never environment listings. The state lives in PostgreSQL with the secret_key backed up separately, so the instance can be rebuilt from the repository plus a database restore.

Note that the provisioned rule shape above follows the documented file format, but confirm field names against your running version with an export: in Grafana you can export an existing alert rule as YAML from its menu, and starting from an export is more reliable than writing the file from memory.

Where you are now

You can predict what a panel will send to its data source and read it back in the inspector. You use $__rate_interval, formats and chained variables deliberately. You order transformations and overrides on purpose, and design dashboards around a question, a hierarchy and links. You configure Grafana through files and environment variables with the right precedence, keep secrets in files, and run it on a real database with a backup that includes the secret_key. You manage dashboards and alerting as code, through provisioning, Terraform or Git Sync, and you know why only one of them owns a given folder.

You understand alerting as a pipeline: evaluation groups, pending periods, the No Data and Error choice, the policy tree, and the difference between silences and evaluation. You know what access control does and does not restrict, in particular that a Viewer can query any data source in the organization. You can debug from the nearest evidence outward, watch Grafana from outside Grafana, and take a 12-to-13 upgrade in a sequence that has a rollback.

The senior level takes on the platform view: high availability and its alerting implications, the trust model and secret rotation, multi-tenancy and cost, upgrade and migration strategy across many instances, and where Grafana stops being the right tool. Before that, the most valuable next step is practical. Pick one hand-built dashboard and one hand-made alert in your team's instance, move both into Git, and review the change like any other.

Sources

• Grafana documentation
• What's new in Grafana 13.0
• What's new in Grafana 13.1
• What's new in Grafana 13.2
• What's new in Grafana 12.0
• Grafana releases on GitHub
• Install Grafana on Docker
• Install Grafana with Helm
• Helm repository migration to grafana-community
• Configure Grafana
• Feature toggles
• Configure security
• Configure authentication
• Set up Grafana for high availability
• Set up Grafana monitoring
• Upgrade Grafana
• Image rendering
• Grafana CLI
• HTTP API
• Dashboard HTTP API
• Provisioning
• Service accounts
• Roles and permissions
• Grafana Alerting
• Alerting fundamentals
• Alert rule state and health
• Notification policies
• Git Sync
• Dashboard best practices
• Troubleshooting
• Image Renderer plugin support removed
• Rendering default auth mode changed to JWT

Part two of three. At this level an interviewer is no longer asking what a dashboard is. They describe a situation from a real monitoring estate ("the alert fired but nobody was paged", "the dashboard takes forty seconds to load") and listen for how you narrow it down. This file is organised by scenario first, then trade-offs, then what separates a strong answer from a weak one. The version is Grafana 13.2, and several answers depend on changes from 12 and 13, which are flagged.

Where this picks up

Scenario: the dashboard is slow

Prompt. "The payments dashboard takes forty seconds to load. Where do you start?"

A strong answer separates four possibilities before touching anything: slow queries, too many queries, a slow Grafana server or database, and slow rendering in the browser. The first move is the browser's network tab and the panel query inspector, to see which /api/ds/query requests are slow and what the data source was actually asked. The interpolated query matters, because variables can turn a short query into an enormous one, for example an "Include All" regex with a thousand alternatives.

Then you take the slowest query and run it directly against the data source with the same range, from the Grafana host. If it is slow there, the dashboard is innocent and the fix is in the query or the backend: a recording rule for an aggregation over many series, a coarser step, a lower max data points, or a shorter default range. If it is fast there but slow in Grafana, look at transformations (they run in the browser over the whole result), the number of panels refreshing together, and the refresh interval.

What a weak answer does. It says "add more CPU to Grafana" or "reduce the panels" without measuring. Grafana's CPU is rarely the constraint; data source concurrency is. Another weak pattern is optimising a panel without checking whether it is the slow one.

Scenario: the alert fired but nobody was notified

Prompt. "An alert went to Firing at 02:10. The on-call engineer heard nothing. Walk me through it."

A strong answer follows the pipeline in order and does not start at the destination. First, confirm the instance truly reached Alerting, not just Pending: a long pending period may mean the instance had not fired yet. Second, read the instance's labels, because routing is entirely label-driven. Third, find which notification policy those labels match; a missing team label sends the alert to the root policy and its default contact point, which might be a channel nobody watches. Fourth, check for a matching silence or mute timing, since either stops notifications but not evaluation. Fifth, check the group wait (30 seconds) and group interval (5 minutes) defaults, which delay notifications by design. Sixth, look in the Grafana log and the contact point's history for Failed to send notification. Only after all that do you suspect Teams, Slack or the paging service.

The part that impresses is the follow-up: what you change so it does not recur. A dead-man's-switch alert that always fires and pages an external service when it stops arriving. A default policy that routes unmatched alerts to a place that is monitored. A required team label enforced in review. Alerting on symptoms (RED signals) rather than a dozen cause alerts.

Scenario: no alert fired when the service died

Prompt. "The checkout service was completely down for twenty minutes and our error-rate alert never fired."

The answer is almost always No Data. An error rate computed as errors divided by requests returns no series when there are no requests, and the rule's No Data handling decides what happens. If it was set to Normal, or the default state was ignored, a dead service looks identical to a healthy one. The fix is per rule: for alerts where absence is the failure, set No Data to Alerting, or add a separate rule on up == 0 or on request rate falling to zero. A strong candidate also mentions that Error (the query failing) is a different state from No Data, and that the default behaviour creates synthetic DatasourceNoData and DatasourceError alerts rather than silence.

Scenario: Grafana itself goes down

Prompt. "How would you know if Grafana stopped evaluating alerts?"

The key insight is that Grafana alerting cannot report that Grafana is down. The answer needs an independent path: a Prometheus rule on up{job="grafana"} == 0 evaluated by Prometheus and routed through a separate Alertmanager, an external uptime probe on /api/health, and a dead-man's-switch that pages when its heartbeat stops. Mention that /api/health returns "database": "ok" or "failing", so a readiness probe should check the body, not just the port. To watch Grafana's internals you enable [metrics] and scrape /metrics.

Scenario: a Viewer sees data they should not

Prompt. "We gave a contractor Viewer access to one folder. Are we safe?"

No. A Viewer can run arbitrary queries against any data source in the organization, for instance in Explore, regardless of which dashboards they were shown. Folder permissions limit dashboards and editing, not what the data source answers. A strong answer lists what constrains queries: data source permissions and label-based access control (Enterprise and Cloud), a separate organization, and least-privilege credentials on the data source, such as a read-only database user with limited schemas. For truly untrusted tenants the answer is a separate Grafana instance. A weak answer talks about hiding the data source from the menu.

Scenario: a dashboard edit keeps reverting

Prompt. "People complain their dashboard changes vanish or cannot be saved."

Two causes, both about provisioning. If the dashboard is provisioned with allowUiUpdates: false, the UI refuses to save, with Cannot save provisioned dashboard. If allowUiUpdates: true, edits may save but the file wins again at the next poll or restart. The correct workflow is to edit, export the JSON, commit it, and let provisioning apply it, or move the folder to Git Sync so the UI and the repository are the same thing. Also check that two mechanisms (for example Terraform and a provisioning file) are not managing the same folder, because each will overwrite the other.

Scenario: after the upgrade, scripts and screenshots break

Prompt. "We upgraded from 12 to 13 and three things broke: a deploy script, alert screenshots, and a data source lookup. What happened?"

This is the 13.0 breaking-change list, and a strong answer names each one. The standalone grafana-cli and grafana-server commands were removed in favour of grafana cli and grafana server. The Image Renderer plugin was removed, and rendering now needs the standalone remote renderer service with a matching renderer_token, because rendering authentication now defaults to JWT. The deprecated numeric-ID data source APIs are disabled by default, so lookups must use UIDs. Mention the others if you can: alerting RBAC enforcement tightened for custom roles, enable_gzip now defaults to true, the /api routes are frozen in favour of /apis, and some Prometheus metrics were deprecated or removed. The operational lesson is the most valuable part: do not upgrade to 13.0.0, because of the Unified Storage migration bug with Git Sync enabled, and always back up the database first.

Trade-off questions

Terraform, Git Sync, or provisioning files?

There is no universally right answer, and the question tests whether you ask who edits. Terraform suits an infrastructure team that owns everything and reviews by pull request; drift is detectable, but dashboard JSON inside HCL is awkward to review. Git Sync (generally available in 13.0) suits product teams who want to keep editing in the UI but need history, review and rollback; it depends on Unified Storage and needs the upgrade care described above. Provisioning files suit immutable, rebuildable instances, typically delivered through Helm sidecars on Kubernetes. The wrong answer is using two against one folder.

Grafana-managed alert rules or data source-managed rules?

Grafana-managed rules evaluate against any data source, can combine several with expressions, and have a single UI and state history. Data source-managed rules run in Prometheus, Mimir or the Loki ruler, are cheaper at large scale, and keep evaluating if Grafana is down. A balanced answer says: rules over one Prometheus estate and owned by the team running it belong in the Prometheus ruler; cross-source or UI-owned rules belong in Grafana. Recording rules exist in both.

One organization or several?

An organization is a hard tenancy boundary inside one instance: separate data sources, dashboards, teams and service accounts. The cost is operational: nothing is shared, provisioning and Terraform apply per organization, and only server admins manage them. A single organization with folder permissions is simpler and adequate for one company. Several organizations suit separated groups within one instance, and separate instances suit untrusted tenants.

Alert on causes or symptoms?

Symptoms, meaning the RED signals users feel. Cause alerts (CPU, disk, pod restarts) page people for things that may have no user impact and are the main source of fatigue. Causes belong on dashboards for diagnosis, with a few exceptions such as a disk that will genuinely fill in an hour.

SQLite or a real database?

SQLite only for evaluation or one developer. One writer at a time means database is locked under concurrency, two replicas cannot share it, and loss of the volume loses everything. Use PostgreSQL 12 or later or MySQL 8.0 or later, and back up the secret_key with the dump.

Pin the image or track latest?

Pin an exact tag. Patches arrive about every two weeks and are often security fixes (13.2.3 carried three CVE fixes), so an unpinned tag turns every pod restart into an unreviewed upgrade. Pinning also lets you patch deliberately: staging first, one replica first, then the rest.

Strong versus weak answers

Common interview questions

1. Where does Grafana store data, and what happens to the data in Prometheus?Grafana stores only its own metadata (dashboards, users, organizations, data source definitions, alert rules and state) in its database. Metrics, logs and traces stay in their data sources, which Grafana queries on demand through its backend.
2. Why does "Save & test" pass and the dashboard still fail?The test calls one health-style endpoint. A real panel sends a specific query over a specific range, and can fail on authentication scope, timeouts, query syntax, variable interpolation or a tenant header that the health test does not exercise.
3. What is $__rate_interval and why use it?A window at least four scrape intervals wide that adapts to zoom level, so rate() always has enough samples. A hard-coded window produces gaps when someone zooms out. It depends on the scrape interval set on the data source.
4. A multi-value variable works with one value and breaks with two. Why?The interpolation format. PromQL regex matchers get an alternation, but SQL needs quoted comma-separated values, so use ${var:singlequote} or the format the data source expects, and read the interpolated text in the inspector.
5. Transformation or data source query, which do you prefer?Shape the data in the data source where possible (SQL join, PromQL matching, recording rule), because transformations run in every viewer's browser on every refresh. Use transformations for presentation and small joins. They never affect alerts.
6. Explain the alert states.Normal, Pending (condition true but not yet for the pending period), Alerting, Recovering (added in 12.0), plus No Data and Error for evaluation failures. Rule health is ok, nodata or error.
7. What are No Data and Error handling, and how do you choose?Per-rule settings: Alerting, Normal, Keep Last State, or the default that creates DatasourceNoData and DatasourceError alerts. Choose by asking whether absence of data means failure for that signal.
8. What are the notification policy defaults?Group wait 30 seconds, group interval 5 minutes, repeat interval 4 hours. They explain the half-minute delay before a first notification and the four-hour reminders.
9. Silence versus mute timing?A silence is ad hoc and matcher-based, for an incident. A mute timing is a recurring schedule. Both suppress notifications, not evaluation.
10. How do you route the same alert to Slack and to a pager?Two child policies with overlapping matchers, and Continue matching on the first so the alert also reaches the second.
11. Why do you set a stable uid on a provisioned data source?Dashboards and alert rules reference it. Without it Grafana generates one, and references break on rebuild. Since 13.0 numeric-ID APIs are disabled by default, so UID is the only stable handle.
12. What is the difference between an API key and a service account token?API keys are superseded by service accounts: a non-human identity with an organization role and multiple tokens (prefix glsa_) with optional expiry. Service accounts are org-scoped and cannot do server-admin operations.
13. How do you keep secrets out of grafana.ini?Value expanders: $__file{/path} for mounted secret files and $__env{VAR} for environment variables. $__vault{} exists for Vault but is Enterprise only. Container environments also support the __FILE suffix.
14. Why can't I change the admin password through the config after the first start?The value only seeds the account when the database is first created. After that it lives in the database; reset it with grafana cli admin reset-admin-password.
15. How do you make Grafana highly available at this level?Several replicas behind a load balancer sharing a PostgreSQL or MySQL database. No sticky sessions are needed. Alerting has its own HA configuration, which is the senior-level detail.
16. Why is OAuth login failing with "missing saved state"?The state cookie was lost: an HTTP versus HTTPS mismatch, a root_url not matching the browser's address, or a proxy interfering with cookies.
17. What changed about plugins in 12 and 13?12.0 removed Angular plugin support entirely. 13.0 removed the Image Renderer plugin in favour of the remote rendering service. Unsigned plugins still need an explicit allow-list entry.
18. How do you install plugins reproducibly in a container?GF_PLUGINS_PREINSTALL with pinned versions in the form id@version, instead of installing by hand in a running container.
19. Why does the Helm chart repo URL matter now?The Grafana chart moved to https://grafana-community.github.io/helm-charts on 30 January 2026. The old repository's grafana chart is deprecated and receives no updates.
20. What is Git Sync and what should you watch when adopting it?Two-way synchronisation of dashboard JSON between a Git repository and a Grafana folder, generally available in 13.0. Keep one source of truth per folder, and treat the 13.0.0 migration bug as the reason to back up and use a patched release.
21. How would you monitor Grafana?Enable [metrics] and scrape /metrics, watch request duration and alerting evaluation metrics, use Grafana Advisor for configuration rot, and add an independent check for Grafana being down.
22. How do you link metrics, logs and traces?Derived fields on Loki to open a trace in Tempo, traces-to-logs and traces-to-metrics on Tempo, exemplars on Prometheus, and consistent service labels across all three.
23. What is your upgrade process?Backup, read "What's new" for each minor crossed, staging on a restored copy, one replica first, verify health, a dashboard, an alert and a login, with database restore as the rollback.

Final self-test

• Name the five stages a query passes through from a panel to a rendered chart, and which run on the server and which in the browser.
• Explain what $__rate_interval does and what breaks when you hard-code [1m].
• Say what Include All interpolates to, and why a custom All value helps.
• Explain why a transformation cannot change an alert, and where an alert's own reduction happens.
• List the alert states, the two failure-to-evaluate states, and how you choose the No Data setting.
• Give the three notification policy timer defaults and what each controls.
• Say what a silence stops and what it does not.
• Explain what a Viewer can do beyond the dashboards they were shown, and list two controls that limit it.
• Explain allowUiUpdates, and why edits to provisioned dashboards appear to vanish.
• Say why uid is mandatory in practice, and what 13.0 did to numeric IDs.
• Name the three places secrets can come from in grafana.ini, and which is Enterprise only.
• List six 13.0 breaking changes and the one patch-level warning about 13.0.0.
• Describe how you would know Grafana had stopped alerting, from outside Grafana.

Part two of three. At this level your dashboards load and your alerts fire, so the problems change character. They are no longer "why is this panel empty" but "why do our graphs have gaps when we zoom out", "why did the page go silent during an outage", and "why did the upgrade delete a folder". Start with the error table, then the practices and practice cards beneath it, then the operational recipes you can paste into a real estate. The version is Grafana 13.2.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

In order of leverage

If you can only do a few things this quarter, do them in this order, because each one reduces the cost of the ones after it. First, get state out of the UI: a dashboard or alert that exists only in a database is a dashboard you can lose. Second, make alerts reach people reliably: fix routing, the No Data choice, and the independent watcher. Third, make the instance rebuildable: real database, secrets in files, pinned versions, backups that include the key. Fourth, tighten access: service accounts with expiries, SSO with role mapping, and the honest understanding that Viewers can query every data source. Fifth, tune the dashboards: recording rules, sensible refresh, fewer panels. The last step is the one most people do first, and it is the lowest leverage.

Practice cards

1. Find a zoom-dependent gapTake a panel with a literal rate window, zoom out until a gap appears, then switch to $__rate_interval and confirm it disappears. Read the step in the inspector at each zoom.
2. Read the interpolated queryAdd a multi-value variable, select two values, and inspect the exact text. Change the format and watch the text change.
3. Reorder a transformationBuild a join and a calculated field, drag them into the opposite order, and note what changes. Use the debug output to see each step's result.
4. Provision and then fight itProvision a dashboard, try to save it in the UI, read the error, and then export the JSON and commit the change through the file instead.
5. Kill the data source on purposeStop Prometheus and watch a rule enter Error or No Data under each setting. Decide what each of your real alerts should do.
6. Time the first notificationFire a rule and time the gap until a webhook receives it. It should be close to the group wait.
7. Silence and observeCreate a silence, and confirm the rule still evaluates and appears in the UI while nothing is delivered.
8. Prove the Viewer caveatGive a test user Viewer on one folder, then have them query another data source through Explore. Write down what actually limited them.
9. Round-trip a restoreDump the database, rebuild a scratch instance with the same secret_key, and run "Save & test" on a data source with a password.
10. Rehearse an upgradeStart the next minor against a copy of production data in a scratch container and read the log for migration lines.

Recipe: a healthy start-up check

Before you trust an instance after any change, run the same five checks. They take under a minute and catch most regressions.

# 1. Is it up, and which version is actually running?
curl -s http://localhost:3000/api/health

# 2. Are the data sources reachable from the server (by uid, not numeric id)?
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/datasources/uid/prom-main/health

# 3. Did the provisioned dashboards load?
curl -s -H "Authorization: Bearer $TOKEN" \
  'http://localhost:3000/api/search?type=dash-db' | jq length

# 4. Is the plugin set what you declared?
grafana cli plugins ls

# 5. Any errors since start?
journalctl -u grafana-server --since "10 minutes ago" | grep -iE "error|fail" | head

The first call should report "database": "ok" and the pinned version. If the second fails with a 404, suspect a data source uid that does not match what your scripts expect. If the third returns zero after a provisioning change, check the provider path and file permissions.

Recipe: back up what actually matters

A Grafana backup is three things, and people routinely take one.

#!/usr/bin/env bash
set -euo pipefail
STAMP=$(date +%Y%m%d-%H%M%S)
DEST=/backups/grafana/$STAMP
mkdir -p "$DEST"

# 1. The database (PostgreSQL shown)
pg_dump --format=custom --file="$DEST/grafana.dump" "postgres://grafana@pg.internal/grafana"

# 2. The key that decrypts secrets stored in that database
cp /run/secrets/grafana_secret_key "$DEST/secret_key"

# 3. The configuration that shaped the instance
tar -czf "$DEST/config.tgz" /etc/grafana/grafana.ini /etc/grafana/provisioning

chmod -R go-rwx "$DEST"

Keep the key in a store with stricter access than the dump, because anyone holding both can read every stored credential. Then schedule the restore test from the previous section. A backup you have never restored is a hope.

Recipe: onboard a service into monitoring in an hour

When a team asks "how do we get our service on Grafana", give them a repeatable path instead of a meeting.

1. Expose metrics and confirm the scrapeThe service exposes Prometheus metrics; confirm the target is Up and the labels include job and a consistent service name.
2. Copy the service dashboard templateA dashboard with $__rate_interval, a service variable and the RED panels. Change only the variable default.
3. Add one symptom alertError rate or latency, with a pending period, a deliberate No Data state, labels team and severity, and a runbook_url.
4. Commit both to the repositoryThe dashboard JSON and the rule file, in the team's folder, reviewed like code.
5. Fire the alert on purposeConfirm it reaches the right channel and that the runbook link opens. An alert that has never been fired is unproven.

Recipe: rotate a service account token without downtime

Tokens without expiry become permanent keys to your instance. Rotate on a schedule.

# Create the new token first, while the old one is still valid
NEW=$(curl -s -X POST -H "Content-Type: application/json" -u admin:"$ADMIN_PASSWORD" \
  -d '{"name":"ci-token-2026-10","secondsToLive":7776000}' \
  "http://localhost:3000/api/serviceaccounts/$SA_ID/tokens" | jq -r .key)

# Store it in your secret manager, redeploy the consumer, then confirm it works
curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $NEW" http://localhost:3000/api/search

# Only then delete the old token in the UI or via the API

The order is what makes it safe: create, distribute, verify, then remove. A token with a ninety-day expiry also means forgotten automation fails loudly instead of living on indefinitely.

Recipe: move a hand-built dashboard into Git

The most common first step toward config-as-code, and a small one.

1. Clean the JSONExport with Export, Export as JSON. Set "id": null, give it a deliberate uid, and replace hard-coded data source names with a data source variable or the UID.
2. Commit itPlace it under the directory your dashboard provider reads, in a path that reflects the folder structure if foldersFromFilesStructure is on.
3. Switch the provider to read-onlyWith allowUiUpdates: false, people can still explore but the repository is the authority.
4. Delete the UI copyIf it lives in the same folder with the same uid, Grafana will complain. Remove the hand-made copy after the provisioned one is verified.
5. Document the edit pathWrite one paragraph in the repository: edit in the UI, export, open a pull request. If people do not know how to change it, they will work around it.

Recipe: read a failing alert in five minutes

When a rule shows Error or unexpected state, read it in this order and stop at the first finding.

1. Open the rule's state history and the instance's annotations. grafana_state_reason names the reason, and the synthetic alert labels name the data source.
2. Identify the error prefix. [sse.dataQueryError] is the data source query failing, [sse.readDataError] is a shape mismatch between the query and an expression, and data source not found is a UID mismatch.
3. Preview the query. Run the rule's query in the editor with the same time range. If it returns data in the editor but fails in evaluation, compare the relative time range and the data source permissions for the rule's folder.
4. Check the reduction. A query that returns several series needs a Reduce that handles each, and a long-format result may need converting before an expression can use it.
5. Check the timeout. Evaluation has a default timeout of thirty seconds with up to three attempts; a slow data source produces Error, not a hung rule.

How to review a dashboard or alert pull request

Once dashboards and rules live in Git, a review is where quality is enforced, and most reviewers do not know what to look for in a wall of JSON. Read in a fixed order and you will catch the problems that later become incidents.

Start with identity. Does every dashboard, data source and rule have a deliberate uid, and is it unique? A copied file with an unchanged uid is the most common cause of a provisioning error, and a missing uid is the most common cause of references that break after a rebuild. Then read the queries, not the layout. Every rate should use $__rate_interval, every service or cluster name should come from a variable, and every data source reference should be a variable or a UID rather than a display name. The diff of a JSON dashboard is noisy, so ask the author to export with stable formatting, and focus on the expr and rawSql fields.

Next, read presentation choices that affect decisions. Units should be set, because a bare number invites misreading. Thresholds should mean something: green should not be the colour of "no traffic". The top row should answer the dashboard's stated question. Then look at cost: how many panels refresh at once, what the default refresh interval is, and whether an expensive aggregation should be a recording rule in the metrics backend instead.

For alert rules, the review is stricter because the change can wake someone. Check the pending period against the detection time you want, and check that the No Data and Error states are chosen and justified in the description. Confirm the routing labels (team, severity) exist and match a policy, and that the runbook_url opens. Ask whether this is a symptom or a cause; if it is a cause, ask which dashboard it belongs on instead. Finally, ask how the author tested it. The right answer is that they fired it on purpose and watched the notification arrive, and the wrong answer is that the threshold looked reasonable.

A short rule makes reviews fast: if a reviewer cannot tell from the pull request what question a dashboard answers or what action an alert demands, the change is not ready, regardless of how good the JSON looks.

Reading Grafana's own signals when something feels off

A final habit is to check Grafana's own evidence before forming a theory. The state history on a rule shows when an instance changed state and why, which settles most "did it fire at 02:10" arguments without guessing. The dashboard version history shows who changed a panel and lets you compare it with the last good version, which matters because the most common cause of a dashboard that suddenly shows nothing is a recent edit rather than a data problem. The server admin settings page shows the effective configuration after every source has been merged, which settles "I changed the file" disputes. The startup log tells you whether provisioning succeeded, and whether a plugin failed signature checks. The /api/health response tells you the running version and whether the database is reachable, and it is the right endpoint for both probes and quick scripts.

When these four sources disagree with your mental model, trust them. The mistakes that cost the most time at this level are almost always an assumption about configuration, such as a file being read, a variable being set, a version being the one you think, that one minute of reading the effective state would have disproved.

Habits worth adopting now

• Put the data source scrape interval in provisioning, so $__rate_interval is right on every rebuild.
• Write the dashboard's question in its description before you add a panel.
• Give every alert rule a runbook_url, a team label and a severity label, and refuse to merge rules without them.
• Make every query use a variable or $__rate_interval, never a literal duration or a hard-coded service name.
• Fire each new alert once on purpose, end to end, before calling it done.
• Keep one owner per folder and one mechanism per resource type.
• Treat a Viewer as someone who can query every data source in the organization, and design credentials accordingly.
• Pin the image, the plugins and the Helm chart, and upgrade deliberately.
• Read "What's new" for each minor you cross, since breaking changes are listed there and not on a separate page.
• Check your scripts for grafana-cli, grafana-server, numeric data source IDs and the old Helm repository URL before any move to 13.

A short checklist

• Queries use $__rate_interval and variables, not literals
• Every data source, dashboard and rule has an explicit uid
• Dashboards and alerting resources are in Git, with one owner per folder
• Each alert has a chosen No Data state, labels for routing, and a runbook link
• Unmatched alerts land in a channel someone reads
• A check outside Grafana detects Grafana being down
• The database is PostgreSQL or MySQL, and the secret_key is backed up with it
• Automation uses service accounts with expiring tokens
• Image, plugins and chart are pinned, and upgrades are rehearsed on a restored copy
• The renderer, if used, is a separate service with a matching token

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further, because each one now has a scale, security or ownership dimension, and it adds the work you own when Grafana is a platform that other teams depend on rather than a tool you happen to use.

Up to now the questions were about making dashboards and alerts work. From here they change: what happens to forty teams when the one database behind Grafana stalls, who can read which data source when a Viewer opens Explore, how an upgrade from 12 to 13 is rehearsed so that nobody loses a dashboard, and how you find out that Grafana itself is down when the alerting that would tell you lives inside Grafana.

Where this picks up

I start with the architecture, because every failure mode below is a consequence of it.

Architecture and failure modes

Grafana is a web application with a Go backend and a React frontend, and the most important sentence in this whole guide is that it is not a datastore. Your metrics live in Prometheus or Mimir, your logs in Loki or Elasticsearch, your traces in Tempo, your business numbers in PostgreSQL. Grafana queries them where they live and draws the result. What Grafana itself stores is metadata: dashboards, folders, users, organizations, data source definitions, alert rules and their state, preferences. That metadata sits in the Grafana database, which is SQLite by default and MySQL 8.0 or later or PostgreSQL 12 or later in any real deployment.

That split gives you a small, clear set of components, and each one fails in its own recognisable way.

Because the server is close to stateless, you scale it by running more replicas behind a load balancer. There is no sticky-session requirement, since sessions are auth tokens stored in the database. The price of that design is that the database is the shared fate of every replica. If PostgreSQL is slow, every replica is slow; if it is unreachable, /api/health returns {"database":"failing"} and logins, dashboard loads and saves all fail, although already-loaded pages may keep painting cached panels for a while. That health endpoint is unauthenticated, and it is the one your liveness and readiness probes should use.

Think of the request path in three hops, because each hop has its own latency budget. A browser asks Grafana for a dashboard, which costs a few database reads. The browser then fires one query request per panel at /api/ds/query, and Grafana's backend proxies each to the data source. The data source does the actual work and answers. Grafana adds little on its own; what users call "Grafana is slow" is overwhelmingly the third hop. A dashboard with thirty panels, a five-second refresh and two hundred open browser tabs is six thousand queries per refresh cycle, all of which land on a Prometheus that was sized for a dozen people.

The components beyond the server are worth naming because each is an operational surface:

• Backend plugins run as separate processes spawned by Grafana. A crashing plugin takes out its own data source, not the server, but it does run code on your host, which is why signing matters.
• The embedded Alertmanager, one per organization, handles grouping, silencing and delivery of Grafana-managed alerts. It is part of the server process, not a separate deployment.
• The remote image renderer is a separate service, headless Chromium behind an HTTP endpoint. Since Grafana 13 it is the only way to render: the old Image Renderer plugin is fully removed, so dashboard screenshots, alert images and scheduled reports all depend on this service existing.
• Unified Storage is the Kubernetes-style resource layer behind the /apis endpoints. Dashboards and folders are served through it, and Git Sync depends on it. You rarely touch it directly, but it explains why the 13.0.0 migration bug involved Git Sync and dashboards together.
• Redis or Memcached, optionally, as the [remote_cache] backend and for Grafana Live or alerting HA in place of gossip.

Failure modes, with how each one looks from outside:

A senior habit to build now: for every alert or complaint, name which of those rows it is before opening a single dashboard. It narrows a forty-minute investigation to four.

Scaling: which component breaks first

Grafana's own CPU is almost never the first thing to break. The sizing guidance in the docs is deliberately modest: a small deployment under 25 concurrent users wants 2 cores and 2 to 4 GB of memory, a medium one of 25 to 200 users wants 4 to 8 cores and 8 to 16 GB, and a large one over 200 users wants 8 to 16 or more cores and 16 to 32 or more GB per instance. Those numbers describe the Grafana process. They say nothing about the systems behind it, and those are what actually fall over.

Here is the order in which things usually break as an estate grows, and what each looks like.

First, the data sources. A Prometheus server answering instant and range queries has a query concurrency limit and a memory budget for each query. One badly written dashboard, say a rate() over a huge range with a high-cardinality by clause at a 5 second refresh, is enough to push it over. The symptom in Grafana is context deadline exceeded or Timeout exceeded, or for Loki and Mimir a 429 too many requests on /api/ds/query, and the cause is the backend's tenant limits, not anything in Grafana. The fixes are all about asking for less: recording rules that precompute the expensive expression, $__rate_interval so the window matches the scrape interval and the panel's resolution, sensible max data points so a thousand-pixel-wide panel does not request ten thousand points, and a dashboard refresh floor.

Second, the refresh rate. [dashboards] min_refresh_interval sets the smallest refresh a user may choose, with a default of 5 seconds. Raising it to 30 seconds cuts worst-case query volume six-fold for dashboards people leave open on a wall display, and it is one of the cheapest knobs you own. Policy, not engineering, is the hard part: people will ask for the 5 second refresh "just for the incident".

Third, the database. Every replica shares one database, and it serves session lookups, dashboard reads and writes, annotation queries, and alert state persistence. The pool settings max_open_conn, max_idle_conn and conn_max_lifetime matter at this point: if each of ten replicas opens a hundred connections to a PostgreSQL that allows two hundred, you find out during a traffic spike. Size the pool to the database's connection limit divided by your maximum replica count, and put a connection pooler in front if you need headroom. Annotation-heavy and alert-heavy instances also grow the database faster than expected, so track its size as a first-class metric.

Fourth, rule evaluation. In the default HA mode every replica evaluates every alert rule. Ten replicas and three thousand rules is thirty thousand evaluations per interval, all hitting your data sources, most of them redundant. The ha_single_node_evaluation setting in [unified_alerting] lets one node evaluate to cut that data source load; evaluate whether you want it, because it changes your failure semantics if that node is the one that dies. Alert rules also have an evaluation timeout, 30 seconds by default, and up to 3 attempts, which together with the minimum evaluation interval of 10 seconds bounds how hard one rule group can hit a backend.

Fifth, the renderer. Headless Chromium is hungry. Scheduled reports or alert screenshots at volume need their own scaling, separate from Grafana, and a render that hangs holds resources until it times out. Keep it on its own nodes with memory limits.

Sixth, Grafana Live and the proxy. Websocket connections are cheap per user but not free at thousands of concurrent viewers. With HA you point [live] at Redis via ha_engine = redis, and the reverse proxy must pass websocket upgrades and use timeouts at least as long as your query timeout.

Two levers apply at every stage. Query caching (Enterprise and Cloud) serves repeated identical queries from a cache, which helps the wall-display case enormously. Splitting the estate is the last resort and the one that actually works: a noisy tenant with a hundred dashboards and no discipline becomes its own organization or its own instance, so its load and its blast radius are bounded.

A worked example of why the multiplication matters. Suppose 150 engineers, 40 of whom keep an incident dashboard open all day. The dashboard has 24 panels and refreshes every 10 seconds. That is 40 times 24 times 6 = 5,760 queries per minute from that one dashboard, 8.3 million per day, and it never appears in any capacity plan because nobody "uses" the dashboard actively. Raising the floor to 30 seconds, using the query cache, and moving the five heaviest expressions to recording rules typically brings this down by an order of magnitude with no change to what the viewer sees.

High availability and the alerting cluster

Running several replicas is easy and running them correctly is not, because two subsystems have state that must agree across replicas: sessions and alerting.

Sessions are the easy one. Login sessions are auth tokens in the shared database, so any replica can serve any request, and the load balancer needs only a health check on /api/health. If you use [remote_cache] with Redis or Memcached you can move some of that load off the database, but it is an optimization, not a requirement. One constraint worth knowing for Enterprise: the license is issued for the cluster's shared hostname, so root_url must be set to that public address and the same on every replica.

Alerting is the hard one. Every replica runs the scheduler and evaluates every Grafana-managed rule, then the replicas gossip alert state and notification logs to each other through the embedded Alertmanager cluster so that only one of them sends a given notification. The settings are these:

[unified_alerting]
enabled = true
execute_alerts = true
evaluation_timeout = 30s
min_interval = 10s
max_attempts = 3

# Gossip-based HA: list the OTHER replicas (headless service in Kubernetes)
ha_peers = grafana-0.grafana-headless:9094,grafana-1.grafana-headless:9094,grafana-2.grafana-headless:9094
ha_listen_address = 0.0.0.0:9094
# ha_advertise_address = 10.0.3.14:9094

# Alternative to gossip: coordinate through Redis
# ha_redis_address = redis:6379

# Evaluate on one node only to cut data source load
# ha_single_node_evaluation = true

Port 9094 has to be open between replicas for both TCP and UDP when you use gossip, and in Kubernetes that means a headless Service so each pod gets a stable DNS name. Forget it and nothing obviously breaks: rules still evaluate and notifications still go out. You only notice when every incident pages the on-call three times, once per replica, because without ha_peers each replica believes it is alone and sends its own copy. That is the classic symptom of a broken alerting cluster, and it is why duplicate notifications are a cluster-health signal rather than a nuisance.

The alert state itself is persisted in the database, which is what lets a restarted replica resume without re-firing everything from Normal. Still, a rolling restart of all replicas at once creates a gap during which nothing is evaluating, so stagger them and watch grafana_alerting_* metrics during a deploy.

There is a second deployment shape worth considering: a dedicated alerting Grafana. Some teams run one replica set for the human-facing UI and another, with execute_alerts true, for evaluation only, so a dashboard traffic spike cannot starve alert evaluation. The trade-off is one more thing to operate and keep configuration-consistent, which only pays off when alerting is treated as critical infrastructure.

The trust model

Security in Grafana starts from one uncomfortable fact, and everything else in this section is a consequence: a Viewer can run arbitrary queries against any data source in their organization, regardless of which dashboards they were shown. The dashboard is a convenience for humans, not an access boundary. Anyone with a Viewer role can open Explore, pick a data source and type their own query, or call /api/ds/query directly. If a data source holds the salary table, the customer logs or the production database, the question "who can read it" is answered by who can log in to that organization, not by which folders they can see.

Follow the consequences through. The credential Grafana stores for a data source is a shared credential. If it is a Prometheus bearer token that can read all tenants, then all Grafana users of that data source read all tenants. If it is a PostgreSQL user with SELECT on every schema, every Viewer can write SQL against every schema. The controls, in the order I would apply them:

1. Give the data source a least-privilege credential. A read-only database role scoped to the views the dashboards need. A Loki or Mimir tenant header that selects one tenant. A cloud service account with a narrow role. This is the only control that holds even if everything above it fails.
2. Split by sensitivity. Put the sensitive data source in its own organization, or its own Grafana instance, and give only the people who need it membership there.
3. Use data source permissions and Label-Based Access Control on Enterprise or Cloud, so a team may query a data source only if granted, and LBAC can restrict which label values a team sees. These are licensed features, and they are the honest answer when one org must serve several audiences.
4. Assume Explore exists. Do not rely on "we only gave them the dashboard link". A link is not a permission.

Anonymous access is the same point made starker. With [auth.anonymous] enabled = true, anyone who can reach the URL acts with the configured org role, and therefore reaches every data source in that org. It is sometimes right for a public status board in an isolated organization with data sources that hold only public data. It is never right on the org that holds your internal data sources.

Public dashboards are the sanctioned way to share a read-only view with unauthenticated people, but remember that they execute the dashboard's queries on the public viewer's behalf, so each public view is load on your data source and exposes whatever the panels return. Treat each one as a publication decision and keep an inventory.

Layered defences around that core, none of them optional at platform scale:

• Put Grafana behind a reverse proxy or an authenticating proxy such as NGINX, oauth2-proxy or Pomerium, terminate TLS there or set [server] protocol = https with cert_file and cert_key, and set [security] cookie_secure = true with an appropriate cookie_samesite.
• Leave allow_embedding false unless a specific trusted site must iframe Grafana, because embedding widens clickjacking and credential-exposure risk.
• Keep content_security_policy and strict_transport_security enabled in [security], and tighten CSP with a template if your plugins allow it.
• Use the data source proxy allow-list, data_source_proxy_whitelist, where Grafana should reach only known hosts, so a compromised editor cannot point a data source at your cloud metadata endpoint. Enterprise adds request security with host allow-listing for the same purpose.
• Set the initial admin password at first boot through GF_SECURITY_ADMIN_PASSWORD, and use disable_initial_admin_creation when you authenticate only through SSO, so no default admin account exists to brute-force.

A final trust point is plugins. Backend plugins execute code on the Grafana host. Install only signed plugins, keep allow_loading_unsigned_plugins empty in production, and treat any entry in it as an exception with an owner. Angular plugins no longer load at all as of Grafana 12, so an old community plugin that stopped working after an upgrade is not a bug you can toggle away.

Authentication, identities and service accounts

Grafana supports OAuth and OIDC providers (GitHub, GitLab, Google, Azure AD, Okta and a generic OAuth provider), LDAP, auth proxy, JWT and basic auth, and on Enterprise and Cloud also SAML, team sync, enhanced LDAP sync and SCIM provisioning. You cannot configure the same authentication type twice, though different types coexist. There is no built-in multi-factor authentication, which means MFA is your identity provider's job, and choosing SSO through an IdP that enforces it is the platform answer.

The settings that matter in production are subtle, and all of them are about what happens after the first login.

Role mapping. With OAuth, role_attribute_path maps a claim to an org role, and allow_assign_grafana_admin decides whether a claim may grant server admin, which you almost never want. If role_attribute_strict = true and the claim produces no valid role, login is refused with an error like no organization role, which is correct and which will still generate help-desk tickets on the day someone's group membership is misconfigured. skip_org_role_sync makes Grafana ignore the IdP's role on later logins, which is how you let admins adjust roles in Grafana without the IdP overwriting them at each login.

Session lifetime and revocation. [auth] login_maximum_inactive_lifetime_duration defaults to 7 days, and OAuth sessions can remain valid for up to 30 days after you revoke access at the IdP unless you enable use_refresh_token, which lets Grafana notice the revoked grant. For a platform holding sensitive data, turn it on, because offboarding that takes a month to bite is an audit finding.

Email-based account matching is convenient and risky. If Grafana links an incoming SSO identity to an existing user by email, then any IdP or provider that lets users set unverified emails becomes a way to take over another user's account. Use only providers where email is verified and unique, and prefer matching on a stable subject identifier.

Service accounts are non-human identities with an org role of Viewer, Editor or Admin and any number of tokens, prefixed glsa_. They replaced API keys, which were migrated to service accounts automatically in 11.x, and the old /api/auth/keys API is gone. They are scoped to one organization, cannot perform server-admin operations, and cannot run the grafana cli, which works against the local database. Token hygiene is where platforms slip: tokens do not expire unless you set an expiry, so a token created for a CI job two years ago still works unless you decided otherwise.

# Create a service account and a token that expires in 30 days
SA_ID=$(curl -s -X POST -H "Content-Type: application/json" -u "$ADMIN" \
  -d '{"name":"ci-dashboards","role":"Editor"}' \
  https://grafana.example.com/api/serviceaccounts | jq -r .id)

curl -s -X POST -H "Content-Type: application/json" -u "$ADMIN" \
  -d '{"name":"ci-token-2026-10","secondsToLive":2592000}' \
  "https://grafana.example.com/api/serviceaccounts/$SA_ID/tokens"

The discipline I suggest: one service account per automation and per environment, the lowest role that works, always a finite secondsToLive, tokens stored in a secrets manager rather than in CI variables that everyone can read, and a quarterly job that lists service accounts and tokens and revokes what has no owner. Enterprise RBAC lets you go below Viewer, Editor and Admin with custom roles, which is the right answer for a Terraform service account that should manage dashboards and folders but not users. Since Grafana 13.0, alerting RBAC enforcement has been tightened for custom roles, roles managed by Terraform and provisioning, particularly those with data-source-specific permissions, so re-test your custom roles after the upgrade instead of assuming they behave as before.

Secrets, encryption and the supply chain

Grafana encrypts the secrets it stores in its database: the secureJsonData of data sources and the secret fields of contact points such as webhook URLs and API keys. The key is [security] secret_key. Two rules follow, and they are the ones that cost teams their weekends.

The secret_key and the database are one backup unit. A database backup without the secret_key that encrypted it is a backup of data sources you can no longer connect to, because the secrets inside cannot be decrypted. Store the key in your secrets manager, back it up alongside every database snapshot, and test a restore that uses the two together.

Changing the key is not a config edit. If you edit secret_key and restart, Grafana cannot decrypt what it stored earlier. Rotation uses the envelope encryption mechanism: configure a provider under [security.encryption] backed by a KMS (AWS KMS, Azure Key Vault, Google Cloud KMS or Vault transit), so data keys are encrypted by a key that your cloud lets you rotate, and follow the documented re-encryption procedure. Confirm the exact keys and commands against the current documentation for your version before relying on them, since I have not reproduced them from the reference here.

Keep secrets out of the configuration files with Grafana's value expanders. $__env{VAR} reads an environment variable, $__file{/path} reads a file and trims the trailing newline, which is precisely what Kubernetes and Docker secrets mount, and $__vault{...} reads from HashiCorp Vault on Enterprise. There is also the GF_<SECTION>_<KEY>__FILE environment pattern. For a deeper treatment of the secrets store itself, see the Vault guide.

apiVersion: 1
prune: true
datasources:
  - name: Prometheus
    uid: prom-main
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s
    secureJsonData:
      httpHeaderValue1: $PROM_TOKEN      # injected from the environment, never committed

Audit logging of data source queries is controlled by log_datasource_query_request_body and log_datasource_query_response_body, and their defaults changed in Grafana 13.0. Request and response bodies can contain the very data you are protecting, so decide on purpose what goes into your audit log and where the log is stored, rather than inheriting a default.

The supply chain for Grafana has three strands. The Grafana image or package: pin an exact tag such as grafana/grafana-enterprise:13.2.3 in production and never latest, because patch releases arrive about every two weeks and are frequently security fixes (13.2.3, released 2026-09-29, contains three CVE fixes), and an unpinned image upgrades itself at the worst time. Verify packages through the repository signing keys the installation docs provide. Plugins: pin versions with GF_PLUGINS_PREINSTALL=id@version, prefer Grafana-signed, and mirror the ones you depend on internally if your environment cannot reach grafana.com. Dashboards as code: generated from a repository, reviewed in pull requests, not hand-edited in production.

Patch hygiene needs an owner and a clock. Subscribe to the Grafana security advisories, define an internal target such as "critical patches to production within seven days", and use the maintained branches: on 2026-09-29 patches shipped for 13.2.3, 13.1.7, 13.0.10 and 12.4.12, which tells you that you have room to stay on an older line, but not forever.

Multi-tenancy: from soft to hard

Every platform team is eventually asked "can team B see team A's dashboards and data", and the honest answer is a spectrum. Grafana gives you four isolation tiers, and choosing the lightest one that satisfies the real requirement is the skill.

Tier 1: folders and teams inside one org (soft tenancy). Folders are the main permission boundary for dashboards, library panels and alert rules. A team owns a folder and gets Edit or Admin there, everyone else gets View or nothing. It is cheap and flexible, it supports nested folders, and it isolates dashboards and alert rules, not data. Viewers of the org still reach every data source, as the trust-model section established.

Tier 2: data source permissions and LBAC (Enterprise or Cloud). This adds the missing piece inside one org: a team may use a data source only when permitted, and Label-Based Access Control restricts the label values a team can query, such as only their namespace. Combined with Mimir, Loki and Tempo tenant headers (X-Scope-OrgID, configured per data source as a custom header), it gives per-team data isolation in one Grafana. RBAC custom roles make the permissions composable.

Tier 3: organizations (hard-ish tenancy). An organization isolates dashboards, data sources, teams, service accounts and tokens. A user can belong to several orgs with a different role in each, and org management needs a server admin. The limits are important: nothing is shared across orgs, so a dashboard you want everywhere has to be provisioned into each; Git Sync and provisioning are configured per org; and the isolation is logical inside one process, one database and one set of server settings. A server admin sees everything, and a vulnerability in Grafana itself affects all orgs equally.

Tier 4: separate instances (hard tenancy). When tenants are mutually untrusting, are in different regulatory zones, or have different upgrade cadences, give each their own Grafana, ideally from the same Helm chart or Terraform module so the operational cost stays sub-linear. For employers in the Gulf and Egypt this is a common shape: one instance per country or legal entity so dashboards and their cached data stay in the required region, with a thin layer above for shared content.

Platform hygiene inside any tier: give every team a folder, an owning team and a contact; set quotas under [quota] for dashboards per org and users so one tenant cannot fill the database; and decide up front how shared things such as the golden-signals dashboard for the organization get distributed to every org or folder.

Configuration as code: provisioning, Terraform and Git Sync

A platform that is configured by clicking is one you cannot rebuild, review or audit. Grafana offers several ways to declare configuration, and the senior skill is picking one owner per resource type and preventing two writers for the same thing.

File provisioning reads YAML at startup for data sources, dashboard providers, plugins and alerting resources from /etc/grafana/provisioning. It is simple, works anywhere, and is what the Helm chart's sidecars and ConfigMaps build on. Its behaviours bite in operations. Provisioned data sources set editable: false so the UI cannot drift from the file. prune: true deletes provisioned data sources removed from the file. Dashboards from a file provider cannot be saved from the UI by default, returning Cannot save provisioned dashboard unless allowUiUpdates: true, in which case edits persist until the file is re-read. The version field on a data source stops an older instance from overwriting newer settings during a rolling update. Always set an explicit, stable uid on data sources and dashboards, or references between them break when a provisioned object is recreated, and as of 13.0 the numeric-ID data source APIs are disabled by default so UIDs are the only reliable handle.

Terraform with the grafana/grafana provider manages folders, data sources, dashboards, alert rule groups, contact points and notification policies, and since 13.2 also saved queries. It gives you plan and apply, drift detection, and a review workflow, and it fits teams that already run Terraform. See the Terraform guide. Its sharp edge is alerting: notification policies are a single tree per org, so two Terraform states that both manage grafana_notification_policy will overwrite each other. One state owns the tree.

Git Sync, generally available since 13.0, keeps dashboards in a Git repository and a Grafana folder in bidirectional sync: edits in the UI become commits or pull requests, merges in Git appear in Grafana. It polls every 60 seconds by default, and webhooks make syncing near-instant; 13.1 added root-level sync, verified commits and README rendering, and 13.2 added GitHub Enterprise, GitLab and Bitbucket webhooks plus commit author override. It depends on Unified Storage. It suits dashboard authors who want pull-request review without learning Terraform, but remember the 13.0.0 warning covered later: migrations with Git Sync enabled are where a bug once lost dashboards.

The Grafana Operator reconciles Grafana, GrafanaDashboard and GrafanaDatasource custom resources in Kubernetes, which is natural if your whole estate is GitOps with Argo CD or Flux; see the Argo CD guide and the Helm guide. The Helm chart's sidecars do a similar job with ConfigMaps labelled grafana_dashboard and grafana_datasource.

A decision table, because "it depends" is not a platform answer:

The rule that prevents most drift is one writer per resource. A dashboard managed by Git Sync is not also edited by Terraform; a policy tree owned by Terraform is not also edited in the UI. Mark provisioned resources as read-only in the UI and explain how to propose a change, or engineers will change production by hand at 3 a.m. and the next apply will silently undo it.

For recovery, provisioned configuration is the cheap part of a disaster: a new instance plus the repositories reproduces everything as code. The expensive part is what lives only in the database (users, exploratory dashboards, annotations, alert state), which is why the database backup and the secret_key remain essential even in a fully as-code shop.

Upgrades and migrations

Grafana's stated principle is that you can upgrade from any supported release to another supported release and that your dashboards carry on. In practice the risk is concentrated in a few places: the database schema migration at startup, removed features, and the behaviour changes that the "What's new" pages list. Since 12.0 there is no separate breaking-changes page; the breaking changes are inside each release's What's new, so reading them is a required step, not a courtesy.

The cadence makes upgrades a routine rather than an event. A minor release arrives every six to ten weeks (12.4.0 on 2026-02-25, 13.1.0 on 2026-07-01, 13.2.0 on 2026-08-18), and patches roughly every two weeks, often security-motivated. On 2026-09-29 the maintained lines with patches were 13.2.3, 13.1.7, 13.0.10 and 12.4.12. Staying a minor behind is fine. Staying a year behind means a large, risky jump with every breaking change arriving at once.

The rollback truth first. Startup runs database migrations automatically, and downgrades are not supported after a schema migration has run. Adopting the dynamic dashboards schema v2 specifically blocks downgrade. So your rollback plan is not "deploy the old image". It is "restore the database snapshot taken before the upgrade, and deploy the old image against it". If you have never restored that snapshot in a rehearsal, you do not have a rollback plan.

The upgrade procedure I would hold a platform to:

1. Read the What's new pages for every version between yours and the target, and write down each item that touches your estate.
2. Back up the database and the secret_key and grafana.ini together, and verify the backup restores.
3. Rehearse in staging with a restored copy of production's database, not an empty instance, because real data is what triggers migration problems.
4. Upgrade one replica first, so the migrations run once, and let it finish before starting the others. Several new-version replicas starting at once against one database race on migrations.
5. Run smoke checks: /api/health shows the new version and "database":"ok", data sources pass their health check, a provisioned dashboard loads, an alert rule evaluates, a notification test delivers, a render works.
6. Roll the rest one at a time, watching error rate and alert evaluation.

# Smoke checks after the first upgraded replica is up
curl -s https://grafana.example.com/api/health
# {"database":"ok","version":"13.2.3","commit":"..."}

# Every data source still healthy? (UIDs, not numeric IDs, since 13.0)
for uid in prom-main loki-main tempo-main; do
  printf '%s ' "$uid"
  curl -s -H "Authorization: Bearer $TOKEN" \
    "https://grafana.example.com/api/datasources/uid/$uid/health" | jq -r .status
done

The 12 to 13 checklist

Grafana 13.0 is the largest breaking release in years, and it is worth carrying the whole list in your head because your teams' scripts will hit several items.

The Helm repository also moved: the Grafana chart now lives at https://grafana-community.github.io/helm-charts, effective 2026-01-30, and the old https://grafana.github.io/helm-charts chart is deprecated and stops getting updates. A pipeline that still points at the old URL does not fail loudly; it simply stays on an old version while you believe you are current.

# Move a Helm-managed Grafana to the current chart repository
helm repo remove grafana 2>/dev/null
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm upgrade my-grafana grafana-community/grafana \
  --namespace monitoring --reuse-values \
  --set image.tag=13.2.3

Two smaller moves deserve a mention. The Docker docs now recommend the grafana/grafana-enterprise image, which contains every OSS feature and runs free without a licence, while grafana/grafana remains the OSS image; decide which you run and pin it. And since kube-prometheus-stack bundles its own Grafana subchart, check which chart repository and version that bundle pulls in, rather than assuming your direct chart is the only Grafana.

Migrating dashboards: schema and API

Dashboards in 13 are all Scenes-powered, and opening an existing dashboard can migrate it to the newer schema used by dynamic dashboards, with rows, tabs and auto-grid layouts. Because v2 adoption blocks downgrade, treat "everyone has been opening dashboards since the upgrade" as the point of no return, and do not let an upgrade's staging rehearsal be the last time you look at dashboard JSON in version control. The newer API lives under /apis/dashboard.grafana.app/v1/namespaces/:namespace/dashboards (and a v2 endpoint for the new schema), with the namespace default for org 1 and org-<id> for others.

# Create or update a dashboard through the resource API (v1 schema)
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://grafana.example.com/apis/dashboard.grafana.app/v1/namespaces/default/dashboards \
  -d '{
    "metadata": {
      "name": "svc-checkout-red",
      "annotations": { "grafana.app/folder": "team-checkout" }
    },
    "spec": { "title": "Checkout RED", "schemaVersion": 41, "panels": [] }
  }'

A 409 here means the UID already exists, which is the useful answer to "did somebody else create this". The legacy POST /api/dashboards/db with overwrite: true still works in 13 but no longer receives updates, so new automation should target /apis. I am describing the path and body shape from the reference documentation; confirm the exact schemaVersion your target release expects before automating.

Cost: what a Grafana estate really costs

Grafana OSS, and Enterprise without a licence key, are free to run. The bill is the infrastructure and the people. It falls into four buckets, and the largest one is usually not the one teams watch.

The data behind it dominates. Storing and querying metrics, logs and traces costs far more than serving their dashboards, and the cost scales with series cardinality, log volume and retention, none of which Grafana controls. The cost control that matters most to a Grafana owner is therefore at the dashboard layer, because queries are where Grafana spends other people's money: heavy queries cost backend CPU, and high-cardinality queries can cost real money on a usage-billed backend.

Compute and database for Grafana itself is comparatively small, following the sizing above, plus a PostgreSQL instance sized for connections rather than data, a Redis if you use it, and renderer nodes if you render at volume.

Licences if you need Enterprise features: RBAC, data source permissions, SAML, team sync, reporting, audit logs, query caching, white labelling, enterprise data sources such as Splunk or Datadog. Buy them for a named requirement, not for comfort. On Grafana Cloud the model is usage-based across active series, log and trace volume and users, with a free tier, so cardinality discipline becomes a literal invoice line.

People: the on-call, upgrades every six to ten weeks, patch rollouts, and the consulting hours the platform team spends helping others write a PromQL query. This is the cost that turns into a headcount argument, and it is the one you reduce with golden paths and self-service.

Cost levers that work, roughly in order of payoff:

Retiring dashboards is a governance act more than a cost one. Usage insights (Enterprise) tell you what is viewed. Without it, a practical approach is a scheduled job that lists dashboards and last-updated times through the API and posts a report to owners, with a quarantine folder before deletion. 13.0 also brought restoration of deleted dashboards, which lowers the fear of deleting.

Observing Grafana itself

A platform has to be able to answer "is it healthy" and "who is hurting it" from its own telemetry, and it has to survive the paradox that a monitoring tool cannot alert you that it is down.

Turn on Grafana's own metrics with [metrics] enabled = true, then scrape /metrics from every replica with Prometheus, using basic_auth_username and basic_auth_password under [metrics] so the endpoint is not open. Backend plugin metrics are at /metrics/plugins/<plugin_id>. Since the HTTP request duration metric grafana_http_request_duration_seconds is exposed as both a native histogram and classic buckets, set classic_http_histogram_enabled = false to cut cardinality once your Prometheus handles native histograms. For traces, [tracing.opentelemetry.otlp] sends OTLP traces for API requests with context propagated to data sources, so a slow panel can be followed into the backend; see the OpenTelemetry guide. Log as JSON for your aggregator, and keep level at info with targeted filters for debugging rather than global debug.

What to alert on, from the Prometheus that scrapes Grafana:

groups:
  - name: grafana-platform
    rules:
      - alert: GrafanaDown
        expr: up{job="grafana"} == 0
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Grafana replica {{ $labels.instance }} is not scraped"
          runbook_url: https://runbooks.example.com/grafana/down

      - alert: GrafanaSlowRequests
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(grafana_http_request_duration_seconds_bucket[5m]))) > 2
        for: 10m
        labels: { severity: ticket }
        annotations:
          summary: "Grafana p95 request latency above 2s for 10 minutes"
          runbook_url: https://runbooks.example.com/grafana/slow

Those rules depend on the metric names in your version, so check the names your /metrics endpoint actually exposes rather than copying mine blindly, particularly since 13.0 deprecated several database metrics in favour of standard Go SQL ones.

Two structural rules. Meta-monitor from outside. The rule that tells you Grafana is down cannot live in the Grafana that is down. Evaluate it in Prometheus with a separate Alertmanager or a hosted service, and ideally add a dead man's switch: an always-firing alert that pages when it stops arriving. Watch the alerting pipeline, not just the server. Count rule evaluation failures, notification failures per contact point, and the number of DatasourceNoData and DatasourceError alerts, because a healthy server with a broken Slack token looks identical to a quiet night.

Grafana Advisor, generally available in 13.0, runs weekly checks for failing data sources, outdated plugins and misconfigured SSO, and it is a cheap first line of hygiene that a platform team should read on a schedule. It does not replace your own metrics, and it is not a paging source.

Alerting at platform scale: governance and storms

When a platform owns alerting for forty teams, the alert rule is no longer the unit of concern. Conventions, routing and volume are. A rule is cheap to write and expensive to own, and most alerting pain is governance pain.

Routing by labels, with mandatory labels. The notification policy tree routes on labels, so the contract you publish is a set of required labels on every rule: team, severity and service at minimum, with the root policy catching anything that lacks them and sending it to a platform "unrouted" channel, which is a feature because it makes badly labelled rules visible. Each team gets a child policy matching team=<name> pointing at that team's contact point, with group_by set sensibly. The defaults are group wait 30 seconds, group interval 5 minutes and repeat interval 4 hours, and children inherit unless they override, so a four-hour repeat on a severity-page route is usually too long and a thirty-second group wait is usually fine.

Templates and runbooks. Message templates are Go templates shared across contact points, and a platform should ship one standard template that always renders the summary, the labels that matter, and the runbook_url annotation. Enforce the annotation in review: an alert without a runbook is a question nobody on call can answer at 3 a.m.

The no-data and error storm. A rule whose query fails or returns nothing produces a synthetic alert named DatasourceNoData or DatasourceError, carrying the rule name, rule UID and datasource_uid labels. If a data source goes down, every rule that uses it fires one, and forty teams get forty pages for one cause. The platform answer is threefold: set per-rule handling where "no data" is normal (Set Normal or Keep Last State), route the DatasourceNoData and DatasourceError alerts to the platform team through a dedicated policy, and add an inhibiting design where one data source health alert owns the incident. Read the grafana_state_reason annotation for why a state was set, and remember that the rule health (ok, nodata, error) is separate from the instance state.

Silences and mute timings stop notifications, never evaluation. A silence is ad hoc, matcher-based and time-boxed, which makes it the maintenance-window tool and also the one people forget; list active silences and their authors in your weekly review. Mute timings are recurring and belong to the policy tree, so they need the same change control as the tree.

Data source-managed versus Grafana-managed rules. Grafana-managed rules can query any data source and are evaluated by Grafana, which makes them flexible and ties them to Grafana's availability and load. Data source-managed rules, such as those in the Prometheus or Mimir ruler, are evaluated by the backend itself and survive a Grafana outage. For critical infrastructure alerts, evaluating in Prometheus or Mimir and delivering through an Alertmanager is the more resilient architecture, while Grafana-managed alerting earns its place where rules need cross-source queries or where teams want self-service. There is also an alert rule migration tool, added in 12.0, for bringing Prometheus-compatible rules across. See the Prometheus guide for the ruler side.

A routing tree as code, which is how I would keep it reviewable:

apiVersion: 1
policies:
  - orgId: 1
    receiver: platform-unrouted
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: platform-datasource-health
        object_matchers:
          - ["alertname", "=~", "DatasourceNoData|DatasourceError"]
      - receiver: team-checkout-pager
        object_matchers:
          - ["team", "=", "checkout"]
          - ["severity", "=", "page"]
        repeat_interval: 1h
      - receiver: team-checkout-slack
        object_matchers:
          - ["team", "=", "checkout"]

Since policy trees are a single object per organization, this file is the only writer, and every change is a pull request with a matching test: a labelled synthetic alert in staging that must land in the right receiver. Verify the exact provisioning key names against the alerting provisioning page for your version, since the file format has evolved across releases.

Incident playbooks

Platform incidents repeat. Write the playbooks down, because the person on call at 3 a.m. will not recall the flag names. Each playbook below is a first-ten-minutes sequence: what to look at, in what order, and the decision it leads to.

Playbook 1: "Grafana is down or very slow for everyone"

1. curl -s https://grafana.example.com/api/health. If it returns "database":"failing", this is a database incident, not a Grafana one. Go to the database team's runbook and stop looking at dashboards.
2. If health is ok but pages are slow, check replica count and load balancer health. A single surviving replica under full load looks like a slow service.
3. Look at the database: connections in use against max_open_conn, long-running queries, locks. A migration holding a lock after an upgrade is a classic.
4. Check recent change: was a replica set upgraded, a plugin installed, a provisioning change merged? Roll back the last change before debugging.
5. Only then look at a data source. A single dead Prometheus makes many panels red, but it does not make the login page slow.

Playbook 2: "Alerts are not arriving" or "everyone got paged three times"

1. Distinguish the two. Duplicates mean the alerting cluster is split; silence means evaluation, routing or delivery.
2. Duplicates: check ha_peers, port 9094 TCP and UDP between replicas, and the headless Service. One missing peer is enough.
3. Silence: find the rule's state and health. If health is error or nodata, the rule is failing to evaluate, and the fault is in the query or data source.
4. If the rule fires, follow the routing: does the label set match a policy, is a silence or mute timing active, does the contact point's test notification work? Failed delivery is usually an expired token, a blocked egress path or a renamed webhook.
5. After a Grafana 13 upgrade, check RBAC: a provisioning or Terraform role that lost alerting permissions fails quietly in pipelines.

Playbook 3: "Dashboards show No data after the upgrade"

1. Open Explore on one affected data source and run a trivial query. If that works, the data source is fine and a panel query or variable is broken.
2. Check for removed or deprecated metric names if the dashboard graphs Grafana's own metrics, and for API changes if a script creates the dashboards.
3. Check whether the data source UID changed because it was recreated, since panels reference UIDs and a recreated data source with a new UID orphans every dashboard that used the old one.
4. If dashboards or folders are missing after a 12 to 13 jump with Git Sync on, stop: do not make further changes, preserve the database, and restore the pre-upgrade snapshot onto a fixed patch.

Playbook 4: "A credential leaked"

1. Rotate at the source first, whether that is a service account token, a data source password, a webhook URL or the IdP client secret. Deleting the file from Git does not help; the secret is already cloned.
2. Find what it could reach: query the audit log and the backend's access logs for its use in the window.
3. If secret_key itself is in doubt, plan the KMS and re-encryption procedure deliberately, not in a panic, because a wrong edit makes every stored secret undecryptable.
4. Write the post-incident change: a scanner in CI, a short token expiry, a scope reduction.

Playbook 5: "Reports and alert images are blank"

1. Confirm the remote renderer is reachable from Grafana at [rendering] server_url, and that Grafana is reachable from the renderer at callback_url.
2. Check renderer_token: with JWT auth on by default since 13.0 it must be a non-default value, identical on both sides. A 401 or Failed to send request to remote rendering service is usually this.
3. On RPM-based renderer images, missing glyphs in screenshots mean missing fonts such as fontconfig, freetype and urw-fonts.

Where Grafana stops

A good platform owner can say what the tool is not for, and Grafana has clear limits.

It is not a datastore. If someone wants to keep data in Grafana, what they want is Prometheus, Mimir, Loki, Tempo, Elasticsearch or a database, and Grafana in front of it. The Prometheus guide and the ELK Stack guide cover two of the usual answers.

It is not a collector. Getting telemetry out of applications is the job of instrumentation and a collector. Grafana Alloy is Grafana's OpenTelemetry Collector distribution and the successor to the end-of-life Grafana Agent; the OpenTelemetry guide covers the instrumentation side. Grafana only reads what arrives.

It is not the highest-availability alerting you can build. Grafana-managed alerting is evaluated inside a stateful web application that shares a database with the UI. For the alerts that decide whether the business is up, evaluate in Prometheus or Mimir with an Alertmanager that has no dependency on Grafana, and keep Grafana alerting for the long tail and for self-service. Pair it with an on-call and incident tool rather than treating contact points as an incident process.

It is not a hard multi-tenancy boundary inside one org. Covered above: folders isolate dashboards, not data, and organizations share a process and a database.

It is not a BI tool for arbitrary business analytics. Grafana can chart SQL beautifully, and many teams run business dashboards on it, but if the need is governed metrics, semantic layers, row-level security over large analytical models, scheduled exports and self-service for non-engineers, a BI product serves that better. For ML specifically, experiment tracking and model comparison belong in MLflow or Weights & Biases; Grafana is for the operational signals around serving, pipelines and GPUs.

It is not where you store configuration of record. Configuration belongs in Git. The database is a runtime cache of what you declared, plus the things you chose not to declare.

Running Grafana as a platform

Owning Grafana for other teams is product management as much as operations. Your users are engineers with deadlines, and the platform succeeds when the right thing is the easy thing.

A golden path. Publish a template repository or Terraform module that gives a team a folder, a team, a service account, a starter RED dashboard for their service, an alert rule group with the mandatory labels, and a runbook stub, in one pull request. Teams that start from it inherit the conventions without reading a wiki. Official best practices help here: the USE method for infrastructure, RED for services, and the Four Golden Signals, with alerts built on RED dashboards, and a maturity model running from uncontrolled dashboards through template variables and hierarchy to scripted generation and deliberate sprawl reduction.

Dashboards as reviewed artifacts. Keep dashboard JSON in version control, link dashboards to each other, avoid duplicating near-identical dashboards in favour of template variables, delete obsolete ones, and keep a separate test environment. With Git Sync or provisioning this is enforced by the path, not by exhortation.

Quotas and limits. Use [quota] settings to cap dashboards per org and users per org, min_refresh_interval for refresh, and data source query timeouts so one runaway panel cannot hold a backend connection for ten minutes. Publish these limits, with the reason, before the first person hits one.

A support model. Decide what the platform team does (runs Grafana, owns SSO, plugins, upgrades, the golden path, the alerting tree) and what teams do (own their dashboards, alerts and runbooks). Put it in a one-page agreement, run an office hour, and track request turnaround, because, as with shared base images elsewhere in this series, teams fork the platform when its responses are slow.

Change management. Announce upgrades with the What's new items that affect each team, give a staging window, and publish the breaking-change table above as a migration guide with owners. The 13.0 list is a good template.

Metrics for the platform itself. Uptime of the UI, p95 page and query latency, alert delivery success rate, data source health failures, age of the running version against the newest patch, number of dashboards with no recent views and no owner, and time to onboard a new team. "Days behind the latest patch" is the single most honest number for a security-minded platform.

Plugin and data source catalogue. Keep an allow-list of approved data sources and plugins, with an owner for each, a pinned version, and a review date. Requests for a new plugin become a short security review because a backend plugin is code on your host.

The review checklist

Use this for pull requests that touch the platform and for design documents of anything that adds a new Grafana dependency. Each item is a question with a reason.

Architecture

• Is the database PostgreSQL or MySQL, backed up, with a rehearsed restore? SQLite cannot be shared between replicas.
• Are there at least two replicas behind a load balancer health-checking /api/health?
• Is ha_peers set, with 9094 open on TCP and UDP, or Redis configured for alerting HA?
• Is the remote renderer deployed, with the same renderer_token on both sides?
• Is root_url the public URL, and is proxy compression not doubled with enable_gzip?

Scale

• Is min_refresh_interval set deliberately, and are heavy expressions recording rules?
• Are pool sizes max_open_conn times maximum replicas within the database's connection limit?
• Is the rule evaluation load understood, with ha_single_node_evaluation decided rather than defaulted?

Security

• Does every data source use a least-privilege credential?
• Is anonymous access off on every org that holds internal data?
• Are plugins signed and pinned, with allow_loading_unsigned_plugins empty?
• Is MFA enforced at the IdP, and is use_refresh_token on?
• Are service accounts single-purpose, finite-lived and owned?
• Is secret_key stored in a secrets manager and backed up with the database?
• Is the image or package pinned to an exact version, with a patch SLA?

Configuration

• One writer per resource? Explicit UIDs? No literal secrets?
• Does the Helm chart come from grafana-community?
• Is the notification policy tree owned by exactly one state?

Alerting

• Do all rules carry team, severity, service and runbook_url?
• Where do DatasourceNoData and DatasourceError go?
• Is Grafana itself monitored from outside Grafana, with a dead man's switch?

Upgrades

• Has the What's new list been read and mapped to owners?
• Was a snapshot restored in staging, and is the first-replica-first rollout written down?
• Are scripts free of grafana-cli, grafana-server, numeric-ID data source calls and old chart URLs?

The complete picture

Here is the whole platform as one flow, from the change a team wants to make to the signal that proves it worked. Every step is a place where something in this guide applies.

Walk a realistic change through it. A team adds a dashboard and an alert for their new service. They open a pull request against their team repository from the golden-path template. Review checks that the alert has team, severity and runbook_url, that panels use $__rate_interval, and that the expensive expression is a recording rule. Merge triggers Git Sync for the dashboard and Terraform for the alert rule group; the notification tree is untouched because one state owns it. Grafana's replicas serve the dashboard through the shared database, the alert rule evaluates once per replica or once in total by your chosen mode, and the data source credential behind it can read only what that team should see. A test notification lands in the team's channel. Meanwhile an external Prometheus scrapes /metrics from every replica and would page you from outside Grafana if any of this went down, and Grafana Advisor reports weekly on the data sources and plugins.

Now the same change going wrong, because that is where the knowledge earns its keep. The alert rule has no team label, so it lands in platform-unrouted and you see it on day one. The panel query is an unbounded rate() at a five second refresh, and the review comment catches it, or the backend query log does a week later. The data source token belongs to a departed engineer, and the service account inventory shows it. The team uses a numeric-ID API call in their pipeline, and the 13.x upgrade staging run fails it before production. Each failure was caught by a control you can name, which is what "platform" means.

The numbers worth memorising for interviews and for on-call:

Where the series leaves you

Across the three levels you have moved from opening Grafana and connecting a first data source to owning it as a platform. The same topics ran through all three, each time with more consequence.

You should now be able to look at a Grafana deployment and see what it permits and what it risks: whether the database is shared and recoverable, whether the alerting cluster is actually a cluster, whether a Viewer can reach data they should not, whether the secret_key is backed up with the database, whether the next upgrade has been rehearsed, and whether Grafana can tell you it is down.

Carry the review instinct out of here. As with the other platforms in this series, most Grafana incidents are decided at review time: in the label that was missing, the credential that was too broad, the upgrade that was never rehearsed. Next, see how the signals it displays are produced in the Prometheus and OpenTelemetry guides, how it is deployed in the Helm and Kubernetes guides, and how its secrets are kept in the Vault guide.

Sources

• Grafana documentation
• What's new in Grafana
• What's new in Grafana v13.0
• What's new in Grafana v13.1
• What's new in Grafana v13.2
• What's new in Grafana v12.0
• Grafana releases on GitHub
• Image renderer plugin support removed
• Rendering default auth mode changed to JWTs
• Install Grafana
• Run Grafana on Docker
• Deploy Grafana on Kubernetes with Helm
• Helm repository migration to grafana-community
• Configure Grafana
• Feature toggles
• Configure security
• Configure authentication
• Set up Grafana for high availability
• Set up Grafana monitoring
• Upgrade Grafana
• Set up image rendering
• Grafana CLI
• HTTP API reference
• Dashboard HTTP API
• Provision Grafana
• Service accounts
• Roles and permissions
• Grafana Alerting
• Alerting fundamentals
• Alert rule state and health
• Notification policies
• Git Sync
• Dashboard best practices
• Troubleshooting

Part three of three. A senior Grafana interview is rarely about which panel type to use. It is about whether you can design a platform other teams depend on, stay calm when it breaks, and argue for the unglamorous work to people who control the budget. About fifty minutes. System-design prompts first, then incident questions, then the leadership conversations, then a rapid-fire round.

How senior answers are judged

Interviewers at this level listen for four things, and you can build each into every answer.

You state the constraint before the design. "How many teams, how many concurrent viewers, which data sources, what happens to us if Grafana is down, which regions must data stay in." A design offered before those questions signals that you design from habit.

You name the failure mode of your own choice. Every option has one. Say it unprompted: "Organizations give us logical isolation, but one process and one database, so a server admin sees everything and a Grafana vulnerability affects every tenant."

You separate the layers. The two most common confusions are the database versus the data source, and the dashboard permission versus the data permission. A candidate who keeps those straight is usually sound elsewhere.

You know what changed recently. Grafana 13 removed a lot. Mentioning the renderer plugin, JWT render auth, UID-only data source APIs, the removed CLI names and the 13.0.0 migration bug shows you run the upgrade path for real rather than read about it.

The thirty-second version of the whole platform

Grafana is a stateless-ish Go web application that queries data where it lives and stores only its own metadata in a SQL database. You scale it by running replicas behind a load balancer on a shared PostgreSQL or MySQL, and alerting needs a gossip or Redis cluster so replicas dedupe notifications. The real limits are the data sources behind it, the trust model (a Viewer can query any data source in the org), the upgrade path, and the fact that the tool that tells you everything is down cannot tell you that it is.

If you can say that in thirty seconds and then unpack any clause on request, you have most of the interview.

System-design prompts

Prompt 1: "Design Grafana for forty product teams across two countries"

What a weak answer does: says "we'll install Grafana on Kubernetes with Helm and give every team a folder," then stops.

A strong answer starts with questions. Are teams mutually trusted? Must data stay in-country for regulatory reasons, which is common for employers in the Gulf and in Egypt? What do the data sources look like: one shared Prometheus and Loki, or per-team tenants? Is there an SSO provider, and does it enforce MFA? Is alerting expected to survive a Grafana outage? What is the budget for licences?

Then the design, layered:

1. Topology. One Grafana instance per legal or data-residency zone, each a set of at least three replicas behind a load balancer health-checking /api/health, all sharing one managed PostgreSQL per zone. Stateless replicas, database as the shared fate.
2. Tenancy. Inside each instance, a folder per team with explicit permissions, because it is the cheapest tier. Data isolation comes from per-tenant data source credentials and, on Enterprise, data source permissions and Label-Based Access Control. If two business units are mutually untrusting, they get separate organizations, and if they are regulated differently, separate instances. I say plainly that folders isolate dashboards, not data.
3. Identity. SSO through the corporate IdP with MFA enforced there, since Grafana has none. Role mapping through role_attribute_path, use_refresh_token on so revocation propagates, SCIM or team sync where licensed so joiners and leavers flow automatically.
4. Configuration as code. Data sources, folders, permissions and the notification policy tree in Terraform with one state owning each; team dashboards through Git Sync or a provisioned repository. One writer per resource.
5. Alerting. ha_peers with a headless Service and port 9094 open for TCP and UDP. Mandatory labels on rules, a policy tree routing on team and severity, and critical infrastructure alerts evaluated in the Prometheus or Mimir ruler so they survive a Grafana outage.
6. Operations. Metrics scraped from every replica by a Prometheus that sits outside this Grafana, a dead man's switch, database backups with the secret_key stored alongside, and a rehearsed restore.
7. Golden path. A module that onboards a team with a folder, a service account, a starter dashboard and a routed alert in one pull request.

Close by naming the trade-off: "this costs us two instances to operate instead of one, which I accept because the alternative is arguing about data residency in an audit."

The follow-up to expect: "Why not one org per team?" The answer is that nothing is shared across orgs, so shared dashboards must be provisioned into each, server admin is needed for management, Git Sync and provisioning are per-org, and the isolation is logical rather than physical. Organizations are right for a few audiences, not for forty teams.

Prompt 2: "Our Grafana is slow. Dashboards take twenty seconds to load"

This is a diagnosis prompt dressed as a design prompt, and the order of your steps is the answer.

1. Is it Grafana or the data source? Open the browser network tab. If /api/ds/query calls are slow, the time is in the backend. If the dashboard JSON call itself is slow, it is the database.
2. If the backend: identify which queries. Look for long ranges with small steps, high-cardinality by clauses, rate() with a fixed [1m] instead of $__rate_interval, and too many panels refreshing at once. Check the backend's own query log and its limits, since 429 and context deadline exceeded come from Loki, Mimir or Prometheus, not Grafana.
3. If the database: connection use against max_open_conn, long queries, lock waits, database CPU, and whether an upgrade migration is still running.
4. Then the multiplication. Panels times viewers times refresh rate. Forty people with a 24-panel dashboard at a 10 second refresh is nearly six thousand queries a minute.
5. Fixes in order of cost: raise min_refresh_interval, move heavy expressions to recording rules, cap max data points, turn on query caching if licensed, and only then add replicas or database capacity.

A strong candidate adds: "Adding Grafana replicas would not have helped, and in default HA mode it would have multiplied alert evaluation against the same struggling backend."

Prompt 3: "Design alerting for a payments platform that must never miss a page"

The word "never" is the prompt. A strong answer refuses the premise gently and then designs around it.

• Two evaluation paths. The most critical rules (payment success rate, settlement lag) are evaluated by the Prometheus or Mimir ruler with its own Alertmanager cluster, which has no dependency on Grafana's database or UI. Grafana-managed alerting covers cross-source and long-tail rules.
• A dead man's switch. An always-firing alert that pages when it stops arriving, sent to an external heartbeat service. This catches Grafana down, network down, Alertmanager down, and a broken contact point in one signal.
• Two notification channels for the highest severity, for example an on-call tool plus an SMS or voice path, so one vendor's outage does not silence you. Contact points also need periodic test notifications, since a token that expired last month looks identical to a quiet night.
• No-data handling as a decision. For rules where silence means trouble, "No Data" should alert, not normalise. For rules where silence is normal, set Normal. The default produces synthetic DatasourceNoData alerts, which I route to the platform team by label so a data source outage is one incident and not forty.
• Routing tested in CI. A synthetic alert per route in staging that must reach the right receiver, run on every change to the policy tree.
• Alert quality. Every paging rule must have a runbook_url, an owner and evidence it fired usefully in the last quarter. Alerts nobody acts on get demoted or deleted.

The trade-off to name: external rulers lose some convenience, such as rules that join across data sources, and you must maintain rules in two places. You accept that for the alerts where independence is the requirement.

Prompt 4: "Plan an upgrade from Grafana 12.4 to 13.2 for the whole estate"

A senior answer is a plan, not an upgrade command.

• Inventory first. What runs: versions per environment, Helm chart source, plugins, Git Sync usage, custom RBAC roles, automation that calls the API, renderer setup, dashboards on Grafana's own metrics.
• Read the notes. The What's new pages for 13.0, 13.1 and 13.2, extracting breaking items per owner.
• Pick the target patch, not x.0.0: go to 13.2.3 or whatever is latest. The 13.0.0 Git Sync migration bug is the reason.
• Rehearse on a restored copy of the production database, timing the restore and the migration.
• Fix the known breakages before the window: move scripts from grafana-cli and grafana-server to grafana cli and grafana server; switch numeric-ID data source calls to UIDs; deploy the standalone renderer and set a non-default renderer_token on both sides; set or remove proxy compression because enable_gzip is now true; move the Helm repository to grafana-community; replace Angular plugins; update dashboards or alerts referencing removed metrics; audit custom RBAC roles.
• Execute: snapshot database and secret_key; upgrade one replica and wait for migrations; smoke tests (health, data source health by UID, a render, an alert test, a provisioned dashboard); then roll the rest.
• Rollback is a restore, because downgrades are unsupported after a migration and v2 dashboards block downgrade. State that the decision point for rollback must be made before users begin saving new dashboards.

Mention the cadence to close: "After this we move to an upgrade every minor, so the next one is small."

Prompt 5: "A security review says Viewers can read the production database through Grafana. Is that a bug?"

It is not a bug, it is the model, and the answer shows whether you understand it. A Viewer can run queries against any data source in the org through Explore or the query API, regardless of which dashboard they were given. So the real control is the data source's stored credential: a read-only role limited to the views the dashboards need. Beyond that, move the sensitive data source into its own organization or instance, use data source permissions and LBAC if licensed, and audit who belongs to that org. Also mention enabling audit logging with a deliberate choice about request and response bodies, since their defaults changed in 13.0 and the bodies can contain sensitive data. The mistake to avoid is proposing "hide the Explore menu", which is cosmetic.

Incident-response questions

Incident questions are about sequence. Always answer with the first three actions and what each decides.

"Everyone got paged three times for one alert"

The alerting cluster is split: each replica believes it is alone and sends its own notification. First check ha_peers and whether port 9094 is open between replicas for TCP and UDP; in Kubernetes confirm the headless Service resolves to all pods. Second, check the Alertmanager cluster status for peer count. Third, look at what changed, such as a new replica or a network policy. It is an incident about the cluster, not about the rule, and the fix is configuration. Prevention is a deploy-pipeline test that fires a synthetic alert and asserts exactly one notification.

"Grafana returns errors after we moved the database to a new host"

Start with /api/health. If it says "database":"failing", it is connectivity or credentials. Check that the [database] settings point at the new host, the user has DDL privileges if a migration will run, TLS settings match (ssl_mode), and security groups or network policies allow the path. Then check that the restored data was encrypted with the same secret_key: if data sources fail to decrypt after the move, the key did not travel with the backup. This is the answer that shows you know the two are a unit.

"After the upgrade, half the dashboards are empty"

Do not keep changing things. First determine scope: empty everywhere, or only for one data source? If Explore works on the data source, the problem is in panels or variables. If the dashboards are missing, not empty, and Git Sync was enabled in a 12 to 13 jump, this matches the 13.0.0 migration bug: preserve the database, restore the pre-upgrade snapshot and upgrade again to a fixed patch. If a data source was recreated, its UID changed and every panel referencing the old UID is orphaned; fix by restoring the UID in provisioning. The senior instinct is to stop, preserve evidence and choose rollback over heroics.

"Alert images and reports went blank this week"

After 13, the Image Renderer plugin is gone and rendering needs the remote service. Check that [rendering] server_url and callback_url are correct, then the token: JWT auth is on by default, so a renderer_token left as - or different on the two sides yields 401 or Failed to send request to remote rendering service. Third, check renderer resources, since Chromium under memory pressure fails quietly. If screenshots have missing text on an RPM-based image, install the fonts.

"We found a service account token in a public repository"

Rotate before anything else: delete or replace the token, since the repository has already been cloned. Then scope the blast radius: what role did the service account have, in which org, and what did the access and audit logs show during the exposure window? Then prevention: secret scanning in CI, finite secondsToLive on every token, tokens in a secrets manager and never in CI variables visible to everyone, one service account per automation so revocation is precise.

"Grafana is down and our alerts about it never fired"

The right answer is not about Grafana. It is that the alert lived inside the thing that failed. The fix is meta-monitoring from outside: Prometheus scrapes /metrics, a separate Alertmanager pages, and a dead man's switch confirms the path. Add a blameless note that the gap was a design gap, not a person's error.

Trade-off conversations with leadership

Leadership questions test whether you can turn a technical position into a decision someone can approve. Use the same structure each time: the risk in their terms, the options, the cost of each, your recommendation and what would change your mind.

"Why do we need Enterprise? OSS is free."

Frame it around named requirements, not features. "We have three requirements that OSS does not meet: we need per-team data source permissions because our teams cannot see each other's customer data; we need SAML and team sync so leavers lose access within minutes without manual work; and we need audit logs for our compliance review." Note that the Enterprise image runs free without a licence, so you are paying for capability, not for the software. If none of the requirements is real, say that too, and recommend OSS. Being willing to argue against spending money earns credibility for the times you argue for it.

"Why are you blocking the five-second refresh? People like it."

Translate into capacity and cost. "One incident dashboard at a five second refresh with thirty viewers produces about eleven thousand queries per minute against our Prometheus, which is why it slows down exactly when we need it. A 30 second floor cuts that six-fold and nobody can perceive the difference on a metric scraped every 15 seconds. I'll keep an exception path for active incidents." Offer a compromise, not a no.

"Can we skip the staging rehearsal? The release notes look fine."

"The release notes told us about a dozen breaking changes in 13.0, including a migration bug that lost dashboards for early adopters with Git Sync. The rehearsal costs us about half a day and uses a copy of production data. Without it, the downside is an unrecoverable dashboard loss for forty teams and an outage while we restore. I'd accept skipping it for a patch on the same minor, not for a major upgrade." That is the pattern: quantify, scope the exception, stay polite.

"Should we move to Grafana Cloud?"

There is no universal answer; give the axes. Managed service removes upgrades, patching, HA and scaling work, at a usage-based price tied to series, logs, traces and users, with cardinality discipline becoming an invoice line. Self-hosted gives control over data residency, networking and cost at scale, and costs people. For Middle East employers, the deciding questions are often which region the stack runs in and whether telemetry may leave the country. Private Data source Connect lets Cloud query private sources without opening inbound paths. A middle route is self-hosted Grafana with selective Cloud features, such as connecting to Grafana Assistant. Recommend based on team size, regulatory constraints and the cost of the on-call load you are currently paying in engineer time.

"Can one team just run their own Grafana?"

Say yes where it is warranted, and set conditions. A separate instance is the right answer for mutual distrust, different regulatory zones, or very different upgrade cadences. The conditions: same Helm chart or Terraform module, same SSO, same patch SLA, metrics scraped by the central monitoring, and an owner. Without those conditions, you have created shadow infrastructure with unpatched versions and default admin passwords.

"What would you cut if you lost half your platform budget?"

A strong answer has an order. First, unused dashboards and data sources and idle refreshes, because they cost nothing to remove and save backend load. Second, licensed features without a live requirement. Third, replica count, to the point that availability targets still hold. Never first: backups, the patch process, meta-monitoring, or the rehearsed restore, because those are what you are paid for on the bad day. Naming what you would protect is as informative as naming what you would cut.

Common interview questions

What does Grafana store, and what does it not? It stores dashboards, folders, users, orgs, data source definitions, alert rules and state, and preferences, in its database. It does not store metrics, logs or traces; those stay in the backends and Grafana queries them.

How do you scale Grafana? Run several replicas behind a load balancer on a shared PostgreSQL or MySQL; no sticky sessions needed. But Grafana CPU is rarely the limit. Scale and protect the data sources, limit refresh rates, use recording rules and caching, and size database connections to replicas.

What is the difference between Grafana-managed and data source-managed alert rules? Grafana-managed rules are evaluated by Grafana against any data source and stored in its database; data source-managed rules live in a Prometheus-compatible or Mimir or Loki ruler and are evaluated there. The second survives a Grafana outage.

What are the default notification policy timings? Group wait 30 seconds, group interval 5 minutes, repeat interval 4 hours, with child policies inheriting unless overridden.

What do DatasourceNoData and DatasourceError mean? Synthetic alerts raised when a rule returns no data or its query fails, under the default handling. They carry the rule name and data source UID, and a single data source failure can produce one per rule.

What is a service account and how does it differ from an API key? A non-human identity with an org role and multiple tokens (prefix glsa_), optionally expiring. API keys were replaced by service accounts. It is scoped to one org and cannot do server-admin operations.

Why does secret_key matter? It encrypts secrets stored in the database. Lose or change it without the procedure and stored data source and contact point secrets cannot be decrypted. Back it up with the database.

Who controls who can query a data source? Ultimately the data source's stored credential. Above that, org membership, and on Enterprise, data source permissions and LBAC.

What is Git Sync? Bidirectional synchronisation between a Git repository and a Grafana folder, generally available since 13.0, polling every 60 seconds with webhooks for instant sync. It depends on Unified Storage.

What changed in the CLI in Grafana 13? The stand-alone grafana-cli and grafana-server commands were removed; use grafana cli and grafana server. The systemd unit is still called grafana-server.

How do you pin Grafana safely? An exact image tag such as grafana/grafana-enterprise:13.2.3, never latest, plus plugin versions via GF_PLUGINS_PREINSTALL=id@version, with a patch SLA because patches are frequent and often security fixes.

Why is /api/health the right probe? It is unauthenticated and reports database health, so it tells the load balancer whether a replica can actually serve.

What a strong answer covers that a weak one misses

Sixty-second self-test

Answer aloud, without looking above.

1. Which two things does Grafana depend on, and how does each failure look?
2. What breaks first as usage grows, and what three levers reduce it?
3. Why do you get three pages for one alert, and what is the fix?
4. A Viewer can reach a data source they should not. Where is the real fix?
5. Name the two things you back up together, and why.
6. Why is rollback after a 12 to 13 upgrade a restore?
7. Name four 13.0 changes that break scripts or pipelines.
8. What does DatasourceNoData mean and how do you stop it becoming a storm?
9. Why must Grafana's own monitoring live outside Grafana?
10. Which tenancy tier would you pick for two mutually untrusting customers, and what is its cost?

If you stumble on any, return to the matching section of the senior detailed guide. The Prometheus, OpenTelemetry and Kubernetes guides cover the neighbouring systems an interviewer may pull you into next.

Part three of three. At this level the problems are operational and they appear at scale: a dashboard forty people leave open all day, an alerting cluster that quietly split, a service account token nobody owns, an upgrade that loses data. Start with the error table, then the practices, the ownership habits, and the review checklists for pull requests and design documents.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These rows describe incidents, not failed first attempts. Each has a symptom, the real cause and the fix.

Notice how many of these share a cause: something was configured once, in one place, and never tested from the point of view of the thing that depends on it.

The practices that pay off most

Practice cards

1. Split an alerting cluster on purposeRun two replicas on one PostgreSQL without ha_peers and a rule that always fires into a webhook. Count the duplicates, then fix them and count again.
2. Read what a Viewer can doLog in as a Viewer, open Explore and list every data source you can query. Repeat with a Viewer service account against /api/ds/query.
3. Lose the key, then recoverRestore a database into a Grafana with a different secret_key and watch a data source fail to decrypt. Then restore with the right key.
4. Kill the databaseStop PostgreSQL in staging. Record what /api/health returns, what users see, and how long recovery takes after restart.
5. Trigger a NoData stormCreate ten rules against one data source, stop the source and watch what arrives. Then add a policy routing the synthetic alerts to a platform channel.
6. Break the renderer tokenLeave renderer_token at - with JWT auth on, reproduce the 401, then set matching tokens on both sides.
7. Price a dashboardTake a live dashboard and compute panels times viewers times refreshes per minute. Apply a 30 second floor and a recording rule and remeasure backend queries.
8. Audit the tokensList every service account and token, mark those with no expiry and no owner, revoke one in staging and confirm the automation fails loudly.

Platform-ownership habits

The habits that separate a person who runs Grafana from one who owns it as a platform.

Keep a written contract with your users. One page: what you run, what they run, which labels are mandatory, which refresh floor applies, how a data source or plugin is requested, the patch SLA, and how to reach you. Every rule in it should have a one-line reason, because unexplained limits are the ones that get worked around.

Treat the What's new page as a change log for your own roadmap. Each release, spend thirty minutes reading it and turn relevant items into tickets with owners before they turn into incidents. The 13.0 page alone contained a dozen breaking items, several of which surfaced later as pipelines failing for a team that never read it.

Keep a version ladder. Know which minor each environment runs, which patch, and how many days behind the newest patch each is. Report it monthly. "Days behind latest patch" is the most honest measure of security posture that a Grafana owner has, and it is cheap to compute from /api/health.

Run a game day each quarter. Pick one playbook, break the thing in staging and have someone who did not write the playbook follow it. The steps they stumble on are the missing steps. Good candidates: database loss, split alerting cluster, leaked token, renderer failure.

Maintain a catalogue, not a free-for-all. Approved data sources and plugins, each with an owner, a pinned version and a review date. A new plugin is a short security review because a backend plugin executes code on your host. Keep allow_loading_unsigned_plugins empty, and treat any entry in it as an exception with a named owner and an expiry.

Retire things on purpose. Dashboards, data sources, service accounts and silences all accumulate. A scheduled job that reports stale objects to their owners, a quarantine folder and a deletion date give you hygiene without drama. Restoring deleted dashboards is available since 13.0, which lowers the fear of cleaning up.

Write the incident's prevention item first. When you close an incident, the first line of the follow-up is what control would have caught this at review time: a synthetic alert test in the pipeline, a scanner for secrets, a token expiry, a mandatory label check. Fixes that live only in people's memory expire when the people move teams.

Be the user of your own golden path. When you onboard a team, use the module the way a new team would, with no insider knowledge. You will find the broken instructions before they do.

Protect the boring work. Backups, restore rehearsals, patching, meta-monitoring and upgrade staging never ship a feature, so they are the first to be cut. Write down what they prevent, in incident and cost terms, so that you can defend them when someone asks why the platform team is not building something new.

Verifying that controls actually work

Controls you have never watched refuse or fire are assumptions. Here is how to check the important ones, with the command you would run and what a healthy result looks like.

# 1. Version and database health on every replica (run against each pod, not only the LB)
for pod in $(kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana -o name); do
  kubectl -n monitoring exec "$pod" -- wget -qO- http://localhost:3000/api/health
done
# Expect the same "version" on all replicas after a rollout, and "database":"ok"

# 2. Is the alerting cluster actually a cluster? Ask the embedded Alertmanager for its peers.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://grafana.example.com/api/alertmanager/grafana/api/v2/status | jq '.cluster'
# Expect every replica listed as a peer and status "ready".
# Note: on 13+, this endpoint needs alert.notifications.system-status:read

# 3. Which tokens never expire?
curl -s -u "$ADMIN" "https://grafana.example.com/api/serviceaccounts/search?perpage=100" \
  | jq -r '.serviceAccounts[] | "\(.id)\t\(.name)\t\(.tokens)"'
# Inspect each account's tokens for a missing expiration, then decide an owner or revoke

I have written the endpoint paths from the reference material; since /api routes are no longer updated in 13, confirm each against the API documentation of the version you run, and prefer a /apis equivalent where one exists.

What healthy output tells you: identical versions mean a rollout finished; a full peer list means duplicates are impossible; a token list with owners means offboarding works. What unhealthy output tells you is where to start.

Review checklist for pull requests

Use these on any change to Grafana configuration, provisioning, Terraform, dashboards or alert rules.

Every pull request

• Is there exactly one writer for each resource this change touches?
• Do data sources and dashboards have explicit, stable UIDs?
• Are there secrets in the diff? They must be $VAR, $__env{} or $__file{}.
• Does this change have a rollback that is not "hope"?

A dashboard change

• Do queries use $__rate_interval and sensible max data points rather than a hard-coded window?
• Is any expensive expression a recording rule instead?
• Does the default refresh respect the platform floor, and is the default time range bounded?
• Are variables used instead of duplicated dashboards, and do high-cardinality variables have a limit?
• Is the JSON in version control, with an owner and a folder?

An alert rule change

• Does it carry team, severity, service and a real runbook_url?
• Is the condition on symptoms a human needs to act on, built on a RED style signal, not on a cause that may never matter?
• What are the no-data and error behaviours, and do they match the meaning of silence for this rule?
• Is the pending period long enough to avoid flapping and short enough to matter?
• Does a synthetic version reach the right receiver in staging?

A notification policy or contact point change

• Is this the only state managing the tree?
• Does every route have a matching test alert?
• Are the repeat interval and group settings sensible for the severity?
• Do secrets come from the environment, and does the contact point's test notification pass?

An auth or permission change

• Does the role mapping avoid granting Grafana admin from a claim unless that is intended?
• Do service accounts have the lowest role that works and a finite expiry?
• If a custom RBAC role is involved, was it retested after the 13 alerting RBAC tightening?
• Does this widen who can query a data source? If so, was the data owner asked?

An upgrade change

• Image tag, plugin versions and chart version pinned to exact values?
• Chart repository set to grafana-community?
• Has the What's new page for every skipped version been read and mapped?
• Scripts free of grafana-cli, grafana-server, numeric-ID data source calls?
• Snapshot taken, restore tested, first replica first?

Review checklist for design documents

A design document that adds a new Grafana dependency, a new tenant, a new data source or a new alerting tier should answer these before it is approved.

1. What is the failure mode? What happens to the feature, and to everyone else on the platform, when Grafana is down, when its database is down and when the new data source is down?
2. What is the load? Panels, viewers and refresh rates, in queries per minute against which backend, and what limit does the backend enforce?
3. What is the trust boundary? Who can query the new data source, through what credential, and what can that credential read?
4. What is the tenancy tier? Folder, permissions, organization or instance, with the reason and the known gap.
5. Where does the configuration live? The owning mechanism, the repository and who reviews changes.
6. How are secrets handled? Where they are stored, how they are injected, how they rotate, what happens when one leaks.
7. How is it monitored, and from where? Which signals and which alerts, evaluated outside Grafana if they matter.
8. How does it upgrade? Which Grafana features does it use, are any in preview or experimental stages that carry no support guarantee, and does it depend on removed or deprecated APIs such as numeric-ID endpoints or Angular plugins?
9. Where must the data live? Region, residency and retention, which for many Gulf and Egyptian employers is a hard constraint and shapes whether the answer is one instance or several.
10. What is the exit? If you had to move this off Grafana, what would be lost and how hard would the export be?

A short checklist

• Database is PostgreSQL or MySQL, backed up, and the restore has been timed.
• secret_key is stored in a secrets manager and backed up with the database.
• At least two replicas, health-checked on /api/health, image pinned to an exact tag.
• ha_peers set with 9094 open on TCP and UDP; duplicate notifications tested.
• min_refresh_interval set deliberately; heavy expressions are recording rules.
• Data source credentials are least-privilege; anonymous access is off where it matters.
• MFA enforced at the IdP, use_refresh_token on, service accounts owned and expiring.
• Configuration in Git with one writer per resource; no secrets in files.
• Every alert rule has team, severity, service and runbook_url.
• DatasourceNoData and DatasourceError have a platform route.
• Grafana is monitored from outside itself, with a dead man's switch.
• Upgrade rehearsed with production data; scripts free of removed CLI names and numeric-ID calls; Helm repo is grafana-community.
• A written playbook exists for database loss, split alerting, leaked token and renderer failure.

Most of these are not hard. They are simply easy to postpone, and they are the difference between a Grafana that teams trust and one they route around. For the systems that feed it and run it, continue with the Prometheus, OpenTelemetry, Helm and Vault guides.