The Complete ELK Stack Guide

This is part one of three. It covers everything you need to do real work with the ELK stack, not a teaser. By the end you can start a working stack on your laptop, put documents into Elasticsearch, search and summarise them, build a dashboard in Kibana, ship a real log file into the stack with Filebeat, and reshape messy log lines with Logstash. You will also be able to read the ten errors that account for most of a beginner's pain. The Mid-level and Senior guides take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and this material only sticks once you have watched your own data arrive, get searched, and occasionally get rejected.

The guide is written against Elastic Stack 9.5.4, released on 15 September 2026. That number matters for a practical reason: 9.5.4 and 9.4.7 fixed a denial-of-service vulnerability in Elasticsearch (ESA-2026-182, CVE-2026-94396) that affects 9.2.0 through 9.4.6 and 9.5.0 through 9.5.3, and Elastic lists no workaround other than upgrading. Whenever you install the stack, install 9.5.4 or later.

What the ELK stack is, and the problem it solves

Every running system writes down what it is doing. A web server records each request, an application records each error, a database records each slow query, a load balancer records each connection. Those records are logs, and on one machine you can read them with tail -f and grep. That stops working the moment you have ten machines, or one machine producing a million lines an hour, or a bug report that says "something was slow at about three in the morning last Tuesday".

The ELK stack exists to answer three questions about that mountain of text. Where is all of it? It gets collected from every machine into one place. How do I find the one line I care about? It gets indexed, so a search across a billion lines returns in milliseconds. What is the shape of it? It can be counted, grouped and charted, so you can see that errors tripled at 03:10 and that all of them came from one host.

That diagram is the whole stack. Read it left to right and you have followed one log line from the moment an application wrote it to the moment a person saw it on a screen.

Before tools like this, the usual answer was a pile of scripts. Someone would rsync log files to a central server nightly, run grep over them, and paste the results into a ticket. Some teams loaded logs into a relational database, which works until the volume grows and every query becomes a full-table scan. Others bought a proprietary log-management product priced by the gigabyte. The stack that became ELK was the open, searchable alternative, and it turned out to be useful well beyond logs: the same engine now powers website search, product catalogues, security analytics, and metrics dashboards.

Two ideas explain why it is fast, and it is worth meeting them now because they shape every command later.

Search is built on an inverted index. A book has an index at the back: look up a word, get the page numbers. Elasticsearch builds the same thing for every word in every document, so finding all documents containing timeout is a lookup rather than a scan. That is why searching a billion lines feels instant, and also why how a field is indexed decides what you can do with it.

Data is spread over shards. An index is split into pieces called shards that can live on different machines, so the work of searching is shared and the data can grow beyond one disk. On your laptop you will have one machine and the machinery will mostly be invisible, but the words "shard" and "replica" will appear in health output, so we define them properly soon.

What people use it for:

For MLOps work specifically, this is where model-serving logs go. When a prediction endpoint starts returning errors, or a feature pipeline stalls, the evidence is in logs and metrics from five different services. A stack that lets you search them together is often the difference between a ten-minute diagnosis and a two-day one. A note for readers working for Gulf or Egyptian employers: logs routinely contain user identifiers and IP addresses, so where the cluster physically lives can be a data-residency question. Self-managed clusters and the regional options of the cloud providers are both possible; decide that before you ship production logs anywhere.

The names: ELK, the Elastic Stack, Beats and Elastic Agent

The vocabulary around this stack confuses almost everybody, because the name changed while the products multiplied. Sort it out once and the documentation stops being baffling.

ELK is an acronym for three products: Elasticsearch, Logstash and Kibana. It is what most people, and most job advertisements, still say. Elastic Stack is the official name, and it includes the three ELK products plus the collection side: Beats and Elastic Agent. You will see both terms, and they describe the same ecosystem. In this guide "ELK" means the three core products and "the stack" means everything.

Here is each piece in one sentence, with the port it listens on, because ports are how you will check that things are running:

• Elasticsearch is the distributed search and analytics engine, built on the Apache Lucene library. It stores your data and answers queries through a JSON REST API on port 9200. Port 9300 is used between Elasticsearch nodes and is not something you call.
• Kibana is the web interface on port 5601. It is where you search, build dashboards, manage the cluster and run commands in a console.
• Logstash is a data-processing pipeline that reads events from inputs, transforms them with filters, and writes them to outputs such as Elasticsearch. Its monitoring API is on port 9600.
• Beats are small, single-purpose shippers written in Go. Filebeat ships log files; there are others for metrics, network data, uptime checks and Windows event logs.
• Elastic Agent is a single unified agent that does the work of several Beats and is managed centrally from Kibana through Fleet. Elastic recommends it for new deployments. Beats are still supported and still shipped, and they remain the simplest thing to learn first, which is why this guide uses Filebeat in its hands-on section.

Two more terms appear constantly. An integration is a packaged recipe for one data source, for example nginx or AWS: the collection settings, the parsing, the field mappings and ready-made dashboards bundled together. ECS, the Elastic Common Schema, is an agreed set of field names (@timestamp, host.name, log.level, source.ip, message) so that logs from different sources can be searched with the same names.

There is also a licensing point that employers sometimes ask about. The default download is under the Elastic License 2.0. Since September 2024 the source is additionally available under the AGPLv3, an OSI-approved open-source licence, so the stack is open source again if you choose that option. The free "Basic" tier already includes security (TLS, users and roles, API keys), Kibana, Fleet and the ES|QL query language, which is everything this guide uses. Some features, such as machine-learning anomaly detection, single sign-on for self-managed clusters and searchable snapshots, need a paid subscription. The local installer used below gives you a 30-day trial of those and then falls back to Basic.

The mental model: the nouns you need

You can run this stack for a year without understanding Lucene internals. You cannot run it for a day without a handful of nouns. Learn these, and the error messages will read like sentences.

A document is one JSON object, the unit of storage: one log line, one product, one user. Every document has an _id and a _source, which is the original JSON you sent.

An index is a named collection of documents that share a purpose, like web-logs or products. If you know relational databases, an index is roughly a table, and a document is roughly a row, with the important difference that the documents are JSON and do not all have to look alike.

A field is one key inside a document. The mapping is the schema: it says what type each field has. The types you will meet first are text, keyword, long, double, date, boolean, and ip. The text and keyword pair is the most important distinction in the whole stack:

• A text field is analysed: broken into lowercase words, so a search for timeout finds "Connection Timeout after 30s". It is for full-text search and cannot be sorted or grouped efficiently.
• A keyword field is stored as one exact value. It is for filters, sorting and counting: status codes, host names, log levels, user ids.

Dynamic mapping means that if you send a field Elasticsearch has not seen, it guesses a type. A JSON string it guesses as text and adds a .keyword sub-field, so host becomes both a searchable host and an exact host.keyword. That convenience is wonderful on day one and a source of trouble on day thirty, which is why we look at explicit mappings later.

A shard is one piece of an index. Internally, each shard is a complete Lucene index. There are two kinds: primary shards hold the original data, and replica shards are copies placed on different nodes for resilience and read speed. The number of primary shards is fixed when you create the index; the number of replicas can change any time.

A node is one running Elasticsearch process, and a cluster is one or more nodes sharing a cluster.name. On your laptop you have a cluster of one node.

Cluster health is reported as a colour, and you will check it constantly:

The surprise that catches every beginner: a single-node cluster with an index that asks for one replica is yellow, forever. A replica may not live on the same node as its primary (a copy on the same machine would protect against nothing), and there is no second node, so the replica has nowhere to go. Yellow on a laptop is normal, not a fault.

Two more nouns round out the set. A data view is a Kibana object that says "these indices (or data streams) are the ones I want to explore"; older material calls it an index pattern. A data stream is an append-only series of hidden indices behind one name, designed for logs and metrics; the shippers in this guide write to data streams, and you will see names like logs-nginx.access-default. Its naming convention is <type>-<dataset>-<namespace>.

Finally, the near-real-time rule. A document you index is not searchable instantly; it becomes visible after the next refresh, which by default happens every second. So if you index and immediately search, you may miss your own document for a moment. This is by design, and it is the reason Elasticsearch is called near-real-time rather than real-time.

Installing the stack on your laptop

There are several ways to install, and choosing well saves an afternoon. For learning, use start-local: a script from Elastic that uses Docker Compose to start Elasticsearch and Kibana with sensible settings in one command. It works on macOS, Linux, and Windows through WSL. For a real server you use the operating-system packages or a container image, and we cover those briefly below so you recognise them. If you are new to containers, the Docker guide explains the ideas this section leans on, and it is worth reading first.

You need Docker running, and Docker Compose version 2 (it ships with current Docker Desktop). Then:

curl -fsSL https://elastic.co/start-local | sh

The script downloads a small project and starts it. When it finishes you have a new folder called elastic-start-local/ containing a Compose file, a hidden .env file, and scripts named start.sh, stop.sh and uninstall.sh. Elasticsearch is at http://localhost:9200 and Kibana at http://localhost:5601.

The .env file is the part to understand. It holds two secrets the script generated for you: ES_LOCAL_PASSWORD, the password for the built-in superuser called elastic, and ES_LOCAL_API_KEY, an API key. Look at them with:

cd elastic-start-local
grep ES_LOCAL_ .env

Two flags are useful. -v <version> pins a version, and --esonly skips Kibana. Because this guide is written for 9.5.4, you can pin it explicitly, but the default already installs a current release:

curl -fsSL https://elastic.co/start-local | sh -s -- -v 9.5.4

Installing on a real machine

On a server you install packages or run an image. You do not need to memorise these steps, but you should know what they look like and what they print. On Debian or Ubuntu, add Elastic's signing key and repository, then install each product from the same repository:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/9.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt-get update && sudo apt-get install elasticsearch
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Read the terminal output of that install. The package prints the automatically generated password for the elastic user exactly once. If you scroll past it, you will need to generate a new one later with elasticsearch-reset-password. Kibana and Logstash install the same way: sudo apt-get install kibana and sudo apt-get install logstash. On Red Hat-family systems the equivalent uses an .repo file and dnf. On macOS or Linux you can also download a .tar.gz archive, extract it, and run ./bin/elasticsearch, which runs in the foreground and prints the password and a Kibana enrollment token to your terminal. On macOS the bundled Java runtime may be blocked by Gatekeeper, and the documented fix is xattr -d -r com.apple.quarantine . inside the extracted folder. Homebrew is not recommended for Elasticsearch: Elastic no longer maintains a tap, and the community formula lags the current release.

On Windows you download the .zip, unzip it, and run .\bin\elasticsearch.bat, or install it as a service with .\bin\elasticsearch-service.bat install.

With Docker alone, without Compose, the official quick start creates a network, runs the image, and resets the password:

docker network create elastic
docker run --name es01 --net elastic -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:9.5.4
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
docker run --name kib01 --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:9.5.4

Two facts about Elasticsearch that surprise people. First, it bundles its own Java, so you do not install Java for it and should not point it at another. Second, every component in one deployment should run the same version: Elasticsearch 9.5.4 with Kibana 9.5.4 and Logstash 9.5.4.

What first-start security does

Since version 8, security is on by default. When a brand-new node starts for the first time, it configures itself: it generates a certificate authority and TLS certificates for both the HTTP and the node-to-node channels, turns on authentication, sets a password for the elastic user, and prints a Kibana enrollment token that is valid for 30 minutes. The certificate for the HTTP channel lives in certs/http_ca.crt under the config directory, and this file is what clients need to trust the server. Package installs put it at /etc/elasticsearch/certs/http_ca.crt.

Connecting Kibana to such a node takes four steps: start Kibana, open the URL it prints (something like http://localhost:5601/?code=123456), paste the enrollment token, then enter the six-digit verification code that bin/kibana-verification-code prints. Then log in as elastic. If the token expired, make a new one with bin/elasticsearch-create-enrollment-token -s kibana.

Auto-configuration is skipped in some situations, all of which are worth knowing because they explain "why did it not generate a password": the data directory was not empty, security settings were already in elasticsearch.yml, standard output was redirected to a file, or discovery settings for a multi-node cluster were already configured.

Checking that the installation works

Never assume a service is healthy because the install command ended without an error. Elasticsearch has a REST interface, so the test is a plain HTTP request, and the habit of checking is worth building now.

With start-local, the password is in .env. Load it into a shell variable so you never paste it into history:

export ELASTIC_PASSWORD=$(grep ES_LOCAL_PASSWORD .env | cut -d= -f2)
curl -u elastic:$ELASTIC_PASSWORD http://localhost:9200

The response is a JSON document describing the node. The parts to read:

{
  "name": "es01",
  "cluster_name": "docker-cluster",
  "cluster_uuid": "k3Xp0s7fT9qH2mVwYl8aBg",
  "version": { "number": "9.5.4" },
  "tagline": "You Know, for Search"
}

Your names and uuid will differ. What matters is version.number, which must match what you meant to install, and the tagline, which proves the request reached Elasticsearch rather than something else on that port.

On a package or archive install, security is on with TLS, so the same check uses https and the CA certificate file:

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200

If you forget --cacert, curl stops with SSL certificate problem: self-signed certificate in certificate chain. That is curl correctly refusing to trust a certificate authority it has never heard of. The fix is to provide the certificate, not to add -k, which switches verification off and teaches a bad habit.

Now ask the cluster how it feels:

curl -u elastic:$ELASTIC_PASSWORD "http://localhost:9200/_cluster/health?pretty"

Read the status field: green, yellow or red, as defined earlier. Also look at number_of_nodes (one on your laptop) and unassigned_shards. On a fresh single-node cluster with no user indices, you should see green. After you create your first index with a replica, expect yellow.

For the other components, the checks are equally direct. Kibana's health is at http://localhost:5601/api/status, or simply the login page loading. Logstash has a smoke test that needs no Elasticsearch at all:

bin/logstash -e 'input { stdin { } } output { stdout {} }'

Type a line, press Enter, and Logstash prints it back as an event. Its monitoring API, once running, answers curl 'localhost:9600/?pretty'. For Filebeat the two checks are filebeat test config -e, which validates the YAML, and filebeat test output, which tests the connection to Elasticsearch. A third, for Elastic Agent, is sudo elastic-agent status.

Your first index: putting data in and getting it out

Everything so far has been setup. This section is the first time you actually store something. Open Dev Tools; all the requests below are in its syntax. The left pane is where you type, the right pane is the response, and the green triangle (or Ctrl+Enter) runs the request under your cursor.

Start by indexing one document. You could create the index first, but Elasticsearch will create it for you on first write, with a guessed mapping:

PUT web-logs/_doc/1
{
  "@timestamp": "2026-09-30T10:00:00Z",
  "host": "web-01",
  "level": "error",
  "status": 500,
  "message": "Connection timeout while calling payment service"
}

The URL reads as: in the index web-logs, create document id 1. The word _doc is fixed; it is the name of the endpoint, not a type you choose. The response includes "result": "created" and a _version of 1. Run the same request again and you get "result": "updated" and version 2, because the same id replaced the document. To let Elasticsearch generate the id, use POST web-logs/_doc with no id.

Read a document back by id, and then change part of it:

GET web-logs/_doc/1

POST web-logs/_update/1
{ "doc": { "level": "critical" } }

DELETE web-logs/_doc/1

Real work involves many documents at once, and one request per document is painfully slow. The bulk API sends many operations in a single request. Its body is unusual: newline-delimited JSON, alternating an action line and a document line, and it must end with a newline:

POST _bulk
{ "index": { "_index": "web-logs" } }
{ "@timestamp": "2026-09-30T10:00:05Z", "host": "web-01", "level": "info", "status": 200, "message": "GET /home served in 34 ms" }
{ "index": { "_index": "web-logs" } }
{ "@timestamp": "2026-09-30T10:00:09Z", "host": "web-02", "level": "error", "status": 502, "message": "Bad gateway from upstream model-server" }
{ "index": { "_index": "web-logs" } }
{ "@timestamp": "2026-09-30T10:01:14Z", "host": "web-01", "level": "warn", "status": 429, "message": "Rate limit reached for client 203.0.113.7" }
{ "index": { "_index": "web-logs" } }
{ "@timestamp": "2026-09-30T10:02:30Z", "host": "web-03", "level": "error", "status": 500, "message": "Connection timeout while calling payment service" }
{ "index": { "_index": "web-logs" } }
{ "@timestamp": "2026-09-30T10:03:41Z", "host": "web-02", "level": "info", "status": 200, "message": "GET /pricing served in 51 ms" }

The response lists one result per line and has a top-level "errors": false if everything worked. Always check that flag. A bulk request that partly fails still returns HTTP 200, with "errors": true and the failing items marked inside, so a script that only checks the status code will silently lose data.

Now look at what you created. Two requests from the _cat family give human-readable tables, and ?v adds column headers:

GET _cat/indices?v
GET web-logs/_count
GET web-logs/_mapping

The first shows your index with its health (yellow on one node, as explained), primary and replica counts (1 and 1 by default), and document count. The mapping shows what Elasticsearch guessed. You will see @timestamp as date, status as long, and host, level and message as text each with a .keyword sub-field. That is dynamic mapping in action.

Searching: the query language you will use every day

Searching uses GET index/_search with a JSON body called the Query DSL. With no body, it returns the first ten documents. The shape of every response is the same: a hits object containing total (how many matched) and hits (the first ten, each with _score and _source).

Full-text search with match

GET web-logs/_search
{
  "query": { "match": { "message": "timeout" } }
}

match is for text fields. It analyses your search text the same way the field was analysed at index time, which is why lowercase timeout finds "Connection timeout while calling payment service". Each hit has a _score, a relevance number: documents where the term is rarer or more prominent score higher. Try "message": "payment timeout", and you get documents with either word, ranked with both-word matches first.

Exact search with term, and filtering with range

For exact values, use term against a keyword field. Because dynamic mapping made host.keyword, that is the one to search:

GET web-logs/_search
{
  "query": { "term": { "host.keyword": "web-01" } }
}

A classic mistake is term on the analysed host field with capital letters; the stored token is lowercase, so nothing matches. The rule of thumb: match for text, term for keyword, numbers and dates.

range works on numbers and dates. Here is every response with a status of 500 or above:

GET web-logs/_search
{
  "query": { "range": { "status": { "gte": 500 } } }
}

Combining conditions with bool

Real questions have several parts: "errors from web-01 in the last hour whose message mentions timeout". The bool query combines clauses, and the four names are worth memorising:

GET web-logs/_search
{
  "query": {
    "bool": {
      "must":   [ { "match": { "message": "timeout" } } ],
      "filter": [
        { "term":  { "level.keyword": "error" } },
        { "range": { "@timestamp": { "gte": "now-1d/d" } } }
      ]
    }
  },
  "sort": [ { "@timestamp": "desc" } ],
  "size": 20,
  "_source": ["@timestamp", "host", "message"]
}

Notice the division of labour: the relevance question ("is about timeouts") sits in must, and the yes/no conditions sit in filter, which is faster because Elasticsearch does not compute a score for them and can cache the result. sort orders by time, size returns twenty rows instead of ten, and _source limits which fields come back. A good habit: any condition that is not about relevance belongs in filter.

Two limits to know now. Results are capped: from plus size cannot exceed 10,000, and deeper paging needs a different technique covered in the Mid-level guide. And a search response has "timed_out" and "_shards" sections; if _shards.failed is above zero, the results are partial.

Mappings: taking control of the schema

Dynamic mapping got you started, but it makes decisions for you, and some of them are wrong for logs. Every string becomes both a text field and a .keyword sub-field, which doubles the storage for fields like host that you only ever filter on. A number that arrives as a string in one document ("status": "200") fixes the type for the whole index; a later document with "status": "unknown" is then rejected. An unexpected field name that appears in one application's logs silently becomes a permanent column. None of these are bugs; they are the price of "it just works".

The alternative is to declare the mapping before you write data. You create the index with the schema you want:

PUT app-logs
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "host":       { "type": "keyword" },
      "level":      { "type": "keyword" },
      "status":     { "type": "integer" },
      "client_ip":  { "type": "ip" },
      "message":    { "type": "text" }
    }
  }
}

Each field now has exactly the type you chose. host and level are pure keywords, so term queries and aggregations work on the field name itself, with no .keyword suffix. client_ip is an ip type, which lets you search by CIDR range such as 203.0.113.0/24. message is full-text only, which is all a log message needs.

You can also set how strict the index is about fields you did not declare, through the dynamic setting in the mapping:

The rule that matters most: you cannot change the type of an existing field. If status was mapped as text and you want integer, the fix is to create a new index with the right mapping and copy the data across with the reindex API. Plan the mapping for anything you will keep, and let dynamic mapping carry only throwaway experiments.

Mapping errors have a distinctive look. Index a document that contradicts the mapping:

POST app-logs/_doc
{ "@timestamp": "2026-09-30T11:00:00Z", "host": "web-01", "level": "info", "status": "not-a-number", "message": "test" }

You get an HTTP 400 and a mapper_parsing_exception with the reason failed to parse field [status] of type [integer]. Read that message left to right: the exception class says the problem is the mapping, the field name tells you where, and the type tells you what was expected. The fix is either to correct the producer, to add an ingest conversion, or to map the field with "ignore_malformed": true so the bad value is dropped from the index while the rest of the document survives.

Another variation of the same idea is index templates, which apply a mapping and settings automatically to any new index or data stream whose name matches a pattern. That is how a fleet of daily or rolling indices all get the same schema without anyone creating them by hand. Templates are a central Mid-level topic. For now, know that when Filebeat or Elastic Agent writes a data stream, a template that the tools installed is what gave it its mapping.

Aggregations: counting and grouping

Search answers "which documents match". Aggregations answer "what does the set of matching documents look like": how many per host, what is the average latency, how many errors per minute. They are what make dashboards possible, and Kibana generates them for you, but you should be able to read and write the basic ones.

An aggregation request goes in the same _search call, under aggs. Setting "size": 0 tells Elasticsearch you do not want the matching documents themselves, only the summary. On web-logs, where host and level have .keyword sub-fields, count documents per level:

GET web-logs/_search
{
  "size": 0,
  "aggs": {
    "by_level": { "terms": { "field": "level.keyword" } }
  }
}

The response has aggregations.by_level.buckets, a list like { "key": "error", "doc_count": 2 }. The word bucket is the central noun: a bucket aggregation sorts documents into groups. The common bucket types are terms (one bucket per distinct value), date_histogram (one bucket per time interval) and range. Metric aggregations compute a number from the documents in a bucket: avg, sum, min, max, cardinality (count of distinct values) and percentiles. They nest, which is the powerful part:

GET web-logs/_search
{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "2026-09-30T00:00:00Z" } } },
  "aggs": {
    "per_minute": {
      "date_histogram": { "field": "@timestamp", "calendar_interval": "1m" },
      "aggs": {
        "errors": { "filter": { "term": { "level.keyword": "error" } } },
        "distinct_hosts": { "cardinality": { "field": "host.keyword" } }
      }
    }
  }
}

Read it from the outside in. The query narrows to one day. The date_histogram makes one bucket per minute. Inside every minute bucket, two sub-aggregations run: how many documents were errors, and how many distinct hosts appeared. That single request is the data behind a typical "errors over time" chart.

The most common aggregation error is this one: Fielddata is disabled on [host] in [web-logs]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting. It appears when you aggregate on a text field. The remedy is in the message: aggregate on the keyword version, host.keyword, or map the field as keyword from the start. A second frequent one is too_many_buckets_exception, which appears when a request would create more than 65,536 buckets, for example a one-second histogram over a year. Use a larger interval or a narrower time range.

Kibana: seeing your data

Everything you did in Dev Tools is the raw interface. Kibana is how most people, including non-engineers, actually use the stack. Three places matter for a beginner: Discover, Dashboards, and Dev Tools (which you already know).

Data views and Discover

Kibana cannot browse an index until you tell it which indices to treat as one dataset. That object is a data view. Open the main menu, then Stack Management, then Data Views, and create one: give it a name like web-logs, type web-logs as the pattern (wildcards such as logs-* work), and choose @timestamp as the time field. The time field is what drives the time picker at the top of every Kibana page.

Now open Discover. It shows a histogram of documents over time, a table of documents below, and a search bar. Three habits make Discover productive:

1. Set the time range first. The top-right picker defaults to a recent window. Your sample documents are from 30 September 2026, so widen the range (for example, "Last 2 years", or absolute dates) or you will see "No results found".
2. Use the search bar with KQL, the Kibana Query Language. level : "error" and host : "web-01" is a valid query. KQL is simpler than the JSON Query DSL and is translated into it for you.
3. Add fields as columns. Hover over a field in the left panel and click add, so the table shows host, level and message instead of one wide JSON blob.

Click any document to expand it and see every field and its mapped type. If a field you expected is missing, the usual cause is that the data view was created before the field existed; refresh the data view's field list in Stack Management.

Building a dashboard

A dashboard is a page of charts on the same time range. The fastest way to build your first is with Lens, Kibana's drag-and-drop chart editor. From the menu choose Dashboards, then Create dashboard, then Create visualization. Pick your data view, drag @timestamp to the horizontal axis and let Lens choose a count of records for the vertical axis: that is an "events over time" chart. Change it to a stacked bar and drag level into "break down by", and you have errors versus warnings over time. Save it, add it to the dashboard, add a second chart for top hosts, and save the dashboard with a name.

The mapping you chose pays off here. Lens only offers fields that can be aggregated for a given chart type, so keyword and numeric fields appear while text fields do not. If a field you want is missing from a chart's field picker, the cause is almost always its type.

ES|QL, the pipe language

During the 8.x series Elasticsearch gained a second query language that reads more like a shell pipeline: ES|QL. It is available in Discover and over the _query API, and on the free tier. Each command feeds the next:

FROM web-logs
| WHERE level.keyword == "error"
| STATS errors = COUNT(*) BY host.keyword
| SORT errors DESC
| LIMIT 10

Read it as a sentence: from the index, keep errors, count them per host, sort the biggest first, keep ten. The equivalent in the Query DSL plus aggregations would be a page of JSON. Over the REST API, send it as POST /_query?format=txt with body { "query": "FROM web-logs | LIMIT 10" }. Two limits to remember: by default a query returns 1,000 rows, with a maximum of 10,000, and since version 9.1 ES|QL may return partial results when some shards fail, flagged by an is_partial field in the response, so look for it if numbers seem low.

For a beginner, the recommendation is: use KQL in Discover for quick finding, Lens for charts, and ES|QL when you want to ask a slightly more complex question in one readable query. You do not need to master the JSON Query DSL to be effective, but you should be able to read it, since documentation and error messages use it.

Getting real logs in with Filebeat

Hand-typed documents taught you the API. Real systems produce logs continuously, and something must read the files and send each new line to Elasticsearch. That is a shipper, and the simplest one to learn is Filebeat: a tiny program that watches files, remembers how far it has read, and sends new lines on. It keeps a record of its position, so if Filebeat or the network fails, it resumes where it stopped instead of losing or duplicating lines.

Install Filebeat from the same repository as the other products (sudo apt-get install filebeat), or download the archive for your platform from Elastic's downloads page. Its configuration lives in one file, filebeat.yml. A complete minimal configuration for an application that writes to /var/log/app/*.log:

filebeat.inputs:
  - type: filestream
    id: app-logs
    paths:
      - /var/log/app/*.log

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  username: "elastic"
  password: "${ES_PASSWORD}"

setup.kibana:
  host: "http://localhost:5601"

This example targets the plain-HTTP start-local stack. Four things deserve explanation.

The input type is filestream. Older tutorials use type: log. That input is deprecated, and in Filebeat 9 it will not start unless you add allow_deprecated_use: true. Use filestream for anything new.

The id is mandatory and must be unique. Filebeat 9 refuses to start if two inputs share an ID, because it tracks file positions per input. Give every input a meaningful, permanent name.

Credentials use a variable. ${ES_PASSWORD} is read from the environment, so the secret is not in the file. On a real server you would use Filebeat's own keystore or an API key instead of a user password.

Against a secured cluster, the output needs the certificate. For an HTTPS cluster the output becomes:

output.elasticsearch:
  hosts: ["https://es-01:9200"]
  api_key: "id:api_key"
  ssl.certificate_authorities: ["/etc/filebeat/http_ca.crt"]

Without the CA line, Filebeat logs x509: certificate signed by unknown authority, which is the shipper's version of curl's self-signed certificate complaint, and the fix is the same: provide the certificate rather than disabling verification.

Now the commands, in the order you should use them:

filebeat test config -e
filebeat test output
sudo filebeat setup -e
sudo systemctl start filebeat

test config checks the YAML for mistakes; test output proves Filebeat can reach and authenticate to Elasticsearch; setup loads the index template and sample dashboards into Elasticsearch and Kibana once; and start begins shipping. If test output succeeds and nothing arrives, check the file's paths and permissions next.

Filebeat also has modules that know how to read common software. filebeat modules enable nginx turns on a module that reads the nginx access and error logs, parses each line into named fields, and brings matching dashboards. Modules are what make "point it at nginx and get a dashboard" work with no parsing effort from you.

One Filebeat 9 behaviour that genuinely confuses people: the filestream input identifies files by a fingerprint of their first 1,024 bytes by default, so a file smaller than 1,024 bytes is not ingested until it grows. If you test with a tiny three-line file and nothing shows up, this is why. Append more text to the file, or write a longer test line.

Once data flows, open Discover. Filebeat writes to a data stream with a name like filebeat-9.5.4, or logs-* when you use Elastic Agent integrations. Create or use a data view matching it, and you will see your log lines with message holding the original text, plus fields describing the host and file.

Logstash: parsing and reshaping

A log line is text. 203.0.113.7 GET /checkout 500 123ms is readable by a person, but searchable structure (a client address, a method, a path, a status number, a duration number) only exists if something pulls those pieces apart. Filebeat modules do that for well-known software. For your own applications, you either parse at the source, or you use a tool built for it. Logstash is that tool: a pipeline that reads events, transforms them, and writes them out.

A Logstash pipeline has three stages, each written as a block in a .conf file:

Inputs include files, TCP and HTTP listeners, message queues such as Kafka, and the beats input that receives from Filebeat on port 5044. Filters include grok (pattern-match text into fields), dissect (split by a fixed layout), date (set the event time from a field), mutate (rename, convert, drop), and geoip (turn an address into a location). Outputs include Elasticsearch, files, and the terminal.

Here is a complete pipeline for the log line shown above. Save it as pipeline.conf:

input {
  stdin { }
}

filter {
  grok {
    match => { "message" => "%{IP:client_ip} %{WORD:method} %{URIPATH:path} %{NUMBER:status:int} %{NUMBER:duration_ms:int}ms" }
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts    => ["http://localhost:9200"]
    user     => "elastic"
    password => "${ES_PASSWORD}"
    index    => "shop-logs"
  }
}

Every line of it has a job. The stdin input reads whatever you type, which makes it a perfect learning tool. The grok filter holds a pattern made of named building blocks: %{IP:client_ip} means "match something shaped like an IP address and store it in the field client_ip", and :int converts the captured text into a number so that it can be averaged later. The stdout output with the rubydebug codec prints each finished event in a readable structure, which is how you debug a pipeline. The elasticsearch output indexes into shop-logs.

Before running anything, test the syntax. Logstash's startup is slow, and a typo found after thirty seconds is a waste:

bin/logstash -f pipeline.conf --config.test_and_exit

Then run it, type a line, and watch:

export ES_PASSWORD="<the password from .env>"
bin/logstash -f pipeline.conf
# now type:
203.0.113.7 GET /checkout 500 123ms

The debug output shows the event with client_ip, method, path, status and duration_ms as separate fields, plus @timestamp (set to the time Logstash received it), the original message, and a host or similar field describing where it ran. If instead you see "tags" => ["_grokparsefailure"], the pattern did not match the line. That tag is Logstash's way of saying "I could not parse this", and it is the single most common Logstash symptom. Fix it by comparing your pattern against the line character by character; the Grok Debugger in Kibana Dev Tools lets you paste a line and a pattern and see what matches.

The Elasticsearch output keeps the secret out of the file too: "${ES_PASSWORD}" is expanded from the environment, and bin/logstash-keystore can hold it on a server. When you move to HTTPS, the option names in version 9 matter: use ssl_certificate_authorities for the CA file, not the old cacert. Logstash 9's Elasticsearch output refuses to start if the removed names (cacert, ssl, keystore) are present. For a real cluster, use an API key with api_key => "id:api_key", scoped to only the indices the pipeline writes.

Two more facts about running Logstash. It needs a newer Java than older tutorials assume, but it bundles its own, so ignore that; and as of version 9 it refuses to run as root (allow_superuser defaults to false). Run it as the logstash user that the package creates. And if you start a second Logstash with the same data directory you get Logstash could not be started because there is already another instance using the configured data directory, so use one instance per data path.

When do you actually need Logstash? Not always. Use it when you need heavy or conditional parsing, enrichment from another source, routing one stream to several destinations, or a buffer in front of Elasticsearch. Skip it when Filebeat modules or Elastic Agent integrations already parse your source, because every extra component is one more thing to run and to monitor.

Configuration you will actually touch

Three products, three configuration files, and a short list of settings that cover your first months. Each file is YAML or a close relative, and each product reads secrets from a keystore or environment variable rather than the file when you ask it to.

elasticsearch.yml

On a package install it lives in /etc/elasticsearch/. A single-node development file needs very little:

cluster.name: learning
node.name: es-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
discovery.type: single-node

cluster.name and node.name are labels; give them meaningful values because they show up in every log and health check. path.data is where the indexed data lives, which should be outside the install folder so an upgrade never touches it. discovery.type: single-node tells Elasticsearch not to look for other nodes; it is for development only.

A critical behaviour: binding to a non-loopback address changes the rules. While network.host stays on localhost, Elasticsearch runs in development mode and only warns about poor settings. The moment you set it to a real address such as 0.0.0.0, it switches to production mode and turns those warnings into bootstrap checks: hard failures that stop the node from starting. That is when beginners meet max virtual memory areas vm.max_map_count [65530] is too low and similar. Those are not bugs; they are Elasticsearch refusing to run in a configuration that is likely to lose data.

For a multi-node cluster you replace discovery.type with discovery.seed_hosts (the list of nodes to contact) and, for the very first start only, cluster.initial_master_nodes. Remove initial_master_nodes afterwards, and never add it to a node joining an existing cluster.

Memory is the other setting people expect to touch, and usually should not. The heap is sized automatically from the node's RAM and roles. If you must override it, put -Xms and -Xmx (equal values) in a file under jvm.options.d/, never in jvm.options itself, and keep the heap at no more than half of the machine's memory.

kibana.yml

server.host: "localhost"
server.port: 5601
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/http_ca.crt"]

Kibana needs to know where Elasticsearch is, and, for HTTPS, which certificate authority to trust. It authenticates as a built-in service identity (the enrollment flow sets that up), never as the elastic superuser; Kibana refuses elasticsearch.username: elastic. To let other machines reach Kibana, change server.host to 0.0.0.0, and put it behind HTTPS before you do.

If you use alerting or Fleet you will later be asked for encryption keys, xpack.encryptedSavedObjects.encryptionKey among them, at least 32 characters. Generate them with bin/kibana-encryption-keys generate, store them somewhere safe, and use the same values on every Kibana instance. Lose them, and the saved data they encrypt becomes unreadable.

logstash.yml and pipelines.yml

logstash.yml holds process-level settings; defaults are fine to start. The ones you will meet are pipeline.workers (defaults to your CPU count), pipeline.batch.size (125), queue.type (memory by default; persisted writes events to disk so a crash does not lose them), and config.reload.automatic, which makes Logstash pick up pipeline edits without a restart, also available as --config.reload.automatic on the command line. The file pipelines.yml lists several independent pipelines, each with a pipeline.id and a path.config.

Secrets

The habit to adopt on day one: no passwords in config files that go into git. Elasticsearch, Kibana and Logstash each have a keystore command (elasticsearch-keystore, kibana-keystore, logstash-keystore), and all three configs expand ${VARIABLE} from the environment. Create API keys for machines rather than sharing the elastic password. An API key can be limited to certain indices and given an expiry, so a leaked key is a bounded problem:

POST /_security/api_key
{
  "name": "filebeat-writer",
  "expiration": "90d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [
        { "names": ["logs-*", "filebeat-*"], "privileges": ["auto_configure", "create_doc"] }
      ]
    }
  }
}

The response contains id, api_key and encoded. Beats and Logstash take the pair as id:api_key; raw HTTP clients send the encoded value in an Authorization: ApiKey <encoded> header. Copy the key when it is displayed; Elasticsearch cannot show it again. Note that elasticsearch-setup-passwords is deprecated; use elasticsearch-reset-password when you need a new password for a built-in user.

Common errors and how to read them

Elasticsearch errors are long, but they are well structured. Nearly all follow the same layout: an HTTP status (400 your request is wrong, 401 or 403 security, 404 not found, 429 the cluster is pushing back), an exception type, and a reason sentence. Read the type first, then the reason; it often contains the fix.

The node will not start

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]. A bootstrap check in production mode. Elasticsearch memory-maps its files and needs many mappings. Fix it on the host: sudo sysctl -w vm.max_map_count=1048576 (the value the 9.x documentation recommends; the check only requires 262144), and persist it in /etc/sysctl.conf. Package installs set this automatically. With Docker Desktop, it must be set inside the VM Docker uses.

max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]. Raise the limit with ulimit -n 65535, or LimitNOFILE=65535 in the systemd unit.

the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured. You bound to a real address without telling the node how to find a cluster. Set discovery.type: single-node for one node, or configure the discovery settings for several.

initial heap size [X] not equal to maximum heap size [Y]. Set -Xms and -Xmx to the same value in jvm.options.d/.

Clients cannot connect or authenticate

received plaintext http traffic on an https channel, closing connection. Elasticsearch's log, after a client used http:// against the TLS port. Use https://.

SSL certificate problem: self-signed certificate in certificate chain from curl, or x509: certificate signed by unknown authority from Filebeat. The client does not trust the cluster's certificate authority. Pass --cacert http_ca.crt to curl, or set ssl.certificate_authorities in Beats.

HTTP 401, missing authentication credentials for REST request or unable to authenticate user. No credentials, or wrong ones. Reset the password with bin/elasticsearch-reset-password -u elastic.

HTTP 403, action [indices:data/write/bulk[s]] is unauthorized for user [x]. You authenticated, but your role lacks the privilege. For data streams, give the writer create_doc and auto_configure on the index pattern.

Kibana: Kibana server is not ready yet. Either Kibana is still starting (wait a minute) or it cannot talk to Elasticsearch. Check that elasticsearch.hosts uses the right scheme, that the CA is trusted, that Elasticsearch is not red, and that both run the same version. The Kibana log tells you which.

Data problems

mapper_parsing_exception … failed to parse field [status] of type [integer]. A value contradicts the mapping. Covered above.

cluster_block_exception … disk usage exceeded flood-stage watermark, index has read-only-allow-delete block. The disk is 95% full, and Elasticsearch protects itself by making indices read-only. Free space or add capacity. The block lifts on its own when usage falls below the high watermark.

circuit_breaking_exception: [parent] Data too large. HTTP 429. A request would use too much heap. Make the query or aggregation smaller, and never aggregate on text fields.

Limit of total fields [1000] has been exceeded. An index has more than 1,000 fields, almost always because dynamic mapping made a field per key of some free-form object. Switch that object to flattened, set dynamic to strict or false, or map explicit fields.

Result window is too large, from + size must be less than or equal to: [10000]. You paged too deep. Use narrower queries now; search_after later.

Shipper problems

Filebeat exits because of a duplicate input ID, or the log input is deprecated. Give each filestream input a unique id, and stop using type: log.

Exiting: data path already locked by another beat. Two Filebeat processes share one data folder. Stop one.

Logstash _grokparsefailure. Your pattern does not match the line. Test it in the Grok Debugger.

Putting it all together

Here is a small project that uses everything above. You will build a tiny log-monitoring setup for a fictional model-serving application, from file to dashboard, in under an hour.

1. Start the stackRun the start-local script, read the password from .env, and confirm GET _cluster/health in Dev Tools.
2. Design the mappingCreate a model-logs index with @timestamp as date, host and level as keyword, latency_ms as integer, status as integer, and message as text.
3. Generate a log fileWrite a short script that appends a line such as 2026-09-30T10:00:05Z web-01 INFO 200 34ms predict ok every second with random hosts, levels, statuses and latencies, to a file, making sure it passes 1,024 bytes.
4. Parse with LogstashWrite a pipeline with a file or stdin input, a grok filter that extracts timestamp, host, level, status and latency as numbers, and a date filter that uses the timestamp as @timestamp. Output to model-logs. Run it with --config.test_and_exit first.
5. Check the dataUse GET model-logs/_count and a terms aggregation on level to confirm the numbers look right, and search for one specific error message.
6. Explore in KibanaCreate a data view, find the errors in Discover with KQL, then build a dashboard with count over time by level, average latency per host, and a table of the ten latest errors.
7. Break it on purposeAdd a line with latency_ms that is not a number and watch both the _grokparsefailure tag and, if you bypass grok, the mapper error.
8. Clean upDelete the index with DELETE model-logs and stop the stack with ./stop.sh.

The result is an end-to-end pipeline you understand rather than one you copied. You can name every stage, explain what each one did, and recognise the typical failure at each step.

What you can now do, and what comes next

You can start and verify a local stack, put data in by hand, in bulk and through shippers, search with match, term, range and bool, summarise with aggregations and ES|QL, set explicit mappings, parse text with Logstash, build a Kibana dashboard, and read the most common errors. That is enough to be useful on a team that already runs the stack.

What this guide deliberately left out: how indices age and get deleted automatically (index lifecycle management and data stream lifecycle), index and component templates in depth, ingest pipelines that parse inside Elasticsearch without Logstash, Fleet and Elastic Agent management, roles and users for a team, multi-node clusters, and upgrades. All of that is the Mid-level guide. Running the stack as a platform for other teams (sizing, security review, failure handling) is the Senior guide.

Natural neighbours in this catalogue: containers and Docker for running the stack itself (Docker), and Kubernetes for running it at scale (Kubernetes). When you deploy model services, the logs they produce are what this stack is for.

Before you go, re-run the Sixty-second self-test in the interview guide and skim the tips guide's checklist. Explaining the stack out loud is the quickest way to find what you only half understood.

Sources

• Elastic documentation
• Elastic Stack release notes
• Elastic Stack 9.5.4 release announcement
• Local development installation quickstart (start-local)
• start-local on GitHub
• Installing Elasticsearch
• Install Elasticsearch with the Debian package
• Install Elasticsearch with Docker
• Automatic security setup
• Bootstrap checks
• Virtual memory (vm.max_map_count)
• Important Elasticsearch configuration
• Elasticsearch breaking changes
• Kibana general settings
• Filebeat installation and configuration
• Beats breaking changes
• Installing Logstash
• Logstash settings file
• Logstash Elasticsearch output plugin
• Logstash breaking changes
• Install Fleet-managed Elastic Agent
• ES|QL reference
• Data streams
• Built-in users
• Fix watermark errors
• Circuit breaker errors
• Mapping explosion
• Elastic support and end-of-life dates

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. Everything here assumes Elastic Stack 9.5.4, so when an answer depends on a version, it says so.

The thirty-second answer

The ELK stack is three open products that turn logs and other event data into something searchable. Elasticsearch stores and indexes JSON documents and answers queries over a REST API; Logstash is an optional pipeline that parses and reshapes events on the way in; Kibana is the web interface for searching, charting and managing it all. Lightweight shippers, Filebeat or Elastic Agent, collect the data at the source, which is why Elastic now calls the whole family the Elastic Stack.

Then the line that shows you have used it: "a string field is either text, analysed for full-text search, or keyword, stored exactly for filtering and grouping, and confusing the two causes most beginner problems."

The model

ELK versus the Elastic Stack

Interviewers like to test vocabulary. ELK is Elasticsearch, Logstash and Kibana. The Elastic Stack is the official name and adds the collection side: Beats and Elastic Agent. If asked "what replaced the ELK stack", the honest answer is that nothing did: the stack grew, and the recommended collection path for new deployments is Elastic Agent with Fleet, while Logstash remains the heavy-duty transformation and routing tier.

It is also worth knowing two recent facts. Elastic's source has been available under the AGPLv3 alongside the Elastic License since September 2024. And Enterprise Search (App Search and Workplace Search) was removed in version 9.0, so a question that mentions them is about an old version.

Text versus keyword

The follow-up: "what does dynamic mapping do with a string?" It creates a text field with a .keyword sub-field, so you can search the first and aggregate the second. That is convenient but doubles the storage for fields you never full-text search, which is why you declare explicit mappings for data you keep.

The commands to have ready

GET /                                     # version and tagline
GET _cluster/health                       # green, yellow, red
GET _cat/indices?v                        # indices with health and doc counts
PUT web-logs/_doc/1 { "level": "error" }  # index one document
POST _bulk                                # many documents, NDJSON body
GET web-logs/_search { "query": { "match": { "message": "timeout" } } }
GET web-logs/_mapping                     # what types were guessed
POST _security/api_key { ... }            # machine credentials
DELETE web-logs                           # drop an index

curl -u elastic:$ELASTIC_PASSWORD http://localhost:9200                       # start-local (HTTP)
curl --cacert http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200 # secured cluster
bin/logstash -f pipeline.conf --config.test_and_exit                          # validate a pipeline
filebeat test config -e && filebeat test output                                # validate Filebeat
sudo elastic-agent status                                                      # Agent health

Cluster health colours

The classic question is "why is my single-node cluster yellow?" A replica is never placed on the same node as its primary, because that would protect against nothing, and with one node there is nowhere else to put it. On a laptop that is fine. In production, yellow means you have lost redundancy, and red means data is unavailable.

Logstash pipelines

input  { beats { port => 5044 } }
filter { grok { match => { "message" => "%{IP:client_ip} %{WORD:method} %{URIPATH:path} %{NUMBER:status:int}" } } }
output { elasticsearch { hosts => ["https://es-01:9200"] api_key => "${ES_API_KEY}" ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"] } }

Three stages: input, filter, output. Say what each does, and say when you would not use Logstash at all: when Filebeat modules or Elastic Agent integrations already parse the source, because an extra component is an extra thing to run, secure and monitor.

A request that cannot be parsed

1. Read the HTTP status400 means your request or document is wrong; 401 and 403 are security; 404 is a missing index; 429 means the cluster is pushing back.
2. Read the exception typemapper_parsing_exception, illegal_argument_exception, security_exception, circuit_breaking_exception. The type names the subsystem.
3. Read the reason sentenceIt usually names the field, the type, or the missing privilege, and often contains the fix.
4. Check the mappingGET index/_mapping shows what Elasticsearch believes the schema is.
5. Check the logsFor startup and cluster problems, the Elasticsearch log under /var/log/elasticsearch carries the detail.

The traps, and why they share one cause

Most beginner failures come from a short list of misunderstandings: field types decide what you can do, security is on by default, and search is near-real-time rather than instant.

Common interview questions

1. What is the ELK stack, and what problem does it solve?Elasticsearch, Logstash and Kibana: store and index event data, transform it on the way in, and explore it visually. It solves the problem that logs and metrics from many machines are useless unless you can collect them in one place, search them quickly, and summarise them. Officially the whole family, including Beats and Elastic Agent, is the Elastic Stack.
2. What does each of the three products do?Elasticsearch is the distributed search and analytics engine, with a REST API on 9200. Logstash is an input, filter and output pipeline for parsing and routing. Kibana is the web UI on 5601 for Discover, dashboards, Dev Tools and management.
3. What is the difference between Beats, Logstash and Elastic Agent?Beats are lightweight single-purpose shippers, Filebeat being the log one. Logstash is a heavier, general-purpose processing pipeline. Elastic Agent is one unified agent that replaces several Beats and is managed centrally through Fleet; it is the recommended path for new deployments. Beats remain supported.
4. When would you use Logstash, and when not?Use it for heavy or conditional parsing, enrichment, routing one stream to several destinations, or buffering in front of Elasticsearch. Skip it when integrations or ingest pipelines already do the job, because it is one more component to run and monitor.
5. What is an index, a document, and a shard?A document is one JSON object. An index is a named collection of documents with a mapping and settings. A shard is one Lucene index that holds a slice of an index's data; primaries hold data and replicas are copies on other nodes.
6. What is an inverted index, and why does search feel instant?A structure mapping each term to the documents containing it, like a book's index. A search is a lookup of terms rather than a scan of every document, so it stays fast as data grows.
7. Explain text versus keyword.Text is analysed into lowercase terms for full-text search; you query it with match. Keyword is stored as one exact value for filters, sorting and aggregations; you query it with term. Dynamic mapping creates both for a string.
8. What does yellow cluster health mean?Every primary shard is assigned but at least one replica is not. On a single node it is normal, because a replica cannot share a node with its primary. In a multi-node cluster it means reduced redundancy and deserves attention.
9. What does red mean, and what do you do?At least one primary shard is unassigned, so some data is unavailable. Use GET _cluster/allocation/explain to see why: a lost node, a full disk, or an allocation filter. Fix the cause rather than forcing allocation.
10. Is Elasticsearch real-time?Near-real-time. A newly indexed document becomes searchable after the next refresh, every second by default. A get by id is immediate; a search may not see it for up to a second.
11. What is the difference between match and term?match analyses the query text the same way the field was analysed, so it suits text. term looks for the exact stored value, so it suits keyword, numbers and dates. Using term on an analysed field with capital letters finds nothing, because the stored terms are lowercase.
12. What are the four clauses of a bool query?must and should contribute to the score; filter and must_not do not. filter is the place for yes-or-no conditions because it is faster and cached.
13. What is an aggregation?A summary computed over matching documents: bucket aggregations group (terms, date histogram), and metric aggregations compute numbers (average, sum, cardinality). They nest, which is how "errors per minute per host" is expressed. Set size: 0 when you only want the summary.
14. Why can you not aggregate on a text field?Because it is not stored in a per-document columnar structure by default; the error reads Fielddata is disabled. Aggregate on the keyword sub-field, or map the field as keyword.
15. What is dynamic mapping and what is its downside?Elasticsearch guesses field types from the first document it sees. It is convenient, but a wrong first guess fixes the type for the index, unexpected fields add up, and a mapping can explode past the 1,000-field default limit. For data you keep, declare the mapping.
16. Can you change a field's type?No. Create a new index with the correct mapping and reindex the data into it.
17. What changed about security in recent versions?Since 8.0, security is on by default: TLS for HTTP and node-to-node traffic, and required authentication. On first start a node generates certificates, sets the elastic password and prints a Kibana enrollment token valid for 30 minutes. That is why curl needs --cacert and credentials.
18. How do you connect Kibana to a new cluster?Generate an enrollment token with elasticsearch-create-enrollment-token -s kibana, paste it into Kibana's setup page, and enter the verification code from bin/kibana-verification-code. Kibana then authenticates with a service identity, never as the elastic superuser.
19. How should machines authenticate?With API keys scoped to the minimum privileges and given an expiry, not with a user password. A writer needs create_doc and auto_configure on the target data streams. Keys can be invalidated individually.
20. What is a data stream?An append-only abstraction over hidden backing indices, meant for time-stamped data such as logs. It needs a matching index template and an @timestamp field, and accepts only create operations. Its naming convention is type, dataset and namespace, for example logs-nginx.access-default.
21. What is a data view in Kibana?A saved definition of which indices or data streams to explore and which field is the time field. It used to be called an index pattern. If Discover is empty, check the time picker, then the data view.
22. What is ES|QL?A piped query language, FROM … | WHERE … | STATS … BY …, available in Discover and through POST /_query. It returns 1,000 rows by default, and since 9.1 it may return partial results, flagged with is_partial.
23. What is the current major version, and what must you know before upgrading?9.x, currently 9.5.4. Upgrades to 9 must come from 8.19.x, downgrades are not supported, and indices created in 7.x must be reindexed or deleted first. Snapshot before any upgrade.
24. Why does the Elasticsearch node refuse to start in production?Binding to a non-loopback address enables bootstrap checks, which are hard failures for unsafe settings, such as a low vm.max_map_count or too few file descriptors. They prevent running in a configuration likely to lose data.
25. Why is Filebeat not shipping my file?Work in order: filebeat test config -e for syntax, filebeat test output for the connection, then the paths and permissions, then file size (a file under 1,024 bytes is not picked up until it grows, because of the fingerprint file identity), then duplicate input ids.
26. How would you ship your application's logs into the stack?Have the app log to a file or stdout, use Filebeat or Elastic Agent to read it, parse at the source or in Logstash or an ingest pipeline, and write to a data stream with ECS field names. Then a data view in Kibana.
27. Why should you never use -k or turn security off?It hides the symptom by removing the protection. Provide the CA certificate instead, so the client still verifies the server.

Explaining the stack with a real story

Interviewers often ask for an example rather than a definition, and a short story is worth more than a perfect list of features. A good one has four beats: the problem, the path the data took, what you found, and what you changed.

A model-serving example works well. A prediction endpoint starts returning intermittent 502 errors, and the application runs on several hosts. On one machine you could have used grep, but across many hosts you would have lost an hour collecting files. Instead, Filebeat on each host ships the service log to Elasticsearch, where each line is a document with a timestamp, host, level and message. In Kibana you open Discover on the right data view, widen the time range to cover the incident, and search for level : "error". The histogram shows that errors began at one exact minute. A Lens chart broken down by host shows that only one host is affected. Expanding a document shows a timeout message from a downstream feature store. You now know the cause, the scope and the start time, in minutes.

Tell it in that order and you have shown that you understand the data flow, the two query interfaces, and the reason the stack exists. Avoid the temptation to list every component you know; an interviewer prefers one concrete path to a tour of the product catalogue.

What a strong answer covers that a weak one misses

A weak answer recites definitions: "Elasticsearch is a search engine, Logstash is a pipeline, Kibana is a dashboard." A strong answer adds the consequences. It says that Elasticsearch is near-real-time, so a just-written document may not be searchable for a second. It says that field types are chosen at index time and cannot be changed later, so the mapping is a design decision. It says that security is on by default, so clients need both credentials and the cluster's certificate authority. It says that Logstash is optional and that Elastic Agent is the recommended collection path for new work. And it says which version it is talking about, because the stack changed a great deal between 7.x and 9.x.

Two small habits also stand out. First, say "I would check" followed by a specific command, such as GET _cluster/health or GET index/_mapping, rather than "I would look at the logs". Second, when you do not know something, say how you would find out: the health API, the allocation explain API, the reason sentence in the error, or the official documentation. Interviewers hire people who can debug, not people who have memorised settings.

Sixty-second self-test

• Give the thirty-second answer, and name the five boxes of the data flow.
• Say what ELK stands for, what the official name is, and what Beats and Elastic Agent add.
• Explain text versus keyword, and say which one you aggregate on.
• Explain why a one-node cluster with a replica is yellow.
• Name the four bool clauses and say which ones score.
• Explain why a search might not see a document you just indexed.
• Name the ports 9200, 5601 and 9600, and what sits on each.
• Explain what the three stages of a Logstash pipeline are.
• Say what _grokparsefailure means and where you debug it.
• Explain why Filebeat 9 needs a unique id on each filestream input.
• Say why -k is the wrong fix for a certificate error.

Part one of three. Most early problems with this stack come from three misunderstandings: field types decide what you can do with a field, security is on by default so every client needs a credential and a certificate, and data takes a moment to become searchable. Start with the table, then the habits, then the checklist.

Common errors at this level

The mistakes beginners actually make

The table is for lookup. These are the mistakes worth understanding properly, each with the symptom, the cause and the fix.

1. Searching a text field with term

Symptom. { "term": { "host": "Web-01" } } returns no hits, although you can see the document in Discover.

Cause. host was dynamically mapped as text, so the stored terms are lowercase (web-01 was split and lowercased at index time). A term query does not analyse what you send; it looks for the exact string Web-01, which does not exist.

Fix. Use match on text fields, and term on keyword fields, numbers and dates. For exact matching on a dynamically mapped string, use the sub-field: host.keyword. Better still, map identifiers such as host as keyword when you create the index, so the question never comes up.

2. Trusting dynamic mapping for data you will keep

Symptom. A field that should be a number behaves like text; a later document is rejected with mapper_parsing_exception; storage is higher than expected; the field count creeps up.

Cause. The first document decided the type. A status code that arrived as the string "200" became text. Every string got a duplicate .keyword field. A free-form object added a new field per key.

Fix. For anything beyond experiments, create the index (or the template) with an explicit mapping, and set dynamic: strict or false where unexpected fields should not silently become columns. You cannot change a field type afterwards; you reindex. So a few minutes at the start saves a migration later.

3. Forgetting that a bulk request can half-fail

Symptom. Your script reports success and the bulk call returned HTTP 200, but the index is missing documents.

Cause. The bulk API returns 200 when the request itself was accepted; individual documents can still fail (mapping errors, version conflicts) and are reported inside the body with "errors": true.

Fix. Check the top-level errors flag, and on true, walk the items for entries with an error key. Retry or route the failures to a dead-letter location. Official clients have helpers that surface these for you.

4. Turning security off, or bypassing certificate checks

Symptom. A tutorial says to add -k, set ssl.verification_mode: none, or disable xpack.security.enabled, and suddenly everything works.

Cause. Elasticsearch creates its own certificate authority on first start. Your client does not trust it by default, so verification fails, correctly.

Fix. Give the client the CA certificate (--cacert, ssl.certificate_authorities, ssl_certificate_authorities in Logstash) or the CA fingerprint for Beats. Keep authentication on. The one exception is start-local on your own laptop, which intentionally uses HTTP, and which you must never expose.

5. Using the elastic superuser for everything

Symptom. Every script, shipper and dashboard tool shares one password with full rights; a leaked config file means a leaked cluster.

Cause. elastic is the easy credential, printed at install.

Fix. Use it to log in and set things up. Create API keys for machines, scoped to specific indices with an expiry, for example create_doc and auto_configure on logs-*. Give people roles that match their job. Kibana itself uses a service identity, not elastic.

6. Expecting a single node to be green

Symptom. Health is yellow and you believe something is broken.

Cause. Default indices ask for one replica, and a replica cannot sit on the same node as its primary.

Fix. Nothing on a laptop. If the yellow bothers you in a lab, set number_of_replicas to 0 on those indices. In production yellow means lost redundancy and should be investigated, and red means unavailable data.

7. Looking at the wrong time range

Symptom. Kibana says no results, or a dashboard is empty, after you indexed data.

Cause. The time picker defaults to a recent window, and your documents' @timestamp values lie outside it, often because a timestamp was missing and Logstash or Filebeat set it to ingest time, or because the test data is old.

Fix. Widen the range first. Then confirm the data view's time field is the one you mean, and that the documents have an @timestamp that parsed as a date. A date filter in Logstash exists for exactly this.

8. Shipping a tiny test file and seeing nothing

Symptom. Filebeat is running, test output passes, and a three-line test file never arrives.

Cause. Filebeat 9's filestream identifies files by a fingerprint of the first 1,024 bytes, so a file smaller than that is not ingested until it grows.

Fix. Use a longer file for tests, or append lines. Do not revert to the deprecated log input to avoid this; if you need the old behaviour, there is a documented setting for it, but the real fix is realistic test data.

9. Copy-pasting from a tutorial for an older version

Symptom. A config option is rejected; a URL returns 404; a command does not exist.

Cause. The tutorial targets 6.x, 7.x or early 8.x. Since then mapping types were removed, Enterprise Search and Logstash modules were removed, elasticsearch-setup-passwords was deprecated, the Logstash output renamed its SSL options, Beats multi-letter flags need a double dash, and Filebeat's log input was deprecated.

Fix. Check the date and version of every tutorial. Prefer the official documentation for 9.5, and treat the version in the page header as part of the answer.

10. Mixing versions

Symptom. Kibana will not start or shows odd errors after an install or partial upgrade.

Cause. Elasticsearch, Kibana, Logstash and Beats should run the same version. A Kibana newer than its Elasticsearch is unsupported.

Fix. Install matching versions from one repository line (9.x), and read the upgrade guide before changing versions. Downgrades are not supported; take a snapshot first.

The habits that pay off most

Verify each hop separately

When data does not appear, do not stare at Kibana. Follow it hop by hop from the source: is the file being written, does Filebeat report it (test output, the log with -e), did Logstash receive it (rubydebug output), did Elasticsearch accept it (GET index/_count), does Discover show it with the right time range. A pipeline is a chain, and the break is always at one link.

Keep secrets out of files

Use ${ES_PASSWORD} in Filebeat and Logstash configuration and supply the value from the environment or the product's keystore. Add your .env (the start-local one holds a password and an API key) to .gitignore the moment you create it. A password committed to git is leaked even if you delete it later, because it stays in history.

Start small, then widen

Begin with one index, one shipper and a handful of documents. Confirm the path works and that you understand every field in the mapping, then add sources. Adding fifteen log sources at once makes every error ambiguous.

Read the error top to bottom

Status code, exception type, reason. The reason often contains the fix, including the name of the field, the limit that was hit, or the privilege that was missing. Copy the exact message into the documentation search when it does not.

Learn the names Elastic uses today

Data view, not index pattern. Integrations, not modules, for Elastic Agent. filestream, not log. Data streams for logs, not daily indices. Using current vocabulary makes searching the documentation work, and it makes you sound current in an interview.

Write down what normal looks like

Spend ten minutes when everything works and record the baseline: the output of GET _cluster/health, the list from GET _cat/indices?v, the document count of your main index, and the shape of the Discover histogram. When something breaks at two in the morning, a saved picture of normal turns "this looks wrong" into "this number changed", which is a far faster way to find the fault. It also teaches you which numbers matter, and it costs nothing.

Delete what you no longer need

Experimental indices, old data views and forgotten test pipelines accumulate quickly and make later debugging confusing, because you cannot tell what is real. Delete practice indices with DELETE index-name when you finish with them, name experiments clearly (a test- prefix helps), and be careful with wildcards: DELETE logs-* on a real cluster is a serious mistake. On a shared cluster, check the name twice before you press run.

Practice cards

1. Index and findIndex five documents with _bulk, then write a bool query with one match in must and two conditions in filter. Expected: the hits you predicted before running it.
2. Break a mappingCreate an index with status as integer and index "abc". Expected: a 400 mapper_parsing_exception naming the field.
3. Explain yellowCreate an index on a one-node cluster and read _cat/indices?v. Then set replicas to 0 and watch it turn green.
4. Group by hostWrite a terms aggregation on host.keyword with size: 0, then rewrite the same question in ES|QL with STATS.
5. Ship and findPoint Filebeat at a file over 1,024 bytes and find the lines in Discover with KQL.
6. Parse and failWrite a Logstash grok pattern, feed it a matching and a non-matching line, and find _grokparsefailure.
7. Revoke a keyCreate an API key, use it, invalidate it with DELETE /_security/api_key, and observe the 401.

Operational recipes

Reset the elastic password. bin/elasticsearch-reset-password -u elastic prints a new random password; add -i to choose it yourself. On Docker: docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic.

Get a fresh Kibana enrollment token. bin/elasticsearch-create-enrollment-token -s kibana. It lasts 30 minutes, so do the enrollment straight away.

Find out why a shard is unassigned. In Dev Tools: GET _cluster/allocation/explain. Read the explanation and the deciders it lists.

See disk use per node. GET _cat/allocation?v. Watch for nodes near the 85% low watermark, well before the 95% flood stage.

Clear a read-only block after freeing disk. Elasticsearch releases it on its own once usage drops below the high watermark. If it does not, PUT */_settings { "index.blocks.read_only_allow_delete": null }.

Validate Logstash configuration. bin/logstash -f pipeline.conf --config.test_and_exit, and use --config.reload.automatic during development so edits apply without a restart.

Check a shipper end to end. filebeat test config -e, filebeat test output, then filebeat -e in the foreground to watch it work.

Take a clean slate. With start-local, ./stop.sh then ./uninstall.sh in the elastic-start-local folder removes the containers and data. Do this when an experiment has left indices and data views you no longer trust.

A short checklist

Before you call a setup finished, check each line.

• Every component runs the same version, 9.5.4 or later.
• GET / returns the expected version.number, and GET _cluster/health is understood (green, or yellow for a good reason).
• Clients reach Elasticsearch over the URL scheme the cluster uses (https unless it is a local start-local), with the CA trusted.
• No password or API key is committed to git; secrets come from the environment or a keystore.
• Machines use scoped API keys with an expiry, not the elastic superuser.
• Any index you will keep has an explicit mapping: keyword, date, number, text chosen on purpose.
• Time fields are real date types named @timestamp, and the Kibana data view points at them.
• Filebeat inputs are filestream with unique ids; Logstash options use the current SSL names.
• Bulk calls check the errors flag.
• You can explain yellow and red, text versus keyword, and why a fresh document might not be searchable for a second.
• You know where disk space is going, and you have not left network.host open to the internet.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. If you already use Elasticsearch at work, you have probably had the experience that it works beautifully for a month and then a disk fills, a shard will not allocate, or a dashboard times out, and nobody on the team can say why. The goal of this level is that you can predict those events from how the system is built, and design the cluster so they do not happen on a Friday night. Everything here is checked against Elastic Stack 9.5.4, released on 15 September 2026.

Where this picks up

Each section begins with the problem it solves and ends, in most cases, with a Try it you can run against a development cluster in a few minutes. All the REST examples are written in Kibana Dev Tools console syntax. If you prefer a terminal, add the https:// host, your CA certificate and credentials exactly as in Beginner.

A note on naming before we start. "ELK" is the common name, but the product Elastic ships is the Elastic Stack, and since version 7.x it has included Beats and Elastic Agent alongside Elasticsearch, Logstash and Kibana. All of those components share one version number, and you should keep them aligned. We use "ELK" and "Elastic Stack" interchangeably below.

How Elasticsearch stores data, precisely

When a document arrives, three things happen that explain almost every performance conversation you will have. Understanding them lets you predict behaviour rather than memorise tuning tips.

First, the document is routed to exactly one primary shard by hashing its ID (or its routing value). A shard is a complete Lucene index in its own right, and an Elasticsearch index is simply a named collection of shards spread across nodes. The number of primary shards is fixed when the index is created. You cannot change it in place, only with the split, shrink or reindex APIs, which is why the decision deserves thought up front. The number of replicas, by contrast, is a dynamic setting you can change at any time.

Second, the document is written into an in-memory buffer and is not yet searchable. It becomes visible to search at the next refresh, which by default happens every second (index.refresh_interval, default 1s). This is what "near real-time" means: a write is acknowledged quickly, but a search issued immediately afterwards may not see it for up to a second. Each refresh writes a small new segment, an immutable set of Lucene files.

Third, because segments are immutable, updates and deletes do not modify anything. A deleted document is merely marked as deleted in its segment, and an update is a delete plus a fresh insert. Background merges combine small segments into larger ones and drop the dead documents for good. That is why a heavily updated index uses more disk than the live data suggests, and why _forcemerge on an index still being written to is almost always a mistake. Force merge is for indices that will never receive another write.

Two data structures live inside each segment, and knowing which one a feature uses tells you its cost. The inverted index maps each term to the documents containing it, and it powers full-text search and term filters. Doc values are a columnar, on-disk structure that stores each field's value per document, and they power sorting and aggregations. A text field gets an inverted index but no doc values, which is exactly why you get the error Fielddata is disabled on [field] when you try to aggregate on it. A keyword field has both. The fix is always to aggregate on the keyword sub-field (message.keyword) or to map the field as keyword in the first place.

The consequence for operations is that refresh rate is a trade. If you are bulk loading a large historical dataset and do not need to search it during the load, raising index.refresh_interval (to 30s, or -1 to disable it during the load) reduces the number of tiny segments and the merge work that follows. Set it back afterwards. For normal logging workloads the default is fine, and the bigger lever is batching your writes through the _bulk API, which a shipper does for you.

Replicas serve two purposes. They are your high availability, because if the node holding a primary dies, a replica on another node is promoted. They also add read throughput, because searches can be served by any copy. They cost disk and indexing work, since every write is applied to the primary and then to each replica. A common and sensible economy is to keep one replica on hot data and none on data you can restore from a snapshot.

Mappings you control

Dynamic mapping is wonderful on day one and expensive by month three. Every new field name that shows up in a document becomes a permanent part of the index mapping and of the cluster state, which the master node must hold and distribute to every node. A logging application that emits a field whose name includes a user ID or a timestamp can generate tens of thousands of fields, and eventually you meet this error:

Limit of total fields [1000] has been exceeded while adding new fields [1]

That is a mapping explosion, and the limit (index.mapping.total_fields.limit, default 1,000) exists to protect the cluster, not to annoy you. Raising it treats the symptom. The real fix is to decide which fields you actually query and to map those explicitly.

The dynamic parameter is the dial. It accepts four values with different consequences:

A production logging template usually wants dynamic: false for the noisy object where arbitrary attributes land, and explicit mappings for the handful of fields that matter. The flattened type is a useful middle path for free-form key-value data such as labels or HTTP headers: the entire object is indexed as one field, so the field count stays at one no matter how many keys appear, at the price that all leaf values are treated as keywords.

PUT _component_template/app-mappings
{
  "template": {
    "mappings": {
      "dynamic": false,
      "properties": {
        "@timestamp":  { "type": "date" },
        "message":     { "type": "text" },
        "log.level":   { "type": "keyword" },
        "user.id":     { "type": "keyword" },
        "http.response.status_code": { "type": "long" },
        "labels":      { "type": "flattened" }
      }
    }
  }
}

Choosing between text and keyword is the decision that matters most. Use text for content people search by words, where the analyzer breaks the string into tokens and lowercases it. Use keyword for identifiers, enumerations, hostnames and anything you filter, sort or aggregate on, where you want the exact string. A field you need both ways can be given a keyword sub-field through a multi-field mapping, which is exactly what dynamic mapping does for strings by default (it creates field as text and field.keyword as keyword). Numbers deserve a thought as well. An HTTP status code is technically a number, but if you never do range queries or arithmetic on it, keyword is faster for exact filters. If you do range queries, use a numeric type.

You cannot change the type of an existing field. When a field is mapped wrongly, the path is to fix the template, create a new index (or roll a data stream over so the next backing index picks up the new mapping), and reindex the old data if you need it. This is why fixing the mapping in the template, and not patching one index, is the habit that matters.

The other mapping error you will meet constantly is a type conflict:

mapper_parsing_exception: failed to parse field [http.response.status_code] of type [long] in document with id '...'

It means the field is mapped as a number and a document arrived with a value that cannot be parsed as one, for example the string "n/a". You have three options. Fix the producer. Convert the value in an ingest pipeline with a convert processor. Or map the field with ignore_malformed: true, which indexes the document but skips the bad value (it remains visible in _source). The third option hides data quality problems, so use it deliberately and monitor how often it fires.

Templates and data streams

Logs and metrics are append-only and time-ordered, and the old way of handling them was a new index per day (app-logs-2026.09.30), created by the shipper, with a hand-maintained alias. It worked, but every part of it was your problem. A data stream is the purpose-built abstraction that replaces it. You write to one stable name, and Elasticsearch manages a series of hidden backing indices behind it, named like .ds-logs-app-default-2026.09.30-000001. The write always goes to the newest backing index, and reads fan out across all of them.

Three rules define a data stream, and most early errors come from forgetting one of them. There must be a matching composable index template with a data_stream: {} object. Every document must have an @timestamp field. And writes must use op_type=create, because data streams are append-only and do not allow updating or deleting individual documents by ID through the stream name. If you try to index with the plain index operation you get:

only write ops with an op_type of create are allowed in data streams

In the console, POST logs-app-default/_doc (auto-generated ID) works, as does PUT logs-app-default/_create/1. Logstash and Elastic Agent handle this for you when configured for data streams.

The naming convention is <type>-<dataset>-<namespace>, for example logs-nginx.access-production. Type says what kind of data (logs, metrics, traces). Dataset says where it came from. Namespace is yours to use for environment or tenant. This is more than tidiness: the Elastic integrations, the built-in templates and the default Kibana data views all key on the pattern, and roles can grant access by pattern such as logs-*-production.

Index templates are composable. A component template holds a reusable block of settings or mappings, and an index template lists component templates in composed_of and adds its own overrides. The legacy _template API is deprecated, so do not write new ones. When several templates match a name, the one with the highest priority wins, and two templates with the same priority and overlapping patterns are rejected.

PUT _index_template/app-logs
{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "priority": 500,
  "composed_of": ["app-mappings"],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-30d",
      "number_of_replicas": 1
    }
  }
}

Priority 500 is deliberate. Elastic's built-in templates for logs-*-* and metrics-*-* sit at a lower number, and a custom template for the same pattern that does not outrank them is silently beaten. The better pattern, where an integration's data is concerned, is not to replace the built-in template at all but to extend it through the @custom component templates the integrations expose, such as logs@custom for every logs data stream, or a dataset-specific one. That way an upgrade to the integration keeps working.

Once a data stream exists you manage it with a few calls you should know by heart:

GET _data_stream/logs-app-default
POST logs-app-default/_rollover
GET logs-app-default/_ilm/explain

The first lists the backing indices and the template and lifecycle in force. The second forces a rollover, which is how you apply a changed template immediately (new mappings only affect newly created backing indices). The third tells you, for each backing index, which lifecycle phase and action it is in and, crucially, why it is stuck if it is.

Lifecycle, tiers and retention

A data stream that only ever grows is a slow outage. Retention is not optional housekeeping but a design input, because storage is the dominant cost of a logging cluster and because a cluster full of old data is also a cluster full of shards it has to keep open. Elasticsearch gives you two mechanisms, and choosing between them is your first decision.

Index lifecycle management (ILM) is the full-featured engine. A policy defines phases, hot, warm, cold, frozen and delete, and each phase lists actions that run once an index reaches a minimum age. In the hot phase the important action is rollover: when the write index hits a condition, a new one is created and becomes the target. The conditions you should use are max_age, max_primary_shard_size and max_docs. The older max_size condition was deprecated in 9.3, because it counted the total including replicas and therefore behaved differently depending on replica count. Prefer max_primary_shard_size.

PUT _ilm/policy/logs-30d
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Read the policy as a story. Data is written to the hot index until it reaches 50 GB per primary shard or a day old, whichever comes first. A week after rollover, it is shrunk to one shard and force-merged to one segment, because by then nothing writes to it and fewer, larger segments are cheaper to search and to store. After 30 days it is deleted. Note that min_age is counted from rollover, not from when the document was written, so a backing index is typically a little older than the number suggests.

Data stream lifecycle is the simpler alternative, designed for data streams only. You state how long to keep data, and Elasticsearch handles rollover and deletion for you:

PUT _data_stream/logs-app-default/_lifecycle
{ "data_retention": "30d" }

It supports downsampling for metrics, and as of 9.5 it can move data to the frozen tier. Elastic Cloud Serverless uses it exclusively, because serverless hides nodes and shards. The trade-off is that it gives you less control: you cannot sequence arbitrary actions across phases. A sensible rule is to use data stream lifecycle when "keep N days, then delete" is the whole requirement, and ILM when you need tiers, shrink, searchable snapshots or custom ordering.

Data tiers are how phases map to hardware. Nodes carry roles such as data_hot, data_warm, data_cold and data_frozen, set in node.roles. Hot nodes take writes and recent searches and deserve fast disks and plenty of CPU. Warm and cold nodes hold older data on cheaper storage with fewer resources. The frozen tier holds data in searchable snapshots, indices whose data lives in a snapshot repository (object storage such as S3) and is cached locally on demand. That gives you long retention at object-storage prices, at the cost of slower queries on cold data. Searchable snapshots need an Enterprise licence, so check your subscription before designing around them. Also remember that frozen indices, the older feature of the same name, were removed in 9.0.

For organisations in the Gulf or Egypt with data residency rules, the snapshot repository deserves as much thought as the nodes. A repository bucket in a cloud region outside your jurisdiction is still a copy of your data outside your jurisdiction. Choose the bucket's region to match the same rules you applied to the cluster.

To find out why a backing index is not moving, ask ILM:

GET logs-app-default/_ilm/explain

The response gives phase, action and step per index, and if an index is stuck you will see a step of ERROR with the failure reason. The most common causes are a rollover target alias or data stream misconfiguration, a shrink that cannot allocate because there is no node with enough space, and an allocate action waiting on a node attribute that does not exist. After fixing the cause, POST _ilm/retry/<index> retries the failed step.

Index modes and what they save

Elasticsearch 9 makes a quiet but important change to how logs are stored. A data stream whose name matches logs-*-* uses the LogsDB index mode by default, conditional on the mapping documentation's criteria. An index mode is a preset that changes how an index is built internally to suit a workload. The available modes are:

The main saving comes from synthetic _source. Normally Elasticsearch stores the original JSON of each document in _source as well as the indexed structures, so the same values are stored twice. With synthetic source, _source is reconstructed from doc values when needed, so the raw JSON is not stored. You configure this through index.mapping.source.mode. (In 9.4 the old location, mappings._source.mode in a template, moved to this setting, so older templates need updating.) The trade-off is that the reconstructed document may differ cosmetically from what you sent: the order of fields can change, arrays of the same type can be sorted, and duplicates may be removed. For logs this is normally acceptable. For a search workload where you show the original document to users verbatim, stay on the standard mode.

There are other 9.x changes at this level you need to be aware of, because they bite during upgrades. From 9.2, norms are disabled for text fields in LogsDB and TSDB modes, which affects relevance scoring but is irrelevant for filtering logs. In 9.4, TSDB indices no longer carry sequence numbers, so optimistic concurrency (if_seq_no) fails on them, and update_by_query and delete_by_query have no conflict detection; setting index.disable_sequence_numbers: false restores the previous behaviour. In 9.5 the TSDB index.look_ahead_time default fell from 30 minutes to 9, so documents timestamped more than nine minutes in the future are rejected. If you have a fleet of machines with clocks that drift, this is a failure you will only notice after the upgrade, and the fix is to correct the clocks (or set the value in your template if you must tolerate skew).

For metrics that age, downsampling replaces full-resolution points with aggregates at a coarser interval, which is the single biggest saving on long-retention metrics. In 9.4 the ILM downsample action stopped force-merging by default, so if you relied on that, set force_merge_index: true or add a forcemerge action explicitly.

Ingest pipelines and where to parse

Raw log lines are strings. To be useful they must become structured fields, with a real timestamp and consistent names. The question at this level is not "how do I parse" but "where should parsing happen", because the three options have different failure modes and costs.

The modern default is: use Elastic Agent with an integration for well-known sources (nginx, AWS, Kubernetes), because the integration ships its own ingest pipeline and mappings and dashboards. Use an ingest pipeline for custom parsing of your own application's logs. Reach for Logstash when you need something the others cannot do well: enrichment from external databases, fan-out to several destinations, buffering in front of a fragile cluster, or complex conditional routing.

An ingest pipeline is a list of processors that run on an ingest node before the document is indexed. The processors you use most are dissect (fast, delimiter-based), grok (regular-expression-based, more flexible and slower), date (parse a timestamp into @timestamp), rename, set, remove, convert, geoip, user_agent and script. When your logs have a stable format, prefer dissect. It is cheaper than grok because it splits on literal delimiters instead of running regular expressions.

PUT _ingest/pipeline/parse-app
{
  "processors": [
    { "dissect": { "field": "message", "pattern": "%{ts} %{log.level} %{msg}" } },
    { "date":    { "field": "ts", "formats": ["ISO8601"] } },
    { "remove":  { "field": "ts" } }
  ],
  "on_failure": [
    { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } }
  ]
}

The on_failure block is the part people leave out and regret. Without it, a document that does not match the pattern makes the whole indexing request fail for that document. With it, the document is still indexed, carrying an error.message you can search for, so a format change in an upstream service shows up as a rising count of tagged documents rather than as silent data loss.

You must test a pipeline against sample documents before attaching it, and the simulate API is how:

POST _ingest/pipeline/parse-app/_simulate
{
  "docs": [
    { "_source": { "message": "2026-09-30T10:00:00Z ERROR payment declined" } }
  ]
}

The response shows the document after each processor if you add ?verbose, which is the fastest way to find which step mangled a field. Since 9.2 the simulate API returns HTTP 400 when a processor definition is itself invalid, where it used to be more forgiving, so a bad pipeline surfaces earlier.

There are two ways to attach a pipeline to an index. Either send ?pipeline=parse-app with each request, which is fragile because every client must remember, or set the index.default_pipeline setting in the template so every document runs through it. There is also index.final_pipeline, which runs after everything else and cannot be bypassed by the request, which makes it the place for organisation-wide rules such as stamping an ingest timestamp or dropping a forbidden field.

Logstash in depth

Logstash is a pipeline of inputs, filters and outputs, with codecs that decode and encode at the edges. You met a single beats input and elasticsearch output in Beginner. At this level the questions are throughput, durability and keeping one slow destination from stalling everything.

Every pipeline runs a number of workers, each pulling batches of events from the queue and running them through the filters and outputs. Three settings in logstash.yml (or per pipeline in pipelines.yml) govern this: pipeline.workers (default: the number of CPU cores), pipeline.batch.size (default 125 events) and pipeline.batch.delay (default 50 ms, how long to wait for a batch to fill). Bigger batches mean fewer, larger bulk requests to Elasticsearch, which is usually more efficient but uses more heap. If workers are idle while the queue is full, your bottleneck is downstream, in Elasticsearch. If workers are busy and CPU is saturated, it is your filters. Do not guess: the monitoring API tells you.

curl -s 'localhost:9600/_node/stats/pipelines?pretty'
curl -s 'localhost:9600/_node/hot_threads?pretty'

In the per-pipeline statistics, compare each plugin's duration_in_millis with its event count to find the slow filter. The hot_threads output shows what the busiest threads are doing, which for a stalled pipeline is often a filter waiting on a network call.

Durability is the next concern. By default the queue sits between input and filters in memory (queue.type: memory), so if Logstash crashes you lose whatever was in flight. Setting queue.type: persisted writes events to disk, which gives at-least-once delivery across restarts and absorbs bursts when Elasticsearch is slow. It is bounded by queue.max_bytes (default 1024mb). When the queue is full, Logstash applies backpressure to its inputs instead of dropping events, which for a Beats input means the shipper backs off and retries.

pipeline.workers: 4
pipeline.batch.size: 250
queue.type: persisted
queue.max_bytes: 8gb
dead_letter_queue.enable: true
api.http.host: 127.0.0.1

The dead letter queue (DLQ) handles a different failure. When Elasticsearch rejects an event permanently, for example with a mapper_parsing_exception because the value does not fit the mapping, retrying will never help. With dead_letter_queue.enable: true, the Elasticsearch output writes that event to the DLQ on disk instead of dropping it. You can later read it with the dead_letter_queue input plugin, fix the data, and reprocess it. It is off by default, which means that by default a mapping conflict silently costs you events, with only an error line in the Logstash log. Turn it on.

Multiple pipelines keep unrelated flows from interfering. Without them, everything in conf.d is concatenated into one pipeline, so a conditional mistake in one file affects all of them and one slow output blocks the lot. Declare them separately in pipelines.yml:

- pipeline.id: apache
  path.config: "/etc/logstash/conf.d/apache.conf"
  pipeline.workers: 2
- pipeline.id: app
  path.config: "/etc/logstash/conf.d/app.conf"
  queue.type: persisted

Pipelines can also feed each other with the pipeline input and output plugins. The documented patterns are the distributor (one input, split by type into separate pipelines), the output isolator (so one failing destination does not stall the other), the forked path and the collector. The output isolator is the one worth remembering: if you send every event both to Elasticsearch and to an archive in object storage, and the archive stalls, a single pipeline stalls both. Two downstream pipelines, each with its own persistent queue, decouple them.

Secrets belong in the Logstash keystore, not in the config file. Create it with bin/logstash-keystore create, add an entry with bin/logstash-keystore add ES_API_KEY, and reference it as ${ES_API_KEY} in the pipeline. Note that the Logstash output wants the API key in the id:api_key form, not the base64 encoded value the API returns, while the HTTP header form uses the encoded one. Mixing these up is a classic source of a 401 that "works in curl".

Version 9 changed several things that break upgraded configurations, and these are the ones an incident ticket usually quotes:

• The elasticsearch output's SSL option names changed and the old ones are removed, so a config that still says cacert fails to start.
• allow_superuser defaults to false, so Logstash refuses to run as root.
• The http.* API settings were replaced by api.*.
• Logstash modules were removed.
• The minimum JDK is 21 from 9.4, though a bundled JDK is included.

The Beats input renamed its own SSL options too (ssl to ssl_enabled, ssl_verify_mode to ssl_client_authentication). In 9.5, the Kafka integration moved to 12.x, removed the default and uniform_sticky partitioner values, and changed the linger_ms default from 0 to 5 ms.

input {
  beats { port => 5044 }
}

filter {
  if [log][level] == "debug" { drop { } }
}

output {
  elasticsearch {
    hosts => ["https://es-01:9200"]
    api_key => "${ES_API_KEY}"
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    data_stream => "true"
  }
}

Always validate before you restart: bin/logstash -f /etc/logstash/conf.d/app.conf --config.test_and_exit parses the file and exits non-zero on a syntax error, which makes it a good CI check on the pipeline repository.

Elastic Agent, Fleet and Filebeat 9

On the collection side there are two generations of tooling, and you will meet both at work. Beats are single-purpose Go shippers: Filebeat for logs, Metricbeat for metrics, Heartbeat for uptime checks, Auditbeat, Winlogbeat and Packetbeat. Elastic Agent is the unified successor, one binary that runs whichever integrations its policy asks for. Elastic recommends Elastic Agent with Fleet for new deployments, and Beats are still supported and shipped. The practical advice is: new hosts get Elastic Agent, and existing Beats estates get migrated when there is a reason, not as a project for its own sake.

An agent can run in two modes. A Fleet-managed agent enrols with Fleet Server (listening on port 8220) using an enrollment token, and thereafter takes its configuration from an agent policy held in Kibana. You change the policy once and every enrolled agent converges, which is the feature that makes a fleet of thousands manageable. A standalone agent reads a local elastic-agent.yml, which suits locked-down hosts and configuration-managed estates where you want the file in Git.

sudo ./elastic-agent install --url=https://fleet-server:8220 \
  --enrollment-token=<token> \
  --certificate-authorities=/path/ca.crt \
  --non-interactive --unprivileged
sudo elastic-agent status
sudo elastic-agent diagnostics

elastic-agent status is your first stop when a host is silent: it lists each component and whether it is healthy, and it tells you if the agent cannot reach Fleet Server or Elasticsearch. elastic-agent diagnostics produces an archive with logs, the rendered configuration and component state that you can attach to a ticket. The --unprivileged flag runs the agent without root where the integrations allow it, which is worth adopting on any host where the policy does not need privileged access.

An integration is a packaged unit for one source: collection configuration, an ingest pipeline, mappings, and dashboards. Adding the nginx integration to a policy gives you parsed access logs in logs-nginx.access-<namespace> and ready dashboards, where the same result by hand would be a day of work. Integration-managed data streams also follow the naming convention discussed above, so the roles and dashboards you write against logs-* keep working.

Migrating a Filebeat config to version 9

Filebeat 9 removed or changed enough behaviour that an unmodified 8.x config may fail or quietly behave differently. The changes to check, in order of how often they bite:

1. Filebeat refuses to start if two inputs share an ID. Every filestream input needs a unique, stable id. The ID is also how Filebeat remembers how far it has read each file, so do not generate it randomly and do not change it casually.
2. The log and container inputs are deprecated. They need allow_deprecated_use: true to run at all. Move to filestream.
3. The default file identity is now fingerprint, not native (inode-based). Filebeat identifies a file by the hash of its first 1,024 bytes, which survives renames and filesystems that reuse inodes. The side effect is that files smaller than 1,024 bytes are not ingested until they grow. If you see a tiny file that never ships, this is why. You can restore the 8.x behaviour with file_identity.native: ~ and prospector.scanner.fingerprint.enabled: false, but think twice: the fingerprint default fixes real data-loss problems with log rotation.
4. Multi-letter command-line flags need a double dash, so -path.config becomes --path.config.

filebeat.inputs:
  - type: filestream
    id: app-logs
    paths:
      - /var/log/app/*.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\d{4}-\d{2}-\d{2}'
          negate: true
          match: after

output.elasticsearch:
  hosts: ["https://es-01:9200"]
  api_key: "id:api_key"
  ssl.certificate_authorities: ["/etc/filebeat/http_ca.crt"]

The multiline parser above stitches stack traces onto the log line that started them: any line that does not begin with a date is appended to the previous event. Without it, a Java exception becomes thirty separate documents, each useless on its own. Test it with filebeat test config -e and filebeat test output, which validate the file and the connection including TLS and credentials before you start the service.

When Filebeat cannot reach Elasticsearch, its log shows Failed to connect to backoff(elasticsearch(https://...:9200)) followed by the underlying cause. x509: certificate signed by unknown authority means the CA is not trusted: set ssl.certificate_authorities, or use ssl.ca_trusted_fingerprint with the fingerprint of the CA certificate. The temptation to set verification to none is strong, and the answer is always no outside a throwaway lab.

Querying: filter context, aggregations and ES|QL

The Query DSL divides every clause into one of two jobs. In query context, a clause answers "how well does this document match?", and produces a relevance score. In filter context, a clause answers "does it match, yes or no?", produces no score, and is eligible for caching. A bool query combines them: must and should run in query context, filter and must_not in filter context. For logs and metrics you almost never care about scores, so almost everything should be in filter.

GET logs-app-default/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "term":  { "log.level": "error" } }
      ]
    }
  },
  "aggs": {
    "by_host": { "terms": { "field": "host.name", "size": 10 } }
  }
}

Three habits make that query fast. The time range is first and in filter context, so Elasticsearch can skip whole shards and segments whose time range cannot match. size: 0 tells it you want only the aggregation, not hits. And term on a keyword field is an exact match, where match would analyse the input. A frequent bug is using term on a text field: the indexed tokens are lowercased and split, so a search for "Payment Declined" with term finds nothing, which looks like a missing-data problem and is actually a query problem. Use match on text and term on keyword.

Aggregations come in three families: bucket aggregations that group documents (terms, date_histogram, range), metric aggregations that compute values over a set (avg, sum, cardinality, percentiles), and pipeline aggregations that operate on other aggregations' output. Two limits matter in practice. A terms aggregation returns approximate counts when a field has many distinct values spread over several shards, because each shard reports only its own top entries. And a single response may not contain more than 65,536 buckets, or you get a too_many_buckets_exception. When you need every bucket, such as exporting all distinct users, use a composite aggregation, which pages through buckets with an after key. The cardinality aggregation is also approximate by design, which is the right trade for "how many unique users", and wrong for anything that must reconcile to the last digit.

Deep pagination is the next trap. A search with from and size makes each shard produce from + size results, so the cost grows with the offset, and Elasticsearch blocks it beyond index.max_result_window (10,000):

Result window is too large, from + size must be less than or equal to: [10000] but was [10100]

Raising the limit is the wrong fix. The right one is search_after with a point in time (PIT). A PIT freezes a consistent view of the indices so your pages do not shift as new data arrives, and search_after continues from the last sort value rather than skipping:

POST logs-app-default/_pit?keep_alive=1m

GET _search
{
  "size": 1000,
  "pit": { "id": "<pit-id>", "keep_alive": "1m" },
  "sort": [ { "@timestamp": "asc" }, { "_shard_doc": "asc" } ],
  "search_after": [ 1790000000000, 42 ]
}

Remember to close the PIT (DELETE _pit) when you finish, since open PITs hold segments in memory. The scroll API is no longer recommended for deep pagination.

ES|QL is the piped query language that Kibana's Discover and the /_query endpoint both speak. It reads top to bottom, which suits log analysis, where you filter, group and sort in that order:

FROM logs-app-default
| WHERE @timestamp > NOW() - 1 hour AND log.level == "error"
| STATS errors = COUNT(*) BY host.name
| SORT errors DESC
| LIMIT 10

Through the REST API the same query goes to POST /_query?format=txt with { "query": "..." }. Several behaviours surprise people. A query returns 1,000 rows by default and at most 10,000, so a missing LIMIT does not mean you saw everything. Since 9.1, ES|QL returns partial results by default when some shards fail, and signals it with is_partial in the response, so check that flag before trusting a count on a degraded cluster; allow_partial_results=false, or the cluster setting esql.query.allow_partial_results, makes failures fatal. Also, from 9.1 a parameter that is a field or function name is written ??name, while a value parameter stays ?name.

The other query languages are good to recognise. KQL and Lucene syntax are what you type in the Kibana search bar. EQL finds sequences of events in a time window (login failure, then success, then a privileged command), which is the security use case. SQL at POST /_sql exists for tools that speak it.

Security in practice

Since version 8.0, security is on by default: TLS protects both the HTTP and the transport layers, and every request must authenticate. That is what the auto-configuration you saw in Beginner sets up on first start. The work at this level is turning that baseline into a model you can operate: who can do what, with which credential, and how to rotate it.

Start with the principle that every machine gets its own credential with the smallest privilege that works, and humans never share the elastic superuser. The right credential for a shipper is an API key. It is created with a role descriptor that limits it, and it can carry an expiration.

POST /_security/api_key
{
  "name": "logstash-writer",
  "expiration": "90d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [
        { "names": ["logs-*"], "privileges": ["auto_configure", "create_doc"] }
      ]
    }
  }
}

The response contains an id, the secret api_key, and encoded, the base64 of id:api_key. The secret is shown only once. Use Authorization: ApiKey <encoded> for raw HTTP. For Logstash and Beats, use the id:api_key form in their configuration. The privilege pair auto_configure and create_doc is what a writer to a data stream needs: it may create the data stream on first write and add documents, but it cannot read, update or delete. A key scoped like this that leaks costs you noise in a logs index, not your data.

When a writer is missing a privilege, the error is explicit, and worth reading in full:

action [indices:data/write/bulk[s]] is unauthorized for user [logstash-writer] with effective roles [writer] on indices [logs-app-default], this action is granted by the index privileges [create_doc,create,index,write,all]

It names the action, the identity, the roles in effect, the target and the privileges that would have satisfied it. Match the last list to a privilege you can grant, and grant that one and no more.

For humans, define roles that map to jobs, and assign users or map external identities to them. A reader role is typically read and view_index_metadata on the relevant patterns, plus Kibana feature privileges scoped to a Space.

PUT /_security/role/logs_reader
{
  "indices": [
    { "names": ["logs-*"], "privileges": ["read", "view_index_metadata"] }
  ]
}

Kibana Spaces divide dashboards and saved objects between teams, but they organise access to saved objects and features, not to data. Data access is still controlled by index privileges. Giving a team a Space does not stop its members searching an index they can read from Dev Tools. Field-level and document-level security, which restrict what a user sees inside an index, are paid-tier features, as are SAML and OIDC single sign-on for self-managed clusters and audit logging.

Kibana itself should connect to Elasticsearch with a service account token rather than a username and password, and never as elastic:

bin/elasticsearch-service-tokens create elastic/kibana my-token

elasticsearch.serviceAccountToken: "<token>"
xpack.encryptedSavedObjects.encryptionKey: "<at least 32 characters>"
xpack.security.encryptionKey: "<at least 32 characters>"
xpack.reporting.encryptionKey: "<at least 32 characters>"
server.ssl.enabled: true

The encryption keys deserve more attention than they usually get. Kibana uses xpack.encryptedSavedObjects.encryptionKey to encrypt sensitive attributes of saved objects, such as connector credentials and Fleet secrets. Without it, alerting and Fleet fail with "Missing encryption key" errors. The key must be identical on every Kibana instance and must be backed up: lose it, and the encrypted saved objects become unreadable. Generate them with bin/kibana-encryption-keys generate and keep them in your secret manager.

Secrets for the services themselves live in keystores: elasticsearch-keystore, kibana-keystore and logstash-keystore. After changing a secure setting on Elasticsearch, POST _nodes/reload_secure_settings reloads it without a restart for settings that support reload. Environment variable substitution (${VAR}) works in all three configuration formats, which suits container deployments. See the Docker guide for how those variables reach the container, and the Kubernetes guide for how to source them from Secrets.

One older tool to avoid: elasticsearch-setup-passwords has been deprecated since 8.0. Use elasticsearch-reset-password -u <user> or the security APIs.

Sizing shards and nodes

Oversharding is the single most common way a healthy-looking cluster degrades. Each shard is a Lucene index with its own memory overhead, file handles and entry in the cluster state, regardless of how little data it holds. A cluster of a few hundred gigabytes spread over thousands of tiny shards uses far more heap than the same data in a few dozen well-sized ones, and the master node suffers every time shards move.

The official guidance is to aim for shards of roughly 10 to 50 GB, and fewer than about 200 million documents per shard. The cluster has a default limit of 1,000 non-frozen shards per node (cluster.max_shards_per_node) and 3,000 on a dedicated frozen node. When you hit it, creating an index fails with:

this action would add [2] shards, but this cluster currently has [1000]/[1000] maximum normal shards open

Raising cluster.max_shards_per_node is a temporary bandage. The durable fixes are to roll over by size instead of by day so small indices do not accumulate, to shrink old indices, to delete what you no longer need, and to stop creating per-tenant or per-service indices with one primary each when a single data stream with a namespace would do. A master node needs roughly 1 GB of heap for every 3,000 indices it tracks, which is a useful number for sanity-checking.

GET _cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node
GET _cat/allocation?v

The first lists your largest shards first, so you can see both oversized and (by sorting ascending) undersized ones. The second shows disk used and available per node, which is the input to the watermark behaviour below.

Memory sizing follows a rule that surprises people: give Elasticsearch no more than half of the machine's RAM as heap, and keep it below the compressed-ordinary-object-pointer threshold, about 26 to 31 GB depending on the JVM. The other half is not wasted. Lucene relies on the operating system's file cache to keep hot segments in memory, and starving it is a common cause of slow searches. Heap is auto-sized from the node's roles and RAM in recent versions, so the common mistake now is to override it with a bad number. If you must, create a file in jvm.options.d/ with equal -Xms and -Xmx, and never edit jvm.options itself.

Disk watermarks protect the cluster from running out of space, and each one changes behaviour:

At the flood stage, writes fail with a cluster_block_exception mentioning TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark. The block releases automatically once usage falls back under the high watermark, but the correct response is to free space or add capacity first, not to edit settings. The emergency removal is PUT */_settings with {"index.blocks.read_only_allow_delete": null}, and it only helps if you have also freed space.

Finally, separate roles once you grow. A minimal production cluster has three master-eligible nodes across three availability zones, so that losing one zone still leaves a majority. Spread replicas across zones using allocation awareness: set node.attr.zone on each node and cluster.routing.allocation.awareness.attributes: zone. When a master-eligible node carries data and heavy searches, garbage collection pauses can cost the cluster its master, which is why larger clusters use dedicated masters. Elasticsearch should run alone on its hosts, and in particular Logstash belongs on its own machines.

Diagnosing a cluster beyond the colour

Cluster health is a summary of one thing: whether shards are assigned. It says nothing about whether the cluster is fast, about to fill a disk or rejecting writes. Good diagnosis moves from the general to the specific through a short list of APIs, and each answers a distinct question.

"Is something wrong, and what does Elasticsearch think?" The health report API returns named indicators with a status, an impact and, when something is wrong, a diagnosis with suggested actions and a link to documentation.

GET _health_report

"Why is this shard unassigned?" Do not guess. The allocation explain API gives the reason for the first unassigned shard, or for one you name:

GET _cluster/allocation/explain
{ "index": "logs-app-default-2026.09.30-000001", "shard": 0, "primary": true }

Its output lists each node and the allocation deciders that refused the shard, with messages such as "the node is above the high watermark cluster setting" or "there are too many copies of the shard allocated to nodes with attribute [zone]". The decider name tells you which mechanism to adjust. If a shard failed to allocate repeatedly and gave up, fix the cause and then run POST _cluster/reroute?retry_failed=true.

"Are we rejecting work?" Look at the thread pools and the rejection counters:

GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected

A growing rejected count on the write pool, together with es_rejected_execution_exception in client logs, means indexing is saturating the node. Clients should back off exponentially and send smaller bulk batches, and you should treat it as a capacity signal, not a transient. The message says whether the pressure was on a coordinating, primary or replica operation, which tells you which tier is the bottleneck. In 9.0, timeouts that used to surface as 5xx errors became HTTP 429, so a 429 can now mean either rejection or timeout, and the body distinguishes them.

"Is memory the problem?" Circuit breakers are the safety valves that reject a request rather than let it take the node down with an out-of-memory error:

circuit_breaking_exception: [parent] Data too large, data for [<http_request>] would be [...], which is larger than the limit of [...]

That parent breaker is the overall limit (95% of real memory by default). Look at GET _nodes/stats/jvm,breaker and, since 9.3, the easier GET _cat/circuit_breakers. The usual cause is a query that pulls a large aggregation or sorts on a field loaded into memory. Reduce the aggregation size, aggregate on keyword doc values instead of loading fielddata, or narrow the time range. If the trips are frequent on a steady workload, you need more heap or more nodes.

"What is this node busy doing?" GET _nodes/hot_threads samples the busiest threads and shows whether they are searching, merging or in garbage collection. GET _tasks?detailed&actions=*search* lists running searches, and POST _tasks/<task_id>/_cancel cancels a runaway one. Kibana's own dashboards can cause this load, so always check which queries are heaviest before blaming the cluster.

Snapshots, restores and upgrades

Replicas protect you from a node failing. They do nothing for an accidental delete, a bad reindex, a corrupted mapping or a ransomware incident, because every copy faithfully applies the mistake. Snapshots are the backup mechanism. A snapshot is an incremental copy of indices (and optionally cluster state) into a repository, which can be a shared file system or object storage such as S3, Google Cloud Storage or Azure. Because snapshots are incremental, after the first one only segments that are new since the last are copied.

PUT _snapshot/s3repo
{ "type": "s3", "settings": { "bucket": "es-snaps", "client": "default" } }

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "s3repo",
  "config": { "indices": "*", "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

Snapshot lifecycle management (SLM) runs that schedule and enforces retention, so backups happen without a cron job and old ones expire. In 9.1 the S3 repository moved to AWS SDK v2, which has consequences: you must set the region explicitly with s3.client.<name>.region in elasticsearch.yml, IMDSv1 is not supported, and credentials go in the keystore (elasticsearch-keystore add s3.client.default.access_key and secret_key). A repository that worked before the upgrade and now fails to verify is usually missing that region. Verify a repository with POST _snapshot/s3repo/_verify, and, more importantly, test a restore. A backup you have never restored is a hypothesis.

POST _snapshot/s3repo/nightly-2026.09.30/_restore
{ "indices": "logs-app-*", "rename_pattern": "(.+)", "rename_replacement": "restored-$1" }

Restoring under a new name, as above, lets you inspect the data without overwriting live indices, which is the safe way to recover a single data stream. Also note that the cluster must have enough room: a restore is a large write.

Upgrades deserve ritual. The rules of a rolling upgrade, in which the cluster stays available, are these. Take a snapshot first, because downgrade is not supported. Upgrade components in this order: Elasticsearch, then Kibana, then Fleet Server, then the agents, with Logstash and Beats after Elasticsearch, and never run a component newer than the Elasticsearch it writes to. Within Elasticsearch, upgrade one node at a time, frozen, cold, warm and hot data nodes first, then other non-master nodes, and the master-eligible nodes last. For each node:

1. Disable replica allocation by setting cluster.routing.allocation.enable to primaries, so the cluster does not start rebuilding replicas while the node is down.
2. Run POST _flush to speed recovery.
3. Stop the node, upgrade it, start it, and wait for it to rejoin.
4. Reset allocation by setting the setting to null.
5. Wait for the cluster to return to green before the next node.

For a major version jump, the path is fixed: 7.17 to 8.19.x, then to 9.x. You cannot go from 7.17 directly to 9. Indices created in 7.x must be reindexed, deleted or marked read-only before the 9.x upgrade. Kibana's Upgrade Assistant and GET _migration/deprecations list what needs attention, and the deprecation log (renamed in 9.0 from deprecation.elasticsearch to elasticsearch.deprecation) records use of deprecated features by your clients. Version 7.17 is out of support as of January 2026. The 8.x line's maintenance ends on 15 January 2027, so a team still on 8.x has a deadline, not a preference.

There is also a reason to stay current inside 9.x. A denial-of-service vulnerability (ESA-2026-182, CVE-2026-94396) in Elasticsearch affects 9.2.0 through 9.4.6 and 9.5.0 through 9.5.3, has no workaround, and is fixed in 9.4.7 and 9.5.4. A patch upgrade is the only remediation, and being able to do one calmly in an afternoon is the point of practising.

Monitoring the stack itself

A logging platform that nobody monitors is a platform that fails silently, usually at the moment you need it. Watch both the infrastructure and the pipeline's flow.

For the cluster, collect metrics with Elastic Agent (the Elasticsearch, Kibana and Logstash integrations) or Metricbeat, and ship them to a separate monitoring cluster. A cluster that reports its own health into itself stops reporting exactly when it is unhealthy. The legacy internal collection, in which the nodes push their own monitoring data, is deprecated in 9.5 and scheduled for removal in 10.0, and the legacy monitoring REST APIs were deprecated in 9.5.3. Elastic also offers AutoOps, which provides hosted diagnostics and recommendations for clusters.

What to alert on, in order of value:

• Ingest lag: the difference between now and the newest @timestamp in each important data stream. Silence is the most dangerous failure, and this catches a broken shipper, a stalled Logstash or a full disk that no error message announced.
• Disk headroom per node against the watermarks, with an alert well before 85%.
• Write rejections and circuit breaker trips, as counts per interval.
• Unassigned shards for longer than a few minutes.
• Logstash queue depth and DLQ growth.
• JVM heap after garbage collection, sustained above about 75%.
• Snapshot failures and the age of the newest successful one.

Kibana's alerting rules can do all of this, with connectors that deliver to Slack, PagerDuty, email or a webhook. Remember that the email connector validates proxy TLS certificates from 9.5.2, with the escape hatch xpack.actions.ssl.proxyVerificationMode: none that you should treat as temporary. Connectors and rules depend on the encryption key discussed earlier.

Elasticsearch also emits slow logs, per index, for searches and indexing over thresholds you set, for example index.search.slowlog.threshold.query.warn: 5s. Treat them as your standing answer to "which query made the cluster slow last night". If you already run Prometheus and Grafana for the rest of your estate, read the Prometheus guide and Grafana guide: Grafana can use Elasticsearch as a data source, and PromQL is generally available in Elasticsearch 9.5. For a vendor-neutral instrumentation route, the OpenTelemetry guide explains the collectors and SDKs that Elastic's native OTLP endpoint receives.

Putting it all together

Here is the design that falls out of everything above, for a typical team that ships the logs of a dozen services on Kubernetes and some virtual machines, with a 30-day retention target. Treat it as a pattern to adapt, not a recipe to copy.

Collection. Elastic Agent runs on every host, and as a DaemonSet in the cluster (see the Kubernetes guide for the pattern, and the Terraform guide if you provision the infrastructure as code). Policies are managed in Fleet and each service's log path has an integration or a filestream input with a stable id. Agents send to Logstash only where enrichment or buffering is needed, and to Elasticsearch directly otherwise.

Schema. One component template holds the shared mappings (ECS field names, dynamic: false on the noisy objects), and one index template per service family composes it. The data stream names follow logs-<service>-<environment>. Custom parsing lives in a tested ingest pipeline wired through index.default_pipeline, with an on_failure that preserves the raw message.

Retention. A logs-30d ILM policy rolls over at 50 GB per primary shard or one day and deletes at 30 days. If there is no tiering, data stream lifecycle with data_retention: 30d is simpler still. LogsDB reduces storage for the price of synthetic source.

Security. Each shipper has an API key with auto_configure and create_doc on its own pattern, expiring in 90 days, stored in a keystore or secret manager. Kibana uses a service account token. The encryption keys are identical across Kibana instances and backed up. Humans use roles, never elastic.

Operations. A nightly SLM policy writes to an object-storage repository in the region that satisfies your residency requirements, and a quarterly restore drill proves it. Monitoring ships to a separate cluster. Alert rules watch ingest lag, disk headroom, rejections, unassigned shards and queue depth. Upgrades follow the rolling procedure with a snapshot before each one.

The same design expressed as a short checklist you can put in a pull request template:

1. Does every new data stream match a template with explicit mappings and a lifecycle?
2. Is dynamic set deliberately, and is the field count bounded?
3. Do new filters go in filter context, and does every query carry a time range?
4. Does each writer have its own least-privilege, expiring credential?
5. Are shards projected to land between 10 and 50 GB?
6. Is there an alert that fires when this data stops arriving?

Where you are now

You can now reason about Elasticsearch from the inside: why a write is not instantly searchable, why a mapping choice is permanent, why a shard count is a cost and not a detail, and why a logging cluster is, above all, a retention problem. You can choose where to parse, keep Logstash durable, migrate a Filebeat configuration to version 9, give each machine its own small credential, and read the cluster's own explanations when something misbehaves. You can also back up, restore and upgrade without hoping.

The Senior level takes the same system and asks what breaks first as it scales, how to isolate tenants and control cost, how to design its security and supply chain, and how to run it as a platform that other teams depend on. Before you go there, it is worth doing two things on a real cluster: run a restore drill, and measure how many of your shards are under 1 GB. The answers tend to set the agenda for the next quarter.

Sources

• Elastic documentation home
• Elasticsearch release notes
• Elasticsearch breaking changes
• Elasticsearch deprecations
• Kibana breaking changes
• Logstash breaking changes
• Beats breaking changes
• Elastic Stack 9.5.4 release announcement
• Security update ESA-2026-182
• Elastic product end-of-life dates
• Data streams
• Size your shards
• Self-managed security auto-configuration
• Built-in users
• Logstash settings file
• Logstash elasticsearch output plugin
• Filebeat installation and configuration
• Elastic Agent command reference
• ES|QL reference
• ES|QL limitations
• Upgrade an Elasticsearch deployment
• Prepare to upgrade
• Fix watermark errors
• Circuit breaker errors
• Rejected requests
• Mapping explosion

Part two of three. A cumulative review of Beginner and Mid-level material, organised by scenario and trade-off rather than by level. At this stage interviewers stop asking "what is a shard" and start asking "the cluster went red at 2 a.m., walk me through it". They are listening for whether you diagnose from evidence, whether you know the cost of each choice, and whether you mention the thing that would bite the team next month. Expect about forty minutes of reading, then practise answering aloud.

Where this picks up

The model in one screen

The Elastic Stack collects data with lightweight shippers, optionally transforms it in Logstash or ingest pipelines, stores it as shards of Lucene indices in Elasticsearch, and exposes it through Kibana. Logs and metrics live in data streams with a lifecycle, and the whole design is constrained by two resources: shards and disk.

Say "Elastic Stack" as well as "ELK", and mention Elastic Agent. It signals that you know Elastic recommends Agent with Fleet for new deployments, and that Beats are still supported.

Scenario questions

"The cluster is red. What do you do?"

A weak answer restarts nodes. A strong answer begins with what red means: at least one primary shard is unassigned, so some data is unavailable. Then it proceeds in a fixed order.

1. Ask the cluster what it thinks: GET _health_report for the indicators and their diagnoses, and GET _cat/shards?v filtered to unassigned shards.
2. Ask why one shard is unassigned: GET _cluster/allocation/explain. The output lists each node and the allocation decider that refused the shard. Typical deciders are the disk watermark, an awareness or attribute rule, a max_retries failure count, or simply that no node holds a copy of the primary.
3. Fix the named cause. If a node is gone, bring it back. If disks are full, free space. If the shard failed to allocate several times, fix the underlying problem and run POST _cluster/reroute?retry_failed=true.
4. Confirm recovery with _cat/recovery or _cat/shards, and only then write up what happened.

What a strong answer adds: it asks whether the data can be restored from a snapshot if the primary's only copy is truly lost, and whether replicas were configured, because a red index with no replica and a dead disk is a restore conversation. It also avoids the dangerous reroute commands that accept data loss unless that decision has been taken deliberately.

"Indexing suddenly slowed and clients are getting 429s."

Name the two families of rejection: a full thread pool queue, and indexing pressure. The exception is es_rejected_execution_exception, and the message says whether a coordinating, primary or replica operation was rejected. Then check:

GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected
GET _nodes/stats/jvm,breaker,indexing_pressure
GET _nodes/hot_threads

Explain the likely causes in order: a burst of traffic bigger than capacity, merges or garbage collection starving indexing, an expensive ingest pipeline (an unanchored grok), too many small bulk requests, or one node that is slower than its peers because of a failing disk. The remedies follow the cause: smaller and fewer bulk requests, client back-off with exponential delay, a cheaper pipeline, raising refresh_interval during a bulk load, more nodes. A strong answer states that raising queue sizes hides the problem, and notes that in 9.0 timeouts also moved to HTTP 429, so the response body should be read to distinguish rejection from timeout.

"Disk filled up and writes started failing with a cluster block."

The error contains TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block. The mechanism is the three watermarks: at 85% (low) the node stops receiving new shards, at 90% (high) Elasticsearch tries to relocate shards off it, and at 95% (flood_stage) indices with a shard on it become read-only apart from deletes. The block lifts automatically when usage drops under the high watermark.

The response is to free space first: delete the oldest indices, or expedite an ILM delete, or add nodes. Only then, if the block has not cleared, remove it with PUT */_settings and {"index.blocks.read_only_allow_delete": null}. The follow-up is the point that distinguishes the answers. Why did we reach 95%? Because retention was not enforced, because one data stream grew tenfold, or because nobody alerted at 80%. The action item is an alert on disk headroom and a retention policy that actually runs, not "watch it more closely".

"The pipeline is slow. What do you check first?"

Decide whether the bottleneck is upstream of Elasticsearch, inside Logstash, or in the cluster, because the remedies differ. For Logstash, use the monitoring API at port 9600:

curl -s 'localhost:9600/_node/stats/pipelines?pretty'
curl -s 'localhost:9600/_node/hot_threads?pretty'

Compare events in and out per plugin and each plugin's duration. If one filter dominates, optimise it. If the output dominates and workers are idle, Elasticsearch is the bottleneck, so look at rejections and bulk sizes. If CPU is saturated across workers, raise pipeline.workers only up to the core count, then batch size (pipeline.batch.size, default 125). If the persistent queue is filling and never draining, downstream cannot keep up. A strong candidate mentions that Logstash applies backpressure to its inputs, so the symptom at the shipper is retries and a growing backlog on disk, not lost events.

"Dashboards time out, but only for the last 30 days."

The likely cause is that queries span too many shards or too much data. Look at the time range filter, which should prune shards; at oversharding, because every shard in range is searched; at the aggregation (a terms aggregation on a high-cardinality field, or a huge date_histogram with a tiny interval); and at text fields used for sorting or aggregation, which produce the Fielddata is disabled error or, if fielddata were enabled, large heap use. Use the profile API and the search slow log for evidence. A strong answer also proposes structural fixes: pre-aggregate with downsampling or a transform for heavy dashboards, use the warm tier for older data, and use ES|QL or filtered dashboards rather than unbounded ones.

"A deployment to production added a field and ingestion for that service stopped."

The probable cause is a mapping conflict: the field already exists with a different type, so the documents containing it are rejected with mapper_parsing_exception: failed to parse field [...] of type [long]. Confirm by reading the bulk response, which lists per-item errors (a 200 bulk can still contain failures, so clients must inspect errors). If the shipper is Logstash, the DLQ would hold the rejected events if it was enabled. Fixes: correct the producer, convert the field in an ingest pipeline, or decide that ignore_malformed is acceptable. The deeper remark to make: the same field name should have the same type everywhere, which is what the Elastic Common Schema exists for, and a template with explicit mappings prevents the first document from deciding the type by accident.

"We upgraded to 9.x and Filebeat stopped shipping small files."

This is the fingerprint file identity change. In Filebeat 9 the default file_identity is fingerprint, which hashes the first 1,024 bytes, so a file smaller than that is not ingested until it grows. The related upgrade failures an interviewer may ask about: Filebeat refuses to start if two inputs share an id, the old log and container inputs need allow_deprecated_use: true and should be replaced by filestream, and multi-letter flags need a double dash. For Logstash, the output's old SSL options such as cacert are removed, so the pipeline fails at startup, and allow_superuser defaults to false, so it refuses to run as root.

Trade-off questions

Logstash or ingest pipelines?

Both transform data, and the honest answer is "it depends on what you need around the transformation". Ingest pipelines run inside Elasticsearch on ingest nodes. They keep the architecture small, are versioned with the cluster, and integrations ship their own. Their cost is CPU on the cluster that also serves searches. Logstash is a separate tier with a rich plugin set, it can enrich from external sources, route to several outputs, buffer with a persistent queue in front of a fragile cluster, and it isolates parse load from the cluster. Its cost is another JVM to size, patch and monitor. A good answer picks the lighter tool first: Agent plus integration, then an ingest pipeline, and Logstash when a specific need justifies it, such as fan-out, external enrichment or buffering.

ILM or data stream lifecycle?

Data stream lifecycle is simpler: you set data_retention, and rollover and deletion are handled, with downsampling support and, in 9.5, movement to the frozen tier. It is also the only mechanism in Serverless. ILM gives explicit phases and actions (shrink, forcemerge, allocate, searchable snapshot, downsample), so it is the choice when you need tiers or a specific sequence. Mention that neither is free: both depend on rollover producing well-sized shards, and min_age counts from rollover.

One big index or many small ones?

Many small shards cost heap, file handles and cluster state, however little data they hold. Elastic's guidance is shards of about 10 to 50 GB, under roughly 200 million documents, with a default cap of 1,000 non-frozen shards per node. One index (or data stream) per tenant or per service is convenient, and it becomes a problem when it means thousands of tiny shards. The alternatives are a shared data stream with a namespace or a tenant field (with filtered aliases or document-level security, a paid feature, for isolation), or size-based rollover so small streams do not generate a shard per day. Hard isolation, for regulatory reasons, may still justify separate clusters, and cross-cluster search can bring them together.

Standard mode or LogsDB?

LogsDB is the default for logs-*-* data streams in 9.x, and it saves storage using sorting, tuned compression and synthetic _source. The cost is that _source is reconstructed and may differ in field order or array handling from the original, which matters if you show originals verbatim or depend on exact representation. A strong answer says to measure on your own data and keep standard mode for search-style workloads where the document is the product.

Self-managed, ECK, or Elastic Cloud?

Self-managed gives full control and demands full responsibility for upgrades, security and capacity. ECK puts the same stack on Kubernetes with an operator, so it suits teams already invested in Kubernetes; ECK 3.x is required for Stack 9.x. Elastic Cloud Hosted offloads operations and runs on AWS, GCP or Azure, which matters for data residency in the Gulf or Egypt, since you choose the cloud region. Serverless removes node and shard management entirely, with fewer knobs. Raise the licensing angle too: the default distribution is under ELv2, AGPLv3 is available as an open-source option since 2024, and several features, such as searchable snapshots and SAML single sign-on on self-managed, need a paid tier.

Direct to Elasticsearch, or through Kafka?

A buffer such as Kafka in front of Logstash absorbs bursts, lets you replay, and decouples producers from consumers. It adds another system to operate. Logstash's persistent queue covers many cases with no extra system, at-least-once across restarts but bounded by disk on one node. A strong answer picks by the requirement: replay or multiple consumers justify Kafka, while modest burst protection justifies the persistent queue.

Replicas: how many?

One replica is the normal production baseline, giving high availability and read throughput. Zero replicas halves storage but means a node loss makes data unavailable until restored, which is tolerable only for data you can rebuild or that is on a tier restorable from a snapshot. More than one helps read-heavy search, but each copy adds write cost and disk. Also mention allocation awareness so that replicas land in a different zone from the primary.

What a strong answer covers that a weak one misses

Commands to have ready

GET _cluster/health?pretty
GET _health_report
GET _cat/nodes?v
GET _cat/shards?v&s=store:desc
GET _cat/allocation?v
GET _cluster/allocation/explain
GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected
GET _nodes/stats/jvm,breaker,indexing_pressure
GET _nodes/hot_threads
GET logs-app-default/_ilm/explain
GET _data_stream/logs-app-default
POST logs-app-default/_rollover
GET _migration/deprecations
POST _cluster/reroute?retry_failed=true

Be able to say in one sentence what each returns. The ones interviewers most often name back at you are allocation/explain, _cat/shards, _ilm/explain and _health_report.

Classic traps and why they happen

Mapping explosion. Every new field becomes permanent cluster state. A service that puts IDs into field names produces tens of thousands, until the 1,000-field limit (Limit of total fields [1000] has been exceeded). The answer is explicit mappings, dynamic: false or strict, or flattened.

text versus keyword. term on a text field finds nothing because the indexed tokens are analysed, and aggregating on text fails with Fielddata is disabled. Use match for text and term or .keyword for exact values.

Deep pagination. from + size beyond 10,000 fails with Result window is too large. Use search_after with a point in time.

Data streams are append-only. only write ops with an op_type of create are allowed in data streams appears when a client sends an update or an index operation with an ID.

Templates only apply at creation. Editing a template changes nothing for existing indices. Roll over the data stream.

cluster.initial_master_nodes left in place. It is a bootstrap setting and should be removed after the first election, or a restarted node may form a separate cluster.

ILM min_age counts from rollover, not from ingestion, so data lives a little longer than the number suggests.

A 200 bulk with errors. The bulk API returns HTTP 200 even when individual items failed. Clients must check the errors flag and the per-item results.

Kibana spaces are not data security. A Space limits saved objects and features. Data access is controlled by index privileges.

Common interview questions

How does a document become searchable? It is routed to a primary shard by hashing its ID, written to an in-memory buffer and replicated. A refresh (every second by default, index.refresh_interval) writes a new segment that search can see. Until then it is acknowledged but not searchable.

Why can't you change the number of primary shards? Routing uses a hash modulo the primary count, so changing it would send existing IDs to the wrong shard. You use the split or shrink APIs, which create a new index, or you reindex.

What is the difference between a filter and a query? A query clause scores relevance. A filter clause is a yes-or-no match that is not scored and can be cached. Logs and metrics should use filters.

What does the Elastic Common Schema give you? Agreed field names (@timestamp, host.name, log.level, source.ip) so dashboards, detections and queries work across sources. It is also converging with the OpenTelemetry semantic conventions.

Why does Elasticsearch need a majority of master-eligible nodes? The cluster elects a master by quorum so that a network partition cannot produce two masters. With three master-eligible nodes, losing one is tolerated and losing two is not. An even number adds cost without adding tolerance.

How would you give an application write-only access to logs? An API key created with a role descriptor granting auto_configure and create_doc on the application's data-stream pattern, with an expiration, stored in the application's secret store.

What is a point in time? A lightweight, consistent view of the indices, kept open for a keep_alive duration, so that paging with search_after does not shift as new data is indexed.

What changed for Logstash in version 9? allow_superuser is false by default, the http.* settings became api.*, modules were removed, old SSL option names were removed from the elasticsearch output and Beats input, and the minimum JDK rose (17 in 9.0, 21 in 9.4).

How do you upgrade without downtime? Snapshot, then a rolling upgrade one node at a time: set allocation to primaries, flush, upgrade and restart the node, reset allocation, wait for green. Masters go last, there is no downgrade, and from 7.x you must pass through 8.19.x.

What are the three disk watermarks? Low (85%), high (90%) and flood stage (95%), with the behaviours described above.

How do you monitor the stack? Elastic Agent or Metricbeat into a separate monitoring cluster, plus alerts on ingest lag, disk headroom, rejections, unassigned shards and queue depth. The legacy internal collection is deprecated in 9.5.

How would you connect this to the rest of a platform? Elastic Agent as a DaemonSet or ECK on Kubernetes, infrastructure from Terraform, metrics in Prometheus and Grafana, and traces through OpenTelemetry, which Elasticsearch accepts through a native OTLP endpoint.

Handling "I don't know"

Interviewers at this level care more about your method than your recall. If you are asked about a setting you do not remember, say what you would check: the settings reference, GET _cluster/settings?include_defaults=true, or the release notes for the version in question. State the version, because Elasticsearch changes quickly: the answer for 7.x, for 8.x and for 9.x can differ, and naming the version is itself a mark of experience. If you cannot recall whether a feature needs a paid tier, say that you would check the subscriptions page before designing around it.

Final self-test

Answer these aloud in under a minute each, without notes.

1. A single primary shard is unassigned and the cluster is red. List your first four actions.
2. Explain near real-time search in three sentences.
3. Why might a cluster with plenty of free disk still refuse to create an index?
4. What is the difference between term and match, and what goes wrong if you confuse them?
5. Name two places to parse a log line and one reason to choose each.
6. What does queue.type: persisted protect against, and what does dead_letter_queue.enable protect against?
7. Write the role descriptor for a least-privilege Logstash writer.
8. Why is from: 20000, size: 10 rejected, and what replaces it?
9. What are the two checks you do before a rolling upgrade, and which nodes go last?
10. Which Filebeat 9 change stops a 500-byte log file from being shipped?
11. Give three levers that reduce storage cost, in order of effect on a typical logging cluster.
12. What would you alert on to detect that a service has silently stopped sending logs?

If you can answer eight of the twelve fluently and explain the reasoning behind the rest, you are ready for a mid-level Elastic Stack interview. The Senior level moves the same questions to system-design scale: what breaks first, how to isolate tenants, and how to argue the trade-off to leadership.

Part two of three. At this level your cluster is up and your dashboards work, so the problems change character. They are no longer "why won't it start" but "why is ingestion lagging", "why did the upgrade break the shipper", and "why is the cluster yellow again". Start with the error table, then read the practices in order of leverage, the practice cards, and the recipes you will reach for at 2 a.m.

Common errors at this level

Cumulative: Beginner's errors still apply. These are the ones that appear once things work. The exact messages are quoted where they help you search logs.

The practices that pay off most

In order of leverage, the things that prevent the most incidents per hour invested.

1. Write explicit templates before the first document. One component template with deliberate mappings prevents mapping explosion, type conflicts and surprise text fields. Everything else is easier when the schema is decided.
2. Put a retention policy on every data stream on day one. Retention is a design input. Disk that fills is the most common cause of a bad night.
3. Size shards by bytes, not by calendar. Roll over at 50 GB per primary shard or a day, whichever comes first, and stop creating a tiny shard per service per day.
4. Give every writer its own expiring API key, scoped to auto_configure and create_doc. Nobody uses elastic, and a leaked key can only add noise.
5. Alert on silence. Ingest lag per important data stream catches the failures that raise no error.
6. Test a restore every quarter. Replicas are not backups. SLM plus a rehearsed restore is.
7. Keep every component on the same, current version, and patch within 9.x. The 9.5.4 release fixed a denial-of-service issue with no workaround.
8. Enable the dead letter queue wherever Logstash writes to Elasticsearch.
9. Put time filters first, in filter context, on every query and every dashboard.
10. Keep Logstash and Elasticsearch configuration in Git, and validate pipelines in CI with --config.test_and_exit.

Practice cards

Making the pipeline reliable, in order

Reliability in a logging pipeline is a chain, and it is only as strong as its weakest hop. Walk it from the source.

At the source. Give each filestream input a stable id, because the registry that remembers read offsets is keyed on it. Use a multiline parser for stack traces. Keep log rotation compatible with the file identity setting: fingerprint identity survives renames, which is one reason to keep it.

In transit. Shippers retry with back-off when the next hop is unavailable, so do not add your own loops on top. Use TLS between every hop, with the CA certificate or fingerprint configured, never verification turned off.

In Logstash. Use a persistent queue sized for the longest outage you want to survive at your peak rate. A rough calculation is peak events per second times average event size times the seconds of outage. Enable the DLQ. Use separate pipelines so a slow destination does not block a fast one. Watch /_node/stats/pipelines for queue depth.

At Elasticsearch. Bulk requests in the region of a few megabytes (not hundreds) per request and a moderate number of concurrent requests keep throughput high without triggering rejections. Always check the errors field of a bulk response, because the HTTP status is 200 even when individual items fail.

After indexing. Alert on ingest lag. It is the end-to-end check that covers every hop above, including the ones you forgot.

Recipes

Find out why a shard is unassigned

GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason
GET _cluster/allocation/explain
POST _cluster/reroute?retry_failed=true

The first lists the reason code, such as NODE_LEFT, ALLOCATION_FAILED or INDEX_CREATED. The second explains the decision node by node. The third retries after you have fixed the cause.

Free disk in an emergency

1. Identify the largest old indices: GET _cat/indices?v&s=store.size:desc.
2. Delete what is past retention (DELETE <index> for a normal index, or delete the oldest backing index of a data stream).
3. Confirm the watermark cleared: GET _cat/allocation?v.
4. If indices are still read-only, remove the block with PUT */_settings and {"index.blocks.read_only_allow_delete": null}.
5. Afterwards, find out why retention did not do its job: GET <stream>/_ilm/explain.

Change a mapping on a live data stream

PUT _component_template/app-mappings
{ "template": { "mappings": { "properties": { "user.id": { "type": "keyword" } } } } }

POST logs-app-default/_rollover

The template change applies to the next backing index. Older backing indices keep the old mapping. If you need the old data in the new shape, reindex it into a new index, and accept that this is a planned operation rather than an edit.

Reindex with a better mapping

POST _reindex?wait_for_completion=false
{
  "source": { "index": "old-index" },
  "dest":   { "index": "new-index" }
}
GET _tasks/<task_id>

The first call returns a task ID. Poll the task. When complete, switch an alias atomically:

POST _aliases
{
  "actions": [
    { "remove": { "index": "old-index", "alias": "app" } },
    { "add":    { "index": "new-index", "alias": "app" } }
  ]
}

Rotate an API key without downtime

1. Create a new key with the same role descriptor and a new name.
2. Deploy it to the shipper (the keystore, or the secret manager behind it) and reload or restart.
3. Confirm writes succeed with the new key.
4. Invalidate the old key with DELETE /_security/api_key and {"ids": ["<old-id>"]}.

Put the expiry on the key so that rotation is a schedule, not a reaction.

Write a least-privilege reader role

PUT /_security/role/logs_reader
{
  "indices": [
    { "names": ["logs-*-production"], "privileges": ["read", "view_index_metadata"] }
  ]
}

Combine it with Kibana feature privileges in a dedicated Space, and test it by logging in as a user who has only that role.

Page through a large result safely

POST logs-app-default/_pit?keep_alive=1m

GET _search
{
  "size": 1000,
  "pit": { "id": "<pit-id>", "keep_alive": "1m" },
  "sort": [ { "@timestamp": "asc" }, { "_shard_doc": "asc" } ]
}

For each following page, pass search_after with the last hit's sort array, and remember to DELETE _pit when done.

Take and verify a snapshot

PUT _snapshot/s3repo/manual-1?wait_for_completion=true
GET _snapshot/s3repo/manual-1
POST _snapshot/s3repo/_verify

Check state: SUCCESS and that failures is empty. For an S3 repository on 9.1 or later, confirm that the client's region is set.

Start a rolling upgrade safely

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }

POST _flush

Stop and upgrade one node, start it, then reset the setting to null and wait for green before the next node. Upgrade master-eligible nodes last, and run GET _migration/deprecations before you begin.

Find slow searches after the fact

PUT logs-app-*/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.indexing.slowlog.threshold.index.warn": "5s"
}

The slow log records offenders with their source, so you can ask "which dashboard panel was it" rather than guess.

Inspect a Logstash pipeline

curl -s 'localhost:9600/_node/stats/pipelines?pretty'
bin/logstash -f /etc/logstash/conf.d/app.conf --config.test_and_exit

The first shows event counts, plugin timings and queue size. The second catches syntax errors before a restart.

Diagnose a silent agent

sudo elastic-agent status
sudo elastic-agent diagnostics
filebeat test output

status lists each component's health, diagnostics bundles logs and the rendered configuration, and test output tries the connection with TLS and credentials.

Query habits

A handful of habits make queries both faster and more correct.

Time first. Put the @timestamp range at the top of the filter list so Elasticsearch can skip shards and segments outside it.

Filters, not musts, for yes-or-no conditions. They are not scored and can be cached. Reserve must and should for where relevance matters.

size: 0 for aggregations. You avoid fetching hits you will not read.

Mind cardinality. A terms aggregation on a field with millions of distinct values is approximate and expensive. Use a composite aggregation for exhaustive listing, and cardinality for counts that can be approximate.

ES|QL needs LIMIT awareness. Results are capped at 1,000 rows by default and 10,000 at most, and partial results are allowed by default since 9.1, so check is_partial before you trust a count on a degraded cluster.

Prefer keyword fields for identifiers. Identifiers, hostnames and status strings belong in keyword. Reserve text for content people search by words.

Habits worth adopting now

• Name data streams deliberately: <type>-<dataset>-<namespace>. Dashboards, roles and templates all key on it.
• Keep one source of truth for templates and pipelines, in version control, applied through a pipeline or Terraform, never hand-edited in a console. See the Terraform guide for the workflow.
• Read the deprecation log on a schedule. It is how you learn what the next major version will break, from your own clients, while it is still cheap.
• Separate monitoring from production. Ship the stack's own metrics to a different cluster.
• Write down the upgrade order and rehearse it on a non-production cluster with real data volumes.
• Mind residency. Where an employer in the Gulf or Egypt has data-location rules, apply them to snapshot repositories and to any cloud region, not only to the nodes.
• Know your licence tier before designing around a feature. Searchable snapshots, field-level security, SAML single sign-on on self-managed clusters and audit logging are not in the free tier.
• Do not chase tuning tips. Measure first with the profile API, the slow log and node statistics, then change one thing.

A weekly checklist

1. Are any shards unassigned, and for how long?
2. What is the highest disk usage on any node, and how far is it from 85%?
3. Are write rejections or breaker trips non-zero?
4. Is ingest lag within tolerance for each important data stream?
5. Did the last snapshot succeed, and how old is it?
6. Is the Logstash queue draining, and is the DLQ empty?
7. Are any API keys within thirty days of expiry?
8. Is any component more than one patch release behind?
9. How many shards are under 1 GB?
10. Did the deprecation log show anything new?

For the ideas behind these checks, the Kubernetes, Prometheus and Grafana guides cover the neighbouring layers, and the Senior level returns to the stack as a platform other teams depend on.

When to escalate rather than tune

Not every problem is a tuning problem, and part of working well at this level is recognising the ones that are not. If _cluster/allocation/explain says no node holds a valid copy of a primary shard, no setting will bring the data back, and the conversation moves to restoring from a snapshot and deciding how much recent data you can afford to lose. If heap after garbage collection sits above roughly three quarters of the limit on every node even when the cluster is quiet, the cluster is undersized for its shard count and mappings, and the honest answer is fewer shards, fewer fields or more nodes, not another circuit breaker tweak. If a vulnerability advisory names your version and offers no workaround, as ESA-2026-182 did for 9.2.0 through 9.4.6 and 9.5.0 through 9.5.3, the fix is the patch release, scheduled with a snapshot behind it.

Keep a short written record of each incident: the symptom, the API output that identified the cause, the fix, and the alert that would have caught it earlier. After a few months that record becomes your team's runbook, and it is far more useful than a generic tuning guide because it describes your data and your failure modes.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further and adds the work you own when the Elastic Stack is not your tool but your team's product: the thing forty engineers ship logs to, the thing the on-call rotation opens at 3 a.m., and the thing finance asks about every quarter.

Up to this point the questions were about making the stack work. From here they are different. What happens to the cluster when one node loses its disk? Which component falls over first when a team doubles its log volume overnight? Who is allowed to read whose data? What does the retention policy cost, and who agreed to it? How do you move from one major version to the next without a downgrade path? This guide is written against Elastic Stack 9.5.4, released on 15 September 2026, and it treats that number as a requirement, not a detail: CVE-2026-94396 (ESA-2026-182) is a resource-consumption denial of service in Elasticsearch that affects 9.2.0 through 9.4.6 and 9.5.0 through 9.5.3, it is fixed in 9.4.7 and 9.5.4, and there is no workaround. A platform owner is the person who notices that advisory and has a tested upgrade path ready.

Where this picks up

I am starting with architecture, because every incident in this guide is an architecture question in disguise.

Architecture and failure modes

You know the nouns from Beginner: nodes, shards, primaries, replicas, segments. At this level you need the layer underneath them, which is cluster state. The cluster state is a single data structure that records which indices exist, what their mappings and settings are, and where every shard lives. Every node holds a copy, but only one node, the elected master, is allowed to change it, and it must publish each change to the other master-eligible nodes and get a majority to acknowledge it before the change counts. That majority rule is the quorum. It is the reason the documentation tells you to run three master-eligible nodes, and the reason an even number of them buys you nothing: four masters tolerate one failure, the same as three, and cost more.

The practical consequence is that the master is a coordination component, not a data component, and the things that hurt it are different from the things that hurt a data node. A master slows down when the cluster state gets large or changes too fast. Both come from the same habit: too many indices, too many fields, too many mapping updates. The sheet of numbers worth memorising is short. The default limit is 1,000 non-frozen shards per node (cluster.max_shards_per_node) and 3,000 on a frozen node, and a master needs roughly 1 GB of heap per 3,000 indices. A cluster that "just grew" to forty thousand small shards does not get slower linearly; it gets slower in how it changes, so index creation, rollover and node recovery all start to take minutes.

What a failure looks like

It helps to classify failures by what the cluster can still promise.

One data node dies. Replicas on other nodes are promoted to primaries, the cluster goes yellow while it rebuilds the missing copies elsewhere, and, provided you had at least one replica and enough disk headroom, nothing is lost. The surprise is the recovery traffic. Re-replicating a node's worth of shards saturates the network and the disks of the survivors, which is exactly when they are also carrying the extra load. Allocation throttling settings exist for this reason, and so does the rule that a cluster should never run so full that losing one node leaves the rest above their disk watermark.

One availability zone dies. If you did not configure shard allocation awareness, primaries and replicas of the same shard may sit in the same zone, and the zone failure is a red cluster. With awareness (node.attr.zone plus cluster.routing.allocation.awareness.attributes: zone) Elasticsearch spreads copies across zones, so the survivors hold at least one copy of everything. This is a design decision made on day one, because it constrains how many replicas and nodes you need per zone.

Masters lose quorum. With three master-eligible nodes across three zones, losing one zone leaves two, which is still a majority. Losing two zones leaves one, which is not. The cluster stops accepting cluster-state changes: no index creation, no mapping updates, no allocation decisions. Existing shards on surviving nodes may keep serving reads for a time, but the cluster cannot heal itself. The fix is to bring a master back; the wrong fix is to improvise a new bootstrap. cluster.initial_master_nodes is for the very first start of a brand-new cluster only, and you remove it afterwards. Re-running it against an existing cluster is how people create a second cluster with the same name and discover later that they have two histories.

A node is slow rather than dead. This is the most expensive failure, because the cluster's own health checks do not call it a failure. A node with a failing disk or a stuck garbage-collection loop still answers pings, so requests keep being routed to it, and every search that touches one of its shards is as slow as that shard. Finding it means looking at per-node numbers rather than the cluster colour: _nodes/stats, _cat/thread_pool, and the slow logs.

The resources that run out, in order

Every cluster has a characteristic order in which resources exhaust. For log and event workloads the order is usually: disk first, then heap, then CPU, then network. Disk runs out first because retention is a promise you made in days while the volume is a number you do not control. The watermarks are the cluster's own defence: at the low watermark (85% by default) no new shards are allocated to the node, at the high watermark (90%) shards are moved away, and at flood stage (95%) indices with a shard on the node get the read_only_allow_delete block, which produces the TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark error that every platform owner meets exactly once in a bad way. The block releases automatically once usage falls below the high watermark, but only if something freed the space.

Heap comes second. Elasticsearch sizes it automatically from node roles and RAM, and your job is to leave it alone except for one rule: never more than half of physical RAM and below the compressed-ordinary-object-pointer threshold, which is roughly 26 to 31 GB depending on the JVM. The rest of the RAM is not wasted: it is the operating system's file cache, which is what makes Lucene fast. A common and costly mistake is giving a 64 GB machine a 48 GB heap. Search gets slower, because the file cache shrank, and garbage collection pauses get longer, because the heap grew.

The memory defence systems are the circuit breakers. indices.breaker.total.limit defaults to 95% of real memory, the request breaker to 60% of heap, and the fielddata breaker to 40%. When a breaker trips you get a circuit_breaking_exception and an HTTP 429, which is Elasticsearch choosing to fail one request instead of letting the JVM die. Treat a rising rate of breaker trips as a capacity signal, in the same way you treat rising queue depth. Since 9.3 GET _cat/circuit_breakers gives you this per node.

Shard economics and the tiers

The shard is the unit you pay for, and understanding why is the single most valuable piece of capacity knowledge in the stack. A shard is a complete Lucene index. It holds its own segments, its own caches and its own file handles, and it costs heap whether it has a billion documents or ten. So the question is never "how many indices", it is "how many shards per node, and how big is each one".

The guidance in the documentation gives you the working range: shards of roughly 10 to 50 GB, and under about 200 million documents each. Smaller than that and you pay the fixed per-shard overhead many times over; larger and recovery, relocation and merges become long, risky operations, because moving a 300 GB shard across the network after a node failure takes hours during which you have reduced redundancy. The limit of 1,000 non-frozen shards per node is a guard rail, not a target. A cluster at 900 shards per node is not "within limits", it is about to be unable to create tomorrow's indices, and the error is this action would add [N] shards, but this cluster currently has [X]/[1000] maximum normal shards open.

Why oversharding happens

It almost never comes from a decision. It comes from defaults that were reasonable for one team and became a pattern for forty: a daily index per service, each with several primaries, each with a replica, each holding thirty megabytes. Multiply that by a hundred services and ninety days of retention and you have tens of thousands of shards holding a few hundred gigabytes. The fix is structural. Use data streams with rollover on size, not on a calendar, so that a low-volume service produces a shard every few weeks while a busy one produces several a day. Set rollover with max_primary_shard_size (the older max_size condition was deprecated in 9.3), and keep max_age as a safety valve so that a quiet stream still rolls eventually and ages out cleanly.

PUT _ilm/policy/platform-logs
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}

Two details in that policy are worth saying out loud. The shrink action reduces the primary count once indexing has stopped, which is how a shard that was split for write throughput becomes a single efficient shard for reading. And forcemerge is safe only on indices that are no longer written to, which is why it lives in the warm phase and never in hot. Force-merging an index that is still receiving writes creates enormous segments that the normal merge policy then cannot reclaim.

Tiers and the frozen tier

Data tiers turn retention into a cost curve. Recent data is on the hot tier with fast storage and replicas, because it is indexed and searched constantly. Older data moves to warm and cold nodes with cheaper disks and often fewer replicas, because it is searched rarely. The frozen tier goes further: indices are backed by searchable snapshots in object storage, and only the parts of a shard that a query actually touches are pulled into a local cache. That makes very long retention affordable, at the price of slower queries. Frozen tier and searchable snapshots need an Enterprise licence, which is a fact a platform owner must put in the cost model before promising a year of searchable logs. Two version facts matter here. Frozen indices, the older mechanism, were removed in 9.0, so anything you read about _freeze is obsolete. And in 9.5 the data stream lifecycle gained support for the frozen tier in tech preview, which is worth watching but not yet worth standardising on.

Index modes that change the economics

Since 9.0, data streams that match logs-*-* use LogsDB index mode by default when the conditions in the mapping documentation are met. LogsDB stores logs with synthetic _source, reconstructing the original document from doc values instead of storing it twice, and sorts segments in a way that compresses log data better. For metrics, the time series mode (TSDB) uses dimensions and metric fields, and in 9.5 its default doc-values codec (ES95) makes doc values roughly 20 to 30 percent smaller. A platform owner should know the behavioural trade-offs as well as the savings:

• Synthetic source means the returned _source is a reconstruction, so field order and some formatting (for example how arrays are normalised) can differ from what was sent. Teams that store the raw original message should keep it in an explicit field.
• TSDB indices stopped tracking sequence numbers in 9.4. if_seq_no conditional updates therefore fail, and update_by_query and delete_by_query no longer detect conflicts. Setting index.disable_sequence_numbers: false restores the old behaviour.
• TSDB's index.look_ahead_time default fell from 30 minutes to 9 minutes in 9.5, so a document timestamped more than 9 minutes in the future is rejected. Sources with clock skew will start failing on upgrade, and the fix is either to correct the clock or to set the value in a template.
• The columnar and logsdb_columnar modes in 9.5 are tech preview. Test them; do not build the platform default on them.

Ingest at scale

Ingest is where platform owners spend most of their operational time, because it is the part of the system where other teams' behaviour arrives as load. The question to ask of any ingest design is: what happens when Elasticsearch is slow, down, or rejecting writes, and how long can the design absorb that before data is lost?

The simplest design is Elastic Agent or Filebeat writing straight to Elasticsearch. It works, and for many teams it is the right answer, but its buffer is the shipper's local disk and registry. A shipper that cannot reach Elasticsearch keeps reading until the source log files rotate away. A shipper that gets HTTP 429 back backs off and retries, which is correct behaviour, but it means a cluster that is rejecting writes pushes the problem out to every host in the fleet at once.

The design that scales, and the one worth standardising for shared platforms, puts a buffer between producers and Elasticsearch. The common shape is Elastic Agent or Beats to Kafka, Kafka to Logstash, Logstash to Elasticsearch. Kafka is durable, replayable and sized in days of retention, so an Elasticsearch outage becomes a consumer lag graph rather than a data-loss incident. Logstash is then free to be the transformation tier it is good at. If running Kafka is more than you want to own, Logstash's persistent queue gives you a smaller buffer on local disk with at-least-once semantics, sized in gigabytes instead of days.

Backpressure, as Elasticsearch experiences it

Elasticsearch protects itself with indexing pressure. The indexing_pressure.memory.limit setting defaults to 10 percent of the heap and caps the bytes of in-flight indexing requests across the coordinating, primary and replica stages. When it is exceeded, bulk requests are rejected with es_rejected_execution_exception and HTTP 429, and the message tells you which stage rejected them. This is the system working as designed. The design failure is on the client side: a client that responds to 429 by immediately retrying at full speed turns a brief rejection into sustained overload. Clients must back off exponentially, and bulk batch sizes should be in the range of a few megabytes, not hundreds.

A platform owner should therefore publish an ingest contract to teams: which endpoints to write to, the maximum bulk size, the required retry behaviour, and the fact that 429 means "slow down", not "failure". Logstash's elasticsearch output and the Beats and Agent outputs already implement backoff; custom clients written by application teams frequently do not.

Design for the traffic you did not plan

Volume spikes are the normal case. A deploy that changes a log level from info to debug can multiply a service's volume by ten in a minute. There are four protections, in increasing order of effort.

1. Per-tenant data streams, so that a noisy service consumes its own shards and its own lifecycle, and the platform can see whose volume it is.
2. Drop and sample at the edge, in the shipper or an ingest pipeline: discard health-check lines, drop fields nobody queries, sample successful requests. Every byte not sent is a byte not paid for on hot storage.
3. Quotas outside Elasticsearch, in Kafka or in a gateway, because Elasticsearch has no native per-tenant ingest quota. If a tenant must be throttled, it is throttled before the cluster.
4. Capacity headroom in the hot tier that is sized for the peak, not the average. A common rule of thumb is to run hot nodes well below the disk and CPU levels at which losing one node would push the rest over their limits.

Logstash as a platform component

Logstash is the part of the stack most likely to be owned by the platform team and misunderstood by everyone else. It is a JVM pipeline with three stages (input, filter, output), a set of worker threads, and an in-memory queue between the input and the workers. By default that queue is in memory, which means a Logstash crash loses whatever was in flight. The platform decision is whether to accept that, and most shared platforms should not.

Persistent queue and delivery guarantees

Setting queue.type: persisted in logstash.yml moves the queue to disk, so events survive a restart and Logstash acknowledges an input only after the event is safely queued. The delivery guarantee you get is at least once: after a crash some events may be delivered twice, which is why idempotent document IDs matter for anything you cannot tolerate duplicated. Three settings govern it: queue.max_bytes (default 1024mb, which is small for a platform), queue.page_capacity (64mb) and, when a queue fills, the backpressure that propagates to the inputs. A full persistent queue does not lose data; it stops accepting more, and the sender has to cope. That is acceptable for Beats, which retry, and unacceptable for a UDP syslog input, which cannot.

pipeline.workers: 8
pipeline.batch.size: 250
pipeline.batch.delay: 50
queue.type: persisted
queue.max_bytes: 20gb
dead_letter_queue.enable: true
api.http.host: 127.0.0.1
allow_superuser: false

The values above are an illustration of which knobs a platform owner touches, not a recommendation: the defaults are pipeline.workers equal to the CPU core count, pipeline.batch.size 125 and pipeline.batch.delay 50 ms. Tune by measurement, using the pipeline statistics API, because the right batch size depends on event size and on what the filters cost.

curl -s 'localhost:9600/_node/stats/pipelines?pretty'
curl -s 'localhost:9600/_node/hot_threads?human=true'

How to read it: in the pipeline stats, compare per-plugin duration_in_millis against events.in to find the slow filter, look at queue sizes to see whether the persistent queue is growing, and use hot threads to see whether workers are busy in a regular expression. A grok with a pattern that backtracks heavily is the classic cause of a Logstash node pinned at full CPU while throughput collapses.

Isolation with multiple pipelines

A single giant pipeline with conditionals per team is a shared failure domain: one slow output blocks everyone. pipelines.yml lets you run several independent pipelines in one process, each with its own workers and queue, and pipeline-to-pipeline communication (a pipeline output feeding a pipeline input) gives you four well-known patterns. The distributor pattern has one input pipeline route events to per-team pipelines. The output isolator pattern gives each destination its own pipeline so that one failing output does not stall the others. The forked path pattern sends a copy to a second destination. The collector pattern merges several inputs into one pipeline. For a platform, the distributor plus output-isolator combination is the one that pays for itself: a team's broken grok pattern stops at its pipeline's boundary.

- pipeline.id: intake
  path.config: "/etc/logstash/conf.d/intake.conf"
  queue.type: persisted
- pipeline.id: team-payments
  path.config: "/etc/logstash/conf.d/team-payments.conf"
  queue.type: persisted
- pipeline.id: team-search
  path.config: "/etc/logstash/conf.d/team-search.conf"
  queue.type: persisted

input  { beats { port => 5044 ssl_enabled => true } }
output {
  if [fields][team] == "payments" { pipeline { send_to => ["payments"] } }
  else if [fields][team] == "search" { pipeline { send_to => ["search"] } }
  else { pipeline { send_to => ["unrouted"] } }
}

Note the ssl_enabled name: in Logstash 9 the legacy ssl option names are gone, and the same applies to the Elasticsearch output, where cacert, keystore, ssl and their relatives were replaced by ssl_certificate_authorities, ssl_keystore_path, ssl_enabled and so on. With the elasticsearch output plugin at v12, leftover legacy options do not produce a warning: Logstash fails to start. That is good news for a pipeline review, because the failure is loud, and bad news for an unplanned upgrade, which is why the next section's upgrade rehearsal exists.

What changed in Logstash 9 that bites platforms

A short list that belongs in every platform runbook:

• allow_superuser defaults to false. A container or unit file running Logstash as root fails on start. Run as the logstash user.
• Minimum JDK moved to 17 in 9.0, and to 21 in 9.4. The bundled JDK is the supported path; overriding it with an old system JVM will stop working.
• The http.* API settings were replaced by api.*, and the modules framework was removed.
• pipeline.buffer.type defaults to heap.
• The 9.5 Kafka integration moved to version 12.x of the plugin. The default and uniform_sticky partitioner values were removed and the linger_ms default changed from 0 to 5 ms, which adds up to 5 ms of latency per batch in exchange for better batching. Output concurrency :legacy was removed, and logstash-output-pipe is no longer bundled.
• The Docker base image is UBI9, so apt in a derived image becomes microdnf.

The operational rule that follows: pin Logstash, its plugins and its pipeline configuration together in version control, and test a pipeline with bin/logstash --config.test_and_exit -f pipeline.conf (note the double dash) in CI before any deploy.

Fleet and Elastic Agent as fleet management

For new deployments Elastic recommends Elastic Agent managed by Fleet, and at platform scale this changes what you operate. Policies live in Kibana; agents check in with Fleet Server on port 8220; integrations bring their own ingest pipelines, mappings and dashboards. The upside is that the platform team changes a policy once and thousands of hosts converge. The downside is that Fleet is now in the critical path of change management, and its failure modes are different from those of the data path.

Fleet Server is a stateless service that you scale horizontally and put behind a load balancer; the agents keep collecting and shipping data if it is unavailable, but they cannot receive new policy. The Kibana encryption keys matter here in a way people discover late: Fleet output and integration secrets and alerting connectors are stored as encrypted saved objects, so xpack.encryptedSavedObjects.encryptionKey must be identical on every Kibana instance and must be backed up. A replacement Kibana started with a new key cannot read what the old one wrote. Fleet also supports output and integration secrets stored in the .fleet-secrets system index, which keeps credentials out of the policy document that agents download in readable form.

Two habits make a Fleet platform tractable. First, treat policies as code: the Kibana Fleet APIs let you manage agent policies and integration settings from version control, and the 9.x API cleanups (hyphenated routes such as enrollment-api-keys became enrollment_api_keys, and service-tokens became service_tokens) mean older scripts must be updated. Second, stage agent upgrades: upgrade Fleet Server first, then agents in rings (a canary group, then a slice of each environment, then the rest), because an agent must not be newer than the Elasticsearch it writes to. The installation itself should be unprivileged where the integration allows it, using elastic-agent install ... --unprivileged, because an agent that runs as root on every host is a large standing attack surface.

sudo ./elastic-agent install \
  --url=https://fleet.example.internal:8220 \
  --enrollment-token="$ENROLLMENT_TOKEN" \
  --certificate-authorities=/etc/pki/elastic/ca.crt \
  --unprivileged \
  --non-interactive
sudo elastic-agent status

An enrollment token is itself a credential that lets any holder join an agent to a policy, so scope each token to one policy and revoke it when the rollout ends. The standalone (non-Fleet) mode remains valid where hosts are locked down and policy changes go through configuration management, and is simpler to reason about in air-gapped environments, including regulated estates in the Gulf where the collection tier must not call out to anything.

The trust model and security

Security is on by default since 8.0: TLS on both the HTTP and transport layers, and authentication required for every request. A platform owner's job is to keep that true at scale, which means owning four things: who is trusted to issue certificates, who holds credentials to write and read, how those credentials are rotated, and how secrets are stored.

Certificates and the CA

On first start a node auto-configures its own CA and certificates, which is ideal for learning and acceptable for a single node. A platform running three masters, a tier of data nodes, Kibana, Logstash and agents should instead use a CA you own, generated once with elasticsearch-certutil, stored offline or in a secrets system, and used to sign node, HTTP and client certificates.

bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --out node-certs.zip
bin/elasticsearch-certutil http

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/node.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12

The passwords for those keystores belong in the Elasticsearch keystore, not in the YAML: bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password. The trust-model point is that the CA private key is the root of everything. Whoever holds it can mint a node certificate that joins the cluster, so its storage and access deserve the treatment you give the root of your internal PKI. Plan certificate expiry as a scheduled operation with an alert weeks ahead, because an expired transport certificate partitions the cluster in a way that looks exactly like a network fault.

On the client side, verify the server. Shippers should set ssl.certificate_authorities or ssl.ca_trusted_fingerprint; Logstash should set ssl_certificate_authorities; and verification_mode: none or curl's -k should appear in no production configuration. The 9.5.2 change in Kibana, which now validates proxy TLS certificates for the email connector (with xpack.actions.ssl.proxyVerificationMode: none as a documented but discouraged escape hatch), is an example of a release turning previously lax configuration into an error.

Credentials: API keys, service accounts, and blast radius

Humans authenticate through a realm: native, LDAP, Active Directory, PKI, SAML, OIDC, Kerberos or JWT. SAML and OIDC single sign-on on self-managed clusters are paid-tier features, so a platform that promises SSO must check the licence. Machines should not use users at all.

• API keys are for shippers and applications. They are created with role descriptors that restrict them, they can carry an expiration, and they can be invalidated individually.
• Service accounts and service tokens are for Elastic components, notably Kibana (elastic/kibana) and Fleet Server. Kibana should authenticate with elasticsearch.serviceAccountToken, not with a password, and never as the elastic superuser: Kibana refuses elasticsearch.username: elastic.

The design rule is one credential per producer, scoped to the smallest index pattern that works. A log writer needs create_doc and auto_configure on its own data-stream pattern and the monitor cluster privilege, nothing more:

POST /_security/api_key
{
  "name": "payments-shipper",
  "expiration": "90d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [
        { "names": ["logs-*-payments"], "privileges": ["auto_configure", "create_doc"] }
      ]
    }
  },
  "metadata": { "team": "payments", "ticket": "PLAT-1234" }
}

The metadata field is underrated: an API key with an owner and a ticket is one you can find and retire later; an anonymous key is a permanent liability. When you list keys (GET /_security/api_key), the ones you cannot attribute are the ones to investigate. The blast radius of a leaked key is exactly its role descriptor, which is the whole argument for narrow ones. Remember a format detail that costs people an afternoon: Beats and the Logstash output take the key as id:api_key, whereas the HTTP header takes the base64 encoded value from the creation response.

Rotation is an operation, not an event. Because keys carry an expiry, rotation means issuing a new key, rolling it to the producer, confirming writes succeed, and then invalidating the old key with DELETE /_security/api_key. Build that as a script or a pipeline, and alert on keys that will expire within two weeks so that the first sign of a forgotten key is a ticket rather than an outage.

Secrets handling

Every component has a keystore: elasticsearch-keystore, kibana-keystore and logstash-keystore. Values in them are referenced without appearing in configuration files, and environment-variable interpolation works in all three. The Elasticsearch keystore's secure settings can be reloaded without a restart with POST _nodes/reload_secure_settings, which matters for credentials like the S3 snapshot keys. A platform should never commit a secret to the configuration repository, including a Helm values file, and a container image should never contain one.

What you do not expose

Port 9200 is never on the internet, and the transport port 9300 is reachable only between nodes. Put Kibana behind your existing identity-aware proxy or VPN, and keep Logstash's API (api.http.host, default 127.0.0.1) on loopback or protect it with api.auth.type. Audit logging (xpack.security.audit.enabled: true) is a paid-tier feature that records authentication and access events; if you are subject to a regulator, it is often the feature that decides the licence tier. For data residency, which is a practical question for Gulf and Egyptian employers, the question to answer in writing is where the snapshots live (an object-storage region in the required jurisdiction) and whether any component, including telemetry and hosted diagnostics such as AutoOps, sends cluster metadata outside it.

Multi-tenancy

"Multi-tenancy" on this stack is not one feature. It is a spectrum, and the platform owner's job is to pick the point on it that matches how much the tenants trust each other, then say so in writing.

Namespaced data streams plus roles is the workhorse. The data stream naming convention is <type>-<dataset>-<namespace>, and using the namespace for the tenant (or the dataset for the service) gives you a pattern to attach privileges to. A tenant's read role says logs-*-payments with read and view_index_metadata, and their shipper key says the same with create_doc. The discipline is that the naming convention is the security boundary, so the platform must enforce it: if a team can write to an arbitrary index name, the role pattern is decoration. Enforce it by giving shippers keys that can only create matching names, and by rejecting anything else at the intake tier.

Kibana Spaces separate saved objects (dashboards, data views, rules) and let you hide features per role. A Space is a convenience boundary for what people see in Kibana; it is not a data boundary by itself, because a user who can read an index can read it through Dev Tools regardless of Space. The security pairing is Space plus an index privilege plus a feature privilege, all three.

Document-level and field-level security let tenants share an index while seeing different rows or columns. They are paid-tier features, and they carry a performance cost because the query is rewritten per request. Use them when sharing is unavoidable (a central security index with per-business-unit views) and avoid them as a substitute for index design.

Separate clusters are the answer when tenants are mutually distrusting, regulated differently, or differently sized by an order of magnitude, and when one tenant's incident must not be able to reach another. Cross-cluster search gives a central team a single query across them, and cross-cluster replication (paid) copies data between regions. Remote clusters connect with the API-key security model on port 9443 or the older certificate model on the transport port; 9.3 disabled cross-cluster search for the _fleet/_fleet_search and _fleet/_fleet_msearch endpoints, which is worth knowing if you built Fleet-related tooling on it.

Noisy-neighbour control is the part with no feature to lean on. Elasticsearch has no per-tenant CPU quota, so you protect the cluster with guard rails that apply to everyone: search.max_buckets (65,536 by default), index.max_result_window (10,000), the per-index field limit (index.mapping.total_fields.limit, 1,000), the depth and nested-field limits, and request timeouts. Slow logs per tenant index, enabled with index.search.slowlog.threshold.query.warn, tell you who is running the expensive queries. A tenant onboarding checklist that sets these in an index template, so that tenants inherit sane limits, is worth more than any amount of after-the-fact policing.

Mapping explosion and the field budget

The tenant-specific failure worth a paragraph of its own is mapping explosion. Dynamic mapping adds a field for every new key it sees, so a service that logs user-supplied keys, for example a map from customer ID to value, creates thousands of fields. Every field is cluster state, and the error is Limit of total fields [1000] has been exceeded. Raising the limit treats the symptom. The fixes are structural: set dynamic: false or strict where the schema is known, map free-form maps as flattened, use index.mapping.total_fields.ignore_dynamic_beyond_limit: true so that surplus fields are dropped and kept in _source rather than failing the write, and push teams toward ECS field names so that the same thing has the same name everywhere.

Cost

Storage dominates the bill of a log platform, and compute follows storage, because data you keep must be indexed, replicated, merged and searched on nodes that have to exist. So the cost conversation is a retention-and-volume conversation, and the platform owner's leverage comes from five levers, listed here from the cheapest change to the most invasive.

1. Drop what nobody reads. Health-check lines, repeated debug messages, unused fields. Doing it in the shipper or an ingest pipeline means the cost is never incurred.
2. Right-size retention per data class. Thirty days of application logs and a year of audit logs is a different bill from a year of everything.
3. Use the efficient index modes. LogsDB with synthetic source for logs; TSDB with downsampling for metrics. Downsampling in ILM no longer force-merges by default since 9.4, so set force_merge_index: true or add a forcemerge action deliberately.
4. Move cold data to cheaper tiers, with fewer replicas on warm and cold, and to the frozen tier with searchable snapshots where the Enterprise licence allows it.
5. Change the tool for data that does not need an index: archive raw logs in object storage and query them on demand.

The mistake to avoid is optimising a lever the workload does not respond to. Before choosing, measure: the per-stream size from _cat/indices, the ratio of hot to total storage, and, crucially, how often each dataset is actually queried. Data that has never been searched in ninety days is a retention-policy candidate, and you can prove it from the search slow logs and Kibana usage rather than guessing.

Replicas deserve a specific note, because they double the storage of a dataset. One replica is the standard for resilience on hot data; a warm tier that is also protected by snapshots can often run with none or with auto_expand_replicas, but note the 9.5.1 change where an ILM allocate action that sets number_of_replicas now also removes auto_expand_replicas. Decide what you are protecting against (node loss versus region loss) before removing copies, and rehearse the restore so the answer is backed by evidence.

Licensing, and showing the bill

The default distribution is under the Elastic License 2.0, and since September 2024 the source has also been available under AGPLv3 and SSPL 1.0, at the user's choice. The Basic tier, which is free, includes security (TLS, RBAC, API keys), ES|QL, Kibana, Fleet and vector search. Paid tiers add, among other things, machine learning anomaly detection, SAML and OIDC for self-managed clusters, field and document-level security, cross-cluster replication and searchable snapshots; check the subscriptions page for the current per-feature mapping before you commit, because a platform that depends on a paid feature and discovers it late has a budget problem, not a technical one. Elastic Cloud Hosted is priced by deployment resources, and Serverless by ingest and retained volume for Observability and Security or by compute units for Search, which I have not verified against the pricing page and which you should confirm before building a model.

Whatever the billing unit, show teams their own number. A monthly report of each tenant's hot and total storage, its ingest volume and its most expensive data streams, produced from _cat/indices and the data stream stats, converts cost from a platform complaint into a tenant decision. Teams cut their own debug logging fastest when they can see what it costs.

Backups, snapshots and disaster recovery

Replicas protect you from losing a node. They do not protect you from deleting an index, from a bad template, from a corrupt write, or from losing a site. Snapshots do, and only if you have practised restoring them. A snapshot is an incremental copy of indices (and optionally cluster state) into a repository: a shared filesystem, S3, Google Cloud Storage or Azure Blob Storage. The first snapshot is full and later ones copy only new segments, so a nightly snapshot is cheap after the first.

PUT _snapshot/platform-s3
{
  "type": "s3",
  "settings": { "bucket": "es-platform-snapshots", "client": "default" }
}

PUT _slm/policy/nightly
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "platform-s3",
  "config": { "indices": "*", "include_global_state": true },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

Since 9.1 the S3 repository uses the AWS SDK v2, so the region must be set explicitly with s3.client.default.region in elasticsearch.yml, IMDSv1 is not supported, V4 signatures are required, and the s3.client.<name>.protocol setting no longer has any effect. The credentials belong in the keystore (s3.client.default.access_key and s3.client.default.secret_key), reloaded with POST _nodes/reload_secure_settings. On an upgrade from 8.x that used S3 snapshots, this is a silent-until-the-next-snapshot breakage; include a manual snapshot in your upgrade rehearsal.

Three points decide whether backups are real:

• Test the restore. Schedule a quarterly drill that restores a snapshot into a scratch cluster and runs a known query. A backup that has never been restored is a hope.
• Decide what a snapshot does not contain. Snapshots do not contain your Kibana encryption keys, your keystores, your elasticsearch.yml, or your certificates. Those live in your secrets manager and configuration repository.
• Know your recovery objectives. Hourly SLM gives you a one-hour recovery point at most; restore time depends on data volume and on how fast the repository and network are, and is the number people guess wrongly. Measure it.

For regional disaster recovery, two models exist. The first, cheaper and slower, is cross-region snapshot copy and restore into a standby. The second, faster and paid, is cross-cluster replication from a leader in one region to a follower in another. Choose by the cost of an outage, not by preference, and remember that restoring into a cluster means that cluster must already be on a compatible version: you cannot restore a snapshot into an older version, which is one more reason downgrade is not a plan.

Upgrades and migrations

Upgrades are where the platform's credibility is won or lost, and 9.x is a real test. The documented path is fixed: you must be on 8.19.x before moving to 9.x, so the route from 7.17 is 7.17 to 8.19.x to 9.5.4. 7.17 reached the end of support on 15 January 2026, so any cluster still there is out of support today, and 8.x reaches end of maintenance on 15 January 2027 and end of support on 15 July 2027. Those dates turn an "upgrade sometime" item into a deadline.

The preparation checklist

1. Read the breaking changes for every version you skip, not just the target. Elasticsearch, Kibana, Logstash and Beats each publish their own page.
2. Run the Upgrade Assistant in Kibana and GET _migration/deprecations, and read the deprecation logs (the logger keyword changed in 9.0 from deprecation.elasticsearch to elasticsearch.deprecation, so any log filters on the old name went blind).
3. Deal with old indices. Indices created in 7.x must be reindexed, deleted or marked read-only before the 9.x upgrade. Frozen indices from older releases are unreadable in 9.0 and have to be handled first.
4. Snapshot, because there is no downgrade.
5. Rehearse on a copy of production data with production-like mappings and pipelines. The silent breakages are the ones a rehearsal finds.
6. Check the components on Kubernetes: ECK must be 3.x for Stack 9.x, and ECE 4.x.

What breaks silently

The breaking-changes list is long, but the items that produce a wrong behaviour rather than an error are the ones to rehearse for:

• Filestream fingerprinting. Filebeat's default file_identity changed to fingerprint, so files under 1,024 bytes are not ingested until they grow. Small, slow files look as if the shipper is broken.
• Duplicate input IDs. Filebeat 9 refuses to start when two inputs share an ID, and the deprecated log and container inputs need allow_deprecated_use: true.
• ES|QL partial results. Since 9.1 ES|QL returns partial results by default; anything that assumes a query either succeeds entirely or fails must check is_partial or set allow_partial_results=false.
• Timeouts as 429. Elasticsearch 9.0 returns 429 instead of 5xx for timeouts, so alerts keyed on 5xx stop firing.
• TSDB look-ahead and sequence numbers, described earlier.
• Kibana internal APIs. Calls to internal Kibana endpoints from outside now return 400, and the old /api/security/v1/* and /api/alerts/* routes are gone.
• Legacy SSL options in Logstash plugins, which stop startup.

The rolling upgrade

The order is Elasticsearch first, then Kibana, then Fleet Server, then agents, Logstash and Beats, with ingest components never newer than Elasticsearch. Within Elasticsearch, upgrade nodes tier by tier: frozen, cold, warm, hot data nodes, then other non-master nodes, and master-eligible nodes last. The per-node sequence follows the documentation:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }

POST _flush

Then stop the node, upgrade it, start it, and re-enable allocation and wait for green before moving on:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }

GET _cluster/health?wait_for_status=green&timeout=10m

A mixed-version cluster is valid only during the upgrade, never as a resting state, and more than two versions in one cluster is not supported. Automate the sequence, but keep a human at the wait-for-green step for the first hundred upgrades, because an automation that proceeds on yellow will eventually proceed on a real problem.

Migrations as a rule set

Moving data between clusters or restructuring indices uses the same few tools. Reindex (POST _reindex, with wait_for_completion=false for anything large) copies documents into a new index with a new mapping. Aliases make the swap atomic: POST _aliases with add and remove actions in one request moves consumers from the old index to the new without a gap. Shrink, split and clone change a shard count without reindexing when the constraints allow it. For moving to a new cluster, reindex-from-remote or a snapshot restore are the options; the trade-off is that reindex gives you a chance to fix the mapping on the way, while restore is faster and keeps the data as it was. Always plan the rollback: for a mapping migration behind an alias, rollback is moving the alias back, provided you kept writing to the old index or can replay.

Monitoring the stack itself

A log platform that cannot tell you it is unhealthy is the worst kind of outage, because the evidence you would use to diagnose it lives inside it. Two rules follow. Monitor the stack from outside itself, and alert on symptoms.

Send Stack Monitoring data to a separate monitoring cluster, collected by Elastic Agent or Metricbeat with the Elasticsearch, Kibana and Logstash integrations. The legacy internal collection was deprecated in 9.5 and is slated for removal in 10.0, and the legacy monitoring REST APIs were deprecated in 9.5.3, so a platform built on them has a migration to schedule now. AutoOps, Elastic's hosted diagnostics service that is also available for self-managed clusters through connected clusters, is an alternative for teams that accept the data leaving the estate; the residency question above decides that.

The alerts worth having are few:

• Disk headroom per tier, alerting well before the 85% low watermark.
• Write rejections (es_rejected_execution_exception) and circuit breaker trips, both rising.
• Shards per node against the 1,000 limit, and pending cluster tasks.
• Ingest lag: the gap between event time and @timestamp at index time, or Kafka consumer lag.
• DLQ growth and persistent queue depth in Logstash.
• Certificate and API key expiry, weeks ahead.
• Snapshot failures and the age of the last successful snapshot.

Slow logs (index.search.slowlog.threshold.query.warn, index.indexing.slowlog.threshold.index.warn) are the tool for finding a single bad query or document stream, and GET _tasks?detailed&actions=*search* shows running searches that you can cancel with POST _tasks/<task_id>/_cancel.

Governance

Governance is what makes the platform scale beyond the people who built it. It is unglamorous: a small set of written rules, enforced by templates and review rather than by memory.

A data catalogue with owners. Every data stream namespace has an owning team, a data classification, a retention, and a contact. Without an owner, "can we delete this" has no answer and the data stays forever.

A naming and field standard. Data streams follow <type>-<dataset>-<namespace>, and fields follow ECS where an ECS name exists (@timestamp, host.name, log.level, source.ip). Elastic is converging ECS with OpenTelemetry semantic conventions, so the standard also prepares you for OTLP ingest, which Elasticsearch now accepts natively, with Elastic Distributions of OpenTelemetry (EDOT) as the collector and SDK option.

Change control for the cluster's own configuration. Templates, ILM policies, roles, Fleet policies and Kibana saved objects live in version control and are applied by a pipeline. Drift, where someone edits a policy in the UI, is detected by comparing the live state to the repository on a schedule.

Data classification and retention rules. Personal data in logs is a liability. Decide what must never be logged, scrub at the edge with processors, and state for each class how long it may be retained. Retention policy is also a legal question in Gulf and Egyptian jurisdictions with data-protection laws, so involve the people who know it.

An exception process. Teams will need an index with a different retention or a higher field limit. Make the request a pull request against the template repository, reviewed by the platform team, and record the reason.

Incident playbooks

Each of these is the shape of a real incident. The format is: how it presents, what to check first, the fix, and the prevention.

Cluster is red or yellow after a node loss

Presents as a health change and unassigned shards. Check GET _cluster/allocation/explain for the first unassigned shard, which names the decider that refused it: disk watermark, awareness, a filter or a shard-count limit. GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason lists them. Fix what the explanation names. If the reason is a failed allocation limit, POST _cluster/reroute?retry_failed=true retries after you fix the cause. Prevent with capacity that survives one node lost, and with awareness settings.

Disk flood stage, writes blocked

Presents as cluster_block_exception with TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark. Check GET _cat/allocation?v for the full nodes. Fix by freeing space: delete or move the oldest indices, not by raising the watermark. The block releases on its own once usage falls below the high watermark; clear it manually with PUT */_settings {"index.blocks.read_only_allow_delete": null} only after space is freed. Prevent with disk alerts at 75 to 80 percent and a retention policy that matches volume.

Heap pressure and circuit breakers

Presents as 429 responses with circuit_breaking_exception, long GC pauses, and nodes leaving and rejoining. Check GET _nodes/stats/jvm,breaker and GET _cat/circuit_breakers (9.3 and later), then GET _tasks?detailed&actions=*search* for the expensive query. Fix by cancelling the offending task, then addressing the cause: an aggregation with an enormous terms size, fielddata on a text field, or too many shards on the node. Prevent with the request guard rails and keyword fields for aggregation.

Indexing rejections

Presents as es_rejected_execution_exception in client logs. Check GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected and the indexing pressure stats. Fix by reducing the producers' bulk size, enforcing backoff, and adding hot capacity if the load is real. Prevent with the ingest contract and a buffer tier.

Mapping explosion or a poison field

Presents as Limit of total fields [1000] has been exceeded, or mapper_parsing_exception when a field's type changes (a status that used to be a number and is now a string). Check the DLQ and the failing document. Fix with an explicit mapping, ignore_malformed, or an ingest convert processor. Prevent with templates that set dynamic behaviour, and a rule that teams do not reuse a field name with a different type.

Split or lost quorum

Presents as master not discovered yet in the logs and a cluster that cannot change. Check which master-eligible nodes are alive and whether they can reach each other on the transport port, then check certificates, because an expired transport certificate looks identical. Fix by restoring the missing masters. Do not re-bootstrap with cluster.initial_master_nodes on a cluster that has existed. Prevent with three masters in three zones and certificate-expiry alerts.

Ingest silence

Presents as a dashboard with a flat line. Check from the source towards the store: is the shipper running (sudo elastic-agent status, filebeat test output), is the buffer growing, is Logstash's persistent queue filling, is Elasticsearch rejecting. The usual causes are a rotated credential, an expired certificate, or an upgrade that changed a default (small files not ingested after the fingerprint change). Prevent with an end-to-end synthetic event written every minute and alerted on when missing, which is the only test that covers every hop.

Where the Elastic Stack stops

A platform owner's maturity shows in knowing when not to use the tool. The stack is excellent at search over semi-structured text and at analytics over time-ordered events, but several jobs belong elsewhere.

• Metrics at very high cardinality, with PromQL-native tooling. Elasticsearch's time series mode and native PromQL support (GA in 9.5) narrow the gap, but a team that lives in Prometheus and Grafana, and whose need is alerting on counters, may be better served by a metrics-native system, with Elasticsearch kept for logs.
• Distributed tracing as the primary workflow. The stack ingests OpenTelemetry natively, but a team that wants only traces and has no other Elastic use may prefer a tracing-specific backend.
• A system of record. Elasticsearch is a search engine with near-real-time visibility and no multi-document transactions. Keep authoritative data in a database and index a copy.
• Cheap, rarely queried archives. Raw logs kept for compliance and read once a year belong in object storage with a query-on-demand tool. The frozen tier helps, and needs a paid licence.
• Heavy relational joins and transactional updates. Use a database.
• A small team with small volume. A hosted log service, or Elastic Cloud, may cost less than an engineer's time. Self-managing a cluster is an opportunity cost.
• Licence constraints. If your organisation's policy excludes Elastic License 2.0 and you cannot use the AGPLv3 option, that settles it before any technical comparison.

Comparing alternatives honestly also means acknowledging where they are better. Loki-style systems index labels rather than content, making them cheaper to run and weaker at free-text search. OpenSearch is the open-source fork with its own roadmap, and a migration between it and Elasticsearch is a real project, not a config change, because APIs and features have diverged. The fact-based way to choose is to write down the query patterns (free-text, aggregations, point lookups), the retention, the volume and the skills on the team, and let those pick the tool.

Running the stack as a platform for other teams

Everything above concerns the cluster. This section concerns the product: how other teams experience it.

Offer a paved road. A team should get logs into the platform with one snippet: an Agent policy or a documented Filebeat/Logstash configuration with the right endpoint, credentials from a provisioning flow, required fields, and sane defaults. The paved road is cheaper than the alternative, which is forty bespoke configurations that you inherit.

Self-service onboarding. Onboarding a service creates, from one request, a data stream namespace, an ILM or lifecycle policy, an API key, a role for readers, a Kibana Space and a starter dashboard. Automate it, through the Elastic APIs or a Terraform provider (both the elastic/ec and elastic/elasticstack providers exist, which I have not verified in detail), so the platform team is not a ticket queue.

Define the service level in writing. What is the ingest-to-searchable latency you promise (the default refresh interval is one second, but your pipeline adds more), what retention per class, what availability, and what support hours. A platform without a stated service level is judged on its worst day.

Publish the contract. The ingest contract from earlier, the field standard, the limits (field counts, bulk size, query timeouts) and the cost model belong on one page that every tenant can find.

Run a deprecation and upgrade calendar. Tenants need notice: end of maintenance dates, the removal of legacy inputs, the Filebeat log input replaced by filestream, legacy monitoring leaving in 10.0. The platform team that announces deprecations a quarter ahead is the one whose upgrades are welcome.

Own the on-call boundary. Decide what the platform team responds to (cluster health, ingest pipeline, Kibana availability) and what tenants own (their mappings, their dashboards, their query cost). Write it down, and review it after every incident.

Measure the platform as a product. Track ingest success rate, search latency percentiles at the tenant level, time to onboard a new service, number of manual interventions per month, and tenant satisfaction. The last two tell you whether you are building a platform or running a help desk.

The review checklist

Use this for a design document or for a pull request that touches the platform. Each question has a concrete thing to look for.

The complete picture

Put the whole platform on one page and it reads as a chain of promises, each of which has a component that keeps it and a failure that breaks it.

Where the series leaves you

Across the three levels you went from a first curl against port 9200 to owning the Elastic Stack as a platform. The same subjects carried all the way through, each time with more consequences:

You should now be able to look at a cluster's settings and a pipeline's configuration and see what they promise: where data waits when Elasticsearch is slow, who can read what, what retention costs, and what the next upgrade will break. The most useful habit to carry out of here is to ask of every design, "what is the first resource to run out, and what will we see when it does?"

Next, the neighbouring guides pick up the ground around the platform: Kubernetes for running it with ECK, Terraform for provisioning the infrastructure under it, and Ansible for configuring self-managed nodes. Most incidents on this stack are prevented at review time, and the checklist above is how you make that true.

Sources

• Elastic documentation home
• Elastic Stack release notes
• Elasticsearch breaking changes
• Elasticsearch deprecations
• Kibana breaking changes
• Logstash breaking changes
• Beats breaking changes
• Elastic Stack 9.5.4 release announcement
• Security update ESA-2026-182
• Product end-of-life dates
• Licensing FAQ
• Subscriptions
• Sizing shards
• Data streams
• Bootstrap checks
• Important system configuration
• Security auto-configuration
• Set up security manually
• Upgrade a deployment or cluster
• Prepare to upgrade
• Upgrade from 7.17
• Logstash settings file
• Elasticsearch output plugin for Logstash
• Elastic Agent command reference
• Fix watermark errors
• Circuit breaker errors
• Rejected requests
• Mapping explosion

Part three of three. At this level an interviewer is not checking that you know what a shard is. They are checking whether you can own a cluster other teams depend on: design it from a vague brief, diagnose it under pressure, and argue for its cost and its upgrades to people who do not care about Lucene. Every answer below is written the way a strong candidate would say it, in prose, with the reasoning visible. Practise saying them out loud.

The version context matters in these answers: Elastic Stack 9.5.4, released 15 September 2026, with 9.4.7 and 8.19.x as maintained neighbours. Naming the version, and the fact that 9.5.4 carries the fix for ESA-2026-182, is a cheap way to show you track the platform.

The thirty-second answer

The Elastic Stack is a search-and-analytics platform: shippers (Elastic Agent or Beats) collect data, optionally through a buffer and Logstash, Elasticsearch stores and indexes it in data streams, and Kibana explores and alerts on it. As a platform owner, I am responsible for four promises: data is not lost, the cluster stays available, only the right people read it, and the cost is predictable. Nearly every decision comes down to shard economics, ingest backpressure, the trust model and retention.

System-design prompts

For each prompt the pattern is the same: clarify requirements out loud, state the numbers you would want, propose a design, name the failure modes, and say what you would measure. Interviewers reward the clarifying questions as much as the answer.

Design a central logging platform for forty teams

Clarify first. How much data per day, and what is the peak-to-average ratio? How long must each class of data be retained? Are there regulated classes, such as audit logs or personal data, and where must they live? What is the tolerated delay between an event happening and being searchable? Is there an existing Kafka? Who is on call for it?

Design. Collection is Elastic Agent managed by Fleet, with policies per environment, because it gives central control and integrations that bring their own pipelines and dashboards. Between the collectors and the cluster I put a durable buffer, Kafka if we already run it, otherwise Logstash with persistent queues, so that an Elasticsearch outage becomes a backlog and not data loss. Logstash is the transformation and routing tier, with a distributor pipeline that hands events to per-team pipelines so one broken parser cannot stall everyone. Elasticsearch runs three dedicated master-eligible nodes across three zones, with allocation awareness set, and hot, warm and cold data tiers, with a frozen tier only if the business pays for the Enterprise licence and accepts slower queries for old data. Each team gets a data stream namespace such as logs-<service>-<team>, a lifecycle policy chosen by data class, a scoped API key for its shipper, a read role on its own pattern, and a Kibana Space.

Sizing. Take daily ingest, multiply by retention and by one plus replicas, and divide by the usable disk per node, leaving headroom so that losing a node keeps the rest under the 85 percent low watermark. Then check shards: target 10 to 50 GB each, rollover on max_primary_shard_size, and confirm we stay well below 1,000 per node.

Failure modes I would name. Oversharding from per-service daily indices. A noisy tenant filling hot disk. Mapping explosion from free-form keys. An expired certificate partitioning the cluster. A restore that has never been tested.

What I would measure. Ingest success rate, ingest lag, rejections, disk headroom per tier, search latency by tenant, and cost per tenant.

Design a multi-tenant platform where two tenants must not see each other's data

Clarify. What does "must not" mean: a policy, a regulatory requirement, or a contractual one? Do the tenants share engineers? Is the data sensitivity the same?

Answer. It is a spectrum, so I would choose by trust level. For cooperating internal teams, one cluster with namespaced data streams, index-pattern role privileges and Kibana Spaces is adequate and cheap, and the naming convention is enforced by giving shippers keys that can only create matching names. For tenants whose data is mutually sensitive I would add document-level security only if sharing an index is unavoidable, because it is a paid feature with a performance cost and a complicated failure mode. For regulated or hostile tenants I would use separate clusters and give a central team cross-cluster search for aggregate views. The point I would stress is that Spaces are a convenience boundary and not a data boundary: a user who can read an index can read it through Dev Tools regardless of the Space.

Design ingest that survives a day of Elasticsearch downtime

Answer. A day of downtime means the buffer must hold a day of peak volume. Kafka sized for that retention is the clean answer: producers keep writing, Logstash consumers lag, and when Elasticsearch returns the consumers catch up, rate-limited so the catch-up does not itself overload the cluster. Without Kafka, Logstash's persistent queue on local disk gives at-least-once delivery, but it must be sized in gigabytes of peak volume times a day, which is often more than anyone allocated, and inputs like UDP syslog cannot be made safe at all. At the shipper, backoff on 429 is mandatory. I would also make the writes idempotent where duplicates matter, using a deterministic document ID, since at-least-once means duplicates are possible. And I would test it by actually stopping Elasticsearch on a staging stack, because the architecture diagram and the behaviour diverge more often than people expect.

Design the migration from a 7.17 cluster to 9.5.4

Answer. The upgrade path is fixed: 7.17 to 8.19.x, then to 9.5.4, and there is no downgrade, so every step begins with a snapshot. 7.17 is already out of support as of 15 January 2026, and 8.x is at end of maintenance on 15 January 2027, which gives the project a deadline. Before the 8.19 to 9 step, indices created in 7.x must be reindexed, deleted or set read-only, frozen indices from older releases have to be dealt with since they are unreadable in 9.0, and ECK, if used, must be 3.x. I would run the Upgrade Assistant and GET _migration/deprecations, rehearse on a copy of production data, and pay special attention to things that change silently: Filebeat's fingerprint file identity, duplicate input IDs, ES|QL partial results, timeouts becoming 429, and the Logstash SSL option renames. The rolling order is Elasticsearch first (frozen, cold, warm, hot, then masters last), then Kibana, Fleet Server, agents, Logstash and Beats. I would offer leadership a timeline with a rehearsal date, a freeze window and a rollback definition, where rollback means restoring from snapshot to the old version, not downgrading in place.

Design observability for the platform itself

Answer. Send monitoring to a separate cluster so that the diagnosis does not depend on the patient. Use Elastic Agent or Metricbeat integrations, because the legacy internal collection is deprecated in 9.5 and leaves in 10.0. Alert on symptoms: disk headroom, write rejections, breaker trips, shards per node, ingest lag, DLQ growth, certificate and key expiry, and the age of the last good snapshot. Add a canary event written end to end every minute; it is the only test that covers every hop.

Incident-response questions

These are scenario questions. A strong answer names the first thing you check and why, states a hypothesis, separates mitigation from fix, and says how you would prevent a repeat.

Writes are failing with a cluster block error. What do you do?

The error TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block tells me at least one node crossed 95 percent disk. First I check GET _cat/allocation?v to see which nodes are full and on which tier. Mitigation is to free space safely: delete or move the oldest data that policy allows, not to raise the watermark, which only delays the same problem. The block lifts automatically when the node falls below the high watermark, and if it does not I remove it with PUT */_settings {"index.blocks.read_only_allow_delete": null} after confirming there is space. The fix is capacity or retention: add nodes, shorten retention for the data class that grew, or cut volume at the source. Prevention is alerting at 75 to 80 percent and a capacity model that assumes one node lost. I would also explain in the post-incident review why we did not page at 85.

The cluster is yellow and stays yellow. Why?

Yellow means all primaries are assigned but some replicas are not. I run GET _cluster/allocation/explain, which names the decider that refused the allocation. The common causes are a single-node cluster with replicas configured, disk watermarks preventing allocation, allocation awareness with too few zones for the replica count, a filter left over from maintenance (cluster.routing.allocation.enable still set to primaries after an upgrade is the classic), or a shard that has exhausted its retry count, for which POST _cluster/reroute?retry_failed=true retries once the cause is fixed. I would mention that yellow is not an emergency but it means you have lost redundancy, so a second failure becomes data loss.

One node is slow and users complain about search latency. Cluster is green.

Green only says shards are assigned. I look at per-node data: GET _nodes/stats/jvm,breaker for heap and GC, GET _cat/thread_pool?v for queues and rejections, GET _nodes/hot_threads for what the node is doing, and the search slow logs for the queries involved. A slow node is typically one of a few things: a failing disk or noisy storage, long garbage-collection pauses from heap pressure, a hot shard because routing or a time-based pattern concentrated load, or an expensive query such as a high-cardinality terms aggregation or a leading-wildcard query. GET _tasks?detailed&actions=*search* finds the running offender, and POST _tasks/<task_id>/_cancel stops it. If the node itself is sick, I exclude it from allocation and let the cluster move its shards, rather than restart it blind.

Logs stopped appearing for one team. The cluster is fine.

I work from the source towards the store, because the cluster being fine means the failure is upstream or in permissions. Is the agent running (sudo elastic-agent status, or filebeat test output)? Did a credential rotate or expire, giving 401 or 403? Did a certificate change, giving x509: certificate signed by unknown authority? Is the buffer growing, or Logstash's persistent queue filling? Are events landing in the dead letter queue with mapping errors? Did a recent upgrade change a default, such as the Filebeat filestream fingerprint identity that skips files under 1,024 bytes until they grow, or a duplicate input ID that prevents startup? Mitigation is to restore the flow and replay from the buffer. Prevention is the canary event per tenant, which tells us before the tenant does.

Kibana says the server is not ready, after an upgrade.

Kibana cannot complete startup, and the causes are few: it cannot reach Elasticsearch (wrong hosts, TLS trust, credentials), Elasticsearch is red, versions are mismatched, or saved-object migration is failing. I read the Kibana log first, since it states which. If the log shows that encrypted saved objects cannot be decrypted, the xpack.encryptedSavedObjects.encryptionKey differs from the one used before, which is why that key is backed up separately and identical on every Kibana instance. I also check that the configuration does not use removed settings; 9.0 removed several, including xpack.actions.whitelistedHosts and xpack.reporting.roles.*.

A developer says "we lost data". How do you establish the truth?

First define "lost": never arrived, arrived and rejected, arrived and was deleted by retention, or is present but unsearchable because of mapping or permissions. I check the source's shipper logs for errors, the buffer for the time window, the Logstash DLQ for rejected events, Elasticsearch for the document by ID or time range, and the lifecycle policy for whether retention deleted it. Most "lost" data is in one of those places. If it really is gone, I look at the snapshot policy and the restore plan. The important part of the answer is the discipline: establish facts with evidence before proposing a cause, and communicate what is known and what is not.

The security team reports an API key leaked in a public repository.

Invalidate it immediately with DELETE /_security/api_key, issue a replacement scoped the same way, and roll it to the producer. Then establish what it could do, which is exactly its role descriptors, and review audit logs if we have them (a paid feature) for use outside expected sources. The design lessons are the answer a senior candidate adds: one key per producer, narrow privileges (create_doc and auto_configure on one pattern), expiry, and owner metadata so we can find keys and their owners. Also add secret scanning to repositories so the next leak is caught at commit time.

How to argue a trade-off to leadership

Leadership does not want a tour of Lucene. They want a decision framed in terms of risk, cost and time, with a recommendation and what you need from them. The structure that works: the decision, the options with their costs and risks, your recommendation, and the consequence of doing nothing.

"Why are we spending so much on logging?"

Start with the driver: storage dominates, and storage is volume times retention times copies. Then show the levers with their effects and risks: drop low-value logs (saves the most, needs teams to agree), shorten retention for debug logs (cheap, loses history), use LogsDB and synthetic source (saves storage, changes how _source is returned), tier older data to cheaper nodes (saves, slows old queries), and the frozen tier (saves most on long retention, needs Enterprise). Recommend the order: measure per-team volume first, drop and sample, then retention per data class, then tiers. Offer the per-team cost report as the durable fix, because teams reduce what they can see. Be honest about what you do not know, for example the exact saving percentage from LogsDB on your data, which you would measure on a sample before promising.

"Why can't we just upgrade whenever?"

Because the path is constrained, the clock is external, and the failure cost is high. Seven-seventeen is out of support, 8.x reaches end of maintenance on 15 January 2027, and there is no downgrade. There are known silent behaviour changes that only a rehearsal will find. The trade-off is engineering time now against an emergency upgrade under a security advisory later, with ESA-2026-182 as a live example: the fix exists only in 9.4.7 and 9.5.4, and a cluster that cannot upgrade in a week is a cluster that stays exposed. Ask for a rehearsal environment and a dated window.

"Should we run it ourselves or buy the hosted service?"

Frame it as cost of ownership, not price. Self-managed gives control of data location and configuration and needs people who can run it: upgrades, capacity, incidents, security. Elastic Cloud Hosted or Serverless trades some control for operations done by the vendor, and Serverless is versionless and autoscaled but has no node or shard management and uses ILM-free lifecycle, with API keys only. For Gulf or Egyptian employers, the deciding factor is often data residency: whether a region exists that satisfies the regulator, and whether metadata leaves it. My recommendation would depend on volume and team size: small volume and a small team favour hosted; large stable volume with compliance constraints and a skilled team favour self-managed or ECK. I would say what data would change my mind.

"Do we need the paid tier?"

List the concrete features the requirement depends on, not the tier. SAML or OIDC single sign-on on self-managed, field and document-level security, audit logging, cross-cluster replication, machine learning and searchable snapshots are the ones that usually decide it. If none is required, Basic covers TLS, RBAC, API keys, ES|QL and Kibana. The mistake I would call out is discovering a paid dependency after launch, so the review checklist asks for the licence tier of every feature in a design.

"Why do we need a separate monitoring cluster?"

Because a monitoring system that shares the failure domain fails together with what it monitors, and the data you would use to diagnose a red cluster would be unavailable. It costs one small cluster. The alternative of an Elastic-hosted diagnostic service is viable if data-leaving-the-estate is acceptable.

Common interview questions

1. Why three master-eligible nodes?The master publishes cluster-state changes and needs a majority of master-eligible nodes to acknowledge them. Three tolerates one failure; four also tolerates only one, so an even number adds cost without resilience. Spreading them across three zones keeps a majority through the loss of a zone. A voting_only node can serve as a tiebreaker.
2. What is the difference between green, yellow and red, and why is green not enough?Green means all shards are assigned, yellow means primaries are assigned but some replicas are not, and red means at least one primary is unassigned. It says nothing about latency, heap pressure, rejections or disk headroom, so I alert on symptoms as well as colour, and read _health_report for diagnoses.
3. How do you decide the number and size of shards?Aim for shards of roughly 10 to 50 GB and under about 200 million documents, and keep shards per node well under the 1,000 default limit. Each shard costs heap regardless of its size, so many small shards waste memory and slow the master. Use rollover on max_primary_shard_size rather than a daily index per service.
4. What happens at the disk watermarks?At the low watermark (85 percent by default) no new shards are allocated to the node, at the high watermark (90 percent) shards are relocated away, and at flood stage (95 percent) indices with a shard there get a read-only-allow-delete block. The block releases when usage drops below the high watermark.
5. How do you size the JVM heap?Leave automatic sizing in place, and if overriding, set equal minimum and maximum in a file under jvm.options.d, never above half of RAM and below the compressed-pointer threshold of roughly 26 to 31 GB. The remaining memory is the file-system cache that Lucene depends on.
6. What is a persistent queue and what does it guarantee?In Logstash, queue.type: persisted stores the in-flight events on disk so they survive a restart and gives at-least-once delivery. Duplicates are possible after a crash, so use deterministic document IDs where they matter. A full queue applies backpressure to the inputs instead of dropping data.
7. When would you use Kafka in the ingest path?When I need a replayable buffer measured in days, when several consumers need the same stream, or when Elasticsearch outages must not affect producers. If those are not requirements, Logstash's persistent queue is simpler to run.
8. Logstash or ingest pipelines?Ingest pipelines run on Elasticsearch ingest nodes, and need no extra tier, which is enough for parsing, renaming and enrichment. Logstash is better for heavy transformation, routing to several destinations, buffering, and inputs Elasticsearch cannot do. Integrations ship with ingest pipelines, which is another reason to start there.
9. How do you authenticate machines to Elasticsearch?API keys for shippers and applications, with role descriptors restricting them, an expiry and owner metadata; service accounts and tokens for Kibana and Fleet Server. Never the elastic superuser, and never a shared credential. Beats and the Logstash output take id:api_key, while HTTP clients send the base64 encoded value in an Authorization: ApiKey header.
10. What is the minimum privilege for a log shipper?The monitor cluster privilege, plus auto_configure and create_doc on the data stream pattern it writes to. Data streams accept only create operations, which is why create_doc is the right index privilege.
11. What are the main multi-tenancy options?Namespaced data streams with role patterns, Kibana Spaces with feature privileges, document and field-level security on paid tiers, and separate clusters joined by cross-cluster search. The right one depends on how much the tenants trust each other, and the naming convention must be enforced for role patterns to mean anything.
12. What is a mapping explosion and how do you prevent it?Dynamic mapping adds a field per new key, so user-supplied keys create thousands of fields, bloating the cluster state. Prevent it with explicit mappings, dynamic: false or strict, the flattened type for free-form maps, and index.mapping.total_fields.ignore_dynamic_beyond_limit.
13. How does a rolling upgrade work?Disable replica allocation with cluster.routing.allocation.enable: primaries, flush, upgrade one node, re-enable allocation, and wait for green before the next. Order the nodes frozen, cold, warm, hot, other non-master nodes, and master-eligible nodes last. Never run more than two versions in one cluster, and snapshot first because there is no downgrade.
14. What must be true before moving to 9.x?The cluster must be on 8.19.x, indices created in 7.x must be reindexed, deleted or made read-only, deprecations resolved via the Upgrade Assistant and _migration/deprecations, and ECK at 3.x if used.
15. ILM or data stream lifecycle?ILM is the full policy engine with phases and actions including shrink, force-merge, allocate and searchable snapshots. The data stream lifecycle is simpler, with retention and downsampling, and Serverless uses only that. In 9.5 it can move data to the frozen tier in tech preview. I choose ILM for tiered clusters and the lifecycle for simple retention.
16. Why shrink and force-merge only in the warm phase?Both are expensive operations suited to indices that no longer receive writes. Force-merging a live index creates huge segments that normal merging cannot reclaim.
17. How do you back up and what doesn't a snapshot contain?SLM policies create incremental snapshots to a repository such as S3. A snapshot does not contain keystores, certificates, configuration files or the Kibana encryption keys, which belong in a secrets manager. The restore must be tested and timed.
18. What changed for S3 snapshots in 9.1?The repository moved to AWS SDK v2: the region must be set with s3.client.<name>.region, IMDSv1 is unsupported, and V4 signatures are required, among other changes. A snapshot repository that worked on 8.x may fail after upgrade until configured.
19. How do you handle deep pagination?Not with from and size, which fails beyond the 10,000 result window. Use search_after with a point in time. Scroll is not recommended for deep pagination.
20. What are circuit breakers?Memory protections that fail a request with a 429 instead of letting the JVM run out of heap. The parent breaker defaults to 95 percent of real memory, the request breaker to 60 percent of heap and the fielddata breaker to 40 percent. Rising trips are a capacity signal.
21. What does a 429 from Elasticsearch mean?Back off. It can be a rejected thread pool task, indexing pressure, a circuit breaker, or, since 9.0, a timeout. Clients must retry with exponential backoff and smaller batches, not retry immediately.
22. What do you do about cost?Drop and sample at the edge, set retention per data class, use LogsDB and TSDB with synthetic source and downsampling, tier data, and show each team its own number. Measure before promising savings.
23. When should you not use Elasticsearch?As a system of record, for transactional or relational workloads, for rarely queried archives better kept in object storage, and when licence terms or team capacity make another choice sensible.
24. How do you secure Kibana and its secrets?TLS, a service account token rather than a password, encryption keys (32 or more characters) that are identical on every instance and backed up, secrets in the Kibana keystore, and Kibana behind an identity-aware proxy. Spaces plus feature and index privileges for access control.
25. How do you monitor the stack?A separate monitoring cluster fed by Elastic Agent or Metricbeat, symptom-based alerts, and an end-to-end canary. Legacy internal monitoring is deprecated in 9.5 and removed in 10.0.

What a strong answer covers that a weak one misses

The difference between a good and a great answer at this level is rarely factual. It is structure.

The traps interviewers set

A few questions are built to catch a recited answer.

• "Add more shards to go faster." Primary shard count is fixed at index creation, more shards cost heap, and past a point they slow things down. The real levers are data tiers, hardware, better queries, and splitting or rolling over when needed.
• "Set cluster.initial_master_nodes on every restart to be safe." That setting is for bootstrapping a new cluster once, and is removed afterwards. Keeping it, or using it against an existing cluster, risks forming a separate cluster.
• "Disable security in the lab, it's simpler." Security is on by default since 8.0 and the auto-configuration makes it easy. Turning it off teaches habits that leak into production. Use the CA certificate and credentials.
• "We'll just raise max_shards_per_node." It is a guard rail, and raising it postpones the problem while making the master's work heavier.
• "Snapshots mean we have a DR plan." Not until you have restored one, timed it, and stored the keys that go with it.
• "Logstash and Beats are deprecated." They are not. Elastic Agent with Fleet is the recommended path for new deployments, Beats are still shipped and supported, and Logstash remains the heavy transform and routing tier. What is deprecated is specific: the Filebeat log input in favour of filestream, legacy monitoring, and some plugin options.

Sixty-second self-test

Answer each in one sentence, without looking above.

1. Why three master-eligible nodes, and why not four?
2. What are the three disk watermarks and what happens at each?
3. Which resource runs out first for logs, and what is the error when it does?
4. What does a persistent queue guarantee, and what does that imply about duplicates?
5. What is the minimum privilege for a log shipper writing to a data stream?
6. What is the upgrade path from 7.17 to 9.5.4, and what must you do before you start?
7. Which three things does a snapshot not contain?
8. Name four cost levers, in order of how invasive they are.
9. Why are Kibana Spaces not a data boundary?
10. Name three cases in which you would pick something other than the Elastic Stack.

If you hesitated on more than three, reread the matching section of the detailed guide, then practise the answer aloud. For the platforms around this one, see Kubernetes for ECK deployments and Terraform for provisioning.

Part three of three. At this level the problems are not failed commands. They are slow degradations that arrive as a page at 3 a.m.: a cluster at 94 percent disk that nobody noticed, a certificate that expired on a Sunday, a tenant whose debug logging doubled the bill. Start with the error table, then the practices, then the checklists you can paste into a pull request template.

Everything here assumes Elastic Stack 9.5.4 (released 15 September 2026). If you are on 9.2.0 through 9.4.6 or 9.5.0 through 9.5.3, the first tip is to upgrade to 9.4.7 or 9.5.4, because ESA-2026-182 (CVE-2026-94396) is a resource-consumption denial of service with no workaround.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These rarely show up in a lab, because they need volume, time, or many tenants to appear.

Three traps worth a longer explanation

Green means assigned, not healthy. The health colour only reports shard assignment. The habit that separates a platform owner from a user is alerting on symptoms: disk headroom by tier, write rejections, breaker trips, shards per node, pending tasks, ingest lag. Teach everyone to open GET _health_report instead of staring at the colour. It returns indicators, each with a diagnosis and an impact, and it is often the fastest way to learn why a cluster is unhappy.

The watermark is a symptom of a retention decision. When disk fills, people reach for the watermark settings because they are easy to change. They are guard rails: the low watermark (85 percent) stops new shards arriving, the high watermark (90 percent) relocates shards away, and flood stage (95 percent) blocks writes. Moving them higher gives you a few hours and a worse incident. The real decision is capacity against retention, and the incident is the moment to record that in writing.

An expired certificate looks like a network fault. When a transport certificate expires, nodes stop talking, masters lose quorum, and the log lines read like connectivity problems. Engineers then spend an hour on network ACLs. Keep an inventory of every certificate (transport, HTTP, Kibana, Logstash, Fleet Server, the CA itself), alert weeks before expiry, and check validity first in any partition incident.

The practices that pay off most

These are ordered roughly by how much incident and cost reduction you get for the effort.

1. Roll over on size and count your shards

Shards are what you pay for, so the first habit is making their number and size a deliberate quantity. A single query from Dev Tools tells you most of what you need:

GET _cat/shards?v&h=index,shard,prirep,store,node&s=store:asc
GET _cat/allocation?v
GET _cluster/settings?include_defaults=true&filter_path=**.max_shards_per_node

How to read it: sort ascending by store size and look at the top of the list; hundreds of shards under 1 GB are the signature of oversharding. Then divide total shards by data nodes and compare against the limit. If the number is trending toward 1,000 per node, schedule the fix before the day a rollover fails.

2. Put guard rails into component templates

Do not rely on documentation for limits. Compose them into every tenant template:

PUT _component_template/platform-guardrails
{
  "template": {
    "settings": {
      "index.mapping.total_fields.limit": 1000,
      "index.mapping.total_fields.ignore_dynamic_beyond_limit": true,
      "index.lifecycle.name": "platform-logs",
      "number_of_replicas": 1
    }
  }
}

PUT _index_template/tenant-payments
{
  "index_patterns": ["logs-*-payments"],
  "data_stream": {},
  "priority": 500,
  "composed_of": ["platform-guardrails"]
}

A template that every tenant inherits cannot be forgotten, and changing one component changes the policy for everyone on their next rollover. Keep the priority above the built-in logs-*-* template, or, better, extend the built-in behaviour through the @custom component template; I have not verified the exact @custom naming on the current docs, so check it before relying on it.

3. Make the ingest contract explicit

Tenants will write bulk requests of the size they find convenient unless told otherwise. Publish a contract: maximum bulk size in megabytes, required exponential backoff on 429, the authentication method, the allowed index patterns, and what a rejection means. Logstash, Beats and Agent already back off correctly; it is the bespoke application client that does not. The contract is also where you say that 429 means "slow down" and not "failure".

4. Alert on the causes you can act on

An alert that nobody can act on becomes noise. Keep the set small and actionable:

# Illustrative alert list, expressed in plain configuration form
- name: hot-tier-disk-high
  condition: disk.used_percent > 78 on data_hot nodes for 10m
  action: page platform on-call
- name: write-rejections
  condition: indexing rejections rate > 0 for 10m
  action: ticket, contact top producers
- name: shards-per-node
  condition: shards per node > 800
  action: ticket, plan rollover and shrink changes
- name: dlq-growing
  condition: Logstash DLQ size increased for 15m
  action: ticket owning tenant
- name: cert-or-key-expiry
  condition: any certificate or API key expires within 21 days
  action: ticket, schedule rotation
- name: canary-missing
  condition: no canary event in the last 5m
  action: page platform on-call

That block is pseudo-configuration to show the shape, not a file any Elastic component reads. The thresholds are starting points: choose yours from your own headroom arithmetic. The canary is the one people skip and the one that catches everything.

5. Test the failure you are promising to survive

For each promise, schedule the test that proves it. Kill a data node during business hours, in a staging cluster first. Stop Elasticsearch for ten minutes with a shipper running. Restore a snapshot. Let a test API key expire. Rotate the Kibana encryption key in a lab and see what stops working. A promise without a test is an assumption, and the assumption fails at the worst time.

Platform-ownership habits

Owning a platform is mostly habits, practised when nothing is on fire.

Keep an upgrade calendar

Write down the dates that matter and review them quarterly: 7.17 is past end of support (15 January 2026); 8.x reaches end of maintenance on 15 January 2027 and end of support on 15 July 2027; 9.x end of maintenance is 15 October 2027 at the minimum, extended once 10.0 ships. Note what is coming: a 9.6.0 Kibana breaking change already lists new privileges for workflow execution APIs, although 9.6.0 was not released as of 30 September 2026. A calendar turns "we should upgrade" into a scheduled piece of work.

Subscribe to the security advisories

Someone on the team owns reading the Elastic security announcements. ESA-2026-182 is the model case: an authenticated denial of service with a CVSS of 6.5, affecting a range of versions, fixed only by upgrading. A platform owner's response is measured in days: read it, assess exposure, schedule the rolling upgrade, confirm afterwards.

Keep a runbook per incident class

The playbooks in the detailed guide become runbooks once they live where on-call can reach them. Each has four lines: how it presents, the first three checks, the reversible mitigation, the follow-up. Review them after every incident and fix them where they were wrong, because a runbook that was wrong once will not be trusted again.

GET _cluster/allocation/explain
GET _cat/allocation?v
GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected
GET _nodes/stats/jvm,breaker,indexing_pressure
GET _cat/circuit_breakers
GET _health_report

Save that list as a Dev Tools collection or a short page linked from the alert. Six requests cover the first five minutes of most incidents.

Hold a small set of standards, and enforce them by template

Naming follows <type>-<dataset>-<namespace>. Fields follow ECS. Retention is chosen per data class. Every API key has an owner. None of these survives on goodwill; all of them can be enforced by a template, a role pattern, or a pipeline step that rejects non-conforming data.

Treat tenants as customers

Publish the service level, the contract, the cost report and the roadmap. Announce deprecations a quarter before they bite, and give a migration guide with each. Keep a short intake process for exceptions, and record the reason for each. The measure of success is that tenants onboard themselves and you hear about problems from the canary before you hear from them.

Keep a blameless incident habit

After each incident, write what happened, the timeline, what was known when, what was done, and what will change. Focus on the system: the alert that was missing, the runbook that was wrong, the limit that was silent. Track the follow-up actions to completion, since the repeated incident is the one that costs credibility.

Protect the keys, not just the data

Three things outlive any cluster and are easy to lose: the CA private key, the Kibana encryption keys, and the keystore passwords. Keep them in a secrets manager with access control and an audit trail, test that you can retrieve them, and include them in the disaster recovery drill. A restore without them is a partial one.

Review checklists

Paste these into your pull request template and design-doc template. A reviewer should be able to answer each question by looking at the change.

Pull request checklist: a change to templates, policies or pipelines

• Does the change set dynamic behaviour and a field limit, or inherit them from the guard-rails component template?
• Does any rollover use max_primary_shard_size, and not the deprecated max_size?
• Is the lifecycle policy named after a data class, and does it match the retention in the data catalogue?
• For a new data stream, is the owner recorded, and does the name follow <type>-<dataset>-<namespace>?
• Does a Logstash change include a --config.test_and_exit run in CI and use current SSL option names (ssl_enabled, ssl_certificate_authorities)?
• Does the pipeline enable the persistent queue and the dead letter queue where loss is unacceptable, and is there an alert on DLQ growth?
• Are Filebeat inputs filestream with unique id values?
• Does a new API key have an expiry, owner metadata and only the privileges it needs?
• Is any secret present in the diff, including in values files and container images?
• Does the change run anywhere that uses verification_mode: none, -k, or the elastic superuser?
• Has the change been applied first to a non-production cluster, with the output attached?

Design document checklist: a new platform capability or tenant

• What are the expected volume, peak ratio, retention and search pattern, and how were they measured?
• What is the isolation level, and what enforces it?
• What buffers an Elasticsearch outage, and for how long?
• What is the cost per month, who pays, and what is the lever to reduce it?
• What happens when a node, a zone, and the whole cluster are lost, and what has been tested?
• What licence tier does each feature need?
• Where do the data and snapshots live, and does that satisfy the residency requirement?
• How will this be upgraded, and what in the breaking-changes lists touches it?
• What will we alert on, and what is the runbook?
• What would make us choose a different tool?

Quarterly platform review

1. Shards per node and smallest common shard size.
2. Disk headroom per tier, projected to next quarter.
3. Cost per tenant, and the biggest movers.
4. Certificates and keys expiring within 90 days.
5. The last restore drill: date, duration, result.
6. The upgrade calendar and the open security advisories.
7. Deprecations on the horizon and their owners.
8. Incidents: count, cause categories and follow-up status.

Practice cards

Operational recipes

Find the biggest data streams by tenant

GET _cat/indices/.ds-*?v&h=index,pri,rep,docs.count,store.size&s=store.size:desc&bytes=gb

Backing indices are named .ds-<name>-<yyyy.MM.dd>-<generation>, so group by the data stream name part to attribute storage to a tenant. Feed the grouped numbers into the monthly cost report.

Take the first look at a slow cluster

GET _cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent
GET _cat/pending_tasks?v
GET _tasks?detailed&actions=*search*

How to read it: heap above roughly 85 percent sustained, a node with CPU far above its peers, pending tasks that do not drain, or a single long-running search task each point to a different branch of the playbook. Cancel a runaway task with POST _tasks/<task_id>/_cancel.

Rotate an API key safely

POST /_security/api_key
{
  "name": "payments-shipper-2026-10",
  "expiration": "90d",
  "role_descriptors": {
    "writer": {
      "cluster": ["monitor"],
      "indices": [ { "names": ["logs-*-payments"], "privileges": ["auto_configure", "create_doc"] } ]
    }
  },
  "metadata": { "team": "payments", "rotated_from": "payments-shipper-2026-07" }
}

Roll the new value to the shipper (remembering the id:api_key form for Beats and Logstash), confirm documents are arriving with the new identity, and only then invalidate the old key with DELETE /_security/api_key and its ids. Doing it in this order means a mistake costs you nothing.

Safely prepare a rolling upgrade step

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": "primaries" } }

POST _flush

After upgrading and restarting the node:

PUT _cluster/settings
{ "persistent": { "cluster.routing.allocation.enable": null } }

GET _cluster/health?wait_for_status=green&timeout=10m

Do not move to the next node until the second request returns green. A stop condition such as "yellow for more than ten minutes" belongs in the runbook, with the name of the person who decides.

Recover writes after a mapping failure

When Logstash reports mapper_parsing_exception and events accumulate in the dead letter queue, fix the mapping or the source first, then reprocess with a short pipeline that reads from the DLQ:

input {
  dead_letter_queue {
    path => "/var/lib/logstash/dead_letter_queue"
    pipeline_id => "team-payments"
    commit_offsets => true
  }
}
output {
  elasticsearch {
    hosts => ["https://es-01:9200"]
    api_key => "${ES_API_KEY}"
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    data_stream => "true"
  }
}

Run it after the fix, check the document counts, and keep the original pipeline's DLQ alert in place. Test the replay on a sample before running it against a full queue.

A closing checklist

• Running 9.5.4 or 9.4.7 or later, with the next upgrade dated.
• Three master-eligible nodes across three zones, and awareness configured.
• Shards of 10 to 50 GB, rollover on max_primary_shard_size, well under 1,000 per node.
• Disk alerts at 75 to 80 percent and headroom for one node lost.
• A buffer tier, tested with a real outage.
• One narrow, expiring, attributed API key per producer.
• Your own CA, certificate expiry alerted weeks ahead, no verification disabled.
• Kibana encryption keys identical on all instances and backed up separately.
• A restore drilled and timed this quarter.
• External monitoring, symptom alerts and a canary.
• A per-tenant cost report, and a retention policy per data class.
• Written criteria for when to choose a different tool.

For the tools around this one, see Kubernetes for running it with ECK and Ansible for configuring self-managed nodes.