The Complete Apache Spark Guide

This is part one of three. It covers everything you need to do real work with Apache Spark on a single machine, not a teaser. By the end you can install Spark, start a session, load a CSV file, clean it, summarise it with both the DataFrame API and SQL, write the result as Parquet, read the web interface to see what your code actually did, and recognise the handful of errors that stop almost every newcomer. Mid-level and Senior take the same topics further; nothing here is thrown away.

The guide targets Apache Spark 4.2, which the official downloads page lists as the current stable release at the time of writing. Patch numbers move quickly, so check https://spark.apache.org/downloads.html on the day you install. Everything below uses Python (PySpark), because that is what most data and machine learning teams reach for first.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and Spark's ideas stick only once you have watched your own job run, finish, and show up in the web interface.

What Spark is, and the problem it solves

Apache Spark is an engine for processing data that is too big, or too slow to process, on one core of one machine. You describe a computation (read these files, keep the rows where this is true, group by that column, average this number) and Spark splits the data into pieces, spreads the pieces across many processor cores or many machines, runs your computation on every piece at once, and stitches the answer back together.

The diagram is the whole idea, and the rest of this guide fills in each box. Notice what is missing: you never write a loop over the pieces, you never open threads, you never decide which machine handles which rows. You write what you want, and Spark works out how.

Why would you need that? Picture a retailer in Cairo or Riyadh with three years of order history: a few hundred million rows across thousands of files. A script that loops through them in plain Python takes hours and may run out of memory halfway. The same question in Spark, running on a laptop with eight cores, already finishes several times faster, because eight pieces are processed at once. Point the same code at a cluster of fifty machines and nothing in your program changes; it just finishes sooner. That property, the same code on a laptop and on a cluster, is the reason Spark became the default tool for large-scale data work.

What people use it for:

Spark is also a family of parts that share one engine: Spark SQL and DataFrames for tables, Structured Streaming for data that keeps arriving, MLlib for machine learning, and GraphX for graphs. This guide stays with the first one, because it is the foundation the others stand on.

What came before Spark

To appreciate the design, you need the story in one paragraph. In the mid-2000s, companies with huge datasets used a model called MapReduce, popularised by Hadoop. You wrote two kinds of functions, a map that processes each record and a reduce that combines results, and the system ran them across a cluster of cheap machines. It worked and it scaled, but every step wrote its intermediate results to disk, and real analyses needed many steps chained together. A ten-step analysis meant ten rounds of writing to and reading from disk, and writing the steps in the MapReduce style was tedious.

Spark kept the central insight, splitting data across machines and moving the computation to the data, and changed two things. First, it keeps intermediate data in memory where it can, so chained steps do not pay the disk round trip each time. Second, it offers a high-level API: instead of hand-writing map and reduce functions you call familiar operations such as filter, join and groupBy, or write SQL, and an optimiser decides how to execute them efficiently.

Two terms you will meet constantly belong here. Hadoop is an ecosystem; its file system, HDFS, and its resource manager, YARN, are still widely used, and Spark can run on YARN and read from HDFS. A cluster is just a group of machines that cooperate on one job. You do not need a cluster to learn Spark. Everything in this guide runs on your laptop, in what Spark calls local mode, where one program plays every role.

One practical note on history, because tutorials online span many years. Older material uses an API called the RDD (resilient distributed dataset), a low-level distributed collection. You will still see it in old blog posts and Stack Overflow answers. For nearly all work today you use DataFrames instead, which are higher level, easier to write and usually faster, because Spark can optimise them. If a tutorial starts with sc.parallelize, it is teaching the older style, and you can skip ahead to its DataFrame section.

A second warning about old tutorials: Spark 4 only supports Scala 2.13, Java 17 or newer, and it removed support for Mesos as a cluster manager. If an article tells you to download a _2.12 jar or run Spark on Mesos, it was written for Spark 3 or earlier.

The mental model: driver, executors, partitions

Spark has a small vocabulary, and nearly every confusing moment later traces back to one of these nouns being fuzzy. Learn them now.

A running Spark program is an application. Every application has exactly one driver: the process that runs your main code, builds the plan for your computation, and hands out work. The driver holds your SparkSession, the object you type spark. on. Work itself happens in executors: processes, usually on other machines, that run tasks and can hold data in memory for reuse. The software that hands out machines and memory to the application is the cluster manager. Spark 4 supports Standalone (Spark's own simple one), Hadoop YARN and Kubernetes; and in local mode, no external manager at all.

The data side has its own nouns. A DataFrame is a distributed table: named columns with types, rows spread over many machines. You treat it like one table, and Spark handles the spreading. The pieces it is spread into are partitions; a partition is the chunk of data that a single task processes. If a DataFrame has 8 partitions and you have 8 free cores, 8 tasks run simultaneously. This is the single most important relationship in Spark: parallelism equals partitions processed at the same time.

Now the part that surprises everyone coming from pandas. Spark operations come in two kinds.

• Transformations describe a new DataFrame from an old one: select, filter, withColumn, groupBy, join. They are lazy. Calling one does not touch any data; it just adds a step to the plan.
• Actions ask for an actual result: show, count, collect, write. An action makes Spark take the whole accumulated plan, optimise it, and execute it.

Think of ordering at a restaurant. Transformations are you telling the waiter what you want, one dish at a time; nothing is cooked. The action is "bring the bill": at that point the kitchen reads the whole order and does the work in the sensible order. The benefit of laziness is that Spark sees your whole recipe before cooking, so it can skip unneeded columns, push filters as early as possible, and combine steps.

Executing an action produces a job. Spark splits a job into stages, and splits each stage into tasks, one per partition. Stage boundaries happen at a shuffle: any step that needs rows with the same key to end up together, such as a groupBy or a join, forces data to be redistributed across executors. Shuffles are the main cost in Spark, because they move data over the network and write it to disk. You do not need to tune them yet, but you should be able to say "this line causes a shuffle" when you see a groupBy.

Summarising the nouns in one table:

Installing Spark and checking it works

The easiest route is pip, because the PySpark package bundles Spark itself. You need two things on the machine first: a Java Development Kit and Python. Spark 4.2 documents support for Java 17, 21 or 25, and Python 3.10 or newer. Install a JDK from your package manager or a vendor you trust, and make sure the JAVA_HOME environment variable points to it.

Check both before anything else:

java -version
python3 --version
echo $JAVA_HOME

If java -version fails, or JAVA_HOME is empty, fix that first; it is the cause of most failed first starts. Now create an isolated Python environment and install PySpark:

python3 -m venv .venv
source .venv/bin/activate          # Windows PowerShell: .venv\Scripts\Activate.ps1
pip install pyspark

A virtual environment (the .venv folder) is a private copy of Python's packages for this project, so installing PySpark does not disturb anything else on your machine. Always activate it in each new terminal before working.

Then run a one-line smoke test that starts Spark, prints its version and prints five numbers:

python -c "from pyspark.sql import SparkSession; s=SparkSession.builder.master('local[*]').getOrCreate(); print(s.version); s.range(5).show()"

You should see log lines, then a version string and a small table:

+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
+---+

If you see that table, Spark works. spark.range(5) generates a one-column DataFrame named id; it is the quickest way to get test data with no files at all.

There are other ways to install, and each suits a different person. With conda, conda install -c conda-forge pyspark inside a new environment does the same job. If you want the full Spark distribution with its command-line tools, download the tarball from the downloads page, unpack it, set SPARK_HOME to that folder and add its bin directory to your PATH:

tar xzvf spark-4.2.0-bin-hadoop3.tgz
cd spark-4.2.0-bin-hadoop3
export SPARK_HOME=$(pwd)
export PATH=$SPARK_HOME/bin:$PATH
pyspark --master "local[2]"

That last command opens an interactive Python shell with a ready-made spark session. The same distribution gives you spark-submit, which is how scripts are launched on clusters, and you can confirm any installation with:

spark-submit --version

If you prefer containers, the project publishes the apache/spark image on Docker Hub; see the Docker guide at /student-guides/docker for the basics before trying it.

Your first session and first DataFrame

Everything in Spark starts from a SparkSession, the entry point object. Create a file called first.py:

from pyspark.sql import SparkSession

spark = (SparkSession.builder
         .appName("first-steps")
         .master("local[*]")
         .getOrCreate())

print("Spark version:", spark.version)

Read the builder chain from top to bottom. appName is a label that appears in the web interface, which helps when several applications run. master("local[*]") says "run here, on this machine, using as many threads as there are cores"; local[2] would use two. getOrCreate() returns the existing session if one is running, or creates one. Using it everywhere keeps you from accidentally starting two.

Now make a tiny DataFrame by hand, so you can experiment without any files:

people = spark.createDataFrame(
    [("Aya", "Egypt", 29), ("Omar", "Saudi Arabia", 41), ("Layla", "Egypt", 35), ("Khaled", "UAE", 17)],
    schema=["name", "country", "age"],
)

people.printSchema()
people.show()

Run it with python first.py. The output has two parts. printSchema() prints the column names and types, like name: string and age: long, along with whether each may contain nulls. show() prints the first rows as a table. The schema (the list of column names and types) matters a great deal in Spark, because the optimiser and the file formats rely on it.

Two methods you will use hundreds of times for orientation: people.columns returns the column names as a Python list, and people.count() returns the number of rows. Note which is an action. count() runs a job; columns only reads the schema, which Spark already knows, so it is instant.

When the script ends, the session ends too. In a notebook or an interactive shell, you can stop it explicitly with spark.stop(), which releases memory and the port the web interface used.

Reading real data: a first project, step by step

Toy data is for learning syntax. Now build something real: a small analysis of a CSV file. Create a file orders.csv in your project folder with these contents (add more rows if you like; the shape is what matters):

order_id,customer,country,category,amount,order_date
1001,Aya,Egypt,books,120.50,2026-01-04
1002,Omar,Saudi Arabia,electronics,899.00,2026-01-04
1003,Layla,Egypt,books,35.25,2026-01-05
1004,Khaled,UAE,electronics,1250.00,2026-01-06
1005,Aya,Egypt,electronics,299.99,2026-01-07
1006,Noor,Jordan,home,64.10,2026-01-07
1007,Omar,Saudi Arabia,home,210.00,2026-01-08
1008,Layla,Egypt,books,18.75,2026-01-09

Step one is reading it. A CSV file has no embedded types, so you must tell Spark two things: that the first line is a header, and how to work out column types.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("orders").master("local[*]").getOrCreate()

orders = spark.read.csv("orders.csv", header=True, inferSchema=True)
orders.printSchema()
orders.show(5)

header=True uses the first line as column names; without it you get _c0, _c1 and a header row mixed into your data. inferSchema=True makes Spark scan the file to guess types, so amount becomes a decimal-capable number rather than text. The cost is an extra pass over the data, which is trivial here and noticeable on huge files. For anything beyond learning, it is better to declare the schema yourself, which is faster and safer:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType, DateType

schema = StructType([
    StructField("order_id", IntegerType()),
    StructField("customer", StringType()),
    StructField("country", StringType()),
    StructField("category", StringType()),
    StructField("amount", DoubleType()),
    StructField("order_date", DateType()),
])

orders = spark.read.csv("orders.csv", header=True, schema=schema)

Declaring a schema also protects you: if next month's file has "N/A" in the amount column, you find out loudly instead of discovering that a column was quietly inferred as text. Other common readers follow the same pattern: spark.read.json(...), spark.read.parquet(...), and spark.read.format("jdbc") with options for reading from a database.

Transforming data: select, filter, withColumn, groupBy

With the DataFrame loaded, you now describe the analysis. The functions live in pyspark.sql.functions, conventionally imported as F, and a column is referenced with F.col("name").

Selecting picks columns, and filtering keeps rows:

big_orders = (orders
    .select("order_id", "customer", "country", "amount")
    .filter(F.col("amount") > 100))
big_orders.show()

Adding or changing a column uses withColumn, which takes the new column's name and an expression. Here we add a rounded amount with a flag column:

enriched = (orders
    .withColumn("amount_rounded", F.round(F.col("amount"), 0))
    .withColumn("is_large", F.col("amount") >= 500))
enriched.show(3)

DataFrames are immutable: withColumn does not change orders, it returns a new DataFrame. If you write orders.withColumn(...) on a line by itself and never assign the result, nothing is kept. This catches nearly everyone once.

Grouping and aggregating is where Spark earns its keep. groupBy names the key, and agg lists the summary calculations, each given a readable name with alias:

by_country = (orders
    .groupBy("country")
    .agg(F.count("*").alias("orders"),
         F.sum("amount").alias("revenue"),
         F.avg("amount").alias("avg_order"))
    .orderBy(F.desc("revenue")))
by_country.show()

The expected output is a table with one row per country, with Saudi Arabia and the UAE at the top for revenue in the sample data. Every groupBy is a shuffle, so this is the first line where Spark has to move rows between partitions so that all of one country's rows meet. On eight rows you cannot feel it; on eight hundred million rows it is the line you would watch.

Two more everyday operations. Joining combines two DataFrames on a key, for example attaching a table of country regions to the orders:

regions = spark.createDataFrame(
    [("Egypt", "North Africa"), ("Saudi Arabia", "Gulf"), ("UAE", "Gulf"), ("Jordan", "Levant")],
    schema=["country", "region"],
)
with_region = orders.join(regions, on="country", how="left")
with_region.select("order_id", "country", "region").show()

The how argument is the join type; left keeps every order even when no region matches, filling the missing region with null. And handling missing data has its own methods: df.dropna() removes rows containing nulls, and df.fillna({"amount": 0}) replaces nulls in named columns with a value. Choose deliberately; silently dropping rows is the usual way analyses go wrong.

Chaining all of this reads like a sentence, and nothing has run yet. Until you call show, count or write, Spark has only a plan.

Spark SQL: the same engine, in SQL

If you know SQL, you can use it directly. Spark SQL and the DataFrame API are two front doors to the same engine, and mixing them is normal. First give the DataFrame a name that SQL can see, called a temporary view:

orders.createOrReplaceTempView("orders")

spark.sql("""
    SELECT country,
           count(*)      AS orders,
           round(sum(amount), 2) AS revenue
    FROM orders
    WHERE order_date >= '2026-01-05'
    GROUP BY country
    ORDER BY revenue DESC
""").show()

spark.sql(...) returns a DataFrame, so you can keep chaining DataFrame operations on its result, or the other way round. A temporary view is only a name for a plan inside this session; it copies no data and disappears when the session ends.

When should you choose which? Use SQL when the logic is a clear query, particularly joins and window-style analyses that you or analysts already write in SQL. Use the DataFrame API when you are assembling steps programmatically, looping over column names, or want your editor to catch typos in function names. Because the optimiser sees both the same way, performance is not the deciding factor.

You can also look at what Spark intends to do. Call explain() on any DataFrame to print the plan:

by_country.explain()

The output is a tree read from the bottom up: a scan of the file, a partial aggregation on each partition, an exchange (that word means shuffle), then the final aggregation. You do not need to understand every line today. What matters is that you can find the word Exchange and know a shuffle happens there. With explain("formatted") you get a more readable layout. Note that in Spark 4.2 the plan may also mention adaptive execution; that is a runtime optimiser that revisits the plan using real data sizes, and it is switched on by default, so there is nothing for you to do.

Writing results, and the Parquet format

Analysis is only useful if the result lands somewhere. The write side mirrors the read side:

by_country.write.mode("overwrite").parquet("out/revenue_by_country")

Three things to understand here. First, write is an action: it is the line that actually runs your whole pipeline. Second, mode("overwrite") tells Spark what to do when the target exists; the options are overwrite, append, ignore and the default error/errorifexists, which refuses to write if the path exists. Third, the path is a directory, not a file.

Open out/revenue_by_country and you will find files with names like part-00000-....snappy.parquet, plus an empty file named _SUCCESS. Each part- file is written by one task for one partition, which is parallelism leaking into your file system. That is why a big job produces many files, and why every Spark reader accepts a directory as input.

Parquet is the format you should default to. It is a columnar format: values of one column are stored together, with compression and the schema embedded. Compared with CSV, a Parquet file is smaller, keeps its types, and lets Spark read only the columns a query needs. Read it back with spark.read.parquet("out/revenue_by_country") and printSchema() shows the original types with no inference. Spark can also write CSV and JSON with .csv(...) and .json(...), which is useful for handing data to people or tools that need it, but for data you will read again, choose Parquet.

When a file-per-partition layout is a problem, you can control how many files appear with coalesce, which reduces the partition count without a full shuffle:

by_country.coalesce(1).write.mode("overwrite").csv("out/revenue_csv", header=True)

That produces a single part file, which is convenient for a small summary and harmful for a large dataset, since one task then does all the writing. For big outputs, leave the partitioning alone, or organise files by a column with partitionBy:

orders.write.mode("overwrite").partitionBy("country").parquet("out/orders_by_country")

This creates sub-folders such as country=Egypt/, so later queries that filter on country read only the matching folders. Choose partition columns with a modest number of distinct values; partitioning by something unique like order_id creates millions of tiny files, a classic performance trap.

For data residency: if your employer in the Gulf or Egypt requires data to stay in-country, the same write call works against cloud object storage in a regional bucket, so "where the files land" is decided by the path you write to and the bucket's region, not by Spark.

Watching your job: the web interface

When a Spark session is alive, it serves a web interface at http://localhost:4040. If that port is busy, because another session is running, Spark takes 4041, then 4042, and so on. This page is how you stop guessing about what Spark did.

To use it, you need a session that stays alive long enough to look at. Add this to the end of a script, run it, and visit the address:

by_country.show()
input("Press Enter to stop Spark... ")
spark.stop()

The page has tabs. Jobs lists one entry per action you ran, so the show() above appears as a job, and a write appears as another. Click into a job and you reach Stages, which are the shuffle-separated sections, each with a count of tasks and the time they took. SQL / DataFrame shows the query plan as a diagram with row counts, which is the friendliest place to see where data shrinks or grows. Executors lists the processes (in local mode, just the driver), with memory use. Storage shows cached data, and Environment lists every configuration value in effect.

What to look for as a beginner: number of tasks in a stage (does it match your partition count?), the longest task compared with the median (a big gap hints that some partition has far more data than the others), and any red failed tasks. Do not try to read everything. Learn to answer "which stage was slow?", because that tells you which line of code to blame.

Spark records its own plan too, which is why explain() and the SQL tab agree. When the session ends, the web interface disappears. Keeping a record of finished applications requires event logging and a History Server, which belong in the Mid-level guide.

Lazy evaluation and ANSI mode: the two behaviours that surprise people

Two behaviours deserve their own section, because they produce the most "but that worked in pandas" confusion.

Laziness, revisited with an example. Run this and notice what is slow:

step = orders.filter(F.col("amount") > 100)      # returns instantly
step2 = step.withColumn("tax", F.col("amount") * 0.14)   # returns instantly
print(step2.count())                              # the work happens here

The first two lines return immediately because they only extend the plan. The third line executes everything. A consequence is that errors can appear far from their cause: a typo in the column name on line one raises an error when Spark analyses the plan, which for DataFrames is usually at line one, but a data problem such as a malformed value only appears when an action runs. Another consequence is repetition: if you call two actions on the same DataFrame, Spark recomputes the plan from the source for both. For an expensive DataFrame that you will reuse, you can mark it for caching with step2.cache() and release it with step2.unpersist(). Caching is an optimisation for later; for now, know that it exists and that recomputation is the default.

ANSI mode. Since Spark 4.0, spark.sql.ansi.enabled is true by default. ANSI mode means Spark follows the SQL standard strictly: invalid operations raise errors instead of quietly returning null. Dividing by zero, casting the text abc to a number, and arithmetic overflow all fail loudly. Try it:

spark.sql("SELECT 1/0").show()

On Spark 4 this raises an error whose class is DIVIDE_BY_ZERO, and the message itself tells you what to do: use try_divide to get null instead, or turn the setting off. Similarly, SELECT CAST('abc' AS INT) raises a CAST_INVALID_INPUT error. If you have used Spark 3, where these returned null and carried on, this is the biggest behaviour change you will notice. Loud failures are usually a good thing for data quality, because bad rows no longer slip through as nulls.

You have two sane ways to deal with it. Use the tolerant functions when you expect bad values:

spark.sql("SELECT try_divide(10, 0) AS safe_division, try_cast('abc' AS INT) AS safe_cast").show()

Both return null rather than failing. Or, for old code you cannot change yet, set the legacy behaviour for the session:

spark.conf.set("spark.sql.ansi.enabled", "false")

Prefer the first approach in new code. Turning ANSI off hides exactly the problems it was introduced to expose, and a fix that makes the error vanish is not the same as a fix that makes the data correct.

Configuration you will actually touch

Spark has hundreds of settings, and you will use a handful. They are set in three places. In code, with builder.config("key", "value") before getOrCreate(), or at runtime with spark.conf.set for SQL settings. On the command line, with --conf key=value passed to spark-submit or pyspark. And in a file, conf/spark-defaults.conf, in a full Spark distribution. When the same key appears in several places, the one set in code wins over command-line flags, which win over the defaults file.

A short list worth knowing as a beginner:

The shuffle partitions default of 200 is a good example of why defaults need thought. On a laptop with a tiny dataset, 200 tiny tasks after each shuffle is wasteful, and you may notice small jobs take longer than they should. For learning, lowering it is reasonable:

spark = (SparkSession.builder
         .appName("orders")
         .master("local[*]")
         .config("spark.sql.shuffle.partitions", "8")
         .getOrCreate())

Adaptive query execution, on by default, already merges small post-shuffle partitions in many cases, so the effect is smaller than it once was, but eight is a sensible number on a laptop. Memory deserves a note too. The driver in local mode is the whole application, so if you are processing a file larger than your RAM you may need to raise it, and for the driver this must be set before the JVM starts: pass --driver-memory 4g on the command line, or --conf spark.driver.memory=4g. Setting it with builder.config after Python has already launched Spark's JVM in an interactive shell does not take effect.

You can inspect any live setting with spark.conf.get("spark.sql.shuffle.partitions"), and the web interface's Environment tab lists them all.

Common errors and how to read them

A Spark error is usually long, because it carries a Python traceback on top of a Java stack trace. Read from the bottom up, and look for the part that starts with a bracketed error class or a line after Caused by:. The real message is nearly always short and near the end.

Walk through one example to practise. You run orders.select("amout") with a typo. The error begins with a Python AnalysisException, then the bracketed class UNRESOLVED_COLUMN.WITH_SUGGESTION, a sentence saying that a column with name amout cannot be resolved, and a "Did you mean one of the following?" list containing amount. The suggestion is the answer, and you found it in the first lines because Spark's current errors are designed to be read, not just searched.

Another worth rehearsing: the file path error. If you write spark.read.csv("order.csv") and the file is orders.csv, the error says the path does not exist. Relative paths resolve from the folder you launched Python in, not from where your script lives, so a script that works in one terminal folder can fail in another. When in doubt, print os.getcwd() or use an absolute path.

And one that looks scary but is not: lots of WARN and INFO lines at startup. These are log messages, not failures. If they drown your output, lower the noise with spark.sparkContext.setLogLevel("WARN") after the session starts; the call is available in a classic session and is simply a convenience here.

Spark next to pandas: when each is the right tool

Most beginners arrive from pandas, so it helps to say plainly how the two relate. pandas runs in one process on one machine and holds the whole table in memory; it is superb for data that fits comfortably in RAM, with a huge ecosystem and immediate feedback. Spark exists for the case where one machine is not enough, or where a job is slow enough that spreading it over many cores is worth the setup.

The practical rule for a student is simple. If your data is a few hundred megabytes and you are exploring, pandas is faster to start and easier to debug, because there is no session to launch and no plan to wait for. If your data is several gigabytes or more, arrives as many files, or has to be processed on a schedule by a team, Spark is the better fit. Spark also has a startup cost: even a tiny query takes a second or two longer than pandas because of the Java process and the planning step. Do not conclude from a ten-row test that Spark is slow; the overhead is fixed, and it is repaid only at scale.

There is a bridge between the two worlds. The Pandas API on Spark, imported as pyspark.pandas, lets you write pandas-style code that runs as Spark jobs; it needs the pyspark[pandas_on_spark] extra. And any small Spark result can be turned into a pandas DataFrame with toPandas(), which is the right way to plot a summary after Spark has shrunk a billion rows down to a hundred. The key word is small: aggregate first, convert second.

summary_small = by_country.limit(100).toPandas()   # safe: at most 100 rows reach the driver
print(summary_small.head())

Notice the order. The heavy lifting (filtering, joining, grouping) happens in Spark, close to the data, and only the tiny final table crosses into pandas. Reversing it, by converting first and aggregating after, throws away everything Spark is good at, and it is the usual cause of the out-of-memory errors listed in the previous section.

Putting it all together

Here is one small end-to-end project that uses every piece. It reads the orders, cleans them, computes revenue per country and category, joins region names, and writes Parquet. Put it in a file called pipeline.py and run it with python pipeline.py.

from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import (StructType, StructField, StringType,
                               IntegerType, DoubleType, DateType)

spark = (SparkSession.builder
         .appName("orders-pipeline")
         .master("local[*]")
         .config("spark.sql.shuffle.partitions", "8")
         .getOrCreate())

schema = StructType([
    StructField("order_id", IntegerType()),
    StructField("customer", StringType()),
    StructField("country", StringType()),
    StructField("category", StringType()),
    StructField("amount", DoubleType()),
    StructField("order_date", DateType()),
])

orders = spark.read.csv("orders.csv", header=True, schema=schema)

clean = (orders
    .dropna(subset=["order_id", "amount"])
    .filter(F.col("amount") > 0)
    .dropDuplicates(["order_id"]))

regions = spark.createDataFrame(
    [("Egypt", "North Africa"), ("Saudi Arabia", "Gulf"),
     ("UAE", "Gulf"), ("Jordan", "Levant")],
    schema=["country", "region"])

summary = (clean
    .join(regions, on="country", how="left")
    .groupBy("region", "country", "category")
    .agg(F.count("*").alias("orders"),
         F.round(F.sum("amount"), 2).alias("revenue"))
    .orderBy(F.desc("revenue")))

summary.show(truncate=False)
summary.write.mode("overwrite").parquet("out/summary")

print("rows written:", spark.read.parquet("out/summary").count())
spark.stop()

Walk through it as Spark does. The read and every transformation until summary.show build a plan without touching data. show is the first action: it triggers a job, and the plan contains one shuffle for the join (possibly a broadcast, which Spark chooses automatically for a small table like regions) and one for the group by. The write is a second job that recomputes the same plan, because we did not cache summary. The final line reads the Parquet back as a sanity check, a third job. You can open the web interface, if you add a pause before stop(), to see all three.

Notice the habits embedded in the file. The schema is explicit, so bad input fails loudly. Cleaning steps are separate and named, so you can inspect clean on its own. The aggregate columns use alias, so downstream readers get meaningful names. And the output goes to Parquet, not CSV, so types survive.

This is also the shape of production Spark jobs: read, clean, join, aggregate, write, with an orchestrator scheduling the script. When you reach that stage, tools like Airflow (/student-guides/airflow) or Dagster (/student-guides/dagster) run the script on a schedule, and the same code can be submitted to a cluster with spark-submit.

What you can now do, and what comes next

You can now install Spark and verify it, explain the driver, executors, partitions, transformations and actions in your own words, read CSV, JSON and Parquet data, apply an explicit schema, filter, select, add columns, group, aggregate and join, express the same logic in SQL, write results to Parquet, read the web interface to find the slow stage, and decode the common error messages. You also know the two traps that bite beginners hardest: pulling everything to the driver with collect(), and forgetting that Spark 4 has ANSI mode on.

The things we deliberately left for later are the ones that matter once your data and your team grow. Mid-level covers how Spark executes a query in detail, shuffles and partition sizing, broadcast joins and skew, caching rules, spark-submit against real clusters, Structured Streaming, MLlib, Spark Connect and testing. Senior covers running Spark on Kubernetes as a platform: security, cost, upgrades, multi-tenancy and incident playbooks.

Natural neighbours in this catalogue: store and version data with Delta Lake (/student-guides/delta-lake), track the models you train from Spark-built features with MLflow (/student-guides/mlflow), and package your jobs with Docker (/student-guides/docker) before running them on Kubernetes (/student-guides/kubernetes).

Your next practical step: take a dataset you genuinely care about, anything from a few hundred thousand to a few million rows, and rebuild one analysis you already do in pandas or a spreadsheet. Compare the results, then look at the web interface to see what Spark did. That comparison is the fastest way to build intuition.

Sources

• Apache Spark documentation overview and requirements: https://spark.apache.org/docs/latest/
• Downloads and release information: https://spark.apache.org/downloads.html
• PySpark installation: https://spark.apache.org/docs/latest/api/python/getting_started/install.html
• SQL migration guide (ANSI mode and other behaviour changes): https://spark.apache.org/docs/latest/sql-migration-guide.html
• Core migration guide: https://spark.apache.org/docs/latest/core-migration-guide.html
• Performance tuning (adaptive execution, shuffle partitions, broadcast): https://spark.apache.org/docs/latest/sql-performance-tuning.html
• Monitoring and the web interface: https://spark.apache.org/docs/latest/monitoring.html
• Submitting applications: https://spark.apache.org/docs/latest/submitting-applications.html
• Configuration reference: https://spark.apache.org/docs/latest/configuration.html

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask. Mid-level reviews this plus its own material; Senior reviews all three. The target is Apache Spark 4.2, so where an answer changed in Spark 4, it says so.

The thirty-second answer

Apache Spark is a distributed data processing engine. You describe a computation with DataFrames or SQL, and Spark splits the data into partitions, runs tasks on those partitions in parallel across the cores of one machine or the executors of a cluster, and combines the results. It is lazy: transformations only build a plan, and an action such as count or write triggers the optimised execution.

Then the line that shows you have used it: "most of the cost is in shuffles, so the first thing I look for in a slow job is the stage after a join or group by."

The model

Learn those ten rows well enough to say them without looking. Every question below is a variation on one of them.

The commands and code to have ready

You will not be asked to recite documentation, but you should be able to write these from memory on a whiteboard or in a shared editor.

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder
         .appName("demo").master("local[*]").getOrCreate())

df = spark.read.csv("data.csv", header=True, inferSchema=True)
df.printSchema()
df.show(5)

out = (df.filter(F.col("age") > 18)
         .withColumn("age_next", F.col("age") + 1)
         .groupBy("country")
         .agg(F.count("*").alias("n"), F.avg("age").alias("avg_age"))
         .orderBy(F.desc("n")))

out.write.mode("overwrite").parquet("out/")

And the SQL equivalent of the same idea:

df.createOrReplaceTempView("people")
spark.sql("SELECT country, count(*) AS n FROM people GROUP BY country").show()

From the shell, the things worth knowing by name:

pip install pyspark
spark-submit --version
./bin/pyspark --master "local[2]"
./bin/spark-submit job.py

Also know that the web interface is at http://localhost:4040 while a session is alive, and that it moves to 4041 and 4042 if the port is taken.

The classic traps, and why they happen

The collect() trap. collect() and toPandas() bring all rows to the driver. On large data the driver runs out of memory. The cause is that distributed data has been forced into one process. The fix is to aggregate or limit first, or to write results to storage.

The "nothing happened" trap. Beginners write a long transformation chain, see no output and no error, and assume it ran. It did not: there was no action. Laziness is the cause.

The immutability trap. df.withColumn("x", ...) does not change df; it returns a new DataFrame. Forgetting to assign the result means the change is lost.

The ANSI mode trap. In Spark 4.0 and later, spark.sql.ansi.enabled defaults to true. Division by zero raises DIVIDE_BY_ZERO and a bad cast raises CAST_INVALID_INPUT, where Spark 3 returned null. The fix is to use try_divide and try_cast, or clean the data; switching ANSI off is a compatibility measure for old code.

The small-data overhead trap. A tiny query in Spark can take longer than in pandas, because of JVM startup and planning, and because the default spark.sql.shuffle.partitions is 200. This is not a sign that Spark is broken; it is a fixed cost that only pays off at scale.

The environment trap. PySpark needs a JDK (17, 21 or 25 for Spark 4.2) and Python 3.10 or newer. A missing JAVA_HOME is the most common failure on a first run.

The relative path trap. spark.read.csv("orders.csv") resolves relative to the directory where you launched Python, not where the script lives.

Common interview questions

What is Spark, and why would you use it instead of pandas? Spark is a distributed engine for processing large datasets in parallel. I would use it when the data does not fit in one machine's memory, arrives as many files, or a job is too slow on one core. For data that fits in memory and is explored interactively, pandas is simpler and faster to start. The two are often combined: heavy aggregation in Spark, then a small result converted to pandas for plotting.

Explain the architecture. An application has one driver and many executors. The driver runs my code, builds the logical plan, optimises it and splits it into stages and tasks. The cluster manager (Standalone, YARN or Kubernetes) provides resources, and executors run the tasks and cache data. In local mode all of this happens in one process on my machine, which is how I develop.

What is the difference between a transformation and an action? A transformation, like select, filter or groupBy, builds a new DataFrame lazily and does no work. An action, like count, show or write, triggers execution of the whole accumulated plan. A single action creates a job.

What is lazy evaluation and why does it matter? Spark records transformations as a plan and only runs it on an action. It matters because the optimiser sees the entire plan, so it can prune columns and push filters early. It also means errors from data can show up late, and that calling two actions on the same DataFrame recomputes the plan twice unless I cache it.

What is a partition? A partition is a slice of a DataFrame that a single task processes. The number of partitions limits parallelism: ten partitions can use at most ten cores at once. After a shuffle the count is controlled by spark.sql.shuffle.partitions, which defaults to 200.

What is a shuffle, and why is it expensive? A shuffle redistributes data across executors so that rows with the same key meet, for a join or a group by. It writes intermediate data to disk and sends it over the network, so it is usually the slowest part of a job. It also marks a stage boundary. In a plan, it appears as an Exchange.

DataFrame, Dataset or RDD? A DataFrame is a distributed table with a schema, and it is the main API because the optimiser understands it. A Dataset is the typed version available in Scala and Java. An RDD is the older low-level distributed collection, without optimisation. In Python I use DataFrames, and I would only reach for RDDs for rare low-level work. Note also that RDDs are not supported in Spark Connect.

What does inferSchema do and why avoid it in production? It makes Spark scan the file to guess column types. It costs an extra pass and can guess wrong, for example treating a column as text because of one bad value. In production I pass an explicit schema, which is faster and makes bad input fail loudly.

Why is Parquet preferred over CSV? Parquet is columnar, compressed and carries its schema, so it is smaller, keeps types and lets Spark read only the columns a query needs. CSV is text with no types and must be parsed in full.

What does mode("overwrite") do, and what are the other modes? It replaces whatever exists at the target path. The others are append, ignore and the default, errorifexists. It is a destructive operation, so I check the path first.

What are the files in an output directory? A Spark write produces a directory, not a file. Each part- file is written by one task for one partition, and an empty _SUCCESS file signals the write finished. That is why readers accept a directory path.

repartition versus coalesce? Both change the number of partitions. repartition(n) can increase or decrease and performs a full shuffle; I can also give it columns to partition by key. coalesce(n) only reduces and avoids a full shuffle, so it is cheaper, but it can leave partitions unevenly sized.

How do you see what a job is doing? Open the web interface on port 4040. The Jobs and Stages tabs show tasks and timing, the SQL tab shows the plan with row counts, and Environment lists the configuration. I also use df.explain() to read the plan and look for Exchange.

What changed in Spark 4 that a beginner should know? ANSI SQL mode is on by default, so invalid casts and divide by zero fail instead of returning null. Spark 4 requires Java 17 or newer, builds only for Scala 2.13, and has dropped Mesos. Tutorials written for Spark 3 may show different results.

How do you handle bad values when casting? With ANSI mode on, I use try_cast or try_divide where nulls are acceptable. Where they are not, I clean or validate the data first, rather than disabling ANSI mode and hiding the problem.

How do you run a script outside the shell? Locally, with python script.py after installing PySpark, or with spark-submit script.py from a full Spark distribution. On a cluster, spark-submit takes --master to choose the cluster manager and options like --conf for settings.

You run df.withColumn(...) and the column is not there. Why? DataFrames are immutable. withColumn returns a new DataFrame, so I must assign it: df = df.withColumn(...).

When would you cache a DataFrame? When I reuse an expensive DataFrame in several actions, using df.cache() and df.unpersist() when done. Caching is not free: it uses executor memory, so I only do it when I have proved recomputation is the cost.

What does spark.sql.shuffle.partitions do? It sets how many partitions a shuffle produces, 200 by default. Too many on small data wastes effort on tiny tasks; too few on large data creates huge tasks. Adaptive query execution, on by default, can merge small post-shuffle partitions automatically.

What is adaptive query execution? A runtime optimiser, enabled by default via spark.sql.adaptive.enabled, that revises the plan using statistics from completed shuffle stages. For example, it can coalesce small partitions and mitigate skewed joins. As a beginner I only need to know it exists and is on.

What is a broadcast join? A join strategy in which a small table is sent to every executor, so the large side does not have to be shuffled. Spark chooses it automatically for tables under spark.sql.autoBroadcastJoinThreshold, 10 MB by default, and I can request it with the broadcast() hint.

Where does the data live in a Gulf or Egyptian deployment where residency matters? Spark does not decide that; the storage path does. If the input and output paths point at a bucket or file system in the required region, the data stays there, and executors read and write it where it lives. I would confirm where intermediate shuffle files and logs go as well.

Questions you should ask them

An interview goes both ways, and good questions signal that you know what real Spark work looks like. Ask which cluster manager the team runs (Kubernetes, YARN or a managed service), which Spark version they are on and whether they are planning the Spark 4 move, how jobs are scheduled and tested, where the data lives, and how they track job cost. Ask also whether they use Spark SQL or DataFrames more, and how they handle schema changes in incoming files. Each of those shows that you understand the job is mostly about reliable data movement rather than clever code.

How to explain a hands-on answer

If asked "tell me about something you built", structure it in four sentences. State the input (for example, eight months of order CSV files). State the transformation (cleaned, joined with a regions table, aggregated per country). State the output (Parquet partitioned by country). Finish with a number or an observation from the web interface: "the group by stage was the longest, so I lowered shuffle partitions from 200 to 8 for my laptop runs and it dropped from a minute to a few seconds." Specific details from a real project beat a perfect definition.

If you have only done exercises, say so and describe the one you understood best. A candidate who can explain why a groupBy creates a second stage is more convincing than one who lists ten APIs.

Whiteboard exercises to practise

Expect one of these, and practise writing them without help.

1. Top N per group. From an orders table, show total revenue per country, sorted descending, top three. This is groupBy, agg, orderBy and limit, or the same in SQL with GROUP BY ... ORDER BY ... LIMIT 3.
2. Clean then summarise. Drop rows where amount is null, remove duplicate order_ids with dropDuplicates, and compute the average amount per category.
3. Join with a lookup. Attach region names from a small table with a left join, then count orders per region. Be ready to say which side Spark could broadcast, and why.
4. Explain a plan. You are shown an explain() output with an Exchange in it. Say what caused it and how many stages result.
5. Fix the memory error. A script calls toPandas() on a billion-row DataFrame and crashes with a heap space error. Describe how you would restructure it.

For each, narrate as you write: what the input looks like, which operations are lazy, where a shuffle occurs, and what the output is. Narrating is itself a skill the interviewer is scoring.

Sixty-second self-test

Answer these aloud without looking back. If you hesitate on more than two, re-read the Beginner Detailed guide.

Mid-level takes every one of those topics further: how a query goes from SQL to physical plan, sizing partitions, broadcast joins and skew handling, the caching rules, spark-submit options and configuration precedence, Structured Streaming with checkpoints, MLlib pipelines, Spark Connect, and testing Spark code. For continuity with the rest of the catalogue, see also the guides on /student-guides/delta-lake and /student-guides/airflow.

Senior then covers what you own when Spark is a platform: running on Kubernetes, memory and overhead tuning, dynamic allocation, security including the lack of a built-in UI authentication, secrets, observability with the History Server and metrics sinks, multi-tenancy, cost control, upgrades between major versions and the incident playbooks that go with them.

Part one of three. Most beginner Spark problems come from four sources: the Java and Python environment, forgetting that Spark is lazy, pulling too much data to the driver, and carrying habits over from pandas or from Spark 3 tutorials. Start with the error table, then read the mistakes in detail, then the habits and the checklist. The guide targets Apache Spark 4.2.

Common errors at this level

Mistake 1: Treating Spark like pandas

Symptom. Your code runs, but it is painfully slow, or it crashes on bigger files. You loop over rows in Python, call collect() to inspect things, or convert to pandas early.

Cause. pandas works on data that sits in your process. Spark's data is spread out, so any step that gathers it all into one place defeats the purpose and can exhaust the driver's memory. A Python loop over rows also runs on the driver, one row at a time, while the parallel machinery sits idle.

Fix. Express the work as column operations that Spark can distribute: filter, withColumn, groupBy, join, and the functions in pyspark.sql.functions. When you need a small result in Python, shrink first and convert second:

small = df.groupBy("country").count().limit(50).toPandas()

If you catch yourself writing for row in df.collect():, stop. There is almost always a built-in function that does it in parallel.

Mistake 2: Expecting code to run when you write it

Symptom. You write ten lines of transformations and nothing seems to happen. Or the line that fails is not the line you edited.

Cause. Spark is lazy. Transformations record a plan; only an action (show, count, collect, write) executes it. Errors that depend on the data therefore appear at the action, which may be many lines below the transformation that caused them.

Fix. When debugging, put a cheap action after each step you doubt, such as df.limit(5).show(), and move it forward until the error appears. Remove the extra actions afterwards, because every action reruns the plan from the source unless the data is cached.

Mistake 3: Losing a transformation to immutability

Symptom. You called withColumn, drop or filter and the next show() looks unchanged.

Cause. DataFrames never change in place. Each method returns a new DataFrame, and the old one is untouched.

Fix. Assign the result, or chain the calls:

orders = orders.withColumn("amount_rounded", F.round("amount", 0))

A useful self-check: if a line of Spark code is a bare method call that is not an action, it is probably a bug.

Mistake 4: Running out of memory by pulling data to the driver

Symptom. java.lang.OutOfMemoryError: Java heap space, often right after collect(), toPandas() or a show(10000000).

Cause. All of those bring rows into the driver's process. The driver's memory is a small fraction of a cluster's, and in local mode it is also the whole application.

Fix. Never collect an entire large DataFrame. Aggregate, filter or limit first, or write the result to Parquet and read it where you need it. If you genuinely need more driver memory, give it at launch, since it cannot be changed after the JVM has started:

spark-submit --driver-memory 4g job.py

Mistake 5: Not setting a schema, then trusting inferred types

Symptom. A numeric column is read as text, dates arrive as strings, or a column changes type between files. Aggregations return odd results or null.

Cause. inferSchema=True samples the file and guesses. One malformed value turns a number column into a string column, and it costs an extra pass over the data.

Fix. Use inferSchema for exploration only. For anything repeated, declare a StructType and pass it to the reader, so unexpected input fails loudly instead of changing your data's meaning. Check the result with df.printSchema() before doing anything else.

Mistake 6: Hitting ANSI mode and switching it off blindly

Symptom. A query that worked in an older tutorial now fails with DIVIDE_BY_ZERO or CAST_INVALID_INPUT.

Cause. Spark 4 turned spark.sql.ansi.enabled on by default. Operations that used to return null on bad input now raise errors.

Fix. Decide whether the bad value is expected. If it is, use try_divide or try_cast so it becomes null deliberately. If it is not, you have found a data quality problem, and the error did its job: fix the data. Setting spark.sql.ansi.enabled to false restores the old behaviour, which is reasonable for legacy code you cannot change yet, but it hides problems.

Mistake 7: Overwriting the wrong path

Symptom. Data you wanted is gone after a rerun, or the write fails because the target exists.

Cause. mode("overwrite") replaces everything at the path, and the default mode refuses to write to an existing path. Both are correct behaviours, applied to the wrong expectation.

Fix. Write into a dedicated output folder that holds only this job's output, never into a directory that contains source data. Print the path before the write when running something new. Use append only when you really mean to add data, and know that rerunning an append job duplicates rows.

Mistake 8: Writing a single huge file, or thousands of tiny ones

Symptom. A coalesce(1) write is slow on big data, or a partitioned write creates an enormous number of small files and later reads crawl.

Cause. One output file means one task does all the writing. Partitioning by a column with many distinct values, such as an id, creates a folder and file for each value.

Fix. For small summaries, coalesce(1) is fine and convenient. For large data, let Spark write several part files. Use partitionBy only on a column with a modest number of values, like a country or a date, and one that your later queries filter on.

Mistake 9: Copying tutorials from the wrong Spark generation

Symptom. Imports fail, jars cannot be found, results differ from the article, or setup steps mention things your install does not have.

Cause. Spark 4 changed things. It needs Java 17 or newer, supports only Scala 2.13, no longer supports Mesos, and defaults to ANSI mode. Older tutorials also teach RDDs first, where DataFrames are the modern path.

Fix. Look at the date and the Spark version in the first lines of any tutorial. Check yours with spark.version. When a _2.12 artifact, a javax servlet reference or an RDD-first walkthrough appears, treat it as Spark 3 material.

Mistake 10: Misreading the web interface, or never opening it

Symptom. A job is slow and you guess at causes, adding settings you found online.

Cause. Spark tells you where time went, in the web interface, but only while the session is alive. Many beginners let the script end before looking.

Fix. Add a pause at the end of a script while you learn:

input("Press Enter to stop Spark... ")
spark.stop()

Then open http://localhost:4040. Find the slowest stage, compare the longest task to the median, and read the plan in the SQL tab. If one task is far slower than the rest, a partition is oversized; if there are hundreds of tiny tasks, there are too many partitions for the data.

Mistake 11: Starting several sessions, or forgetting to stop one

Symptom. Port warnings, memory pressure on a laptop, or a notebook that behaves differently after a rerun.

Cause. Each session is a JVM. Several can run at once, each taking memory and a web port.

Fix. Always use getOrCreate(), and call spark.stop() when a script is done. In notebooks, restart the kernel when settings that must be applied at launch, such as driver memory, need to change.

Mistake 12: Ignoring the Python environment

Symptom. The code works in one terminal but not another, or the worker complains about Python versions.

Cause. The virtual environment is not active, or the system Python differs from the one PySpark imports.

Fix. Activate your .venv in every new terminal. If Spark still picks another interpreter, export both variables:

export PYSPARK_PYTHON=$(which python)
export PYSPARK_DRIVER_PYTHON=$(which python)

Run them after activating the environment, so which python points at the environment's interpreter.

Habits worth adopting on day one

Declare schemas for any file you will read twice. It is the habit that prevents the most silent bugs and costs ten lines.

Look before you change. Before adding a setting, open the web interface and name the slow stage. Changing configuration without measuring teaches you nothing, because you cannot tell whether it helped.

Read errors bottom-up. Find the bracketed error class and the Caused by: line. Search for those exact words.

Prefer built-in functions. The functions in pyspark.sql.functions are optimised and run inside the engine. A custom Python function applied row by row is much slower, because every row crosses between the JVM and Python. Search the function list before writing your own.

Write Parquet, not CSV, for anything you will read again. You keep types, you save space, and later reads touch fewer columns.

Name things. appName shows in the interface; alias names columns in aggregations. Both cost nothing and save time later.

Keep one small input for fast iteration. Develop against a thousand-row sample, then run on the full data once the logic is right. The code does not change.

Keep a pandas answer as a check. For a small dataset, compute the same figure with pandas or a spreadsheet, and compare it with Spark's number. Agreement is real evidence that your pipeline does what you think.

Pin your versions. Write pyspark==<version> in a requirements file with the version you tested, so a colleague gets the same behaviour. Check the downloads page for the current release before you choose.

Mind where data lives. If your employer has data-residency rules, such as requirements to keep customer data inside Saudi Arabia, the UAE or Egypt, decide the storage location before you write the first job. Spark reads and writes wherever the path points, so the rule has to be enforced in your paths and bucket choices, not assumed.

Practice cards

Each card takes ten minutes. Do them in order.

Card 1: the environment. Create a virtual environment, install PySpark, run the one-line smoke test, then break it on purpose by unsetting JAVA_HOME and read the error. Fix it again.

Card 2: laziness. Build a three-step transformation chain and time it. Add a count() and time again. Write down which line took the time and why.

Card 3: schema. Take any CSV you have. Read it with inferSchema, print the schema, then write an explicit schema and compare. Introduce one bad value in a numeric column and watch the two approaches behave differently.

Card 4: ANSI. Run SELECT 1/0 and a bad cast, read both errors, then rewrite each with try_divide and try_cast.

Card 5: the plan. Run a groupBy, call explain(), and find Exchange. Open the web interface and find the same shuffle as a stage boundary.

Card 6: the output. Write a result as Parquet, list the folder, read it back, and check the schema. Then write it partitioned by a column and see the folders.

Card 7: the bridge. Aggregate in Spark, convert the small result with toPandas(), and plot it. Explain to a friend why the order of those steps matters.

When to ask for help, and how

There comes a point where reading the error yourself stops paying off, and the skill is recognising it early. If you have checked the environment, read the bottom of the error, and confirmed the plan and the data, it is time to ask. A good question carries four things: the Spark version (print spark.version), the Java and Python versions, the smallest piece of code that still fails, and the exact error text starting at the bracketed class. With those four, a colleague or a community forum can usually answer in one reply, instead of three rounds of clarifying questions.

Shrinking the example is itself a debugging tool. Replace your real file with a five-row DataFrame built by spark.createDataFrame, remove steps until the failure disappears, and the last step you removed is the cause. Half of all questions answer themselves during this exercise. Keep the habit even when you are not asking anyone, because it turns a mysterious failure over millions of rows into a clear one over five.

A short checklist

Before you run a new Spark job, tick these:

• Is java -version showing Java 17, 21 or 25, and is JAVA_HOME set?
• Is the virtual environment active, and does spark.version print what you expect?
• Does the script set a master (local[*] for local runs)?
• Is the schema declared for files you read repeatedly?
• Does every transformation chain end in an action, and is every result assigned?
• Is anything calling collect() or toPandas() on data that might be large?
• Is the output path a folder you are happy to overwrite?
• Have you handled bad values with try_cast or try_divide where nulls are acceptable?
• Is spark.stop() called when the script ends?
• Did you open http://localhost:4040 at least once to see what the job did?

When something fails, work through the same order: environment first, then the bottom of the error message, then the plan, then the data. Most beginner problems are found in the first two. When you outgrow this list, the Mid-level guide covers partition sizing, joins, caching and running on real clusters, and the catalogue's guides on /student-guides/docker and /student-guides/mlflow are natural next stops for packaging your jobs and tracking what they produce.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. It is written for Spark 4.2, the current line at the time of writing, so the defaults and breaking changes below are the ones you will meet on a new cluster. Where you already know the basics, we go straight to the depth.

Where this picks up

Each section states the problem it solves first and finishes with a Try it you can run on a laptop in a few minutes. Every example uses PySpark. The behaviour is the same in Scala, and the configuration keys are identical.

From query to tasks: how a job is actually built

The Beginner level gave you the vocabulary: application, job, stage, task. At work, you need the mechanism, because every performance question reduces to "where did the stage boundary land and how skewed were its tasks".

When you write DataFrame code, nothing runs. Each transformation adds a node to a logical plan, an unresolved description of what you asked for. When an action arrives (count, write, collect, show), the Catalyst optimiser takes over in four passes. It resolves names against the catalog, producing an analysed plan; it rewrites that plan with rules such as predicate pushdown and column pruning, producing the optimised logical plan; it chooses physical operators, for example a sort-merge join or a broadcast hash join, producing one or more physical plans and picking one; and it generates code for the chosen plan. The result is a tree of physical operators that the scheduler cuts into stages.

The cut points are shuffle boundaries. Operators that can run on a partition of data in isolation (filter, project, a map-side step) are pipelined together into one stage. An operator that needs rows with the same key to meet (a join, a groupBy, a repartition, a window over a partition key) forces an exchange: every task in the producing stage writes its output split by key, and every task in the consuming stage reads its slice from all of them. That exchange is the shuffle. It writes to local disk, moves data across the network, and is the single largest cost in most jobs.

Two consequences let you predict behaviour before you run anything. First, the number of tasks in a stage equals the number of partitions it processes: for a file scan, that is driven by how the files split; after a shuffle, by spark.sql.shuffle.partitions (default 200) unless adaptive execution coalesces them. Second, a stage cannot start until the stages it reads from have finished, so one slow task in a stage holds up everything behind it. Cluster time is spent waiting on the slowest task far more often than on aggregate throughput.

You read the plan with explain. The formatted mode is the most readable because it numbers each operator and lists its details separately:

orders = spark.read.parquet("s3a://lake/orders/")
customers = spark.read.parquet("s3a://lake/customers/")

daily = (orders.join(customers, "customer_id")
               .groupBy("country", F.to_date("ts").alias("day"))
               .agg(F.sum("amount").alias("revenue")))

daily.explain("formatted")   # numbered operators, then a details section per operator
daily.explain(True)          # parsed, analysed, optimised logical plans, then the physical plan

Read a physical plan from the bottom up, the way data flows. The bottom nodes are scans (FileScan parquet), and the scan details are where you confirm that pushdown worked: PushedFilters lists the predicates handed to the Parquet reader, and ReadSchema lists only the columns that are actually read. Above them you will see Exchange nodes (a shuffle, with its partitioning such as hashpartitioning(customer_id, 200)), then join operators (BroadcastHashJoin or SortMergeJoin), then HashAggregate pairs, one before the exchange for the partial aggregate and one after for the final merge. A plan with an AdaptiveSparkPlan node at the top has adaptive execution on, which means the printed plan is the initial one and the final shape is only known after running. The SQL tab in the UI shows the final plan after adaptation.

Reading the Spark UI like a profiler

The UI at http://localhost:4040 (or the driver's address on a cluster) is the profiler that ships with Spark. Most people look only at the Jobs tab, which tells you almost nothing. The useful path is SQL, then Stages, then Executors.

Start in the SQL tab, which lists each query with its duration and, when you open one, a plan graph annotated with runtime metrics: rows output per operator, time in each, spill sizes, and for shuffles the data size written. This is where you find which operator is expensive. A scan that outputs a billion rows and a filter above it that keeps ten thousand tells you the filter did not push down. A join whose output row count far exceeds both inputs tells you the key is not unique on one side and you have a row explosion, which no amount of memory tuning will fix.

Next, open the stage that the slow operator belongs to. The Stages tab shows summary metrics across tasks as quantiles: minimum, 25th percentile, median, 75th, maximum. Read the gap between median and maximum. If the median task takes two seconds and the maximum takes four minutes, you have skew, one partition holding far more data than the rest. If all tasks are slow and evenly so, you have too little parallelism or too much work per row. Look at three columns in particular: Shuffle Read Size per task (skew shows here first), Spill (memory) and Spill (disk) (tasks ran out of execution memory and wrote to disk), and GC Time (if it is more than a small fraction of task time, the executors are under memory pressure).

The Executors tab shows per-executor totals: task time, GC time, shuffle read and write, and failed tasks. One executor with far more failed tasks than the rest points at a bad node. Uniformly high GC time points at executor memory sizing. The Storage tab shows what is cached, in which storage level, and what fraction actually fits in memory, which is how you check that a cache() is doing what you think. The Environment tab lists the resolved configuration, and it is the authoritative answer to "what value did this job actually run with", because it reflects the merge of defaults, flags and code that the next section explains.

Shuffles, partitions and adaptive query execution

A shuffle has two ends. In the map side each task partitions its output by key and writes it to local disk; in the reduce side each task fetches its slice of every map output over the network. The cost is disk writes, network transfer, serialisation, and the sort or hash that organises the data. The number of reduce-side partitions is the number of tasks in the next stage.

Before adaptive query execution (AQE), you had to guess that number. The default of 200 for spark.sql.shuffle.partitions was wrong for almost every job: too many tiny tasks for small data, too few oversized tasks for large data. AQE, on by default through spark.sql.adaptive.enabled=true, fixes most of this by re-planning at shuffle boundaries using real statistics. After each shuffle stage finishes, Spark knows the exact size of every output partition, and it can adjust the rest of the plan.

AQE does three things you should know by name. Coalescing: adjacent small shuffle partitions are merged toward an advisory size, spark.sql.adaptive.coalescePartitions.enabled (true) with spark.sql.adaptive.advisoryPartitionSizeInBytes at 64 MB; a floor of spark.sql.adaptive.coalescePartitions.minPartitionSize (1 MB) stops it merging below that. Join strategy switching: if one side of a planned sort-merge join turns out small enough at runtime, the join is converted to a broadcast join. Skew handling: a partition that is both much larger than the median and larger than an absolute threshold is split into several tasks, which is the next section's subject.

One setting surprises people. spark.sql.adaptive.coalescePartitions.parallelismFirst defaults to true, which means coalescing prioritises keeping parallelism high and may ignore the advisory size, using the minimum partition size instead. If your jobs produce more small partitions than the advisory size suggests, set it to false so the 64 MB target is honoured.

spark.conf.set("spark.sql.adaptive.coalescePartitions.parallelismFirst", "false")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128m")

Then there are the places where you still choose partition counts yourself. repartition(n) or repartition(n, "key") performs a full shuffle to produce exactly n partitions, optionally hash-partitioned by a key. coalesce(n) only reduces the partition count by merging existing partitions without a full shuffle, which makes it cheap, and also makes it unable to rebalance: if the input was lopsided, the output is too. A common mistake is calling coalesce(1) before a write to get a single output file; that forces the whole upstream computation for that stage into one task, because narrow operators are pipelined into the same stage. If you need one file, accept a final repartition(1) so the upstream work stays parallel and only the last step funnels.

AQE has a 4.2 detail you should be aware of if you upgrade. Order-independent shuffle checksums are on by default in 4.2: they detect inconsistent data when an indeterminate shuffle stage is retried, roll back and re-run the dependent stages on a mismatch. If, after an upgrade, you see stages being re-run that were not before, compare behaviour by setting both spark.sql.shuffle.orderIndependentChecksum.enabled=false and spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch=false. Treat that as a diagnostic, not a fix: the feature exists because non-deterministic stages can silently corrupt results on retry.

Joins, broadcast and skew

Joins are where most slow jobs live, so it helps to know the strategies by cost. A broadcast hash join sends the small side to every executor and joins without shuffling the large side at all; it is by far the cheapest. A sort-merge join shuffles both sides by the join key, sorts each partition, and merges; it scales to any size but pays for two shuffles. Spark picks broadcast automatically when one side's estimated size is under spark.sql.autoBroadcastJoinThreshold, which defaults to 10485760 bytes (10 MB). With AQE the decision can also be made at runtime, once real sizes are known.

That default is conservative. If you have a dimension table of 200 MB and executors with several gigabytes, a broadcast is better than a shuffle of your fact table, and you can ask for it explicitly with a hint, which wins regardless of the threshold:

from pyspark.sql.functions import broadcast

enriched = events.join(broadcast(dim_customers), "customer_id")

# or raise the threshold for the session (value in bytes)
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", str(200 * 1024 * 1024))

Broadcasting has its own failure modes. The small side is collected on the driver and then sent to executors, so a table that is "small" in your head but large when decompressed can exhaust driver memory. When building the broadcast takes too long you get a TimeoutException: Futures timed out after [300 seconds], where 300 is the default for spark.sql.broadcastTimeout. The right fix is rarely to raise the timeout; check whether the table really is small and whether the upstream computing it is slow. If it is not small, remove the hint and let the sort-merge join run.

Skew is a join (or aggregation) key with a few hot values. All rows for one key land in one partition, so one task does most of the work. The UI shows it as a stage where the maximum shuffle read dwarfs the median. AQE's skew join handling is on by default through spark.sql.adaptive.skewJoin.enabled. A partition is considered skewed when it is larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor (5.0) times the median partition size and also larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (256 MB). Both conditions must hold, which is why a skew that exists but stays below 256 MB per partition is not split. If your partitions are hot at 100 MB, lower the threshold.

AQE handles skew only in sort-merge joins. For skewed aggregations, or when the hot key is something like a null or a default value that should not be joined at all, fix it in the data. Two options earn their keep. First, filter the hot key out and handle it separately: join the common rows normally, handle the rows with the hot key with a broadcast of the matching small slice, and union the results. Second, salting: add a random suffix to the key on the large side, replicate the small side across the same suffix range, join on the pair, and aggregate again. Salting is a heavy tool, and you should reach for it only after confirming skew in the UI.

Caching and the memory model

Caching solves one problem: a DataFrame is recomputed from its source every time an action needs it, so a DataFrame used by three actions is computed three times. df.cache() marks it to be kept the first time it is computed. df.persist(StorageLevel.MEMORY_AND_DISK) does the same with an explicit level, and df.unpersist() frees it. For SQL tables, spark.catalog.cacheTable("t") and spark.catalog.uncacheTable("t") do the equivalent. Caching is lazy: nothing is stored until an action runs, which is why a cache() followed by a count() is the standard way to materialise it.

Caching is not free and often not worth it. A DataFrame that feeds one action should never be cached; the storing costs more than recomputing. A DataFrame read from fast columnar storage with cheap transformations is often faster to re-read than to hold, because the cached copy competes for memory with execution. The cases where it pays are a DataFrame that is expensive to compute (many joins, an expensive UDF) and is reused by several actions, or an iterative algorithm that revisits the same data. Check the Storage tab: if "Fraction Cached" is below 100% for a memory-only level, partitions that do not fit are recomputed on every use, which can make the job slower than no cache at all. MEMORY_AND_DISK spills the remainder to disk instead.

To size executors sensibly you need the memory layout. An executor JVM has a heap (spark.executor.memory) and, outside the heap, an overhead region (spark.executor.memoryOverhead) that holds off-heap allocations, native libraries and Python worker processes. Inside the heap, spark.memory.fraction governs how much is a unified region shared between execution (shuffles, joins, sorts, aggregations) and storage (cached data), and spark.memory.storageFraction sets how much of that region is protected from eviction by execution. Execution can evict cached blocks when it needs room, but cached data cannot evict running execution memory, which is why a heavy cache can starve a join without ever causing an error: the join simply spills.

The failure that matters in practice is exceeding the container, not the heap. On Kubernetes the symptom is a pod killed with OOMKilled and exit code 137; on YARN, a message that the container was killed for exceeding memory limits. The JVM heap was fine; the sum of heap plus overhead plus Python workers was not. The fix is spark.executor.memoryOverhead, and for PySpark jobs with heavy Python-side work, spark.executor.pyspark.memory as well. Raising spark.executor.memory alone does not help and makes the container bigger without adding any overhead headroom.

File layout: partitioning, small files and safe writes

Most steady-state Spark cost is decided before the job starts, by how the data is laid out. Columnar formats (Parquet, and ORC) let Spark read only the columns you select and skip row groups using statistics, so writing to Parquet instead of CSV or JSON often matters more than any setting. Spark 4 changed the ORC default compression codec from snappy to zstd, and renamed the Parquet codec lz4raw to lz4_raw, which matters if old configuration sets it by name.

Partitioned writes put each distinct value of a column into its own directory, so queries filtering on it skip whole directories:

(daily.write
      .mode("overwrite")
      .partitionBy("day")
      .parquet("s3a://lake/gold/daily_revenue/"))

Choose partition columns with low cardinality that queries actually filter on: a date, a region. Partitioning by a high-cardinality column such as a user ID creates millions of tiny directories and files, and slows both writing and listing. The goal is files of a useful size, commonly hundreds of megabytes rather than kilobytes, though that figure is general guidance rather than a Spark rule.

The small-file problem is the usual symptom of getting this wrong. Each task writes its own file per output partition value, so a job with 200 shuffle partitions writing to 365 date directories can produce up to 73,000 files. Listing them is slow, opening them is slower, and the object store charges per request. The remedy is to control the number of writers per partition value: repartition by the partition column before writing so each value is handled by few tasks.

(daily.repartition("day")           # all rows for a day land in one task
      .write.mode("overwrite")
      .partitionBy("day")
      .parquet(path))

That pattern has the opposite risk, a single huge day and one oversized task, so measure before choosing. Spark 4 also changed spark.sql.maxSinglePartitionBytes from unbounded to 128m, which affects how single-partition plans are treated, and is one more reason to read the migration guide before an upgrade.

mode("overwrite") with partitionBy has a trap. By default it replaces the entire target path, not only the partitions present in your data. If you intended to rewrite one day and instead deleted the history, you have met it. Spark has a dynamic partition overwrite mode that replaces only the partitions it writes, selected by the spark.sql.sources.partitionOverwriteMode setting set to dynamic; confirm it behaves as you expect on a copy of the data before trusting it with real history. Even better, avoid the question: a table format such as Delta Lake gives you transactional overwrites and merges, covered in the Delta Lake guide, and it is the usual answer once multiple jobs write to the same dataset.

Idempotence is the property that matters for production: running a job twice must produce the same result as running it once. Overwriting a deterministic partition by its date key is idempotent; appending is not, so a retry duplicates rows. Design every batch job as "compute partition X, replace partition X".

UDFs, Arrow and the pandas API on Spark

A Python UDF solves a real problem, logic that built-in functions cannot express, at a real cost. Spark's engine runs on the JVM, so a row-at-a-time Python UDF serialises each row out to a Python worker process, calls your function, and serialises the result back. Catalyst also cannot see inside it, so it cannot push filters through it or reorder around it. That is why the rule at this level is: use built-in functions first, and treat every UDF as a performance decision.

The built-in pyspark.sql.functions module is large. Before writing a UDF for string manipulation, date arithmetic, conditionals, arrays or maps, search it. A when/otherwise chain, transform and filter on arrays, and regexp_extract cover a surprising share of what people write UDFs for.

When you do need Python, prefer an Arrow-based pandas UDF, which moves batches of rows between the JVM and Python in columnar form and operates on whole pandas Series at once:

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def zscore(v: pd.Series) -> pd.Series:
    return (v - v.mean()) / v.std()

scored = df.withColumn("amount_z", zscore("amount"))

Note what this does: mean and std are computed per Arrow batch, not over the whole column, so the result is not a global z-score. This is the single most common pandas UDF bug. Use it for row-wise or batch-local logic such as model scoring, where the batch boundary does not matter, and use window or aggregate functions for statistics over the full dataset.

Arrow-based paths need compatible Python dependencies. For Spark 4.2 the install documentation lists pandas between 2.2.0 and below 3.0.0 and pyarrow at 18.0.0 or newer for SQL and Connect, so a ModuleNotFoundError for pyarrow or a pandas version complaint after an environment rebuild is a packaging problem, not a Spark problem. Pin both in your lock file.

The pandas API on Spark (pyspark.pandas) offers the pandas interface backed by Spark DataFrames, which helps teams migrate existing pandas code:

import pyspark.pandas as ps

psdf = ps.read_parquet("s3a://lake/gold/daily_revenue/")
top = psdf.groupby("country")["revenue"].sum().sort_values(ascending=False).head(10)

It is not pandas, and treating it as a drop-in is how people get surprised. The official guidance is to avoid single-partition computations and avoid shuffles where possible, and to choose a distributed or distributed-sequence default index rather than the default sequence index, which pulls ordering through one place. Operations that depend on row order are expensive in a distributed system, and calls like to_pandas() collect everything onto the driver exactly as toPandas() does. If a job needs ordering semantics everywhere, it probably should be a plain DataFrame job.

ANSI mode and the Spark 4 upgrade hazards

Many teams at this level are mid-migration from 3.5, and the migration guides are where the real surprises live. The change that bites first is ANSI SQL mode on by default since Spark 4.0 (spark.sql.ansi.enabled=true). Under it, operations that used to return NULL quietly now fail: an invalid cast, arithmetic overflow, division by zero. The messages carry error classes you can search for.

[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead.
[CAST_INVALID_INPUT] The value 'abc' of the type "STRING" cannot be cast to "INT" because it is malformed.

Read ANSI errors as the system telling you the data is not what the code assumed, which is a feature on a new pipeline and a nuisance on a legacy one. There are three responses, in order of preference. Fix the data or the logic. Use the tolerant function that states your intent: try_cast and try_divide return NULL on failure, and the code then documents that it expects bad values. Or, for legacy code you cannot yet change, restore the old behaviour with spark.sql.ansi.enabled=false (the environment variable SPARK_ANSI_SQL_MODE=false also works). Treat that last option as a loan: it hides data-quality problems, and the flag may not exist forever.

The 4.0 breaking changes that cause the most incidents, collected in the migration guide:

• Java and Scala. Java 17, 21 or 25 is required (Java 25 before 25.0.3 is deprecated), and Scala 2.12 artifacts do not exist for Spark 4. Libraries built as _2.12 jars produce ClassNotFoundException or NoSuchMethodError at run time; rebuild for Scala 2.13.
• Table creation. CREATE TABLE without USING or STORED AS now uses spark.sql.sources.default, which is Parquet, instead of Hive. Restore with spark.sql.legacy.createHiveTableByDefault=true.
• JDBC type mappings changed for MySQL, Postgres, Oracle, SQL Server and DB2, so a table read or written through JDBC can come back with different Spark types. Each has a spark.sql.legacy.<db>.* flag.
• Encoding. encode() and decode() accept only a short list of charsets, and unmappable characters raise MALFORMED_CHARACTER_CODING.
• Config renames. spark.sql.legacy.parquet.int96RebaseModeInWrite became spark.sql.parquet.int96RebaseModeInWrite, with the datetime-rebase and Avro variants renamed likewise.
• Operations. Event log rolling and compression are on by default, worker cleanup is on, speculation is less aggressive, and Kubernetes persistent volume claims use ReadWriteOncePod.

The upgrade method that works is mechanical. Read the SQL, Core and PySpark migration guides for every version you cross. Run the old and new versions side by side on a copy of production data and diff the outputs, not only the success status. When something differs, toggle one legacy flag at a time to isolate which behaviour change is responsible, then fix the code and remove the flag. Pin the image tag throughout so you know what you tested.

Structured Streaming in production

Structured Streaming applies the DataFrame API to unbounded data. You write the transformation as if the data were a table that keeps growing, and Spark runs it incrementally, in micro-batches by default. The mental model that helps: a streaming query is a batch query plus state (what it has processed and what it must remember) stored in a checkpoint.

The checkpoint is the contract. It records the offsets processed and, for stateful operations, the aggregation state. Restart a query with the same checkpoint location and it resumes where it stopped. Delete or change the location and it starts over, reprocessing or skipping data. Most sinks require the location: the error Checkpoint location must be specified either through option("checkpointLocation", ...) or SparkSession.conf.set("spark.sql.streaming.checkpointLocation", ...) is the reminder. Put it on durable shared storage such as object storage or HDFS, not on a local disk, and give every query its own location.

from pyspark.sql import functions as F

events = (spark.readStream.format("rate").option("rowsPerSecond", 50).load())

windowed = (events
    .withWatermark("timestamp", "10 minutes")
    .groupBy(F.window("timestamp", "5 minutes"))
    .count())

query = (windowed.writeStream
    .outputMode("append")
    .format("console")
    .option("checkpointLocation", "/tmp/ckpt/windowed")
    .trigger(processingTime="1 minute")
    .start())

Three concepts decide whether a streaming job is healthy. The trigger sets how often a micro-batch runs; processingTime runs on an interval, and availableNow=True processes everything available and then stops, which makes it a cost-effective way to run incremental batch jobs on a schedule with streaming semantics and checkpointed progress. The watermark tells Spark how late data can arrive: withWatermark("timestamp", "10 minutes") means state for windows older than the latest event time minus ten minutes can be dropped. Without a watermark, stateful operations such as windowed aggregations and stream-stream joins keep state forever and eventually exhaust memory. The output mode (append, update, complete) controls what is written each batch; append with a watermark emits a window only when it is final.

The restart contract has limits that catch teams. Changing the query's logic can be incompatible with the stored state: adding a grouping column, changing a window size, or altering a stateful operator's schema usually requires a new checkpoint and a reprocessing plan. Changing a stateless transformation is generally safe. Treat the checkpoint as part of the deployed artifact and version it with the job. The same applies to upgrading Spark: test restarting from a checkpoint written by the previous version before relying on it.

Monitor a running query through its lastProgress and status properties, or the Streaming tab in the UI: the number to watch is input rate versus processing rate. If processing is persistently slower than input, the query falls behind without ever failing, and latency grows without an alarm. Alert on that gap.

Configuration management and spark-submit

Spark has several places to set a value, and misconfigured jobs are usually victims of precedence rather than ignorance. In order of precedence, highest first: values set in code through SparkConf or SparkSession.builder.config, then flags passed to spark-submit, then conf/spark-defaults.conf. The Environment tab shows what won.

The consequence for production is that code should set almost nothing. Application code that hard-codes .master("local[*]") or spark.executor.memory overrides the cluster's flags and is the cause of the classic bug where a job tuned on a cluster ignores the tuning. Code should hold only the settings that are part of the job's logic (for example a spark.sql.shuffle.partitions that the algorithm depends on), and everything about resources and environment should come from outside.

./bin/spark-submit \
  --master k8s://https://<apiserver>:<port> \
  --deploy-mode cluster \
  --name daily-revenue \
  --driver-memory 4g --executor-memory 8g --executor-cores 4 \
  --conf spark.sql.shuffle.partitions=400 \
  --py-files deps.zip \
  job.py --date 2026-10-01

A pattern that holds up in teams is layered config files: a base file of defaults checked into the repository, one per environment (dev, staging, production) holding resource sizes and endpoints, and a thin launcher that merges them into --conf flags. The files are reviewed like code, which means a pull request that doubles executor memory is visible. Never put passwords in --conf: values on the command line show up in process listings and in the UI's Environment tab. Pass credentials as mounted files or environment variables from your platform's secret store, or use cloud identity for storage access; spark.redaction.regex hides matching values in the UI and logs, but it is a safety net, not a plan.

Running Spark on Kubernetes

Kubernetes is the most common target for new Spark deployments, and the model is simple. spark-submit in cluster mode creates a driver pod; the driver then requests executor pods from the Kubernetes API and tears them down when the application ends. Spark 4 supports Standalone, YARN and Kubernetes (Mesos was removed in 4.0), and local[*] for a single machine.

./bin/spark-submit \
  --master k8s://https://<apiserver-host>:<port> \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=5 \
  --conf spark.kubernetes.container.image=apache/spark:4.2.0 \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/spark/examples/jars/spark-examples_2.13-4.2.0.jar

The local:// scheme means the file is already inside the image. The service account is the first thing that fails on a new cluster: the driver needs permission to create, watch and delete pods in its namespace, and without a Role and RoleBinding it fails with an error saying the service account cannot create resource "pods". Create the service account and binding before the first submission.

Build your own image on top of the official one with bin/docker-image-tool.sh or a plain Dockerfile (see the Docker guide for layering and caching), install your Python dependencies in it, tag it with an immutable version, and reference it through spark.kubernetes.container.image. There are separate keys for the driver and executor images when they differ. Pod templates (spark.kubernetes.driver.podTemplateFile, spark.kubernetes.executor.podTemplateFile) cover what flags cannot: labels for cost allocation, tolerations, node selectors, sidecars. Kubernetes-specific resource requests use spark.kubernetes.driver.request.cores and spark.kubernetes.executor.request.cores.

Dynamic allocation lets the application grow and shrink its executor count with load, but on Kubernetes there is no external shuffle service, so executors holding shuffle files cannot simply be removed. The answer is shuffle tracking:

--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.shuffleTracking.enabled=true \
--conf spark.dynamicAllocation.minExecutors=2 \
--conf spark.dynamicAllocation.maxExecutors=40

Set maxExecutors always; an unbounded job on a shared cluster is an outage waiting for a large input. Note that Spark 4.2 raises the executor pod allocation batch size from 10 to 20 (spark.kubernetes.allocation.batch.size), so ramp-up creates pods faster; check your namespace quotas, and set it back to 10 if bursts trouble your control plane.

Secrets reach pods through Kubernetes primitives, not the command line: spark.kubernetes.driver.secrets.<secret>=/mount/path mounts a secret as files, and spark.kubernetes.driver.secretKeyRef.ENV_NAME=secret:key injects one value as an environment variable, each with an executor counterpart. For storage access prefer workload identity or an IAM role bound to the service account over long-lived keys.

Spark 4.2 adds one default that surprises people running tools against executors: executor pods now get a restrictive NetworkPolicy ingress. If you run in client mode, or connect a tool directly to executors, and connections stop working after an upgrade, that is why. You can exclude the step with spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep, but do that only when you understand what you are opening.

Operating a running application uses ordinary Kubernetes tools plus two spark-submit options:

kubectl -n spark-jobs logs -f <driver-pod>
kubectl -n spark-jobs port-forward <driver-pod> 4040:4040     # the UI of a live job
./bin/spark-submit --status spark-jobs:daily-revenue* --master k8s://https://<apiserver>:<port>
./bin/spark-submit --kill   spark-jobs:<driver-pod>   --master k8s://https://<apiserver>:<port>

Spark Connect: a thin client for a shared server

Classic PySpark runs the driver inside your process: a JVM starts beside your Python, and the two talk through a bridge. That ties your application to a local Java installation and to the exact Spark version on the cluster. Spark Connect splits the architecture. A Spark server runs somewhere (on a cluster, in a container) and exposes a gRPC endpoint, by default port 15002; your application is a thin client that builds the DataFrame plan and sends it over the wire for the server to execute.

./sbin/start-connect-server.sh                  # starts the server, port 15002
./bin/pyspark --remote "sc://localhost"         # a shell attached to it
export SPARK_REMOTE="sc://localhost"            # or set it for any client

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
spark.range(5).show()

The benefits are practical. A notebook, service or CI job needs no JVM: pip install pyspark-client provides a pure-Python Connect client, and pyspark-connect makes Connect the default. Client dependencies are isolated from the server. The server can be upgraded independently of its clients, which reduces the blast radius of an upgrade. And many applications can share one long-running server rather than each starting its own driver.

The constraints are what you give up. Connect does not support RDDs, the SparkContext, or private attributes such as df._jdf, and only session-level configuration can be set from the client. Code that touches those fails with errors like "RDD is not supported" or a JVM-attribute error. Before moving a codebase, grep it for .rdd, sparkContext, and _jdf; what remains is DataFrame code that moves cleanly.

One caution on tenancy: a shared Connect server gives each client its own session, but sessions share a JVM. It is a convenience boundary, not a security boundary between teams.

Observability: event logs, the History Server and metrics

The live UI disappears when the application ends, which is exactly when you need it. Event logs persist what the UI displays. Enable them with spark.eventLog.enabled=true and point spark.eventLog.dir at shared storage the History Server can read. Since Spark 4.0 rolling and compression are on by default, so a long-running streaming application does not produce one enormous file.

The History Server reads those logs and serves the same UI for finished applications. Start it with ./sbin/start-history-server.sh and set spark.history.fs.logDirectory to the event log path. It listens on port 18080 (spark.history.ui.port), re-scans the directory every ten seconds by default and keeps 50 applications in memory. The log cleaner (spark.history.fs.cleaner.enabled) is off by default, so logs accumulate until you enable it and set spark.history.fs.cleaner.maxAge (7 days by default). Spark does not set permissions on the event log directory for you.

For automation, the same data is served over a REST API under /api/v1: /applications, /applications/<id>/jobs, /stages, /executors, /sql. A script that reads a nightly run's stages and records task-time quantiles gives you a regression detector for free.

Metrics use the Dropwizard system, configured through conf/metrics.properties or spark.metrics.conf.*. Sinks include console, CSV, JMX, Graphite, StatsD and an experimental PrometheusServlet, which serves the driver's metrics at :4040/metrics/prometheus/ and executor metrics at :4040/metrics/executors/prometheus/. Alert on a short list: executor JVMHeapMemory and totalGCTime for memory pressure, failedTasks for flakiness, and the spill metrics for undersized tasks. For batch jobs, also record each run's duration and input and output row counts: a job that outputs half the usual rows without erroring is more dangerous than one that fails. Remember that Spark has no built-in authentication for its UI, so never expose it, or the Master REST API enabled by default since 4.1, to an untrusted network.

Testing PySpark code

Spark code is easy to leave untested because "it needs a cluster". It does not. A local session with two cores runs real Catalyst plans in a second or two. The structure that makes this work is pure transformation functions: write each step as a function from DataFrame to DataFrame, keep reading and writing at the edges, and test the functions.

import pytest
from pyspark.sql import SparkSession

@pytest.fixture(scope="session")
def spark():
    session = (SparkSession.builder
               .master("local[2]")
               .config("spark.sql.shuffle.partitions", "2")
               .config("spark.ui.enabled", "false")
               .getOrCreate())
    yield session
    session.stop()

from jobs.revenue import daily_revenue

def test_sums_per_country_and_day(spark):
    orders = spark.createDataFrame(
        [("c1", "2026-10-01 09:00:00", 10.0), ("c1", "2026-10-01 10:00:00", 5.0)],
        "customer_id string, ts string, amount double")
    customers = spark.createDataFrame([("c1", "EG")], "customer_id string, country string")
    result = daily_revenue(orders, customers)
    assert result.collect()[0]["revenue"] == 15.0

Three fixture settings matter. A small spark.sql.shuffle.partitions keeps a three-row test from creating 200 tasks. Disabling the UI avoids port conflicts. A session-scoped fixture starts Spark once for the whole run, which is the difference between seconds and minutes. Use explicit schemas rather than inference, test nulls, empty inputs and values ANSI mode rejects, and compare results without depending on row order, since Spark promises none after a shuffle. Beyond unit tests, run data checks against real output (row counts, key uniqueness, null rates), covered by the Great Expectations guide, and run the suite inside your job's container image in CI so Python, Java, pandas and pyarrow match production.

Debugging beyond the basics

Most Spark errors are wrappers. Job aborted due to stage failure: Task N in stage S failed 4 times tells you only that a task failed four times; the cause is further down under Caused by, and often in the executor's log rather than the driver's. Read from the bottom of the stack trace upward, find the first frame from your code or a data-format library, and work from there.

A reliable order of questions for a slow job: Is it one stage? (Stages tab, sort by duration.) Is it skewed? (median versus maximum task time.) Is it spilling? (spill columns.) Is it scanning too much? (SQL tab, scan rows against what the query needs.) Is it waiting on executors? (Executors tab, and on Kubernetes kubectl get pods for Pending executors.) Is it a UDF? (a Python evaluation operator in the plan.) Answer these in that order and you will find most causes before touching a setting.

For a failing job, classify the error. Memory errors split into driver heap (a collect, toPandas or large broadcast), executor heap (too little parallelism, too much per task) and container kill (OOMKilled, which means overhead). Analysis errors, such as UNRESOLVED_COLUMN.WITH_SUGGESTION which lists columns that look like the one you typed, happen at plan time and never need a cluster to reproduce. Serialisation and class errors after an upgrade point to a Scala 2.12 jar or a javax dependency. Python environment errors, such as "Python worker failed to connect back", usually mean driver and workers use different interpreters.

Two techniques cut debugging time sharply. Reproduce small: take a sample of the failing partition's data (df.limit(1000) after the filter that isolates the key) and run it on a local session with the same code. Most logic bugs reproduce in seconds there. Bisect the plan: write intermediate DataFrames to a scratch location, or call count() after each step, to find the transformation where rows go missing or multiply. Remove the checks afterwards, since each count() is a full job.

Spark in the rest of your stack

Spark rarely stands alone. It reads and writes object storage (S3 through S3A, GCS, ADLS, HDFS) and, increasingly, table formats. A table format adds transactions, schema evolution and time travel on top of files, and solves the concurrent-write and partial-overwrite problems described earlier. The Delta Lake guide covers one such format; Iceberg and Hudi are alternatives. These are third-party projects, so check each one's compatibility with your Spark version before upgrading Spark, since they ship connector jars tied to a Spark release.

For orchestration, a scheduler launches your spark-submit or Kubernetes job, passes the logical date as an argument, and retries on failure. Airflow and Dagster both fit; Prefect and Kubeflow do too. Whatever you pick, the contract with Spark is the same: the job takes the partition to process as a parameter, writes idempotently, and exits non-zero on failure. Everything else, such as retry policy, alerting and backfills, belongs in the orchestrator.

For machine learning, Spark's strengths are feature engineering over large data and batch scoring. MLlib (pyspark.ml) provides distributed classical algorithms with a DataFrame-based Pipeline of transformers and estimators, and the pipeline object can be saved and loaded. Deep learning training is usually done elsewhere; the common shape is Spark preparing features, a separate framework training the model, and a pandas UDF applying it for batch scoring as described above. Track experiments and register models with MLflow and keep the model version your job loads as an explicit parameter, not "latest". Feature definitions that must match between training and serving are the job of a feature store such as Feast.

Cost and residency decisions cross all of these. Executors are interruptible and recomputable, so spot or preemptible capacity is a common saving for them, with an on-demand driver; this is general practice rather than an official recommendation, so test how your job behaves when executors disappear. In the Middle East and North Africa, pick the cloud region that satisfies the data-residency rules of your employer's regulator and keep storage and compute in the same region, since cross-region reads are slow and billed.

Putting it all together

Here is one production-shaped job that combines the pieces: parameterised, config-free, idempotent, testable. The transformation is a pure function in its own module; the entry point handles arguments and I/O.

from pyspark.sql import DataFrame, SparkSession, functions as F
from pyspark.sql.functions import broadcast

def daily_revenue(orders: DataFrame, customers: DataFrame) -> DataFrame:
    slim_customers = customers.select("customer_id", "country")
    return (orders
            .select("customer_id", "ts", "amount")
            .join(broadcast(slim_customers), "customer_id")
            .groupBy("country", F.to_date("ts").alias("day"))
            .agg(F.sum("amount").alias("revenue")))

def main(day: str, orders_path: str, customers_path: str, out_path: str) -> None:
    spark = SparkSession.builder.appName(f"daily-revenue-{day}").getOrCreate()
    orders = (spark.read.parquet(orders_path)
                   .where(F.to_date("ts") == F.lit(day).cast("date")))
    customers = spark.read.parquet(customers_path)

    result = daily_revenue(orders, customers)
    (result.repartition("day")
           .write.mode("overwrite")
           .partitionBy("day")
           .parquet(out_path))

if __name__ == "__main__":
    import sys
    main(*sys.argv[1:5])

./bin/spark-submit \
  --master k8s://https://<apiserver>:<port> \
  --deploy-mode cluster \
  --name daily-revenue \
  --conf spark.kubernetes.container.image=registry.example.com/jobs/revenue:1.4.0 \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
  --conf spark.dynamicAllocation.maxExecutors=20 \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=s3a://lake/_spark-events \
  local:///opt/app/jobs/revenue.py 2026-10-01 s3a://lake/orders s3a://lake/customers s3a://lake/gold/daily_revenue

Where you are now

You can predict how a query becomes stages and tasks, read a physical plan and the SQL and Stages tabs to find the one operator that matters, and decide between a broadcast, a sort-merge join and a skew fix on evidence. You size memory with the execution, storage and overhead split in mind, so you can tell a heap error from a container kill. You write files other jobs can read efficiently and rewrite them safely. You know the cost of a UDF, you handle ANSI mode and the Spark 4 breaking changes deliberately, and you can run Structured Streaming with a checkpoint you understand. You submit to Kubernetes with config that lives outside the code, you test transformations locally, and you can observe finished jobs through event logs, the History Server and metrics.

The Senior level treats Spark as a platform other teams depend on: failure modes at scale, the security model, multi-tenancy and quotas, cost governance, upgrade programmes, and the question of when Spark is the wrong tool. If you want to practise first, take a job you own, add one test, one idempotent write and one alert, and keep the SQL tab open while you do it.

Sources

• Apache Spark documentation overview
• Downloads and release information
• SQL performance tuning (AQE, joins, broadcast, caching)
• SQL migration guide
• Core migration guide
• Running Spark on Kubernetes
• Monitoring and instrumentation
• Spark Connect overview
• Security
• PySpark installation
• Pandas API on Spark
• Structured Streaming programming guide
• Submitting applications
• Configuration

Part two of three. Interview preparation for someone who runs Spark at work. Interviewers at this level stop asking what a stage is and start asking what you did when one stage took four hours. Every answer below is written the way a strong candidate would give it: a claim, the evidence you would look at, and the trade-off you accepted. The examples assume Spark 4.2.

Where this picks up

What a strong answer does

A weak answer names a setting. A strong answer names a symptom, a place to see it, a cause and a fix, in that order, and admits what it does not know. When asked "the job is slow, what do you do?", the weak candidate says "increase executors" or "raise the shuffle partitions". The strong candidate says that they would open the SQL tab to find the expensive operator, then the stage's task quantiles to see whether it is skewed, spilling or just large, and would decide only after that. The difference is whether your first move is a measurement or a guess.

Three habits distinguish strong answers throughout this file:

• Measure before tuning. Name the tab, the metric or the explain output you would read.
• State the cost of the fix. Broadcasting trades driver memory for shuffle time; caching trades executor memory for recomputation; salting trades code complexity for balance.
• Know the version. Spark 4 changed defaults (ANSI mode, Scala 2.13, Java 17 or newer, event log rolling). An answer that ignores that sounds like a Spark 2 answer.

Scenario: the job got slow

Q. A nightly job that took 20 minutes now takes three hours. Nothing in the code changed. What do you check first?

Start with what changed in the world, since the code did not. Input volume is the most likely cause, then the shape of the data, then the cluster, then the platform. I would compare the input size and row counts of tonight's run with last week's, using the History Server for the old run and the REST API if I want it scripted. If volume is roughly equal, I look for a data shape change: a new hot key, a new null or default value in a join column, a partition that is suddenly much larger.

Then I go to the slow run's UI. In the SQL tab I find the operator with the largest time or row count. In that stage's task table I compare the median and maximum task duration and shuffle read. If the maximum is far above the median, it is skew. If many tasks show spill, the tasks do not fit in execution memory. If the tasks are short but there are very many, scheduling overhead dominates, which points to too many small partitions or small input files. If the stage itself is fine but there is a long gap before it, the executors may not have arrived: on Kubernetes I would check for Pending pods and quota.

A strong answer also mentions the platform: a node pool change, a different Spark image, or a dependency upgrade. A weak answer jumps to "add executors", which does nothing for skew.

Q. One task in a stage runs for 40 minutes while the other 199 finish in 30 seconds. Why, and what do you do?

That is data skew: a key whose rows all hash to one partition. The Shuffle Read Size column for that task will be far larger than the median. Adaptive query execution splits skewed partitions in sort-merge joins by default, but only when a partition is more than five times the median size and also above 256 MB by default, so a moderately skewed partition escapes it. My order of action: confirm AQE skew handling is enabled and lower the byte threshold if the hot partitions are below it; check whether the hot key is meaningful, because it is often a null or a placeholder that should be filtered out before the join; if the key is real and the other side is small, broadcast it; and as a last resort, salt the key. If the skew is in an aggregation rather than a join, AQE's skew join handling does not apply, so I would pre-aggregate with a salted key and then merge.

What I would not do is raise executor memory. The slow task would still own all the rows.

Q. Tasks are finishing but the Stages tab shows heavy "Spill (disk)". What does it mean?

Execution memory ran out during a shuffle, sort, join or aggregation, so Spark wrote intermediate data to disk and read it back. The job still finishes, but slowly. The cause is too much data per task. Options in order of preference: increase the number of partitions so each task holds less (raise spark.sql.shuffle.partitions, or lower the advisory partition size when AQE coalescing is merging too aggressively); reduce the data before the shuffle by projecting fewer columns and filtering earlier; give executors more memory; and check whether a large cache is crowding execution, which the Storage tab shows. I would also check whether one skewed task is the only one spilling, in which case the issue is skew and not memory.

Q. The driver dies with java.lang.OutOfMemoryError: Java heap space. Executors look idle. What happened?

The driver collected too much. Common causes are collect(), toPandas(), a very large broadcast (the small side is materialised on the driver first), or an enormous number of tasks and files making the plan or the listing too big. I would find the action in the driver stack trace. The fix is to keep the data distributed: write the result out instead of collecting it, aggregate before toPandas(), or use limit() for inspection. Raising spark.driver.memory is a stopgap that moves the failure to a larger input.

Q. On Kubernetes, executors disappear with OOMKilled and exit code 137, but the JVM heap graph looks fine. Why?

The container limit is heap plus overhead. Overhead covers off-heap memory, native libraries and, for PySpark, the Python worker processes. The JVM was not out of heap; the pod exceeded its limit and the kernel killed it. I would raise spark.executor.memoryOverhead, and for Python-heavy work look at spark.executor.pyspark.memory. Raising spark.executor.memory alone makes the pod larger without addressing the real shortfall. I would also check for a Python UDF or a pandas UDF holding large batches, since that lives outside the heap.

Scenario: the job is wrong, not slow

Q. After upgrading from Spark 3.5 to 4.x, several jobs fail with errors about casts and division. What changed and what do you do?

ANSI SQL mode is on by default since Spark 4.0 (spark.sql.ansi.enabled=true). Invalid casts, arithmetic overflow and division by zero now raise errors such as CAST_INVALID_INPUT and DIVIDE_BY_ZERO instead of producing NULL. The errors are revealing real data quality problems that were silently hidden. My approach is to triage each failure: if the bad value is a data bug, fix the source or add a filter; if the code intentionally tolerates bad values, make that explicit with try_cast or try_divide; and only for legacy code that cannot change yet, set spark.sql.ansi.enabled=false as a temporary measure with an owner and a date. I would never disable it globally and move on.

A strong candidate also lists the other 4.0 breaking changes unprompted: Scala 2.13 only, Java 17 or newer, CREATE TABLE defaulting to Parquet rather than Hive, JDBC type mapping changes, and the javax to jakarta move that breaks old libraries.

Q. A job produced duplicate rows after a retry. How would you make it safe?

The write was not idempotent. A retry reran the append, so rows landed twice. The fix is to design the job as "compute partition X, replace partition X": parameterise it by the date or partition key, filter the input to that key, and overwrite exactly that output partition. In plain Parquet this needs care, because mode("overwrite") with partitionBy can replace the whole path unless dynamic partition overwrite is configured and verified on a copy of the data. A table format such as Delta Lake gives transactional overwrites and merges and is the more robust answer when several writers share a table; the Delta Lake guide covers it. For streaming, the equivalent is using the batch ID inside foreachBatch to make each micro-batch's write idempotent.

Scenario: streaming

Q. A streaming query stops and is restarted. How does it know where to resume, and what can go wrong?

It resumes from its checkpoint, which stores the processed offsets and, for stateful operations, the aggregation state. As long as the checkpoint location is intact and the query is compatible with it, the query continues without loss or duplication at the engine level. Problems arise in three places. If the checkpoint is on local disk of a pod that was replaced, it is gone and the query starts over. If the query logic changed in a way that is incompatible with the stored state, for example a new grouping column or a different window size in a stateful operation, the restart fails or must use a new checkpoint with a reprocessing plan. And if the sink is not idempotent, the replay of the last uncommitted batch writes twice. So: durable checkpoint storage, one checkpoint per query, version it with the job, and use batch IDs in foreachBatch.

Q. What is a watermark and what happens without one?

A watermark declares how late event-time data may arrive, for example withWatermark("timestamp", "10 minutes"). It lets Spark finalise windows and discard state older than the latest event time minus the delay. Without it, stateful operations such as windowed aggregations and stream-stream joins keep state forever, so memory grows until the job fails, and in append output mode a window may never be emitted. The trade-off is lateness versus state size: a longer delay captures more late events but holds more state and delays output.

Q. Your streaming job shows no errors but the dashboard is hours behind. What is the likely cause?

Processing rate is lower than input rate, so the query is falling behind without failing. I would read lastProgress or the Streaming tab for input rows per second against processed rows per second and batch duration. Causes include skew in the micro-batch, a slow sink, too few partitions, state growth, or a trigger interval shorter than the batch duration. The long-term fix is an alert on that gap, because a stream that falls behind quietly is worse than one that crashes loudly.

Scenario: deployment and operations

Q. Walk me through how you would run a Spark batch job on Kubernetes.

The job is packaged in a container image built on the official Spark image, with Python dependencies installed and the tag pinned to a version. spark-submit runs in cluster mode against k8s://https://<apiserver>:<port>, which creates a driver pod; the driver requests executor pods. The driver's service account needs a Role that lets it create, watch and delete pods in its namespace, and a missing one is the classic first failure. Resource settings, image, namespace and dynamic allocation live in the submit command or a per-environment config file, not in code. Dynamic allocation on Kubernetes needs spark.dynamicAllocation.shuffleTracking.enabled=true because there is no external shuffle service, and I always set maxExecutors so one job cannot take the cluster. Secrets are mounted from Kubernetes secrets or injected with secretKeyRef, and never put into --conf values, which appear in process lists and the UI. Event logs go to shared storage so the History Server can show finished runs.

Q. What is Spark Connect and when would you use it?

It separates the client from the Spark driver. A server exposes a gRPC endpoint on port 15002, and the client builds DataFrame operations and sends them for execution, so the client needs no JVM and can be upgraded independently. It suits notebooks, services and CI jobs that should not each start a driver. The limits are that RDDs, SparkContext, df._jdf and non-session configuration are not available, so code that uses them cannot move as is. It also does not give hard isolation between tenants, since sessions share a server JVM.

Trade-off questions

Q. Broadcast join or sort-merge join?

Broadcast when one side is genuinely small, since it avoids shuffling the large side entirely. The default threshold is 10 MB, deliberately conservative, and a hint can override it. The costs are driver memory (the table is collected there first), executor memory for every copy, and the broadcast timeout of 300 seconds. Sort-merge scales to any size at the price of two shuffles and sorts. With AQE the engine can switch from sort-merge to broadcast at runtime when the real size turns out small, which reduces the need for hints. I would hint only where I have measured a win and can say why the estimate was wrong.

Q. Cache or recompute?

Cache when a DataFrame is expensive to build and reused by several actions, or revisited iteratively. Do not cache something used once, or something cheap to re-read from columnar storage, because the cache competes with execution for memory and unevictable blocks can cause spill. With memory-only storage, partitions that do not fit are recomputed on each use, which can be slower than no cache. I verify on the Storage tab that the fraction cached is what I expect, and I unpersist when finished.

Q. repartition or coalesce?

repartition(n) performs a full shuffle and can increase or decrease partitions with even distribution. coalesce(n) only decreases, merging partitions without a full shuffle, which is cheaper but cannot rebalance skewed input. The trap is coalesce(1) before a write: because narrow operators pipeline into the same stage, the entire upstream work for that stage runs in one task. If I truly need one file I use repartition(1) so upstream work stays parallel.

What strong answers cover that weak ones miss

Common interview questions

Q. What does AQE do?

It re-optimises the plan at runtime using statistics from completed shuffle stages. It coalesces small shuffle partitions toward an advisory size (64 MB by default), converts sort-merge joins to broadcast joins when a side turns out small, and splits skewed partitions in joins. It is on by default, and the SQL tab shows the final adapted plan.

Q. How does Spark decide the number of partitions?

For file scans, from the input files and their split sizes. After a shuffle, from spark.sql.shuffle.partitions (default 200), which AQE can reduce by coalescing. Explicitly, through repartition and coalesce. For writes, the number of partitions of the DataFrame being written determines the number of files per output partition value.

Final self-test

Answer aloud without notes; a sentence or two each.

1. Name the four things you read in the UI, in order, when a job is slow.
2. What are the two conditions a partition must meet before AQE treats it as skewed, and their defaults?
3. Why does raising executor memory not fix a skewed stage?
4. A pod is OOMKilled with exit code 137. Which setting do you reach for first?
5. Why is coalesce(1) before a write dangerous?
6. What does ANSI mode change, and what are the three ways to respond to the errors?
7. What does a streaming checkpoint contain, and when must you create a new one?
8. What makes a batch job idempotent?
9. What does Spark Connect remove from the client, and what does it not provide?
10. Why must maxExecutors be set with dynamic allocation on Kubernetes, and what extra setting does it need?

If you hesitated on more than three, reread the matching sections of the Mid-level detailed guide, then try the questions again on a real job's SQL tab. The practice tips in the next file turn these answers into habits.

Part two of three. Practical tips for someone who runs Spark at work: the errors you will actually meet, the practices that pay off most, cards to rehearse, and recipes to keep. It assumes Spark 4.2 and PySpark.

Errors you will meet at this level

Beginner errors are about getting a session to start. Mid-level errors are about jobs that start fine and then fail, crawl or quietly return the wrong thing. Each entry gives the symptom, the cause and the fix.

1. Job aborted due to stage failure: Task N in stage S failed 4 times. This is a wrapper, not a cause. Scroll to the Caused by lines, then open the failed task in the Stages tab and read the executor's stderr. The same wrapper covers out-of-memory kills, bad input rows, missing files and library mismatches.

2. java.lang.OutOfMemoryError: Java heap space on the driver. Something pulled data to the driver: collect(), toPandas(), a very large broadcast, or a plan built over a huge number of files. Keep data distributed, write results instead of collecting them, and use limit() to inspect. Raising spark.driver.memory only postpones the failure.

3. The same error on an executor during a wide aggregation. Each task holds too much. Raise the number of partitions or lower the advisory size, project fewer columns before the shuffle, and check for skew before adding memory.

4. Pod OOMKilled, exit code 137 (or YARN "Container killed for exceeding memory limits"). The container, not the heap, ran out. Heap plus overhead plus Python workers exceeded the limit. Raise spark.executor.memoryOverhead first, and spark.executor.pyspark.memory for Python-heavy jobs. Raising only spark.executor.memory does not help.

5. [DIVIDE_BY_ZERO] or [CAST_INVALID_INPUT] after an upgrade. ANSI mode is on by default in Spark 4. Fix the data, use try_divide and try_cast where tolerance is intended, and use spark.sql.ansi.enabled=false only as a labelled, temporary measure for legacy code.

6. ClassNotFoundException or NoSuchMethodError after moving to Spark 4. A library was built for Scala 2.12 (_2.12 artifacts do not exist for Spark 4) or still uses javax.servlet. Rebuild or upgrade the dependency to a Scala 2.13, jakarta-compatible release.

7. java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]. A broadcast took longer than spark.sql.broadcastTimeout (300 seconds). Check whether the table is truly small and why the query producing it is slow before raising the timeout. If it is large, remove the hint and let a sort-merge join run.

8. SchemaColumnConvertNotSupportedException reading Parquet. Files in one path disagree about a column's type. Supply an explicit schema, or use mergeSchema when columns are legitimately added over time. Find the offending files by reading directories one at a time.

9. Python in worker has different version ... than that in driver. Driver and executors use different Python minor versions. Bake the same interpreter into the image and set PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to it.

10. ModuleNotFoundError: No module named 'pyarrow', or pandas version complaints. Arrow-based features need compatible packages on both driver and executors. The Spark 4.2 install documentation lists pandas 2.2.0 or newer but below 3.0.0 and pyarrow 18.0.0 or newer for SQL and Connect. Pin both in the image.

11. Pods stay Pending, or the driver reports pods is forbidden. Pending means the cluster lacks resources or quota (kubectl describe the pod). Forbidden means the driver's service account has no Role to create pods in its namespace. Create the Role and RoleBinding before the first submit.

12. Executors are deleted in client mode on Kubernetes. The executors have no owner reference to the driver. Set spark.kubernetes.driver.pod.name to the pod hosting the driver.

13. After upgrading to 4.2, connections to executors stop working. Executor pods now get a restrictive NetworkPolicy ingress by default. If you genuinely need direct connections, exclude the feature step with spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep, understanding what that opens.

14. A write replaced far more data than intended. mode("overwrite") with partitionBy can replace the whole target path. Test on a scratch copy and list the directory before and after. Use dynamic partition overwrite only after verifying it, or use a table format with transactional overwrites.

15. Millions of tiny files. Many shuffle partitions each writing to many partition directories. Repartition by the partition column before the write, partition by low-cardinality columns, and check file counts after every change to the write path.

16. DUPLICATED_CTE_NAMES, or schemas named builtin or session stop resolving (4.2). Duplicate CTE names are now detected case-insensitively, so rename them. The 4.2 virtual system catalog resolves before your schemas; use an explicit catalog prefix or, temporarily, spark.sql.legacy.persistentCatalogFirst=true.

17. RDD is not supported, or an attribute error for _jdf, over Spark Connect. Connect has no RDDs, no SparkContext and no private JVM attributes. Rewrite with the DataFrame API or run that code against a classic session.

18. A streaming query will not restart, or reprocesses everything. The checkpoint was deleted, moved, or is incompatible with changed stateful logic. Keep checkpoints on durable storage, one per query, and version them with the job.

The practices that pay off most, in order of leverage

These are ranked by how much trouble each prevents per minute spent.

1. Read the plan and the SQL tab before changing anything. Every other practice depends on knowing which operator is expensive. df.explain("formatted") before running, the SQL tab after. Count the Exchange nodes, check PushedFilters, and find the operator with the largest row count or time. Most "tuning" done without this is superstition.

2. Make every batch job idempotent and parameterised. The job takes a date or partition key as an argument, filters its input to it, and replaces exactly that output partition. A retry then produces the same result as a first run, and backfills are a loop over dates. This single habit removes duplicate-row incidents and makes scheduling trivial.

3. Keep resource configuration out of code. Code sets only what the logic depends on. Executor memory, cores, master URL, image and endpoints come from spark-submit flags or per-environment files that are reviewed like code. Check the Environment tab after a deploy to confirm what actually won, since code beats flags beats spark-defaults.conf.

4. Enable event logs everywhere, and run a History Server. Set spark.eventLog.enabled=true and a shared spark.eventLog.dir. Without logs, a failed job cannot be inspected after the application ends. Enable the History Server cleaner and a retention age so logs do not accumulate forever.

5. Treat built-in functions as the default and UDFs as a decision. Search pyspark.sql.functions first. If Python is unavoidable, use an Arrow-based pandas UDF, and load models once per partition rather than per row or per batch.

6. Control output file layout. Choose partition columns with low cardinality, repartition by the partition column before writing, and look at file counts and sizes. Layout decided at write time is paid for by every later read.

7. Test transformations locally. Pure functions from DataFrame to DataFrame, a session-scoped local session with two cores, a handful of shuffle partitions, UI disabled, explicit schemas, and cases for nulls and empty input. Run the suite in the same container image as production.

8. Pin the environment as one unit. Spark version, Java, Python, pandas and pyarrow, and your image tag are tested together. Rebuilding an image with an unpinned dependency is how Monday mornings fail.

9. Bound everything on a shared cluster. Set maxExecutors with dynamic allocation, and on Kubernetes enable shuffle tracking. Use namespace quotas. A job without limits will eventually meet an input large enough to take the cluster.

10. Move secrets out of --conf. Use Kubernetes secrets mounted as files or secretKeyRef environment variables, or cloud workload identity. spark.redaction.regex is a safety net, not a policy.

11. Alert on shape, not just failure. Record duration, input rows and output rows per run, and alert on large deviations. For streaming, alert when the processing rate falls below the input rate. Quiet wrongness is worse than a crash.

12. Plan upgrades like projects. Read the SQL, Core and PySpark migration guides for every version you cross, run old and new side by side on a copy of production data, diff the outputs, and toggle legacy flags one at a time. Keep a ledger of flags with owners.

Practice cards

Work through each card on a real job, or a generated dataset, and write down the result. Each takes about ten minutes.

Operational recipes

Keep these in your team's runbook. Replace placeholders with your values.

Submit to Kubernetes with bounded dynamic allocation.

./bin/spark-submit \
  --master k8s://https://<apiserver>:<port> \
  --deploy-mode cluster \
  --name nightly-job \
  --conf spark.kubernetes.container.image=<registry>/<image>:<immutable-tag> \
  --conf spark.kubernetes.namespace=<namespace> \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=<service-account> \
  --conf spark.dynamicAllocation.enabled=true \
  --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
  --conf spark.dynamicAllocation.minExecutors=2 \
  --conf spark.dynamicAllocation.maxExecutors=20 \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=<shared-path> \
  local:///opt/app/job.py <date>

Watch and manage a running job.

kubectl -n <namespace> logs -f <driver-pod>
kubectl -n <namespace> port-forward <driver-pod> 4040:4040    # then open http://localhost:4040
./bin/spark-submit --status <namespace>:<driver-pod> --master k8s://https://<apiserver>:<port>
./bin/spark-submit --kill   <namespace>:<driver-pod> --master k8s://https://<apiserver>:<port>

Start a History Server for finished jobs.

./sbin/start-history-server.sh     # with spark.history.fs.logDirectory set to your event log path
curl -s http://localhost:18080/api/v1/applications | head

Inspect resolved configuration inside a session.

for k, v in sorted(spark.sparkContext.getConf().getAll()):
    print(k, "=", v)
print(spark.conf.get("spark.sql.shuffle.partitions"))

This prints the static configuration; use spark.conf.get for SQL settings, and the Environment tab for the authoritative merged view. Over Spark Connect, only spark.conf is available, since there is no SparkContext.

Compare partition sizes after a read.

from pyspark.sql import functions as F

(df.withColumn("pid", F.spark_partition_id())
   .groupBy("pid").count()
   .orderBy(F.desc("count"))
   .show(10))

If the top partition holds many times more rows than the median, you have skew or a lopsided file layout, before the job has even started shuffling.

Find the hot keys.

(df.groupBy("customer_id").count()
   .orderBy(F.desc("count"))
   .show(20))

Check the top of that list for nulls, defaults and placeholders. These are often the whole skew problem.

Reproduce on a sample, locally.

sample = (spark.read.parquet("s3a://lake/orders/")
               .where("ts >= '2026-10-01' and ts < '2026-10-02'")
               .limit(100000))
sample.write.mode("overwrite").parquet("/tmp/sample_orders")

Then run the failing function against /tmp/sample_orders in a local session with the production settings. Most logic bugs reproduce in seconds.

Make a streaming query self-reporting.

import json

p = query.lastProgress
if p:
    print(json.dumps({
        "batch": p["batchId"],
        "input_rps": p.get("inputRowsPerSecond"),
        "processed_rps": p.get("processedRowsPerSecond"),
    }))

Alert when the processed rate stays below the input rate.

A pre-merge checklist for Spark changes.

• Does the job take its partition as a parameter and replace only that partition?
• Is any resource setting hard-coded in code?
• Are there new UDFs, and could a built-in function do the work?
• Does explain show the expected join strategy and pushed filters?
• Is there a unit test with null and empty inputs?
• Are credentials passed through secrets and not through --conf?
• Is the image tag immutable, and does the test suite run in that image?
• Do event logs go to shared storage?

Choosing what to fix first

When a job is both slow and expensive, the order of work matters more than any individual fix. Start with correctness, because a fast wrong job is worse than a slow right one: confirm that retries are safe and that the output row count matches expectations. Then remove work that should never have been done. Filters that did not push down, columns read and never used, a join against an unfiltered table and a cached DataFrame used once all fall here, and each is a code change that costs no cluster money. Only then look at how the remaining work is spread: skew, partition counts and spill. Tune the engine last, because every earlier change alters what the engine has to do.

A useful rule is to ask, for each candidate fix, what number in the UI will move if it works. Broadcasting a table should remove an Exchange and shrink shuffle write. Repartitioning before a write should reduce the file count. Raising overhead should stop container kills. If you cannot name the number, you are guessing, and you should go back to the stage's task quantiles. Write the number down before and after, in the pull request, so the next person can see the effect and the reason.

Finally, resist fixing everything at once. Teams that change the image, the Spark version and three configuration values in one deployment cannot tell which change caused the next incident. One change per run, compared in the History Server, is slower for a day and faster for a year.

Habits worth keeping

Open the SQL tab on every new job, even a fast one, and learn what normal looks like for it. Write down the input row count and runtime of the first successful run in the job's README, so there is a baseline. When a job fails, classify before fixing: memory (driver, executor or container), analysis, data, environment, or platform. Keep a short list of the legacy flags you have set, and remove one a month. Review the Spark migration guide whenever the platform team announces a new version, even if you do not own the upgrade.

The Senior guide takes this from running your own jobs well to running Spark as a platform for other people's: failure modes at scale, security and multi-tenancy, cost, and the decision of when another tool is the right answer.

Where this picks up

Part three of three. The earlier levels taught you to write jobs that run, read a plan, and tune a shuffle. This level is about owning Spark as a platform that other teams submit work to: you answer for the cluster manager, the images, the security posture, the upgrade path, the bill and the 3 a.m. page. This guide was verified against Apache Spark 4.2 documentation; patch releases and dates differ between the downloads and releases pages, so check https://spark.apache.org/downloads.html on the day you pin a version.

The thread through everything below is a single question: when something goes wrong in a shared Spark estate, can you say which team caused it, which component failed, and what you will change so it does not recur?

Architecture and failure modes

A Spark application is one driver process plus N executor processes, managed by a cluster manager. In Spark 4 the cluster managers are Standalone, Hadoop YARN and Kubernetes; Mesos support was removed in 4.0, so any estate still on Mesos is on a migration clock whether or not anyone has noticed. Spark 4 is built only for Scala 2.13 and needs Java 17, 21 or 25 (Java 25 before 25.0.3 is deprecated), Python 3.10 or newer, and treats R as deprecated. Those are platform decisions: they dictate your base images, your CI matrix, and what every team's library jars must be compiled against.

The most important architectural fact for an owner is that the driver is a single point of failure for its application. It holds the SparkSession, the query plan, the scheduler state and the results of anything collected. If the driver dies, the application dies; no executor can continue without it. Executors, by contrast, are expendable: lose one and the scheduler retries its tasks elsewhere, up to the task failure limit. This asymmetry shapes every operational choice. Give drivers reliable, on-demand capacity and generous memory; let executors run on cheap, interruptible capacity.

The failure modes cluster by process:

• Driver out of memory. Almost always caused by collect(), toPandas(), an enormous broadcast, or a plan so large (thousands of unions, deeply iterated loops) that the plan itself is the memory consumer. The signature is a java.lang.OutOfMemoryError: Java heap space on the driver, or, on Kubernetes, a driver pod killed with exit code 137.
• Executor lost. An executor exits because its container exceeded the memory limit, its node was reclaimed, or it was preempted. The scheduler reports ExecutorLostFailure. Isolated losses are normal; a pattern across many executors with the same reason is a sizing bug.
• Stage failure after repeated task failure. The wrapper message Job aborted due to stage failure: Task N in stage S failed 4 times hides the real error in the Caused by chain. Teach every team to read the chain from the bottom.
• Shuffle data loss. When an executor holding shuffle files disappears, downstream stages cannot fetch its output, and the producing stage must be recomputed. Dynamic allocation and spot capacity both raise the odds of this, which is why shuffle storage strategy appears again under scaling and cost.
• Straggler tasks. One partition holds far more data than its neighbours. Adaptive Query Execution (AQE) splits skewed join partitions automatically (spark.sql.adaptive.skewJoin.enabled, on by default, triggered when a partition is more than skewedPartitionFactor times the median and larger than skewedPartitionThresholdInBytes, 5.0 and 256 MB by default), but skew in aggregations or in a non-join key still needs a human.
• Control plane failure. On Kubernetes the Spark driver talks to the API server to create and watch executor pods. An API server under pressure, an exhausted quota or a missing role will look to the user like "my job is stuck", and to you like a Kubernetes incident.

Spark 4.2 added a failure-detection mechanism worth knowing by name: order-independent shuffle checksums are on by default. When an indeterminate stage (one whose output order or content can vary between attempts) is retried, Spark compares checksums, and on a mismatch it rolls back and re-runs dependent stages instead of silently combining inconsistent data. That is a correctness gain, and a surprise if you do not expect to see stages re-run after an upgrade. The switches are spark.sql.shuffle.orderIndependentChecksum.enabled and spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch; setting both to false restores the old behaviour. Treat that as a diagnostic move, not a fix: a retried indeterminate stage means a job whose output was not deterministic, and the owner of the job should know.

Scaling: which component breaks first

Spark scales along several axes at once (data volume, number of concurrent applications, number of tenants, number of tasks per stage), and each axis has a different first bottleneck. Naming them lets you watch the right dashboard instead of adding nodes.

Data volume per job stresses the shuffle. Every wide transformation (join, group by, repartition) moves data across the network and spills to local disk. Before adding executors, look at the stage detail in the UI for memoryBytesSpilled and diskBytesSpilled: spill means executors are already short of execution memory, and more of the same executors will spill the same way. The levers, in order of leverage, are avoiding the shuffle (broadcast the small side; the default spark.sql.autoBroadcastJoinThreshold is 10 MB), shrinking what is shuffled (select columns and filter before the join), and only then sizing partitions. With AQE on, spark.sql.adaptive.coalescePartitions.enabled merges small post-shuffle partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB by default), so a high spark.sql.shuffle.partitions (default 200) is less harmful than it used to be.

Concurrent applications stress the cluster manager and the storage layer. Each application brings its own driver and executors, so the cluster manager sees thousands of pod creations rather than a few large processes. On Kubernetes that means API server load, scheduler throughput and, in the 4.2 line, a default allocation batch size that went from 10 to 20 executor pods per round (spark.kubernetes.allocation.batch.size). A burst of twenty pods from each of fifty applications arrives all at once; if namespace quotas or node capacity cannot absorb it, pods sit Pending. Lower the batch size for tenants on a constrained cluster, and make quotas explicit rather than discovering them during an incident.

Shuffle storage is the quiet limit. Executors write shuffle files to local disk. With dynamic allocation on Kubernetes there is no external shuffle service, so Spark relies on shuffle tracking: spark.dynamicAllocation.enabled=true together with spark.dynamicAllocation.shuffleTracking.enabled=true, bounded by spark.dynamicAllocation.minExecutors and maxExecutors. Shuffle tracking keeps executors alive while they hold shuffle data that downstream stages still need, which makes scale-down slower and more conservative than people expect. If your cost model assumes executors vanish the instant a stage finishes, it is wrong for shuffle-heavy jobs. Executor local disks must also be large enough for the biggest shuffle; the Kubernetes page documents on-demand persistent volume claims for executors (spark.kubernetes.executor.volumes.persistentVolumeClaim.<name>.options.claimName=OnDemand with storageClass and sizeLimit options) for exactly this reason.

The History Server is the first component many teams forget to scale. It parses event logs on demand, and a busy estate produces a great many of them. Its knobs are spark.history.retainedApplications (default 50), spark.history.fs.update.interval (10 s) and the cleaner, which is off by default (spark.history.fs.cleaner.enabled, with maxAge 7 days when you enable it). A log directory that grows forever is a slow incident: nothing breaks until the listing takes minutes.

Running Spark on Kubernetes as a platform

Kubernetes is the cluster manager most new platforms choose, and it is where an owner has the most levers. In cluster mode the submission client creates a driver pod; the driver then requests executor pods from the API server and owns them through owner references, so deleting the driver garbage-collects its executors. The reference submission from the official page looks like this:

./bin/spark-submit \
  --master k8s://https://<apiserver-host>:<port> \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=5 \
  --conf spark.kubernetes.container.image=apache/spark:4.2.0 \
  --conf spark.kubernetes.namespace=spark-jobs \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/spark/examples/jars/spark-examples_2.13-4.2.0.jar

Every part of that command is a decision you now make centrally rather than per team. The k8s:// master URL points at an API server. The image is yours to build and pin (the project provides bin/docker-image-tool.sh for building images, and apache/spark images on Docker Hub as a starting point). The namespace and service account define the blast radius and the permissions. The local:// scheme in the last line tells Spark the jar is already inside the image, which is the right default for a platform: the artifact that ran in CI is the artifact in the image, not something fetched at submit time.

Pod templates are how a platform injects policy without trusting each team to remember it. spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile point at ordinary Kubernetes pod specs that Spark merges its own settings into. Put the things you need on every pod there: node selectors or tolerations that separate driver capacity from executor capacity, labels for cost attribution, a security context, and sidecars such as a log shipper. Keep the templates in version control next to the images, and treat a template change as a platform release.

Service accounts and RBAC. The driver needs permission to create, watch and delete pods (and typically services and config maps) in its namespace. A missing role shows up as pods is forbidden: User "system:serviceaccount:..." cannot create resource "pods". The fix is a Role and RoleBinding for the service account named in spark.kubernetes.authenticate.driver.serviceAccountName, scoped to that one namespace. Resist the temptation to bind a cluster-wide role to "make it work": a driver that can create pods anywhere is a tenant that can run anything anywhere.

Network policy. In Spark 4.2 executor pods get a restrictive NetworkPolicy ingress by default. This is good platform news and a compatibility trap: tools or client-mode drivers that connect inbound to executors can stop working after the upgrade. The documented escape hatch is to exclude the feature step with spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep. Prefer to understand the connection that broke and open that path in a policy of your own, rather than disabling the default for everyone.

Operating running jobs. You will use these constantly:

# Status of drivers matching a prefix, and killing one
spark-submit --status spark-jobs:spark-pi* --master k8s://https://<apiserver-host>:<port>
spark-submit --kill spark-jobs:<driver-pod> --master k8s://https://<apiserver-host>:<port>

# Logs and the UI of a live driver
kubectl -n spark-jobs logs -f <driver-pod>
kubectl -n spark-jobs port-forward <driver-pod> 4040:4040

The failure table in the official page maps neatly onto platform tickets. Pods Pending indefinitely means insufficient resources or quota: check kubectl describe nodes and the namespace quota. Image pull errors mean registry credentials: use spark.kubernetes.container.image.pullSecrets. Connection refused at submit time means a wrong API server URL: kubectl cluster-info shows the right one. Publish this mapping as a runbook, because the first three questions on any ticket will be the same.

Memory: the taxonomy behind most pages

More Spark incidents come from memory than from any other cause, and most are misdiagnosed because "out of memory" names three different failures. Learn to separate them by who killed the process.

The JVM ran out of heap. The process throws java.lang.OutOfMemoryError: Java heap space and the stack trace tells you where. The remedies are in your hands: raise spark.executor.memory (or spark.driver.memory), reduce the data held per task, increase the number of partitions so each is smaller, or stop collecting to the driver.

The container exceeded its limit. The JVM never saw an error; the operating system or the orchestrator killed the process because total memory (heap plus everything else) crossed the container limit. On Kubernetes the pod shows OOMKilled with exit code 137; on YARN the message is Container killed by YARN for exceeding memory limits. What counts beyond the heap is overhead: off-heap allocations, native libraries, Python worker processes in PySpark, thread stacks and network buffers. The setting that governs it is spark.executor.memoryOverhead. PySpark workloads that use pandas UDFs, Arrow or heavy Python libraries need considerably more overhead than a Scala job with the same heap, and spark.executor.pyspark.memory can additionally cap the Python side. The classic platform mistake is giving every team the same executor template: a Python-heavy team hits OOMKilled while its Java heap sits half empty, and raising spark.executor.memory makes it worse because it grows the container without growing the overhead share you needed.

The node ran out of memory. If requests are far below limits and many pods land on one node, the kernel kills something when the node is exhausted, and Kubernetes evicts pods under pressure. The symptom is executors lost with no application-level message. The remedy is on your side of the fence: requests that reflect real usage, and limits equal to requests for executors so scheduling is honest.

For a platform, the right response is defaults plus guardrails. Set sane defaults for overhead per language, publish a sizing guide with measured examples, and enforce ceilings per team with namespace quotas. Then give teams a way to see their own waste: the executor page shows memory used against allocated, and the metrics endpoints expose JVMHeapMemory and totalGCTime. A team that requests 32 GB executors and peaks at 6 GB is paying for the difference.

Security: the trust model

Spark's security page opens with a statement a platform owner should memorise: security features are off by default. Out of the box there is no authentication on the RPC layer, no encryption in transit, no encryption of spilled data, and no authentication on the web UI. The default posture assumes a trusted network. Your first job is to decide whether that assumption holds in your estate, and for a shared platform it almost never does.

Think in layers and decide a position for each.

RPC authentication and encryption. spark.authenticate=true makes processes in an application prove they share a secret. On YARN and Kubernetes, Spark generates a unique secret per application, which is the right model: a secret shared across applications would let one tenant's process talk to another's. On Kubernetes, prefer file-based secrets (spark.authenticate.secret.file, with .driver.file and .executor.file variants) over passing a literal. For encryption, the SSL namespaces are spark.ssl (the default), spark.ssl.ui, spark.ssl.rpc, spark.ssl.standalone and spark.ssl.historyServer, each with the usual properties: enabled, protocol, keyStore, keyStorePassword, trustStore, trustStorePassword, keyPassword and enabledAlgorithms. One subtlety that catches people: the RPC namespace must be enabled explicitly (spark.ssl.rpc.enabled) and does not simply inherit from spark.ssl.enabled. An older AES-based option exists (spark.network.crypto.enabled, with spark.network.crypto.authEngineVersion=2), but TLS for RPC is the direction the documentation points.

Local disk encryption. Shuffle files, spills and cached blocks land on executor local disks. spark.io.encryption.enabled=true encrypts them, with spark.io.encryption.keySizeBits set to 128, 192 or 256. It costs CPU. Decide by data classification: for regulated data, particularly where Gulf or Egyptian data-residency and sector rules apply to anything written to disk, turn it on or encrypt the node volumes by other means and write down which.

The web UI. This is the most commonly exposed surface. Spark ACLs exist (spark.acls.enable, spark.admin.acls, spark.ui.view.acls, spark.modify.acls, with .groups variants), but there is a crucial caveat in the docs: Spark ships no built-in authentication filter. ACLs decide what an authenticated user may do, but nothing authenticates the user until you install a servlet filter with spark.ui.filters. The documentation mentions org.apache.spark.ui.JWSFilter with a base64url secret key (spark.org.apache.spark.ui.JWSFilter.param.secretKey) as an example. The common production patterns are therefore to keep UIs on a private network, put an authenticating reverse proxy in front, or both. The History Server has its own ACL settings (spark.history.ui.acls.enable, spark.history.ui.admin.acls). Harden the HTTP side with spark.ui.strictTransportSecurity, spark.ui.xContentTypeOptions.enabled, spark.ui.contentSecurityPolicy.enabled and spark.ui.allowFramingFrom where they apply.

The trust model to document for your tenants is short. A Spark job is arbitrary code. Anything a user can write in a transformation or a UDF runs in the executor with the pod's identity and network reach. Spark's security settings protect traffic and consoles; they do not sandbox user code. Isolation between tenants therefore comes from the layer beneath: separate namespaces, service accounts, network policies, and storage permissions scoped to each team.

Secrets and the supply chain

Spark gives secrets many ways to leak, and nearly all of them come from convenience. The rule is simple: no credential belongs in a --conf literal. Command-line arguments are visible in process listings and shell history, and configuration values appear in the UI's Environment tab and in event logs, which the History Server serves to anyone with view access to that application.

Use three mechanisms instead, in this order of preference. First, avoid long-lived credentials entirely: cloud storage access through IAM roles or workload identity attached to the service account means there is nothing to leak. Second, mount Kubernetes secrets as files (spark.kubernetes.driver.secrets.<secret>=/mount/path and the executor equivalent) so the value never appears in configuration. Third, expose a secret as an environment variable with spark.kubernetes.driver.secretKeyRef.ENV_NAME=secret:key (and the executor variant) when a library demands one. For anything that does pass through configuration, spark.redaction.regex masks matching values in the UI and logs; confirm its default pattern in your version's configuration page, and extend it for your own naming conventions rather than trusting it to catch everything. For rotation and central storage, our Vault guide covers the pattern of short-lived dynamic credentials injected at start.

The supply chain for Spark has more links than most teams count:

1. The base image. Whether you build from the official apache/spark image or your own, pin by tag and ideally by digest, and rebuild on a schedule. A Spark image carries a JVM, Python, OS packages and the Spark distribution, and each has its own advisories. The security page on the project site tells you which release lines receive patches; the downloads page currently lists 4.2.0 alongside patch lines for 4.1, 4.0 and 3.5. Older versions can carry known vulnerabilities, so "we are on an old line because it works" is a risk statement, not a neutral one.
2. The distribution itself. Verify downloaded tarballs against the project signatures using the published KEYS file at https://downloads.apache.org/spark/KEYS. Do it in the image build, not by hand.
3. Dependencies. Python wheels, JVM jars and connector packages are installed into images. Jobs that use --packages to resolve dependencies at start-up pull from the internet at runtime, which defeats pinning and fails when a repository does. Bake dependencies into the image, and vendor connector jars you rely on.
4. User code. Teams build their application layer on your base image. Make the base image the only supported route and scan the result in CI; our Trivy guide covers image scanning, and Docker covers the build-side controls.

A practical gate: the platform builds the base image weekly, scans it, and publishes it with a date-stamped immutable tag. Team images must derive from a base no older than a stated number of weeks, and the admission policy on the cluster can enforce the registry and the age rule. This turns "please upgrade" into something the platform checks mechanically.

Multi-tenancy

A shared Spark platform multiplies every problem by the number of tenants, and the central design question is where the hard boundary sits. On Kubernetes the answer is the namespace: one per team or per environment, each with a ResourceQuota capping total CPU, memory and pod count, a service account scoped to that namespace, a NetworkPolicy, and storage credentials limited to that team's data. Teams point their jobs at their namespace with spark.kubernetes.namespace. On YARN the equivalent is the queue (--queue), with capacity limits configured in the scheduler. Standalone clusters offer limited isolation and are a poor fit for sharing between untrusting teams.

Quotas are what stop one team's backfill from starving another's production job, and they work together with Spark's own controls. Dynamic allocation bounded by spark.dynamicAllocation.maxExecutors keeps an individual application from asking for the moon; the namespace quota is the backstop when a team ignores the setting. Enforce the application-level limit in your submission wrapper so users get a clear message ("your job asks for 800 executors, the maximum is 200") instead of a pile of Pending pods. The allocation batch size matters here: a quota-bound namespace with a large batch makes failures noisy, and lowering it spreads requests out.

Inside one application, Spark can share executors between concurrent jobs. The scheduler mode spark.scheduler.mode=FAIR with named pools is documented on the job scheduling page; confirm the pool configuration against that page for your version before depending on it. Pools help a long-lived shared application (a notebook server, a Connect server) divide time between users. They are fairness controls, not security controls.

Spark Connect and tenancy. A Connect server speaks gRPC to thin clients (default port 15002), so a team can connect from a notebook or a service without a local JVM and without sharing dependencies with the server. That is excellent for platform operations: you can upgrade the server independently of the clients, centralise secrets and storage access on the server side, and keep client environments small. But be precise about what it isolates. Sessions are isolated from each other logically, yet a shared server is a shared JVM: one user's runaway query can exhaust memory for all, and code that runs in the server runs with the server's identity. Treat a shared Connect server as a convenience for trusted peers, and give separate servers (one per team, in each team's namespace) to tenants that must not share fate. Know the feature limits as well: there is no RDD API, no SparkContext, no df._jdf, and only session-level configuration, so code from older notebooks will fail on contact. Check the Spark Connect overview page for the current list before promising migration.

Observability for a platform

The questions a platform owner asks are different from a job author's. The author asks why a stage is slow. You ask which team's jobs are failing, whether the History Server is healthy, which namespaces are near quota, and whether executor pods are being created faster than they are scheduled. Four sources answer them.

Event logs and the History Server. Enable spark.eventLog.enabled=true and point spark.eventLog.dir at shared storage (object store or HDFS). Since Spark 4.0 rolling (spark.eventLog.rolling.enabled) and compression (spark.eventLog.compress) are on by default, which makes long streaming applications' logs manageable. Run the History Server (./sbin/start-history-server.sh) with spark.history.fs.logDirectory pointing at the same location; it listens on spark.history.ui.port, default 18080. Give each team its own prefix if your permission model needs it, set the cleaner and a retention age, and remember that the event log contains configuration values, so redaction matters here too.

Metrics. Spark uses Dropwizard metrics, configured in conf/metrics.properties or through spark.metrics.conf.*. Sinks include console, CSV, JMX, a metrics servlet, a PrometheusServlet (marked experimental), Graphite, Slf4j and StatsD. The built-in Prometheus-format endpoints are :4040/metrics/prometheus/ on the driver, :4040/metrics/executors/prometheus/ for executor metrics aggregated at the driver, :8080/metrics/master/prometheus/ on a Standalone master and :8081/metrics/prometheus/ on a worker. On Kubernetes, scraping each short-lived driver is the awkward part; pod annotations in your template plus service discovery in Prometheus solve it, and dashboards in Grafana make it legible. The signals to keep are executor JVMHeapMemory and totalGCTime, failedTasks, shuffle read and write volume, and task-level executorRunTime, memoryBytesSpilled, diskBytesSpilled and peakExecutionMemory. Long GC time against run time means memory pressure; rising spill means partitions or memory are mis-sized.

Alerts worth having on day one are few. Applications failing at a rate above the baseline per namespace; pods Pending longer than a threshold; History Server unreachable or its log listing taking too long; event log directory growth; executor loss rate by reason; and any UI or REST port reachable from outside the intended network. Page on the last, ticket on the rest.

Cost

Spark bills show up as compute, storage and network, and most of the controllable cost is in compute that sits idle or recomputes work. The levers, from most to least effective, are these.

Make idle executors disappear. Dynamic allocation with shuffle tracking releases executors that hold no needed shuffle data, within the min and max you set. A job that holds 100 executors through a long driver-only phase, such as a collect followed by Python processing, is paying for nothing. Teach teams to look at the Executors tab over time and to set a realistic minimum (often zero or one) and a ceiling.

Match capacity types to roles. Drivers on on-demand nodes; executors on spot or preemptible nodes for jobs that tolerate task and executor loss. This is general cloud practice rather than something the Spark documentation prescribes, so test it with your own workloads. Jobs that are shuffle-heavy and long-running lose the most to preemption, because lost executors hold shuffle files; the more a job shuffles, the more conservative its capacity should be.

Attribute spend. You cannot discuss cost without ownership. Put a team and a cost-center label on every driver and executor pod through the pod templates; the configuration also supports per-job labels with spark.kubernetes.driver.label.* and spark.kubernetes.executor.label.*, which you should confirm against the configuration page for your version. Then a cost tool or your cloud's billing export can group by label. Without labels, you will argue about the bill instead of fixing it. Your Terraform estate and its tagging policy are the place to make labels mandatory for the node pools themselves.

Upgrades and migrations

Spark's pace means an upgrade programme, not an upgrade event. Spark 3.5 patch releases, plus 4.0, 4.1 and 4.2 lines are all listed as maintained at the time of writing, which gives you room to plan, but not to stand still: older versions can carry known vulnerabilities, and each skipped hop makes the next harder.

The method that works is the same every time. Read all three migration guides (SQL, Core and PySpark) for every version you cross. Run the new version against a copy of production data, never only against unit fixtures. Pin the new image tag, and run old and new side by side on the same inputs, comparing outputs. When behaviour differs, use the documented legacy flags as a diagnostic: flip one, see whether the difference disappears, and you have found the cause. Then decide whether to fix the code or carry the flag temporarily with an owner and an expiry date.

The biggest hop in recent history is 3.5 to 4.0. Its items, in the order they usually hurt:

1. ANSI SQL mode is on by default (spark.sql.ansi.enabled=true). Invalid casts, arithmetic overflow and division by zero now raise errors rather than returning null. This is the most common regression: jobs that "worked" for years while quietly producing nulls now fail on the first bad row. Errors read like [DIVIDE_BY_ZERO] Division by zero, naming try_divide and the config as ways out. The right fix is usually to use the try_ variants where nulls are the intended behaviour, and to let real data problems surface. The flag is a short-term escape.
2. Scala 2.13 only, Java 17 or newer, and jakarta. Any in-house jar compiled for Scala 2.12 fails with NoSuchMethodError or ClassNotFoundException; anything reaching javax.servlet breaks because the servlet API moved to jakarta. Libraries need rebuilding or replacing, and that is calendar time you cannot compress.
3. Table and type defaults. CREATE TABLE without USING or STORED AS now creates a table using spark.sql.sources.default (Parquet) rather than a Hive table (spark.sql.legacy.createHiveTableByDefault=true restores it). JDBC type mappings changed for MySQL, Postgres, Oracle and others, each with a spark.sql.legacy.<db>.* flag. The ORC default codec moved to zstd, and spark.sql.maxSinglePartitionBytes changed from unlimited to 128m.
4. Operational defaults. Event log rolling and compression, worker cleanup, RocksDB as the default backend for the shuffle service database, ReadWriteOncePod for Kubernetes persistent volume claims, and a less aggressive speculation default (multiplier 3, quantile 0.9). These change how your platform behaves rather than how jobs compute.
5. Removals. Mesos support, Hive metastore clients older than 2.0.0, and the old config spellings for Parquet rebase modes, which were renamed without the legacy segment.

4.0 to 4.1 brought the Master REST API on by default, S3A magic committer on by default, RDD checkpoint compression, and a native Netty I/O mode default, each with a documented switch to restore the old behaviour, plus Parquet reader and Thrift server ordinal position changes. 4.1 to 4.2 brings the order-independent shuffle checksums covered earlier, the virtual system catalog (with system.builtin and system.session, which can shadow persistent schemas named builtin or session; spark.sql.legacy.persistentCatalogFirst=true restores the old resolution order), case-insensitive duplicate CTE detection that fails at parse time with DUPLICATED_CTE_NAMES and cannot be switched off, NATURAL JOIN honouring spark.sql.caseSensitive, the Kubernetes batch size and NetworkPolicy changes, and the deprecation of the Derby JDBC source.

For a platform, three structural habits ease every future upgrade. First, publish a version policy: which versions are supported, for how long, and the date each is retired. Second, run a canary suite: a handful of representative jobs from different teams, run against each candidate image nightly, so you find breakage before the teams do. Third, use Spark Connect where it fits, because decoupling client and server versions lets you upgrade the server while clients move at their own pace.

Governance and incident playbooks

Governance for Spark is the set of defaults you own, the limits you enforce, and the evidence you keep. Keep it small enough that people follow it.

Own the defaults. Publish a platform spark-defaults.conf baked into the image, remembering the precedence: values set in code beat spark-submit flags, which beat the defaults file. That means a team can override anything you set, which is why the things you must guarantee (authentication, redaction, labels, namespace) belong in pod templates, admission policy and the submission wrapper, not only in a defaults file. Enforce the few that matter: executor ceiling, image source, labels, secret handling. Record exceptions with an owner and an expiry date, especially for legacy flags.

Below are playbooks for the incidents you will actually see. Each starts with what to look at, in order, and what you change.

Playbook 1: jobs stuck, executors never start. Pods Pending: kubectl describe pod and kubectl describe nodes will say insufficient CPU or memory, an unbound claim, or a quota rejection. If it is quota, the team asked for more than the namespace allows; lower their request or raise the quota deliberately. If it is capacity, check whether the autoscaler is provisioning nodes and how long it takes relative to the job's patience. If pods fail with image errors, check the pull secret and the tag. If there is no pod at all, the driver lacks permission: read the driver log for forbidden.

Playbook 2: executors killed, OOMKilled. Confirm with the pod status and exit code 137. Compare heap use with the pod limit, then raise spark.executor.memoryOverhead first, and the Python memory cap if PySpark. Check for a skew-driven task that holds one huge partition. Only then raise heap. Afterwards, update the sizing profile so the next team does not repeat the failure.

Playbook 3: after an upgrade, jobs fail on data that was fine. This is almost always ANSI mode. Read the error class (CAST_INVALID_INPUT, DIVIDE_BY_ZERO), find the offending expression, and decide between try_cast, try_divide or cleaning the data. Use spark.sql.ansi.enabled=false only per job with an expiry, never platform-wide.

Where Spark stops

A senior engineer is credited with knowing when not to use the tool. Spark is a distributed batch and micro-batch engine built for data that does not fit on a machine, and its costs (a JVM per executor, a scheduler, shuffles, start-up latency) pay off only above a certain scale.

Small data. If the data fits comfortably in one machine's memory, a single-node engine will usually be faster and far cheaper to run than a cluster, because it has no distributed overhead. Spark's local mode exists for development, not as a justification for a cluster. Ask before building a platform: how big is the largest real dataset, and how big will it be in two years?

Deep learning training. MLlib covers distributed classical machine learning, and Spark excels at feature engineering and batch scoring with pandas UDFs. Training large neural networks is usually better done in dedicated frameworks, with Spark handing prepared data over; our Ray guide covers a common pairing. Treat that as an architectural judgement rather than an official statement from the Spark project.

Managed alternatives. If your team does not want to operate Kubernetes, clusters, images, upgrades and security hardening, a managed service moves much of that burden to a vendor. Databricks, EMR, Dataproc and others exist, and our Databricks guide covers one; what they take on, what they cost, and what lock-in they introduce are questions for the business case, not for this guide's facts. The honest comparison is the total cost of your platform team plus infrastructure versus the managed premium, and for many organisations in the region the data-residency options of each provider are decisive.

Running Spark as a platform for other teams

Everything above converges on an operating model. A platform is a product with customers, and its quality shows up in how little teams need to think about it.

A paved road. Offer one supported way to run a job: a base image, a submission wrapper or operator, templates, defaults, a namespace-per-team scheme, and logging and metrics already wired. Teams that stay on the road get security, observability and cost attribution for free. Teams that leave it own the consequences, in writing.

Feedback and metrics for the platform itself. Time to onboard a team; percentage of jobs on supported versions; median time from submission to the first task running; job failure rate by cause (user code, platform, infrastructure); share of spend with an owner label; age of the oldest base image in use. These are the numbers you bring to leadership, and they are easier to defend than a general claim that the platform is "healthy".

The review checklist

Use this for pull requests that touch Spark jobs, platform templates or design documents. A review that asks these questions catches most production problems earlier than any dashboard.

For a job or its configuration:

• Is the Spark and Python or Scala version the supported one, and are all jars built for Scala 2.13?
• Does it rely on ANSI-off behaviour, a legacy flag or a deprecated configuration name? Is there an owner and expiry?
• Are secrets passed through mounted files, environment references or IAM, and never as --conf literals?
• Is there any collect(), toPandas() or unbounded show() on data that can grow?
• Are joins planned deliberately: small side broadcast, keys checked for skew, columns pruned before the join?
• Are input files a sensible size, and does the write partition by a column with reasonable cardinality?
• Is caching justified, and is there an unpersist()?
• Are Python UDFs avoided in favour of built-in functions or pandas UDFs, and is overhead sized for Python?
• Are dynamic allocation bounds set, with shuffle tracking on Kubernetes?
• For streaming, is the checkpoint location durable, unique to the query and access-controlled?

For a platform change or design document:

• Which namespaces, quotas, service accounts and network policies are affected, and what is the blast radius if this is wrong?
• Does the change alter a default that teams rely on, and how is it announced and rolled back?
• Is the driver on capacity that will not be reclaimed, and executors on capacity that can be?
• Are UIs, the Master REST API and the History Server unreachable from outside the intended network, and has that been verified from outside?
• Is there authentication in front of every UI, not just ACLs?
• Are pods labelled for cost attribution, and are metrics and event logs collected for the new component?
• Which images are used, how old are they, were they scanned, and who rebuilds them?
• What is the upgrade path, and has a canary job proven it on this change?
• What alerts exist for the new failure modes, and who receives them?
• What happens when it fails at three in the morning, and is that written in a runbook?

The complete picture

Pull the threads together. A Spark application is a driver plus executors under a cluster manager; the driver is the single point of failure and executors are disposable. Scale is bounded in turn by shuffle volume, task count, concurrency against the cluster manager, shuffle storage and the History Server. On Kubernetes you control the platform through images, pod templates, service accounts, quotas, network policy and the submission wrapper. Memory failures divide into heap, container overhead and node pressure, and the diagnosis starts with who killed the process.

Costs fall with idle executors released, driver and executor capacity chosen separately, less shuffle, right-sized requests and labelled spend. Upgrades are a programme: three migration guides per hop, legacy flags as diagnostics, side-by-side runs, canary jobs and a published version policy, with 3.5 to 4.0 as the large hop and ANSI mode as its most common regression. Governance is a few enforced defaults and recorded exceptions; incidents are handled from runbooks; and the platform knows what it is not for.

Where the series leaves you

You started this series with a local session and a DataFrame. You now have a way to reason about Spark at the level of an organisation: where it fails, what it costs, who can reach it, how to upgrade it, and when to choose something else. The skills that remain are practice. Run the game days. Measure your own platform against the checklist. Read the migration guides before the release, not after the incident. For the neighbouring pieces of the stack, continue with Kubernetes for the substrate, MLflow for tracking what the jobs produce, Delta Lake for the table layer, and Airflow or Dagster for orchestration.

Above all, keep one habit from this level: when something breaks in a shared estate, ask which component failed, which team's work was affected, and what change to the platform makes the next failure smaller. That habit, more than any flag, is what ownership of Spark consists of.

Sources

• Apache Spark documentation overview and requirements: https://spark.apache.org/docs/latest/
• Downloads, versions, Maven coordinates: https://spark.apache.org/downloads.html
• Release index: https://spark.apache.org/releases/
• Running Spark on Kubernetes: https://spark.apache.org/docs/latest/running-on-kubernetes.html
• Security: https://spark.apache.org/docs/latest/security.html
• Monitoring and instrumentation: https://spark.apache.org/docs/latest/monitoring.html
• SQL migration guide: https://spark.apache.org/docs/latest/sql-migration-guide.html
• Core migration guide: https://spark.apache.org/docs/latest/core-migration-guide.html
• SQL performance tuning: https://spark.apache.org/docs/latest/sql-performance-tuning.html
• Spark Connect overview: https://spark.apache.org/docs/latest/spark-connect-overview.html
• Job scheduling: https://spark.apache.org/docs/latest/job-scheduling.html
• Configuration: https://spark.apache.org/docs/latest/configuration.html
• Signing keys: https://downloads.apache.org/spark/KEYS

Where this picks up

Part three of three. At senior level an interviewer is no longer checking that you know what a shuffle is. They want to hear how you would run Spark for sixty engineers, what you do when the cluster is on fire, and how you would defend a budget. Expect open prompts with no single right answer, and expect follow-ups that probe the weakest part of whatever you just said. This guide was written against Apache Spark 4.2; say the version you are talking about in an interview, because defaults changed enough between 3.5 and 4.x that an unqualified answer can be wrong.

The pattern that wins in these interviews has four moves. Clarify the workload and constraints before designing. State the first bottleneck you expect, rather than listing every component. Name the failure modes and how you would detect them. Say what you would not build and why. The model answers below follow that shape. They are written as speech, so rehearse the reasoning rather than memorising the wording.

The thirty-second answer

"Spark is a distributed engine for batch and micro-batch data processing. An application is one driver plus many executors under a cluster manager (Kubernetes, YARN or Standalone in Spark 4). As a platform owner I care about four things: the driver is a single point of failure so it gets reliable capacity, the shuffle is where scale breaks first, every security feature is off by default so I turn them on deliberately, and upgrades are a programme because Spark 4 changed defaults such as ANSI mode, Scala 2.13 and Java 17 or newer. I also know where Spark is the wrong tool: small data, low-latency serving and workflow orchestration."

Practise saying that in under thirty seconds. It signals that you think in failure modes and ownership rather than in API calls.

System-design prompts

Design a multi-tenant Spark platform on Kubernetes for fifty data engineers

Clarify first. How many concurrent applications at peak? Batch only, or streaming and interactive too? Which data stores and which regions? Any regulated data? What does the team do today? A strong candidate asks at least three of these before drawing anything.

A model answer. "I would use Kubernetes as the cluster manager because the organisation already runs it and Spark supports it natively, with a pod per driver and executor and no external shuffle service. Each team gets a namespace with a ResourceQuota, a service account whose Role covers pods, services and config maps in that namespace only, and a NetworkPolicy. Jobs are submitted through a thin wrapper or operator, so teams declare an image, a profile and resource limits rather than assembling spark-submit flags. The platform owns the base image, pinned and rebuilt weekly, and the pod templates, set through spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile. The templates put drivers on on-demand nodes and executors on spot nodes, apply a team and cost-center label, and set a security context. Dynamic allocation is on with shuffle tracking (spark.dynamicAllocation.shuffleTracking.enabled=true) and a ceiling enforced by the wrapper. Event logs go to object storage with rolling and compression, which are default since 4.0, and a History Server reads them. Metrics are scraped through the Prometheus endpoints and shown in Grafana. Authentication and TLS are enabled explicitly because Spark ships with them off, and the UIs sit behind an authenticating proxy because Spark has no built-in authentication filter."

First bottleneck. "At fifty engineers I expect the first pain to be quota and burst behaviour rather than raw compute: many applications starting at once create many pods, and Spark 4.2 raised the executor allocation batch size from 10 to 20. I would watch Pending pods and API server latency, and lower the batch size for constrained namespaces."

Failure modes to name. Driver on reclaimed capacity; OOMKilled from too little memory overhead on Python jobs; History Server slowness from unbounded event logs; a tenant requesting more than its quota; and UI exposure.

What I would not build. "A shared long-lived cluster that all teams submit to without namespaces. It is cheap until the first tenant's runaway job, and then there is nothing to attach a quota or an audit trail to."

What a weak answer does. Starts with a diagram of Spark internals, never asks about the workload, and treats security as a final slide.

Design a platform that lets notebooks and services use Spark without everyone running a driver

A model answer. "That is the use case for Spark Connect, available as a gRPC server (default port 15002) with thin clients, so a notebook or service does not need a local JVM or the server's dependencies. It decouples client and server versions, which helps upgrades, and lets me centralise storage credentials on the server. I would be explicit about the limits: no RDD API, no SparkContext, no df._jdf, and only session-level configuration, so some legacy notebooks will fail and need rewriting. On tenancy I would not claim that Connect isolates users strongly. Sessions are separate, but a shared server is one JVM, so one user's memory-hungry query can affect everyone. I would run a server per team in the team's namespace for tenants that must not share fate, and use a shared server only among trusted peers with quotas on what each session can do."

Follow-up an interviewer might ask. "Why not just give each analyst a driver pod?" A good answer weighs cost (idle drivers that hold memory all day), dependency drift between notebooks, and credential sprawl against the loss of the full RDD and SparkContext API under Connect. Neither is free.

Incident-response questions

For each incident, interviewers want an order of investigation, not a list of possibilities. Say what you look at first and why.

Many jobs are stuck and executors never start

"First I establish whether pods exist. If drivers exist but executors are Pending, kubectl describe pod and kubectl describe nodes will show insufficient resources, an unbound volume claim or a quota rejection. If it is quota, one team asked for more than its namespace allows. If it is node capacity, I check whether the autoscaler is adding nodes and how long that takes. If executor pods do not exist at all, the driver may lack permission: the driver log will contain pods is forbidden and the fix is a Role and RoleBinding for its service account. If pods exist but fail to start with image errors, I check the pull secret and tag. Then I look at what changed: a new batch size after an upgrade, a template edit, a quota edit."

Executor pods are killed with exit code 137

"137 plus OOMKilled means the container exceeded its memory limit and was killed from outside, so there is no Java stack trace. The heap may be fine. I compare actual heap with the pod limit and raise spark.executor.memoryOverhead first, because overhead covers off-heap memory, native code and Python workers. For PySpark with pandas UDFs, Python memory is a large part of it, and spark.executor.pyspark.memory can cap it. I also check for skew: one task with a huge partition can push a single executor over. Raising spark.executor.memory alone can make the pod larger without fixing the ratio, so it is not my first move. Afterwards I update the sizing profile so the next team does not repeat the failure."

After an upgrade to Spark 4, jobs that ran for years now fail

"I suspect ANSI mode first, because spark.sql.ansi.enabled became true by default in 4.0. Invalid casts, overflow and division by zero now raise errors such as CAST_INVALID_INPUT or DIVIDE_BY_ZERO where Spark 3 returned null. I would read the error class, find the expression, and decide whether the right fix is try_cast or try_divide where nulls are intended, or cleaning the source data. For a deadline I would allow spark.sql.ansi.enabled=false per job, with an owner and expiry, never platform-wide. If errors are not ANSI, I check for Scala 2.12 jars (NoSuchMethodError, ClassNotFoundException), the javax to jakarta move, and the migration guide's table and JDBC type changes, each of which has a legacy flag I can use to confirm the diagnosis."

A job got much slower overnight with no code change

"I compare the slow run with a good run in the History Server, stage by stage. Typical causes: input grew and a table crossed the broadcast threshold (default 10 MB), so a broadcast join turned into a shuffle join; a skewed key appeared; a source started producing many small files; or spill increased. I check the plans with explain('formatted') on both runs and diff them. If an upgrade happened, I also consider that Spark 4.2 enables order-independent shuffle checksums, which can roll back and re-run stages when an indeterminate stage is retried, so I look for retried stages. I would not touch cluster size until I know which stage dominates."

The driver keeps dying on a job that worked last month

"Driver death is application death, so I look for what grew on the driver: a collect() or toPandas() on data that grew, an enormous broadcast, a plan that got very large, or a huge number of tasks from small files. The signature is heap exhaustion in the driver log or exit code 137 on the driver pod. I fix the cause rather than the symptom: remove the collect, compact the files, or restructure the loop. I raise spark.driver.memory only as a stopgap."

Someone finds a database password in the Spark UI

"Rotate it immediately, then investigate. A secret in the Environment tab usually means it was passed as a --conf value, or embedded in a JDBC URL, and the redaction pattern did not match the key name. The same value is probably in event logs, which the History Server serves, and in process listings. I would remove the literal in favour of a mounted Kubernetes secret, an environment reference with spark.kubernetes.driver.secretKeyRef.ENV_NAME, or better workload identity so there is no password. Then I extend spark.redaction.regex for the naming pattern and add a CI check for secret-looking --conf flags."

The History Server is empty or painfully slow

"Empty: check that spark.eventLog.enabled is true, that the log directory is writable by users (Spark does not set its permissions; the documented model is a sticky, world-writable directory owned by the History Server superuser), and that spark.history.fs.logDirectory matches. Slow: the log directory has grown without bound, because the cleaner (spark.history.fs.cleaner.enabled) is off by default. I would enable it with a retention age, and consider whether spark.history.retainedApplications and update interval suit the volume."

Questions that probe depth

Why is the driver a special case when you design capacity?

"Every other process can be lost and replaced. Lose an executor and the scheduler reruns its tasks. Lose the driver and the application is gone, together with everything it computed in memory. So drivers go on dependable capacity and executors may use cheaper, interruptible capacity. A candidate who says 'put everything on spot' has not thought about this."

Explain dynamic allocation on Kubernetes and its trap

"With no external shuffle service, Spark uses shuffle tracking: executors holding shuffle data still needed downstream are kept alive. So scale-down is slower than people expect, and a cost model that assumes executors vanish the instant a stage finishes will overestimate the savings for shuffle-heavy jobs. The configuration is spark.dynamicAllocation.enabled=true and spark.dynamicAllocation.shuffleTracking.enabled=true with min and max executors. I would also mention the burst effect of the allocation batch size."

What does Spark's security model protect, and what does it not?

"Spark protects traffic and consoles: RPC authentication with a shared secret (a unique secret per application on Kubernetes and YARN), TLS, encryption of spilled and shuffle data on local disk, and UI ACLs. All of it is off by default. It does not protect against user code, because a job is arbitrary code running with the pod's identity and network reach. And the ACLs only decide what an identified user may do: there is no built-in authentication filter, so without a filter or an authenticating proxy, ACLs protect nothing. Tenant isolation therefore comes from namespaces, service accounts, network policy and storage permissions."

When would you choose a managed service over operating Spark yourself?

"When the organisation does not want to own images, upgrades, security hardening, and an on-call rotation for a cluster manager, and when the managed premium is lower than the cost of the team that would otherwise do that. Managed offerings such as Databricks, EMR or Dataproc each have their own regions, pricing and lock-in; I would compare total cost and data-residency options rather than quote features. The honest answer includes the case for staying: portability, control over versions, and existing Kubernetes expertise."

Arguing a trade-off to leadership

At this level you are judged on how you frame decisions for people who do not care about executors. The structure is: the decision, the options, what each costs and risks, your recommendation, and what would change your mind. Keep technical terms to the minimum, and attach a number to every claim.

"Why does the platform cost so much, and can we halve it?"

"Spend breaks into idle capacity, work that could be avoided and unlabelled usage. I would first show cost by team, which requires cost-center labels on pods, because we cannot discuss what we cannot attribute. Then three levers: release idle executors with dynamic allocation, move executors to interruptible nodes for jobs that tolerate loss while keeping drivers on stable ones, and cut requested-versus-used memory waste, which is typically the largest hidden line. I would promise a measured reduction on the jobs we have profiled, not a blanket figure, and say that shuffle-heavy long jobs gain least from interruptible capacity. The risk to name is reliability: more interruptions mean more retried work."

"Why can't we skip the Spark 4 upgrade?"

"Staying on 3.5 is possible for now, since it still gets patch releases, but each hop skipped makes the next one larger, and older versions can carry known vulnerabilities. Spark 4 requires Java 17 or newer and Scala 2.13, which means rebuilding any internal jars, and that work does not shrink with time. The risk of upgrading is behaviour change, mainly ANSI mode, which I would manage with side-by-side runs and canary jobs. I would propose a dated plan with one pilot team, a published support window and a rollback path, rather than a big-bang migration."

"Can't every team just share one big cluster?"

"It is the cheapest option on paper, and I would call it a risk acceptance. One team's runaway job, memory leak or misconfiguration becomes everyone's outage, we lose cost attribution, and a security flaw in one tenant's code reaches all. Namespace-per-team costs some idle overhead and needs automation, but contains failures and makes spend visible. I would offer shared capacity only among trusted peers with quotas."

What a strong answer covers that a weak one misses

Interviewers describe the gap between candidates in surprisingly consistent terms.

A weak answer lists features. It treats Spark as a library, describes the executor and driver but never says which one matters more for availability, and gives numbers (partitions, memory) as if they were universal. It reaches for "add more nodes" first. It says security is handled by the network. It does not mention versions, so its defaults may be the wrong ones for Spark 4.

A strong answer starts with questions, names the first bottleneck, and gives failure modes alongside the design. It separates the three kinds of out-of-memory failure by who killed the process. It treats upgrades as a programme and knows the headline changes: ANSI mode default, Scala 2.13 only, Java 17 or newer, jakarta, Mesos removed. It knows that Spark Connect is a convenience boundary, not a tenant boundary. It attaches owners to exceptions and expiry dates to flags. It says what it would not build. And it can say, without defensiveness, when a different tool would be the right choice.

Sixty-second self-test

Answer each aloud without notes, then check yourself against the sections above.

1. Why does the driver get on-demand capacity and the executors get spot?
2. Name the three kinds of "out of memory" and who kills the process in each.
3. What is shuffle tracking and why does Kubernetes need it for dynamic allocation?
4. Which Spark security features are on by default? What is the catch with UI ACLs?
5. What changed by default in Spark 4.0 that causes the most upgrade failures?
6. What are the two Kubernetes changes in Spark 4.2 that affect a platform owner?
7. Why is a shared Spark Connect server not a tenant boundary?
8. What would you say to leadership about cost, in three levers?
9. Name two cases where Spark is the wrong tool.
10. What is the first thing you ask when given an open design prompt?

If you hesitated on more than two, reread the matching section and try again tomorrow. For the surrounding stack, the Kubernetes guide and the Airflow guide give you the neighbouring vocabulary an interviewer will expect you to move between.

Where this picks up

Part three of three. At this level the problems are not syntax errors. They are failures that appear only when many teams, large data and long uptimes combine: a pod burst that exhausts a quota, a History Server that slowly stops responding, an upgrade that turns silent nulls into hard failures. This guide is organised as traps first, then habits, then review checklists. Facts here were checked against the Apache Spark 4.2 documentation; where the documentation did not settle something, the text says so and you should test it in your own cluster.

Production traps that only appear at scale

Each trap is written as symptom, cause and fix, because that is the order in which you meet them.

Executors start in a burst and the namespace stalls

Symptom. Several jobs start together, many executor pods sit Pending, and some applications make no progress for minutes.

Cause. Spark creates executor pods in batches, and in 4.2 the default batch size is 20 instead of 10 (spark.kubernetes.allocation.batch.size). Fifty applications each asking for a batch of 20 is a thousand pod requests at the same moment, against a quota or a node pool that was sized for the old behaviour.

Fix. Make the batch size part of your tenant profile, lowering it to 10 for constrained namespaces. Compute the worst-case burst (applications that can start together multiplied by the batch size) and compare it to the namespace quota. Make the wrapper reject requests above the executor ceiling with a readable message instead of letting them queue.

The driver sits on the same cheap capacity as the executors

Symptom. Long jobs die at random hours with the whole application lost, even though the executors are designed to survive loss.

Cause. The driver is a single point of failure for its application. Putting it on interruptible nodes because "everything runs there" means a node reclaim kills the entire job and everything held in memory.

Fix. Use the driver and executor pod templates to put drivers on on-demand nodes and executors on spot or preemptible nodes, with node labels and tolerations in the templates. This is general cloud practice, not a Spark rule, so measure the interruption rate of your own spot pools before relying on it.

OOMKilled with a half-empty heap

Symptom. Executor pods end with exit code 137 and OOMKilled; the Spark UI shows heap use well under the limit; there is no Java stack trace.

Cause. The container exceeded its limit through memory outside the heap: native allocations, network buffers and, in PySpark, the Python worker processes. The JVM never raised an error because the operating system killed the process from outside.

Fix. Raise spark.executor.memoryOverhead before spark.executor.memory, and cap the Python side with spark.executor.pyspark.memory where appropriate. Give Python-heavy teams their own sizing profile. Raising heap alone grows the container without growing the share you were short of.

Shuffle-heavy jobs on spot nodes recompute forever

Symptom. A long job on interruptible executors keeps rerunning earlier stages, and run times vary wildly between nights.

Cause. Executors hold shuffle files on local disk. Losing one forces the producing stage to run again for the missing output, and with enough interruptions the job spends more time recomputing than progressing.

Fix. Tier jobs by shuffle volume and capacity: shuffle-heavy and long-running jobs get stable executors, light jobs get spot. Keep shuffle tracking enabled with sensible min and max executors, and be aware that on Kubernetes shuffle tracking deliberately keeps executors alive while they hold needed shuffle data, so scale-down is slower than your cost model might assume.

An ACL that protects nothing

Symptom. The security review says UI ACLs are enabled, yet an unauthenticated request to a driver or History Server port returns the page.

Cause. Spark ships no built-in authentication filter. spark.acls.enable decides what an authenticated identity may do, but nothing authenticates the user until you install a servlet filter through spark.ui.filters or place an authenticating proxy in front.

Fix. Test from outside the cluster network, not from inside it. Put UIs behind a proxy that authenticates users, or configure a filter, and then use the ACL settings to decide who may view or modify. Treat the check as part of the platform's acceptance test.

The History Server decays slowly

Symptom. The History Server takes minutes to list applications, then stops responding. Nobody changed anything recently.

Cause. The log directory grew without bound. The cleaner (spark.history.fs.cleaner.enabled) is off by default, and each finished application, plus every long streaming job's rolling logs, adds to what the server has to scan.

Fix. Enable the cleaner with a retention age (the default maxAge when enabled is 7 days), size spark.history.retainedApplications for your volume, and alert on directory size. Do the same for the driver log directory if you persist driver logs.

Upgrading turns silent nulls into failures

Symptom. After moving to Spark 4, jobs that ran for years fail with CAST_INVALID_INPUT or DIVIDE_BY_ZERO.

Cause. ANSI SQL mode became the default in 4.0 (spark.sql.ansi.enabled=true). Invalid casts, overflow and division by zero now raise errors; Spark 3 returned null and moved on. The jobs were always producing bad rows. They are now telling you.

Fix. Use try_cast or try_divide where null is the intended behaviour, and fix the data where it is not. Allow spark.sql.ansi.enabled=false only per job, with an owner and an expiry date, never as a platform default.

Stages that re-run after the 4.2 upgrade

Symptom. After moving to 4.2, some jobs show retried stages or longer run times that were not there before.

Cause. Order-independent shuffle checksums are on by default in 4.2. When an indeterminate stage is retried and its output differs, Spark rolls back and re-runs dependent stages to avoid combining inconsistent data. The behaviour is a correctness feature, and the retries mean the job's output was not deterministic.

Fix. Do not turn it off first. Find the nondeterministic step (a non-deterministic function, an unordered operation feeding a retry-sensitive stage) and fix it. If you need to compare behaviour, spark.sql.shuffle.orderIndependentChecksum.enabled=false and spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch=false restore the old behaviour for the comparison. Whether a given job's retries come from this feature should be confirmed in a real run; the migration guide describes the mechanism but your jobs will tell you the symptoms.

Tools that stop reaching executors

Symptom. After 4.2, a client-mode driver, a debugging tool or a metrics scraper that connects inbound to executor pods times out.

Cause. Executor pods now get a restrictive NetworkPolicy ingress by default.

Fix. Work out the exact connection that broke and allow it in a policy of your own. The documented escape hatch is excluding the feature step with spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep, but that removes the default for those jobs and should be a recorded exception rather than a global setting.

Scala 2.12 jars and javax imports

Symptom. NoSuchMethodError, ClassNotFoundException or servlet-related errors after the move to Spark 4.

Cause. Spark 4 is built only for Scala 2.13 and its servlet API moved from javax to jakarta. Libraries compiled for _2.12 or reaching for javax.servlet fail at run time, often deep in a stack.

Fix. Search internal jars for _2.12 before the upgrade, rebuild or replace them, and put the check in CI. This is calendar time you cannot compress, so start it first.

Small files quietly eat the driver

Symptom. A job that scans a table is slow to start and the driver uses large amounts of memory even though the computation is light.

Cause. The table is made of millions of small files. Planning and scheduling a task per tiny split loads the driver, and the number of tasks inflates scheduler bookkeeping.

Fix. Compact files in a scheduled job, write partitioned Parquet on columns of reasonable cardinality with partitionBy, and avoid writing one file per input per run. A platform that does not own file layout will keep paying for it in driver memory.

Secrets visible in the UI

Symptom. A password shows up in the Environment tab, a log line or the event log.

Cause. The credential was passed as a --conf literal or embedded in a connection string. Command lines appear in process listings, configuration appears in the UI and in event logs, and redaction only masks keys that match spark.redaction.regex.

Fix. Rotate first. Then replace the literal with a mounted Kubernetes secret (spark.kubernetes.driver.secrets.<secret>=/mount/path and the executor equivalent), an environment reference (spark.kubernetes.driver.secretKeyRef.ENV_NAME=secret:key) or workload identity so no secret exists. Extend the redaction pattern for your naming conventions and add a CI check.

Platform-ownership habits

The traps above recur because platforms accrete exceptions. These habits keep that from happening, in rough order of leverage.

Own the image and the templates, not the flags. Teams will always pass different flags. What you can control is the base image, the pod templates and the submission wrapper, because those are where policy is applied. A security context, labels, node selectors and a metrics annotation placed in the template reach every pod without relying on anyone's memory. Keep templates and image recipes in version control and review changes as platform releases.

Remember the precedence order. Configuration set in code overrides spark-submit flags, which override spark-defaults.conf. A default in a file is therefore a suggestion, not a guarantee. If something must always hold, enforce it where the job cannot override it: admission policy, quotas, templates and the wrapper.

Write exceptions down with an expiry. Every legacy flag, every disabled network policy step and every raised quota is a decision someone made for a reason. Record the owner and the date it expires, and review the list quarterly. Flags without expiry dates become permanent, and permanent flags hide the next upgrade's breakage.

Pin everything and rebuild on a schedule. Pin the base image by tag and ideally digest, vendor the jars you depend on rather than resolving packages at start-up, and rebuild weekly so that fixes in the JVM, Python and operating system arrive. Verify downloaded Spark tarballs against the project's KEYS file during the image build, and scan the result. See the Trivy guide for scanning and the Docker guide for the build side.

Measure the platform, not just the jobs. Track time to onboard a team, share of jobs on supported versions, time from submission to first running task, failure rate by cause (user code, platform, infrastructure), share of spend with an owner label, and age of the oldest base image in use. These are the numbers that justify headcount and that show drift before it becomes an incident.

Make cost visible per team. Labels on every pod through the templates, a monthly report of requested versus used memory per team, and the top ten jobs by spend. Visibility changes behaviour more than any policy. Back it with node-pool tagging in your infrastructure code (Terraform guide).

Run game days. Pick a playbook (pods Pending, OOMKilled, an ANSI failure after an upgrade, a leaked secret) and cause it in a test namespace. Time how long it takes an on-call engineer to diagnose with only the runbook. The runbooks that survive a game day are the ones that work at three in the morning.

Keep a canary suite for upgrades. A handful of representative jobs from different teams, run against each candidate image every night, finds breakage before teams do. Pair it with a published version policy: which versions are supported, until when, and when each is retired. Remember that Mesos is gone, Scala 2.12 is gone, and R is deprecated.

Decide who retries. Spark retries tasks, schedulers retry jobs, and teams add retry loops of their own. Written down once, the rule (task retries inside Spark, one job-level retry in the scheduler, none in user code) prevents a failure from multiplying load during an incident. For schedulers, see Airflow and Dagster.

Protect streaming checkpoints. A Structured Streaming query recovers from its checkpoint, so the checkpoint location must be durable, access-controlled and unique to the query. Never share one between queries, and treat a change to stateful logic as a schema change that needs a migration plan.

Know when to say no. If a workload fits on one machine, if it needs millisecond latency, or if it is really workflow logic, the right answer is a different tool. Write the decision record once and point to it. See the Ray guide for one neighbour that often takes over where Spark ends.

Review checklist for pull requests

Use these on pull requests that change a Spark job or its configuration.

• Does the job run on the supported Spark version, with jars built for Scala 2.13?
• Does it depend on spark.sql.ansi.enabled=false, a spark.sql.legacy.* flag or a renamed configuration? If so, is there an owner and an expiry date in the pull request?
• Are all credentials supplied through mounted secrets, environment references or workload identity, with no --conf literal and no password in a connection string?
• Is there a collect(), toPandas() or unlimited show() on data that can grow? Could it be a limit(), an aggregation or a write?
• Are joins deliberate? Is the small side broadcast (the default threshold is 10 MB), are skewed keys known, and are columns and rows pruned before the join?
• Is each cached dataset reused enough to justify memory, and is it released with unpersist()?
• Are Python UDFs replaced by built-in functions, or at least by pandas UDFs, and does the memory profile include overhead for the Python workers?
• Are the dynamic allocation minimum and maximum set, and does the maximum fit the namespace quota?
• Are input files a sensible size, and does the write partition by a column that will not explode into thousands of tiny directories?
• For streaming, is the checkpoint location durable, unique and access-controlled, and is the trigger and state growth understood?
• Is the output written idempotently, so a retry at any level cannot duplicate or corrupt it?
• Does the job log enough context (application id, input, row counts) for the on-call engineer to diagnose it without reading code?
• Is the data stored and processed in the region the data-residency policy requires?

Review checklist for design documents and platform changes

Use these when a design document proposes a new platform component, a new tenant, a new Spark version or a change to a default.

• What is the blast radius if this is wrong: which namespaces, tenants and data stores are affected?
• Which defaults change for existing users, how is the change announced, and what is the rollback?
• Where do drivers run, and where do executors run, and why are these different or the same?
• Is every UI, History Server and API port unreachable from outside its intended network, and has that been verified from outside rather than assumed?
• Is there real authentication in front of every UI, not only ACLs?
• Are RPC authentication, encryption in transit and local disk encryption configured, or is their absence documented as an accepted risk with an owner?
• How are secrets delivered, rotated and kept out of event logs?
• Which images are used, how old are they, who rebuilds them, who scans them, and what blocks a stale image from running?
• How are quotas, service accounts and network policies defined for a new tenant, and how long does onboarding take?
• How is spend attributed, and what is the owner label on every pod?
• What metrics, logs and alerts exist for the new component, and who receives each alert?
• What is the upgrade path, which migration guides were read, and which canary jobs prove it?
• What happens at three in the morning when this fails, and is that written in a runbook that has been exercised?
• What would make us choose a different tool for this workload, and is that threshold written down?

A short practice card

Pick one item a week and do it in a test namespace.

1. Reproduce OOMKilled with a small overhead and read the pod status. Fix it and note the ratio.
2. Run a job with ANSI mode on and off against a column of dirty data, and rewrite one expression with try_cast.
3. Enable the History Server cleaner with a short retention and watch it work.
4. Probe each UI port from a machine outside the cluster and record what answers.
5. Apply a namespace template for a new tenant and time the onboarding.
6. Label every pod through the templates and group a day's usage by team.
7. Diff the plan of a slow run and a fast run with explain('formatted') and name the cause.
8. Search your jars for _2.12 and list what blocks a Spark 4 move.

When all eight are routine, you are operating Spark as a platform rather than using it as a tool. The Kubernetes guide and the Prometheus guide are the natural next stops for the substrate and the monitoring that sit underneath.