The Complete Databricks Guide

This is part one of three. It covers everything you need to do real work on Databricks, starting from a blank browser tab. By the end you can open a workspace, run code in a notebook, create governed tables with the three-level naming that Unity Catalog uses, load a file, travel back in time on a table, schedule a job, track a first machine learning run, and talk to the platform from your terminal with the command line interface. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. Databricks is a platform you learn by clicking and running, and the ideas stick only after you have seen your own table appear in the catalog and your own query come back with rows.

The guide is written against Databricks Runtime 18 LTS, which runs Apache Spark 4.1, and the Databricks command line interface in its 1.x line. The platform itself is updated continuously, so button labels in the web interface move around from month to month. Where a label matters I describe what the control does as well as what it is called, so you can still find it after a redesign.

What Databricks is, and what you can do by the end

Databricks is a managed platform for working with data at scale. You log in through a browser, you get a place to write code, a way to run that code on computers somebody else operates, a catalog where your tables live, and a scheduler that reruns your work every night. On top of that sit the machine learning and generative AI features: experiment tracking, model registry, model serving, vector search and more. You do not install a server, you do not patch a database, and you do not size a cluster before you can run your first line.

The platform is built on a handful of open technologies. Apache Spark does the distributed processing. Delta Lake is the table format that gives files on cheap cloud storage the behaviour of database tables, including transactions and history. MLflow tracks experiments and models. Unity Catalog governs who can see what. Databricks did not invent most of these pieces; it integrates them and runs them for you.

That diagram is the whole platform at the level a beginner needs. Code runs on compute, compute reads and writes tables, and the catalog decides which names exist and who may touch them. Every later section fills in one box.

What people actually use it for falls into a few groups.

This guide stays on the foundations that every one of those groups shares: notebooks, compute, Unity Catalog, Delta tables, files, jobs, a first MLflow run and the command line. A reader who has these can pick up any of the four groups afterwards.

You need very little to start: a web browser, an email address, and a willingness to type small snippets of Python and SQL. Python knowledge at the level of variables, functions and loops is plenty. SQL at the level of SELECT, WHERE and GROUP BY is plenty too. If you have never written either, spend a weekend on the basics first, because Databricks assumes you can already read a query.

The problem it solves and what came before

To see why Databricks looks the way it does, look at what teams did before it. For a long time there were two separate worlds. The first was the data warehouse: a database engine built for analytical queries over clean, structured tables. Warehouses were fast and well governed, but loading them was slow and expensive, they handled raw files, images and text poorly, and machine learning code could not run inside them. The second was the data lake: a pile of files in cheap object storage such as Amazon S3 or Azure storage. Lakes held anything and cost little, but they had no transactions, no schema enforcement, no easy permissions and no history. Two jobs writing to the same folder could leave it half updated, and nobody could tell which version of a file a model had been trained on.

Most organisations ran both. Raw data landed in the lake, a pipeline copied a cleaned subset into the warehouse, and data scientists pulled a third copy onto a laptop or a separate machine learning cluster. Every copy was a place for numbers to disagree and for access rules to be forgotten.

The lakehouse idea is to keep the data in the lake, in open file formats, and add the missing warehouse features on top. Delta Lake adds a transaction log to Parquet files, which gives you atomic writes, schema checks and a history of every change. Unity Catalog adds names, permissions and lineage. SQL engines and Spark both read the same tables. One copy of the data serves reporting, engineering and machine learning.

The practical consequence is that "which version of Databricks" is a slightly odd question. The thing with a version number is the Databricks Runtime, the bundle of Spark, Delta Lake and libraries that your compute runs. At the time of writing the newest long-term-support release is Runtime 18 LTS, based on Spark 4.1, with support to June 2029. Runtime 17.3 LTS is still supported too. Serverless compute is different: it is called versionless, because Databricks upgrades the runtime for everyone at once.

The mental model: account, workspace, compute and catalog

Databricks has a lot of product names, but the beginner's mental model needs only a few nouns. Learn these five and every page of documentation becomes easier to place.

The account is the top-level entity that your organisation signs up for. It holds the people, groups, billing, and the list of workspaces. Account administrators work in a separate web page called the account console. Most beginners never need it, but you will hear the word.

A workspace is the environment you log into. It has its own web address that looks like https://<something>.cloud.databricks.com on Amazon Web Services, and inside it live notebooks, jobs, compute and queries. An account usually has several workspaces, for example one for development and one for production, so that experiments cannot break a nightly pipeline.

The control plane and the compute plane explain where things physically run. The control plane is operated by Databricks. It serves the web interface, stores your notebooks, and runs the job scheduler. The compute plane is where your data is actually processed. With classic compute the machines run inside your organisation's own cloud account. With serverless compute they run inside Databricks' account and you never see them. Your data stays in cloud object storage either way.

Compute is the set of machines that run your code. A notebook has no power of its own. It is a document, and it needs to be attached to compute before any cell can run. Section six covers the options.

Unity Catalog is the governance layer. It decides what tables, files and models exist, what they are called, and who may use them. Every object has a three-level name: catalog, then schema, then the object itself. You will type names like main.default.my_table for the rest of your Databricks career.

If you are working in the Middle East or North Africa, one more word matters early: region. A workspace is created in one cloud region, and the metastore that holds Unity Catalog metadata is tied to a region too. Where your data physically lives can be a legal question for Gulf and Egyptian employers. Cloud providers offer regions in the area, but which Databricks features are available in each region varies, so check the current cloud and region documentation for your provider before you promise a client that data never leaves the country.

Getting a workspace

The platform needs no installation, but you do need somewhere to log in. For learning, Databricks offers a Free Edition that you can sign up for without a company account. The sign-up page is the authority on its current limits, which change, so read what it says about compute allowances and supported features before you plan a project around it. Another route is a trial through a cloud marketplace, and a third is the workspace your employer or course provides.

The steps are the same everywhere in spirit.

1. Go to the Databricks sign-up page and choose the free option for learning.
2. Sign in with an email address or an existing Google or Microsoft account and follow the prompts.
3. Wait for the workspace to be created. When it is ready you land on a home page inside your own workspace.
4. Copy the address from your browser bar and save it somewhere. This is your workspace URL, and the command line interface will ask for it later.

Unity Catalog is turned on automatically for workspaces created after 8 November 2023, so a new workspace starts with a catalog called main and, inside it, a schema called default. You may also see a catalog called samples containing small ready-made datasets that Databricks provides for tutorials. We will use one of them shortly.

Once you are in, spend a few minutes just looking at the left sidebar. It lists the main areas: a workspace area for your files and notebooks, a catalog browser, compute, jobs and workflows, and a place for SQL. The exact names and order change as the product evolves. If something is not where this guide says, use the search box at the top of the page, which finds notebooks, tables and settings by name.

Your first notebook

A notebook is the main place you write code on Databricks. It is a document made of cells. Each cell holds a few lines of code or some text, and you run cells one at a time. The output appears directly below the cell, whether it is a number, a table of rows or a chart. Notebooks suit data work because you rarely know the answer in advance. You look at the data, try a transformation, look again, and keep the conversation with the data on the page.

A notebook has a default language, usually Python, but each cell can switch language with a magic command on its first line. %sql makes the cell SQL, %python makes it Python, %md makes it formatted text. This lets one notebook mix a SQL query with a Python function, which is common in practice.

Create one now. In the workspace area choose to create a new notebook, give it a name such as first-steps, and attach it to compute using the selector at the top. On a new workspace the selector offers serverless compute, which needs no setup. Then type this into the first cell and run it with the run button or Shift+Enter.

print("hello from Databricks")
spark.version

The first line prints text. The second asks the Spark session, a ready-made object named spark that every notebook has, which version of Spark it is connected to. You never create spark yourself in a notebook. Databricks creates it and hands it to you, which is one of the small conveniences that makes the first hour pleasant.

Now read real data. The samples catalog contains a taxi trips table you can query without uploading anything.

df = spark.read.table("samples.nyctaxi.trips")
df.display()

spark.read.table loads a table by its three-level name into a DataFrame, which is Spark's name for a table of rows and named columns held in a form Spark can process in parallel. display() is a Databricks addition that renders a DataFrame as an interactive table with a button for turning it into a chart. Plain Spark has show(), which prints text. You will see both in examples online, and display() only works inside Databricks.

A detail that surprises beginners is that Spark is lazy. Reading a table or filtering a DataFrame does not immediately do the work. Spark records what you asked for as a plan, and only starts computing when something demands a result, such as display(), show(), a write, or a count. This lets Spark look at your whole plan, reorder it, and avoid reading data it does not need. The practical effect is that an error in an early step may only appear when a later step finally runs.

Switch to SQL in a new cell to see the same data from another angle.

%sql
SELECT pickup_zip, COUNT(*) AS trips, ROUND(AVG(fare_amount), 2) AS avg_fare
FROM samples.nyctaxi.trips
GROUP BY pickup_zip
ORDER BY trips DESC
LIMIT 10

The first line, %sql, is the magic command. The query counts trips per pickup postal code and shows the ten busiest. SQL and the DataFrame API compile to the same plan underneath, so choose whichever reads better for the question in front of you. Analysts usually start in SQL, engineers often prefer Python, and a good notebook uses each where it is clearer.

Compute: what actually runs your code

A notebook is only text until it is attached to compute. Beginners often ask why Databricks has so many kinds, so it helps to see them as answers to different questions.

Serverless compute is the default for most new work. You attach a notebook or a job to it and it starts in moments, because Databricks keeps capacity ready and manages everything for you. There is no cluster to size, no runtime to choose, and no machine left running after you stop. Databricks upgrades the underlying runtime for everyone at once, which is why it is called versionless. There are limits: older low-level Spark interfaces such as RDDs and the Spark context are not available, and some Spark settings are not allowed. For a beginner's work these rarely matter.

All-purpose clusters are classic compute that you create and control. You choose the runtime, the machine type and the number of workers, and you can leave the cluster running for interactive work with many people. It is flexible and it is also the easiest way to waste money, because it bills while it runs, including while you are at lunch. Always set auto-termination so the cluster shuts itself down after a period of inactivity.

Job clusters are classic clusters that exist only for the length of a scheduled job. They start when the job starts and disappear when it ends. For repeated work they are cheaper than leaving an all-purpose cluster up.

SQL warehouses are compute specialised for SQL, dashboards and business intelligence tools. They come in sizes and in serverless, pro and classic flavours, and they stop themselves after a quiet period. When an analyst opens a dashboard or connects Power BI, a SQL warehouse answers.

When you do create a classic cluster, you pick a Databricks Runtime version. Choose the newest LTS release unless a library forces you elsewhere, and choose the ML variant, written for example as "18 LTS ML", if you will train models, because it bundles MLflow, scikit-learn, PyTorch and similar libraries so you do not install them yourself. You will also see a checkbox for Photon, Databricks' native vectorised query engine. It speeds up SQL and DataFrame work, and it is billed at a higher rate per hour, so it pays off for heavy queries and not for tiny experiments.

Unity Catalog and the three-level name

Every table, volume, function and model in modern Databricks lives in Unity Catalog. It is a single place that stores the names of your data objects and the rules about who may use them. Understanding its shape early saves a great deal of confusion, because nearly every error message in the first week is about a name or a permission.

The hierarchy has three levels you use every day, under a container that you rarely touch.

The metastore is the top-level container for metadata in a region. Your account has one per region, and workspaces attach to it. A catalog is the first level you name, and teams commonly use one per environment or per business domain. A schema, sometimes called a database, groups related tables. The object at the bottom might be a table, a view, a volume for files, a function, or a registered model.

You always address an object as catalog.schema.object. A table called events in schema bronze in catalog course is course.bronze.events. Writing the full name every time is the safest habit, because it works from any notebook without depending on settings. You may also set a default with USE CATALOG course and USE SCHEMA bronze, after which events alone resolves, but defaults make notebooks fragile when someone else runs them with different settings.

Tables come in two ownership styles. A managed table is fully controlled by Unity Catalog, which chooses where its files live and removes them when the table is dropped. An external table points at storage you own, and Unity Catalog governs access to it but does not own the files. Begin with managed tables. They are simpler, and they let the platform apply maintenance for you.

Permissions are granted to users, groups and service principals, a service principal being a non-human identity for automation. Access follows the hierarchy, and one rule trips up almost everyone. To read a table you need SELECT on the table, plus USE SCHEMA on its schema, plus USE CATALOG on its catalog. Having the table permission alone is not enough, because you must be allowed to walk down the tree to reach it. As an owner of your own catalog you hold these automatically, so you will first meet the rule when you share your work with someone else.

GRANT USE CATALOG ON CATALOG course TO `data-team`;
GRANT USE SCHEMA, SELECT ON SCHEMA course.bronze TO `data-team`;
SHOW GRANTS ON SCHEMA course.bronze;

That example gives a group called data-team the ability to find and read everything in the bronze schema. It is shown here so that you recognise the shape. Granting to groups and not to individuals is the standard practice, because people join and leave and the group stays.

Your first project: from raw trips to a table you own

Reading a sample table is useful, but the real skill is creating your own governed table. This section builds a small project from start to finish. Everything runs in a notebook on serverless compute, and everything is a few lines.

The goal is a table of the busiest taxi pickup zones that you own, stored in your own catalog, with history you can inspect. First create a catalog and a schema. If your workspace does not allow you to create a catalog, use the main catalog and your own schema instead, and replace course with main in every example that follows.

CREATE CATALOG IF NOT EXISTS course;
CREATE SCHEMA IF NOT EXISTS course.bronze;

IF NOT EXISTS makes the statement safe to rerun, which matters because notebooks get rerun constantly. A command that fails on its second execution is a command you will dread.

Next load a cleaned slice of the sample trips into a new managed table. The CREATE TABLE ... AS SELECT pattern creates a table from a query's result in one step.

CREATE OR REPLACE TABLE course.bronze.trips_clean AS
SELECT pickup_zip, dropoff_zip, trip_distance, fare_amount
FROM samples.nyctaxi.trips
WHERE fare_amount > 0 AND trip_distance > 0;

Run it, then check the result with a query.

SELECT COUNT(*) AS rows_loaded FROM course.bronze.trips_clean;

The same thing in Python uses the DataFrame writer. This is the form you will use when the transformation is easier to express in Python than in SQL.

from pyspark.sql import functions as F

trips = spark.read.table("course.bronze.trips_clean")

busiest = (
    trips.groupBy("pickup_zip")
         .agg(F.count("*").alias("trips"),
              F.round(F.avg("fare_amount"), 2).alias("avg_fare"))
         .orderBy(F.desc("trips"))
)

busiest.write.mode("overwrite").saveAsTable("course.bronze.busiest_zones")
spark.read.table("course.bronze.busiest_zones").display()

Read it slowly. groupBy and agg compute a count and an average per pickup zone, orderBy sorts the result, write.mode("overwrite") replaces the table if it already exists, and saveAsTable registers the result in Unity Catalog under the name you gave it. The final line reads it back, which proves that it exists in the catalog and not just in memory.

Notice what you did not do. You did not choose a file format, because Delta is the default for tables in Databricks. You did not decide where the files go, because the table is managed. You did not set up permissions, because you own what you created. The lakehouse made several decisions on your behalf, and the defaults are sensible.

Delta tables: history, time travel and safe mistakes

The tables you just created are Delta tables. A Delta table is a folder of Parquet data files in cloud storage plus a transaction log that records every change as an ordered list of commits. The log is what turns a pile of files into something with database behaviour. A write either appears completely or not at all, a reader never sees half of a change, and the table remembers its past.

That memory is a gift for beginners, because it makes mistakes recoverable. Overwrite your table by accident and the old version is still there. Look at the history first.

DESCRIBE HISTORY course.bronze.busiest_zones;

The output has one row per commit, with a version number starting at zero, a timestamp, the user, and the operation, such as CREATE OR REPLACE TABLE AS SELECT or WRITE. Each time you ran your Python cell with overwrite, a new version was added. Nothing was lost.

To read an older version, ask for it by number.

SELECT * FROM course.bronze.busiest_zones VERSION AS OF 0;

This is time travel. You can also ask for a table as it was at a timestamp. And if a bad write needs undoing, you can restore the table to an earlier state rather than rebuild it.

RESTORE TABLE course.bronze.busiest_zones TO VERSION AS OF 0;

Restoring is itself a new commit in the log, so it does not erase history. That is worth noticing, because it means that even the undo is undoable.

Time travel is not unlimited. Old data files are cleaned up by a command called VACUUM, which removes files no longer referenced by recent versions, with a default retention of seven days. After vacuum has run, versions older than the retention window can no longer be read. Do not treat Delta history as a backup system. It is a short safety net for human error, not a replacement for proper retention of data you must keep.

There are two other maintenance commands you will see often. OPTIMIZE compacts many small files into fewer larger ones so queries read less overhead. Small files accumulate quickly when a stream or frequent small writes feed a table. On managed Unity Catalog tables, Databricks can run this housekeeping for you automatically through a feature called predictive optimization, so as a beginner you mostly need to recognise the names and not schedule them yourself.

Files, volumes and secrets

Tables are not the only thing you work with. Sooner or later someone hands you a CSV, a JSON export or a folder of images. Unity Catalog stores files in volumes, which are governed folders that sit inside a schema, so files get the same names and permissions as tables. A volume path looks like /Volumes/<catalog>/<schema>/<volume>/<file>.

Create one and upload a file.

CREATE VOLUME IF NOT EXISTS course.bronze.raw;

Then use the catalog browser to open the volume and upload a small CSV through the interface. Once it is there, read it from a notebook.

df = (spark.read
        .option("header", True)
        .option("inferSchema", True)
        .csv("/Volumes/course/bronze/raw/my_file.csv"))
df.display()

header says the first line holds column names and inferSchema asks Spark to guess column types by scanning the data. Inference is convenient for exploration and risky for production, because a column that happens to look numeric in today's file might hold text tomorrow. Later you will declare schemas explicitly.

To look around, list the contents of a volume with dbutils, the utilities object that every notebook has.

dbutils.fs.ls("/Volumes/course/bronze/raw")

You will find old tutorials that use dbfs:/ paths, /FileStore, or mounts created with dbutils.fs.mount. These belong to an older way of working on the Databricks File System root. For new work, prefer Unity Catalog volumes and tables, because they carry permissions and lineage and the older locations do not.

The dbutils object does more than list files. Its secrets utility is how code reads a password or an API key without writing it into the notebook. A secret lives in a secret scope, and you fetch it by scope and key.

api_key = dbutils.secrets.get(scope="my-scope", key="api-key")

If you print a secret in a notebook, Databricks shows [REDACTED] instead of the value, a safeguard against leaking it into shared output. That safeguard is a courtesy and not a security boundary, so never rely on it to protect a value you have already stored in plain text somewhere else. Secrets are created with the command line interface or the API by someone with the right permission, and you should never paste a real token into a cell, even in a throwaway notebook, because notebooks get shared and exported.

The command line interface

Everything so far happened in a browser. The Databricks command line interface, usually just "the CLI", lets you do some of the same things from a terminal, which is where automation begins. You will use it to check that you are logged into the right workspace, to list resources, to trigger jobs, and later to deploy whole projects.

Install it for your system. On macOS with Homebrew:

brew tap databricks/tap
brew install databricks

The current documentation also shows a brew trust databricks/tap step on recent Homebrew versions, so run it between the two lines if your Homebrew asks for it. On Linux the project provides an install script:

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

On Windows use WinGet or Chocolatey:

winget install Databricks.DatabricksCLI
choco install databricks-cli

Alternatively download a release archive from the project's GitHub releases page and put the binary on your path. Then check the installation.

databricks -v

You want a version in the 1.x line. The CLI reached 1.0 and is now updated often, so upgrade with the same tool you installed it with, for example brew upgrade databricks. If a team shares a project, everyone should use compatible CLI versions. A recent change to the way deployment state is stored means that very old clients cannot work with projects deployed by new ones.

Next, log in. The recommended way for a person is OAuth in the browser, called user-to-machine authentication.

databricks auth login --host https://<your-workspace-url>

The command opens your browser, you approve, and the CLI stores a token so you are not asked again. Since version 1.0 the token goes into your operating system's secure store, such as the macOS Keychain or Windows Credential Manager. On a headless Linux machine with no such store you can set the environment variable DATABRICKS_AUTH_STORAGE=plaintext to fall back to a plain file, which is acceptable for a throwaway machine and a poor habit anywhere else.

The CLI saves what it learns in a profile inside a file named .databrickscfg in your home directory. A profile is a named set of connection details, so you can keep one for development and one for production and switch between them with -p.

databricks auth profiles
databricks current-user me
databricks clusters list
databricks workspace list /Users/<you>
databricks fs ls /Volumes/course/bronze/raw/
databricks jobs list
databricks clusters list -p <profile>

current-user me is the best first command because it answers "who am I, and on which workspace?" in one line. If it returns your email, authentication works. The other commands list clusters, workspace folders, volume contents and jobs. Output is plain text by default and can be switched to JSON when you want to process it with other tools.

Jobs: running your work on a schedule

A notebook you run by hand is an experiment. A notebook that runs every morning without you is a pipeline. Databricks calls the scheduling feature Jobs, part of the product family now branded Lakeflow. A job is a set of tasks with a trigger. A task can run a notebook, a Python file, a SQL query, a dbt project or a declarative pipeline. Tasks can depend on each other, so "ingest" runs before "clean", which runs before "publish".

Create one from the web interface. Open the jobs area in the sidebar and choose to create a job. Give the first task a name, choose the type notebook, point it at your notebook, and select serverless compute if it is offered. Run it once with the run-now control and watch the run page. It shows the status of each task, the output of the notebook, and the time taken.

To run it on a schedule, add a trigger. The usual one is a schedule written as a time and frequency, such as every day at six in the morning in a named time zone. Other triggers start a job when new files arrive in a location, or when another job finishes. You can also attach notifications so that a failure sends an email, which is the single most important setting on any job nobody watches.

From the terminal you can inspect and trigger the same job.

databricks jobs list
databricks jobs run-now <job-id>
databricks jobs get-run <run-id>

run-now starts a run and prints its identifier, and get-run reports on it. These commands become the building blocks for scripts and continuous integration pipelines. If you later keep jobs as code in a project deployed with the CLI, which the Declarative Automation Bundles feature supports, you will define the job in a file and deploy it with databricks bundle deploy. Older articles call that feature Databricks Asset Bundles. It is the same tool under a new name, and the commands did not change.

One principle matters from the first job: make the notebook safe to run twice. Schedulers retry failed runs, and humans click run again. A notebook that appends the same rows every time it runs will double your data. Prefer writes that replace a defined slice, or use merge statements that update matching rows, so that rerunning gives the same result as running once. This property is called idempotence, and it is the difference between a pipeline that recovers from failures and one that needs a person to clean up after it.

MLflow: your first tracked run

Machine learning adds one new problem to data work: you will train the same model many times with different settings, and weeks later you will need to know which run produced which result. MLflow solves this by recording every attempt. Databricks hosts a managed MLflow, so there is no server to set up. The ML runtimes include it, and on serverless compute you can install the library with %pip install mlflow at the top of a notebook.

MLflow has a few nouns. An experiment is a named collection of runs, usually one per problem. A run is one execution of your training code. It records parameters, the settings you chose, metrics, the numbers you measured, and artifacts, the files produced, including the model itself. Here is the smallest useful example, using a tiny built-in dataset.

import mlflow
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

with mlflow.start_run():
    alpha = 0.5
    model = Ridge(alpha=alpha).fit(X_train, y_train)
    rmse = mean_squared_error(y_test, model.predict(X_test)) ** 0.5
    mlflow.log_param("alpha", alpha)
    mlflow.log_metric("rmse", rmse)

Run the cell, then open the experiments area from the sidebar, or follow the link that appears in the cell output. Your run is listed with its parameter and metric. Change alpha, run again, and the two runs sit side by side so you can compare them in a chart. That comparison, repeated across dozens of attempts, is what makes experiment tracking valuable.

Models you want to keep go into the model registry, and on current Databricks that registry lives in Unity Catalog, so a model gets a three-level name like any table, for example course.models.churn. Versions of a model are labelled with aliases such as champion, which your serving or batch code loads by name. The older Workspace Model Registry with its fixed stages named Staging and Production is deprecated, so if a tutorial calls transition_model_version_stage, it describes the legacy way. The Mid-level guide covers registering and promoting models, and the MLflow guide goes deeper into the library itself.

Common errors and how to read them

Databricks errors look intimidating because they combine Spark, Delta, Unity Catalog and cloud messages in one stack trace. The trick is to read from the top for the error class in square brackets, such as [TABLE_OR_VIEW_NOT_FOUND], and then to scan for the first line that mentions something you wrote. The long Java frames below it are almost never the cause. The exact wording varies by runtime version, so use the patterns below as a guide to what to look for and check the error reference in the documentation for the precise text.

Table or view not found. The message says that a table or view cannot be found. The usual causes are a missing catalog or schema prefix, a typo, or a missing permission, because Unity Catalog may report "not found" for objects you are not allowed to see. Check the three-level name against the catalog browser and then check your grants.

Insufficient privileges. The message names a missing privilege such as USE SCHEMA on a schema. Remember that SELECT on a table is not sufficient on its own. The reader also needs USE CATALOG and USE SCHEMA on the parents. Ask the owner for the grant to the group you belong to.

Cannot configure default credentials. The CLI or the Python SDK prints that it cannot configure default credentials and points you at the authentication documentation. It means no profile, host or token could be found. Run databricks auth login --host <url> or set DATABRICKS_HOST and a credential, then retry.

Unauthorized, 401 or invalid access token. Your token expired or was revoked. Log in again with databricks auth login.

Cluster terminated, cloud provider launch failure. A classic cluster could not start, typically because your cloud account lacks quota or capacity for the machine type. Choose a different instance type or ask your cloud administrator to raise the limit. Serverless avoids this class of problem for you.

Schema mismatch when writing to a Delta table. You tried to write columns that differ from the table. Decide whether the new shape is intended. If it is, enable mergeSchema for that write, and if it is not, fix the DataFrame.

Putting it all together

Here is one small project that uses everything above. You will ingest a file, build a table, schedule it, and track a model. Budget an hour and use a fresh notebook.

1. Prepare the home. Create a catalog course with schemas bronze and models, and a volume course.bronze.raw.
2. Ingest. Upload a small CSV of your choice into the volume, read it with spark.read, and save it as course.bronze.my_data with saveAsTable.
3. Inspect. Run DESCRIBE HISTORY on the table, then overwrite it once and read the earlier version with VERSION AS OF 0.
4. Summarise. Build a second table, course.bronze.my_summary, with a groupBy aggregation that answers one question about your data.
5. Schedule. Put steps two to four in a notebook, create a one-task job on serverless compute, add an email notification on failure, and run it once.
6. Track. In another notebook, train a small scikit-learn model on your summary or on the diabetes dataset, and log a parameter and a metric with MLflow.
7. Drive it from the terminal. Log in with the CLI, run databricks current-user me, list your jobs, and trigger the job with databricks jobs run-now.

The finished result is modest, and it contains every moving part of a real project: governed storage, a repeatable transformation, a schedule, a tracked experiment and a scripted entry point. A production system is the same shape with more tables, more tasks, stronger permissions and a deployment process.

What you can now do, and what comes next

You can now explain what the lakehouse is and why it replaced the lake-plus-warehouse pair. You can describe an account, a workspace, the two planes and the catalog, and say which of them a given problem belongs to. You can run code in a notebook, choose between serverless compute, a cluster and a SQL warehouse, and avoid the most expensive beginner mistake of leaving compute running. You can create catalogs, schemas, tables and volumes with correct three-level names, read the history of a Delta table, travel back in time and restore it. You can read the common errors and fix the usual causes. You can schedule a notebook as a job, track a first MLflow run, and use the command line interface to authenticate and drive the platform.

The Mid-level guide picks up where this one stops. It covers Unity Catalog permissions in depth, incremental ingestion with Auto Loader, merge statements and table maintenance, registering models in Unity Catalog with aliases, the Python SDK, Databricks Connect for developing from your own editor, and how to run jobs with service principals instead of your personal identity. The Senior guide then covers deployment with Declarative Automation Bundles, environment separation, cost governance, security, and the points where Databricks is not the right tool.

Neighbouring guides help here: Delta Lake and Spark explain the engines underneath, and Terraform and GitHub Actions pair with bundles.

The best next step is to keep the project from the last section alive and extend it a little each week: add a second source, add a permission, add a test. Platforms are learned by owning something small that has to keep working.

Sources

• Databricks documentation home
• Platform release notes
• Databricks Runtime release notes
• Supported runtimes and end-of-support dates
• Databricks CLI installation
• Databricks CLI authentication
• Databricks CLI releases on GitHub
• Unity Catalog
• Serverless compute
• MLflow on Databricks
• Declarative Automation Bundles

This is the interview companion to the Beginner Databricks guide. Interviewers for junior data and machine learning roles are rarely trying to catch you out on exotic features. They want to hear that you understand what the platform is for, that you can name its parts accurately, that you know the handful of commands to reach for, and that you have already made, and understood, the classic mistakes. Read the main guide first. This page is about turning that knowledge into short, confident answers.

The thirty-second answer

If you are asked "what is Databricks?", this is the answer to have ready.

Databricks is a managed platform that lets teams run data engineering, analytics and machine learning on one copy of their data, stored as open Delta tables in cloud storage and governed by Unity Catalog. You write code in notebooks, SQL or jobs, it runs on compute the platform operates, and the same tables serve reports, pipelines and models.

Then add the one sentence that shows you understand the history. Before the lakehouse, teams kept raw files in a data lake and copied cleaned data into a warehouse, which created several drifting copies and separate permission systems. The lakehouse puts warehouse features, transactions, schemas and governance, on top of the lake, so there is one copy to trust.

Practise saying this aloud until it takes under thirty seconds and does not sound recited. Interviewers notice the difference between a memorised definition and an explanation, and they usually follow up on whichever noun you used, so only say words you can unpack.

The model you should be able to draw

Interviewers often hand you a marker and ask you to draw how the pieces fit. The picture to have in your head has these elements.

An account contains workspaces, users, groups and billing. A workspace is the environment with its own address. Databricks runs a control plane, which serves the interface and schedules jobs, and your code runs in the compute plane. Classic compute lives in the customer's cloud account and serverless compute lives in Databricks' account. Data stays in the customer's cloud object storage in both cases. Unity Catalog sits across all of it with a hierarchy of metastore, catalog, schema and object, and objects are addressed as catalog.schema.object.

Be ready to say which noun owns which responsibility, because that is what the question really tests.

A strong candidate also separates Delta Lake, the table format, from Spark, the processing engine, from Databricks, the platform that runs and manages both. A weak candidate says "Databricks is Spark". The correction is easy: Spark is an open engine you can run anywhere, Delta Lake is an open table format, and Databricks packages them with governance, scheduling, notebooks and managed machine learning.

On versions, say something precise and modest. The versioned thing is the Databricks Runtime, and the newest long-term-support release is 18 LTS, built on Spark 4.1. Serverless compute is versionless. Saying "I would check the supported runtimes page" is a fine answer to a question about dates you do not remember.

The commands and snippets to have ready

You do not need to memorise a manual, but a few snippets should come from your fingers without searching. Interviewers sometimes share a screen and ask you to "show me how you would".

From a notebook, reading and exploring:

df = spark.read.table("samples.nyctaxi.trips")
df.display()
spark.sql("SELECT * FROM course.bronze.events LIMIT 10").show()
df.write.mode("overwrite").saveAsTable("course.bronze.out")
dbutils.fs.ls("/Volumes/course/bronze/raw")

From SQL, creating objects and reading history:

CREATE CATALOG IF NOT EXISTS course;
CREATE SCHEMA IF NOT EXISTS course.bronze;
DESCRIBE HISTORY course.bronze.events;
SELECT * FROM course.bronze.events VERSION AS OF 3;
GRANT USE CATALOG ON CATALOG course TO `data-team`;

From the terminal:

databricks auth login --host https://<workspace-url>
databricks current-user me
databricks clusters list
databricks jobs list
databricks jobs run-now <job-id>

For each line, be ready to say what it does and one thing that can go wrong. For saveAsTable with overwrite, the thing that goes wrong is replacing a table you meant to append to. For DESCRIBE HISTORY, the answer is that history is not a backup, because vacuum removes old files after the retention period. For auth login, it is that you must check which workspace you landed in.

Two details separate people who have used the platform from people who have read about it. The first is that display() is Databricks-only, while show() is plain Spark. The second is that spark and dbutils are provided in a notebook, so you never create them there. Mention these naturally in a live exercise and the interviewer will quietly note that you have used the tool.

The classic traps and why they happen

Most junior mistakes on Databricks fall into five groups. Knowing them by name, with the cause, is worth more than any definition.

The idle cluster. Classic all-purpose clusters bill while running. A cluster without auto-termination left on over a weekend produces a bill nobody planned. The cause is that interactive clusters are meant to be shared and persistent, so they do not stop unless told to. The fix is auto-termination, job clusters for scheduled work, and serverless where it is supported.

The permission that is not enough. A user has SELECT on a table and still cannot read it. The cause is that Unity Catalog requires USE CATALOG and USE SCHEMA on the parents too. A related trap is that Unity Catalog may answer "table not found" when the real problem is a missing privilege, since it avoids confirming that hidden objects exist.

The stale cell. A notebook shows a result that the code above it no longer produces, because cells ran out of order. The fix is to restart and run all before trusting or sharing a result.

The unsafe rerun. A job that appends on every run duplicates data when retried. The cause is non-idempotent writes. The fix is to overwrite a defined slice or use a merge so that a second run changes nothing.

The legacy path. Writing to dbfs:/FileStore, mounting storage, or using the old workspace model registry with stages. These work in old tutorials and are deprecated in current guidance. Use Unity Catalog volumes and tables, and Unity Catalog models with aliases.

Several of these have a deeper reason you can mention. Spark is lazy, so an error in an early transformation can surface only when an action such as display() finally forces the computation, which confuses people reading a stack trace from the bottom. collect() and toPandas() bring all rows to one machine, so on a large table they cause out-of-memory failures on the driver. Inferring a schema from a file is convenient but unsafe in production, because the guess depends on whichever rows Spark happened to sample.

Common interview questions

Each answer below is a model. Use your own words, and tailor the example to a project you have actually done.

A lakehouse stores data in open formats in cheap cloud storage, as a lake does, and adds the reliability and governance of a warehouse on top. It exists because organisations were maintaining two systems and copying data between them. Delta Lake adds transactions, schema enforcement and history to files, and Unity Catalog adds names and permissions. One copy of the data then serves SQL reporting, engineering pipelines and machine learning. A strong answer adds that the benefit is fewer copies and a single access-control layer, and not just speed.

It is an open table format. A Delta table is a folder of Parquet files plus a transaction log that records every commit. The log gives you atomic writes, so readers never see half a change, schema enforcement, and history that supports time travel. Mention that DESCRIBE HISTORY shows commits and VERSION AS OF reads an older one, and that vacuum eventually deletes files that old versions need.

Unity Catalog is the governance layer for data and AI assets. Objects live in a hierarchy of metastore, catalog and schema, and you address a table as catalog.schema.table. It stores the names, enforces permissions, and records lineage and audit information. Add that grants follow the hierarchy, so reading a table needs SELECT plus USE SCHEMA and USE CATALOG on its parents.

With a managed table, Unity Catalog controls the storage location and the lifecycle, so dropping the table removes its data. With an external table, the files live in a location you own and Unity Catalog governs access without owning them, so dropping the table leaves the files. Managed is the default recommendation for new work because it is simpler and lets the platform maintain the table.

Serverless is the default when it is supported, because it starts quickly, needs no sizing and upgrades itself. A classic cluster is for cases that need control: a specific runtime version, custom libraries or settings, or low-level Spark APIs like RDDs that serverless does not offer. For scheduled work on classic compute, use job clusters rather than an all-purpose cluster, because they exist only while the job runs.

An all-purpose cluster is for interactive work and is shared and long-lived. A job cluster is created for a single job run and destroyed after, which makes it cheaper for scheduled work. A SQL warehouse is compute specialised for SQL queries, dashboards and BI tools, with sizes and automatic stop. The first two run notebooks and code of any kind, while a warehouse answers SQL.

A Databricks Unit is the billing measure of processing per hour, priced differently per type of compute. On classic compute you also pay the cloud provider for the virtual machines underneath, and on serverless both are included in the rate. I would not quote prices in an interview. I would say how to reduce the bill: auto-termination, job clusters, right-sized compute, and tags so that spending can be attributed.

For a person, run databricks auth login --host <workspace-url>, which opens a browser for OAuth and stores a token, in the operating system's secure store from version 1.0 onward. The connection details are kept in a profile in .databrickscfg. For automation, use a service principal with OAuth client credentials, and not a personal access token. Personal access tokens still work but the documentation recommends OAuth because tokens are long-lived secrets.

Store it in a secret scope and read it with dbutils.secrets.get(scope=..., key=...). The value is redacted if printed. Never paste it in a cell, a widget default or a committed file. For deployed code, prefer identities that need no password at all, such as a service principal with the right grants.

MLflow records machine learning experiments. A run holds parameters, which are the settings you chose, metrics, which are the numbers you measured, and artifacts such as the model. Databricks hosts it as a managed service. I would log the parameters that change between attempts, the evaluation metrics, and the fitted model, so that I can compare runs and reproduce the best one. For model versions I would register them in Unity Catalog and refer to them by alias.

show() is plain Spark and prints a text table. display() is Databricks-specific and renders an interactive table with built-in charting. Code that must also run outside Databricks should avoid display().

I would check the full three-level name for typos or a changed default catalog or schema. Then I would check whether the table was dropped or renamed, by looking in the catalog browser and the history of the schema. Finally I would check permissions, because Unity Catalog can report not-found for objects the user may not see. A strong answer also mentions asking whether someone changed the session defaults with USE CATALOG.

Run DESCRIBE HISTORY to find the version before the bad write, check that it is still readable with VERSION AS OF, and restore with RESTORE TABLE ... TO VERSION AS OF. Then explain the limit: this works only while the old data files exist, so it is a safety net rather than a backup. Mention that restore is itself a commit and so can be reversed.

Make its writes idempotent. Overwrite a clearly defined slice, such as one date's partition, or use MERGE so matching rows are updated and new rows inserted, which means a second run leaves the same result as one. Add a failure notification, pass the date in as a parameter, and avoid relying on state kept in the notebook session.

Spark records transformations as a plan and only executes when an action asks for a result. That allows it to optimise the whole plan, for example by reading only needed columns and pushing filters down, and by avoiding work for results never requested. The consequence for a beginner is that errors can appear at the action and not at the line that caused them.

What a strong answer covers that a weak one misses

Interviewers listen for a few habits more than for facts. A strong answer states the why before the how: the lakehouse exists to reduce copies, serverless exists to remove sizing, volumes exist so files have permissions. A strong answer names a trade-off, for instance that serverless gives convenience and loses low-level control. A strong answer is honest about limits of memory, saying "I would check the docs for the exact setting" when appropriate, and does not invent a flag. And a strong answer includes a small example from your own work, such as the table you built, the job you scheduled or the experiment you tracked.

Weak answers have recognisable shapes. They define a term with another term from the same family, such as "Unity Catalog is the catalog for the lakehouse". They describe only the interface, saying "you click here", with no idea of what happens underneath. They quote outdated material, such as workspace-registry stages, mounts or personal access tokens in a continuous integration pipeline, as if it were current. And they claim certainty about prices, limits or dates that change.

If the interviewer asks about something you have not used, such as Lakeflow pipelines, Auto Loader or Model Serving, say so plainly, then reason from what you know. "I have not built one, but I understand it as incremental file ingestion, and I would start from the documentation and a small test" is a perfectly good answer for a beginner. Honest reasoning beats a confident guess.

Sixty-second self-test

Answer each aloud without looking. If you hesitate on more than two, reread the matching section of the main guide.

1. What is Databricks, in two sentences?
2. What is the difference between a control plane and a compute plane?
3. Spell out a three-level table name and say which part is the catalog.
4. Which three privileges are needed to read one table?
5. What is the difference between a managed and an external table?
6. When would you choose serverless over a classic cluster, and when not?
7. What does DESCRIBE HISTORY show, and how long does time travel last by default after vacuum?
8. What does the command databricks current-user me tell you?
9. How would you stop a scheduled job from duplicating rows when it retries?
10. Name the three things an MLflow run records.

When you can answer all ten without hesitation, you are ready for a beginner-level Databricks conversation. The Mid-level guide prepares you for scenario questions, such as what to check when a pipeline is slow or a job fails after an upgrade, and for trade-off questions about compute, ingestion and permissions.

This page collects the mistakes that beginners on Databricks actually make, in the order they usually happen, followed by habits worth adopting on your first day and a checklist to keep. Each mistake is written the same way: the symptom you see, the cause underneath, and the fix. Skim the headings now, and come back when something goes wrong, because recognising a symptom is most of the cure.

The mistakes beginners actually make

1. Leaving compute running

Symptom. Nothing is visibly wrong, but a colleague, a finance team or a cloud invoice tells you that a cluster ran for days.

Cause. A classic all-purpose cluster bills for as long as it is running, whether or not anyone is using it. People start one for a morning exercise and close the laptop without stopping it. If auto-termination was left off or set to a long time, the cluster keeps going.

Fix. Set auto-termination to a short period whenever you create a cluster. Prefer serverless compute for notebooks when it is available, because it does not leave machines behind. Use job clusters for scheduled work, since they disappear when the run ends. At the end of a session, open the compute page and look at what is still running before you close the tab.

2. Table not found, when the table exists

Symptom. A query fails with an error class such as TABLE_OR_VIEW_NOT_FOUND, yet you are sure the table exists because you saw it yesterday.

Cause. There are three common reasons. The name is missing its catalog or schema prefix and resolves against a default that is not what you assumed. The name has a typo, often in the schema. Or you lack permission, because Unity Catalog may say "not found" for objects you are not allowed to see.

Fix. Write the full three-level name every time: catalog.schema.table. Open the catalog browser and copy the name from there, since a copied name has no typos. If the table is visible to a colleague and not to you, ask them to check your grants.

3. Insufficient privileges on something you can plainly see

Symptom. You have been granted SELECT on a table, and a query still fails with an error about insufficient privileges, naming USE SCHEMA or USE CATALOG.

Cause. Unity Catalog permissions follow the hierarchy. Reading a table needs SELECT on the table, USE SCHEMA on its schema and USE CATALOG on its catalog. A grant on the table alone is not enough, because you must be allowed to reach it through its parents.

Fix. Ask the owner to grant the missing parent privilege, ideally to a group you belong to and not to you personally. When you share your own work, grant all three levels at once.

GRANT USE CATALOG ON CATALOG course TO `data-team`;
GRANT USE SCHEMA, SELECT ON SCHEMA course.bronze TO `data-team`;
SHOW GRANTS ON SCHEMA course.bronze;

4. Trusting a notebook that ran out of order

Symptom. A notebook gives one answer on your screen and a different answer when a colleague, or a scheduled job, runs it from the top. Or a variable is "undefined" in a job that worked in the editor.

Cause. Cells run in the order you click, not the order they appear. A cell you edited and never reran, or a variable defined in a cell that you later deleted, still lives in the session's memory and hides the problem.

Fix. Before you rely on a result, detach and reattach the compute or restart the session, then run all cells from the top. Do it before every share and before turning a notebook into a job. A notebook that passes a clean run is one you can trust.

5. Appending when you meant to replace, and the reverse

Symptom. A table has twice the rows you expected after a rerun, or a table you meant to extend now holds only the latest batch.

Cause. The DataFrame writer's mode decides what happens when the target exists. append adds rows, so a rerun duplicates them. overwrite replaces the table's contents, so a partial load destroys earlier data. Both are correct for some jobs and disastrous for others.

Fix. Decide the intended behaviour before writing, and make the job safe to run twice. Overwrite a defined slice, or use a MERGE that updates matching rows and inserts new ones. If you do get it wrong, Delta history is your friend for a few days.

DESCRIBE HISTORY course.bronze.events;
RESTORE TABLE course.bronze.events TO VERSION AS OF 3;

6. Treating Delta history as a backup

Symptom. Months later you try to time travel to a version and get an error that the files are gone.

Cause. Time travel depends on old data files still existing. VACUUM deletes files that recent versions no longer reference. History is a safety net for recent mistakes and nothing more.

Fix. Keep real backups or retention copies of data you must be able to restore, using the means your organisation provides. Leave the default retention alone unless you have a documented reason to shorten it.

7. Collecting everything to the driver

Symptom. A cell that worked on a small sample runs for a long time and then fails with an out-of-memory message, or the message that the driver is up but not responsive.

Cause. collect() and toPandas() bring every row of a distributed DataFrame onto one machine, the driver. On a big table that machine runs out of memory. The code was written as if the data fitted in a laptop.

Fix. Filter, aggregate and join in Spark, and bring back only small results. If you need pandas, take a sample or an aggregate first, and use limit while developing.

small = (spark.read.table("course.bronze.trips_clean")
              .groupBy("pickup_zip").count()
              .limit(100)
              .toPandas())

8. Pasting a secret into a cell

Symptom. A token or password sits in a notebook, a Git history or a screenshot.

Cause. It was quicker to paste it than to set up a secret scope, and a notebook feels private. Notebooks get shared, exported and committed, and Git history never forgets.

Fix. Store the value in a secret scope and read it with dbutils.secrets.get. If one has already leaked, treat it as compromised: revoke it, create a new one and tell whoever owns the system. Deleting it from the notebook is not enough. Prefer identities that need no stored secret.

api_key = dbutils.secrets.get(scope="my-scope", key="api-key")

9. Following an old tutorial into deprecated territory

Symptom. A tutorial tells you to write to /FileStore, mount a storage account, use model stages named Staging and Production, or paste a personal access token into a configuration file. It sort of works, and then it does not work at work.

Cause. The platform changed and the web did not. Databricks moved governance into Unity Catalog, so volumes and tables replace mounts and the root file store, Unity Catalog models with aliases replace stages, and OAuth is recommended in place of personal access tokens. The program called Databricks Asset Bundles is now named Declarative Automation Bundles, which causes confusion when you search for it.

Fix. Check the date on any guide, and when it disagrees with the official documentation, trust the documentation. Prefer /Volumes/<catalog>/<schema>/<volume>/ paths, three-level table names, and databricks auth login.

10. Talking to the wrong workspace

Symptom. A CLI command lists jobs you do not recognise, or a deploy lands somewhere unexpected.

Cause. The CLI resolves its connection from several places, and environment variables take priority over a profile in the configuration file. A leftover DATABRICKS_HOST in your shell, or an old default profile, points the tool at a different workspace from the one you are looking at in the browser.

Fix. Make databricks current-user me your first command in any terminal session, and name profiles clearly so you can pass -p <profile>. Run databricks auth profiles to see what exists. When a result is surprising, suspect the target before the code.

11. Expecting serverless to behave like a classic cluster

Symptom. Code that ran on a cluster fails on serverless with a message that a feature is not supported, or a Spark setting cannot be changed.

Cause. Serverless trades low-level control for convenience. The older RDD interface, direct use of the Spark context, custom JAR files and some Spark configuration settings are not available.

Fix. Rewrite using DataFrames and SQL, which cover nearly all beginner work. If you truly need the low-level feature, use a classic cluster and say so in the notebook's first cell, so that the next reader knows why.

12. Letting inferSchema decide in a pipeline

Symptom. A pipeline works for weeks, then fails or silently produces nulls after a source file changes.

Cause. inferSchema samples the data and guesses types. A column that was all numbers in the sample may contain text in the next file, so the guess changes. Delta then refuses the write, or casts it to something you did not intend.

Fix. Use schema inference for exploration only. In anything scheduled, declare the schema explicitly, and let a mismatch fail loudly so that you notice it. If a change is intended, allow it deliberately for that write with the schema merge option.

Habits worth adopting on day one

The habits below cost almost nothing at the start and save a great deal later. They are the difference between someone who is productive in a week and someone who is productive in a month.

Use full three-level names. course.bronze.events works from any notebook, job and terminal. Names that depend on session defaults break the first time someone else runs your code.

Put your code in Git from the beginning. Use a Git folder for notebooks and commit often. Notebooks in a workspace have no review and no easy recovery, and Git gives you both. A notebook in Git is also one step from a deployable job.

Name things for their stage. Use schemas such as bronze, silver and gold, and name tables for what they hold. A reader should be able to tell from the name how close a table is to the raw source and what one row represents.

Prefer groups to people in grants. Grant to a group and add people to the group. When someone changes team, you change one membership and not forty grants.

Write idempotent jobs. Ask of every write: "if this runs twice, what happens?" The answer should be "nothing new the second time". This one question prevents the majority of pipeline incidents.

Parameterise instead of editing. Pass dates, table names and environments into a notebook as widgets or job parameters. A notebook you must edit before running is a notebook that will be run wrongly.

Add a failure notification to every job. An unwatched job that fails silently is worse than no job, because people assume the data is fresh. Email costs nothing.

Look at the data before you trust it. After every load, count rows, check for nulls in key columns and glance at a sample. Five lines of checking catch a lot of broken sources.

df = spark.read.table("course.bronze.trips_clean")
print(df.count())
df.select([(df[c].isNull().cast("int")).alias(c) for c in df.columns]).groupBy().sum().display()
df.limit(5).display()

Read the error class first. The bracketed class at the top of an error message, such as [TABLE_OR_VIEW_NOT_FOUND], is the stable part. Search the Databricks error reference for it, and read the explanation before you read any stack frames.

Keep a tidy sandbox. Create your own schema for experiments and drop what you no longer need. Shared catalogs fill with half-finished tables called test2_final, and nobody can tell which ones matter.

Cost awareness for beginners

You do not need to understand pricing in detail to avoid the common surprises. Remember three ideas. First, classic compute costs money while it runs, both in Databricks units and in cloud machines, so stopping it matters. Second, work that does not need to be interactive belongs in a job, not an open cluster. Third, scanning a huge table repeatedly costs more than writing the aggregate you need once into a smaller table.

A few cheap habits follow. Use limit while you explore, and do not run a full-table query to see what the columns look like. Cache or save intermediate results you reuse, in a table, rather than recomputing them in every cell. Choose the smallest compute that does the job, and turn on Photon only when a query is large enough to benefit, since it carries a higher rate. On a company account, ask whether tags are required on compute, because tagged spending can be attributed to a project, and untagged spending tends to be questioned later.

If you are on a free learning workspace, the limits you meet are about allowances and not invoices, but form the habits anyway. The reflexes you build there are the ones your employer will expect.

Working with a team

Databricks is shared by nature. A few courtesies make a team workspace pleasant. Put your personal experiments under your own folder and your own schema, and keep shared catalogs for things the team has agreed on. Do not edit another person's notebook in place, copy it or use Git branches. Leave a short description on tables you create, since the catalog browser shows it to everyone, and an undescribed table is a question waiting to be asked.

When you ask for help, bring information. Say which workspace and compute you used, paste the full three-level name and the first line of the error including its bracketed class, and say what you already tried. Questions like that get answered in minutes, and "it doesn't work" gets answered with more questions.

A short checklist

Keep this beside you for your first month.

When these habits feel automatic, move on to the Mid-level guide. It deals with the next layer of problems: permissions at scale, incremental ingestion, running jobs as service principals, and developing from your own editor with Databricks Connect.

This is part two of three. It assumes you already run notebooks and the odd job at work, and takes you from "it runs" to "I can predict what it will do, what it will cost and how to ship it". The runtime used for examples is Databricks Runtime 18 LTS (Spark 4.1), the newest long-term-support release as of this writing, and the CLI is the 1.x line.

Where this picks up

Each section starts with the problem it solves and ends with a Try it you can run in your own workspace.

What a Delta table really is

When something goes wrong with a table, such as a write that fails with a conflict, a query that scans far too much, or a VACUUM that breaks a reader, the cause is almost always in how Delta stores data. Knowing the structure lets you predict the behaviour.

A Delta table is a directory in cloud storage containing two kinds of thing. The first is ordinary Parquet data files. The second is a folder called _delta_log holding a numbered sequence of JSON commit files (00000000000000000000.json, 00000000000000000001.json, and so on), with periodic checkpoint files that summarise the state so a reader does not replay thousands of commits. Each commit records actions: "add this file", "remove that file", "change the schema", along with per-file statistics such as the minimum and maximum of the first columns and the row count.

The table's current contents are defined as the result of replaying the log. Nothing is ever edited in place. An UPDATE writes new Parquet files containing the changed rows and then commits a log entry that removes the old files and adds the new ones. Three things follow directly.

Time travel is free until you vacuum. The old files are still on storage after a commit removes them from the current state, which is why VERSION AS OF 3 works. VACUUM deletes files that no current or recent version needs and are older than the retention threshold, which defaults to seven days. After that, those versions cannot be read and RESTORE to them fails. If an audit requirement says you must be able to go back thirty days, you raise delta.deletedFileRetentionDuration, you do not just avoid running VACUUM.

Queries skip files using the statistics. Because the log records min and max values per file, a filter such as WHERE event_date = '2026-09-01' lets the engine discard every file whose range excludes that date without opening it. That is data skipping, and it is why file layout matters: if every file contains every date, nothing can be skipped.

Writers coordinate by optimistic concurrency. A writer reads the table at version N, does its work, then tries to commit version N+1. If someone else already committed N+1, the loser checks whether the other commit touched anything it read. If not, it retries on top of the new version silently. If so, it fails with a conflict exception (ConcurrentAppendException, ConcurrentDeleteReadException and relatives, usually shown with a DELTA_CONCURRENT_... error class). Two jobs appending to different regions of a table will usually coexist; two jobs running MERGE against overlapping rows will not.

DESCRIBE DETAIL course.silver.users;            -- format, numFiles, sizeInBytes, clustering columns
DESCRIBE HISTORY course.silver.users LIMIT 5;   -- operation, operationParameters, operationMetrics
RESTORE TABLE course.silver.users TO VERSION AS OF 12;

DESCRIBE HISTORY is the best forensic tool you have. Its operationMetrics column shows, per commit, how many files were added and removed and how many rows were written. A MERGE that rewrote 4,000 files to change 200 rows is visible there, and that is your first clue to a performance problem.

Unity Catalog in practice

Beginner level showed the three-level name and a first GRANT. At work the problems are different: who owns what, how to avoid granting to individuals, and how to keep development from touching production data.

Ownership is separate from privileges. Every securable object has exactly one owner, normally the principal that created it. The owner can grant on it and manage it. Production objects created by a person leave a hole in the plan, because when that person leaves, the owner has gone. Make groups or service principals the owners of shared objects, and create production objects from automation rather than from a laptop session.

Grant to groups, never to people. Groups come from your identity provider (Entra ID, Okta) via SCIM provisioning, so access follows team membership. A grant to alice@company.com outlives her move to another team. A grant to analytics-readers does not.

Privileges are checked at every level of the hierarchy. To read course.silver.users a principal needs USE CATALOG on course, USE SCHEMA on course.silver and SELECT on the table. Missing the first two is the most common reason a "granted" user still cannot read anything. Privileges inherit downward, so SELECT on a schema covers every current and future table in it, which is convenient and also the reason to grant at the schema level only when you mean it.

GRANT USE CATALOG ON CATALOG course_prod TO `analytics-readers`;
GRANT USE SCHEMA, SELECT ON SCHEMA course_prod.gold TO `analytics-readers`;
GRANT ALL PRIVILEGES ON SCHEMA course_dev.silver TO `data-engineers`;
SHOW GRANTS ON SCHEMA course_prod.gold;
SHOW GRANTS `analytics-readers` ON CATALOG course_prod;

An environment is a catalog. The pattern that scales is one catalog per environment (course_dev, course_prod) with identical schema and table names inside, so code written against ${catalog}.silver.users runs everywhere with the catalog supplied as a parameter. You can also bind a catalog to specific workspaces, so production data is reachable only from the production workspace even for a user who has the grants.

Managed versus external. A managed table's files live in storage that Unity Catalog controls, and dropping the table eventually removes the data. An external table points at a path you own, governed through an external location (a storage path paired with a storage credential, the cloud identity used to access it). Prefer managed for anything new: you get the platform's maintenance features, and nobody can modify the files from outside the governance layer. Use external when other systems must write the same files, or when the data already exists in a bucket you cannot move. The same distinction applies to volumes, which are the Unity Catalog home for non-tabular files such as images, model artefacts and raw drops, addressed as /Volumes/<catalog>/<schema>/<volume>/....

Row filters and column masks let you restrict rows and mask values inside a shared table, so one gold table can serve several audiences. Attribute-based access control policies on views and at metastore level are in beta as of September 2026, so treat them as something to evaluate rather than rely on. For data residency, remember the metastore is regional: a Gulf or Egyptian employer with a residency requirement should confirm which cloud region the workspace and its metastore sit in before any data lands.

Choosing compute

Compute is where money and time go, and there is a decision rule that covers most cases.

Serverless first. For notebooks, jobs and SQL, serverless compute removes sizing, start-up time and runtime upgrades: Databricks manages the machines in its own account and upgrades the runtime for everyone at once. Because the runtime is versionless, serverless notebooks and jobs use an environment version that you pin so behaviour changes only when you decide. The costs of that convenience are restrictions: no RDD APIs and no direct sparkContext access, no custom JARs, and only the Spark configurations Databricks allows. If your code uses those, you will see an error such as "feature not supported on serverless compute", and classic compute is your answer.

Classic job clusters for scheduled work that needs control. A job cluster is created for the run and destroyed afterwards. It costs less per DBU than an all-purpose cluster and cannot be left running by accident. Configure autoscaling with a minimum and maximum worker count, and use spot instances with on-demand fallback for tolerant batch work.

All-purpose clusters for interactive work only. They are billed at a higher rate, and the typical waste is a large cluster left up overnight. Always set auto-termination, and never schedule production jobs onto one.

SQL warehouses for SQL and BI. They come in classic, pro and serverless types, with t-shirt sizes, a min and max cluster count for concurrency, and auto-stop. Serverless warehouses start in seconds, which makes aggressive auto-stop cheap.

Access modes decide isolation. Recent documentation calls the two main modes standard (formerly shared) and dedicated (formerly single user). Standard mode lets several users share compute with isolation enforced between them, and is the right default for team clusters. Dedicated mode ties compute to one user or group identity, and is for workloads that need capabilities standard mode lacks. Check your runtime's documentation for which features each supports.

Photon is the native vectorised engine. It accelerates SQL and DataFrame operations (scans, joins, aggregations, Delta writes) and bills at a higher DBU rate, so it pays off when it shortens runtime by more than it raises the rate. It does nothing for Python UDF-heavy code, where time is spent outside the engine. Measure rather than assume: run the same job twice with and without it and compare the total cost, not just the duration.

Cluster policies are the control that stops cost problems at the source. A policy is a set of rules over cluster attributes: allowed node types, maximum workers, required runtime versions, mandatory tags, forced auto-termination. Give teams a policy rather than free-form creation rights. The tag rule matters most, because without a team or project tag on every cluster you cannot attribute spend later.

{
  "autotermination_minutes": {"type": "range", "maxValue": 60, "defaultValue": 30},
  "custom_tags.team": {"type": "fixed", "value": "churn-ml"},
  "autoscale.max_workers": {"type": "range", "maxValue": 8}
}

For classic compute, pin the runtime to a long-term-support release (for example 18 LTS) rather than a floating latest, and schedule upgrades deliberately. The upgrade section below covers how.

Incremental ingestion with Auto Loader

Batch pipelines that re-read everything are slow and expensive; pipelines that track what is new are cheap. Auto Loader is Databricks' answer for files landing in cloud storage or a volume. It is a Structured Streaming source named cloudFiles that remembers which files it has processed.

Two locations matter, and confusing them is the most common beginner-to-mid error. The checkpoint location belongs to the stream: it records progress, meaning which files are done and the state of the write. The schema location belongs to the source: it stores the schema Auto Loader inferred and how it evolved. Both must be on durable storage you control, typically a volume, and neither should be shared between streams.

raw = "/Volumes/course_dev/bronze/raw/events"
chk = "/Volumes/course_dev/bronze/chk/events"

(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", f"{chk}/schema")
    .option("cloudFiles.inferColumnTypes", "true")
    .load(raw)
    .writeStream
    .option("checkpointLocation", f"{chk}/stream")
    .option("mergeSchema", "true")
    .trigger(availableNow=True)
    .toTable("course_dev.bronze.events"))

availableNow=True is incremental batch. The stream starts, processes everything that arrived since the last run, and stops. You get exactly-once file processing through the checkpoint, no always-on cluster, and the job is an ordinary scheduled task. Use continuous streaming only when latency requirements demand it. Most pipelines that people build as continuous streams are happier as availableNow runs every few minutes.

Schema evolution is a policy, not a default. When new columns appear in the source, Auto Loader's behaviour depends on its schema evolution mode. The default fails the stream once to record the new schema and succeeds on restart, which surprises people who expect it to just work. That is deliberate: unannounced upstream changes should be noticed. Decide per source whether you want to add new columns automatically, rescue unexpected data into a rescued-data column, or fail. The destination table also needs to accept the change, which is what mergeSchema on the write controls, separately from the source setting.

Bronze should be forgiving, silver strict. Land raw data with minimal interpretation, keeping unparsed content where you can, so a downstream bug never forces you to re-fetch from the source system. Enforce types and quality rules at the silver step, where a failure is visible and fixable.

Never delete or move the checkpoint casually. Deleting it makes the stream start from scratch and reprocess every file, duplicating rows in an append-only target. If you must reprocess deliberately, point at a new checkpoint and a fresh target, validate, then swap.

For upserts into silver, use foreachBatch with a MERGE, so each micro-batch is merged by key rather than appended:

from delta.tables import DeltaTable

def upsert(batch_df, batch_id):
    target = DeltaTable.forName(spark, "course_dev.silver.users")
    (target.alias("t")
        .merge(batch_df.dropDuplicates(["id"]).alias("s"), "t.id = s.id")
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute())

(spark.readStream.table("course_dev.bronze.users")
    .writeStream
    .foreachBatch(upsert)
    .option("checkpointLocation", "/Volumes/course_dev/silver/chk/users")
    .trigger(availableNow=True)
    .start())

Note the dropDuplicates inside the batch. A MERGE fails if several source rows match the same target row, so deduplicate within the batch first, by whatever rule defines the winner (usually the latest timestamp).

Table design: clustering, maintenance and change feeds

File layout determines how much a query reads. The old approach was partitioning by a column plus ZORDER during OPTIMIZE. For new tables, the current approach is liquid clustering.

Partitioning splits a table into physical directories by column value. It helps only for low-cardinality columns that you always filter on, and is hard to change later; over-partitioned tables end up with millions of tiny files. Liquid clustering replaces it with a clustering key you can declare and change without rewriting the whole table, and clustering is applied incrementally as you write and as you run OPTIMIZE.

CREATE TABLE course_dev.silver.events (
  id BIGINT, user_id BIGINT, event_date DATE, kind STRING, payload STRING
) CLUSTER BY (event_date, user_id);

ALTER TABLE course_dev.silver.events CLUSTER BY (user_id, event_date);  -- change the key later
OPTIMIZE course_dev.silver.events;                                      -- cluster newly written data

Choose clustering keys from the columns you filter and join on most. Do not combine liquid clustering with partitioning or ZORDER on the same table; they are alternatives.

Small files are the silent cost. Many small writes produce many small files, and every file costs a metadata operation and an open. Two mechanisms counter that: optimized writes, which size files better as you write (enabled with spark.databricks.delta.optimizeWrite.enabled where your compute does not already do it for you), and OPTIMIZE, which compacts existing files. Check DESCRIBE DETAIL: an average file size of a few megabytes on a large table is the signal.

Let the platform do it where you can. Predictive optimization runs OPTIMIZE and VACUUM for managed tables automatically, so for new work on Unity Catalog managed tables check whether it is enabled for your catalog before writing a maintenance job. If it is not available to you, schedule OPTIMIZE and VACUUM as a job.

VACUUM retention is a contract. Default retention is seven days. Everything that reads or streams from a table needs its files to exist: a long-running query, a stream that has been stopped for more than a week and restarts, a time-travel audit. Shorten retention only after considering all of them. The relevant table properties are delta.deletedFileRetentionDuration for data files and delta.logRetentionDuration for history.

Change data feed records row-level changes (insert, update, delete, with before and after images), so downstream consumers can process only what changed. Enable it per table, then read the changes by version:

ALTER TABLE course_dev.silver.users SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT _change_type, _commit_version, id, email
FROM table_changes('course_dev.silver.users', 5)
WHERE _change_type IN ('update_postimage', 'insert');

Databricks announced in September 2026 that automatic change data feed computed at query time is generally available, which reduces the need to enable the property in advance for some tables; check the current documentation for which table types are covered before depending on it.

Constraints and expectations. Delta supports NOT NULL and CHECK constraints that reject bad writes at the door. They are the cheapest quality gate you have, and they turn silent corruption into a loud failure at the writer.

ALTER TABLE course_dev.silver.users ADD CONSTRAINT valid_email CHECK (email LIKE '%@%');

Making slow Spark fast

Most slow jobs have one of four causes: reading too much, shuffling too much, skew, or spilling to disk. The method is to find which one with evidence before changing anything.

Start with what the query reads. In the SQL Query Profile, or the Spark UI's SQL tab for notebook work, find the scan nodes. They report files and bytes read, and how many were pruned. If a query for one day reads the whole table, the layout is wrong (see clustering) or the filter is not pushed down because you wrapped the column in a function. WHERE to_date(ts) = '2026-09-01' can defeat skipping that WHERE ts >= '2026-09-01' AND ts < '2026-09-02' would use.

Then look at shuffles. A shuffle moves data across the network so that rows with the same key meet: joins, group-bys and window functions cause them. The exchange nodes show how much data moved. Ways to reduce it: filter and select columns before the join, so less data travels; broadcast the small side of a join so the big side does not move at all; avoid repeated shuffles on the same key by doing the dependent work together.

from pyspark.sql.functions import broadcast
joined = events.join(broadcast(dim_country), "country_code")

Skew is the one that looks like a hang. Most tasks in a stage finish in seconds and one runs for twenty minutes, because one key (a null, a default value, one giant customer) holds most of the rows. In the Spark UI's stage page, compare the maximum task duration and size with the median. A large gap is skew. Adaptive Query Execution, which is on by default, can split skewed partitions in joins at runtime. When it is not enough, filter out or separately handle the hot key, or salt it.

Spill and out-of-memory errors. "Spill" in the stage metrics means the data did not fit in memory and went to disk. Occasional spill is fine; large spill with slow tasks suggests too few partitions for the data volume. On classic compute you can raise partitions or use larger machines; on serverless the platform tunes spark.sql.shuffle.partitions for you. A driver out-of-memory error ("Driver is up but is not responsive, likely due to GC", or OutOfMemoryError: Java heap space) is almost always collect(), toPandas() or a huge broadcast on data that was too large. Keep results on the cluster, write them to a table, and sample or aggregate before pulling anything to the driver.

Python UDFs cost you. Row-at-a-time Python user-defined functions move data between the JVM and Python and defeat Photon. Prefer built-in functions, which are optimised and cover most needs. When you do need custom logic, Unity Catalog Python UDFs are available on all compute types as of September 2026, and vectorised pandas UDFs are better than scalar Python ones for batch work.

Cache sparingly. df.cache() helps when one expensive result is reused several times within a job, and hurts otherwise by consuming memory. It is not a substitute for writing an intermediate result to a table when the result is reused across jobs.

Spark 4 and ANSI mode. Runtimes based on Spark 4 (DBR 17 LTS and later) turn ANSI SQL mode on by default, which makes invalid casts and arithmetic overflow raise errors where Spark 3 returned null. It is a good change for correctness and a common source of "this worked before the upgrade" failures. Fix the data or use the try_ functions (try_cast, try_divide) where nulls are the intended behaviour.

SELECT try_cast(amount AS DECIMAL(12,2)) AS amount FROM course_dev.bronze.orders;

Jobs and orchestration

A scheduled notebook is a script. A job is a system: tasks with dependencies, retries, parameters, notifications and ownership. Getting those right is most of operational maturity.

Model the workflow as tasks in one job. Each task is a notebook, Python wheel, SQL file, dbt project or pipeline, and depends_on expresses order. This replaces chaining notebooks with dbutils.notebook.run, which gives you no per-step retry, no per-step status and no repair. Task values (dbutils.jobs.taskValues) pass small results between tasks.

Parameters, not edited notebooks. Define job parameters and read them in the notebook with widgets, or pass them to a wheel's entry point. The same code then runs in dev with a dev catalog and in prod with a prod catalog.

dbutils.widgets.text("catalog", "course_dev")
catalog = dbutils.widgets.get("catalog")
df = spark.read.table(f"{catalog}.silver.users")

Retries need idempotency. A task retry reruns the whole task. If the task appends and fails halfway, a retry duplicates rows. Design tasks so a rerun produces the same result: MERGE by key, or INSERT OVERWRITE for a partition or replaceWhere for a slice, or availableNow streams whose checkpoint tracks progress. Then set retries (a small number with a delay) only on tasks that are safe to repeat and fail for transient reasons.

Repair, don't rerun. When a task fails in a multi-task job, the repair run action reruns only the failed task and those downstream of it, reusing the successful results. A full rerun repeats expensive work and can duplicate effects.

Triggers. Besides cron schedules, jobs can run on file arrival in a volume or external location, and on table updates, which removes polling logic. Choose the trigger that expresses your real dependency: "when new data lands" is better than "every hour, hoping it landed".

Notifications and timeouts. Every production job needs failure notifications to a channel somebody watches, and a timeout on tasks so a hung run does not hold compute forever. Alerts to one person's email are a single point of failure.

Run as a service principal. A production job should run as an identity that is not a person, so it keeps working when staff change and so its permissions are exactly what production needs. Declare it as run_as in the bundle (next section).

Operate from the CLI. Everything above is scriptable:

databricks jobs list
databricks jobs run-now 123456789
databricks jobs get-run 987654321

Declarative Automation Bundles

Everything you configure by clicking in the UI is invisible to review, impossible to reproduce and drifts between environments. Declarative Automation Bundles (the new name for Databricks Asset Bundles, so older articles say "DABs" and the CLI group is still databricks bundle) put the definition in a repository. One databricks.yml describes jobs, pipelines, dashboards and serving endpoints, plus the targets they deploy to.

The mental model has four nouns: bundle (the project and its name), resources (the things to create), targets (environments such as dev and prod, each with a workspace and overrides), and variables (values that differ per target, referenced as ${var.name}).

bundle:
  name: churn-pipeline

variables:
  catalog:
    default: course_dev

resources:
  jobs:
    train:
      name: train-${bundle.target}
      tasks:
        - task_key: train
          notebook_task:
            notebook_path: ./notebooks/train.py
            base_parameters:
              catalog: ${var.catalog}
          environment_key: default

targets:
  dev:
    default: true
    mode: development
    workspace:
      host: https://<dev-workspace>.cloud.databricks.com
  prod:
    mode: production
    workspace:
      host: https://<prod-workspace>.cloud.databricks.com
      root_path: /Workspace/bundles/${bundle.name}/prod
    run_as:
      service_principal_name: "<service-principal-application-id>"
    variables:
      catalog: course_prod

The two modes encode good practice. In development mode the CLI prefixes resource names with [dev <your-username>] and pauses schedules, so your experiments neither collide with a colleague's nor run by themselves. In production mode it requires an explicit identity to run as and a workspace path that is not under one user's folder, which prevents the "prod runs as Alice" mistake. The top-level run_as takes a single identity.

The daily loop is five commands:

databricks bundle validate -t dev      # catches schema and reference errors, no changes made
databricks bundle deploy -t dev        # create or update the resources
databricks bundle run -t dev train     # start the job and follow it
databricks bundle destroy -t dev       # remove what this bundle created
databricks bundle deploy -t prod       # from CI, as the service principal

To adopt something built in the UI, databricks bundle generate job --existing-job-id <id> writes the YAML for it, and databricks bundle deployment bind attaches an existing resource to the bundle so it is updated rather than duplicated.

The deployment engine changed. Bundles used to deploy through Terraform, which the CLI downloaded and ran. From CLI v1.14.0, bundles that used Terraform state migrate automatically to the direct engine after a successful deploy, and you opt out with engine: terraform in the bundle configuration. The practical consequence arrived in v1.17.0: the direct deployment state version was bumped to 3, and clients older than v1.8.0 reject bundles deployed with it. When a team shares bundles, upgrade the CLI on laptops and in CI together, and pin the CLI version in CI rather than installing whatever is newest.

Bundles pair naturally with Git: the repository holds the code and the bundle, a pull request validates it, and a merge deploys it. Terraform remains the right tool for account-level and workspace-administration resources (catalogs, grants, cluster policies, groups) where a platform team owns them; bundles are for the projects teams ship on top. See Terraform for that side.

Authentication for people and automation

Personal access tokens are the habit most worth breaking. A token is a long-lived password tied to a person, copied into CI secrets and chat messages, and valid until someone remembers to revoke it. The documentation recommends OAuth instead, and the CLI and SDKs support it through unified authentication.

People use user-to-machine OAuth. You log in once through the browser and the CLI keeps refreshable tokens:

databricks auth login --host https://<workspace-url>
databricks auth profiles
databricks clusters list -p <profile>

Since CLI 1.0, those tokens are stored in your operating system's secure store (Keychain on macOS, Credential Manager on Windows, Secret Service over D-Bus on Linux). On a headless Linux machine or in a container without one, login fails, and the documented fallback is DATABRICKS_AUTH_STORAGE=plaintext. Use that on disposable machines only, and use machine OAuth in CI.

Automation uses machine-to-machine OAuth with a service principal. A service principal is a non-human identity. You create it, add it to the workspace, grant it exactly what the workload needs, and generate an OAuth secret for it. Its credentials go in a profile or environment variables:

[prod]
host          = https://<workspace-url>
client_id     = <service-principal-client-id>
client_secret = <oauth-secret>

In CI the same values come from the pipeline's secret store as DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, and the CLI and SDK pick them up with no code. If your CI provider supports it, the documentation describes OAuth token federation, which exchanges the pipeline's own identity token for a Databricks one and removes the stored secret entirely; check the current page for your provider before relying on it.

Know the precedence when behaviour surprises you. Bundle settings win over environment variables, which win over config profiles. A stray DATABRICKS_HOST in your shell can silently override the profile you thought you were using, so when a command targets the wrong workspace, print the environment before anything else. databricks current-user me tells you who and where you are authenticated as, in one line.

Secrets in notebooks come from secret scopes. Create them from the CLI and read them with dbutils.secrets.get; output is redacted as [REDACTED] if you try to print them. Treat redaction as a courtesy rather than a boundary, since code that can read a secret can transform it.

databricks secrets create-scope pipeline
databricks secrets put-secret pipeline api-key

key = dbutils.secrets.get(scope="pipeline", key="api-key")

The error you will meet is default auth: cannot configure default credentials when no credentials are configured at all, and 401 or an invalid-token message when a token or OAuth session has expired. For the second, rerun databricks auth login.

MLflow models in Unity Catalog

MLflow is built into Databricks, and at this level the work is moving from "I logged a run" to "a model has a governed identity and a safe path to production". Databricks Runtime 18 LTS ships MLflow 3.8.1. Rely on the runtime's version and check the MLflow docs for exact API changes between releases, since MLflow 3 changed how models are logged.

Point the registry at Unity Catalog. Registered models then live in catalog.schema.model and use the same grants as tables. The older workspace model registry, with its Staging and Production stages, is legacy; do not start new work on it.

import mlflow
from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")
mlflow.set_experiment("/Users/<you>/churn")

with mlflow.start_run():
    mlflow.log_param("alpha", 0.1)
    mlflow.log_metric("rmse", 0.42)
    mlflow.sklearn.log_model(
        model, "model",
        registered_model_name="course_dev.models.churn",
    )

Log a signature. A model signature records the input and output schema. It is what lets serving validate requests and what tells a colleague what the model expects. mlflow.models.infer_signature(X_train, model.predict(X_train)) produces one, and passing it as signature= to log_model stores it. Models in Unity Catalog require a signature.

Aliases are the promotion mechanism. An alias such as champion is a movable pointer to a version. Consumers load models:/course_prod.models.churn@champion, and promotion is moving the pointer, not editing consumers:

client = MlflowClient()
client.set_registered_model_alias("course_prod.models.churn", "champion", 3)
model = mlflow.pyfunc.load_model("models:/course_prod.models.churn@champion")

The stage-based calls (transition_model_version_stage, models:/name/Production) belong to the legacy registry; do not use them against Unity Catalog models.

Promote across environments by copying, with gates. Because dev and prod are separate catalogs, promotion is a copy of a model version from course_dev.models.churn to course_prod.models.churn (the client has copy_model_version for this), followed by setting the alias after checks pass. Put the checks in code: evaluate the candidate on a fixed holdout, compare against the current champion, and fail the pipeline when it is worse. The gate that matters is not accuracy alone but whether the schema matches what consumers send.

client.copy_model_version(
    src_model_uri="models:/course_dev.models.churn/5",
    dst_name="course_prod.models.churn",
)

Serving. A registered model can be exposed as a REST endpoint with Mosaic AI Model Serving, created in the UI, with the SDK (w.serving_endpoints.create) or as a bundle resource, which is the version you can review. Serving endpoints can scale down when idle, which saves cost and means the first request after quiet pays a cold start; decide whether your latency requirement allows that. Service-to-service callers should authenticate as service principals, not with someone's token.

Reproducibility. Log what produced the model: the code version (Git commit as a tag), the input table version (the Delta version you trained on, from DESCRIBE HISTORY), parameters and the environment. Months later, "what data trained this?" should have an answer in the run.

For a wider treatment of tracking and registry concepts outside Databricks, see MLflow, and for the table format itself Delta Lake.

Developing locally and testing

The notebook is a good place to explore and a poor place to build something you must trust. At this level, move reusable logic into Python modules in a repository, test it, and deploy it through a bundle.

Databricks Connect lets code running on your laptop (in an editor, a debugger or pytest) execute Spark operations on Databricks compute. You write ordinary PySpark, and the query runs remotely.

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.serverless(True).getOrCreate()
spark.read.table("samples.nyctaxi.trips").limit(5).show()

Two rules prevent most install pain. First, the client version must match the runtime you connect to; Databricks Connect supports DBR 13.3 LTS and above, and the package version tracks the runtime, so a DBR 18 target wants a matching 18.x client (check the install page for the exact pin rule). Second, install it in a clean virtual environment and do not install pyspark alongside it: the two conflict. The error to recognise is that a cluster ID or serverless was required but not specified, which means the session builder had nothing to connect to; set serverless(True), name a cluster, or configure a profile.

Structure code so most of it needs no cluster. Write transformations as functions that take a DataFrame and return one, with table names and paths passed in. Then most tests can build a tiny DataFrame, call the function and assert on the result.

# src/churn/features.py
from pyspark.sql import DataFrame, functions as F

def add_tenure_days(df: DataFrame) -> DataFrame:
    return df.withColumn("tenure_days", F.datediff("end_date", "start_date"))

# tests/test_features.py
import datetime as dt
from churn.features import add_tenure_days

def test_tenure(spark):
    df = spark.createDataFrame(
        [(dt.date(2026, 1, 1), dt.date(2026, 1, 11))], ["start_date", "end_date"]
    )
    assert add_tenure_days(df).first()["tenure_days"] == 10

The spark fixture goes in conftest.py and builds a DatabricksSession for integration runs. Keep two layers of test: fast tests of pure functions, and a few integration tests of a pipeline against a dev catalog, run in CI with the service principal. Tests that write should write to a schema created for the run and drop it afterwards, so parallel runs never collide.

Test data quality, not just code. A transformation can be correct and still produce nonsense when upstream data changes. Add checks inside the pipeline (constraints, row-count thresholds, uniqueness assertions on keys) so a bad input fails the run instead of reaching a dashboard. For a fuller treatment of data tests see Great Expectations.

The SDK for automation. Anything the CLI can do, the Python SDK can do from code, with the same unified authentication:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
for job in w.jobs.list(limit=10):
    print(job.job_id, job.settings.name)

Lint the bundle in CI. A reasonable pull-request pipeline runs unit tests, then databricks bundle validate -t dev, and on merge databricks bundle deploy -t prod as the service principal. This is also where GitHub Actions or your CI of choice meets the platform.

Cost and observability with system tables

You cannot control what you cannot see. Databricks exposes operational data as system tables in the system catalog, queryable like any other table once an admin has granted access. The three to know are system.billing.usage (consumption), system.access.audit (who did what) and system.query.history (SQL queries across the account, generally available as of September 2026).

Bill = DBUs times the SKU rate, plus cloud infrastructure on classic compute. A DBU is a unit of processing capability per hour, priced differently per product. Serverless bundles the infrastructure into the DBU rate, so one line item covers both. Billing tables can lag, by up to about a day for serverless, so do not alarm on today's number.

SELECT usage_date,
       sku_name,
       custom_tags['team'] AS team,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY dbus DESC;

This query only becomes useful if the team tag is present, which loops back to cluster policies that make tags mandatory. For jobs and serverless there are budget policies that apply tags to usage, and query tags for SQL warehouses reached general availability in September 2026 for attributing warehouse spend.

The levers, by size of effect. Move scheduled work from all-purpose to job clusters or serverless. Set auto-termination and warehouse auto-stop. Cap autoscaling maximums with policies. Use spot capacity for tolerant batch jobs. Cut the data you read with clustering and filter pushdown. Stop re-running full refreshes when an incremental path exists. Review the top ten consumers monthly: costs concentrate in a few jobs.

Observe jobs, not just infrastructure. The jobs UI and API show run durations. Watch for trends, not just failures: a job that creeps from ten to twenty-five minutes over a month is telling you the data is growing faster than the design. System tables carry job and compute usage too, so a dashboard of duration and cost per job is a few queries.

Auditing. system.access.audit records actions such as grants, logins and table access. Use it to answer "who changed this grant" and to check that your service principals do only what you expect.

Metrics inside the platform. The Spark UI and compute metrics show executor memory and CPU, and Query Profile shows SQL plans. For models and GenAI applications, MLflow tracing and evaluation capture behaviour of the application, and Databricks Apps can emit OpenTelemetry telemetry (generally available as of September 2026) to your existing tools such as OpenTelemetry collectors and Grafana.

Upgrades, deprecations and keeping current

Databricks moves quickly. The skill is knowing what changes under you and what you control.

Three release streams. The runtime (DBR) changes on a published schedule, with long-term-support releases supported for three years: 14.3 LTS and 15.4 LTS are older LTS releases, 16.4 LTS and 17.3 LTS are current, and 18 LTS (Spark 4.1) is the newest. A non-LTS release such as DBR 19 (Spark 4.2) is for early evaluation, not for production. Serverless is versionless and updated for you, with behaviour governed by the environment version you select. The CLI, SDKs, Databricks Connect and Terraform provider release independently on GitHub and PyPI.

Upgrade the runtime as a project. Read the runtime release notes and the Spark migration guide for each Spark version you cross, rerun your test suite against a cluster on the new runtime, and look especially at: stricter ANSI behaviour in Spark 4, library compatibility (Scala 2.13 only, newer Java), and anything that used removed or deprecated APIs. Do it in dev first, then roll production job by job. Because a LTS is supported for three years, a yearly upgrade rhythm keeps you comfortably inside support.

Retire the legacy patterns. These still work in places and will bite you in new code:

Delta Live Tables has also been renamed, to Lakeflow Spark Declarative Pipelines. Older tutorials use the old name and the old dlt module, so check which API your runtime expects before copying code.

Watch the release notes. Skim the monthly platform release notes. They announce deprecations with dates; for example, notes from September 2026 deprecated a beta Genie MCP endpoint on 31 October 2026 and retire several hosted models on 30 October. Putting a recurring calendar item for that reading costs ten minutes a month and saves an incident.

Putting it all together

Here is the whole level as one small project: a daily churn pipeline built so that a new teammate could deploy it, and an auditor could understand it.

1. Layout. A repository with src/churn/ (functions), tests/, notebooks/ (thin entry points), and databricks.yml.
2. Governance. Catalogs course_dev and course_prod, schemas bronze, silver, gold and models. A group churn-engineers has all privileges in dev. In prod only the service principal churn-sp can write; analytics-readers have USE CATALOG, USE SCHEMA and SELECT on gold.
3. Ingestion. An Auto Loader task with availableNow lands raw JSON from a volume into bronze.events, with schema and checkpoint locations on the volume.
4. Transformation. A foreachBatch merge keyed on id produces silver.users, clustered by the columns reports filter on, with a CHECK constraint on the key fields. Tasks are idempotent.
5. Training. A task trains a model on a recorded Delta version of silver, logs it with a signature to course_dev.models.churn, and tags the run with the Git commit and the table version.
6. Promotion. A gate task evaluates the new version against the current champion on a fixed holdout. When it passes in dev, the pipeline copies the version to course_prod.models.churn and moves the champion alias.
7. Deployment. CI runs pytest, then databricks bundle validate -t dev. On merge to main, a job authenticated as the service principal runs databricks bundle deploy -t prod. The CLI version is pinned.
8. Operations. Jobs have failure alerts to a team channel, task timeouts and retries only on safe tasks. A policy forces a team tag and auto-termination, and a weekly query over system.billing.usage reports spend per tag.

Where you are now

You can explain Delta from its transaction log, which means you can predict conflicts, skipping behaviour and what VACUUM will break. You grant on groups through the three-level hierarchy and keep environments apart with catalogs. You choose compute by a rule rather than habit, ingest incrementally, design tables with clustering and checks, and diagnose slow queries from scans, shuffles and skew. Your jobs are idempotent and repairable, your projects live in bundles, your automation uses OAuth service principals, and your models have governed names and aliases. You can see cost in system tables and plan runtime upgrades.

The senior level steps back to running Databricks as a platform for other teams: workspace and network architecture, security and compliance models, multi-tenancy, governance at scale, migrations and incident handling. If you work with the neighbouring tools, Spark covers the engine in more depth, and Delta Lake the open table format.

Sources

• Databricks documentation
• Platform release notes
• Databricks Runtime release notes
• Supported runtimes and end-of-support dates
• September 2026 release notes
• Databricks CLI install
• Databricks CLI authentication
• Databricks CLI releases on GitHub
• Declarative Automation Bundles
• Bundle settings
• Databricks Connect for Python
• Unity Catalog
• MLflow on Databricks
• Serverless compute
• Apache Spark migration guides

Part two of three. At this level interviewers stop asking what Databricks is and start asking what you do when something breaks, how you would design a piece of it, and why you chose one option over another. This file is organised as scenarios first, trade-offs second, and a note on what separates strong answers from weak ones. Read it with the detailed guide beside you; the answers here are compressed on purpose.

Where this picks up

The sixty-second frame

If you are asked to describe how you work with Databricks, say something like this. "Code lives in Git and deploys through Declarative Automation Bundles, so dev and prod are the same definition with different targets. Data is in Unity Catalog with one catalog per environment, access is granted to groups, and production runs as a service principal using OAuth. Pipelines are incremental and idempotent, tables are managed and clustered, and I watch cost through tagged compute and the billing system tables." That sentence covers six topics, and each one is a follow-up question you should be ready for.

Scenario questions

A job that ran in twenty minutes now takes two hours. What do you check first?

Start by establishing what changed rather than guessing at tuning. Compare the slow run with a fast one in the jobs UI: same code version, same cluster configuration, same parameters? Then compare the data: did input volume jump? DESCRIBE HISTORY on the input tables shows new commits and how large they were.

If code and configuration are unchanged, open the slow run's Spark UI or Query Profile and look at three numbers in order. Bytes read per scan, because a table that lost its layout (or a filter that stopped pushing down) reads everything. Shuffle size, because a join that used to broadcast may have turned into a shuffle when a dimension table grew past the threshold. And per-stage task times, because one task running far longer than the median is skew from a new hot key, often a surge of null or default values.

A strong answer names the order of investigation and the evidence for each hypothesis. A weak answer jumps to "add more workers", which pays more to run the same bad plan.

Two jobs write to the same Delta table and one fails with a concurrent-modification error. Why, and what do you do?

Delta uses optimistic concurrency: each writer reads a version, works, and tries to commit the next one. If another commit arrived first and touched data this writer read, the commit fails with a conflict exception such as ConcurrentAppendException. Plain appends to different data usually coexist; overlapping MERGE, UPDATE or DELETE operations do not.

Fixes in order of preference: stop them overlapping by design (one writer per table, or per partition or slice with the filter written into the merge condition so the engine can see they are disjoint), cluster the table so row-level concurrency can apply to your operation, serialise the jobs with task dependencies, and as a last resort retry with backoff. Retrying is safe only if the task is idempotent. A strong answer mentions that retries reapply the whole task and ties the fix to the design rather than blindly wrapping the call.

A user says they have access to a table but get "table not found". Walk me through it.

Unity Catalog hides what you cannot access, so a privilege problem often looks like a missing object. Check, in order: spelling and the full three-level name; whether the user has USE CATALOG on the catalog and USE SCHEMA on the schema, since SELECT on the table alone is not enough; whether the grant went to the user's group and the user is really in it; whether the catalog is bound to the workspace they are using; and whether the compute's access mode supports the operation. SHOW GRANTS at each level, run by someone who can see the objects, answers most of it.

A strong answer knows the "not found instead of denied" behaviour. A weak one re-grants SELECT repeatedly and wonders why nothing changes.

Your CI pipeline deploys a bundle and fails after an upgrade, though nothing in the bundle changed. Suspects?

The deployment engine and its state. From CLI v1.14.0 bundles that used Terraform state migrate to the direct engine after a successful deploy, and in v1.17.0 the direct deployment state version moved to 3. Clients older than v1.8.0 reject bundles deployed with that release. So the first questions are: which CLI version is CI using, which version did the last successful deploy use, and does a colleague on an old laptop deploy to the same target? The remedy is to pin one CLI version across everyone and upgrade together. Also check authentication: a changed environment variable can override a profile, since environment variables outrank config profiles and bundle settings outrank both.

The nightly cost report doubled. Where do you look?

Start with system.billing.usage grouped by SKU and day to find which product jumped, then by usage_metadata or custom tags to find which job, warehouse or cluster. Common causes: an all-purpose cluster left running or without auto-termination, a job that moved from incremental to full refresh, a new serverless workload without a budget policy, a warehouse with auto-stop disabled, or retries multiplying a failing job's cost. Remember that billing data can lag, so confirm on more than one day. Then fix the control, not just the instance: a policy forcing tags and auto-termination, so it cannot recur silently.

A model must be promoted from dev to prod with approval. How do you build it?

Use separate catalogs, course_dev.models.x and course_prod.models.x. The pipeline registers a candidate in dev with a signature and tags (Git commit, training table version). A gate step evaluates the candidate against the current prod champion on a fixed holdout and fails if it is worse or if the schema changed. On success, the pipeline copies the version into the prod catalog and moves the champion alias to it. Consumers load by alias, so promotion needs no consumer change, and rollback is moving the alias back. The prod write runs as a service principal, and a human approval can sit as a manual step in CI before the copy. A strong answer rejects the workspace registry's stages as legacy.

A streaming job restarts and reprocesses everything. What happened?

Most likely the checkpoint was lost, moved or pointed at a different path, so the stream believed it had never run. With Auto Loader, a missing checkpoint means every file in the source looks new. Check that the checkpoint location is durable, unique to this stream and unchanged in the latest deploy. If the deploy changed a path through a variable, that is the cause. To recover without duplicates in an append-only target, deduplicate through a merge, or restore the table to the pre-incident version with RESTORE and restart cleanly.

VACUUM ran and now a long-running reader fails with a missing file. Why?

VACUUM deletes files that are no longer part of the current table version and are older than the retention threshold, default seven days. A reader holding an old snapshot longer than that, or a stream paused for more than the retention window, refers to files that no longer exist. Increase retention to exceed your longest reader and recovery window, and never set retention near zero on tables others read.

Trade-off questions

Serverless or classic compute?

Serverless removes sizing, startup delay and upgrades, and is my default for notebooks, jobs and SQL when the code is compatible. Classic wins when you need things serverless restricts: RDD APIs, custom JARs, particular Spark configurations, specific instance types or a private network arrangement that serverless does not support yet. Cost is workload-dependent: serverless includes infrastructure in the DBU price and wins on bursty or short workloads where classic would sit idle or start slowly; a steady, long, tuned batch job on spot instances can be cheaper on classic. Answer with a rule and a measurement plan, not a slogan.

Managed or external tables?

Managed by default: Unity Catalog controls the storage and lifecycle, maintenance features like predictive optimization apply, and nothing outside governance can alter the files. External when another system must write or read the files directly, or data already exists in storage you cannot move. The cost of external is that you own cleanup and layout.

Liquid clustering or partitioning?

Liquid clustering for new tables: you can change keys without a rewrite, it handles high-cardinality columns, and you avoid the small-file problem of over-partitioning. Partitioning remains in legacy tables, and may suit very large low-cardinality tables with strict lifecycle needs, such as dropping a day at a time. Never mix both mechanisms on the same table.

Continuous streaming or availableNow batches?

If the latency requirement is minutes, availableNow on a short schedule gives you incremental processing with no always-on cluster and normal job semantics. Continuous streaming is justified when seconds matter, and then you pay for always-on compute and take on monitoring of a long-lived process. Most requirements stated as "real time" are not.

Bundles or Terraform?

Both, with a boundary. Terraform for account and workspace administration that a platform team owns: catalogs, grants, policies, groups, workspaces. Bundles for the project resources teams deploy: jobs, pipelines, dashboards, serving endpoints, alongside the code. Mixing creates ownership fights, so decide per resource type who manages it and enforce that.

Personal access tokens or OAuth?

OAuth. Tokens are long-lived credentials tied to a person, easy to leak and rarely rotated. Interactive work uses user OAuth, automation uses a service principal with machine OAuth, or federation where the CI provider supports it. The cost is some setup work, which is paid once.

One workspace or several?

Several, typically one per environment, sharing a regional metastore with separate catalogs and catalog bindings. A single workspace is simpler but mixes blast radius and makes network and access separation harder. Add workspaces when you need different networking, compliance posture or administrative ownership.

Short technical questions with model answers

What does a Delta commit contain? A JSON file in _delta_log listing actions such as added and removed files and schema changes, with per-file statistics used for data skipping.

Why does MERGE fail with multiple matches? More than one source row matched the same target row, so the result of the update would be ambiguous. Deduplicate the source by key, usually by latest timestamp, before merging.

What is the difference between a checkpoint location and a schema location in Auto Loader? The checkpoint tracks the stream's progress; the schema location stores the inferred and evolving source schema. Separate paths, both durable, neither shared.

How do you pin behaviour on serverless? By selecting an environment version, since the runtime is versionless and upgraded by Databricks.

Which LTS would you start a new project on today? DBR 18 LTS, the newest long-term-support release (Spark 4.1), supported for three years. Mention that Spark 4 enables ANSI mode by default, so cast and overflow behaviour is stricter than Spark 3.

What does mode: development do in a bundle target? It prefixes deployed resource names with [dev <username>] and pauses schedules, so experiments are isolated and do not run on their own. production mode requires an explicit identity to run as and a non-user workspace path.

How do you test PySpark code? Keep transformations as functions from DataFrame to DataFrame, unit test them with small DataFrames, and run a few integration tests through Databricks Connect against a dev catalog in CI.

What is wrong with dbutils.notebook.run for orchestration? No per-step retries, status, repair or dependency graph. Use job tasks.

How does Databricks Connect misbehave? Installing it next to pyspark, or with a client version that does not match the runtime. Use a clean environment and match versions.

Where would you find who changed a grant? The system.access.audit system table.

What a strong answer covers that a weak one misses

Strong candidates share habits you can adopt.

They state the mechanism, not only the remedy. For a conflict error they explain optimistic concurrency before they propose a retry. For a slow job they explain which metric points at which cause.

They investigate in a stated order and say what evidence would change their mind. A weak answer lists every possible tuning knob.

They mention cost and operations unprompted: tags, auto-termination, alerts, who gets paged. A weak answer treats a pipeline as finished when it runs once.

They separate people from automation: production runs as a service principal, grants go to groups, tokens are not shared.

They know what is deprecated and say so: stages in the workspace registry, mounts, partition-plus-ZORDER on new tables, tokens for CI, the old Asset Bundles name, Delta Live Tables naming. Naming the replacement shows currency.

They admit limits. "I would check the current documentation for the exact setting" beats confidently inventing a configuration key. Several Databricks features are in beta or recently renamed, and an honest "that was in beta when I last checked" is credible.

They connect adjacent tools when relevant: Terraform for platform administration (Terraform), MLflow concepts for tracking (MLflow), Spark internals for performance (Spark).

Design prompts in miniature

"Design a daily ingestion pipeline for files from a partner." Land files in a volume, Auto Loader with availableNow into bronze, a keyed merge into silver, checks that fail loudly, a job with failure alerts and timeouts, deployed by a bundle, running as a service principal. Mention schema evolution policy, and what happens on a bad file.

"How would you give analysts access to sensitive customer data?" Grant SELECT on a gold schema to an analyst group, use row filters and column masks on the sensitive columns inside the table rather than building copies, restrict the catalog to the right workspaces, and audit access through the audit system table. Mention data residency if the employer has such a requirement.

"The team has fifty notebooks and no tests. Where do you start?" Pick the pipeline that matters most, extract transformations into functions, add unit tests for those, move deployment to a bundle, then expand. Do not rewrite everything.

Sixty-second self-test

Answer aloud without looking up.

1. What are the two kinds of thing in a Delta table directory, and what does VACUUM remove?
2. Name the three privileges a user needs to select from a table.
3. Why does a stream reprocess everything after a deploy?
4. What is the first number you read in a slow query's profile?
5. What does mode: production require in a bundle?
6. Which credential type do you use in CI, and what replaces a stored secret where supported?
7. How do you promote a model without changing its consumers?
8. Where do you look when the cost report doubles?

If any answer was a guess, reread the matching section of the detailed guide.

Part two of three. At this level your notebooks run and your jobs are scheduled, so the problems change character. They are no longer "why won't this start" but "why did this cost four times what we expected", "why does it work in dev and fail in prod", and "why did the deploy break when nothing changed". Start with the error table, then the practices in order of leverage, then practice cards and operational recipes.

Common errors at this level

Cumulative: beginner errors still apply. Error text below is representative; exact wording varies by runtime, so search the error class name in the Databricks error-messages reference when yours differs.

The practices that pay off most

Ordered by leverage: the first few change your cost and reliability the most for the least effort.

Practice 1: Tag, then measure

Cost control is a measurement problem before it is an engineering one. Make the team (or project) tag mandatory through cluster policies and budget policies, then query system.billing.usage grouped by that tag weekly. Costs concentrate in a handful of jobs, so reviewing the top ten each month is worth more than any single optimisation.

Practice 2: Idempotent by construction

For each task, ask: if this runs twice, is the result still correct? If it appends, make it a keyed MERGE or a replaceWhere for the slice it owns. If it is a stream, make sure its checkpoint is stable. Only then add retries, and keep the count small with a delay, because a retry storm against a failing dependency costs money and hides the real error.

Practice 3: Separate code from configuration

Notebooks should receive the catalog, schema and paths as parameters. A notebook that hard-codes course_prod cannot be tested in dev without risk. With bundles, per-target variables supply those values, so the same code runs in every environment.

Practice 4: Test the logic where it is cheap

Pure DataFrame functions can be tested in seconds with tiny inputs. A few integration tests against a dev catalog confirm the wiring. Write the first test the day you extract the first function; the habit matters more than the coverage number.

Practice 5: Guard the schema

A write that fails on schema mismatch is doing its job. Decide who may change the schema and how: additive changes through mergeSchema on the write, anything destructive through a reviewed migration. Add NOT NULL and CHECK constraints at the tables others depend on, so bad data stops at the writer instead of in a dashboard.

Practice 6: Know what is legacy

Stages in the workspace model registry, mounts, the DBFS root, personal access tokens in CI, dbutils.notebook.run chains and partition-plus-ZORDER on new tables all still work and all age badly. When you touch code that uses one, replace it with its current form (Unity Catalog aliases, volumes, OAuth, job tasks, liquid clustering).

Habits

Print who you are before you debug. databricks current-user me and databricks auth profiles take five seconds and settle half of all "it works on my machine" questions about workspaces and identities.

Validate before you deploy. databricks bundle validate -t <target> is cheap and catches most reference and schema mistakes before they reach a workspace.

Pin things that change under you. The CLI version in CI, the runtime on classic jobs, the environment version on serverless, the model version behind an alias. Upgrade each one on purpose, and tell the team.

Keep notebooks thin. The notebook is an entry point that parses parameters and calls tested functions in a package.

Look at DESCRIBE HISTORY first. When a table misbehaves, the log tells you who wrote what and how much. It is the black-box recorder.

Read the monthly release notes. Deprecations come with dates. Ten minutes a month beats an outage.

Operational recipes

Recipe: find out why a table is slow to query

DESCRIBE DETAIL course_prod.silver.events;

Look at numFiles and sizeInBytes. Average file size in the low megabytes on a large table means too many small files. Run OPTIMIZE, or confirm predictive optimization is enabled for the table. Then check the clustering columns against the filters your queries actually use. If the filter column is not a clustering key, files cannot be skipped.

Recipe: roll back a bad write

DESCRIBE HISTORY course_prod.silver.users LIMIT 10;
RESTORE TABLE course_prod.silver.users TO VERSION AS OF 41;

Pick the version before the bad commit, restore, and check the row count against what you expect. This works only while the old files exist, so it is limited by the VACUUM retention. If the table feeds a stream, think about what the restore does to downstream checkpoints before you run it.

Recipe: audit who can read a schema

SHOW GRANTS ON SCHEMA course_prod.gold;
SHOW GRANTS ON CATALOG course_prod;

Run both. Privileges inherit downward, so the catalog's grants matter as well. Look for individuals and for broad groups with more than SELECT.

Recipe: authenticate CI without a stored password

For GitHub Actions or another provider that supports it, use OAuth token federation so the pipeline exchanges its own identity for a Databricks token. Check the current Databricks documentation for your provider's steps. Where federation is unavailable, store a service principal's client ID and secret in the pipeline secret store as DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, give that principal only the privileges the deploy needs, and rotate the secret on a schedule.

Recipe: a safe pull-request pipeline

pip install -r requirements-dev.txt
pytest tests/
databricks bundle validate -t dev

On merge to main, a second job deploys to production as the service principal:

databricks bundle deploy -t prod

Pin the CLI version in both jobs so that they agree with what developers run locally.

Recipe: attribute spend to a team

SELECT custom_tags['team'] AS team, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY dbus DESC;

A large row with a null team is the list of things to bring under policy. Remember the billing table can lag, so look at completed days.

Recipe: promote a model

from mlflow import MlflowClient

client = MlflowClient()
client.copy_model_version(
    src_model_uri="models:/course_dev.models.churn/5",
    dst_name="course_prod.models.churn",
)
client.set_registered_model_alias("course_prod.models.churn", "champion", 1)

Run this from a pipeline that has first evaluated the candidate against the current champion. To roll back, point the alias at the previous version. Consumers load by alias and notice nothing.

Recipe: upgrade a runtime safely

1. Read the release notes for the target runtime, and the Spark migration guide for each Spark version you cross.
2. Create a dev copy of the job on the new runtime and run your tests.
3. Compare outputs with the old runtime on the same input, looking for cast, null and ordering differences.
4. Promote job by job, starting with the least critical.

Practice cards

Work through these on a real project. Each takes ten to thirty minutes.

Card: the grant audit. List every grant on your production schemas. Replace each individual grant with a group grant. Record who owns each production object, and move ownership to a group or service principal where you can.

Card: the cluster sweep. List clusters with databricks clusters list. For each all-purpose cluster, record size, auto-termination and the last time anyone used it. Delete what is unused and set a termination time on the rest.

Card: the idempotency review. Pick your most important job. Read each task and write down what happens if it runs twice. Fix the first one that duplicates data.

Card: the first bundle. Put one existing job into a bundle with databricks bundle generate job --existing-job-id <id>. Add a dev and a prod target, deploy dev, and compare to the UI-built original.

Card: the profile read. Take one slow query. Open the profile, write down bytes read, shuffle size and the slowest stage, and make one change. Record before and after.

Card: the token hunt. Search for DATABRICKS_TOKEN and dapi in your repositories and CI configuration. List every place a personal token is used and plan its replacement with a service principal.

Card: the model alias. Register a model in Unity Catalog, set a champion alias, load by alias in a second notebook, and move the alias to a new version. Confirm the loading code did not change.

Card: the connect loop. Set up Databricks Connect in a clean environment, extract one function from a notebook and run its unit test from your laptop.

Pitfalls that look fine until they don't

A green job that processed nothing. An incremental task with an empty input finishes successfully. Add a check for expected volume, or alert when a normally busy table receives no new rows.

A cheap query that scans everything. On a small dev table nothing looks wrong. The same query on production data reads a hundred times more. Look at bytes read on a realistic sample, not at duration on a toy.

A schedule that overlaps itself. If a job takes longer than its interval, runs pile up. Set the maximum concurrent runs to one for jobs that must not overlap, and alert on duration.

A dev bundle that touches prod data. Dev targets must point at dev catalogs. Use per-target variables and review them in code review; a catalog name hard-coded in a notebook defeats the whole structure.

A copied notebook that became the production system. If a notebook lives only in a personal folder, it has no history, no review and an owner who may leave. Anything scheduled belongs in Git and in a bundle.

A secret in a notebook cell. Even once deleted, it remains in notebook history and in any exported copy. Use secret scopes, and rotate anything that was exposed.

How to read an unfamiliar failure

Most errors at this level fall into one of four families, and naming the family first saves time. Identity errors (credentials, privileges, wrong workspace) are answered by asking who you are authenticated as and what that identity may do; check databricks current-user me, then SHOW GRANTS. Concurrency errors (conflicts, missing files after cleanup) are answered by asking which other process touched the same data and when; DESCRIBE HISTORY gives the timeline. Capacity errors (quota, out of memory, slow stages) are answered by the profile: what was read, shuffled and spilled. Compatibility errors (serverless restrictions, Connect version mismatch, ANSI casts after an upgrade) are answered by asking what changed in the runtime, client or environment version since it last worked.

Write down the family before you change anything. It stops the common failure of tuning cluster size to fix what is really a permission problem, and it gives a colleague a clear starting point when you hand the problem over.

A short checklist

Before you call a pipeline production-ready:

• Lives in Git and deploys through a bundle, with dev and prod targets.
• Runs as a service principal; humans have no write access to production data.
• Grants are to groups; owners are groups or service principals.
• Every task is idempotent, and retries are limited to safe ones.
• Failure alerts go to a monitored channel, with timeouts set.
• Compute is tagged and governed by a policy; scheduled work is not on an all-purpose cluster.
• Tables are managed, clustered and have constraints on the keys.
• Runtime or environment version is pinned, and the CLI version is pinned in CI.
• Tests exist for the transformations and run on every pull request.
• Someone owns the monthly cost and release-notes review.

For related tools, see MLflow for tracking concepts, Terraform for platform administration, and Spark for engine internals.

This is part three of three. The earlier levels taught you to build on Databricks and to use it well. This one is about owning it: the workspaces, the catalogs, the identities, the bill and the upgrade calendar that other teams depend on without thinking about them.

The questions change. They stop being "how do I make this job run" and become "what happens to forty teams when this component fails", "who can read this table and how would I prove it", "why did the bill double in March" and "how do we move to the next runtime without a quarter of broken pipelines". Everything below is organised around those questions, against Databricks Runtime 18 LTS (Spark 4.1) as the default runtime and Databricks CLI 1.19 as the tooling baseline.

Where this picks up

I will start with the architecture, because every failure mode you will be paged for sits on one side of a boundary that the architecture draws.

Architecture and failure modes

Databricks splits into a control plane and a compute plane. The control plane is hosted by Databricks and runs the web application, the REST APIs, the job scheduler, the notebook service and the metadata for Unity Catalog. The compute plane is where data is processed. In classic compute it lives in your cloud account, in a network you control, and the data sits in your object storage as Delta files. In serverless compute it lives in Databricks' own account, in an environment Databricks sizes and upgrades for you.

Learn the boundary by learning what breaks across it. If the control plane has an incident, running streaming queries and already-started clusters usually carry on, but nobody can start a new cluster, launch a job run, or log in. If your cloud provider runs out of a VM type in a zone, the control plane is perfectly healthy and every classic job that needs that instance type fails at launch. If your storage permissions are changed by an infrastructure team, queries fail with permission errors even though nothing in Databricks changed. Three different owners, three different symptoms, and most of the first ten minutes of an incident is deciding which one you are in.

The hierarchy above the workspace matters just as much. One account holds many workspaces, users, groups, service principals, metastores and billing. A metastore is regional, and every workspace in that region attaches to it, which is why governance in Databricks is an account-level discipline and not a per-workspace one. A workspace is a deployment unit: it has a URL, its own compute and jobs, and its own network configuration. A catalog is a governance unit. Keeping those two ideas apart, workspace for where compute runs and catalog for who may see data, is the single most useful habit at this level.

Region deserves a sentence of its own for readers whose employers are in the Gulf or Egypt. Both the cloud provider and the region you choose determine where control plane metadata and your data live, and feature availability can differ between regions and between clouds. Confirm both against the official documentation for your provider before you commit a residency promise to a regulator or a customer. Azure Databricks has its own documentation and uses Entra ID for identity, and some features arrive on one cloud before another.

Unity Catalog as an operating model

At the mid level you used grants. At the platform level the question is which objects exist and who owns them. Unity Catalog organises everything as metastore, catalog, schema and then tables, views, volumes, functions and models, always addressed as catalog.schema.object. A layout is a decision you will live with for years, because moving a table between catalogs changes every query that names it.

A durable layout separates by environment first and by domain second. A catalog per environment per domain, such as payments_dev and payments_prod, lets you bind a catalog to specific workspaces, so production data is not even visible from a development workspace. Inside a catalog, schemas map to the medallion layers or to data products. Ownership belongs to groups, never individuals: an object owned by a person who leaves becomes an administrative puzzle, and an object owned by a group synchronised from your identity provider does not.

CREATE CATALOG IF NOT EXISTS payments_prod;
GRANT USE CATALOG ON CATALOG payments_prod TO `payments-readers`;
GRANT USE SCHEMA, SELECT ON SCHEMA payments_prod.gold TO `payments-readers`;
GRANT CREATE TABLE ON SCHEMA payments_prod.silver TO `payments-engineers`;
SHOW GRANTS ON SCHEMA payments_prod.gold;

Grants inherit downwards, so a grant on a schema covers every present and future table in it. That is convenient and dangerous in equal measure: a SELECT granted at catalog level quietly exposes a table created next month. Prefer granting at the schema level to a group, and reserve catalog-level grants for platform roles. The permissions you need for any read are three layered ones, USE CATALOG, USE SCHEMA and SELECT, and a missing one produces an INSUFFICIENT_PERMISSIONS error or, for objects the caller cannot see at all, a TABLE_OR_VIEW_NOT_FOUND. Support teams spend a great deal of time on that second message, because it looks like a typo and is really a privilege problem.

Managed and external storage is the other early decision. Managed tables and volumes let Unity Catalog control the storage location and the lifecycle, which enables features like predictive optimization and automatic clean-up. External tables govern a path you own and keep, which suits data that other systems also write. Defaulting to managed and requiring a written reason for external is a reasonable platform rule. External locations and storage credentials are the metastore-level objects that make external access possible, and only a small group should be able to create them.

Fine-grained control comes through row filters and column masks, which attach a function to a table so the same table returns different rows or masked values to different groups. Attribute-based access control policies, which assign rules by tags rather than object by object, are in beta on views and at metastore level as of September 2026, so evaluate them but do not yet build a compliance claim on them. Everything you apply must be reproducible, so manage catalogs, schemas and grants through code (the Terraform provider or bundles) and treat manual grants in the UI as drift.

Workspace topology and environments

How many workspaces you run is a trade-off between isolation and overhead. A workspace per environment (dev, staging, prod) is the common baseline, because production compute, jobs and network rules should not share a blast radius with experiments. Some organisations add a workspace per business unit when separate networking, separate administrators or separate compliance scope is required. Each additional workspace multiplies what you must configure: network, identity, cluster policies, secrets and jobs.

Because the metastore is shared across a region, the catalog is your isolation tool within it. Bind production catalogs to production workspaces so that even an administrator in the development workspace cannot query production data from there. Verify the exact binding controls in the current Unity Catalog documentation, since the console wording changes between releases. The point to carry is the principle: isolation by workspace and by catalog together is stronger than either alone.

Identity should be provisioned, not typed. Synchronise users and groups from your identity provider (Entra ID or Okta) into the account using SCIM, assign groups to workspaces, and grant privileges to those groups. When an engineer changes teams, one change in the identity provider moves their access everywhere. Service principals are the equivalent for automation and belong to the account too; give each pipeline or each team its own so that audit entries identify the automation that acted.

A small number of workspace-level settings repay the effort of managing them as code: cluster policies, IP access lists, workspace-level configuration and the secret scopes the platform itself needs. The Terraform provider for Databricks covers account and workspace administration including catalogs, grants, cluster policies and SCIM groups, and it is the right tool for the slow-moving platform layer. Declarative Automation Bundles, covered below, are the right tool for the fast-moving project layer. Keeping those two layers apart avoids the classic conflict where a platform team and a product team both want to own the same resource.

Compute strategy and scaling: what breaks first

Databricks compute scales until something external stops it, so the useful skill is predicting what stops it. In rough order of how often it bites: cloud quota and capacity, then driver memory, then skew, then metadata and small-file overhead, then concurrency conflicts on shared tables.

Cloud quota and capacity come first on classic compute. A job cluster that cannot launch reports a termination reason such as Cloud Provider Launch Failure, INSUFFICIENT_INSTANCE_CAPACITY or a quota error, and the cause is a vCPU quota or an instance type that is unavailable in the zone. Request quota increases ahead of launches, allow several instance types in policy so a job can fall back, and use spot instances with on-demand fallback for cost. Serverless compute removes this class of failure from your account, which is its strongest argument beyond convenience.

Driver memory and skew are the code-level limits. collect() on large data, a broadcast of something that is not small, and one hot key in a join produce messages like Driver is up but is not responsive, likely due to GC or executor OutOfMemoryError. Adaptive Query Execution is on by default and rebalances many skewed joins, so the remaining cases are usually genuine data problems: a null key shared by millions of rows, or a table that should have been filtered earlier. The remedy is rarely a bigger cluster.

Compute type is a policy decision, not a personal one. Use job clusters, or serverless jobs, for scheduled work, since they exist only for the run and cost less than all-purpose clusters. Keep all-purpose clusters for interactive work and set auto-termination. SQL warehouses are sized in T-shirt sizes with a minimum and maximum cluster count and auto-stop, and serverless warehouses start in seconds. Photon, the native vectorised engine, accelerates SQL and DataFrame work but is billed at a higher DBU rate and does little for code dominated by Python UDFs, so enable it where you can show a benefit.

Cluster policies are your control surface. A policy restricts node types, maximum size, runtime versions and mandatory tags, and assigning teams to policies is how you prevent a single mis-sized cluster from consuming a quarter's budget. Serverless has its own limits you should publish to users: no RDD APIs, no direct sparkContext use, restricted custom JARs, libraries supplied through environment configuration, and a narrower set of allowed Spark configurations. A job that depends on any of them fails with an unsupported-feature error and needs classic compute.

Security: the trust model, identities and secrets

Start a security review by drawing who can act as whom. A person authenticates to a workspace through your identity provider. A job runs as an identity, either its creator or a service principal named in run_as. A cluster reads storage through a credential that Unity Catalog vends according to the grants of whoever is running the query. Every arrow in that picture is a place where privilege can be borrowed, so the review is a hunt for arrows that are broader than they need to be.

Automation should authenticate with OAuth machine-to-machine credentials: a service principal with a client ID and secret, supplied through the unified authentication environment variables DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, or a profile in ~/.databrickscfg. Personal access tokens still work, but the documentation recommends OAuth, and a PAT in a CI system is a long-lived bearer credential tied to a human who will eventually leave. For GitHub Actions and similar systems, prefer OAuth token federation so the pipeline exchanges a short-lived identity token for access instead of storing a secret. Follow the official authentication page for the current setup steps, and test that no long-lived secret remains in the CI configuration.

Resolution order matters for incident forensics: bundle settings, then environment variables, then profiles. A pipeline that "mysteriously" acts as the wrong identity is nearly always an environment variable overriding a profile. On the person side, since CLI 1.0 the interactive OAuth tokens live in the operating system's secure store (Keychain on macOS, Credential Manager on Windows, Secret Service on Linux). A headless build machine without one fails, and the supported fallback is DATABRICKS_AUTH_STORAGE=plaintext. Use that only on throwaway machines, and prefer machine-to-machine credentials for anything that is not a person.

Secrets belong in secret scopes, read with dbutils.secrets.get(scope, key), which redacts the value in notebook output. Scopes can be Databricks-backed or, on Azure, Azure Key Vault-backed. Redaction is a convenience, not a control: a user who can read a secret can print it character by character, so limit who can read each scope. Unity Catalog external secrets backed by AWS Secrets Manager are in beta as of September 2026, so keep them out of your production baseline for now. A failing lookup reports Secret does not exist with scope: X and key: Y, which usually means the wrong workspace, since scopes are per workspace.

Network posture is where regulated employers spend their effort. The levers are private connectivity so the control plane is reached without the public internet, deploying classic compute into your own network with secure cluster connectivity so nodes have no public IPs, IP access lists on the workspace, and network policies governing what serverless compute may reach. The exact feature names and per-cloud availability vary, so verify them against the documentation for your provider and region. Encryption can use platform-managed keys or customer-managed keys for workspace storage and managed services, and the compliance security profile adds controls required by standards such as HIPAA and PCI. As of September 2026, Databricks Apps are enabled by default in workspaces with the compliance security profile, which is the kind of change worth reading release notes for.

The supply chain is smaller than in container platforms but real. Libraries installed on clusters come from public package indexes unless you point at an internal mirror, init scripts run as root on every node, and a notebook can pull code from anywhere. Keep init scripts in workspace or volume paths (init scripts on DBFS are a legacy pattern), pin library versions through your build, and use cluster policies to restrict which libraries and init scripts users may attach. Pin the CLI and the bundle files in version control, and review changes to them like application code.

Delivery with Declarative Automation Bundles

Bundles, until recently called Databricks Asset Bundles, are how you describe a project's jobs, pipelines, dashboards and serving endpoints in a databricks.yml file and deploy them across targets. Older articles and the bundle command group still use the previous name; the commands are unchanged. At platform level the question is how teams use them consistently and how a deployment stays safe.

A target is an environment: its workspace, its variables and its mode. Development mode prefixes resource names with [dev <user>] and pauses schedules, so two engineers do not collide and a test job does not run on a timer. Production mode expects explicit identity and a workspace root path that is not tied to one user, and the deployment should run as a service principal.

bundle:
  name: churn-pipeline
variables:
  catalog: {default: payments_dev}
resources:
  jobs:
    train:
      name: train-${bundle.target}
      tasks:
        - task_key: train
          notebook_task: {notebook_path: ./notebooks/train.py}
          environment_key: default
targets:
  dev:
    default: true
    mode: development
    workspace: {host: https://<dev>.cloud.databricks.com}
  prod:
    mode: production
    workspace:
      host: https://<prod>.cloud.databricks.com
      root_path: /Workspace/bundles/${bundle.name}/prod
    run_as: {service_principal_name: "<sp-application-id>"}
    variables: {catalog: payments_prod}

The commands you will standardise are databricks bundle validate, bundle deploy -t prod, bundle run -t prod train, bundle destroy -t dev, bundle generate job --existing-job-id <id> for turning a hand-built job into code, and bundle deployment bind for adopting an existing resource without recreating it. A pipeline for production should validate on every pull request, deploy to a staging target on merge, and promote to production only from the same commit.

Two engine facts have operational weight. Since CLI v1.14.0, a bundle that previously used Terraform state migrates automatically to the direct deployment engine after a successful deploy, and you can opt out with engine: terraform in the bundle configuration. And CLI v1.17.0 raised the direct-deployment state version to 3, so clients older than v1.8.0 refuse to work with a bundle deployed by it. The practical rule is that the CLI version is shared infrastructure: pin it in CI and in your developer tooling, upgrade everyone together, and announce it. A laptop on an old CLI will fail against a production bundle and the error will not mention the cause clearly. In the run_as setting, give a single identity as a mapping, as in the example above.

Table design and pipelines at scale

Storage costs little; reading badly is what costs. Delta tables are Parquet files plus a transaction log, and most large-scale problems are file layout problems wearing other symptoms. Too many small files make listing and planning slow. Too coarse a partitioning forces full scans. Concurrent writers collide in the log.

For new tables, prefer liquid clustering, declared with CLUSTER BY, over the older combination of partitioning and ZORDER. Liquid clustering lets you change clustering keys later without rewriting the table design, and it pairs with row-level concurrency to reduce write conflicts. Deletion vectors speed up updates and deletes by marking rows instead of rewriting files. Predictive optimization can run OPTIMIZE and VACUUM for managed tables automatically; confirm its default for your account before relying on it. VACUUM has a default retention of seven days, and time travel depends on files that vacuum has not yet removed, so shortening retention shortens how far back you can recover.

Concurrent writes still conflict. The errors, DELTA_CONCURRENT_APPEND, ConcurrentAppendException and ConcurrentDeleteReadException, mean two transactions touched overlapping data. The right responses are to narrow what each writer touches, to use liquid clustering with row-level concurrency, and to retry idempotently. Schema changes are the other classic: a write that does not match fails with a schema-mismatch error, and the fix is to enable mergeSchema or overwriteSchema deliberately, not blanket-allow it in shared jobs, because it lets an upstream change silently alter a table that dozens of consumers read.

Streaming ingestion with Auto Loader and trigger(availableNow=True) gives incremental batch: cheap, restartable and easy to reason about. Keep the checkpoint location with the table's lifecycle, because deleting a checkpoint replays or loses data depending on the source. For declarative pipelines, the product formerly called Delta Live Tables now sits under the Lakeflow name; confirm the current product name and decorator module in the documentation before standardising, since older code still uses import dlt. Expectations let a pipeline drop or fail on bad rows, and as of September 2026 deleting a Unity Catalog pipeline retains its tables by default, with cascade: true to remove them as well. That is a safer default, and a reason to audit scripts that assumed deletion cascaded.

Change data feed has a September 2026 change: automatic change data feed is computed at query time and is generally available, so you may not need to enable delta.enableChangeDataFeed up front for every table that downstream systems read incrementally. Test it against your own consumers before removing existing settings.

The ML and serving platform

A shared ML platform has three jobs: make models discoverable and governed, make promotion repeatable, and make serving cheap enough to be used. Unity Catalog models do the first. Models register under a three-level name such as payments_prod.models.churn, access is controlled by grants like any other object, and lineage connects a model to the data it was trained from. Use aliases such as champion to point at the version that should serve, and treat the older workspace registry and its stages as deprecated: transition_model_version_stage and models:/name/Production belong to the legacy path.

import mlflow
from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")
client = MlflowClient()
client.set_registered_model_alias("payments_prod.models.churn", "champion", 7)
model = mlflow.pyfunc.load_model("models:/payments_prod.models.churn@champion")

Promotion should be a pipeline, not a click. A common shape is training in the development catalog, evaluation against a held-out set with an explicit threshold, registration to the production catalog by a service principal, and an alias move only if the gate passes. MLflow 3 adds Logged Models and deployment jobs for exactly this kind of workflow; its log_model call is moving from artifact_path towards name, so check the current signature before copying older examples. The runtime ships a matching MLflow, DBR 18 LTS with MLflow 3.8.1, so version skew between a client in CI and the cluster is a real source of subtle differences.

Mosaic AI Model Serving exposes custom models, foundation models and external models as REST endpoints. Platform decisions are scale-to-zero (cheap, with cold-start latency), authentication (endpoints sit behind workspace identity), and rate limits and logging through the gateway layer. Unity Gateway, which routes to third-party model providers, gained API, Terraform, CLI and SDK support in September 2026, which makes it manageable as code. Model retirements are announced in release notes, with several third-party models retiring on 2026-10-30, so subscribe to those notes and keep a list of which endpoints depend on which provider models. Exact field names for endpoint configuration should be taken from the current SDK reference rather than from older posts.

Feature Engineering in Unity Catalog, Vector Search and Agent Bricks are further shared services. The same ownership questions recur for each: who may create one, how is it named, who pays and how is it retired. A platform that lets anyone create endpoints without tags and an expiry accumulates idle spend very quickly.

Multi-tenancy

Many teams will share your platform, and the design question is how much they share. There are three layers to choose from, and strong platforms use more than one.

At the data layer, isolate tenants by catalog or by schema, with grants to tenant groups. A catalog per team gives the cleanest separation and the simplest cost story; a schema per team inside a shared catalog gives less overhead and a looser boundary. For shared tables that hold several tenants' rows, row filters and column masks enforce the split inside the table, but they are a policy you must test, since a filter function with a bug is a leak with an audit trail.

At the compute layer, standard access mode (formerly called shared) provides user isolation on shared compute, which is what makes serverless and shared clusters safe for multiple people. Dedicated access mode (formerly single user) assigns compute to one identity for workloads that need it. The console may show either name during the transition, so use the mode names your current console shows. Cluster policies and budget policies bound what each team can spend and tag everything it launches.

At the workspace layer, a separate workspace gives separate networking, separate administrators and separate limits, at the price of more to operate. Reach for it when a tenant has a different compliance scope or must not share a control surface, not as a default.

Sharing outward uses Delta Sharing, now surfaced as OpenSharing in the latest notes, with September 2026 support for sharing foreign Delta, Iceberg and schema-level objects. Treat a share as a published interface: version it, document it and restrict who can create one.

Cost: attribution, control and anomalies

The bill has two parts on classic compute, DBUs multiplied by the SKU rate plus the cloud provider's charge for the machines, and one on serverless, where the infrastructure is folded into the DBU rate. Two consequences follow. Cloud-side cost tools will not see serverless spend, and Databricks-side tools will not see your cloud storage and network costs. You need both, reconciled.

Attribution comes first. Without it you cannot tell a team that it spent too much. Require tags on compute and jobs through cluster policies, use budget policies to attach serverless usage to a team, and, since September 2026, use query tags on SQL warehouses to attribute queries. Then query system.billing.usage, which carries a usage_metadata structure naming the job, cluster or warehouse behind a record. Billing tables can lag by up to 24 hours on serverless, so a daily report is reliable and an hourly one is not.

SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= date_sub(current_date(), 30)
GROUP BY usage_date, sku_name
ORDER BY usage_date;

Control comes from a short list of levers, in the order they usually pay off: job clusters or serverless in place of all-purpose clusters, auto-termination everywhere, policy limits on maximum size, spot instances with fallback for tolerant jobs, auto-stop on warehouses, and avoiding repeated full scans through good table layout and predictive optimization. Photon is a cost and speed trade rather than a free upgrade, because it carries a higher DBU rate.

Anomaly detection is a scheduled query comparing today's spend per tag with a trailing average, with an alert to the owning team. An alert that goes to a shared mailbox is ignored; send it to the team whose tag it is. Show-back (reporting) comes before charge-back (billing), and trust in the numbers is what makes either work, so publish your attribution method and its known gaps, for example untagged spend and the serverless lag.

Observability and system tables

Databricks gives you more telemetry about itself than most platforms, and most of it is in tables you can query. The system tables expose billing usage, audit events, query history (account-wide as of September 2026), compute activity and lineage. A platform team builds dashboards and alerts on these instead of scraping the UI.

Four operational questions deserve standing queries. Which jobs are failing or slowing down, from job run history and job health rules that alert when a duration crosses a threshold. Which queries are expensive, from query history, with Query Profile for the single query you must understand. Who accessed sensitive data, from the audit table. What depends on a table you are about to change, from lineage. Answering the last one before a schema change is the cheapest incident prevention available.

Alerts go to the people who can act on them: failure notifications on jobs by email, Slack or webhook, health rules for duration, and your own scheduled queries over system tables for the account-level signals. For exporting to external tools such as Datadog, Grafana or Splunk, use cluster log delivery to storage and the APIs, and verify the integration path with the tool's own documentation. Databricks Apps now emit OpenTelemetry telemetry (generally available in September 2026), which lets application-level traces join the rest of your observability stack. For GenAI applications, MLflow Tracing and evaluation give you request-level visibility.

Upgrades and migrations

Upgrades on Databricks are several independent calendars. The runtime, the CLI and SDKs, Databricks Connect, serverless environments and the platform itself each move at their own pace.

For runtimes, use long-term support versions. DBR 18 LTS (Spark 4.1) was released on 2026-06-10 and is supported until 2029-06-10; DBR 17.3 LTS runs to 2028-10-22; 16.4 LTS to 2028-05-09; 15.4 LTS to 2027-08-19; and 14.3 LTS to 2027-02-01. Each is supported for three years. DBR 19, on Spark 4.2, is not an LTS, so keep it for experiments and do not standardise on it. The programme is simple to state: know which runtime each production job uses, schedule moves well before end of support, and test on the next LTS ahead of time.

The hard part is the Spark major version. Spark 4 turns ANSI SQL mode on by default, so casts and arithmetic that silently returned null now raise errors; that is a correctness improvement and an upgrade hazard in equal measure. Spark 4 is Scala 2.13 only and needs a recent Java, so custom JARs built for Scala 2.12 must be rebuilt. Read the Spark migration guide for each version and the Databricks runtime notes before you start. Run the new runtime in staging against production-shaped data, compare outputs and not just success, and move jobs in waves. Pin the runtime in each classic job so a default does not change under you.

Serverless is versionless: Databricks upgrades the runtime for everyone, and behaviour changes are communicated through environment versions that you pin in jobs and notebooks. That removes the migration project and replaces it with a requirement to read release notes. Databricks Connect must match the cluster runtime version, and installing pyspark beside it breaks things, so keep it in a clean virtual environment. The CLI is the one you control directly: upgrade the whole team together, because of the state-version change in v1.17.0 described above.

Legacy retirements that are your job to track include the workspace model registry and stages, personal access tokens, mounts and DBFS root usage, Hive metastore tables, and the older pipeline naming. Keep a migration register with owner, deadline and status for each, and review it monthly.

Governance and incident playbooks

Governance is the set of rules you can demonstrate, not merely the set you hold. For Databricks the demonstrable rules are: every object has an owning group; every production change arrives through a bundle or Terraform plan; every grant goes to a group; every cost has a tag; every secret has a scope with a short reader list. The audit table lets you test each one, which is what turns a policy into a control.

Playbooks should be short and ordered by what you check first. These four cover most pages.

A job fails at launch. Read the termination reason. Capacity or quota means your cloud provider: try another instance type or zone, and request quota. A permission or network error means the infrastructure team's changes. If the error names the workspace itself, check the Databricks status page and the release notes before touching anything.

A user says a table vanished. TABLE_OR_VIEW_NOT_FOUND is a privilege problem until proven otherwise. Check the three layers, USE CATALOG, USE SCHEMA, SELECT, with SHOW GRANTS, then the audit table for a drop. A dropped managed table may be recoverable depending on retention; check the documentation before you promise anything.

A bundle deploy fails across the team. Compare CLI versions first. After the v1.17.0 state bump, an old client cannot work with a newer state. Then check authentication precedence, since an environment variable may override the intended profile.

The bill spiked. Group system.billing.usage by day and tag to find the day and owner, then drill down by usage_metadata to the job or warehouse. The usual culprits are an all-purpose cluster left running, a job in a retry loop, or a warehouse with auto-stop disabled. Fix the cause, then add the policy that would have prevented it.

Where Databricks stops

Honest platform owners know the edges. Databricks is strongest for large-scale data engineering, analytics and ML on a shared governed foundation. It is a poor fit in several cases.

For small data, a platform with a control plane, catalogs and a billing model has real overhead. If the whole workload fits on one machine, a managed Postgres or a warehouse elsewhere is simpler and cheaper. For latency-critical transactional serving, Databricks offers managed Postgres in Lakebase and model serving endpoints, but a dedicated serving stack such as the ones covered in the KServe or BentoML guides may suit strict latency and deployment-control needs better. For heavy custom Kubernetes-native workflows, platforms built on Kubernetes give you control that a managed service deliberately withholds.

Workflows that cross many non-Databricks systems may be better orchestrated by a tool like Airflow or Dagster, with Databricks as one task type. Lakeflow Jobs is strong inside the platform and less so across it. Teams that want open, portable table formats can use Delta Lake and Spark themselves, trading the managed experience for control. Where lock-in matters, the mitigations are open formats, Unity Catalog's interoperability features and keeping business logic in plain code, so that moving is a project and not a rewrite. Experiment tracking and registry needs alone may be served by MLflow without the rest of the platform.

Running Databricks as a platform for other teams

A platform is a product with users, and your users are engineers who would rather not think about you. Your job is to make the right path the easy path and to make deviations visible.

Start with a paved road: a bundle template with environments, service principal, tags and policy included; a base set of cluster policies; a catalog-naming standard; a CI workflow that validates, deploys and gates promotion; and a short onboarding guide that gets a new team to a first deployed job in a day. Measure it by time to first production deployment and by the share of resources created outside the road.

Second, give teams self-service within limits. They should be able to create schemas, jobs and endpoints in their own catalogs without a ticket, and they should hit a policy ceiling, not a human, when they exceed their budget. Reserve tickets for what is genuinely cross-cutting: new catalogs, storage credentials, network changes.

Third, communicate change. Subscribe to the platform and runtime release notes, publish a monthly summary of what affects your users (for instance the September 2026 retirements and deprecations), and give deprecations a date and a migration path. Keep a support rota and a channel, and feed the recurring questions back into templates and documentation.

Fourth, keep yourselves honest with a small set of platform metrics: spend per team and its trend, share of untagged spend, failed-run rate by termination reason, runtimes within a year of end of support, long-lived credentials remaining, and median time to onboard. Review them monthly with the teams, not just among yourselves.

The review checklist

Use these in design reviews and pull requests for anything that will run on the shared platform.

• Does every new object live in a catalog and schema that match the naming standard, with an owning group and not an individual?
• Are grants made to groups at schema level, with no broad catalog-level grants added?
• Is the job deployed through a bundle, with a production run_as service principal and no personal token?
• Is compute a job cluster or serverless, under a cluster policy with a size cap, auto-termination and a cost tag?
• Is the runtime an LTS and pinned, or the serverless environment version pinned?
• Is the new table clustered with CLUSTER BY where appropriate, and does the write path tolerate concurrent writers and schema change deliberately?
• Do models register in Unity Catalog with an alias, with an evaluation gate before the alias moves?
• Does any serving endpoint have an owner, tags, and a retirement date?
• Are secrets read from a scope with a short reader list, and is nothing hard-coded?
• Is there an alert for failure and for duration, going to the owning team?
• Does the change depend on a beta feature, and if so is that acknowledged in writing?
• Is the CLI version pinned in CI, and does the change need a team-wide upgrade?

The complete picture

Databricks is a control plane and a compute plane over your own storage. Unity Catalog governs everything on top of it through one three-level naming scheme and one grant model, bound to workspaces and owned by groups. Compute is chosen by policy: serverless and job clusters by default, quotas planned, sizes capped. Delivery is a bundle deployed by a service principal, with the CLI version treated as shared infrastructure. Delta tables are designed for how they are read, pipelines are idempotent, and models move by alias through a gated promotion. Cost is attributed by tag and watched through system tables, observability is queries and alerts over those same tables, and upgrades are a calendar of runtimes, SDKs and legacy retirements that someone owns. Around all of it sit the playbooks, the metrics and the paved road that let other teams work without needing you for every step.

Where the series leaves you

Beginner taught the three-level name, the notebook and the managed table. Mid taught grants, Auto Loader, MLflow in Unity Catalog and bundles for one project. This level asked you to hold the platform together: the boundaries that fail independently, the trust model, the cost model and the upgrade calendar.

The natural next steps are practical. Put the review checklist into your pull request template. Write the four playbooks as runbooks and rehearse one. Build the spend-by-tag query and send it to its owners. Then widen outwards: Terraform for the platform layer, GitHub Actions for the delivery path, MLflow and Delta Lake for the layers beneath, and Spark for the engine that runs it all. Verify every version and feature named here against the current release notes before you act on it, since the platform changes monthly.

Sources

• Databricks documentation home
• Platform release notes
• Databricks Runtime release notes
• Supported runtimes and end-of-support dates
• September 2026 release notes
• Databricks CLI authentication
• Databricks CLI releases
• Declarative Automation Bundles
• Bundle settings
• Unity Catalog
• MLflow on Databricks
• Serverless compute
• Databricks Connect for Python

Senior interviews about Databricks rarely ask what a command does. They hand you a messy situation and watch how you structure it: what you ask first, what you refuse to assume, which trade-offs you name and how you would know you were wrong. This file gives you the prompts that come up most, with the reasoning a strong answer contains. Versions referenced are Databricks Runtime 18 LTS (Spark 4.1) and CLI 1.19.

The sixty-second platform pitch

Have a short answer ready for "how would you describe running Databricks for a company". A good one: "Databricks is a control plane that Databricks hosts and a compute plane that runs either in our cloud account or in theirs, over data that stays in our object storage as Delta. I treat Unity Catalog as the governance spine: catalogs per environment and domain, grants to groups from the identity provider, and workspaces bound to the catalogs they should see. Delivery is bundles deployed by service principals, compute is chosen by policy, cost is attributed by tag and watched in system tables, and I run runtime and CLI upgrades as a calendar."

That answer works because it names the planes, the governance model, the delivery path, the cost model and the upgrade discipline in one breath. A weak answer lists products. Practise yours aloud until it takes under a minute.

System design prompts

Design a multi-team platform for twelve teams on one cloud

Start with questions, because the right design depends on the answers. How many regions, and are there residency requirements? Which regulators and which data classifications? How do teams currently authenticate? What is the cost accountability model? A candidate who starts drawing before asking has missed the first test.

Then propose the skeleton. One account, with workspaces per environment (dev, staging, prod) in the region the data must stay in. One metastore per region. Catalogs per team per environment, with production catalogs bound to the production workspace only. Groups synchronised from the identity provider through SCIM and used for every grant. Service principals per team for jobs and CI. Cluster policies per tier (interactive, scheduled, large-batch) with size caps, auto-termination and mandatory cost tags. Bundles with a shared template for delivery. System tables for cost and audit.

Then say what you would defer and why. You would not build twelve workspaces; you would start with the three environments and add a workspace only for a team with a different compliance scope. You would not enable every beta feature; attribute-based access control is in beta on views and at metastore level as of September 2026, so you would pilot it rather than depend on it.

Strong answers also describe the failure modes of their own design. The shared metastore means a bad grant has regional reach, so metastore-level objects are restricted and changed by review. Capacity and quota are the cloud provider's, so you request quota ahead. Strong answers close by saying how they would measure success: time to first production deployment for a new team, share of untagged spend, and failed-run rate.

Design CI/CD for data and ML projects

A good answer separates the platform layer from the project layer. Terraform manages accounts, catalogs, grants, cluster policies and groups. Bundles manage each project's jobs, pipelines and endpoints. Pull requests run databricks bundle validate and unit tests. A merge deploys to a staging target. Promotion to production deploys the same commit with bundle deploy -t prod, and production runs as a service principal using run_as.

Be ready for the identity question: how does CI authenticate? The answer is OAuth machine-to-machine with a service principal, or token federation so no long-lived secret is stored. A personal access token in CI is the answer that loses points. Be ready for the CLI question too: why pin the CLI? Because v1.17.0 raised the direct-deployment state version to 3 and clients older than v1.8.0 reject it, so a laptop on an old CLI can fail against a production bundle. Pin it in CI and upgrade everyone together.

Mention what you test. Unit tests for transformation logic run without a cluster, and Databricks Connect or a staging target covers integration. Compare outputs between staging and production-shaped data and do not only check that the job succeeded. Mention rollback: bundles are declarative, so rolling back is redeploying the previous commit, and data changes need table history or a restore plan, which is why you shorten VACUUM retention carefully.

Design the promotion path for a model from training to serving

Training in the development catalog logs to MLflow. The model registers in Unity Catalog under catalog.schema.model. An evaluation job runs against a held-out dataset and compares against the current champion with an explicit threshold. Only if the gate passes does a service principal register to the production catalog and move the champion alias. Serving reads models:/catalog.schema.model@champion, and the endpoint is defined in code.

Name what is deprecated: workspace registry stages and transition_model_version_stage. Name the operational concerns: who owns the endpoint, scale-to-zero cold starts versus always-on cost, rate limits and logging at the gateway, and model retirements announced by providers, for instance several third-party models scheduled for retirement on 2026-10-30. A strong answer adds how rollback works: move the alias back to the previous version, which is instant and auditable.

Design cost attribution for a shared platform

Say: tags first, then budget policies, then system tables. Cluster policies make tags mandatory on classic compute. Budget policies attach serverless usage to teams. Query tags attribute SQL warehouse queries, generally available since September 2026. system.billing.usage with usage_metadata gives you the join to the job, cluster or warehouse. Mention the caveats unprompted: billing tables can lag by up to 24 hours on serverless, so daily reporting is reliable and hourly is not, and cloud-side bills cover the machines and storage that Databricks-side tables do not see.

Finish with the human side. Show-back before charge-back, and a published method with its known gaps, because trust in the numbers is the real deliverable.

Incident-response questions

A scheduled job that has run for months now fails at launch

Read the termination reason before anything else. Cloud Provider Launch Failure, INSUFFICIENT_INSTANCE_CAPACITY or a quota error points to the cloud provider: the instance type is unavailable in that zone, or the vCPU quota is exhausted. Immediate mitigation is to allow alternative instance types in the policy, or switch the job to serverless. Longer term, raise quota and allow multiple node types so the job can fall back. If the error is a permission or network error, ask what the infrastructure team changed. If the failure is cluster-wide and the status page shows an incident, wait rather than change things.

The reasoning to show is isolation of ownership: three parties, each with a typical symptom. Then show that you would add a standing alert for launch failures and a runbook so the next one takes minutes.

A user reports that a table has disappeared

Treat TABLE_OR_VIEW_NOT_FOUND as a privilege problem first, because Unity Catalog can report a missing object when the caller has no rights to see it. Check USE CATALOG, USE SCHEMA and SELECT with SHOW GRANTS. Then check system.access.audit for a drop or a change of ownership. Check the catalog binding too: a catalog bound to different workspaces is invisible from the wrong one. Only then conclude that the table was dropped, and consult table history and retention to see whether it is recoverable.

A weak answer jumps to "restore from backup". A strong answer separates "cannot see" from "does not exist", and uses the audit table to find who and when.

The monthly bill doubled

Find the day, then the tag, then the resource. Query system.billing.usage grouped by day to find the step change, group by tag to find the owner, and drill into usage_metadata for the job, cluster or warehouse. The usual causes: an all-purpose cluster without auto-termination, a job stuck in a retry loop, a warehouse with auto-stop disabled, or a new workload on Photon with no benefit. Then close the loop: add the cluster policy or alert that would have caught it, and tell the owning team in writing. If a large share of spend is untagged, say that attribution is itself the first remediation.

Every engineer's bundle deploy starts failing the same day

Compare CLI versions first. A newer deployment in the shared target with the state version bump from v1.17.0 makes clients older than v1.8.0 fail. Second suspect: authentication precedence. Bundle settings, then environment variables, then profiles, so a stray DATABRICKS_HOST or token variable can send the CLI to the wrong workspace. Third suspect: engine migration. Since v1.14.0, bundles that used Terraform state migrate to the direct engine after a successful deploy, and engine: terraform opts out. The lesson to state is that the CLI is shared infrastructure and needs a pinned, announced version.

A job fails on the new runtime after an upgrade

Ask what changed, and suspect Spark 4 behaviour. ANSI SQL mode is on by default, so casts that once returned null now raise errors; Scala 2.13 is required, so custom JARs built against 2.12 fail. Compare against the Spark migration guide and the runtime release notes. Mitigate by pinning the job to the previous LTS while you fix the code, then fix the cast or rebuild the JAR rather than disabling the stricter behaviour fleet-wide. Say that the root cause is process: upgrades should have been tested in staging with output comparison.

Two pipelines writing to one table keep failing

The errors DELTA_CONCURRENT_APPEND and ConcurrentDeleteReadException indicate overlapping transactions. Narrow what each writer touches, use liquid clustering with row-level concurrency, and retry idempotently. Ask whether two pipelines should write one table at all; often the better design is separate tables merged downstream.

Trade-off questions

Serverless or classic compute

Serverless removes sizing, startup delay and quota problems, and upgrades itself; you pay a higher DBU rate and accept its limits: no RDD APIs, no direct sparkContext, restricted custom JARs, and libraries through environment configuration. Classic gives you control over instance types, spot pricing and custom runtime needs, with the operational burden of quota and capacity. A balanced answer defaults to serverless for jobs and SQL, uses classic where a requirement forces it, and checks the real bill. State that you would measure, not assume.

One workspace or many

More workspaces buy isolation, separate networking and separate administration, and cost configuration effort and fragmented visibility. The catalog is the governance unit and the workspace is the deployment unit, so many needs for separation are met by catalogs and workspace binding inside one workspace per environment. Add workspaces for different compliance scope, network zones or administrator boundaries.

Managed or external tables

Managed tables let Unity Catalog control storage and lifecycle and enable features such as predictive optimization. External tables keep data in a path you own for other writers or readers. Default to managed with a written reason for each exception.

Terraform or bundles

They manage different layers. Terraform for slow-moving platform resources (catalogs, grants, policies, groups). Bundles for fast-moving project resources (jobs, pipelines, endpoints). Letting both manage the same resource causes drift fights.

Build on Databricks or assemble open-source pieces

Databricks reduces integration work and gives governance in one place; open-source assembly gives control and portability and costs engineering time. The mitigation for lock-in is open table formats, logic in plain libraries, and thin platform wrappers. Be explicit that you would choose according to team size and the cost of operating the alternative.

Arguing a trade-off to leadership

Leadership asks about money, risk and speed, in that order. Translate technical choices into those units, and bring a recommendation, not a menu.

For example, to propose moving scheduled jobs to serverless: "This removes capacity failures that caused four incidents last quarter and cuts platform engineering time, at a higher unit rate. I propose moving the twenty jobs with the most failures first and comparing the bill after a month. If cost rises more than we can justify against saved incident time, we revert those jobs to classic, and the change is a one-line edit in each bundle."

Notice the structure: the problem in business terms, the proposal, the cost honestly stated, a small first step, and a defined way back. The same shape works for asking for budget to build a paved road ("time to first deployment drops from three weeks to one day"), for retiring personal access tokens ("removes the largest credential-leak risk, and takes one sprint per team"), and for pausing a feature launch because a beta dependency has no support commitment.

When you must say no, say what you can offer. "We cannot put that data in a shared catalog without a classification review, but I can set up a restricted catalog with row filters in two days and review in parallel."

What a strong answer covers that a weak one misses

Strong answers ask about constraints before designing. Weak answers start naming products.

Strong answers separate ownership: Databricks, the cloud provider and your own infrastructure. Weak answers blame "Databricks" for every failure.

Strong answers state failure modes of their own design, such as the shared metastore. Weak answers describe only the happy path.

Strong answers use the audit and billing system tables as evidence. Weak answers rely on screenshots and recollection.

Strong answers know what is deprecated and what replaced it: stages by aliases, personal tokens by OAuth, mounts by volumes, Hive metastore by Unity Catalog, the Asset Bundles name by Declarative Automation Bundles. Weak answers teach the old way.

Strong answers talk about people: onboarding time, deprecation notices, the support rota. Weak answers treat a platform as only technology.

Strong answers say what they do not know and how they would find out. Say "I would check the release notes for the current default" instead of guessing a flag.

Rapid-fire facts to have ready

• Hierarchy: metastore, catalog, schema, then tables, views, volumes, functions, models. One metastore per region.
• Read permissions: USE CATALOG, USE SCHEMA, SELECT.
• Default LTS to teach: DBR 18 LTS on Spark 4.1, supported to 2029-06-10; each LTS is supported for three years.
• DBR 19 (Spark 4.2) is not LTS.
• CLI authentication precedence: bundle settings, environment variables, then profiles.
• Interactive OAuth tokens live in the OS secure store; DATABRICKS_AUTH_STORAGE=plaintext is the headless fallback.
• Billing system tables can lag up to 24 hours on serverless.
• VACUUM default retention is seven days.
• Liquid clustering replaces partitioning plus ZORDER for new tables.
• Model promotion uses aliases such as champion, not stages.

Questions to ask the interviewer

Strong candidates interview back. Ask how many workspaces and metastores they run and why. Ask how production jobs authenticate. Ask what percentage of spend is tagged. Ask which runtime their oldest production job uses and when it goes out of support. Ask who owns the platform and how teams request changes. The answers tell you whether the job is building a platform or rescuing one, and showing that you know to ask is a senior signal in itself.

Sixty-second self-test

Answer these aloud without notes.

1. Which boundary separates what Databricks hosts from what runs in your account, and what fails on each side?
2. Why is a catalog-level SELECT grant risky?
3. What identity should a production job run as, and how does CI authenticate?
4. Why must the CLI version be pinned, and what changed in v1.17.0?
5. What are the first three things you check when the bill doubles?
6. Why is Spark 4 an upgrade hazard, and how do you reduce the risk?
7. How does a model move to production, and what replaced stages?
8. Name two cases where you would pick something other than Databricks.

If you stumble on any of these, return to the matching section of the detailed file. Related guides for the surrounding stack are Terraform, MLflow and Delta Lake.

These are the lessons that only appear once a Databricks platform has many teams, many jobs and a monthly bill that somebody reads. Each trap below follows one pattern: the symptom you will see, the cause and the fix. The later sections turn the lessons into habits and review checklists. Versions referenced are Databricks Runtime 18 LTS (Spark 4.1) and Databricks CLI 1.19.

Production traps that appear at scale

The grant that exposed next month's table

Symptom. A privacy review finds that a broad group can read a table created last week that nobody meant to share.

Cause. Privileges inherit downwards in Unity Catalog. A SELECT granted at catalog level covers every schema and table that exists now and every one created later. It was convenient on the day, and quietly dangerous afterwards.

Fix. Grant at schema level to groups, and keep catalog-level grants for platform roles only. Add a scheduled query over system.access.audit and the grant tables that lists catalog-level grants to anything other than platform groups. A control you cannot query is a control you cannot prove.

The table that "does not exist"

Symptom. A user reports TABLE_OR_VIEW_NOT_FOUND for a table you can see perfectly well.

Cause. Unity Catalog can return not-found when the caller lacks the privilege to know the object exists. A second cause is a catalog bound to different workspaces than the one the user is in.

Fix. Check USE CATALOG, USE SCHEMA and SELECT first, then the workspace binding. Put this as the first line of your support runbook, because it saves hours of looking for a drop that never happened.

The shared CLI that stopped working for half the team

Symptom. Bundle deploys fail for some engineers and CI, while others are fine.

Cause. CLI v1.17.0 raised the direct-deployment state version to 3, and clients older than v1.8.0 reject a bundle deployed with it. Before that, v1.14.0 migrated Terraform-state bundles to the direct engine after a successful deploy.

Fix. Treat the CLI as shared infrastructure. Pin the version in CI, publish the supported version, and upgrade laptops and pipelines together. If you must stay on the Terraform engine for a while, say so explicitly with engine: terraform in the bundle configuration, and write down when you will stop.

The headless machine that cannot log in

Symptom. The CLI fails on a build agent or container where an interactive login makes no sense.

Cause. Since CLI 1.0, user OAuth tokens go to the operating system's secure store, and a headless Linux machine may have no Secret Service to hold them.

Fix. Do not use interactive login for automation at all. Use a service principal with OAuth machine-to-machine credentials, through DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET. Reserve DATABRICKS_AUTH_STORAGE=plaintext for disposable machines where you accept the risk.

The wrong identity acting in CI

Symptom. A pipeline deploys to the wrong workspace or acts as a person.

Cause. Authentication precedence is bundle settings, then environment variables, then config profiles. A leftover environment variable such as DATABRICKS_TOKEN or DATABRICKS_HOST silently overrides the intended profile.

Fix. In CI, set only the variables you intend and print which identity is acting before the deploy, for example with databricks current-user me. Remove personal access tokens from pipelines entirely.

Spark 4 and the silent cast

Symptom. After moving to DBR 17 or 18 LTS, jobs that ran for years start failing on casts, or a downstream metric changes.

Cause. Spark 4 turns ANSI SQL mode on by default, so conversions that quietly returned null now raise errors. Spark 4 is also Scala 2.13 only, so custom JARs built for 2.12 break.

Fix. Test the new runtime in staging with real data and compare outputs, not just success. Fix the casts and rebuild the JARs. Resist the temptation to switch the stricter behaviour off fleet-wide, because that postpones the problem and hides it. Pin classic jobs to a runtime so a default never moves under you.

The all-purpose cluster nobody switched off

Symptom. A flat daily charge for a cluster that is idle most of the time.

Cause. Scheduled work running on an all-purpose cluster, or an interactive cluster without auto-termination.

Fix. Scheduled work goes to job clusters or serverless. Every all-purpose cluster has auto-termination, enforced in the policy and not requested in a wiki. Query system.billing.usage weekly for compute that bills at all hours.

Billing that lags behind the incident

Symptom. You react to a cost spike a day late.

Cause. Billing system tables can lag by up to 24 hours on serverless, so an hourly alert has nothing to see.

Fix. Alert on leading indicators you can see now: number of running clusters, warehouse uptime, job run counts and retry loops. Keep billing for the daily review.

Concurrent writers fighting over one table

Symptom. DELTA_CONCURRENT_APPEND or ConcurrentDeleteReadException appears at peak load and disappears when you rerun.

Cause. Two transactions touched overlapping data and the second lost.

Fix. Narrow what each writer touches, use liquid clustering with row-level concurrency, and retry idempotently. Consider whether two pipelines should write one table at all.

The schema change that rippled

Symptom. A downstream dashboard breaks the morning after an upstream job changed a column.

Cause. mergeSchema or overwriteSchema was enabled broadly, so upstream changes flowed into a table many people read.

Fix. Make schema changes deliberate. Check lineage before changing anything shared: the lineage system tables tell you who reads what. Announce changes and version them through bundles.

Serverless limits discovered in production

Symptom. A job that works on a classic cluster fails with an unsupported-feature error on serverless.

Cause. Serverless has no RDD APIs, no direct sparkContext use, restricted custom JARs and a narrower set of Spark configurations.

Fix. Publish the limits to users, run a staging job on serverless before migrating, and keep classic available for workloads that need it.

Platform-ownership habits

Keep a migration register

Most platform pain is a legacy pattern that outlived its replacement. Keep a short register with an owner and a date for each: workspace model registry and stages (replace with Unity Catalog models and aliases), personal access tokens (OAuth), mounts and the DBFS root (volumes and external locations), Hive metastore tables (Unity Catalog), init scripts on DBFS (workspace or volume paths), dbutils.notebook.run for orchestration (job tasks) and partitioning plus ZORDER on new tables (liquid clustering). Review it monthly. Teams will migrate when someone gives them a date and a recipe, and not before.

Treat the beta label as a risk rating

As of September 2026, attribute-based access control on views and at metastore level, managed agent memory and sessions, the Agent Bricks CLI and Unity Catalog external secrets backed by AWS Secrets Manager are beta. Beta features can change shape or behaviour. Adopt them as pilots with an exit plan and keep them out of your compliance story until they are generally available.

Watch the retirement dates

Release notes carry hard dates. The beta Genie MCP endpoint /api/2.0/mcp/genie is deprecated on 2026-10-31, and several third-party models on Unity Gateway retire on 2026-10-30. Keep a list of which of your endpoints and applications depend on each, and put the dates in the team calendar. These are the easy outages to avoid and the embarrassing ones to explain.

Build a paved road and measure it

Your platform's real product is the shortest path to a safe production deployment. Measure median time from a new team's first request to its first production job. Measure the share of resources created outside the template. When either number is bad, the fix is almost always a better template, not a stricter lecture.

Run a support rota and learn from it

Every recurring question is a missing template, policy or document. Tag support requests by cause for a quarter and fix the top three causes in code. The goal is a platform that needs you less each month.

Reading the signals before they page you

Most of the traps above announce themselves days earlier in data you already own. The habit worth building is a short weekly review of four signals, run by whoever holds the platform rota, with each one answered by a query and not by memory.

The first signal is failed runs grouped by termination reason. A rising count of launch failures means quota or capacity is tightening, and you can fix that with a policy change before it becomes an outage. A rising count of out-of-memory failures usually means data growth or a skewed key, which belongs to the owning team and deserves a conversation, not a bigger cluster.

The second is spend by tag, compared with the trailing average. A step change on one tag is nearly always one resource, and the earlier you look the fewer days of waste you pay for. Include the percentage of untagged spend in the same report, since an untagged slice growing is a policy hole.

The third is access. Look in system.access.audit for grants made outside your pipeline, for new storage credentials and external locations, and for service principals created without an owner. Each is a small event and an early sign that someone found a faster path around the road you built.

The fourth is the calendar: runtimes within twelve months of end of support, announced retirements, and the gap between the CLI version in CI and the latest release. Because DBR 14.3 LTS is supported to 2027-02-01 and 15.4 LTS to 2027-08-19, jobs on those runtimes already belong on the migration register with a date.

Write down what you decide each week, even when the decision is to do nothing. After a quarter, the notes show which signals earned their place and which you can retire, and they are the best evidence you will have when someone asks whether the platform is getting healthier.

Review checklists

For a pull request to a bundle

1. Targets. Does production use mode: production, an explicit run_as service principal and a non-user root path?
2. Variables. Are catalogs and other environment values variables, with no production names hard-coded in notebooks?
3. Compute. Serverless or a job cluster under a policy, with a size cap and tags?
4. Runtime. An LTS runtime, or a pinned environment version?
5. Secrets. Read from a scope, never in the file or the notebook?
6. Alerts. Failure notification and a duration rule, going to the owning team?
7. CLI. Does the change need a newer CLI than the pinned one?

For a design document

1. Data. Which catalogs and schemas, owned by which groups, with which grants and which classification?
2. Residency. Which cloud and region, and does every dependency, including model providers, respect it?
3. Failure. What fails if the control plane is down, if capacity runs out, or if a storage permission changes?
4. Cost. What does it cost per run and per month, and where does the tag land?
5. Upgrade. Which runtime does it need, and what happens at its end of support?
6. Beta. Does anything depend on a beta feature, and what is the exit?
7. Exit. How would we move this off Databricks, and what would be hard?

For a Unity Catalog change

1. Scope. Is the grant at the narrowest level that works, and to a group?
2. Inheritance. What future objects will this grant also cover?
3. Evidence. Can an audit query show the grant was made through code?
4. Test. Has someone who should not have access tried and failed?

Practice cards

1. Audit your grants. List every principal with a direct grant and mark the people. Convert each to a group grant and re-test access.
2. Find the idle spend. Group system.billing.usage by day and tag for the last thirty days, and identify one cluster or warehouse that bills while nobody uses it.
3. Rehearse the CLI upgrade. Upgrade the CLI in a staging pipeline and on one laptop, deploy a bundle from each, and record anything surprising.
4. Replace one token. Take one pipeline that uses a personal access token and move it to a service principal with OAuth.
5. Run an upgrade rehearsal. Take one production job, run it on DBR 18 LTS in staging, and diff the output against production.
6. Test a denial. Pick a table with a row filter and prove that a user outside the group cannot see the filtered rows.
7. Retire an endpoint. List serving endpoints with no traffic for a month and retire one with its owner's agreement.

Operational recipes

Three short recipes cover a lot of platform work. Keep them in your runbook.

Find out who and what is acting from a terminal:

databricks -v
databricks current-user me
databricks auth profiles

The first command confirms the CLI version against your pinned one, the second shows which identity the active configuration resolves to, and the third lists the profiles available so you can see which one a script would use. Run all three whenever a deploy behaves strangely.

Validate and deploy a bundle through the same path in every environment:

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run -t prod train

Adopt an existing hand-built job into code without recreating it:

databricks bundle generate job --existing-job-id 123456789
databricks bundle deployment bind train 123456789 -t prod

Look at recent usage by SKU to start any cost investigation:

SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= date_sub(current_date(), 14)
GROUP BY usage_date, sku_name
ORDER BY usage_date DESC, dbus DESC;

When you run the queries, remember the serverless lag: yesterday's numbers may still be incomplete. If a number looks wrong, wait a day before concluding anything.

Cross-links and next steps

For the platform layer, read the Terraform guide. For delivery, read GitHub Actions. For the table format and the engine underneath, read Delta Lake and Spark. For the experiment and model layer, read MLflow.

Final checklist

• Every object is owned by a group, and every grant goes to a group at the narrowest sensible level.
• Production runs as a service principal and CI uses OAuth with no long-lived tokens.
• The CLI version, the runtime and any serverless environment version are pinned and announced.
• Compute is chosen by policy, with size caps, auto-termination and mandatory tags.
• Spend is attributed by tag and reviewed daily, with the serverless lag in mind.
• A migration register tracks each legacy pattern with an owner and a date.
• Beta features are pilots with an exit, and dated retirements are in the calendar.
• The four incident playbooks exist, and one has been rehearsed this quarter.
• Someone has verified that a denied grant is actually denied.