The Complete Great Expectations Guide

This is part one of three. It covers everything you need to do real work with Great Expectations, not a teaser. By the end you can install the library, connect it to a dataframe, a folder of files or a database, write tests for your data, run them as a single repeatable step, read the results, publish a readable report, and make a pipeline stop when the data is bad. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and these ideas only stick once you have watched your own data fail a test and worked out why.

One warning before we start. Most blog posts, videos and chat answers about this tool describe a version that no longer exists. Great Expectations changed its entire API in 2024, and its hosted product was shut down in 2026. This guide teaches the 1.x library, called GX Core, and is checked against release 1.23.2. If a tutorial tells you to type great_expectations init or to call context.sources, it is describing the old tool, and that advice will not work here.

What Great Expectations is, and the problem it solves

Great Expectations, usually shortened to GX, is a Python library for testing data. You write down what you believe is true about a table or a file ("every order has an ID, IDs are never repeated, prices are positive, the country column only contains countries we ship to"), and GX checks the data against those beliefs and tells you precisely which ones failed and how badly.

Each of those beliefs is called an Expectation. They are assertions about data, in the same spirit as the assertions you write in a unit test for code, but aimed at the rows and columns flowing through your pipeline instead of at a function's return value.

Why would you need this? Think about how data problems actually reach you. A column that was always filled starts arriving empty because an upstream team renamed a field. A currency amount that used to be in riyals is suddenly in fils, a hundred times larger. A nightly export delivers half the usual number of rows because a job timed out. A date column quietly starts containing the string "N/A". None of these crash your code. Your pipeline happily reads the data, trains a model on it, or publishes a dashboard from it. The first sign of trouble is usually a person asking why the numbers look strange, days later, after decisions have been made.

Code fails loudly and data fails silently. That sentence is the whole reason this tool exists. A typo in your Python raises an exception on the first run. A typo in somebody else's data produces no exception at all; it produces wrong answers that look plausible.

Before tools like this, teams handled the problem in one of three ways. Some wrote ad hoc assert statements or if df["price"].min() < 0 checks scattered through their scripts, which works until the scripts multiply and nobody remembers what is being checked. Some wrote SQL queries that count bad rows and pasted them into a scheduled job, which works until the person who wrote them leaves. And many simply did nothing and relied on users to notice. GX gives those checks a shared vocabulary, a standard place to live, a standard format for results, and a way to produce documentation that non-programmers can read.

For machine learning work the stakes are higher than for reporting. A model trained on data with a silently broken column does not raise an error; it learns the breakage. A feature that is suddenly always null can drag accuracy down for weeks before anyone connects the drop to the data. Checking data at the boundaries, where it enters your pipeline and where it enters training, is one of the cheapest reliability improvements available.

Three facts about how GX works will save you confusion later.

It is a library, not a service. There is no server to install, no daemon running in the background and no web application to log into. You pip install it, import it into Python, and call it from a notebook, a script, an Airflow task or a CI job. Everything runs inside your own Python process.

It tests data where the data lives. If your data is a pandas dataframe in memory, GX computes its checks with pandas. If it is a table in PostgreSQL or Snowflake, GX translates the checks into SQL and lets the database do the counting, so you do not pull a hundred million rows into your laptop. If it is a Spark dataframe, the checks run on the Spark cluster.

It is free and open source. GX Core is released under the Apache 2.0 licence. There used to be a paid hosted product called GX Cloud. It was shut down in June 2026 (release 1.18.0), and the library now refuses to start in cloud mode, raising an error that says so. Anything you read about a GX Cloud account, an access token, or a hosted dashboard is obsolete.

The mental model: five nouns and a loop

GX has a fairly long vocabulary, but the day-to-day picture needs only a handful of words. Learn them in the order the data flows through them.

A Data Source tells GX how to reach a place where data lives. It might be "the pandas library in this Python process", "a folder on disk", "this PostgreSQL database" or "this S3 bucket". It holds the connection details and nothing else.

A Data Asset is a specific collection of records inside a Data Source: one table, one SQL query, one family of CSV files, or one dataframe. If the Data Source is the warehouse, the asset is one shelf in it.

A Batch Definition says how to slice the asset into pieces you will test. You might test the whole table every time, or one specific file, or only one day's rows, or one month's. The definition is the rule for slicing.

A Batch is what you get when you apply that rule: the actual group of records ready to be tested. You usually ask the Batch Definition for one, sometimes giving it parameters such as {"year": 2024, "month": 2} to say which slice you want.

Now the other half of the picture: what you assert, and how you run the assertions.

An Expectation is a single verifiable statement about data, such as "the column order_id has no null values". In code it is a Python class you create with arguments.

An Expectation Suite is a named collection of Expectations that together describe one dataset. "orders_suite" might hold twelve Expectations about the orders table.

A Validation Definition links one Suite to one Batch Definition. It answers the question "test this data against these rules". Running it produces a result.

A Checkpoint bundles one or more Validation Definitions and adds Actions, which are things to do after the tests run: send a Slack message, rebuild the documentation site, post to a webhook. The Checkpoint is what you run in production. The docs call it the primary means for validating data in a deployment.

Holding all of these together is the Data Context, the object you create first and use to reach everything else. It stores the configuration of your project and gives you methods to add and fetch Data Sources, Suites, Validation Definitions and Checkpoints.

That is a lot of nouns, so here is the same story told as a loop, which is how you will actually work:

1. Create a context.
2. Tell it where the data is (source, asset, batch definition).
3. Write Expectations and collect them into a suite.
4. Try the suite on a batch and read the results.
5. Wrap the working suite into a Validation Definition and a Checkpoint.
6. Run the Checkpoint whenever new data arrives, and react to the outcome.

You do not have to use every noun on day one. For exploration you can skip straight to testing one Expectation on one batch. The extra objects earn their keep when you want a repeatable, automated check.

Two kinds of context

The Data Context comes in two forms that matter to a beginner.

An Ephemeral Data Context lives in memory. When your Python process ends, everything you defined is gone. It is ideal for notebooks, tests and pipeline jobs that rebuild their setup from code each run.

A File Data Context saves its configuration into a folder named gx/ in your project. Suites, Validation Definitions, Checkpoints and Data Source settings persist between runs, and you can commit most of the folder to git. It is ideal when you want to build up a project over time.

There used to be a third, the Cloud Data Context. It is dead, as explained above.

Installing and checking your setup

GX needs Python 3.10, 3.11, 3.12 or 3.13. Python 3.14 is not supported at the time of writing, and 3.9 was dropped in late 2025. The official documentation lists macOS and Linux as supported platforms, and says that although Windows is not officially supported, many people run it there successfully. If you are on Windows and something behaves oddly, the Windows Subsystem for Linux (WSL2) is a sensible fallback, though that is our suggestion rather than an official statement.

First check your Python version:

python --version

If it prints something between 3.10 and 3.13, you are fine. On some systems the command is python3. Then create a virtual environment, which is a private folder of installed packages for one project, so that GX's dependencies do not collide with anything else on your machine. On macOS and Linux:

python -m venv my_venv
source my_venv/bin/activate
python -m pip install --upgrade pip
pip install great_expectations

On Windows PowerShell the same idea looks slightly different:

py -3.12 -m venv my_venv
.\my_venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install great_expectations

After activation your prompt usually shows (my_venv) at the start. Everything you install now goes into that folder and is deleted if you delete the folder.

The core install gives you pandas support and SQLite through Python's standard library. To talk to other systems you install extras, optional groups of dependencies named in square brackets. You quote the name so your shell does not try to interpret the brackets:

pip install 'great_expectations[postgresql]'
pip install 'great_expectations[snowflake]'
pip install 'great_expectations[spark]'
pip install 'great_expectations[s3]'

Use double quotes on Windows. The list of extras is long; it includes bigquery, databricks, sql-server, oracle, redshift, trino, gcp, azure and others. Notice that the old mssql extra was replaced by sql-server, and that there is no cloud extra any more.

Installing with the extra you need is not cosmetic. In September 2026 a new SQLAlchemy release (2.1) broke older GX versions on Python 3.11 and above. The fix in GX 1.23.2 is partly a version cap applied through the extras, so installing great_expectations[snowflake] gets you the protection while a bare install plus a separate Snowflake package might not. If you cannot upgrade, pinning sqlalchemy<2.1 works around it.

Now verify the install from Python:

import great_expectations as gx

print(gx.__version__)
context = gx.get_context()
print(type(context).__name__)

Expected output looks like this:

1.23.2
EphemeralDataContext

The version may be newer when you read this; releases arrive every couple of weeks. The type will be EphemeralDataContext if no gx/ folder was found, or FileDataContext if one was. That single line of output is worth understanding, because it is the source of the most common beginner confusion, which we return to in the errors section.

From a shell you can also run pip show great_expectations to see the installed version and where it lives.

Your first validation in five minutes

Before building any project structure, let us see GX catch a real problem. Create a small CSV file called orders.csv with a deliberate mess in it:

order_id,customer,country,amount
1001,Amal,EG,120.50
1002,Omar,SA,89.00
1003,Layla,AE,45.25
1003,Youssef,EG,300.00
1005,,SA,-15.00
1006,Nour,XX,60.00

Look at it for a moment. Order 1003 appears twice. Order 1005 has no customer and a negative amount. Order 1006 has a country code, XX, that does not exist. A human notices these by reading six rows; nobody notices them in six million. Now let GX do the noticing.

import great_expectations as gx

context = gx.get_context()

batch = context.data_sources.pandas_default.read_csv("orders.csv")
print(batch.head())

pandas_default is a built-in Data Source that GX creates for you so that you can explore without any setup. Calling read_csv on it reads the file into pandas and hands you back a Batch directly. The head() call shows the first rows, just as in pandas.

Now create an Expectation and test it:

not_null = gx.expectations.ExpectColumnValuesToNotBeNull(column="customer")
result = batch.validate(not_null)
print(result)

The printed result is a structured object. The important parts look like this:

{
  "success": false,
  "expectation_config": {
    "type": "expect_column_values_to_not_be_null",
    "kwargs": { "column": "customer", "batch_id": "pandas_default-#ephemeral_pandas_asset" }
  },
  "result": {
    "element_count": 6,
    "unexpected_count": 1,
    "unexpected_percent": 16.666666666666664,
    "partial_unexpected_list": [null]
  }
}

Read it top to bottom. success: false is the verdict. expectation_config repeats what you asked: the snake_case type string is the same Expectation as the class name ExpectColumnValuesToNotBeNull, written the way it appears in saved files. element_count is how many rows were examined, unexpected_count is how many broke the rule, and unexpected_percent turns that into a percentage. The exact batch identifier will differ on your machine.

Try a range check on the amount column:

positive = gx.expectations.ExpectColumnValuesToBeBetween(
    column="amount", min_value=0, max_value=10000
)
result = batch.validate(positive)
print(result.success)
print(result.result["partial_unexpected_list"])

That prints False and [-15.0], which is the offending value. You can tweak an Expectation and re-test it instantly. Set the minimum to -100 and run it again and it passes, because the rule is now looser:

positive.min_value = -100
print(batch.validate(positive).success)

This exploratory loop (create, validate, read, tweak) is the most useful habit in GX. You are not committing anything; you are asking questions of the data and learning what "normal" looks like. Only when an Expectation captures something you truly believe do you keep it.

A few more will demonstrate the range of what is available. Each is a class under gx.expectations:

unique_ids = gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
valid_country = gx.expectations.ExpectColumnValuesToBeInSet(
    column="country", value_set=["EG", "SA", "AE", "KW", "QA"]
)
row_count = gx.expectations.ExpectTableRowCountToBeBetween(min_value=1, max_value=1000000)

for exp in (unique_ids, valid_country, row_count):
    r = batch.validate(exp)
    print(exp.__class__.__name__, r.success)

You should see that the uniqueness and country checks fail (order 1003 repeats and XX is not in the set) while the row count passes. In three minutes you found all three problems in the file.

Building a reusable suite from a dataframe

Testing one Expectation at a time is fine for exploring, but you want to keep the good ones together and run them as a unit. That is what an Expectation Suite is for. We will also stop using the built-in shortcut and set up the objects properly, because that is what you will do in every real project.

In a real pipeline your data usually arrives as a pandas dataframe that an earlier step produced. Here is the full setup, written to work in an in-memory context so that nothing is left on disk:

import pandas as pd
import great_expectations as gx

df = pd.read_csv("orders.csv")

context = gx.get_context(mode="ephemeral")

data_source = context.data_sources.add_pandas(name="my_pandas")
asset = data_source.add_dataframe_asset(name="orders_df")
batch_definition = asset.add_batch_definition_whole_dataframe("whole_orders")

batch = batch_definition.get_batch(batch_parameters={"dataframe": df})

Walk through it. add_pandas creates a Data Source for pandas. add_dataframe_asset adds an asset meaning "a dataframe". add_batch_definition_whole_dataframe says "the whole thing, no slicing". And get_batch needs to know which dataframe, since a dataframe asset has no fixed content, so you pass it at run time in batch_parameters. The same asset and definition can be reused tomorrow with tomorrow's dataframe.

Notice the argument mode="ephemeral". Passing the mode explicitly removes guesswork. With no argument, gx.get_context() looks for an existing project and falls back to in-memory if it finds none; with mode="ephemeral" you always get memory, even if a gx/ folder exists nearby.

Now build the suite:

suite = context.suites.add(gx.ExpectationSuite(name="orders_suite"))

suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(
    column="amount", min_value=0, max_value=10000))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(
    column="country", value_set=["EG", "SA", "AE", "KW", "QA"]))
suite.add_expectation(gx.expectations.ExpectTableRowCountToBeBetween(min_value=1))

context.suites.add(...) registers the suite with the context and returns it. Each add_expectation call attaches an Expectation and, in a context that persists, saves it. Now run the whole suite against the batch:

result = batch.validate(suite)
print(result.success)
print(result.statistics)

The first line prints False (the file has problems) and the second prints a small summary with counts such as evaluated_expectations, successful_expectations, unsuccessful_expectations and success_percent. One failing Expectation makes the whole suite fail.

You can look at each Expectation's individual result:

for r in result.results:
    print(r.expectation_config.type, r.success)

That lists one line per Expectation, which is the quickest way to see which rules broke.

Changing an Expectation you already saved

Beginners often try to fix a rule by editing the object and then wonder why nothing changed. To change an Expectation that is already in a suite, edit its attributes and call save() on it:

for exp in suite.expectations:
    if exp.__class__.__name__ == "ExpectColumnValuesToBeBetween":
        exp.max_value = 50000
        exp.save()

To fetch a suite you saved earlier, use context.suites.get(name="orders_suite"). If you edit the suite itself, for example by deleting an Expectation, call suite.save() afterwards. GX will remind you with an error saying the suite "has changed since it has last been saved" if you forget.

Choosing and tuning Expectations

GX ships around sixty core Expectation classes, and you can browse every one, with examples, in the Expectation Gallery on the GX website. You will not memorise them. What you should develop is a sense of the families, so that when you have a data worry you know which family to reach for.

Shape of the table. ExpectTableRowCountToBeBetween and ExpectTableRowCountToEqual check volume, which catches the half-empty export. ExpectTableColumnsToMatchSet and ExpectTableColumnsToMatchOrderedList check that the expected columns exist, which catches a renamed or dropped field. ExpectColumnToExist is the gentlest version of that check.

Missing and unique values. ExpectColumnValuesToNotBeNull and ExpectColumnValuesToBeNull are the basics. ExpectColumnValuesToBeUnique protects keys. ExpectCompoundColumnsToBeUnique protects composite keys such as the pair (customer_id, order_date).

Allowed values and ranges. ExpectColumnValuesToBeInSet and ExpectColumnValuesToNotBeInSet handle categories. ExpectColumnValuesToBeBetween handles numeric and date ranges.

Text patterns and lengths. ExpectColumnValuesToMatchRegex checks formats such as phone numbers or IDs. ExpectColumnValueLengthsToBeBetween catches truncated or runaway text. ExpectColumnValuesToMatchStrftimeFormat checks that a string column holds dates in a particular format.

Types. ExpectColumnValuesToBeOfType and ExpectColumnValuesToBeInTypeList check that a column holds the type you expect, which catches numbers arriving as strings.

Statistics of a whole column. ExpectColumnMeanToBeBetween, ExpectColumnMedianToBeBetween, ExpectColumnMinToBeBetween, ExpectColumnMaxToBeBetween and ExpectColumnStdevToBeBetween check aggregates. These are the ones that catch the silent unit change: if amounts were averaging 120 and suddenly average 12,000, a range check per row may still pass but the mean check fails.

Relationships between columns. ExpectColumnPairValuesAToBeGreaterThanB catches an end date before its start date. ExpectMulticolumnSumToEqual checks that parts add up to a total.

There are two kinds of Expectation worth distinguishing, because it affects how you read results. Some judge each row: "every value in amount is between 0 and 10000". These produce a count of unexpected rows. Others judge the column or table as a whole: "the mean of amount is between 50 and 500". These produce a single observed value, which the result shows under observed_value.

The mostly argument

Real data is rarely perfect, and a rule that demands 100 percent purity fails every day and gets ignored. Row-level Expectations accept a mostly argument, a fraction between 0 and 1 that says how much of the data must satisfy the rule:

gx.expectations.ExpectColumnValuesToNotBeNull(column="customer", mostly=0.95)

This passes if at least 95 percent of customers are filled in. Use mostly when the business accepts a small, known amount of mess, and keep it tight enough that a real regression still trips it. The number is a statement about your data, so write it down where a colleague can question it.

Severity and descriptions

Two optional arguments make Expectations kinder to the humans reading results. severity can be "critical" (the default), "warning" or "info", and later you can tell actions to notify only on a given severity. description is a sentence that appears in the documentation site:

gx.expectations.ExpectColumnValuesToBeBetween(
    column="amount",
    min_value=0,
    max_value=10000,
    severity="warning",
    description="Order amounts are positive and below 10,000 in local currency.",
)

A description turns a technical check into something a product manager can read and challenge.

Reading validation results

A result object carries a lot of information. Knowing which parts to read keeps you from drowning in it.

At the top level a suite result has success (true only if every Expectation passed), statistics (the counts), results (one entry per Expectation) and meta (run information). Each entry in results has its own success, the expectation_config you asked for, and a result dictionary whose contents depend on the result format.

The result format controls how much detail you get back. There are four levels:

You choose the level when you validate:

result = batch.validate(suite, result_format="COMPLETE")

In a Checkpoint you pass result_format to the constructor, and in a Validation Definition run you pass it as result_format={"result_format": "SUMMARY"}. The choice is a trade-off. Detailed results help you debug, but they can be very large on a big table, and they contain actual data values, which matters if the data is sensitive. A sensible habit is SUMMARY by default and COMPLETE only when you are investigating.

Finding the bad rows

The failing rows are what you really want. By default results list only the offending values, which is not enough when a value like -15.0 could appear in a hundred rows. Ask GX to report which rows by naming a column that identifies them:

result = batch.validate(
    suite,
    result_format={
        "result_format": "COMPLETE",
        "unexpected_index_column_names": ["order_id"],
    },
)

Now each failing Expectation includes an unexpected_index_list, a list of dictionaries holding the order_id and the bad value for each failing row. A few other options are worth knowing: partial_unexpected_count controls how many example values are shown (set it to 0 to show none), and include_unexpected_rows returns up to 200 complete failing rows for the row-level Expectation types.

You can print a readable summary of a Checkpoint result with result.describe(), which returns a JSON-style string of what passed and failed. There is also describe_dict() if you would rather work with a Python dictionary.

Connecting to files and databases

Dataframes are convenient, but in production your data more often sits in files or in a database. The setup has the same shape (source, asset, batch definition) with different method names.

A folder of files

For CSV, Parquet, JSON and similar files on disk, use the pandas filesystem Data Source pointed at a base folder:

data_source = context.data_sources.add_pandas_filesystem(
    name="files", base_directory="./data"
)
csv_asset = data_source.add_csv_asset(name="taxi_csv")

There are sibling methods for other formats, such as add_parquet_asset, add_json_asset and add_excel_asset. Now define how to slice it. The simplest is one specific file:

january = csv_asset.add_batch_definition_path(
    name="january", path="yellow_tripdata_sample_2019-01.csv"
)
batch = january.get_batch()

When the files follow a naming pattern with a date in it, define a monthly batch definition with a regular expression that has named groups called year and month:

monthly = csv_asset.add_batch_definition_monthly(
    name="monthly",
    regex=r"yellow_tripdata_sample_(?P<year>\d{4})-(?P<month>\d{2})\.csv",
)
batch = monthly.get_batch(batch_parameters={"year": 2019, "month": 1})

A regular expression is a compact pattern language for matching text; \d{4} means "four digits". The named groups (?P<year>...) tell GX which part of the file name is the year. The pattern is matched relative to the base directory. Batch parameters should be integers; older versions accepted digit strings like "2019", and 1.21.0 deprecated that. If you call get_batch() with no parameters on a partitioned definition, you receive the latest matching batch.

A database table

For SQL databases, use the Data Source matching your database. PostgreSQL is a good example:

data_source = context.data_sources.add_postgres(
    name="warehouse",
    connection_string="${POSTGRES_CONNECTION_STRING}",
)
asset = data_source.add_table_asset(name="trips", table_name="public.trips")
whole = asset.add_batch_definition_whole_table("FULL_TABLE")

Two details matter here. First, the connection string uses the ${POSTGRES_CONNECTION_STRING} placeholder instead of the literal password; the configuration section explains why. Second, the table name is schema-qualified (public.trips). Older code used a separate schema_name argument, which was deprecated in 1.14.0 and is scheduled for removal in 2.0, so always put the schema inside table_name.

To test only part of a table, define a date-partitioned batch definition using a date column:

daily = asset.add_batch_definition_daily(name="DAILY", column="pickup_datetime")
batch = daily.get_batch(batch_parameters={"year": 2020, "month": 1, "day": 14})

This makes GX add a filter on pickup_datetime so that only one day's rows are tested, and the database does the work. Validating yesterday's increment instead of the whole table keeps queries fast and your database bill small. There are also add_batch_definition_monthly and add_batch_definition_yearly.

Other SQL factories exist on context.data_sources, including add_sqlite, add_snowflake, add_bigquery, add_databricks_sql, add_redshift, add_sql_server and a generic add_sql that accepts any SQLAlchemy connection URL. You can also test the result of a query instead of a table with add_query_asset, provided the SQL starts with SELECT (a query beginning with WITH is rejected; wrap it in a subquery or create a view).

GX issues queries as the database user you give it. Use an account with read-only access (SELECT on the tables you validate) so that a mistake in a test can never change data.

Validation Definitions and Checkpoints

So far you have called batch.validate(suite) by hand. That is a good way to explore but not a good way to run something every night. Two objects turn it into a repeatable operation.

A Validation Definition pairs one Batch Definition with one Suite and gives the pair a name:

validation_definition = context.validation_definitions.add(
    gx.ValidationDefinition(
        name="orders_whole_vd",
        data=batch_definition,
        suite=suite,
    )
)

You run it by supplying batch parameters, if the definition needs any:

vd_result = validation_definition.run(
    batch_parameters={"dataframe": df},
    result_format={"result_format": "SUMMARY"},
)
print(vd_result.success)

A Validation Definition is all you need when you just want the verdict for one dataset. A Checkpoint adds two things: it can group several Validation Definitions, and it can trigger Actions afterwards.

checkpoint = context.checkpoints.add(
    gx.Checkpoint(
        name="orders_checkpoint",
        validation_definitions=[validation_definition],
        result_format={"result_format": "SUMMARY"},
    )
)

checkpoint_result = checkpoint.run(batch_parameters={"dataframe": df})
print(checkpoint_result.success)

The batch parameters you pass to checkpoint.run apply to every Validation Definition inside it. So group definitions together only when they accept the same kind of parameters. A monthly definition expects year and month; handing it dataframe produces an error about the keys not being valid.

Using the result to control a pipeline

The reason to have a Checkpoint is to make a decision. The simplest decision is to stop:

if not checkpoint_result.success:
    print(checkpoint_result.describe())
    raise SystemExit(1)

Exiting with a non-zero status code makes schedulers such as Airflow, cron wrappers or CI systems mark the step as failed, which stops downstream steps from consuming bad data. That is the moment where data testing stops being a report and becomes a safety mechanism.

Data Docs: a report people can read

Validation results are data, and most people would rather read a web page. Data Docs are static websites GX generates from your Expectations and validation results. They show each suite as a readable list of rules (using your description text), and each validation run as a pass or fail table with the offending values. You can open them in a browser without any server, and you can share them by publishing the folder to a web host.

Data Docs need somewhere to live, which means a File Data Context. Create one:

import great_expectations as gx

context = gx.get_context(mode="file", project_root_dir="./my_project")

If the folder ./my_project/gx does not exist, GX creates it. Otherwise it loads what is there. Mode "file" without a root directory looks for the nearest existing project, or creates gx/ in the current directory.

The folder layout

After you create a File Data Context, your project contains:

my_project/
  gx/
    great_expectations.yml
    expectations/
    validation_definitions/
    checkpoints/
    plugins/
    uncommitted/
      config_variables.yml
      validations/
      data_docs/local_site/

The pieces matter. great_expectations.yml is the main configuration file: it lists the stores and also records your Data Sources, assets and batch definitions. The expectations, validation_definitions and checkpoints folders hold your suites and other definitions as JSON files. You commit all of these to git, and a colleague who clones the repository gets your whole setup.

The uncommitted folder is for things that should not go into git: the secrets file config_variables.yml, the stored validation results, and the generated Data Docs. GX writes a .gitignore that excludes it. Validation results and Data Docs can contain real data values, so treat them as sensitive.

Building and opening the docs

The UpdateDataDocsAction is the normal way to rebuild the site after each run. You can also trigger a build by hand:

context.build_data_docs()
context.open_data_docs()

build_data_docs regenerates the site under gx/uncommitted/data_docs/local_site/, and open_data_docs launches it in your default browser. The home page lists suites and recent validation runs. Click a run to see each Expectation with a green tick or red cross, the observed value, and the unexpected values.

A limitation worth knowing: since version 1.13.0 (February 2026) the only place GX can write stores and Data Docs is a local or network file system. The old S3, GCS, Azure and database store backends were removed. If you need your docs on a shared website, you copy or sync the folder yourself, for example to a protected bucket or an internal web server. The official Data Docs guidance also states that only the base_directory key of the site configuration is supported.

Actions: doing something with the result

An Action runs after a Checkpoint's validations finish and reacts to the result. You list Actions when creating the Checkpoint. They execute in order, once per Checkpoint run.

Version 1 of GX has no default Actions. In the old API, storing results and updating documentation happened automatically. Now if you want the Data Docs to refresh after a run, you must add UpdateDataDocsAction yourself. Forgetting it is a classic source of "why is my report out of date".

The built-in Actions include UpdateDataDocsAction, SlackNotificationAction, MicrosoftTeamsNotificationAction, EmailAction, PagerdutyAlertAction, OpsgenieAlertAction, SNSNotificationAction and APINotificationAction, which posts to a URL. The official compatibility reference lists Email, Microsoft Teams, Slack and custom Actions as supported; treat the rest as available but less formally covered.

A Checkpoint with documentation and a Slack alert looks like this:

from great_expectations.checkpoint import (
    SlackNotificationAction,
    UpdateDataDocsAction,
)

checkpoint = context.checkpoints.add_or_update(
    gx.Checkpoint(
        name="orders_checkpoint",
        validation_definitions=[validation_definition],
        actions=[
            UpdateDataDocsAction(name="update_docs"),
            SlackNotificationAction(
                name="slack_on_fail",
                slack_token="${SLACK_BOT_TOKEN}",
                slack_channel="${SLACK_CHANNEL}",
                notify_on="failure",
                show_failed_expectations=True,
            ),
        ],
        result_format={"result_format": "COMPLETE"},
    )
)

The notify_on argument decides when a notification is sent. The accepted values are all, success, failure, critical, warning and info. The failure value fires on any failure regardless of severity. critical fires on a critical failure, or when an Expectation could not execute at all. warning fires when the worst failure is a warning, and info when the worst is informational. The highest severity present wins, so one critical failure in a run of mostly warnings still counts as critical.

Tokens never appear as literals in the code. The ${SLACK_BOT_TOKEN} placeholder is replaced at run time from an environment variable or the secrets file, which is the topic of the next section.

A note of realism: alerting well is harder than alerting at all. If every run posts a message, people mute the channel within a week. Start with notify_on="failure", include the failed Expectations so the message is actionable, and route the alert to the people who own the data source, not to a general channel.

Configuration, credentials and project settings

Your first configuration problem will be credentials. A connection string for a real database contains a password, and you must never put that password in a file that goes into git.

GX solves this with substitution variables. Anywhere a credential field accepts it, you write ${NAME} instead of the literal value. When GX loads the configuration it replaces ${NAME} with the value from one of two places: an environment variable called NAME, or an entry called NAME in gx/uncommitted/config_variables.yml. If both exist the environment variable wins. You can also pass a runtime_environment dictionary to get_context, which overrides both.

POSTGRES_CONNECTION_STRING: "postgresql+psycopg2://report_user:s3cret@db.example.com:5432/analytics"
SLACK_BOT_TOKEN: "xoxb-your-token"
SLACK_CHANNEL: "data-quality-alerts"

Because that file lives under uncommitted/, git ignores it. In containers and CI, prefer environment variables:

export POSTGRES_CONNECTION_STRING="postgresql+psycopg2://report_user:s3cret@db.example.com:5432/analytics"
python run_checkpoint.py

When a password contains a literal dollar sign, escape it as \$, or GX will try to substitute it and fail. GX also restricts substitution to fields that are designed for it, such as connection_string and tokens. If you try it elsewhere you get an error saying that only certain fields may use config substitution. For larger teams, GX can also read secrets from AWS Secrets Manager, Google Secret Manager and Azure Key Vault, which the Mid-level guide covers.

Common errors and how to read them

GX errors are usually specific, and the message usually contains the fix. Here are the ones beginners meet most, with the real text to match against.

GreatExpectationsError: GX Cloud has been shut down, so this no longer functions and will be removed in great_expectations 2.0. You asked for cloud mode, directly or by accident. The accident is usually leftover environment variables whose names begin with GX_CLOUD_, which make get_context() try cloud mode. Unset them, or call gx.get_context(mode="file") or mode="ephemeral".

MissingConfigVariableError: Unable to find a match for a config substitution variable. A ${NAME} placeholder has no value. Export the environment variable or add it to config_variables.yml. If the value itself contains a dollar sign, escape it as \$.

DataContextError: ExpectationSuite with name orders_suite was not found. The suite does not exist in the context you are using. Nine times out of ten you are in an ephemeral context that forgot it, or you loaded a different project. Run context.suites.all() to see what exists, and use mode="file" with project_root_dir so you always load the same project. The same message pattern exists for Validation Definitions and Checkpoints.

DataContextError: Cannot add ExpectationSuite with name orders_suite because it already exists. You re-ran setup code. Use add_or_update, or fetch with get. Data Sources produce a similar message: "Can not write the fluent datasource ... because a datasource of that name already exists".

ExpectationSuite 'orders_suite' has changed since it has last been saved. Please update with <SUITE_OBJECT>.save(), then try your action again. You edited a suite in memory and then used it. Call suite.save().

ExpectationSuite ... must be added to the DataContext before it can be updated. You built a suite object but never registered it. Call context.suites.add(suite) first.

TestConnectionError: Path: ... does not exist. The base_directory of a filesystem Data Source is wrong. Relative paths resolve from your current working directory, which differs between a notebook and a scheduled job. Use an absolute path in anything that runs unattended.

query must start with 'SELECT' followed by a whitespace. A query asset begins with WITH, a comment or some other keyword. Begin with SELECT, or move the logic into a view.

Batch parameters should only contain keys from the following set: ... which is not valid. You passed parameters the Batch Definition does not accept: day to a monthly definition, or dataframe to a table. Match the keys to the definition, and remember that all definitions in a Checkpoint share one set of parameters.

ModuleNotFoundError: No module named 'psycopg2' (or snowflake, pyodbc, pyspark, boto3). You are missing an extra. Install great_expectations[postgresql] or whichever extra matches. SQL Server additionally needs the Microsoft ODBC driver installed on the operating system, which pip cannot do.

A different class of problem produces no error at all. An Expectation passes on an empty batch. Always check element_count. Saved objects disappear between runs. You are in an ephemeral context. A regular expression Expectation behaves differently after upgrading to 1.23.2 on Snowflake. That release made Snowflake use substring matching like the other back ends, where it previously required the whole value to match. Anchor patterns with ^ and $ when you mean a full match.

Putting it all together

Let us build one small project from start to finish, the kind you could put in a repository and run every night. It validates a daily orders CSV, rebuilds the documentation, and fails the pipeline if the data is bad.

The folder looks like this:

orders_quality/
  data/orders_2024-06-01.csv
  run_quality.py
  requirements.txt

The requirements file pins the version:

great_expectations==1.23.2
pandas

The script uses a File Data Context so that the docs and results persist, and add_or_update everywhere so that it can be re-run:

import sys
import pandas as pd
import great_expectations as gx
from great_expectations.checkpoint import UpdateDataDocsAction

df = pd.read_csv("data/orders_2024-06-01.csv")

context = gx.get_context(mode="file", project_root_dir=".")

data_source = context.data_sources.add_or_update_pandas(name="orders_source")
asset = data_source.add_dataframe_asset(name="orders_df")
batch_definition = asset.add_batch_definition_whole_dataframe("whole_orders")

suite = gx.ExpectationSuite(name="orders_suite")
suite.add_expectation(gx.expectations.ExpectColumnToExist(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(
    column="customer", mostly=0.98,
    description="At least 98 percent of orders name a customer."))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(
    column="amount", min_value=0, max_value=10000,
    description="Order amounts are positive and below the approval limit."))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeInSet(
    column="country", value_set=["EG", "SA", "AE", "KW", "QA"],
    severity="warning"))
suite.add_expectation(gx.expectations.ExpectTableRowCountToBeBetween(min_value=100))
suite = context.suites.add_or_update(suite)

validation_definition = context.validation_definitions.add_or_update(
    gx.ValidationDefinition(name="orders_vd", data=batch_definition, suite=suite)
)

checkpoint = context.checkpoints.add_or_update(
    gx.Checkpoint(
        name="orders_checkpoint",
        validation_definitions=[validation_definition],
        actions=[UpdateDataDocsAction(name="update_docs")],
        result_format={
            "result_format": "SUMMARY",
            "unexpected_index_column_names": ["order_id"],
        },
    )
)

result = checkpoint.run(batch_parameters={"dataframe": df})
print(result.describe())

if not result.success:
    print("Data quality check failed; see gx/uncommitted/data_docs/local_site/index.html")
    sys.exit(1)

Read it as a story. The script loads the day's data, opens the project, and declares its Data Source, asset and batch definition using calls that are safe to repeat. It declares the rules, each with a business-facing description where it helps, and uses mostly for the one rule where small gaps are tolerated and severity="warning" for the country list, which we want to hear about but not block on. It registers the suite, links it to the batch definition in a Validation Definition, and wraps that in a Checkpoint whose only Action rebuilds the documentation. Finally it runs the Checkpoint, prints a description and exits with status 1 if anything failed.

Note one subtlety in this script: a warning-severity failure still makes result.success false, so it blocks here. That is deliberate for a beginner project; blocking and then choosing what to relax is safer than silently letting bad data through. When you later want warnings not to block, you would inspect severities in the result instead of the single flag.

Run it from a scheduler. A cron line such as 0 6 * * * cd /srv/orders_quality && ./venv/bin/python run_quality.py would run it at 06:00 daily. In Airflow the same script becomes a task, and the non-zero exit marks it failed; the Airflow guide shows how tasks fit into a larger workflow, and GX also publishes an official Airflow provider. If you run pipelines in containers, the Docker guide covers packaging this script with its dependencies.

Commit the gx/ folder, excluding what .gitignore already excludes, so teammates get the same suites and Checkpoints. Open gx/uncommitted/data_docs/local_site/index.html after a failing run and look at how the report names the failing rows by order_id.

Where does this sit in a larger pipeline? The official guidance suggests three natural places: at ingestion, to quarantine bad raw records before they spread; after transformation, to decide whether downstream jobs should run; and at delivery, to tell genuine business anomalies apart from data problems. For machine learning, the same idea applies at the point where training data is assembled. If you track experiments, put the data check before the run starts, so a failed check prevents a wasted training job; the MLflow guide shows how to record what data a run used, and Evidently covers a different question, whether live data has drifted from the training data. GX asks "is this data valid?"; drift tools ask "is this data different?". They complement each other.

What you can now do, and what comes next

If you worked through the Try it tasks, you can now do things that many working data teams still do not. You can explain what GX is for and why the old tutorials are wrong. You can install it with the right extras and confirm the version. You can explore a dataset with throwaway Expectations, collect the good ones into a suite, and change them safely. You can connect to a dataframe, a folder of files, or a database table and slice the data into batches. You can turn a suite into a Validation Definition and a Checkpoint, choose a result format, find the exact rows that failed, and make a pipeline stop when the data is bad. You can keep credentials out of git, build a readable Data Docs site, and read the common errors instead of guessing.

Before you move on, two habits are worth forming. First, treat Expectations as code that deserves review: a rule is a claim about the business, so someone who understands the business should agree with it. Second, treat failures as information. A check that nobody ever sees fail is either testing something trivial or not being run. A check that fails daily and is ignored is worse than no check. Aim for a small set of meaningful rules that people trust.

Natural next steps in this catalogue are the Delta Lake guide for storage that already enforces some constraints, the Spark guide for validating large datasets, and the Dagster and Prefect guides for orchestrators that can run a Checkpoint as one step. If you work on Databricks, see the Databricks guide.

Sources

• Great Expectations documentation home
• GX Core introduction
• GX overview
• Try GX
• Changelog and deprecation timeline
• Compatibility reference
• Glossary
• Install Python
• Install GX
• Install additional dependencies
• Create a Data Context
• Connect to data
• Connect to SQL data
• Connect to filesystem data
• Connect to dataframes
• Create an Expectation
• Test an Expectation
• Organize Expectation Suites
• Create a Validation Definition
• Run a Validation Definition
• Create a Checkpoint with Actions
• Run a Checkpoint
• Choose a result format
• Configure credentials
• Configure Data Docs
• GX in your data pipeline
• Expectation Gallery
• great-expectations on PyPI
• Source repository and releases

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Great Expectations. Mid-level reviews this plus its own material; Senior reviews all three. Everything here refers to the 1.x library called GX Core, checked against release 1.23.2.

The thirty-second answer

Great Expectations is an open-source Python library for testing data. You declare assertions called Expectations, such as "this column has no nulls" or "these values are unique", group them into a suite, and run them against a batch of data. The checks run where the data lives: pandas in memory, SQL pushed down to the database, or Spark. The output is a structured validation result you can act on, plus a generated Data Docs website.

Then the sentence that shows you have used the current version: "There is no command-line interface and no hosted cloud any more; everything is the Python API, and a Checkpoint's success flag is what a pipeline gates on."

The model

Two contexts to distinguish. An Ephemeral context lives in memory and is gone when the process ends. A File context persists configuration in a gx/ folder you can commit to git, apart from the uncommitted/ subfolder, which holds secrets, results and generated docs.

The code to have ready

An interviewer who says "show me" will usually want to see the smallest complete loop. Know this by heart.

import great_expectations as gx

context = gx.get_context(mode="ephemeral")
source = context.data_sources.add_pandas(name="src")
asset = source.add_dataframe_asset(name="orders")
batch_def = asset.add_batch_definition_whole_dataframe("whole")
batch = batch_def.get_batch(batch_parameters={"dataframe": df})

suite = context.suites.add(gx.ExpectationSuite(name="orders_suite"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"))

result = batch.validate(suite)
print(result.success)

And the production half:

vd = context.validation_definitions.add(
    gx.ValidationDefinition(name="orders_vd", data=batch_def, suite=suite))
cp = context.checkpoints.add(
    gx.Checkpoint(name="orders_cp", validation_definitions=[vd]))
res = cp.run(batch_parameters={"dataframe": df})
if not res.success:
    raise SystemExit(1)

Quick exploration without any setup uses the built-in source: context.data_sources.pandas_default.read_csv("file.csv") returns a batch you can validate one Expectation against. Installing looks like pip install great_expectations, with extras in quotes such as 'great_expectations[postgresql]'. Checking the version is gx.__version__. Rebuilding and viewing documentation is context.build_data_docs() and context.open_data_docs().

The classic traps and why they happen

Following 0.x material. The library was rewritten in 1.0 (August 2024). great_expectations init, context.sources, validators calling expect_... methods, BatchRequest, profilers and great_expectations checkpoint run are all gone. Candidates fall into this because search results and chat answers are full of the old API. The tell in an interview is someone describing a "validator" as the central object or mentioning the CLI.

Assuming GX Cloud exists. Cloud was shut down in release 1.18.0 (June 2026). get_context(mode="cloud"), a CloudDataContext, or leftover GX_CLOUD_* environment variables now raise GreatExpectationsError: GX Cloud has been shut down. Use mode="file" or mode="ephemeral".

Losing work to an ephemeral context. gx.get_context() with no argument returns a File context if it finds a project and an Ephemeral one otherwise. Run it from the wrong folder, or with a stale GX_HOME, and you silently get an empty in-memory project. Saved suites appear to vanish. The defence is an explicit mode="file" and project_root_dir.

Expecting default Actions. In 1.x no Actions run by default. Data Docs only update if the Checkpoint includes UpdateDataDocsAction.

Batch parameter mismatches. Every Validation Definition in a Checkpoint receives the same batch_parameters. A daily definition needs year, month and day; a dataframe definition needs dataframe. Mixing them gives "Batch parameters should only contain keys from the following set". Numbers should be integers; digit strings have been deprecated since 1.21.0.

Forgetting to save. Editing an Expectation object already stored in a suite needs expectation.save(), and editing the suite needs suite.save(). Otherwise you get "has changed since it has last been saved".

Passing on empty data. An Expectation evaluated over zero rows can pass. Check element_count in the result, and add a row-count Expectation.

Secrets in code. Connection strings with passwords belong in ${VARIABLE} placeholders resolved from environment variables or gx/uncommitted/config_variables.yml, never in committed files.

Expecting cloud storage for results. Since 1.13.0 stores and Data Docs can only be written to a file system. Reading data from S3 or GCS still works.

Common interview questions

What problem does Great Expectations solve? Data fails silently. A renamed column, a unit change or a half-empty export raises no exception in your code; it just produces wrong numbers downstream. GX lets you state what must be true about data, tests it automatically, reports exactly which rows broke which rule, and can stop a pipeline before bad data spreads. For ML, it protects training data, because a model trained on a broken column learns the breakage without complaint.

How is it different from unit tests? Unit tests check code with fixed inputs you control. GX checks real, changing data at run time. Code changes when you deploy; data changes whenever an upstream system does. They complement each other: tests tell you the transformation is correct, Expectations tell you today's input and output are sane.

Explain Expectation, Suite, Validation Definition and Checkpoint. An Expectation is one assertion. A suite groups them for one dataset. A Validation Definition pairs a suite with a batch definition, answering "test this data against these rules". A Checkpoint groups Validation Definitions and adds Actions, and is what you run in production. The separation lets one suite be reused against dev and prod tables, and one Checkpoint notify once for several datasets.

What is a Batch, and why not just validate the table? A Batch is a slice of an asset chosen by a Batch Definition: the whole table, one file, or a daily or monthly partition. Validating yesterday's partition instead of the whole table keeps queries fast and cheap and means you check new data when it arrives.

Where does the computation happen? Where the data is. Pandas data is computed in memory, SQL sources are translated into queries the database runs, and Spark runs on the cluster. That is why a SQL Data Source scales to large tables while a pandas batch is limited by RAM. Each validation issues queries, so cost scales with the number of Expectations and batches.

What is the difference between a File and an Ephemeral context? Ephemeral is in memory and disappears, good for notebooks and jobs that rebuild setup from code. File persists to a gx/ folder, good for building a project over time and committing suites to git. uncommitted/ is git-ignored and holds secrets, validation results and Data Docs.

What does mostly do? It is an argument on row-level Expectations giving the fraction of rows that must pass. mostly=0.95 tolerates up to five percent violations. Use it for known, acceptable mess, and keep it tight so a real regression still fails. It does not apply to whole-column statistics such as a mean check.

What are the result formats? BOOLEAN_ONLY, BASIC, SUMMARY (the Checkpoint default) and COMPLETE. Higher levels add counts, then unexpected value frequencies and row positions, then the full lists and a query to fetch failing rows. Detail helps debugging but enlarges output and includes real values, which matters for sensitive data.

How do you find the failing rows? Set unexpected_index_column_names in the result format to a key column such as order_id; the result then lists identifiers with their bad values. include_unexpected_rows can return up to 200 complete rows for row-level Expectation types.

What does "unexpected" mean in a result? Simply that a value violated the Expectation. It says nothing about rarity. unexpected_percent is the share of examined rows that broke the rule.

How would you make a pipeline stop on bad data? Run a Checkpoint and exit non-zero when result.success is false, so the scheduler marks the step failed and downstream steps do not run. A refinement is to use severity (critical, warning, info) and block only on critical failures.

What are Actions, and what is the catch? Actions run after validation: UpdateDataDocsAction, Slack, Microsoft Teams, email, PagerDuty, Opsgenie, SNS and an API action. The catch is that 1.x has no default Actions, so docs do not refresh unless you add the action. notify_on accepts all, success, failure, critical, warning and info.

How do you handle credentials? ${NAME} substitution in fields such as connection_string, resolved from environment variables first and gx/uncommitted/config_variables.yml second. Escape a literal dollar as \$. Give GX a read-only database user.

Do you need a server or a cloud account? No. It is a library under Apache 2.0, running in your own process. The hosted GX Cloud no longer works.

How do you install it for PostgreSQL? pip install 'great_expectations[postgresql]' inside a virtual environment, on Python 3.10 to 3.13. Install with the extra, because in 2026 a SQLAlchemy 2.1 release broke older GX versions on Python 3.11+ and the fix in 1.23.2 relies partly on caps applied through extras.

Can it read from S3 or Azure Blob Storage? Yes, reading data from S3, GCS and Azure Blob is supported through dedicated Data Sources and extras. Writing results and Data Docs to those stores is not supported any more; they go to a file system.

How does it fit with drift monitoring? GX asks whether data is valid against rules you wrote. Drift tools such as Evidently ask whether data has changed compared with a reference. A pipeline can use both: GX at ingestion and before training, drift monitoring on live traffic.

How would you introduce it to a team that has none? Start with the one table that hurt most. Write five to ten Expectations that reflect real business rules, run them nightly through a Checkpoint, send failures to one channel with the failing Expectations included, and review which rules fire. Tighten or loosen rules deliberately. Expand only once people trust the alerts.

Where do the Expectations come from? From the people who understand the data and from exploration: load a batch, try an Expectation, see what passes, and ask whether the bounds are a business fact or an accident of the sample. The automatic profilers of 0.x are gone.

What would you check first if a saved suite cannot be found? The type of context, since an ephemeral context has no memory. Then the folder it read (project_root_dir, GX_HOME, the working directory), then the suite list with context.suites.all().

What a strong answer covers that a weak one misses

Interviewers also listen for judgement. A strong candidate says where they would place checks (at ingestion, after transformation, at delivery), says that a rule nobody acts on is noise, and treats Expectations as reviewable claims about the business rather than numbers copied from last week's data. They mention mostly and severity as ways to keep a check useful instead of turning it off. They remember that results and Data Docs contain real values and treat them as sensitive.

Expect a short coding prompt too: "given this dataframe, write checks that would have caught this bug". Good habits under pressure: state the belief in words first, choose the Expectation family (missing, unique, range, set, pattern, row count), write the suite, and say what you would do when it fails, not just how you would detect it.

Mini scenarios

"A nightly job loaded 40 percent fewer rows than normal and nobody noticed for a day." Add ExpectTableRowCountToBeBetween with a floor from the business, run daily on the new partition, and block the pipeline on failure. Mention that a row-count range from last week's number is brittle; use a range the business agrees with and revisit it.

"A price column changed from riyals to fils." Row-level range checks might still pass. Add a statistic Expectation, such as ExpectColumnMeanToBeBetween or a median check, which catches a hundredfold shift. That is why column statistics sit next to row rules.

"The Checkpoint passes but the report looks empty." Check element_count, confirm the batch parameters select the intended slice, and confirm the Data Docs action is in the Checkpoint.

"Two teams want different rules for the same table." Use separate suites, each in its own Validation Definition, which can reuse one Batch Definition. Keep the gx/ project in version control so changes are reviewed.

Sixty-second self-test

Answer out loud, without scrolling up.

1. Name the four nouns that describe the data side and the four that describe the rules side.
2. Why is gx.get_context() with no argument risky, and what do you write instead?
3. What does mostly=0.9 mean, and on which kind of Expectation does it apply?
4. What are the four result formats, and which is the Checkpoint default?
5. Which Action must you add yourself to refresh Data Docs?
6. What happens if you pass {"dataframe": df} to a daily partitioned definition?
7. Where do you keep a database password?
8. Name two things removed in 1.x that old tutorials still use.
9. How do you make a scheduler mark a step as failed when data is bad?
10. Why should you install with an extra such as [postgresql]?

Answers: (1) Data Source, Data Asset, Batch Definition, Batch; Expectation, Suite, Validation Definition, Checkpoint. (2) It silently falls back to an ephemeral context; use mode="file" with project_root_dir. (3) At least 90 percent of rows must satisfy the rule; row-level Expectations. (4) BOOLEAN_ONLY, BASIC, SUMMARY, COMPLETE; SUMMARY. (5) UpdateDataDocsAction. (6) An error that the batch parameter keys are not valid. (7) In an environment variable or uncommitted/config_variables.yml, referenced as ${NAME}. (8) Any two of the CLI, GX Cloud, context.sources, validators with expect_ methods, profilers. (9) Exit non-zero when checkpoint_result.success is false. (10) So that dependency caps and drivers for your database are installed.

If you hesitated on more than three, reread the sections on the mental model, Checkpoints and errors before moving on.

Part one of three. Most beginner problems with Great Expectations come from five causes: following documentation for the old 0.x API, working in an ephemeral context without realising it, forgetting that nothing happens automatically in 1.x, writing rules copied from a sample instead of from the business, and running checks nobody reads. Start with the table, then the habits and the checklist. Everything is checked against GX Core 1.23.2.

Common errors at this level

The rest of this guide explains the mistakes behind the rows that cost the most time.

Mistake: following a tutorial for the wrong version

Symptom. You copy code from a blog post or a chat answer and get errors about missing attributes, unknown commands or removed classes. Nothing you try makes the examples work.

Cause. Great Expectations rewrote its API in version 1.0 in August 2024. Older material uses a command-line interface, context.sources, validator objects with expect_... methods, BatchRequest, profilers and data assistants, and a great_expectations/ project folder with config_version: 3.0. All are gone. There is also plenty of material about GX Cloud, which was shut down in June 2026.

Fix. Check the date and the code. Modern examples call gx.get_context(), use context.data_sources, instantiate classes from gx.expectations, and run Checkpoints with checkpoint.run(...). When in doubt, use the official docs at docs.greatexpectations.io, which render the current 1.x version, and print gx.__version__ to see what you have. The legacy 0.18 line reached end of life in October 2025, so do not start new work on it.

Mistake: not knowing which kind of context you have

Symptom. You define a suite on Monday, run a script on Tuesday and GX cannot find it. Or everything works in a notebook and fails in a scheduled job.

Cause. gx.get_context() with no arguments returns a File context only if it finds a gx/ folder, searching the path in the GX_HOME environment variable, then the current directory, then its parents. Otherwise it gives you an in-memory context that forgets everything when the process ends. A scheduler usually runs from a different working directory than your notebook, and a stale GX_HOME that points at a folder without great_expectations.yml does not raise an error; it just falls back to memory.

Fix. Be explicit. Use gx.get_context(mode="ephemeral") when you mean memory, and gx.get_context(mode="file", project_root_dir="/absolute/path/to/project") when you mean a saved project. Print type(context).__name__ at the top of a script while debugging; it says FileDataContext or EphemeralDataContext.

Mistake: expecting things to happen by themselves

Symptom. The Checkpoint runs but Data Docs show an old result, nothing is posted to Slack, or results are not stored where you looked.

Cause. In 1.x, Checkpoints have no default Actions. The old behaviour, where results were stored and Data Docs refreshed automatically, was removed. You must list each Action explicitly. Likewise, editing a saved object does not persist until you call save().

Fix. Always include UpdateDataDocsAction(name="update_docs") when you want a current report, add notification Actions deliberately, and call expectation.save() or suite.save() after editing. A quick self-check after creating a Checkpoint is to print the list of its actions.

Mistake: writing rules from one week of data

Symptom. A check you wrote last month starts failing for reasons nobody can explain, people learn to ignore it, and eventually someone switches it off.

Cause. The bounds came from the data you happened to load: a minimum of 44.99, a maximum of 312.40, a value set of exactly the countries seen that week. The rule describes the sample rather than the business, so any legitimate variation triggers it.

Fix. Ask of every Expectation, "is this a fact about the business or about my sample?" Take bounds from rules people can state ("no negative amounts, nothing above the approval limit"). For things that are mostly true, use mostly with a number you can defend. For things that matter less, set severity="warning". Write a description so that a product manager can read the rule in Data Docs and disagree with it.

Mistake: passing on nothing

Symptom. A validation reports success although the data is obviously wrong. Or the element_count in the result is 0.

Cause. The Expectation was evaluated over zero rows, because a file was empty, a filter or a partition selected nothing, or an upstream step wrote nothing. A rule like "no nulls in this column" is trivially true when there are no rows.

Fix. Always include a volume check in a suite: ExpectTableRowCountToBeBetween(min_value=...) with a floor the business accepts. Read element_count whenever a result looks too good. Confirm that the batch parameters select the slice you intend. A check that cannot fail is not a check.

Mistake: mixing up batch parameters

Symptom. An error saying "Batch parameters should only contain keys from the following set", or saying a part such as year must be specified.

Cause. Each Batch Definition accepts specific keys. A dataframe definition takes dataframe. A yearly definition takes year, a monthly one year and month, a daily one year, month and day. A Checkpoint passes one set of parameters to all its Validation Definitions, so a definition with different needs breaks the run. Passing digit strings like "2024" is deprecated since 1.21.0.

Fix. Pass integers: {"year": 2024, "month": 2}. Group into one Checkpoint only the Validation Definitions that take the same keys, and use separate Checkpoints otherwise. Keep a line in your README listing the parameters each Checkpoint expects.

Mistake: secrets in the repository

Symptom. A connection string with a password sits in a script or in great_expectations.yml, and later appears in a pull request or a chat.

Cause. It is the easiest way to get a demo working.

Fix. Use ${NAME} placeholders in credential fields such as connection_string and Action tokens. Put the values in environment variables, or in gx/uncommitted/config_variables.yml, which the generated .gitignore excludes. Environment variables win if both exist. Escape a literal dollar sign as \$. Use a database account with only SELECT on the tables you validate. If a secret has ever been committed, rotate it; deleting the file later does not remove it from history.

POSTGRES_CONNECTION_STRING: "postgresql+psycopg2://report_user:REPLACE_ME@db.example.com:5432/analytics"

Mistake: adding an Expectation object to two suites

Symptom. RuntimeError: Cannot add Expectation because it already belongs to an ExpectationSuite.

Cause. Expectation objects become attached to a suite and carry an identifier. Adding the same object to another suite would make two suites share one rule.

Fix. Write a small function that builds a fresh Expectation each time you need it, or copy the object with copy.copy(expectation) and set expectation.id = None. Functions are better: they make reuse explicit and keep shared rules in one place.

def order_id_rules():
    return [
        gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"),
        gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"),
    ]

for exp in order_id_rules():
    suite_a.add_expectation(exp)
for exp in order_id_rules():
    suite_b.add_expectation(exp)

Mistake: running heavy checks on huge data through pandas

Symptom. A job that was fast in testing runs out of memory or takes hours in production.

Cause. A pandas batch loads the whole dataset into memory. A SQL Data Source, in contrast, turns checks into queries that run inside the database, and Spark runs them on the cluster.

Fix. For large data use a SQL or Spark Data Source. Validate a partition (yesterday, this month) through add_batch_definition_daily or monthly instead of the whole table. Use the SUMMARY result format by default and COMPLETE only when debugging, since it returns full lists of bad values. Remember each Expectation costs queries, so keep suites focused.

Mistake: assuming failures will be noticed

Symptom. The Checkpoint fails every night and the pipeline carries on regardless.

Cause. A failed validation is only a result object. Nothing stops anything unless your code acts on it.

Fix. After checkpoint.run(...), inspect result.success and exit non-zero, or raise an exception, when it is false. In Airflow or a CI job that marks the step failed and blocks downstream steps. Add a notification Action so a person sees it.

result = checkpoint.run(batch_parameters={"dataframe": df})
if not result.success:
    print(result.describe())
    raise SystemExit(1)

Habits worth adopting on day one

Pin the version and install with extras. Put great_expectations[postgresql]==1.23.2 (with your own extra) in requirements.txt. GX releases about every two weeks and keeps a public deprecation timeline. The September 2026 SQLAlchemy 2.1 incident showed the value of pins and of installing with extras, which carry dependency caps.

Read deprecation warnings. They name the replacement and the removal version, usually 2.0.0. Fix them while the change is small: digit-string batch parameters, schema_name on table assets (use a schema-qualified table_name), string row conditions, context.get_datasource.

Explore before you commit. Use context.data_sources.pandas_default.read_csv(...) and throwaway Expectations to learn what normal looks like, then keep only the rules that state a belief you would defend.

Start small and run for real. Five to ten meaningful Expectations on your most painful table, run nightly, beat a hundred rules that never run. Add rules when a real incident shows you a gap; each incident is a free requirements document.

Name things consistently. Use names such as orders_suite, orders_vd and orders_checkpoint, so context.suites.all() reads like a catalogue. Put the dataset name first.

Make setup scripts repeatable. Use add_or_update for suites, Validation Definitions, Checkpoints and Data Sources, so running the script twice is harmless.

Write descriptions. A description string turns a technical check into something a colleague can read in Data Docs. Do this for every Expectation you would have to explain in a meeting.

Use severity deliberately. Mark the rules that must block a pipeline as critical (the default), and those that merely deserve attention as warning or info. Later you can notify or block on severity.

Treat results as sensitive. COMPLETE results, uncommitted/validations and Data Docs contain real values and rows. Do not commit them and do not publish them openly. For employers with data-residency obligations in the Gulf or Egypt, remember GX reads data where it sits and writes its reports to local or network files, so decide where those folders live as you would for any other file containing real records.

Commit the project, not the secrets. Commit gx/great_expectations.yml, expectations/, validation_definitions/ and checkpoints/. Leave uncommitted/ out. Review changes to suites in pull requests like any other change, because a changed rule is a changed claim about the business.

Run it in CI as well. Running your suites against a small fixture dataset with known defects proves that the Checkpoint still fails when it should. A suite that cannot fail is worse than none.

Mistake: changing a rule without telling anyone

Symptom. A failing check is "fixed" by widening its bounds, and two weeks later a real problem slips through the widened rule. Nobody remembers why the number changed.

Cause. Suites feel like configuration, so people edit them casually. But each Expectation is a claim about what the business considers valid, and loosening it changes what the pipeline lets through.

Fix. Keep suites in git and change them through pull requests. Write the reason in the commit message and, where it helps, in the Expectation's description. When a rule fails, first decide whether the data is wrong or the rule is wrong. Only in the second case should the rule change, and ideally someone who owns the data agrees. A short conversation with the upstream team often finds a real defect that widening the bounds would have hidden. Record who approved each change, even informally, so that the next person who sees the rule can trace why it says what it says and whom to ask before touching it again.

Practice cards

Use these as short drills. Do each from a blank file.

1. The six-row mess. Create a CSV with a duplicate ID, a null and a negative amount. Validate five Expectations against it with pandas_default.read_csv. Predict which fail before you run.
2. Suite from scratch. Build an ephemeral context, a dataframe asset and a whole-dataframe batch definition. Create a suite of five rules and print result.statistics.
3. Find the rows. Re-run with result_format={"result_format": "COMPLETE", "unexpected_index_column_names": ["order_id"]} and print the failing IDs.
4. Persist it. Switch to mode="file" with project_root_dir. Close Python, reopen, and fetch the suite with context.suites.get(name=...).
5. Gate a script. Wrap the suite in a Validation Definition and Checkpoint. Exit with status 1 on failure and confirm with echo $?.
6. Report. Add UpdateDataDocsAction, run twice with different data, and open the docs.
7. Break it on purpose. Add a suite twice; read an unknown suite; pass the wrong batch parameter keys. Match each message to its fix.
8. Secrets. Move a connection string to an environment variable and use ${...}. Confirm git does not show it.
9. Tolerance. Add mostly=0.9 to a null check on data with 8 percent nulls, then 15 percent. Explain both outcomes.
10. Severity. Mark one rule warning. Run a failing batch and work out which notify_on values would fire.

A short checklist

Before you call a GX setup finished, check each item.

• I know which Python version and GX version I am on, and both are pinned.
• I installed with the extras my data sources need.
• I chose mode="file" or mode="ephemeral" on purpose and used absolute paths for anything unattended.
• Setup code uses add_or_update so it can be re-run.
• Every rule states a belief about the business, and the important ones have a description.
• The suite includes a row-count check, so an empty batch cannot pass quietly.
• mostly and severity are set deliberately, not copied.
• The Checkpoint lists its Actions, including UpdateDataDocsAction if I want a report.
• The script exits non-zero when result.success is false, and someone receives the alert.
• Credentials use ${...} placeholders; uncommitted/ is git-ignored; the database user is read-only.
• Results and Data Docs are stored and shared as sensitive data.
• A bad fixture proves the Checkpoint can fail.

If you can tick every line, you are using Great Expectations the way the documentation intends, and you are ready for the Mid-level guide, which explains how validations are computed and how to run them in a production pipeline. Related guides in this catalogue include Airflow for scheduling a Checkpoint, Docker for packaging it, and Evidently for drift checks that complement validity checks.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Everything here targets GX Core 1.23, the open-source Python library. Where your older notes or blog posts disagree, trust this page: most of what is written online describes the 0.x API or GX Cloud, and neither exists in a usable form today.

Where this picks up

Each section starts with the problem it solves, and most end with a Try it you can run against a real table in a few minutes.

What a run really does

Beginner gave you the nouns. To use the tool well you need the verbs, because every surprising behaviour in production is a consequence of the order in which GX does things.

A Checkpoint receives batch parameters and expectation parameters and hands them to each of its Validation Definitions. Each definition asks its Batch Definition for a Batch: one concrete slice of the data, such as yesterday's partition. The Expectations in the suite do not touch data themselves. They declare which Metrics they need (a null count, a row count, a column minimum), the execution engine computes those Metrics where the data lives, and the Expectation compares the observed number with what you asked for.

The execution engine is chosen by the data source. A pandas data source uses PandasExecutionEngine, which means the batch is loaded into memory and Metrics are dataframe operations. A Spark data source uses SparkDFExecutionEngine and computes distributed. Every SQL data source uses SqlAlchemyExecutionEngine, which compiles Metrics into SQL and runs it in your warehouse. You rarely touch an engine directly, but this split is the single most useful fact for predicting cost and speed: on a SQL source GX sends queries and receives numbers, while on a pandas source the whole batch crosses the network into your process.

GX bundles Metrics into as few queries as it can, so a suite of thirty Expectations on one table is not thirty round trips. The bundling is also why a single expensive Expectation, such as a quantile or a distinct-value comparison over a huge column, shows up as one slow query rather than smearing its cost across the run.

Actions come last and run once per Checkpoint run, after every validation has finished. They receive the whole CheckpointResult. That ordering explains two behaviours people trip over. First, an alert never fires halfway through a run, so a long Checkpoint is silent until it ends. Second, in 1.x there are no default actions: nothing stores a result or builds Data Docs unless you add the Action that does it.

Choosing and managing the Data Context

Beginner showed gx.get_context() and left you to trust it. At work you need to know exactly which context you get, because the wrong one fails silently.

gx.get_context() with no arguments returns a File Data Context if it can find one, and an Ephemeral Data Context otherwise. It looks in the directory named by the GX_HOME environment variable, then the current directory, then the parent directories. An Ephemeral context lives only in memory; when the Python process ends, everything you added is gone.

The silent failure is the hard part. If GX_HOME points at a directory that has no great_expectations.yml, GX does not raise. Discovery falls back to an ephemeral context, your saved suites are simply absent, and the first symptom you see is DataContextError: ExpectationSuite with name orders_suite was not found. When saved objects "vanish" between runs, check three things in order: what type(context).__name__ says, what GX_HOME holds, and what the working directory of the job was. Cron jobs and container entrypoints often run from a different directory than your terminal.

The fix is to stop relying on discovery in anything that runs unattended:

import great_expectations as gx

# Durable, committed configuration: load the project you mean
context = gx.get_context(mode="file", project_root_dir="/opt/pipeline")

# Stateless job: build everything in code, nothing persisted
context = gx.get_context(mode="ephemeral")

mode="file" finds the nearest File context or creates a new gx/ folder in the working directory, which is convenient on a laptop and a hazard in production: a typo in the path yields a fresh empty project rather than an error. mode="ephemeral" is always in-memory, even when a gx/ directory exists.

There is one more trap, and it is new. GX Cloud was shut down in 1.18.0. If complete GX_CLOUD_* environment variables are present in your environment, even leftovers from an old project, a bare get_context() raises GreatExpectationsError: GX Cloud has been shut down. Passing an explicit mode avoids the issue, and unsetting the variables removes it.

Which context should you choose? A File context suits a team that wants suites reviewed in pull requests: the suites, Validation Definitions and Checkpoints are JSON and YAML files in git. An Ephemeral context suits a job that declares everything in code on each run and wants no shared mutable state, which also makes it the simplest choice inside a container. Both are legitimate; what matters is choosing deliberately.

SQL data sources and what gets pushed down

A SQL data source is where most production use lives, because validating data in place avoids moving it. You add one with a factory on context.data_sources:

ds = context.data_sources.add_postgres(
    name="warehouse",
    connection_string="${POSTGRES_CONNECTION_STRING}",
)
trips = ds.add_table_asset(name="trips", table_name="public.trips")
recent = ds.add_query_asset(
    name="recent_trips",
    query="SELECT * FROM public.trips WHERE fare_amount > 0",
)

The connection string is a ${...} placeholder, resolved at run time, so the secret never appears in the saved configuration. There are dedicated factories for Snowflake, BigQuery, Databricks SQL, Redshift, SQL Server, Fabric, AlloyDB, Aurora, Citus and Neon, plus add_sql for any SQLAlchemy URL and add_sqlite for local work. Each has an update_* and an add_or_update_* sibling, and context.data_sources.delete(name) removes one.

Two asset types, two different contracts. A table asset names a table, and you should schema-qualify it inside table_name, as in public.trips. The old schema_name argument is deprecated since 1.14.0 and removed in 2.0.0, so any tutorial that passes it is out of date. A query asset wraps a SELECT statement and treats its result as the table. The query must begin with SELECT followed by whitespace. A query that opens with WITH, a comment, or unusual casing fails with query must start with 'SELECT' followed by a whitespace. If you need a CTE, wrap it in a subquery or create a view and use a table asset. Trino data sources do not support query assets at all.

A cautious engineer asks what privileges all this needs. GX has no access control of its own; it uses the identity of the process. The right grant is a dedicated database user with SELECT on exactly the objects you validate and nothing else. A wrong schema or a missing grant shows up as Attempt to connect to table: "trips" failed because the test query failed. Ensure the table exists and the user has access to select data from the table, and the fix is on the database side, not in GX.

Since 1.23.0 the SQL execution engine is cached per data source rather than rebuilt on every call, which removes repeated setup cost in a process that validates several assets against one warehouse. Since 1.13.0, distinct-value set comparisons are pushed into the database instead of pulling values back to compare them in Python. The general rule holds: on SQL, the warehouse does the work, so the cost of a suite is the cost of the queries GX issues, and you pay it on every run.

There is one regex behaviour change to know about. As of 1.23.2, regex Expectations on Snowflake use substring semantics like every other backend. Previously Snowflake's REGEXP anchored to the whole value. A pattern such as [0-9]{5} that used to demand exactly five digits on Snowflake now passes for any value that merely contains five digits. If you mean a full match, anchor the pattern with ^ and $.

Files, regexes and batch definitions

The second big family is files: local directories, S3, GCS and Azure Blob Storage, read through pandas or Spark. The shape is the same as SQL, with a different asset vocabulary.

ds = context.data_sources.add_pandas_filesystem(
    name="files", base_directory="./data"
)
csv_asset = ds.add_csv_asset(name="taxi_csv")

one_file = csv_asset.add_batch_definition_path(
    name="jan", path="yellow_tripdata_sample_2019-01.csv"
)
monthly = csv_asset.add_batch_definition_monthly(
    name="monthly",
    regex=r"yellow_tripdata_sample_(?P<year>\d{4})-(?P<month>\d{2})\.csv",
)
batch = monthly.get_batch(batch_parameters={"year": 2019, "month": 1})

The factory names follow a pattern: add_pandas_filesystem, add_spark_filesystem, add_pandas_s3, add_spark_s3, add_pandas_gcs, add_spark_gcs, add_pandas_abs, add_spark_abs. Cloud variants take the bucket or container plus client options, for example add_pandas_s3(name=, bucket=, boto3_options={}). For S3 you need the s3 extra. Reading data from object storage is fully supported; it is only the metadata stores (results, suites) that no longer support it, which a later section returns to.

The regex in a monthly or daily definition is where careful work pays off. It must contain named groups called year, month and day, and for a filesystem data source it is matched against paths relative to base_directory. The named groups are what turn a filename into batch parameters: GX reads 2019 and 01 out of the filename and matches them against the integers you pass. A regex with unnamed groups, or one that matches the full absolute path, finds no files, and you get an empty-looking failure rather than a helpful message.

Directory assets, for Spark and S3-style layouts where a folder of part-files is one logical table, use add_directory_csv_asset(name=, data_directory=) with add_batch_definition_whole_directory(...), or with a daily definition keyed on a column.

What get_batch returns when you pass nothing

Calling get_batch() with no batch parameters returns the latest valid batch, ordered by the definition's sort_ascending setting. That is a convenient default in a notebook and a hazard in a pipeline. If today's file is late, "latest" quietly becomes yesterday's file, you validate stale data, and the Checkpoint passes. For anything scheduled, pass parameters explicitly and let a missing batch fail loudly.

Integers, not strings

Since 1.21.0, batch parameters are integers: {"year": 2024, "month": 2}. Digit strings such as "2024" still work but raise a DeprecationWarning and will be removed in 2.0.0. The error Batch parameters should only contain keys from the following set: {...} means you passed a key the definition does not use, such as day to a monthly definition. And 'day' must be specified in the batch parameters means a daily SQL definition got no day. Match the parameters to the partition granularity, every time.

Spark dataframes

For Spark, context.data_sources.add_spark(name=...) followed by add_dataframe_asset and add_batch_definition_whole_dataframe lets you pass {"dataframe": spark_df} at run time. Create the SparkSession yourself. GX's own get_or_create_spark_session() helpers are deprecated since 1.0.0, and the force_reuse_spark_context argument has been replaced by spark_config. Spark 4 has been supported since 1.19.0, with pyspark>=2.3.2,<4.2.

Suites as reviewed code

A suite that lives only in a notebook is a suite nobody maintains. The mid-level habit is to treat the suite as code: versioned, reviewed, and changed deliberately.

import great_expectations as gx

suite = context.suites.add_or_update(gx.ExpectationSuite(name="trips_suite"))

suite.add_expectation(
    gx.expectations.ExpectColumnValuesToNotBeNull(column="trip_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeUnique(column="trip_id")
)
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="fare_amount", min_value=0, max_value=500, mostly=0.99,
        description="Fares are non-negative and under 500, allowing 1% noise.",
    )
)

add_or_update is what makes a setup script safe to run twice. Plain context.suites.add(...) on a name that already exists raises DataContextError: Cannot add ExpectationSuite with name trips_suite because it already exists. The same pattern applies to Validation Definitions and Checkpoints.

Three save rules cause most of the confusion. Changing an Expectation that is already in a suite means modifying its attributes and calling expectation.save(). Changing the suite itself, for instance adding an Expectation, is saved with suite.save() when it is already persisted. And using a suite that has unsaved edits in a run fails with ExpectationSuite 'trips_suite' has changed since it has last been saved. Please update with <SUITE_OBJECT>.save(), then try your action again. The message is long but the instruction is exact.

Two further errors involve ownership. Adding the same Expectation object to a second suite raises RuntimeError: Cannot add Expectation because it already belongs to an ExpectationSuite. To reuse an Expectation, copy it with copy.copy(expectation) and set .id = None on the copy. And a suite that was never added to the context raises ExpectationSuite '<name>' must be added to the DataContext before it can be updated when you build a Validation Definition from it. Add it first.

Description and meta

Every Expectation accepts a description, which Data Docs renders in place of the generated sentence, and a meta dictionary for your own annotations. Write descriptions in business language: "Every trip has a pickup time" reads better to an analyst than "values must not be null". This is a cheap way to make Data Docs useful to people who do not read Python.

Suite parameters

Any argument of any Expectation can be a runtime placeholder, {"$PARAMETER": "max_fare"}, supplied when you run: expectation_parameters={"max_fare": 500}. This has worked in every Expectation argument since 1.5.2. It is the right tool when a threshold depends on the run, such as an expected row count derived from an upstream system, and the wrong tool for constants, which belong in the suite where a reviewer can see them.

Row conditions, severity and targeted checks

Real data has subsets that follow different rules. A discount column may only be non-null for promotional orders. A row condition restricts an Expectation to the rows that should obey it.

from great_expectations.expectations.row_conditions import Column

promo_only = (Column("order_type") == "promo") & (Column("country") == "EG")

gx.expectations.ExpectColumnValuesToNotBeNull(
    column="discount_code", row_condition=promo_only
)

Since 1.9.0 the documented API is these condition objects; the older string form is deprecated and removed in 2.0.0. The operators are ==, !=, >, <, >=, <=, .is_in([...]), .is_not_in([...]), .is_null() and .is_not_null(). Comparing with None is rejected, so use is_null().

The grammar has limits that reward knowing them early. Conditions must be an OR of ANDs, so "A and (B or C)" has to be rewritten as "(A and B) or (A and C)", and a nested OR inside an AND raises an error. An Expectation may carry at most 100 condition statements. A handful of Expectations ignore row conditions entirely and reject them: expect_column_to_exist, the table column-list and column-count Expectations, expect_query_results_to_match_comparison, and unexpected_rows_expectation. Finally, two Expectations that differ only in their row condition are distinct Expectations, so you can have the same check twice in one suite for different subsets.

Severity

Every Expectation has a severity: "critical" (the default), "warning" or "info". Severity lets one suite hold hard requirements and soft signals together, and lets the Checkpoint react differently to each.

gx.expectations.ExpectColumnValuesToMatchRegex(
    column="phone", regex=r"^\+?[0-9]{8,15}$", mostly=0.95,
    severity="warning",
    description="Most phone numbers are well-formed; a dip is worth a look.",
)

A failed warning or info does not have to page anyone, and, as a later section shows, the notify_on setting of an Action can be tied to severity. Reserve critical for checks whose failure means downstream consumers must not proceed.

Custom SQL and multi-source Expectations

The built-in catalogue covers 59 core Expectation classes, and most business rules fit. When one does not, you have two routes that stay inside the library's supported surface.

Subclassing for defaults

If you check the same thing in many places, a subclass presets the arguments and the description:

class ExpectValidPassengerCount(gx.expectations.ExpectColumnValuesToBeBetween):
    column: str = "passenger_count"
    min_value: int = 1
    max_value: int = 6
    description: str = "There should be between **1** and **6** passengers."

This is configuration reuse, not new logic. It keeps a rule defined once while many suites use it.

Custom SQL with UnexpectedRowsExpectation

When the rule is a query, UnexpectedRowsExpectation passes exactly when the query returns zero rows. The placeholder {batch} is replaced with the batch's selectable:

orphans = gx.expectations.UnexpectedRowsExpectation(
    unexpected_rows_query="""
        SELECT * FROM {batch}
        WHERE shipped_at IS NOT NULL AND paid_at IS NULL
    """,
    description="Nothing ships before it is paid for.",
)
suite.add_expectation(orphans)

This is the most flexible tool in the library, and a good answer to "GX cannot express my rule". It is also where cost can hide, since you wrote the query and the warehouse will run it on every validation. It does not accept row conditions, which is natural because the query already contains its own WHERE.

When it fails, you often want every failing row rather than a sample. The result format can include up to 200 rows, but ValidationDefinition.get_unexpected_rows(), added in 1.15.0, returns all failing rows with no cap:

from great_expectations.expectations import UnexpectedRowsExpectation

for evr in vd_result.results:
    if isinstance(evr.expectation, UnexpectedRowsExpectation) and not evr.success:
        rows = vd.get_unexpected_rows(
            evr.expectation, batch_parameters=vd_result.batch_parameters
        )

It works only for UnexpectedRowsExpectation; any other type raises Only UnexpectedRowsExpectation is currently supported. Because there is no cap, a bad day can return millions of rows into memory, so look at the counts in the result before you fetch.

Comparing two sources

ExpectQueryResultsToMatchComparison compares the result of a query against one data source with a query against another, taking base_query, comparison_data_source_name, comparison_query and an optional mostly. This is the tool for reconciliation: does the warehouse table agree with the source system it was loaded from. Keep both queries cheap and aggregated, such as counts and sums per day, rather than comparing rows.

Validation Definitions and Checkpoints

Beginner validated a batch against a suite by hand. Production needs two named, stored, reusable objects.

A Validation Definition binds exactly one Batch Definition to exactly one Suite. It is the smallest thing you can name, store and run: vd.run(batch_parameters=...) returns a suite-level result. A Checkpoint groups Validation Definitions, runs them with shared parameters, then fires Actions. The Checkpoint is the unit you schedule.

vd = context.validation_definitions.add_or_update(
    gx.ValidationDefinition(name="trips_daily_vd", data=bd_daily, suite=suite)
)

checkpoint = context.checkpoints.add_or_update(
    gx.Checkpoint(
        name="trips_checkpoint",
        validation_definitions=[vd],
        actions=[UpdateDataDocsAction(name="update_docs")],
        result_format={"result_format": "SUMMARY"},
    )
)

result = checkpoint.run(
    batch_parameters={"year": 2020, "month": 1, "day": 17}
)

The design has one consequence you must internalise: batch parameters are shared by every Validation Definition in a Checkpoint. If one definition is daily and another is monthly, a single {"year":..., "month":..., "day":...} dictionary is wrong for one of them and you meet the Batch parameters should only contain keys error. Group definitions that partition the same way into one Checkpoint, and use separate Checkpoints for separate granularities.

Reading the result and gating the pipeline

checkpoint.run() returns a CheckpointResult. Its .success is the overall verdict, .describe() gives a readable summary, .describe_dict() gives a dictionary you can ship elsewhere, and .run_results holds per-definition detail. The pipeline-gating pattern is short and worth writing once, in a shared helper:

result = checkpoint.run(batch_parameters=params)
if not result.success:
    print(result.describe())
    raise SystemExit(1)

A non-zero exit fails the orchestrator task, and downstream tasks do not run. This is the point of a Checkpoint: the quality decision becomes part of the pipeline's control flow, instead of a report someone may or may not read.

Decide deliberately what "success" should mean. result.success is false if any Expectation failed, including a warning. If you want only critical failures to stop the pipeline, inspect the per-Expectation results and their severity instead of gating on the boolean, and let the warning ones alert without blocking. The script at the end of this guide gates on the plain boolean for simplicity.

Actions and alerting

An Action is a component that reacts to a Validation Result. The built-ins live in great_expectations.checkpoint: UpdateDataDocsAction, SlackNotificationAction, MicrosoftTeamsNotificationAction, EmailAction, PagerdutyAlertAction (needs the pagerduty extra), OpsgenieAlertAction, SNSNotificationAction and APINotificationAction, which POSTs to a URL of your choosing. Of these, the compatibility reference lists Email, Microsoft Teams, Slack and custom Actions as officially supported.

from great_expectations.checkpoint import (
    SlackNotificationAction, UpdateDataDocsAction,
)

actions = [
    SlackNotificationAction(
        name="slack_on_fail",
        slack_token="${SLACK_BOT_TOKEN}",
        slack_channel="${SLACK_CHANNEL}",
        notify_on="failure",
        show_failed_expectations=True,
    ),
    UpdateDataDocsAction(name="update_docs"),
]

The token and channel are placeholders resolved from the environment, so the credentials are not stored in the Checkpoint file.

notify_on and severity

notify_on decides which outcomes trigger a notification, and since the severity work in 1.5.9 and 1.5.11 it understands severity. The values are:

The highest severity present wins, which is what makes tiering work. Two Slack Actions, one with notify_on="critical" pointed at an on-call channel and one with notify_on="warning" pointed at a team channel, give you paging for the serious failures and a quieter feed for the rest, from a single suite. Note the subtle clause in critical: an Expectation that failed to execute, for example because a column is missing, counts. A run that could not evaluate a check is treated as seriously as one that evaluated and failed.

A custom Action

When you need to send results somewhere GX does not cover, such as your own warehouse table, subclass ValidationAction:

from typing import Literal
from great_expectations.checkpoint import (
    ValidationAction, CheckpointResult, ActionContext,
)

class ShipResultToWarehouse(ValidationAction):
    type: Literal["ship_result_to_warehouse"] = "ship_result_to_warehouse"
    table: str

    def run(
        self,
        checkpoint_result: CheckpointResult,
        action_context: ActionContext | None,
    ) -> dict:
        payload = checkpoint_result.describe_dict()
        # write payload to self.table with your own client here
        return {"rows_written": 1}

The action_context carries the outputs of Actions that ran earlier in the same Checkpoint, so you can order Actions to depend on each other. GX supports the pattern; the domain logic and its failure handling are yours. Keep a custom Action small, and make it tolerant: an Action that raises because the warehouse was briefly unreachable should not hide the validation result it was trying to record.

Result formats and unexpected rows

The default result format for Checkpoints is SUMMARY. Choosing a different one is a cost decision as much as a diagnostic one, because it decides how much data travels back to Python and lands in stored results.

You set it on a Checkpoint, a vd.run(...), or a single batch.validate(...), and you can pass a dictionary with extra keys: {"result_format": "COMPLETE", "unexpected_index_column_names": ["trip_id"]}. That last key matters most in practice. By default a failing value tells you what was wrong but not which row. Naming your primary-key columns in unexpected_index_column_names makes the result identify the offending rows by key, which is what a data engineer needs to go and fix them.

Other keys worth knowing: return_unexpected_index_query, which produces a query that re-selects the failing rows and defaults to true only for COMPLETE; partial_unexpected_count, where 0 suppresses the partial lists; exclude_unexpected_values, which keeps raw values out of the result; and include_unexpected_rows, which returns up to 200 full rows for the map-style Expectations and UnexpectedRowsExpectation.

Results contain your data

This is the security point that surprises people. A COMPLETE result, include_unexpected_rows, and the Data Docs built from them contain raw values from your tables: real email addresses, real amounts. Treat the validation results folder and Data Docs as sensitive as the data itself. If the data is personal data under the rules that apply to your employer, for instance a Gulf or Egyptian organisation with residency obligations, decide where those artefacts may be stored before you turn on COMPLETE in a shared location. exclude_unexpected_values is the lever for keeping values out while retaining counts.

Size

COMPLETE on a column with millions of failures returns millions of values. The habit that scales is to default to SUMMARY, raise to COMPLETE for a targeted investigation on one Expectation, and pair it with unexpected_index_column_names so you learn which rows rather than shipping all of them.

The gx folder, credentials and secrets

With a File Data Context, configuration is a folder you commit. Knowing what is in it tells you what to review and what to protect.

gx/
  great_expectations.yml          # config_version: 4, commit this
  expectations/                   # suites as JSON
  validation_definitions/
  checkpoints/
  plugins/                        # added to sys.path for custom code
  uncommitted/                    # git-ignored
    config_variables.yml          # secrets and substitution variables
    validations/                  # validation results
    data_docs/local_site/

great_expectations.yml also holds fluent_datasources, which is where your data sources, assets and batch definitions are serialised. That is good news for review: adding a table asset is a readable diff. The uncommitted/ folder is excluded by the generated .gitignore, and it is where secrets and results belong. Do not remove that exclusion.

Since 1.20.0 a File context can be stood up on a read-only filesystem, which suits immutable containers, but results and Data Docs still need somewhere writable, so mount a writable path for uncommitted/ or redirect the stores.

Substitution

Credentials enter through ${VAR} placeholders. GX resolves them from environment variables first, then from gx/uncommitted/config_variables.yml, and a dictionary passed as runtime_environment overrides both. A literal dollar sign in a value must be escaped as \$. Only certain fields accept substitution, the credential-bearing ones such as connection_string and tokens; putting ${...} anywhere else raises a message beginning Only ... may use config substitution.

The two failures are:

• MissingConfigVariableError: Unable to find a match for a config substitution variable. The variable is undefined in this process. Check the environment of the job, not of your shell.
• A password containing $ that was not escaped, which produces the same error in a confusing way. Escape it as \$.

Secrets managers

For production you usually want the value to come from a secrets manager rather than a file. GX supports AWS Secrets Manager, GCP Secret Manager and Azure Key Vault through a secret| prefix convention, using the aws-secrets, gcp and azure-secrets extras. The AWS form, placed in config_variables.yml, looks like this:

my_conn: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|connection_string

The text after the trailing | is a key inside a JSON secret, and you may append :<version-uuid> to pin a version. The important warning is that if the key does not match, no substitution occurs, silently. Your connection string then stays as the literal text secret|arn:... and the failure appears later as a confusing connection error. When a secret-backed connection fails, print the resolved configuration first. Secrets-manager substitution also runs after the ordinary ${} substitutions, which is why ${AWS_REGION} inside the ARN works.

Data Docs and where results live

Data Docs are a static website generated from your suites and results: a readable page per suite and per run. They are the part of GX that non-engineers actually open.

context.add_data_docs_site(
    site_name="team_site",
    site_config={
        "class_name": "SiteBuilder",
        "site_index_builder": {"class_name": "DefaultSiteIndexBuilder"},
        "store_backend": {
            "class_name": "TupleFilesystemStoreBackend",
            "base_directory": "uncommitted/data_docs/team_site/",
        },
    },
)
context.build_data_docs(site_names="team_site")
context.open_data_docs()

Adding a site name that already exists raises InvalidKeyError: Data Docs Site team_site already exists in the Data Context., so use a new name or update the configuration.

The storage limit that shapes your architecture

Since 1.13.0 the only store backends left are the filesystem one, an in-memory one and an inline one. The S3, GCS and Azure Blob store backends and the database store backend were removed. The documentation states that GX writes Data Docs to the directory named by base_directory, and that configuring other keys is not supported. So suites, results and Data Docs can only be written to a local or networked filesystem. Reading data from object storage is unaffected; it is the metadata that is filesystem-bound. Do not follow older tutorials that configure TupleS3StoreBackend or similar: those classes no longer exist.

Teams that want results in shared storage have three workable designs:

1. A networked filesystem. Mount a shared volume so every job writes to the same place.
2. Sync after the run. Write locally, then copy uncommitted/ or the Data Docs folder to object storage with your own tooling as the last pipeline step.
3. A custom Action. Ship describe_dict() from the Checkpoint to a warehouse table, as shown earlier, and build dashboards there.

Data Docs themselves are just static files, so publishing them is an ordinary web-hosting problem. Put them on a protected internal web server or a bucket with authentication in front, because, as noted earlier, they can contain raw data values.

Running GX inside your pipeline

GX Core is a library, not a service. There is no server, no daemon and, since Cloud closed, no hosted interface. It runs inside whatever executes your Python: an orchestrator task, a container, a CI job.

Where to place checks

The official guidance on pipeline placement identifies three natural positions. Validate at ingestion to quarantine bad raw records before they spread. Validate after transformation to decide whether downstream steps should run. Validate at delivery to separate genuine business anomalies from data-quality problems. Most teams begin with the middle position, because it protects the tables other people query.

Airflow

The official provider is airflow-provider-great-expectations 1.0.0, which requires great-expectations>=1.7.0,<2 and Airflow >=2.1,<4; Airflow 2.9.0 and later is the officially supported orchestrator. It offers three operators: GXValidateDataFrameOperator, GXValidateBatchOperator and GXValidateCheckpointOperator. Only the Checkpoint operator supports File contexts, with context_type set to "ephemeral" or "file". Two caveats from the provider's documentation are stale against current GX. It still offers context_type="cloud", which raises the shutdown error, and it still mentions a mssql extra, which no longer exists after 1.13.0. For SQL Server install great_expectations[sql-server] directly. See the Airflow guide for the orchestration side.

dbt and other orchestrators

For dbt, the pattern in the official tutorial is to let dbt build the tables and then validate them through an ordinary SQL data source, orchestrated by Airflow. GX does not need to know dbt exists; the tables are just tables. With Dagster, Prefect, cron or a CI runner the recipe is identical: load the context, run a Checkpoint, check result.success, exit non-zero on failure. The Dagster guide and Prefect guide cover those runtimes. And when a table is validated on a schedule alongside versioned datasets, the DVC guide covers the data-versioning side of the same concern.

Containers and CI

A File context committed to git plus a base image with GX installed gives you a reproducible validator. Pin the version exactly, including extras, in your requirements: great_expectations[postgresql]==1.23.2. Supply credentials as environment variables at run time. In CI, running your suites against a small fixture database on every pull request catches a broken suite before it reaches production.

Performance and cost

Because SQL Expectations become queries, performance work is mostly the discipline of asking the warehouse fewer and cheaper questions. The levers, in order of leverage:

1. Validate partitions, not whole tables. A daily Batch Definition validates a day's rows instead of the lifetime of the table. For a table that grows, this converts a cost that rises forever into a constant.
2. Use SQL or Spark for large data, not pandas. A pandas data source loads the batch into your process's memory. That is fine for files that fit and a mistake for a billion-row table.
3. Choose the result format deliberately. SUMMARY by default, COMPLETE only when investigating.
4. Watch for expensive single Expectations. Quantiles, distribution divergence and custom SQL with joins dominate run time. Find them by timing, and move them to a slower schedule or a sample.
5. Do not validate the same table repeatedly. If three suites read the same table, consider one Checkpoint so the Metrics are bundled.

Known limits are worth having in your head: include_unexpected_rows caps at 200 rows; get_unexpected_rows() has no cap; a row condition holds at most 100 statements; and Databricks queries are automatically split when they would exceed 256 parameters.

Concurrency has a version requirement. Building validators on multiple threads had bugs around batch identity and deadlocks that were fixed in 1.23.1 and 1.23.2, so require 1.23.2 or later if you validate in threads.

What it costs

GX Core is free under Apache 2.0, and there is no paid tier any more. The bill is the compute it drives, meaning warehouse query time or Spark cluster time, plus the storage of results and Data Docs. On a warehouse that charges per query or per bytes scanned, a suite on a large unpartitioned table is a real line item. Measure it: run a suite once, look at the warehouse's own query history for what GX issued, and multiply by your schedule.

Testing suites and upgrading safely

The thing GX tests is your data, but the suite itself is code that can be wrong. Two kinds of tests keep it honest.

Test that a suite fails when it should. Build a tiny dataframe that violates each rule on purpose, validate it with batch.validate(suite), and assert that the specific Expectations fail. A suite whose rules never fail on bad data is worse than no suite, because it reads as assurance. This test belongs in CI beside your other tests, and it runs in an Ephemeral context with no warehouse at all.

Test that a suite passes on known-good data. Keep a small representative sample, commit it or generate it, and assert the suite passes. This catches a threshold tightened by mistake.

Because batch.validate(exp) tests a single Expectation, you can iterate on thresholds interactively, then copy the settled version into the suite.

Upgrades

GX releases a minor version roughly every two weeks, so upgrades are frequent and small. The discipline that keeps them boring:

• Pin exactly in production, including extras, and upgrade on purpose.
• Read the changelog line for each release. Since 1.23.1 the changelog also carries a deprecation timeline table listing what will be removed in 2.0.0.
• Run your suites in CI against the new version before rolling it out. Behaviour changes do happen: the Snowflake regex change in 1.23.2 is one, and the SQLAlchemy 2.1 incident is another.
• Treat deprecation warnings as a to-do list. Digit-string batch parameters, schema_name, string row conditions, context.get_datasource, and the DBFS data sources are all flagged for removal in 2.0.0, with replacements you can adopt today.

Teams that have 0.x projects face a rewrite rather than an upgrade, since 0.18 reached end of life on 1 October 2025. Checkpoints, suites and data sources must be recreated in the 1.x model, and there is no automated converter to rely on.

Debugging beyond the basics

Most GX problems fall into a small number of families, and recognising the family saves time.

"It worked yesterday and nothing changed." Suspect the environment. A new release with a behaviour change, a dependency that moved (SQLAlchemy 2.1 is the recent example), or an expired credential. Compare gx.__version__ and pip freeze between a passing and failing environment first.

"My objects disappeared." An ephemeral context, as covered earlier. Print the context type, GX_HOME and the working directory.

"Nothing matched." A file regex with no named groups, a base_directory that resolves differently in the job (TestConnectionError: Path: <abs path> does not exist), or batch parameters that select no batch. Use absolute paths in jobs.

"Already exists" errors. Setup scripts written with add instead of add_or_update, including the data-source variant Can not write the fluent datasource warehouse because a datasource of that name already exists in the data context. Use context.data_sources.get(name) or an add_or_update_<type> call.

"Import errors." ModuleNotFoundError: No module named 'psycopg2' and its cousins mean a missing extra. Install great_expectations[postgresql], and for SQL Server remember the operating-system ODBC driver as well.

A Checkpoint that is slow. Turn on debug logging for the great_expectations logger, find the slow statements, and apply the performance levers from the previous section.

When you must investigate a failure, the most productive sequence is: read result.describe(), raise the result format for the one failing Expectation, add unexpected_index_column_names to find the rows, then query those keys in the source. For custom SQL, get_unexpected_rows() gives the complete set.

Coding agents

Since 1.21.0, GX ships agent skills for coding assistants. python -m great_expectations skills install installs three skills (gx-configure-data-source, gx-configure-expectations, gx-configure-checkpoint) into .agents/skills/ and .claude/skills/, with --target, --project-root, --symlink and --force flags, and python -m great_expectations skills list shows them. It will not overwrite directories GX did not install, and there is no uninstall command, so delete the folders by hand. Re-run the install after upgrading GX so the text matches the version. This is the only command-line surface the package has. The 0.x great_expectations init and checkpoint run commands were removed in 1.0.

Putting it all together

Here is a compact but realistic pipeline step. It loads a committed File context, validates yesterday's partition of a warehouse table, alerts by severity, ships the result to a history table, and fails the task when validation fails. Setup happens once, in a separate script, and this is the script the orchestrator runs every day.

import datetime as dt
import sys

import great_expectations as gx

yesterday = dt.date.today() - dt.timedelta(days=1)

context = gx.get_context(mode="file", project_root_dir="/opt/pipeline")
checkpoint = context.checkpoints.get("trips_checkpoint")

result = checkpoint.run(
    batch_parameters={
        "year": yesterday.year,
        "month": yesterday.month,
        "day": yesterday.day,
    },
    expectation_parameters={"max_fare": 500},
)

print(result.describe())

sys.exit(0 if result.success else 1)

The gx/ folder beside it holds the suite reviewed in pull requests, a Validation Definition on a daily partition of the table, and a Checkpoint with three Actions: a Slack notification with notify_on="critical" to the on-call channel, a second with notify_on="warning" to the team channel, and UpdateDataDocsAction. A custom Action writes describe_dict() to a history table. The warehouse user is read-only. The version is pinned, and CI runs the suite-fails-on-bad-data tests before any release.

That arrangement uses almost everything in this part: explicit context mode, partitioned batches, integer batch parameters, severity tiers, parameters at run time, result shipping despite the filesystem-only stores, and a gate that blocks downstream work when validation fails. For the data-versioning side, pair it with the lakeFS guide, and for the orchestration side the Airflow guide.

Where you are now

You can now choose a context deliberately and predict which one you will get, connect SQL and file sources with the right batch definitions, write suites as reviewed code with mostly, row conditions, severity and runtime parameters, and reach for custom SQL when a rule does not fit the catalogue. You can build Validation Definitions and Checkpoints, route alerts by severity, ship results where the filesystem-only stores cannot go, keep credentials out of git, control the cost of every run, test the suites themselves, and upgrade on a two-week cadence without surprises.

The next part moves from using GX well to owning it as a platform for a team: its failure modes, how it behaves on very large data, the trust model around results that contain raw values, multi-team isolation, upgrade and migration governance, and when a different tool is the right answer.

Sources

• GX Core documentation home
• GX Core introduction
• Changelog and deprecation timeline
• Compatibility reference
• Glossary
• Create a Data Context
• Install additional dependencies
• Install agent skills
• Connect to SQL data
• Connect to filesystem data
• Connect to dataframes
• Create an Expectation
• Organize Expectation Suites
• Row conditions
• Define a custom Expectation class
• Use SQL to define a custom Expectation
• Define a multi-source Expectation
• Create a Validation Definition
• Retrieve all unexpected rows
• Create a Checkpoint with Actions
• Choose a result format
• Create a custom Action
• Configure credentials
• Access secrets managers
• Configure metadata Stores
• Configure Data Docs
• GX in your data pipeline
• Airflow provider for Great Expectations
• great-expectations on PyPI

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than level. At this level interviewers stop asking what an Expectation is and start asking what you would do when a pipeline is failing at seven in the morning. Fast review first, scenarios and trade-offs after. Senior reviews all three. Everything assumes GX Core 1.23.

Where this picks up

Foundations in one screen

Great Expectations Core is a Python library for declarative data tests. You describe what good data looks like as Expectations, it evaluates them where the data lives (in memory for pandas, distributed for Spark, pushed down as SQL for warehouses), and it produces structured results, triggers Actions, and renders Data Docs. It is a library inside your pipeline, not a server. GX Cloud was shut down in 1.18.0, and the 0.x command line was removed in 1.0.

If you can say that paragraph without hesitation, you have the thirty-second answer. The rest of this file is what separates an engineer who has run it from one who has read about it.

Scenario questions

The Checkpoint passed, but the dashboard is wrong. What do you check first?

A strong answer starts by asking whether the Checkpoint validated the data the dashboard reads. The likeliest culprit is get_batch() or a Checkpoint run with no batch parameters, which returns the latest valid batch. If today's partition was late, "latest" was yesterday, yesterday passed, and the stale data was never tested. Next, check whether the suite is too permissive: a mostly of 0.8 where the business needs 0.999, or no freshness or row-count Expectation at all. Then check that the Expectation tests the table the dashboard uses rather than a staging copy. Only after these do you suspect GX itself.

A weak answer says "the suite must be missing a check" and stops. The strong answer names a mechanism and a way to confirm it, such as printing result.batch_parameters from the stored result to see which partition was actually validated.

Saved suites keep disappearing between runs. Why?

Because you are getting an Ephemeral Data Context. gx.get_context() returns a File context only if it finds one, searching GX_HOME, then the current directory, then parent directories. A job running from a different working directory finds nothing, and a stale GX_HOME pointing at a directory without great_expectations.yml does not raise; it falls back to ephemeral silently. The symptom is DataContextError: ExpectationSuite with name ... was not found. The fix is to stop using discovery in unattended jobs and call gx.get_context(mode="file", project_root_dir="..."), or to build everything in code with mode="ephemeral" and add_or_update. A strong answer also says to print type(context).__name__ first.

The pipeline is slow and the warehouse bill rose. Where do you look?

Start with what GX actually sends. On a SQL data source it compiles Metrics into queries and runs them in the warehouse, so the cost is the warehouse's query history filtered to the validation user. Then apply the levers in order. Is the Batch Definition whole-table when it could be daily? A whole-table definition validates the lifetime of the table on every run. Is any one Expectation dominating, such as a quantile, a distribution comparison or a custom SQL query with a join? Is the result format COMPLETE on a column with many failures? Is the same table validated by several Checkpoints that could share one? And is the data source pandas on a table too big for memory? A strong answer leads with partitioning because it changes the cost curve; a weak answer says "add more compute".

A notification channel has been muted because it is noisy. How do you fix it?

This is a severity problem, not a Slack problem. If every Expectation has the default critical severity, a cosmetic failure alerts like a missing key. The fix is to triage: mark structural and key checks critical, drift and format checks warning or info, and configure two Actions with different notify_on values, critical to on-call and warning to a team channel. The highest severity present decides which fires, so one suite can serve both audiences. Also revisit mostly: an Expectation that fails on one bad row in ten million should either tolerate it with a documented threshold or be a different check. A strong answer adds that people will only un-mute a channel that has become trustworthy.

After upgrading GX, a Snowflake regex check started passing rows it used to fail. What happened?

In 1.23.2, regex Expectations on Snowflake changed to substring semantics like every other backend. Previously Snowflake's REGEXP anchored to the whole value. A pattern meant as a full match now succeeds if the value merely contains a match. The fix is to anchor patterns with ^ and $. The broader lesson interviewers listen for is that you read changelogs, pin versions, and run suites in CI against a new version before rolling it out. Mentioning the SQLAlchemy 2.1 incident in the same release, and installing with extras so the sqlalchemy<2.1 cap applies, is a strong addition.

Someone wants Data Docs and results stored in S3. Can GX do that?

Not directly. Since 1.13.0 the cloud-storage and database store backends were removed. Only the filesystem, in-memory and inline backends remain, and the documentation says Data Docs are written to the directory given as base_directory, with other configuration keys unsupported. Reading data from S3 is unaffected. A strong answer then offers the three workable designs: a networked filesystem mount, a final pipeline step that syncs the folders to object storage, or a custom Action that ships describe_dict() to a warehouse table. A weak answer produces a TupleS3StoreBackend snippet from an old tutorial, which no longer exists.

A Checkpoint run raises an error mentioning GX Cloud, though you never used Cloud.

The message is GreatExpectationsError: GX Cloud has been shut down. It fires when a bare get_context() sees complete GX_CLOUD_* environment variables, a leftover great_expectations.conf cloud section, or cloud arguments. The fix is to unset the variables and pass an explicit mode="file" or mode="ephemeral". The same error appears if an Airflow operator is given context_type="cloud".

You need to check that shipped orders were all paid for. The catalogue has no such Expectation.

Use UnexpectedRowsExpectation with a query that selects the violating rows: it passes exactly when the query returns zero rows, and {batch} stands in for the batch's table. If the rule compares two systems, ExpectQueryResultsToMatchComparison takes a base query, a comparison data source and a comparison query. A strong answer volunteers the costs and risks: you wrote the SQL, so you own its performance; it accepts no row conditions; and it runs with the connection's privileges, so the user should be read-only. To retrieve every failing row rather than a 200-row sample, call get_unexpected_rows() on the Validation Definition, adding that it returns all rows and so can be large.

A run failed, and you need to tell the data team which rows are bad.

Raise the result format for the failing Expectation and set unexpected_index_column_names to the table's primary key. A failing value on its own says what was wrong, not where. With key columns named, the result identifies rows by key, and with COMPLETE the unexpected_index_query re-selects them. A strong answer also flags that COMPLETE results and include_unexpected_rows contain raw data, so the results folder and Data Docs must be protected like the data.

Trade-off questions

File Data Context or Ephemeral?

A File context stores suites, Validation Definitions and Checkpoints as JSON and YAML in a committed gx/ folder, so changes are reviewed as diffs and shared across jobs. The cost is that you must manage the folder's location, and a mistake silently gives you an empty context. An Ephemeral context rebuilds everything from code each run, so there is no shared mutable state and no path to misconfigure, but you re-declare everything and cannot browse saved state. Choose File when several people curate suites and reviews matter; choose Ephemeral for a single job or a notebook. A strong answer names the failure mode of each and does not pretend one is universally better.

Validate at ingestion, after transformation, or at delivery?

The official guidance describes all three. At ingestion you quarantine bad raw records before they spread, at the price of checking data whose shape you do not control. After transformation you decide whether downstream steps run, which protects the tables other people query. At delivery you separate true business anomalies from data-quality failures. Most teams start in the middle because it has the best return, then add ingestion checks for the sources that burn them. A strong answer ties the choice to where a bad row is cheapest to stop.

Fail the pipeline on any failure, or only on critical ones?

Gating on result.success is simple and strict: any failed Expectation, of any severity, stops the run. Gating on severity lets warnings alert without blocking, which keeps throughput high but requires you to inspect individual results and trust your severity assignments. The right answer depends on the consumer. A regulatory report should block on anything; an internal exploratory table should block only on structural failures. A strong answer says that the decision must be written down, because otherwise severities drift.

pandas, Spark, or SQL execution?

SQL pushes computation to the warehouse, so only numbers come back, and cost is the warehouse's. It is the natural choice when the data already lives there. Spark distributes work for large files and lakehouse tables. Pandas loads the whole batch into process memory, which is fine for small files and risky beyond RAM. A strong answer connects the engine to batch size and notes that partitioned Batch Definitions bound the size in every case.

Write a custom Expectation, or use a SQL one?

For a rule that is one query, UnexpectedRowsExpectation is usually enough and stays inside the supported surface. Subclassing an existing Expectation to preset arguments and a description is right when you repeat a rule across suites. Anything that needs new Metric logic is a deeper investment that you should justify, since you then maintain it across a release every two weeks. A strong answer reaches for the simplest mechanism that works.

Pin the version, or float?

Pin exactly, with extras, in production. GX releases roughly every two weeks, behaviour changes do occur, and the SQLAlchemy 2.1 incident showed that a dependency can break you overnight. Floating is acceptable in a scratch environment that you expect to rebuild. The mitigation that makes pinning sustainable is a scheduled, small upgrade with your suites run in CI against the new version.

Commands and API calls to have ready

What a strong answer covers that a weak one misses

• A mechanism. A weak answer says "it checks data quality". A strong one says Metrics are computed by an engine, and SQL is pushed down.
• Where the failure is silent. The ephemeral fallback, the latest-batch default, secrets substitution that does not substitute, and Cloud leftovers.
• Cost as a first-class concern. Partitioning, result format, expensive Expectations, and the warehouse query history.
• Honesty about what is gone. GX Cloud, the old CLI, profilers, and the cloud store backends. Naming them as removed, rather than describing them as if they work, tells the interviewer you have used the current version.
• Security awareness. Results and Data Docs contain raw values, the database user should be read-only, and credentials belong in ${VAR} placeholders or a secrets manager.
• Version discipline. Pinning, changelog reading, and the deprecations heading for 2.0.0.

Common questions with short answers

What does mostly do? It sets the fraction of rows, between 0 and 1, that must pass for a row-level Expectation to succeed. It turns "never" into a documented tolerance.

What is the difference between a Validation Definition and a Checkpoint? A Validation Definition links one Batch Definition to one Suite and runs them. A Checkpoint groups Validation Definitions, shares parameters across them, and fires Actions afterwards.

When do Actions run? Once per Checkpoint run, after all validations finish, and they can read the earlier Actions' outputs through action_context.

Why did my Checkpoint fail with a batch-parameters error? Batch parameters are shared by every definition in the Checkpoint, so a daily and a monthly definition cannot share one dictionary. Group by granularity.

How do you keep a password out of git? Use a ${VAR} placeholder in a credential field, resolved from the environment or uncommitted/config_variables.yml, or use a secrets-manager reference.

How do severities interact with notifications? notify_on accepts critical, warning and info as well as all, success and failure, and the highest severity present wins.

Does GX have a command line? Not for project work. The only command surface is python -m great_expectations skills install and skills list, which install agent skills. Everything else is Python.

How does GX fit with Airflow? Through the official provider, with three operators; only the Checkpoint operator supports File contexts. Airflow 2.9.0 and later is the supported orchestrator. See the Airflow guide for the orchestration side.

How does it relate to drift monitoring? GX asserts rules you write about a batch. For statistical monitoring of model inputs and outputs over time, the Evidently guide covers the neighbouring tool.

Sixty-second self-test

Answer each aloud, then check yourself against the sections above.

1. Which context do you get from a bare get_context(), and how can it silently differ between your laptop and a cron job?
2. What does get_batch() return with no parameters, and why is that dangerous in a schedule?
3. Why must batch parameters match across every Validation Definition in a Checkpoint?
4. What decides which notify_on Action fires when an Expectation fails?
5. Name two places where a result can leak raw data, and one control for each.
6. Where can Data Docs be written in 1.23, and what are your three options for central storage?
7. What changed for Snowflake regex Expectations in 1.23.2, and how do you keep a full-match meaning?
8. Write the query shape for an UnexpectedRowsExpectation and say when it passes.
9. Why should you install great_expectations[snowflake] rather than the bare package?
10. Name three things that used to be part of GX and are now gone.

Part two of three. At this level your suites run and your Checkpoints pass, so the problems change character. They are no longer "why won't this import" but "why is the warehouse bill higher", "why did this pass on stale data", and "why did the alerts stop meaning anything". Start with the error table, then the practices in order of leverage, then practice cards and operational recipes. Everything targets GX Core 1.23.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

Two of these deserve a sentence more. The secrets one is nasty because nothing errors at substitution time: the literal text is passed to your database driver, and the failure looks like a bad connection string. And the get_batch()-with-no-arguments hazard, which is not an error at all, is covered below because it produces a green Checkpoint over stale data.

The practices that pay off most, in order of leverage

1. Pass batch parameters explicitly in anything scheduled

A call with no batch parameters returns the latest valid batch. On a day when the new partition is late, "latest" is yesterday's, yesterday passes, and the pipeline proceeds on data nobody checked. Pass the exact partition, computed from the run's logical date, so that a missing batch is a failure you see. This one habit prevents the most expensive class of silent error.

2. Validate partitions, not whole tables

A daily or monthly Batch Definition turns a cost that grows with the table into a cost that is constant per run. It also makes failures specific: "the 17th failed" is actionable, and "the table has some bad rows somewhere" is not. Whole-table definitions have their place for small dimension tables and periodic full audits, but they should be a choice.

3. Use an explicit context mode in unattended jobs

Call gx.get_context(mode="file", project_root_dir=...) or mode="ephemeral". Reliance on discovery is what produces the vanishing-suites incident, and a stray GX_CLOUD_* variable is what produces the shutdown error. Explicit code has neither problem.

4. Gate the pipeline on the Checkpoint result

A check that produces a report nobody reads is not a control. Run the Checkpoint, test result.success, and exit non-zero so the orchestrator stops the downstream tasks. Decide in writing whether warnings should block, and if they should not, inspect severities rather than the boolean.

5. Triage with severity and route alerts by it

Keep critical for failures that mean downstream consumers must not proceed. Use warning and info for drift and hygiene. Configure separate Actions with notify_on="critical" and notify_on="warning". The alert channel that people trust is the one that only pages for real problems.

6. Put a documented tolerance on row-level checks

mostly=0.99 with a description that says why is more honest than an unexamined 100% that fails on one stray row and is then quietly loosened by whoever is on call. The description appears in Data Docs, so it becomes documentation.

7. Keep credentials in placeholders and the warehouse user read-only

${VAR} placeholders in credential fields, values in the environment or uncommitted/config_variables.yml, or a secrets-manager reference. Give the validation user SELECT on exactly the objects it validates. GX has no access control of its own, so the database user is the control.

8. Test the suite itself

Feed a deliberately bad dataframe to the suite in CI and assert that the intended Expectations fail, and feed a known-good sample and assert success. A suite that cannot fail is false comfort, and this test needs no warehouse.

9. Treat results as sensitive

COMPLETE results, include_unexpected_rows and Data Docs contain raw values. Keep them in protected storage, use exclude_unexpected_values where counts are enough, and decide where they may live before enabling them, which matters for employers with data-residency obligations in the Gulf or Egypt.

10. Pin versions and read the changelog

A release arrives roughly every two weeks. Pin exactly, with extras, run your suites in CI against the candidate version, and read the deprecation timeline. Upgrades done this way are small and dull.

Practice cards

Each card is a small exercise you can do on a real project in ten to twenty minutes.

Operational recipes

Recipe: a safe, idempotent setup script

Run once per environment, or at the start of each ephemeral job. Every call is an add_or_update, so re-running is harmless.

import great_expectations as gx
from great_expectations.checkpoint import UpdateDataDocsAction

context = gx.get_context(mode="file", project_root_dir="/opt/pipeline")

ds = context.data_sources.add_or_update_postgres(
    name="warehouse",
    connection_string="${POSTGRES_CONNECTION_STRING}",
)
asset = ds.add_table_asset(name="trips", table_name="public.trips")
daily = asset.add_batch_definition_daily(name="DAILY", column="pickup_datetime")

suite = context.suites.add_or_update(gx.ExpectationSuite(name="trips_suite"))

vd = context.validation_definitions.add_or_update(
    gx.ValidationDefinition(name="trips_daily_vd", data=daily, suite=suite)
)
context.checkpoints.add_or_update(
    gx.Checkpoint(
        name="trips_checkpoint",
        validation_definitions=[vd],
        actions=[UpdateDataDocsAction(name="update_docs")],
    )
)

If the asset or batch definition already exists on a re-run you will meet the already exists message for those names, so in a real script fetch them with ds.get_asset("trips") and asset.get_batch_definition("DAILY") when present.

Recipe: the gate

import datetime as dt
import sys

import great_expectations as gx

day = dt.date.today() - dt.timedelta(days=1)
context = gx.get_context(mode="file", project_root_dir="/opt/pipeline")
checkpoint = context.checkpoints.get("trips_checkpoint")

result = checkpoint.run(
    batch_parameters={"year": day.year, "month": day.month, "day": day.day}
)
print(result.describe())
sys.exit(0 if result.success else 1)

Recipe: find the failing rows by key

Raise the format for the investigation, not for every run:

result = checkpoint.run(
    batch_parameters={"year": 2020, "month": 1, "day": 17},
    result_format={
        "result_format": "COMPLETE",
        "unexpected_index_column_names": ["trip_id"],
    },
)

Then read the failing keys from the result, or use the unexpected_index_query it returns to re-select the rows in your warehouse client.

Recipe: a regression test for a suite

import great_expectations as gx
import pandas as pd


def test_suite_rejects_bad_fares():
    context = gx.get_context(mode="ephemeral")
    ds = context.data_sources.add_pandas(name="mem")
    asset = ds.add_dataframe_asset(name="trips_df")
    bd = asset.add_batch_definition_whole_dataframe("whole")

    suite = gx.ExpectationSuite(name="trips_suite")
    suite.add_expectation(
        gx.expectations.ExpectColumnValuesToBeBetween(
            column="fare_amount", min_value=0, max_value=500
        )
    )

    bad = pd.DataFrame({"fare_amount": [10, -5, 900]})
    batch = bd.get_batch(batch_parameters={"dataframe": bad})
    assert not batch.validate(suite).success

Recipe: ship the result somewhere queryable

The results store is filesystem-only, so use a custom Action to build a history you can chart. The Action receives the CheckpointResult; describe_dict() gives a dictionary to write to your own table. Keep it small and have it log and continue on failure, so a brief warehouse outage does not hide the validation outcome. Order matters: action_context carries earlier Actions' outputs, so put it after the ones it depends on.

Recipe: secrets from a manager

For AWS, install great_expectations[aws-secrets] and put the reference in uncommitted/config_variables.yml:

warehouse_conn: secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:prod_db_credentials|connection_string

Reference it from the data source as ${warehouse_conn}. If it fails, the first check is that the JSON key after the final | exists, since a mismatch means silent non-substitution.

Recipe: an upgrade in five steps

1. Read the changelog entries between your pinned version and the target, including any "Compatibility" line and the deprecation timeline.
2. Create a branch that changes the pin, extras included.
3. Run your suite regression tests and a dry run of each Checkpoint against a non-production copy.
4. Run with -W error::DeprecationWarning once, and list what will break in 2.0.0.
5. Roll out, and re-run python -m great_expectations skills install if you use agent skills, so the installed text matches.

Why these errors keep happening

The error table lists symptoms, but the same few root causes sit behind most of them, and learning the causes lets you diagnose an error you have never seen.

Silent fallbacks. GX tries to be helpful when it cannot find something, and the helpfulness is the danger. A missing project becomes an ephemeral context. A missing batch parameter becomes "the latest batch". A secrets reference that does not match becomes literal text. In each case nothing raises at the point of the mistake, and the failure surfaces later and somewhere else. The defence is to make intent explicit at the point of use: name the context mode, pass the partition, and print resolved configuration when a connection fails. When something behaves oddly, ask first which default GX might have quietly chosen for you.

Two objects with one name. Suites, Validation Definitions, Checkpoints, data sources and Data Docs sites all live in named collections, and most of the "already exists" and "was not found" errors are a script disagreeing with the collection about whether a name is taken. The add_or_update family exists to make setup idempotent, and the get family exists for lookups. Use add only when you genuinely expect the name to be new.

State that was edited but not saved. A suite is an object in memory and a document in a store, and they can diverge. The long "has changed since it has last been saved" message is GX refusing to run a suite that differs from its stored copy, which is the safe behaviour. Save, then run.

Granularity mismatches. Batch Definitions partition data at a specific grain, and Checkpoints share one dictionary of batch parameters across every definition they contain. Most batch-parameter errors are a day key meeting a monthly definition. Keep one grain per Checkpoint and the class of error disappears.

Version drift. Much of what goes wrong after an upgrade is the ecosystem rather than GX, as the SQLAlchemy 2.1 incident showed, and the rest is deliberate behaviour change such as the Snowflake regex semantics. Pinning exactly and testing the candidate version in CI turns these from outages into pull-request comments.

Leftovers from older material. Tutorials, blog posts and model answers from chat assistants overwhelmingly describe the 0.x API or GX Cloud. Symptoms include great_expectations init not existing, context.sources being missing, a TupleS3StoreBackend class that cannot be found, and a Cloud error on a project that never used Cloud. When a snippet does not work, check its date and its API style before debugging your own code. If it uses batch_request, Validator-centred workflows, profilers, or evaluation_parameters, it is describing a version that is gone.

Reading a failure like a diagnosis

When a Checkpoint fails, resist the urge to open the data first. Read the failure in layers, because each layer rules out a class of cause cheaply.

Layer one: did it run at all? An exception before any validation means configuration: context, connection, missing variable, missing extra. These errors are loud, and the table above maps the message to the fix.

Layer two: did it validate the right thing? Look at which batch was validated. If the run used a default batch rather than the intended partition, the verdict is about the wrong data. Confirm the batch parameters in the stored result before you believe a pass or a fail.

Layer three: which Expectations failed, and how badly? Read result.describe(). A failure with a large unexpected_percent is a real data problem, while a failure that narrowly misses a threshold may be a tolerance problem, where mostly needs thought rather than the data needing repair. An Expectation that failed to execute, rather than evaluating and failing, usually means a column or table changed shape, which is a schema problem worth treating as its own category.

Layer four: which rows? Only now raise the result format for the failing Expectation and add unexpected_index_column_names, so the result gives you keys. Query those keys in the source system. The question to answer is whether the rows are wrong at source, wrong after your transformation, or correct but newly unusual.

Layer five: what changed? Compare with the last passing run: a new upstream release, a backfill, a timezone change, a new category value. Data problems usually have an origin event, and finding it is worth more than fixing the rows.

This sequence takes minutes once it is a habit, and it keeps you from the common waste of querying a large table before you know which partition and which rule you are chasing.

Keeping a suite healthy over time

A suite is not finished when it passes. It decays in three ways, and a little maintenance stops each.

Thresholds go stale. A maximum fare that was generous last year may be too tight after a price change, and a row-count range written for one market is wrong after launching in another. Revisit numeric thresholds when the business changes, and keep the reason in the description so the next person knows whether a number is a rule or a guess. Runtime parameters are the right home for values that track something outside the suite.

Noise accumulates. Every check that fails and is ignored teaches people to ignore the next one. Once a quarter, list the Expectations that failed most often and ask of each whether it should be fixed at the source, relaxed with a documented mostly, downgraded to warning or info, or deleted. Deleting a check nobody acts on is a healthy act: it removes warehouse cost and restores the signal of the rest.

Coverage drifts. New tables and columns appear without new Expectations. A light process helps: when a table is added to the pipeline, the pull request that adds it also adds a minimal suite with the structural checks (columns exist, keys are unique and non-null, row count is non-zero) and one freshness or volume check. Grow from that floor rather than aiming for exhaustive suites that nobody maintains.

It also helps to keep the suite small enough to read. A suite with five hundred Expectations that no one has reviewed is a liability, while forty well-described ones that a teammate has read is an asset. Prefer fewer, more meaningful checks, and push generic structure checks into a reusable subclass so they are defined once.

Two cost traps

Re-validating the same table. It is easy for three teams to build three suites that each scan the same large table daily. The warehouse does three times the work for overlapping questions. Put shared checks in one suite and one Checkpoint where you can, and note that the SQL engine is cached per data source since 1.23.0, so several assets on one data source in a single process avoid repeated setup.

Expensive Expectations on the critical path. Quantile, divergence and join-based custom SQL checks can dominate a run. If a check is valuable but slow, move it off the critical path: run it in a separate Checkpoint on a slower schedule, or on a sampled or smaller partition, and let the fast structural checks gate the pipeline. The gate should be quick enough that nobody is tempted to bypass it.

Habits worth building

• Name things for humans. Suites, definitions and Checkpoints that say what they protect, such as trips_daily_vd, are easy to find in a failing run.
• Write descriptions in business language. They become the sentences in Data Docs.
• Keep one Checkpoint per granularity. It keeps batch parameters consistent and failures legible.
• Put thresholds in the suite and run-specific values in parameters. A constant hidden in a script is invisible to reviewers.
• Review the suite in pull requests. With a File context, a new Expectation is a readable diff, which is the main advantage of the format.
• Keep the validation user read-only and the results folder protected.
• Delete Expectations that nobody acts on. A check whose failure is always ignored costs money and trust.

A short checklist

Before a suite goes to production, confirm that:

• the context mode is explicit, and the project_root_dir is absolute;
• every scheduled run passes integer batch parameters computed from its logical date;
• each Expectation has a considered severity, and row-level ones a documented mostly where needed;
• alerts route by notify_on to the right channels;
• a Checkpoint failure stops the downstream tasks;
• credentials are placeholders or secret references, and the database user is read-only;
• UpdateDataDocsAction is present if Data Docs matter to anyone;
• results and Data Docs are stored somewhere protected, with a plan for central history;
• the version is pinned with extras, and CI runs the suite regression tests;
• a teammate other than the author has read the suite.

If you can tick all ten, you are using GX at the level this part describes. The next part covers owning it as a platform: how it behaves on very large data, what it means to run it for several teams, and when another tool is the better answer. For the neighbouring topics, see the Airflow guide for scheduling, and the Delta Lake guide for the tables you may be validating.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further and adds the work you own when Great Expectations is a platform that other teams depend on, rather than a library you call from your own pipeline.

Up to this point the questions were about making validations work. From here they change: what happens to the validations of forty teams when one of them ships a bad suite, who is allowed to change a suite that gates a finance report, what a release of the library can break overnight, and how you explain to leadership what a data-quality tool costs when the tool itself is free.

Where this picks up

I am starting with the architecture, because every later decision depends on what kind of thing Great Expectations actually is.

Architecture and failure modes

GX Core is a Python library. There is no server, no daemon, no scheduler and no user interface. Since the hosted product was shut down in version 1.18.0, nothing about GX runs unless your own process imports it. That single fact shapes everything a platform owner does. You are not operating a service with an uptime; you are operating a convention, a set of versioned artifacts and a pinned dependency, and the "platform" is the discipline that keeps forty teams on the same convention.

A validation run has a small number of moving parts. Your process creates a Data Context. The context loads configuration, which names data sources, suites, validation definitions and checkpoints. A checkpoint run asks each Validation Definition for its batch, hands the Expectations to an Execution Engine, and the engine turns them into Metrics. For SQL data those metrics become queries that run inside the warehouse; for Spark they become distributed jobs; for pandas the whole batch is loaded into the memory of your process. The results flow back, get written to a results store, and then the Actions run.

Read the diagram as a list of ways a run can go wrong, because the platform owner's job is to make each of them visible. There are four distinct failure categories, and they must never be conflated.

The first is a validation failure: the job ran, the data was examined, and the data does not meet an Expectation. This is the tool working. checkpoint_result.success is false, and what happens next is a policy decision.

The second is an execution failure of a single Expectation: the Expectation could not be evaluated, for example because a column does not exist or a query timed out. GX records it in the result as an Expectation that failed to execute, and the severity logic treats it as critical for notify_on="critical". This is usually a schema change upstream, and it deserves a different response from a data-value failure.

The third is an infrastructure failure: the data source is unreachable, credentials expired, the filesystem is full or read-only, or the process ran out of memory. Here there may be no result at all. The most dangerous version of this category is the silent one, where the job that runs the checkpoint never ran, so no failure and no success is recorded.

The fourth is a configuration failure that does not raise. The documented example is a stale GX_HOME environment variable pointing at a directory without a great_expectations.yml. Discovery does not fail; it silently falls back to an Ephemeral context. Your saved suites appear to have vanished, and a job that looks up a suite by name fails with ExpectationSuite with name <name> was not found, but a job that builds everything in code may quietly validate nothing you intended. A related silent case is secrets-manager substitution: if the key in a secret reference does not match, no substitution happens and the literal reference string is used as the value.

The design consequence is that a green pipeline is not evidence that data was validated. A platform worth trusting emits a positive signal that a checkpoint ran, against which batch, with how many Expectations evaluated, and alerts on the absence of that signal. Hold the principle: absence of failure is not presence of success.

The two context models

The most consequential architectural choice you make for other teams is which Data Context they use.

A File Data Context persists configuration in a gx/ folder: great_expectations.yml, plus folders for expectations, validation definitions and checkpoints, and an uncommitted/ folder that holds secrets, validation results and Data Docs and is git-ignored by the generated .gitignore. Its strength is that the configuration is a reviewable artifact. Suites are JSON files. A change to a threshold is a pull request with a diff that a data owner can read. Since 1.20 a File context can be loaded from a read-only filesystem, which suits immutable containers, although results and Data Docs then need a writable path elsewhere.

An Ephemeral Data Context lives in memory and is gone when the process ends. Its strength is that there is no shared mutable state: every job builds what it needs with add_or_update calls and the job is self-describing. Its weakness is that configuration is scattered through Python code, so reviewing a suite means reading code, and nothing survives a run unless you export it.

My recommendation for a platform is a hybrid that many mature teams converge on. Suites, validation definitions and checkpoints live as a File context in git, are reviewed like code, and are loaded in production with get_context(mode="file", project_root_dir=...). Anything that varies by environment, meaning connection strings and tokens, comes from ${VAR} substitution and never from the committed file. The project root is baked into the job image, so the running job cannot drift from what was reviewed.

import sys
import great_expectations as gx

def run(project_root: str, checkpoint_name: str, batch_parameters: dict) -> int:
    # Explicit mode: never let discovery decide. A stale GX_HOME or a missing
    # gx/ folder must fail loudly, not fall back to an empty ephemeral context.
    context = gx.get_context(mode="file", project_root_dir=project_root)
    if type(context).__name__ != "FileDataContext":
        print("refusing to run: expected a FileDataContext", file=sys.stderr)
        return 2
    checkpoint = context.checkpoints.get(name=checkpoint_name)
    result = checkpoint.run(batch_parameters=batch_parameters)
    print(result.describe())
    return 0 if result.success else 1

if __name__ == "__main__":
    sys.exit(run(sys.argv[1], sys.argv[2], {"year": 2026, "month": 9, "day": 30}))

The wrapper does three things a platform must insist on. It uses an explicit mode. It asserts the context type, so the "my suites disappeared" failure becomes a hard error. And it turns the result into an exit code, which is the contract every orchestrator understands. Note that mode="file" creates a new gx/ folder in the working directory if none is found, so passing project_root_dir matters: it targets a specific folder rather than wandering.

What the shutdown of GX Cloud taught

If your organisation adopted GX Cloud, the shutdown in 1.18.0 is a case study in platform risk. Constructing a CloudDataContext, or calling get_context(mode="cloud"), now raises GreatExpectationsError: GX Cloud has been shut down, so this no longer functions and will be removed in great_expectations 2.0. The classes remain importable as shells through 1.x, and removal is scheduled for 2.0.0. The replacement is mode="file" or mode="ephemeral". Any internal documentation that mentions Data Health dashboards, workspaces, the GX Agent or a cloud access token is obsolete, and some pages of the official documentation and of the Airflow provider still mention Cloud and are stale too. A platform owner should grep internal runbooks for these terms once, and treat the lesson as a principle: a dependency on a hosted control plane is a dependency on someone else's roadmap, and open-source library usage with artifacts in your own git is the variant that survives.

Scaling: which component breaks first

Because GX is a library that pushes work elsewhere, "scaling GX" is really scaling what it drives. The question to ask is which component saturates first, and the answer depends on the backend.

Pandas breaks first, and breaks in memory

With a pandas data source, the batch is loaded into the memory of the validating process. The first thing to break is RAM. A 4 GB Parquet file does not stay 4 GB once decoded, and the process that validates it must also hold metric computation intermediates. The symptoms are an out-of-memory kill with exit code 137 in a container, or swapping that makes a job ten times slower without failing. The fix is structural, not a larger machine: validate partitions rather than whole tables, or move the validation to Spark or to the warehouse. A reasonable platform policy is that pandas is for dataframes that a pipeline step already holds in memory, for samples, and for tests; anything that is a table of record is validated where it lives.

SQL pushes the cost into the warehouse

With a SQL data source, GX compiles metrics into queries and runs them in the warehouse, bundling metrics into as few queries as it can. Since 1.13 comparisons of distinct-value sets are pushed into the database, and since 1.23.0 the SQL execution engine is cached per data source rather than rebuilt on each call. This is excellent for memory and awful for naive cost estimates, because the cost of a checkpoint is now the cost of the queries it issues: roughly, the number of Expectations that need separate scans, times the size of the batch, per run. A suite of eighty Expectations against a five-billion-row table, run hourly over the whole table, is a bill.

The scaling lever is the Batch Definition. A whole-table batch scans everything every run. A daily partitioned batch, defined with add_batch_definition_daily(name="DAILY", column="event_ts") and run with {"year": 2026, "month": 9, "day": 30}, scans one day, assuming the warehouse can prune on that column. Whether it can is a property of your table design, not of GX. If the partition column is not the warehouse's clustering or partitioning key, the query still scans everything and you have gained nothing. The platform owner's first review question for any large-table suite is therefore not "which Expectations" but "which batch definition, and does the warehouse prune on it".

table_asset = datasource.add_table_asset(name="events", table_name="analytics.events")

# Whole table: scans everything every run. Acceptable for small dimension tables.
full = table_asset.add_batch_definition_whole_table("FULL_TABLE")

# Daily: one partition per run, so cost tracks the increment, not the history.
daily = table_asset.add_batch_definition_daily(name="DAILY", column="event_ts")

# Integers, not digit strings: strings are deprecated since 1.21.0, removal in 2.0.0.
batch = daily.get_batch(batch_parameters={"year": 2026, "month": 9, "day": 30})

Notice the schema-qualified table_name. The schema_name argument of add_table_asset has been deprecated since 1.14.0 with removal in 2.0.0, so a platform template should already use the qualified form.

Result size is the quiet one

Teams discover late that results can be large. The result format controls this. BOOLEAN_ONLY returns only success; BASIC adds counts and a partial list of unexpected values; SUMMARY, the checkpoint default, adds partial counts and partial index lists; COMPLETE adds the full unexpected_list, the full unexpected_index_list and the index query. On a column with ten million failures, COMPLETE produces a result that is itself a large dataset, stored on disk, loaded by Data Docs, and possibly posted to an alert channel. A platform default of SUMMARY, with COMPLETE only on request for a specific investigation, prevents an entire category of incidents. The option include_unexpected_rows is capped at 200 rows; get_unexpected_rows() on a Validation Definition, available since 1.15.0, has no cap and for an UnexpectedRowsExpectation can return every failing row, so call it only when you mean to.

Security: the trust model, secrets and the supply chain

GX has no access control of its own. There is no login, no role, no permission model inside the library, because there is no server to enforce one. It runs with the identity of the process that imports it, and everything it can read, that process can read. This is the most important sentence in the security section, because it moves every control to places you already operate: the database grants, the job's service account, the filesystem permissions and the network.

Least privilege lives in the database

The correct identity for a validation job is a read-only database user with SELECT on exactly the objects it validates, and nothing else. GX issues queries; it does not need to write to validated tables, and a suite that can be edited by a team member must not be able to turn into a write path. Watch one specific feature: UnexpectedRowsExpectation runs SQL that a suite author wrote, with {batch} replaced by the batch's selectable. The query is author-controlled text executed under the job's identity. With a read-only user, the worst a malicious or careless query can do is read what that user can read, or run an expensive scan. With a privileged user it can do anything the grants allow. Treat custom SQL in suites as code that needs review, and make the grant the backstop.

Query assets have a related guard: the SQL must start with SELECT followed by whitespace, otherwise creation fails with query must start with 'SELECT' followed by a whitespace. This is a usability rule more than a security boundary, since a SELECT can still be expensive, so do not rely on it as a control.

Credentials: three tiers

The documented mechanism is ${VAR} substitution in configuration values, resolved from environment variables or from gx/uncommitted/config_variables.yml. Environment variables take precedence, and the runtime_environment argument of get_context overrides both. A literal dollar sign is escaped as \$. Only fields defined as substitutable accept the syntax, such as connection_string and tokens; using it elsewhere raises an error that says that field does not allow config substitution.

For a platform, I recommend three tiers and a rule for choosing.

• Tier one, environment variables injected by the runner. The job's orchestrator or secret operator puts the connection string in the environment, and the committed great_expectations.yml contains only ${WAREHOUSE_CONNECTION_STRING}. Nothing secret is on disk. This is the default, and it composes with Kubernetes secrets, a Vault agent or a CI secret store. See the Vault guide for the injection side.
• Tier two, config_variables.yml in uncommitted/. Acceptable for local development, because the folder is git-ignored. Not acceptable as the production mechanism, because the file is a plaintext secret on a disk that also holds validation results.
• Tier three, secrets-manager references. GX can resolve values from AWS Secrets Manager, GCP Secret Manager and Azure Key Vault using a secret|... prefix in config_variables.yml. For AWS, install the aws_secrets extra and write a value such as secret|arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT_ID}:secret:dev_db_credentials|connection_string, where the trailing |connection_string selects a key in a JSON secret and an optional version UUID may be appended. Secrets-manager substitution runs after the ordinary ${} substitutions.

The failure mode of tier three is the silent one. If the JSON key does not match, no substitution occurs, and the literal reference string is used as the value. The job then fails later with a connection error that mentions a string that looks like an ARN. When you see a connection error whose "password" is a secret|arn: string, the substitution did not happen, and the cause is a wrong key, a missing permission on the secret, or a missing extra.

fluent_datasources:
  warehouse:
    type: postgres
    connection_string: ${WAREHOUSE_CONNECTION_STRING}

GX masks connection strings in its output, and recent releases fixed masking of Azure connection-string AccountKey regardless of position (1.22.0) and keep the database and query parts of a masked URL verbatim (1.23.2). Masking is a safety net, not a substitute for keeping secrets out of logs, because a custom Action or a debug print can bypass it.

Results are data

This is the point teams miss. Validation results, at COMPLETE and often at SUMMARY, contain raw values from your tables: the unexpected values, sometimes whole rows with include_unexpected_rows. Data Docs renders those into static HTML. So gx/uncommitted/validations and the Data Docs site are copies of your data, in a place with a different access model from the warehouse. A validation of a column of national identity numbers, failing on a format check, writes those identity numbers into a result file and onto a web page.

For employers in the Gulf and Egypt, where personal-data rules and sector regulators constrain where and how personal data may be stored and who may see it, this is not academic. A warehouse in an in-region cloud location with strict access control can have a Data Docs site on a shared laptop folder that undoes all of it. The platform response has four parts.

1. Classify validation output by the most sensitive table it touches, and apply that table's access policy to the results store and the Data Docs.
2. Use exclude_unexpected_values and partial_unexpected_count set to zero in the result format for sensitive columns, so the failure count and the expectation are recorded without the values. Identify failing rows by a primary key with unexpected_index_column_names instead of by value.
3. Serve Data Docs only behind authentication. The site is a static directory; GX does not host it. Publish it to a protected web location that you control, and never to a public bucket.
4. Set retention on the results folder, and delete results on a schedule consistent with the data's retention rules.

# Keep the failure signal, drop the values: safe for columns holding personal data.
result = checkpoint.run(
    batch_parameters={"year": 2026, "month": 9, "day": 30},
    result_format={
        "result_format": "SUMMARY",
        "unexpected_index_column_names": ["customer_id"],
        "exclude_unexpected_values": True,
        "partial_unexpected_count": 0,
    },
)

The optional keys above are documented result-format options; confirm on your installed version with a small test before you standardise them across teams, because result-format behaviour is exactly the kind of thing that changes between minor releases.

The store constraint

There is a constraint that shapes the storage design. Since 1.13.0 the cloud-object-storage and database store backends were removed; the only remaining backends are the filesystem one, an in-memory one and an inline one. Expectation stores, validation result stores and Data Docs can only be written to a local or networked filesystem. Do not teach or copy examples that use TupleS3StoreBackend, TupleGCSStoreBackend, TupleAzureBlobStoreBackend or DatabaseStoreBackend; they no longer exist, and many blog posts still show them. The documentation for Data Docs says GX writes sites to the directory named by base_directory, and that configuring other keys is not supported and may be removed.

In practice you have three options for central results. Mount a networked filesystem that all jobs write to. Sync the uncommitted folders to object storage after each run, treating GX's local write as a spool. Or, better for most platforms, write a custom Action that ships the serialised result to a warehouse table and a log pipeline, and stop relying on the file store as the system of record. The third option gives you queryable history, which is what you need for trend dashboards and for the incident questions later in this guide.

The supply chain

GX is a fast-moving dependency with a large transitive tree, which includes pandas, numpy, SQLAlchemy, Pydantic and the drivers that your extras pull in. The platform owner's supply-chain duties are the standard ones, applied with the specifics of this library.

Pin the exact version and the extras together, for example great_expectations[postgresql]==1.23.2, and pin a lock file for the whole tree. Install with extras, always, because extras carry version caps that protect you: the snowflake and databricks extras are capped at sqlalchemy<2.1. Scan the built image, and see the Trivy guide for wiring that into CI. Mirror packages through an internal index if your network requires it. The project's source repository moved to the Fivetran organisation (github.com/fivetran/great_expectations), and documentation links were updated in 1.23.0, so allowlists that name the old organisation need updating. Usage analytics were removed in 1.7.0 and the posthog dependency is no longer installed, so there is no outbound telemetry to block, and the documentation page about toggling analytics is stale. For FIPS hosts, 1.22.0 made non-security md5 calls pass usedforsecurity=False. Vulnerabilities are reported through a private GitHub security advisory on the repository.

The supported environment is also a supply-chain fact. Python 3.10 to 3.13 is supported and 3.14 is not; the documentation says macOS and Linux are supported and that Windows is not officially supported though users have deployed on it. A platform image on Linux with a supported Python removes the question.

Multi-tenancy, ownership and governance

GX has no multi-tenancy. There are no tenants, projects, workspaces or permissions as objects. Everything a platform calls multi-tenancy is convention built out of the things you can isolate: repositories or folders, credentials, runners and naming. The design question is how much isolation each team needs, and the answer should follow the blast radius of a mistake.

Isolation models

There are three workable models, each with a clear trade-off.

One repository, one File context per domain or team. Each team owns a gx/ folder in its own repository, reviewed by that team and run by that team's pipeline. The platform provides the base image, the wrapper, the conventions and the shared Actions as a package. Isolation is strong: a bad suite affects only its owner, credentials are per team, and the upgrade is per team on the platform's schedule. The cost is duplication and drift, which the platform counters with templates and a scheduled upgrade bot.

A single central repository with a folder per team. Simpler to govern and to audit, with one place to enforce review rules and one CI. The costs are a shared blast radius, since a broken commit can break everyone's checkpoints if they share a context, and contention on review. This suits an organisation with a strong central data team and relatively few authors.

Ephemeral contexts built in code per job. No shared state at all. Each job builds and runs its own context, often from a small declarative spec that the platform interprets. This gives the best isolation and the worst reviewability unless the spec is well designed. It is the right answer when teams are numerous, suites are simple, and you want to offer validation as a service call rather than as a repository to maintain.

Most organisations end with the first model plus a platform package, sometimes with the third model for long-tail teams. Whichever you choose, use separate credentials per data source and per team, so that a leaked credential exposes one team's tables, and use naming conventions that encode the owner, such as finance.orders.daily for a checkpoint, so that an alert carries its owner in its name.

Ownership is the real governance problem

Data-quality programmes fail socially long before they fail technically. A suite with no owner is a source of alerts nobody acts on, and within a quarter it is muted. The governance rules that matter are few.

Every suite and checkpoint has a named owning team, recorded in meta on the Expectation or in a CODEOWNERS file next to the suite JSON. Changes to a suite are reviewed by the owner of the data and by a platform reviewer for structural concerns. There is an expiry or a review date on suites that gate important outputs, so that thresholds derived from last year's distribution do not silently become wrong. And there is a written answer, per checkpoint, to the question "what happens when this fails", which is a policy, not a technical detail.

Severity as a policy tool

GX gives you the mechanism for that policy. Every Expectation has a severity of critical, which is the default, warning or info. Actions can filter on it through notify_on, whose values are all, success, failure, critical, warning and info. The documented meanings matter, so get them exactly right. failure fires on any failure regardless of severity. critical fires on a critical failure, or on any Expectation that failed to execute. warning fires on a warning failure when there is no critical failure, and info fires on an info failure when there is no warning or critical failure. The highest severity present wins.

This lets a platform express tiered policy without custom code. Critical Expectations, such as uniqueness of a key or non-null of a required field, block the pipeline and page the owner. Warning Expectations, such as a distribution drifting, notify a channel but do not block. Info Expectations accumulate into Data Docs for trend reading. The pipeline gate then reads the result and decides based on severity instead of on the coarse success flag, so that a drift warning never stops the nightly load while a duplicate primary key always does.

from great_expectations.checkpoint import SlackNotificationAction, UpdateDataDocsAction

checkpoint = context.checkpoints.add_or_update(gx.Checkpoint(
    name="finance.orders.daily",
    validation_definitions=[orders_vd],
    actions=[
        SlackNotificationAction(
            name="slack_critical",
            slack_token="${SLACK_BOT_TOKEN}",
            slack_channel="${ORDERS_ALERT_CHANNEL}",
            notify_on="critical",
            show_failed_expectations=True,
        ),
        UpdateDataDocsAction(name="update_docs"),
    ],
))

There are two design warnings about severity. First, it only routes alerts; it does not by itself stop a pipeline. Blocking is your code reading the result. Second, a default of critical means authors who never set severity produce an all-critical suite, which is noisy. Consider making severity a required field in your review checklist so that every Expectation is a deliberate choice.

Remember also that V1 has no default Actions. The old behaviour of storing results and updating Data Docs automatically is gone from the migration's point of view, and UpdateDataDocsAction must be added explicitly. A checkpoint with an empty actions list runs and returns a result and does nothing else, which is a common reason a migrated team sees no alerts and no docs.

Shared and custom Actions

The built-in Actions cover Slack, Microsoft Teams, email, PagerDuty, Opsgenie, SNS and a generic API POST, and the compatibility reference lists email, Teams, Slack and custom as officially supported. The platform should wrap these in one internal package so that every team alerts the same way, and should add one custom Action that every checkpoint includes: a result shipper.

from typing import Literal
from great_expectations.checkpoint import ValidationAction, CheckpointResult, ActionContext

class ShipResultAction(ValidationAction):
    type: Literal["ship_result"] = "ship_result"
    sink_url: str

    def run(self, checkpoint_result: CheckpointResult,
            action_context: ActionContext | None) -> dict:
        payload = checkpoint_result.describe_dict()
        # Post payload to your warehouse loader or log pipeline here.
        return {"shipped": True, "success": checkpoint_result.success}

The Action interface is documented; GX supports the pattern but not your domain logic, so test this class like any production code. The action_context argument carries the results of earlier Actions in the same run, which lets a shipper record whether the alert went out.

An important subtlety is that an Action that raises can turn a successful validation into a failed job. Decide deliberately whether a failing alert sink should fail the pipeline, and in most platforms it should not: catch the exception, log it loudly, and emit a metric.

Cost

GX Core is free under the Apache 2.0 licence, and since the shutdown of the hosted product there is no paid tier. That does not mean validation is free. The costs are real and come from four places, and a platform owner should be able to state each to a finance partner.

The largest is compute. Every Expectation on a SQL source contributes to queries that consume warehouse credits or slot time, and every Spark validation consumes cluster time. This is the dominant line, and it scales with batch size, number of checkpoints and run frequency. The levers are the ones from the scaling section: partitioned batch definitions, running at the frequency that data actually changes rather than on a fixed timer, and removing Expectations that nobody acts on. A useful governance rule is that every Expectation must justify its scan. An Expectation on a column that only ever changes through a constrained upstream process is paying for assurance the source already gives.

The second is storage, which is small but grows: result files and Data Docs accumulate forever unless you set retention. At COMPLETE on wide failures they grow quickly.

The third is people. Suites need authors, reviewers and owners, alerts need responders, and the upgrade cadence needs an engineer. This is usually the largest real cost, and it is why a platform team that reduces per-team effort through templates and automation pays for itself.

The fourth is the cost of not validating: the incident where a bad figure reached an executive report. That is the argument for the programme, and it should be quantified in your own organisation's terms rather than asserted.

For leadership, the useful framing is a unit cost: the warehouse cost per checkpoint run, multiplied by runs per month, per domain. A platform that publishes this table lets teams trade coverage against cost knowingly. If your warehouse is Databricks or another platform with its own quality features, compare the marginal cost honestly, and see the "Where GX stops" section.

Upgrades and migrations

The release cadence is the defining operational fact. Releases arrive roughly every one to three weeks, with a minor bump about every two weeks: 1.18.0 in early June 2026, 1.19.0 in July, 1.20.0, 1.21.0 and 1.22.0 across August, 1.23.0 on 10 September, and 1.23.2 on 25 September. The project follows semantic-style versioning with deprecations announced in the changelog and removals targeted at 2.0.0. You cannot treat this library as a dependency you upgrade once a year. You need a process that absorbs a release every two weeks without an incident.

What a safe upgrade process looks like

The process has five parts. Keep them boring, and automate them.

1. Pin exactly, upgrade deliberately. Everything runs on an exact pinned version from the platform base image. Teams do not choose their own.
2. Read the changelog line called "Compatibility". Each release's notes flag behaviour changes. Read them, and record which affect you.
3. Run the suites in CI against the candidate version before any production rollout. A scheduled job upgrades the pin on a branch, runs a representative set of checkpoints against test data, and reports differences in results between the old and new versions.
4. Canary a few teams, then roll out. Because GX runs inside each team's job, the rollout is a base-image bump, so it can be staged by team.
5. Keep a rollback. The previous image stays available, and the pin can be reverted in one commit.

Recent releases that would have bitten you

The changelog of the last few months is a good list of the kinds of surprise to test for.

• 1.23.2 and the SQLAlchemy 2.1 incident. SQLAlchemy 2.1.0 was released on 24 September 2026 and broke GX 1.23.1 and earlier on Python 3.11 and later, with Snowflake import failures, Databricks query problems, driverless postgresql:// URLs, BigQuery float comparisons and SQL Server mixed-case tables affected. Version 1.23.2 fixed it, and the snowflake and databricks extras are capped at sqlalchemy<2.1. The lesson is that an unpinned transitive dependency broke a release that had not changed. If you could not upgrade immediately, pinning sqlalchemy<2.1 was the documented workaround, and exact lock files would have prevented the outage entirely.
• Snowflake regex semantics in 1.23.2. Regex Expectations on Snowflake now use substring semantics like every other backend, where previously Snowflake's REGEXP anchored to the whole value. A suite that passed with an unanchored pattern may now fail or pass differently. Anchor patterns with ^ and $ when a full match is intended, and re-baseline Snowflake suites on upgrade.
• Engine caching in 1.23.0. The SQL engine is cached across calls. That is a performance win, and it made a latent batch-identity bug reachable, fixed in 1.23.1. Anything that runs validations concurrently was exposed.
• Integer batch parameters in 1.21.0. Batch parameters are now integers on every data source family. Digit strings such as "2024" are deprecated and produce a DeprecationWarning, with removal in 2.0.0. A platform template that accepts parameters from a scheduler usually receives strings, so convert them in the wrapper.
• Store removals in 1.13.0. Cloud-store backends disappeared, and any configuration that referenced them stopped loading.
• Extra renames. The mssql extra was removed in 1.13.0 in favour of sql-server, and gx-redshift became a deprecated alias of redshift in 1.21.0. A pip install line in a template that names a removed extra fails at build time, which is the better time.

Turn deprecations into a backlog

The changelog contains a deprecation timeline table, and a platform should mirror it in an internal backlog with owners. Items targeting 2.0.0 that affect typical estates include string-style row_condition in favour of RowCondition expression objects, context.get_datasource in favour of context.data_sources.get, DataContext.add_or_update_datasource in favour of context.data_sources.add_* or update_*, the schema_name argument in favour of schema-qualified table names, PandasDBFSDatasource and SparkDBFSDatasource in favour of the filesystem data sources, the Spark session helpers, the cloud-mode parts of get_context, and digit-string batch parameters. A simple CI step that runs your test suites with -W error::DeprecationWarning turns each of these into a failing build on the day it appears, instead of a surprise on the day 2.0.0 lands.

Migrating from 0.x

If any team still runs a 0.18 project, note that 0.18 reached end of life on 1 October 2025, and its migration guide survives only in the 0.18 documentation. A migration is a rewrite, not a conversion. The CLI is gone, so there is no great_expectations init, and the only command surface today is python -m great_expectations skills install and skills list, which install agent skill files. Suites need re-creating with the 1.x API, with expectation_suite_name now name and evaluation parameters now suite parameters. context.sources became context.data_sources, batching moved into Batch Definitions, checkpoints gained first-class Validation Definitions, profilers and Data Assistants are gone, and config_version moved from 3.0 to 4.0. The pragmatic plan is to migrate by data domain: for each, write the new suite from the old one's intent, run both in parallel for a period, compare pass and fail outcomes, and retire the old one when they agree.

Incident playbooks

Incidents in a data-quality platform come in a few recurring shapes. For each, a playbook beats improvisation, because the people on call will not have read this guide. Each entry below gives the symptom, the first check, the likely causes and the fix.

Playbook one: every checkpoint started failing at once

Symptom: dozens of unrelated checkpoints fail or error within the same hour, across teams and data sources. The first check is what changed in the platform, not in the data. A simultaneous failure across unrelated data is almost never a data problem. Look at the base image, the pinned versions, the lock file and transitive dependencies. If the failures are import errors or query errors mentioning SQLAlchemy on Python 3.11 or later, suspect the SQLAlchemy 2.1 incident: upgrade to 1.23.2 with extras, or pin sqlalchemy<2.1. If the errors began after a base-image bump, roll the image back first and diagnose afterwards. A platform that cannot roll back in minutes has not finished its design.

Playbook two: a checkpoint has been green for weeks, and the data was wrong

Symptom: a downstream consumer found bad data, and the checkpoint never complained. This is the most damaging class, and the cause is nearly always that the validation did not test what the owner thought it did. Work through four questions in order.

First, did the checkpoint actually run against the bad batch? Look at the shipped results for that date. If there is no record, the job did not run, or ran with the wrong batch parameters, or silently fell back to an empty context. Second, which Expectations were in the suite on that date? Check git history for the suite JSON. Third, did a mostly threshold hide it? An Expectation with mostly=0.95 passes when five percent of rows are bad, which may be a million rows. Fourth, was the failing Expectation of a severity that the gate ignores? A warning that nobody read is a common culprit.

The fixes are usually a new Expectation and, for the platform, a structural one: add a positive "ran" signal and a dashboard of Expectations evaluated per checkpoint per day, with an alert when the count drops to zero or falls sharply.

Playbook three: saved suites vanished, or "was not found"

Symptom: DataContextError: ExpectationSuite with name <name> was not found. or the same for a Validation Definition or Checkpoint, in a job that worked yesterday. The cause is nearly always a different context being loaded. The candidates are an Ephemeral context because no gx/ was found, a wrong working directory, or a stale GX_HOME pointing at a directory with no great_expectations.yml, which silently yields an Ephemeral context. The platform wrapper from the architecture section prevents this class by asserting the context type. To diagnose by hand, print type(context).__name__ and context.suites.all() in the failing environment and compare with a known-good one.

Playbook four: an alert storm

Symptom: a channel receives hundreds of notifications, and people mute it. Causes include a critical default severity across a whole suite, a checkpoint run per partition in a backfill, and an upstream outage that fails every downstream checkpoint at once. The immediate step is to silence at the channel or routing layer, not to delete suites. The structural fixes are severity discipline, one notification per checkpoint run rather than one per Expectation, and dependency-aware suppression: when an upstream freshness check fails, downstream checks should not also page. GX does not do suppression for you, so it belongs in your orchestration or your alert router.

Where Great Expectations stops

A platform owner's maturity shows in knowing when not to use the tool. GX is excellent at one thing: explicit, reviewable, per-dataset assertions that run inside your pipeline and produce structured results. It is weak or absent elsewhere.

It is not an observability product. There is no server, no hosted user interface, no built-in metrics exporter, no anomaly detection that learns your data, and no lineage. It tests what you wrote down. If your need is to discover that a table changed in ways nobody anticipated, across thousands of tables, without anyone authoring tests, a data-observability product or your warehouse's native monitoring is a better fit. GX's strength is the opposite: intention made explicit.

It is not a central results store. The filesystem-only stores mean you must build the history, the trends and the dashboards yourself, typically with a custom Action. If you need that out of the box, account for the work.

It is not orchestration. It does not schedule, retry or manage dependencies. Pair it with Airflow, Dagster or Prefect. The official Airflow provider, airflow-provider-great-expectations, offers operators for dataframes, batches and checkpoints. Its documentation still mentions a cloud context type and a mssql extra, both broken against current GX: use great_expectations[sql-server] directly and never a cloud context. Only the checkpoint operator supports File contexts, and Airflow 2.9.0 and later is the officially supported orchestrator. Anything that runs Python works.

It is not a transformation-time test framework for SQL models. If your team lives in dbt, dbt's own tests cover uniqueness, not-null and relationships at build time with less setup. The official tutorial runs dbt and then GX validations orchestrated by Airflow, which shows the common split: model-level structural tests in dbt, richer statistical and cross-source assertions in GX.

It does not version your data. To know which version of a dataset a result refers to, pair it with a versioned table format or data versioning. Delta Lake tables and lakeFS branches give you a stable snapshot to validate and a place to block a bad write before it reaches consumers; the same idea of validating on a branch before merging fits the platform story well. DVC plays a related role for file-based datasets.

It does not watch models in production. Drift detection on model inputs and predictions belongs to a monitoring tool such as Evidently or NannyML. GX's distribution Expectations, for example ExpectColumnKLDivergenceToBeLessThan and the quantile and mean checks, can approximate data drift gates for a training pipeline, but they are assertions with thresholds you choose, not a monitoring system.

It does not handle every data shape equally. The officially tested data sources are a specific list: AlloyDB, Aurora PostgreSQL, S3, Azure Blob Storage, BigQuery, Citus, Databricks SQL, GCS, Microsoft Fabric, SQL Server, Neon, Oracle, pandas, PostgreSQL, Redshift, Snowflake, Spark, SQLite and Trino. Others, including Athena, Glue, MySQL, Teradata and Vertica, are reported working without a guarantee. If your critical warehouse is in the second group, weigh that.

A reasonable decision rule for a platform owner is this. Choose GX when teams need explicit, reviewable, code-owned assertions that run in their own pipelines, and when you are willing to own the result history. Choose native warehouse or dbt tests for simple structural checks close to the model. Choose an observability product when the need is discovery without authorship. And do not choose any of them as a substitute for fixing the contract at the source: a producer-side schema contract prevents a class of failures that no downstream test can.

Running Great Expectations as a platform for other teams

Everything so far combines into an operating model. This section describes the parts of the platform you build, in the order I would build them.

The golden path

The platform's product is a golden path: the shortest way for a team to get a validated dataset that meets your standards. It has five artifacts.

1. A base image with a pinned Python in the 3.10 to 3.13 range, a pinned GX version with the extras your estate needs, a locked dependency tree, and your wrapper.
2. A repository template with a gx/ folder, a .gitignore that keeps uncommitted/ out, CODEOWNERS, a pre-commit hook and a CI workflow.
3. A wrapper like the one in the architecture section: explicit context mode, context-type assertion, integer batch parameters, result-format defaults, exit codes and the result-shipping Action.
4. Shared Actions for alerting and shipping, versioned as a small internal package.
5. Documentation and examples for the five most common cases: a table, a partitioned table, a file in object storage, a Spark dataframe and a cross-table comparison.

Testing suites as code

Suites are code, so test them. The cheapest effective test is a fixture-based one: a small dataset that should pass and a mutated copy that should fail each critical Expectation, run in CI. This catches the most common authoring error, an Expectation that cannot fail. Another good test is a lint over the suite JSON, enforcing your rules: severity set, description present on critical Expectations, owner recorded in meta, no string-style row_condition, and no more than a sensible number of Expectations per suite.

import pandas as pd
import great_expectations as gx

def _run(context, suite, df):
    ds = context.data_sources.add_or_update_pandas(name="test_pandas")
    asset = ds.add_dataframe_asset(name="orders_df")
    bd = asset.add_batch_definition_whole_dataframe("whole")
    batch = bd.get_batch(batch_parameters={"dataframe": df})
    return batch.validate(suite)

def test_good_data_passes_and_duplicate_key_fails():
    context = gx.get_context(mode="ephemeral")
    suite = context.suites.add(gx.ExpectationSuite(name="orders_suite"))
    suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"))
    good = pd.DataFrame({"order_id": [1, 2, 3]})
    bad = pd.DataFrame({"order_id": [1, 1, 3]})
    assert _run(context, suite, good).success
    assert not _run(context, suite, bad).success

The pattern inside the test, an Ephemeral context and a dataframe asset, is the quickest way to exercise an Expectation with no infrastructure. In the test above, the second call to add_dataframe_asset with the same name would collide if you reused a context across tests, which is why each test builds its own context, and why the data source uses add_or_update_pandas. The error you would otherwise see is Can not write the fluent datasource <name> because a datasource of that name already exists in the data context.

The review checklist

Use this list in design reviews for a new validation programme and in pull-request reviews for changes to suites and platform code.

• Which Data Context type does this use, is the mode explicit, and does a wrapper assert the context type?
• Is the GX version and every extra pinned exactly, with a lock file for the whole tree, on Python 3.10 to 3.13?
• Do credentials come from the runner or a secrets manager through ${VAR} or a secret| reference, with nothing secret in the committed great_expectations.yml?
• Does the job use a read-only database identity with SELECT on only the validated objects?
• Is every custom SQL Expectation reviewed as code, including its cost?
• For large tables, which Batch Definition is used, and does the warehouse prune on its column?
• Are batch parameters integers?
• Is the result format SUMMARY by default, and does sensitive data avoid COMPLETE and unexpected values?
• Where do results and Data Docs live, who can read them, and what is the retention?
• Does every Expectation have a deliberate severity, and is notify_on matched to the response policy?
• Is there an explicit UpdateDataDocsAction if Data Docs are wanted, and a result-shipping Action?
• Does the pipeline gate read the result and exit non-zero on the right severity?
• Is there a positive signal that the checkpoint ran, and an alert when it does not?
• Does each mostly threshold have a written reason?
• Are suite owners recorded, and do suites have a review date?
• Are deprecated APIs absent: string row conditions, schema_name, context.get_datasource, DBFS data sources, cloud mode?
• Does CI run the suites against the candidate GX version, and is there a rollback?
• Is there a runbook entry for each failure category, and has it been exercised?

The complete picture

Put the series together. At Beginner you learned the vocabulary: a Data Context, a Data Source, an Asset, a Batch Definition, an Expectation, a Suite, a Validation Definition and a Checkpoint with Actions. At Mid you learned to connect real data, to partition batches, to run checkpoints in a pipeline and to read results. At this level you treat the same objects as an organisation's shared infrastructure.

The shape of that infrastructure is now clear. GX is a library, so the platform is a convention: a pinned image, a wrapper, a repository template, reviewed suites in git, and shared Actions. Computation happens in your warehouse or cluster, so cost and scale are governed through batch definitions, frequency and result format. There is no access control in the library, so security is the database grant, the runner's identity, and the handling of results and Data Docs as copies of your data. There is no multi-tenancy, so isolation is repositories, credentials and naming. There is a two-week release cadence, so upgrades are a pipeline, with exact pins, CI against candidates, canaries and rollback. And there is no server, so observability is whatever you ship from a custom Action.

The deepest lesson is about what validation means. A test that never fails is decoration, a test that always fails is noise, and a test that fails and gets ignored is worse than none because it teaches people to ignore alerts. The platform's success is measured by whether failures are acted on, and that is decided by ownership, severity and clarity, far more than by the choice of Expectation.

Where the series leaves you

You can now own Great Expectations for a team or an organisation. You can design a File-context-based platform with an explicit golden path, scale it by partitioning and by choosing where computation happens, secure it with least privilege and careful handling of results, run it across teams with severity-based policy and ownership, absorb a two-week release cadence safely, respond to its failure modes with practised playbooks, and say plainly where it stops.

From here, the natural neighbours in this catalogue are the tools that complement it. Airflow and Dagster run and gate your checkpoints. Delta Lake and lakeFS give you versioned data to validate. Feast depends on the quality of the features you validate before they are served. Evidently continues where batch validation ends, watching data and models in production. And MLflow can record the validation outcome alongside the run that produced the model, so that an audit can tie a model to the quality of the data it learned from.

The most valuable next step is not a new tool. Pick the one dataset in your organisation whose failure would be most embarrassing, write its five critical Expectations, give them an owner, run them in the pinned image, and watch what happens when the first one fails. That loop, done well once, is the platform.

Sources

• GX Core documentation home
• GX Core introduction
• Changelog and deprecation timeline
• Compatibility reference
• Glossary
• Create a Data Context
• Install additional dependencies
• Install agent skills
• Configure credentials
• Access secrets managers
• Configure metadata stores
• Configure Data Docs
• Row conditions
• Define a custom Expectation class
• Use SQL to define a custom Expectation
• Retrieve all unexpected rows
• Create a Checkpoint with Actions
• Choose a result format
• Create a custom Action
• GX in your data pipeline
• Data pipeline tutorial with Airflow
• dbt tutorial
• 0.18 migration guide
• great-expectations on PyPI
• Source repository and releases
• Airflow provider for Great Expectations

How senior interviews for this tool differ

A senior interview about Great Expectations is rarely about syntax. The interviewer assumes you can build a checkpoint. They want to know whether you can own a validation programme that dozens of people depend on: how you design it, how you keep it cheap and safe, how you respond when it fails, and how you argue for it. The questions below are the kind asked of someone applying to run data quality as a platform. Each has a model answer and a note on what separates a strong answer from a weak one.

Keep one framing in mind throughout. Great Expectations is a Python library with no server, no access control and no hosted interface. The hosted product was shut down in version 1.18.0. Every senior answer should reflect that reality, because candidates who describe a dashboard, a login or a cloud workspace reveal that their knowledge is out of date. The current stable line is 1.23.x, and releases arrive roughly every two weeks.

System-design prompts

Design data validation for a company with forty data teams

The prompt is deliberately open. Start by asking clarifying questions: which warehouses and orchestrators are in use, how sensitive the data is, what the regulatory constraints are, how many tables are business-critical, and who is on call.

A strong answer then lays out a platform made of conventions rather than a service. Standardise on a File Data Context in each team's repository, with suites, validation definitions and checkpoints as reviewed JSON in git and the uncommitted folder ignored. Provide one pinned base image with a locked dependency tree, a wrapper that loads the context with an explicit mode and asserts its type, and a small shared package of Actions for alerting and result shipping. Give each team its own credentials and a read-only database identity. Define severity policy: critical blocks and pages, warning notifies, info is recorded. Ship every result to a central table through a custom Action, because the file store is the only built-in option and is not a good system of record. Add a positive "ran" signal so that a missing run is itself an alert. Finish with the operating model: owners in CODEOWNERS, a review date on suites, a rollout process for the two-week release cadence, and metrics for the platform itself.

A weak answer describes a central server that everyone connects to, which does not exist, or lists Expectations without saying who owns them or how they are reviewed. A very strong answer names the trade-off between one repository per team, which isolates blast radius, and one central repository, which simplifies governance, and picks one with a reason.

Validate a 5 TB events table every hour without a large bill

Expect the interviewer to probe cost. The core move is the Batch Definition. A whole-table batch scans everything; a daily or monthly partitioned batch scans an increment, provided the warehouse can prune on the column used. Explain that GX compiles metrics into queries that run in the warehouse, so cost is the number of queries and bytes scanned per run. Propose validating only the newest partition hourly with a daily batch definition and integer batch parameters, running a slower full-table suite weekly, and using SUMMARY results. Insist on checking the warehouse query history for bytes scanned rather than trusting an estimate.

Add the subtlety that separates strong answers: the partition column must correspond to the table's clustering or partitioning, otherwise the query still scans everything. Mention that Expectations cost differently, and that dropping ones nobody acts on is a cost lever. If the table is in Spark rather than SQL, mention giving validation its own small pool so it cannot starve production jobs.

Where should results and Data Docs live?

The trap here is the answer that uses an object-store backend. Since 1.13.0 the cloud-storage and database store backends were removed, and the remaining backends are filesystem, in-memory and inline. Results and Data Docs go to a local or networked filesystem. A strong answer offers three designs: a mounted network filesystem, a sync of the uncommitted folders to object storage as a spool, or, preferred, a custom Action that writes describe_dict() output to a warehouse table. Explain why the third is best: queryable history, trend dashboards, and no dependence on file layout.

Then raise the security point unprompted. Results and Data Docs contain raw data values, so they inherit the sensitivity of the most sensitive table validated. Data Docs is a static site that GX does not host, so you host it behind authentication. For Gulf and Egyptian employers, where personal-data rules can limit who may see identifiers and where they may be stored, this is a first-order design constraint.

Build a pipeline gate that does not cause alert fatigue

The prompt tests severity. Every Expectation carries a severity of critical, warning or info, and the notify_on values on Actions are all, success, failure, critical, warning and info, with the highest severity present winning. A strong design has duplicate-key and null-key checks as critical, which block the load and page the owner; drift and volume bands as warning, which notify a channel; and informational profiling checks that accumulate in Data Docs. Emphasise that severity routes alerts but does not stop a pipeline, so the gate is your code reading the checkpoint result and exiting non-zero on the right condition.

Finish by addressing noise: one notification per run rather than per Expectation, suppression of downstream alerts when an upstream freshness check fails, and a review of alert precision every quarter.

Migrate an estate from 0.18 to 1.x

The interviewer wants to hear that you know it is a rewrite. Version 0.18 reached end of life on 1 October 2025. The CLI is gone, so there is no great_expectations init. Suites are recreated in the 1.x API; expectation_suite_name becomes name, evaluation parameters become suite parameters, context.sources becomes context.data_sources, batching moves into Batch Definitions, checkpoints reference Validation Definitions, and profilers and Data Assistants no longer exist. V1 has no default Actions, so UpdateDataDocsAction has to be added explicitly. Propose migrating domain by domain with old and new suites in parallel, comparing outcomes, and retiring the old when they agree. Mention that the migration guide survives only in the 0.18 documentation.

Offer validation as a service to teams that will not maintain a repository

This tests whether you know the Ephemeral option. Propose a small declarative spec that teams submit, which your service interprets into an Ephemeral Data Context built with add_or_update calls per run. Isolation is excellent because there is no shared state, and reviewability depends on the spec, so make the spec a reviewed file in git. The trade-off is that you now own an interpreter against an API that changes every two weeks.

Incident-response questions

Every checkpoint in the company started failing at once. What do you do?

The key insight is that simultaneous failure across unrelated data is a platform change, not a data problem. Check what changed: the base image, a pinned version, a transitive dependency. If the errors mention SQLAlchemy on Python 3.11 or later, recall that SQLAlchemy 2.1.0 was released on 24 September 2026 and broke GX 1.23.1 and earlier; 1.23.2 fixes it, and the snowflake and databricks extras are capped below 2.1. Roll back the image first, restore service, then diagnose. In the review afterwards, explain that exact lock files and a canary rollout would have prevented the outage.

A weak answer starts inspecting individual tables. A strong answer states the order: contain, roll back, then investigate.

A checkpoint was green for a month, and the data was wrong

This is the incident that ends programmes. Walk through four checks. Did the checkpoint run against that batch at all? Look at shipped results, because a job that never ran looks identical to a job that passed. Which Expectations were in the suite that day? Did a mostly threshold hide the failure, since mostly=0.95 tolerates five percent bad rows? Was the failing Expectation of a severity the gate ignores?

Then describe the structural fixes: a positive run signal, a dashboard of Expectations evaluated per checkpoint per day, and an alert when that falls to zero. A strong candidate also mentions the silent context fallback, where a stale GX_HOME pointing at a directory without great_expectations.yml yields an Ephemeral context without raising.

Your saved suites disappeared

The error is DataContextError: ExpectationSuite with name <name> was not found. The cause is nearly always a different context. Candidates are an Ephemeral context because no gx/ was found, the wrong working directory, or a stale GX_HOME. Show the diagnostic, which is printing the context type and listing suites in the failing environment, and the durable fix, which is an explicit mode="file" with project_root_dir and an assertion on the type.

Data Docs with personal data was found on a shared drive

Contain first, by removing access, then establish what was exposed, to whom and for how long, and involve security and legal according to your organisation's process. Do not improvise notification duties in an interview; say that they follow the regulator and contracts that apply. Then explain the preventive controls: result formats that omit values for sensitive columns through exclude_unexpected_values and partial_unexpected_count set to zero, failing rows identified by primary key through unexpected_index_column_names, authenticated hosting, and retention.

The alert channel has hundreds of messages and everyone muted it

Silence at the routing layer rather than deleting suites. Identify the cause: an all-critical default, a backfill that ran a checkpoint per partition, or an upstream outage. Then the structural fixes of severity discipline, one notification per run, and dependency-aware suppression, which GX does not provide and your orchestration or alert router must.

A team says their suite passes on Snowflake but failed after the upgrade

This points to the 1.23.2 change. Regex Expectations on Snowflake now use substring semantics like other backends, where previously REGEXP anchored to the whole value. The fix is to anchor patterns with ^ and $ where a full match is intended. The lesson for the platform is to re-baseline suites per backend during upgrade testing.

Trade-off questions

File context or Ephemeral context?

File contexts make configuration a reviewable artifact in git, survive sessions, and since 1.20 can be loaded from a read-only filesystem. Ephemeral contexts have no shared state and suit jobs that build their configuration in code, at the price of scattering the configuration through Python. My default for a platform is a File context in git, loaded explicitly in production, with secrets injected from the environment. I switch to Ephemeral for long-tail teams through a declarative spec, and for unit tests of Expectations.

pandas, Spark or SQL execution?

Pandas loads the batch into process memory and breaks first on size, so it suits dataframes already in memory, samples and tests. SQL pushes metrics into the warehouse, which is light on the validating process and costs warehouse compute. Spark is distributed and needs isolation from production jobs. Pick the execution engine by where the data already lives, and validate increments in all three.

Custom Expectation class or configuration of a built-in?

Prefer configuring a built-in Expectation. If you need a named business rule, subclass with preset defaults and a description, which reads well in Data Docs. If you need a bespoke check, use UnexpectedRowsExpectation with SQL, reviewed as code because it runs under the job's database identity. Write a new Expectation class with custom metrics last, because it is code you maintain against a fast-moving library.

GX, dbt tests, or an observability product?

dbt tests are the lowest-friction choice for structural checks on models. GX is stronger for explicit, reviewable assertions across sources, statistical checks and pipelines that are not dbt-based. An observability product is stronger when you need to discover anomalies without authoring tests, at scale. They are not exclusive. The official tutorial runs dbt then GX under Airflow, which shows the typical split.

Fail the pipeline on any failure, or only on critical ones?

Failing on everything produces a pipeline nobody trusts, and failing on nothing produces a check nobody respects. Block on critical failures and on Expectations that fail to execute, which notify_on="critical" also covers. Notify on warnings. Record info. Review severity assignments with the data owner, because severity is a business judgement about the cost of bad data, not a technical property.

Arguing a trade-off to leadership

Why fund a data-quality platform when the tool is free?

Open with the honest framing: the library is free under Apache 2.0, and since the hosted product was shut down there is no paid tier, but the programme has costs in compute, storage and people. Then quantify the benefit in the organisation's own terms. Count the past incidents where bad data reached a report, a model or a customer, estimate the hours of rework and the decision cost, and compare with the programme cost. Offer a unit cost, warehouse spend per checkpoint run, so leadership sees that coverage is a dial.

A strong answer avoids promising zero incidents. It promises earlier detection, clearer ownership and a measurable trend.

Why not buy a commercial data-observability product?

Acknowledge that it may be the right answer for discovery across thousands of tables. Argue for GX where the need is explicit assertions owned by the teams that know the data, with artifacts in your own git and no dependency on a hosted control plane. Cite the shutdown of GX Cloud as the concrete example of why that matters. Propose a time-boxed comparison with agreed criteria rather than a debate of preferences.

The release cadence is too fast. Why not freeze the version?

Freezing stops breaking changes and starts accumulating risk: deprecations land in 2.0.0, bugs are fixed in later releases, and dependency breakage such as the SQLAlchemy 2.1 incident requires a fix you cannot take. The answer is a pipeline that makes upgrades cheap: exact pins, CI against candidates, canaries, rollback, and a deprecation backlog mirrored from the changelog's timeline table. Present it as a fixed small monthly cost that prevents a rare large one.

How do you show the platform is working?

Name a few measures: the share of critical tables covered by a checkpoint that ran in the last 24 hours, time to acknowledge and resolve by owner, alert precision, warehouse cost per checkpoint, and the age of the oldest pinned version. Explain that alert precision is the most honest one, because it measures whether people believe the alerts.

What a strong answer covers that a weak one misses

• Current facts. The strong candidate knows there is no CLI for project work, no GX Cloud, and no cloud-object-store backends for results. The weak one describes 0.x workflows.
• Failure categories. The strong candidate separates validation failures, execution failures, infrastructure failures and silent configuration failures. The weak one says "the test failed".
• Cost as a design input. The strong candidate talks about batch definitions and bytes scanned. The weak one lists Expectations.
• Results as sensitive data. The strong candidate raises it unprompted.
• Ownership. The strong candidate names who reviews and who responds. The weak one describes only tooling.
• Humility about limits. The strong candidate says where GX stops and points to Airflow, Delta Lake, lakeFS or Evidently. See the Airflow guide, the Delta Lake guide and the Evidently guide.

Sixty-second senior self-test

Answer each aloud in under fifteen seconds.

1. Why is a green pipeline not evidence that data was validated?
2. Which component breaks first with pandas, and what do you do about it?
3. Name the four result formats and say which is the checkpoint default.
4. Which store backends remain, and what does that imply for Data Docs?
5. What does notify_on="critical" fire on?
6. What does the silent fallback of get_context() look like, and how do you prevent it?
7. What changed for Snowflake regex Expectations in 1.23.2?
8. Why pin extras together with the version?
9. What would you put in a golden path for a new team?
10. When would you not use Great Expectations?

If you hesitated on more than two, reread the architecture, security and upgrade sections of the senior detailed guide, then return here.

Production traps that only appear at scale

The mistakes that hurt a platform owner are rarely syntax errors. They are decisions that look fine with three checkpoints and fail with three hundred. Each trap below gives the symptom, the cause and the fix, in the order of how often I see them.

Trap: the whole-table batch on a growing table

Symptom: a checkpoint that took a minute in the first month takes forty minutes a year later, and the warehouse admin asks why validation leads the credit report. Cause: the suite uses add_batch_definition_whole_table, so every run scans the full history. Fix: move large tables to a partitioned Batch Definition such as add_batch_definition_daily(name="DAILY", column="event_ts") and run it with integer batch parameters, for example {"year": 2026, "month": 9, "day": 30}. Then confirm in the warehouse query history that the scan is actually pruned. If the column is not the table's clustering or partitioning key, the query still reads everything and the change bought nothing.

Trap: loading a big table through pandas

Symptom: a job is killed with exit code 137, or slows to a crawl on swap. Cause: a pandas data source loads the batch into the process's memory, and decoded data is much larger than the file on disk. Fix: validate where the data lives, through a SQL or Spark data source, and reserve pandas for dataframes a pipeline step already holds, samples and unit tests.

Trap: COMPLETE results by default

Symptom: the results folder grows by gigabytes, Data Docs pages take forever to load, and an alert message is truncated or rejected. Cause: the COMPLETE result format adds the full unexpected list, the full index list and the index query, which on a column with millions of failures is itself a dataset. Fix: make SUMMARY, which is the checkpoint default, your platform default, and use COMPLETE for a targeted investigation. Remember that include_unexpected_rows returns at most 200 rows, whereas get_unexpected_rows() on a Validation Definition has no cap and can return every failing row for an UnexpectedRowsExpectation.

Trap: results and Data Docs as an unmanaged copy of your data

Symptom: nothing visible, until a review finds identity numbers on a web page. Cause: results and Data Docs include raw unexpected values. Fix: classify the results store by the most sensitive table it touches, host Data Docs behind authentication, set retention, and for sensitive columns use exclude_unexpected_values and a partial_unexpected_count of zero, identifying failing rows by primary key with unexpected_index_column_names.

Trap: object-storage store backends copied from old posts

Symptom: ImportError or a configuration that fails to load on an upgrade. Cause: TupleS3StoreBackend, TupleGCSStoreBackend, TupleAzureBlobStoreBackend and DatabaseStoreBackend were removed in 1.13.0, and only filesystem, in-memory and inline backends remain. Fix: write results to a filesystem path or, better, to your own table through a custom Action. Reading data from S3, GCS and Azure Blob is unaffected; the removal concerns metadata and Data Docs stores.

Trap: the silent fallback to an empty context

Symptom: DataContextError: ExpectationSuite with name <name> was not found. in a job that worked yesterday, or worse, a job that validates nothing and exits green. Cause: a stale GX_HOME pointing at a directory with no great_expectations.yml does not raise; discovery falls back to an Ephemeral context. A different working directory has the same effect. Fix: always pass mode="file" with project_root_dir, and assert type(context).__name__ == "FileDataContext" in your wrapper.

Trap: a floating dependency on a moving library

Symptom: a deploy nobody touched breaks. Cause: releases come every one to three weeks and transitive dependencies move independently. The clearest case is SQLAlchemy 2.1.0, released on 24 September 2026, which broke GX 1.23.1 and earlier on Python 3.11 and later. Fix: pin the exact version together with the extras, for example great_expectations[postgresql]==1.23.2, lock the whole tree, and upgrade through CI and a canary. Install with extras, always, because the snowflake and databricks extras carry a sqlalchemy<2.1 cap that protects you. If you were caught and cannot upgrade, pin sqlalchemy<2.1 as the stopgap.

Trap: behaviour changes that look like data changes

Symptom: suites that passed for months fail after an upgrade, with no data change. Cause: a semantic change in the library. The 1.23.2 example is regex Expectations on Snowflake, which now use substring semantics like every other backend, where REGEXP previously anchored to the whole value. Fix: anchor patterns with ^ and $ when a full match is intended, read the "Compatibility" line in each release's notes, and compare old and new results in CI before rolling out.

Trap: digit-string batch parameters

Symptom: a DeprecationWarning flood, and a future break. Cause: schedulers pass "2026" as a string, and since 1.21.0 string values for numeric batch parameters are deprecated with removal in 2.0.0. Fix: convert to integers in your wrapper so that every team inherits the fix.

Trap: severity left at the default

Symptom: every failure is critical, the channel is noisy, and people mute it. Cause: the default severity is critical, and authors never change it. Fix: make severity a required review item, and map it to policy: critical blocks and pages, warning notifies, info is recorded.

Trap: an empty actions list after a migration

Symptom: checkpoints run, results appear, and nobody gets alerts and Data Docs never update. Cause: V1 has no default Actions. The old automatic storing and Data Docs behaviour is gone, and UpdateDataDocsAction must be added explicitly. Fix: make your platform checkpoint factory always include the update-docs and result-shipping Actions.

Trap: a mostly threshold that hides a real problem

Symptom: bad data reaches a consumer and the suite was green. Cause: mostly=0.95 passes with five percent of rows failing, which on a large table is a large absolute number. Fix: require a written reason beside each mostly, and pair a tolerant per-row check with a strict count or proportion check.

Trap: secrets-manager references that do not resolve

Symptom: a connection error in which the "password" is a string beginning secret|arn:. Cause: if the key after the final pipe does not match, no substitution occurs and the literal reference is used. Fix: check the JSON key, the secret's permissions and the installed extra (aws_secrets for AWS, gcp and azure-secrets for the others). Remember that secrets-manager substitution runs after ordinary ${} substitution.

Trap: multi-threaded validation on an old version

Symptom: intermittent hangs, or results that report the wrong batch. Cause: bugs fixed in 1.23.1 and 1.23.2, a batch-identity issue made reachable by engine caching in 1.23.0 and a deadlock in multi-threaded validator construction on Python 3.10 and 3.11. Fix: require 1.23.2 or later if you validate in threads.

Platform-ownership habits

Make a green run mean something

Emit a positive signal for every checkpoint run: which checkpoint, which batch, how many Expectations evaluated, how many passed, and how long it took. Alert on the absence of that signal and on a sudden drop in Expectations evaluated. A silent failure is the failure you will not learn about until a consumer tells you.

Standardise through a wrapper, not through a wiki

Conventions that live in a document decay. Conventions that live in a wrapper are enforced. Put the explicit context mode, the context-type assertion, integer conversion of batch parameters, result-format defaults, exit codes and the shipping Action in one function that every team calls, and version it.

def run(checkpoint_name: str, batch_parameters: dict) -> int:
    """Platform contract: explicit mode, typed context, integer parameters, exit code."""
    params = {k: int(v) for k, v in batch_parameters.items()}  # digit strings are deprecated
    context = gx.get_context(mode="file", project_root_dir=PROJECT_ROOT)
    assert type(context).__name__ == "FileDataContext"
    result = context.checkpoints.get(name=checkpoint_name).run(batch_parameters=params)
    return 0 if result.success else 1

Keep one person responsible for the version

Somebody owns the pinned GX version, the lock file and the upgrade calendar. With a release every two weeks, an unowned pin ages badly. A scheduled job that proposes the bump on a branch, runs a representative set of checkpoints and posts the diff in results is worth a day of engineering.

Treat deprecations as a backlog

The changelog has a deprecation timeline table. Copy its items into your tracker with owners. Run CI with python -W error::DeprecationWarning so each deprecation becomes a failing test on the day it appears. The items likely to matter most are string-style row_condition, schema_name on table assets, context.get_datasource, DataContext.add_or_update_datasource, the DBFS data sources, the Spark session helpers, digit-string batch parameters and any cloud-mode call.

Give every Expectation a reason to exist

Each Expectation costs scan time and attention. Ask for a description on critical ones, an owner in meta, and a severity. Delete Expectations nobody has acted on in a quarter. A smaller suite that people trust beats a large one that is ignored.

Own the cost number

Publish warehouse cost per checkpoint run and per domain. Teams that see the number trade coverage against cost sensibly. Teams that do not see it either over-validate or drop validation when the bill is questioned.

Isolate by credentials, not by hope

GX has no access control. The isolation you have is separate credentials per team and per data source, a read-only database identity with SELECT on only the validated objects, and separate runners for sensitive domains. Review UnexpectedRowsExpectation SQL like any other code, because it runs under the job's identity.

Drill the playbooks

Run a game day each quarter. Break something on purpose, such as pointing GX_HOME at an empty directory, and time the on-call engineer. The playbooks in the detailed guide are only as good as the last time someone used them.

Keep the custom surface small

Every custom Expectation class, metric or Action is code you own against a library that changes every two weeks. Prefer configuring built-in Expectations, then subclass presets that set a column, bounds and a description, then UnexpectedRowsExpectation with SQL, and only then new classes. Keep custom code in gx/plugins, under test.

Test suites like code

A suite with an Expectation that cannot fail is decoration. For each critical Expectation, keep a fixture that must pass and a mutated copy that must fail, run in CI with an Ephemeral context and a dataframe asset. Lint suite JSON for severity, description and owner.

Place validation deliberately

Validate at ingestion to quarantine bad raw records, after transformation to condition downstream steps, and at delivery to tell business anomalies from data-quality problems. Each placement has a different severity mix. Pair with an orchestrator such as Airflow, Dagster or Prefect, and with a versioned data layer such as Delta Lake or lakeFS so a result refers to a stable snapshot.

Review checklist for pull requests that change suites

Use this when reviewing a change to a suite, a validation definition or a checkpoint.

Intent

• Does the pull request say which failure it is meant to catch, and who asked for it?
• Is there a named owner, recorded in meta or CODEOWNERS?
• Does each new critical Expectation have a description a business reader can understand?

Correctness

• Can each new Expectation fail? Is there a fixture that makes it fail?
• Is each mostly justified in writing, and is there a stricter count or proportion check beside it?
• Are regex patterns anchored with ^ and $ where a full match is meant, particularly on Snowflake after 1.23.2?
• Are row conditions written with the object API and limited to OR-of-ANDs, with no None comparison values?
• Does any Expectation use an argument that the target version removed or deprecated?

Policy

• Is severity set deliberately, and does it match the response: block and page, notify, or record?
• Does notify_on on the Actions match the severity policy?
• Is the gate reading the result and exiting non-zero on the right condition?

Cost and scale

• Which Batch Definition does this use, and does the warehouse prune on its column?
• Are batch parameters integers?
• Is the result format SUMMARY unless there is a reason?
• How many queries and bytes does one run cost, measured in the warehouse history?

Security

• Are credentials referenced through ${VAR} or a secrets-manager reference, with nothing secret committed?
• Does any custom SQL run under a read-only identity?
• For sensitive columns, are values excluded from results?

Operations

• Does the checkpoint include the result-shipping Action and, if Data Docs are wanted, an explicit UpdateDataDocsAction?
• Is there a positive run signal and an alert when it is missing?
• Does CI run this suite against the candidate GX version?

Review checklist for design documents

Use this when a team proposes a new validation programme or a change to the platform.

Scope and ownership. Which datasets, and why these? Who owns each suite, who responds to alerts, and what is the escalation path? What is the written answer to "what happens when this fails"?

Architecture. Which Data Context type, with an explicit mode? Where does computation happen, and what is its cost model? Where do results go, and is the file store treated as a spool or as the system of record? How does a missing run become visible?

Security and privacy. What identity does the job run as, and what can it read? Where are secrets injected from? What data values appear in results and Data Docs, who can read them, and how long are they kept? For employers in the Gulf and Egypt, which residency and sector rules apply to that storage?

Scale and cost. What is the largest table, and which Batch Definition bounds the scan? What is the expected cost per run and per month, and who approves growth? What is the plan if the warehouse is shared with production workloads?

Operations. Which version and extras are pinned, on which Python in the 3.10 to 3.13 range? How are upgrades tested and rolled back? Which deprecations targeting 2.0.0 affect this design? What are the playbooks, and when were they last exercised?

Exit. What would make us stop using GX for this domain? What is the migration path if we did? Is any part of the design dependent on a hosted service, given that the hosted product was shut down?

Practice cards

Each card is a prompt to answer aloud or in writing.

Card 1. A team's checkpoint is green and a finance report is wrong. List, in order, the four things you check.

Card 2. Design a severity policy for a customers table with a primary key, an email column, a country column and a signup timestamp. Which Expectations are critical, warning and info, and why?

Card 3. Write the wrapper function for your platform in five lines. Which three mistakes does it prevent?

Card 4. A 2 TB table is validated hourly. Propose a batch strategy and explain how you would prove the cost fell.

Card 5. A regulator asks where validation results containing national identifiers are stored and who can read them. What do you need to be able to answer?

Card 6. A release lands tomorrow. Describe the upgrade pipeline from pin bump to canary to rollback.

Card 7. List the Expectations and features you would refuse to maintain as custom code, and what you would use instead.

Card 8. Write the one-paragraph case, for leadership, for funding the platform when the library is free.

Operational recipes

Recipe: inspect what a job actually loaded

Use this first when a team says their saved objects vanished.

import great_expectations as gx

context = gx.get_context(mode="file", project_root_dir="/app/project")
print(type(context).__name__)                      # must be FileDataContext
print([s.name for s in context.suites.all()])      # the suites this job can see
print([c.name for c in context.checkpoints.all()]) # and the checkpoints

If the type is EphemeralDataContext, check GX_HOME and the working directory.

Recipe: list everything a platform might have configured

for ds in context.data_sources.all():
    print(ds.name, type(ds).__name__)

Then fetch one with context.data_sources.get("warehouse") and list its assets through get_asset. A KeyError from get means the name is not present.

Recipe: make setup scripts re-runnable

Setup code that uses add fails on the second run with Cannot add ExpectationSuite with name <name> because it already exists. Use add_or_update for suites, validation definitions and checkpoints, and add_or_update_<type> for data sources, which avoids Can not write the fluent datasource <name> because a datasource of that name already exists in the data context.

Recipe: gate on severity, not only on success

result = checkpoint.run(batch_parameters={"year": 2026, "month": 9, "day": 30})
summary = result.describe_dict()
print(summary)       # inspect the structure once, then decide what to gate on
if not result.success:
    raise SystemExit(1)

Inspect the structure of describe_dict() on your installed version before writing severity-aware gating against it, since the shape is version-specific, and keep that code in the platform wrapper.

Recipe: upgrade check in CI

python -m pip install "great_expectations[postgresql]==1.23.2"
python -W error::DeprecationWarning -m pytest tests/

Run it against the candidate version on a branch, compare results to the current pin, and only then propose the bump. Pair it with an image scan; see the Trivy guide.

Recipe: install the agent skills into a repository template

python -m great_expectations skills list
python -m great_expectations skills install --target all

Add --symlink if you want links rather than copies, and --force only to overwrite edited copies. There is no uninstall command; delete the directories.

Recipe: a fast local test of one Expectation

import pandas as pd
import great_expectations as gx

context = gx.get_context(mode="ephemeral")
batch = context.data_sources.pandas_default.read_dataframe(pd.DataFrame({"n": [1, 2, 9]}))
print(batch.validate(gx.expectations.ExpectColumnValuesToBeBetween(column="n", min_value=1, max_value=6)))

This gives a pass or fail result in seconds with no infrastructure. If this call is not available on your installed version, use the dataframe asset pattern from the Beginner guide instead.

A final checklist for the platform owner

• The version and extras are pinned exactly, on a supported Python, with a locked tree and a tested rollback.
• Every job loads its context with an explicit mode and asserts the type.
• Credentials are injected from the runner or a secrets manager, and the database identity is read-only.
• Large tables use partitioned Batch Definitions, verified in the warehouse history.
• Results default to SUMMARY, and sensitive values are excluded.
• Results and Data Docs are access-controlled, retained on a schedule, and shipped to a queryable store.
• Severity is deliberate, routed, and read by the pipeline gate.
• A missing run is an alert.
• Deprecations are tracked to 2.0.0, and CI fails on new ones.
• Playbooks exist and were exercised this quarter.
• Every suite has an owner.