The Complete Alibi Detect Guide

This is part one of three. It covers everything you need to start using Alibi Detect for real work: asking, in code, whether the data reaching your model still looks like the data the model learned from. By the end you can install the library, run a drift check on a table of numbers, read the result without guessing, tune the one setting that controls false alarms, save a detector to disk and load it back, and explain to a manager what the whole thing does and does not prove. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and drift detection only makes sense once you have watched a detector stay quiet on normal data and then fire on data you deliberately damaged.

Why a model that worked yesterday can be wrong today

A trained model is a summary of the data it saw. It learned that, in that data, certain input patterns go with certain outputs. The moment you deploy it, the world keeps moving and the model does not. Customers change their behaviour, a sensor gets recalibrated, an upstream team renames a category, a marketing campaign brings in a different kind of user, a currency devalues. The model has no way to notice any of this. It receives numbers, produces numbers, and returns them with the same confidence as on its first day.

This is the quiet failure that makes machine learning different from ordinary software. A web service that breaks returns errors, and your monitoring lights up. A model that has gone stale returns perfectly valid-looking answers that are simply worse. Nothing crashes. The only symptom is that the business outcome slowly degrades, and by the time someone notices, weeks of bad predictions are already in the world.

There are two ways to find out. The first is to wait for the true answers (the labels) to arrive and measure accuracy. That is the gold standard, but labels are often late: you learn whether a loan defaulted after months, whether a customer churned after a quarter. The second way is to watch the inputs. If the data arriving today no longer resembles the data the model was trained on, you have a strong early warning that the model is operating outside the territory it understands, long before any label arrives.

Alibi Detect is a Python library for that second way. Its own documentation describes it as a source-available Python library focused on outlier, adversarial and drift detection. You give it a sample of data that represents "normal", and it gives you back a yes-or-no answer, plus supporting numbers, about whether new data is different enough to worry about.

Notice what the library does not do. It does not retrain your model, it does not fix anything, and it does not tell you why the data changed. It raises a hand and says "this looks different". What you do next is a human or pipeline decision, and a good part of the later guides is about making that decision well.

What people did before, and where this fits

Before dedicated libraries, teams monitored models in one of three ways, and you will still meet all of them.

The first was no monitoring at all, relying on users or analysts to complain. This works until it does not, and it fails silently.

The second was hand-rolled summary statistics. An engineer computes the mean and standard deviation of each feature every week, plots them, and eyeballs the chart. It catches a feature that doubles in value. It misses a change in the relationship between features, and it has no principled answer to the question "is this wobble normal?", because nothing tells you how much a mean should wobble from sampling noise alone.

The third was hand-rolled statistical tests, calling a two-sample test from SciPy for each column. This is the right idea, and it is exactly what the simplest Alibi Detect detector does. What the library adds is the surrounding machinery: tests that work on many features at once with proper correction for repeated testing, tests that work on images and text by first compressing them, versions that process one instance at a time instead of a batch, a consistent return format, and a way to save and reload a configured detector.

Alibi Detect also sits next to, not instead of, other tools in this catalogue. If you want ready-made drift reports and dashboards for tabular data, look at Evidently and NannyML. If you want to track experiments and models, MLflow is the usual choice. Alibi Detect is the lower-level, more programmable building block, and it is the one with the widest set of detector types, which is why people reach for it when the data is images, text, or a stream.

The mental model: four nouns

Everything in the library is built from four ideas. Learn these and the documentation becomes readable.

The detector. A detector is an ordinary Python object. You create it, and then you call its predict() method on data. There are three families. Drift detectors ask whether a whole batch of data differs from the reference. Outlier detectors ask whether a single instance looks unlike the training data. Adversarial detectors ask whether an input was deliberately crafted to fool a model. This guide is mostly about drift, because it is the most commonly useful and the easiest first step.

Reference data, written x_ref in the code. This is the sample the detector treats as normal. It is usually a slice of your training or validation data. The quality of this sample decides the quality of everything else, so it deserves more thought than beginners give it. A reference set that is too small gives a jumpy detector. A reference set that does not represent the real normal gives you alarms about a "problem" that was always there.

The test and its threshold. A drift detector runs a statistical test comparing the new batch with the reference. The test produces a p-value: roughly, the probability of seeing a difference this large if the two datasets really came from the same source. You choose a threshold, called p_val in the code, commonly 0.05. If the p-value falls below the threshold, the detector declares drift. We will unpack what that really means shortly, because the usual misreading causes most beginner confusion.

The result. Calling predict() returns a dictionary with two parts: data, which holds the verdict and the numbers, and meta, which describes the detector that produced it. The single most important field is is_drift, which is 1 when drift was found and 0 when it was not.

Two more words you will see early. Data drift, also called covariate drift, means the distribution of the inputs changed. This is what the main detectors test. Concept drift means the relationship between inputs and the correct answer changed even if the inputs look the same, for example a fraud pattern that evolves. Input-only tests cannot confirm concept drift on their own, and you usually need labels for that. Alibi Detect has detectors based on model uncertainty that help without labels, but treat them as hints rather than proof.

Finally, detectors come in two styles. Offline detectors take a whole batch and compare it with the reference each time you call them. Online detectors take one instance at a time, keep a sliding window, and track a running timestep. Online detectors are for streams and they introduce two extra settings, ert and window_size, which we will meet briefly later. Start offline.

Read this before you build anything: the licence

Most tools in this catalogue are open source in the usual sense. Alibi Detect changed. Versions up to 0.11.4 were released under the Apache 2.0 licence. Starting with version 0.11.5, released in January 2024, the project moved to the Business Source License 1.1. The current release is 0.13.0, from December 2025, and it is under the Business Source License too. PyPI lists it as "Other/Proprietary License", and the project describes itself as source-available, not open source.

In plain terms, the licence lets you read the code, copy it, modify it and redistribute it, and it lets you use it for non-production purposes. It also contains an additional grant that allows production use by non-profit educational institutions, as long as that use does not involve commercialising the library or putting it into models, products or services sold, licensed or marketed to third parties. Production use by anyone else needs a commercial licence from Seldon, the company behind the project, whose pricing page is linked in the sources.

What does this mean for you as a student? Learning, coursework, personal experiments and portfolio projects are non-production use, and you are fine. The care is needed later. If you join an employer or take a client and you put Alibi Detect into a live system, somebody must check that the licence covers that use. This is not a hypothetical worry: the MLServer project, which is also maintained by Seldon, has an open issue about removing Alibi Detect from its base container image because of exactly this licence mismatch.

The other practical consequence is about alternatives. Versions up to 0.11.4 are Apache 2.0, but they date from the Python 3.7 to 3.11 era and carry dependency and security costs, so pinning an old version is a poor default. For a real product, comparing with other drift tools in this catalogue is a sensible step. Everything you learn here about drift (reference data, tests, thresholds, false alarms) transfers to those tools, so your effort is not wasted either way.

Installing and checking your setup

Alibi Detect needs Python 3.9 or newer, and Python 3.12 support arrived in release 0.13.0. As with every Python project, work in a virtual environment so that the library and its dependencies do not collide with anything else on your machine.

python -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install alibi-detect

That default install gives you the basic detectors and, importantly, no deep-learning framework. That is enough for everything in this guide. The simplest drift detector, the Kolmogorov-Smirnov test, runs on plain NumPy and SciPy.

The library also offers extras for heavier detectors. You will need these later for image or text data, or for detectors that train a small neural network.

pip install "alibi-detect[tensorflow]"   # TensorFlow backend
pip install "alibi-detect[torch]"        # PyTorch backend
pip install "alibi-detect[all]"          # everything

Two details trip people up. First, quote the brackets on macOS and in zsh: without quotes the shell tries to expand the square brackets as a pattern and the command fails. Second, the pip extra is called torch, but when you later pass a backend to a detector the string is 'pytorch'. The names differ, and mixing them up is a classic first-week error.

Platform notes. On Linux everything works with pip or conda. On macOS pip works, but do not use the keops extra, which is not officially supported there. On Windows the docs advise installing and testing PyTorch before installing Alibi Detect if you want GPU use, and the keops extra is Linux only. If you are a beginner on any platform, the default install is the right start.

One more trap: the TensorFlow version the library allows is capped below 2.19 as of release 0.13.0. If you install a very new TensorFlow first and then add Alibi Detect, pip may fight you. If you only need the default install, you can ignore this entirely, which is one more reason to start there.

Now check that it works:

import alibi_detect
print(alibi_detect.__version__)
from alibi_detect.cd import KSDrift

You should see a version string such as 0.13.0 and no error on the import. The alibi_detect.cd package holds the drift detectors. The outlier detectors live in alibi_detect.od.

Your first drift check

Time to build something. We will fake a tiny, honest scenario. Imagine a model that takes five numeric features per customer. Your reference data is a thousand past customers. We will generate it randomly so you can run this anywhere, then show the detector a "today" batch that is normal, and another that is shifted.

import numpy as np
from alibi_detect.cd import KSDrift

rng = np.random.default_rng(42)

# Reference data: 1000 instances, 5 features, drawn from a standard normal
x_ref = rng.standard_normal((1000, 5)).astype(np.float32)

# Build the detector. Drift detectors take the reference data at construction.
cd = KSDrift(x_ref, p_val=0.05)

# A batch that comes from the same source as the reference
x_same = rng.standard_normal((200, 5)).astype(np.float32)
print("same source:", cd.predict(x_same)["data"]["is_drift"])

# A batch where every feature has been shifted by 0.5
x_shifted = (rng.standard_normal((200, 5)) + 0.5).astype(np.float32)
print("shifted:    ", cd.predict(x_shifted)["data"]["is_drift"])

Run it. The shifted batch should print 1, meaning drift. The same-source batch should print 0 on most runs, and the reason it is not guaranteed is explained in the next two sections, so do not be alarmed if you ever see a 1 there.

Walk through what each part does, because every later example is a variation on this one.

The reference array has shape (1000, 5): one row per instance, one column per feature. This rows are instances, columns are features layout is how every detector in the library expects tabular data. The call to astype(np.float32) converts numbers to 32-bit floats. For the simple KS detector this is optional, but it is a good habit, because the deep-learning backends you will meet later are fussy about it.

KSDrift(x_ref, p_val=0.05) constructs the detector. Notice that there is no fit step. Unlike many scikit-learn objects, a drift detector receives its reference data at construction and is ready immediately. KS stands for Kolmogorov-Smirnov, a classic test that compares the shape of two distributions one feature at a time. It is cheap, it needs no neural network, and it is the right default for a first look at tabular data.

cd.predict(x) runs the test on the new batch and returns the result dictionary. We read ["data"]["is_drift"] to get the verdict.

Reading the result without fooling yourself

The verdict is one field in a larger dictionary. Print the whole thing once to see what is available:

preds = cd.predict(x_shifted)
print(preds["meta"])
print(preds["data"])

The meta part describes the detector: its name, its detector_type (which is 'drift' for these, 'outlier' or 'adversarial' for the others), its data_type, whether it is online, and the library version. This is useful when you log results from several detectors and need to know afterwards which one said what.

The data part holds the verdict and numbers. For KSDrift, you get is_drift, the threshold, and, if you ask, the p-value and the test statistic for each feature. You ask by passing flags to predict:

preds = cd.predict(x_shifted, drift_type="feature", return_p_val=True, return_distance=True)
print(preds["data"]["p_val"])      # one p-value per feature
print(preds["data"]["distance"])   # one K-S statistic per feature

Here drift_type="feature" tells the detector to report per feature rather than a single combined answer. Look at the two arrays. The p-value array says, for each column, how surprising the difference is: smaller means more surprising. The distance array holds the K-S statistic, which is the largest gap between the two distributions' cumulative curves for that column: bigger means a larger difference. The p-value tells you whether to believe a difference exists; the distance tells you how big it is. Both matter, because with a huge sample even a trivial difference produces a tiny p-value, and the distance is what tells you whether it is a trivial difference.

Now the misreading. A p-value of 0.03 does not mean "there is a 97 percent chance the data has drifted". It means that if there were no real difference, you would see a gap this large only about 3 percent of the time. That is a statement about the test under the assumption of no drift. It is a useful alarm trigger, but it is not a probability of drift, and it says nothing about whether the drift is large enough to hurt your model.

And the corollary that matters most for operations: with a threshold of 0.05, a perfectly stable system will still fire about one time in twenty per test, simply by chance. If you run a check every hour on stable data, you should expect roughly one false alarm a day, and with a batch of features each tested separately, more than that. This is not a bug. It is what the number means. The next section shows how the library helps you cope.

Many features at once: p_val and correction

Our detector tested five columns. When you test several features separately and ask "did any of them drift", chance works against you. If each column has a 5 percent chance of a false alarm, the chance that at least one of five fires is much larger than 5 percent. With a hundred columns, a false alarm on any given batch becomes nearly certain.

Statisticians call this the multiple comparisons problem, and the standard remedy is to make the per-column threshold stricter in a principled way. KSDrift offers this through the correction argument, which takes 'bonferroni' or 'fdr':

cd = KSDrift(x_ref, p_val=0.05, correction="bonferroni")

Bonferroni is the conservative option. It divides the threshold by the number of features, so that the chance of any false alarm across all features stays near your chosen p_val. Its cost is that it can miss a real but modest drift in one column. FDR, short for false discovery rate, is the more permissive option. Instead of controlling the chance of any false alarm, it controls the expected fraction of flagged features that are false alarms, which suits cases where you are happy to follow up a few spurious flags in exchange for catching more real ones.

Which should you choose? A reasonable beginner rule is to start with Bonferroni when an alarm pages a human and false alarms are expensive, and with FDR when you are exploring and will look at the per-feature numbers anyway. Either is better than ignoring the issue.

The other control is alternative, which defaults to 'two-sided', meaning the test flags a change in either direction. You can pass 'less' or 'greater' to look for one direction only, but until you have a reason, leave the default.

You also decide how to read the result. With drift_type="batch" the detector combines the per-feature outcomes, after correction, into a single yes-or-no answer. With drift_type="feature" you see each column's own result. A practical pattern is to alert on the batch answer and log the feature answer, so that when an alert fires, the first thing you read tells you which columns moved.

Choosing and caring for your reference data

The detector is only as good as its idea of normal, so spend real thought here. Several decisions are hiding in the phrase "a sample of training data".

Size. Too few instances and the reference distribution is poorly estimated, so the detector fires on noise or misses real change. A few hundred to a few thousand instances is a sensible range for the simple tests. More is not always better: for the heavier kernel-based detectors, cost grows quickly with the reference size, as the later guides explain.

Representativeness. The reference should look like the data the model is expected to see in healthy operation, not just the training set. If your training data was rebalanced or filtered, the reference should reflect what production normally looks like, otherwise you will get alarms from day one and learn to ignore them.

Freshness and the refresh trap. Reality moves, so some people refresh the reference with recent data. The library supports this through update_x_ref, which takes forms such as {'last': 1000} or {'reservoir_sampling': 1000} to keep a rolling or sampled reference. This is powerful and dangerous. If the reference always follows today's data, a slow drift is absorbed and never alerts, because each day looks like the previous day. Only update the reference when you deliberately want the detector to accept the new normal. As a beginner, keep a fixed reference and revisit it on purpose.

Privacy. The reference is a copy of part of your training data. If that data contains personal information, so does the file you save. In the Gulf and in Egypt, employers in banking, telecoms and health care often have strict rules about where such data may be stored and who may touch it. Treat the reference file with the same care as the training set: restricted access, and no casual copies to a laptop or a public notebook.

Order and structure. Rows should be independent instances. If your data is a time series in which neighbouring rows are strongly related, a basic per-feature test will overstate its confidence. That is an advanced topic, but be aware that the simple tests assume the rows are not copies of each other.

Outliers and adversarial inputs: the other two families

Drift detection answers a question about a whole batch. Sometimes the question is about one instance: is this single request so strange that I should not trust the model's answer? That is outlier detection, and it lives in alibi_detect.od. The catalogue includes Isolation Forest, Mahalanobis distance, auto-encoders (AE and VAE), the AEGMM and VAEGMM variants, likelihood ratios, a Prophet-based detector for time series, a spectral residual detector, and a sequence-to-sequence detector.

The idea differs from drift in one important way. Many outlier detectors are fitted on normal data first, and then they need a threshold on an instance score. You choose the threshold so that a chosen share of normal data would be flagged, for example the 99th percentile of scores on clean data. The library provides an infer_threshold method for this on the detectors that need it. The result of predict is again a dictionary, with an is_outlier flag per instance rather than a single is_drift flag.

Because the exact constructor arguments vary a lot between these detectors, and they change the examples considerably, this guide does not reproduce them. Use the outlier methods page in the Sources, pick the detector that matches your data type, and copy its example. For tabular data, Isolation Forest and Mahalanobis distance are the lightest starting points and need no deep-learning backend.

Adversarial detection is the third family, for inputs crafted on purpose to fool a model, usually images with tiny, human-invisible changes. The library offers an Adversarial Auto-Encoder detector and a Model Distillation detector. This is a specialised area; as a beginner, it is enough to know it exists and where it lives.

Why not start with outliers? Because drift gives a more actionable first signal for a deployed model: it is cheap, it needs no training for the simple tests, and it matches the operational question "should I worry about my model today?". Outlier detection becomes valuable when single bad requests matter, such as fraud, safety, or rejecting garbage input before it reaches an expensive model.

Images, text and big tables: preprocessing in one paragraph

The simple KS test works on numeric columns. Real data is often richer. An image has thousands of pixels; a sentence is not a number at all. Testing raw pixels directly works poorly, because pixel space is huge and noisy and almost any two image batches look "different" there.

The standard remedy is a preprocessing function, passed as preprocess_fn. Its job is to reduce each instance to a compact, meaningful representation before the statistical test runs. Typical choices are the output of an encoder network, or the probabilities your own classifier assigns, since a shift in what your model sees is exactly what you care about. The detector then compares the compressed representations instead of the raw inputs.

This is where the optional backends come in: building and running an encoder needs TensorFlow or PyTorch. It is also where the library is most powerful, because the same drift test now works on images and text. You do not need it for today's tabular work, and the mid-level guide takes this up properly. The takeaway for now is the shape of the idea: reduce first, then test.

Offline versus online, briefly

Everything so far was offline: a batch arrives, you compare it with the reference, you get a verdict. This fits a nightly job that checks yesterday's traffic, which is the most common setup in practice.

Some systems need a verdict as data streams in, one instance at a time. Online detectors exist for this, with names such as MMDDriftOnline, LSDDDriftOnline, CVMDriftOnline and FETDriftOnline. They keep a sliding window and a counter of how many instances they have seen. Two settings control them. The expected run-time, ert, is the average number of instances you are willing to wait before a false alarm on stable data. A larger value means fewer false alarms but slower detection. The window size, window_size, sets how much recent data each test looks at: small windows react fast to severe drift, larger windows are more sensitive to subtle drift.

Online detectors have a one-time setup cost, because the constructor simulates many runs to calibrate its threshold for your chosen ert and window_size. That can take noticeable time on a large reference set, so you do it once at start-up, not on every request. Online detectors are also the ones where state matters: the detector remembers where it is, and after an alert you reset it with reset_state().

If you are new, do not start here. Master the offline pattern first. The mid-level and senior guides cover online detectors, their state handling and their operational traps.

Saving a detector and loading it back

A detector you build in a notebook is useless to a scheduled job on another machine unless you can store it. Alibi Detect provides two functions for this:

from alibi_detect.saving import save_detector, load_detector

save_detector(cd, "./my_detector/")
cd = load_detector("./my_detector/")

The first call writes a folder; the second reads it and returns a ready detector. Try the round trip and call predict on the loaded detector to prove it behaves the same.

There are two storage formats, and understanding them avoids a lot of confusion. The config format is the default for drift detectors. It writes a human-readable file called config.toml plus supporting files such as the reference data as a .npy array. Because it is readable text, you can open it, edit a setting, and keep it in version control, which makes changes reviewable. A minimal config looks like this:

name = "KSDrift"
x_ref = "x_ref.npy"
p_val = 0.05

The name line is the detector class and every other line is a constructor argument. The example above is the shape of the file; open the one your own save produces to see exactly what your version writes.

The legacy format uses a Python serialisation library called dill. It is the only format available for outlier and adversarial detectors, and you can request it for drift detectors with save_detector(cd, path, legacy=True). It is not supported for some detectors, including the online ones. One further note from the 0.13.0 release: saving in legacy format with recent TensorFlow needs the environment variable TF_USE_LEGACY_KERAS=1 so that TensorFlow uses Keras 2.

Two safety points every beginner should absorb now. First, match versions. The docs warn that cross-version loading is not guaranteed: loading a detector saved by a different Alibi Detect version issues a warning, and the docs strongly recommend matching the library, Python and dependency versions when you save and load. Pin them in your requirements file. Second, only load detector files from sources you trust. Dill-based files can execute arbitrary code when loaded, exactly like a pickle file, so an untrusted detector folder is as dangerous as an untrusted program.

When things go wrong: configuration and common errors

Most beginner problems fall into a short list. Reading them in advance saves an evening.

The backend is missing. If you build a detector that needs a deep-learning framework, for example by passing a backend argument, and the matching extra is not installed, the construction fails, typically with an import-related error. The cure is to install the extra, alibi-detect[tensorflow] or alibi-detect[torch]. Remember the naming quirk: the extra is torch but the backend string is 'pytorch'.

pip cannot resolve a TensorFlow version. Alibi Detect caps TensorFlow below 2.19 in release 0.13.0. If the environment already holds a newer TensorFlow, the resolver complains or pulls an older one. Use a clean environment, or choose the PyTorch backend.

Binary incompatibility with NumPy. If you see an error that talks about binary incompatibility or a changed dtype size, two compiled packages were built against different NumPy generations. Align the versions of NumPy and the libraries that depend on it, or rebuild the environment from scratch.

A hang when importing both TensorFlow and PyTorch. PyTorch 2.0.0 to 2.0.1 could hang in this situation; the docs say it is fixed in 2.1.0 and later.

Shape and type problems in predict. The new data must have the same feature layout as the reference: same number of columns, same order. Offline detectors expect a batch, so passing a single row gives a meaningless test. Cast the arrays to float32 if a deep backend complains.

A loaded detector warns about versions. This is the expected behaviour described above. Match versions, or re-save the detector with the current library.

Alerts that never come, or never stop. These are not errors. A detector that never fires may have a reference that drifted along with the data, or a threshold that is too strict. A detector that fires constantly may have a reference that does not represent normal, or too many features and no correction. Reread the sections on reference data and correction before blaming the library.

When you meet an error not on this list, read the last line of the traceback first: it names the real problem. Then search the repository's issue tracker, and check the release notes in the changelog for something that changed in your version. A habit worth building early: always report your versions of Python, Alibi Detect, NumPy and any deep-learning library when you ask for help, because most problems in this area are version problems.

A short playbook for when an alert fires

Sooner or later your detector will say is_drift == 1. What you do in the next ten minutes decides whether the monitor earns trust or gets muted. Here is a calm sequence a beginner can follow, in order, and each step is cheaper than the one after it.

First, check that the check itself is sound. Did the batch contain enough rows? A batch of twelve instances gives a test very little to work with. Did the columns arrive in the same order as the reference? Did a file fail to load and get replaced by zeros or empty values? A surprising share of alerts are caused by a broken pipeline step rather than by the world changing, and finding that is good news, because the fix is ordinary engineering.

Second, look at which features moved. Run the per-feature report with drift_type="feature" and sort by distance. If one column carries nearly all the difference, ask what feeds that column. A unit change (kilometres to metres), a new category code, a join that started returning nulls, or a timezone shift each leave a clear fingerprint in a single column. If many columns moved a little together, the cause is more likely a change in who your users are, or a change in how the data is collected for everyone.

Third, look at the size, not just the p-value. Compare the K-S distances with those from earlier healthy batches. A distance that is only slightly above the usual range with a tiny p-value usually means the batch was large and the test was very powerful, not that anything serious happened. A distance several times the usual range deserves attention.

Fourth, check persistence. Run the detector on the next batch, and the one after. A single alert among many quiet days is the one-in-twenty chance you read about earlier. Three alerts in a row, all pointing at the same column, is a pattern.

Fifth, connect it to the model. If labels have arrived for any of the recent data, measure the model on them. If you have no labels yet, look at the distribution of the model's own predictions: a model that suddenly predicts one class far more often than before is telling you something. Drift in the inputs raises a question about the model, and only performance evidence answers it.

Finally, decide, and write it down. The options are to do nothing and keep watching, to fix the upstream problem, to retrain on recent data, or to update the reference because the new data is the new normal. Record the choice and the reason. Teams that write these decisions down learn which kinds of alert matter, and over a few months they tune their thresholds with evidence instead of instinct.

Where Alibi Detect fits in a real system

A detector on a laptop is a learning tool. In a real system it is one part of a loop. The common shape is a scheduled batch job: a scheduler such as Airflow starts a script every night. The script loads the saved detector with load_detector, reads the last day of production inputs from wherever they are logged, calls predict, and writes the verdict, the p-values and the distances to a store. A dashboard in Grafana or an alert rule in Prometheus turns those numbers into something a person sees.

Alibi Detect also has an official deployment path with Seldon Core and Knative eventing, in which model requests are logged as events and fanned out to detector services. The documentation for that path describes Seldon Core version 1, so check how it fits current Seldon and KServe setups before copying it. For a beginner, the batch job is the right model: it is simple, it is easy to test, and it uses only what you already learned.

Packaging the job in a container makes it reproducible; the Docker guide covers that. And if you track model versions in MLflow, store the detector folder and the reference file with the model version they belong to. A detector separated from the model it protects is a common source of confusion later.

Two habits make the loop trustworthy. Log the metadata (the detector name and library version from meta) alongside every result, so you can tie any alert to the code that raised it. And alert on sustained drift, not a single batch, for the reason covered earlier: a lone is_drift == 1 is often chance.

Putting it all together

Here is one small end-to-end project that uses everything above. It builds a reference, saves a detector, then simulates a week of daily batches in which drift begins on day five, and prints a daily report. Run it and watch the detector stay quiet and then fire.

import numpy as np
from alibi_detect.cd import KSDrift
from alibi_detect.saving import save_detector, load_detector

rng = np.random.default_rng(7)

# 1. Reference data: 2000 healthy instances, 8 features
x_ref = rng.standard_normal((2000, 8)).astype(np.float32)

# 2. Build, save, and reload (as a scheduled job would)
detector = KSDrift(x_ref, p_val=0.05, correction="bonferroni")
save_detector(detector, "./kst_detector/")
detector = load_detector("./kst_detector/")

# 3. Simulate seven days of production batches
for day in range(1, 8):
    batch = rng.standard_normal((500, 8)).astype(np.float32)
    if day >= 5:
        batch[:, 2] += 0.6          # feature 2 slowly goes wrong from day 5

    preds = detector.predict(batch, drift_type="feature",
                             return_p_val=True, return_distance=True)
    p_vals = preds["data"]["p_val"]
    dist = preds["data"]["distance"]
    worst = int(np.argmax(dist))
    print(f"day {day}: drift={preds['data']['is_drift']} "
          f"worst feature={worst} distance={dist[worst]:.3f} p={p_vals[worst]:.4f}")

Read the output the way an operator would. Days one to four should show no drift, and from day five the worst feature should be feature 2 with a large distance and a tiny p-value. Notice what you did without effort: you built the detector once, saved it, reloaded it as a separate job would, and produced a report that names the column responsible. That is the core of a real drift monitor.

Now extend it, as an exercise rather than a copy-paste. Make the shift on day five small (0.1) and see whether you still catch it with 500 instances per batch. Add correction="fdr" and compare. Write each day's line to a CSV file instead of printing it. Wrap the detection in a function that returns a dictionary of the fields you want to log, including the library version from preds["meta"]. Each change teaches a different lesson about sensitivity, correction or logging.

What you can now do, and what comes next

You can now explain what drift is and why a model fails silently without it. You know the three detector families and the four nouns: detector, reference data, test with a threshold, and result. You can install the library in a clean environment and check it. You have run a Kolmogorov-Smirnov drift check, read the verdict, p-values and distances, and you know that a p-value is not a probability of drift. You can control false alarms with p_val and correction, you understand the risk of refreshing the reference, you can save and reload a detector, and you can name the common errors and their causes. You also know the licence, which matters more than any technical detail when the work moves from learning to shipping.

What the next level adds: heavier detectors such as MMD and learned-kernel drift, preprocessing with encoders so that images and text can be tested, the details of the config files and the registry, the online detectors in depth, and testing and performance. After that, the senior guide covers designing monitoring for a team, failure modes at scale, security of serialised detectors, and when to choose something else. If your interest is the wider monitoring picture, read Evidently and NannyML next, and compare how they answer the same questions.

The most valuable thing you can do now is to use a detector on data you care about. Take a public dataset, split it by time or by category so that the halves differ, and see what the detector says. Intuition for drift comes from watching real data misbehave.

Sources

• Alibi Detect documentation
• Getting started and installation
• Algorithm overview
• Drift detection overview
• Kolmogorov-Smirnov drift detector
• Online MMD drift detector
• Outlier detection methods
• Adversarial detection
• Saving and loading detectors
• Config files
• Deployment with Seldon Core and Knative
• GitHub repository
• Changelog
• Licence
• PyPI package page

This file prepares you for the questions an interviewer asks when your CV says Alibi Detect, or more often when it says "model monitoring" and the interviewer wants to know whether you understand drift. Read the model first, then rehearse the answers out loud. An answer you can say in your own words beats one you memorised.

The thirty-second answer

Alibi Detect is a Python library from Seldon for outlier, adversarial and drift detection. You give a detector a reference sample that represents normal data, and it tests whether new data differs from it, returning a dictionary whose key field, is_drift, is 1 when it found a difference.

It matters because a deployed model degrades silently: it keeps returning valid-looking answers while the inputs move away from the training data. Watching the inputs gives an early warning before the true labels arrive. One caveat to volunteer without being asked: since version 0.11.5 the library is under the Business Source License, so production use at a company needs a commercial licence.

That last sentence is worth practising. Interviewers for MLOps roles are impressed by candidates who raise licensing before being asked, because it shows you have shipped or thought about shipping.

The model to keep in your head

Four nouns carry almost every answer. A detector is a Python object with predict(), in one of three families: drift, outlier, adversarial. Reference data, x_ref, is the sample treated as normal. The test and threshold is a statistical test that yields a p-value, compared with your chosen p_val. The result is a dictionary with a data part (verdict and numbers) and a meta part (detector name, type, version).

Add the two distinctions interviewers love. Offline versus online: offline detectors compare a whole batch with the reference on each call, while online detectors take one instance at a time with a sliding window, controlled by ert and window_size. Data drift versus concept drift: data drift is a change in the input distribution, concept drift is a change in the relationship between inputs and the right answer. Input tests detect the first directly and can only hint at the second.

If you can draw the loop, do it: reference data and a new batch go into a detector, a verdict comes out, a human or pipeline decides what to do. The library detects; it does not retrain or fix.

The commands and code to have ready

You should be able to write these from memory on a whiteboard.

Installing, with the quoting trap:

pip install alibi-detect
pip install "alibi-detect[torch]"

A first drift check:

import numpy as np
from alibi_detect.cd import KSDrift

x_ref = np.random.randn(1000, 5).astype(np.float32)
cd = KSDrift(x_ref, p_val=0.05)
preds = cd.predict(x_new)
print(preds["data"]["is_drift"])

Per-feature detail:

preds = cd.predict(x_new, drift_type="feature", return_p_val=True, return_distance=True)

Save and load:

from alibi_detect.saving import save_detector, load_detector
save_detector(cd, "./my_detector/")
cd = load_detector("./my_detector/")

Be ready to explain each line. Why float32? Why no fit? Why does the second call pass drift_type="feature"? The honest answers: float32 because the deep-learning backends expect it and it is a good habit; no fit because drift detectors take the reference at construction; per-feature because it tells you which column moved.

The classic traps and why they happen

Misreading the p-value. The trap: saying a p-value of 0.02 means a 98 percent chance of drift. The truth: it is the probability of a difference at least this large if there were no drift. It is an alarm trigger, not a probability of drift.

False alarms from repeated testing. The trap: running many checks, or many features, at p_val=0.05 and treating every alert as real. At 0.05 a stable system fires about one time in twenty per test. With many features, the chance that at least one fires is large. The remedy is the correction argument: 'bonferroni', conservative, or 'fdr', which controls the expected share of false flags.

Refreshing the reference blindly. The trap: using update_x_ref so the reference always follows recent data. A slow drift is then absorbed and never alerts. Use it only when you deliberately accept the new normal.

Drift is not performance loss. The trap: assuming an alert means the model got worse. Inputs can change in ways the model handles fine, and the model can fail with no input drift at all (concept drift). Confirm with labels or prediction distributions.

Version and format mismatches. The trap: loading a detector saved by another library version. The docs say cross-version loading is not guaranteed and strongly recommend matching versions. Also, the legacy format uses dill, so loading an untrusted file can run arbitrary code.

Backend naming. The pip extra is torch, the backend string is 'pytorch'. Tiny, but it has cost many people an hour.

Licence surprise. Apache 2.0 up to 0.11.4, Business Source License 1.1 from 0.11.5. Non-production use is allowed, and production use by non-profit educational institutions is allowed under the additional grant, while other production use needs a commercial licence from Seldon.

Common interview questions

What is drift and why monitor it? Drift means the data the model sees in production differs from the data it was trained on. A model is a summary of its training data, so when inputs move it operates outside what it learned, but it does not crash. It returns plausible, worse answers. Monitoring inputs gives early warning, often before labels arrive, which can be weeks or months for outcomes like loan default or churn.

How does Alibi Detect decide there is drift? A drift detector runs a statistical two-sample test between the reference data and the new batch. For the Kolmogorov-Smirnov detector that is a per-feature test; the p-values are combined with a correction when several features are tested. If the result falls below the threshold p_val, is_drift is 1. Other detectors use different tests, such as maximum mean discrepancy, but the shape is the same.

What goes in the reference data, and how big should it be? A sample representing healthy production operation, usually from training or validation data, not necessarily the raw training set if that was rebalanced. A few hundred to a few thousand rows is a sensible range for simple tests. Too small a sample gives a jumpy detector. For kernel-based detectors, cost grows quickly with size, so you keep it modest. I would also treat the file as sensitive, because it is a copy of training data.

Why do you need a correction when testing many features? Each per-feature test has its own chance of a false alarm. Ask whether any of fifty features drifted and the chance of at least one false alarm is high even on stable data. Bonferroni tightens the per-feature threshold so the overall chance stays near the chosen level, at the price of missing modest real drift. FDR allows an expected fraction of false flags, which suits exploratory use.

What is the difference between an outlier detector and a drift detector? A drift detector asks whether a batch differs from the reference as a whole, a statement about distributions. An outlier detector asks whether one instance is unlike the training data, a statement about a single point. Many outlier detectors are fitted on normal data first and then need a threshold on an instance score, set with infer_threshold.

What are online detectors, and what are ert and window_size? Online detectors process one instance at a time with a sliding window and a running timestep, for streams. ert is the expected run-time in the absence of drift, the average number of instances before a false alarm; larger means fewer false alarms and slower detection. window_size sets how much recent data is tested; small windows catch severe drift fast, large windows are more sensitive to subtle drift. They also need an up-front threshold calibration, so you do it once, not per request.

What would you do when is_drift is 1? Check the check first: batch size, column order, broken upstream steps. Then look at per-feature distances to see which columns moved. Compare the size with healthy batches, check persistence over the next batches, and connect it to model performance or prediction distributions. Then decide: watch, fix upstream, retrain, or update the reference, and record the reason.

Does drift always mean you must retrain? No. Drift is a reason to investigate. It may be a pipeline bug, a harmless change, or a real shift that needs retraining. Retraining on drifted data without understanding the cause can bake in a bug.

How do you handle images or text? Raw pixels or tokens are too high-dimensional for a direct test. You pass a preprocess_fn that reduces each instance to a compact representation, such as an encoder's output or your own model's probabilities, and test that. The principle is reduce first, then test.

How do you save and ship a detector? With save_detector and load_detector. The default config format for drift detectors writes a readable config.toml plus artefacts such as the reference array, which can sit in version control. The legacy format uses dill. I would pin the library and Python versions, store the detector alongside the model version, and only load files from trusted sources.

Can you use Alibi Detect in a commercial product? Not freely. Since 0.11.5 it is under the Business Source License 1.1. Non-production use is permitted, and so is production use by non-profit educational institutions under the additional use grant. Other production use needs a commercial licence from Seldon. I would check the LICENSE for the exact version and involve whoever owns legal decisions, and I would note the MLServer issue about removing the library from a base image for this reason.

How does it compare with Evidently or NannyML? All answer similar questions. Alibi Detect is a lower-level, programmable library with a wide set of detectors for tabular, image, text and time-series data and online variants. Evidently and NannyML lean towards reports and dashboards for tabular data. I would choose by data type, by whether I need a stream, and by licence. See the Evidently and NannyML guides.

Where would the detector run in a real system? Most often as a scheduled batch job: load the saved detector, read the last window of production inputs, call predict, and write is_drift, p-values and distances to a metrics store that feeds dashboards and alerts. Tools such as Airflow, Prometheus and Grafana fit around it.

What data-residency concerns apply? The reference file is a copy of training data and may contain personal information. In Gulf and Egyptian organisations, banking, telecom and health data often have rules on where they may be stored. I would store the reference in the approved region, encrypt it, and apply the same access controls as the training set.

Follow-up questions interviewers use to probe depth

These are the second questions, the ones that separate someone who has run a detector from someone who has read about one.

You said the test has a false-alarm rate. How would you measure yours? Take a stretch of historical data you believe was healthy, split it into many batches of the size you will use in production, and run the detector on each against the reference. The share of batches that fire is your empirical false-alarm rate. If it is far above what p_val and the correction suggest, the reference probably does not represent normal operation, or the rows are not independent, for example because several rows come from the same customer. This is the single most convincing thing a candidate can describe, because it turns a vague worry into a number.

Your detector never fires. Is that good news? Not by itself. A silent detector is only good news if you have shown it can fire. I would run the known-answer test: a batch from the reference distribution and a deliberately shifted batch. If the shifted batch does not alert, the detector is too insensitive, perhaps because the reference is refreshed automatically and follows the data, or because the shift is too small for the batch size. Silence proves nothing until sensitivity is demonstrated.

One feature drifted and the other forty-nine did not. Does the batch-level answer catch it? It depends on the correction. With Bonferroni the per-feature threshold is tightened, so a modest shift in one column may not cross it, while a large shift will. This is why I alert on the batch answer but also log the per-feature numbers, and why for critical columns I might run a separate detector with its own threshold.

Why not just compare means and standard deviations? Summary statistics miss changes in shape, such as a distribution becoming bimodal with the same mean, and they have no principled answer to how much wobble is normal. A two-sample test compares whole distributions and gives a calibrated p-value. Means are still useful as a quick dashboard, but they are not a test.

What would make you choose not to use Alibi Detect? Licence is the first reason: a commercial production system without a Seldon licence. The second is fit: if I only need ready-made tabular reports and dashboards, a report-oriented tool may be simpler. The third is dependency pressure: the TensorFlow cap below 2.19 in the current release can clash with a platform base image, though the PyTorch backend avoids that.

How would you explain drift to a non-technical manager? I would say that a model is like a student who studied last year's exam. If this year's questions look different, the student still answers confidently but may be wrong, and nobody tells us until the results come out. Drift detection is a quick check of whether this year's questions look like last year's, so we can look into it before the results arrive.

What is a reasonable first deliverable for a drift monitor? A nightly job for one model: a fixed reference set, a KS detector with a chosen correction, results logged with metadata, a known-answer test in CI, and a one-page playbook saying what to check when an alert fires. It is small, it is explainable, and it produces numbers you can improve on.

What a strong answer includes

Interviewers listen for four things. First, you separate drift from performance: you never claim an alert equals a worse model. Second, you know false alarms are built in to any threshold and you can say how to manage them. Third, you treat the reference data as a design decision rather than an afterthought. Fourth, you mention licence and versions unprompted. A weak answer recites detector names. A strong answer tells a short story: "we had a reference of two thousand rows, a nightly job, Bonferroni correction because pages were costly, and when an alert fired we looked at the per-feature distances first."

If you have no production experience, say so, and describe a project you ran on public data where you deliberately damaged a column and watched the detector respond. That honest, concrete story is worth more than borrowed vocabulary.

Sixty-second self-test

Answer aloud, then check against this file.

1. Name the three detector families and what question each answers.
2. What does x_ref hold, and why is there no fit call for a drift detector?
3. What does a p-value of 0.03 mean, and what does it not mean?
4. At p_val=0.05, how often does a stable system false-alarm per test?
5. What do Bonferroni and FDR trade off?
6. What does drift_type="feature" change?
7. Why can refreshing the reference hide a slow drift?
8. What are ert and window_size?
9. Why must library versions match between save and load, and why must you trust the file?
10. From which version is the licence a Business Source License, and who may use it in production?

If you stumbled on more than two, reread the matching section of the detailed guide and try again tomorrow.

Sources

• Alibi Detect documentation
• Saving and loading
• Licence

These are the mistakes beginners actually make with Alibi Detect, each with the symptom you will see, the cause, and the fix. Then come the habits worth adopting on day one, and a checklist you can paste into a project README. Read it once now, and again after your first week with real data.

Mistake 1: treating every alert as an incident

Symptom. The nightly job fires is_drift == 1 about once every few weeks on a system where nothing changed, someone investigates for an hour, finds nothing, and the team starts ignoring the alert.

Cause. A threshold of p_val=0.05 means a stable system will falsely alarm about one time in twenty per test. If you check many features or check often, false alarms are guaranteed. The alert was never a statement of certainty.

Fix. Alert on persistence: require the same signal in several consecutive batches before paging anyone. Use the correction argument when testing many features. Log every result, including the quiet ones, so you can measure your own false-alarm rate instead of guessing it. A monitor that cries wolf is worse than none, because it trains people to look away.

Mistake 2: reading the p-value as the probability of drift

Symptom. A report says "97 percent probability the data has drifted" because the p-value was 0.03.

Cause. The p-value is the chance of a difference this large if there were no drift. It is not the chance that drift exists, and it says nothing about size.

Fix. Report the p-value and the distance together. The p-value says whether to believe a difference exists. The distance says how large it is. With a large batch, a trivial difference gets a tiny p-value, so a distance close to your usual healthy range is a reason to relax.

Mistake 3: a reference set that is wrong in a quiet way

Symptom. The detector fires constantly from the first day, or never fires even when you know the data changed.

Cause. The reference does not represent healthy production. It might be a rebalanced training set, an old period before a known change, or so small that the test is jumpy. Or it might be refreshed automatically, so it follows the data and absorbs slow drift.

Fix. Choose the reference deliberately: a few hundred to a few thousand rows that look like normal operation. Keep it fixed at first. If you use update_x_ref with {'last': N} or {'reservoir_sampling': N}, do so on purpose, knowing that the detector will then accept slow change as the new normal. Write down where the reference came from and when.

Mistake 4: the wrong shape, type or column order

Symptom. A shape or type error in predict, or worse, no error and a nonsense verdict.

Cause. The new data must have the same feature layout as the reference: rows are instances, columns are features, in the same order. Swapping two columns raises no error; it compares the wrong features. Offline detectors expect a batch, so a single row gives a meaningless test. Deep backends can complain about 64-bit floats.

Fix. Build one function that turns raw data into the array, and use it for both the reference and the new batches. Assert the shape before calling predict. Cast to float32. Keep the list of column names next to the saved detector so that nobody reorders them later.

Mistake 5: installing the wrong extra, or forgetting the quotes

Symptom. Zsh says "no matches found: alibi-detect[torch]", or a detector fails to construct because a backend is missing.

Cause. In zsh, unquoted square brackets are treated as a pattern. Separately, detectors that use a deep-learning backend need the matching extra, and the names differ: the pip extra is torch, the backend string is 'pytorch'.

Fix. Quote it: pip install "alibi-detect[torch]". Install only the extras you need. For KS drift on tables, the default install is enough, and avoiding unnecessary extras avoids unnecessary dependency fights.

Mistake 6: fighting TensorFlow and NumPy versions

Symptom. pip reports it cannot find a compatible set of packages, or an import fails with an error about binary incompatibility.

Cause. As of release 0.13.0, TensorFlow is capped below 2.19, so a very new TensorFlow does not fit. Separately, compiled packages built against different NumPy generations clash.

Fix. Use a fresh virtual environment per project. Start from the default install, and add one extra at a time. If the TensorFlow cap bothers you, use the PyTorch backend. Record the full set of versions with pip freeze once things work.

Mistake 7: saving a detector with no record of the versions

Symptom. Months later a new environment loads the detector and prints a version warning, or fails to load it.

Cause. The docs say loading across versions is not guaranteed, and they strongly recommend matching the library, Python and dependency versions between save and load. Detectors saved in the legacy dill format are also sensitive to the Python version.

Fix. Write alibi_detect.__version__, the Python version and the dependency list into a small text file next to the saved folder. Pin them in your requirements. After any upgrade, load every saved detector in a test and re-save if needed. If you hit an old config that mentions config_spec, that field was removed in 0.11.0, and the docs say to delete the line by hand.

Mistake 8: loading a detector you did not make

Symptom. None, until it matters. A colleague sends a detector folder, you load it, and you have run code you never read.

Cause. The legacy format uses dill, and PyTorch models use dill pickling. Like any pickle, loading can execute arbitrary code.

Fix. Treat detector folders like executables. Load only from sources you control, keep them in a registry with checksums, and prefer the readable config format where you can, because you can read config.toml before loading it.

Mistake 9: drift equals "the model is broken"

Symptom. An alert triggers a retraining job that changes nothing, or an expensive rollback.

Cause. Drift in inputs is a question, not an answer. The model may handle the change well. The reverse is also true: concept drift can hurt the model while the inputs look the same, and no input test will catch it.

Fix. Pair drift alerts with performance signals. When labels arrive, measure the model on them. Until then, watch the distribution of the model's predictions. Decide using both. The MLflow guide covers tracking model versions and metrics.

Mistake 10: skipping the licence

Symptom. A prototype built on Alibi Detect goes into a client's production system, and months later procurement or legal asks which licence applies.

Cause. The project changed from Apache 2.0 to Business Source License 1.1 in 0.11.5 (January 2024). Non-production use is allowed, and non-profit educational institutions may use it in production under the additional grant, but other production use needs a commercial licence from Seldon.

Fix. Read the LICENSE file for the exact version before the first commit that touches a client. Decide early whether the project can use the library, or should use an alternative from this catalogue, such as Evidently or NannyML. Put the decision in the README.

Mistake 11: leaving the reference data lying around

Symptom. A file named x_ref.npy sits in a shared folder or a public repository.

Cause. The reference is a copy of part of the training data. If that data includes personal or confidential information, so does the file. For Gulf and Egyptian employers in banking, telecom and health, there are often rules on where such data may live.

Fix. Treat the file as you treat the training set: restricted access, approved region, encryption, and keep it out of Git unless the data is public. Use a data version control tool or an artefact store with access control.

Mistake 12: testing high-dimensional data directly

Symptom. A detector on raw images or long text vectors fires on almost everything, or on nothing.

Cause. In very high dimensions a per-feature or kernel test has little power and a lot of noise. The library's answer is to reduce first.

Fix. Pass a preprocess_fn that maps each instance to a compact representation, such as an encoder's output or your own model's probabilities, and test that. This needs a deep-learning backend extra, and the mid-level guide covers it in detail.

Habits worth adopting on day one

Run a known-answer test first. Before you trust a detector on real data, show it a batch from the reference's own distribution and a deliberately shifted batch. If it does not say quiet and then loud, something in your setup is wrong. Keep this as an automated test.

Make the pipeline reproducible. Fix random seeds in examples, pin versions, and keep the script and the saved detector in version control or an artefact store. Package the job in a container so it runs the same everywhere; the Docker guide shows how.

Log everything with its metadata. Store is_drift, the per-feature p-values and distances, the detector name and the library version from meta, the batch size, and the date. When someone asks in March why an alert fired in January, the answer should be one query away.

Report effect size, not only significance. Teach everyone who reads your report to look at the distance next to the p-value.

Prefer the simplest detector that answers the question. KS drift on tabular data needs no deep-learning backend and is easy to explain. Reach for kernel-based and learned detectors only when the simple one is clearly inadequate.

Keep the reference fixed until you have a reason. Change it on purpose, record why, and record the date.

Write the alert playbook before the first alert. Decide in advance who looks at an alert, which three things they check, and what the possible outcomes are: watch, fix upstream, retrain, or accept the new normal.

Start with the question, not the detector. Ask what you are afraid will happen to the model. A broken pipeline, a new customer segment, a changed unit, and an evolving fraud pattern all call for different checks. Pick the detector last.

Test on data you understand. Split a public dataset by time or category so that the halves really differ. When you know the answer in advance, you learn how the detector behaves.

Check the documentation for your exact version. The library changed licence, Python support and serialisation details across releases. The changelog is short and worth a skim whenever you upgrade.

Reading your first week of results

The first week with a real detector is a learning exercise about your own data, more than a monitoring exercise. Use it deliberately.

Expect noise, and write down how much. In the first days, record every verdict, even the quiet ones, and note which alerts you could explain. By the end of the week you will have a rough rate of unexplained alerts. That rate is the honest baseline for your system, and any later threshold change should be judged against it. If you tune the threshold before measuring the baseline, you are guessing.

Look at the distances over time, not just the verdict. Plot the largest per-feature distance for each day. A healthy system shows a band of ordinary wobble. A real drift shows a rise that stays up, and a one-off spike that returns to the band is almost always chance or a one-time data glitch. The plot teaches your eye what normal looks like much faster than reading a column of zeros and ones.

Find the columns that are always noisy. Some features are naturally unstable: counts that depend on the day of the week, values that spike at month end. If one column causes most of your false alarms, you have three honest options. Build the reference to include those cycles, check batches of a comparable period (weekday against weekday), or test that column separately with a stricter threshold. Do not silently drop it, because the column you drop is the one that breaks later.

Compare like with like. A batch of Friday traffic compared with a reference built mostly from Tuesday traffic will look different for ordinary reasons. Seasonality, payday effects and holiday periods all matter, and in the Gulf and Egypt the timing of Ramadan, Eid and national holidays can change behaviour sharply. If your data has such cycles, make sure the reference covers them, and expect alerts around the cycle boundaries until it does.

Keep a short diary. For each alert, write one line: what it was, what you found, what you did. After a month, that diary is your tuning guide and your evidence when somebody asks whether the monitor is worth keeping.

Do not tune on the alert you just saw. The temptation, after a false alarm, is to loosen the threshold until it goes away. Resist it for at least a week of data. A threshold tuned to silence one alert tends to silence the real one next month.

Share the first report with someone who knows the data. An analyst or engineer who owns the upstream tables can often explain an alert in a minute: a campaign started, a form field changed, a partner system was migrated. Ask them to look at your per-feature table before you build anything more elaborate. Their explanations become the first entries in your playbook, and they are also the quickest way to learn which alerts the business actually cares about, which is the question no threshold can answer for you.

A short checklist

Copy this into your project README and tick it off.

• Python 3.9 or newer, in a fresh virtual environment, with the versions of Python, Alibi Detect, NumPy and any deep-learning library recorded.
• Brackets quoted when installing extras, and only the extras I need installed.
• Licence for the exact version read, and the production-use decision written down.
• Reference data chosen on purpose: size, source, date, owner, storage location, access rules.
• Reference and new data built by the same function, same column order, float32.
• p_val chosen deliberately, and correction set when testing many features.
• Known-answer test in place: quiet on same-source data, loud on shifted data.
• Result logging includes verdict, per-feature p-values and distances, detector name, library version, batch size and date.
• Alert rule requires persistence, not a single batch.
• An alert playbook written: who checks what, and the four possible outcomes.
• Detector saved with save_detector, versions recorded beside it, and a test that loads it.
• Detector folders loaded only from trusted sources.
• A plan to connect drift alerts to model performance when labels arrive.

Sources

• Alibi Detect documentation
• Changelog

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. If you already run a KSDrift check in a notebook, this level is about turning that check into something a team can trust at three in the morning.

Where this picks up

Each section starts with the problem it solves, and ends with a Try it you can do on a real project in a few minutes. Everything here targets Alibi Detect 0.13, the release that added Python 3.12 support, and where an older release behaved differently the text says so.

What a drift detector actually does

The beginner path showed you cd.predict(x_new) and a number that said 0 or 1. To use a detector well you need to know what was computed, because every false alarm and every missed alarm traces back to it.

A drift detector holds a reference sample, x_ref, and answers one question about each batch you hand it: is it plausible that this batch and the reference were drawn from the same distribution? That is a two-sample hypothesis test. The null hypothesis says there is no difference. The test computes a statistic from the two samples, converts it into a p-value, the probability of seeing a difference at least this large if the null were true, and compares that p-value with the threshold you set through p_val. A p-value below the threshold is reported as drift, is_drift equal to 1.

Three consequences follow that surprise people the first time.

First, the answer depends on both samples. A batch of 20 instances has little statistical power, so a real shift can pass unnoticed, while a batch of 50,000 instances will flag differences too small to matter to anyone. The detector reports statistical significance, not business impact. If you feed it huge batches, expect it to find drift constantly and decide separately what size of shift deserves a human.

Second, the test is about the input distribution, p(x), unless you deliberately point it at model outputs. Alibi Detect's drift detectors do not need labels, which is their great advantage in production, and also their limit: input drift is a warning sign, not proof that the model got worse. The relationship between inputs and labels can change without the inputs moving at all, and inputs can move without hurting accuracy.

Third, the detector is stateful only in the sense that it stores x_ref and whatever it derived from it. An offline detector such as KSDrift or MMDDrift treats each predict call as independent. Nothing about the previous batch influences the next result unless you configured reference updates. This makes offline detectors easy to reason about and easy to run in parallel.

The return value is a dictionary with two keys, and it pays to know it by heart:

preds = cd.predict(x_new, drift_type='feature', return_p_val=True, return_distance=True)

preds['meta']   # name, detector_type, data_type, online, version
preds['data']   # is_drift, p_val, threshold, distance

meta identifies the detector that produced the result. Since release 0.11, meta['detector_type'] says 'drift', 'outlier' or 'adversarial', and a separate online field says whether the detector is online. Older code that tested detector_type for the words online or offline breaks silently on current versions, which is worth grepping for if you inherited a monitoring script.

data holds the verdict. For KSDrift, drift_type='batch' returns one decision after a multiple-testing correction, while drift_type='feature' returns the verdict feature by feature. return_p_val and return_distance are opt-in, so a monitoring job that wants to chart them must request them. The distance for the Kolmogorov-Smirnov detector is the K-S statistic per feature, the largest vertical gap between the two empirical distribution functions. It is bounded between 0 and 1 and is much more useful on a dashboard than the p-value, because it behaves like an effect size: it does not shrink toward zero just because your batch got bigger.

Choosing the detector for the data you have

Alibi Detect ships a long list of drift detectors, and picking by name recognition is the commonest mid-level mistake. The decision follows from three properties of your data: whether features are numeric or categorical, how many dimensions you test at once, and whether you have a model whose outputs you can monitor.

Univariate tests per feature. KSDrift runs a Kolmogorov-Smirnov test on every feature separately and combines the results. CVMDrift does the same with the Cramér-von Mises test, which weighs the whole distribution more evenly and is more sensitive to differences in the middle of it. FETDrift uses Fisher's exact test for binary features. ChiSquareDrift handles categorical features. These detectors are cheap, they need no deep-learning backend, and most importantly their output is explainable: when one fires, you know which column moved. For tabular data with tens of columns, start here.

Mixed-type tabular data. TabularDrift applies the appropriate test per column, a K-S test for numeric columns and a chi-squared test for categorical ones, and combines them. Real business tables are almost always mixed, so this is the natural entry point for a feature table that includes a country code or a plan type next to numeric values.

Multivariate kernel tests. MMDDrift computes the Maximum Mean Discrepancy between the two samples, and LSDDDrift uses the Least-Squares Density Difference. Both look at the joint distribution, so they can notice that two features which were uncorrelated are now correlated even though each marginal looks unchanged. A per-feature test cannot see that. The price is cost: the p-value comes from a permutation test controlled by n_permutations, and each permutation involves kernel evaluations across the pooled sample, so time and memory grow roughly with the square of the sample size. Both need a backend, 'tensorflow', 'pytorch', or for MMD also 'keops' on Linux with CUDA.

Learned detectors. ClassifierDrift trains a classifier to separate reference from test instances. If a classifier can tell them apart better than chance on held-out data, the distributions differ. It scales to high-dimensional data and, as a bonus, can point to the instances it found most distinctive. LearnedKernelDrift trains the kernel itself on part of the reference to make an MMD test more powerful. SpotTheDiffDrift is a more interpretable variant of the classifier approach. ContextMMDDrift conditions on a context variable, such as the time of day or the region, so that expected seasonal variation does not trigger alarms.

Model-output detectors. ClassifierUncertaintyDrift and RegressorUncertaintyDrift monitor the uncertainty of your model's predictions rather than the raw inputs. They work without labels and aim at what you care about: is the model becoming less sure than it used to be? They are an excellent complement to input tests because they reduce a high-dimensional problem to a single number per instance.

A practical decision rule is to climb the ladder only as far as you need. Begin with univariate tests per feature. If you suspect interactions, add one MMDDrift over a reduced representation. If the data is images or text, go straight to a preprocessing step plus a kernel or classifier detector, which the next section covers. Keep the model-uncertainty detector in mind whenever the model exposes probabilities.

Preprocessing: testing the right representation

Statistical tests degrade quickly as dimension rises. A 224 by 224 colour image has about 150,000 values. Running a K-S test on each pixel yields 150,000 p-values that mostly reflect lighting noise, and an MMD test in that space loses power because distances between points in very high dimensions become similar. The fix is to reduce the data to a compact representation that keeps what matters, and then test that. In Alibi Detect, the hook for this is preprocess_fn.

preprocess_fn is any callable that maps a batch of raw data to a batch of features. The detector applies it to x_ref and to every batch you pass to predict. Three sources of preprocessing are common in practice:

• An untrained or randomly initialised encoder. Surprisingly, a random convolutional encoder already works as a drift-sensitive projection, and it needs no training data labels. It is a good baseline.
• A trained encoder or the penultimate layer of your own model. This focuses the test on features your model finds meaningful.
• Your model's softmax output. Testing the predicted probabilities collapses the problem to the number of classes and ties drift to the model's own view of the data.

Alibi Detect provides a helper, preprocess_drift, for wrapping a model for use as preprocess_fn. Combine it with functools.partial so the detector receives a one-argument callable:

from functools import partial
from alibi_detect.cd import MMDDrift
from alibi_detect.cd.tensorflow import preprocess_drift

preprocess_fn = partial(preprocess_drift, model=encoder, batch_size=128)
cd = MMDDrift(x_ref, backend='tensorflow', p_val=.05, preprocess_fn=preprocess_fn)

A PyTorch equivalent lives under alibi_detect.cd.pytorch, and the backend string for PyTorch is 'pytorch' even though the pip extra is spelled torch. That naming mismatch costs people an afternoon, so it is worth memorising.

Two constructor flags control when the preprocessing runs, and they matter for startup time and memory. With preprocess_at_init=True, the default, the detector encodes x_ref once when you build it and stores the encoded version, so each predict call only encodes the new batch. If you already have the embeddings, for example because they were computed by a feature pipeline, pass them with x_ref_preprocessed=True and the detector skips the encoding step entirely. Set preprocess_at_init=False only in the unusual case where the preprocessing depends on both the reference and the test data, since then the reference must be re-encoded on every call.

Data types need attention too. Deep backends expect float32, and passing float64 arrays is a frequent source of dtype complaints in TensorFlow and PyTorch. Cast once at the boundary of your detector service rather than scattering astype calls through the code.

Choosing the encoder is a modelling decision with monitoring consequences. An encoder trained on data from last year is a frozen ruler: it measures drift in the directions it learned to care about, and it is blind to changes along directions it never saw. If your input distribution can change in ways the encoder was not trained to represent, add a second, cheaper signal such as a univariate test on a handful of hand-picked summary statistics, for example image brightness or text length. Simple statistics catch gross changes that sophisticated embeddings sometimes absorb.

Thresholds, p-values and the arithmetic of false alarms

A drift monitor that cries wolf gets muted, and a muted monitor is worse than none. Most of the engineering effort at this level goes into making alerts rare enough to be believed and sensitive enough to be useful. That starts with arithmetic.

Under no drift at all, a test at p_val=0.05 still reports drift in about one run out of twenty, because that is what a 5 percent significance level means. If you check once a day, you will see roughly one false alarm every three weeks on perfectly healthy data. That is the floor for a single test. Now multiply by the number of features. A KSDrift over 40 features performs 40 tests. With no correction, the chance that at least one reports drift on stable data is about 1 minus 0.95 to the power of 40, which is roughly 87 percent. A monitor configured that way would fire nearly every day for nothing.

The correction argument exists for this. It takes 'bonferroni', which divides the threshold by the number of tests and so controls the chance of any false positive, or 'fdr', which controls the expected fraction of false positives among the features flagged, a less conservative choice that keeps more power when many features truly drift. With drift_type='batch', the detector combines the per-feature results after correction into a single batch-level decision. Use Bonferroni when one false alarm is expensive and you have few features. Use the false discovery rate when you have many features and can tolerate a few spurious ones.

Beyond a single detector, think about alert policy as separate from detection. A few patterns work well:

• Require persistence. Alert on drift in, say, three of the last five batches, rather than a single one. A single flagged batch is noise at a rate you can compute, while three in a row is evidence.
• Alert on magnitude as well as significance. Combine is_drift with a floor on the statistic, so that a statistically significant but tiny difference on a large batch does not page anyone.
• Separate warning and critical. Use a lenient p_val for a dashboard colour and a strict one for paging.
• Budget batch size. Fix the batch size per detector. Power changes with sample size, so a monitor whose batches vary from 200 to 20,000 rows behaves like a different monitor each day.

Sampling interacts with all of this. If production traffic is large, sampling a fixed number of instances per window keeps cost predictable and the statistical power constant. A thousand instances per window is usually enough to detect a moderate shift in one feature, though the right number depends on the shift you care about, and you should measure it by injecting synthetic drift rather than guess.

That injection experiment is the single most valuable calibration step. Take your reference data, perturb it by a known amount, and see what fraction of windows your detector flags. Repeat for a range of perturbation sizes and you get a power curve that tells you, in the units of your own data, how large a shift must be before the monitor notices it. A detector without a power curve is a detector you do not yet understand.

Reference data as a managed artifact

x_ref looks like a harmless array, but it defines what "normal" means for every future decision. Treating it casually is the second most common source of bad monitors, after thresholds.

What belongs in it. The reference should represent the conditions your model was validated under. Usually that is a sample of the training or validation data. It should not include data you already know to be corrupted, and it should match the preprocessing state of production: if serving applies a scaler, the reference must be scaled with the same fitted scaler, or every batch will look drifted from the first minute. This single mismatch, a reference taken before normalisation and production data taken after, accounts for a remarkable share of "the monitor fires immediately" reports.

How big. More reference data gives a more stable test, but with diminishing returns, and for kernel tests the cost grows with its size. For univariate tests, a few thousand rows is plenty. For MMDDrift without KeOps, keep the reference to a few thousand instances too, because the permutation test becomes slow and memory-hungry beyond that. If you have millions of training rows, take a stratified or random subsample, fix the random seed, and store the subsample's identity with the detector.

Does it refresh? The update_x_ref argument lets the reference move with the data. It takes {'last': N}, which keeps the most recent N instances, or {'reservoir_sampling': N}, which maintains a uniform random sample of everything seen. Both make the detector adaptive, and that is a decision with teeth. A reference that follows the data absorbs slow drift: if the distribution creeps by a tiny amount each week, each batch looks like the last one, and the monitor never fires even as the data ends up somewhere entirely different. Use updating when you genuinely want to detect only abrupt change, and keep a fixed reference when you want to know how far today's data has travelled from what the model was trained on. Many teams run both: a fixed reference for the long-term view and a rolling one for sudden breaks.

Where it lives. The reference is a copy of part of your training data. If that data contains personal information, so does x_ref.npy, and it needs the same access controls, encryption at rest, and retention rules as the dataset it came from. This matters in regulated settings across the Gulf and Egypt, where personal data often must stay inside specific jurisdictions: if the training data must stay in-region, the detector artifact that embeds a sample of it must too. Embeddings are somewhat safer than raw rows, but they are not anonymous by default, and you should not assume they are.

Versioning. Store the reference, the encoder, the detector config and the library version together, and give the bundle an identifier. When an alert fires, the first debugging question is always "against what?", and you can only answer it if the bundle identifier appears in the alert. The config file format described in the saving section makes this natural, because it is a text file you can commit.

Online detection: ert, window size and state

Offline detectors answer "has this batch drifted?" Sometimes you do not have batches, or you cannot wait for one. Instances arrive one at a time and you want to know as soon as the stream departs from the reference. Repeating an offline test after every instance would be statistically invalid, since running a test again and again on overlapping data inflates false alarms. Online detectors exist for exactly this and are designed for sequential monitoring: MMDDriftOnline, LSDDDriftOnline, CVMDriftOnline and FETDriftOnline.

Online detectors replace p_val with a different control, the expected run-time, ert. It is the average number of instances you can process before a false alarm when nothing has changed. Set ert=100 and, on healthy data, the detector fires on average once every 100 instances. A higher ert means fewer false alarms and slower detection of real drift, a trade-off you tune explicitly. The second parameter, window_size, sets how many of the most recent instances the detector compares with the reference. Small windows react quickly to severe shifts. Large windows are more sensitive to subtle ones but respond more slowly.

from alibi_detect.cd import MMDDriftOnline

cd = MMDDriftOnline(x_ref, ert=100, window_size=10,
                    backend='tensorflow', n_bootstraps=2000)

for x_t in stream:                    # one instance at a time
    pred = cd.predict(x_t, return_test_stat=True)
    if pred['data']['is_drift']:
        break

Look at what happens at construction. The detector must pick a threshold such that the chosen ert holds for the chosen window_size. To do that it simulates n_bootstraps runs over the reference data, and that is the expensive part. On a large reference this setup can take a long time, and it is paid once per construction. The lesson is architectural: configure and build the detector at deploy or build time, save it, and load it in the serving process. Reconstructing it at every pod start turns a deploy into a wait.

The other half is state. An online detector tracks the instances it has seen and the time step t. The predict result carries time, the number of instances processed, and when you ask for it test_stat and threshold. Three methods manage state: save_state(path), load_state(path) and reset_state(), which returns the detector to t=0. A few operational rules follow:

• State is per process. If two replicas each hold a copy, each sees half the stream and neither tests the whole stream. Route each stream to a single consumer, or persist and restore state when you move it.
• Reset after an alert. Once the detector flags drift, it stays in a flagged situation. Decide what the next phase looks like: either reset and watch for a new change, or rebuild the reference from the new regime.
• A detector saved with state is not stateless. save_detector preserves online state when t is greater than zero. If you want a clean artifact to ship, call reset_state() first.
• Online detectors cannot use the legacy format. They are saved in the config format only.

Online detection is also where "it never fires" is most often a configuration outcome rather than a bug. An ert of 10,000 with a window of five instances will take a very long time to flag a mild shift. Use a short synthetic experiment to estimate the detection delay: shift the stream after instance 500 and record how many instances pass before the alert. Do that for two or three window sizes and choose the one that matches how quickly you must react.

Saving and loading: config, legacy and the registry

A detector that exists only in a notebook is not a monitor. Alibi Detect offers two serialization formats, and the choice shapes how your team deploys and reviews them.

The API is small:

from alibi_detect.saving import save_detector, load_detector

save_detector(cd, './my_detector/')
cd = load_detector('./my_detector/')

The config format is the default for drift detectors. It writes a directory containing a human-readable config.toml plus artifacts: .npy files for arrays and subfolders for models. Because the main file is text, you can read it, diff it in a pull request, edit it, and keep it in version control. A minimal config is only a few lines:

name = "MMDDrift"
x_ref = "x_ref.npy"
p_val = 0.05

name is the detector class and every other key is a constructor argument. Backend-specific settings go in the same file. For anything that is not a plain value, the config offers three routes: a path to a local file, with .npy for data and .dill for functions; a registry reference using @ syntax such as "@cd.tensorflow.preprocess.preprocess_drift" or "@utils.tensorflow.kernels.GaussianRBF", for components pre-registered by the library; and nested TOML sections for complex artifacts.

The legacy format serializes with dill. It is the only format available for outlier and adversarial detectors, and drift detectors can opt in with save_detector(cd, path, legacy=True). It is not supported for the Cramér-von Mises and Fisher's exact detectors or for any online detector. On TensorFlow newer than 2.15, saving with legacy=True needs TF_USE_LEGACY_KERAS=1 in the environment so TensorFlow uses Keras 2, a requirement noted in the 0.13 release.

For your own preprocessing code, register a function so the config can name it rather than pickle it:

from alibi_detect.saving import registry

@registry.register('my_function.v1')
def my_function(x):
    return (x - x.mean()) / x.std()

Then set preprocess_fn = "@my_function.v1" in the config. When loading, the module containing that decorator must have been imported first, or the registry lookup fails. In a service, that means an explicit import at the top of your loader module. The registered name is effectively an API: version it (.v1, .v2) and never change the behaviour behind an existing name, or old configs will load and quietly behave differently.

Validation. Alibi Detect checks configs in two stages through validate_config. The first, "unresolved", verifies field names and types before any artifact is loaded. The second, "resolved", checks the loaded artifacts before the detector is built. The point is to fail fast: a typo in a key name is caught in milliseconds instead of after a large model loads. Config validation relies on pydantic, with pydantic v2 supported through the v1 compatibility layer since 0.12.0. Run load_detector against every committed config in CI and you get a regression test for free.

Versions. The documentation does not guarantee that a detector saved with one version of Alibi Detect loads correctly under another. Loading across versions emits a warning, and you should read that as a real signal. Two concrete historical changes show why. In 0.11.0 the config_spec field was removed from config files, so detectors saved earlier needed that line deleted by hand before loading. In the same release, the meaning of meta['detector_type'] changed. The safe discipline is to pin the library, the backend, numpy and the Python minor version, build the detector in CI under that same environment, and load it under the same environment in production. Legacy dill files are even more sensitive and may fail to load on a different Python minor version.

Outlier detection in practice

Drift asks whether a batch differs from a reference. Outlier detection asks whether a single instance looks like the training data, and it has a different workflow. Alibi Detect's outlier detectors include Isolation Forest, Mahalanobis distance, Auto-Encoder, Variational Auto-Encoder, AEGMM, VAEGMM, Likelihood Ratios, Prophet, Spectral Residual and Seq2Seq, covering tabular, image and time-series data.

Most of these are fitted on data you consider normal and then calibrated. The scoring step produces an instance score, for example a reconstruction error from an auto-encoder or an isolation score from a forest, and a threshold converts the score into a decision. The library provides an infer_threshold method on detectors that need one: you pass a held-out sample of normal data and a percentile, and the threshold is set so that roughly that percentage of normal data would be considered normal. Choosing 99 means you accept about 1 percent of healthy instances being flagged, and on a stream of a million requests a day that is ten thousand flags, which is not an alert policy. Outlier flags per instance are best aggregated: count them per window, compare the rate against the expected rate, and alert on the rate.

Fit and threshold data must be separate. Fitting on the same instances you use to infer the threshold makes the model look better at reconstructing "normal" data than it will be on new normal data, and the threshold comes out too tight, flooding you with false positives. Reserve a validation slice.

Remember the serialization consequence: outlier detectors save in the legacy dill format, so every caution above about pinning versions and trusting the file source applies with more force. For time series, the Prophet detector needs its own extra, alibi-detect[prophet], and brings Prophet's own installation requirements, so install it only where you use it.

Outlier and drift detection complement each other. A rising outlier rate in a window is itself a drift signal, and a drift alert on a feature is a prompt to look at the specific outliers behind it. For tabular data with moderate dimension, many teams find a drift test per feature gives the best signal for the effort, and keep an outlier detector for the cases where a single bad input is costly, such as a sensor reading outside any plausible range that should never reach a model.

Production patterns and the neighbouring tools

Alibi Detect is a library, not a service. It has no scheduler, no storage and no dashboard. Production use means choosing where the predict call runs and where its output goes. Three patterns cover most deployments.

A scheduled batch job. The detector is loaded with load_detector, the job reads the most recent window of production inputs from a warehouse or object store, calls predict, and writes the results to a metrics store. The scheduler can be Airflow, a Kubeflow pipeline (see Kubeflow), a Kubernetes CronJob, or plain cron. This is the right default: it is simple, the compute is bounded, a failed run is easy to retry, and daily or hourly cadence suits most business models. Package the job as a container (see Docker), bake the pinned dependencies and the saved detector into the image, and run it as a Kubernetes CronJob.

A streaming consumer. For online detectors, a long-running process reads instances from a queue and calls predict per instance. The earlier rules on state apply: one consumer per stream, saved state on shutdown, restored state on start, and threshold setup done before deployment.

Event-driven monitoring next to the model server. The official deployment page for Alibi Detect describes Seldon Core with Knative eventing. The model server's payload logger sends each request to a Knative broker, and triggers fan events out to serverless detector services for outlier, drift or adversarial detection, whose results other components can consume for alerting or storage. The documented drift example buffers instances before testing, with a batch size of 5,000 in the example. That page targets Seldon Core v1, and the docs may lag newer Seldon products, so check which version of the server you run before copying the manifests. See Seldon Core and KServe for the serving side.

Whichever pattern you choose, a few integration points repeat.

• Where results go. Emit is_drift as a gauge and the per-feature distances and p-values as histograms or labelled gauges, then chart and alert in Prometheus and Grafana. Include the detector name and library version from meta as labels so you can tie a result to the code that produced it.
• Where reports go. If you want human-readable drift reports for data scientists, pair Alibi Detect's decisions with a reporting tool such as Evidently or NannyML, which also estimates performance without labels. They answer overlapping questions with different strengths, and some teams use Alibi Detect for the statistical decision and another tool for the explanation.
• Where artifacts live. Store detector bundles next to the model they monitor, in the same registry, with the same version identifier. MLflow can log the bundle directory as an artifact tied to the model version.
• Where features come from. If a feature store feeds your model, compute the reference and the production windows from the same source, so training-time and serving-time features are identical. See Feast and Great Expectations: data validation catches schema and range problems that a drift test is not built for.

Scale comes down to two levers. Kernel tests cost roughly the square of the sample size, so keep references and windows modest, or use the KeOps backend for MMD on large data, which is supported on Linux with CUDA and a C++ compiler only. And encoding dominates cost: running a deep encoder over every batch is the expensive step, so sample production traffic, for instance one to ten percent, and pre-encode the reference once.

The licence, and what it means for your employer

Alibi Detect is source-available, not open source. Up to release 0.11.4 it used Apache 2.0. From 0.11.5 in January 2024 it moved to the Business Source License 1.1, and PyPI lists the licence as "Other/Proprietary". Version 0.13.0 is under BSL.

In plain terms, BSL lets you copy, modify and redistribute the code and use it for non-production purposes. The additional use grant permits production use by non-profit educational institutions, provided it does not involve commercialising the library or embedding it in products or services sold, licensed or marketed to third parties. Anyone else running it in production needs a commercial licence from Seldon. Each version converts to Apache 2.0 on a change date, the fourth anniversary of that version's first public distribution, so older releases become permissively licensed in time. Read the LICENSE file for the exact date of the version you use.

Why does this belong in an engineering guide? Because it turns up late and expensively. A team prototypes a monitor, ships it, and months later a procurement or legal review flags a dependency with a non-open-source licence. The licence has also had real consequences in the wider ecosystem: an issue in the MLServer repository asks for Alibi Detect and Alibi Explain to be removed from the MLServer base image because of a licence mismatch. If your organisation builds base images, that is the kind of problem you inherit.

Practical advice:

1. Check at the start, not at the end. Before you propose Alibi Detect for a production system at work, ask whoever owns licence compliance, and note the version.
2. Learning and evaluation are fine. Using it to study, prototype and run comparisons is within the non-production grant.
3. Know the alternatives. Evidently and NannyML cover much of the same ground and appear in this catalogue. Check each licence yourself before relying on it, rather than trusting a summary.
4. Offering detectors as a service to third parties may count as commercial use. Ask Seldon or your legal team.

Security deserves its own short list because the licence is not the only risk. Treat saved detectors as code, as covered above. Treat x_ref as data with the sensitivity of its source. Keep secrets out of the config, since the library has no credentials of its own and the surrounding service owns access to storage, brokers and metrics stores. Use your platform's secret manager for those.

Testing detectors and debugging them

Monitoring code is rarely tested, and that is why its failures are silent. A drift detector has well-defined statistical behaviour, so you can test it more rigorously than most application code.

Test the plumbing. The cheapest tests cover loading and shape. In CI, call load_detector on every saved bundle, run predict on a small fixture, and assert that the result has the keys your metrics exporter reads: is_drift, plus p_val and distance if you requested them. This catches version mismatches, missing registry imports and renamed artifacts at merge time.

Test the statistics. Use seeded random data. Draw a "healthy" batch from the same distribution as the reference and assert no drift for most seeds. Draw a deliberately shifted batch and assert drift. Because the tests are probabilistic, do not assert on a single draw at p=0.05, which would fail one run in twenty. Fix the seed so the result is deterministic, or loop over many seeds and assert on the proportion. Keep a few such cases as regression tests for your own configuration, since a change to p_val, correction or the preprocessing silently shifts sensitivity.

Test the end-to-end alert path. Inject a known synthetic shift into a staging copy of the data and check that an alert actually arrives where it should. A monitor that computes drift correctly but publishes to a dead metrics endpoint is the commonest total failure. A scheduled "canary" window with a guaranteed shift, run daily and expected to fire, proves the path is alive.

Debugging follows from the model in the first section. When a detector behaves oddly, walk through a short list:

1. Is the shape right? x must have the same feature shape as x_ref. Offline detectors need a batch, and a single instance gives a meaningless test. Online detectors take one instance at a time.
2. Is the dtype right? Cast to float32 for deep backends.
3. Do reference and production match in preprocessing? Compare feature means and ranges side by side.
4. Which features fired? Re-run with drift_type='feature' and return_p_val=True to see the culprits, then plot those columns over time.
5. Is it the sample size? Rerun on a subsample. If a huge batch fires and a subsample does not, the shift is real but tiny.
6. Is the backend what you think? Missing optional dependencies show up as import errors or failed validation when you request a backend that is not installed. Install the matching extra, alibi-detect[tensorflow] or alibi-detect[torch], and remember the backend value is 'pytorch'.
7. Is a version change involved? A warning on load, a numpy binary incompatibility or a TensorFlow resolver conflict usually means the environment drifted from the one that built the detector. TensorFlow is capped below 2.19 in 0.13, and a very new TensorFlow will not resolve.

The dependency story deserves care. Importing TensorFlow and PyTorch together hung on PyTorch 2.0.0 to 2.0.1, fixed from 2.1.0. On Windows, the docs recommend installing and testing PyTorch before Alibi Detect. KeOps is supported on Linux only. Zsh users must quote extras, as in pip install "alibi-detect[torch]". Capture these in the Dockerfile and a lock file so nobody rediscovers them.

Operating a monitor over time

A monitor is a product with users, and its quality shows up as alert fatigue or silence. Operating it well is mostly about what you record and how you review it.

Record every run, not only alerts. For each window, store the detector name and library version from meta, the bundle identifier, the batch size, the timestamp range, is_drift, and the per-feature distances. A table of this history answers questions an alert cannot: how often the monitor ran, what normal looks like for each feature, and whether a recent change in alert volume came from the data or from a config edit.

Review alerts like incidents. Each alert should end in a recorded verdict: real drift, pipeline bug, expected seasonal change or false alarm. Tally these monthly. A high share of false alarms means settings need loosening or a correction applied, and a high share of pipeline bugs is a sign your data validation is thin and a tool like Great Expectations should sit upstream.

Connect drift to outcomes. When labels arrive, compare drift alerts with model quality over the same windows. Some alerts will precede degradation, and many will not. Use that record to decide which features deserve a strict threshold and which are noisy. Without labels, the uncertainty-based detectors give a model-level signal that sits closer to outcomes than raw input tests.

Plan upgrades. Because cross-version loading is not guaranteed, an upgrade is a small project. Build the new environment, re-create and re-save every detector under it, run the plumbing and statistical tests, compare results against the old detectors on the same historical windows, and only then switch. Also read the release notes for licence terms and dependency ceilings, since the TensorFlow cap and Keras behaviour have both changed in recent releases.

Putting it all together

Here is a small but realistic monitor for a tabular model with 12 numeric features, built the way this level recommends. Each step maps to a section above.

1. Reference. Take a 3,000-row random sample of the validation set, scaled with the same fitted scaler that serving uses. Save it as x_ref.npy and record the sample seed.
2. Detector. Use KSDrift with correction='fdr' and an explicit p_val, never relying on a default. Add one MMDDrift on the same features if you care about interactions, with a modest reference.
3. Calibrate. Run the detector over 200 bootstrap batches of healthy held-out data to measure the real false-alarm rate. Inject shifts of increasing size and record the power curve.
4. Alert policy. Alert when drift appears in three of the last five daily windows with a distance above a floor you chose from the power curve. Use a lenient setting for a dashboard colour and the strict one for paging.
5. Save and test. Save in the config format. Commit config.toml. In CI, run load_detector, a seeded healthy case and a seeded shifted case, under the pinned environment.
6. Package. Build a container with pinned versions of Alibi Detect, numpy and Python, containing the saved detector. Run it as a CronJob that reads yesterday's window, samples a fixed number of rows, and calls predict with drift_type='feature', return_p_val=True and return_distance=True.
7. Publish. Emit is_drift and per-feature distances with the version and bundle identifier as labels to Prometheus. Chart them in Grafana. Add the canary window.
8. Operate. Review alerts monthly, record verdicts, and re-save the detector after any upgrade or reference refresh.

import numpy as np
from alibi_detect.cd import KSDrift
from alibi_detect.saving import save_detector, load_detector

x_ref = np.load('x_ref.npy').astype(np.float32)
cd = KSDrift(x_ref, p_val=0.05, correction='fdr')
save_detector(cd, './ks_monitor/')

cd = load_detector('./ks_monitor/')
x_win = np.load('yesterday.npy').astype(np.float32)
res = cd.predict(x_win, drift_type='feature', return_p_val=True, return_distance=True)

record = {
    'detector': res['meta']['name'],
    'version': res['meta']['version'],
    'is_drift': res['data']['is_drift'],
    'distance': [float(d) for d in res['data']['distance']],
}
print(record)

Nothing in this code is clever, and that is the point. The leverage is in the surrounding discipline: a defensible reference, a measured false-alarm rate, a persistence rule, a tested load path and a recorded history.

Where you are now

You can now explain what a drift test compares and why batch size changes its behaviour, choose among univariate, kernel, classifier and uncertainty detectors, reduce high-dimensional data before testing it, compute the false-alarm arithmetic for your own settings, and manage a reference set as a versioned, sensitive artifact. You understand ert, window_size and online state, the config and legacy formats and the registry, and you can test, deploy and operate a monitor around the library. You also know the licence and can raise it before it becomes a problem.

The senior level goes further: it designs monitoring for many models and teams, reasons about which component fails first at scale, sets up governance and incident playbooks, and says plainly when Alibi Detect is the wrong tool. If you are weighing it against other tools, start with Evidently and NannyML.

Sources

• Alibi Detect documentation
• Getting started and installation
• Drift detection overview
• Drift detection methods
• Kolmogorov-Smirnov drift detector
• Online MMD drift detector
• Outlier detection
• Saving and loading detectors
• Config files
• Deployment with Seldon Core and Knative
• Alibi Detect on GitHub
• Changelog
• Licence
• Alibi Detect on PyPI

Mid-level interviews about Alibi Detect rarely ask you to recite a constructor. They hand you a situation, a monitor that fires too often or a pipeline that never fires, and watch how you reason. This file gives you the scenarios you are likely to meet, the reasoning a strong answer follows, and the gaps that mark a weak one.

How these questions are judged

Interviewers listening to a drift-monitoring answer are checking for four habits. The first is statistical honesty: do you know that a drift test reports significance and not importance, and that a 5 percent threshold produces false alarms by construction? The second is systems thinking: do you place the detector in a pipeline with a reference, a schedule, storage and alerting, or do you treat it as a function call? The third is calibration: do you measure false-alarm rate and power on your own data instead of trusting defaults? The fourth is risk awareness, which at this tool includes the licence, the deserialization risk and version pinning.

A strong answer names the mechanism, gives a concrete number or setting, and says what you would measure to confirm. A weak answer names a detector and stops.

Scenario questions

The monitor fires every day, and nobody believes it any more. What do you check first?

Start by establishing whether the alarms are false or real, because the fix differs. Pull a month of stored results and look at the per-feature statistics, not the verdicts. If the distances are small and stable and only the p-values dip below the threshold, you are looking at significance without effect, which usually means batches are very large or the threshold is too lenient for the number of tests.

Then check the arithmetic. A single test at p_val=0.05 fires in about one run in twenty on healthy data. With 40 features and no correction, the chance of at least one false positive per run is about 87 percent, so a daily alert is exactly what the settings predict. The fix is a correction, either 'bonferroni' for strict control or 'fdr' when you have many features and tolerate a few spurious ones, plus a persistence rule such as drift in three of the last five windows.

Next rule out a pipeline mismatch. If the reference was scaled and production data is not, or the reference was taken before a cleaning step that production applies, every batch looks drifted from the first run. Compare means and ranges of one feature side by side.

A strong answer also mentions measuring the real false-alarm rate by running the detector on resampled healthy data. A weak answer says "raise the p-value threshold" without noticing that changing a threshold on an uncorrected multi-feature test barely helps and costs sensitivity.

The monitor has never fired, and the model quality has fallen. Why?

There are several candidate causes, and a good answer orders them by likelihood. First, the reference may be updating. With update_x_ref set to {'last': N} or reservoir sampling, the reference follows the data, so a slow drift looks like normal variation at every step. If you want to detect gradual change, use a fixed reference.

Second, the drift may not be in the inputs. Alibi Detect's drift detectors test p(x). A change in the relationship between inputs and labels, concept drift, can degrade the model without moving the inputs at all. That is the case for a model-output detector such as ClassifierUncertaintyDrift or RegressorUncertaintyDrift, and for performance monitoring once labels arrive.

Third, power. Small batches cannot detect small shifts. Running a power curve with synthetic shifts shows the smallest change the monitor can see.

Fourth, the representation. On images or text, testing raw values or a poor encoder can mask a change the model is sensitive to. And finally, an operational cause: the job stopped running, or it writes to a dead metrics endpoint. A canary window with a guaranteed shift catches that.

Weak answers blame the library. Strong ones go through reference, signal type, power, representation and plumbing in that order.

You must monitor an image classifier. How do you design it?

Raw pixels are the wrong space to test. The dimension is huge and most variation is nuisance, so a per-pixel test is noisy and a kernel test loses power. Reduce first with a preprocess_fn: an encoder, a random one as a baseline or a trained one such as your model's penultimate layer, or the softmax output. Wrap it with preprocess_drift and functools.partial, then run an MMDDrift on the embeddings with a backend, 'tensorflow' or 'pytorch'.

Encode the reference once, which preprocess_at_init=True does by default, or pass pre-computed embeddings with x_ref_preprocessed=True. Sample production traffic to keep encoder cost bounded. Cast to float32.

Add a second signal that does not depend on the encoder, such as a univariate test on brightness or image size, because an encoder only sees the directions it learned. In addition, monitor the model's prediction uncertainty, since it ties the signal to the thing you care about. Mention that the encoder and reference form one versioned unit.

How would you detect drift on a stream with no batches?

Use an online detector: MMDDriftOnline, LSDDDriftOnline, CVMDriftOnline or FETDriftOnline. Explain why you cannot simply rerun an offline test after every instance: repeated testing on overlapping data inflates false alarms. Online detectors control the expected run-time ert, the average number of instances before a false alarm under no drift, and use a window_size of recent instances.

The follow-up is almost always about setup and state. Construction runs n_bootstraps simulations to find a threshold for the chosen ert and window, which is slow on large references, so build and save the detector ahead of time. State is per process: use one consumer per stream, or persist with save_state and restore with load_state, and reset_state after an alert. Strong candidates mention a delay experiment: shift the stream and measure how many instances pass before detection.

Your detector worked in CI and fails to load in production. What happened?

Think version skew. The docs do not guarantee that a detector saved by one version loads under another. Loading across versions emits a warning, and mismatched Python, numpy, TensorFlow or PyTorch versions cause failures. Legacy dill files are especially sensitive to the Python minor version.

Other candidates: a custom function referenced as @my_function.v1 in the config that was not imported before load_detector, so the registry lookup fails; a legacy save on TensorFlow newer than 2.15 without TF_USE_LEGACY_KERAS=1; and an old config still containing the config_spec line removed in 0.11.0.

The remedy is process: pin the library, backend, numpy and Python, build and save in CI under the same image that production uses, and make loading every committed detector a CI test.

The team wants to put Alibi Detect into a customer-facing product. What do you say?

Raise the licence before anything else. From 0.11.5 the library is under the Business Source License 1.1 and is not open source. Non-production use is allowed, and production use by non-profit educational institutions is allowed under conditions. Other production use, especially embedding it in something sold or offered to third parties, needs a commercial licence from Seldon. Versions through 0.11.4 were Apache 2.0, but they carry old Python and dependency ceilings.

Then propose a decision, not a veto: involve legal early, record the version, and evaluate alternatives in this catalogue such as Evidently or NannyML, checking their licences directly. Candidates who have never heard of the licence change are exposing a gap that matters in a real company.

Trade-off questions

Univariate tests per feature or a multivariate kernel test?

Univariate detectors such as KSDrift are cheap, need no backend, and tell you which feature moved, which makes alerts actionable. Their weakness is that they see only marginals, so a change in the correlation between two features can go unnoticed, and many tests need a multiple-testing correction. MMDDrift and LSDDDrift test the joint distribution and can catch interaction changes, but they rely on permutation tests whose cost grows with the square of the sample, give a single verdict with little explanation, and need a deep-learning backend.

A good answer recommends a layered approach: univariate tests as the workhorse for explainability, one multivariate test on a compact representation for interactions, and a clear rule for which one pages someone.

A fixed reference or a rolling one?

A fixed reference answers "how far has the data moved from what the model was trained on?" and catches slow drift, but it will eventually fire permanently if the world legitimately changes, and then you must decide when to refresh it. A rolling reference with update_x_ref answers "did something change recently?" and stays quiet through gradual evolution, which is also its danger. Many teams run both. State the cost of each failure: a fixed reference risks alert fatigue after a legitimate regime change, and a rolling one risks silent decay.

Batch job or streaming detector?

Batch is simpler, cheaper, easier to retry and enough for most models. Its downside is delay, equal to the window. Streaming online detectors react faster and suit cases where acting within minutes matters, but they add state, threshold setup cost, and a single-consumer constraint. Choose streaming only when the cost of delayed detection justifies the operational burden. An answer that picks streaming for prestige is a red flag.

Sample production traffic or test everything?

Sampling bounds compute, since encoder passes and kernel tests dominate cost, and a fixed sample size keeps statistical power constant from day to day. Testing everything gives more power for small shifts but costs more and makes the batch size vary with traffic, which changes sensitivity. A defensible answer samples a fixed number per window, chosen using a power curve, and monitors rare but important segments separately so sampling does not hide them.

Config format or legacy format?

The config format is readable, diffable, reviewable in a pull request and the default for drift detectors, and it supports online detectors. Legacy dill is the only option for outlier and adversarial detectors, is opaque, and carries stronger version and security caveats. Prefer config wherever the detector supports it. For legacy files, treat them as executable code: load only from trusted storage and verify a checksum.

Alibi Detect or a reporting tool such as Evidently?

Alibi Detect gives a broad set of statistical detectors, including online and learned ones, as a library you embed. Reporting-oriented tools give prebuilt reports and dashboards. They overlap. The honest answer is that the choice depends on whether you need custom detectors and sequential monitoring or fast human-readable reports, and on licence constraints. See Evidently.

Questions on specifics

What does ert mean, and how do you choose it?

It is the expected run-time without drift: the average number of instances before a false alarm. A higher value gives fewer false alarms and slower detection. You choose it from the cost of a false alarm and the delay you can tolerate, then verify with a simulation. Pair it with window_size: small windows catch severe shifts quickly, large ones catch subtle shifts.

What is the difference between correction='bonferroni' and 'fdr'?

Bonferroni divides the threshold by the number of tests and controls the probability of any false positive. It is conservative and loses power when many features truly drift. The false discovery rate approach controls the expected fraction of false positives among those flagged, which keeps more power when many features move.

Why does preprocess_fn matter?

Because statistical tests lose power in high dimensions. The preprocessing reduces data to a representation worth testing: an encoder, a model's penultimate layer, or softmax outputs. The detector applies it to the reference and every batch. Whether that happens once at construction or each call is governed by preprocess_at_init and x_ref_preprocessed.

What are the backends and how are they named?

'tensorflow', 'pytorch' and, for MMD on large data, 'keops'. The pip extras are tensorflow, torch and keops, so the PyTorch backend string differs from its extra. KeOps works on Linux with a C++ compiler and CUDA.

What changed around version 0.11?

The meta['detector_type'] value became 'drift', 'outlier' or 'adversarial' with a separate online field, and config_spec was removed from config files. Version 0.11.5 changed the licence to BSL 1.1. Version 0.13.0 added Python 3.12 support, raised the TensorFlow bound to below 2.19 and requires TF_USE_LEGACY_KERAS=1 for legacy saves on newer TensorFlow.

How do you make a detector deployable?

Build it in CI, save it with save_detector in the config format, commit the config.toml, load it in the service with load_detector, and run a plumbing test on every commit. For online detectors, do threshold setup at build time and reset state before saving a clean artifact.

What strong answers have that weak ones lack

Whiteboard prompt: design a drift monitor for a credit-scoring model

You are asked to design monitoring for a model that scores applications from a Gulf lender. A strong walkthrough covers these points in order.

Clarify the data. Mixed numeric and categorical features, so TabularDrift or per-type detectors, and a reference taken from validation data scaled with serving's own fitted scaler. Clarify residency: the reference embeds applicant data, so it stays in the same region and under the same access rules as the training set.

Choose signals. Per-feature tests with correction='fdr' for explainability, a model-uncertainty detector for a label-free outcome signal, and, when repayment labels arrive months later, a lagged performance check. Say why labels lag and why drift alerts matter in the meantime.

Set policy. Daily windows with a fixed sample size, a persistence rule, a magnitude floor chosen from a power curve, and a warning versus critical split.

Deploy. A containerised CronJob with pinned versions, the saved config-format detector, load tests in CI, results pushed to Prometheus and charted in Grafana, with detector version and bundle identifier as labels, and a daily canary window.

Operate. Monthly review of alerts with recorded verdicts, a documented response for each class, and an upgrade procedure that re-saves and revalidates detectors. Finish with the licence check, since a lender will have a compliance review.

Interviewers reward the order here: data, signals, policy, deployment, operation, risk.

Questions to ask back

Good candidates ask questions that show they have run monitors. Ask how large the production batches are and whether sizes vary, whether labels arrive and how late, whether anyone owns the alert response, which feature pipeline produces the reference and whether it matches serving, and whether the organisation has already cleared the licence. Each question points at a real failure cause from this guide.

Sixty-second self-test

Answer aloud without notes.

1. Why does a 40-feature KSDrift with no correction fire almost daily on healthy data?
2. Name two reasons a monitor stays silent while the model degrades.
3. What do ert and window_size control?
4. Why is x_ref sensitive data?
5. What goes wrong if you load a detector saved under a different version, and how do you prevent it?
6. Which saving format do outlier detectors use, and what is the security consequence?
7. State the licence position from 0.11.5 onward.

If any answer took more than a few seconds, reread the matching section of the detailed file.

These are the lessons teams pay for when they move from a notebook drift check to a monitor people rely on. They are ordered with errors first, then the habits that pay off most, then practice cards and operational recipes. Everything targets Alibi Detect 0.13.

Errors at this level

Each entry gives the symptom, the cause and the fix.

The monitor fires from the first run

Symptom. A new detector flags drift on its very first production batch and keeps flagging it.

Cause. Reference and production data passed through different preprocessing. The usual culprit is a reference taken before scaling or cleaning while serving applies a fitted scaler, or a different handling of missing values or categorical encoding.

Fix. Compare the mean, range and missing-value rate of one feature in x_ref and in a production batch. If they differ by orders of magnitude, you have a pipeline mismatch, not drift. Build the reference by running validation data through the exact serving transformation code, and keep that code in one shared module.

Daily alerts on healthy data

Symptom. A multi-feature KSDrift pages almost every day, and all the numbers look ordinary.

Cause. Multiple testing. At p_val=0.05, each test fires about one time in twenty on healthy data, and 40 uncorrected tests give roughly an 87 percent chance of at least one alarm per run.

Fix. Set correction='bonferroni' or 'fdr', require drift in several recent windows before paging, and measure the real false-alarm rate by running the detector over resampled healthy data.

Significant but meaningless drift

Symptom. Tiny p-values with per-feature distances close to zero.

Cause. Very large batches give a test enough power to detect trivial differences. Statistical significance is not practical importance.

Fix. Use a fixed batch or sample size per window, and alert on magnitude as well: require the distance to exceed a floor you chose from a power curve.

The monitor never fires and the model decays

Symptom. Quiet dashboards, falling business metrics.

Cause. One of several: update_x_ref made the reference follow the data and absorb slow drift, the real change is in the relationship between inputs and labels, the batches are too small to see the shift, or the job quietly stopped.

Fix. Keep a fixed reference alongside any rolling one, add a model-uncertainty detector, measure power with synthetic shifts, and run a canary window that must always fire through the real alert path.

Wrong backend name

Symptom. An error complaining about the backend, or a failure on construction of MMDDrift or LSDDDrift.

Cause. The pip extra is torch but the backend value is 'pytorch'. Or the optional dependency for the backend is not installed.

Fix. Install alibi-detect[tensorflow] or alibi-detect[torch] as needed, and pass backend='tensorflow' or backend='pytorch'. On zsh, quote the extras: pip install "alibi-detect[torch]".

Dtype and shape surprises

Symptom. Errors or nonsense results from predict on a deep backend, or a test on a single instance that always says no drift.

Cause. float64 arrays passed to float32 backends, a feature shape that does not match x_ref, or a single instance passed to an offline detector, which has too little data for a test.

Fix. Cast to float32 at the boundary of your service, assert the shape against x_ref in a test, and pass batches to offline detectors and single instances to online ones.

Online detector setup takes forever

Symptom. A service restart hangs for minutes before it is ready, and the readiness probe fails.

Cause. Constructing an online detector simulates n_bootstraps runs to set the threshold for your ert and window_size. On a large reference this is slow, and the service repeats it at every start.

Fix. Build the detector once in CI or at image build time, save it, and load it in the service. Reduce the reference size if you can.

The online detector sees half the stream

Symptom. With two replicas, detection is slower than your simulation predicted, or results differ between pods.

Cause. Online state is per process, and a load balancer split the stream so each replica saw part of it.

Fix. Route each stream to a single consumer, or persist with save_state and restore with load_state when moving it. Call reset_state after an alert so the detector starts a fresh run.

A detector loads on one machine and fails on another

Symptom. load_detector works in CI and breaks in production, or prints a version warning.

Cause. Version skew. The docs do not guarantee cross-version loading. Python, numpy, TensorFlow or PyTorch differ, or a legacy dill file is being loaded under another Python minor version.

Fix. Pin everything, build the detector under the production image, and run load_detector in CI against every committed bundle. Treat a version warning as a failure.

Registry lookup fails on a custom function

Symptom. Loading a config that contains preprocess_fn = "@my_function.v1" fails.

Cause. The module containing the @registry.register decorator was never imported in the loading process.

Fix. Import that module explicitly before calling load_detector, and version the registered name so behaviour behind a name never changes.

Legacy save fails on new TensorFlow

Symptom. A Keras serialization failure when saving with legacy=True.

Cause. On TensorFlow newer than 2.15, Keras 3 is the default and the legacy path needs Keras 2.

Fix. Set export TF_USE_LEGACY_KERAS=1 before Python starts. Better, use the config format where the detector supports it.

Install conflicts

Symptom. pip cannot resolve TensorFlow, or pulls an unexpectedly old one, or numpy complains about binary incompatibility.

Cause. The release caps TensorFlow below 2.19, and compiled packages disagree on the numpy version.

Fix. Use a matching TensorFlow, or choose the PyTorch backend. Align numpy with the compiled dependencies, and keep a lock file. If an import hangs when both TensorFlow and PyTorch are loaded, check that PyTorch is 2.1.0 or later.

Practices that pay off, in order of leverage

1. Measure your own false-alarm rate and power

Nothing else changes monitor quality as much. Run the detector over many resampled healthy batches to measure how often it fires, and over deliberately shifted data to measure what it catches. A power curve turns tuning from guesswork into engineering and gives you a defensible answer when someone asks how sensitive the monitor is.

2. Make the reference a versioned artifact

Store x_ref, the encoder, the config and the library version together with an identifier, and put that identifier in every alert. When something fires, the first question is "against what?" Treat the reference with the same access controls as the training data it was sampled from, because it contains some of it.

3. Separate detection from alerting

The detector says whether a batch differs. Your alert policy decides whether anyone is bothered. Add persistence (drift in several recent windows), a magnitude floor, and a split between a dashboard warning and a page. This is where most of the noise reduction comes from.

4. Fix the batch size

Statistical power depends on sample size, so a monitor with varying batches is a different monitor every day. Sample a fixed number of instances per window, chosen from your power curve.

5. Test loading in CI

A one-line test that runs load_detector on each committed bundle and calls predict on a fixture catches version skew, missing registry imports and renamed artifacts before they reach production.

6. Use the config format and review it

A text file in a pull request is a conversation. Someone will ask why p_val changed. Opaque pickles cannot be reviewed, and a reviewed config doubles as the audit trail for why an alert fired.

7. Reduce before testing high-dimensional data

Use a preprocess_fn: an encoder, a model layer or softmax outputs. Pre-encode the reference once. Pair it with a simple encoder-independent signal such as a summary statistic, since an encoder only sees the directions it learned.

8. Monitor the monitor

Run a canary window that must always fire, record every run and not only alerts, and alert when runs stop appearing. Silent failure is the worst failure of a monitor.

9. Check the licence first

From 0.11.5, Alibi Detect uses the Business Source License 1.1. Write the version and licence in the repository README and involve whoever owns compliance before production use at work.

10. Review alerts as incidents

End each alert with a verdict: real drift, pipeline bug, seasonal change or false alarm. A monthly tally tells you whether to loosen settings or fix upstream data validation, and pairs naturally with Great Expectations.

Practice cards

Each card is a ten-minute exercise.

Card 1: the arithmetic of alarms. Generate 200 healthy batches for 30 features. Run KSDrift without correction, then with 'bonferroni', then with 'fdr'. Count alarms for each. Write down the numbers and which setting you would pick for a daily page.

Card 2: size versus significance. Shift one feature by 0.1 standard deviations. Test batches of 100, 1,000 and 20,000. Record p_val and distance at each size. Notice which one keeps its value as the batch grows.

Card 3: the creeping reference. Build one fixed detector and one with update_x_ref={'last': 500}. Feed 60 batches that each shift the mean a little. Note which fires and when.

Card 4: the interaction a univariate test misses. Create two samples with equal marginals but opposite correlation between two columns. Compare KSDrift with MMDDrift using a backend.

Card 5: embeddings. Take an image or text dataset and compare an MMDDrift with a random encoder against one with your model's features after a mild perturbation.

Card 6: online delay. Build an MMDDriftOnline with ert=200 and window_size=20. Measure the average run length on healthy data across ten runs. Shift the stream and measure the delay to alert. Repeat with window_size=50.

Card 7: round trip. Save a detector, edit config.toml by hand, reload it, and confirm the change. Then register a custom function and observe the failure when you forget the import.

Card 8: break loading on purpose. Create a detector in one environment and try loading it in another with a different numpy or Python minor version. Record what the warning or failure looks like so you recognise it later.

Card 9: the canary. Write a job that sends a guaranteed shift through your alert path and fails loudly if no alert arrives.

Card 10: read the licence. Open the LICENSE file, find the additional use grant and the change date, and write two sentences a manager could act on.

Operational recipes

Recipe: build and save a detector in CI

python -m venv .venv && . .venv/bin/activate
pip install "alibi-detect[torch]==0.13.0" numpy
python build_detector.py          # constructs the detector and calls save_detector
python -c "from alibi_detect.saving import load_detector; load_detector('./ks_monitor/')"

The last line is the load test. Commit the bundle's config.toml and publish the artifacts to your registry with the model version. Pin the same versions in the serving image.

Recipe: a plumbing test

import numpy as np
from alibi_detect.saving import load_detector

def test_detector_loads_and_predicts():
    cd = load_detector('./ks_monitor/')
    x = np.load('tests/fixture_window.npy').astype(np.float32)
    out = cd.predict(x, drift_type='feature', return_p_val=True, return_distance=True)
    assert set(['is_drift', 'p_val', 'distance']) <= set(out['data'])
    assert out['meta']['version']

Add a seeded healthy case and a seeded shifted case beside it, and assert on those results too, using a fixed seed so the test is deterministic.

Recipe: calibrate false alarms

Resample healthy held-out data into many batches of your production size, run the detector on each, and record the fraction flagged. Compare it with your target. If it is higher, add a correction or tighten p_val. Store the result next to the bundle so the next person knows how it was tuned.

Recipe: a Kubernetes CronJob for a batch monitor

apiVersion: batch/v1
kind: CronJob
metadata:
  name: drift-monitor
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: monitor
              image: registry.example.com/drift-monitor:2026-10-01
              args: ["--window", "yesterday", "--sample", "5000"]

The image contains the pinned dependencies and the saved detector. concurrencyPolicy: Forbid prevents two runs from overlapping. See Kubernetes and Docker for the surrounding pieces.

Recipe: ship results to metrics

Turn each predict result into a flat record: detector name and library version from meta, bundle identifier, window start and end, batch size, is_drift, and per-feature distances. Push is_drift and distances as labelled gauges to Prometheus, and chart them in Grafana. Alert on persistence, for instance drift in three of the last five windows, rather than on a single run.

Recipe: an upgrade checklist

1. Read the release notes and changelog for licence changes and dependency ceilings.
2. Build the new environment in a branch with all versions pinned.
3. Re-create and re-save each detector under it.
4. Run plumbing and statistical tests.
5. Replay the same historical windows through old and new detectors and compare results.
6. Switch the production image, keep the old one for rollback, and watch the first week of results.

Recipe: handling an alert

1. Identify the bundle and window from the alert labels.
2. Rerun with drift_type='feature' to find which features fired, and read their distances.
3. Compare the means and ranges of those features with the reference to rule out a pipeline mismatch.
4. Check upstream data validation for schema or range breaks.
5. If the drift is real, check the model's uncertainty detector and any available labels.
6. Record the verdict, and decide whether to retrain, roll back an upstream change, or refresh the reference.

Recipe: protecting artifacts

Store saved detectors in a bucket with versioning, record a checksum when you publish, and verify it before load_detector. Encrypt the bucket, restrict access to the same group that may read the training data, and keep the bucket in the same region as that data when residency rules require it. Never embed credentials in config.toml.

A habit worth keeping

Keep a short log beside each detector bundle. Every time you change a setting, such as p_val, correction, the window size or the reference sample, write one line saying what you changed and what you saw that made you change it. Six months later, when someone asks why the monitor is so quiet or so loud, that log is the only record of the reasoning, and it spares you from reverse-engineering your own past decisions from a diff of numbers.

A short checklist

• The reference passes through the same preprocessing as serving.
• p_val is set explicitly, and a correction is used with many features.
• Batch size is fixed and chosen from a power curve.
• The real false-alarm rate has been measured.
• update_x_ref is a deliberate choice, with a fixed reference kept alongside.
• High-dimensional data is reduced with a preprocess_fn, and the encoder is versioned with the reference.
• Online detectors are built ahead of time, run with one consumer per stream, and reset after alerts.
• Detectors are saved in the config format wherever supported, and loading is tested in CI.
• Library, backend, numpy and Python versions are pinned.
• Saved files come only from trusted storage and are checksummed.
• Every run is recorded, and a canary window proves the alert path.
• The licence has been checked and written down.

Where to go next

If your monitor works but you need reports for humans, look at Evidently. If you need to estimate model quality without labels, look at NannyML. For serving and event-driven deployment, read Seldon Core and KServe. For storing detector bundles beside models, see MLflow.

This is part three of three. Beginner showed you a detector firing on a shifted batch. Mid showed you backends, preprocessing and the saving formats. Here the question changes: you own Alibi Detect as a platform that other teams depend on, which means you answer for its false alarms, its silence, its licence, its artifacts and its bill, and you answer for them across many models you did not build.

Drift detection has an unusual failure profile. A web service that breaks produces errors. A drift monitor that breaks produces nothing, and nothing looks exactly like "the data is fine". Most of this guide is about making sure that silence is earned.

Where this picks up

I am starting with the shape of the system, because every failure mode below attaches to one of its parts.

The moving parts of a monitoring platform

A detector is a small Python object with a predict() method. A platform built on detectors is something else: it is the pipeline that turns production traffic into the batches a detector can test, the store that holds reference data and saved detectors, the scheduler or consumer that runs them, and the path that carries results to a person who can act. Alibi Detect gives you the second box from the left and nothing else, which is the right division of labour and also the reason ownership is hard.

Notice what the library does not have. It has no scheduler, no storage, no alerting, no tenancy model, no authentication and no network code of its own. That is a strength for security, since there is no service to harden inside the library, and a weakness for adoption, because every team will assemble those pieces differently unless you give them a paved path. The rest of this guide is mostly that paved path.

Three call shapes you will support

Every use of Alibi Detect in your organisation will be one of three shapes, and naming them lets you reason about failure.

Batch. A scheduled job loads a saved detector with load_detector, takes the last window of production inputs, calls predict, and writes is_drift, p-values and distances to a metrics store. This is the dominant shape and the one to make the default. The schedulers you already run (see Airflow) fit it, state lives in the store rather than the process, and a failed run is retryable.

Streaming. An online detector such as MMDDriftOnline receives one instance at a time and keeps a window and a timestep t in memory. This gives faster detection of severe drift but moves state into your runtime, with all the problems that implies: restarts lose it, replicas split it, and a test that is meant to see a single ordered stream does not survive being load-balanced.

Event-driven. The official deployment page describes Seldon Core with Knative eventing: a payload logger sends request events to a broker, which fans them out to serverless detector services for outlier, drift or adversarial detection, and further consumers handle alerting and storage. The example drift wrapper there batches drift_batch_size = 5000 instances before running the test. Treat this as a reference architecture rather than a recipe, because those documents target Seldon Core v1 and may lag the current product; see the Seldon Core guide and the KServe guide for the serving side.

How a drift monitor lies to you

Before scale and security, the failure that matters most is statistical, because it is invisible. A drift monitor can lie in two directions, and a platform owner has to design against both.

The alert rate you actually bought

A detector with p_val=0.05 is a hypothesis test. On perfectly stable data it will declare drift about one time in twenty, per test. That sounds harmless until you multiply. A team monitoring thirty features with a univariate detector such as KSDrift, checked hourly, has made 720 feature-level tests per day. Without correction, several of them will fire every day on data that has not moved.

The library gives you the main lever. Univariate detectors accept correction='bonferroni', which is conservative, or 'fdr', which allows an expected fraction of false positives among the features flagged. The default call is also not the same as per-feature reporting: predict with drift_type='batch' applies the correction and returns a single is_drift, whereas drift_type='feature' returns the per-feature view you use for diagnosis.

import numpy as np
from alibi_detect.cd import KSDrift

x_ref = np.load("x_ref.npy").astype(np.float32)

# Batch decision: corrected across features, one yes/no for alerting.
cd = KSDrift(x_ref, p_val=0.01, correction="fdr")

def check(x: np.ndarray) -> dict:
    batch = cd.predict(x, drift_type="batch")
    out = {"is_drift": int(batch["data"]["is_drift"])}
    if out["is_drift"]:
        # Per-feature view only when someone needs to look.
        feat = cd.predict(x, drift_type="feature",
                          return_p_val=True, return_distance=True)
        out["p_val"] = feat["data"]["p_val"].tolist()
        out["distance"] = feat["data"]["distance"].tolist()
    return out

The platform decision hidden in that snippet is which p-value teams may choose. If you let every team pick p_val freely, you will end up with a fleet of monitors with incomparable alarm rates, and the dashboard that shows "drift detected on 14 models" means nothing. Standardise the corrected threshold, publish the expected false-alarm budget, and let teams request an exception with a reason.

Alert on persistence, not on a batch

Even a well-corrected test fires occasionally on stable data. A single is_drift = 1 is evidence, not an incident. The practice that holds up is to alert on a rule over results: drift in at least k of the last n batches, or a distance that stays above a level. You build this in the metrics store, outside the library, which is another reason to export raw results rather than only the boolean.

Statistical power runs the other way

The mirror image is silence. Two things make a detector fail to notice real drift:

• Too little data per test. A test on 50 instances cannot see a small shift. Window size is a power decision, not a convenience decision.
• Testing the wrong space. On images or text, raw inputs are very high dimensional and a test on raw pixels works poorly. The detector needs a preprocess_fn that reduces dimension first, such as an encoder, a projection, or a model's softmax output. If that reduction throws away the very thing that moved, the monitor is blind.

Online detectors add their own trade, and it has a name. The expected run-time, ert, is the average number of instances before a false alarm when nothing has changed. A larger ert means fewer false alarms and slower detection. A smaller window_size catches severe drift fast, a larger one is more sensitive to subtle drift. There is no setting that is good at both, so a platform should expose these two parameters with defaults per use case rather than leave them to guesswork.

The encoder is part of the detector

The most common way a mature drift platform silently breaks is not in the statistics at all. It is a change to something the detector depends on.

For any detector that uses a preprocess_fn, the reference data and the live data are compared after the same transformation. If that transformation is an untrained encoder, a trained encoder, or a model's output, then it is part of the detector's definition. Change it, and the reference set, which was embedded by the old version, no longer lives in the same space as the new batches. The test then compares two different coordinate systems and reports spectacular drift, or, worse, a plausible-looking result that is meaningless.

This produces three rules a platform should enforce:

1. Version the encoder with the detector. The saved artifact, the reference embedding and the encoder weights are one unit with one version identifier.
2. Embed the reference once. The constructor argument preprocess_at_init defaults to true, and x_ref_preprocessed lets you pass reference data that is already embedded. Both exist to avoid re-encoding x_ref on every start-up. Use them deliberately, and remember that x_ref_preprocessed=True means you are vouching that the embedding matches the encoder.
3. Set preprocess_at_init=False only when preprocessing genuinely depends on both reference and test data.

A model you serve may also be the encoder, for instance when you test its softmax outputs. Then a model rollout is a detector change, and your release process must say so. Link the two in the registry, which the artifacts section below returns to.

Scaling: which component breaks first

Alibi Detect's cost has a very uneven shape across detectors, so "scaling the platform" means something different depending on which ones your teams chose. Work through the components in the order they typically give out.

The reference set, and the cost of permutation tests

Univariate tests such as KSDrift are cheap. The kernel detectors are not. MMDDrift and LSDDDrift derive their p-value from a permutation test controlled by n_permutations, and each permutation evaluates kernels across the pooled reference and test data, so cost grows roughly with the square of the number of instances. A reference set that was fine at a thousand rows becomes painful at a hundred thousand.

Three levers exist, in order of preference:

• Keep x_ref small. A few thousand instances is usually enough statistical power for a kernel test, and more rows mostly buy compute cost. Choose the reference by sampling the training or validation data, not by taking all of it.
• Reduce dimension before testing. An encoder or projection lowers the per-kernel cost and, as discussed, improves power on images and text.
• Change the backend. The keops extra exists for MMD on large data, but it is supported on Linux only, needs a C++ compiler (g++ 7 or later, or clang++ 8 or later) and the CUDA toolkit. Do not put it in a base image that must also run on macOS laptops.

Online threshold setup

The constructor of an online detector such as MMDDriftOnline runs n_bootstraps simulations to set the detection threshold for your chosen ert and window_size. On a large reference set this is slow, and it is a one-time cost. The operational consequence is clear: never construct an online detector inside a request path or a pod's start-up if you can avoid it. Construct it once in the build or deploy step, save it, and load the saved detector in the serving process.

from alibi_detect.cd import MMDDriftOnline
from alibi_detect.saving import save_detector

cd = MMDDriftOnline(
    x_ref, ert=500, window_size=20,
    backend="pytorch", n_bootstraps=2000,
)
cd.reset_state()                      # a stateless detector saves cleanly
save_detector(cd, "artifacts/fraud-online/v14/")

The comment matters. save_detector() keeps online state when t > 0, so to ship a clean detector you call reset_state() first, otherwise you bake one process's partial window into every replica that loads it.

State and replicas

Online detectors keep per-stream state, which breaks the usual horizontal scaling reflex. Two replicas of an online consumer each see half the stream, each fills its own window with non-contiguous data, and each tests something that is not the stream you meant. The patterns that work are one consumer per stream (partitioned by a key such as tenant or model), or save_state and load_state for controlled handover. Autoscaling that treats the consumer as stateless is a bug.

Detector artifacts as a supply chain

A saved detector is a bundle of data and code. Once other teams load them, you are running a small artifact supply chain, and the concerns are the same as for model files and container images: where they come from, whether they can be tampered with, and whether the one you load is the one you meant.

Two formats, two risk profiles

save_detector(cd, path) writes the config format by default for drift detectors: a human-readable config.toml plus artifacts such as .npy data and model folders. It can be read in review, diffed and kept in version control. The legacy format serialises with dill. It is the only format for outlier and adversarial detectors, drift detectors can opt in with legacy=True, and it is not supported for Cramér-von Mises, Fisher's Exact Test or the online detectors.

For a platform the choice is nearly always the config format wherever it is available, for three reasons: the artifact is reviewable, you can run validate_config() on it before anything heavy loads, and it removes pickling of the detector object from the trust model. A minimal config shows how small the reviewable surface is:

name = "MMDDrift"
x_ref = "x_ref.npy"
p_val = 0.05

name is the detector class, and every other key is a constructor argument. Backend options go in the same file. Complex pieces can reference registry entries with the @reference.v1 syntax, such as "@cd.tensorflow.preprocess.preprocess_drift", or your own functions registered with registry.register.

Validation as a gate

validate_config() runs in two stages. The unresolved stage checks field names and types before any artifact loads, and the resolved stage checks loaded artifacts before the detector is built. For a platform this is a pre-merge check: a pull request that changes a detector config runs the unresolved stage in seconds, and a nightly job runs the full load_detector round trip. Failing fast before a slow model load saves real money in CI.

Version pinning is the contract

Cross-version serialization is not guaranteed by the docs, and loading an artifact saved by a different alibi-detect version only issues a warning, which is easy to miss in a log. The documentation strongly recommends matching alibi-detect, Python and dependency versions between save and load. Legacy dill artifacts add a sharper edge: they can fail to load on a different Python minor version.

So the platform contract is simple to state and must be enforced by tooling:

1. Pin alibi-detect, Python, numpy, and TensorFlow or PyTorch exactly in the runtime image.
2. Record those versions beside every artifact, including meta['version'] from the detector.
3. Run a CI job that loads every promoted artifact in the exact runtime image and calls predict on a fixture.
4. Treat a version-mismatch warning at load time as a failure in production, by promoting it to an error in your loader wrapper.

import warnings
from alibi_detect.saving import load_detector

def load_strict(path: str):
    with warnings.catch_warnings():
        warnings.simplefilter("error")      # a version-mismatch warning becomes a failure
        return load_detector(path)

The trust model and security

The library makes no network calls and holds no credentials, so there is no service inside it to attack. The risks are in what it loads and what it stores.

Deserialization is code execution

The legacy format uses dill, and PyTorch models are saved through dill-compatible pickling. Loading a pickle-family file can execute arbitrary code. If an attacker can write to your artifact store, or if a team can load an artifact from a path they do not control, then load_detector is a remote code execution primitive running with the detector service's permissions.

The controls are the standard ones for pickled artifacts, applied firmly:

• Only load from locations you control. The detector runtime's service account gets read-only access to a single artifact prefix, and no one else gets write access to it except the promotion pipeline.
• Checksum or sign every artifact at promotion and verify before load. A digest recorded in the registry entry is the minimum; signing, as you would for container images (see the Trivy guide for the scanning side), is better.
• Prefer the config format, which removes pickling of the detector itself, while remembering that a config can still reference .dill files and registry functions. Review those references like code.
• Run loading with least privilege: no cloud credentials beyond the artifact prefix, no write access to the metrics store from the loading step, and a network policy that denies egress you do not need.

The reference set is training data

x_ref.npy is a copy of part of the training or validation data, which means it may contain personal data. For Gulf and Egyptian employers this has a concrete form: data-residency and data-protection rules that apply to the training set apply to the reference set too, and a reference file copied into a convenient bucket in another region is a compliance finding. Store it encrypted, restrict access the same way as the source dataset, and where possible reference an embedded or aggregated form rather than raw records. Because x_ref_preprocessed=True lets you hold only embeddings, that is a legitimate way to reduce what is stored, subject to the embedding itself not being invertible enough to leak the originals, which you should assess for images and text rather than assume.

The licence is an architectural constraint

This is the fact that should be on the first page of every design document. Up to 0.11.4, Alibi Detect was Apache 2.0. From 0.11.5 (January 2024) it is licensed under the Business Source License 1.1, and PyPI classifies it as "Other/Proprietary License". The docs describe the project as source-available, and it is not open source under the OSI definition.

The practical content of the licence, as it stands in the repository:

• You may copy, modify, create derivative works, redistribute, and make non-production use.
• The Additional Use Grant allows production use by non-profit educational institutions, provided that use does not include commercialising the licensed work or putting it into models, products or services sold, licensed or marketed to third parties.
• Anyone else needs a commercial licence from Seldon for production use.
• Each version converts to Apache 2.0 on a change date, the fourth anniversary of that version's first public distribution. For early BSL versions that would be around January 2028, though you must read the exact date from the LICENSE file rather than trust this guide.

For a platform owner this changes the design in four ways.

It is a procurement decision, not an engineering one. Adopting the library in a production path at a company is a licensing event. Have legal or procurement confirm the position before teams build on it, and record the decision. Learning and prototyping are fine; shipping is where the licence applies.

It can leak into images you did not mean to ship. There is a public issue in the MLServer project, titled "Remove alibi-detect and alibi-explain from MLServer base image due to license mismatch", which shows how a source-available dependency inside a shared base image causes problems for downstream users. The lesson for you is to run licence scanning on your own base images and to keep Alibi Detect in a dedicated image rather than a shared one that many unrelated services inherit.

It complicates hosting for others. If your platform offers detection as a service to other business units, or to external customers, whether that counts as offering a service in the sense of the grant is a question for the licensor and your lawyers. The fact sheet behind this guide could not settle it, so do not assume.

It creates a clean escape hatch to plan for. Because the interface you expose to teams is your wrapper (a check() function returning a plain dictionary), you can substitute another engine behind it. Versions up to 0.11.4 are Apache 2.0, but they are the Python 3.7 to 3.11 era and carry dependency and security costs, so pinning to one is a stopgap, not a strategy. Alternatives in this catalogue include Evidently and NannyML; read their licences yourself before relying on this guide's summary of anyone.

Multi-tenancy

Alibi Detect has no concept of a tenant. A detector is an object, a reference set is an array, and nothing in the library separates one team's data from another's. Multi-tenancy is therefore a layout you impose, and the decisions are about isolation, memory and noise.

The unit of tenancy is the detector instance

Use one detector instance and one reference set per tenant and per model version, stored under separate, access-controlled paths. A path scheme that encodes the owner, the model, the model version, the encoder version and the detector version gives you four things at once: access control by prefix, a clear record of what produced an alert, safe rollback, and no way for one team's refresh to overwrite another's reference.

drift-artifacts/
  team-risk/
    fraud-scorer/
      model-v22/
        encoder-v3/
          detector-v14/
            config.toml
            x_ref.npy
            MANIFEST.sha256

Online detectors need one more rule: state is per stream, so every online detector instance is keyed to exactly one stream, and the key is part of its identity. Mixing two tenants' instances into one window does not just leak data between them, it makes both of their tests meaningless.

Memory is the noisy-neighbour resource

Each loaded detector holds its x_ref and possibly a model in memory. A runtime that loads every tenant's detector into one process will be limited by memory long before CPU. Two patterns hold up: a pool of workers each loading the detectors for a bounded set of tenants, or one short-lived job per detector in batch mode, which isolates memory by construction and lets the scheduler enforce resource limits. The second is simpler and is what the batch default gives you for free.

For kernel detectors, one tenant with an oversized reference set can monopolise a worker for minutes. Enforce the reference-size ceiling at promotion time, and set resource requests and limits on the job so that one tenant's cost shows up on their own line of the bill rather than in a shared pool.

Reference data policy

The update_x_ref argument decides how the reference set changes over time, and it is the most consequential single setting on a drift detector, because it controls what the detector is willing to call normal.

The options are {'last': N}, which keeps the most recent N instances, and {'reservoir_sampling': N}, which keeps a uniform sample over everything seen. Both make the reference follow the data. That is exactly right when you want to absorb gradual, harmless change, and exactly wrong when you want to detect it: a reference that tracks the data will never alert on slow drift, because the baseline moves with it.

So reference policy is a governance decision with three honest answers.

Frozen. The reference is a sample of training or validation data and never changes until a person re-baselines it. Slow drift is detected indefinitely, and so is every harmless seasonal change, which means alerts that tell you the world has moved on and nothing else. Frozen references are right for models whose training distribution is the contract.

Rolling. update_x_ref keeps the reference current. This answers "did something change recently" rather than "is production different from training". It needs a separate, slower control to catch slow drift, otherwise you have built a monitor that is blind to the most common kind.

Re-baselined on a schedule or an event. A person or a pipeline replaces the reference after a retrain, a deliberate migration or a documented seasonal shift. This is the most useful in practice and the most process-heavy, because every re-baseline is a change to what the monitor means and belongs in the change log.

The robust platform pattern combines two detectors on the same feature set: a frozen-reference detector for "different from training" and a rolling one for "different from last week". Their disagreement is informative: frozen fires and rolling does not means slow drift or a regime change that has settled; rolling fires and frozen does not means a short-lived blip that returned to normal.

Deployment patterns in production

With the pieces above fixed, the deployment question is how the runtime is hosted. The three shapes from the start of this guide each have an operational profile.

Batch on a scheduler

A job per model per window. It loads the artifact (strictly, as above), pulls the window from the capture store, runs the check, writes results, and exits. This is the cheapest, most testable and most observable option. Its weaknesses are latency, which is the window length, and the capture store, which must retain enough history for backfills. Backfills deserve a design: when a detector is fixed or a new one is added, you will want to replay history through it, and the job must accept a time range rather than only "the last window".

Make the job idempotent. Key results by model, detector version and window, so a retry overwrites rather than duplicates, and a dashboard never shows the same window twice.

Event-driven on Kubernetes

The official pattern pairs Seldon Core's payload logger with Knative eventing. It assumes a Kubernetes cluster with Istio and a load balancer, Seldon Core installed and the Istio ingress address known, and it fans events from a broker to serverless detector services. The attraction is that detection is decoupled from the serving path: a slow or failing detector never adds latency to predictions. The costs are operational: more moving parts, event delivery semantics to understand, and a dependency on documentation that targets Seldon Core v1. Before adopting it, confirm the current state of the Seldon Core and MLServer story with the vendor, because the fact sheet behind this guide could not.

If you already run Kubernetes and KServe, a simpler equivalent is to log inference payloads to a queue or object store from the serving layer and run the batch shape against that store. You lose seconds of latency and gain a design the whole team can debug.

Observability and alert design

A platform that reports drift must itself be observable, because its output is trusted most when it is quiet. Export results to the metrics system you already run (see Prometheus and Grafana), and keep three families of signal separate.

Result signals. is_drift as a gauge per model and detector, per-feature p-values and distances as histograms, and for online detectors time and test_stat. Attach meta['version'] and the detector name to every result so that an alert can be tied to the code that produced it.

Pipeline signals. The signals that stop a quiet monitor from being mistaken for a healthy one: the timestamp of the last successful check per detector, the number of instances in the last window, the age of the reference set, and the number of tests skipped for too little data. The most valuable alert you can write is "no check completed for this model in twice the expected interval", because it catches every upstream failure in the first diagram of this guide.

Alert rules sit on top of these: persistence over several windows, a suppression while a known deployment is in progress, and routing to the model's owner rather than a shared channel. Where labels arrive later, add the performance metric to the same dashboard and teach readers to read drift as a reason to look, not a verdict. If you use OpenTelemetry, tracing a check from window selection to result write makes slow checks diagnosable.

Upgrades and migrations

Upgrades are risky here for a reason specific to this library: dependency ceilings. Alibi Detect pins TensorFlow below 2.19 as of 0.13.0, requires Python 3.9 or later, and the release history shows steady churn in numpy, scipy, pandas and Pydantic bounds. Release 0.13.0 (December 2025) added Python 3.12 support, raised the TensorFlow bound, fixed numerical stability in the spectral residual detector and a scipy compatibility problem, and made saving with legacy=True require TF_USE_LEGACY_KERAS=1 so TensorFlow uses Keras 2 instead of Keras 3. Check PyPI for any later release before you plan, since the picture moves.

A safe upgrade sequence

1. Read the changelog and licence of the target version. The licence changed once without a major version number; assume it can again.
2. Build the new runtime image in a branch with every pin explicit, and scan it.
3. Load every promoted artifact in the new image with the strict loader and run predict on fixtures. Record the outcomes.
4. Compare outputs old against new on identical fixtures. Numerical changes between versions, such as the spectral residual fix, are expected in places; a changed is_drift on a stable fixture is a finding.
5. Re-save artifacts under the new version and promote them as new detector versions, never in place.
6. Run both detector versions in parallel for a period and compare alert counts before retiring the old one.

Migration traps, by version

The release history gives you a short, concrete checklist. From before 0.11.0, delete config_spec from saved configs and re-save. Code that read meta['detector_type'] to learn whether a detector was online must now read the separate online field, because detector_type since 0.11.0 says drift, outlier or adversarial. For TensorFlow users moving past 2.15, legacy saves need the Keras environment variable. Python 3.7 support ended in 0.11.4, and the package has long since moved on to requiring 3.9.

When the base image must move to a TensorFlow version the pins forbid, the honest options are to stay on the PyTorch backend for new detectors, or to isolate the detector runtime in its own image so that the rest of the estate is free to upgrade. Because the platform wrapper hides the library from teams, either path is invisible to them.

Cost

Costs come from four places. Licence: commercial production use needs a licence from Seldon, the line item most often forgotten in build-versus-buy estimates. Compute: dominated by the encoder forward pass and by permutation tests, so sample traffic, shrink the reference and check slow-moving models less often. Storage: small per model but large across hundreds of versions, so keep promoted versions and a bounded history. People: the most expensive and least measured, because a noisy platform costs more in triage attention than in compute. Track alerts per model per week and the fraction that led to action. Attribute cost per tenant using the artifact layout and job resource limits, and show each team the monitoring cost beside its serving cost.

Governance

Governance for a drift platform is the small set of rules that stop it from becoming either ignored or untrustworthy. It does not need a committee; it needs a handful of decisions written down.

Ownership. Every detector has a named owning team and an on-call route. A detector with no owner is deleted after notice; it is worse than none because it creates false assurance.

Standard defaults. The corrected p_val, the persistence rule, the maximum reference size, the allowed backends, the artifact layout, the encoder-versioning rule. Teams may deviate with a recorded reason and an expiry.

Change control. Detector configs live in version control and change through pull requests. Reference replacements, encoder changes and model rollouts that affect a detector's input are changes to the monitor and are reviewed as such.

Licence and security review. New production use of the library passes the licence check; new artifact sources pass the trust review; the runtime image is scanned and rebuilt on a schedule.

Incident playbooks

These four situations account for most pages. Each playbook starts with the question that narrows the cause fastest.

Playbook 1: a burst of drift alerts

First question: did the data change, or did our pipeline? Check, in this order: did a model, encoder or feature-pipeline release happen in the window; did the capture volume or sampling rate change; did the reference or detector version change; did an upstream schema or unit change (a currency, a timezone, a null-handling default). Per-feature results (drift_type='feature' with p-values and distances) show whether one feature moved, which points at a pipeline or schema cause, or all of them, which points at a transform applied to everything, including the encoder. Contain by suppressing the alert with a time limit, not by editing the threshold. Fix the cause, then record whether the reference needs re-baselining.

Playbook 2: the monitor went quiet

First question: is it quiet because the data is stable or because nothing is being checked? Look at the pipeline signals: last successful check, window size, skipped tests. If checks stopped, find which arrow of the architecture diagram broke. If checks ran on a window far smaller than intended, power collapsed and the silence is meaningless. If a rolling reference has absorbed a slow shift, compare against the frozen detector. Then prove the monitor can still see: replay a known-drifted fixture through the live artifact.

Playbook 4: an online detector alerts forever, or never

First question: is it state or parameters? A detector that keeps alerting after the incident has probably not been reset: call reset_state() as part of the post-alert action. A detector that never alerts may have an ert too large, a window_size too small for subtle drift, or state that was lost and restarted so often that the window never fills. Check time, the instance count, to see whether the detector has actually been processing a continuous stream.

Where Alibi Detect stops

A platform owner earns their keep by knowing when a tool is the wrong one. Alibi Detect is a library of detectors: strong on algorithm breadth (offline and online drift, outlier and adversarial detection across tabular, text, image and time-series data), and deliberately silent on everything around them. Choose something else, or add something, in these cases.

• You want reports and dashboards out of the box. Alibi Detect returns dictionaries. If your teams want a data-quality or drift report with visuals and no code, Evidently is closer to what they are asking for, and NannyML addresses performance estimation without labels, which a drift test does not.
• You cannot take the licence. Production use at a company needs a commercial licence. If that is a blocker, the answer is a different engine behind your wrapper, not an old pinned version.
• You need a managed service. There is no hosted offering inside the library. Cloud platforms and the Seldon commercial products exist for that; evaluate them on their own terms.
• Your problem is data quality, not distribution shift. Nulls, schema breaks and range violations are better caught by validation tooling before any statistical test. Great Expectations is the catalogue's answer, and it belongs upstream of drift detection.
• You need concept-drift verdicts. The library tests inputs, and offers uncertainty-based detectors (ClassifierUncertaintyDrift, RegressorUncertaintyDrift) that work without labels, but confirming concept drift usually needs labels. If labels are the bottleneck, the problem is a labelling pipeline.
• You need LLM-specific monitoring. Embedding-based drift tests can run on text, but evaluating generative output quality, retrieval quality or safety is a different job, covered by the tools in the LLM part of this catalogue.
• Your constraint is the Python and TensorFlow window. If your estate must run a TensorFlow or Python version outside the pins, you will fight the packaging constantly.

The mark of a good decision here is that it is written down with its trigger. "We use Alibi Detect behind our wrapper until legal says no or until the pins block our base image" is a decision. "We use Alibi Detect" is a habit.

Running Alibi Detect as a platform for other teams

Everything above converges on one idea: teams should not use Alibi Detect, they should use your platform, which uses Alibi Detect. Concretely, the platform is a small set of things.

A paved path. A template repository with the artifact layout, a build step that constructs, validates and saves a detector, a CI job that loads it strictly and checks fixtures, and a deployment that runs batch jobs. A team onboarding a model supplies reference data, an encoder if needed, and a short configuration of features and cadence.

A narrow interface. Teams call check(model, window) and receive a plain dictionary with a fixed schema: the decision, the per-feature view, the detector and artifact versions, the window and the sample size. They never import the library. This is what makes an engine swap, a licence change or an upgrade a platform concern rather than forty teams' concern.

The wrapper below shows how little code the narrow interface needs. The platform's value is in everything around it.

import time
from dataclasses import dataclass, asdict

from alibi_detect.saving import load_detector

@dataclass
class CheckResult:
    model: str
    detector_version: str
    library_version: str
    window: str
    n_instances: int
    is_drift: int
    p_val: list | None
    distance: list | None
    duration_s: float

def check(model: str, version: str, window: str, x, path: str) -> dict:
    t0 = time.time()
    cd = load_detector(path)                      # use the strict loader in production
    batch = cd.predict(x, drift_type="batch")
    flag = int(batch["data"]["is_drift"])
    p_val = distance = None
    if flag:
        feat = cd.predict(x, drift_type="feature",
                          return_p_val=True, return_distance=True)
        p_val = feat["data"]["p_val"].tolist()
        distance = feat["data"]["distance"].tolist()
    return asdict(CheckResult(
        model, version, batch["meta"]["version"], window,
        len(x), flag, p_val, distance, time.time() - t0,
    ))

Orchestration of such checks fits any scheduler; tie the results to the pipelines that retrain and redeploy the model (the Kubeflow and Airflow guides cover those), so that a confirmed drift can trigger a retraining proposal that a person approves.

The review checklist

Use this for pull requests that add or change a detector, and for design documents that propose monitoring. A reviewer should be able to answer each line from the change itself.

Statistics

• The p_val and correction are the platform defaults, or the deviation is justified.
• The expected false-alarm count per week is stated.
• The test window is large enough to see the effect size the team cares about.
• High-dimensional inputs are reduced before testing, and the reduction is justified.
• Alerts are rules over several windows, not a single batch.

Reference and encoder

• The reference is a documented sample, with its source and date.
• The reference policy is stated: frozen, rolling or re-baselined, with a reason.
• The encoder is versioned with the detector, and rollouts that change it are linked.
• The reference size is under the platform ceiling.

Artifacts and security

• The config format is used wherever available.
• validate_config() runs in CI, and a strict load plus fixture predict runs against the runtime image.
• Artifacts are checksummed or signed, stored under an immutable versioned path, and loaded read-only.
• x_ref has the same access, encryption and residency rules as the training data.

Operations

• The job is idempotent and replayable.
• Online detectors have one stream each, a checkpoint plan and a post-alert action.
• Pipeline signals exist: last successful check, window size, reference age.
• The alert has an owner and a route.

Licence and cost

• The production use of the library has been cleared against the Business Source License.
• The runtime is pinned exactly, scanned and on the rebuild schedule.
• Cost per check is estimated, and sampling is considered.

The complete picture

Put the series together and the shape is consistent. A detector is a statistical test with a reference set. Its output is only as meaningful as the alarm rate you chose, the space you tested in, the baseline you kept and the pipeline that fed it. Around that small core sit the things this level is about: an artifact supply chain that can execute code if you trust the wrong source, a licence that turns production use into a procurement decision, state that makes online detection awkward to scale, and a failure mode in which the monitor goes quiet and looks healthy.

Where the series leaves you

You can now operate Alibi Detect for other people rather than only use it. That means you can state what an alert costs and what it means, keep the artifacts that produce it trustworthy, scale the right component, upgrade without silent behavioural change, and say plainly when the library is the wrong tool. Take the instinct forward into the neighbouring guides: Evidently for report-driven monitoring, NannyML for estimating performance without labels, Seldon Core and KServe for the serving layers detectors attach to, and Prometheus and Grafana for the signals that keep the whole system honest.

The habit that matters most is the one the whole guide has circled: do not trust silence you have not tested. Replay known drift through the live path on a schedule, and the rest follows.

Sources

• Alibi Detect documentation
• Getting started and installation
• Algorithm overview
• Saving and loading detectors
• Config files
• Drift detection
• Drift detection methods
• Kolmogorov-Smirnov drift detector
• Online MMD drift detector
• Outlier detection
• Adversarial detection
• Deployment with Seldon Core and Knative
• Alibi Detect on GitHub
• Release notes
• Changelog
• Licence
• alibi-detect on PyPI
• MLServer licence issue

Part three of three. At this level an interviewer is not checking whether you can call predict. They want to know whether you can design a monitoring platform around a library with sharp edges, respond when it goes wrong, and argue for the money and the constraints it needs. Every answer below is written the way a strong candidate would speak it, followed by what separates it from a weak one.

The ninety-second answer

If you are asked to describe your ownership of Alibi Detect in an interview, this is the shape to aim for:

"Alibi Detect is a Python library of drift, outlier and adversarial detectors. A drift detector tests whether a batch of production data differs from a reference sample, and returns a dictionary with an is_drift flag. It has no scheduler, storage or alerting, so a platform is everything around it: capture, a versioned store for reference sets and saved detectors, a runtime, and an alert path. The three risks I manage are statistical, operational and legal. Statistically, the alarm rate across many features and checks, and whether the test has the power to notice real change. Operationally, state in online detectors, artifact versioning and silent failure of the monitor itself. Legally, the licence has been Business Source License 1.1 since 0.11.5, so production use needs a commercial licence unless you are an eligible non-profit educational institution."

Three sentences from that answer carry the most weight: the library does not include a platform, the monitor can fail silently, and the licence is a design constraint. Candidates who miss all three sound like users. Those who include them sound like owners.

System-design prompts

Prompt 1: "Design drift monitoring for forty models across six teams."

Start by asking questions, because the answer changes with them. How are predictions served (batch or online)? Are labels available, and how late? What data types are involved: tabular, text, images? What is the tolerance for detection delay: hours or days? Is there a regulatory constraint on where reference data may live? Has anyone cleared the licence for production use?

Then describe the architecture in four boxes. Capture: sample inputs from the serving layer into a store, at a rate chosen per model. Reference store: each model gets a reference sample, an encoder if the data is high dimensional, and a saved detector, all under an immutable versioned path keyed by team, model, model version, encoder version and detector version. Runtime: batch jobs on the scheduler you already operate, one per model per window, loading artifacts through a strict loader that treats a version-mismatch warning as an error. Results: a fixed result schema exported as metrics, with alerts evaluated over several windows rather than one.

Then state the defaults you would impose: a corrected threshold (correction='fdr' or 'bonferroni' with a stricter p_val), a cap on reference size, the config save format wherever it is supported, and a persistence rule for alerts. Explain that teams interact with a check() wrapper and never import the library, because that gives you an upgrade, licence and engine-swap seam.

Close with how you would know it works: pipeline signals (last successful check, window size, reference age), a synthetic-drift replay on a schedule, and the fraction of alerts that led to action.

What a strong answer covers that a weak one misses. The weak answer draws a detector and a dashboard. The strong one covers the false-alarm budget across forty models, ownership of each arrow's liveness, the licence question asked at the start, and how a team onboards without help.

Prompt 2: "Add real-time drift detection to a fraud model that scores thousands of requests per second."

The first move is to challenge "real-time". Ask what happens in the business between a drift occurring and a person acting on an alert. If the answer is hours, a batch job over the last few minutes of sampled traffic is simpler, cheaper and testable, and it is the right call. If the answer is truly minutes, an online detector is justified, and the design must handle its properties.

State the properties. An online detector such as MMDDriftOnline takes ert (expected run-time to a false alarm with no drift) and window_size, runs n_bootstraps simulations in its constructor to set the threshold, and keeps state per stream. From those facts follow the decisions. Build and save the detector at deploy time, not at pod start, calling reset_state() before saving so the artifact is stateless. Partition the stream by a key so each consumer owns one detector instance and one ordered stream; do not load-balance one stream across replicas. Do not call the detector in the scoring path; feed it from a queue or log so a slow detector can never slow a fraud decision. Checkpoint state with save_state and decide in advance whether a crash restores with load_state or accepts a gap with reset_state(). Define the post-alert action, since the detector does not reset itself.

Be candid about trade-offs: a larger ert means fewer false alarms and slower detection, a smaller window catches severe drift quickly and misses subtle drift. Then say what you would pair it with: a slower batch detector against a frozen reference, because an online detector with a window cannot notice a baseline that has been moving for weeks.

What separates the answers. Weak answers add an online detector to the request path and scale it with replicas. Strong answers keep it off the hot path, name the state problem, and offer batch as the default.

Prompt 4: "The data science team wants to use Alibi Detect in a customer-facing product. What do you do?"

This is a licence question disguised as a design question, and the right first answer is "I check the licence before anyone builds". Since 0.11.5 (January 2024) the library is under Business Source License 1.1: non-production use and some other rights are granted, production use by non-profit educational institutions is allowed under conditions, and anyone else needs a commercial licence from Seldon. Embedding detection into a product sold or marketed to third parties falls squarely in the area the grant excludes.

So the sequence is: bring in procurement or legal, get a written position on the use, and in parallel design so that the decision is reversible. The reversible design is a thin wrapper with a fixed result schema, so another engine such as those covered in the Evidently and NannyML guides can sit behind it if the licence cannot be agreed. Say plainly that pinning to the last Apache 2.0 release (0.11.4) is not a strategy, since it is the Python 3.7 to 3.11 era and carries dependency and security costs.

What separates the answers. A weak answer says "it is open source, so fine". A strong answer knows the licence history, the grant's shape, and that you do not guess about hosting detection as a service for other parties; you ask.

Incident-response questions

Interviewers use incidents to test how you narrow a problem under uncertainty. Structure each answer as: the first question, the checks in order, the containment, the fix, and the lesson.

"A burst of drift alerts fired across twelve models at 03:00."

The first question is whether the data changed or the pipeline did. Twelve models at once is the clue: independent real-world drift across twelve unrelated models at the same minute is improbable, so you look for a shared dependency. Did a shared feature pipeline release? Did a schema, unit or null-handling default change upstream? Did the shared encoder or the runtime image change? Did the capture sampling change? Per-feature output (drift_type='feature' with p-values and distances) tells you whether one feature moved, which points at a source schema problem, or everything moved, which points at a shared transformation.

Contain by suppressing the alert with an expiry, and never by loosening the threshold. Fix the cause, then decide whether any reference needs re-baselining, and record the decision. The lesson is a new alert for changes to shared pipelines, joined to the drift dashboard.

"Nothing has alerted for three months. Is that good?"

This question tests whether you distrust silence. The right answer is "I cannot tell from the alerts alone". Check pipeline signals: last successful check per detector, the number of instances in each window, skipped tests, and the age of the reference. Check whether a rolling reference (update_x_ref) has absorbed a slow shift. Then prove the monitor can still see by replaying a known-drifted fixture through the live artifact in staging and confirming that it alerts.

Add the point that the three most common causes of a silent monitor are not statistical: capture stopped, the job stopped, or the alert route changed. The statistical causes are windows too small for the effect size and a test space that discards what moved.

"After an upgrade, detectors fail to load in production."

First question: what changed in the runtime? Almost always a version of alibi-detect, Python, numpy, scipy, TensorFlow or PyTorch. Read the version-mismatch warning, compare with the detector's recorded meta['version'] and the pins, and reproduce in the image that saved the artifact. Two release-specific causes to name: a config saved before 0.11.0 with a config_spec line that must be deleted, and a legacy save on newer TensorFlow that needs TF_USE_LEGACY_KERAS=1. A custom function named in a config fails if the module that registers it was not imported before loading.

Restore service first by rolling back the image or flipping the artifact pointer. Investigate afterwards. The lesson is a CI job that loads every promoted artifact in the candidate image and compares predict output on fixtures, so that this failure appears in a pull request.

"A teammate loaded a detector from a shared bucket and our service made outbound calls to an unknown host."

Treat it as a security incident. Deserialization of pickle-family files, which includes the legacy format and PyTorch models saved through dill-compatible pickling, can run arbitrary code. Contain: isolate the workload, revoke its credentials, and block the egress. Establish what could be reached with the service's permissions and rotate it. Find who could write to the bucket and how the file got there. Recover from an artifact you can verify against a recorded checksum.

The durable fixes are write access restricted to the promotion pipeline, checksum or signature verification before load, preference for the config format, and no unneeded egress from the detector runtime. A strong answer says "rotate first" rather than "delete the file", because deletion is not containment.

"A model owner says your monitor missed a real accuracy drop."

Do not defend the monitor. Ask what changed: if the input distribution did not move but the relationship between inputs and labels did, that is concept drift, which a test on inputs does not detect. Alibi Detect does offer uncertainty-based detectors (ClassifierUncertaintyDrift and RegressorUncertaintyDrift) that work without labels, and the platform should join drift output with performance metrics when labels arrive. If the input did shift but the monitor was silent, check power: was the window large enough, was the test space reduced in a way that discarded the change, did a rolling reference absorb it? Then fix the monitoring gap you found and say publicly what drift detection does not promise.

Arguing a trade-off to leadership

At this level a good share of the interview is persuasion. Leadership cares about risk, cost and speed. You need to translate statistics into those terms without distorting them.

"Why are we paying for, or building around, a monitoring tool at all?"

Argue from the cost of silent failure. A model that degrades unnoticed costs revenue or exposes the company to harm until someone complains. Drift monitoring is cheap insurance against the unnoticed period. Then be honest about what it buys: an early reason to look, not a verdict. Offer a measurable promise: models monitored, median time from a data change to a person looking, and the fraction of alerts that led to action. Commit to reporting those numbers.

"The licence costs money. Can we use the free version?"

Lay out the options with their costs, and let the decision-maker choose. Option one: a commercial licence from Seldon, with a known recurring cost and vendor support. Option two: another monitoring tool whose licence fits, with migration and capability costs. Option three: build detection in-house on statistical libraries, which carries a high engineering cost and a long tail of maintenance. Option four: pin an old Apache release, which I would recommend against because of dependency and security debt. Recommend one with reasons, and state what would change your mind. Never present "just use it" as free, because an audit finding costs more than the licence.

"Product wants alerts on every model, immediately."

Translate the request into the alarm budget. Show the arithmetic: features times checks per day times the significance level equals expected false alarms per day without correction. Propose a staged rollout: monitor in shadow mode for two weeks, measure the real alert rate, tune the corrected threshold and persistence rule, then enable paging. Frame it as protecting the credibility of the alerts. A pager that cries wolf in the first month teaches teams to ignore it, and that is the harder problem to undo.

Trade-off questions

These are short, and each rewards a two-sided answer.

Offline or online detectors as the platform default? Offline. State stays in the store, retries are safe, and tests are reproducible. Offer online detectors for use cases where delay is costly, and only with stream-keyed consumers and a checkpoint plan.

Config format or legacy format? Config where the detector supports it: it is readable, validated by validate_config(), reviewable in a pull request, and avoids pickling the detector object. Legacy is the only option for outlier and adversarial detectors, so those artifacts need the strictest provenance controls.

Frozen or rolling reference? Frozen answers "is production different from training", and alerts on harmless seasonal change as well as slow drift. Rolling answers "did something change recently", and can be blind to slow drift. Run both on important models and read their disagreement.

TensorFlow or PyTorch backend? The one your serving stack already uses, bearing in mind that TensorFlow is capped below 2.19 in 0.13.0 and the backend string for PyTorch is 'pytorch' while the pip extra is [torch]. Where the TensorFlow cap blocks an image upgrade, PyTorch is the pressure valve.

Build a wrapper or let teams use the library directly? Wrapper. The cost is a small piece of code to maintain, and the benefit is control of licence, upgrades, defaults and the option to change engine.

What a strong answer covers that a weak one misses

Across all of these, a few habits mark the senior candidate. They ask about the business consequence before choosing a detector. They quantify the false-alarm rate rather than describing it. They treat artifacts as a supply chain and say "dill" and "pickle" in the security answer. They know the licence history, and they treat it as an input to design rather than a footnote. They distrust silence and describe a way to test for it. They distinguish drift from accuracy loss, and they say what the tool does not do. Finally, they volunteer the cost of their own recommendation, which is the clearest signal that someone has operated what they are describing.

Sixty-second self-test

Answer without looking back.

1. What did the licence change to in 0.11.5, and what does that mean for production use?
2. Why does p_val=0.05 become a problem across thirty features checked hourly, and which argument is the first lever?
3. What is ert, and which way does it trade against detection speed?
4. Why must an online detector's constructor run at build time, and why call reset_state() before saving?
5. Why do online detectors not scale by adding replicas?
6. Name two risks of the legacy save format and one reason to prefer the config format.
7. Why is the encoder part of the detector's version?
8. What does a rolling reference (update_x_ref) stop you detecting?
9. A burst of alerts across many models: what is your first question?
10. What is the most valuable alert on the platform itself?

If you hesitate on questions 1, 4 or 10, reread the licence, scaling and observability sections of the detailed guide, then rehearse the incident answers aloud. For the neighbouring systems an interviewer may ask about, see Evidently, NannyML, Seldon Core and KServe.

Part three of three. At this level the problems are not failed calls but quiet ones: a monitor that stopped watching, a reference set that went stale, a pager that nobody trusts, an artifact that loaded when it should not have. Start with the error table, then the platform habits, the review checklists and the practice cards.

Production traps that appear at scale

Cumulative: everything from Beginner and Mid still applies. These traps rarely show up in a notebook, because they need many models, many teams or many months.

Three of these deserve a closer read

The reference that follows the data. The row about slow drift is the most dangerous because it looks like success. update_x_ref={'last': 1000} or {'reservoir_sampling': 1000} is the right tool when you want to absorb change. It makes the detector blind to exactly the gradual change many teams most want to catch. A rolling detector tells you what changed recently; only a frozen one tells you how far you have moved from training. Run both on models that matter.

The warning that is the real error. A version mismatch at load time issues a Warning, and in production logs a warning is invisible. The documentation says cross-version serialization is not guaranteed. A load wrapper that turns warnings into exceptions (warnings.simplefilter("error") around load_detector) converts the library's polite hint into a deploy-blocking failure, which is what it should be.

The online detector's hidden state. Online detectors keep a timestep t and a sliding window. Saving after processing data bakes that state into the artifact, loading it into several replicas duplicates it, and a restart loses it. All three are mistakes of treating a stateful object as a stateless one.

Platform-ownership habits

These are the practices that pay off most, in order of leverage. None is clever. All are about making the platform trustworthy to people who did not build it.

1. Publish a false-alarm budget

For every monitored model, compute and publish the expected number of false alarms per week: tests per day times the significance level, after correction. When the observed rate differs sharply, you have learned something: the data is moving, the test is mis-specified, or the cadence is wrong. Teams that can see the budget stop arguing about whether alerts are "too many" and start discussing whether the budget is right.

2. Alert on persistence

A single is_drift = 1 is evidence. Page on a rule over several windows, such as drift in at least k of the last n checks, or a distance staying above a level. Keep the raw per-window results so the rule can change without re-running history.

3. Monitor the monitor

Export four things for every detector: the time of the last successful check, the instance count in the last window, the age of the reference set, and the number of skipped checks. Alert on "no completed check in twice the interval". It is the single alert that catches the largest number of different upstream failures.

4. Replay known drift on a schedule

Keep three fixtures per detector: stable data, clearly drifted data and borderline data. Run them through the live artifact in the live runtime weekly, and alert if the drifted fixture stops alerting. This turns the claim "the monitor works" into a test that runs.

5. Hide the library behind a wrapper

Teams call check() and receive a dictionary with a fixed schema: decision, per-feature view, detector version, library version, window, sample size and duration. They never import alibi_detect. This is how you handle a licence change, an upgrade or an engine swap as a platform task rather than forty team tasks.

6. Make artifacts immutable and promote by pointer

A saved detector is built once, written to a versioned path, tested and promoted by moving a pointer. A consumer in the middle of loading latest/ while you overwrite it will read half of each bundle. Include the encoder version and the library version in the path or the manifest.

7. Prefer the config format, and review it like code

config.toml can be read in a pull request and checked with validate_config(), whose first stage runs before any artifact loads and so is cheap enough for every commit. Review every .dill reference and every registry function in a config as you would review a new dependency.

8. Treat the reference set as sensitive data

x_ref.npy is part of the training set. Give it the same encryption, access and residency rules, and store embeddings rather than raw records when that is enough. For Gulf and Egyptian employers, keeping it in-region with the training data is the simplest defensible position.

9. Pin everything and test the load

Pin alibi-detect, Python, numpy and the backend exactly. A CI job loads every promoted artifact in the exact runtime image and calls predict on a fixture. Run it nightly as well as on change, so that a rebuilt base image cannot break artifacts unnoticed.

10. Ask the licence question first

Before a new production use, confirm the position under the Business Source License 1.1 and record it. Keep a list of where the library runs. Keep it in its own image, not a shared base.

11. Write down why every baseline moved

Reference replacements, encoder changes and model rollouts that affect a detector's input are changes to the monitor. Each gets a ticket, an owner and a reason, so that "the alerts stopped" can be traced to "we changed the question".

12. Delete detectors without owners

A detector nobody owns creates false assurance. Give notice, then remove it.

Review checklists

Use the first for pull requests that add or change a detector, and the second for design documents that propose monitoring for a model.

Pull request checklist

Detector and statistics

• Detector family matches the data: univariate test for tabular features, kernel or classifier detector with a reduction for images and text.
• p_val and correction are the platform defaults, or the deviation is justified in the description.
• Expected false alarms per week are stated.
• The window size is large enough for the smallest shift the team cares about.
• float32 arrays are passed to deep backends, and shapes match x_ref.

Reference and encoder

• Reference source, size and date are recorded.
• Reference size is under the platform ceiling for the detector type.
• update_x_ref is set deliberately, and the reason is written.
• If x_ref_preprocessed=True, the embedding is tied to a named encoder version.
• preprocess_fn is part of the saved artifact, not a notebook cell.

Artifact and runtime

• Saved in the config format, or the legacy format has a written reason.
• validate_config() passes in CI, and a strict load plus fixture predict passes in the runtime image.
• The artifact path is immutable and versioned, with a checksum in the manifest.
• Custom registered functions are imported before load_detector.
• Online detectors are saved after reset_state() and keyed to one stream.

Operations

• The job is idempotent, with results keyed by model, detector version and window.
• A backfill path accepts a time range.
• Staleness, window size and reference age are exported.
• The alert has an owner, a route and a persistence rule.

Design document checklist

• The business consequence of undetected drift is stated, with the delay the business can tolerate.
• The document says why batch is not enough before proposing online detectors.
• Capture, reference store, runtime and alert path are drawn, with a liveness signal on every arrow.
• Reference policy (frozen, rolling or re-baselined) is chosen per model with reasons.
• The relationship between drift and performance metrics is explained, including how labels arrive.
• The trust model names who can write artifacts and what the loader can reach.
• Data residency and access for reference data are addressed.
• The licence position is cleared and recorded, and the exit path is described.
• Upgrade plan covers the TensorFlow ceiling and the Python version.
• Cost estimate includes licence, compute, storage and triage time.
• Success measures are named, and so is the result that would end the project.

Practice cards

Each card is a short exercise to run in staging. Do them over a few weeks, one at a time.

Card 1: the alarm budget. Take a stable dataset with 20 features. Split it into hourly-sized batches. Run KSDrift with no correction, then correction='fdr', then 'bonferroni'. Count alerts per hundred batches in each case and compare them with the prediction from tests times p_val. Note how the correction changes sensitivity to a real shift by adding a small shift to one feature.

Card 2: the blind monitor. Build a rolling-reference detector with update_x_ref={'last': 1000} and a frozen one. Feed a slow linear shift. Record when each alerts. Write the paragraph you would send to a team explaining why you run both.

Card 3: the swapped encoder. Embed a reference with encoder A, then compute test batches with encoder B. Read the result. Then add the encoder version to the artifact path and write a CI check that refuses a mismatch.

Card 4: the strict load. Save a detector under one dependency set, load it under another, and find the warning. Wrap the loader so the warning fails the job. Add a fixture that compares predict output between the two.

Card 5: the online lifecycle. Build an online detector, save it with t > 0, load it twice and observe duplicated state. Rebuild, call reset_state(), save, load and compare. Then simulate a crash and compare load_state with reset_state().

Card 6: the silent night. In staging, stop the capture job and measure how long until anything alerts. Add the staleness alert and repeat. Record the improvement in minutes.

Card 7: the hostile artifact. In an isolated sandbox with no credentials, create a harmless pickle-family file that writes a marker file when loaded, and load it with your runtime. Observe it execute. Then add checksum verification and show that the loader refuses a modified artifact.

Card 8: the licence map. Search all repositories and images for alibi-detect, read the LICENSE at the version you run, and write a one-page position with the grant, the change date and the open questions for legal.

Operational recipes

Prove the monitor can see

import numpy as np
from alibi_detect.saving import load_detector

def test_live_artifact_detects_known_drift():
    cd = load_detector("artifacts/fraud-scorer/model-v22/encoder-v3/detector-v14/")
    drifted = np.load("fixtures/drifted.npy").astype(np.float32)
    stable = np.load("fixtures/stable.npy").astype(np.float32)
    assert cd.predict(drifted)["data"]["is_drift"] == 1
    assert cd.predict(stable)["data"]["is_drift"] == 0

Run it against the production artifact, in the production image, on a schedule. When the stable fixture alerts on a borderline-sized sample, widen the fixture rather than the threshold.

Make a saved online detector stateless

from alibi_detect.saving import save_detector

cd.reset_state()                                   # back to t = 0
save_detector(cd, "artifacts/fraud-online/v15/")   # no partial window baked in

Compute a per-tenant false-alarm budget

def expected_false_alarms_per_week(n_features: int, checks_per_day: float,
                                   p_val: float, corrected: bool) -> float:
    """Upper-bound view: uncorrected tests are n_features per check.
    With a family-wise correction the per-check rate is roughly p_val."""
    tests_per_check = 1 if corrected else n_features
    return tests_per_check * checks_per_day * 7 * p_val

This is arithmetic, not a measurement. Compare it with observed alerts in shadow mode and treat a large gap as the thing to investigate.

Pin a runtime image

alibi-detect==0.13.0
numpy==<pinned version>
tensorflow==<pinned version below 2.19>   # or torch, depending on the backend

Fill the placeholders from a resolved lock file, rebuild on a schedule, scan the image, and keep the lock file next to the artifacts it was used to save.

A closing checklist for the platform owner

• Every model has a named owner, a reference policy and a false-alarm budget.
• A staleness alert exists, and a weekly replay of drifted fixtures passes.
• Artifacts are immutable, checksummed, loaded strictly and written only by the promotion pipeline.
• The reference set is protected like the training set.
• The library is behind a wrapper, in its own image, with the licence position recorded.
• Upgrades are rehearsed with fixtures, and rollback is one pointer.
• You can say, for each detector, what drift it would miss.

If you want to see how neighbouring tools tackle the same problems, read the Evidently and NannyML guides, then the serving layers in Seldon Core and KServe.