The Complete NannyML Guide

This is part one of three. It covers everything you need to do real work with NannyML, not a teaser. By the end you can install the library, load a reference set and a production set, estimate how well a classification model is doing before any labels arrive, find which input columns have drifted, compare your estimates with the real numbers once labels show up, and read every alert the library raises without guessing what it means. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and the ideas only stick once you have watched your own chunks turn into a chart and seen an alert fire.

This guide is written against NannyML 0.13.1, the latest release on PyPI at the time of writing, which supports Python 3.9 to 3.12. A section near the end is honest about the project's maintenance status, because you should know it before you build on the library.

The problem: a model that fails without telling anyone

A model in production is a strange kind of software. A web service that breaks returns errors, and somebody notices within minutes. A model that breaks keeps returning answers. The answers are well-formed, they arrive on time, the HTTP status is 200, and every dashboard about latency and uptime stays green. Meanwhile the answers have quietly become worse, because the world the model was trained on is no longer the world it is being asked about.

Here is a concrete case. A bank in Cairo trains a model to predict whether a loan applicant will repay. It works well on last year's applicants. Then a new salary-transfer partnership brings in a different kind of customer, interest rates move, and the typical income in the application form shifts. The model still produces a probability of default for every applicant. Nobody sees an error. Three months later the risk team notices that defaults are higher than expected, and by then the bad loans are on the books.

The slow part of that story is the labels. You only know whether a borrower repaid after months have passed. For fraud, you often learn the truth after a chargeback dispute weeks later. For a churn model, you learn after the customer has already left. This delay between a prediction and the truth is the central difficulty of monitoring, and it is the reason a plain "compute accuracy every day" job does not work for most real models: the data for it does not exist yet.

NannyML exists for exactly this gap. It is an open-source Python library whose headline idea is to estimate the performance of a model without access to the true labels, and to tell you when the estimate drops. It also detects changes in the input data (called drift) and, importantly, helps you separate drift that matters from drift that does not.

What came before NannyML, and why it was not enough

Before libraries like this, teams monitored models in two common ways, and both have a flaw you can feel.

The first is monitoring the outputs and inputs with simple statistics: track the average prediction, track the mean of each input column, and alert if they move. This is cheap and it is a reasonable start. The flaw is that it produces a flood of alerts. Data always moves a little. A feature's average changes on Fridays, at month end, during Ramadan, in summer. Most of those movements do not hurt the model, so the team learns to ignore the alerts, and then the one that mattered is ignored too.

The second is waiting for labels and computing realized performance. This is the gold standard because it measures the thing you actually care about. The flaw is the delay described above. By the time the number is available, the damage is done.

Drift detection tools improved the first approach by applying proper statistical tests to distributions, and several of them are good. But a statistical test answers the question "did this distribution change?" and not the question "did my model get worse?". Those are different questions. A feature can shift dramatically and the model can be unaffected, if the model hardly uses that feature or the shift stays inside the range the model already knows. The reverse also happens: a small shift in an important feature can wreck accuracy.

NannyML's contribution is to put performance first. You start from an estimate of how the model is doing, and you use drift detection as the tool for explaining a drop in that estimate, not as the alarm itself. In the docs' own framing, the workflow is to estimate performance, check whether there is a problem, and then look for the cause in the data.

If you want to compare with a tool that focuses on rich drift reports and dashboards, read the Evidently guide at /student-guides/evidently, and for a library centred on statistical detectors, /student-guides/alibi-detect. Seeing two approaches side by side is one of the best ways to understand what each is for.

The mental model: four nouns

NannyML has a lot of classes, but a handful of ideas carry nearly everything. Learn these four nouns and the rest of the API will look familiar.

Reference data is a period where you trust the model. It is usually your held-out test set, or an early stretch of production for which you already have labels. It must contain the true outcomes, because the library uses it to learn what "normal" looks like: how much the metrics wobble from one slice to the next, and how the data usually looks. Think of it as the calibration measurement of an instrument.

Analysis data is the data you are monitoring: recent production inputs and the model's predictions. For performance estimation it does not need labels. That is the whole point. If labels exist for some analysis data, you can use them later to check the estimates, but they are not required to produce the estimates.

Chunks are the unit of everything. NannyML does not produce one number for your whole analysis set. It cuts the data into groups, either by number of rows, by number of groups, or by calendar period such as a week or a quarter, and computes each metric once per chunk. The result is a time series. A chunk has a key, a start and end, and a label saying whether it came from the reference or the analysis period. Chunking is the single idea most beginners underestimate, so a later section is devoted to it.

Thresholds and alerts turn numbers into decisions. During fitting on the reference data, NannyML measures how much each metric normally varies across reference chunks. By default the thresholds sit three standard deviations around the reference mean. In the analysis period, a chunk whose value falls outside those bounds is marked with an alert. So an alert does not mean "this is bad in an absolute sense". It means "this chunk is outside the range the reference data taught us to expect".

Two more names will appear immediately. Calculators and estimators are the objects you create: CBPE estimates performance for classification, UnivariateDriftCalculator measures drift column by column, and PerformanceCalculator computes realized performance. All of them follow the same rhythm you may know from scikit-learn: create the object, call fit() on reference data, then call estimate() or calculate() on analysis data. The result is a results object that you can filter, turn into a DataFrame, or plot.

Installing NannyML and checking the setup

NannyML is a normal Python package. The one decision that matters is the Python version: 0.13.1 declares support for Python 3.9 through 3.12. Python 3.13 is not listed, and on a newer interpreter pip may refuse to find a compatible release. If you have a choice, use 3.11 or 3.12.

Always create a virtual environment. A monitoring library pulls in pandas, scikit-learn, LightGBM, Plotly and others, and you do not want to disturb the rest of your machine.

python3.12 -m venv .venv
source .venv/bin/activate
pip install nannyml

On Windows, the recommended route for a course like this is WSL2, which gives you a Linux shell and avoids most binary-wheel surprises. If you prefer conda, the documentation also shows conda install -c conda-forge nannyml.

Verify the install by importing the package and printing its version:

python -c "import nannyml; print(nannyml.__version__)"

You should see 0.13.1. If you see something older, an earlier environment is shadowing the new one, so check which python you are running.

NannyML depends on LightGBM, which needs a native OpenMP runtime. On a Mac that means Homebrew's libomp; on a slim Linux container it means the libgomp1 package. If the import fails with a message about a missing dynamic library, install the matching runtime and try again:

brew install libomp                      # macOS
sudo apt-get install -y libgomp1         # Debian or Ubuntu

There is also an optional extra. Since version 0.12, the database dependencies are not installed by default. You only need pip install "nannyml[db]" if you plan to write results into a SQL database, which is not a beginner task.

If you work behind a company network that inspects TLS traffic, pip install can fail with CERTIFICATE_VERIFY_FAILED. Do not turn verification off. Ask your IT team for the proxy's root certificate and point pip at it with the PIP_CERT environment variable.

Your first project: estimating performance without labels

Now the main event. NannyML ships with sample datasets so you can practice without preparing your own. We will use the one from the official quickstart: a model that predicts whether a person in Massachusetts is employed, built from US census records. The data is split into a reference set from 2015 and an analysis set covering 2016 to 2018. A third table holds the true outcomes for the analysis rows, which we will keep aside to play the role of "labels that arrive later".

import nannyml as nml

reference_df, analysis_df, analysis_targets_df = nml.load_us_census_ma_employment_data()

print(reference_df.shape, analysis_df.shape)
print(reference_df.columns.tolist())
print(reference_df.head())

Spend a minute on that output. Among the columns you will see the input features, a timestamp, the model's predicted_probability (how sure it is that the person is employed), its prediction (the final yes or no), and employed, the true outcome. The reference table includes the true outcome. The analysis table does not, and that is deliberate.

The estimator for a binary classifier is called CBPE, short for Confidence-Based Performance Estimation. The idea is simple to say. If the model's predicted probabilities are well calibrated, then a prediction of 0.9 is right about 90 percent of the time. If you know the probability attached to every prediction, you can work out the expected number of true positives, false positives and so on, without seeing a single label, and from those compute a metric such as ROC AUC. The technique rests on two assumptions you must remember: the model's probabilities are calibrated, and the relationship between inputs and outcomes has not changed (no concept drift). We return to these in the limits section.

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc'],
    chunk_size=5000,
)

estimator.fit(reference_df)
results = estimator.estimate(analysis_df)

Read the constructor argument by argument, because each one is a promise about your data.

problem_type tells NannyML which family of metrics applies. The values are classification_binary, classification_multiclass and regression. y_pred_proba is the name of the column holding predicted probabilities, y_pred the column with the predicted class, and y_true the column with the real outcome (needed in the reference data only). timestamp_column_name points at the date column, which gives your results real dates on the chart axis. metrics is the list of metrics to estimate, here only ROC AUC, and chunk_size is the number of rows per chunk.

Then the rhythm: fit() learns from the reference data, and estimate() applies the estimator to the analysis data. Calling estimate() before fit() raises an error telling you the estimator was not fitted.

Finally, look at the output as a table and a picture:

print(results.filter(period='analysis').to_df())
results.plot().show()

The plot opens in your browser, or inline in a notebook. You will see a line of estimated ROC AUC over chunks, a shaded confidence band around it, and horizontal threshold lines. When the line leaves the thresholds, the chunk is highlighted. That highlight is your alert.

Reading the results: values, bands, thresholds and alerts

A results object is more than a chart. Each row of the DataFrame describes one chunk, and knowing the columns is how you stop treating the library as a black box.

For an estimated metric, a row carries the value (the estimate), the sampling error, the confidence boundaries (upper and lower), the thresholds (upper and lower) and a boolean alert. Chunk metadata sits next to them: the chunk key, start and end index, start and end date, and the period, which is reference or analysis. Call .to_df() and print .columns to see exact names in your installed version, since the DataFrame groups them by metric.

These columns answer different questions, and mixing them up is a classic beginner error.

Sampling error is the statistical noise in a metric computed from a limited number of rows. If you compute a ROC AUC from 200 rows, it could easily land a few points above or below the true value just by luck. NannyML estimates this noise for each chunk and draws a confidence band of plus or minus three sampling errors around the value. A small chunk gives a wide band, a big chunk a narrow one. The practical lesson is that a wobble inside the band is not evidence of anything.

Thresholds are different. They are not about noise in one chunk; they are about the normal range of the metric learned from the reference data. By default they are set at three standard deviations around the reference mean of the metric across reference chunks.

An alert is raised when the estimated value is outside the thresholds. Notice what this means: alerts are relative to what the reference taught the library. A reference set that is too short, or chunks that are too small, produce thresholds that are too narrow or too noisy, and you will see alerts on healthy data. A reference that is unrepresentative gives the opposite problem.

You can pull out only the alerting chunks with ordinary pandas. With multilevel=False the DataFrame is flat and easier to filter:

df = results.filter(period='analysis').to_df(multilevel=False)
print(df.columns.tolist())
alert_cols = [c for c in df.columns if c.endswith('alert')]
print(df[df[alert_cols[0]]][['chunk_key']])

Beyond the main plot, results.filter(period='analysis') limits the view to production chunks, and results.filter(metrics=['roc_auc']) limits it to one metric. These filters return a new results object, so you can chain them with .plot() or .to_df().

Another useful habit is looking at the reference period on the plot too. You want to see the reference chunks sitting comfortably inside the thresholds. If the reference chunks themselves already alert, the setup is wrong before you even look at production: the reference is too small, the chunks too small, or the reference data was not as clean as you thought.

Chunking: the setting that changes every result

Because every number in NannyML is per chunk, how you chunk decides what you can and cannot see. The documentation describes three ways to define chunks, and you pick one per calculator.

By size with chunk_size=5000: every chunk has that many rows. Use this when your traffic is steady and you want comparable statistical power per chunk. If the last group has fewer rows, it is kept as a smaller chunk by default, which is worth knowing because a tiny final chunk has a wide band.

By period with chunk_period, for example chunk_period='W' for weeks or 'M' for months. The documented aliases include 'D', 'W', 'M', 'Q' and 'A' among others. This needs a timestamp column, because the library has to know which period each row belongs to. Use this when your business thinks in calendar terms: a weekly report, a monthly review. The cost is that chunks have different sizes, so a quiet week gets a wider uncertainty band than a busy one.

By count with chunk_number, for example chunk_number=9: the data is divided into that many chunks of equal size. Use it when you care about having a fixed number of points on a chart.

If you pass none of them, NannyML falls back to a count-based split into ten chunks. That default is fine for a first look and a poor choice for monitoring, so set your own.

How do you choose a size? Think about two opposing forces. Larger chunks give narrower confidence bands and smoother lines, so alerts are more trustworthy, but they average over a longer time, which delays detection and can hide a short incident. Smaller chunks react faster but are noisier, and the sampling error can be wider than the effect you are looking for. A reasonable beginner rule: choose the smallest chunk for which the confidence band is narrow enough that a drop you care about would clearly leave it. For a fraud model that needs a daily answer but sees 800 transactions a day, chunking by day may give a band too wide to be useful, and a three-day or weekly chunk is the honest choice.

You also need enough chunks in the reference period. Thresholds come from the spread of the metric across reference chunks, and a spread estimated from two or three chunks is unreliable. If your reference set is small, that is a reason to use smaller chunks there, not a reason to shrink the band by hand.

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc'],
    chunk_period='M',
)

Passing both chunk_size and chunk_number is an error, since they describe incompatible plans. Pick one chunking style per calculator.

Detecting data drift, one column at a time

An alert on estimated performance tells you that something is wrong. The next question is why, and that is where drift detection comes in. Data drift means the distribution of an input column in production differs from the distribution in the reference data. The tool for beginners is the univariate drift calculator: "univariate" means it examines one column at a time.

Start by choosing which columns to examine. The sample data has many features, and for a first pass it is fine to look at a few. Selecting them from the frame's own columns avoids typing a name that does not exist:

non_features = ['timestamp', 'predicted_probability', 'prediction', 'employed']
feature_columns = [c for c in reference_df.columns if c not in non_features]
print(feature_columns)

Then build the calculator. It follows the same create, fit, calculate pattern:

drift_calculator = nml.UnivariateDriftCalculator(
    column_names=feature_columns[:4],
    timestamp_column_name='timestamp',
    continuous_methods=['kolmogorov_smirnov', 'jensen_shannon'],
    categorical_methods=['chi2', 'jensen_shannon'],
    chunk_size=5000,
)
drift_calculator.fit(reference_df)
drift = drift_calculator.calculate(analysis_df)

Two arguments deserve an explanation. NannyML needs to know whether each column is continuous (numbers on a scale, like age) or categorical (labels, like a marital status code). It infers this from the data type, and a numeric column that really holds categories can be declared with treat_as_categorical. The continuous_methods and categorical_methods lists choose the statistical measures to apply to each kind.

The measures answer "how different are these two distributions?" in different ways. For continuous columns the documentation lists kolmogorov_smirnov, jensen_shannon, wasserstein and hellinger. For categorical columns it lists chi2, jensen_shannon, l_infinity and hellinger. The Kolmogorov-Smirnov and chi-squared options are classic statistical tests that produce a statistic and a p-value style signal, and are sensitive to small changes in large samples. Jensen-Shannon distance, Wasserstein distance and Hellinger distance are distance measures, which give a magnitude of change instead of a yes or no. If you only remember one rule: with very large chunks, tests flag changes too small to matter, and a distance measure lets you judge size. Using Jensen-Shannon alongside a test is a sensible default.

Now look at the output:

print(drift.filter(period='analysis', methods=['jensen_shannon']).to_df(multilevel=False))
drift.filter(methods=['jensen_shannon']).plot(kind='drift').show()
drift.filter(methods=['jensen_shannon']).plot(kind='distribution').show()

The drift plot is a line chart of the measure per chunk with thresholds and alert markers, one panel per column. The distribution plot shows what the column actually looks like in each chunk: a joyplot for continuous columns and stacked bars for categorical ones. Always look at both. The line tells you when the measure crossed the line, and the distribution plot shows you what changed, such as an age group growing, a category appearing more often, or the middle of the range shifting. Without the second plot you know that a column drifted and not what that means for your business.

This is the part where discipline matters. Drift in a column is a suspect, not a verdict. Many columns will alert, and few of them are why the model got worse. The correct sequence is: look at the estimated performance first, and when it drops, use the drift results to find candidate causes. NannyML even offers rankers that order drifting columns by how many alerts they raised or by how strongly they correlate with the performance change, but those are mid-level tools. For now, the habit is enough: performance drops first, drift explains.

Realized performance: checking the estimate when labels arrive

An estimate that you never compare with reality is a belief, not a measurement. Eventually labels arrive, and NannyML lets you compute the real metric with PerformanceCalculator. It uses the same pattern, but it needs the true outcomes for the analysis rows.

In the sample data the targets live in a separate table. Join them to the analysis predictions using the index, which is how the two frames line up:

analysis_with_targets = analysis_df.merge(
    analysis_targets_df, left_index=True, right_index=True
)

calculator = nml.PerformanceCalculator(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    metrics=['roc_auc'],
    timestamp_column_name='timestamp',
    chunk_size=5000,
)
calculator.fit(reference_df)
realized = calculator.calculate(analysis_with_targets)

print(realized.filter(period='analysis').to_df())
realized.plot().show()

Use the same chunk_size as the estimator. If the chunks differ, the two curves describe different slices of time, and you cannot compare them point by point. Then plot the realized line next to the estimated one, or print the two DataFrames and look at the value columns for matching chunks.

What do you expect to see? When the CBPE assumptions hold, the estimate should track the realized line closely, and both should fall in the same chunks. That agreement is the evidence that the method works on your data. When they diverge, the divergence is itself information. Typical causes are a model whose probabilities are not calibrated, or a change in the relationship between inputs and outcomes that estimation cannot see because no labels are involved.

This comparison also builds trust with the people who read your reports. A team lead is far more likely to act on "the estimate said performance fell in March, and the labels that arrived in June confirmed it" than on a drift chart. In practice, you will run estimation continuously and realized calculation whenever a batch of labels arrives, and keep both.

Other metrics, other problem types, and the limits you must remember

So far we used ROC AUC. For binary classification, CBPE supports a family of metrics. The documentation lists, among others, roc_auc, accuracy, f1, precision, recall, specificity, average_precision, confusion_matrix and business_value. You pass several at once in a list and get one result per metric, so a single estimator can feed a whole dashboard. Choose metrics that match how the model is used: for a fraud model with rare positives, precision and recall tell you far more than accuracy, which stays high even for a useless model.

For problems that are not binary classification, NannyML changes the estimator. Regression models use DLE, Direct Loss Estimation, which trains an internal LightGBM model to predict your model's error, then aggregates the predicted errors into metrics like mean absolute error. Multiclass classification uses problem_type='classification_multiclass', where the probability columns are given as a mapping from class to column. Both appear in the Mid-level guide. The mental model you learned here, reference, fit, analysis, chunks and thresholds, carries over unchanged.

Now the limits. They matter more than any feature, because a monitoring tool that is trusted beyond its limits is more dangerous than none.

Estimation assumes calibrated probabilities. If your model says 0.9 when the true rate is 0.7, the estimate inherits that error. Check calibration on your reference set before trusting CBPE, and consider calibrating the model's scores first.

Estimation assumes no concept drift. Concept drift means the rule linking inputs to outcomes changes, for example, because customer behaviour changes in response to a new policy. Estimation looks only at inputs and predictions, so it cannot see this, and the estimate may stay flat while real performance collapses. This is why realized performance, once labels arrive, remains essential.

Covariate shift can push the model outside what it knows. If production inputs land in regions that the reference never covered, the calibration learned there no longer applies. Drift detection helps you notice this.

Estimates are early warnings, not proof. Treat an alert as a prompt to investigate. Confirm with labels, with a sample you review by hand, or with a business metric.

Configuration you will touch, and the errors you will meet

Most of what you configure lives in the constructor arguments you have already seen. A short map helps you remember where to look when something is off.

If the results look wrong, the first suspects are the column names. y_pred_proba, y_pred, y_true and timestamp_column_name must match the DataFrame exactly, including case. A typo gives you an error about a missing column. Second is the reference data: is it long enough, and does it really represent the good period? Third is the chunking discussed above. Only after those three should you doubt the method.

The thresholds can also be changed. By default they come from the reference data, but you can supply your own threshold objects from nannyml.thresholds, such as a constant lower bound for a metric, which suits cases where the business has a fixed minimum, for example "ROC AUC below 0.7 is unacceptable". That belongs to the Mid-level guide, but it is useful to know that the automatic thresholds are a default and not a law.

Here are the errors you are most likely to meet, how to read them, and what to do.

• Missing native library on import. A message such as Library not loaded: libomp.dylib on macOS or libgomp.so.1: cannot open shared object file on Linux means LightGBM's OpenMP runtime is missing. Install libomp with Homebrew or libgomp1 with apt.
• No matching distribution on a new Python. If pip reports no matching distribution for nannyml on Python 3.13, the interpreter is outside the supported range. Create a 3.12 environment.
• Not fitted. An error saying the calculator was not fitted means you called calculate() or estimate() before fit(). Always fit on the reference data first.
• Invalid arguments. NannyML raises an invalid-arguments error for conflicting or missing settings, for example giving both chunk_size and chunk_number, or omitting y_pred_proba for CBPE. Read the message: it names the argument.
• Missing columns. A column name that is not in the DataFrame produces an error that lists the missing names. Print df.columns and compare.
• Period chunking without timestamps. Using chunk_period needs the timestamp column, as described earlier.
• Saving a figure as an image. Calling write_image on a Plotly figure needs pip install kaleido. Use write_html if you do not want another dependency.
• Pydantic import errors. If something like cannot import name 'BaseSettings' from 'pydantic' appears, another package in the environment pinned an incompatible Pydantic. A fresh virtual environment is the cleanest fix.
• SSL errors from pip. CERTIFICATE_VERIFY_FAILED behind a TLS-inspecting proxy needs the proxy's CA certificate, never disabled verification.

There are also errors that are not exceptions but wrong-looking results. An alert on nearly every reference chunk means the reference is too small or the chunks too small. No alerts ever means the chunks may be so large that they smooth away all variation. An estimate far from the realized value means calibration or concept drift.

Where NannyML stands today: a note on maintenance

A responsible guide tells you what you are building on. In June 2025, the data quality company Soda announced that it had acquired NannyML. Soda stated that the open-source library would remain open and maintained, and it mentioned a future 1.0 release. The latest release on PyPI, 0.13.1, is from July 2025. At the time of writing, a community issue on the project's GitHub repository asks about the pace of activity, and no newer release has appeared.

What does that mean for you as a beginner? It is not a reason to avoid the library. The version you install works, the ideas are sound, and the code is Apache 2.0 licensed. It is a reason to be careful in three ways. First, pin the version (nannyml==0.13.1) in your requirements file so that a surprise upgrade cannot change behaviour. Second, pin its neighbours, such as pandas, NumPy and scikit-learn, with a lock file, because the library's constraints on them were last updated in 2025 and newer releases of them may not fit. Third, stay on Python 3.9 to 3.12. Check the repository's releases page before starting a long project, in case that situation has changed.

The same maintenance question is a good habit for every tool in this catalogue. Look at the last release date, the open issues, and who owns the project before you commit a team to it. Learning what NannyML does well, especially label-free performance estimation, remains valuable even if you later move to another tool, because the ideas of reference, chunks, thresholds and sampling error exist in all of them. For experiment tracking that pairs well with monitoring results, see /student-guides/mlflow.

Preparing your own data for NannyML

The sample datasets are convenient, but the day will come when you point NannyML at your own model. Most of the trouble at that point has nothing to do with the library. It comes from the shape of the data, so it is worth knowing what NannyML expects before you start.

You need two tables, each as a pandas DataFrame. The reference table has one row per prediction from a period you trust. It holds the input features, the model's predicted probability, the predicted class, the true outcome and a timestamp. The analysis table has the same columns for the period you want to watch, usually without the true outcome. Both tables must use identical column names and compatible types. If a feature is an integer in the reference and a string in production, you will get either an error or a nonsense comparison, so fix the types in your own loading code and not by hand in a notebook.

Where do the predictions come from? The cleanest answer is that your serving system logs every prediction together with its inputs and a timestamp, to a table or a set of files. If your model does not log, start doing it today. A model that does not record what it saw and what it said cannot be monitored by anyone, with any tool. The predicted probability matters most for NannyML: store the probability, not only the final class. A model that outputs only a yes or no label gives CBPE nothing to work with.

Think about the reference period with the same care as the model itself. A common and sound choice is the held-out test set the model was evaluated on, because you know how the model performed there and it has labels. Another is the first weeks of production, once their labels have arrived. A poor choice is a period that includes a known incident, a holiday that will not repeat, or a data pipeline bug, because the thresholds will learn that abnormal behaviour as normal. Also make sure that the reference contains enough rows to form several chunks. A useful rule of thumb is at least ten chunks at the chunk size you plan to use, and more is better, since the thresholds come from the spread across those chunks.

Time matters too. Keep the timestamps in a real datetime type and in one time zone. A model serving customers in Riyadh, Cairo and Dubai across several time zones can end up with rows from the same day falling into different calendar chunks if timestamps are a mixture of local times and UTC. Convert to UTC when you log, and convert for display only.

Finally, think about privacy before the data leaves its home. The tables hold copies of production inputs, which may contain personal data. Many employers in the Gulf and Egypt require that such data stays in a given region or inside a given network. Run the monitoring job where the data lives, and keep the outputs, which can reveal distributions of sensitive columns, under the same access rules as the data itself.

A quick checklist before running anything: print reference_df.dtypes and analysis_df.dtypes and confirm they match, confirm the timestamp column is a datetime, confirm the probability column holds values between zero and one, and confirm the reference has labels while the analysis does not need them. Five minutes here saves the confusing errors that appear later as strange charts.

Writing up a finding that someone will act on

A chart is not a result until a person understands it and does something. NannyML gives you numbers and plots, and your value as an engineer is turning them into a short message that a product owner or a risk manager can use. A good habit is to write every finding with the same five parts.

First, what happened, in one sentence and in business words: "Estimated ROC AUC fell below its lower threshold in the chunks starting in March." Avoid describing the library; describe the model.

Second, how sure you are. State the chunk size, mention whether the confidence band is wide, and say whether the effect is a single chunk or several in a row. One alerting chunk out of forty is often noise within three standard deviations, since by construction a small share of healthy chunks will cross the line. Several consecutive chunks drifting downward is a pattern.

Third, what changed in the data. Use the drift results and the distribution plots: "The share of applicants in the youngest age group doubled, and the income column shifted upward." This is the explanation part, and you give it as a candidate cause and not as a proven cause.

Fourth, how you checked. If labels have arrived for any part of the period, say what the realized metric shows. If not, say when labels are expected, and what other check you used, such as a manual review of a sample of predictions.

Fifth, what you recommend. Retrain on newer data, investigate an upstream data change, add a threshold in a downstream rule, or keep watching. Be ready to say what evidence would change your recommendation.

Writing in this shape protects you in two ways. It keeps you honest about uncertainty, which matters because the estimate rests on assumptions you cannot fully verify. And it makes your work easy to audit later: when the labels arrive and settle the question, you can check your own message against the truth and learn how reliable your monitor is on this model. Teams that do this for a few months tend to know exactly how much to trust each alert, and that knowledge is worth more than any tuning of thresholds.

Putting it all together: a small end-to-end project

Let us finish with one script that does everything in this guide, saves its outputs, and is safe to rerun. You will estimate performance, compute realized performance, run drift detection on a few columns, and write the findings to files you can open later.

import nannyml as nml

# 1. Load the sample data: reference (with labels), analysis (without), and the late-arriving targets.
reference_df, analysis_df, analysis_targets_df = nml.load_us_census_ma_employment_data()

CHUNK = 5000
non_features = ['timestamp', 'predicted_probability', 'prediction', 'employed']
features = [c for c in reference_df.columns if c not in non_features][:4]

# 2. Estimate performance without labels.
estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc', 'f1'],
    chunk_size=CHUNK,
)
estimator.fit(reference_df)
estimated = estimator.estimate(analysis_df)

# 3. Realized performance, using the labels that "arrived later".
full = analysis_df.merge(analysis_targets_df, left_index=True, right_index=True)
calculator = nml.PerformanceCalculator(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    metrics=['roc_auc', 'f1'],
    timestamp_column_name='timestamp',
    chunk_size=CHUNK,
)
calculator.fit(reference_df)
realized = calculator.calculate(full)

# 4. Drift on a few columns, to explain any drop.
drift_calc = nml.UnivariateDriftCalculator(
    column_names=features,
    timestamp_column_name='timestamp',
    continuous_methods=['jensen_shannon'],
    categorical_methods=['jensen_shannon'],
    chunk_size=CHUNK,
)
drift_calc.fit(reference_df)
drift = drift_calc.calculate(analysis_df)

# 5. Save tables and interactive charts.
estimated.filter(period='analysis').to_df(multilevel=False).to_csv('estimated.csv', index=False)
realized.filter(period='analysis').to_df(multilevel=False).to_csv('realized.csv', index=False)
drift.filter(period='analysis').to_df(multilevel=False).to_csv('drift.csv', index=False)

estimated.plot().write_html('estimated.html')
realized.plot().write_html('realized.html')
drift.plot(kind='drift').write_html('drift.html')
drift.plot(kind='distribution').write_html('distribution.html')

print('Done: open estimated.html, realized.html, drift.html and distribution.html')

Run it with python monitor.py and then work through the outputs in the order a real investigation would follow. Open estimated.html first: does the estimate leave its thresholds, and in which chunk? Open realized.html and ask whether the real numbers agree. If performance dropped, open drift.html and distribution.html and look for columns whose alerts start in the same chunk. Write a short note with the structure that a stakeholder would want: what dropped, since when, which inputs changed, and what you would check next.

Notice how the structure mirrors the practice rather than the library. The estimate raises the question, the realized numbers test it, the drift results suggest a cause, and a human decides what to do. NannyML does the statistics; the judgement stays with you.

To make it yours, replace the sample loader with your own pandas DataFrames. The reference frame needs the features, the predicted probabilities, the predicted classes, the true labels and a timestamp. The analysis frame needs the same columns except the labels. If the model you monitor is hosted in a cloud region such as the Gulf or Egypt for data-residency reasons, run this job inside the same environment as the data. The inputs to a monitoring job are copies of your production inputs, so they carry the same privacy obligations.

What you can now do, and what comes next

You now understand the problem NannyML was built to solve: models degrade silently, and labels arrive late. You can install the library on a supported Python and diagnose the usual native-library problems. You know the four nouns, reference, analysis, chunks and thresholds, and you can explain why an alert is relative to the reference rather than an absolute verdict. You can estimate classification performance with CBPE, read the value, confidence band, thresholds and alert columns, choose a sensible chunking style and size, run univariate drift detection and read its line and distribution plots, compute realized performance, and compare it with your estimate. You also know the assumptions that can make the estimate wrong, and the maintenance situation of the library, so you can pin versions and decide with open eyes.

The Mid-level guide takes you into the machinery. It covers custom chunkers and thresholds, regression with DLE, multiclass models, multivariate drift with PCA reconstruction and domain classifiers, data quality checks, and the rankers that link drift to performance impact. The Senior guide covers running monitoring as a scheduled production job, persisting fitted calculators, storing results in a database, scaling limits and governance.

Natural neighbours in this catalogue: if you want to log the monitor's results next to your experiments, read /student-guides/mlflow. If you need to schedule the monitoring script to run every day, /student-guides/airflow and /student-guides/prefect are the usual choices. If you want to package the job so it behaves the same everywhere, /student-guides/docker is the place to start. And to validate the data before it ever reaches the model, see /student-guides/great-expectations.

Sources

• NannyML documentation
• Quickstart
• Estimating performance for binary classification (CBPE)
• Univariate drift detection
• Detecting data drift overview
• Chunking
• NannyML on PyPI
• NannyML on GitHub
• NannyML releases
• LightGBM installation guide
• Soda announces the NannyML acquisition

This page prepares you to talk about NannyML in an interview for a junior MLOps or machine learning engineering role. It assumes you have read the beginner guide. Interviewers rarely care whether you remember every argument. They care whether you understand the problem, can explain the model in plain words, and know where the method breaks. Practise saying the answers aloud, because a correct answer that you cannot say smoothly sounds like a guess.

The thirty-second answer

What is NannyML? It is an open-source Python library for monitoring machine learning models after deployment. Its distinctive feature is that it estimates a model's performance without needing the true labels, using the model's own predicted probabilities, and it then uses data drift detection to explain why performance changed.

If the interviewer wants one more sentence: the point is that labels often arrive weeks or months after predictions, so waiting for them means finding out about a failure too late, and NannyML gives an early estimate that you later confirm against the real numbers.

Notice what this answer does. It states the problem (late labels), the tool's approach (estimate, then explain) and the honest follow-up (confirm later). Candidates who only say "it detects drift" miss what makes the library different, and the interviewer will often probe exactly there.

The model you should be able to draw

You may be asked to explain the workflow on a whiteboard. Draw four boxes in a row and say one sentence about each.

Reference data is a period where the model is known to behave well, with true labels. The library fits its baselines and thresholds on it. Analysis data is the production data you want to watch, with predictions but possibly without labels. Chunks are the groups of rows, defined by size, count or calendar period, over which each metric is calculated, so the output is a time series rather than a single number. Thresholds and alerts come from the spread of each metric across the reference chunks, by default three standard deviations around the mean, and a chunk whose value falls outside is flagged.

Then say the pattern that every calculator follows: create it, call fit() on reference data, then call estimate() or calculate() on analysis data, and receive a results object that you can filter, convert to a DataFrame, or plot. If you say this without hesitation, you have shown you understand the library's shape, which is worth more than any single parameter.

A strong extra detail is the order of use. You look at estimated performance first, and you use drift results to explain a drop, not as the primary alarm. Many candidates reverse this, and explaining why the order matters sets you apart. Drift is common and often harmless, so alerting on it alone produces noise that teams learn to ignore.

The commands and calls to have ready

You will not be asked to type from memory in most interviews, but you may be asked to write pseudo-code or read a snippet. Be comfortable with these four.

pip install nannyml
python -c "import nannyml; print(nannyml.__version__)"

Be ready to say that the current release is 0.13.1, that it supports Python 3.9 to 3.12, and that LightGBM needs an OpenMP runtime on some systems.

import nannyml as nml

reference_df, analysis_df, analysis_targets_df = nml.load_us_census_ma_employment_data()

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc'],
    chunk_size=5000,
)
estimator.fit(reference_df)
results = estimator.estimate(analysis_df)
results.filter(period='analysis').to_df()
results.plot().show()

Be able to say what each argument means. problem_type selects the family of metrics. y_pred_proba is the predicted-probability column and y_pred the predicted class. y_true is needed in the reference data. The timestamp column dates the chunks. chunk_size is the rows per chunk.

drift_calculator = nml.UnivariateDriftCalculator(
    column_names=['some_feature'],
    timestamp_column_name='timestamp',
    continuous_methods=['kolmogorov_smirnov', 'jensen_shannon'],
    categorical_methods=['chi2', 'jensen_shannon'],
    chunk_size=5000,
)
drift_calculator.fit(reference_df)
drift = drift_calculator.calculate(analysis_df)
drift.plot(kind='drift').show()
drift.plot(kind='distribution').show()

And for realized performance once labels arrive, nml.PerformanceCalculator with the same arguments as the estimator, fitted on reference data and calculated on analysis data joined with its targets. Point out that you keep the chunking identical so that estimated and realized curves compare like with like.

The classic traps and why they happen

Interviewers like questions that begin "what could go wrong". Know these, and know the reason behind each, because the reason is what they are listening for.

The estimate is wrong because the probabilities are not calibrated. CBPE treats a predicted probability of 0.8 as meaning "right about 80 percent of the time". If the model is overconfident, as many boosted trees and neural networks are, the arithmetic built on those probabilities is off. The fix is to check and improve calibration before trusting the estimate.

The estimate stays flat while the real performance drops, because of concept drift. Concept drift is a change in the relationship between inputs and outcomes. The estimator only sees inputs and predictions, so it cannot notice a change that only affects labels. This is why you keep computing realized performance when labels arrive.

Alerts fire on healthy data, because the reference is too small or the chunks are too small. Thresholds come from the spread across reference chunks. A short reference gives few chunks, a noisy spread, and unreliable thresholds. Small chunks give large sampling error, so the metric jumps around for statistical reasons alone.

Nothing ever alerts, because chunks are too large. Large chunks average away short incidents, and the confidence band becomes so tight and the line so smooth that real events are diluted. A two-day outage inside a three-month chunk is invisible.

Period chunking fails without a timestamp. If you ask for weekly chunks and have not told the library which column holds the dates, it cannot assign rows to weeks.

The install fails on a new Python. Version 0.13.1 does not declare support for Python 3.13. Use 3.12 and pin the version.

A drift alert is treated as a performance problem. Drift in a column the model barely uses does nothing. Drift is a candidate explanation, and you must check it against performance.

If you can attach a cause to each symptom, you sound like someone who has used the tool rather than read about it.

Common interview questions

Q: Why can't you just calculate accuracy in production? Because accuracy needs the true label for each prediction, and in many applications the label arrives late or never. A loan model learns whether a borrower defaults months later. A recommendation model may never learn whether a recommendation was the best choice. Until labels arrive, you are blind, and NannyML narrows that blindness by estimating the metric from the predicted probabilities.

Q: How does CBPE estimate performance without labels? If the model is well calibrated, a predicted probability equals the chance the prediction is correct. So for each row you can compute the expected contribution to the confusion matrix, summing the probabilities across a chunk gives expected counts of true and false positives and negatives, and standard metrics follow from those counts. It is an expected-value calculation, not a guess. The method depends on calibration and on there being no concept drift.

Q: What is a chunk, and why does NannyML use them? A chunk is a group of rows, defined by a number of rows, a number of groups or a calendar period. Every metric is computed per chunk, producing a time series. This is needed because a single number for the whole production period hides when things changed, and because the library needs several chunks in the reference period to learn how much each metric normally varies.

Q: How do you choose a chunk size? There is a trade-off. Bigger chunks reduce sampling error and make alerts more trustworthy but delay detection and blur short events. Smaller chunks react faster but are noisy. I would start from how often the business reviews results, check how wide the confidence band is at that size, and pick the smallest size where a drop I care about would clearly leave the band. I also make sure the reference period has enough chunks to give stable thresholds.

Q: What is sampling error? It is the statistical noise in a metric computed from a finite number of rows. NannyML estimates it per chunk and draws a confidence band of plus or minus three sampling errors around the value. Movements inside the band are not evidence of change.

Q: What is the difference between the confidence band and the thresholds? The band reflects uncertainty about one chunk's value, due to limited rows. The thresholds reflect the normal range of the metric learned from the reference chunks. An alert depends on the value crossing the thresholds. Mixing them up is a common mistake.

Q: Which drift methods does NannyML offer for a single column? For continuous columns, Kolmogorov-Smirnov, Jensen-Shannon, Wasserstein and Hellinger. For categorical columns, chi-squared, Jensen-Shannon, L-infinity and Hellinger. I would use a distance measure such as Jensen-Shannon next to a test, because with large samples tests flag tiny changes, and a distance tells you how big the change is.

Q: If a feature drifts, does the model get worse? Not necessarily. It depends on how much the model relies on that feature and whether the new values stay within the range it handles well. That is why the workflow starts from estimated performance and uses drift to explain a drop. Alerting on drift alone creates noise.

Q: What would you do if the estimated performance drops? First I would check that it is not an artefact: is the chunk small, is the confidence band wide, did the data pipeline change? Then I would look at drift results for the same period and at the distribution plots to see what changed. If labels are available for any part of that period, I would compute realized performance. I would tell the model owner what I found and with what confidence, and suggest a decision such as retraining or investigating an upstream data change.

Q: What are the limits of label-free estimation? It assumes calibrated probabilities and no concept drift. It cannot detect label-only changes. Under heavy covariate shift into regions the reference never covered, the calibration may not hold. So I treat it as an early warning and confirm with realized metrics.

Q: How is it different from a drift-only tool? A drift tool tells you that data changed. NannyML tries to tell you whether the model's performance probably changed, then use drift to explain. Both views are useful, and for a comparison with a report-centred tool, I would look at Evidently.

Q: Would you use it in production today? I would, with care. The library is Apache 2.0 licensed and the current release works, but it was acquired by Soda in 2025, and the last release on PyPI is from July 2025. I would pin the version and its dependencies, stay on a supported Python version, keep an eye on the repository for a 1.0 release, and make sure the monitoring logic is simple enough to migrate if I had to.

Q: What does the output look like, and how would you act on it? The output is a results object, from which I can produce a DataFrame with a value, sampling error, thresholds and an alert flag per chunk, and Plotly charts. The library does not send notifications on its own in the open-source version, so I would add a small step that reads the alert column and posts to the team's channel or opens a ticket.

Q: How would you schedule it? It is a batch library. I would run it from a scheduler such as Airflow or Prefect, reading recent predictions from storage, calling estimate() and calculate(), and saving the results. I would write the code so that it can run inside a container for consistency.

Q: Where does data residency come in? Monitoring jobs read copies of production inputs and predictions, which can contain personal data. In a Gulf or Egyptian setting with local data rules, I would run the job inside the same region and network as the data, restrict who can read the result tables, and avoid shipping raw rows to external services.

Questions to ask back

At the end of an interview you can show maturity by asking about the team's reality. Ask how they find out today that a model has degraded. Ask how long labels take to arrive for their main models. Ask whether they have a reference dataset that is versioned, because without one a monitor has nothing to compare to. These questions show that you understand that the hard part of monitoring is not the library but the data and the process around it.

How strong and weak answers differ

A weak answer lists features: "it detects drift and shows charts". It does not mention that labels are late, does not name the assumptions, and treats every alert as a failure. When asked about chunks, it says "the data is split".

A strong answer leads with the problem, uses the words reference, analysis, chunk, sampling error and threshold correctly, explains the order of use (performance first, drift to explain), names at least two limits of the method, and proposes confirming with realized metrics. It stays honest about the project's maintenance status. It also admits what it has not used, rather than bluffing, and describes how it would find out.

A useful rehearsal technique is to answer each question in two lengths. A fifteen-second version for when the interviewer is moving fast, and a ninety-second version with an example for when they lean in. Prepare one example from your own practice project, for instance the census data from the quickstart, where you saw the estimate and the realized values diverge or agree.

Sixty-second self-test

Without looking back, answer these aloud. If you hesitate on more than two, reread the matching section of the beginner guide.

1. In two sentences, what is NannyML and what makes it different from a drift tool?
2. Name the four nouns of the mental model.
3. What are the three calls every calculator follows, and on which data does each run?
4. What does CBPE stand for, and which two assumptions does it need?
5. What is the difference between sampling error and a threshold?
6. Which three ways can you define chunks, and what does each need?
7. Name two drift methods for continuous columns and two for categorical ones.
8. Why should you look at estimated performance before drift?
9. Name three reasons the estimate might be wrong or alerts might be misleading.
10. Which Python versions does the current release support, and what should you do about the project's maintenance situation?

Check yourself: (1) late labels, estimate then explain; (2) reference, analysis, chunks, thresholds and alerts; (3) create, fit() on reference, estimate() or calculate() on analysis; (4) Confidence-Based Performance Estimation, calibrated probabilities and no concept drift; (5) noise within one chunk versus the normal range from reference; (6) by size, by count, by period (which needs timestamps); (7) for example Kolmogorov-Smirnov and Jensen-Shannon, chi-squared and Jensen-Shannon; (8) drift is common and often harmless, performance is what matters; (9) uncalibrated probabilities, concept drift, chunks or reference too small or too large; (10) 3.9 to 3.12, and pin versions and watch the repository.

When you can do all ten smoothly, move on to the Mid-level interview page for scenario questions and trade-offs. If you want the neighbouring tools in your answers, see /student-guides/evidently and /student-guides/alibi-detect.

This page collects the mistakes beginners actually make with NannyML, in the same shape each time: the symptom you see, the cause underneath, and the fix. After the mistakes come the habits worth adopting on day one, and a checklist you can paste into your own project notes. Read the mistakes before you hit them. Each takes minutes to fix once you know what it is and an afternoon if you do not.

Mistakes beginners make

1. The import fails with a missing library message

Symptom. import nannyml raises an error such as Library not loaded: @rpath/libomp.dylib on a Mac, or libgomp.so.1: cannot open shared object file on Linux.

Cause. NannyML depends on LightGBM, and LightGBM needs an OpenMP runtime from the operating system. Python packages cannot always bring it with them. On a slim container image the library is simply absent.

Fix. On macOS run brew install libomp. On Debian or Ubuntu run sudo apt-get install -y libgomp1. Then start a fresh Python process. If you are building a Docker image, add the package to the image instead of fixing it by hand on one machine, so that the next person does not repeat the discovery.

2. pip says there is no matching distribution

Symptom. pip install nannyml ends with No matching distribution found, or a line saying versions were ignored because they require a different Python.

Cause. The current release, 0.13.1, declares support for Python 3.9 to 3.12. If your interpreter is 3.13 or newer, pip filters out every release as incompatible. Older interpreters have the same problem in reverse: Python 3.8 was dropped in 0.13.0, and 3.7 in 0.12.

Fix. Check with python --version, then create a virtual environment from a supported interpreter, for example python3.12 -m venv .venv. Do not try to force the install on an unsupported Python. You might succeed in installing and then fail later in a dependency, which is a harder problem to diagnose.

3. The install works but the wrong version is used

Symptom. You installed NannyML, but nannyml.__version__ shows an old number, or an API from a tutorial does not exist.

Cause. Two environments are fighting. Often the pip on your path belongs to a different Python from the one running your notebook. A second cause is following an old blog post: older tutorials use names that were later renamed, such as the old statistical drift calculator that is now called UnivariateDriftCalculator.

Fix. Install with python -m pip install nannyml so that the interpreter and the installer match, and print the version from inside the same session where you run your code. When you copy code from a tutorial, check its date and compare it with the current documentation. If a class does not exist, look for its current name before assuming the library is broken.

4. pip fails with a certificate error

Symptom. CERTIFICATE_VERIFY_FAILED or an SSL error when installing packages, usually at work or on a university network.

Cause. The network inspects encrypted traffic and re-signs it with its own certificate authority, which Python does not trust by default.

Fix. Get the organisation's root certificate and point the tools at it, using the PIP_CERT environment variable for pip, and REQUESTS_CA_BUNDLE or SSL_CERT_FILE for libraries that use them. Never switch verification off. A disabled check protects nothing and teaches the habit of ignoring a warning that can be real somewhere else.

5. Calling estimate or calculate before fit

Symptom. An exception saying the calculator was not fitted and asking you to call fit() first.

Cause. Estimators and calculators learn from the reference data before they can process analysis data. Thresholds, baselines and internal models are created in fit(). Skipping it leaves nothing to compare against.

Fix. Always follow the three-step rhythm: construct, fit(reference_df), then estimate(analysis_df) or calculate(analysis_df). If you restructure notebook cells and run them out of order, rerun from the top. Putting the three calls next to each other in one cell prevents the problem.

6. Column name typos and missing columns

Symptom. An error complaining about missing columns, or about an argument that is required.

Cause. y_pred_proba, y_pred, y_true and timestamp_column_name take the names of columns as strings, and they must match your DataFrame exactly, including case and underscores. The reference DataFrame also needs y_true, while the analysis one usually does not.

Fix. Print reference_df.columns.tolist() and copy the names from the output. Write a tiny helper that checks that every name you pass exists in both frames before you create the calculator. For drift calculators, select feature columns by excluding the non-feature columns, rather than typing each name.

7. Using chunk_period without a timestamp

Symptom. An error about the timestamp when you pass chunk_period='W' or 'M'.

Cause. To group rows into weeks or months, the library must know the date of each row. Without timestamp_column_name it has no way to do it.

Fix. Pass the timestamp column name, and make sure the column holds real datetimes rather than strings that merely look like dates. If you cannot provide one, use chunk_size or chunk_number.

8. Giving two incompatible chunking settings

Symptom. An invalid-arguments error when you pass more than one of chunk_size, chunk_number and chunk_period.

Cause. They are three different plans for cutting the same data. The library cannot follow two at once, so it refuses.

Fix. Choose one. If you are unsure, start with chunk_size, because it is the easiest to reason about, and revisit when you know the rhythm of your data.

9. Alerts on almost every chunk

Symptom. The chart is a wall of red, including on the reference period.

Cause. Thresholds come from how much a metric varies across reference chunks, three standard deviations around the mean by default. If the reference has very few chunks, or the chunks are tiny, the spread is poorly estimated, and the real analysis data does not fit. Another cause is a reference period that is not truly representative, for instance one that contains a past incident.

Fix. Increase the reference size or reduce the chunk size so that the reference contains enough chunks, increase the analysis chunk size so that sampling error shrinks, and review whether the reference was really a healthy period. Then plot the reference chunks alone and confirm they sit inside the thresholds before you look at anything else.

10. Alerts never appear

Symptom. Every chunk is inside its thresholds, even when you know something broke.

Cause. Chunks that are too large smooth away short events, and thresholds that are too wide hide moderate changes. Or the metric you picked does not react to the kind of failure you have, for instance accuracy on a dataset with very rare positives.

Fix. Try a smaller chunk size and see whether the picture changes. Add metrics suited to the problem, such as precision, recall and F1 for imbalanced data. Test your monitor by injecting a known problem into a copy of the data, for example shifting a feature or scrambling some predictions, and confirming that the monitor notices. A monitor you have never seen fire is not yet trustworthy.

11. Trusting the estimate blindly

Symptom. The estimated performance looks fine, but the business reports worse results.

Cause. The estimate assumes calibrated probabilities and no concept drift. If either assumption fails, the estimate can look calm while the model is wrong.

Fix. Check calibration of the model's scores on the reference data. Compute realized performance whenever labels arrive and compare it with the estimate, using identical chunking. If the two diverge, believe the realized numbers and investigate the assumptions. Treat the estimate as an early warning.

12. Treating every drift alert as an emergency

Symptom. Dozens of columns alert, the team panics, and then nothing turns out to be wrong.

Cause. Drift measures whether a distribution changed, not whether it matters. With large chunks, statistical tests detect changes too small to matter. And a model may barely use some drifting features.

Fix. Make performance the main alarm and use drift to explain it. Look at the distribution plot of a drifting column to see how it changed. Use a distance measure such as Jensen-Shannon next to a test to judge magnitude. Later, learn the rankers that order columns by their link to performance.

13. Comparing estimated and realized curves with different chunks

Symptom. The estimated and realized lines do not seem to describe the same thing, and you cannot match points.

Cause. The two calculators were created with different chunk settings, so chunk boundaries do not line up.

Fix. Define the chunk setting once, in a variable, and use it for every calculator in the script. Keep the same reference data too.

14. Wrong types for categorical and continuous columns

Symptom. A column of category codes is treated as a number, and the drift result looks strange, or a text column raises an error.

Cause. The drift calculator infers the type from the data. A column of integers that actually means categories, such as a status code, looks continuous.

Fix. Declare it with treat_as_categorical. Look at the distribution plot to check that it makes sense, bars for categories and a joyplot for numbers.

15. The plot does not appear, or saving a picture fails

Symptom. .show() does nothing in a plain script, or write_image raises an error about a missing engine.

Cause. The plots are Plotly figures. In a script without a browser they have nowhere to display, and exporting static images needs the separate kaleido package.

Fix. Save interactive charts with .write_html('name.html') and open the file, or install kaleido if you need PNG files. In a notebook, .show() works inline.

16. Dependency clashes in a shared environment

Symptom. An import error mentioning Pydantic, or pip complaining about incompatible versions after installing another library.

Cause. NannyML pins ranges for packages such as Pydantic, pandas and scikit-learn, and another tool in the same environment may want something different. A big environment gradually collects such conflicts.

Fix. Give each project its own virtual environment. Save the exact versions with pip freeze > requirements.txt once your script works. If you must share an environment, install NannyML first, and add the others after, checking what pip changes.

17. Putting raw production data in public places

Symptom. A monitoring notebook with real customer rows is pushed to a public repository or pasted into a chat.

Cause. Monitoring inputs are copies of production data, and notebooks save their output, including printed tables.

Fix. Practise on the sample datasets and synthetic data. For real data, clear notebook outputs before committing, keep data outside the repository, and follow the data-residency and privacy rules of your employer, which in many Gulf and Egyptian organisations require data to stay in region.

Habits worth adopting on day one

Pin the version. Write nannyml==0.13.1 in your requirements file. The project's release pace has slowed since the 2025 acquisition by Soda, so an unexpected upgrade is unlikely, but the libraries around it keep moving, and a pinned set makes your results reproducible.

Use a virtual environment for every project. It costs ten seconds and saves hours. Name it the same way everywhere, for example .venv, and add it to .gitignore.

Write the chunk setting once. Keep it in a constant at the top of the script, and reference it everywhere. When you later change it, you change one line.

Plot the reference period every time. Before looking at production, confirm that reference chunks sit inside the thresholds. It is the quickest test of whether the setup is sane.

Always run the realized comparison when labels arrive. Build the habit of asking whether the estimate was right. Keep a small table of estimated and realized values for the same chunks, and look at the gap over time.

Look at the distribution, not just the drift score. One minute with a joyplot or a stacked bar chart tells you more than a p-value.

Test your monitor with an injected problem. Take a copy of the data, change a feature or damage predictions on purpose, and confirm that an alert fires in the right chunk. This is the only way to know that the monitor works.

Save outputs as files. Write the result DataFrames to CSV or Parquet and the charts to HTML. Files are easy to share with colleagues who did not run your notebook and easy to compare between runs.

Record the context. Write down the reference period, chunk size, library version, metrics and Python version next to the results. In three months you will not remember which settings produced a chart, and a monitor whose settings are unknown cannot be trusted.

Keep notebooks and scripts separate. Explore in a notebook, then move the working logic into a script such as monitor.py that you can run from a scheduler. Scripts are easier to review, test and rerun than notebooks with hidden state.

Read the docs page for each calculator once. Every calculator has a tutorial page with the arguments and an example. Spending ten minutes there is cheaper than guessing. When the documentation and a blog post disagree, the documentation for your installed version wins.

Learn the neighbours. A monitoring job lives inside a larger pipeline. Skim /student-guides/mlflow for logging the monitor's outputs, /student-guides/evidently for a report-oriented alternative, /student-guides/great-expectations for validating data before it reaches the model, and /student-guides/docker for packaging the job.

A worked debugging story

To make the habits concrete, here is a typical afternoon. A student builds a monitor for a churn model on a telecom dataset. The first run shows alerts on every chunk, including the reference period. The temptation is to widen the thresholds by hand. Instead the student applies the checklist. First, the reference plot shows the problem: the reference has only 3,000 rows and the chunk size is 1,000, so there are three reference chunks. The thresholds are built from three data points. Second, the student gets a larger reference set from an earlier month and reduces the chunk size in the reference to produce a dozen chunks. Now the reference sits inside the thresholds. Third, the analysis plot shows one genuine alert in the final week. Fourth, the drift results for that week show a categorical column, the tariff plan, with a new category that never appeared in the reference. The distribution plot confirms it: a plan launched that week. Fifth, labels arrive a month later, and realized performance confirms the drop, with a gap from the estimate small enough to trust the method.

Notice that the student never tuned anything to make the chart look better. Every change was justified by a reason about the data: too few reference chunks, a new category, a confirmed drop. That is the mindset to copy. A monitor is an instrument, and you calibrate instruments against reality, not against your hopes.

A short checklist

Copy this into your project notes and tick it before you call a monitor finished.

• Python is 3.9 to 3.12, in a dedicated virtual environment, and nannyml.__version__ prints the version you expect.
• The version of NannyML and its main dependencies are pinned in a requirements file.
• LightGBM's OpenMP runtime is installed on the operating system, and the Docker image, if any, includes it.
• Column names for predictions, probabilities, labels and timestamps were copied from df.columns.
• The reference data is a period you trust, has labels, and yields enough chunks.
• Exactly one chunking setting is used, defined once, shared by every calculator.
• The reference chunks sit inside the thresholds on the plot.
• You can say whether the model's probabilities are calibrated.
• Estimated performance is the primary alarm, and drift results are used to explain it.
• Realized performance is computed when labels arrive and compared with the estimate with identical chunking.
• A deliberately injected problem triggers an alert in the right chunk.
• Results, charts, settings and the reference period are saved as files.
• No real customer data is in a public place, and data handling follows your employer's regional rules.
• You know the project's maintenance status and have a plan to replace the monitor if needed.

When every box is ticked, you have a monitor you can defend in a review. The Mid-level tips page covers the next layer: custom thresholds, regression and multiclass settings, multivariate drift, and operating the job on a schedule.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Where you already know the basics, we go straight to the depth.

Where this picks up

You can already fit a calculator on reference data, run it on analysis data and read the plot. The jump at work is that someone now relies on the output. A false alert wakes a person, and a missed alert costs money. Each section below therefore starts with the question "why does it behave this way", because you cannot tune a monitor you cannot predict.

One piece of context first. The version this guide is checked against is NannyML 0.13.1, released in July 2025, which is the latest release at the time of writing. Soda acquired NannyML in June 2025 and has said the open-source library stays open. There has been no release since July 2025, so treat the library as usable and stable but pin your versions, which we return to near the end.

Chunks, precisely

Everything NannyML reports is a time series of chunks. A chunk is a group of rows, and every metric, estimate and drift score is computed once per chunk. That single design decision explains most of the behaviour you will see, so it is worth knowing exactly how chunks are formed.

There are three ways to ask for chunks, and you pass exactly one of them to a calculator or estimator:

• chunk_size=5000 makes chunks of a fixed number of rows. If the data does not divide evenly, the leftover rows form an incomplete final chunk by default.
• chunk_number=9 splits the data into that many chunks of roughly equal size.
• chunk_period='M' groups rows by calendar period using the timestamp column. The aliases are S for seconds, T or min for minutes, H for hours, D for days, W for weeks, M for months, Q for quarters and A or y for years.

If you give none of them, NannyML falls back to count-based chunking with ten chunks. That default is convenient in a notebook and wrong for production, because the chunk size then depends on how much data happened to be in the window. Next week's run, with more rows, silently changes the meaning of every point on the chart.

Passing two of the options together is an error, and chunk_period without a timestamp_column_name has nothing to group by, so it fails too. Both raise exceptions at construction or fit time rather than producing odd results, which is the behaviour you want.

When you need control over the leftover rows, use the chunker classes directly:

import nannyml as nml
from nannyml.chunk import SizeBasedChunker, CountBasedChunker

# Fixed-size chunks; throw away a trailing partial chunk
chunker = SizeBasedChunker(chunk_size=3500, incomplete='drop')

# Nine chunks; fold the remainder into the last one
alt_chunker = CountBasedChunker(chunk_number=9, incomplete='append')

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc'],
    chunker=chunker,
)

The incomplete argument accepts 'drop' or 'append'. Dropping is the safer choice for monitoring, because a final chunk with 300 rows next to chunks of 5,000 has a much wider sampling error and will trip an alert or hide one for reasons unrelated to the model. Appending avoids losing data but makes the last chunk larger than the others, which changes its statistical power in the other direction.

Which scheme to pick is a statistical question and a business question. Period-based chunks line up with how people talk ("March looked bad") and with when labels arrive, but their sizes vary: a holiday week has far fewer rows than a normal one. Size-based chunks keep the statistics uniform but make the x-axis irregular. A common production compromise is to chunk by period and add a guard in your own code that ignores any chunk with fewer rows than a minimum you trust.

Where thresholds come from

An alert in NannyML is not a judgment about whether performance is good. It is a statement that a chunk's value left the range the reference period taught the monitor to expect. Understanding that difference prevents most arguments about "false alarms".

When you call fit(), the calculator computes the metric on every reference chunk and then derives a lower and an upper threshold from how those values varied. By default the thresholds sit three standard deviations either side of the mean of the reference chunk values. Each result row then carries lower_threshold, upper_threshold and a boolean alert, which is true when the value falls outside that band.

Two things follow. First, a reference set with only a handful of chunks gives a noisy estimate of the standard deviation, so the band can be too tight or too loose by luck. Second, a reference period that was unusually quiet gives a tight band, and normal production variation will trip it constantly. If you see alerts on nearly every analysis chunk, suspect the reference before you suspect the model.

You can replace the defaults with threshold objects, passed per metric in a dictionary:

from nannyml.thresholds import ConstantThreshold

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc', 'f1'],
    chunk_size=5000,
    thresholds={'roc_auc': ConstantThreshold(lower=0.7, upper=None)},
)

ConstantThreshold pins the band to values you choose, and None on a side means that side never alerts. This is the right tool when the business has a floor: "an AUROC under 0.7 means the fraud queue is unworkable" is a contract with the stakeholders, not something to be inferred from last year's noise. The library also offers StandardDeviationThreshold for the data-driven band, and it is the default behaviour. Older tutorials pass plain tuples or rely on defaults; the threshold objects are the current form, so prefer them in new code.

A practical pattern is to use both ideas on different metrics. Keep the learned band on drift scores, where nobody has a business floor in mind, and use constant thresholds on the headline performance metric, where someone does. Document the choice next to the code, since the next engineer will otherwise assume the defaults.

Sampling error and confidence bands

Every chunk value is an estimate from a finite sample, and NannyML makes that visible. Alongside each estimated or realized value you get a sampling_error and confidence boundaries set at three times that error. The shaded region on the plot is exactly that band.

Sampling error exists because a metric computed on 500 rows would differ from the same metric on another 500 rows even if nothing about the world changed. It shrinks as chunk size grows, roughly with the square root of the sample, and it grows when the metric depends on a rare class. An F1 score on a problem with 1% positives, in a chunk of 1,000 rows, rests on about ten positive examples, so the value jumps around.

This is the lever for choosing chunk size. Look at the sampling error on your reference chunks. If the confidence band is wider than the change you care about, no alert logic can rescue you: you either need larger chunks, a longer period, or a metric that is less sensitive to the rare class. The reverse mistake is chunks so large that a sharp, short-lived problem is averaged away. A fraud model that fails for two days inside a month-long chunk will look fine.

A useful working procedure is to write down the smallest drop in the metric that would make someone act, say two points of AUROC. Then pick the smallest chunk size whose sampling error is comfortably below that. The two numbers, effect size and sampling error, trade against each other and against how fast you need to hear about problems.

The sampling error also tells you when to distrust a single alert. A value just outside its threshold, with a confidence band that still overlaps the threshold, is weak evidence. A value far outside, with the band entirely beyond the threshold, is strong. Plots draw the band for this reason; when you build your own alerting from the dataframe, consider carrying that distinction through, for example by escalating only when the whole confidence interval is outside the threshold.

df = results.filter(period='analysis').to_df(multilevel=False)

# Strong signal: the whole confidence band is below the lower threshold
strong = df[df['roc_auc_upper_confidence_boundary'] < df['roc_auc_lower_threshold']]

# Weak signal: only the point estimate crossed
weak = df[df['roc_auc_alert'] & ~df.index.isin(strong.index)]

How CBPE estimates performance without labels

Confidence-Based Performance Estimation is the feature that makes NannyML different from a drift dashboard. It estimates classification metrics for chunks that have no ground truth yet. Knowing how, and when it is unsound, is the most important thing in this guide.

The idea is that a classifier which outputs calibrated probabilities already tells you how often it expects to be wrong. If the model says 0.9 for a thousand rows, and it is calibrated, about 900 of them should be positive. From those probabilities and the thresholded predictions you can compute the expected entries of the confusion matrix for each chunk, and from the expected confusion matrix any metric built on it: accuracy, precision, recall, F1, specificity, and for AUROC a similar expectation over the score distribution. The estimator is fitted on the reference data, where labels exist, and that fit includes a calibration step so that the probabilities it reasons about are trustworthy.

Two assumptions carry the whole method.

No concept drift. The relationship between inputs and the label must be the same in production as it was in the reference period. If fraudsters change tactics so that the same features now mean something different, the model's probabilities are still calibrated to the old world, and the estimate is confidently wrong. CBPE cannot see this, because it never sees a label.

No covariate shift into regions the model has not seen. The calibration was learned where the reference data lived. Inputs that move into unfamiliar territory produce probabilities the calibrator has no evidence about.

The consequence for how you work is that CBPE is an early warning, not a measurement. Treat its output as "if the world has not changed its rules, this is what performance is likely to be". When labels arrive, compare the estimate with realized performance and keep that comparison as a permanent record. If the estimate tracks reality across several periods, your trust in it is earned. If it diverges, you have discovered either concept drift or a calibration problem, and both are valuable findings.

The constructor arguments to know are y_pred_proba, y_pred, y_true (needed for the reference fit), timestamp_column_name, metrics, chunk_size or its alternatives, problem_type and thresholds. The result carries value, sampling_error, realized, the confidence boundaries, the thresholds and alert. The realized column is filled for reference chunks, where you have labels, which is why plots show the estimate and reality overlapping in the reference period.

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    timestamp_column_name='timestamp',
    metrics=['roc_auc', 'f1', 'precision', 'recall'],
    chunk_size=5000,
)
estimator.fit(reference_df)
results = estimator.estimate(analysis_df)

# Fitting on data the model has already trained on is the classic trap; see below
ref = results.filter(period='reference').to_df(multilevel=False)
print(ref[['roc_auc_value', 'roc_auc_realized']].head())

Check the reference period first. If estimated and realized values do not agree on reference chunks, where both are known, the calibration is not working and the analysis-period estimates are not worth reading. This is a one-minute test and it saves weeks of misplaced confidence.

DLE and regression models

Regression has no probabilities to lean on, so NannyML uses a different method: Direct Loss Estimation. Instead of reasoning about confidence, it trains a small internal model to predict the loss your model will incur on each row.

On the reference data, where you know both the prediction and the true value, DLE computes the per-row error of your model and fits a LightGBM regressor that predicts that error from the features and the prediction. In production it applies that internal model to unlabeled rows, aggregates the predicted losses per chunk, and converts them into the metric you asked for. The reference data therefore serves a second purpose: it is the training set for the loss estimator, so the quality of DLE depends on how representative it is.

dle = nml.DLE(
    problem_type='regression',
    feature_column_names=['km_driven', 'age', 'engine_size'],
    y_pred='y_pred',
    y_true='y_true',
    timestamp_column_name='timestamp',
    metrics=['mae', 'rmse'],
    chunk_size=6000,
)
dle.fit(reference_df)
dle_results = dle.estimate(analysis_df)
dle_results.filter(period='analysis').plot().show()

Note the argument that CBPE does not have: feature_column_names. The loss estimator needs the inputs because it learns where the model tends to be wrong. The supported metrics include mae, mape, mse, msle, rmse and rmsle.

The assumptions mirror CBPE's. If the relationship between inputs and the target changes, the learned map from features to loss is stale. If features drift into new regions, a tree model predicts a constant there and under-reports the damage. Both are reasons to pair DLE with drift detection, which the later sections cover.

There are cost implications that classification does not have. DLE trains a gradient-boosting model during fit(), so fitting is not instant on a large reference set, and the fitted estimator is worth persisting rather than refitting on every run. Also keep the feature list identical between fit and estimate. A feature that exists in production but not in reference, or the other way round, is a frequent source of confusing exceptions.

Metric choice matters in the same way as elsewhere. mape blows up when true values are near zero, and msle and rmsle require non-negative values. Pick the metric your stakeholders already use; the library will compute it, but you will be explaining the number in meetings.

Multiclass models

Multiclass classification changes one thing in the interface and several things in how you read results. Set problem_type='classification_multiclass', and y_pred_proba becomes a dictionary mapping each class label to the column that holds its probability.

estimator = nml.CBPE(
    problem_type='classification_multiclass',
    y_pred_proba={
        'prepaid_card': 'y_pred_proba_prepaid_card',
        'highstreet_card': 'y_pred_proba_highstreet_card',
        'upmarket_card': 'y_pred_proba_upmarket_card',
    },
    y_pred='y_pred',
    y_true='y_true',
    timestamp_column_name='timestamp',
    metrics=['roc_auc', 'f1'],
    chunk_size=6000,
)

The dictionary keys must match the labels that appear in y_pred and y_true exactly, in type as well as spelling. A label that is an integer in one column and a string in another is a quiet source of errors, so normalise it upstream.

Multiclass support for performance metrics arrived in version 0.12, so older tutorials will not show it. The metrics are computed in a one-versus-rest fashion and averaged, so a single headline number can hide a class that has collapsed. When a macro-averaged metric moves, look at the per-class breakdown before concluding anything, and consider tracking the rare, expensive class separately.

Sampling error compounds here, because the rare classes have the fewest rows per chunk. A three-class model with a 3% minority class needs larger chunks than a binary model of the same traffic. If your minority class is also the one the business cares about, size the chunks for it, not for the majority.

Univariate drift: choosing a method per column

Univariate drift compares the distribution of each column in a chunk with the reference. You met the calculator at Beginner level; at work the real decision is which method to use, because the choice changes what counts as drift.

For continuous columns the available methods are kolmogorov_smirnov, jensen_shannon, wasserstein and hellinger. For categorical columns they are chi2, jensen_shannon, l_infinity and hellinger. They answer slightly different questions.

• Kolmogorov-Smirnov measures the largest gap between the two cumulative distributions. It is sensitive to a shift anywhere in the distribution and produces a statistic bounded between 0 and 1. With large chunks it becomes very sensitive: it will flag differences too small to matter, because given enough rows any difference is statistically visible.
• Jensen-Shannon is a symmetric, bounded divergence between the two distributions, computed over bins. It behaves smoothly and is a sensible default when you want one method for both column types.
• Wasserstein measures how far you would have to move probability mass to turn one distribution into the other, in the units of the feature itself. That makes it the easiest to explain to a domain expert ("the typical loan amount moved by roughly this much"), but its scale depends on the feature, so thresholds are not comparable across columns.
• Hellinger is another bounded distance, and it is available for both kinds of column.
• Chi-squared tests whether category frequencies changed. For categories it is the textbook choice but, like Kolmogorov-Smirnov, grows sensitive with sample size.
• L-infinity reports the largest change in any single category's share, which reads naturally: "the share of one category moved by eight points".

The pattern that works in practice is to pick per column type and per audience rather than to enable everything. Running four methods across 80 features produces 320 time series and a great many alerts, most of which are the same event counted repeatedly.

calc = nml.UnivariateDriftCalculator(
    column_names=['loan_amount', 'age', 'channel'],
    treat_as_categorical=['channel'],
    timestamp_column_name='timestamp',
    continuous_methods=['jensen_shannon', 'wasserstein'],
    categorical_methods=['l_infinity'],
    chunk_size=5000,
)
calc.fit(reference_df)
drift = calc.calculate(analysis_df)

alerts = (
    drift.filter(period='analysis', methods=['jensen_shannon'])
    .to_df(multilevel=False)
)
drift.filter(column_names=['loan_amount']).plot(kind='distribution').show()

Use the distribution plot beside the drift plot. A drift score tells you that something changed; the joyplot for a continuous column or the stacked bars for a categorical one show what. Half of all drift alerts turn out to be a pipeline change, such as a unit switched from dollars to cents or a new category code, and the picture shows it in seconds.

Be deliberate about which columns to monitor. Monitoring every column, including identifiers and timestamps, creates alerts that carry no meaning, since an incrementing ID always "drifts". Restrict the list to features the model consumes and the handful of business fields that explain them.

Multivariate drift: when features move together

Univariate checks miss the case where every column looks normal on its own but their combination is new. Age and income can each stay in range while the pairing of young and very high income appears for the first time. Multivariate methods look at the joint structure, and NannyML ships two.

Data reconstruction drift uses PCA. During fit() it learns a low-dimensional representation of the reference features, compresses and then reconstructs each row, and records the average reconstruction error per chunk. If production rows follow the same correlations as the reference, the error stays steady. If the correlation structure changes, rows reconstruct badly and the error rises. It produces a single time series for the whole feature set, which is attractive when you do not want 80 separate charts.

The domain classifier takes the more direct route. It trains a classifier to tell reference rows from analysis rows and reports the cross-validated AUROC. An AUROC near 0.5 means the classifier cannot distinguish the two sets, so they are statistically alike; a value that climbs towards 1 means they are easy to separate, which is drift in the strongest sense.

features = ['loan_amount', 'age', 'income', 'tenure']

recon = nml.DataReconstructionDriftCalculator(
    column_names=features,
    timestamp_column_name='timestamp',
    chunk_size=5000,
)
recon.fit(reference_df)
recon_results = recon.calculate(analysis_df)

domain = nml.DomainClassifierCalculator(
    feature_column_names=features,
    timestamp_column_name='timestamp',
    chunk_size=5000,
)
domain.fit(reference_df)
domain_results = domain.calculate(analysis_df)

The two have different costs. Reconstruction is cheap and works on continuous and encoded categorical data through the PCA step. The domain classifier trains LightGBM models, so it is slower, but it handles mixed data naturally and has an interpretation anyone can follow. A reasonable arrangement is reconstruction as the always-on, low-cost multivariate signal, and the domain classifier as a deeper check you run on a schedule or when reconstruction alerts.

Interpretation needs care. A multivariate alert says the joint distribution moved, not which feature is responsible. Use it as the trigger and then use univariate results, distribution plots and the rankers in the next section to locate the cause. And an alert on a multivariate method with no corresponding drop in performance is not a false alarm; it is drift that has not yet hurt. Whether you act on it depends on how much headroom the model has.

Linking drift to performance with rankers

A monitoring system that produces forty drift alerts and one performance alert leaves the engineer with a question: which of those forty matters? Rankers exist to answer it.

AlertCountRanker is the simple one. It ranks columns by how many alerts they raised over the analysis period, so the columns that drifted most often rise to the top. It needs only the drift results.

CorrelationRanker goes a step further. It looks at how each column's drift moves alongside changes in estimated performance, and ranks columns by that correlation. The aim is not just "this feature drifted" but "this feature drifted at the times performance fell".

ranker = nml.CorrelationRanker()
ranker.fit(results.filter(period='reference'))
ranked = ranker.rank(drift, results)
print(ranked.head())

alert_ranked = nml.AlertCountRanker.rank(drift)

The correlation ranker is fitted on the performance results for the reference period, then applied to the drift results and the performance results of the analysis period. That means you need both a drift calculator and a performance estimator or calculator, chunked in the same way, so that chunk n of one aligns with chunk n of the other. A mismatch in chunk size will give you a ranking built on misaligned rows, with no warning that the answer is meaningless.

Treat the ranking as a lead, not a verdict. Correlation across a handful of chunks is weak evidence, and two features that drift because of the same upstream change will rank together. The ranker narrows forty candidates to three; your domain knowledge and a look at the distribution plots do the rest. Use it in the triage runbook: when a performance alert fires, run the ranker, open the plots for the top columns, and check the data pipeline for those columns first.

Keep in mind how few chunks the analysis typically contains. With nine chunks, a correlation can look strong by chance. The more history you accumulate, the more the ranking means, which is a reason to store results over time instead of recomputing from scratch each run.

Realized performance and closing the loop

Labels eventually arrive: the loan defaults or does not, the customer churns or stays. When they do, PerformanceCalculator computes the real metrics, and the interesting work is comparing them with what CBPE or DLE predicted.

full = analysis_df.merge(analysis_targets_df, left_index=True, right_index=True)

pc = nml.PerformanceCalculator(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='employed',
    metrics=['roc_auc'],
    timestamp_column_name='timestamp',
    chunk_size=5000,
)
pc.fit(reference_df)
realized = pc.calculate(full)

Delayed labels are the normal case, and they create a design problem. Labels for January may arrive in April. Your monitoring then needs a rule for which chunks are estimated, which are realized, and how to avoid overwriting a realized result with an estimate. Store both, tagged by type, and by the date they were computed. A table with columns for chunk start, chunk end, metric, value, kind (estimated or realized) and run date lets you answer "what did we believe in February, and what turned out to be true" without recomputation.

The merge itself is where mistakes creep in. The example joins on the index; in production you join on a stable identifier, and you must handle labels that never arrive, rows that arrive twice and rows whose label is missing for a systematic reason. If only the positive outcomes get recorded, the realized metric is biased, and you will misread a labelling gap as a performance problem. NannyML computes whatever data you give it; the soundness of the join is your responsibility.

Think also about what an estimate-versus-realized gap means. A small, noisy gap is the sampling error you already know about. A persistent gap in one direction means CBPE's assumptions fail for this model, and the most likely cause is concept drift or a poorly calibrated model. That discovery should change your policy: if the estimate is systematically optimistic, lower the thresholds you act on or recalibrate the model.

Over time this comparison is the evidence for how far to trust label-free monitoring on each model. Keep it, review it, and let it inform where you invest in faster label collection.

Choosing and protecting the reference data

Almost every misbehaving NannyML deployment traces back to the reference set. It defines the thresholds, the calibration, the loss estimator and the baseline distributions, so a flaw there spreads into every output.

A good reference period has four properties. It was held out from training, so scores are honest. It contains labels, because CBPE and DLE need them to fit. It is representative of normal production, including seasonality you consider normal. And it is long enough to produce enough chunks for stable thresholds, with enough rows per chunk for a small sampling error.

The choices that cause trouble:

• Using the test split from training time. It can be fine if it resembles production, but it often comes from a different period or sampling procedure, and the monitor then alerts on the first day because production is not like the test set.
• Using the first weeks after launch. It is realistic, but if the model was already degraded or the launch population was unusual, you bake that in.
• Using a window containing a known incident. The thresholds then widen to accommodate a problem you wanted to catch.

Version the reference data deliberately. Record which dataset, which date range and which model version produced it, and store a hash or snapshot reference. When someone retrains the model, the reference moves with it, and monitoring continuity across that boundary is a decision you should make explicitly, not discover from a chart with a cliff in it.

How long to keep a reference set before refreshing it is a judgment about how fast your world changes. A model of consumer behaviour in a region with strong seasonal patterns, such as Ramadan and the shopping seasons around it, needs a reference that covers a full cycle, or a plan to compare like with like, otherwise the calendar itself triggers drift alerts every year. Document the policy. Several teams use a rule such as: refit the monitor whenever the model is retrained, and review it every quarter regardless.

Do not refit the reference to make alerts go away. If a refit follows every alert, the monitor measures nothing. Refit when the definition of normal has changed on purpose, such as after a retrain or a deliberate product change, and record why.

Datasets such as feature stores and versioned data lakes help here, because they let you rebuild the exact reference snapshot later. If you already use a feature store, see the Feast guide; for dataset versioning, the DVC guide covers pinning data alongside code.

Running NannyML as a scheduled job

NannyML is a library, not a service. It does not run in the background, it has no server and it never sends a page. In production you wrap it in a job, and the shape of that job decides whether the monitoring is reliable.

A sound job has five steps. It loads the reference data and the fitted calculators; it loads the slice of production data for the period being monitored; it computes results; it writes them somewhere durable; and it evaluates the alert column and notifies someone. The scheduler is whatever you already operate. The Airflow and Prefect guides cover orchestrators, and a Kubernetes CronJob (see the Kubernetes guide) is a fine fit for a self-contained container.

Fit once, not on every run. Fitting reads the reference data, trains calibrators, loss estimators and PCA models, and takes time proportional to the reference size. Fit when the reference changes, persist the fitted object, and load it in the daily job. NannyML includes a store in nannyml.io.store for this, but any pickle-based persistence works, with the usual warning that pickles are code and must only be loaded from storage you trust. Record the NannyML version beside the pickle, since a fitted object written by one version is not guaranteed to load in another.

import pickle
import nannyml as nml

# --- run once when the reference changes -----------------------------
estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred_proba='predicted_probability',
    y_pred='prediction',
    y_true='label',
    timestamp_column_name='timestamp',
    metrics=['roc_auc'],
    chunk_period='D',
)
estimator.fit(reference_df)
with open('cbpe-fraud-v7.pkl', 'wb') as f:
    pickle.dump(estimator, f)

# --- run every day ---------------------------------------------------
with open('cbpe-fraud-v7.pkl', 'rb') as f:
    estimator = pickle.load(f)
results = estimator.estimate(todays_df)
frame = results.filter(period='analysis').to_df(multilevel=False)
frame['model_id'] = 'fraud-v7'
frame.to_parquet('s3://bucket/monitoring/fraud-v7/2026-09-30.parquet')

Writing results with a model_id column and a run date makes a shared table queryable by several models, and a BI tool or Grafana panel can read it directly. NannyML's own writers and runner can also send results to files or a database; database writers need the optional extra, installed as pip install "nannyml[db]", because those dependencies stopped being included by default in version 0.12. Check the current documentation for the runner's configuration format before depending on it, since this is the part of the project that changed most between versions.

Alerts are your code. The open-source library flags chunks, and nothing else; there is no built-in Slack or pager integration. Read the alert column, decide which alerts are strong enough to page and which belong in a daily digest, and send them through whatever your team already uses.

And monitor the monitor. The most dangerous failure is silence: a job that stopped running, or ran on an empty slice and produced zero chunks, looks exactly like a model with no problems. Alert on missing output, on a zero chunk count, and on a run date that is too old.

Testing the monitor itself

A monitor is software that decides when people get interrupted, and it deserves tests like anything else. The challenge is that its correct output is statistical, so the tests are about behaviour on data whose answer you know.

The most useful trick is synthetic injection. Take the reference data, copy it, and break it on purpose: shift a feature's mean, swap a category, delete values, shuffle a label relationship. Then assert that the monitor behaves as it should. A shifted feature must raise a univariate alert; an unchanged copy must not. Run these in CI with a fixed random seed.

import numpy as np
import pandas as pd
import nannyml as nml


def make_drifted(ref: pd.DataFrame, col: str, shift: float) -> pd.DataFrame:
    out = ref.copy()
    out[col] = out[col] + shift
    return out


def test_shift_is_detected(reference_df):
    calc = nml.UnivariateDriftCalculator(
        column_names=['age'],
        timestamp_column_name='timestamp',
        continuous_methods=['jensen_shannon'],
        chunk_size=2000,
    )
    calc.fit(reference_df)
    drifted = make_drifted(reference_df, 'age', shift=15)
    frame = calc.calculate(drifted).filter(period='analysis').to_df(multilevel=False)
    assert frame[('age', 'jensen_shannon', 'alert')].any()


def test_identical_data_is_quiet(reference_df):
    calc = nml.UnivariateDriftCalculator(
        column_names=['age'],
        timestamp_column_name='timestamp',
        continuous_methods=['jensen_shannon'],
        chunk_size=2000,
    )
    calc.fit(reference_df)
    frame = calc.calculate(reference_df).filter(period='analysis').to_df(multilevel=False)
    assert not frame[('age', 'jensen_shannon', 'alert')].mean() > 0.2

Check the column naming in your installed version before copying the assertions, because the flattened column names depend on the multilevel argument. Print frame.columns once and write the test against what you see.

The second class of test is contract tests for the data going in. Assert that the columns exist, that probabilities are between 0 and 1 and sum to one for multiclass, that timestamps are parseable and increasing within the slice, and that the row count is plausible. Most real incidents in monitoring are a column that changed type or a join that silently dropped half the rows, and these checks catch them at the door. Great Expectations is a good home for such checks if you already use it; see the Great Expectations guide.

The third class is regression tests on the results shape. Keep a small saved dataset and its expected number of chunks, and fail the build when an upgrade changes it. Since upgrades are rare but breaking changes do exist in this project's history, such as the dropped Python versions and the optional database dependencies, a small suite that runs on every dependency bump buys real safety.

Finally, test the false-positive rate, not just the detection rate. Split your reference data in two, fit on the first half, run on the second, and count alerts. A well-set-up monitor should alert rarely on data from the same distribution, and a high count means your chunks are too small or your reference too short.

Performance and memory

NannyML works on in-memory pandas dataframes and has no distributed mode. For most models that is fine, and the bottlenecks are predictable once you know what each component costs.

Memory scales with the size of the analysis dataframe plus whatever the calculators copy. The cheap defence is to process one period at a time: load a day or a week from the warehouse, run the calculators, write the results, discard the dataframe. Because the calculators are already fitted on the reference, each period is independent, and the work fits in a small container. Select only the columns you need, and convert wide string columns to categories before you load them.

CPU cost depends on the method. Univariate drift is cheap per column but scales linearly with the number of columns and methods. Reconstruction drift costs a PCA transform. The domain classifier and DLE train gradient-boosting models and are the heaviest; for those, fit once, persist, and pay only the cost of inference afterwards. If the domain classifier is slow on a large reference, subsample the reference to a representative size, as the extra rows add little once the classifier has enough to learn from.

When you have billions of rows, the answer is to reduce before NannyML sees them. Aggregate or sample in SQL or Spark first, keeping the columns the monitor needs. A uniform sample with a fixed seed preserves the distributions that drift methods compare, though it widens the sampling error, which you can read directly from the results. Stratify by the label or by a rare class when the minority carries the business value, otherwise the sample may contain too few of them. For heavy preprocessing the Spark guide is the right companion.

Plotting has its own costs. Plots are Plotly figures; building hundreds of them in a loop is slow and memory hungry. Filter before plotting, and only generate figures when a person will look at them. To save a figure as HTML use figure.write_html('report.html'), and for static images write_image needs the separate kaleido package.

Two small tips complete the picture. Run on a recent supported Python, which at the time of writing means 3.9 to 3.12, because wheels for LightGBM and other dependencies are built for those versions. And measure before optimising: time fit and calculate separately on a realistic day of data, so you know which one to move out of the daily path.

Environment, versions and the maintenance question

The practical risks around NannyML at this level are less about the API and more about the environment around it.

Pin everything. The last release was 0.13.1 in July 2025, and the project has seen few commits since Soda's acquisition. A community issue on the repository asks about the stale state of the project. None of this makes the library unsafe to use, but it means that newer versions of pandas, NumPy, scikit-learn or Pydantic can outrun it, and nobody will release a compatibility fix on a schedule. Pin nannyml==0.13.1 and the libraries around it in a lock file, build the image once, and test upgrades of the neighbours in CI before taking them.

Python support is 3.9 to 3.12. Version 0.13.0 added 3.12 and dropped 3.8, and 0.12 had dropped 3.7. Python 3.13 is not declared as supported, so on a 3.13 interpreter pip may report that no matching distribution is found. Use 3.11 or 3.12 for the container.

System libraries. LightGBM needs OpenMP. On macOS, brew install libomp fixes an error mentioning libomp.dylib. On slim Linux images the equivalent is apt-get install -y libgomp1, which fixes libgomp.so.1: cannot open shared object file. Bake it into the image so the first run does not fail at 3 a.m.

Database extras. Since 0.12, the database dependencies are optional. If your job writes to a database through NannyML's writers and you see an import error mentioning sqlmodel or a database driver, install nannyml[db].

Corporate networks. If your build runs behind TLS inspection, pip may fail with a certificate error. Export the proxy's root certificate and point PIP_CERT or SSL_CERT_FILE at it; never switch verification off. Check with your security team where the certificate bundle lives.

Telemetry. The library can send anonymous usage information, and the documentation describes how to opt out. In regulated environments, such as banks and government workloads in the Gulf and Egypt where outbound traffic is controlled, confirm the current opt-out setting in the documentation, disable it, and verify with a network policy that the container cannot reach the internet at all.

Data residency. The monitor reads model inputs, which often contain personal data. Run it inside the same region and network boundary as the model, and restrict who can read the result tables, because feature distributions can reveal sensitive information.

Finally, have a plan for the library's future. The announced direction from the maintainers includes a future major version, and nothing is promised on its timing. Keep your monitoring logic thin and well tested, so that if you ever move to another tool such as Evidently, the work is a swap of a calculator and not a rewrite.

Debugging beyond the basics

When the monitor misbehaves, the symptom usually points at one of a small number of causes. The sections above give the reasons; this one gives the order of questions.

Every chunk alerts, including the reference. Chunks are too small for the metric, or the reference is too short or too calm. Look at the sampling error, count the reference chunks, and widen both.

Nothing ever alerts. Chunks may be too large, hiding short events; thresholds may be wide because the reference contained an incident; or a ConstantThreshold is set at a value that can never be reached. Inject a synthetic shift and see whether it is caught.

Estimated performance is far from realized. First compare them on the reference period, where both are known. If those disagree, calibration is the issue and you should recalibrate the model. If the reference agrees but analysis chunks diverge, suspect concept drift, a labelling gap, or covariate shift into unseen regions.

Drift alerts with stable performance. The model may be robust to that feature, or the shifted feature may be unimportant. This is information, not an error; check feature importance from the training run before reacting.

Exceptions. NannyML raises its own exception types for common mistakes. InvalidArgumentsException signals a bad combination of arguments or a missing column. A calculator that has not been fitted raises an error telling you to call fit() first. Chunking by period without a timestamp column also fails. Read the message first, because the wording names the argument. Empty chunks or warnings about no samples usually mean a chunk_period produced a period with no rows, or the reference is too small.

Wrong numbers, no error. These are the dangerous ones, and they come from data. Check dtypes (a probability column loaded as strings), timestamps in mixed time zones, duplicate rows after a join, and probability columns that do not sum to one in multiclass. Print analysis_df.describe() and a few raw rows before you debug the library.

A habit that pays: log, per run, the reference window, the analysis window, the chunk count, the NannyML version, and the number of alerts. Most of the questions above can be answered from that line without a debugger. If you track experiments in MLflow, it is a reasonable place to log those run parameters and attach the result dataframe as an artifact.

Putting it all together

Here is how the pieces fit for a binary fraud model with delayed labels. Treat it as a design you can adapt, not a recipe to paste.

1. Reference. A held-out, labelled window of six weeks that covers a normal pay cycle, stored as a versioned parquet snapshot with its date range recorded. The model version is part of its name.
2. Chunking. Daily chunks, because the business reacts daily and the volume gives a sampling error well below the two-point AUROC change the team cares about. A guard in the job ignores chunks below a minimum row count.
3. Fit once. CBPE for roc_auc and recall, univariate drift using Jensen-Shannon on twelve model features, and reconstruction drift as the multivariate signal. Fitted objects are pickled with the NannyML version.
4. Thresholds. A learned band on drift, and a ConstantThreshold floor on recall, because the fraud operations team has a number below which their queue is unmanageable.
5. Daily job. A scheduled task loads yesterday's scored transactions, estimates performance, calculates drift, writes a tagged parquet file and updates a results table with model_id, run date and kind.
6. Alerting. A separate step reads the result frame. A recall floor breach with the whole confidence band beyond it pages the on-call; drift-only alerts go into a daily message with the top three columns from the ranker.
7. Closing the loop. When labels arrive weeks later, a second job computes realized performance and appends it, and a weekly report plots estimate against realized for each model.
8. Watching the watcher. A freshness check alerts if no result file is newer than 30 hours or the analysis chunk count is zero.
9. Tests. Synthetic-shift tests and contract tests run in CI on every dependency change, against a pinned image.

The sequence reads as a pipeline, but the thinking behind it is a set of decisions: what counts as normal, how quickly you need to know, how much noise you can live with, how you will verify the estimates and who is told. The code is the short part.

# The job's container, pinned and reproducible
docker build -t fraud-monitor:2026-10 .
docker run --rm \
  -e MODEL_ID=fraud-v7 \
  -e RUN_DATE=2026-09-30 \
  fraud-monitor:2026-10 python run_monitor.py

The Docker guide covers the image-building practices behind this, including pinned bases and digest-referenced tags.

Where you are now

You can now explain how a chunk is formed and why the choice changes everything downstream, where thresholds and confidence bands come from, and how CBPE and DLE estimate performance, including the assumptions that make their output an early warning and not a measurement. You can choose drift methods per column, add multivariate checks, rank drifting features against performance changes, and compare estimates with realized results once labels arrive. You have a view on reference data, a design for a scheduled job, a way to test the monitor, and a plan for running a library whose release pace has slowed.

The Senior level takes the same tool as a platform: how to run monitoring for many models and teams, how to handle custom metrics, persistence and the runner at scale, how to govern thresholds and alerts, and when to choose something else. Before that, the best next step is to apply this on a real model at work and keep the estimate-versus-realized record for a quarter. That record is the evidence that tells you how far to trust the tool.

Sources

• NannyML documentation
• Quickstart
• Chunking
• Binary performance estimation with CBPE
• Detecting data drift
• Univariate drift detection
• NannyML on PyPI
• NannyML GitHub repository
• NannyML releases
• LightGBM installation guide

This file prepares you for the mid-level conversation about NannyML. At this level an interviewer is less interested in whether you can recite the API and more interested in whether you can reason about a monitor that people depend on. Each answer below is written the way a strong candidate would say it: the first move, the reasoning, and the caveat. Use them as models for structure, then replace the examples with your own.

How these interviews go

Expect three kinds of question. Scenario questions give you a symptom ("the monitor alerts every day") and watch how you narrow the cause. Trade-off questions ask you to choose between two reasonable designs and defend the pick. Probing questions take something you said and push on the assumption underneath it, typically "what if there are no labels" or "how would you know".

The interviewer is listening for four habits. Do you ask what the model is for and how fast labels arrive before choosing a design? Do you separate what the tool measures from what you infer? Do you mention the failure mode of your own answer? And do you have a plan for the monitor itself failing silently? A candidate who says "I would set up CBPE and alert on the alert column" has described a demo. A candidate who adds "and I would check the estimate against realized values on the reference period first, because the method assumes calibration" has described a system.

Scenario questions

The monitor alerts on almost every chunk, including some reference chunks. What do you check?

Alerts on reference chunks mean the thresholds are too tight for the natural variation, which points at the inputs to the thresholds, not at the model. I would check three things in order.

First, the sampling error. If the confidence band on a chunk is wide relative to the gap between the value and the threshold, the chunk is too small for the metric. Rare-class metrics are the usual cause: an F1 score over a 1% positive class with 1,000 rows per chunk rests on about ten examples.

Second, how many reference chunks there are. The default thresholds come from the spread of the metric across reference chunks, three standard deviations either side of the mean. With a handful of chunks that standard deviation is itself poorly estimated, so the band can be too narrow by luck. I would lengthen the reference or shrink the chunk size so there are more chunks, but only if the sampling error allows it.

Third, whether the reference period was unusually quiet. If it was, the band reflects a calm that production never has.

The fix is to refit with a longer, more representative reference, or larger chunks. What I would not do is widen thresholds by hand until the alerts stop, because that hides the problem.

A strong answer names sampling error and reference length, and says refit instead of patching the output. A weak one says "make the thresholds bigger".

Performance is estimated to be fine, but the business says the model is getting worse. How do you find out which is right?

The business is more likely to be right, because the estimate rests on assumptions and the complaint rests on outcomes. CBPE assumes no concept drift and calibrated probabilities, so a change in the relationship between inputs and label is invisible to it.

I would first compare the estimate with realized performance on whichever recent chunks already have labels. If realized is lower than estimated, the assumptions have failed and I have a finding. Then I would look at calibration on the labelled data, since a reliability curve showing that the probabilities no longer match observed frequencies explains the gap directly. If calibration is fine but the realized metric is still lower, the likely cause is concept drift, and the fix is retraining, not monitoring changes.

If labels have not arrived yet, I would use the drift results to look for covariate shift into regions the model has not seen, because there the estimate is least reliable. And I would ask the business for specifics: which segment, which time, which kind of error. A segment-level problem can be averaged away in a whole-population metric.

A strong answer treats the estimate as a hypothesis to be checked and has a concrete way to check it. A weak one defends the tool.

Labels arrive six weeks late. How do you design the monitoring?

I would run estimation immediately and realized performance when labels land, and keep both. Estimation is the point of the library in this situation: six weeks is longer than the business can wait, so a label-free estimate is the early warning, and drift detection backs it up.

The design details matter. I would store results in a table with the chunk dates, the metric, the value, the kind (estimated or realized) and the run date, so I can see what we believed at each point in time. When labels arrive, a second job computes realized performance for those chunks and appends it, never overwriting the estimate. A weekly comparison of estimated against realized per model tells me how far to trust the estimate on that model.

The strong answer includes the append-only record and the comparison loop. The weak answer stops at "use CBPE".

Your team wants to monitor every feature with every drift method. How do you respond?

I would push back, politely and with arithmetic. Eighty features times four methods is 320 series, and most alerts will be the same event counted several times. People stop reading a channel like that.

I would propose a smaller design. Pick one method per column type, with a reason. Jensen-Shannon is a reasonable default because it works for both continuous and categorical columns and is bounded. Restrict the columns to features the model actually consumes, and drop identifiers, since an incrementing ID always drifts. Add one multivariate signal, such as reconstruction drift, to catch joint movement. Then add a ranker to point at the likely culprits when performance moves.

I would also tie alerting to action. Drift without a performance change goes into a digest; drift with a performance alert pages someone. If after a quarter a particular series has never led to an action, I would remove it.

A strong answer reduces the signal count and links alerts to decisions. A weak one agrees to collect everything.

The nightly job succeeded for two weeks and shows no alerts. Is the model healthy?

I cannot tell from that, and I would say so. A job that "succeeds" can still produce nothing: the input slice may be empty, producing zero analysis chunks; a join may have dropped half the rows; or the scheduler may have stopped and the last output is old.

I would check the run record: the number of chunks, the row counts per chunk, the output timestamp. Then I would inject a synthetic shift into a copy of recent data and confirm the monitor catches it, which tests that the machinery detects anything at all.

For the long term I would add a freshness alert that fires when no new results exist, and a minimum-row-count guard. Silence from a monitor is only evidence of health if you also know the monitor is running and reading data.

A strong answer distrusts silence. A weak one says "great, no alerts".

A multivariate drift alert fires but univariate checks are all quiet. What does that mean and what do you do?

It means no single column moved much, but their combination did. Typical causes are a change in correlations, for example a new customer segment whose age and income pairing did not exist before, or an upstream change that altered one column in a way that interacts with another.

For reconstruction drift, the PCA model reconstructs rows badly when the correlation structure changes, so the error rises even if every marginal distribution is stable. For the domain classifier, a rising AUROC says reference and analysis rows can now be told apart.

My next steps: find which columns are responsible. I would run the correlation ranker if I have performance results, check pairwise plots of the likely candidates, and ask the data owners what changed upstream. For a domain classifier I could also look at which features it relies on most. I would check whether performance moved. If it did not, the alert is drift that has not yet hurt, and the decision is whether the model has headroom.

A strong answer explains why the combination can move while marginals do not. A weak one calls it a false positive.

How would you roll out NannyML for ten models at once?

I would not copy ten notebooks. I would write a thin shared wrapper that takes a small config per model: column names, problem type, chunk settings, reference location, thresholds. That gives one place to fix bugs and one results schema with a model_id column.

Each model gets its own reference data and its own fitted estimator, versioned together with the model. Results go to a shared table that a dashboard reads. Alert routing goes through the team's existing channels. I would start with two or three models that have good labels, prove that the estimates track reality, and then extend, because the effort of choosing chunk sizes and references is per model and cannot be automated away.

I would also set expectations: the library has no tenant concept, so isolation, access control and naming are on us.

Trade-off questions

Period-based chunks or size-based chunks?

Period chunks match how people talk and when labels arrive, so reports line up with the business calendar. Their weakness is uneven sizes: a holiday has fewer rows, so its sampling error is wider and the alert logic behaves differently. Size-based chunks keep the statistics uniform but make the time axis irregular, which confuses readers.

My choice depends on volume stability. With steady traffic, I would use periods. With uneven traffic I would either use size-based chunks or periods with a guard that ignores chunks below a minimum row count. I would say that the reference must be chunked the same way, and that changing the scheme means refitting.

Learned thresholds or constant thresholds?

Learned thresholds adapt to the metric's natural variation, so they suit signals nobody has an opinion about, such as drift scores. Constant thresholds encode a business requirement, such as a minimum recall, and are independent of how good or bad the reference period was.

I would use both: learned on drift, constant on the headline metric the business cares about. The risk with learned bands is that a poor reference gives a poor band. The risk with constant ones is that they go stale when the business changes, so I would put an owner and a review date on each.

CBPE and DLE, or just wait for labels?

If labels arrive quickly relative to the business risk, compute realized performance and skip estimation: it is simpler and it is real. If labels are slow, estimation is the only early signal. The cost is the assumptions and the extra component to trust.

A balanced answer: use estimation where the label delay is long, always back it with drift detection, and always compare against realized values when they come. Estimation without that comparison is faith.

Univariate or multivariate drift?

Univariate is cheap, explainable and tells you which column moved. Multivariate catches joint changes but gives one number that does not say where. I would run both: multivariate as the compact signal, univariate as the diagnosis.

On cost, reconstruction drift is light, while the domain classifier trains models and is heavier but easier to explain. I would default to reconstruction and bring in the domain classifier for periodic deeper checks.

NannyML or Evidently?

They overlap on drift. NannyML's distinguishing feature is label-free performance estimation with confidence bands. Evidently has richer reports and a service-style UI. If the main risk is delayed labels, NannyML fits. If the need is broad reports and a dashboard for many data-quality and drift checks, Evidently may be easier. I would also mention the maintenance factor: NannyML's last release was in July 2025, after the Soda acquisition, so I would pin versions and keep my wrapper thin enough to swap. See the Evidently guide for the other side.

Fit on every run, or persist the fitted calculator?

Persist. Fitting trains calibrators, PCA models and loss estimators on the reference, so it is slow and gives a result that should not change from run to run. A daily job that refits has a moving baseline and can hide drift. I would fit when the reference changes, store the fitted object with the NannyML version and the reference snapshot identifier, and load it in the daily job. The caveat is that pickles are code, so only load them from storage you control.

Run it on all the data or on a sample?

Sampling upstream is usually right once the data is large, because NannyML is in-memory pandas. A fixed-seed uniform sample keeps the distributions, and the sampling error column shows what it cost. I would stratify when a rare class matters. If the volume is moderate, I would use all rows and process one period at a time.

Probing questions

What does "no concept drift" mean and why does CBPE need it?

Concept drift is a change in the relationship between inputs and the label. CBPE estimates performance from the model's own probabilities, which were calibrated on reference data. If the relationship changes, the probabilities are calibrated to a world that no longer exists, so the estimate is wrong while still looking confident. Since the method never sees labels, it cannot detect that. That is why I treat the estimate as an early warning and compare it with realized values.

Why shouldn't reference data come from training data?

A model scores the rows it trained on with unrealistic confidence and accuracy. A reference set made of them teaches the estimator a baseline that production will never match, and the calibration fitted on it is flattering. A held-out set is honest. If the held-out set differs in time or sampling from production, I would say that too, since it affects the thresholds.

How do you read the confidence band on the plot?

It is three times the sampling error either side of the value. It tells you how much the metric would move from resampling alone. If a value is outside its threshold but the band still overlaps the threshold, the evidence is weak. If the whole band lies beyond the threshold, it is strong. I would build the paging rule on the strong case.

What happens if you change the chunk size?

The meaning of every number changes. Sampling error shrinks or grows, thresholds learned on the old chunks are wrong for the new ones, and old results are not comparable. I would refit, record the change, and mark the discontinuity in the results table.

What are the risks of using this library now?

The last release was 0.13.1 in July 2025, and the repository has seen little activity since the acquisition by Soda, with a community issue asking about it. The maintainers have said the library remains open. The risk is dependency drift: newer pandas, NumPy or Pydantic may break it with no timely fix. I would pin versions, support only Python 3.9 to 3.12, test dependency upgrades in CI and keep my monitoring logic thin enough to replace.

What strong answers have in common

Strong answers start from the purpose of the model and the timeline of its labels. They name the assumption behind each method and what breaks when it fails. They give a procedure with an order ("first this, then that") instead of a list of possibilities. They mention how the answer itself could be wrong. And they include the operations around the library: where results are stored, how freshness is checked, how versions are pinned.

Weak answers have the opposite shape. They list features, they trust the output, they treat an alert as a fact about the model, and they stop at the notebook.

A short self-test

Answer each aloud in under a minute, then compare with the sections above.

1. Why do alerts appear on reference chunks, and what do you change?
2. What two assumptions does CBPE make, and how do you test them?
3. How does DLE estimate regression loss, and what do you need to pass in that CBPE does not need?
4. When do you prefer reconstruction drift to a domain classifier?
5. Why append estimates and realized results instead of overwriting?
6. What would you log on every run?
7. What is your answer if asked whether you would still choose NannyML today?

If you stumble on one, return to the matching section of the detailed guide and rehearse it with your own project's numbers.

For related ground, see the MLflow guide for tracking monitor runs, the Prefect guide for scheduling, and the Great Expectations guide for input checks.

This file collects what goes wrong when you run NannyML for real, what pays off most, and a few recipes you can lift into your own jobs. It assumes you have read the mid-level detailed guide and have a calculator or estimator running on at least one model.

Errors you will meet at this level

Beginner errors are about installation and syntax. At work the errors are subtler: the code runs, and the numbers are wrong or the job dies in production but not on your laptop. Each entry gives the symptom, the cause and the fix.

Every chunk alerts, even in the reference period

Symptom. The plot is a wall of red markers from the first chunk.

Cause. The thresholds come from the spread of the metric across reference chunks. If chunks are too small, the metric is noisy and, paradoxically, the band is wide but the sampling error is wider still. If the reference has very few chunks, the standard deviation is a poor estimate. If the reference was unusually calm, the band is too tight for normal production.

Fix. Look at sampling_error on reference chunks. Increase chunk size until the confidence band is smaller than the change you care about, extend the reference so it holds more chunks, and refit. Do not widen the thresholds by hand to quiet the chart.

No alerts ever, even when you know something changed

Symptom. A known incident, such as a unit change in one column, leaves the chart flat.

Cause. Chunks so large that a short event is averaged away, a reference that contained an earlier incident and so learned wide bands, or a ConstantThreshold set at a level the metric never reaches.

Fix. Inject a synthetic shift into a copy of the data and see what it takes to make the monitor react. Shrink chunks if the sampling error allows, rebuild the reference without the incident, and check your constant thresholds against actual values.

Estimated performance disagrees with realized performance

Symptom. CBPE reports a stable AUROC, and when labels arrive the real value is lower.

Cause. The two assumptions behind the method failed: concept drift, where the input-to-label relationship changed, or poor calibration, where the model's probabilities do not match observed frequencies. A third cause is a flawed reference, such as training data reused as reference.

Fix. Compare estimate and realized on the reference period first. If they differ there, fix calibration or the reference. If they match on reference but not later, suspect concept drift and consider retraining. Record the gap per period.

InvalidArgumentsException at construction or fit

Symptom. An exception naming a missing or conflicting argument.

Cause. A column name that does not exist in the dataframe, a missing y_pred_proba for a probabilistic problem, or conflicting chunk options, such as giving both a size and a number.

Fix. Read the message, because it names the argument. Print df.columns and compare with the names you pass. Give exactly one chunking option.

Calling calculate or estimate before fit

Symptom. An exception stating that the calculator was not fitted and asking you to call fit().

Cause. The object was created but never fitted, or you loaded a pickle that was saved before fitting.

Fix. Fit on the reference data first. In a daily job, save the fitted object and add an assertion right after loading it.

Chunking by period without a timestamp

Symptom. An exception when you pass chunk_period and no timestamp_column_name.

Cause. Period chunking groups rows by their timestamps and has nothing to group by.

Fix. Pass the timestamp column name, and check that the column is a datetime type and not a string.

Empty chunks and warnings about no samples

Symptom. Warnings about empty chunks, or a metric that is missing for a period.

Cause. With chunk_period, a period had no rows, for example a weekend in a business-days system. Or the reference is too small for the chunk size.

Fix. Choose a period that matches the data's rhythm, or switch to size-based chunks. Check the row count per chunk before trusting any value.

ModuleNotFoundError for sqlmodel or a database driver

Symptom. A job that wrote to a database through NannyML starts failing after you rebuild the image.

Cause. Since version 0.12, the database dependencies are optional.

Fix. Install the extra: pip install "nannyml[db]", and add it to the lock file.

LightGBM library errors

Symptom. On macOS, Library not loaded: @rpath/libomp.dylib. On slim Linux images, libgomp.so.1: cannot open shared object file.

Cause. LightGBM needs the OpenMP runtime, and NannyML depends on LightGBM.

Fix. brew install libomp on macOS, and apt-get install -y libgomp1 in the Dockerfile on Debian-based images. Bake it into the image.

No matching distribution found for nannyml on a new Python

Symptom. pip refuses to install on Python 3.13.

Cause. NannyML 0.13.1 declares support for Python 3.9 to 3.12.

Fix. Use 3.11 or 3.12 in your image and in CI.

Import errors that mention Pydantic

Symptom. An error such as cannot import name 'BaseSettings' from 'pydantic', usually after upgrading another package.

Cause. Two libraries in the environment disagree about which major version of Pydantic they need.

Fix. Pin the Pydantic version in your lock file and test the environment in CI. Do not fix it by installing things at random in production.

Plot export fails

Symptom. An error from write_image.

Cause. Static image export needs the kaleido package.

Fix. pip install kaleido, or save to HTML with write_html.

Certificate errors during pip install on a corporate network

Symptom. SSLError: CERTIFICATE_VERIFY_FAILED.

Cause. A TLS-inspecting proxy re-signs traffic with a certificate your Python does not trust.

Fix. Export the proxy's root certificate and set PIP_CERT or SSL_CERT_FILE to it. Never disable verification.

Silent errors: wrong numbers, no exception

These cost the most because nothing tells you. Typical causes are probability columns loaded as strings, timestamps in mixed time zones, duplicated rows after a join, multiclass probabilities that do not sum to one, and a class label that is an integer in one column and a string in another. Print describe() and a few raw rows before debugging the library.

Practices that pay off most, in order of leverage

These are ordered by how much trouble each one prevents per hour spent on it.

1. Validate the reference with the estimate-versus-realized check

Before you read a single analysis result, plot estimated against realized values on the reference chunks. If they do not agree where you know the answer, nothing downstream is trustworthy. It takes a minute and catches calibration problems, leaked training data and wrong column mappings in one go.

2. Choose chunk size from sampling error and the smallest change that matters

Write down the smallest drop in the metric that someone would act on. Choose the smallest chunk size whose sampling error is comfortably below it. Revisit when traffic changes materially, and refit whenever you change it.

3. Fit once and persist

Fit when the reference changes, and store the fitted object with the NannyML version and a reference snapshot identifier. A daily refit gives a moving baseline and wastes compute. Remember that a pickle is code: load only from storage you control.

4. Keep the estimate and the realized result, both, forever

Append results with columns for chunk dates, metric, value, kind (estimated or realized) and run date. Never overwrite. The history is your evidence for how far to trust the estimate on each model.

5. Alert on the strong case

A point outside the threshold with a confidence band that still overlaps it is weak evidence. Page on cases where the whole band is beyond the threshold, and send the rest to a digest. Fewer, more believable alerts keep people reading.

6. Use constant thresholds where the business has a floor

Learned bands are for signals nobody has an opinion about. For the headline metric, put the real requirement in a ConstantThreshold, and record the owner and review date.

7. Monitor the monitor

Alert when no new result exists, when the chunk count is zero, or when row counts are far below normal. A broken scheduler looks identical to a healthy model.

8. Pin versions and test upgrades

Pin nannyml==0.13.1 and its neighbours. The project's last release was July 2025, so you cannot rely on quick fixes. Run a small suite with synthetic shifts and a known-chunk-count check in CI whenever a dependency changes.

9. Reduce the signal count

Choose one drift method per column type, only model features, one multivariate signal and a ranker. A small monitor that people read beats a large one they mute.

10. Sample or aggregate upstream when data is large

NannyML is in-memory pandas. Reduce rows in SQL or Spark first, keep the columns you need, use a fixed seed, and stratify on rare classes. The sampling error column will tell you what the reduction cost.

11. Log a structured line per run

Reference window, analysis window, chunk count, row count, version and number of alerts. Most investigations can be answered from this without a debugger.

12. Version and document the reference policy

Which dataset, which dates, which model version, when to refresh. A written rule beats a surprise cliff in a chart after a retrain.

Practice cards

Each card is a short exercise. Do them on a real or synthetic dataset, and write down what you saw, because writing it down is what makes it stick.

Card 1: The chunk size experiment

Take a fitted estimator and run it at three chunk sizes. Record the number of chunks, the median sampling error and the number of alerts for each. Pick the one you would defend in a review and write the sentence that defends it.

Card 2: Inject and detect

Copy your reference data, shift one feature by a chosen amount, and run a univariate drift calculator on it. Find the smallest shift that triggers an alert. Repeat with a method from a different family, such as kolmogorov_smirnov versus wasserstein, and compare.

Card 3: The false-alarm rate

Split the reference in half, fit on the first half and run on the second. Count alerts. Change chunk size and see how the rate responds. Decide on an acceptable rate for your team and write it down.

Card 4: Estimated against realized

Using a dataset that has labels for the analysis period, run CBPE and PerformanceCalculator over the same chunks. Plot both and compute the gap per chunk relative to the sampling error. Write a verdict: do you trust the estimate for this model?

Card 5: Break the calibration

Take your analysis probabilities and push them toward 0 or 1, for example by applying a monotonic transform, while keeping predictions the same. Run CBPE and watch the estimate move even though the labels did not. Explain in two sentences why this shows the method depends on calibration.

Card 6: Rank the suspects

Run drift on all features and a performance estimator on a period where performance fell. Run both rankers and compare their top three. Open the distribution plots and decide which ranking pointed to the real cause.

Card 7: The silent failure

Run your daily job with an empty input file. Does it fail loudly, produce zero chunks quietly, or write an empty result? Add the check that makes it fail loudly.

Card 8: Rebuild from the pickle

Fit, pickle, restart your Python process and load. Run an estimate, then compare it with a run where you fitted fresh. They should match; if not, find out what was not saved.

Card 9: Multiclass wiring

Take a three-class problem and wire up the y_pred_proba dictionary from a single list of class labels. Break it on purpose by renaming one key, and read the exception you get.

Card 10: The reference policy

Write one paragraph: where the reference comes from, which dates, how it is stored, when you refresh it and who approves. Ask a colleague to find a hole in it.

Operational recipes

Recipe: a pinned, reproducible image

Python 3.12, the OpenMP runtime and pinned packages. Build once, run many times. See the Docker guide for digest pinning and multi-stage builds.

FROM python:3.12-slim

RUN apt-get update \
 && apt-get install -y --no-install-recommends libgomp1 \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

COPY run_monitor.py .
CMD ["python", "run_monitor.py"]

nannyml==0.13.1

Generate the full lock file from your environment so that pandas, NumPy, scikit-learn and Pydantic are pinned alongside it.

Recipe: a freshness and sanity check

Run this after the monitor job, in the same pipeline, so silence becomes a failure.

import sys
import pandas as pd

frame = pd.read_parquet(sys.argv[1])

problems = []
if frame.empty:
    problems.append('result file has no rows')
if 'chunk_key' in frame.columns and frame['chunk_key'].nunique() == 0:
    problems.append('zero analysis chunks')

if problems:
    print('MONITOR UNHEALTHY: ' + '; '.join(problems))
    sys.exit(1)
print(f'ok: {len(frame)} rows')

Use the same pattern to compare the result's newest chunk end date against today and fail if it is older than your tolerance. Check your own result schema for the column names before relying on the ones above.

Recipe: strong and weak alerts

Separate evidence levels so that only the strong one pages. Adapt the column names after printing frame.columns for your metric.

frame = results.filter(period='analysis').to_df(multilevel=False)

metric = 'roc_auc'
upper_ci = frame[f'{metric}_upper_confidence_boundary']
lower_th = frame[f'{metric}_lower_threshold']

strong = frame[upper_ci < lower_th]          # whole band below the floor
weak = frame[frame[f'{metric}_alert']].drop(strong.index)

Recipe: appending results with provenance

Write each run to its own file and include enough columns to rebuild the story later.

from datetime import date

frame = results.filter(period='analysis').to_df(multilevel=False)
frame['model_id'] = 'fraud-v7'
frame['kind'] = 'estimated'
frame['run_date'] = date.today().isoformat()
frame['nannyml_version'] = nml.__version__
frame.to_parquet(f"s3://bucket/monitoring/fraud-v7/{frame['run_date'].iloc[0]}.parquet")

The column names after flattening can include tuples or joined strings depending on the options you use, so write one normalisation step and test it.

Recipe: opting out of network access

A monitor that reads sensitive features should not reach the internet. In Kubernetes, give the pod a network policy that allows only your warehouse and storage endpoints. This also covers any usage telemetry the library may send, so you do not have to rely on a setting alone. The Kubernetes guide covers network policies. Check the NannyML documentation for the current opt-out setting and apply both.

Recipe: a CronJob wrapper

Run the job on a schedule with a retry limit and a time limit, so a stuck run does not block the next one.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: fraud-monitor
spec:
  schedule: "30 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: monitor
              image: registry.example.com/fraud-monitor:2026-10
              env:
                - name: MODEL_ID
                  value: fraud-v7
              resources:
                requests: { cpu: "1", memory: 4Gi }
                limits: { memory: 8Gi }

Size memory from a measured run on a realistic day, not from a guess. The container reads credentials from a mounted secret or the cluster's identity mechanism, never from the image.

Recipe: where to send the results

Write results to a table keyed by model_id, chunk dates and metric, and chart them in a dashboard tool. The Grafana guide shows panels over SQL data sources, and the Prometheus guide is the route if you prefer to export the latest values as metrics and alert through your existing rules.

A checklist before you call a monitor done

• The reference is held out, labelled, representative and versioned.
• Estimate and realized agree on the reference chunks.
• Chunk size is justified by sampling error and a stated minimum change.
• Thresholds are documented: which are learned, which are constant, and who owns them.
• Fitted objects are persisted with the NannyML version.
• Results append with model, kind and run date.
• Strong and weak alerts are separated, and each goes somewhere a person reads.
• A freshness and zero-chunk check fails loudly.
• Versions are pinned, Python is 3.12 or earlier, and the OpenMP runtime is in the image.
• A synthetic-shift test and a false-alarm test run in CI.
• The data path stays inside your network boundary, and results are access controlled.
• Someone owns the reference refresh policy.

When every line is true, your monitor tells you something you can act on, and equally important, you will notice when it stops telling you anything at all.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when NannyML is not your notebook but a platform other teams depend on: the assumptions that decide whether an alert means anything, the shape of the jobs you run, the data you are trusted with, and the question nobody on the other teams will ask until it is too late, which is what happens when the library stops moving.

Up to now the questions were about how to get a calculator to produce a sensible chart. From here they are different. Who decides what the reference data is, and who can change it? When a team tells you "the estimate said our model was fine and it was not", can you show why? What does a monitoring job cost across forty models? And what is your plan if the upstream project goes quiet?

Where this picks up

I will start with what the library is architecturally, because every operational decision follows from the fact that it is not a service.

What you are actually operating

NannyML is a Python library, not a server. Nothing listens on a port, nothing stays resident, and nothing remembers what happened yesterday unless you wrote it down. A calculator is an object: you construct it, call fit() on reference data to learn baselines and thresholds, then call calculate() or estimate() on analysis data to get a results object. Everything else, scheduling, storage, alerting, access control, a dashboard, is yours to build or to borrow from the tools around it.

That sounds like a limitation until you see what it buys you. A batch library has a very small failure surface. There is no cluster to patch and no database to migrate for the library itself. A run either completes and produces rows, or it fails and produces none. The price is that the platform behaviour people expect from "a monitoring tool" does not exist unless you create it, and the people on other teams will assume it does.

Read that diagram as the list of things you own. The library is the third box. The scheduler is your orchestrator, and the Airflow and Prefect guides cover running it well. The results table and the alert router are the platform proper, and most of this guide is about them.

The official runner, the nml run command, reads a configuration file and can run on a schedule. It is convenient for a single model. For a platform I recommend the Python API wrapped in your own job, for a reason that has nothing to do with features: the API is what the documentation covers in depth and what changes least visibly between versions, and a wrapper lets you enforce your results schema, your tenant tags and your logging in one place. Whichever you choose, treat the runner's configuration file as code, review it, and never put credentials in it.

One more architectural fact belongs here, because it shapes security. The library is pandas-based and in-memory. It reads the dataframe you hand it. It does not push computation down to your warehouse and it has no distributed mode. Every capacity decision later in this guide traces back to that sentence.

The statistical contract you are selling

The single most important thing a platform owner does is decide what claims the dashboard is allowed to make. NannyML's headline feature is performance estimation without labels, and its estimates are valid only under stated assumptions. If you present an estimate as a measurement, you will eventually be wrong in front of a stakeholder, and the cause will be one of the assumptions you never mentioned.

CBPE, the classification estimator, works from the model's own predicted probabilities. If those probabilities are calibrated, meaning that among predictions of 0.8 roughly eighty percent are truly positive, then the expected confusion matrix for a chunk can be computed without any labels, and metrics such as ROC AUC, precision and recall follow. It leans on two conditions. The scores must be well calibrated, and there must be no concept drift, meaning the relationship between inputs and the true outcome has not changed. A shift in the inputs is acceptable provided the model has seen that region of the input space; a shift into a region it never saw breaks the calibration assumption, because the scores there were never tested against reality.

DLE, the regression estimator, trains an internal LightGBM model to predict the loss of your model from its inputs and then aggregates that into metrics such as MAE or RMSE. It shares the spirit of the assumption: the loss model learned on reference data must still describe how your model errs in production. It also inherits the cost of training a model, which matters at scale.

Put plainly, there are three situations, and your platform's documentation should name each:

1. Covariate shift inside the known space. The estimator works. This is the case it was built for, and it is where the tool earns its keep.
2. Concept drift. The world changed what the label means: fraud tactics moved, a policy altered who gets approved, a seasonal pattern broke. The estimator is blind to it by construction, because nothing it observes carries the information. Estimated performance can look flat while real performance collapses.
3. Calibration failure. The model was never calibrated, or was calibrated on data unlike production. The estimate is biased from day one, and a reference-period check against known labels will show it.

The practical countermeasure is a calibration gate at onboarding. Before a model joins the platform, run its reference data through the realized-performance calculator and the estimator side by side and require them to agree within the estimator's own confidence band. A model that fails the gate either needs recalibration or should be onboarded with performance estimation switched off and drift monitoring only. Saying "no estimate for this model" is a legitimate platform decision, and far better than a confident wrong one.

Then measure the estimator itself over time. Each time labels arrive for a past period, you hold an estimate and a realized value for the same chunk. Store both, store the gap, and review the gap distribution per model. A model whose gap widens is telling you its calibration or the world has moved, even if no individual alert fired. This is the closest thing the platform has to a health check on its own central claim.

Reference data is a governed artifact

At Beginner level the reference set was a dataframe. At platform level it is the most security-relevant and quality-relevant input you have, because every threshold, every baseline and every alert is defined relative to it. A bad reference set does not produce errors. It produces confident, consistent, wrong monitoring.

Think of the failure modes. A reference period that includes an unusual event, a campaign, an outage or a holiday, bakes that abnormality into the baseline and widens thresholds so real drift hides. A reference set that is too small gives noisy thresholds and floods you with alerts on reference-like data. A reference set silently refreshed to "last month" lets slow drift become the new normal; the monitor learns to accept the very change it exists to catch. And a reference set that someone can edit without review lets an engineer make an alert go away by changing the question.

So treat it like a model artifact. Give each reference set an identifier, a content hash, the date range it covers, the model version it belongs to, the row count and the person or team who approved it. Store it immutably, in object storage with versioning, and refer to it by identifier in the job configuration. When a model is retrained, the reference set is replaced as part of the same change, with the same review. The lineage tools in this catalogue help: MLflow can hold the identifier and hash as run metadata, and a data versioning tool can hold the data itself.

Fitting is the other half. fit() on reference data is cheap compared with calculate() on a long analysis window, but it is not free, and DLE and the domain classifier train models inside it. Fit once per model version and persist the fitted object, so every scheduled run reuses identical baselines. Python's standard pickle module works for this, with the usual caution: unpickling executes code, so only load fitted objects that your own pipeline wrote, from a location your own pipeline controls, and record the NannyML and dependency versions next to the file because a pickle is not guaranteed to load across versions. If that makes you uneasy, refitting from the versioned reference set on every run is a legitimate trade: more compute, no serialized artifact to protect.

import hashlib
import pickle
import nannyml as nml

def fit_and_store(reference_df, model_id, ref_version, path):
    """Fit a CBPE on a versioned reference set and write it out with provenance."""
    digest = hashlib.sha256(
        reference_df.to_parquet(index=True)
    ).hexdigest()

    estimator = nml.CBPE(
        problem_type="classification_binary",
        y_pred_proba="predicted_probability",
        y_pred="prediction",
        y_true="label",
        timestamp_column_name="timestamp",
        metrics=["roc_auc", "f1"],
        chunk_period="W",
    )
    estimator.fit(reference_df)

    with open(path, "wb") as fh:
        pickle.dump(
            {
                "model_id": model_id,
                "reference_version": ref_version,
                "reference_sha256": digest,
                "nannyml_version": nml.__version__,
                "estimator": estimator,
            },
            fh,
        )
    return digest

The weekly period in that example is a deliberate choice: time-based chunks align monitoring with the way people discuss the model ("last week") and make a chunk's meaning stable regardless of traffic. The cost is that a quiet week produces a small chunk with a wide band, which the next section addresses.

Chunking, sampling error and the economics of alerts

Every NannyML metric is computed per chunk, and every chunk carries statistical uncertainty. The documentation expresses the confidence band as three times the sampling error, and sampling error shrinks with the number of rows. This is simple to state and surprisingly hard to operate, because on a fleet the arithmetic of false alarms compounds.

Consider what an alert is. By default thresholds come from reference data: the mean of the metric across reference chunks, plus and minus three standard deviations. Three standard deviations sounds conservative, and for one metric on one chunk it is. But a single model might track three performance metrics and fifty columns with two drift methods each, producing a hundred and fifty series evaluated every week. Even if each series is individually well behaved, with that many independent chances some will cross a threshold by luck alone. Multiply by forty models and "no alerts" is not a realistic expectation, so a team that is paged on every alert will stop trusting the pages within a month.

The platform response is to decide which signals may page a human and which are merely recorded. A workable policy has three tiers:

• Page: estimated or realized performance below a business-set floor for a sustained number of consecutive chunks. A single chunk is a blip; two or three in a row is an event.
• Ticket: multivariate drift alerts, or several univariate alerts on features known to matter. These need a person to look within days.
• Record only: single univariate drift alerts, which feed the ranking and investigation tools but do not interrupt anyone.

Sustained-alert logic belongs in your wrapper, not in NannyML, because the library gives you a boolean per chunk and nothing about persistence. Implementing "three of the last four chunks" over your results table is a few lines of SQL and does more for trust than any threshold tuning.

Chunk size is the other lever, and it trades sensitivity for stability. Too many small chunks and the bands are wide and noisy; too few large chunks and a real shift is averaged away, giving you the opposite symptom, a monitor that never alerts. Choose the chunk from the decision cadence and the volume: how often could anyone act on a finding, and how many predictions arrive in that period? If a model makes two hundred predictions a week, weekly chunks give you bands too wide to mean anything, and you should aggregate monthly or decide the model is too low-volume for statistical monitoring. Document the minimum rows per chunk your platform will accept, and refuse to onboard below it.

NannyML offers three styles: time-based with chunk_period, fixed-size with chunk_size, and fixed-count with chunk_number, with explicit chunker classes such as SizeBasedChunker and CountBasedChunker for control over incomplete leftover chunks. For a platform, time-based chunks are usually right for reporting, and you should pick how incomplete trailing chunks are treated deliberately. An incomplete chunk is a half-filled bucket; if it is allowed into the results it can raise a spurious alert on Monday morning, and if it is dropped you lose the newest data. Many teams run the job after the period closes and drop incomplete chunks, accepting a built-in delay.

Scaling: which component breaks first

Because the library is in-memory pandas, the first thing that breaks is memory, and it breaks in a specific way: the dataframe you pass must hold all columns you configure for the analysis window, plus working copies. A monitoring job for a model with five hundred features and a year of hourly predictions will not fit on a small worker, and it fails with the operating system's out-of-memory killer rather than a friendly exception. Plan for roughly a small multiple of the in-memory size of your analysis dataframe, and measure it on a real run rather than guessing.

The second thing to break is wall-clock time, and where it comes from depends on the calculator. Univariate drift is a loop over columns and methods: cheap per column, but it scales linearly with the number of columns and with row count, and some methods cost more than others. The Kolmogorov-Smirnov test and Wasserstein distance work on continuous values and handle large chunks, while chi-squared and Jensen-Shannon on categorical columns depend on the cardinality. Multivariate drift via PCA reconstruction is moderately expensive. The domain classifier and DLE are the heavy ones, because they train LightGBM models, and on a large analysis window that is real CPU time. If you notice jobs running past their schedule, profile per calculator before reaching for bigger machines; one expensive calculator often accounts for most of the run.

The third limit is I/O and orchestration. Many models on one schedule mean many jobs reading the same warehouse at the same hour. Stagger them, and put the heavy reads behind a query that selects only the columns each model needs.

What you do about it, in order of how much it helps:

1. Select columns. Monitor the features that matter, not all of them. A ranked set from previous drift investigations is better than the full schema, and it cuts both memory and alert noise.
2. Process windows incrementally. Fit once, then calculate on each new period rather than recomputing history. Store results per chunk and append.
3. Pre-aggregate upstream. For very large volumes, sample deterministically in SQL or Spark before the data reaches pandas. Sampling reduces rows but keeps the statistics valid as long as the sample is random with respect to the quantities you measure; do not sample by something correlated with the features, and note in the results that a sample was used, since sampling error then has two sources.
4. Right-size workers per model tier. A scheduler that can assign a memory limit per task, as Kubernetes jobs and most orchestrators can, lets small models run cheaply and large ones get the memory they need. Kubernetes resource requests are the natural fit.
5. Move to a warehouse-native approach only when pandas genuinely cannot cope, and accept that you are then leaving NannyML's calculators behind for those series.

There is no distributed mode to turn on, so "scale out" means more independent jobs rather than a bigger single one. That fits the multi-tenant design below, where each model is its own unit of work.

The platform wrapper

A platform needs one job shape that every model uses. The wrapper takes a model identifier, loads that model's configuration and fitted objects, reads the analysis window, runs the configured calculators, stamps every row with tenant and version tags, writes to the shared results table, and checks its own work. The calculators are the interchangeable part; the wrapper is the contract.

import nannyml as nml
import pandas as pd

def run_monitor(cfg, analysis_df, stored):
    """Run one model's monitoring and return a tagged long-format frame."""
    if len(analysis_df) < cfg["min_rows_per_chunk"]:
        raise RuntimeError(f"{cfg['model_id']}: window too small to monitor")

    estimator = stored["estimator"]
    estimates = estimator.estimate(analysis_df)

    drift_calc = nml.UnivariateDriftCalculator(
        column_names=cfg["features"],
        treat_as_categorical=cfg["categorical"],
        timestamp_column_name="timestamp",
        continuous_methods=["jensen_shannon"],
        categorical_methods=["jensen_shannon"],
        chunk_period=cfg["chunk_period"],
    )
    drift_calc.fit(stored["reference_df"])
    drift = drift_calc.calculate(analysis_df)

    frames = []
    for name, res in (("estimate", estimates), ("drift", drift)):
        df = res.filter(period="analysis").to_df(multilevel=False)
        df["model_id"] = cfg["model_id"]
        df["tenant"] = cfg["tenant"]
        df["reference_version"] = stored["reference_version"]
        df["nannyml_version"] = nml.__version__
        df["kind"] = name
        frames.append(df)

    if not frames or all(f.empty for f in frames):
        raise RuntimeError(f"{cfg['model_id']}: produced no analysis chunks")
    return frames

Two details in that sketch carry most of the value. The row-count guard at the top and the empty-result guard at the bottom turn the silent failure above into a visible one. And the stamped columns, tenant, reference version, library version, make every row traceable, which is what you need the day someone asks why last quarter's chart changed. Wide and long layouts differ between calculators, so decide in the wrapper how each result is flattened into your table rather than leaving it to each team; the to_df(multilevel=False) call gives column names you can map consistently.

Keep the wrapper deliberately boring. Resist adding a plugin system. Three or four patterns cover most models: classification with estimation, regression with DLE, drift-only for models without usable probabilities, and a data-quality pattern using the missing-values and unseen-values calculators. If a team needs something else, the wrapper's configuration should say so explicitly, and you decide whether it is worth supporting.

For the exported charts that go to stakeholders, remember the results are Plotly figures, so .write_html() produces a self-contained file that can be attached to a report or hosted as a static page. Image export needs the separate kaleido package, which is worth pinning in the job image if you rely on it.

Security and the trust model

A monitoring platform reads model inputs and predictions, and that makes it a data-protection system whether or not anyone designed it as one. Features frequently include personal data, financial attributes or health details. The monitoring job needs to read them, the results table may repeat some of them, and the plots can embed raw values. For employers in the Gulf and Egypt, where data-residency and protection rules often constrain where personal data may be processed, the first question is not "does the tool work" but "where does this job run and what does it write down".

Walk the trust boundaries in order.

Inputs. The job's credentials should give read access to exactly the tables or objects for the models it monitors, nothing wider. Use short-lived credentials from your secret manager or workload identity rather than long-lived keys; Vault is one way to issue them. Never place connection strings or keys in a runner configuration file or in code, because both end up in version control. If you use the database dependencies, installed with the nannyml[db] extra, the connection string is a secret like any other.

Compute. Run jobs inside your own network, in the region your data may live in, on workers that are ephemeral and cleaned between runs. Because the library does all its work locally, there is no need for any data to leave your environment for monitoring to function, and you should verify that claim for your own deployment with network egress rules rather than assuming it. The library also sends anonymous usage telemetry unless you opt out; the documentation describes how to disable it. In regulated settings, read the current documentation, disable it explicitly in the job environment, and block outbound traffic that is not required, so the setting is enforced twice.

Outputs. Aggregates are safer than rows, but not automatically safe. A drift chart for a categorical column with a tiny group can disclose that a particular rare category exists in a particular week. Decide per model which columns may appear in shared dashboards, and give the results table the same access controls as the source data for sensitive models. Distribution plots and any exported HTML should be treated as containing data, not as harmless pictures.

Artifacts. A pickled fitted calculator is executable on load, as noted earlier. Write it only from your pipeline, store it where only the pipeline can write, and verify its hash before loading. A fitted calculator also embeds statistics learned from reference data, which can themselves be sensitive for small populations.

The supply chain. NannyML depends on pandas, NumPy, scikit-learn, LightGBM, Plotly, SQLAlchemy-family libraries and Pydantic, so installing it means trusting a sizeable tree. Resolve once, produce a lock file with hashes, build a container image from it, scan the image, and pin the base image by digest. The Docker guide covers the hardening, and the Trivy guide the scanning. If you use the published nannyml/nannyml image, pin it by digest rather than a floating tag, and consider building your own from a locked requirements file, since you then control the Python version and the system libraries, including the OpenMP runtime LightGBM needs on slim base images.

Behind a TLS-inspecting corporate proxy, pip will fail certificate verification when it cannot trust the proxy's CA. The fix is to point pip and Python at the exported CA bundle through the standard environment variables, for example PIP_CERT and SSL_CERT_FILE, never to disable verification, and to bake that CA into your build image so the problem is solved once.

Multi-tenancy and the results schema

NannyML has no concept of a tenant. It knows about one set of calculators, one reference dataset and one analysis dataset at a time. Everything multi-tenant is therefore something you add, and the place to add it is the results schema.

The design that holds up is a single long-format results table with a small set of mandatory columns: a model identifier, a tenant or team identifier, the reference version, the library version, the run identifier, the metric or column name, the method, the chunk key and boundaries, the period, the value, the lower and upper thresholds, the alert flag, and the time the row was written. Long format means one row per chunk per metric or method, which makes queries uniform: the alert-routing query and the dashboard query do not need to know which calculator produced a row.

Isolation then comes from three places. Partition or prefix by tenant so storage and access control align. Apply row-level or table-level permissions in your warehouse so a team sees only its own models. And run each model as an independent job with its own identity, so one team's bad data or crash cannot affect another's results. This is the same reasoning that makes you partition by tenant in any shared store, applied to monitoring output.

Retention is a design decision with privacy consequences. Aggregated metric rows are small and cheap to keep for years, which lets you answer "when did this start" long after the fact. Raw analysis data is large and sensitive and usually should not be kept by the monitoring platform at all: read it from the source of record, compute, write aggregates, discard. If an investigation needs the rows behind an alert, a documented process should fetch them from the source, with the access controls the source enforces.

Naming matters more than it seems. A model identifier must be stable across retrains, with the model version as a separate column, otherwise a retrain breaks the time series. Decide early whether the unit of monitoring is a model, a model version, or a deployment, and write it down. Many painful dashboards come from not deciding.

For dashboards, the results table is a natural source for Grafana or any BI tool, and that is typically where stakeholders look. The platform's value is that every team's charts look the same and mean the same thing, so keep the dashboard definitions as code as well.

Cost

The software is free under its open-source licence, so cost is compute, storage, and people. Compute is the interesting part, because it is easy to overspend invisibly.

The cost drivers, from largest to smallest in most deployments: the training cost of DLE and the domain classifier, which fit LightGBM models; wide drift calculations across many columns; reading large windows from the warehouse, which is billed by whoever owns the warehouse; and idle capacity if you run persistent workers for jobs that take minutes. Batch jobs on ephemeral workers that exist only for the run keep the last one near zero.

Control it by tier. Not every model deserves the same monitoring. A revenue-critical model with fast labels deserves frequent windows, estimation and realized performance. A low-stakes model deserves a weekly drift check on a short list of columns. Writing down the tiers and what each includes lets you answer "why is this model not monitored daily" with policy instead of opinion, and puts the cost conversation in terms leadership can use: this tier costs this much compute per month and covers this many models.

Measure the platform's own cost per model per run, from job duration and worker size, and put it in the same table as the results. Teams change behaviour when they can see the number. Also account for the human cost, which is usually larger than compute: the time spent investigating alerts. An alert policy that wastes engineer hours is the most expensive part of the system, which is another reason to keep the paging tier narrow.

Upgrades, pinning and the maintenance question

Here is a fact you must plan around. The latest stable release at the time of writing is 0.13.1, published in July 2025, and the project was acquired by Soda in June 2025. Soda's announcement states that the open-source library remains available and that a future 1.0 is planned, but a long gap without a release has prompted community questions about activity in the project's issue tracker. The honest posture is: the library is usable and the supported version works, but you are depending on a project whose future cadence you cannot predict, and your platform should be built so that this does not become an emergency.

The upgrade history shows what kind of changes to expect. Recent versions dropped Python 3.7 and then 3.8, added Python 3.12, made the database dependencies optional behind the nannyml[db] extra, updated Pydantic and SQLModel, and added multiclass metrics and a data quality checker. The supported Python range on PyPI is 3.9 to 3.12, and Python 3.13 is not listed, so an organisation that moves its default interpreter forward can find NannyML unavailable on the new default. None of that is dramatic, but it shows that dependency pressure, not feature work, is what moves this library now.

So the policy is:

• Pin everything. nannyml==0.13.1 plus a full lock file covering pandas, NumPy, scikit-learn, LightGBM and Pydantic. Treat the lock file as the unit of upgrade, not the single package.
• Pin the interpreter. Use a Python version the library declares support for, currently 3.11 or 3.12 are sensible choices, and make the container image the single place that version is set.
• Run a contract test in CI. A small fixed dataset with known expected outputs, run on every dependency change. NannyML's sample datasets are useful here. The test asserts that the key results, chunk counts and alert flags, stay within tolerance. When a transitive dependency update changes numerical behaviour, this is how you find out before production does.
• Watch the project. Subscribe to the repository's releases and to the vendor's announcements. A new major version may break APIs, so schedule a deliberate evaluation instead of upgrading casually.
• Keep an exit. Isolate NannyML behind your wrapper so that replacing a calculator with another implementation, or with your own, touches one module. The wrapper's results schema is the interface; if the library ever stalls, your dashboards and alerting survive.

Migration between your own platform versions follows the same logic. When you change the results schema, add columns rather than renaming them, version the schema, and backfill. When you change a reference set or chunk setting, mark the discontinuity in the results so a chart does not present a methodological change as a real-world change.

Governance and ownership

Governance is the part of platform work that does not show up in code and decides whether the platform lasts. Four questions need explicit answers.

Who owns a model's monitoring? The model owner owns the response to alerts; the platform team owns the machinery. State this in a short onboarding document, and make accepting it a condition of joining. Alerts routed to a platform team that cannot act on them are a recipe for resentment.

Who may change thresholds, chunking or the reference set? A change that loosens monitoring should require a reviewer who is not the person asking, with a recorded reason. A change that tightens it can be lighter. Keep the configuration in version control so the review is the pull request.

What is the service level? State it plainly: the job runs on this schedule, results land within this delay, missing runs are detected within this time. Teams plan around what you promise, so promise what you can measure. Include the monitoring of the monitor: a heartbeat that records each job's last successful run and an alert when it is stale.

What does onboarding require? A checklist: reference data identified and approved, minimum volume met, calibration gate passed or estimation disabled, owner and escalation named, label latency documented, sensitive columns classified. Onboarding is where you prevent most future incidents.

Label latency deserves its own line. How long between a prediction and its true label determines what the platform can honestly offer. Where labels arrive in days, realized performance is the main signal and estimation is a bonus. Where labels arrive in months or never, estimation and drift are the only signals, and the limits in the earlier section matter most. Record the latency per model, and show it on the dashboard so nobody assumes labels are faster than they are.

When drift is detected, the investigation workflow should be standard. NannyML's ranking tools help here: ranking drifting features by the number of alerts, or by their correlation with changes in performance, gives investigators a starting point. Link the investigation to the model's lineage, and for model-level questions the Evidently guide compares a neighbouring tool, and the Alibi Detect guide covers detectors that work at the instance level, which are different instruments for different questions.

Incident playbooks

Platform incidents come in two families: the monitoring says something is wrong with a model, and the monitoring itself is wrong. Both need playbooks written before the night they are used.

Playbook: a performance alert fires.

1. Confirm the alert is real. Check the chunk size and row count: is this a thin chunk with a wide band? Check whether it is the first alert or the third consecutive one.
2. Check the data pipeline before the model. A broken upstream join, a feature that suddenly arrives as null, a unit change, a late-loading table: these cause most "drift" and look dramatic. The data quality calculators for missing and unseen values help rule them in or out quickly.
3. Check the estimator's assumptions. If realized labels exist for earlier chunks, compare estimate and realized for them. If they disagree, the estimator is the suspect and the alert may be an artifact.
4. Rank the drifting features and look at their distributions with the distribution plots, comparing against reference.
5. Decide: rollback, retrain, recalibrate, or accept. Record the decision and the evidence in the model's history.

Playbook: drift on one feature, performance flat. Common and often benign. If the feature has low importance to the model, note it and let the ranking track it. If it is an important feature and estimation says performance is stable, check for concept drift by looking for realized labels; flat estimates are exactly what concept drift would hide.

Playbook: estimates and realized performance disagree. The estimator's contract has failed. Look first for calibration: plot reliability on the newest labelled data. Look next for concept drift. Report plainly to the model's owner that estimation is unreliable for this model and switch the dashboard to realized metrics until it is understood.

Playbook: the monitor went quiet. No results for a model. Check the scheduler, the upstream table's freshness, the job's logs for the empty-window guard, then credentials expiry. This is the incident the heartbeat exists to catch, and the recovery step is a backfill: because results are stored per chunk and the job is incremental, rerunning for the missed period should be safe and idempotent. Make that a design requirement: running the same window twice must not duplicate rows, so key results on model, chunk and run identity and upsert.

Playbook: a flood of alerts across many models at once. Suspect something shared before suspecting many models. A change in a common upstream feature table, a time-zone shift, a daylight-saving change, a new default in a shared library: these hit everyone together. Check for a platform change on the same day, including your own upgrade.

Playbook: the job fails after a dependency change. Roll back to the previous image by digest, run the contract test, and bisect the lock file change. This is why deploying only from images built from lock files matters; the rollback is one reference.

Where NannyML stops

Knowing when not to use a tool is part of owning it. NannyML is strong at one thing: estimating classification and regression performance without labels, and tying drift to it. It is not the right choice in several situations.

• Large language model applications. Its calculators assume tabular features, probabilities or numeric predictions. Evaluating generated text, retrieval quality or agent traces needs tools built for that; the Langfuse and RAGAS guides cover those.
• Real-time or per-request detection. It is a batch tool over chunks. If you need to flag an individual input as out of distribution at serving time, look at instance-level detectors such as those in Alibi Detect.
• Very large volumes without sampling. Because it is in-memory, a warehouse-native approach or sampling pipeline is needed.
• Concept drift as the main risk. Where the label definition itself shifts quickly, label-free estimation is least reliable, and you need fast labels and realized metrics.
• Teams that want a monitoring UI out of the box. A tool with reports and a service interface may serve them faster; the Evidently guide describes one. The commercial NannyML Cloud product is a separate offering and this guide does not cover it.
• A strict long-term support requirement. Given the release cadence, if your organisation needs a vendor-backed support commitment for open-source components, evaluate that before standardising.

Choosing something else is not a failure of the platform. A good platform team can say "for that use, use this" and link to it.

Running it as a platform for other teams

Pulling the threads together, the thing you are building is a small product with customers. It has an interface, an agreement and a feedback loop.

The interface is the onboarding configuration and the results table. A team gives you a model identifier, a reference set, a column mapping, a problem type, chunk settings and an owner; they receive a dashboard and an alert feed. Keep the interface small and stable. Every option you expose is something you must support through upgrades.

The agreement is the service level, the alert policy and the statistical contract. Publish it: what the estimate means, when it should not be trusted, how quickly results arrive, what happens when the job fails. A one-page document that teams must read before onboarding saves hundreds of questions later.

The feedback loop is how the platform improves. Review the alert log monthly: which alerts led to action, which were ignored, which were missed. Review the estimator gap per model. Review cost per model per run. Review onboarding time. These four numbers tell you whether the platform is working, and they are the basis for any case to leadership for more investment or for retiring a part that is not paying for itself.

Enablement matters as much as machinery. Offer a template repository with the wrapper, an example configuration and a contract test, so a new team gets a working monitor in an afternoon. Run a short session on reading the charts, because teams that misread a confidence band will either panic or ignore it. Point them to the earlier levels of this guide and to related material: MLflow for tracking models and reference lineage, Airflow or Prefect for scheduling, and Kubernetes for running the jobs.

Finally, plan the platform's own retirement or replacement. Everything about the wrapper, the schema and the policy is meant to outlive any one library. If in two years you replace NannyML's calculators with another tool's, the teams should see a changelog entry and not a migration project.

The review checklist

Use this for a design review of a new model joining the platform, and again quarterly for existing ones.

Statistical fitness

• The problem type is supported, and probabilities are available if estimation is wanted.
• Calibration was checked on reference data against realized labels, and the result is recorded.
• Minimum rows per chunk are met at the chosen chunk size, and the expected band width is acceptable.
• Label latency is documented, and the dashboard shows it.

Reference data

• It has an identifier, a hash, a date range and an approver.
• It excludes known anomalies, or the reason for including them is recorded.
• It is immutable and stored with versioning, and replacing it follows the same review as a retrain.

Operations

• The job is idempotent and incremental, and guards against empty and short windows.
• A heartbeat exists and an alert fires on a stale run.
• Memory and duration were measured on a real run and the worker is sized accordingly.
• Re-running a window does not duplicate rows.

Security and data

• The job's identity has the minimum read access and writes only to its own prefix.
• No secrets in configuration files or code.
• Usage telemetry is disabled in the job environment where policy requires it, and egress is restricted.
• Pickled artifacts are written and read only by the pipeline, with a hash check.
• The job runs in an approved region for the data it reads.

Supply chain and upgrades

• The image is built from a lock file, the base is pinned by digest, and the image is scanned.
• Python is within the supported range.
• A contract test runs on every dependency change.

Alerting and governance

• Only sustained performance breaches can page, and the owner is named.
• Threshold loosening needs an independent reviewer.
• Alert volume per model is tracked.

The complete picture

Taken together, the three levels describe one idea at three depths. At Beginner you learned that NannyML turns a production dataset into a time series of chunks and estimates how a model is doing even without labels. At Mid-level you learned to choose chunks, thresholds and calculators deliberately and to read the results with their uncertainty. At this level you learned to treat the whole thing as a service: a governed reference artifact, a stated statistical contract, a single job shape, a tenant-aware results table, a security posture, a cost model, and a plan for the library's uncertain future.

The recurring principle is honesty about uncertainty. The estimator is honest only when its assumptions hold; the alert is honest only when chunk size and policy are chosen with the false-alarm arithmetic in mind; the platform is honest only when it tells teams what it can and cannot see. A monitoring system that overclaims is worse than none, because it teaches people to stop looking.

Where the series leaves you

You can now design, defend and run a model monitoring platform built on NannyML, and also say where it should not be used. In an interview or a design review you can explain the estimator's assumptions, choose chunking from volume and decision cadence, size the system, secure it for regulated data, and describe the incident playbooks.

From here the useful next steps sit beside this guide. Read Evidently to compare approaches, Alibi Detect for instance-level detection, MLflow for the lineage your reference sets need, and Airflow or Prefect for the orchestration around the jobs. Then pick one real model in your organisation and run it through the onboarding checklist above. The gaps you find are the platform's backlog.

Sources

• NannyML documentation (stable)
• Quickstart
• Binary performance estimation with CBPE
• Detecting data drift
• Univariate drift detection
• Chunking
• nannyml on PyPI
• NannyML on GitHub
• NannyML releases
• Soda acquires NannyML
• LightGBM installation guide

Part three of three. At this level an interviewer is not checking whether you can name a calculator. They want to know whether you can design a monitoring service for other teams, defend what it claims, respond when it is wrong, and argue its cost and limits to people who do not care about chunking. Read the model first, then the design prompts, the incident questions, the leadership arguments, and finish with the self-test.

The ninety-second answer

If you only get one chance to describe NannyML at senior level, say this.

NannyML is an open-source Python library for post-deployment monitoring of tabular models. Its distinctive capability is estimating performance without labels: CBPE for classification uses calibrated probabilities, DLE for regression learns the model's loss. It also does univariate and multivariate drift detection and data quality checks, all computed per chunk against reference data. It is a batch library, not a service, so running it for other teams means building the scheduling, results schema, alerting and access control around it, and being honest that estimates hold only when their assumptions do.

Notice what that answer does. It names the headline feature, states the architecture, and ends with the limitation. Interviewers at this level reward the candidate who volunteers the limitation, because it shows you have been burned by it or have thought hard enough to predict it.

The model to carry into every answer

Three ideas organise almost every senior question.

The estimate is conditional. CBPE needs calibrated scores and no concept drift. Covariate shift inside the region the model has seen is fine; a shift into unseen territory or a change in what the label means is not. DLE needs the learned loss model to still describe how the model errs. Whenever someone asks "can we trust this number?", the answer starts with those conditions.

An alert is a statistical event, not a fact. Thresholds default to the reference mean plus and minus three standard deviations, and the confidence band is three times the sampling error. With many metrics, columns, chunks and models, some alerts will occur by chance. Your policy decides which ones reach a human.

The tool is a library, so the platform is yours. Nothing in the library schedules, stores, routes or secures. The wrapper, the results table, the heartbeat, the tenant boundaries and the upgrade policy are the engineering you are being hired for.

Add a fourth for the current period: the upstream project's cadence is uncertain. The latest stable release is 0.13.1 from July 2025, the company behind it was acquired by Soda in June 2025, and Python 3.13 is not a declared target. A senior answer plans for that without drama.

System-design prompts

Prompt 1: monitor forty models across six teams

"Design a monitoring service on NannyML for forty production models owned by six teams. Some have labels within days, some never."

A strong answer starts by classifying the models before drawing anything. For each, you need the problem type, whether probabilities are exposed, label latency, prediction volume and data sensitivity. Those five facts decide what each model can be offered: estimation plus realized metrics, drift only, or something lighter.

Then describe the shape. One job template, parameterised by a small versioned configuration per model, runs on your orchestrator on a schedule. It reads the analysis window from the source of record, loads fitted objects built from a versioned reference set, runs the configured calculators, stamps rows with model, tenant, reference version and library version, and appends to a long-format results table. A guard fails the run on empty or short windows. A heartbeat table records the last successful run per model. An alert router reads the results table, applies sustained-breach logic, and routes by owner.

Name the first thing that breaks. Memory, because everything is in-memory pandas, so you size workers per model tier and select columns instead of passing the full schema. Next, wall-clock time from the calculators that train models, DLE and the domain classifier. Then scheduling collisions when forty jobs hit the warehouse at the same hour, so you stagger them.

A weak answer draws a dashboard and a box labelled NannyML. The missing parts are the ones that decide whether the service works: tenancy, guards, the heartbeat, and what happens on models that fail the calibration gate.

Prompt 2: what should the reference data policy be?

"A team wants to refresh their reference data monthly to keep it current. Do you allow it?"

This is a trap about silent normalisation. Refreshing the reference to the latest month means slow drift becomes the new baseline, and the monitor stops flagging the change it exists to catch. A strong answer says: the reference represents a period when the model was known to behave well, so it changes when the model changes, with review, not on a calendar. If the team's concern is seasonality, the answer is a reference that covers a full cycle, or separate references per season, or a comparison to the same period a year earlier, not a rolling window.

Then say how you would govern it: an identifier, a content hash, a date range, a model version, an approver, immutable storage with versioning, and replacement as part of the retrain change. Mention that a reference set that anyone can edit lets an engineer silence an alert by changing the question.

Prompt 3: regulated data

"The model scores loan applicants and the features contain personal data. Your employer is in the Gulf and data must stay in-country. How do you deploy monitoring?"

Everything runs inside the approved region and network. The library computes locally, so no data needs to leave, but you verify that with egress rules rather than trusting it, and you disable the library's anonymous usage telemetry through the setting described in the current documentation. Credentials are short-lived and scoped per model. The results table holds aggregates, but you assess whether small categorical groups in drift plots could identify people, and you apply source-level access controls to the results for sensitive models. Raw analysis data is not retained by the platform. Exported HTML plots are treated as data. Pickled fitted artifacts are written and loaded only by the pipeline and hash-checked, since unpickling executes code.

Show that you know a second concern: the reference set and fitted baselines are derived from personal data too, so they live under the same retention and residency rules as the source.

Prompt 4: five hundred features, hourly predictions

"A model has five hundred features and predicts hourly. The job runs out of memory. What do you do?"

First, measure: one real run, with the memory profile per calculator. Then in order of leverage. Select columns, using feature importance and past drift investigations to monitor a ranked subset. Process incrementally, so the job holds one period rather than a year. Choose chunking from decision cadence, since hourly chunks of a few hundred rows give wide bands and daily might be right. Sample deterministically upstream in SQL if needed, noting that sampling adds uncertainty. Assign a larger memory limit through the orchestrator for this model's tier. Mention what you would not do: bolt on a distributed engine that the library does not support.

Prompt 5: replacing NannyML

"In two years you may need to swap the library out. How did you design for that?"

The wrapper and the results schema are the interface. Dashboards and alert logic read the table, not the library's objects. Configuration is expressed in your terms, not the library's. A contract test with a fixed dataset pins expected outputs, so a replacement can be compared against it. The cost of a swap is then one module and a validation exercise, not a migration project for every team.

Incident-response questions

The estimate looked fine and the model failed

"Last week estimated ROC AUC was flat. Labels arrived and the realized value had dropped sharply. What happened and what do you do?"

The first hypothesis is concept drift, because it is the case the estimator cannot see: nothing in the inputs and scores carries the information that the label's relationship to them changed. The second is calibration failure in the new region. Show your reasoning order. Compare estimated and realized for earlier labelled chunks to see whether the gap is new or chronic. Check the reliability of scores on the newest labelled data. Look for a business event, a policy change, a new fraud pattern, a change in how labels are produced. Then act: report to the model owner that estimation is unreliable for this model right now, switch the dashboard to realized metrics, and open a retraining or recalibration decision.

The follow-up they are listening for is what you change in the platform afterwards: the estimator gap becomes a stored, reviewed metric per model, and the dashboard shows realized beside estimated whenever labels exist.

Alerts everywhere at once

"At 03:00 on Monday, thirty models alert simultaneously."

Suspect something shared. Look for a common upstream change: a feature table's pipeline, a time-zone or daylight-saving shift, a schema change, your own platform change or dependency upgrade. A flood across unrelated models is almost never thirty simultaneous real shifts. Check whether the alerts are on the same feature names, whether they are first-chunk alerts caused by an incomplete trailing chunk, and whether the job image changed. Contain it by pausing paging, not monitoring, so history stays complete, and tell owners what you know. Then fix the cause and backfill. Afterwards, add a platform-level rule: if more than a set fraction of models alert in one run, page the platform team instead of thirty owners.

The monitor went silent

"A model's dashboard has not changed in two weeks. Nobody noticed. How did that happen, and how do you prevent it?"

A job that read an empty window and finished without error, an expired credential that the orchestrator swallowed, a schedule disabled during a migration. The cause matters less than the detection gap. The prevention is a heartbeat per model with an alert when the last successful run is older than the schedule allows, guards that fail on empty or short windows, and a monthly review of models with no new chunks. State the principle: a monitor that stops producing results looks identical to a healthy model.

A dependency upgrade changed the numbers

"After a routine image rebuild the drift scores shifted slightly and a few borderline alerts appeared. Is the model worse?"

Probably not. Suspect the environment first. Compare the lock files of the old and new images, roll back by digest, and run the contract test on both. The root cause is typically a transitive dependency changing numerical behaviour. The platform lesson is that images are built from lock files with hashes, upgrades happen as deliberate changes with the contract test as the gate, and results are stamped with library version so a methodological discontinuity is visible on the chart rather than read as a real-world change.

A team asks you to loosen thresholds

"A team is tired of alerts and wants wider thresholds. What do you say?"

Ask what the alerts are costing them and what they ignore. Often the right fix is not the threshold but the policy: page only on sustained breaches of a business floor, send single drift alerts to a ranked list, and check whether chunks are too small for the volume. If a threshold does need to move, it goes through a pull request on the configuration, with a reviewer who is not the requester and a recorded reason, and the change is marked on the chart. Loosening because of annoyance, with no analysis, trains the platform to be ignored.

Trade-off questions

Estimate or wait for labels? Estimation gives an early warning when labels are slow, at the price of its assumptions. Realized metrics are the truth but late. A mature answer uses both, and shows the realized series beside the estimate as soon as labels arrive.

Small chunks or large chunks? Small chunks detect shifts sooner and produce wider bands and more false alarms. Large chunks are stable and can average a real shift away. Choose from decision cadence and volume, and refuse to onboard models too low-volume for the cadence wanted.

Univariate or multivariate drift? Univariate is interpretable and points to the column. Multivariate catches changes in relationships that no single column shows, but tells you less about where. Use multivariate as the cheap tripwire and univariate for investigation, and be clear that neither alone says the model got worse.

Library API or the nml run runner? The runner suits a single model and a simple schedule. A wrapper around the API suits a platform, because you control tenant tags, guards, logging and the results schema. The cost is that you own the wrapper.

Build on NannyML or use a heavier monitoring product? NannyML is lightweight and unique in label-free estimation, but you build the service. A product with a UI gets teams started sooner and constrains you more. The right answer depends on team size and appetite to own a platform, and you should say which you would choose for your organisation and what would change your mind.

Arguing a trade-off to leadership

Leadership questions test whether you can turn technical constraints into decisions. Keep each answer to the structure: what we recommend, what it costs, what risk remains, what we will measure.

"Why do we need this when we have dashboards for accuracy?" Because accuracy needs labels and labels arrive late, sometimes months late. In that gap the model can degrade unseen. This tool gives an early estimate and shows which inputs changed. Be candid: the estimate is an early warning with stated assumptions, and we confirm it with real labels when they come.

"Why not monitor everything daily?" Cost and noise. Each added metric and column adds compute and false alarms, and teams that get noisy pages stop reading them. We tier models by business impact and label speed, give high-impact models frequent monitoring and realized metrics, and give low-impact ones a lighter weekly check. The tiers are written down so the cost is a decision, not an accident.

"Is it safe to depend on this library?" It is open source under a permissive licence, the supported version works, and we have pinned and tested it. The honest risk is that the last release was in July 2025 and the company was acquired, so future cadence is uncertain. We mitigate by pinning the full environment, running a contract test on every dependency change, and keeping the library behind our own wrapper so replacement is a bounded project. We will review that risk each quarter.

"How do we know the monitoring itself is working?" We monitor the monitor. Every model has a heartbeat, every run checks its own row counts, and we track the gap between estimates and realized performance. If the estimator drifts from reality for a model, we see it and stop presenting estimates for that model.

"Can we put this on the sensitive customer model?" Yes, inside our approved region and network, with scoped short-lived credentials, telemetry disabled and egress restricted, no raw data retained, and results access-controlled like the source. The conversation to have with the security and data-protection teams is about aggregates in shared dashboards and small categorical groups, and we will bring that to them rather than waiting to be asked.

What strong answers cover that weak ones miss

A strong senior answer reliably does these things. It states assumptions before claims. It treats reference data as a governed artifact. It distinguishes the library from the platform and lists what the platform adds. It reasons about false-alarm arithmetic across a fleet. It names what breaks first at scale. It brings up security and data residency without being asked. It has a plan for the project's uncertain future. It names when to use another tool, pointing to Evidently for report-style monitoring, Alibi Detect for instance-level detection, and Langfuse for LLM applications.

A weak answer lists calculators, recites the three-standard-deviation rule without noticing it causes fleet-wide noise, treats estimated performance as measured performance, assumes refreshing the reference is harmless, and has no story for what happens when the job silently stops.

Sixty-second self-test

Answer each aloud without notes. If you hesitate, reread the matching section.

1. State the two assumptions behind CBPE and what happens when each fails.
2. Why does refreshing the reference data monthly defeat the monitor?
3. Explain why forty models with a hundred metrics each cannot be expected to have no alerts, and what you do about it.
4. Name the three things you stamp on every results row and why.
5. What breaks first at scale, and what are your three first moves?
6. List the five trust boundaries in the security walk and one control for each.
7. What do you say when asked whether the library is safe to depend on?
8. Give the playbook for a flood of simultaneous alerts across unrelated models.
9. When would you recommend a different tool, and which?
10. Describe the heartbeat and the two guards that prevent a silent monitor.

If you can answer all ten crisply, you are ready for the senior conversation on this tool. The senior tips file holds the production traps and checklists behind these answers.

Part three of three. At this level the problems are not failed imports. They are monitors that run green for months while saying nothing true, alert streams that teach people to stop reading, and a library whose upstream has gone quiet. Start with the table of production traps, then the habits of platform ownership, the review checklists for pull requests and design documents, and the operational recipes.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These appear when many models, many teams and many months are involved.

Two of these deserve a longer word because they cause the most damage.

The silent empty run. A job that reads nothing and writes nothing usually ends with exit code zero. The orchestrator is green, the dashboard has a line that simply stops, and a stopped line looks like a stable model. Two guards fix it: fail when the analysis window has fewer rows than your documented minimum, and fail when the run produces no analysis chunks. Then the heartbeat catches the cases the guards cannot, such as a schedule that was disabled and never ran at all.

The estimate presented as a measurement. The mistake is rarely in the code. It is in how the number is shown. If the chart does not say "estimated" and does not state the assumptions, a stakeholder reads a flat line as a guarantee. Put the word on the axis, put the assumptions under the chart, and put the realized series on the same chart whenever labels exist. This single presentation habit prevents the conversation where a team feels the tool lied to them.

Traps in how teams use the results

Treating a single alert as an incident. One chunk outside a three-standard-deviation band is a statistical event. Two or three consecutive chunks is a pattern. Encode persistence in your router and say so in the policy, so a Monday-morning blip does not wake anyone.

Reading univariate drift as performance impact. A feature can drift hard without changing performance, and performance can fall with no single feature alerting. Use the ranking tools to connect the two, and treat a drift alert as a reason to look, not a verdict. The ranker that orders features by correlation with performance change gives investigators a better starting point than alert counts alone, but both need realized or estimated performance to exist.

Chasing the alert by loosening the band. The reflex when alerts are noisy is to widen thresholds. That treats the symptom. Check chunk size against volume first, then the reference period, then whether too many low-value columns are monitored. A loosened threshold that was not analysed is a monitor with a hole in it.

Comparing results across chunk settings. A change in chunk period or size changes the sampling error and the number of points. A chart that spans the change misleads. When settings change, mark the discontinuity in the results and the dashboard.

Forgetting that labels are late. The most recent realized chunk is always older than the most recent estimated one. If the dashboard puts them on a shared axis with no marker, readers see a realized line that "stops early" and assume a fault. Label the latency.

Platform-ownership habits

These are habits, not tools. They are what separates a platform that lasts from a script that spread.

1. Write the statistical contract down. One page: what CBPE and DLE assume, when they fail, and what the platform will do when they do. Link it from every dashboard. When someone is surprised, you point to a document everyone agreed to, not a conversation nobody remembers.
2. Treat the reference set as a release artifact. Identifier, hash, date range, model version, approver, immutable storage. Replace it only with the model.
3. Put configuration in version control. One small file per model: reference identifier, chunk settings, tier, owner, minimum rows. Loosening monitoring needs an independent reviewer; tightening it can be light.
4. Keep one results schema. Long format, with model, tenant, reference version, library version, run identifier, metric, method, chunk boundaries, value, thresholds, alert flag and write time. Add columns, never rename them.
5. Monitor the monitor. Heartbeat, freshness and completeness, plus the estimator gap. Review them monthly.
6. Tier the models. Write down what each tier gets and costs. Cost conversations then become choices.
7. Pin the world. Locked, hashed dependencies, a pinned interpreter in the supported range, an image built from the lock file, a base image pinned by digest. Never pip install at job start.
8. Keep an exit. The library sits behind your wrapper; your dashboards and alerts read your table. If the project goes quiet or a 1.0 breaks APIs, the change is one module.
9. Review the alert log. Monthly: what led to action, what was ignored, what was missed. Keep a record of the monitor's own mistakes and their causes.
10. Teach the reading. A short session on confidence bands and the estimated-versus-realized chart prevents panic and indifference alike.

Security habits that cost almost nothing

• One identity per model. Read access to its own inputs, write access to its own results prefix. A shared monitoring super-user cannot be audited and cannot be revoked per team.
• No secrets in configuration. Connection strings and storage credentials come from a secret manager or workload identity. The runner's configuration file and your repository should contain only references. The Vault guide covers one way to issue short-lived credentials.
• Disable usage telemetry where policy requires it, using the setting in the current documentation, and enforce it again with egress rules. A setting in a file is a hope; a blocked route is a control.
• Trust pickles only from your pipeline. Unpickling runs code. Write fitted objects from the pipeline into a location only the pipeline can write, check a hash before loading, and record versions next to the file.
• Treat exported plots as data. HTML exports from Plotly embed the values behind the chart. A distribution plot for a column with a rare category can disclose that the category exists. Classify columns before they reach shared dashboards.
• Stay in region. Run jobs and store results where the data is allowed to live. For many Gulf and Egyptian employers that is a contractual or regulatory line, and it is far easier to hold from the start than to move later.
• Scan the image. NannyML pulls a large dependency tree. Scan with a tool such as the one in the Trivy guide, gate on fixable high-severity findings, and rebuild on a schedule so the base does not age.
• Fix TLS, do not bypass it. Behind a TLS-inspecting proxy, point pip and Python at the exported corporate CA through PIP_CERT and SSL_CERT_FILE and bake that CA into the build image. Disabling verification to make an install pass is how a supply-chain incident starts.

Review checklist for a pull request

Use this on any change to the wrapper, a model's configuration, or the job image.

Configuration changes

• Does it loosen monitoring: wider thresholds, larger chunks, fewer columns, estimation switched off? If so, is there a reviewer other than the author and a written reason?
• Does it change the reference set? Is the identifier, hash and approver recorded, and is it tied to a model version?
• Does it change chunk settings? Is the discontinuity marked in the results?
• Is the minimum row count still met at the new chunk size?

Wrapper code

• Are the empty-window and empty-result guards still in place?
• Is the job idempotent: does running the same window twice upsert instead of duplicating?
• Are the mandatory columns stamped on every row?
• Is anything read from the environment that could carry a secret into a log?
• Are pickled artifacts only loaded from the approved location, after a hash check?

Image and dependencies

• Is the change built from the lock file, with the base pinned by digest?
• Does the contract test pass, and are its tolerances unchanged? A loosened tolerance in a dependency PR is a smell.
• Is the Python version inside the declared range for the pinned NannyML release?
• Did the library version in the results change, and does the dashboard mark it?

Operations

• Was memory and duration measured on a realistic run for any model whose data volume changed?
• Does the heartbeat cover the new job?
• Is the alert route and owner named?

Review checklist for a design document

Use this when a team proposes monitoring a new model, or proposes a new platform feature.

1. Fitness. Is it tabular? Are probabilities exposed? Is the volume high enough per chunk at the cadence they want? If not, the answer may be a lighter tier or a different tool.
2. Calibration. Was it checked against reference labels? What was the gap between estimated and realized on the reference period?
3. Labels. What is the latency, and what share of predictions ever receives a label? Do they know realized metrics will lag?
4. Reference. Which period, why that one, who approved it, how will it be replaced?
5. Alert policy. What pages, what tickets, what is only recorded? Who owns the response? What business floor defines "bad"?
6. Data protection. Which columns are sensitive, where does the job run, who can read the results, what is retained?
7. Cost. Which calculators, how often, on which worker size, and what is the monthly compute per model?
8. Failure. What happens if the job does not run, runs empty, or the library is unavailable on a new interpreter?
9. Exit. If NannyML were replaced, what would change for this team?
10. Alternatives. If the model serves text, needs per-request detection, or needs a built-in UI, why is this the right tool? The Evidently and Alibi Detect guides cover neighbours.

A design document that cannot answer items 2, 3 and 5 is not ready, however polished its architecture diagram.

Operational recipes

Recipe 1: the contract test

Keep a small fixed dataset and a script that runs one estimator and one drift calculator on it, then asserts chunk counts, alert flags for known chunks and key values within a tolerance. Run it in CI on every dependency or image change.

import nannyml as nml

def test_cbpe_contract():
    reference, analysis, _ = nml.load_us_census_ma_employment_data()
    est = nml.CBPE(
        problem_type="classification_binary",
        y_pred_proba="predicted_probability",
        y_pred="prediction",
        y_true="employed",
        timestamp_column_name="timestamp",
        metrics=["roc_auc"],
        chunk_size=5000,
    )
    est.fit(reference)
    out = est.estimate(analysis).filter(period="analysis").to_df(multilevel=False)

    assert len(out) > 0, "no analysis chunks produced"
    assert out.shape[0] == EXPECTED_CHUNKS            # recorded from a known-good run
    assert out["alert"].sum() == EXPECTED_ALERTS      # ditto

Record EXPECTED_CHUNKS and EXPECTED_ALERTS from a known-good run and store them with the test. The test's value is that it fails loudly when a dependency changes behaviour, before production does. Check the exact output column names against your installed version when you first record the expectations, since the flattened column names come from the library.

Recipe 2: the heartbeat query

Keep a table with one row per model and the end date of its newest chunk and the time of its last successful run. A scheduled query finds the stale ones.

SELECT model_id, tenant, MAX(chunk_end) AS newest_chunk, MAX(written_at) AS last_run
FROM monitoring_results
GROUP BY model_id, tenant
HAVING MAX(written_at) < CURRENT_TIMESTAMP - INTERVAL '2' DAY;

Adjust the interval per tier, and route the result to the platform team, not the model owners, since a stale monitor is your fault before it is theirs. The interval syntax differs between warehouses.

Recipe 3: sustained-breach logic

A page requires a metric below its floor in several consecutive chunks. Over the results table, order chunks per model and metric and test the last few.

WITH recent AS (
  SELECT model_id, metric, chunk_end, value, lower_threshold,
         ROW_NUMBER() OVER (PARTITION BY model_id, metric ORDER BY chunk_end DESC) AS rn
  FROM monitoring_results
  WHERE kind = 'estimate' AND period = 'analysis'
)
SELECT model_id, metric
FROM recent
WHERE rn <= 3
GROUP BY model_id, metric
HAVING SUM(CASE WHEN value < lower_threshold THEN 1 ELSE 0 END) = 3;

Substitute a business-set floor column for lower_threshold when the policy is a floor rather than the statistical band. Note that this assumes your wrapper writes kind and period columns, which are your own schema choices, not library fields.

Recipe 4: onboarding a model in an afternoon

1. Copy the template repository and fill in the model's configuration file.
2. Register the reference set: identifier, hash, date range, approver.
3. Run the calibration gate: fit the estimator, compare with realized on a labelled period, and record the gap.
4. Run the contract test and one dry run on a recent window, recording memory and duration.
5. Open the pull request; the reviewer works from the pull request checklist above.
6. After merge, confirm the heartbeat shows the first run and the dashboard shows the new model with its tier and label latency.

Recipe 5: investigating an alert

1. Check chunk row count and whether the chunk is complete.
2. Check data quality first with the missing-values and unseen-values calculators; broken pipelines look like dramatic drift.
3. Compare estimated and realized for earlier chunks to judge whether the estimator is currently trustworthy.
4. Rank drifting features by alert count and by correlation with performance, and read their distribution plots against reference.
5. Decide and record: rollback, retrain, recalibrate or accept, with the evidence attached to the model's history.

Recipe 6: upgrading the library

1. Read the release notes and the issue tracker for the target version.
2. Build a candidate image from a new lock file in a branch, with the interpreter still in the supported range.
3. Run the contract test, then a shadow run: the candidate writes to a separate table for a week while the current image keeps writing to production.
4. Compare the two tables for chunk counts and alert flags, and investigate every difference.
5. Promote by digest, mark the version change on dashboards, and keep the previous image for rollback.

The short checklist

Keep this on the wall of whoever owns the platform.

• The statistical contract is written, linked from dashboards, and agreed.
• Every model has a calibration result, a label latency and an owner.
• Reference sets are versioned, hashed, approved and immutable.
• Configuration lives in version control and loosening needs independent review.
• One results schema, additive changes only, every row stamped.
• Only sustained performance breaches page; everything else is ranked and recorded.
• Guards on empty windows and empty results; heartbeat per model.
• Memory and duration measured per tier; workers sized to match.
• One identity per model; no secrets in files; telemetry disabled where required; egress restricted.
• Images built from lock files, base pinned by digest, scanned, interpreter in range.
• Contract test on every dependency change; shadow run before upgrades.
• The wrapper isolates the library so replacement is one module.
• Monthly review of alerts, estimator gaps, cost per model and onboarding time.

Where a check feels like overhead, remember who it protects: the next team that will trust a flat line on a chart, and the engineer who will be asked, months later, why it said what it said.