The Complete BentoML Guide

This is part one of three. It covers everything you need to take a model that works in a notebook and turn it into an HTTP service someone else can call, packaged as an artifact you can hand to a colleague or a cluster. By the end you can write a BentoML service, serve it locally, call it from curl and from Python, save a model into BentoML's store, build a versioned Bento, turn that Bento into a container image, and read the errors that come up along the way. Mid-level and Senior take the same topics further — batching, concurrency, multi-service graphs, autoscaling, security — and nothing here is thrown away when you get there.

Each section ends with a Try it task. Do them as you go. Serving is one of those subjects that looks obvious on the page and teaches you something the first time your own service refuses to start.

Everything below is written against BentoML 1.4.x and the current service API. That qualifier matters more than usual here, and the section on the old API explains why.

What BentoML is, and the problem it solves

BentoML is a Python framework for turning model inference code into a running HTTP service, and for packaging that service into a single versioned artifact. You write a normal Python class, decorate it, and BentoML gives you a web server, request validation, an OpenAPI description, Prometheus metrics, a build command that produces a reproducible bundle, and a command that turns that bundle into a container image.

That list is deliberately boring. The interesting claim BentoML makes is not that it can serve a model — anything can serve a model — but that serving and packaging are the same problem, and that solving them separately is what goes wrong in practice.

Read that left to right as the arc of this guide. A trained model on its own is a file. Wrapped in a service it becomes something callable. Built into a Bento it becomes something storable, listable and shippable. Containerized it becomes something a platform team can run without knowing any Python at all.

To see why that arc is worth a framework, look at what people build without one. The usual first attempt is a small web application — a Flask or FastAPI file that loads a pickle at import time and exposes one POST route that pulls JSON out of the request body, calls model.predict, and returns JSON. It takes twenty minutes and it works. It is also where a long list of problems starts.

The model loads once per worker process, or once per request if you got it wrong, and nothing in the code tells you which. The input is parsed by hand, so a client sending a string where a float was expected produces a five-hundred error and a stack trace in the logs rather than a clear four-hundred response. There is no description of the endpoint, so the team consuming it reads your source code or guesses. The requirements.txt next to the file drifts from the environment you actually trained in. The weights live somewhere outside version control — a shared drive, an S3 bucket, someone's home directory — so "which model is in production" is answered by asking around. And when a second model arrives, the whole thing is copied and the copies diverge.

None of those are hard problems individually. Together they are the reason a model that worked in March is unreproducible in June. BentoML's answer is to make the service definition carry all of it: the code, the dependency specification, the model references and the runtime configuration travel inside one artifact with a version string on it.

Two consequences of that design shape everything that follows.

A Bento is an immutable, self-describing bundle. When you run bentoml build, BentoML collects your source files, records the Python packages, notes which service class to start and which models to include, and writes the whole thing into the local Bento store under a name and an auto-generated version. You do not edit a Bento; you build a new one. That is what makes "roll back to the version that worked" a real option rather than a hopeful git checkout.

Your service is ordinary Python. There is no configuration language for describing inputs, no graph to declare, no model-server plugin to write. You load the model in __init__ and you write methods. Type hints on those methods become the request and response schema. This is the part people find surprisingly pleasant: the thing you test in a unit test is the thing that serves traffic.

What teams actually use it for:

You need very little to follow along: Python 3.9 or newer, a terminal, and — for the container sections only — a container runtime such as Docker or Podman. A model is optional at first; the first service we write does not need one, which is on purpose, because the point of the first exercise is the serving machinery and not the machine learning.

Before you read any tutorial: the old API

This section is out of order on purpose. BentoML changed its public API in version 1.2, and the overwhelming majority of blog posts, Stack Overflow answers, course videos and generated code still show the older style. If you do not know what the old style looks like, you will copy it, and it will not run.

The old pattern, from 1.1 and earlier, looked like this. It is shown here only so you can recognise it:

import bentoml
from bentoml.io import NumpyNdarray

runner = bentoml.sklearn.get("iris_clf:latest").to_runner()
svc = bentoml.Service("iris_classifier", runners=[runner])

@svc.api(input=NumpyNdarray(), output=NumpyNdarray())
def classify(input_series):
    return runner.predict.run(input_series)

Three tells identify it. There is a module-level svc = bentoml.Service(...) object. There are IO descriptors imported from bentoml.io — NumpyNdarray, JSON, Image — passed as input= and output=. And there are Runners, created with .to_runner() and called with runner.predict.run(...).

The current style, from 1.2 onward, replaces all three. The service is a class decorated with @bentoml.service. The endpoints are methods decorated with @bentoml.api. The input and output types are expressed as ordinary Python type hints instead of descriptor objects. And where you previously used a Runner to put model inference in its own process, you now use bentoml.depends() to depend on another service — a mid-level topic, but worth knowing the name of now.

import bentoml

@bentoml.service
class IrisClassifier:
    def __init__(self) -> None:
        ...

    @bentoml.api
    def classify(self, sepal_length: float) -> str:
        ...

Why spend a whole section on this? Because it is the single most common way beginners lose an afternoon with BentoML, and because knowing the history makes the current design easier to read. The old API asked you to describe your inputs in BentoML's vocabulary. The new one reads your Python type hints instead. Everything that looks elegant about the current API is the removal of a translation layer.

The four nouns

BentoML has a small vocabulary, and nearly every error message and documentation page uses it precisely. Four nouns carry most of the weight.

A service is the unit of deployment. One service becomes one set of worker processes with its own resource requests and its own scaling behaviour. Its __init__ runs once per worker, which is where model loading belongs: load the weights there and every request after the first finds them already in memory. Anything expensive and per-process goes in __init__.

An API is one HTTP endpoint. By default the route is the method name, so a method called summarize answers at /summarize. You can override that with @bentoml.api(route="/custom/path") when the endpoint must match a contract you do not control. The parameters and return annotation of the method define what the endpoint accepts and returns, and BentoML validates incoming requests against them before your code runs.

A model in BentoML's sense is an entry in the local model store: weights plus metadata, addressed as name:version, with name:latest resolving to the newest. This matters because it separates "the model" from "the code that serves it". Two services can reference the same stored model; a new model version can be saved without touching the service; and the Bento records which model version it was built with. Models can also be references to external sources rather than local copies — a Hugging Face model reference, for instance — which keeps large weights out of your artifact.

A Bento is everything above, frozen. bentoml build produces one, tagged name:version, where the name comes from your service class and the version is generated for you. It sits in the local Bento store until you export it to a file, push it to BentoCloud, or containerize it.

Two more words you will meet early, worth defining now even though they belong to later sections. A deployment is a running instance of a Bento on BentoCloud, the hosted platform. Adaptive batching is BentoML grouping concurrent requests into a single model call, switched on per API with batchable=True; it is the main throughput lever and the subject of the mid-level guide.

Installing BentoML and checking the setup

BentoML is a Python package. The only hard requirement is Python 3.9 or newer. Install it into a virtual environment — not because BentoML is unusually fragile, but because BentoML reads your environment to decide what to put in the Bento, so a clean environment is a clean artifact.

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install bentoml
bentoml --version

On Windows, use PowerShell:

py -3.11 -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -U pip
pip install bentoml
bentoml --version

If PowerShell refuses to run the activation script, Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once will allow it. For the container sections later on, Windows users are better served by working inside WSL2, where the container runtime and the Linux filesystem behave the way the documentation assumes.

The second command to run is the one nobody runs until they need help:

bentoml env

This prints your system information and the versions of the relevant packages. Run it now so you know what healthy looks like, and include its output whenever you ask a colleague or a maintainer for help — most "it does not work on my machine" threads are answered by one line of it.

For production, pin the version rather than floating:

pip install "bentoml==1.4.39"

Pinning matters here more than for most libraries, for two reasons. BentoML's API changed meaningfully at 1.2, so an unpinned install in a long-lived environment can pick up a version your code does not match. And the 1.4 patch series has carried several security fixes — file path resolution on user input, symlink validation when extracting archives, hardening of the template engine that generates Dockerfiles — which is an argument for pinning to a specific patch and then deliberately moving that pin forward, rather than never pinning and never thinking about it.

Telemetry is worth a word because it surprises people. BentoML's CLI reports anonymous usage by default. Every command accepts --do-not-track, and the behaviour can be switched off for a whole environment with the BENTOML_DO_NOT_TRACK environment variable. If you work somewhere that reviews outbound traffic, set it before your first command rather than explaining it afterwards.

Your first service

We will build a text summarization service, because it needs no training step and exercises every part of the machinery: a model loaded once, an endpoint with a typed input, a real dependency, and something interesting to look at in the response.

Install the two extra packages:

pip install torch transformers

Create a file called service.py:

import bentoml
from transformers import pipeline


@bentoml.service
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline("summarization")

    @bentoml.api
    def summarize(self, text: str) -> str:
        result = self.pipeline(text)
        return result[0]["summary_text"]

Eleven lines, and every one of them is doing something worth naming.

@bentoml.service on the class is what makes this more than a class. It tells BentoML this is a deployable unit, and it is where configuration goes later — resources, worker count, timeouts, metrics settings all become arguments to this decorator.

__init__ runs once per worker process when the server starts. Downloading and constructing a Hugging Face pipeline takes seconds; doing it here means those seconds are paid at startup, once, and not on every request. Note that there is nothing BentoML-specific inside it. It is the code you would write in a notebook.

@bentoml.api on summarize turns the method into an endpoint. The route defaults to the method name, so this one answers at /summarize.

The annotations text: str and -> str are the contract. BentoML reads them, builds a request schema from them, validates incoming JSON against that schema, and publishes the result as an OpenAPI description. A caller who sends a number instead of a string is rejected before your pipeline is ever invoked, with a response that says what was wrong.

Start it:

bentoml serve service:Summarization

The argument is module:ClassName — the Python module (service, from service.py) and the class inside it. Get either wrong and the server will tell you it could not import the service. Run it from the directory that contains service.py, because that is what makes the module importable.

The server listens on port 3000 by default. The first start is slow, because the model downloads; later starts are fast. When it is up, the logs tell you the address, and BentoML serves an interactive API page where you can see every endpoint it found and try one from the browser. That page is generated from your type hints, which is the first moment the design pays off visibly: you wrote two annotations and got documentation.

Call it from another terminal:

curl -X POST http://localhost:3000/summarize \
  -H "Content-Type: application/json" \
  -d '{"text": "BentoML is a Python framework for serving and packaging machine learning models. It turns inference code into an HTTP service and bundles the code, its dependencies and its model references into a single versioned artifact that can be containerized and deployed."}'

Three details in that command are worth fixing in your memory, because every one of them is a question people ask. The method is POST, not GET. The content type is JSON. And the JSON keys are the parameter names of your method — text here, because the parameter is called text. Rename the parameter and the request body changes with it. That coupling is intentional: there is one definition of the interface and it is the Python signature.

Calling your service three ways

A service that only answers curl is not finished. Three callers matter in practice, and each teaches something different.

The interactive page. BentoML serves a browser UI generated from the OpenAPI description of your service. Use it to check what a client will see: the endpoint list, the field names, the types, whether a field is required. When a frontend developer asks you for the API contract, this page is the answer, and because it comes from your type hints it cannot drift from the implementation.

curl or any HTTP client. The endpoint is an ordinary JSON POST. Nothing about calling it requires BentoML on the client side, which is the point: a Java backend, a Node service or a cron job can call it with whatever HTTP library it already has. This is also how you will test from CI.

The Python client. When the caller is Python, BentoML ships a client that makes the remote endpoints look like methods:

import bentoml

with bentoml.SyncHTTPClient("http://localhost:3000") as client:
    summary = client.summarize(
        text="BentoML turns inference code into an HTTP service and packages it."
    )
    print(summary)

There is an asynchronous equivalent for callers that already run an event loop. Use the client for integration tests and for one service calling another over HTTP during development. Note that the method name and keyword argument match your service exactly — the same contract again, from a third angle.

Which to reach for depends on what you are checking. If you are asking "is the server up and the model loaded", curl is fastest. If you are asking "what will the consuming team see", use the browser page. If you are writing a test, use the Python client, because a failure gives you a Python exception instead of a string you have to parse.

One endpoint you get for free and should know about: BentoML exposes Prometheus metrics on /metrics, on the same port, enabled by default. Open it now. The counters and histograms there — request totals, in-progress requests, request duration — are what monitoring will scrape later, and seeing them exist from the first service is worth more than reading about them in the senior guide.

Types are the interface

The single idea that most repays attention at this level is that your type hints are the API specification. Getting comfortable with that turns a lot of later work into type design rather than framework configuration.

Simple scalars work as you would expect: str, int, float, bool. Lists of them work. For anything with structure, use a Pydantic model, and your endpoint gains a documented, validated object schema:

from typing import List

import bentoml
from pydantic import BaseModel


class SummarizeRequest(BaseModel):
    text: str
    max_words: int = 60


class SummarizeResponse(BaseModel):
    summary: str
    input_words: int


@bentoml.service
class Summarization:
    def __init__(self) -> None:
        from transformers import pipeline

        self.pipeline = pipeline("summarization")

    @bentoml.api
    def summarize(self, request: SummarizeRequest) -> SummarizeResponse:
        result = self.pipeline(request.text)
        return SummarizeResponse(
            summary=result[0]["summary_text"],
            input_words=len(request.text.split()),
        )

Now the request body is a nested object, max_words has a default so clients may omit it, and the response has named fields instead of a bare string. The browser page updates itself. No descriptor classes, no schema file, no duplication between the validation layer and the implementation.

Two more type choices come up constantly in model serving.

Arrays. A numpy array annotation is accepted directly, which is what you want for a classical model whose predict takes a matrix. A list of floats is often the friendlier choice for a beginner API, because JSON clients do not have to think about shapes.

Files. For images, audio or documents, annotate with pathlib.Path. BentoML handles the upload and hands your method a path to a file on disk, which means your code opens a file like any other Python code rather than decoding a multipart body by hand.

from pathlib import Path

import bentoml


@bentoml.service
class ImageInfo:
    @bentoml.api
    def describe(self, image: Path) -> dict:
        return {"name": image.name, "bytes": image.stat().st_size}

A word of advice about the return type. dict works and is tempting, but an untyped dictionary gives the caller no schema and gives you no protection against renaming a key. Prefer a Pydantic response model for anything another team consumes. The extra five lines are the cheapest documentation you will ever write.

Saving and loading a model

So far the model has been downloaded by transformers at startup. That is fine for a demo and wrong for anything you need to reproduce, because "the model" is then whatever the internet served that morning. BentoML's model store exists to fix that.

The store is a local, versioned registry of saved models, managed through bentoml models. You save a trained object into it under a name; BentoML assigns a version and records metadata. From then on the model is addressable as name:version, or name:latest for the newest.

Train and save a small scikit-learn model so the whole loop is visible:

import bentoml
from sklearn import datasets, svm

X, y = datasets.load_iris(return_X_y=True)
clf = svm.SVC(gamma="scale")
clf.fit(X, y)

saved = bentoml.sklearn.save_model("iris_clf", clf)
print(saved.tag)

pip install scikit-learn
python train.py
bentoml models list

The printed tag looks like iris_clf:xxxxxxxxxxxx, with a generated version. Run train.py twice and bentoml models list shows two versions; iris_clf:latest points at the second. That is the whole mechanism, and it is enough to answer "which model is this" for the rest of your career.

Now serve it. The model is loaded in __init__, from the store, by tag:

import bentoml
import numpy as np


@bentoml.service
class IrisClassifier:
    def __init__(self) -> None:
        self.model = bentoml.sklearn.load_model("iris_clf:latest")

    @bentoml.api
    def classify(self, features: list[float]) -> int:
        prediction = self.model.predict(np.array([features]))
        return int(prediction[0])

bentoml serve service:IrisClassifier
curl -X POST http://localhost:3000/classify \
  -H "Content-Type: application/json" \
  -d '{"features": [5.1, 3.5, 1.4, 0.2]}'

Each supported framework has its own module with the same shape — save_model and load_model — so the pattern transfers. For deep learning work where the weights come from Hugging Face, BentoML can reference the external model instead of copying gigabytes into your store and your artifact, which keeps builds fast.

Loading models from untrusted sources deserves one sentence now rather than a surprise later: many model formats are pickles, and loading a pickle executes code. The same applies to Bento archives you import from elsewhere. Treat both like running a script someone sent you, because that is what it is.

Building a Bento

Your service runs. It runs on your laptop, in your virtual environment, from your working directory. bentoml build is the step that removes all three of those dependencies.

Build configuration lives in bentofile.yaml next to your code, or in pyproject.toml under [tool.bentoml.build] if you prefer one file. The minimum is which service to start and what to include:

service: "service:IrisClassifier"
include:
  - "*.py"
python:
  packages:
    - scikit-learn
    - numpy

bentoml build
bentoml list

The build prints a tag like iris_classifier:xxxxxxxxxxxx and stores the Bento locally. bentoml list shows what you have; bentoml get iris_classifier:latest shows the detail of one, including the path on disk — open that directory once, because seeing the structure makes the concept concrete. Inside you will find your source files, the recorded Python dependencies, the service configuration and the model references.

The fields of bentofile.yaml worth knowing on day one:

Be deliberate with include. The default temptation is to include everything, which quietly puts your notebooks, your test data, your .env and your local virtual environment into a shareable artifact. List what the service needs. A Bento is something you hand to other people.

python.lock_packages is the field that earns its keep. With it on — which is the default — the build resolves your dependency specification into a fully pinned set and records that. Two builds from the same source therefore install the same versions, which is the difference between an artifact and a wish.

You can build from a different directory or config file when your repository layout calls for it:

bentoml build -f ./deploy/bentofile.yaml ./

And you can move Bentos between machines without any platform at all:

bentoml export iris_classifier:latest ./iris_classifier.bento
bentoml import ./iris_classifier.bento
bentoml delete iris_classifier:latest

That export file is the simplest possible answer to "how do I give this to the ops team". It is also, as noted above, executable content: import only what you trust.

Serving a Bento, and containerizing it

A built Bento can be served directly by tag, which is how you check that the artifact — not your working directory — actually works:

bentoml serve iris_classifier:latest

This is a genuinely important test and people skip it. Serving from source uses your environment; serving the Bento uses what the Bento recorded. If the Bento is missing a file or a package, this is where you find out, in two seconds, instead of in a container build ten minutes later.

Then the step that makes the artifact portable:

bentoml containerize iris_classifier:latest

BentoML generates a Dockerfile from the Bento's configuration and builds an image through your local container backend. Docker is the common choice; Podman and Buildah also work. The image is an ordinary OCI image with no BentoML-specific runtime requirement on the host, which is exactly what a platform team wants to receive.

Run it:

docker run --rm -p 3000:3000 iris_classifier:xxxxxxxxxxxx

Use the tag the containerize command printed. Then call http://localhost:3000/classify exactly as before. The response should be identical to the one from bentoml serve, and confirming that identity is the point of the exercise: the thing you tested is the thing that ships.

Two practical notes. First, the container build installs your dependencies, so the first one is slow — expect minutes, not seconds, for anything with PyTorch in it. Second, if you are on an Apple Silicon Mac and the target is an x86 cloud instance, the image you build locally is ARM by default and will not run there; building for the target architecture is a container-level concern covered in the Docker guide. The honest advice for a beginner is to build deployment images in CI, on Linux, and keep local containerize runs for verification.

The commands you will actually use

Grouped by intent rather than alphabetically, because that is how you will reach for them.

While writing a service

bentoml serve service:MyService    # run from source
bentoml env                        # environment report for bug reports
bentoml --version

Working with models

bentoml models list
bentoml models get iris_clf:latest
bentoml models delete iris_clf:xxxxxxxxxxxx

Building and inspecting Bentos

bentoml build
bentoml build -f ./deploy/bentofile.yaml ./
bentoml list
bentoml get iris_classifier:latest
bentoml serve iris_classifier:latest
bentoml delete iris_classifier:xxxxxxxxxxxx

Moving Bentos around

bentoml containerize iris_classifier:latest
bentoml export iris_classifier:latest ./out.bento
bentoml import ./out.bento
bentoml push iris_classifier:latest        # to BentoCloud
bentoml pull iris_classifier:latest

Four options are shared by every command and are worth knowing as a set: --verbose when you need to see what BentoML is doing, -q or --quiet in scripts, --do-not-track to suppress telemetry for one invocation, and --context to select which configured remote a command talks to.

A habit worth forming now: after any build, run list and then serve the tag. Three commands, ten seconds, and it catches the two most common packaging mistakes — a missing file and a missing package — at the cheapest possible moment.

Configuring the service

Configuration in current BentoML is mostly arguments to the @bentoml.service decorator, which means it lives next to the code it configures and travels inside the Bento. If you find a tutorial editing a global YAML configuration file, you are reading about the older style again.

import bentoml


@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 30},
    workers=2,
    path_prefix="/v1",
)
class IrisClassifier:
    ...

resources declares what one replica of this service needs — CPU, memory, and for model work a GPU. traffic controls request handling, most usefully timeout, in seconds, after which a request is abandoned. workers sets how many worker processes serve this service; remember that __init__ runs once per worker, so four workers means four copies of your model in memory. path_prefix puts every endpoint of the service under a prefix, which is how you version an API or fit into an existing URL scheme. There is also a metrics argument for tuning what Prometheus sees, including the histogram buckets for request duration.

The one you should think about on day one is workers, and the reason is memory arithmetic. A large model with several workers will exhaust the memory of a modest machine, and the symptom is a process being killed with no Python traceback at all. Start with one worker, measure, then raise it. Concurrency and batching — the other two throughput levers — are mid-level topics for a good reason: they interact, and tuning them without measurement makes things worse.

Two lifecycle hooks are worth knowing by name. @bentoml.on_deployment runs once before any worker starts, which is where one-time preparation such as downloading weights belongs; doing it in __init__ would repeat the work per worker. @bentoml.on_shutdown runs on the way out, for cleanup. Use the first sparingly and the second rarely, but recognise them when you read someone else's service.

A last word on secrets, because it is a mistake that is easy to make and hard to undo. The build configuration accepts an envs list, and it is tempting to put an API key there. Do not. The values are baked into the Bento, and a Bento is an artifact you export, push and share. Secrets belong in whatever your runtime provides — Kubernetes Secrets, Vault, a cloud secrets manager, or the hosted platform's secret store — injected as environment variables when the container starts.

Reading the errors

Serving errors are mostly import errors, port errors and version errors wearing different clothes. Learning the five shapes below covers nearly everything you will hit in your first month.

The command is not found. bentoml: command not found, or on Windows a message about an unrecognised command. Your virtual environment is not active, or the scripts directory is not on PATH. Activate the environment. If you need to confirm which interpreter owns the install, python -m bentoml --version bypasses PATH entirely.

The service cannot be imported. An import failure when you run bentoml serve service:MyService almost always means one of three things: the module name is wrong (it is the filename without .py), the class name is wrong or misspelled, or you are in the wrong directory so Python cannot find the module. Check in that order. Read the traceback to the bottom, too — sometimes the real failure is inside your own module, a missing dependency or a syntax error, and the service import message is just the outer wrapper.

Legacy API errors. A ModuleNotFoundError for bentoml.io, or an AttributeError saying bentoml has no attribute Service, means you are running 1.1-era code on a current install. Rewrite the file in the class-based style. The instinct to pin an old BentoML instead is understandable and wrong: you will be writing the migration eventually, and the old series does not get the security patches.

The port is taken. An address-already-in-use error on port 3000 means something else is listening — nine times out of ten a bentoml serve you started in another terminal and forgot. Stop it and start again.

The Bento or model is not found. A not-found error naming a tag means the store does not have what you asked for. Run bentoml list or bentoml models list and compare the tag carefully, including the version. This bites hardest in containers and CI, where the store is not the one on your laptop: a model saved locally is not inside a Bento unless the build included it.

Two more that are worth recognising because the cause is not in your code. GPU code that reports no device available in a container is usually a mismatch between the image's CUDA version and the host driver rather than a BentoML problem; upgrading to a current patch release is the first thing to try, since CUDA base image corrections have shipped in the 1.4 series. And failures during pip install, model download or any call to a hosted service that mention certificate verification are the TLS-interception problem from the installation section, not a BentoML bug.

The general technique matters more than the list. When a service misbehaves, the order is: read the server log in the terminal where it runs; reproduce with curl so you see the status code and body; re-run the command with --verbose; and if the artifact is involved, bentoml serve <tag> to separate "my code is broken" from "my Bento is broken". Collect bentoml env output before asking anyone for help.

Putting it all together

One small end-to-end project, using everything above. A sentiment classifier, trained, saved, served with a typed interface, built, containerized and called from a container.

import bentoml
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

texts = [
    "excellent service, very fast delivery",
    "the product arrived broken and late",
    "great quality for the price",
    "terrible experience, would not buy again",
    "friendly support and a quick refund",
    "completely useless, waste of money",
]
labels = [1, 0, 1, 0, 1, 0]

model = make_pipeline(TfidfVectorizer(), LogisticRegression())
model.fit(texts, labels)

saved = bentoml.sklearn.save_model("sentiment", model)
print("saved:", saved.tag)

import bentoml
from pydantic import BaseModel


class Review(BaseModel):
    text: str


class Sentiment(BaseModel):
    label: str
    score: float


@bentoml.service(resources={"cpu": "1"}, traffic={"timeout": 15})
class SentimentClassifier:
    def __init__(self) -> None:
        self.model = bentoml.sklearn.load_model("sentiment:latest")

    @bentoml.api
    def classify(self, review: Review) -> Sentiment:
        probabilities = self.model.predict_proba([review.text])[0]
        positive = float(probabilities[1])
        return Sentiment(
            label="positive" if positive >= 0.5 else "negative",
            score=round(max(positive, 1 - positive), 4),
        )

service: "service:SentimentClassifier"
description: "TF-IDF and logistic regression sentiment classifier"
labels:
  owner: mlops-course
include:
  - "service.py"
python:
  packages:
    - scikit-learn
models:
  - "sentiment:latest"

Then the whole loop, in order:

python train.py
bentoml models list

bentoml serve service:SentimentClassifier
# in a second terminal:
curl -X POST http://localhost:3000/classify \
  -H "Content-Type: application/json" \
  -d '{"review": {"text": "the delivery was quick and the product is great"}}'

bentoml build
bentoml list
bentoml serve sentiment_classifier:latest

bentoml containerize sentiment_classifier:latest
docker run --rm -p 3000:3000 sentiment_classifier:xxxxxxxxxxxx

Walk through what each stage proved. train.py put weights in the model store with a version, so the model is now a named thing rather than a file. Serving from source proved the code works. The typed interface means a client sending the wrong shape gets a validation error, and the browser page documents the fields without you writing documentation. bentoml build froze code, pinned dependencies and the model reference into one tagged artifact — note the models entry in the build file, which is what puts the stored model inside the Bento so the container does not need your laptop's store. Serving the Bento proved the artifact is complete. Containerizing produced an image that any platform can run. Running it and getting the same answer proved the chain end to end.

Six commands, and the gap between "it works in my notebook" and "another team can deploy this" is closed. That gap is where most model projects die, so closing it reliably is a more valuable skill than it sounds.

What you can now do, and what comes next

You can write a BentoML service in the current API and recognise the legacy one on sight. You can serve it, call it from three kinds of client, and read the generated API page. You can express an interface in type hints and Pydantic models and let BentoML validate it. You can save a model into the store with a version, load it by tag, and understand why pinning matters. You can write a build file, produce a Bento, inspect it, serve it, export it and containerize it. You can set resources, timeouts, workers and a path prefix, and you know where secrets must not go. And you can diagnose the five errors that account for most first-month frustration.

That is enough to ship a real internal service. It is not yet enough to run one under load, and the honest list of what you do not yet know is short and specific.

Throughput. Adaptive batching groups concurrent requests into one model call and is the main lever for GPU utilisation. It interacts with traffic.concurrency and with workers, and tuning it without load testing makes latency worse. That is the centre of the mid-level guide.

Multi-service graphs. bentoml.depends() lets one service call another as if it were a local object, so CPU preprocessing and GPU inference can be separate services with separate resources and separate scaling. This replaces the old Runner concept.

Async, streaming and tasks. API methods can be async def; generators give streaming responses, which matters for anything token-by-token; and @bentoml.task handles long-running work through a queue the client polls.

Operations. Metrics beyond the defaults, tracing, autoscaling with minimum and maximum replicas and stabilization windows, secrets management, cost control on GPU replicas, and the question of self-hosting on Kubernetes versus the hosted platform. That is the senior guide.

Where to go next in this catalogue depends on what you are missing. If model versioning and experiment tracking are the weak link upstream of serving, read MLflow — the pairing of MLflow for tracking and BentoML for serving is a common and sensible stack. If the serving target is Kubernetes and you want to compare BentoML's packaging-first approach with a Kubernetes-native one, read KServe and Seldon Core. If you want the container fundamentals under bentoml containerize — layers, platforms, image size, running as a non-root user — read Docker, and then Kubernetes for where the image goes. If what you are serving is a language model and you care about tokens per second, vLLM covers the inference engine that BentoML would wrap. And for watching a deployed service, Prometheus and Grafana are what consume that /metrics endpoint you opened earlier.

Before you move on, do one thing: take a model that already exists in your work or your coursework and put it through the loop in the previous section. Not the iris dataset — something of yours, with a real input shape and a real dependency list. Every rough edge you hit doing that is a thing the next two guides will explain, and you will read them very differently having felt the problem first.

Then read the Interview Prep and Tips files. The first turns this into answers you can give under pressure; the second is the list of mistakes that cost other people their afternoons.

Sources

• BentoML documentation
• Install BentoML
• Services
• Adaptive batching
• Bento build options
• CLI reference
• Metrics
• Parallelize requests
• BentoML releases
• bentoml on PyPI

This file turns the detailed guide into answers you can give out loud, under time pressure, to someone who may know BentoML better than you do. Interviewers at this level are not testing whether you have memorised the CLI. They are testing whether you understand what problem a serving framework solves, whether you know where the boundaries are between model, service and artifact, and whether you have actually run the thing.

Everything here matches BentoML 1.4.x and the current service API.

The thirty-second answer

BentoML is a Python framework for turning model inference code into an HTTP service and packaging that service into a single versioned artifact called a Bento. You write a class decorated with @bentoml.service with methods decorated with @bentoml.api, and you get a server, request validation from your type hints, an OpenAPI description, Prometheus metrics, a reproducible build and a container image.

If they want a second sentence, make it the one about why it exists:

The point is that serving and packaging are the same problem. A hand-written Flask wrapper serves the model but does not capture the dependencies, the model version or the configuration, so it is unreproducible by the time anyone needs to roll it back. A Bento captures all of it under one tag.

Resist the urge to list features. Two sentences, then stop and let them steer. If you keep talking you will end up on autoscaling, which is not a beginner topic and not where you want to be.

The mental model to state first

Four nouns, and an interviewer will often ask you to define them.

A service is a Python class decorated with @bentoml.service. It is the unit of deployment: one service becomes its own set of worker processes with its own resource requests. Model loading goes in __init__, which runs once per worker.

An API is a method decorated with @bentoml.api, exposed as an HTTP endpoint. The route defaults to the method name and can be overridden with route=. Its parameters and return annotation are the request and response schema, validated before your code runs.

A model is an entry in BentoML's local model store, saved with a framework module such as bentoml.sklearn.save_model and addressed as name:version, with name:latest resolving to the newest.

A Bento is the built bundle produced by bentoml build: your code, the pinned dependency set, model references and service configuration, tagged name:version and stored locally until you export, push or containerize it.

The sentence that ties them together and shows you understand the design: a model is weights, a Bento is weights plus the code and environment that serve them. That distinction is the most common follow-up question in disguise.

Commands to have ready

You will not be asked to recite these, but you will be asked "walk me through getting a model into production", and the answer should sound like someone who has typed them.

bentoml serve service:MyService         # run from source during development
bentoml serve my_service:latest         # run the built artifact
bentoml models list                     # what is in the model store
bentoml build                           # produce a Bento from bentofile.yaml
bentoml list                            # what is in the Bento store
bentoml get my_service:latest           # detail, including the path on disk
bentoml containerize my_service:latest  # generate and build an OCI image
bentoml export my_service:latest ./out.bento
bentoml import ./out.bento
bentoml env                             # environment report, for bug reports

The workflow answer that lands well is short: write the service, bentoml serve service:Class to check it, bentoml build, then bentoml serve <tag> to check the artifact rather than your working directory, then bentoml containerize and hand the image to whatever runs containers. Saying that fourth step unprompted is a strong signal, because it is the step people skip and it is the one that catches a missing file or package.

The classic traps, and why they happen

Copying the legacy API. Most examples online use bentoml.Service(...), IO descriptors from bentoml.io, and Runners created with .to_runner(). That is the pre-1.2 style. On a current install it fails with a ModuleNotFoundError for bentoml.io or an AttributeError about bentoml having no attribute Service. The cause is that 1.2 replaced descriptors with plain type hints and Runners with bentoml.depends(). The fix is to rewrite, not to pin an old version — the old series does not receive the security patches that the 1.4 line has been shipping.

Loading the model inside the API method. It works, and it reloads the weights on every request. The cause is not knowing that __init__ runs once per worker. The symptom is latency that is uniformly terrible rather than occasionally bad.

Pinning to :latest in a shared environment. latest resolves at load time, so someone retraining changes what production serves with no code change and nothing in git to explain it. Use latest while iterating locally; pin an explicit version in anything you build for others.

Forgetting models in the build file. The service loads sentiment:latest from the local store, works on your laptop, and fails in the container because the store there is empty. Listing the model under models in bentofile.yaml is what puts it inside the Bento.

Putting secrets in envs. The build configuration accepts environment variables, and they are baked into the artifact. A Bento is something you export and share. Secrets belong in the runtime's secret store, injected when the container starts.

Too many workers for the memory available. __init__ runs per worker, so four workers means four copies of the model. The failure mode is a process killed by the kernel with no Python traceback, which is confusing the first time. Start with one worker and measure.

Including everything in the Bento. A broad include pattern quietly ships notebooks, test data, a local virtual environment and sometimes a .env file inside an artifact you hand to other people.

Common interview questions

"What does BentoML do that FastAPI does not?"

FastAPI gives you the HTTP layer and validation; BentoML gives you that plus the packaging and the model lifecycle. With FastAPI you still have to decide how the model gets into the image, how dependencies are pinned, which model version is running, and how to produce a reproducible artifact. BentoML answers those with the model store, lock_packages at build time, the models reference in the build file, and a tagged Bento. I would also mention that BentoML adds serving-specific features FastAPI has no opinion about — adaptive batching, per-service resource and worker configuration, metrics on /metrics by default.

"Why not just write a Dockerfile?"

You can, and bentoml containerize produces an ordinary OCI image, so nothing stops you doing both. The argument for letting BentoML generate it is that the Dockerfile is then derived from the same build configuration that pins the dependencies and records the model, so the image cannot drift from the artifact. The build file exposes the fields that matter — docker.distro, docker.python_version, docker.system_packages — so you keep the control you need without maintaining the file.

"Where does model loading go, and why?"

In __init__ of the service class, which runs once per worker process at startup. That moves the cost of constructing the model to start-up time and out of the request path. Putting it in the API method reloads per request; putting one-time preparation that should happen before any worker starts — downloading weights, say — in @bentoml.on_deployment avoids repeating it per worker.

"How does a caller know what to send?"

From the type hints. The parameters and the return annotation of the @bentoml.api method define the schema; BentoML validates incoming requests against it and publishes an OpenAPI description, which is served as an interactive page you can open in a browser. The JSON keys in the request body are the parameter names of the method, so renaming a parameter changes the contract — which is the right behaviour, because there is only one definition of the interface.

"What is the difference between bentoml serve service:MyService and bentoml serve my_service:latest?"

The first runs from your source directory in your current environment. The second runs the built Bento, with the files and pinned dependencies the build recorded. I always run the second before containerizing, because it separates "my code is broken" from "my Bento is incomplete" in two seconds rather than after a ten-minute image build.

"How would you give this service to the platform team?"

Two options depending on what they want. bentoml containerize gives them an OCI image with no BentoML requirement on the host, which drops into Kubernetes, ECS or Cloud Run like any other image — that is the usual answer. If they want the artifact rather than the image, bentoml export writes a single .bento file they can bentoml import and build themselves. I would add that a Bento archive is executable content, so importing one from an untrusted source is the same risk as running a script someone sent you.

"The service starts but every request times out. What do you check?"

In order: the server log in the terminal where it runs, because the real error is usually there; then curl directly so I see the status code and body rather than a client's interpretation; then whether traffic.timeout is shorter than the model's actual inference time; then whether the model is being loaded per request instead of in __init__; then memory, because a worker killed by the kernel leaves no traceback. If the artifact is involved I would also serve the Bento by tag to rule out a packaging problem.

"What are these Runners people keep mentioning?"

The pre-1.2 mechanism for running model inference in a separate process from the API server. They were created with .to_runner() and called with runner.predict.run(...). In current BentoML the equivalent is to make the model its own service and depend on it with bentoml.depends(), which gets you the same process separation plus independent resources and scaling per service. Runners are legacy and I would not write new code with them.

"What do you get for monitoring, out of the box?"

Prometheus metrics on /metrics, on the service port, enabled by default — request counts, requests in progress, request duration as a histogram, and batch size once batching is on. The histogram buckets are configurable through the metrics argument of @bentoml.service. Anything beyond that, like tracing, is configuration rather than code.

"Have you deployed one?" — and the honest answer

If you have only done the exercises, say so, and say precisely what you did: trained a model, saved it to the store, served it, built the Bento, containerized it, ran the container and confirmed the same response. That is a real end-to-end loop and describing it concretely is far stronger than implying production experience you cannot defend. Then name what you know you have not done — load testing with batching on, autoscaling, a multi-service graph — and that you know which guide covers it. Interviewers at this level are calibrating your self-assessment at least as much as your knowledge.

What a strong answer sounds like

Two candidates answer "how do you deploy a model with BentoML" and the difference is not knowledge, it is shape.

The weak answer is a feature list: BentoML does serving, batching, containers, autoscaling, monitoring, it has a cloud platform. All true, none of it evidence that the speaker has run anything. It also invites a follow-up they cannot survive, because an interviewer will pick the most advanced thing mentioned and ask about it.

The strong answer is a sequence with a reason at each step: I write the service class, load the model in __init__, type the endpoint with Pydantic models; I serve from source to check the code; I write the build file with the model pinned and a narrow include; I build and then serve the Bento by tag, because that tests the artifact rather than my working directory; then containerize and hand over an image. Same material, but every clause is something you can be asked about and answer.

Three habits make the difference audible. Name the failure each step prevents — serving the Bento by tag catches a missing file, pinning the model tag catches a silent retrain. Say what you have not done, and say which problem it solves, so the interviewer knows you have read far enough to know the shape of what is missing. And give concrete numbers and names where you have them: port 3000, /metrics, __init__ runs once per worker. Vague answers are indistinguishable from guesses, and interviewers assume the worse of the two.

One trap in the other direction: do not volunteer opinions about tools you have not used. "BentoML is better than KServe" from a beginner reads as repeated marketing. "BentoML is Python-code-first and packaging-centric, KServe is Kubernetes-CRD-first, so the choice depends on whether your team lives in Python or in YAML" is a defensible sentence you can actually stand behind, and if you cannot, say you have not compared them.

Two questions people get wrong

"Does BentoML need Kubernetes?"

No. bentoml serve runs it locally, and bentoml containerize produces an ordinary OCI image with no BentoML requirement on the host, so it runs anywhere containers run — Kubernetes, ECS, Cloud Run, or one VM with a container runtime. BentoCloud is the hosted option, and it is where features like autoscaling policies and the external request queue live, but nothing about the packaging model depends on it. The useful framing is that BentoML stops at the image boundary on purpose.

"What happens to a request that violates the type hints?"

It is rejected before your method runs, with a validation error that names the field and the expected type, rather than reaching your model and producing an exception. That is the practical payoff of the type hints being the schema: bad input is a client error with a useful message instead of a server error with a stack trace in your logs. Worth saying because it shows you know validation is happening at a layer you did not write.

Sixty-second self-test

Answer out loud, without notes. Anything you hesitate on, reread that section of the detailed guide.

1. What are the four nouns, and which one has a version you choose?
2. Which decorator makes a class deployable, and which makes a method an endpoint?
3. Where does the route come from by default, and how do you override it?
4. Where does model loading go, and how many times does that code run?
5. What produces a Bento, and what does the build file need at minimum?
6. What does python.lock_packages do, and is it on by default?
7. Name three tells that an example is written against the legacy API.
8. Why is :latest fine locally and risky in a shared environment?
9. What makes a stored model end up inside a Bento?
10. What port does bentoml serve use, and what is served on /metrics?
11. Why serve the Bento by tag before containerizing it?
12. Where should an API key live, and where must it not?

The answers, compressed: service, API, model, Bento — the model version is generated, but the model name is yours and the tag you pin is your choice, while the Bento version is generated. @bentoml.service and @bentoml.api. The route is the method name; route= overrides it. Loading goes in __init__, once per worker. bentoml build, and the build file needs at least service plus include and the Python packages. lock_packages pins the resolved dependency set at build time and is on by default. The legacy tells are bentoml.io, bentoml.Service(, and .to_runner(). :latest resolves at load time, so it can change behaviour with no code change. A models entry in the build file. Port 3000, and Prometheus metrics. Serving by tag tests the artifact rather than your working directory. Secrets go in the runtime's secret store, never in build-time envs.

If you can get through that list cleanly, you can hold a beginner-level BentoML conversation without bluffing — and the three things you should be ready to say you have not yet done are batching, multi-service graphs and autoscaling, which is exactly where the mid-level and senior guides pick up.

This file is the list of mistakes that cost other people their afternoons, each with the symptom you will actually see, the cause underneath it, and the fix. Then the habits worth adopting before you have formed the opposite ones, and a checklist to run before you tell anyone your service is ready.

All of it applies to BentoML 1.4.x and the current service API.

The mistakes beginners actually make

Copying a legacy example

Symptom. ModuleNotFoundError: No module named 'bentoml.io', or an AttributeError saying the bentoml module has no attribute Service, on code you copied from a tutorial that looked current.

Cause. BentoML changed its public API at version 1.2. The old style used a module-level svc = bentoml.Service(...), IO descriptor objects imported from bentoml.io, and Runners created with .to_runner(). Most material on the web predates the change, and code generators reproduce it confidently.

Fix. Rewrite in the current style: a class decorated with @bentoml.service, methods decorated with @bentoml.api, plain Python type hints instead of descriptors. Do not pin an old BentoML to make the old code run — the 1.4 line has been shipping security fixes, and you will have to migrate eventually anyway. Before copying anything, search the snippet for bentoml.io, bentoml.Service( and to_runner. Three seconds, every time.

Loading the model in the API method

Symptom. Every request is slow, by the same large amount. The first request is no slower than the hundredth.

Cause. The model is constructed inside the @bentoml.api method, so it is rebuilt or re-read on every call.

Fix. Move it to __init__, which runs once per worker process at start-up. If the expensive part is something that should happen once for the whole deployment rather than once per worker — downloading weights, for instance — use @bentoml.on_deployment, which runs before any worker spawns.

Getting the module:Class argument wrong

Symptom. bentoml serve fails to import the service.

Cause. One of three things, in order of likelihood: the module name is wrong (it is the filename without .py), the class name is misspelled, or you are running from a directory where Python cannot import the module.

Fix. Check those three. Then read the traceback to the bottom, because sometimes the import failure is inside your own module — a missing third-party package or a syntax error — and the service-import message is only the outer wrapper.

:latest in something other than your laptop

Symptom. Production behaviour changes and nothing in git explains it. Or two replicas disagree because they started at different times.

Cause. name:latest resolves when the model is loaded, not when the code is written. Anyone who retrains changes what the service serves.

Fix. Use latest while iterating locally. Pin an explicit version in anything you build for a shared environment, so the Bento records exactly which weights it serves. If you want the convenience of latest, resolve it at build time rather than at load time by pinning the tag in the build file.

The model is not in the Bento

Symptom. The service works under bentoml serve service:MyService, and the container built from the same project fails at start-up saying the model cannot be found.

Cause. The model lives in your local model store. The container has its own, empty store. Nothing put the model inside the Bento.

Fix. List it under models in bentofile.yaml. The quick way to catch this class of mistake generally is to run bentoml serve <tag> on the built Bento before containerizing: it uses what the build recorded, so it fails the same way the container will, in two seconds instead of ten minutes.

Shipping your whole working directory

Symptom. The exported .bento file is hundreds of megabytes for a small model, and the container build is slow.

Cause. An include pattern broad enough to pick up notebooks, sample data, model checkpoints, caches, and sometimes the local virtual environment. Occasionally a .env.

Fix. List what the service needs rather than excluding what it does not: include: ["service.py"] is a perfectly good starting point. Add a .bentoignore, which works like .gitignore, for the patterns you want excluded everywhere. Then check: bentoml get <tag> prints the path on disk, and looking inside once is the fastest way to find what should not be there.

Secrets in the build configuration

Symptom. None, which is the problem. The credential works, and it is now inside an artifact that has been exported, pushed and shared.

Cause. envs in the build file is a convenient place to put an API key, and the values travel inside the Bento.

Fix. Secrets are injected by the runtime at start-up — Kubernetes Secrets, Vault, a cloud secrets manager, or the hosted platform's secret store. Use envs only for non-sensitive configuration. If a key has already shipped inside a Bento, rotate it; deleting the Bento is not enough once it has been exported.

Too many workers

Symptom. The server dies during start-up or under load, with no Python traceback — just a killed process, or a container that restarts.

Cause. __init__ runs once per worker, so workers=4 means four copies of the model in memory. The kernel's out-of-memory killer does not produce a stack trace.

Fix. Start with one worker. Measure memory per worker, then raise the count against the memory you actually have, leaving headroom. Do not reach for more workers as a latency fix before you have checked whether the request path is doing something it should not, like loading the model.

Debugging without reading the server log

Symptom. Twenty minutes of guessing about a four-hundred or five-hundred response.

Cause. The server runs in one terminal and you are typing in another, so the log line that explains everything is behind a window.

Fix. Two terminals, always: one with the server in the foreground, one for curl. The order of operations when something misbehaves is: read the server log, reproduce with curl so you see the status code and body, re-run with --verbose, and if the artifact is involved, serve the Bento by tag to isolate code from packaging.

Fighting a TLS-inspecting proxy

Symptom. pip install, a Hugging Face download, or any call to a hosted platform fails with SSL: CERTIFICATE_VERIFY_FAILED or SELF_SIGNED_CERT_IN_CHAIN. The same URL opens fine in your browser.

Cause. A corporate proxy or Zero Trust agent terminates TLS and re-signs with its own certificate authority, which Python does not trust.

Fix. Export the proxy's CA certificate and point REQUESTS_CA_BUNDLE, SSL_CERT_FILE and PIP_CERT at the bundle. Do not disable verification, and do not set it as a permanent environment default without thinking: the habit follows you into something that matters.

Forgetting the virtual environment

Symptom. bentoml: command not found, or on Windows a message about an unrecognised command — after it worked ten minutes ago.

Cause. A new terminal, with no environment activated.

Fix. Activate it. python -m bentoml --version confirms which interpreter actually owns the install, which is useful when you have more than one environment and are no longer sure.

Habits worth adopting on day one

Type your interface properly. dict in and dict out works and gives the caller nothing. A Pydantic request model and a Pydantic response model cost five lines and buy you validation, a documented schema on the browser page, and a renamed field becoming a visible change rather than a silent break. Do it from the first service and you will never have to retrofit it.

Serve the Bento, not just the source. Make bentoml build && bentoml list && bentoml serve <tag> a reflex after any change to the build file. It is ten seconds and it catches the two most common packaging mistakes — a missing file, a missing package — before a container build or a CI run pays for them.

Pin BentoML itself. pip install "bentoml==1.4.39" in your requirements, and in your build file's packages. The 1.2 API change is reason enough; the steady stream of security fixes in the 1.4 patches is the reason to move the pin forward deliberately rather than never.

Keep __init__ honest. Everything expensive and per-process goes there. Nothing expensive goes in the API method. If you are unsure which, add a print statement and count how many times it appears when you make three requests.

Open /metrics on your first service. Not because you need monitoring yet, but because seeing the request counter move teaches you that the server is instrumented by default, and you will reach for it instinctively later when someone asks about p95 latency.

Treat Bentos and model files as executable. bentoml import, and loading a pickled model, both run code. Import only what you or your CI produced. This is a one-line habit that avoids an incident category entirely.

Write the build file early, not at the end. People develop for a week against bentoml serve service:Class and then discover at packaging time that the project layout, the model loading and the dependency list all need rework. Build on day one, even a trivial Bento, and the artifact grows with the code.

Read release notes when you move the pin. The 1.4 series has carried fixes for file path resolution on user input, symlink validation when extracting archives, template hardening, SQLite concurrency and CUDA base images. Those are all things that look like mysterious bugs if you do not know they were fixed.

Measure before you tune. Workers, concurrency and batching interact, and all three can make latency worse when set by intuition. At beginner level the correct setting is usually the default; the mid-level guide is about how to measure before changing them.

Confusing a slow start-up with a broken service

Symptom. You run bentoml serve, wait, see nothing, and conclude it has hung. Or the container restarts repeatedly on a platform that health-checks it immediately.

Cause. The first start downloads model weights and constructs the model in __init__, which for anything transformer-shaped takes tens of seconds and sometimes minutes. Nothing is wrong; the work is just front-loaded by design.

Fix. Locally, wait and watch the log rather than restarting, because restarting throws away whatever had already downloaded. For anything deployed, give the platform a start-up grace period generous enough for a cold model load, and prefer to have the weights inside the image or the Bento rather than downloaded at start-up, so the slow path only happens at build time.

Assuming the request body shape

Symptom. A four-hundred response saying a field is missing, on a body you are sure is correct.

Cause. The JSON keys are the parameter names of your method. If the parameter is a Pydantic model called review, the body is {"review": {...}} and not the model's fields at the top level. People reliably get this wrong once.

Fix. Open the interactive page in a browser and look at the example body it generates, rather than reasoning about it. That page comes from your type hints, so it cannot be wrong about your own service.

Three small things that save hours

Name the service class after the thing, not the file. The Bento's name is derived from your service class, so a class called Service or MyService produces artifacts nobody can identify in bentoml list six weeks later. Call it SentimentClassifier and the tag reads like documentation.

Put a description and labels in the build file. They cost two lines, they travel inside the Bento, and they are visible from bentoml get. The first time a colleague asks "what is this Bento and who owns it", those two lines answer it. The habit matters more as the store fills up, which it will faster than you expect.

Clean the stores. Iterating produces a Bento and sometimes a model version per build, and nothing removes them. bentoml list and bentoml models list grow, disk fills, and the useful tags get lost among the experiments. bentoml delete and bentoml models delete are the only cleanup, so run them when you finish a session rather than when the disk complains.

Reading output rather than skimming it

Three outputs reward actually reading, and all three are routinely skimmed.

The server start-up log tells you which service it imported, which endpoints it found, and the address it is listening on. If an endpoint you wrote is not in that list, you have your answer immediately — usually a missing @bentoml.api decorator — and no amount of curl debugging will find it faster.

The bentoml get <tag> output gives you the Bento's path on disk, its recorded dependencies, its models and its configuration. This is the authoritative answer to "what is actually in the artifact", and it beats reasoning about what your build file should have done.

The bentoml env output is what you paste when you ask anyone for help. Collecting it first changes the conversation from a guessing game to a diagnosis, and it frequently answers the question before you send it, because the Python version or package version you had forgotten about is right there.

The general principle, which applies well beyond BentoML: when a tool prints something at start-up, it is telling you what it decided. Most confusing behaviour is the tool doing exactly what it announced and nobody reading the announcement.

The checklist

Run through this before you tell a colleague your service is ready.

• No bentoml.io, bentoml.Service( or to_runner anywhere in the code.
• The model loads in __init__, and nothing expensive happens in the API method.
• The model tag is pinned explicitly, not :latest, for anything beyond your laptop.
• Request and response are Pydantic models, or at least properly annotated — not bare dict.
• bentofile.yaml exists, with service, a deliberate include, the Python packages, and a models entry if the service loads from the store.
• python.lock_packages is left on, so the dependency set is pinned at build time.
• No secrets in envs or anywhere else inside the Bento.
• bentoml build, then bentoml serve <tag> against the built Bento, both clean.
• You have looked inside the Bento directory once and know what is in it.
• bentoml containerize, then docker run, returns the same response as bentoml serve.
• /metrics responds and the request counter moves when you call the endpoint.
• workers is set against measured memory, not guessed.
• traffic.timeout is longer than the model's real inference time, with margin.
• BentoML is pinned to a specific patch version in the requirements and the build file.
• You can say, in one sentence, which model version this Bento serves and where its weights came from.

The last item is the one that matters most and the only one no tool can check for you. If you cannot answer it, nothing else on the list will save you the day someone asks why the predictions changed.

When all of that is routine, go to the Mid-level guide. It starts where this leaves off: adaptive batching and how to load-test it, concurrency and worker tuning with measurement rather than intuition, multi-service graphs with bentoml.depends(), async and streaming endpoints, and what the metrics are actually telling you. Serving a model is the easy half; serving it under load without surprises is the half people get paid for.

This is part two of three. It picks up where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Where you already know the basics, we go straight to the depth.

Where this picks up

Everything here is written against BentoML 1.4.x. If you learned BentoML before 2024, note the warning in the next section before you copy anything from an old project: the pre-1.2 API is a different library wearing the same name.

Each section states the problem it solves first, and most end with a Try it you can run on a real service in a few minutes.

What actually runs when you serve a Bento

Beginner treated bentoml serve as a black box that produced a URL. To tune anything, you need the shape of the thing it starts.

A served Bento is not a single Python process. BentoML runs an HTTP server that accepts connections and dispatches them to worker processes, and each service in the Bento gets its own set of workers. The number of workers for a service is what the workers argument of @bentoml.service controls. This matters more than it looks, because of one consequence that catches almost everyone the first time they serve a large model:

__init__ runs once per worker, not once per service. If your constructor loads a 6 GB model into memory, then workers=4 loads it four times. On CPU that is 24 GB of RAM. On a single GPU it is an out-of-memory crash during startup, and the error will come from your framework rather than from BentoML, which is why it reads as a model problem rather than a configuration one.

So the first design decision at this level is where the expensive thing lives:

That last row is the lifecycle question. BentoML gives you @bentoml.on_deployment for setup that must run once before any worker starts — downloading weights to a shared path, warming a cache, creating a directory — and @bentoml.on_shutdown for cleanup when the service stops. Use the deployment hook for work that is wasteful or incorrect when repeated per worker, and __init__ for work each worker genuinely needs its own copy of, such as loading the model into that process's memory.

import bentoml

@bentoml.service(
    resources={"cpu": "2", "memory": "4GiB"},
    traffic={"timeout": 30, "concurrency": 16},
    workers=2,
    path_prefix="/v1",
)
class Classifier:
    def __init__(self) -> None:
        # Runs in every worker process. Keep it to per-process state.
        self.model = load_model_from_disk("/models/clf")

    @bentoml.api
    def classify(self, text: str) -> dict:
        return self.model(text)

Four arguments are doing distinct jobs there. resources declares what one replica needs, which the deployment platform turns into requests and limits; it does not enforce anything on your laptop. traffic sets the per-replica request policy. workers sets process count inside the replica. path_prefix moves every route under /v1, which is how you give a service a versioned URL without touching the method names.

The Bento as a build artifact

Beginner ran bentoml build and got a tag. The reason to care about the details is that a Bento is the unit you promote between environments, and an unreproducible Bento is an unreproducible deployment.

A Bento is a versioned directory containing your code, a dependency specification, references to models, and the service configuration, stored in the local Bento store and addressed as name:version. Build configuration lives either in bentofile.yaml or in pyproject.toml under [tool.bentoml.build]. The second form is worth preferring in a repository that already has a pyproject.toml, because it keeps one file as the project's source of truth.

service: "service:Classifier"
description: "Text classifier API"
labels:
  team: ml-platform
  owner: aya
include:
  - "service.py"
  - "src/**/*.py"
exclude:
  - "tests/**"
python:
  requirements_txt: "./requirements.txt"
  lock_packages: true
docker:
  distro: debian
  python_version: "3.11"
  system_packages:
    - libgomp1
envs:
  - name: LOG_LEVEL
    value: "info"

The fields that change outcomes rather than cosmetics:

include and exclude, plus .bentoignore. include takes wildcards; anything not matched does not enter the Bento. A .bentoignore file works like .gitignore and is the better place for the blanket rules — notebooks, checkpoints, .git, caches. The failure mode of being sloppy here is not subtle: either you ship a multi-gigabyte Bento full of experiment outputs, or you ship one that is missing a module and fails on import inside the container, long after the build passed.

python.lock_packages, on by default. It resolves and pins the whole dependency set at build time, so the Bento records exact versions rather than ranges. That is what makes "the Bento we tested" and "the Bento in production" the same software. Turn it off only when you have a deliberate reason, and understand that you have traded reproducibility for build speed.

docker.distro and friends. distro accepts debian, alpine, ubi8 and amazonlinux. system_packages installs OS-level dependencies you would otherwise discover as a missing shared library at runtime. setup_script runs your own commands during the image build, and base_image replaces the generated base entirely, which you want when your organisation mandates a hardened base.

envs. Build-time declaration of environment variables baked into the Bento. This is a configuration mechanism, not a secrets mechanism — see the configuration section.

bentoml build                                  # from the project directory
bentoml build -f ./deploy/bentofile.yaml ./    # explicit config and build context
bentoml list                                   # local Bentos
bentoml get classifier:latest                  # full metadata
bentoml export classifier:latest ./classifier.bento
bentoml import ./classifier.bento
bentoml delete classifier:latest

export and import are how a Bento travels without a registry: a single file you can put in object storage and pull into another environment. They are also a trust boundary. A .bento archive contains code and model files that will be executed and deserialised on import, so treat an untrusted Bento the way you would treat an untrusted pickle, which is to say do not import it. BentoML has shipped repeated hardening in this area in the 1.4 line — symlink validation during archive extraction, stricter file path resolution, sandboxed template rendering — which is a good reason to stay current on patches rather than pinning to an old 1.4 release forever.

Concurrency, workers and the traffic policy

Here is the problem this section solves. Your service works. Under load it either falls over, or it stays up but every request takes ten seconds. Both are concurrency problems, and they have different fixes.

traffic on @bentoml.service carries the two settings that matter:

@bentoml.service(traffic={"timeout": 60, "concurrency": 32})
class Classifier: ...

timeout is how long a request may take before the server gives up on it, in seconds. The default is generous enough for small models and far too generous for a service fronting a slow generative model, because a client that has already disconnected still holds a worker. Set it from your actual latency distribution: a little above the p99 you are willing to serve, not above the worst case you have ever seen.

concurrency is the number of requests a single replica is designed to handle at once. This is the number that an autoscaler reasons about, and getting it wrong in either direction is expensive. Set it too low and you pay for replicas that sit idle while requests queue. Set it too high and each replica accepts more work than it can do, so latency climbs for everyone and the autoscaler sees a healthy replica count while your p95 is ruined.

The honest way to find it is measurement. Pick a fixed arrival rate, raise it in steps, and watch where latency stops being flat:

# A quick saturation sweep with hey
for rate in 5 10 20 40 80; do
  echo "== $rate rps =="
  hey -z 30s -q "$rate" -c 50 -m POST \
    -H 'Content-Type: application/json' \
    -d '{"text":"a representative input of realistic length"}' \
    http://localhost:3000/v1/classify | grep -E 'Requests/sec|95%|Error'
done

Read that output as a curve. While the service is below saturation, throughput tracks the offered rate and p95 is flat. At saturation, throughput stops rising and p95 climbs roughly linearly with the extra load, because the extra requests are queueing rather than running. The last rate where p95 was still acceptable, divided by replicas, is a defensible concurrency value.

Then there is the relationship between the three knobs, which is where people get lost:

concurrency governs how much work is admitted to a replica. workers governs how many OS processes exist to do it. Batching governs how many admitted requests are merged into a single model call. Raising workers without raising concurrency gives you idle processes; raising concurrency without either more workers or batching gives you a longer queue and worse latency.

On BentoCloud there is a fourth piece: external_queue: True moves excess requests into a platform-managed queue so a replica is never handed more than its concurrency value. That is the setting that protects a GPU replica from being trampled, and it is BentoCloud-only — when it is enabled, the platform's max_concurrency cap is inactive. Self-hosted, the equivalent protection is a queue or rate limit in front of the service, which the Kubernetes section comes back to.

Adaptive batching, precisely

A GPU running one request at a time is mostly idle. Matrix multiplication is efficient in proportion to how much work you hand it at once, so sixteen inputs in one call cost far less than sixteen calls. Adaptive batching is BentoML's server-side mechanism for getting that for free, without asking clients to batch anything.

You enable it per API:

from typing import List
import bentoml

@bentoml.service(resources={"gpu": 1}, traffic={"timeout": 60, "concurrency": 64})
class Summarizer:
    def __init__(self) -> None:
        from transformers import pipeline
        self.pipeline = pipeline("summarization")

    @bentoml.api(batchable=True, max_batch_size=32, max_latency_ms=1000)
    def summarize(self, texts: List[str]) -> List[str]:
        results = self.pipeline(texts)
        return [item["summary_text"] for item in results]

Clients still send one input each. The server holds arriving requests briefly, merges them into a list, calls your method once, and splits the results back out by position. Four parameters control it:

The constraint that generates most of the errors: a batchable API takes one parameter carrying many requests (plus, optionally, the request context). The type must be a container — List[str], a NumPy array, a list of Pydantic models. If you need several fields per request, define a Pydantic model and batch a list of it. A signature like def summarize(self, text: str, max_len: int) cannot be batched, because the server has no way to know which argument varies per request and which does not.

batch_dim exists because frameworks disagree about axis order. If your model consumes (batch, features) and returns (batch, classes), the default is right. If the output stacks along a different axis, a tuple tells BentoML the input axis and the output axis separately, and getting it wrong shows up as results returned to the wrong callers — the worst class of bug this feature produces, because nothing crashes.

The real decision is max_latency_ms. This is a straight trade: you are spending latency to buy throughput. The budget is how long a request will sit waiting for company. Pick it from the service's own latency, not from a round number. If the model takes 400 ms per batch, a 1000 ms budget can more than triple your p99 at low traffic, where no company ever arrives and every request waits out the full window before being dispatched alone. If the model takes 8 seconds, a 1000 ms window is noise and almost pure profit.

So batching is excellent for throughput-oriented workloads — bulk scoring, embedding generation, asynchronous pipelines — and needs care for interactive ones. The way to tell whether it is actually helping is the metric BentoML exposes for exactly this purpose, bentoml_service_adaptive_batch_size. It is a histogram of realised batch sizes. If it sits at 1, batching is costing you latency and buying nothing, because traffic is too sparse for batches to form. That is a signal to lower the budget, consolidate traffic onto fewer replicas, or switch batching off.

Async APIs and work that waits

Batching helps when the bottleneck is the model. A different bottleneck appears when your API spends its time waiting — on an embedding service, a vector database, object storage, or a third-party model API. A synchronous worker blocked on a network call is a whole process doing nothing, and adding workers to fix that is the expensive way out.

BentoML API methods may be async def. An async API lets a single worker keep many requests in flight, because each one yields the event loop while it waits:

import bentoml
import httpx

@bentoml.service(traffic={"timeout": 30, "concurrency": 64}, workers=2)
class Enricher:
    def __init__(self) -> None:
        self.http = httpx.AsyncClient(timeout=10.0)

    @bentoml.api
    async def enrich(self, doc_id: str) -> dict:
        resp = await self.http.get(f"http://metadata-svc/doc/{doc_id}")
        resp.raise_for_status()
        return resp.json()

Two rules keep this honest. First, one blocking call poisons the loop. A requests.get or a synchronous database driver inside an async def stalls every other request that worker is handling, which presents as uniformly terrible latency under load with low CPU usage. Use async clients, or keep the method synchronous and let BentoML run it in its own worker. For calling a synchronous method without blocking, BentoML provides .to_async on the method, which gives you an awaitable form of it.

Second, async raises concurrency, so it raises your exposure to the thing you are calling. A single worker holding 64 in-flight requests is 64 simultaneous calls to the metadata service. Give your client a timeout — the example above does — and size your own concurrency with the downstream's capacity in mind, not just your own.

Generators are the third shape. An API that yields produces a streaming response, which is how you serve token-by-token output from a text generation model so that time-to-first-token is short even when the full completion takes twenty seconds. Streaming changes the error model: once the first byte is sent the status code is fixed, so a failure halfway through arrives as a truncated stream rather than a 500. Clients need to handle that, and your timeout needs to cover the whole stream rather than the first chunk.

If you are serving large language models specifically, this is the point where BentoML becomes the packaging and serving layer around a dedicated inference engine rather than the thing doing the generation. A common production shape is a BentoML service wrapping vLLM or TGI: the engine owns batching and KV-cache management on the GPU, and BentoML owns the API surface, the dependency spec, the container and the deployment. In that arrangement you usually leave BentoML's own adaptive batching off, because two batching layers fighting each other is worse than either alone.

Multi-service graphs with bentoml.depends

A single service that preprocesses on CPU and infers on GPU is a scaling problem with no good answer. The two halves want different hardware and scale at different rates, but they share one replica, so you buy GPUs to run tokenisation.

bentoml.depends() splits them. You define each stage as its own service, and declare the dependency as a class attribute. The caller then uses the dependency as if it were a local object, and BentoML handles the transport.

import bentoml
from typing import List

@bentoml.service(resources={"gpu": 1}, traffic={"timeout": 60, "concurrency": 32})
class Embedder:
    def __init__(self) -> None:
        self.model = load_embedding_model()

    @bentoml.api(batchable=True, max_batch_size=64, max_latency_ms=500)
    def embed(self, texts: List[str]) -> List[List[float]]:
        return self.model.encode(texts).tolist()


@bentoml.service(resources={"cpu": "2"}, workers=4, traffic={"timeout": 60})
class SearchGateway:
    embedder = bentoml.depends(Embedder)

    @bentoml.api
    def search(self, query: str, top_k: int = 5) -> dict:
        vector = self.embedder.embed([self._clean(query)])[0]
        return {"hits": self._lookup(vector, top_k)}

    def _clean(self, text: str) -> str:
        return " ".join(text.split()).lower()[:2000]

    def _lookup(self, vector, top_k):
        ...

Four things follow from this structure, and they are the reason to use it.

Independent scaling. The gateway and the embedder are separate services with their own resources, workers and traffic, so a deployment can run eight cheap gateway replicas in front of two GPU replicas. The CPU work no longer rides on GPU hardware.

Batching where batching pays. The gateway's work is per-request and cheap, so it stays unbatched. The embedder's work is a GPU matrix operation, so it batches. With depends, the embedder sees calls from every gateway replica, which is precisely the traffic consolidation that makes batches form.

One deployable unit. The graph still builds into a single Bento with one service: entry point, and still serves locally with one bentoml serve. You are not operating two repositories or two release trains.

A real network boundary. The call looks local and is not. Arguments and return values are serialised, so a dependency call is a request with its own latency and its own failure modes. Keep the payloads sensible — passing a 50 MB array between services costs real time — and remember the inner service's timeout must be shorter than the outer one's, or the gateway will still be waiting on a call the embedder has already abandoned.

Where to draw the boundary is a judgement call. Split when the halves want different hardware, scale at different rates, or are owned by different teams. Do not split a fast function into its own service for tidiness; you will have added a network hop and a failure mode to save nothing.

Containerizing, and what runs it

bentoml containerize classifier:latest turns a Bento into an OCI image. That image is the deployment unit for everything outside BentoCloud — Kubernetes, ECS, Cloud Run, a single VM. Any of Docker, Podman or Buildah can serve as the build backend.

bentoml build
bentoml containerize classifier:latest
docker run --rm -p 3000:3000 classifier:<version>

The generated image is a real Docker image and all your usual rules apply to it: pin by digest for deploys, scan it in CI with Trivy, and keep the base current. The docker section of the build config is where you influence what goes in — distro for the flavour, system_packages for shared libraries, base_image when a hardened base is mandated, setup_script for anything else.

Two practical notes. Building on an Apple Silicon laptop for an x86 cloud node produces an image that will not start there, so build for the target platform explicitly or build in CI on the target architecture. And if your organisation routes traffic through a TLS-inspecting proxy, the image build is where it bites: pip install and model downloads fail with CERTIFICATE_VERIFY_FAILED or SELF_SIGNED_CERT_IN_CHAIN. Point REQUESTS_CA_BUNDLE, SSL_CERT_FILE and PIP_CERT at your exported CA bundle. Never disable verification to make a build pass; you will have published that workaround to everyone who copies the Dockerfile.

On Kubernetes, a BentoML image is an ordinary Deployment plus a Service plus an Ingress, and the interesting part is making the Kubernetes settings agree with the BentoML ones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: classifier
spec:
  replicas: 2
  selector:
    matchLabels: { app: classifier }
  template:
    metadata:
      labels: { app: classifier }
    spec:
      containers:
        - name: classifier
          image: registry.example.com/classifier@sha256:...
          ports:
            - containerPort: 3000
          resources:
            requests: { cpu: "2", memory: "4Gi" }
            limits: { cpu: "4", memory: "6Gi" }
          env:
            - name: LOG_LEVEL
              value: info

Memory is the one to get right. The container limit must cover workers copies of the model plus the runtime, and a limit below that gets you an OOM kill during startup that looks like a crash loop with no Python traceback. CPU requests should reflect what one replica actually uses at your chosen concurrency, because that is what the scheduler and any CPU-based autoscaler act on.

For horizontal scaling, the metric to drive an HPA on is the one that reflects admitted work rather than CPU: bentoml_service_request_in_progress compared against your configured concurrency is a much better signal for a GPU service than CPU utilisation, which barely moves while the accelerator is saturated. Exposing it to the HPA needs a Prometheus adapter, which the observability section sets up the scraping half of.

Self-hosted, you also own the admission control that external_queue would have given you on BentoCloud: an ingress-level rate limit, or a queue in front of the service for work that can be deferred. A GPU service with no admission control degrades badly, because every extra request consumes memory and attention without making anything finish sooner.

If your platform is Kubernetes-native and your team would rather describe serving with custom resources than with a Python class, KServe and Seldon Core are the neighbours to compare against; Triton is the one to compare against when you need multi-framework inference and maximum GPU efficiency above all. BentoML's distinguishing position is that the service is Python code and the packaging is first-class.

Configuration, models and secrets

Three kinds of input reach a running service, and conflating them is the source of most "it works in staging" incidents.

Build-time configuration is anything recorded in the Bento: the service entry point, dependency pins, envs, labels, the Docker settings. It is fixed when you build, which is exactly what you want for the things that define the artifact.

Runtime configuration is anything that differs between environments — endpoints, log levels, feature flags, model aliases. This belongs in environment variables supplied by the platform, read with ordinary os.environ in your service. The test is simple: if changing a value requires rebuilding the Bento, you have put it in the wrong place, and you have also broken the promise that the artifact you tested is the artifact you deployed.

Secrets belong in neither. A Bento is a shareable archive and its envs are baked in, so a token in bentofile.yaml is a token in every copy of that Bento, in the image, in the registry, and in anyone's bentoml export. Inject secrets at runtime from the platform's store — Kubernetes Secrets, Vault, a cloud secret manager, or BentoCloud's own secrets — and have the service read them from the environment.

import os
import bentoml

@bentoml.service
class Scorer:
    def __init__(self) -> None:
        self.feature_store_url = os.environ["FEATURE_STORE_URL"]   # runtime config
        self.api_key = os.environ["FEATURE_STORE_KEY"]             # injected secret
        self.model_alias = os.environ.get("MODEL_ALIAS", "production")

Failing loudly on a missing required variable, as os.environ[...] does, is deliberate. A service that starts with a silently empty key fails later, per request, in a way that looks like a model problem.

Model weights are the fourth kind of input and the one with a cost attached. There are two defensible patterns. Bake the weights into the Bento — register them in the model store or reference them from the build — and you get a large, slow-to-pull image with a fast, deterministic start, and a Bento tag that uniquely identifies both code and weights. Fetch them at deployment time, using @bentoml.on_deployment to download once before workers spawn, and you get a small image, a slower first start, and the need to pin the version you fetch so the deployment stays reproducible. Downloading in __init__ instead means every worker downloads, which multiplies both time and egress.

Cold start is the number that decides between them. If you want to scale to zero or scale up quickly under bursts, pulling a 20 GB image and then downloading 6 GB of weights is a multi-minute cold start that your autoscaler cannot hide. Bake weights, keep a warm minimum replica count, or both.

This is also the seam where BentoML meets the rest of an MLOps stack. A common arrangement in production is MLflow as the registry and lineage system, BentoML as the thing that packages a specific model version into a servable artifact, and Evidently watching the inputs and outputs for drift once it is live. Hugging Face models can be referenced directly through BentoML's model APIs rather than being copied by hand, which keeps the resolved revision recorded in the Bento.

Metrics, and what to watch

Beginner knew the service worked because a curl returned the right answer. Production asks a harder question: is it slow, and if so, where? BentoML answers it with Prometheus metrics on /metrics, enabled by default, served on the same port as the API.

curl -s http://localhost:3000/metrics | grep bentoml_service

Four default series carry most of the diagnostic weight:

The histogram buckets deserve one configuration decision, because a histogram can only resolve latency at bucket boundaries. The defaults span a wide range; if your service answers in 30 ms, every request lands in the first bucket and your p95 is unknowable. Set buckets around your actual distribution:

@bentoml.service(
    metrics={
        "enabled": True,
        "namespace": "bentoml_service",
        "duration": {"buckets": [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5]},
    }
)
class Classifier: ...

The duration block takes either explicit buckets or a min/max/factor triple for exponential buckets. Pick buckets that bracket your p50 and p99 with several steps between them.

Beyond the defaults, the bentoml.metrics module gives you Counter, Histogram and Gauge wrappers for the things only you know are interesting: predicted class distribution, input length, cache hit rate, how often a request fell back to a smaller model. These are what turn a dashboard from "the service is up" into "the service is behaving".

Scraping is the standard arrangement — Prometheus scraping port 3000, Grafana on top — and for request-level causality rather than aggregates, BentoML emits OpenTelemetry traces, which is what you want when a gateway-plus-model graph is slow and you need to know which hop owns the time.

A dashboard worth having answers, in this order: what is the error rate per endpoint; what is p95 latency and is it moving; how many requests are in progress compared with configured concurrency; and what batch sizes are actually forming. Those four together distinguish the three failure shapes you will meet. Errors up with latency flat is a code or dependency fault. Latency up with in-progress near concurrency is saturation, so add replicas or raise throughput. Latency up with in-progress low is something downstream being slow, or batching waiting out its window for company that never arrives.

Testing a service you can actually trust

Model code is notoriously undertested, and the usual reason is that the only test anyone writes needs a GPU and a running server. BentoML services are classes, which gives you a cheaper layer.

Test in three tiers, cheapest first.

Unit tests on the logic. The methods on a service that are not decorated with @bentoml.api are ordinary methods. Preprocessing, validation, formatting, the fallback path — all testable with no model and no server. Writing a service so that the API method is thin and delegates to testable helpers is not purity, it is how you get a fast test suite.

from service import SearchGateway

def test_clean_collapses_whitespace_and_truncates():
    g = SearchGateway.__new__(SearchGateway)   # no __init__, no model load
    assert g._clean("  Hello   WORLD ") == "hello world"
    assert len(g._clean("x" * 5000)) == 2000

Contract tests on the schema. Because inputs and outputs are validated from type hints, the schema is part of the code and can drift from what clients expect. A test that asserts the response keys and types — ideally generated from the same Pydantic model the clients use — catches the renamed field before a consumer does.

One end-to-end test against a served instance. Start the service, send a handful of representative requests, assert on status and shape. Keep it small and keep it in CI, because this is the test that catches the failures the others structurally cannot: a missing module that was excluded from include, a system package that is absent from the image, a model path that only exists on the developer's laptop.

bentoml serve service:Classifier &
SERVER_PID=$!
sleep 15
curl -sf -X POST http://localhost:3000/v1/classify \
  -H 'Content-Type: application/json' \
  -d '{"text":"a representative request"}' | tee /tmp/out.json
python -c "import json;d=json.load(open('/tmp/out.json'));assert 'label' in d, d"
kill $SERVER_PID

Two further tests pay for themselves at this level. A build test that runs bentoml build in CI proves the dependency spec resolves and the include list is complete, which is a failure you absolutely want at merge time rather than at deploy time. And a regression test on predictions: a frozen set of inputs with recorded outputs, asserted within a tolerance. It will not tell you the model is good, but it will tell you loudly when a dependency upgrade quietly changed the numbers, which is the failure mode that otherwise reaches users first.

CI/CD and promoting Bentos

The pipeline shape that works mirrors the artifact: build once, promote the same thing. A Bento's version is computed at build time, so the tag you produce in CI is the identity you carry through every environment.

name: bento
on:
  push:
    branches: [main]
jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install
        run: pip install "bentoml==1.4.39" -r requirements.txt
      - name: Test
        run: pytest -q
      - name: Build Bento
        run: |
          bentoml build
          bentoml list
      - name: Containerize
        run: |
          TAG=$(bentoml list -o json | python -c "import json,sys;print(json.load(sys.stdin)[0]['tag'])")
          echo "BENTO_TAG=$TAG" >> "$GITHUB_ENV"
          bentoml containerize "$TAG"

Four conventions make this reliable.

Pin BentoML itself. pip install "bentoml==1.4.39" in CI and the same pin in python.packages. An unpinned install means the tool that builds your artifact changes without a commit, and the one thing you do not want to debug is a build difference you cannot reproduce.

Capture the built tag as an output. Everything downstream — the image, the deploy manifest, the release note — refers to that tag. Parsing it once and passing it along is what keeps staging and production provably identical.

Separate build from deploy. The build job produces and pushes an artifact. Deployment jobs, gated per environment, consume a tag that already exists. Promotion becomes "deploy the tag that passed staging", with no rebuild, which is the entire point.

Scan and sign the image before promotion. A Bento image contains your dependency tree, and the 1.4 line has had enough security fixes of its own to make patch currency a real concern. Trivy in the pipeline, failing on high severities, plus whatever signing your platform expects.

For the deploy half, BentoCloud takes a config file:

services:
  Classifier:
    scaling:
      min_replicas: 1
      max_replicas: 6
      policy:
        scale_up_stabilization_window: 180
        scale_down_stabilization_window: 600

bentoml deploy classifier:<version> -n classifier-prod -f deployment.yaml

The stabilization windows are the settings people skip and then regret. Both are in seconds, in the range 0 to 3600. The scale-up window keeps a traffic spike from immediately provisioning replicas that will be idle by the time they finish starting — relevant when starting means pulling a large image and loading weights. The scale-down window keeps a brief lull from destroying capacity you are about to need, which is exactly the thrashing pattern that makes a GPU service both expensive and slow. Make scale-down considerably longer than scale-up, and make scale-up longer than your cold start is short.

If you are deploying to your own cluster instead, the equivalent is an image digest in a Deployment manifest, reconciled by Argo CD or Flux, with an HPA keyed on in-progress requests. The artifact discipline is the same; only the actuator changes.

Work that does not fit in a request

Some inference does not belong behind a synchronous HTTP call. Transcribing an hour of audio, scoring a 200,000-row file, generating a long document — these take minutes, and a request/response API is the wrong shape for them. Held open, they burn a worker slot for the whole duration, they break the moment a load balancer's own idle timeout expires, and a client that disconnects takes the result with it even though the work completed.

Raising traffic.timeout to cover the worst case does not fix this; it generalises the problem. A timeout sized for the slowest job is also the timeout that lets a stuck fast request hold a slot for ten minutes, so you lose the protection timeouts exist to give you.

BentoML's answer is @bentoml.task. A task endpoint accepts the work, puts it on a queue, and returns immediately with a handle. The client comes back later for the result, and the server runs the job in the background without a connection held open for it.

import bentoml

@bentoml.service(traffic={"timeout": 60, "concurrency": 8}, workers=2)
class BulkScorer:
    def __init__(self) -> None:
        self.model = load_model("/models/clf")

    @bentoml.api
    def score(self, text: str) -> float:
        """Interactive path: fast, synchronous, tight timeout."""
        return float(self.model.predict_proba([text])[0])

    @bentoml.task
    def score_file(self, uri: str) -> dict:
        """Batch path: minutes of work, submitted and polled."""
        rows = read_rows(uri)
        scores = self.model.predict_proba(rows)
        out = write_results(uri, scores)
        return {"rows": len(rows), "output": out}

Three consequences shape how you design around it.

Submission and execution are now separate failures. A task can be accepted and still fail later, so "the call returned 200" no longer means "the work happened". Whatever you build on top needs to handle a job that was accepted and then died — which means the result has to be durable somewhere you control, usually object storage with the output path derived from the input, not just a value sitting in memory waiting to be collected.

Make the work idempotent. Clients retry, deployments restart, and a queued job may be picked up more than once in the lifetime of a flaky environment. If running the same submission twice produces two charges, two rows or two emails, you have built a correctness bug that only appears under the conditions where you least want one. Derive a deterministic output key from the input, and make a second run overwrite rather than append.

Size concurrency for the long job, not the short one. A service exposing both an interactive API and a task API has two very different workloads sharing one replica's resources. The example above keeps concurrency low precisely because a handful of file-sized jobs will use all of the memory and CPU the replica has. If the two workloads diverge much further than that, split them: a task service and an interactive service, each with its own resources, workers and scaling, composed with bentoml.depends if they share a model.

The judgement call is simply where the boundary sits. Anything a human is waiting on, in a browser, stays synchronous and gets a timeout in seconds. Anything measured in minutes becomes a task. The uncomfortable middle — ten to thirty seconds — is usually best served by streaming, so the caller gets output progressively instead of staring at a blank response, which the async section covered.

Debugging beyond the basics

The failures at this level are rarely "it does not start". They are "it starts and behaves unlike the thing you tested". Four classes cover almost all of them.

Old-API errors. ModuleNotFoundError: No module named 'bentoml.io', or an AttributeError about bentoml.Service, means pre-1.2 code on a 1.4 runtime. This happens most often when someone copies a snippet from a blog post. The fix is the rewrite described at the top of this guide, not a version downgrade.

Import and entry-point errors. A failure to import the service almost always means the module:Class string is wrong, you are in the wrong directory, or — the one that only appears after a build — a module that exists locally was never included in the Bento. That last case is why bentoml build belongs in CI and why a smoke test against the container is worth the thirty seconds it costs.

Memory kills with no traceback. A worker that dies during startup with no Python error, repeatedly, is nearly always the container memory limit against workers copies of the model. Count the copies before you suspect the model.

GPU invisibility. Torch reporting no GPU inside a container that is scheduled on a GPU node is a driver-and-CUDA mismatch or a missing device request, not a BentoML fault. The 1.4 line has corrected its generated CUDA base images at least once, so being on an old patch is worth ruling out early.

The investigative tools, in the order they usually help:

bentoml env                       # system and package versions — first thing in any bug report
bentoml --verbose serve service:Classifier
bentoml get classifier:latest     # what is actually in the Bento: deps, models, config
bentoml list                      # confirm the tag you think you are serving exists
curl -s localhost:3000/metrics | grep -E 'in_progress|duration_seconds_count'

bentoml get is the underused one. When behaviour differs between environments, the question is almost always "what is in this artifact", and that command answers it with the recorded dependency set and configuration rather than your assumptions about them.

One habit worth building: when a service is slow, decide first whether it is saturated or waiting, using in-progress requests against configured concurrency. Saturated means the fix is capacity or throughput — replicas, workers, batching. Waiting means the fix is somewhere else — a downstream call, a batching window, a lock. Teams waste days adding replicas to a service that was waiting, and the metric that distinguishes the two was there the whole time.

Putting it all together

Here is one service that uses everything above, as a shape to copy. A CPU gateway validates and prepares requests; a GPU service batches the model call; configuration arrives from the environment; metrics are tuned to the real latency.

import os
from typing import List

import bentoml
from pydantic import BaseModel


class ScoreRequest(BaseModel):
    text: str
    threshold: float = 0.5


@bentoml.service(
    resources={"gpu": 1},
    traffic={"timeout": 30, "concurrency": 32},
    workers=1,
    metrics={
        "enabled": True,
        "duration": {"buckets": [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5]},
    },
)
class Model:
    def __init__(self) -> None:
        self.model = load_model(os.environ.get("MODEL_PATH", "/models/clf"))

    @bentoml.api(batchable=True, max_batch_size=32, max_latency_ms=200)
    def score(self, texts: List[str]) -> List[float]:
        return [float(p) for p in self.model.predict_proba(texts)]


@bentoml.service(
    resources={"cpu": "2", "memory": "2GiB"},
    traffic={"timeout": 60, "concurrency": 64},
    workers=4,
    path_prefix="/v1",
)
class Gateway:
    model = bentoml.depends(Model)

    @bentoml.api
    def classify(self, req: ScoreRequest) -> dict:
        cleaned = " ".join(req.text.split())[:4000]
        if not cleaned:
            return {"label": "empty", "score": 0.0}
        score = self.model.score([cleaned])[0]
        return {
            "label": "positive" if score >= req.threshold else "negative",
            "score": score,
        }

service: "service:Gateway"
description: "Text classifier: CPU gateway in front of a batched GPU model"
labels:
  team: ml-platform
include:
  - "service.py"
  - "src/**/*.py"
python:
  requirements_txt: "./requirements.txt"
  lock_packages: true
docker:
  distro: debian
  python_version: "3.11"

bentoml serve service:Gateway                # both services, one command
bentoml build
bentoml containerize gateway:<version>

Read the choices rather than the code. One GPU worker, because a second copy of the weights would not fit. Batching on the inner service only, with a 200 ms window chosen to be close to one batch's own duration. concurrency higher on the gateway than on the model, because the gateway's work is cheap and it is deliberately the queueing layer. The inner timeout shorter than the outer, so the model gives up before the gateway does. Model path from the environment, so staging and production share a Bento. Narrow metric buckets, so p95 is a number rather than a guess.

Where you are now

You can predict what a served Bento does: how many processes exist, what each one loads, what admits and queues requests, and where a batch is assembled. You can read a slow service and say whether it is saturated or waiting, because you know which metric distinguishes them. You can split a service along a hardware boundary with bentoml.depends and scale the halves separately. You can build a Bento that is reproducible, promote one artifact through environments without rebuilding, and keep secrets out of an archive that other people can read.

What is left is the senior layer: owning this as a platform. Failure modes under real load and which component breaks first, the trust model and supply chain in full, multi-tenancy, cost control on GPUs, upgrade and migration strategy, and the judgement call about when BentoML is the wrong tool and KServe, Ray Serve or Triton is the right one. Before you go there, spend a week with the knob-versus-effect table from the last section. Nothing in the senior material lands properly until you have measured one real service yourself.

Sources

• BentoML documentation
• Install BentoML
• Services
• Adaptive batching
• Parallelize requests
• Bento build options
• CLI reference
• Metrics
• Create deployments
• Concurrency and autoscaling
• BentoML releases
• bentoml on PyPI

Mid-level BentoML interviews are not vocabulary tests. The interviewer wants to know whether you have run a model service under load and learned something from it. The questions below are the scenarios that come up, with the reasoning a strong answer shows and the gap a weak one leaves.

How this level is judged

A beginner answer names features. A mid-level answer names a mechanism and a measurement. Interviewers listen for three things: do you know what actually runs when a Bento is served, do you reach for a metric before a knob, and do you state the trade-off rather than claiming a setting is simply better.

The single most common disqualifier at this level is answering in the pre-1.2 API. If you describe Runners and bentoml.io descriptors, an interviewer running BentoML 1.4 concludes you last touched it in 2023. Know that the old API existed, say so deliberately, and answer in the current one.

Scenario: the service is slow under load

"Your BentoML service answers in 80 ms in testing. In production at 60 requests per second, p95 is four seconds. What do you check first?"

Do not start guessing at settings. Start by deciding whether the service is saturated or waiting, because the two have opposite fixes and the metrics tell you which within a minute.

Pull bentoml_service_request_in_progress and compare it with the concurrency value configured in traffic. If in-progress sits at or above concurrency, the replica is admitting all the work it is designed for and the rest is queueing: this is saturation, and the fix is capacity or throughput — more replicas, more workers if the work is CPU-bound Python, or batching if it is a GPU model. If in-progress is low while latency is high, nothing is queueing at the gate and the time is being spent inside the request: a slow downstream call, a lock, or a batching window waiting out its full max_latency_ms for company that never arrives.

Then check the error rate from bentoml_service_request_total by status, because a service that is slow and erroring is usually one problem, not two, and p95 computed over a mix of fast failures and slow successes is misleading.

A strong answer also says what it would rule out: whether the model itself regressed (compare single-request latency outside the server), and whether the latency histogram has the resolution to answer the question at all — default buckets spanning seconds cannot distinguish 80 ms from 300 ms.

What a weak answer misses: it jumps straight to "add more workers". Workers without concurrency headroom gives you idle processes, and on a GPU service a second worker may simply fail to load a second copy of the weights.

Scenario: batching made things worse

"You enabled adaptive batching on a GPU endpoint and p99 went up with no throughput gain. Explain."

This is the diagnostic the interviewer most wants to see you do correctly, because it tests whether you understand that batching spends latency to buy throughput.

Read bentoml_service_adaptive_batch_size. If the distribution is pinned at 1, no batches are forming: traffic is too sparse for a second request to arrive inside the window, so every request waits out max_latency_ms and is then dispatched alone. You have paid the full latency cost for none of the benefit. The causes are low arrival rate, or traffic spread so thinly across replicas that no single one sees concurrent requests.

Three fixes, in order of preference. Lower max_latency_ms toward something like one batch's own execution time, so the waiting cost is proportionate. Consolidate traffic onto fewer replicas so batches can form — this is the counter-intuitive one, and it is also the argument for putting the batched model behind a gateway with bentoml.depends, since the gateway's replicas all feed one model service. Or accept that this workload is interactive and sparse, and turn batching off.

If the batch sizes are healthy and throughput still did not improve, the model is not getting cheaper per item with batch size, which you verify offline by timing it at sizes 1, 8, 32 and 64. Some models and some hardware simply do not amortise.

What a strong answer adds: the point that batching is right for bulk and asynchronous work and questionable for interactive paths, and that the two can coexist as separate endpoints with different settings.

Scenario: out of memory on startup

"A service with workers=4 works on a laptop and crash-loops in Kubernetes with no Python traceback. Why?"

Because __init__ runs once per worker process, not once per service. Four workers load four copies of the model. If the container memory limit covers one copy and the runtime, the fourth worker — or the second — gets killed by the kernel, which produces no traceback because Python never got to raise anything. A crash loop with no error text is the signature.

The fix depends on the shape of the work. For a large model, one worker per replica and throughput from batching. For a small model with CPU-bound Python, more workers are right and the memory limit needs to cover them all. If something expensive genuinely needs to happen once for the whole deployment rather than per process — downloading weights to a shared path, say — that belongs in an @bentoml.on_deployment hook, which runs before workers spawn.

What a weak answer misses: it blames the model or the cluster. The interviewer is specifically checking whether you know the per-worker cost, because it is the first thing that bites anyone moving a BentoML service from a laptop to a limited container.

Scenario: staging and production disagree

"The same code is deployed to staging and production. Staging is fine, production returns different predictions. How do you find it?"

Stop comparing branches and compare artifacts. bentoml get on both Bento tags prints the recorded dependency set, the model references and the service configuration. Nine times out of ten the difference is visible immediately: a different dependency version, a different model path or alias, a different environment variable.

Then ask the structural question: are those two deployments running the same Bento version at all? If the pipeline rebuilds per environment rather than promoting one artifact, they are different software by construction, and no amount of comparing commits will prove otherwise. The fix is build-once-promote-many, with a resolved Bento version and image digest in each deploy rather than :latest.

The second likely cause is a model alias resolved at runtime. If the service reads something like a production alias from a registry, a registry change becomes a production change with no deployment to point at. Resolve aliases at build or deploy time and record what they resolved to.

What a strong answer adds: lock_packages is on by default precisely to make this debuggable; if someone turned it off for build speed, the Bento records ranges instead of pins and two builds a week apart are legitimately different.

Scenario: the cold start is unacceptable

"Your autoscaler adds a replica during a traffic spike and the new replica takes four minutes to serve its first request. What do you do?"

Break the four minutes into its parts before changing anything, because each part has a different remedy: pulling the image, loading the model into process memory, and any weights downloaded at startup.

If weights are being fetched at start, that is usually the biggest slice. Fetching them in __init__ is the worst case, because every worker downloads its own copy — the hook that exists for this is @bentoml.on_deployment, which runs once before workers spawn. Better still, bake the weights into the Bento so the image pull is the only transfer and the start is deterministic; you trade a bigger, slower-pulling image for a predictable start, and for a service that scales under bursts that trade is almost always correct.

If the image pull dominates, the levers are image size — distro, system_packages, and not shipping notebooks and datasets through loose include patterns — and node-level caching so a scaled-up replica lands somewhere the layers already exist.

Then make the scaling policy honest about what is left. Keep min_replicas at one or more rather than scaling to zero, and set scale_up_stabilization_window so a momentary spike does not provision replicas that will be idle by the time they are ready. Keep scale_down_stabilization_window considerably longer, so a short lull does not destroy capacity you are about to need; thrashing a GPU service is how you get both the bad latency and the big bill.

What a strong answer adds: that scale-to-zero is a product decision, not a technical one. If the first user after an idle period can wait, it saves real money on GPUs. If they cannot, holding a warm replica is the cheaper option once you price the lost requests.

Scenario: inheriting a 1.1 service

"You inherit a BentoML service written against 1.1, with Runners and IO descriptors. Your task is to get it onto the current version. How do you plan it?"

Call it a rewrite, not an upgrade, and say so in the first sentence. The 1.2 change replaced the service definition, the schema mechanism and the composition mechanism at once: bentoml.Service(...) with @svc.api(input=..., output=...) becomes a class with @bentoml.service and @bentoml.api; IO descriptors become ordinary type hints and Pydantic models; runners become bentoml.depends. Almost every line of the service file changes, even though the model code underneath it usually does not.

The plan that works: pin the existing service so it keeps running, and build the new one beside it rather than editing in place. Write the regression fixture first — a frozen set of inputs with the old service's recorded outputs — because that is the only thing that will tell you the rewrite preserved behaviour. Map each runner to either a method on one service or a separate service with depends, deciding by hardware and scaling rather than by how the old code was arranged. Then re-derive the tuning: concurrency, workers and batching settings from the old configuration are not transferable, because the process model and the batching implementation are not the same.

Run both in parallel and compare outputs on live traffic if you can. Cut over by DNS or ingress, keep the old deployment available for a rollback window, and only then delete it.

What a weak answer misses: it treats the type hints as cosmetic. They are the schema, so the HTTP contract can change during the rewrite — a field renamed or a response shape flattened — and clients break. Diffing the generated OpenAPI schema before and after is the cheap way to catch that.

Trade-off: batch size versus latency budget

"How do you choose max_batch_size and max_latency_ms?"

Measure, then set. Time the model offline at increasing batch sizes and find where per-item latency stops improving — that is max_batch_size, and going beyond it buys nothing while increasing the memory one call needs and the latency the whole batch pays.

max_latency_ms is the queueing budget and should be set relative to the model's own batch duration, not to a round number. If a batch takes 400 ms, a 1000 ms window means a lone request at low traffic can take 1.4 seconds. If a batch takes 8 seconds, a 1000 ms window is noise. Then validate under real traffic with the batch-size histogram, and state the service's latency SLO out loud: the budget has to fit inside it with room for the model.

The answer that impresses is the one that says what it would not batch: anything a user is waiting on interactively with a tight SLO, and any endpoint whose request signature cannot be expressed as one batched container parameter.

Trade-off: workers, concurrency, replicas

"You need three times the throughput. Which do you change?"

It depends on what the bottleneck is, and saying so is the answer.

CPU-bound Python work scales with workers, because each worker is a separate process with its own interpreter lock — up to the cores and memory a replica has. I/O-bound work does not need more workers at all; it needs async def APIs so one worker holds many in-flight requests, plus concurrency raised to admit them. GPU-bound work usually cannot use more workers, because a second copy of the weights will not fit, so throughput comes from adaptive batching and then from replicas.

concurrency is not a throughput lever in itself. It is a declaration of how much work one replica is designed to handle at once, and it is what an autoscaler reasons about. Raising it without adding capacity just lengthens the queue and ruins latency while the replica count still looks healthy.

What a strong answer adds: cost. Replicas are the lever that always works and always costs money, especially on GPUs. Batching raises utilisation of hardware you are already paying for, which is why it comes first.

Trade-off: one service or several

"When do you split a BentoML service into multiple services with bentoml.depends?"

Split along a hardware boundary, a scaling boundary or an ownership boundary. The canonical case is CPU preprocessing in front of GPU inference: in one service, the two share a replica, so you buy GPU hardware to run tokenisation and you cannot scale the cheap half independently. Split, and the gateway runs eight cheap replicas in front of two GPU replicas. A secondary benefit is that batching works better, because the model service sees consolidated traffic from every gateway replica.

Say what you would not split. A fast pure function does not need its own service; you would have added a network hop, a serialisation cost and a failure mode for nothing. And note that the dependency call looks local but is not — payloads are serialised, so passing large arrays between services costs real time, and the inner service's timeout must be shorter than the outer one's or the gateway waits on a call that was already abandoned.

Trade-off: BentoML or something else

"Why BentoML rather than KServe, Ray Serve or Triton?"

Answer on the axis of what each one is organised around. BentoML is Python-code-first and packaging-centric: the service is a class, the dependency spec and model references are part of a versioned artifact, and that artifact becomes an OCI image you can run anywhere. That suits a team whose serving logic is real Python — custom pre- and post-processing, a multi-stage graph, business rules — and who want one artifact to promote.

KServe and Seldon Core are Kubernetes-native: serving is described in custom resources and the platform owns the lifecycle. Better fit when the team is a Kubernetes platform team and wants declarative, GitOps-managed inference. Triton is NVIDIA's multi-framework server, the choice when squeezing a GPU is the dominant requirement and you are willing to work within its model-repository conventions. Ray Serve fits teams already running a Ray cluster for distributed Python.

What a weak answer misses: it argues about features. The interviewer is checking whether you choose tools by the shape of the team and the workload, and whether you can name a case where you would not pick BentoML — which is worth volunteering.

Questions you should ask them

Interviews at this level go better when you ask the questions an operator would ask. What is the latency SLO on the inference path, and is anything interactive? Are you on GPUs, and what is current utilisation? Is a cold start acceptable, or do you hold warm replicas? Do you build once and promote, or build per environment? Where do model weights live — baked into the image or fetched at start? And is any of this constrained by data residency, which in Gulf and Egyptian deployments often decides between a hosted platform and a self-managed cluster in a local region before any technical argument gets made.

What separates a strong answer

Four habits, in order of how much they move an interviewer.

A metric before a knob. Every scenario answer should name the series you would look at — in-progress requests, the duration histogram, the batch-size histogram — before naming a setting you would change.

The mechanism, not the feature. "__init__ runs per worker" beats "you can configure workers". "Batching spends latency to buy throughput" beats "batching improves performance".

The trade-off stated out loud. Nothing here is free. Batching costs latency, workers cost memory, replicas cost money, splitting services costs a network hop. Naming the cost is what distinguishes someone who has run this from someone who has read about it.

The current API. Classes with @bentoml.service, methods with @bentoml.api, type hints for schemas, bentoml.depends for composition. If you mention Runners, mention them as the legacy pattern you would migrate away from.

These are the mistakes that show up once BentoML is carrying real traffic, the practices that pay back most, and a set of recipes to run when something is wrong. Everything here is for BentoML 1.4.x.

Errors at this level

You copied a tutorial and it imports bentoml.io. Symptom: ModuleNotFoundError: No module named 'bentoml.io', or an AttributeError about bentoml.Service or runners. Cause: the code predates 1.2, when services became classes decorated with @bentoml.service and IO descriptors were replaced by plain type hints. Fix: rewrite it in the current API. Do not downgrade BentoML to make an old snippet run — you will be pinning yourself to an unsupported line to save twenty minutes.

The container crash-loops with no traceback. Symptom: workers die during startup, repeatedly, and the logs end mid-initialisation with nothing Python-shaped in them. Cause: the container memory limit against workers copies of the model, because __init__ runs once per worker process. Fix: either reduce workers or raise the limit to cover every copy plus the runtime, and leave headroom for the request path.

It serves locally and fails to import in the container. Symptom: a build and containerize that both succeed, then an import error inside the image naming one of your own modules. Cause: the module is not in include and so never entered the Bento. Fix: widen include ("src/**/*.py"), and add bentoml build plus a container smoke test to CI so this fails at merge time.

The Bento is enormous. Symptom: a build that takes minutes and an image measured in tens of gigabytes. Cause: include patterns or a missing .bentoignore pulling in notebooks, checkpoints, datasets and .git. Fix: a .bentoignore with the blanket exclusions and an explicit include list. Check the size after every change to those patterns.

Latency got worse when you enabled batching. Symptom: p99 rises by roughly max_latency_ms, throughput unchanged. Cause: traffic is too sparse for batches to form, so every request waits out the whole window and is dispatched alone. Fix: read bentoml_service_adaptive_batch_size; if it is pinned at 1, lower the window, consolidate traffic onto fewer replicas, or turn batching off for this endpoint.

Batching raises a signature error. Symptom: an error about the API signature when batchable=True is added. Cause: a batchable API takes exactly one parameter carrying many requests, plus optionally the request context, and that parameter must be a container type. Fix: change the signature to List[str], a NumPy array, or a list of a Pydantic model that holds the several fields you were passing separately.

Results go to the wrong callers. Symptom: no error, but responses are mismatched under concurrency. Cause: batch_dim is wrong, so the server splits the model's output along an axis that is not the batch axis. Fix: set batch_dim explicitly, as a tuple when the input and output axes differ, and add a test that sends distinguishable concurrent requests and checks each gets its own answer.

Everything is slow and CPU is idle. Symptom: uniformly bad latency across all requests on an async def endpoint. Cause: a blocking call inside the event loop — requests, a synchronous database driver — stalling every other in-flight request in that worker. Fix: use async clients, or make the method synchronous so BentoML runs it in its own worker; .to_async gives you an awaitable form of a synchronous method when you need one.

A dependency call times out from the wrong side. Symptom: the gateway returns a timeout while the inner service logs a completed call, or the reverse. Cause: the inner service's traffic.timeout is longer than the caller's, so the caller abandons work the callee is still doing. Fix: timeouts decrease inward. The inner budget must fit inside the outer one with margin.

Torch reports no GPU inside the container. Symptom: a service scheduled on a GPU node that silently runs on CPU, or fails to allocate. Cause: a CUDA-and-driver mismatch, or a missing device request in the pod spec. Fix: check the pod actually requested a GPU, then check your patch level — the 1.4 line corrected its generated NVIDIA CUDA base images, so being on an old patch is worth ruling out first.

pip install fails inside the build behind the corporate proxy. Symptom: CERTIFICATE_VERIFY_FAILED or SELF_SIGNED_CERT_IN_CHAIN during containerize, or when downloading model weights. Cause: a TLS-inspecting proxy the build does not trust. Fix: export the proxy's CA bundle and point REQUESTS_CA_BUNDLE, SSL_CERT_FILE and PIP_CERT at it. Never disable verification; that workaround gets copied.

The practices that pay off, in order of leverage

1. Build once, promote the artifact. A Bento's version is computed at build time, and that tag is the identity you carry through every environment. One build in CI, then deploys that reference an existing tag and an image digest. Everything downstream gets easier: rollback is deploying the previous tag, and "staging and production differ" becomes a question bentoml get can answer. Rebuilding per environment is the single decision that makes model serving undebuggable.

2. Keep lock_packages on. It is the default, and turning it off for build speed trades the only guarantee you have that the Bento you tested is the Bento you shipped. An ML dependency tree is exactly the kind that drifts quietly and changes numeric output.

3. Count the model copies before sizing anything. One number — resident memory of the loaded model times workers — decides your container limit, your node size and whether the service starts at all. Write it in the service's README.

4. Measure before you tune. A saturation sweep takes ten minutes and tells you your real concurrency, your real p95 and whether batching helps. Every setting discussed here is a guess without it, and a guess that is written into a manifest is a guess nobody revisits.

5. Tune the metric buckets. A duration histogram whose buckets span seconds cannot resolve a service that answers in 30 ms, so your p95 is fiction. Set metrics={"duration": {"buckets": [...]}} to bracket your measured p50 and p99. Do this before you build a dashboard, not after you mistrust one.

6. Split along the hardware boundary. CPU preprocessing in front of GPU inference, composed with bentoml.depends, is the highest-value structural change available: it stops you buying GPU time for tokenisation, lets the halves scale independently, and consolidates traffic so batches actually form.

7. Secrets never enter the Bento. A Bento is a shareable archive and its envs are baked in. Inject credentials at runtime from the platform's secret store. Grep your build config for anything credential-shaped today.

8. Keep patches current. The 1.4 line has shipped repeated security fixes — archive symlink validation, file path resolution, template sandboxing — alongside concurrency and container fixes. Pin an exact version so builds are reproducible, and raise that pin on a schedule rather than when something breaks.

9. Make a long job a task. Anything measured in minutes belongs on @bentoml.task rather than behind a timeout you had to inflate. An inflated timeout removes the protection timeouts exist to give you.

10. Put bentoml env in the runbook. It prints system and package versions, it is the first thing any maintainer asks for, and copying it into an incident note costs nothing.

The metrics endpoint is exposed to the internet. Symptom: /metrics answering from outside the cluster. Cause: metrics are served on the same port as the API and are enabled by default, so anything that routes the API routes them too. Fix: keep the service port internal and route only your API paths through the ingress, or scrape from inside the cluster. The series are not secret in themselves, but request counts per endpoint and latency distributions are more operational detail than you want public.

A task was accepted and the result vanished. Symptom: a @bentoml.task submission returned successfully, the client polls, and there is nothing to collect. Cause: the service restarted, or the job failed after acceptance — submission and execution are separate events with separate failures. Fix: write results to durable storage under a key derived from the input, make the work idempotent so a retry overwrites rather than duplicates, and treat a 200 on submission as "queued", never as "done".

The build is reproducible and the deployment is not. Symptom: two deployments of the same Bento version behaving differently. Cause: a model alias resolved at runtime, so a registry change became a production change with no deployment to point at. Fix: resolve aliases at build or deploy time and record what they resolved to, alongside the Bento version.

Habits that prevent the above

Write the sizing arithmetic down, in the repository. Three numbers belong in the service's README: resident memory of one loaded model, the workers count, and the resulting container limit with headroom. Without them, the next person to change workers finds out about the memory ceiling from a crash loop, and the person after that raises the limit without understanding why it was set.

Treat the type hints as the public contract. Because schemas come from annotations, renaming a field or flattening a response is an API change made by editing a function signature, which does not look like an API change in a diff. Define request and response shapes as Pydantic models, keep them in one module, and review changes to that module the way you would review a change to an OpenAPI document.

Give every service one latency budget and derive the rest from it. The SLO comes first; traffic.timeout, max_latency_ms and the metric buckets are all consequences of it. Teams that set those three independently end up with a timeout that cannot be hit, a batching window that does not fit, and a histogram that cannot measure whether they met the target.

Keep the interactive path and the bulk path apart. The same model often serves both a user-facing endpoint with a one-second budget and a bulk job measured in minutes. Different endpoints at minimum, different services when the resource profiles diverge. Sharing one set of traffic settings between them means tuning for one and breaking the other.

Smoke-test the container, not the repository. A test that runs against bentoml serve in your working directory cannot catch a missing include entry or an absent system library, because your laptop has both. Thirty seconds of docker run plus one real request in CI catches the class of failure that otherwise reaches production.

Review the Bento, not just the diff. In a pull request that touches bentofile.yaml, ask for the output of bentoml get on the resulting tag. Dependency changes, model references and size regressions are visible there and invisible in the YAML.

Practice cards

Operational recipes

A service is slow. Which kind of slow?

curl -s http://localhost:3000/metrics | grep -E \
  'bentoml_service_request_in_progress|bentoml_service_adaptive_batch_size|bentoml_service_request_total'

In-progress at or above configured concurrency means saturation: add replicas, or raise throughput with batching or workers depending on the bottleneck. In-progress low with latency high means waiting: look downstream, or at a batching window that is timing out. Batch size pinned at 1 on a batched endpoint means batching is costing latency and returning nothing.

Deciding what to change for more throughput.

A deployment behaves unlike the one you tested.

bentoml list
bentoml get myservice:<staging-version>
bentoml get myservice:<prod-version>
bentoml env

Compare the recorded dependencies, model references and configuration. If the two tags differ, your pipeline is rebuilding per environment and that is the bug to fix before this one.

Before you promote anything.

pytest -q
bentoml build
bentoml get $(bentoml list -o json | python -c "import json,sys;print(json.load(sys.stdin)[0]['tag'])")
bentoml containerize <tag>
docker run --rm -p 3000:3000 <image> &
sleep 20 && curl -sf -X POST localhost:3000/v1/classify \
  -H 'Content-Type: application/json' -d '{"text":"smoke"}'

Four things are verified there: the tests pass, the dependency spec resolves, the Bento contains what you think, and the image actually serves. The last one is the check that catches missing modules and absent system libraries.

Sizing a deployment's scaling policy.

services:
  Classifier:
    scaling:
      min_replicas: 1
      max_replicas: 6
      policy:
        scale_up_stabilization_window: 120
        scale_down_stabilization_window: 600

Both windows are in seconds and accept 0 to 3600. Make scale-down several times longer than scale-up so a brief lull does not destroy capacity you are about to need — the thrashing pattern that makes a GPU service both slow and expensive. Keep min_replicas at one or more unless your cold start, including the image pull and any weight download, genuinely fits inside your SLO.

Rolling back.

Deploy the previous Bento version and image digest. That is the whole procedure, and it only works if you promoted an artifact rather than rebuilding, which is why practice one on this page is practice one.

This is part three of three. It takes every topic from Beginner and Mid one level further, because each one now has a security, a cost, or an ownership dimension, and it adds the work that only exists when BentoML is a platform you run for other teams rather than a library you import.

Up to this point the questions have been about getting a model to answer HTTP requests correctly and quickly. From here they are different: what happens when four teams share one GPU pool, what an attacker gets if they can hand you a Bento, who is allowed to deploy to production, what you do at 02:00 when p99 latency has tripled and the only new thing in the last day was somebody else's model, and how you move a hundred services from one minor version of BentoML to the next without a flag day.

The version this guide is written against is BentoML 1.4.x, which requires Python 3.9 or newer and is Apache-2.0 licensed.

Where this picks up

I start with the runtime topology, because almost every scaling, cost and incident decision below is downstream of knowing which process is doing what.

The architecture you are actually operating

A reader coming from the Mid level knows that bentoml serve service:MyService starts something that answers HTTP on port 3000. What you own as a platform engineer is more specific than that, and the specifics are what let you predict behaviour.

A served Bento is a main server process that owns the listening socket and the HTTP layer, plus a set of worker processes that hold the model and execute your API methods. The workers argument to @bentoml.service sets how many workers that service gets. Each worker runs __init__ independently, which means each worker loads the model independently, which means your memory footprint is roughly the model's resident size multiplied by the worker count, not the model size. This single fact explains more BentoML out-of-memory incidents than anything else on this page. A 6 GB model with workers=4 is a 24 GB container, and nobody wrote 24 anywhere.

When your Bento contains more than one service, each service is its own group of processes with its own worker count and its own resources block. bentoml.depends() wires them together so that calling self.other.predict(x) looks like a local method call. What is happening underneath is a call from one service's worker into the other service, and on a platform that can place services on different machines, a call across the network. That distinction matters for two reasons: the inner call consumes one unit of the inner service's concurrency budget, and its failures surface in the outer service as ordinary Python exceptions, which is exactly how a GPU service's saturation shows up as a 500 in a CPU gateway with no obvious cause.

The resources argument ({"cpu": "2", "memory": "500MiB"}, and a GPU request for accelerated services) is a declaration of what one replica needs. It is honoured by the platform that schedules the replica. Locally it is documentation. That gap is worth saying out loud to teams, because a service that works on a laptop with resources={"cpu": "2"} and quietly uses eight cores will behave very differently the first time something enforces the limit.

Two more pieces of the topology are easy to miss and both show up in incidents. The first is lifecycle hooks. @bentoml.on_deployment runs once before workers spawn, and @bentoml.on_shutdown runs on the way down. The hook that runs once is where expensive, idempotent, shared setup belongs: fetching weights into a shared location, warming a cache, validating that a required secret is present. Putting that work in __init__ instead means every worker does it, which multiplies your cold start by the worker count and, if the work hits a rate-limited download endpoint, turns a deploy into a throttling incident.

The second is background tasks. @bentoml.task turns an API into a queue-backed job: the caller submits work, gets a handle, and polls for the result later. For a platform owner this is the escape hatch for workloads whose natural runtime exceeds any sane HTTP timeout — batch scoring, document ingestion, long generations. The reason to care at this level is that tasks change your failure model. A synchronous endpoint under overload rejects or times out, and the client knows. A task endpoint under overload accepts work and grows a backlog, and nobody knows until someone asks why yesterday's job has not finished. If you offer tasks on your platform, you owe your tenants a backlog metric and an alert on it.

Which component breaks first

Platform ownership is largely the skill of predicting the next bottleneck before a tenant finds it. In BentoML, the order is fairly stable, and knowing it saves you from tuning the wrong thing.

The GPU breaks first, if there is one. Not the HTTP layer, not Python, not the network. A single GPU replica has a fixed amount of memory and a fixed amount of compute, and a large model occupies most of the memory before the first request arrives. The symptom is not an error; it is latency that climbs linearly with offered load while utilisation sits pinned near the top. The fix is never "more workers" — more workers on one GPU means more copies of the weights in the same memory, and you will run out before you get faster.

Memory breaks second, through the worker multiplier described above. The symptom is a replica that dies without a Python traceback, because the container runtime killed it rather than the process failing. If your logs end mid-request with no exception, suspect the limit, not the code.

Then the concurrency budget. traffic={"concurrency": N} tells the platform how many requests one replica can handle at once. If it is set too high, requests pile into a replica that cannot serve them and every one of them gets slow; if it is set too low, you scale out earlier than you need and pay for replicas that are idle. The honest way to pick it is measurement: find the offered concurrency at which per-request latency starts rising faster than throughput, and set it a little below that.

Then the batching window. max_latency_ms is a promise: the server will not hold a request longer than that waiting for batch-mates. Set it too low and you never form batches, so you paid for the complexity and got nothing. Set it too high and you have added latency to every request on a quiet morning for throughput you only need at peak.

Python's own overhead breaks last, and on a GPU service it almost never matters, because the model dominates. On a small CPU model it can dominate instead, and that is the case where workers genuinely helps: several independent processes using several cores on a model small enough to duplicate cheaply.

@bentoml.service(
    resources={"cpu": "4", "memory": "8GiB"},
    traffic={"timeout": 30, "concurrency": 16},
    workers=2,
)
class Scorer:
    @bentoml.api(batchable=True, max_batch_size=32, max_latency_ms=200)
    def score(self, texts: list[str]) -> list[float]:
        return self.model.predict(texts).tolist()

Read that block as four separate decisions with four separate justifications, and insist in review that each number has one. "It was in the example" is the most expensive sentence in model serving.

Batching as a budget, read from metrics

Adaptive batching is the single highest-leverage feature BentoML gives you on expensive hardware, and the one most often enabled without ever being verified. The mechanism is simple: with batchable=True on an API, the server collects concurrent requests and passes them to your method as one container — a list, an array — and splits the result back out. max_batch_size caps how many it will group, max_latency_ms caps how long it will wait, and batch_dim tells BentoML which axis to concatenate and split along when the natural answer is not the first one.

The constraint that trips people is on the signature. A batchable API takes one batched parameter, optionally alongside a bentoml.Context. If your endpoint needs three fields per request, you do not get three batched arguments; you wrap the three fields in a Pydantic model and batch a list of those. Teams that do not know this write a three-argument endpoint, see no error, and quietly run unbatched.

Which is why the metric matters more than the configuration. BentoML exposes bentoml_service_adaptive_batch_size as a histogram, and it is the only honest answer to "is batching working". If that histogram is concentrated at 1, you are not batching, whatever the decorator says. The usual causes are, in order: traffic that is genuinely serial, a max_latency_ms too small for your inter-arrival time, a signature that silently defeats grouping, or a load generator sending one request at a time.

histogram_quantile(0.5, sum by (le) (rate(bentoml_service_adaptive_batch_size_bucket[5m])))

Put that next to p95 of bentoml_service_request_duration_seconds on one dashboard panel and you can see the trade in real time: as the median batch size rises, throughput rises and latency rises with it. That picture is also the single most useful artifact for a conversation with a product owner about whether 200 ms of added latency is worth a third of the GPU bill.

The Bento as a supply-chain artifact

bentoml build produces a Bento: a versioned, self-contained directory, tagged name:version, holding your code, a dependency specification, build configuration, and whatever models the build pulled in. It is exportable with bentoml export, importable with bentoml import, pushable to a registry with bentoml push, and turnable into an OCI image with bentoml containerize. In other words it is a distributable package, and everything you know about distributable packages applies.

Three consequences shape platform policy.

A Bento is a shareable archive, so it must contain no secrets. The envs field in the build configuration is tempting and wrong for anything sensitive, because the value travels with the artifact. API keys, database passwords and model-registry tokens belong in the runtime environment, injected by your platform's secret store — Kubernetes Secrets, Vault, a cloud secret manager, or BentoCloud's secrets — and read from the process environment at startup. The build-time rule I enforce is blunt: envs may contain configuration that would be safe to publish, and nothing else. A cheap review grep for KEY, TOKEN, SECRET and PASSWORD in bentofile.yaml catches most of it.

bentoml import is code execution, eventually. A Bento carries Python source and, in many cases, serialized model files. Importing one from an untrusted source and serving it is equivalent to running a stranger's program with your credentials in the environment. On a platform this means Bentos must come from your own build pipeline, identified by tag, and the path from a developer's laptop into the production Bento store should not exist. Several BentoML patch releases in the 1.4 line were specifically about hardening this surface: 1.4.34 fixed file-path resolution on user input, 1.4.36 added symlink validation when extracting tarfiles, 1.4.38 moved Jinja template rendering to a sandboxed environment, and 1.4.39 fixed symlink handling in the Bento store. That is four path-and-archive fixes in six patch releases, which tells you where the sharp edges are and why "we are on 1.4, we are fine" is not a patch policy.

Dependency locking is a reproducibility feature with a staleness cost. python.lock_packages defaults to true and pins the resolved dependency set at build time, which is what you want: the image you test is the image you ship. The cost is that a Bento built three months ago pins three-month-old transitive dependencies with three months of published vulnerabilities. The resolution is not to turn locking off; it is to rebuild on a schedule so that the lock is fresh, and to scan the resulting image in CI with Trivy or an equivalent so you know what you are shipping.

service: "service:Scorer"
description: "file: ./README.md"
labels:
  team: risk-platform
  cost-centre: ml-serving
include:
  - "service.py"
  - "src/**/*.py"
exclude:
  - "tests/**"
  - "notebooks/**"
python:
  requirements_txt: "./requirements.txt"
  lock_packages: true
docker:
  distro: debian
  python_version: "3.11"
  system_packages:
    - libgomp1

Two details in that file are platform policy rather than taste. include is an allow-list and exclude is a deny-list, and the allow-list is the safer default: the failure mode of a too-narrow include is a loud ImportError at serve time, while the failure mode of a too-narrow exclude is a notebook with a pasted credential travelling inside a shared artifact. A .bentoignore file, gitignore-style, covers the rest. And labels is how chargeback becomes possible at all — a tag you can group by in your platform's cost tooling is worth more than any amount of spreadsheet archaeology later.

Security and the trust model

State the trust model plainly, because teams assume BentoML provides more than it does. A served Bento is an unauthenticated HTTP server. It validates the shape of inputs against your type hints, exposes your APIs, serves an OpenAPI description and a metrics endpoint, and makes no decision about who may call it. Everything in the authentication and authorisation layer is yours to add.

There are two honest places to add it. The first is in front: an ingress, API gateway, or service mesh that terminates TLS, checks a token, applies per-tenant rate limits, and forwards to the Bento on a private network. This is where it belongs for a platform, because the policy is then uniform across every tenant service and no individual team can get it wrong. The second is inside, using bentoml.Context to read request headers in the API method itself. This is the right tool for fine-grained, model-specific decisions — which tenant's feature table to read, which prompt template applies, whether this caller may request the expensive variant — and the wrong tool for the baseline auth check, because it has to be correct in every service independently.

@bentoml.service(path_prefix="/v1")
class Scorer:
    @bentoml.api
    def score(self, text: str, ctx: bentoml.Context) -> dict:
        tenant = ctx.request.headers.get("x-tenant-id")
        if not tenant:
            ctx.response.status_code = 400
            return {"error": "missing tenant"}
        return {"tenant": tenant, "score": self.model.predict([text])[0]}

The gateway in front has authenticated the caller and has set x-tenant-id from a verified token; the service is reading it for routing and attribution, not for a security decision. Keep that division crisp in review, because a header the service trusts is a header a caller can forge the moment the network boundary moves.

The rest of the hardening pass is the ordinary container discipline covered in the Docker guide, applied to the image bentoml containerize produces: run as a non-root user, mount the root filesystem read-only where the framework tolerates it, drop capabilities, set memory and CPU limits that match the resources declaration, and pin the base image by digest so that the distro and python_version you tested are the ones that ship. The 1.4.38 release notes corrected the NVIDIA CUDA base images used for GPU builds, which is a reminder that the base image is part of your dependency surface even when you did not choose it explicitly.

One more item belongs on the list because it is invisible: BentoML's CLI has usage telemetry, disabled per-command with --do-not-track or globally through its environment variable. For regulated environments in the Gulf and Egypt where outbound traffic from build agents is audited, turn it off in the CI image and document that you did, so the next person reviewing egress rules does not have to rediscover what the connection was.

Multi-tenancy, which BentoML does not have

There is no tenant concept in BentoML. Nothing in the framework isolates one team's data, prompts, or load from another's. Say this out loud when you design the platform, because the alternative is discovering it when one tenant's traffic spike becomes another tenant's outage.

You have three patterns, and the choice is a straight trade between isolation and GPU utilisation.

One deployment per tenant is the strongest isolation and the most expensive. Each tenant gets its own replicas, its own scaling policy, its own secrets, its own metrics, and its own blast radius. On GPUs it is often unaffordable, because a tenant with light traffic still holds a whole accelerator.

One shared deployment, tenant-aware at the edge is the cheapest and the one that needs the most care. All tenants share replicas; the gateway authenticates and stamps a tenant identifier; the service routes on it; per-tenant rate limits at the gateway stop one tenant monopolising the concurrency budget. The risks are real and specific: a shared batch now contains several tenants' inputs at once, so any per-request state in your service is a cross-tenant leak waiting to happen, and a shared queue means a noisy tenant's latency is everyone's latency.

A hybrid, which is what most mature platforms land on: shared capacity for the long tail of small tenants, dedicated deployments for the few whose volume or data-residency requirements justify them. path_prefix gives you clean versioned or per-tenant URL namespaces without touching the service code, and labels give you the attribution to decide who graduates to dedicated capacity.

For MENA teams there is a hard constraint that overrides the economics. If a customer's contract or a regulator requires that data stays in-country — and in the Gulf and Egypt that is common for financial, health and public-sector work — then that tenant cannot share a deployment that runs outside the permitted region, however good the utilisation story is. Put region in the tenant model from day one. Retrofitting it means re-deploying everything.

Scaling and autoscaling

Autoscaling a model server is not autoscaling a web server, and the difference is cold start. A stateless web replica is useful a second after it starts. A model replica must pull a container image, load weights that may be gigabytes, move them onto an accelerator, and only then accept traffic. If that takes two minutes, no reactive autoscaler can save you from a thirty-second spike — by the time capacity arrives, the spike is over and you are paying for it.

This is why the scaling configuration exposes stabilization windows rather than only thresholds.

services:
  Scorer:
    scaling:
      min_replicas: 2
      max_replicas: 8
      policy:
        scale_up_stabilization_window: 60
        scale_down_stabilization_window: 600

The asymmetry is deliberate and is the main thing to teach your tenants. Scale up quickly, because the cost of being short is user-visible errors. Scale down slowly, because the cost of being wrong is paying the entire cold start again on the next request, and a flapping GPU deployment can spend more on repeated loading than it ever saved on idle time. The windows accept values from zero to 3600 seconds, and the first number I set for a new GPU service is a scale-down window several times the measured cold-start time.

The complement to stabilization is queueing. On BentoCloud, external_queue: True holds excess requests at the platform level so a replica never receives more concurrent work than its declared concurrency; when it is on, the separate maximum-concurrency rejection behaviour is inactive. For a GPU service this is the difference between a saturated replica degrading gracefully and a saturated replica thrashing. It is a BentoCloud feature, so if you are self-hosting on Kubernetes, the equivalent is yours to build: a request-queue sidecar, a mesh-level concurrency limit, or an ingress with a bounded queue and a fast rejection path. Do not skip it and hope that concurrency alone will protect the replica; concurrency is a signal to the scheduler, not a gate on the socket.

Self-hosted scaling uses the metrics BentoML already publishes. bentoml_service_request_in_progress is the natural input, because it is the direct measure of how much work a replica is holding, and comparing it to the declared concurrency gives you a dimensionless utilisation number that an autoscaler can act on across services of very different shapes.

Observability that predicts incidents

BentoML enables Prometheus metrics by default and serves them on /metrics on the service port. Four series carry most of the diagnostic weight, and knowing what each one tells you is the difference between a five-minute triage and an hour of guessing.

bentoml_service_request_total is a counter, so rate and error ratio come from it. bentoml_service_request_duration_seconds is a histogram, so quantiles come from it — and the default buckets may not straddle your actual latencies, which is why the metrics argument lets you set them per service. A histogram whose buckets all sit below your p50 reports a useless quantile with total confidence. bentoml_service_request_in_progress is a gauge and the best single saturation signal. bentoml_service_adaptive_batch_size is the histogram discussed above and the only proof that batching is real.

@bentoml.service(
    metrics={
        "enabled": True,
        "namespace": "bentoml_service",
        "duration": {"buckets": [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30]},
    },
)
class Scorer: ...

Set those buckets from a measurement, not from a template. For a service whose p50 is 40 ms and whose timeout is 30 s, a bucket list that stops at 10 s cannot distinguish "slow" from "about to time out", and that is precisely the distinction you need during an incident.

On top of the four, add two things BentoML cannot know for you. The first is the model version as a label on your own custom metrics, so that a latency or quality regression can be attributed to a deployment rather than correlated with one by eye. The second is a request identifier propagated from the gateway through the service and into your logs, so a tenant's complaint about one slow call becomes a log query rather than an archaeology project. If you run OpenTelemetry collection already, the BentoML service is a span-producing component like any other and plugs into the same pipeline described in the OpenTelemetry guide; pair it with Prometheus and Grafana for the numeric side and with Evidently when the question is about the predictions rather than the server.

Four alerts earn their place. Error ratio above a small threshold for five minutes, because a model server's errors are usually a dependency or an input-shape change. Saturation — in-progress over declared concurrency — because it predicts the latency alert before users feel it. p95 latency over the service's own objective, per service, because a platform-wide latency threshold is meaningless across services with different shapes. And task backlog growth, if you offer tasks, because it is otherwise silent.

Cost, and who pays for it

On a serving platform, the GPU is the bill. Everything else is noise, and the levers that matter are the ones that raise accelerator utilisation or shorten the time an accelerator sits idle.

Batching is the first lever and usually the largest. Doubling the median batch size on a saturated GPU service is close to halving the per-request cost of that service, which is why the batch-size histogram belongs in a cost review and not only in a dashboard.

Separating CPU work from GPU work is the second. A service graph built with bentoml.depends() lets tokenisation, image decoding, feature lookup and post-processing live in a CPU service with a high worker count and cheap replicas, while the GPU service does nothing but inference. The win is not architectural elegance; it is that the GPU is no longer paying for base64 decoding. The cost is a network hop and a second thing to operate, so it pays off at the point where GPU utilisation is high and provably spent on non-model work.

Cold-start reduction is the third, and it is what makes aggressive scale-down affordable. Baking weights into the image makes the image enormous but the start fast; downloading in an @bentoml.on_deployment hook keeps the image small and does the work once per deployment rather than once per worker; a shared cache volume sits in between. Which one is right depends on whether image pull or weight download is slower in your environment, and that is measurable in an afternoon.

Allocation is the fourth and is organisational. Labels on Bentos and deployments, carried through to your platform's cost reporting, turn "the ML platform costs a lot" into "this team's recommendation service costs this much per thousand requests". That number changes behaviour in a way that no amount of advocacy does. It is also what lets you offer tenants a real choice: shared capacity at a lower rate with shared latency risk, or dedicated capacity at cost.

Upgrades and migrations

BentoML's history has one genuine discontinuity and a steady stream of ordinary patches, and you treat them completely differently.

The discontinuity is 1.1 to 1.2. The service API was replaced. The old world had a module-level bentoml.Service(...) object, APIs declared with IO descriptors that described the wire format explicitly, and Runners as the mechanism for getting work onto a model. The current world has a class decorated with @bentoml.service, methods decorated with @bentoml.api, plain Python type hints — Pydantic models, numpy arrays, pathlib.Path — in place of IO descriptors, and bentoml.depends() in place of runners. Code written for the old API does not run on the new one, and the symptom is an import or attribute error pointing at a name that no longer exists.

This matters to you as a platform owner for a reason beyond migration: most BentoML material on the internet, including much that ranks well, is written for the old API. A new team following a tutorial will write runner-based code, hit an error they cannot interpret, and conclude the platform is broken. The cheapest intervention available to you is a two-paragraph note at the top of your internal getting-started page saying exactly this, with a correct minimal service next to it.

The patches are ordinary, and the policy should be boring. Pin an exact version in the dependency specification and in CI. Read the release notes for every patch you take, because the 1.4 line has repeatedly shipped security fixes in path handling, archive extraction and template rendering, and those are the ones you do not want to be six months behind on. Rebuild Bentos after upgrading the framework rather than assuming an older artifact will run on a newer runtime; the rebuild is cheap and the alternative is an incident whose cause is a version skew nobody logged.

For a fleet, the mechanism that makes this sustainable is a canary path. One low-traffic, well-instrumented service upgrades first and runs for a week. The platform team owns that service, so there is no negotiation about who absorbs the risk. If its error ratio, p95 and batch-size histogram are unchanged, the version is promoted to the default in your shared base configuration and tenants pick it up on their next build. If something moved, you have one service to debug rather than fifty.

One deprecation to know about rather than plan around: Yatai, the older self-hosted Kubernetes platform for BentoML, is no longer part of the 1.4 documentation. If you inherit a deployment that depends on it, treat that as a migration to plan, not a component to extend. Self-hosted serving today means the container image from bentoml containerize running as an ordinary workload on your own cluster, with your own ingress, autoscaler and secret management around it.

Governance for a shared platform

Governance on a model-serving platform is mostly about three questions having written answers: what may be deployed, by whom, and how you find out later what was running.

What may be deployed. The gate is CI, and the Bento is the artifact it gates. A build that passes should mean: the service imports and serves; a smoke request returns the expected shape; dependencies are locked; the image scan has no fixable high-severity findings; the build config contains no credential-shaped environment entries; resources, traffic.timeout and a concurrency value are all set explicitly; and the service declares an owner label. None of that is BentoML-specific machinery — it is a handful of checks in your pipeline — and all of it is the difference between a platform and a shared staging environment.

set -euo pipefail
bentoml build --do-not-track
BENTO=$(bentoml list -o json | python -c 'import json,sys; print(json.load(sys.stdin)[0]["tag"])')
bentoml containerize "$BENTO" --do-not-track
bentoml export "$BENTO" "./artifacts/${BENTO/:/-}.bento"

By whom. Production deployment should be a pipeline action triggered by a merge, not a human running a command with their own credentials. The practical shape is that developers can build and deploy to a development target freely, and the production target accepts only artifacts promoted by the pipeline, by tag. The reason is less about trust than about reconstruction: when a deployment is a pipeline run, the question "what changed at 14:30" has an answer.

How you find out later. Keep the mapping from deployed tag to Git commit, and keep it outside the running system. bentoml get <tag> tells you what is in a Bento, and bentoml list tells you what exists in a store, but neither survives the store. A small record per promotion — tag, commit, model version, who approved, when — is the thing you will want during an audit or a post-incident review, and the thing nobody adds until after the first time they needed it.

Finally, publish service-level objectives per service rather than per platform. A fraud-scoring endpoint with a 50 ms budget and a document-summarisation endpoint with a 30 s budget cannot share an alert threshold, and pretending they can produces alerts that everyone learns to ignore.

Incident playbooks

These are the five incidents I have actually seen on BentoML platforms, each with the first thing to look at. The value of writing them down is that the first three minutes of an incident are when people guess, and a playbook replaces guessing with looking.

Latency tripled, no deploy. Check saturation first: bentoml_service_request_in_progress against declared concurrency. If saturation is high and the request rate is also high, this is capacity and the answer is replicas. If saturation is high and the rate is flat, something downstream got slower — a feature store, a vector database, an external API — and you are looking at the wrong service. If saturation is low and latency is high, suspect the batching window or a single pathological input.

A replica died with no traceback. Memory. Resident use per worker times workers against the container limit. The absence of a Python exception is the signal: the process was killed, it did not fail. Reduce workers, raise the limit, or find what grew — a cache on self that nothing evicts is the usual culprit.

Errors spiked right after a deploy. Compare the input validation. BentoML validates from type hints, so a changed signature produces clean 400s rather than confusing 500s, which is a feature: the error class tells you whether the problem is the caller or the model. Many 400s means a contract change; many 500s means your code. Roll back by tag, then diagnose.

Batching stopped helping. Look at the batch-size histogram, not the configuration. A collapse to 1 after a deploy almost always means the API signature changed in a way that defeats grouping — an added second parameter, a switch from a list to a scalar.

Cold starts are hurting during scale-up. Measure the three phases separately. Image pull, weight load, first request. Each has a different fix, and the common mistake is optimising the one that is not dominant. If pull dominates, shrink the image or pre-pull on the nodes. If load dominates, move the download into an @bentoml.on_deployment hook or a cache. If the first request dominates, you need a warm-up call before the replica is marked ready.

Where BentoML stops

Advocating for a tool well requires knowing where it is the wrong answer, and BentoML has clear edges.

It stops where Python stops. BentoML's core idea is that your service is a Python class, which is exactly why it is pleasant and exactly why it cannot help you serve a model from a non-Python runtime. If your inference path must be C++ or Rust end to end for latency reasons, BentoML is packaging overhead around something you are not using.

It stops where you want serving to be declarative Kubernetes. KServe and Seldon Core express a deployment as a custom resource reconciled by a controller, which fits an organisation whose whole platform is GitOps and whose model servers should look like every other workload. BentoML's unit is a Python project and an image, and the deployment story is either BentoCloud or whatever you build. If "a model deployment is a YAML file in a repo and a controller makes it true" is your organising principle, the CRD-based servers meet you where you are.

It stops at extreme single-model throughput. Triton exists because squeezing the last factor from NVIDIA hardware — concurrent model instances, tuned execution, framework backends in one process — is specialised work. For high-volume LLM serving specifically, vLLM and the inference servers built around continuous batching and paged attention will beat a general-purpose server on tokens per GPU-hour. The mature pattern is not either-or: BentoML as the packaging, API and operations layer with a specialised engine doing the inference inside it.

It stops where your workload is not a request. Scheduled scoring of a hundred million rows is a job, not an API. Ray for distributed compute, Spark for batch over large tables, and a scheduler like Prefect or Dagster around them. @bentoml.task covers long-running single requests; it is not a batch engine.

And it does not replace a model registry. The Bento records which models it contains, but the question "which version of the fraud model is approved for production, who approved it, and what were its evaluation numbers" belongs in MLflow or an equivalent. The clean division is registry owns lineage and approval, BentoML owns packaging and serving, and the build reads the approved version from the registry. For agent and LLM-shaped services, pair the serving layer with Langfuse for trace-level visibility that server metrics cannot give you.

Running BentoML as a platform for other teams

Everything above is technical. What makes the platform work is a small number of decisions you make once, centrally, so that fifty teams do not make them fifty times badly.

Publish a template repository. A correct 1.2-plus-style service, a bentofile with locking and labels, a CI pipeline with the gates listed earlier, a Dockerfile policy, a dashboard definition, and a README that says what each number means. Most teams will change the model and nothing else, which is the point: the defaults you ship are the defaults your platform runs.

Own the shared base image. Pinned distro, pinned Python version, pinned CUDA for GPU services, system packages everyone needs, non-root user, scanned on a schedule. When a CVE lands you rebuild one image and tell teams to rebuild; without it you negotiate with every team individually.

Define a tier model and attach real numbers to it. Two or three tiers — shared CPU, shared GPU, dedicated — each with a stated latency objective, a stated concurrency allocation, a stated scaling policy and a stated cost. A tenant choosing a tier is a tenant who has agreed to a trade-off, which is a far better conversation than a tenant discovering one.

Keep the paved road paved. The platform's value is that the easy path is the correct path. Every time a team has to read the BentoML documentation to do something routine, that is a gap in your template. Every time a service reaches production without explicit resources and traffic, that is a gap in your CI gate.

And keep a deprecation calendar. Framework minor versions, base images, Python versions, and the services still on patterns you have moved away from. Announce, support both for a window, then remove. The alternative is a platform that is the union of every decision anyone ever made, which is the state most ML platforms are in.

The review checklist

This is what I actually read when a BentoML service comes up for review.

A service that passes all eighteen is not guaranteed to behave, but it will fail in ways you can diagnose, which is the realistic goal.

The complete picture

Put the three levels together and the shape of BentoML is clear. At Beginner level it is a decorator that turns a Python class into an HTTP API, and a build command that turns a directory into a versioned artifact. At Mid level it is a set of knobs — workers, concurrency, timeouts, batching, service graphs — that let you trade latency for throughput deliberately, plus the metrics that tell you whether the trade worked. At Senior level it is a packaging and serving layer inside a platform that you own: an artifact with supply-chain properties, a process topology with predictable failure modes, a cost model dominated by accelerator utilisation, and a set of policies that make fifty teams' services reviewable by one team.

The thread running through all of it is that BentoML is deliberately a small amount of framework around a large amount of your Python. That is its strength — there is no DSL to learn and no controller to fight — and it is the source of every responsibility listed on this page, because the framework is not going to add authentication, tenancy, queueing or policy on your behalf. Owning BentoML well means being clear-eyed about which half of the problem the tool solves.

Where the series leaves you

You should now be able to look at a BentoML service and a deployment configuration and see not just what they do but what they cost, what they permit, and how they will fail.

Carry the review habit out of here. Most model-serving incidents are not clever: they are a worker count nobody multiplied, a timeout nobody set, a secret nobody moved, or a batch size nobody graphed — and every one of them is cheaper to catch in a pull request than at 02:00.

Sources

• BentoML documentation
• Install BentoML
• Services
• Adaptive batching
• Parallelize requests
• Bento build options
• CLI reference
• Metrics
• Create deployments
• Concurrency and autoscaling
• BentoML releases
• bentoml on PyPI

Senior BentoML interviews rarely test syntax. They test whether you can be handed a model-serving platform and not lose money, data or a weekend. The questions below are the ones that come up, grouped by what they are really probing, with what a strong answer covers and what a weak one leaves out.

Part one: system-design prompts

"Design a model-serving platform for twelve teams on a shared GPU pool."

Start by separating the three layers, because candidates who do not will spend the interview arguing about the wrong one. There is the edge — TLS, authentication, per-tenant rate limits, request identifiers — which BentoML does not provide and which you own. There is the serving layer, which is Bentos built from a template and run as containers with declared resources and traffic. And there is the platform layer — CI gates, a shared base image, a tier model, cost attribution, a deprecation calendar.

Then make the tenancy call explicitly. One deployment per team is the strongest isolation and the worst GPU utilisation. One shared deployment with tenant identity stamped at the edge is the cheapest and needs per-tenant rate limits or the largest tenant sets everyone's p99. Most platforms end up hybrid: shared capacity for the long tail, dedicated deployments for the few teams whose volume or data-residency obligations justify it. Say that residency belongs in the tenant model from the start — for Gulf and Egyptian customers in finance, health and public sector it is a contractual constraint, and retrofitting region means redeploying everything.

Close on the two platform-wide numbers you would put on one dashboard: error ratio per service, and bentoml_service_request_in_progress divided by each service's declared concurrency. The second is dimensionless, so it means the same thing on a 40 ms CPU classifier and a 20 s GPU summariser.

A weak answer designs the Kubernetes topology and never mentions that BentoML has no authentication and no tenant concept.

"Two models, one of them GPU-bound. How do you lay it out?"

Split them into two services joined by bentoml.depends(). The CPU service does tokenisation, decoding, feature lookup and post-processing with a high worker count and cheap replicas; the GPU service does inference and nothing else. The justification is not elegance, it is that the GPU stops paying for base64 decoding, and the two now scale on their own signals.

Name the costs honestly: a network hop, a second thing to operate and alert on, and a failure mode where the inner service's saturation arrives in the outer service as an ordinary Python exception with no obvious cause. That last one is why every depends() target needs its own alerting rather than inheriting the gateway's.

Also say when not to split: if GPU utilisation is already high and provably spent on the model, splitting adds latency and operational surface for nothing. The trigger is evidence that the accelerator is doing non-model work.

"You need p99 under 200 ms and the model takes 120 ms. Walk me through it."

Work the budget backwards. The model is 120 ms, so you have 80 ms for everything else: network, queueing, serialisation, post-processing. That immediately constrains batching — max_latency_ms of 1000 is not available to you, and a value above about 40 ms is already a large fraction of your remaining headroom. State that batching trades latency for throughput and that at this budget you are mostly buying throughput from replicas, not from batches.

Then the knobs. traffic.timeout set below the caller's timeout so you fail fast rather than letting a caller give up on a request you are still computing. traffic.concurrency derived from a load test at the knee where latency starts rising faster than throughput. min_replicas at least one, and realistically two, because a latency-sensitive path cannot absorb a cold start. Duration histogram buckets that straddle 200 ms, or your p99 is a number with no resolution where you need it.

Finally, say what you would refuse. If the model is 120 ms and the budget is 200 ms, there is no configuration that survives a 2× traffic spike with scale-to-zero, and you would rather negotiate the objective or pre-provision than promise something the cold start makes impossible.

"Should we use BentoML, KServe, Triton or vLLM?"

The answer a strong candidate gives is a decision rule, not a preference.

BentoML when the service is Python, the team owns pre- and post-processing logic that belongs next to the model, and you want packaging, versioning and an API layer without a DSL. KServe or Seldon Core when your organisation's whole platform is declarative Kubernetes and GitOps, and a model deployment should be a custom resource reconciled by a controller like every other workload. Triton when you are squeezing the last factor out of NVIDIA hardware with concurrent model instances and tuned framework backends. vLLM when the workload is LLM token generation and the economics are tokens per GPU-hour.

Then the sentence that shows seniority: these are not exclusive. The mature pattern is BentoML as the packaging, API, metrics and operations layer, with a specialised engine doing the inference inside the service. You get the Python ergonomics and the throughput.

"Design the CI/CD path from a merge to production."

The artifact is the Bento and the gate is CI. A passing build means the service imports and serves, a smoke request returns the expected shape, dependencies are locked with lock_packages, the container image scan has no fixable high-severity findings, the build configuration contains no credential-shaped envs entries, and resources, traffic.timeout and a concurrency value are all set explicitly with an owner label present.

Promotion is by tag, by pipeline, never by a human with their own credentials. Development targets are open; the production target accepts only promoted tags. The reason is reconstruction rather than trust: when a deployment is a pipeline run, "what changed at 14:30" has an answer.

Keep the mapping from deployed tag to Git commit and model version outside the running system. bentoml get and bentoml list tell you about artifacts that exist now; neither survives the store being rebuilt, and the audit happens later.

Part two: incident response

"p99 tripled, nothing was deployed. First three things."

Saturation, rate, downstream — in that order, and say why.

First bentoml_service_request_in_progress against the declared concurrency. If saturation is high and the request rate is also high, it is capacity and the answer is replicas. If saturation is high and the rate is flat, the service is holding requests longer for the same load, which means something downstream got slower — a feature store, a vector database, an external API — and you are looking at the wrong service. If saturation is low and latency is high, suspect the batching window or one pathological input shape.

What separates a senior answer is refusing to touch configuration before reading the signal. Restarting replicas or raising workers at this point destroys the evidence and sometimes fixes it by accident, which is worse than not fixing it.

"A replica vanished with no traceback."

That is the signature of a kill, not a crash: the process did not fail, something stopped it. Almost always memory, and almost always the worker multiplier — each worker runs __init__ and loads its own copy of the model, so a 6 GB model with workers=4 needs roughly 24 GB and nobody wrote 24 anywhere.

The diagnostic is resident memory per worker times the worker count against the container limit. The fixes, in order of preference: reduce workers, raise the limit to match reality, or find what grew — an unbounded cache on self is the usual culprit, and it grows across requests in a way no single-request test catches.

"Errors spiked immediately after a deploy."

Read the error class before the logs. BentoML validates inputs against the API's type hints, so a changed signature produces clean 400s rather than confusing 500s. Many 400s means the contract moved and the callers are now wrong — or you are. Many 500s means your code. That split tells you in seconds whether this is a rollback or a code fix.

Roll back by tag first, diagnose second. The senior instinct is that restoring service and finding the cause are separate activities with different urgencies, and a rollback you have drilled is cheap.

"Batching stopped helping after a refactor."

Look at bentoml_service_adaptive_batch_size, not the decorator. A histogram that collapsed to 1 means grouping is no longer happening. The most common cause is the signature constraint: a batchable API takes one batched parameter, optionally alongside a bentoml.Context. Somebody added a second argument, saw no error, and silently turned batching off. The fix is to wrap the fields in a Pydantic model and batch a list of those.

The broader point to make: the configuration is a request and the metric is the answer. Any claim about batching that is not on a graph is a guess.

"Scale-up is making things worse, not better."

Cold start. New replicas pull an image, load weights that may be gigabytes, move them onto an accelerator, and are useless until all of that finishes — so a reactive scale-up during a thirty-second spike arrives after the spike and costs money. Worse, an aggressive scale-down window makes the deployment flap, paying the cold start repeatedly.

Measure the three phases separately — image pull, weight load, first successful request — because each has a different fix and the common mistake is optimising the one that is not dominant. Then set scale_down_stabilization_window to several times the measured cold start, keep scale_up_stabilization_window short, and raise min_replicas for latency-sensitive paths. On BentoCloud, add external_queue so a saturated replica queues rather than thrashes; self-hosted, that queue is yours to build.

"A tenant reports behaviour you cannot reproduce."

Ask for bentoml env before anything else. It prints the system and package information for their environment, and about half the time the difference is a BentoML or Python version. The second question is which Bento tag they are running, because "latest" means different things in two stores.

The third question is whether their code uses runners or IO descriptors. Most BentoML material online, including pages that rank well, was written for the 1.1 API, which was replaced in 1.2. A team following a tutorial writes runner-based code, hits an attribute or import error on a name that no longer exists, and concludes the platform is broken.

Part three: arguing trade-offs to leadership

"Justify a 200 ms latency increase to cut the GPU bill by a third."

Do not argue it in the abstract; bring the two curves. As median batch size rises, throughput rises and latency rises with it, and both are on one dashboard panel next to the per-thousand-request cost. The decision then belongs to whoever owns the product objective, which is where it should be.

Frame it as a tier choice rather than a global setting. Latency-sensitive endpoints keep a tight max_latency_ms and pay more per request; tolerant endpoints — batch scoring, summarisation, anything behind a job queue — take the larger batch and the lower cost. A platform that offers one latency policy for everything is leaving money on the table at one end and breaking an objective at the other.

"Why can we not just scale to zero?"

Because the replica is not useful when it starts, it is useful when the weights are loaded, and the gap is the cold start you measured. Give them the number. If a first request after idle takes ninety seconds, scale-to-zero on a user-facing path is not a cost optimisation, it is an availability decision disguised as one.

Then offer the real options: keep a warm floor on user-facing paths and let batch or internal paths scale down aggressively; shorten the cold start by baking weights into the image or downloading once in an @bentoml.on_deployment hook rather than per worker; or accept a documented first-request penalty on genuinely tolerant endpoints. Presenting three options with costs attached is what makes the recommendation credible.

"The platform team wants to gate deployments. Why should we accept the friction?"

Translate each gate into the incident it prevents, with the cheapest honest framing. A resources and traffic.timeout check prevents the replica killed by a limit nobody computed. A secret scan on build envs prevents a credential travelling inside a shareable archive to whoever has the Bento. A locked dependency set prevents the image that tested clean and shipped different. An owner label prevents the thirty minutes during an incident spent finding out whose service this is.

Then concede the real cost rather than pretending there is none: gates add minutes to a build and occasionally block a legitimate change at an inconvenient moment. The mitigation is that the paved road is paved — a template repository where all of this is already configured, so compliance is the default and not a task.

"Should we self-host or use the managed platform?"

Lay out what the managed platform does that you would otherwise build: autoscaling with stabilization windows, the external request queue that protects a saturated GPU replica, secret management, and deployment orchestration. Then price the alternative honestly — on your own cluster those become an autoscaler configuration, a queueing layer you build, your existing secret store, and your existing delivery pipeline.

The decision usually turns on two things rather than cost: whether you already run Kubernetes competently, and whether data residency permits the managed regions. For a team that already operates clusters well, the container image from bentoml containerize is an ordinary workload and self-hosting is cheap. For a small team without that muscle, building a queue and an autoscaler to serve three models is how a quarter disappears. Say which situation you are in rather than which option is better in general.

"Why is this upgrade a project and not a version bump?"

Distinguish the two kinds of change. Patches inside the 1.4 line are routine and should be taken promptly, because that line has repeatedly shipped security fixes in path handling, archive extraction and template rendering. The policy is boring: pin an exact version, read the notes, rebuild Bentos after upgrading rather than assuming an old artifact runs on a new runtime.

The 1.1 to 1.2 change was different in kind. The service API was replaced — module-level service objects, IO descriptors and runners gave way to @bentoml.service classes, @bentoml.api methods with plain type hints, and bentoml.depends(). Code does not carry over, so that is a rewrite scoped per service. For a fleet, describe the canary path: one low-traffic service owned by the platform team upgrades first and runs for a week with error ratio, p95 and batch size watched; if nothing moved, the version becomes the template default and teams pick it up on their next build.

Final self-test

These are the things that only show up once BentoML is carrying other teams' models: the traps that are invisible at one service and expensive at fifty, the habits that make a shared platform survivable, and the checklists I actually run against pull requests and design documents.

Production traps that only appear at scale

The worker multiplier. workers=N runs __init__ N times, so N copies of the model live in memory. On a laptop with one worker nothing is wrong; on a platform where someone raised workers to use spare CPU, the replica is killed by the container runtime at the next cold start. The signature is a log that ends mid-request with no Python traceback — the process was stopped, it did not fail. Fix the memory declaration to match workers × resident model size plus headroom, and treat any pull request that raises workers without touching resources.memory as incomplete.

Batching that was never on. A batchable API must take one batched parameter, optionally alongside a bentoml.Context. Add a second argument and BentoML raises nothing visible at review time; grouping simply stops. The only honest check is the metric: if bentoml_service_adaptive_batch_size sits at 1, batching is off whatever the decorator says. Wrap multiple fields in a Pydantic model and batch a list of those.

Duration buckets that cannot see the problem. The duration histogram's buckets decide what quantiles you can compute. If every bucket sits below your p50, the p95 you alert on is a number with total confidence and no resolution. Set metrics={"duration": {"buckets": [...]}} from a measurement, and make sure the list straddles both your objective and your timeout, or you cannot distinguish "slow" from "about to give up".

Concurrency mistaken for capacity. traffic.concurrency tells the platform how many requests one replica can hold at once. It does not make the replica faster. Set too high, every request on a saturated replica gets slow; set too low, you scale out before you need to and pay for idle. Derive it from a load test at the knee where latency starts rising faster than throughput, and write the test result in the service README so the next person does not re-guess.

Cold start eating the autoscaler. A model replica is useless until the image is pulled and the weights are loaded. Reactive scale-up during a short spike arrives after the spike. An aggressive scale-down window makes the deployment flap and pays the cold start repeatedly, which on GPUs can cost more than the idle time it saved. Measure image pull, weight load and first successful request separately, then set the scale-down window to several times the total.

The inner service's saturation surfacing as someone else's 500. A bentoml.depends() target that is full makes the caller's method raise an ordinary Python exception. On a dashboard that only watches the gateway, that looks like the gateway broke. Every dependency target needs its own saturation and error alerting, not inherited coverage.

Secrets riding inside the artifact. The envs field in build configuration travels with the Bento, and a Bento is a shareable archive you can export and email. Anything credential-shaped belongs in the runtime environment, injected by your platform's secret store. One grep in CI for KEY, TOKEN, SECRET and PASSWORD in the build config catches most of it in under a second.

A build context that includes more than the service. exclude is a deny-list and will miss things; include is an allow-list and fails loudly when it is too narrow. Prefer the allow-list. A too-narrow include produces an ImportError at serve time, which you notice; a too-narrow exclude ships a notebook with a pasted credential, which you do not.

A locked dependency set going stale. lock_packages defaults to true and that is correct — the image you tested is the image you ship. The cost is that a three-month-old Bento pins three-month-old transitive dependencies with three months of published advisories. The answer is a scheduled rebuild plus an image scan, not turning locking off.

Patch drift on a line that ships security fixes. The 1.4 series has repeatedly fixed path resolution on user input, symlink handling in archive extraction and the Bento store, and template rendering sandboxing. Four of those landed across six patch releases. "We are on 1.4" is a version, not a patch policy.

Tasks with an invisible backlog. A synchronous endpoint under overload rejects or times out and the caller knows. A @bentoml.task endpoint accepts work and grows a queue, and nobody finds out until someone asks why yesterday's job has not finished. If you offer tasks, you owe tenants a backlog metric and an alert on it.

Tutorials from the previous API. Most BentoML material online predates 1.2 and uses module-level service objects, IO descriptors and runners. A new team follows one, gets an attribute or import error on a name that no longer exists, and concludes your platform is broken. A two-paragraph note at the top of your internal getting-started page, next to a correct minimal service, is the cheapest support ticket you will ever prevent.

Platform-ownership habits

Own the template, not the advice. Documentation telling teams what to do is a worse investment than a template repository where it is already done: a correct service, a build config with locking and labels, a CI pipeline with your gates, a dashboard definition, and a README explaining what each number means. Most teams change the model and nothing else, so your defaults become the platform's behaviour.

Own the base image. One pinned distro, one pinned Python version, pinned CUDA for accelerated services, the system packages everyone needs, a non-root user, scanned on a schedule. When an advisory lands you rebuild once and tell teams to rebuild. Without it you negotiate with every team separately, and the slowest team sets your exposure.

Make every number in a service declaration defensible. resources, workers, traffic.timeout, traffic.concurrency, max_batch_size, max_latency_ms. Six numbers, six justifications. "It was in the example" is the most expensive sentence in model serving, and it scales linearly with the number of services you run.

Publish a tier model with real figures. Two or three tiers — shared CPU, shared GPU, dedicated — each with a stated latency objective, concurrency allocation, scaling policy and cost per thousand requests. A tenant who picked a tier has agreed to a trade-off. A tenant with no tier discovers one during an incident.

Attribute cost before you try to reduce it. Labels on Bentos and deployments carried into your cost reporting turn "the ML platform is expensive" into "this service costs this much per thousand requests". That sentence changes behaviour in a way no amount of advocacy does, and it is what makes the shared-versus-dedicated conversation honest.

Put region in the tenant model on day one. For Gulf and Egyptian customers in financial services, health and the public sector, data residency is frequently contractual. A residency-constrained tenant cannot share a deployment running outside the permitted region, however good the utilisation story is. Retrofitting region means redeploying everything.

Keep a deprecation calendar and use it. Framework minor versions, base images, Python versions, and services still on patterns you have moved away from. Announce, support both for a window, remove. Without it the platform becomes the union of every decision anyone ever made, which is the state most ML platforms are in. Yatai, the older self-hosted Kubernetes platform, is no longer in the 1.4 documentation; if you inherited it, it belongs on that calendar as a migration rather than a component to extend.

Drill the rollback. A previous tag that exists is not a rollback; a previous tag you have restored at least once under time pressure is. Put it in the quarterly routine with the on-call rotation, and measure how long it took.

Be clear about what BentoML does not do. No authentication, no tenancy, no rate limiting, no request queue outside the managed platform. Every one of those is yours, and the failure mode of being vague about it is a team assuming the framework covered something it never claimed to.

Review checklists

For a pull request touching a service

For a design document proposing a new service

Operational recipes

Sizing a new service in one afternoon. Serve it with workers=1 and record resident memory. Load-test one replica at rising concurrency and plot latency against throughput; the knee is your traffic.concurrency. Time the three cold-start phases. Those four measurements fill in every number in the service declaration, and taking the afternoon is cheaper than any of the incidents in the first section.

Proving batching pays. Graph the median of the batch-size histogram next to p95 duration, under constant load. Raise max_batch_size, re-measure, and record requests per accelerator-hour before and after. The pair of numbers is the artifact for the cost conversation; the dashboard alone is not.

Auditing an estate you inherited. List what exists in the Bento store, then for each one check the service declaration for the six numbers, the build config for secrets and locking, and the labels for an owner. Services with no owner label are the real finding: you cannot ask anyone about them, and they are the ones still running pre-1.2 patterns.

Triaging an unreproducible report. Ask for bentoml env output and the Bento tag before anything else. Version and environment skew explains a large share of "works for me" reports, and latest means different things in two stores.

Deciding whether to split a service. Measure what fraction of accelerator time is spent outside the model call. If it is small, splitting adds a hop and an operational surface for nothing. If it is large, move the pre- and post-processing into a CPU service behind bentoml.depends() and let each side scale on its own signal.

Upgrading the fleet. Pin an exact version. Upgrade one low-traffic service the platform team owns, and watch error ratio, p95 and batch size for a week. If nothing moved, make it the template default and let teams pick it up on their next build. Rebuild Bentos rather than assuming an old artifact runs on a new runtime.

Turning off telemetry in regulated environments. BentoML's CLI has usage telemetry, disabled per command with --do-not-track and globally through its environment variable. Set it in the CI image and write down that you did, so the next person auditing egress does not have to work out what the connection was.

Cost levers, in order of leverage

Raise the batch size. On a saturated accelerator, doubling the median batch size is close to halving the per-request cost of that service. Nothing else on this list comes near it, which is why the batch-size histogram belongs in the cost review and not only on the dashboard. The limit is the latency budget, so the honest framing is always "how much latency may I spend", not "how big may the batch be".

Stop paying GPU rates for CPU work. Tokenisation, image decoding, feature lookup, JSON assembly and post-processing do not need an accelerator. Moving them into a CPU service behind bentoml.depends() means the GPU does inference and nothing else. Measure before you split: if the accelerator is already spending almost all its time inside the model call, the hop costs latency and buys nothing.

Shorten the cold start so scale-down becomes affordable. The three options trade against each other. Baking weights into the image makes it large and the start fast. Downloading in an @bentoml.on_deployment hook keeps the image small and does the work once per deployment instead of once per worker. A shared cache volume sits between them. Which one wins depends on whether image pull or weight download is slower in your environment, and that is an afternoon's measurement rather than a debate.

Right-size before you scale out. A replica with workers set higher than its cores, or memory set to twice what it uses, wastes capacity on every replica you then add. Fix the unit before multiplying it.

Attribute, then negotiate. Per-service cost per thousand requests, grouped by the labels you put on every Bento. Teams optimise a number they are shown and ignore a pool they are not. This lever is organisational rather than technical, and on most platforms it moves more money than any configuration change.

The short list

If you remember five things from this level, make them these.

Memory multiplies with workers, and that is the most common way a BentoML replica dies. Batching is a claim until the batch-size histogram proves it. The Bento is a shareable archive, so secrets never go in it and imports never come from outside your pipeline. BentoML provides no authentication and no tenancy, so both are yours and both need to be uniform rather than per-service. And cold start is the fact that makes model serving different from web serving: it sets your scale-down window, your replica floor, and whether scale-to-zero was ever an option.

Everything else on this page is a consequence of one of those five.