The Complete Seldon Core Guide

This is part one of three. It covers everything you need to put a real model behind a real HTTP endpoint with Seldon Core 2, on a cluster you control. By the end you can install Seldon Core 2 on a Kubernetes cluster, deploy a trained model as a Model resource, call it over the Open Inference Protocol, understand why the scheduler sometimes refuses to place your model, run your own pinned inference server, and chain two models into a Pipeline. Mid-level and Senior take the same material further into scaling, Kafka tuning, security and platform ownership; nothing here is throwaway.

Each section ends with a Try it task. Do them in order on a throwaway cluster. Seldon Core 2 is a distributed system, and the only way the vocabulary sticks is to watch your own model go Ready, then break it on purpose and read the condition it reports.

Before anything else, one warning that will save you hours, because it is the single most common way beginners lose an afternoon.

What Seldon Core is, and the problem it solves

Training a model is a self-contained piece of work. You have data, you have a notebook, and at the end you have an artifact on disk: a pickled scikit-learn pipeline, an XGBoost booster, a saved TensorFlow graph. That artifact is worth nothing to the business until something can be asked, thousands of times a second, "here is a row of features, what do you predict?" and get an answer back in milliseconds, reliably, with a record of what was asked and what was answered.

Serving is where most machine-learning projects actually stall, and the reasons are depressingly repetitive. Somebody writes a small Flask application that loads the pickle at startup and exposes /predict. It works. Then there are nine models, so there are nine Flask applications, each with a slightly different request format, each pinned to a different scikit-learn version, each running one replica on a node with eight idle gigabytes of RAM. Then somebody asks what the model predicted for customer 48213 last Tuesday, and nobody knows, because the only record is a log line that was rotated away. Then a data scientist wants to A/B test a new version against the old one, and the only mechanism available is an if statement in the application code.

Seldon Core 2 is a Kubernetes-native framework that replaces that pile of bespoke services with a small set of declarative resources. The official documentation describes it as a framework for deploying and managing machine-learning and LLM systems at scale, and organises its own value around four claims worth unpacking in plain terms:

The second and fourth of those are the ones that change your daily life. Standardisation means your client code does not care which framework trained the model. Multi-model serving means a server replica is a host for models rather than a wrapper around one model: a single MLServer pod can hold a dozen small models, loading and unloading them on demand. If you have forty small models, that is the difference between forty pods and three.

The third claim deserves its own sentence, because it is the philosophical centre of the project. Seldon calls its approach data-centric MLOps: the data flowing through your models is treated as a first-class concern rather than an afterthought. That is why Apache Kafka sits in the middle of the architecture rather than at the edge of it. When you build a pipeline, the inputs and outputs of every step are written to Kafka topics, which is what makes an audit trail and a replay possible at all.

The version landscape, and what to install

Pinning down which version you are on is not bureaucracy here; it decides which features exist and which bugs you will hit.

The current v2 release line is 2.10.x, with v2.10.2 published on 19 December 2025. The 2.10 series is where a number of things changed in ways a beginner should know about up front, because they contradict older documentation.

Three practical consequences follow.

Install 2.10.2, not an earlier 2.10. The two patch releases after 2.10.0 fix problems you are likely to meet: pipelines that will not create or delete, and models that quietly come back on fewer replicas than you asked for after a node reboot. Starting on the patched version is free; debugging a known bug is not.

Model-native autoscaling was deliberately disabled in 2.10.0 pending wider fixes. If you read an older page describing automatic replica scaling driven by the model itself, that path is not available on a current install. The official advice is to scale inference servers with the Kubernetes Horizontal Pod Autoscaler or with KEDA instead. The Mid-level guide covers how; at beginner level, set replica counts by hand and know that the automatic route is off.

The documentation moved. Older URLs under /user-guide/kubernetes-resources/... now return 404. The current pages are flat — /seldon-core-2/user-guide/models, /servers, /pipelines, /quickstart — and historical versions live under versioned paths such as /seldon-core-2/v2.10/.... If a search result 404s, strip it back to the docs home and navigate down.

The four nouns

Almost everything in Seldon Core 2 is one of four Kubernetes custom resources. Learn these four words properly and the rest of the system becomes readable.

A Server is a running inference server: a StatefulSet of replicas able to hold models. Two server "farms" ship with the default installation, each with one replica: MLServer, Seldon's own Python-based server, and NVIDIA Triton. You do not usually create a Server to deploy your first model; you use one that is already there.

A Model is the resource you will write most often. It is a Kubernetes object that points at a trained artifact and says what the artifact needs in order to run. Critically, a Model is not a pod. It is a request to the scheduler to load an artifact onto some Server replica that can host it. That indirection is multi-model serving, and it is the thing newcomers most often get wrong when they go looking for "my model's pod" and cannot find one.

A Model is also more general than the word suggests. The documentation is explicit that the same resource represents a predictive model, a drift detector, an outlier detector, an explainer, a feature transformation or a routing model. Anything that takes tensors in and gives tensors out is a Model.

A Pipeline is a directed graph of Models, wired together through Kafka topics. You use one when a single prediction needs more than one model: a feature transformer followed by a classifier, or a classifier with a drift detector watching the same input.

An Experiment splits traffic between candidate models or pipelines by weight, which is how you run an A/B test without putting an if statement in your client. It is listed here for completeness; it belongs to the Mid-level guide.

Four supporting components complete the picture, and their names appear in logs often enough to be worth memorising. The operator, also called the controller manager, watches your custom resources and reconciles them. The scheduler is the brain of the control plane: it decides which Server replica each Model should live on. The agent is a sidecar container on every server pod that downloads artifacts and asks the inference server to load and unload them. And Envoy, exposed through a Kubernetes Service called seldon-mesh, is the single front door for the data plane: every inference request enters there and is routed to the right model.

What you need before you install

Seldon Core 2 is a Kubernetes system and only a Kubernetes system. There is no native macOS or Windows installer, and no single binary that serves a model on your laptop. What you need is a cluster.

The production installation page states the requirements plainly: Kubernetes 1.27 or later, plus kubectl and helm. For learning, a local cluster is fine, and kind — Kubernetes in Docker — is the least troublesome option.

# Linux or macOS, with Docker already running
kind create cluster --name seldon
kubectl cluster-info --context kind-seldon
kubectl version

The last command matters: confirm the server version is 1.27 or newer before you go further. Installing onto an older cluster produces Helm and CRD errors that look mysterious until you remember to check.

On macOS, Homebrew gets you the tools: brew install kubectl helm kind. You need a container runtime for kind to use, which means Docker Desktop, Colima or OrbStack. On Apple Silicon, check that the inference server images you intend to use publish an arm64 manifest before you rely on them. MLServer and Triton are large images with native dependencies and architecture support is not something to assume; if a pod sits in a crash loop with an exec format error, that is what happened.

On Windows, use WSL2 with an Ubuntu distribution plus Docker Desktop's WSL integration, and run every command from the WSL shell. kubectl and helm work natively in PowerShell, but the sample scripts and the curl invocations in the documentation are written for bash, so a Linux shell saves you constant translation.

Two optional pieces are worth knowing about now. Kafka is required for pipelines and not required for calling models directly, so you can do the first half of this guide without it. A Prometheus stack in a seldon-monitoring namespace gives you the metrics the Mid-level guide builds on. The documentation also describes a learning environment with a self-hosted Kafka setup, which is the shortest path to a laptop cluster that can run pipelines.

Installing Seldon Core 2 with Helm

The installation is four Helm charts applied in order. The order is not cosmetic, and understanding why teaches you something about how Kubernetes extensions work.

kubectl create ns seldon-mesh || echo "Namespace seldon-mesh already exists"
kubectl create ns seldon-monitoring || echo "Namespace seldon-monitoring already exists"

helm repo add seldon-charts https://seldonio.github.io/helm-charts/
helm repo update seldon-charts

Two namespaces: seldon-mesh for the runtime and your models, seldon-monitoring for the metrics stack. Now the charts.

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
  --namespace default --install

helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh --set controller.clusterwide=true --install

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
  --namespace seldon-mesh --install

kubectl get pods -n seldon-mesh

What each chart does, in the order they must be applied:

1. seldon-core-v2-crds installs the custom resource definitions — the API extensions that teach your cluster the words Model, Server, Pipeline and Experiment. CRDs are cluster-scoped, which is why this one goes into default rather than seldon-mesh. Nothing else can work until the cluster knows these kinds exist.
2. seldon-core-v2-setup installs the operator, the controller that watches those resources and acts on them. controller.clusterwide=true tells it to watch every namespace. The alternative is a namespaced operator: --set "controller.watchNamespaces={ns1,ns2}", which is how multi-tenant installations isolate teams.
3. seldon-core-v2-runtime installs the per-namespace runtime: the scheduler, Envoy, the model and pipeline gateways, and the dataflow engine. This is the machinery that actually places and serves models.
4. seldon-core-v2-servers installs the default Servers — one MLServer and one Triton, one replica each.

The helm upgrade ... --install form is used throughout rather than helm install, because it is idempotent: the same command creates the release the first time and updates it afterwards, which makes it safe to put in a script or re-run after a typo.

Give the pods a minute, then look:

kubectl get pods -n seldon-mesh
kubectl get servers -n seldon-mesh

You should see the controller manager, the scheduler, Envoy, the dataflow engine, the model gateway, the pipeline gateway, and StatefulSets for mlserver and triton. Exact pod names vary with the version and your values, so read what your cluster actually prints rather than matching a list from a blog post. kubectl get servers is the command to remember: it is how you find out what your models are allowed to be scheduled onto.

Uninstalling runs in reverse: servers, runtime, setup, then CRDs. Leave the CRD chart for last and treat it with respect — deleting the CRDs deletes every Model, Server and Pipeline in the cluster, because Kubernetes garbage-collects the custom objects when their definition disappears. More than one person has removed the "harmless CRDs chart" first and taken production with it.

Your first model

The canonical first model is an iris classifier: a scikit-learn model that takes four flower measurements and predicts a species. Seldon publishes the trained artifact in a public bucket, so you do not need to train anything.

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.5.0/iris-sklearn"
  requirements:
  - sklearn
  memory: 100Ki

Four lines of spec carry the whole idea, so read each one.

apiVersion: mlops.seldon.io/v1alpha1 is the API group and version. Note the v1alpha1: there is no v1 API for Core 2 yet. Every Core 2 resource you write uses this exact string, and getting it wrong produces no matches for kind "Model", which is the same error you get when the CRDs are not installed.

storageUri points at the artifact. The value is an rclone-style URI, which means Seldon supports the storage backends rclone supports: gs:// for Google Cloud Storage, s3:// for S3 and S3-compatible stores, local paths on a persistent volume, and more. Notice the mlserver_1.5.0 segment in the sample path — Seldon publishes the same sample artifacts rebuilt for different MLServer versions, because a pickle saved by one scikit-learn version may not load in another. That detail is a preview of a real production concern.

requirements is a list of capability tags. This is the most important field to understand, and the next section is devoted to it. For now: sklearn means "this artifact needs a server that can run scikit-learn models".

memory is a declared budget the scheduler uses when deciding how many models fit on a replica. The 100Ki in the sample is a toy value chosen because the iris model genuinely is tiny. For anything real, measure and declare honestly; the consequences of lying are covered below.

Apply it and wait:

kubectl apply -f iris.yaml -n seldon-mesh
kubectl get model iris -n seldon-mesh
kubectl wait --for condition=ready --timeout=300s model iris -n seldon-mesh

kubectl wait is worth adopting as a habit rather than refreshing get in a loop. It blocks until the resource reports itself ready, or fails after the timeout — which makes it usable in scripts and in CI, and makes "it never became ready" an explicit outcome rather than something you give up waiting for.

While you are waiting, look for your model's pod. There isn't one. kubectl get pods -n seldon-mesh shows the same pods as before. The iris model was loaded into the existing MLServer replica by the agent sidecar, which fetched the artifact from the bucket and asked MLServer to load it. That is multi-model serving in action, and it is why the second, third and tenth model you deploy cost you almost nothing.

Calling the model: the Open Inference Protocol

Seldon Core 2 speaks the Open Inference Protocol, or OIP — the standard formerly known as the V2 inference protocol and still widely called KServe V2. It is a shared contract, which is the practical meaning of Seldon's "learn once, deploy anywhere" claim: KServe speaks it too, so client code written against one works against the other.

The protocol is available over HTTP/REST and over gRPC. The inference path has a fixed shape:

POST /v2/models/<model-name>/infer

First you need an address. On a cloud cluster with a load balancer, the mesh has an external IP:

kubectl get svc seldon-mesh -n seldon-mesh \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

On kind, minikube or any laptop cluster there is no load-balancer controller, so that command prints nothing. Forward the service port instead:

kubectl get svc seldon-mesh -n seldon-mesh
kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80

Run the get svc first and read the port column; map your local 8080 to whatever port the service actually exposes rather than trusting a number from a tutorial. With the forward running in one terminal, call the model from another:

curl -X POST http://localhost:8080/v2/models/iris/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: iris" \
  -d '{"inputs":[{"name":"predict","shape":[1,4],"datatype":"FP32","data":[[1,2,3,4]]}]}'

Two things in that request decide whether it works.

The Seldon-Model header is how Envoy routes. There is one seldon-mesh Service for every model in the namespace, and the header is what tells the mesh which model you mean. Omit it and the request does not reach your model, no matter how correct the URL path is. This trips up almost everyone exactly once, because the path already contains the model name and it feels redundant. It is not: the path is part of the OIP contract that the inference server sees, and the header is the mesh's routing key.

The body is a tensor description, not a row of JSON. Each entry in inputs has a name the model expects, a shape, a datatype from the OIP type list, and data. The datatype must match what the model was trained on: FP32 for the iris floats, INT64 for integer features. Sending FP32 where INT64 is expected gets you an error from the inference server rather than a wrong prediction, which is the right behaviour but still a surprise the first time.

Here is the shape for a different sample, the income classifier, which takes twelve integer features:

{"inputs":[{"name":"income","datatype":"INT64","shape":[1,12],"data":[53,4,0,2,8,4,2,0,0,0,60,9]}]}

Note that the tensor name changed with the model. Tensor names are part of the model's own contract, not Seldon's, and reading them off the model's metadata or its training code is part of deploying it.

When you are done, remove the model the same way you created it:

kubectl delete -f iris.yaml -n seldon-mesh

Declarative all the way through: the file is the source of truth, and apply and delete are the only two verbs you need.

Capabilities and scheduling

This section explains the error you are most likely to hit, so it is worth reading even if everything is working.

Every Server advertises a set of capabilities: tags describing what it can run. Every Model declares requirements: tags describing what it needs. The scheduler places a Model on a Server only when the Model's requirements are a subset of the Server's capabilities. That is the whole matching rule, and it is string comparison — no clever inference, no fuzzy matching.

The default MLServer advertises these capabilities:

mlserver, alibi-detect, alibi-explain, huggingface, lightgbm,
mlflow, python, sklearn, spark-mlib, xgboost

The default Triton advertises these:

triton, dali, fil, onnx, openvino, python, pytorch,
tensorflow, tensorrt

Read those two lists together and the division of labour is obvious. Classical machine learning — scikit-learn, XGBoost, LightGBM, MLflow-packaged models, Hugging Face transformers — goes to MLServer. Deep-learning graphs and optimised runtimes — ONNX, PyTorch, TensorFlow, TensorRT, OpenVINO — go to Triton. Note spark-mlib, spelled exactly that way in the capability list, which is a reminder that these are opaque strings rather than names with meaning: you match the string that is there, including its typo.

Capability matching is only the first of three filters. The scheduler also checks that enough Server replicas exist for the Model's requested replica count, and that each candidate replica has enough free memory for the Model's declared memory. Fail any of the three and the Model reports a condition ModelReady of False with the reason ScheduleFailed.

On memory, one point matters enough to repeat: it is a budget the scheduler reasons with, not a limit the kernel enforces. Declaring 100Ki for a model that really needs two gigabytes does not cap it at 100Ki. It tells the scheduler the model is tiny, so the scheduler happily packs many more models onto that replica than will fit, and then the pod is killed for exceeding its container memory limit. The symptom appears far from the cause: a server replica restarting repeatedly, taking unrelated models down with it. Measure what your model's resident size actually is and declare that.

Since 2.9, the scheduler also supports partial scheduling: when capacity is short, a model may run on fewer replicas than requested rather than failing outright. This is usually what you want — degraded service beats no service — but it means "ready" does not always mean "at full requested capacity", so check the replica count rather than only the condition. The 2.10.1 release fixed a regression in exactly this area where models came back on too few replicas after pod restarts, which is another reason to be on a current patch version.

Running your own Server

The default Servers get you started, but eventually you need control over the image: a specific MLServer version, extra Python dependencies your model imports, or a GPU node. That means writing a Server resource.

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-custom
spec:
  serverConfig: mlserver
  capabilities:
  - income-classifier-deps
  podSpec:
    containers:
    - image: seldonio/mlserver:1.6.0
      name: mlserver

Three fields do the work. serverConfig: mlserver names a ServerConfig, a template that defines the pod shape, the agent sidecar and the rclone sidecar. You are not writing an inference server from scratch; you are starting from Seldon's MLServer template and overriding part of it. podSpec is that override, and because it is an ordinary Kubernetes pod spec you can set resources, node selectors, tolerations and environment variables there — which is how GPU scheduling is done, by requesting nvidia.com/gpu and targeting GPU nodes.

capabilities is the field to be careful with. Setting capabilities replaces the defaults entirely. The server above advertises exactly one tag, income-classifier-deps, and nothing else — so no model requiring sklearn can land on it. That is sometimes exactly what you want: a dedicated server for one model with one specific dependency set, with a tag that no other model will accidentally match. When instead you want to keep the defaults and add to them, use extraCapabilities. If both are set, capabilities wins.

A model targeting that server names the matching tag:

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-classifier
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.4.0/income-sklearn/classifier"
  requirements:
  - income-classifier-deps
  memory: 100Ki

This gives you a clean pattern for version pinning. The Servers documentation shows the same idea with a version tag: a server running image seldonio/mlserver:1.3.4 advertising the capability mlserver-1.3.4, and models that need that exact runtime requiring that exact tag. The capability string becomes a contract between an artifact and the runtime that can load it — which is much better than discovering the mismatch as an unpickling error in a container log.

Private artifact stores need credentials, and credentials belong in Kubernetes Secrets in rclone configuration format, referenced from the Model or configured for the agent — never inline in YAML you commit. Getting the exact field names right is worth doing against the current CRD reference rather than from memory, and the Mid-level guide covers the production pattern alongside external secret managers.

Your first Pipeline

A Pipeline wires Models into a graph. The classic shape is a preprocessor feeding a classifier, so that the client sends raw input and the transformation happens server-side where it is versioned alongside the model.

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: income-classifier-app
spec:
  steps:
    - name: preprocessor
    - name: income-classifier
      inputs:
      - preprocessor
  output:
    steps:
    - income-classifier

Each entry in steps names a Model that is already deployed. A Pipeline does not deploy models; it connects them. If a named step does not exist as a ready Model, the Pipeline will not become ready either, and the fix is upstream.

inputs is where the wiring lives: listing preprocessor under income-classifier means the classifier consumes the preprocessor's output. The general form uses dot notation — <stepName|pipelineName>.<inputs|outputs>.<tensorName> — which lets a step consume one specific named tensor rather than everything upstream produced.

output.steps declares which step's output is the pipeline's answer. It is required for synchronous REST and gRPC calls: without it, there is nothing to return to the caller.

Calling a pipeline looks almost identical to calling a model, with one difference that is easy to miss:

curl -X POST http://localhost:8080/v2/models/income-classifier-app/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: income-classifier-app.pipeline" \
  -d '{"inputs":[{"name":"income","datatype":"INT64","shape":[1,12],"data":[53,4,0,2,8,4,2,0,0,0,60,9]}]}'

The .pipeline suffix in the Seldon-Model header is what tells the mesh you mean the pipeline rather than a model of the same name. The URL path does not carry it. Forget the suffix and you get a routing failure that reads as "model not found", which sends people hunting for a deployment problem that does not exist.

Underneath, each step's input and output is a Kafka topic. That is why Kafka is a hard dependency for pipelines and not for direct model calls, and it is also where the audit story comes from: the data that flowed through every step is sitting on a topic, so it can be inspected, replayed, or fed into Kafka Connect or ksqlDB. Since 2.10 a Kafka Schema Registry integration makes the schema contracts of those topics visible too.

Three more pipeline features are worth knowing by name so you recognise them when you need them. Joins control what happens when a step has several inputs: inner waits for all of them, outer waits for a window set by joinWindowMs and then proceeds with what arrived, and any proceeds on the first arrival. tensorMap renames tensors between steps when the producer's output name does not match the consumer's expected input name — the alternative being to retrain or rewrap a model over a naming mismatch. And triggers start a step when another step completes without passing that step's data along, which is how you express "run this only after that finished".

Reading errors

Seldon Core 2 reports problems in three places, and knowing which to read saves most of the time people waste on it. Kubernetes conditions tell you about scheduling. The agent container's logs tell you about fetching and loading artifacts. The inference server's logs tell you about running the model.

Here is the routine, in order:

kubectl get model,server,pipeline -n seldon-mesh
kubectl describe model iris -n seldon-mesh
kubectl logs -n seldon-mesh <server-pod> -c agent
kubectl logs -n seldon-mesh <server-pod> -c mlserver

The first command is the overview: it shows every resource and its readiness in one screen, which immediately tells you whether the problem is one model or the whole runtime. The second reads conditions. The third and fourth are the two containers that matter on a server pod — the agent, which fetched your artifact, and the inference server, which tried to load and run it.

The failures you are most likely to meet:

Two of those rows deserve emphasis because they are version answers rather than configuration answers. "Pipelines fail to create or delete" and "gRPC streams block" were both long-standing bugs fixed in 2.10.2. If you are debugging either on an earlier version, you are debugging someone else's already-fixed bug. Check your version before you check your YAML is a good general habit with this project, given how much the 2.10 patch line changed.

Configuration you will actually touch

Most of Seldon Core 2's configuration lives in Helm values and in two custom resources, and at beginner level you need to recognise them rather than master them.

SeldonRuntime is the resource that installs the per-namespace runtime — scheduler, Envoy, gateways, dataflow engine. The seldon-core-v2-runtime chart creates one for you. If you want Seldon serving models in a second namespace, a second SeldonRuntime is the mechanism.

SeldonConfig holds the runtime's configuration: Kafka connection details, tracing, and since 2.10 a scaling configuration block. The point of centralising it in a resource is that configuration can be changed after the cluster is deployed rather than only at install time. Two settings from 2.10 are worth knowing exist: maxShardCountMultiplier, which lets pipeline components scale horizontally beyond the limit that Kafka partition counts used to impose, and the new scaling config block under config.ScalingConfig. The upgrade notes say to set the new scaling configuration before upgrading to 2.11 — and that with Helm the 2.9-to-2.10 upgrade fills the new options from your existing values automatically, while a non-Helm install must set maxShardCountMultiplier by hand.

Operator scope is the decision you make at install time. controller.clusterwide=true watches everything; controller.watchNamespaces={ns1,ns2} restricts it. The namespaced form is the foundation of multi-tenancy, and it pairs with per-namespace SeldonRuntimes and separate Kafka credentials per team.

Kafka security is the configuration most likely to block your first real pipeline. TLS is configured by referencing a Secret under security.kafka.ssl.client.secret in the Helm values, and SASL_SSL — which is what managed Kafka services such as Confluent Cloud use — needs both the SASL mechanism and the client credentials set. The documentation has worked examples for Confluent SASL and OAuth 2.0. The failure mode is recognisable: broker timeouts and connection errors in the dataflow-engine and model-gateway logs, almost always because a Secret name is wrong or the Secret is in the wrong namespace.

Monitoring comes from Prometheus. The install flow creates a seldon-monitoring namespace for a metrics stack, and Seldon exposes metrics including request rates such as infer_rps, which is also the metric the documented HPA pattern scales on. You do not need this for your first model, but install it before you need to answer "is it slow?" rather than after.

Finally, there is a seldon command-line tool, distributed as a release binary, which talks to the scheduler directly to load, inspect and call models and pipelines. It is genuinely useful — particularly pipeline inspect, which reads the Kafka topics behind a pipeline so you can see what actually flowed through each step. On Kubernetes, though, prefer kubectl apply and let the operator reconcile: that keeps your cluster's state matching files in git, which is the entire point of a declarative system. Check the current CLI documentation for exact syntax and connection flags before scripting against it.

Putting it all together

Here is the whole beginner path as one exercise. Start from an empty cluster and end with a two-step pipeline you can call. Do it without scrolling up if you can.

# 1. Cluster
kind create cluster --name seldon
kubectl version

# 2. Namespaces and chart repository
kubectl create ns seldon-mesh || echo "Namespace seldon-mesh already exists"
kubectl create ns seldon-monitoring || echo "Namespace seldon-monitoring already exists"
helm repo add seldon-charts https://seldonio.github.io/helm-charts/
helm repo update seldon-charts

# 3. Four charts, in order
helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
  --namespace default --install
helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh --set controller.clusterwide=true --install
helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install
helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
  --namespace seldon-mesh --install

# 4. Confirm
kubectl get pods -n seldon-mesh
kubectl get servers -n seldon-mesh

Then the model, the wait, and the call:

kubectl apply -f iris.yaml -n seldon-mesh
kubectl wait --for condition=ready --timeout=300s model iris -n seldon-mesh

kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80 &

curl -X POST http://localhost:8080/v2/models/iris/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: iris" \
  -d '{"inputs":[{"name":"predict","shape":[1,4],"datatype":"FP32","data":[[1,2,3,4]]}]}'

Four checkpoints, each with a specific thing to verify. After the charts, kubectl get servers lists two Servers. After the model, kubectl get pods shows no new pod — the model is inside an existing replica. After the call, you get a tensor back. And after you deliberately drop the Seldon-Model header, you get a routing failure that you now recognise on sight.

For the pipeline half, deploy the two quickstart models, apply the Pipeline, and call it with Seldon-Model: income-classifier-app.pipeline. You will need Kafka for this, which is what the learning-environment documentation sets up.

Clean up in reverse order, and remember what the last step does:

kubectl delete -f iris.yaml -n seldon-mesh
helm uninstall seldon-core-v2-servers -n seldon-mesh
helm uninstall seldon-core-v2-runtime -n seldon-mesh
helm uninstall seldon-core-v2-setup -n seldon-mesh
helm uninstall seldon-core-v2-crds -n default
kind delete cluster --name seldon

That CRD uninstall is the one to be careful with on any cluster you did not create for practice.

What you can now do, and what comes next

You can install Seldon Core 2 on a Kubernetes cluster from four Helm charts in the right order, and explain what each chart contributes. You can deploy a trained artifact as a Model, understand that it lives inside a shared Server replica rather than a pod of its own, and know what each field of its spec means. You can call it over the Open Inference Protocol and you know why the Seldon-Model header exists. You can read a ScheduleFailed condition and name the three filters the scheduler applies. You can write your own Server with a pinned image, and you know the difference between capabilities and extraCapabilities. You can chain models into a Pipeline, and you know why calling one needs the .pipeline suffix. And you can tell Seldon Core 2 apart from Seldon Core 1 on sight, which is more than most search results manage.

What the Mid-level guide takes up next, in the order it matters:

Scaling properly. Native model autoscaling is off in 2.10, so scaling means HPA or KEDA on the Servers. The documented pattern has real constraints worth understanding before you rely on it: only custom Prometheus metrics, matched HPAs on both the Model and the Server applied together, and a one-to-one mapping between Models and Servers that gives up the multi-model-serving benefit. Also: the scheduler does not create Server replicas, so a Model that scales ahead of its Server reports ScheduleFailed while the existing replicas keep serving.

Pipelines in anger. Join semantics, tensorMap, triggers, conditional routing, cross-pipeline externalInputs and externalTriggers, and cyclic pipelines with bounded iterations. Plus the Kafka side: partitions, retention and the maxShardCountMultiplier that lets pipeline components scale past partition limits.

Experiments. Weighted traffic splits for A/B tests, and shadow traffic for validating a candidate against production requests without serving its answers.

Monitoring both kinds. Operational monitoring with Prometheus and Grafana, and data-science monitoring by deploying drift and outlier detectors — which are Models with the alibi-detect capability — inside your pipelines.

Where to go in this catalogue. KServe is the closest comparison and also speaks the Open Inference Protocol; the architectures differ sharply, with Seldon's Kafka pipelines and multi-model servers against KServe's one-InferenceService-per-model model. Triton is the other default server here and is worth knowing directly. MLflow is where the artifacts you deploy usually come from, and MLServer can load MLflow-packaged models through the mlflow capability. BentoML is an alternative packaging-and-serving route worth comparing. Kubernetes and Helm are the ground this all stands on, and everything in this guide is easier if those are solid. Prometheus and Grafana are the monitoring stack, Alibi Detect is the drift and outlier library behind the data-science monitoring story, and Evidently covers the same ground from a different angle.

One closing note on region. If you are deploying for a Gulf or Egyptian employer with data-residency obligations, Seldon Core 2's being plain Kubernetes is the point: it runs wherever your cluster runs, on-premises or in a UAE or Saudi cloud region, with no dependency on a vendor-hosted control plane. But do check your Kafka: a managed Kafka in a distant region moves every pipeline payload across a border, and with pipelines that payload is also retained on topics there. Keep the cluster and its Kafka in the same region, and decide topic retention with your compliance obligations in mind rather than as a performance setting.

Sources

• Seldon Core 2 documentation home
• Concepts
• Core features
• Architecture
• Quickstart
• Production environment installation
• Learning environment installation
• Upgrading
• Models
• Model scheduling
• Servers
• Pipelines
• Custom servers example
• Single-model-serving HPA
• Operational monitoring
• Pipeline advanced configuration
• Seldon Core releases
• Seldon Helm charts repository

Seldon Core comes up in interviews for MLOps, platform and machine-learning engineering roles, usually as the "how would you serve this?" question. What interviewers are testing is rarely whether you memorised a YAML field. It is whether you understand that serving is a distributed-systems problem and can talk about it without hand-waving. This file gives you the answers to have ready at beginner level, the traps that catch people, and a self-test.

The thirty-second answer

Seldon Core 2 is a Kubernetes-native framework for deploying and managing machine-learning and LLM systems. You describe what you want with custom resources — a Model pointing at a trained artifact, a Server that hosts models, a Pipeline that wires models into a graph through Kafka, an Experiment that splits traffic — and the operator and scheduler make it real, serving everything behind one endpoint that speaks the Open Inference Protocol.

Two sentences, and every noun in them is load-bearing. If you have thirty seconds more, add the two differentiators:

Two things make it distinctive. Multi-model serving means a server replica hosts many models rather than one, so forty small models do not cost forty pods. And it is data-centric: pipeline inputs and outputs go through Kafka topics, so predictions are auditable and replayable rather than just logged.

Do not say "it makes deployment easy" or anything else that sounds like a brochure. Interviewers hear that from everyone.

The first thing to get right: which Seldon

This is the question behind the question, and getting it right early signals that you have actually used the tool rather than read about it.

There are two separate products. Seldon Core 1 is built around a single SeldonDeployment custom resource and a Python wrapper library you subclass. Seldon Core 2 is a different architecture with four resources — Model, Server, Pipeline, Experiment — under the API group mlops.seldon.io/v1alpha1. The v1 line is still maintained, with v1.19.0 released in January 2026 as a security and maintenance release, but it is not a path towards v2 and the resources are not compatible.

Why this earns points: most online material about Seldon is about v1, so someone who learned from search results will describe v1's architecture while claiming to know the current product. Saying "I should check which line you are on, because they are different systems" is a strong opening, and it is also true.

The model, in four nouns

Expect to be asked to explain the architecture. Structure the answer around the four resources plus the two planes.

A Server is a deployment of an inference server as a StatefulSet of replicas. Two come with a default install: MLServer, Seldon's Python-based server, and NVIDIA Triton. Each hosts many models.

A Model is a custom resource pointing at a trained artifact. Its important fields are storageUri, an rclone-style URI to the artifact; requirements, a list of capability tags describing what the artifact needs; and memory, a declared size budget. The key insight to state out loud: a Model is not a pod. It is a request to the scheduler to load an artifact onto some Server replica that can host it.

A Pipeline is a directed graph of Models wired through Kafka topics, with steps, inputs and a required output.steps. A Experiment splits traffic between candidates by weight for A/B testing.

Then the two planes. The control plane is the operator, which reconciles your resources; the scheduler, which decides where each model goes; and the agent, a sidecar on every server pod that fetches artifacts and loads them into the inference server. The data plane is Envoy, exposed as a Kubernetes Service called seldon-mesh, plus the inference servers themselves. Pipelines add Kafka, a model gateway, a pipeline gateway and a dataflow engine between the two.

If you can draw that on a whiteboard in thirty seconds you are ahead of most candidates for a beginner-level role.

Commands to have ready

Nobody will ask you to type YAML from memory, but being fluent with the verbs reads as experience.

kubectl get servers -n seldon-mesh
kubectl apply -f iris.yaml -n seldon-mesh
kubectl wait --for condition=ready --timeout=300s model iris -n seldon-mesh
kubectl get model,server,pipeline -n seldon-mesh
kubectl describe model iris -n seldon-mesh
kubectl logs -n seldon-mesh <server-pod> -c agent

The install, as four charts in a fixed order — CRDs, setup, runtime, servers:

helm repo add seldon-charts https://seldonio.github.io/helm-charts/
helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
  --namespace default --install
helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh --set controller.clusterwide=true --install
helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install
helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
  --namespace seldon-mesh --install

And an inference call, which is the one thing worth being able to reproduce exactly:

curl -X POST http://${MESH_IP}/v2/models/iris/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: iris" \
  -d '{"inputs":[{"name":"predict","shape":[1,4],"datatype":"FP32","data":[[1,2,3,4]]}]}'

The classic traps, and why they happen

Interviewers love these because the answers reveal whether you have debugged the thing or only deployed it once from a tutorial.

The missing Seldon-Model header. The URL path already names the model, so the header looks redundant and people drop it. It is not redundant: the path is part of the Open Inference Protocol contract that the inference server sees, while the header is how Envoy decides which model behind the single seldon-mesh Service the request belongs to. Without it the request never reaches your model.

The .pipeline suffix. Calling a pipeline uses the same URL shape but the header must be Seldon-Model: <name>.pipeline. Omit the suffix and the failure reads like "model not found", which sends people looking for a deployment problem that does not exist.

ScheduleFailed from a capability typo. The scheduler matches strings. scikit-learn is not sklearn, and the capability list even contains spark-mlib spelled that way — these are opaque tags, not names with meaning. Say that the matching rule is "the Model's requirements must be a subset of the Server's capabilities" and you have answered the question properly.

capabilities replaces, extraCapabilities extends. Setting capabilities on a Server wipes out the defaults, so a server you meant to specialise can no longer host anything requiring sklearn. If both fields are set, capabilities wins.

Treating memory as a limit. It is a budget the scheduler reasons with when packing models onto replicas, not an enforced cap. Under-declare it and the scheduler over-packs the replica, the pod gets killed for exceeding its container memory limit, and unrelated models on the same replica go down with it. The symptom appears nowhere near the cause.

Deleting the CRDs chart first. CRD deletion cascades: every Model, Server and Pipeline in the cluster disappears with the definition. Uninstall in reverse order.

Expecting a pod per model. People deploy a model, look for its pod, find none, and conclude the install is broken. That absence is multi-model serving working.

Common interview questions

"What problem does Seldon Core solve that a Flask app does not?"

Start concrete. One Flask app serving one model works. The problems appear at scale: nine models means nine services with nine request formats and nine sets of pinned dependencies, each holding a replica's worth of idle memory. Seldon Core 2 gives you one request format for every framework through the Open Inference Protocol, multi-model serving so replicas are shared rather than dedicated, declarative resources so the cluster's state matches files in git, and — for pipelines — an audit trail on Kafka. The honest framing is that it is infrastructure: worth it when you have many models or need auditability, overkill for one endpoint.

"Walk me through deploying a scikit-learn model."

Install the four Helm charts in order. Put the trained artifact somewhere rclone can reach — a GCS or S3 bucket. Write a Model with storageUri pointing at it, requirements: [sklearn], and a memory figure you measured. kubectl apply it, then kubectl wait --for condition=ready. The scheduler places it on an MLServer replica that advertises sklearn; the agent sidecar fetches the artifact and asks MLServer to load it. Then call /v2/models/<name>/infer through the seldon-mesh Service with the Seldon-Model header set. Mention that you would check kubectl get servers first to see what capabilities are available.

"A model is stuck not-ready. How do you debug it?"

Name the layers, in order, because that is the real answer. kubectl describe model and read the conditions — if it says ScheduleFailed, this is a scheduling problem: check capability strings letter by letter, check whether you asked for more replicas than the Server has, check whether the declared memory can fit. If the conditions look fine but the server pod is restarting, it is an artifact problem: read the agent container's log for an rclone fetch failure, then the inference server's log for a load failure. Kubernetes conditions report scheduling, the agent reports fetching, the inference server reports loading and running. Knowing which layer owns which failure is the skill.

"How do you scale a model under load?"

The current answer is HPA or KEDA on the Servers, because model-native autoscaling was deliberately disabled in version 2.10.0 pending wider fixes. Mentioning that is the whole point of the question: it shows you read release notes rather than older documentation. Add the caveats if you know them — the documented HPA pattern supports only custom Prometheus metrics, needs matched autoscalers on the Model and the Server applied together, and requires a one-to-one Model-to-Server mapping, which gives up the multi-model-serving saving.

"When would you use a Pipeline instead of just calling a model?"

When you genuinely have a graph: a feature transformer feeding a classifier, or a classifier with a drift detector watching the same input. Then name the cost, because that is what separates a good answer from a confident one. Every pipeline request goes through Kafka, which means topics, partitions, retention and extra latency. That buys auditability and replay. For a single model behind an endpoint it buys nothing, so use a direct model call.

"Seldon Core or KServe?"

Do not pick a winner; describe the difference. Both speak the Open Inference Protocol, so client code is portable between them. Seldon Core 2 is data-centric: Kafka-backed pipelines, multi-model servers, an audit trail on topics. KServe is serverless-style with one InferenceService per model. If the team's need is "many models, cheap, with a record of every prediction", Seldon's design fits. If it is "one model per service, scale to zero", KServe's does. Then say you would want to benchmark rather than assert performance — interviewers respect the refusal to make up numbers.

"What is the Open Inference Protocol?"

The standard request and response format Seldon Core 2 uses, formerly called the V2 inference protocol or KServe V2, available over HTTP/REST and gRPC. The inference path is /v2/models/<name>/infer and the body describes tensors: each input has a name, a shape, a datatype and data. The value of it is that the format does not change with the framework, and that other servers implement it too.

Three more questions you should not be surprised by

"Why is Kafka in there at all?"

Because Seldon describes its approach as data-centric MLOps: the data flowing through models is treated as a first-class concern rather than a side effect. Concretely, every step of a Pipeline reads its input from a Kafka topic and writes its output to one. That is what makes the audit trail and replay possible — the data that flowed through each step is still sitting on a topic, so you can inspect it, replay it, or feed it to Kafka Connect or ksqlDB, and since 2.10 see its schema through the Schema Registry integration. The trade-off to name without being asked: Kafka is a hard dependency for pipelines and completely unnecessary for direct model calls, and its retention and partition count are a real operating cost.

"How does a model get from a bucket into a running server?"

This question is really asking whether you understand the control plane. You apply a Model. The operator sees the new resource and reconciles it. The scheduler picks a Server replica that advertises all the Model's requirements, has capacity for the requested replicas, and has free memory for the declared memory. The agent sidecar on that pod fetches the artifact from storageUri using rclone, then asks the inference server — MLServer or Triton — to load it. Once loaded, Envoy will route requests carrying that model's Seldon-Model header to it. Four components, in that order: operator, scheduler, agent, inference server.

"What would you do differently in production from what you did on your laptop?"

A fair question for a beginner, and the honest answer is a list. Pin Helm chart versions so the install is reproducible. Declare memory from measurement rather than copying the sample's 100Ki. Put artifact-store and Kafka credentials in Kubernetes Secrets, or behind an external secrets manager, never in committed values files. Install the Prometheus stack before you need to answer "is it slow?" rather than after. Decide whether the operator is cluster-wide or namespaced, because the namespaced form with controller.watchNamespaces is how teams get isolated from each other. Put an API gateway or service mesh in front of the inference endpoint rather than assuming the mesh authenticates callers. And know that deleting the CRDs chart cascades to every Model, Server and Pipeline in the cluster.

What a strong answer adds

At beginner level nobody expects you to have run Seldon Core 2 at scale. What separates a strong candidate is three habits of speech rather than three extra facts.

Name the cost of the thing you are recommending. "Use a Pipeline" is a weak answer. "Use a Pipeline, and accept Kafka topics, retention and extra latency in exchange for an audit trail" is a strong one. Every architectural choice in this system is a trade, and interviewers are listening for whether you know which trade you just made.

Say which layer owns a failure. Candidates who have only deployed from a tutorial describe debugging as "check the logs". Candidates who have actually broken something say "conditions for scheduling, the agent container for fetching, the inference server for loading". The second answer is short, specific and almost impossible to fake.

Distinguish what you verified from what you read. If you do not know whether a field exists, say you would check the CRD reference rather than inventing a name. If you are asked whether Seldon is faster than KServe, say you would benchmark it on the workload rather than quoting a number. Interviewers are hiring someone who will not fabricate configuration under pressure, and demonstrating that costs you nothing.

Sixty-second self-test

Answer out loud, then check yourself against the sections above.

1. Name the four Seldon Core 2 custom resources and the API group they belong to.
2. Where does a Model actually run, if not in its own pod?
3. What are the three things the scheduler checks before placing a model?
4. What does ScheduleFailed mean and what is the first command you run?
5. Which HTTP header routes an inference request, and what does a pipeline add to it?
6. What is the difference between capabilities and extraCapabilities on a Server?
7. Is memory on a Model a limit or a budget, and what breaks if you get it wrong?
8. Name the four Helm charts in install order, and say what happens if you uninstall them in that same order.
9. What changed about autoscaling in 2.10.0, and what should you use instead?
10. How do you tell a Seldon Core 1 tutorial from a Core 2 one in one glance?

Eight or more without hesitating is a solid beginner footing. If questions 3, 5 or 7 gave you trouble, those three are where real deployments actually fail, so go back and re-read them before any interview.

These are the mistakes people actually make with Seldon Core 2 in their first weeks, each with the symptom you will see, the cause underneath it, and the fix. Then the habits worth adopting on day one, and a checklist to run before you tell anyone your model is deployed.

Mistake 1: following a Seldon Core 1 tutorial

Symptom. kubectl apply fails immediately with no matches for kind "SeldonDeployment". Or the reverse: you find a tutorial telling you to pip install seldon-core and subclass a Python class, and nothing in it resembles your cluster.

Cause. Seldon Core 1 and Seldon Core 2 are different architectures sharing a name. Core 1 has one custom resource, SeldonDeployment, and a Python wrapper library. Core 2 has Model, Server, Pipeline and Experiment under mlops.seldon.io/v1alpha1. The v1 line is still released — v1.19.0 came out in January 2026 as a maintenance and security release — which makes it even more confusing, because recency does not tell you which product a page is about. Most Seldon material online is about v1.

Fix. Before trusting any Seldon page, look for the giveaway. mlops.seldon.io means Core 2. SeldonDeployment means Core 1. Treat the official Core 2 documentation as your primary source and blog posts as hints to verify, not instructions to follow. Do not install the v1 charts while following v2 instructions.

Mistake 2: looking for your model's pod

Symptom. You deploy a Model, it reports ready, and kubectl get pods shows exactly the pods that were there before. You conclude the install is broken.

Cause. A Model is not a pod. It is a request to the scheduler to load an artifact into a Server replica that can host it, and a Server replica hosts many models at once. That is multi-model serving, and it is the main reason Seldon Core 2 is cheaper than one-service-per-model designs.

Fix. Change where you look. kubectl get model,server -n seldon-mesh is your status view. To see what the model is doing at the container level, find the server pod and read its containers: the agent sidecar fetched the artifact, the inference server loaded it. Internalise the sentence "models live inside servers" and this stops being confusing permanently.

Mistake 3: dropping the Seldon-Model header

Symptom. A curl to /v2/models/iris/infer returns a 404 or a not-found error even though kubectl get model iris says ready.

Cause. There is one seldon-mesh Service for every model in the namespace. Envoy routes on the Seldon-Model header, not on the URL path. The path is part of the Open Inference Protocol contract that the inference server sees; the header is the mesh's routing key. Because the path already contains the model name, the header looks redundant and people leave it out.

Fix. Always send -H "Seldon-Model: <name>". When you wrap inference in a client library or a service, put the header in the client's default headers so it cannot be forgotten per call.

Mistake 4: calling a pipeline without the .pipeline suffix

Symptom. Models work. The pipeline is ready. Calling the pipeline fails as if the name does not exist.

Cause. A pipeline is addressed with the suffix in the header: Seldon-Model: income-classifier-app.pipeline. The URL path does not carry it. That suffix is what distinguishes a pipeline from a model of the same name.

Fix. Memorise the asymmetry — path without, header with. When a pipeline call fails, check the header before you check anything else; it is the cause far more often than the pipeline is.

Mistake 5: a capability typo

Symptom. The Model never becomes ready. kubectl describe model shows ModelReady: False with reason ScheduleFailed.

Cause. The scheduler matches capability strings exactly. A Model is placed only when its requirements are a subset of some Server's capabilities. scikit-learn is not sklearn. The default MLServer capability list includes spark-mlib spelled precisely that way, which is a useful reminder that these are opaque tags rather than words with meaning.

Fix. Run kubectl get servers -n seldon-mesh, read the capability list your servers actually advertise, and copy the string rather than typing it. When you cannot see why it will not place, the three things the scheduler checks are capability match, enough replicas for the requested count, and enough free memory for the declared memory. Work through all three rather than guessing at one.

Mistake 6: replacing a server's capabilities by accident

Symptom. You add a custom capability to a Server so one model can target it, and every other model on that server stops being schedulable.

Cause. capabilities on a Server replaces the defaults entirely. A server declaring only income-classifier-deps advertises nothing else, so no model requiring sklearn can land on it. extraCapabilities is the field that extends the defaults. If both are set, capabilities wins.

Fix. Use extraCapabilities when you want to add, capabilities when you want a dedicated server that hosts only what you explicitly allow. Both are legitimate; decide which you mean and leave a comment in the YAML saying so.

Mistake 7: treating memory as a limit

Symptom. A server replica restarts repeatedly. Models that have nothing to do with your change go down with it. Nothing in your model's own conditions explains it.

Cause. memory on a Model is a budget the scheduler reasons with when packing models onto replicas, not a cap the kernel enforces. The documentation's sample value of 100Ki is honest for the iris model and absurd for anything real. Declare 100Ki for a model that needs two gigabytes and the scheduler will cheerfully place dozens of them on one replica, which then exceeds its container memory limit and is killed.

Fix. Measure. Load the artifact, look at the resident memory, and declare something close to it with headroom. The cost of over-declaring is a few wasted megabytes of scheduling budget; the cost of under-declaring is a server replica taking down other people's models.

Mistake 8: installing or uninstalling the charts in the wrong order

Symptom. On install, no matches for kind "Model" or operator pods crash-looping. On uninstall, every Model, Server and Pipeline in the cluster vanishes at once.

Cause. The four charts have a dependency order: seldon-core-v2-crds teaches the cluster the resource kinds, seldon-core-v2-setup installs the operator that watches them, seldon-core-v2-runtime installs the per-namespace runtime, and seldon-core-v2-servers installs the default MLServer and Triton. Nothing works before the CRDs exist. And because Kubernetes garbage-collects custom objects when their definition is removed, deleting the CRDs cascades to everything built on them.

Fix. Install in that order, uninstall in reverse. Treat the CRD chart as the most dangerous Helm release in the cluster and never run helm uninstall on it to "clean up" without knowing what else is using those kinds.

Mistake 9: debugging an already-fixed bug

Symptom. Pipelines that will not create or delete. gRPC streaming calls that hang forever. Models that come back on fewer replicas than requested after a node reboot.

Cause. All three are known bugs with known fixes. The partial-scheduling regression was fixed in 2.10.1. Pipelines failing to create or delete, gRPC stream blocking, operator reconciliation problems and MLServer parallel-worker configuration were all fixed in 2.10.2.

Fix. Check your version before you check your YAML. With this project specifically, the 2.10 patch line changed enough that "which version am I on?" should be the second question you ask about any strange behaviour, right after "which Seldon is this?".

Mistake 10: expecting native model autoscaling

Symptom. You configure automatic replica scaling on a Model from an older documentation page and nothing happens.

Cause. Model-native autoscaling was deliberately disabled in 2.10.0 pending wider fixes. The official advice is to scale inference servers with the Horizontal Pod Autoscaler or with KEDA instead.

Fix. At beginner level, set replica counts by hand and know the automatic route is off. When you do get to HPA, note that the scheduler does not create Server replicas, so a Model that scales ahead of its Server reports ScheduleFailed while the existing replicas keep serving traffic — which is exactly the behaviour you want, but alarming if you did not expect it.

Mistake 11: assuming the install path works on your laptop

Symptom. kubectl get svc seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}' prints nothing. Or a server pod crash-loops with an exec format error on an Apple Silicon machine. Or image pulls fail with x509: certificate signed by unknown authority.

Cause. Three different laptop realities. A kind or minikube cluster has no load-balancer controller, so there is no external IP to print. MLServer and Triton are large images with native dependencies, and arm64 manifests are not something to assume. And on a corporate machine, something between you and the registry is usually inspecting TLS.

Fix. For the address, port-forward instead: run kubectl get svc seldon-mesh -n seldon-mesh to read the real port, then forward to it. On Apple Silicon, check the registry for an arm64 manifest before committing to an image. For the certificate error, export the interceptor's root certificate authority and add it to the trust store your container runtime uses — and do not disable verification to get past it, because that setting will outlive your debugging session and become somebody's security incident.

Mistake 12: reaching for a Pipeline too early

Symptom. A single-model deployment that now needs Kafka, has higher latency, and is harder to debug than the direct call it replaced.

Cause. A Pipeline routes every request through Kafka topics. That is what buys auditability and replay, and it is a real cost: brokers, partitions, retention, and extra hops. Kafka is a hard dependency for pipelines and not needed at all for direct model calls.

Fix. Use a Pipeline when you have a genuine graph — a transformer feeding a classifier, a drift detector watching the same input. For one model behind an endpoint, call the model.

Mistake 13: trusting an old documentation URL

Symptom. A search result or a bookmarked link under /user-guide/kubernetes-resources/... returns 404, and you conclude the documentation for the feature is gone.

Cause. The Seldon Core 2 documentation was restructured. The current pages are flat — /seldon-core-2/user-guide/models, /servers, /pipelines, /quickstart — and historical versions are kept under versioned paths such as /seldon-core-2/v2.10/.... Search engines still hold the old shapes.

Fix. When a Seldon docs link 404s, strip it back to the documentation home and navigate down rather than giving up on the page. And be deliberate about versioned paths: a page under a v2.8 prefix is describing a release two minors behind you, which is exactly how people end up configuring autoscaling that no longer exists.

Mistake 14: mismatching the artifact and the server version

Symptom. The Model schedules fine, then the server pod crash-loops. The inference server's log shows an unpickling or deserialisation error rather than anything about Seldon.

Cause. A model artifact is only loadable by a runtime with compatible libraries. Seldon's own sample buckets make this visible: the same models are published under paths like mlserver_1.4.0 and mlserver_1.5.0 because an artifact saved against one library version may not load under another. Bumping a server image changes which versions load your artifacts.

Fix. Keep the artifact and the runtime versioned together, and make the pairing explicit rather than implicit. The capability-tag pattern above is the cleanest way: give the server a tag naming its image version and have the model require that tag, so a mismatch becomes a scheduling refusal instead of a crash loop in a container log nobody is watching.

Habits worth adopting on day one

Read the release notes before installing. This project changes meaningfully between patch versions. Autoscaling being disabled in 2.10.0 and three classes of bug being fixed in 2.10.2 are both things you learn in ten minutes of reading and lose a day to otherwise.

Pin your chart versions. Add --version to each helm upgrade. Without it, re-running your install script months later gives you a different cluster than the one you tested, and an unplanned minor upgrade is a bad way to start a debugging session.

Use helm upgrade --install, not helm install. It is idempotent, so the same command works the first time and every time after. That makes it safe in scripts and safe to re-run after a typo.

Use kubectl wait instead of refreshing get. kubectl wait --for condition=ready --timeout=300s model iris -n seldon-mesh blocks until the model is ready or fails loudly at the timeout. It works in scripts and in CI, and it turns "it never came up" into an explicit outcome.

Keep a teardown-and-rebuild script. Being able to recreate your whole learning cluster in one command removes the fear that makes people avoid experimenting. Everything in Seldon Core 2 is declarative, so the script is short.

Learn which layer reports which failure. Kubernetes conditions report scheduling. The agent container's logs report artifact fetching. The inference server's logs report loading and running. Going to the right place first is most of what debugging speed actually is.

Keep credentials out of YAML. Artifact-store credentials belong in Kubernetes Secrets in rclone configuration format, and Kafka credentials belong in Secrets referenced from Helm values. Never inline a key in a manifest or a values file you commit, even on a practice cluster — the habit is what you are building.

Name the tensor, not just the model. Tensor names like predict or income are part of the model's own contract, not Seldon's, and they change per model. Record them next to the artifact when you deploy it, so the next person does not have to read the training code to construct a request.

Check the licence before anything commercial. Seldon changed Core's licence away from Apache 2.0, and production use may require a commercial licence. Read the LICENSE file in the repository and take the question to whoever owns legal risk, before you build a product rather than after.

The checklist

Run through this before you tell anyone a model is deployed.

Fourteen lines, most of them a single command. The three that catch people most often are the capability strings, the memory figure and the header — so if you skip any, do not skip those.

This is part two of three. It assumes you can already install Seldon Core 2 on a cluster, apply a Model, and get an inference response back. From here on the question changes from "how do I deploy a model" to "why did the platform behave like that, and what will it do when the load doubles".

Where this picks up

Everything here is written against Seldon Core 2.10 (latest confirmed patch v2.10.2, December 2025) on Kubernetes 1.27 or later. One warning before anything else, because it will waste a day otherwise.

Note the API group too: v1alpha1. There is no stable v1 yet, which is a useful signal about how much field-level churn to expect between minor releases, and why reading the release notes is not optional.

The control plane, step by step

At the beginner level a Model resource felt like a deployment: you applied YAML and a model appeared. It is not a deployment. Nothing in Core 2 creates a pod when you apply a Model. Understanding what actually happens is what lets you predict failures.

The sequence for a Model is this. The operator sees the new custom resource and hands it to the scheduler, the control-plane component that decides where the model should live. The scheduler looks at the Server resources it knows about, each of which is already running as a StatefulSet of replicas, and filters them: does this server advertise every capability the model's requirements list? Does it have at least as many replicas as the model asks for? Does each candidate replica have enough unused memory for the model's declared memory? If a server survives all three filters, the scheduler assigns the model to specific replicas and tells the agent sidecar on each of those pods.

The agent is the component that does physical work. It fetches the artifact from storageUri using rclone, writes it to the pod's local storage, and asks the inference server process in the same pod to load it. Only when the server reports the model loaded does the agent report success back to the scheduler, which flips the ModelReady condition on the custom resource to True.

Four consequences follow, and they explain most of the confusing behaviour at this level.

A model that cannot be placed stays pending forever, not crash-looping. There is no pod to crash. kubectl get model shows it not ready, and kubectl describe model gives the reason ScheduleFailed. Nothing retries into a different shape on your behalf; you have to change either the model's demands or the servers' supply.

Scaling a model never creates server capacity. The scheduler places models; it does not provision replicas. If you raise a model's replica count past what its server offers, the model reports ModelReady: False with ScheduleFailed while the replicas it already has keep serving traffic normally. That is a deliberate design: a failed scale-up degrades your headroom, not your availability.

Loading is a data transfer, so readiness time scales with artifact size and object-store bandwidth. A multi-gigabyte artifact pulled from a bucket in another region is slow every time a replica restarts, because the agent's local copy is not durable. This is the single most common reason a kubectl wait --for condition=ready times out on a model that is actually fine.

One logical scheduler per runtime is a control-plane choke point. While it is down, nothing new can be placed and conditions go stale, although traffic to already-loaded models continues through Envoy. Release 2.10 added Kubernetes lifecycle probes to all Core 2 components precisely to make these restarts cleaner, and 2.10.2 fixed operator reconciliation bugs that showed up as resources stuck in an intermediate state.

On the data-plane side, there is one entry point per runtime: the seldon-mesh Service, backed by Envoy. It does not route by URL path alone. Routing is driven by the Seldon-Model header, which is why the same path works for every model:

kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80

curl -X POST http://localhost:8080/v2/models/iris/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: iris" \
  -d '{"inputs":[{"name":"predict","shape":[1,4],"datatype":"FP32","data":[[1,2,3,4]]}]}'

The payload shape is the Open Inference Protocol (OIP, formerly the V2 inference protocol), the same contract KServe and Triton speak. That portability is the point of the "learn once, deploy anywhere" claim in the docs: a client written against OIP can move between KServe, a bare Triton server and Seldon without changing its request builder.

Capabilities arithmetic and multi-model serving

Capabilities are the matching language between models and servers, and almost all "it won't schedule" tickets are a capabilities problem.

A Server advertises a set of capability tags. The defaults come from the ServerConfig it references. The stock MLServer config advertises mlserver, alibi-detect, alibi-explain, huggingface, lightgbm, mlflow, python, sklearn, spark-mlib, xgboost. The stock Triton config advertises triton, dali, fil, onnx, openvino, python, pytorch, tensorflow, tensorrt. A Model lists requirements, and the rule is a subset test: every requirement must appear in the server's capability set. One missing or misspelled tag fails the whole match.

Two fields let you change a server's advertised set, and the distinction matters:

• capabilities replaces the defaults entirely.
• extraCapabilities adds to the defaults.
• If both are set, capabilities wins — so a server with both will not advertise what you listed under extraCapabilities.

Replacing rather than extending is the right move when you want a server that only accepts models you intend for it. This is how you get isolation without a separate cluster: invent a private tag, put it on the server, and require it on the models.

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-custom
spec:
  serverConfig: mlserver
  capabilities:
  - income-classifier-deps
  podSpec:
    containers:
    - image: seldonio/mlserver:1.6.0
      name: mlserver

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: income-classifier
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.4.0/income-sklearn/classifier"
  requirements:
  - income-classifier-deps
  memory: 100Ki

Two things in that Server are production habits rather than examples. The podSpec override pins the MLServer image to an explicit tag, seldonio/mlserver:1.6.0, instead of inheriting whatever the chart's default happens to be after an upgrade. And the capability name describes a dependency set, not a framework. That naming is what makes the second habit possible: use the capability tag to encode the runtime contract.

The docs' own pattern for this is a version-shaped tag. A server running seldonio/mlserver:1.3.4 advertises mlserver-1.3.4, and models whose artifacts were pickled against that version require mlserver-1.3.4. When you introduce a newer server, you add it alongside with its own tag, migrate models one at a time by editing their requirements, and retire the old server when nothing requires it. Without such a tag, a server image upgrade silently re-loads every model on a runtime it was never tested against, and the failure mode is not a clean error but subtly different predictions from a library that changed its defaults.

The reason all of this machinery exists is multi-model serving. A Core 2 Server is not one model's host; it is a pool of replicas onto which the scheduler packs many models, loading and unloading them dynamically. For a team with forty small scikit-learn models, this is the difference between forty deployments each holding an idle pod and three server replicas holding all forty models. That is the main cost argument for Core 2 over a one-pod-per-model design, and it is worth being precise about when it applies: it pays off for many small models with uneven traffic, and it pays off far less for a handful of large GPU models that each saturate a node.

Memory as a budget, not a limit

The memory: 100Ki in every example is the field people copy without reading, and it is one of the few numbers in Core 2 that quietly decides whether your cluster is stable.

spec.memory on a Model is a declaration to the scheduler, used for bin-packing. The scheduler subtracts it from each replica's available memory when it places the model and refuses to place models whose declared total exceeds what a replica has. It is not a cgroup limit. Nothing enforces it at runtime. The actual memory the model consumes is whatever the inference server allocates when it loads the artifact.

That asymmetry produces a specific and nasty failure. Declare 100Ki for a model that really needs 800Mi and the scheduler believes it can pack hundreds of such models onto one replica. It does. The replica's real memory use climbs past its container limit, the kernel OOM-kills it, the StatefulSet restarts it, the agent reloads every model that was assigned there, and the restart storm looks like a network or Kafka problem because nothing in the Seldon resources says "out of memory". The 100Ki in the docs is a toy value for a toy iris model; it is not a default to inherit.

The practice that fixes it:

1. Measure the real resident footprint of each model, once, by loading it alone on a replica and reading container memory from your Prometheus data.
2. Declare memory at that measured value plus headroom for request-time allocations — inference buffers scale with batch size, not just with weights.
3. Make the server's own container memory request and limit consistent with the sum you are willing to pack, so the scheduler's arithmetic and the kernel's arithmetic agree.
4. Re-measure when you change the runtime image, because a new runtime can change the footprint of an unchanged artifact.

Memory-aware scheduling arrived in 2.9, alongside partial scheduling: when capacity is short, a model may be placed on fewer replicas than it requested rather than failing outright. This is good for availability and confusing for operators, because a model can be ModelReady while running at half the redundancy you asked for. It is also the area where 2.10.1 fixed a regression that had existed since 2.9.0, in which models could end up on fewer replicas than requested after pod restarts and stay there. If you are on 2.9.x or 2.10.0 and see redundancy silently decay after a node drain, that is the bug, and the fix is the upgrade rather than any configuration change.

Pipelines and the Kafka substrate

A Pipeline wires models into a DAG. The mechanism is what makes Core 2 distinctive: each step's inputs and outputs are Kafka topics, and the dataflow engine runs the graph as a Kafka Streams application. A pipeline is not an in-process chain of function calls with a convenience wrapper; it is an event-driven topology that happens to expose a synchronous HTTP front door.

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: income-classifier-app
spec:
  steps:
    - name: preprocessor
    - name: income-classifier
      inputs:
      - preprocessor
  output:
    steps:
    - income-classifier

Each steps[].name must be the name of a Model that is already deployed in the same namespace; a pipeline does not define models, it references them. output.steps is what makes a synchronous REST or gRPC response possible: it tells the pipeline gateway which step's output to wait for and return. A pipeline with no output is a valid fire-and-forget topology whose results you read from Kafka, and omitting output by accident is a common reason a caller hangs or gets nothing useful back.

Calling a pipeline uses the same URL shape as a model, with one difference that costs people an afternoon:

curl -X POST http://localhost:8080/v2/models/income-classifier-app/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: income-classifier-app.pipeline" \
  -d '{"inputs":[{"name":"income","datatype":"INT64","shape":[1,12],"data":[53,4,0,2,8,4,2,0,0,0,60,9]}]}'

The .pipeline suffix in the header is what tells Envoy this name is a pipeline rather than a model. The path keeps the plain name. Forget the suffix and you get a not-found error that reads as though the pipeline does not exist, which sends you to check the resource instead of the header.

Wiring between steps uses dot notation, <stepName|pipelineName>.<inputs|outputs>.<tensorName>, so a step can consume the pipeline's original request, a previous step's output, or a single named tensor out of either. That last form is how you avoid passing whole payloads around: a classifier that needs only the features tensor can reference it directly rather than taking the preprocessor's entire output. When two steps agree on the data but disagree on the names, tensorMap renames tensors between them — the right fix when you are composing a model you do not own and cannot re-export.

The architectural consequences of Kafka being in the middle are the reason to choose pipelines at all:

• Audit. Every step's input and output is a real topic with real retention, so "what exactly did the model see when it produced that score" is answerable after the fact. For teams under financial-regulator scrutiny in the Gulf, or anywhere a model decision affects a customer's credit, that is a compliance capability rather than a debugging convenience.
• Replay. The same retention lets you re-run historical traffic through a changed graph.
• Decoupling. A slow downstream step does not block the upstream one at the transport level, because the queue absorbs it.

And the costs, which are equally real:

• Kafka is a hard dependency for pipelines. Direct model calls do not touch it; pipelines cannot work without it.
• Latency is higher than an in-process chain. Each hop is a produce and a consume.
• Retention and partition count are a genuine cloud bill, driven by your payload sizes.

Joins, triggers and conditional routing

Multi-input steps need a rule for what to do when inputs arrive at different times, and Core 2 makes you choose it explicitly. The join types are:

inner is the safe default and the one that hangs when an upstream step silently stops producing. outer with a joinWindowMs is how you bound that: the pipeline degrades to a partial input rather than stalling, which is the right trade when the missing input is a feature lookup from a system like Feast that may be slow. any is a fan-in for alternatives, not for duplicates — two arrivals both proceed.

Triggers are the other control you get. A trigger makes a step wait for another step to complete without consuming its data. That separation of "ordering" from "dataflow" is what lets you gate a model behind a check: an outlier detector produces nothing the predictor needs, but you may want the predictor to run only after it has. triggersJoinType takes the same inner, outer and any semantics for multiple triggers.

Conditional routing falls out of the tensor-level wiring rather than being a separate feature. A routing step emits different named tensors depending on its decision, and each downstream branch listens for one specific name. A branch whose tensor never appears simply never runs. This is how you express "cheap model for ordinary requests, expensive model for the hard ones" without any imperative branching logic.

Two further capabilities belong on your mental map. externalInputs and externalTriggers reference another pipeline's inputs, outputs or a specific step, using the form <pipelineName>.(inputs|outputs|step).<reference>, which lets you compose pipelines across team boundaries instead of building one unmaintainable graph. And since 2.9.1, cyclic pipelines with bounded iterations are supported, which is the shape an agentic or iterative-refinement flow needs; the bound is what keeps it from being an infinite loop in production.

Experiments and progressive rollout

An Experiment is the custom resource for splitting traffic between models or between pipelines: A/B tests, canaries and shadow comparisons. It lists candidates with weights and applies to both HTTP and gRPC calls, so the split happens at the mesh rather than in your client.

The operational value is less about statistics than about rollout safety. Deploying a new model version in Core 2 is cheap — it is another Model on an existing server, not a new deployment — so a canary is a weight change rather than an infrastructure change. The discipline that makes it worth doing is deciding the comparison before shifting traffic: which metric, measured over how many requests, and what value sends you back. Without that, a traffic split is just two models in production and a guess.

That command is worth generalising. Because the API group is v1alpha1, kubectl explain on your own cluster is the most reliable field reference you have:

kubectl explain model.spec
kubectl explain server.spec
kubectl explain pipeline.spec.steps
kubectl explain experiment.spec

Artifact storage and credentials

storageUri is an rclone-style URI, which is why it accepts gs://, s3:// and local PVC paths with the same syntax. The agent sidecar is the component that resolves it, so credentials are the agent's concern, not the inference server's.

Credentials are supplied as Kubernetes Secrets holding rclone configuration, either referenced from the model or configured once for the server's agent. The field names here are an area the docs have moved around, so confirm with kubectl explain model.spec on your cluster rather than copying a field from a blog post. What does not change is the discipline around them:

• Credentials never appear in a manifest or in a Helm values file that is committed. If a storageUri works and no Secret exists, something is injecting credentials from the node's identity, and you should know which thing that is.
• Prefer workload identity or an instance role over long-lived keys where your cloud supports it. For a cluster in a Gulf or Egyptian region, this also keeps the model artifacts and the credentials to read them inside the same account and region, which is usually the easier data-residency story to defend.
• An external secrets operator or Vault-backed injection is the normal production answer, so that key rotation is not a kubectl edit.
• Give each team or namespace its own credential scoped to its own bucket prefix. Core 2's open-source inference endpoint has no built-in user authentication, so bucket scoping is one of the few places you get real isolation for free.

The failure mode to recognise: a server pod in CrashLoopBackOff or a model that never becomes ready, with the real message in the agent container rather than the model's conditions. The agent is where "no such bucket", "permission denied" and TLS trust failures appear.

kubectl logs -n seldon-mesh <server-pod> -c agent --tail=100
kubectl logs -n seldon-mesh <server-pod> -c mlserver --tail=100

If image or artifact pulls fail with x509: certificate signed by unknown authority, the cause is usually a proxy or TLS-inspecting gateway between the node and the registry, not Seldon. Add the inspecting CA to the node's trust store; do not turn verification off.

Scaling: HPA, KEDA, and what changed in 2.10

This is the part of Core 2 that has moved most recently, and getting it wrong is the difference between a platform that absorbs a traffic spike and one that reports ScheduleFailed during it.

Native model autoscaling was disabled in 2.10.0, pending wider fixes. The official advice is to autoscale servers with HPA or KEDA instead. If you are reading a 2.9-era tutorial that configures model-level autoscaling, it does not apply to a 2.10 cluster.

The HPA pattern the docs prescribe for single-model serving has three constraints that are easy to miss and expensive to discover:

1. Only custom Prometheus metrics are supported. Native CPU and memory targets cannot be used, because two HPAs would end up controlling the same pods. You need a metrics adapter — the Prometheus adapter or KEDA — to expose the metric to the autoscaling API.
2. You apply two HPAs together, one with scaleTargetRef.kind: Model and one targeting the Server, using the same metric, the same minReplicas and maxReplicas, and the same behaviour policies. The documented example is an Object metric infer_rps whose describedObject is the Model, with a target AverageValue of 3.
3. This pattern requires a one-to-one mapping between models and servers, so it gives up the multi-model-serving benefit. That is the real trade: pack many models per server and scale manually, or scale automatically and accept one server per model.

The ordering constraint from the control plane reappears here. The scheduler does not create server replicas, so if the model HPA scales before the server HPA has delivered pods, the model reports ModelReady: False with ScheduleFailed while existing replicas keep serving. That is why the two HPAs share policies: you are trying to keep model demand and server supply moving together.

The docs' scaling best practices are worth restating as rules:

• Do not configure a scale-up that asks for more replicas than your cluster can actually produce in the window — if nodes take two minutes to join, a thirty-second target is a promise the autoscaler cannot keep.
• Tune stabilisation windows deliberately. Inference traffic is spiky, and an eager scale-down followed by a cold reload of every model is worse than holding a replica.
• Remember that every new replica reloads its assigned models from the artifact store. Scale-up latency is model-load latency, not pod-start latency.

Pipeline components scale on a different axis. Since 2.10, the dataflow engine, pipeline gateway and model gateway can scale horizontally beyond the Kafka partition count, controlled by the maxShardCountMultiplier setting in SeldonConfig. Before that, partitions were the ceiling on pipeline throughput. If you install with Helm, an upgrade to 2.10 fills this in from your existing values; if you install without Helm, you must set it manually.

Observability, and the audit trail you already have

Core 2 gives you two distinct kinds of monitoring, and conflating them is why teams end up with dashboards nobody trusts.

Operational monitoring is Prometheus metrics from the Seldon components and inference servers: request rates, latencies, error rates, and the model-level infer_rps that the HPA pattern depends on. The install flow puts a Prometheus stack in a seldon-monitoring namespace, and Grafana dashboards are published alongside. The four signals worth alerting on:

• Request rate and latency per model, at a percentile rather than a mean — an inference service's mean hides the batch that timed out.
• Error rate at the mesh, split from error rate at the server, so you can tell a routing problem from a model problem.
• Model readiness, as a count of models not in ModelReady state. This is the alert that catches ScheduleFailed before a user does.
• Server replica memory, against the sum of declared memory for models placed there. A growing gap is your bin-packing assumptions drifting from reality.

Data-science monitoring is the other half, and in Core 2 it is not a separate product: drift detectors, outlier detectors and explainers are deployed as ordinary Model resources — that is what the alibi-detect and alibi-explain capabilities on the default MLServer config are for — and placed as steps inside a pipeline. A drift detector is a model that consumes the same tensor the predictor consumes and emits a score. Because it is a pipeline step, its outputs land on a Kafka topic like any other, so drift becomes queryable history rather than a log line. If you also run Evidently or Alibi Detect outside the serving path, the Core 2 version's advantage is that it sees exactly the payload the model saw, with no sampling or reconstruction step in between.

OpenTelemetry tracing is configurable in the runtime configuration, which is what lets a pipeline request appear as one trace across the gateway, the Kafka hops and each model. Check the current configuration keys in the docs for your version before wiring it, since this is configured through SeldonConfig rather than per-resource annotations.

The Kafka topics deserve naming as an observability tool in their own right. seldon pipeline inspect <name> reads them directly, which means you can see the actual tensors that flowed between two steps, after the fact, without adding any logging to any model. There is no equivalent in a request-response serving stack: the data is simply not kept.

The latency budget, and where it actually goes

Performance work on Core 2 goes wrong when people tune the model and ignore the path. Before changing anything, write down the budget for a request and attribute each part of it, because the levers are different at every hop.

For a direct model call the path is client, Envoy, the inference server process, the model. Envoy's contribution is small and roughly constant. The inference server's contribution is where the interesting choices live: how many worker processes it runs, how it batches concurrent requests, and whether the model releases the interpreter while it computes. MLServer's parallel-worker configuration is the main dial here — more workers absorb concurrency at the cost of memory per replica, which immediately interacts with your declared memory budget and therefore with how many models the scheduler will pack alongside. It is worth knowing that this configuration was itself corrected in 2.10.2, so a cluster on 2.10.0 or 2.10.1 may not be honouring what you set.

For a pipeline, add a produce and a consume for every hop, plus the time the dataflow engine spends in the Kafka Streams topology. This cost is per step and essentially fixed, which is why the useful optimisation for a latency-bound pipeline is usually to remove a step rather than to speed one up. Merging a trivial preprocessor into the predicting model removes two Kafka round trips in exchange for slightly less reusable code; for a service with a tight latency objective that is often the right trade, and it is a decision to make deliberately rather than discover.

Three structural levers are worth more than micro-tuning:

• Payload size. OIP tensors are JSON by default over REST, and large float arrays are expensive to serialise at both ends and expensive to store in Kafka retention. gRPC avoids some of that cost, and sending the smallest tensor a step actually needs — using the dot-notation wiring to pull one named tensor rather than a whole upstream output — avoids more.
• Batch shape. An inference server's throughput per replica usually improves with batching and its tail latency usually gets worse. Decide which of the two your service level cares about before tuning, and measure at a percentile, since a mean latency hides exactly the requests your users complain about.
• Cold load. A request that arrives while a replica is still fetching an artifact is not slow for any reason you can tune. Measure the time from pod deletion to all its models ready, and treat that number as the floor on any autoscaling or rollout plan.

Testing and CI/CD for Seldon resources

Seldon resources are Kubernetes manifests, so the whole GitOps toolchain applies, and the question is only what to assert at each stage. A pipeline that catches real problems looks like this, in increasing cost:

Static validation. Lint the YAML, then validate against the cluster's own schemas with a dry run. Because the API group is v1alpha1, a client-side schema copy goes stale; --dry-run=server asks the real CRD:

kubectl apply --dry-run=server -f manifests/ -n seldon-mesh

Capability consistency. This is the check nobody writes and everybody needs: for each Model, assert that some Server in the target namespace advertises every tag in requirements. It is a small script over the same manifests, and it turns the most common production failure into a failed pull request. The same script should assert that memory is set to something other than the copied-in 100Ki.

Deploy and wait. In a staging namespace, apply and block on readiness, with a timeout generous enough for the artifact to download:

kubectl apply -f manifests/ -n seldon-staging
kubectl wait --for condition=ready --timeout=300s model income-classifier -n seldon-staging

Contract test. Send one known payload and assert on the response shape and a known-good score. Because the interface is OIP, this test is a plain HTTP call and does not need any Seldon client library:

curl -fsS -X POST http://localhost:8080/v2/models/income-classifier/infer \
  -H "Content-Type: application/json" \
  -H "Seldon-Model: income-classifier" \
  -d @test/fixtures/income.json

Pipeline test. Same call with the .pipeline suffix in the header, then seldon pipeline inspect to assert the intermediate steps ran, not just that a response came back. A pipeline can return a plausible answer while a middle step silently produced defaults.

For delivery itself, treat the four Helm charts as infrastructure owned by the platform and the Model, Pipeline and Experiment resources as application artifacts owned by the teams. The split matters because their release cadences differ by an order of magnitude: charts change at upgrades, models change weekly. Argo CD or Flux syncing a repository of model manifests gives you the audit trail of who deployed which storageUri, which is the question an auditor actually asks. Keep the storageUri pointing at an immutable, versioned path in the bucket; a mutable latest/ path makes the git history a lie, because the manifest no longer determines what runs.

Where a model is produced by a training pipeline — MLflow registering a run, or a Kubeflow job writing to a bucket — the clean handoff is for training to publish a versioned artifact path and for CI to open a pull request changing storageUri. Resist having the training job apply the Model resource directly: you lose review, and you lose the ability to roll back by reverting a commit.

Debugging beyond the basics

A routine beats inspiration here, because Core 2's failures are distributed across the control plane and the data plane and each layer tells you a different part of the story. Work outward:

kubectl get model,server,pipeline -n seldon-mesh
kubectl describe model <name> -n seldon-mesh
kubectl get servers -n seldon-mesh
kubectl logs -n seldon-mesh <server-pod> -c agent
kubectl logs -n seldon-mesh <server-pod> -c mlserver

The first command answers "is this a placement problem or a traffic problem". The describe gives you the condition and reason — ModelReady and, when it fails, ScheduleFailed. The server listing tells you what capacity and capabilities exist to satisfy the model. The agent log covers artifact fetching; the runtime log covers loading and inference.

Mapping symptoms to layers:

Two of those rows are version bugs rather than configuration errors, which is itself a lesson: on a v1alpha1 API, "check the release notes for the exact patch you are running" is a debugging step, not an afterthought. A cluster on 2.10.0 with pipelines that intermittently fail to create is not misconfigured; it is out of date.

Configuration management and upgrades

Core 2's configuration lives in two custom resources, and knowing which is which saves you from editing the wrong thing.

SeldonRuntime installs the per-namespace runtime: scheduler, Envoy, the model and pipeline gateways, the dataflow engine. SeldonConfig holds the configuration those components read — Kafka connection and security, tracing, and since 2.10 a scaling configuration block including maxShardCountMultiplier. The stated purpose of centralising this in 2.10 was to make Core 2 reconfigurable after the cluster is deployed, rather than requiring a reinstall.

The install is four Helm charts, and the order is the whole lesson:

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
  --namespace default --install

helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh --set controller.clusterwide=true --install

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
  --namespace seldon-mesh --install

CRDs come first and live in their own chart because Helm does not upgrade CRDs packaged inside another chart's subdirectory. Then the operator, then the runtime, then the default servers. Upgrades follow the same order; uninstalls go in reverse.

Operator scope is a decision you make once and regret changing. --set controller.clusterwide=true gives one operator watching everything, which is simple and correct for a single team. --set "controller.watchNamespaces={ns1,ns2}" restricts it, which is what you want when separate teams must not see each other's resources. Pair the namespaced form with per-namespace SeldonRuntime instances, Kubernetes RBAC, NetworkPolicies and separate Kafka credentials per team, and you have workable multi-tenancy.

The upgrade habits that matter, in order of how much pain they save:

1. Pin chart versions with --version in whatever your GitOps repository holds. An unpinned helm upgrade is a release you did not choose.
2. Read the release notes every time. The 2.9 to 2.10 CRD changes were backward compatible with existing resources, and 2.10's notes say to set the new scaling configuration before upgrading to 2.11. That is the kind of instruction that only exists in the notes.
3. Upgrade patches promptly on this line. 2.10.1 and 2.10.2 fixed scheduling, pipeline lifecycle and gRPC streaming bugs. Sitting on 2.10.0 means debugging problems that are already solved.
4. Treat server image upgrades as model migrations, using the capability-tag pattern, rather than as a chart value change.
5. Read LICENSE in the repository before production use. Seldon's licensing for Core has changed, and a commercial licence may be required. That is a question for whoever signs contracts, not one to answer from a guide.

Putting it all together

Here is the shape of a small but real deployment that exercises everything above: an income classifier with a preprocessor and a drift detector, served to an internal application, on a cluster you can defend in a review.

Servers. One Server named for its runtime contract, with capabilities: [income-deps] and a pinned seldonio/mlserver:1.6.0 image, sized so that the sum of the declared memory of the models you intend to place on it fits comfortably inside the container's memory limit. Three replicas, so a node drain costs redundancy rather than availability.

Models. Three Model resources — preprocessor, classifier, drift detector — each requiring income-deps, each with storageUri pointing at an immutable versioned path in a bucket in your own region, and each with memory set to a measured value. The drift detector requires the alibi-detect capability too, so either that tag joins the server's capabilities list or the detector gets its own server.

Pipeline. A Pipeline with the preprocessor feeding both the classifier and the detector, the classifier as output.steps, and a joinWindowMs-bounded outer join anywhere an optional enrichment arrives. The detector is deliberately not on the response path: it consumes the same tensors and writes to its topic, so drift monitoring cannot add latency to or fail a prediction.

Traffic. Callers hit seldon-mesh with Seldon-Model: income-classifier-app.pipeline. Nothing is exposed publicly; an ingress or service mesh in front handles authentication, because the open-source inference endpoint does not.

Scaling. Server replicas under KEDA or HPA on infer_rps with a stabilisation window tuned to your artifact load time. If you have taken the one-model-per-server trade to get autoscaling, both HPAs ship together with matched policies.

Observability. Prometheus for rates, latency percentiles and a model-readiness count; the Kafka topics for payload-level audit; the drift detector's topic as the data-quality signal. One alert on models not ready, one on error rate at the mesh, one on server memory against declared budget.

Delivery. The four charts pinned and applied by the platform team. The Model and Pipeline manifests in a repository synced by Argo CD, with CI running a server-side dry run, the capability-consistency check, a staging deploy with kubectl wait, and one contract request per model and per pipeline.

Rollout. A new classifier version arrives as a second Model on the same server; an Experiment shifts a small weight to it; the metric and the sample size were agreed before the weight changed; the old model stays loaded until the comparison is done, because unloading it is a one-line revert.

None of those choices is exotic. What makes them work together is that each one respects a mechanism from earlier in this guide: capabilities decide placement, declared memory decides packing, Kafka decides what you can audit, and the scheduler never creates capacity for you.

Where you are now

You can predict Core 2's behaviour rather than react to it. You know that applying a Model starts a placement decision rather than a deployment, that ScheduleFailed is a supply-and-demand statement you can read off kubectl get servers, and that the agent sidecar is where artifact problems announce themselves. You know memory is a scheduler budget whose relationship to reality is entirely your responsibility, and what happens when that relationship breaks. You know pipelines are a Kafka topology with an HTTP front door, which is why they give you audit and replay and why they cost latency and retention. You know that native model autoscaling is off in 2.10 and what the two-HPA pattern costs in multi-model serving. And you know the upgrade order, why CRDs are a separate chart, and what deleting them does.

What the senior level adds is ownership. How the architecture fails under real load and which component gives first; what the scheduler's single logical instance means for a multi-tenant platform; the trust model, since there is no authentication on the inference endpoint; cost modelling where Kafka retention and GPU packing dominate; migrations between runtime versions and between serving stacks; the governance and incident playbooks that let other teams deploy models without you in the loop. If you are weighing Core 2 against alternatives, the comparison worth preparing is with KServe — both speak the Open Inference Protocol, and the real differences are data-centric Kafka pipelines and multi-model servers on one side against a serverless-style one-service-per-model design on the other — and with BentoML or Ray Serve when the problem is packaging and Python-level composition rather than Kubernetes-level orchestration.

Sources

• Seldon Core 2 documentation
• Concepts
• Architecture
• Core features
• Quickstart
• Models
• Model scheduling
• Servers
• Pipelines
• Single-model serving with HPA
• Operational monitoring
• Performance tuning
• Pipeline advanced configuration
• Production environment installation
• Upgrading
• Custom servers example
• Releases on GitHub

At this level interviewers stop asking what Seldon Core is and start asking what you did when it misbehaved. The questions below are the ones that actually come up for a mid-level MLOps or platform role, with the reasoning a strong answer shows.

One framing point to establish in your first sentence, every time: you mean Seldon Core 2, with Model, Server, Pipeline and Experiment resources under mlops.seldon.io/v1alpha1. Saying so signals that you know Core 1 and its SeldonDeployment resource are a separate, maintenance-only architecture. Candidates who blur the two lose credibility in the first minute, because every follow-up question then has an ambiguous answer.

Scenario: a model has been "not ready" for twenty minutes

The honest first move is to say what you would not do: restart the server StatefulSet. Nothing in a Core 2 model is a pod, so there is nothing to restart into a better state.

Then the routine. kubectl describe model <name> to read the condition and its reason. If the reason is ScheduleFailed, this is a placement problem and there are exactly three candidate causes: no server advertises every tag in the model's requirements, no server has as many replicas as the model requested, or no replica has enough free declared memory for the model's memory. kubectl get servers tells you the supply side, and comparing the two lists resolves it in under a minute. A capability typo is the single most common answer.

If the model is not ScheduleFailed but simply never becomes ready, placement succeeded and the agent sidecar is working on it. That means artifact download or model load, so kubectl logs <server-pod> -c agent for fetch errors and -c mlserver for load errors. A large artifact pulled cross-region is slow on every replica restart, because the agent's copy is not durable — so a kubectl wait timeout can mean "too impatient" rather than "broken".

What separates a strong answer: naming the layer boundary explicitly. The scheduler decides placement, the agent moves data, the runtime loads the model, and each one logs somewhere different. A weak answer lists commands without that map and so cannot explain why it ran them in that order.

Scenario: traffic doubled and latency went up, not throughput

Start by asking whether the calls are to models or pipelines, because the answer changes everything.

For direct model calls, work through the chain. Are requests reaching more than one server replica, or is the model placed on fewer replicas than you think? Partial scheduling means a model can be ModelReady while running on fewer replicas than it requested, so redundancy can be quietly lower than the manifest claims. If you are on a version before 2.10.1, a known scheduling regression could have left it there permanently after a pod restart.

Next, is the server itself saturated or is it thrashing? Multi-model serving means several models share a replica, so a noisy neighbour model is a real cause. Check server replica memory against the sum of the declared memory of the models placed there. If the real figure is much higher, your declared budgets are fiction and the kernel will eventually resolve the disagreement with an OOM kill and a reload storm.

For pipelines, add the Kafka hops. Every step is a produce and a consume, so a four-step pipeline has a floor you cannot optimise away in the models. Before 2.10, pipeline throughput was capped by the Kafka partition count; from 2.10 the dataflow engine, pipeline gateway and model gateway can scale past it via maxShardCountMultiplier in SeldonConfig. The question to ask is whether this workload needed a pipeline at all — a single-step pipeline should be a direct model call.

A strong answer ends with the capacity conclusion rather than a fix: scale-up latency in Core 2 is dominated by model load time, not pod start time, so if your traffic doubles in thirty seconds and your artifacts take ninety to download, no autoscaler configuration saves you. The fix is headroom.

Scenario: you need to autoscale, and your tutorial is out of date

This is a current-events question, and it is a good one, because the answer changed recently.

Native model autoscaling was disabled in 2.10.0 pending wider fixes. The official guidance is to autoscale servers with HPA or KEDA. So the first thing a strong candidate says is "which version are we on", because a 2.9-era configuration does not apply.

Then the three constraints of the documented HPA pattern. Only custom Prometheus metrics work, because CPU and memory targets would mean two HPAs controlling the same pods — so you need a metrics adapter, the Prometheus adapter or KEDA. You apply two HPAs together, one targeting the Model and one the Server, with the same metric, the same minReplicas and maxReplicas, and the same policies; the documented example uses an Object metric infer_rps described by the Model with a target average value of 3. And the pattern needs a one-to-one model-to-server mapping, which means giving up multi-model serving.

That last point is where you show judgement. The trade is explicit: pack many models onto shared replicas and scale by hand with generous headroom, or autoscale and run one server per model. For forty small models with uneven traffic, packing wins on cost by a wide margin. For three large models that each saturate a node, there is nothing to pack and autoscaling is free. Say which situation you would choose which for.

The ordering trap is the detail that marks experience: the scheduler never creates server replicas. If the model HPA fires first, the model reports ModelReady: False with ScheduleFailed while its existing replicas keep serving. Nothing breaks, but your scale-up did not happen, and the alert you get names the wrong problem.

Scenario: the pipeline returns a plausible answer that is wrong

The useful observation here is that Core 2 gives you a tool nothing else in the serving category does: every pipeline step's inputs and outputs are real Kafka topics with real retention, so you can read exactly what each step saw for the request in question, after the fact, without having added any logging. seldon pipeline inspect <name> is the command.

From there the causes are structural rather than mysterious. A tensorMap renaming the wrong tensor produces valid-looking garbage. A step wired to the pipeline's original inputs when it should read the previous step's outputs skips a transformation silently. An outer join whose joinWindowMs expired proceeded with a missing input, which is exactly what you configured it to do — the model then scored on defaults. And a conditional route whose expected tensor name never appeared means a branch simply did not run, with no error anywhere.

The strong answer names the join semantics precisely, because they are the usual culprit: inner waits for every input and will hang if one stops arriving, outer waits up to a window and then proceeds without it, any proceeds on the first arrival. Choosing outer is choosing silent degradation over a stall, which is often correct and must be monitored. A weak answer says "check the logs", which for a Kafka topology tells you much less than the topics do.

Scenario: an upgrade broke something, or fixed nothing

Interviewers like this one because it tests whether you read release notes.

The frame: Core 2 is on a v1alpha1 API, and recent patches have fixed real behavioural bugs. 2.10.1 fixed a partial-scheduling regression that had existed since 2.9.0 and left models on fewer replicas than requested after pod restarts. 2.10.2 fixed pipelines failing to create or delete, gRPC stream blocking, operator reconciliation problems and MLServer parallel-workers configuration. If someone describes pipelines that intermittently will not delete on 2.10.0, the answer is the upgrade, not a configuration change.

On the mechanics, the order is the content of the answer: CRDs chart first, then setup (the operator), then runtime, then servers. CRDs live in a separate chart because Helm does not upgrade CRDs packaged in another chart's subdirectory. Uninstall reverses that order, and deleting the CRD chart while resources exist cascades — every Model, Server and Pipeline in every namespace goes with it.

Two specifics that read as real experience. Going 2.9 to 2.10, the CRD changes were backward compatible with existing resources, and with Helm the existing values fill in the new options; without Helm you must set maxShardCountMultiplier by hand. And 2.10's notes say to set the new scaling configuration before upgrading to 2.11 — the kind of instruction that exists only in the notes, which is the argument for reading them.

Trade-off: pipeline or direct model call

Say the default first: if a request goes to one model and returns, call the model directly. Lower latency, no Kafka dependency, fewer components that can fail.

Then what a pipeline buys. Multiple steps composed as a DAG. An auditable record of every intermediate tensor, which for a credit or fraud decision under a financial regulator is a compliance capability rather than a convenience. Replay of historical traffic through a changed graph. Drift and outlier detectors running beside the predictor as ordinary Model resources — that is what the alibi-detect capability on the default MLServer config is for — rather than as a separate monitoring stack reconstructing payloads from samples.

And what it costs. Kafka becomes a hard dependency; direct model calls do not need it at all. Every hop adds a produce and a consume. Retention and partition count are a real cloud bill driven by your payload sizes. A strong answer puts a number on the shape of the decision even without exact figures: a four-step pipeline on sub-10ms models is mostly transport, and if your service-level objective is tight you should compose in one model instead.

The nuance that impresses: you can keep detectors off the response path. Wire the detector from the preprocessor's output but leave it out of output.steps, so drift monitoring cannot add latency to a prediction or fail one.

Trade-off: Seldon Core 2 or KServe

Keep this one factual, because interviewers have usually chosen one and will notice marketing language.

Both speak the Open Inference Protocol, so the client contract is portable and a migration does not rewrite callers. The architectural difference is the honest answer: Core 2 is data-centric, with Kafka-backed pipelines and multi-model servers that pack many models per replica. KServe is serverless-style, one InferenceService per model, with optional Knative for scale-to-zero.

From that, the selection criteria follow. Choose Core 2 when you need multi-step inference graphs with audit and replay, or when you have many small models whose per-pod cost dominates. Choose KServe when each model is independent, scale-to-zero matters because traffic is bursty or development-heavy, and you would rather not run Kafka. If you are asked about raw performance, decline to claim a winner without a benchmark on the actual workload — that answer is more credible than a number you cannot defend.

Trade-off: how much to trust memory

This is a quiet question that reveals a lot. spec.memory on a Model is a budget declared to the scheduler for bin-packing, not an enforced limit. Nothing stops the model using more.

The failure chain is worth reciting because it is so unobvious in production: a model declared at the docs' sample 100Ki that really needs hundreds of megabytes lets the scheduler pack far too many models onto a replica; the replica's real usage passes its container limit; the kernel OOM-kills it; the StatefulSet restarts it; the agent reloads every model assigned there; and the incident presents as a Kafka or network problem because no Seldon resource says "out of memory".

So the practice is: measure the real footprint by loading each model alone on a replica and reading container memory from Prometheus, declare that plus headroom for request-time buffers, keep the server container's own memory request consistent with the sum you intend to pack, and re-measure after a runtime image change. Memory-aware scheduling arrived in 2.9, which makes the declaration matter more, not less.

Scenario: a regulator asks what the model saw

This question is increasingly common for teams in banking and insurance, and it is where Core 2 answers better than most of its alternatives — provided you designed for it.

The mechanism is the answer. Every pipeline step's inputs and outputs are Kafka topics with configured retention, so the payload each step saw for a given request is recorded by construction, not by instrumentation someone remembered to add. seldon pipeline inspect reads those topics, and the same data is available to Kafka Connect or ksqlDB for longer-term storage; the 2.10 Schema Registry integration gives you the schema contracts for those topics, which is what makes downstream consumers maintainable rather than brittle.

The caveats are what make the answer credible. Retention is a configuration and a cost: if your topics keep three days and the question is about last quarter, the pipeline alone does not answer it, so you need a sink into durable storage. Direct model calls bypass Kafka entirely, so a model served without a pipeline has no such record. And the record proves what the model received, not which artifact produced the score — that link comes from your git history, which only holds if storageUri points at an immutable versioned path rather than a mutable latest/.

For a Gulf or Egyptian employer there is usually a residency dimension too, and the clean framing is that artifacts, Kafka topics and the credentials that read them all live in the same account and region as the cluster. Saying that unprompted tends to land well.

What a strong answer covers that a weak one misses

Across every question above, the same five things separate the two.

Version literacy. A weak answer describes Seldon in general. A strong one says which version the behaviour belongs to, and knows that native model autoscaling was disabled in 2.10.0, that streaming needs MLServer 1.6.0 or later, and that 2.10.1 and 2.10.2 fixed scheduling and pipeline bugs.

Layer discipline. A weak answer names commands. A strong one explains that the scheduler places, the agent fetches, the runtime loads and Envoy routes, and therefore which log holds the answer.

The declaration-versus-enforcement distinction. requirements and memory are declarations the scheduler reasons about, not runtime guarantees. Most surprising Core 2 behaviour is a gap between a declaration and reality.

Honest trade-offs. Multi-model serving and HPA autoscaling genuinely conflict in the documented pattern. Kafka pipelines genuinely cost latency and retention. A candidate who presents Core 2 as free of trade-offs has not run it.

Knowing what to verify. The API group is v1alpha1, field names move, and old documentation paths return 404. Saying "I would check kubectl explain model.spec on the target cluster rather than trust an example" is a better answer than a confidently wrong field name.

Sixty-second self-test

Answer these out loud before an interview. If any one stalls you, go back to the section it came from.

1. What happens, component by component, between kubectl apply -f model.yaml and ModelReady: True?
2. Your model shows ScheduleFailed. Name the three possible causes and the command that distinguishes them.
3. What is the difference between capabilities and extraCapabilities on a Server, and which wins if both are set?
4. Why does a pipeline call need .pipeline in the header but not in the URL path?
5. Name the three join types and say what each does when one input never arrives.
6. Why can you not autoscale a Seldon server on CPU with the documented HPA pattern, and what do you use instead?
7. What does spec.memory actually do, and what is the failure chain when it is understated?
8. In what order do you upgrade the four Helm charts, and what happens if you delete the CRDs chart first?
9. Which recent patch releases fixed scheduling, pipeline lifecycle and gRPC streaming bugs?
10. Give one reason to choose Core 2 over KServe and one reason to choose KServe over Core 2.

These are the Seldon Core 2 mistakes that survive past the beginner stage, the practices that pay back the most for the effort, and a set of recipes to keep beside the terminal. Everything here assumes Core 2 on Kubernetes 1.27 or later.

The errors that cost the most time

Copying memory: 100Ki from the docs. The sample value belongs to a toy iris model. Because memory is a scheduler budget rather than an enforced limit, understating it makes the scheduler pack many more models onto a replica than will fit. The symptom is not an error message; it is a server replica OOM-killed by the kernel, restarted, reloading every model assigned to it, and an incident that looks like Kafka or networking because no Seldon resource mentions memory. Measure each model alone, declare the measured figure plus headroom for request-time buffers, and keep the server container's own memory consistent with the sum you intend to pack.

Treating a capability typo as a Seldon bug. requirements must be a subset of the server's capabilities, and the match is exact string equality. sklean for sklearn gives you ScheduleFailed and no hint about which tag failed. Before filing anything, diff the two lists yourself with kubectl get servers and kubectl describe model.

Setting both capabilities and extraCapabilities. capabilities replaces the server's defaults; extraCapabilities extends them; if both are present, capabilities wins and your extra tags are silently absent. This produces the most confusing version of the previous bug, because the tag you added is in the manifest and not in the server.

Forgetting .pipeline in the Seldon-Model header. The URL path uses the plain name, the header needs the .pipeline suffix. The resulting not-found error reads as though the pipeline does not exist, so people go and check the resource, re-apply it, restart things, and lose an hour to a header.

Scaling a model and expecting server capacity to appear. The scheduler places models; it never creates server replicas. Raise a model's replica count past the server's and you get ModelReady: False with ScheduleFailed while the existing replicas keep serving. Nothing is broken and nothing scaled.

Following Core 1 instructions. Most Seldon content online predates Core 2 and uses SeldonDeployment and the Python wrapper library. The tell-tale error is no matches for kind "SeldonDeployment". The 1.x line is still released — v1.19.0 is maintenance, security hardening and a move to Python 3.12 — so you can install it by accident and get a cluster where none of your Core 2 manifests apply.

Configuring native model autoscaling. It was disabled in 2.10.0 pending wider fixes. Any tutorial that sets it up is pre-2.10. Autoscale servers with HPA or KEDA instead.

Reading the model's conditions and nothing else. Artifact fetch failures live in the agent sidecar's log, and load failures live in the runtime container's. A model stuck not-ready with no condition reason is almost always one of those two, and kubectl describe model will never tell you which.

Letting storageUri point at a mutable path. A latest/ prefix in the bucket means the manifest no longer determines what runs, your git history is no longer an audit trail, and a replica restart can load a different model than its neighbour. Use immutable versioned paths.

Upgrading the MLServer image as if it were a chart value. Changing the server image re-loads every model placed on it under a runtime they were never tested against. A scikit-learn artifact can load cleanly under a different library version and score differently. Treat it as a migration, with the capability-tag pattern below.

Deleting the CRDs chart to "clean up". CRD deletion cascades to every Model, Server and Pipeline in every namespace, without confirmation. Uninstall in reverse install order: servers, runtime, setup, CRDs.

Assuming a pipeline that returns 200 ran correctly. A middle step can be skipped entirely — a conditional branch whose tensor name never appeared, or an outer join whose window expired — and the pipeline will still answer with something plausible built from defaults. A contract test that only asserts on the final response passes happily. Assert on the intermediate topics too, or at least alert on a step's throughput falling to zero while the pipeline's stays flat.

Expecting streaming to work on an old runtime. Inference response streaming, over SSE and gRPC, needs MLServer 1.6.0 or later. Separately, gRPC stream blocking was a real bug fixed in 2.10.2, so a hanging stream on 2.10.0 is not your client.

Building one enormous pipeline across team boundaries. externalInputs and externalTriggers let one pipeline reference another's inputs, outputs or a specific step, which is how two teams compose without sharing one unmaintainable graph and one blast radius. Reach for composition before you add a tenth step.

Sitting on a minor release and debugging fixed bugs. 2.10.1 fixed a partial-scheduling regression present since 2.9.0 in which models ended up on fewer replicas than requested after pod restarts. 2.10.2 fixed pipelines failing to create or delete, gRPC stream blocking, operator reconciliation problems and MLServer parallel-workers configuration. On a v1alpha1 API, checking the patch notes is a debugging step.

Practices by leverage

1. Encode the runtime contract in the capability tag. This is the highest-value habit in Core 2 because it converts an invisible coupling into a declared one. A server running seldonio/mlserver:1.6.0 advertises a tag naming that contract; models whose artifacts were built against it require that tag. Introducing a new runtime means adding a server with a new tag and migrating models one requirements edit at a time, each independently revertible. Without it, every image bump is an untested mass migration.

2. Measure and declare memory honestly, once per model. Half an hour of measurement removes an entire class of restart-storm incidents, and it is the only way the scheduler's bin-packing can be trusted.

3. Pin every image and every chart version. --version on each helm upgrade, an explicit tag in Server.spec.podSpec, and an immutable versioned storageUri. Three pins, and your cluster becomes a function of your repository.

4. Default to direct model calls; use a pipeline when you need what it gives. Multiple steps, audit, replay, or detectors beside the predictor. A single-step pipeline is Kafka latency and a Kafka dependency bought for nothing.

5. Keep detectors off the response path. Wire a drift or outlier detector from the preprocessor's output and leave it out of output.steps. Monitoring then cannot add latency to or fail a prediction, and its results still land on a Kafka topic as queryable history.

6. Add a capability-consistency check to CI. For each Model, assert that some Server in the target namespace advertises every tag in requirements, and that memory is not the sample value. A short script turns the most common production failure into a failed pull request.

7. Alert on model readiness as a count. One alert on "models not in ModelReady" catches ScheduleFailed, capability drift and capacity shortfalls before a user does. Pair it with error rate split between the mesh and the servers, so routing problems and model problems are distinguishable.

8. Keep platform and application release cadences separate. The four Helm charts belong to the platform team and change at upgrades. Model, Pipeline and Experiment resources belong to the model teams and change weekly, synced from git by Argo CD or Flux. Mixing them means every model deployment is a platform change review.

9. Decide the metric before you shift traffic. An Experiment makes a weight change trivial, which is exactly why it is easy to run a split with no decision rule and learn nothing. Agree the metric, the sample size and the rollback threshold in the pull request that creates the experiment.

10. Use scoped credentials per team. The open-source inference endpoint has no built-in user authentication, so put an ingress or service mesh in front of seldon-mesh and give each namespace its own artifact-store credential scoped to its own bucket prefix. For a cluster in a Gulf or Egyptian region, keeping artifacts and the credentials that read them in the same account and region is also the simplest data-residency position to defend.

11. Write down the latency budget before tuning anything. Attribute a request across Envoy, the inference server, the model, and — for pipelines — one produce and one consume per hop. The Kafka hops are roughly fixed per step, so a latency-bound pipeline usually improves by losing a step rather than by speeding one up. Measure at a percentile; a mean hides the requests users complain about. If tracing is configured in the runtime, one pipeline request is one trace across every hop, which answers "who owns the latency" in seconds.

12. Know your cold-load time as a number. Delete a server pod, time how long until its models are ready, and write it on the runbook. That figure is the floor under every autoscaling stabilisation window, rollout plan and kubectl wait timeout you will ever set, and it changes when artifacts grow or the bucket moves region.

Practice cards

Work through these on a kind cluster. Each one builds a reflex you will otherwise build during an incident.

Operational recipes

Triage a not-ready model.

kubectl get model,server,pipeline -n seldon-mesh
kubectl describe model <name> -n seldon-mesh
kubectl get servers -n seldon-mesh
kubectl logs -n seldon-mesh <server-pod> -c agent --tail=100
kubectl logs -n seldon-mesh <server-pod> -c mlserver --tail=100

Placement first, then artifact fetch, then load. The first command tells you which of the three you are in.

Confirm a call reaches the right thing. Models and pipelines share the URL shape and differ only in the header suffix.

kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80

# model
curl -fsS -X POST http://localhost:8080/v2/models/iris/infer \
  -H "Content-Type: application/json" -H "Seldon-Model: iris" \
  -d '{"inputs":[{"name":"predict","shape":[1,4],"datatype":"FP32","data":[[1,2,3,4]]}]}'

# pipeline — note the .pipeline suffix in the header only
curl -fsS -X POST http://localhost:8080/v2/models/income-classifier-app/infer \
  -H "Content-Type: application/json" -H "Seldon-Model: income-classifier-app.pipeline" \
  -d '{"inputs":[{"name":"income","datatype":"INT64","shape":[1,12],"data":[53,4,0,2,8,4,2,0,0,0,60,9]}]}'

A staging gate for CI. Server-side dry run, deploy, wait, then one contract request per model and per pipeline.

kubectl apply --dry-run=server -f manifests/ -n seldon-staging
kubectl apply -f manifests/ -n seldon-staging
kubectl wait --for condition=ready --timeout=300s model income-classifier -n seldon-staging

Give the timeout room for the artifact download; a failing wait on a large model is often impatience rather than a fault.

Stand up an isolated server for one model family.

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-custom
spec:
  serverConfig: mlserver
  capabilities:
  - income-classifier-deps
  podSpec:
    containers:
    - image: seldonio/mlserver:1.6.0
      name: mlserver

capabilities replaces the defaults, so this server accepts only models that require that tag. That is isolation without a second cluster.

Upgrade the platform. Order is the whole recipe: CRDs, operator, runtime, servers.

helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
  --namespace default --install

helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh --set controller.clusterwide=true --install

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install

helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
  --namespace seldon-mesh --install

kubectl get pods -n seldon-mesh

Add --version to each. Read the release notes first: the 2.9-to-2.10 CRD changes were backward compatible, Helm fills new options from existing values, and without Helm you must set maxShardCountMultiplier yourself. Set the new scaling configuration before you go to 2.11.

Restrict the operator for multi-tenancy.

helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
  --namespace seldon-mesh \
  --set "controller.watchNamespaces={team-risk,team-fraud}" --install

Pair it with per-namespace runtimes, RBAC, NetworkPolicies and separate Kafka credentials and topics per team.

Inspect a pipeline's real dataflow during an incident.

seldon pipeline status income-classifier-app
seldon pipeline inspect income-classifier-app

The status tells you whether the graph is live; the inspect reads the step topics and shows the tensors that actually moved, which is the only way to tell a skipped step from a wrong one. Confirm the CLI's connection flags against its own documentation for your version before relying on it in a runbook, since the CLI talks to the scheduler directly rather than through the Kubernetes API.

The checklist before a Seldon change goes live

Run this over any pull request that touches a Model, Server or Pipeline.

• Every Model's requirements are a subset of some Server's capabilities in the target namespace, checked mechanically rather than by eye.
• memory is a measured value, not the sample 100Ki, and the sum of declared memory for a server's models fits inside its container memory limit.
• storageUri points at an immutable versioned path, and the credential for it comes from a Secret or an external secrets operator, never from the manifest.
• The server image is pinned to an explicit tag, and its capability tag reflects that runtime contract.
• Pipelines have output.steps if a caller expects a synchronous response, and every multi-input step has a deliberately chosen join type — with joinWindowMs set wherever the join is outer.
• No pipeline exists for a single model.
• Detectors are off the response path unless a prediction genuinely must fail when they do.
• Any autoscaling is on servers via HPA or KEDA with a custom Prometheus metric, both HPAs ship together with matched policies, and the stabilisation window respects your measured cold-load time.
• Chart versions are pinned, the install order is respected, and the release notes for the exact target patch have been read.
• A readiness alert and a mesh error-rate alert exist and have been seen to fire at least once in staging.
• Someone has read LICENSE in the Seldon repository and confirmed what your intended production use requires. Seldon's licensing for Core has changed from its original terms, and a commercial licence may apply — that is a question for whoever signs contracts, not one to settle from a guide.

This is part three of three. The earlier levels were about getting a model to answer a request and getting a pipeline to pass data between steps. This level is about the fact that other teams now depend on you for both, and that the failure modes you will actually be paged for are not the ones that appear in a quickstart.

Seldon Core 2 is unusual among serving stacks because its control plane is stateful in a way most Kubernetes operators are not, and because its pipeline layer has a hard dependency on Kafka. Those two facts generate most of the operational work. If you remember nothing else from this guide, remember that owning Seldon Core 2 as a platform means owning a scheduler's placement state and a Kafka cluster's health, not just a set of CRs.

Where this picks up

I start with architecture, because every scaling and security decision below follows from which plane a component sits in.

The architecture you now own

Seldon Core 2 splits cleanly into a control plane that decides where models live and a data plane that carries inference traffic. The split matters operationally because the two planes fail independently and have opposite availability characteristics: the data plane keeps serving when the control plane is down, and the control plane is useless when the data plane is saturated.

The operator is the only component whose scope you choose at install time. With controller.clusterwide=true it watches every namespace; with --set "controller.watchNamespaces={ns1,ns2}" it watches a fixed list. That single flag is the root of your multi-tenancy design, which is why it appears again further down.

The scheduler is the component seniors consistently under-weight. It is not a stateless admission webhook: it holds the mapping from models to server replicas, and it is the thing that decides your model is unschedulable. Lifecycle probes were added to all Core 2 components in 2.10.0 precisely because operators needed Kubernetes to notice when one of these processes had stopped being useful, and the 2.10.2 patch fixed operator reconciliation problems that had become more visible across the 2.10.x line.

The agents are where artifact fetching happens, which means they are where credential problems surface. When a model will not become ready and the server pod is in CrashLoopBackOff, the agent container's log is usually the shortest path to the answer, because rclone failures against storageUri print there rather than in the inference server's log.

The scheduler is the component that breaks first

Capacity planning conversations about serving platforms usually jump to GPUs. In Seldon Core 2 the first thing that breaks under organisational load is almost never compute — it is placement. The scheduler filters candidate servers on three conditions: every entry in the model's requirements must appear in the server's capabilities, the server must have at least as many replicas as the model's replicas, and each replica must have enough free declared memory for the model's memory. If no server satisfies all three, the model reports ModelReady: False with reason ScheduleFailed.

Each of those three conditions fails for a different organisational reason. Capability mismatches are a naming problem: a team writes sklearn where the platform standard is a pinned capability like income-classifier-deps, or someone adds a capability to a Server with capabilities rather than extraCapabilities and silently drops the whole default set, because capabilities overrides the defaults while extraCapabilities extends them. Replica-count failures are a sequencing problem, covered in the autoscaling section. Memory failures are an honesty problem, covered in the section after that.

# The three-command triage for ScheduleFailed, in order
kubectl describe model fraud-scorer -n team-risk | sed -n '/Conditions/,$p'
kubectl get servers -n team-risk -o custom-columns=\
'NAME:.metadata.name,CONFIG:.spec.serverConfig,REPLICAS:.spec.replicas,CAPS:.spec.capabilities'
kubectl logs -n team-risk statefulset/mlserver -c agent --tail=100

Partial scheduling, introduced in 2.9.0, changes how this looks. A model may now run on fewer replicas than requested when capacity is short, rather than refusing to run at all. That is the right behaviour for availability and the wrong behaviour for a quiet platform, because a model serving on one replica instead of three is degraded without being down. A regression in this area existed from 2.9.0 onward and became visible in 2.10.0 — after pod restarts, models could end up deployed to fewer server replicas than requested — and it was fixed in 2.10.1. If you are running anything in the 2.9.x or 2.10.0 range, that fix alone justifies the upgrade.

The scheduler is also a single logical component per SeldonRuntime, which has two consequences. First, it is a concentration of state: a scheduler that restarts has to rediscover placement, and the robustness work in 2.10.0 — faster recovery after component restarts and finer-grained Pipeline status updates — exists because that rediscovery used to be slower and noisier than operators wanted. Second, it is a natural blast-radius boundary. If you want one team's bad deploy to be unable to delay another team's rollout, the answer is a separate SeldonRuntime in a separate namespace, not a bigger scheduler.

Sizing memory honestly

The memory field on a Model is a declared budget the scheduler uses for packing. It is not a Kubernetes resource limit and nothing enforces it at runtime. That distinction is the source of a failure mode that is very hard to diagnose after the fact: if every team under-declares, the scheduler happily packs far more models onto a server replica than the pod can hold, the pod is OOM-killed by the kernel, every model on that replica goes down together, and the postmortem blames Kubernetes.

The quickstart's memory: 100Ki is a toy value, and the most valuable platform policy you can write is that it may never appear in a production manifest. Replace it with a measured number.

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: fraud-scorer
spec:
  storageUri: "s3://risk-artifacts/fraud/2026-09/classifier"
  requirements:
  - sklearn
  memory: 1200Mi      # measured resident set under load, plus headroom
  replicas: 3

Measuring is unglamorous and takes an afternoon. Load the model onto an otherwise empty server replica, drive representative traffic at it, read the container's working set, and declare that plus headroom. Do it once per model family rather than once per model, and keep the numbers in the same repository as the manifests so a reviewer can see when a new model claims an implausible figure.

Memory-aware model scheduling arrived in 2.9.0, which means the scheduler's behaviour here is relatively recent and worth re-reading in the release notes rather than assumed from older blog posts. The same release introduced inference response streaming over SSE and gRPC, which requires MLServer 1.6.0 or later — a version constraint that matters because changing the MLServer image also changes the server's capabilities, and therefore which models can be scheduled onto it.

Scaling when model autoscaling was taken away

This is the area where stale advice does the most damage, because native model autoscaling was deliberately disabled in 2.10.0 pending wider fixes, and the official guidance is now to autoscale servers with HPA or KEDA instead. Anything you read that configures autoscaling on the Model alone and expects it to work is describing a version you should not be running.

The supported pattern for single-model serving is more involved than people expect, and its constraints are worth stating plainly. Only custom Prometheus metrics are supported; native CPU and memory metrics cannot be used, because multiple HPAs cannot be allowed to control the same pods. You therefore need a metrics adapter — the Prometheus adapter or KEDA — in the cluster before any of this functions. You then apply two HorizontalPodAutoscalers together: one targeting the Model and one targeting the Server, with the same metric, the same minReplicas and maxReplicas, and the same policies.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: fraud-scorer-model
spec:
  scaleTargetRef:
    apiVersion: mlops.seldon.io/v1alpha1
    kind: Model
    name: fraud-scorer
  minReplicas: 1
  maxReplicas: 6
  metrics:
  - type: Object
    object:
      metric:
        name: infer_rps
      describedObject:
        apiVersion: mlops.seldon.io/v1alpha1
        kind: Model
        name: fraud-scorer
      target:
        type: AverageValue
        averageValue: 3

The second HPA is the same document with kind: Server and the server's name as the target, and the same metric and bounds. The reason both are needed is that the scheduler does not create server replicas. If the model scales first, it asks for more replicas than exist, reports ModelReady: False with ScheduleFailed, and — importantly — the replicas that already exist keep serving. That is a degraded state rather than an outage, but it is also exactly the state an engineer will misread as "the autoscaler is broken".

Two practices from the official guidance deserve to be platform policy rather than advice. Do not configure scale-up to request more replicas than can realistically become available in time, because asking for capacity that cannot arrive turns an autoscaling event into a sustained ScheduleFailed. And tune stabilisation windows deliberately: inference servers load artifacts on startup, so they are slow to become useful, and an aggressive scale-down followed by a scale-up is far more expensive here than for a stateless web service.

Pipeline components scale on a different axis entirely. Before 2.10.0 the dataflow engine, pipeline gateway and model gateway were effectively bounded by Kafka partition counts. The maxShardCountMultiplier setting in SeldonConfig, added in 2.10.0, lets them scale horizontally beyond that limit. That same release introduced a new scaling configuration block under config.ScalingConfig in SeldonConfig, whose stated purpose is to centralise Core 2 configuration so it can be changed after the cluster is deployed rather than only at install time.

Kafka is a production system, not a dependency

For direct model calls Kafka is irrelevant. For pipelines it is the substrate: each step's input and output is a topic, which is precisely what gives you the audit trail, the replay capability and the data-centric tracing that distinguish Core 2 from a plain model server. The cost of that design is that you now operate, or pay someone to operate, a Kafka cluster with production SLOs.

The practical decision is whether to self-host. The learning-environment documentation covers self-hosted Kafka, and it is the right choice for a laptop cluster. For production the realistic options are a managed service — Confluent Cloud or AWS MSK are both documented integration targets — or a Kafka team that already exists in your organisation. In the Gulf and in Egypt the data-residency question usually settles this: if prediction inputs and outputs are regulated personal or financial data, and your pipeline topics therefore hold that data at rest for the retention period, then the region in which the Kafka cluster lives is a compliance fact and not an infrastructure preference. Managed Kafka in a Middle East region, or self-hosted inside the same cluster, are the two defensible answers; a managed cluster in another continent is usually not.

Security configuration is Helm values plus Kubernetes Secrets. For TLS you reference a Secret through security.kafka.ssl.client.secret. For SASL_SSL, which is what Confluent Cloud needs, the values set the protocol, the mechanism and the client secrets for both the SASL and the SSL sides, and official Confluent SASL and OAuth 2.0 examples exist in the documentation — check the current version's path, because these examples have lived under versioned paths such as the v2.8 tree.

# Secrets first, values second — the chart expects the Secret to exist
kubectl create secret generic kafka-client-tls \
  --from-file=ca.crt --from-file=tls.crt --from-file=tls.key -n seldon-mesh

helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
  --namespace seldon-mesh --install --version 2.10.2 \
  --set security.kafka.ssl.client.secret=kafka-client-tls

Kafka is also where pipeline cost accumulates. Retention and partition count are the two levers, and both are easy to set generously and forget. A pipeline that carries full feature vectors through four steps writes that payload to four topics, and if retention is seven days you are paying to store a week of every intermediate tensor. The discipline that follows is simple and frequently ignored: do not build a pipeline where a direct model call would do. A pipeline earns its Kafka bill when you need the audit trail, the replay, a join across inputs, or conditional routing. A single preprocessing step followed by a single model often does not need any of those.

The trust model

Be precise about this when leadership asks, because the honest answer is uncomfortable. The open-source product does not give you user authentication on the inference endpoint. There is one seldon-mesh Service and Envoy routes to models based on the Seldon-Model header. Anything that can reach that Service can address any model behind it, by name, by setting a header. There is no per-model authorisation step in that path.

That is not a defect so much as a scope decision, and it has one correct response: put something in front. An ingress gateway or a service mesh terminates client identity, and only it is permitted to reach seldon-mesh. A NetworkPolicy enforces that, so a compromised workload elsewhere in the cluster cannot bypass the gateway and talk to Envoy directly. This is the control that most Core 2 installations are missing, and it is cheap.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: seldon-mesh-from-gateway-only
  namespace: seldon-mesh
spec:
  podSelector:
    matchLabels:
      app: seldon-envoy
  policyTypes: [Ingress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-system

Match the podSelector to the labels your installation actually puts on the Envoy pods, which you can read from kubectl get pods -n seldon-mesh --show-labels; chart label schemes change between versions and a policy with a selector that matches nothing is worse than no policy, because it looks like a control.

Beyond the endpoint, the trust boundaries worth writing down are the artifact store, Kafka and the Kubernetes API. The artifact store is a code-execution path: storageUri points at something an agent downloads and an inference server loads, and for several frameworks loading an artifact means deserialising attacker-controllable data. Whoever can write to the bucket path a Model references can, in effect, run code inside your serving pods. Treat write access to model artifact buckets with the same seriousness as write access to a container registry, and prefer immutable, versioned paths — the official samples' convention of pathing by MLServer version, as in mlserver_1.5.0, is a reasonable habit to borrow for your own artifacts.

Kafka credentials are the second boundary. Per-team credentials and per-team topic scoping mean a tenant cannot read another tenant's prediction data out of the pipeline topics. Shared credentials across the platform mean they can, and since those topics hold exactly the inputs and outputs your audit story relies on, that is a data-protection finding waiting to be written.

The licence is the boundary nobody thinks of as security but legal will. Seldon relicensed Core from Apache 2.0 to the Business Source License, and production use may require a commercial licence. Read LICENSE in the repository and get a written answer from your legal function before you build a platform on it. I have flagged this as unverified for your reviewer rather than paraphrasing terms I did not open this session.

Secrets and the artifact supply chain

Artifact credentials reach the agent as Kubernetes Secrets in rclone configuration format, because storageUri is an rclone-style URI — gs://, s3://, a local PVC path, and the other schemes rclone supports. They can be referenced per Model or supplied globally through the server configuration and agent environment. Either way, the rule is the same: nothing with a key in it goes into a manifest or into Helm values that land in git.

In practice that means an external secret manager. The External Secrets Operator or Vault materialise the rclone Secret in the namespace from a source of truth that has rotation, audit and access control, and your manifests reference a Secret name rather than a credential. Prefer workload identity where your cloud offers it, so there is no long-lived key to rotate at all; that is the strongest version of this control and is available on the major providers' Kubernetes offerings.

The supply chain has three links and each wants a different control. The container images — MLServer, Triton, and whatever custom servers you build — should be pinned by digest in your Server specs and scanned on a schedule, exactly as you would treat any other production image. The model artifacts want provenance: a record of which training run produced the file at that path, which is what an experiment tracker like MLflow gives you if the handoff from training to serving goes through it rather than through a manual upload. And the Helm charts want version pinning with --version, so an upgrade is a deliberate act rather than whatever the repository served that morning.

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver-risk
spec:
  serverConfig: mlserver
  capabilities:
  - mlserver-1.6.0
  - sklearn
  podSpec:
    containers:
    - name: mlserver
      image: seldonio/mlserver:1.6.0

Note what that manifest does deliberately. It uses capabilities rather than extraCapabilities, so the server advertises exactly the two tags listed and nothing from the default set — which for MLServer would otherwise include mlserver, alibi-detect, alibi-explain, huggingface, lightgbm, mlflow, python, sklearn, spark-mlib and xgboost. Advertising a capability you have not tested means the scheduler will place models you cannot actually serve. Pinning the capability name to the image version, as the Servers documentation demonstrates, makes the coupling between artifact and runtime explicit: a model that needs MLServer 1.6.0 behaviour asks for mlserver-1.6.0 and cannot be scheduled onto an older image by accident.

Multi-tenancy that actually isolates

Multi-tenancy in Core 2 is decided by two choices: the operator's scope, and how many SeldonRuntime instances you run. Everything else is conventional Kubernetes isolation.

The cluster-wide operator with controller.clusterwide=true is the simplest install and the weakest boundary. One operator reconciles every namespace, and one SeldonRuntime in a shared namespace means one scheduler, one Envoy and one set of gateways for all tenants. Noisy-neighbour effects go straight through: a tenant that floods the mesh affects everyone, and a tenant whose bad Pipeline wedges the dataflow engine affects everyone's pipelines.

The namespaced form, --set "controller.watchNamespaces={team-risk,team-growth}", with a SeldonRuntime per tenant namespace, gives each tenant its own scheduler, Envoy and gateways. You pay for that in fixed overhead per tenant — several pods that exist whether the tenant serves one model or fifty — and in operational surface, because now you upgrade and monitor N runtimes. It is the right trade for tenants with different availability expectations or different regulatory scope, and the wrong trade for twenty small internal teams that all accept the same SLO.

A practical middle path is two tiers. A shared runtime hosts the long tail of internal models, where density matters and an occasional noisy neighbour is acceptable. Dedicated runtimes host the handful of workloads with real SLOs or regulated data. Publish which tier a workload qualifies for and what it costs, so the choice is made by the requesting team against a price rather than by you against a backlog.

Whichever shape you choose, the conventional controls still have to be there: RBAC so a tenant can create Models and Pipelines in their own namespace and nothing in anyone else's, resource quotas so one tenant's server replicas cannot consume the cluster, NetworkPolicies so tenants cannot address each other's mesh directly, and separate Kafka credentials with per-tenant topic scoping. Kafka topic naming is prefixed per namespace and runtime, which is what makes that scoping possible, though you should verify the exact prefix your version produces before writing an ACL against it.

Cost, and the levers that actually move it

Three things dominate the bill: server replicas, GPU nodes, and Kafka. They respond to different levers and it is worth knowing which lever you are pulling.

Multi-model serving is the biggest single lever on replica count, because packing many models onto one replica and loading or unloading them dynamically is what lets a hundred low-traffic models share a handful of pods. This is the structural advantage Core 2 has over one-pod-per-model designs, and it is also the thing the documented HPA pattern forces you to give up for the models you autoscale. The resolution is segmentation: the long tail of low-traffic models lives on packed shared servers with no autoscaling, and the small number of high-traffic models gets dedicated servers with paired HPAs. Mixing the two policies on one server farm gives you the costs of both and the benefits of neither.

Right-sizing memory is the second lever, and it cuts both ways. Over-declaring wastes capacity because the scheduler refuses to pack more models onto a replica that is actually half empty. Under-declaring causes the OOM incident from the sizing section. The measured number is the cheap number.

Scale-down behaviour is the third. Stabilisation windows that are too short cause thrash, and thrash is expensive here because every scale-up reloads artifacts. Windows that are too long mean you pay for peak capacity most of the day. Pick the windows from your traffic shape and revisit them quarterly rather than at install time.

Kafka is the fourth and the one that surprises people, because it does not appear on a GPU dashboard. Partitions and retention set the floor, pipeline depth multiplies it, and the fix is architectural: fewer pipelines, shallower pipelines, and retention per topic class.

Upgrades and migrations

Upgrades are mechanical and the order is not optional. Helm does not upgrade CRDs held in a subchart, which is exactly why Core 2 ships CRDs as their own chart. Upgrade seldon-core-v2-crds first, then seldon-core-v2-setup for the operator, then seldon-core-v2-runtime, then seldon-core-v2-servers. Uninstall runs in reverse, and deleting the CRDs cascades: every Model, Server and Pipeline in the cluster goes with them. Put that sentence in your runbook in bold, because someone will eventually try to "clean up" the CRD chart.

for chart in crds setup runtime servers; do
  helm upgrade "seldon-core-v2-${chart}" "seldon-charts/seldon-core-v2-${chart}" \
    --namespace "$([ "$chart" = crds ] && echo default || echo seldon-mesh)" \
    --install --version 2.10.2 --wait
done
kubectl get pods -n seldon-mesh

The 2.9-to-2.10 path is a gentle one. All CRD changes are backward compatible with existing CRs, and with Helm the existing values fill in the new options, so the upgrade is seamless. The one action item is forward-looking: set the new scaling configuration before you upgrade to 2.11. If you install without Helm, maxShardCountMultiplier has to be set manually, because nothing fills it in for you.

The 2.10.x patches are worth taking promptly rather than batching. 2.10.1 fixed the partial-scheduling regression that could leave models on fewer replicas than requested after pod restarts. 2.10.2 fixed several long-standing issues that became more visible across 2.10.x: pipelines failing to create or delete, gRPC streams blocking, operator reconciliation problems, and an incorrect MLServer parallel-workers configuration. Each of those is a plausible page. If you are debugging any of those four symptoms on 2.10.0, check your version before you check anything else.

MLServer version bumps deserve their own change window, because the image's version determines its capabilities and therefore what the scheduler will place on it. Artifacts are built against a runtime version — the official samples are pathed by it, as in mlserver_1.4.0 and mlserver_1.5.0 — and an artifact that loads under one version may not under another. Roll the image on a canary server with pinned capability names, move a representative model family across, and only then roll the farm.

Governance, observability and audit

The strongest governance argument for Core 2 is that pipeline data is retained in Kafka, so prediction inputs and outputs are auditable and replayable rather than merely logged. That turns several conversations that are usually hand-waved — what did the model see, what did it return, would the new version have behaved differently on last Tuesday's traffic — into queries. It is worth building on deliberately rather than treating as a side effect.

Operationally, the install flow puts Prometheus in a seldon-monitoring namespace and Core 2 exposes metrics including infer_rps, which is the same metric the HPA pattern consumes. The dashboards that matter to a platform owner are not the per-model latency charts the teams want; they are the ones that show the platform's own health. Requested versus actual replicas per model, catching degraded partial scheduling. Scheduler and operator reconciliation health, catching control-plane stalls. Kafka consumer lag on pipeline topics, catching a dataflow engine that has fallen behind. Agent artifact-fetch failures, catching credential expiry before a deploy does. Tracing is configurable through OpenTelemetry, which pairs naturally with OpenTelemetry if your organisation already collects spans.

Data-science monitoring is a different axis and Core 2 treats it as a first-class deployment concern rather than an external add-on: drift and outlier detectors are themselves Models, scheduled onto servers with the alibi-detect capability, and placed inside pipelines so they see the real traffic. That is a genuinely good design, because the detector runs on the same data path as the model and its outputs land on Kafka topics alongside everything else. If your organisation already uses Alibi Detect or Evidently, the platform decision is where the detection runs: inside the pipeline where it is operationally coupled and automatically fed, or outside on a batch export where it is independent but delayed.

Governance also means saying who may do what. A model that handles regulated data should not be deployable without a review, and the cheapest enforcement is to make the manifests live in git with required reviewers and let Argo CD apply them, rather than giving teams kubectl apply rights in production namespaces. That also gives you the change history you will want during an incident.

Incident playbooks

Four incidents account for most pages on a Core 2 platform. Each has a different first move, and the point of writing them down is that the first move is where time is lost.

Models will not schedule. The condition says ScheduleFailed. Check capabilities, then replica counts, then declared memory against free memory — in that order, because capability typos are the most common and the fastest to confirm. If several models failed at once, suspect a Server change that replaced the default capability set with capabilities instead of extending it with extraCapabilities.

Inference returns 404 or complains the model is not found. Almost always routing rather than serving. Confirm the Seldon-Model header is present and spelled correctly, and confirm that a pipeline target carries the .pipeline suffix — income-classifier-app.pipeline rather than the bare name. Then confirm the model has finished loading, because a model that is still loading is not yet addressable.

A pipeline will not become ready, or will not delete. Check the version first: this was a known issue fixed in 2.10.2. If you are current, move to Kafka — connectivity, credentials, and then the dataflow engine's logs. Kafka timeouts and broker errors from the dataflow engine or model gateway point at wrong security.kafka.* values or unreachable brokers, and the usual root cause is a Secret that does not exist in the namespace the component reads from.

Server pods are crash-looping. Read the agent container's log before the inference server's. The two dominant causes are an rclone failure fetching storageUri — bad credentials or a wrong bucket path — and an MLServer version that cannot load the artifact it was given. ImagePullBackOff with x509: certificate signed by unknown authority is a different animal entirely: a proxy or TLS-inspecting middlebox between the node and the registry, which in corporate environments is often an endpoint agent rather than anything in your cluster.

# The sequence that resolves most of the above
kubectl get model,server,pipeline -n team-risk
kubectl describe model fraud-scorer -n team-risk
kubectl logs -n team-risk <server-pod> -c agent --tail=200
kubectl logs -n team-risk <server-pod> -c mlserver --tail=200
kubectl logs -n team-risk deploy/seldon-scheduler --tail=200

For pipelines, seldon pipeline inspect <name> reads the Kafka topics directly and is the fastest way to see where in a DAG the data stopped. Verify the CLI's exact invocation against the current CLI documentation for your version before putting it in a runbook; the binary is distributed as a release artifact and talks to the scheduler directly rather than through the Kubernetes API, which also means it needs scheduler reachability that a restricted on-call environment may not have.

Where Seldon Core stops

A platform owner's credibility depends on knowing the boundaries of their own choice. Core 2 is a strong fit when you have many models to serve densely, when the data path itself needs auditing and replaying, and when multi-step inference graphs with joins, triggers and conditional routing are genuinely part of the product. It is a poor fit in several identifiable situations.

If you have a handful of models and no pipeline requirements, you are paying for a scheduler, a mesh, gateways and possibly a Kafka cluster to do something a simpler serving layer does with less. KServe takes the other design position — one InferenceService per model, serverless-style scaling, no Kafka in the request path — and both speak the Open Inference Protocol, so the client contract is comparable. For a single model behind an API, BentoML or a plain FastAPI service on Kubernetes is less machinery to own.

If your workload is large-model inference where throughput per GPU is the whole game, the serving runtime matters more than the orchestration around it, and you will end up comparing vLLM or Triton on their own terms. Core 2 can front those, and 2.10.0's OpenAI API translation layer — which converts OpenAI-style REST calls to and from the Open Inference Protocol so standard OpenAI client libraries can talk to LLMs deployed on Core 2 — makes that integration much more palatable than it was. But choose the runtime on runtime criteria first.

If your inference is fundamentally batch, pipelines over Kafka are the wrong shape for a nightly scoring job, and an orchestrator belongs there instead. And if the licence answer from your legal function is unfavourable, that settles it regardless of the technical fit.

Running it as a platform for other teams

The difference between operating Core 2 and offering it is the contract. Teams need to know what they get, what they must supply, and what you will not do.

Publish the tiers from the tenancy section with their SLOs and their costs. Publish the capability names your servers advertise, because a team cannot write a correct requirements list by guessing. Publish the memory-declaration policy, with the measurement procedure, because this is the field that causes shared-server incidents. Publish the pipeline policy: when a pipeline is justified and when a direct model call is expected, with the Kafka cost named so the choice has a price attached.

Then make the paved road genuinely easier than the alternative. A template repository with a Model manifest, a Server manifest with pinned image and capability names, the paired HPAs if the workload qualifies, and an Argo CD application is a morning's work and removes most of the review burden permanently. Teams that start from a correct template produce correct manifests.

Keep two things for yourself. Scheduler and runtime health is yours, not theirs, and the dashboards that show it should not be in a tenant's view. And version management is yours: chart versions, MLServer images and the upgrade calendar are platform decisions, communicated with notice, not negotiated per team.

The review checklist

Hand this to your reviewers. It is ordered so that the items that cause incidents come first.

Models. Is memory a measured number rather than a copied one? Is replicas matched by a server that can satisfy it? Do the requirements name capabilities that a server in this namespace actually advertises? Is storageUri an immutable, versioned path? Is the artifact's provenance traceable to a training run?

Servers. Is the image pinned, by digest if possible? Does the manifest use capabilities or extraCapabilities, and is that choice deliberate rather than accidental? Are the advertised capabilities ones that have been tested on this image? Are pod requests and limits set above the sum of the declared model budgets?

Pipelines. Is there one sentence naming the capability — audit, replay, join, conditional routing — that justifies this being a pipeline? Is output.steps set, given that it is required for synchronous responses? Are the join types deliberate, with joinWindowMs set where outer is used? Is topic retention appropriate to the data class rather than inherited from a default?

Autoscaling. Are both HPAs present, on the Model and on the Server, with matching metrics and bounds? Is the metric a custom Prometheus metric, since CPU and memory are unavailable here? Is the scale-up target achievable with the capacity that exists? Are stabilisation windows set for a workload that loads artifacts on startup?

Security. Is there an ingress or mesh in front of seldon-mesh, with a NetworkPolicy that actually enforces it? Are credentials external rather than inline? Are Kafka credentials scoped per tenant? Is write access to artifact buckets restricted to the systems that need it?

Change. Are chart versions pinned? Was the upgrade order CRD-first? Were the release notes read? Is the scaling configuration set ahead of 2.11? Is this a Core 2 manifest rather than a v1 SeldonDeployment?

The complete picture

Put the three levels together and the shape of Seldon Core 2 is consistent. At the beginner level it is a few custom resources: a Server that runs models, a Model that points at an artifact and declares what it needs, and an inference call with a header. At the mid level it is a data-centric system: pipelines as DAGs over Kafka topics, experiments for traffic splitting, capabilities as the contract between artifacts and runtimes. At this level it is a platform with two planes, one stateful scheduler, a hard Kafka dependency for part of its functionality, an inference endpoint with no built-in authentication, and a licence question that belongs to your legal team.

The through-line is the scheduler. Nearly every senior concern reduces to it: capability naming is how you talk to it, memory declaration is how you feed it, autoscaling is a dance between it and two HPAs, multi-tenancy is a decision about how many of them to run, and partial scheduling is its way of telling you that capacity is short without telling you loudly. An engineer who reasons fluently about placement is operating this platform well.

The second through-line is Kafka, and it is the honest cost of the thing that makes Core 2 distinctive. You cannot have auditable, replayable prediction data without storing prediction data, and storing it means a cluster, credentials, retention policy and a bill. The platforms that get this right are the ones where pipelines are used where they earn that cost and direct model calls are used everywhere else.

What has changed recently is worth holding in mind as a set. 2.9.0 brought streaming, partial scheduling and memory-aware placement. 2.10.0 brought pipeline components that scale past Kafka partition limits, the centralised scaling configuration, Schema Registry integration, the OpenAI translation layer, lifecycle probes on every component — and it disabled native model autoscaling, redirecting you to HPA or KEDA. 2.10.1 and 2.10.2 fixed scheduling, pipeline lifecycle and gRPC streaming bugs that are each a plausible page. A platform owner who is current on those five releases will diagnose faster than one working from a two-year-old mental model.

Where the series leaves you

You can now install Core 2 in a defensible production shape, decide whether a workload belongs on a shared or a dedicated runtime, size memory from measurement, build autoscaling that works on the current release, operate the Kafka dependency with its costs understood, put a real authentication boundary in front of an endpoint that has none of its own, upgrade four charts in the right order without losing every CR, and argue honestly about when a simpler serving stack would serve your organisation better.

The next things worth your time are adjacent rather than deeper. KServe is the comparison you will be asked to make, and running both briefly teaches more than reading about either. Prometheus and Grafana are where your platform-health dashboards live, and the custom-metrics work the HPA pattern requires is Prometheus work rather than Seldon work. Alibi Detect is the natural next step if the data-science monitoring side of the platform is yours too. And if the licence answer sends you looking, BentoML and Ray Serve are the two alternatives whose design positions differ enough from Core 2 to be genuinely instructive.

Sources

• Seldon Core 2 documentation
• Concepts
• Core features
• Architecture
• Quickstart
• Production environment install
• Learning environment
• Upgrading
• Models
• Model scheduling
• Servers
• Pipelines
• Single-model serving with HPA
• Operational monitoring
• Performance tuning
• Advanced pipeline configuration
• Custom servers example
• Releases on GitHub
• Helm charts

Senior interviews about a serving platform rarely test whether you can write a Model manifest. They test whether you understand what you are on the hook for: which component concentrates state, what happens when the dependency you did not choose goes down, how you would isolate two teams who do not trust each other, and whether you can tell an executive that the tool you champion is the wrong one for a given workload. Seldon Core 2 gives an interviewer unusually good material for all four, because its design makes its trade-offs visible.

Everything below assumes Seldon Core 2 on the 2.10.x line. If an interviewer's question presupposes native model autoscaling or the SeldonDeployment CRD, that is itself a signal worth handling, and the last section covers how.

The two-minute framing

If you get one open question — "tell me how you would run Seldon Core 2 for a platform" — this is the shape of a strong answer, and it is worth having ready because it sets the agenda for everything after it.

Seldon Core 2 splits into a control plane and a data plane. The control plane is an operator, a scheduler and per-pod agents; the scheduler holds model-to-replica placement and is the stateful component. The data plane is Envoy behind a single seldon-mesh Service, inference servers — MLServer or Triton — as StatefulSets that each hold many models, and for pipelines a set of gateways plus a dataflow engine over Kafka. Direct model calls do not touch Kafka; pipelines cannot exist without it.

From there, name the three things you own that are not obvious: the scheduler's placement state, a Kafka cluster with production SLOs if you offer pipelines, and an authentication boundary in front of an inference endpoint that has none of its own. A candidate who names those three has demonstrated ownership rather than familiarity, and most interviewers will follow one of them rather than asking their own next question.

System design: a multi-tenant serving platform

The prompt. "Six product teams want to serve models. Two handle regulated customer data. The rest have maybe eighty small models between them, mostly low traffic. Design the platform."

The answer that lands. Start with the tenancy decision, because it determines everything else. Two tiers. A shared SeldonRuntime in one namespace carries the long tail: eighty small models packed densely onto a handful of MLServer replicas using multi-model serving, which is the structural reason to choose Core 2 at all. The two regulated workloads get dedicated runtimes in their own namespaces, with the operator installed namespaced via controller.watchNamespaces rather than cluster-wide, so each has its own scheduler, Envoy and gateways and no shared blast radius.

Then be explicit about the cost of that choice: a dedicated runtime is several fixed pods whether it serves one model or fifty, and you now upgrade and monitor three runtimes instead of one. Say that you measured the per-runtime overhead and published it as the price of the dedicated tier, so teams choose against a number.

Next, the data path. One ingress or service mesh terminates client identity, and a NetworkPolicy ensures only that gateway can reach seldon-mesh in each namespace — because Envoy routes purely on the Seldon-Model header and anything that can reach the Service can address any model behind it by name. This is where a lot of candidates stop too early; the NetworkPolicy is what makes the gateway a boundary rather than a convention.

Then the artifact path. Per-tenant buckets with versioned, immutable paths, rclone credentials delivered as externally managed Secrets, and write access to artifact paths treated as equivalent to write access to a container registry, because storageUri is a code-execution path: whoever writes the artifact controls what the inference server deserialises.

Finally, pipelines. Only the regulated tenants get them initially, because Kafka topics are what deliver the audit trail and replay they need, and the storage cost is justified there. The long tail gets direct model calls, with a published policy naming when a pipeline is warranted — audit, replay, a join across inputs, or conditional routing — so teams do not reach for one by default.

What a weak answer misses. It designs for the eighty models and forgets the two regulated ones. It never mentions that the inference endpoint has no authentication. It treats Kafka as a checkbox rather than a cluster someone operates. And it does not price the isolation it recommends.

System design: autoscaling a high-traffic model

The prompt. "One model takes most of the platform's traffic and it is spiky. Make it elastic."

The answer that lands. Open by establishing the version constraint, because this is where outdated knowledge shows immediately: native model autoscaling was disabled in 2.10.0 pending wider fixes, and the official guidance is to scale servers with HPA or KEDA instead. So the design is HPA-based.

Describe the documented pattern precisely. Two HorizontalPodAutoscalers applied together: one with scaleTargetRef.kind: Model and one with kind: Server, both using the same custom metric, the same minReplicas and maxReplicas, and the same policies. The metric must be a custom Prometheus metric — infer_rps as an Object metric whose describedObject is the Model, with an AverageValue target, is the documented example — because native CPU and memory metrics are not supported here: multiple HPAs cannot be allowed to control the same pods. That requires a metrics adapter, Prometheus adapter or KEDA, in the cluster first.

Then explain why both are mandatory, which is the part that distinguishes a candidate who has run this from one who has read about it. The scheduler does not create server replicas. If the Model scales and the Server does not, the model asks for more replicas than exist and reports ModelReady: False with reason ScheduleFailed — while the replicas that already exist keep serving. So the symptom is a degraded model and a scary-looking condition, not an outage, and an engineer who does not know this will misread it as a broken autoscaler.

Close with the trade-off, unprompted. This pattern needs a one-to-one mapping of Models to Servers, so the autoscaled model cannot share a replica with others and you give up multi-model serving for it. That is why you segment: the high-traffic model gets a dedicated server with paired HPAs, and the long tail stays packed with no autoscaling. Mixing both policies on one farm gives you the costs of both and the benefits of neither.

Add the two operational cautions from the documentation: do not configure scale-up to request more replicas than can become available in time, and set stabilisation windows generously, because inference servers load artifacts on startup and are therefore slow to become useful — thrash is much more expensive here than for a stateless service.

Incident response: models will not schedule

The prompt. "A deploy goes out and twelve models across three namespaces go ScheduleFailed. Walk me through it."

The answer that lands. The scheduler filters servers on exactly three conditions, and the triage follows their likelihood rather than their order in the docs. Every entry in the model's requirements must appear in the server's capabilities; the server must have at least as many replicas as the model's replicas; and each replica must have enough free declared memory for the model's memory.

Twelve models at once across namespaces points at a Server change rather than twelve bad models. The specific thing I would check first is whether someone set capabilities on a Server when they meant extraCapabilities — capabilities overrides the default set entirely, while extraCapabilities extends it, so one line can strip a server of sklearn, xgboost, mlflow and the rest and unschedule everything that asked for them. That single confusion explains more mass scheduling failures than anything else.

The commands: kubectl describe model <name> for the condition and reason, kubectl get servers with the capabilities column to see what is actually advertised, then the agent container's log on the server pod. Then say what you would do next time: alert on the gap between requested and actual replicas per model, not just on ModelReady, because partial scheduling since 2.9.0 means a model can be "ready" on fewer replicas than it asked for and look healthy on a dashboard while being one traffic peak from trouble.

Incident response: pipelines will not become ready

The prompt. "Pipelines stopped creating. Direct model calls are fine."

The answer that lands. The fact that direct calls are fine is the diagnosis's first gift: it localises the problem to the Kafka side of the system, because pipelines depend on Kafka and direct model calls do not. So the control plane and the mesh are probably healthy.

The first move is to check the version, and saying so out loud is a strong signal. Pipelines failing to create or delete was a known issue fixed in 2.10.2, alongside gRPC stream blocking and operator reconciliation problems — all of which were long-standing issues that became more visible across the 2.10.x releases. If the cluster is on 2.10.0, that is the likely answer and the fix is an upgrade.

If the cluster is current, go to Kafka: connectivity, then credentials, then the dataflow engine's logs. Broker errors and timeouts from the dataflow engine or model gateway usually mean wrong security.kafka.* Helm values or brokers that are unreachable, and the most common concrete cause is a referenced Secret that does not exist in the namespace the component reads from. seldon pipeline inspect <name> reads the topics directly and shows where in the DAG the data stopped, which is the fastest way to localise a partially-working pipeline — verify its exact invocation for your version, and note that the CLI talks to the scheduler directly rather than through the Kubernetes API, so it needs network reachability a locked-down on-call environment may not have.

Arguing the trade-off to leadership

Three conversations recur, and each is won by naming the cost rather than hiding it.

"Why do we need Kafka to serve models?" We do not, for direct model calls. We need it for pipelines, and what pipelines buy is that every step's input and output is retained, so prediction data is auditable and replayable. That is the capability that lets us answer what a model saw on a given day and test whether a new version would have behaved differently on real historical traffic. If we do not need those answers, we should not be running pipelines, and I have a policy that says so. The honest framing is that Kafka is the price of the audit story, and we should pay it where the audit story matters and not elsewhere.

"Should we move to KServe?" They solve the same problem from opposite ends, and both speak the Open Inference Protocol so the client contract is comparable. KServe is one InferenceService per model with serverless-style scaling and no Kafka in the request path, which is less machinery for a handful of models. Core 2 is multi-model serving with many models per replica and a Kafka-backed data path, which is cheaper at high model counts and gives us the audit trail. With eighty models, density is worth real money to us; with eight, I would be recommending the move rather than defending against it. That symmetry is what makes the answer credible.

"Is this free?" No, and this is the one to raise before someone else does. Seldon relicensed Core from Apache 2.0 to the Business Source License, and production use may require a commercial licence. I would say plainly that the licence terms need a written answer from legal before we commit a platform to it, and that I have not interpreted the terms myself because that is not my call to make. An interviewer hearing a candidate volunteer a licensing risk about their own preferred tool will read that as judgement.

Questions to expect, briefly answered

"What breaks first as the platform grows?" Placement, not compute. The scheduler is the bottleneck and ScheduleFailed is the symptom, usually driven by capability naming drift and under-declared memory rather than by a shortage of nodes.

"What is memory on a Model?" A declared budget the scheduler uses for packing, not an enforced Kubernetes limit. Nothing stops a model exceeding it. If teams under-declare, the scheduler over-packs a replica, the kernel OOM-kills the pod, and every model on it goes down together. The quickstart's 100Ki should be banned from production manifests and replaced with a measured number.

"What happens if the scheduler dies?" Existing loaded models keep serving, because the data plane is independent. New applies and changes stall. It is a change freeze, not an outage — a different severity, and your runbook should say so. 2.10.0 added lifecycle probes to all components and improved recovery after restarts for exactly this reason.

"Models versus Servers versus capabilities?" A Server is a StatefulSet of inference-server replicas that advertises capability tags. A Model declares requirements, which must be a subset of some server's capabilities for the scheduler to place it. That subset relation is the whole contract between artifacts and runtimes, and pinning capability names to image versions — mlserver-1.6.0 with seldonio/mlserver:1.6.0 — makes it explicit instead of implicit.

"How do you upgrade?" Four charts in order: seldon-core-v2-crds, then seldon-core-v2-setup, then seldon-core-v2-runtime, then seldon-core-v2-servers, with versions pinned. CRDs are a separate chart because Helm does not upgrade CRDs from a subchart. Uninstall is the reverse order, and deleting the CRDs cascades to every Model, Server and Pipeline in the cluster. The 2.9-to-2.10 CRD changes are backward compatible, and the forward-looking action is to set the new scaling configuration before upgrading to 2.11.

"How do you route to a pipeline?" The same URL shape, /v2/models/<name>/infer, but the Seldon-Model header carries a .pipeline suffix — income-classifier-app.pipeline. A missing or wrong header is the usual cause of a model-not-found response, and that suffix is the specific trap for pipelines.

Handling an interviewer who is on the wrong version

This happens often, because most writing about Seldon on the public internet describes Core 1 — a separate product line built on the SeldonDeployment CRD and the v1 Python wrapper, still releasing as maintenance (v1.19.0 in January 2026, platform maintenance, security hardening and a move to Python 3.12) and not compatible with Core 2 custom resources. If the question assumes SeldonDeployment or the v1 wrapper, answer the question as asked and then say which line you are describing and how the Core 2 equivalent differs. Correcting an interviewer flatly is a poor trade; demonstrating that you know two architectures exist and which one is current is a good one.

The same applies to native model autoscaling. If someone asks you to configure it, say that it was disabled in 2.10.0 pending wider fixes, that the official guidance is HPA or KEDA on Servers, and then describe the paired-HPA pattern. The signal you are sending is that you track release notes, which for a platform owner is the most relevant habit there is.

Sixty-second self-test

Answer these out loud. If any takes more than ten seconds, that is the section to re-read.

Which plane keeps working when the scheduler dies, and what does that do to your incident severity? What are the scheduler's three filter conditions, in the order you would check them? What is the difference between capabilities and extraCapabilities, and which one wins if both are set? Why does the HPA pattern need two autoscalers, and what does it cost you? Why can you not use CPU metrics for it? What does memory on a Model actually do, and what happens when a team under-declares it? What does Kafka buy you, and what does it cost? What authentication protects the seldon-mesh endpoint out of the box? What suffix does a pipeline need in the Seldon-Model header? In what order do the four charts upgrade, and what happens if you delete the CRD chart? Which two patch releases fixed scheduling and pipeline-lifecycle bugs you would otherwise debug for a day? And what is the licence, and whose decision is it?

These are the things that do not appear until a Seldon Core 2 installation has real tenants, real traffic and a real on-call rotation. Each one has cost somebody a weekend. They assume the 2.10.x line.

The traps that only appear at scale

The toy memory value, propagated. The quickstart uses memory: 100Ki. It gets copied into the first real model, then into the template, then into eighty models. Because memory is a declared budget for the scheduler and not an enforced Kubernetes limit, nothing complains — the scheduler simply believes it and packs accordingly. The failure arrives much later as a kernel OOM kill on a shared server replica, taking down every model on that replica at once, and the postmortem blames Kubernetes memory pressure rather than the declaration. The fix is a platform policy: 100Ki is banned in production manifests, every model declares a number measured from an isolated replica under representative load, and the server pod's own memory limit sits above the sum of the declared budgets with headroom.

Partial scheduling as a silent degradation. Since 2.9.0 a model may run on fewer replicas than requested when capacity is short, rather than refusing to run. That is correct for availability and dangerous for a quiet platform, because ModelReady is true and the dashboard is green while the model is at a third of its intended capacity. There was also a regression here from 2.9.0 that became visible in 2.10.0 — after pod restarts, models could end up on fewer replicas than requested — fixed in 2.10.1. Alert on the gap between requested and actual replicas per model. This is the single highest-value Seldon-specific alert available to you and almost nobody has it.

capabilities where extraCapabilities was meant. Setting capabilities on a Server replaces the default set entirely; extraCapabilities extends it; capabilities wins if both are present. One line of YAML can therefore strip an MLServer farm of sklearn, xgboost, mlflow, huggingface, lightgbm and the rest, and every model that asked for them goes ScheduleFailed simultaneously. When a dozen models fail at once across namespaces, this is the first thing to check, not the last.

MLServer image bumps as capability changes. Changing the inference server image changes what the server can do, and therefore what the scheduler will place on it. Artifacts are built against a runtime version — the official samples are literally pathed by it, as in mlserver_1.4.0 and mlserver_1.5.0 — so an artifact that loads under one version may fail under another. An image bump treated as a routine patch becomes a serving incident. Treat it as a change window: canary server with pinned capability names, one representative model family moved across, then the farm.

Kafka retention set once and forgotten. Every pipeline step's input and output is a topic. A four-step pipeline carrying full feature vectors writes that payload four times, and generous retention means you store a week of every intermediate tensor. This does not show up on a GPU dashboard and it compounds quietly. Set retention per topic class rather than globally — pipeline inputs and final outputs usually need a longer window than intermediates — and derive the window from the actual investigation period your compliance or data-science teams will use.

Pipelines where a model call would do. The most expensive architectural habit on a Core 2 platform. Pipelines earn their Kafka cost when you need audit, replay, a join across inputs, or conditional routing. A single preprocessing step followed by a single model usually needs none of those and should be a direct model call. Requiring one sentence per pipeline naming the capability that justifies it typically removes a third of the pipelines on a platform that grew organically.

The endpoint with no authentication. Envoy routes on the Seldon-Model header and there is one seldon-mesh Service. Anything that can reach that Service can address any model behind it by setting a header. In the open-source product there is no per-model authorisation in that path. Most installations have an ingress in front and no NetworkPolicy enforcing that the ingress is the only route, which means the boundary is a convention rather than a control. Demonstrate the gap from an unrelated pod with curl once; the fix gets prioritised immediately.

The artifact bucket as a code-execution path. storageUri points at something an agent downloads and an inference server loads, and for several frameworks loading means deserialising. Write access to a model artifact path is therefore equivalent to the ability to run code inside your serving pods. Audit it with the same seriousness as registry write access, and use immutable versioned paths so a silent overwrite is impossible.

The autoscaling pattern that quietly costs you density. The documented HPA approach needs a one-to-one mapping of Models to Servers, so an autoscaled model cannot share a replica with others. Teams adopt it per model without noticing that they are dismantling multi-model serving — the structural reason to run Core 2 at all — one workload at a time. Segment deliberately: high-traffic models get dedicated servers with paired HPAs, the long tail stays packed with no autoscaling.

Core 1 answers applied to Core 2. The v1 line is a different architecture built on SeldonDeployment, still releasing as maintenance — v1.19.0 in January 2026 brought platform maintenance, security hardening, CVE remediation and a move to Python 3.12 — and its CRs are not compatible with Core 2. Most search results and Stack Overflow answers describe it. no matches for kind "SeldonDeployment" is the error that tells you someone followed v1 instructions on a v2 cluster.

Deleting the CRD chart to "clean up". CRD deletion cascades to every Model, Server and Pipeline in the cluster. Uninstall order is the reverse of install order, and the CRD chart goes last. Put this in bold in the runbook.

Platform-ownership habits

Read every release note before every bump, and keep a one-paragraph summary per version. On this platform the release notes are operational documentation, not marketing. 2.10.0 disabled native model autoscaling, added maxShardCountMultiplier so pipeline components can scale past Kafka partition limits, introduced a centralised scaling configuration block in SeldonConfig, added Kafka Schema Registry integration and an OpenAI API translation layer, and put lifecycle probes on all components. 2.10.1 fixed partial scheduling after pod restarts. 2.10.2 fixed pipelines failing to create or delete, gRPC stream blocking, operator reconciliation and MLServer parallel-workers configuration. Knowing that list turns four plausible multi-hour debugging sessions into a version check.

Own the four platform-health panels and nothing else on that dashboard. Requested versus actual replicas per model. Scheduler and operator reconciliation health. Kafka consumer lag on pipeline topics. Agent artifact-fetch failures. Per-model latency charts belong to the tenants; these four belong to you, and a dashboard that mixes them gets read by nobody.

Publish the capability names your servers advertise. A team cannot write a correct requirements list by guessing, and a wrong guess is a ScheduleFailed ticket in your queue. Pin capability names to image versions, so the coupling between artifact and runtime is explicit and a model that needs 1.6.0 behaviour cannot land on an older image by accident.

Price your tenancy tiers from a measured number. Stand up a second SeldonRuntime, measure its fixed pod overhead, and publish the dedicated tier's cost from that figure. Teams then self-select instead of appealing to you, and the number survives a finance conversation in a way an estimate does not.

Maintain a template repository and migrate teams onto it. A Model manifest with a realistic memory declaration, a Server with pinned image and capability names, the paired HPAs for workloads that qualify, and an Argo CD application. It is a morning's work and it permanently removes most of your review burden, because teams that start from a correct template produce correct manifests.

Keep version management yours. Chart versions, MLServer images and the upgrade calendar are platform decisions communicated with notice, not negotiated per team. Pin every chart with --version so an upgrade is deliberate rather than whatever the repository served that morning.

Run game days against the four standing incidents. Break Kafka credentials in staging and time the on-call. Then add whatever they checked first and wrongly to the top of the runbook page, because that is where the time actually goes. Runbooks written from real game days are the ones people open.

Get the licence answered in writing, once. Seldon relicensed Core from Apache 2.0 to the Business Source License and production use may require a commercial licence. Read LICENSE in the repository, send it to legal, and attach their written answer to your platform design document. This is not a question to answer yourself, and it is not one to discover during a procurement review.

Keep data residency in the Kafka decision. If pipeline topics hold regulated personal or financial data at rest for the retention period — which for most pipelines they do, since that is the point — then the region the Kafka cluster lives in is a compliance fact. For Gulf and Egyptian employers the defensible answers are managed Kafka in a Middle East region or self-hosted inside the same cluster. A managed cluster on another continent usually is not, and it is much cheaper to establish that before the platform is built.

Separate the two outage classes in your paging policy, out loud. A scheduler that dies does not stop inference on models that are already loaded, because the data plane is independent of the control plane: new applies stall, so it is a change freeze rather than an outage. A Kafka outage is the inverse — every pipeline stops immediately while direct model calls carry on untouched. Most teams page identically for both, which means either the control-plane page wakes someone for nothing or the Kafka page arrives at the same leisurely priority as a config stall. Write the two severities down with the reasoning attached, so the next person to edit the alerting rules does not collapse them back together.

Instrument the artifact fetch path before you need it. Credential expiry on an artifact bucket does not announce itself; it announces itself as a server pod in CrashLoopBackOff during someone's deploy, usually at the worst possible moment, and the evidence lives in the agent container's log rather than the inference server's. Reading the agent log first is the habit that saves the most time per incident on this platform, and it is worth writing at the top of the crash-loop runbook page in preference to anything cleverer. The same log is where a wrong bucket path, a typo in a versioned artifact prefix and a region mismatch all surface, which makes it the single most information-dense place to look.

Revisit stabilisation windows quarterly, not at install time. Traffic shape drifts as products grow, and the window you chose when the platform had three models is almost certainly wrong once it has eighty. Windows that are too short cause thrash, and thrash on this platform is unusually expensive because every scale-up reloads artifacts from object storage before the replica is useful. Windows that are too long mean you pay for peak capacity through the quiet hours. A calendar reminder with the previous quarter's traffic chart attached is enough process; the failure mode is not getting the number wrong, it is never looking again.

The pull-request checklist

Ordered so the items that cause incidents come first.

On a Model. Is memory measured rather than copied, and does the number look plausible for the artifact? Is replicas satisfiable by a server in this namespace? Does every entry in requirements name a capability that a server here actually advertises? Is storageUri an immutable versioned path rather than a mutable one? Can the artifact be traced to a training run?

On a Server. Is the image pinned, by digest where possible? Is the choice between capabilities and extraCapabilities deliberate, and does the author know that the former replaces the default set? Have the advertised capabilities been tested on this image? Are the pod's requests and limits above the sum of the declared model budgets?

On a Pipeline. Is there one sentence naming the capability — audit, replay, join, conditional routing — that justifies this being a pipeline rather than a direct model call? Is output.steps set, since it is required for synchronous REST and gRPC responses? Are join types deliberate, with joinWindowMs set wherever outer is used? Does the topic retention match the data class?

On autoscaling. Are both HPAs present, one on the Model and one on the Server, with identical metrics, bounds and policies? Is the metric a custom Prometheus metric, given that CPU and memory are unavailable here because multiple HPAs cannot control the same pods? Is the metrics adapter actually installed? Is the scale-up target achievable with capacity that can arrive in time? Are stabilisation windows set for a workload that loads artifacts on startup?

On security. Does an ingress or mesh sit in front of seldon-mesh, with a NetworkPolicy whose selector matches the real Envoy pod labels? Are credentials externally managed rather than inline in manifests or committed Helm values? Are Kafka credentials scoped per tenant, so one tenant cannot read another's prediction data off the topics?

The design-document checklist

Does it say which tenancy tier the workload belongs to, and why? Shared runtime or dedicated, with the SLO and the cost of each named.

Does it name the capability that makes Core 2 the right choice here? If the answer is not multi-model density, a Kafka-backed audit trail, or a genuinely multi-step graph, then the document should compare honestly against KServe, BentoML, or a plain service on Kubernetes before defending the current choice.

Does it state the authentication boundary explicitly? Not "behind the ingress" but which component terminates client identity and what enforces that it is the only route.

Does it account for Kafka? Who operates it, in which region, with what retention, at what monthly cost, and with which credentials scoped to whom.

Does it plan the upgrade path? Pinned chart versions, CRD-first order, the scaling configuration set ahead of 2.11, and a named change window for MLServer bumps.

Does it say what is monitored and who is paged? Specifically, does it include the requested-versus-actual replica alert, and does it distinguish a scheduler outage — a change freeze — from a Kafka outage, which stops every pipeline immediately while leaving direct model calls untouched? Those are two different severities and a document that treats them identically will produce the wrong page at the wrong hour.

Does it answer the licence question with a document rather than an opinion?

The habit underneath all of them

Nearly every item above reduces to one of two things: the scheduler's view of the world, or Kafka's bill. Capability naming, memory declaration, autoscaling, multi-tenancy and partial scheduling are all ways of talking to the scheduler. Pipeline depth, retention, partitions and the audit story are all Kafka. An engineer who keeps those two models clear in their head will diagnose faster, design more cheaply and argue more honestly about this platform than one who knows every field in the CRDs.