The Complete Triton Inference Server Guide

This is part one of three. It takes you from never having heard of Triton Inference Server to serving a model over HTTP, sending it requests from Python, reading its metrics, and fixing the errors that stop most newcomers. By the end you will have a working model repository on your laptop, you will know what every line of its configuration means, and you will be able to explain to a colleague why a team would put Triton between their trained model and their users. Mid-level and Senior take the same ground further; nothing in this part is a teaser.

The guide was checked against Triton Inference Server 2.72.0, shipped as container release 26.08. Triton releases monthly and tags its containers by year and month, so by the time you read this a newer tag may exist. Everything taught here is stable across releases, but always pin the tag you tested with rather than guessing.

Each section ends with a Try it task. Do them as you go. The ideas here stick only once you have watched your own server start, reject a bad request, and start again properly.

What Triton is, and the problem it solves

You have trained a model. On your laptop it is a file, perhaps model.onnx or model.pt, plus a notebook cell that loads it and calls it on some numbers. That is fine for you. It is not fine for a mobile app, a website backend, or another team's batch job, because none of them can import your notebook. They need a network service: something they can send a request to and get a prediction back from, reliably, many times a second.

The obvious first attempt is to write that service yourself. You wrap the model in a small web application, load the file at startup, and call it inside a request handler. This works, and for a lot of projects it is the right answer; the FastAPI guide on this site (/student-guides/fastapi) covers exactly that. But as soon as the traffic grows, the same set of problems appears every time:

• The GPU sits idle. A GPU is fast when it processes many inputs at once and wasteful when it processes one. A hand-written handler takes one request, runs it, returns it, and takes the next. The expensive hardware spends most of its time waiting.
• Every framework needs its own glue. One team has a PyTorch model, another has ONNX, a third has a TensorRT engine, a fourth has a gradient-boosted tree. Four formats mean four services with four sets of conventions for health checks, metrics, and error handling.
• Operations are rebuilt every time. Loading and unloading models, running two versions side by side, exposing metrics, limiting concurrency: each of these is code someone has to write and then maintain.

Triton Inference Server is NVIDIA's answer to those three problems. It is a ready-made server program, not a library. You do not write a web application around your model. You describe your model in a small configuration file, put it in a folder with a particular layout, and start Triton pointing at that folder. Triton loads the model, opens network ports, and answers requests. Along the way it does the things you would otherwise build by hand: it can group requests from different clients into one batch for the GPU, run several copies of a model at once, serve many different model formats from the same process, and publish metrics that a monitoring system can scrape.

The diagram is the whole idea. Everything else in this guide is detail about one of those four boxes.

A word on the name, because it confuses people. Triton was originally called the TensorRT Inference Server, and you will still see old blog posts using that name. It now serves many kinds of models, not only TensorRT ones, and the name was changed accordingly. Do not confuse it with the Triton language and compiler for writing GPU kernels, which is an unrelated project that happens to share the name.

It also helps to know where Triton sits among its neighbours. If you want a general-purpose Python service where you control every line, FastAPI is the lighter choice. If your whole team lives in PyTorch and wants a PyTorch-specific server, there is TorchServe (/student-guides/torchserve). If you are deploying large language models and want that single job done as well as possible, vLLM (/student-guides/vllm) and TensorRT-LLM (/student-guides/tensorrt-llm) are specialised engines, and Triton can host both of them as backends. Triton's particular strength is being the one server that speaks to all of these formats at once, which is why managed cloud services such as SageMaker (/student-guides/sagemaker) and Vertex AI (/student-guides/vertex-ai) use it as a serving runtime.

For readers in the Gulf and Egypt, one practical point deserves mention early. Triton runs wherever you can run a container, which means it runs inside your own data centre or in a cloud region of your choosing. Employers in finance, government, and healthcare often require that model inputs, which can be customer data, never leave the country. A self-hosted inference server satisfies that requirement in a way a third-party prediction API cannot.

The mental model: four nouns

Triton has a lot of features, but a beginner needs only four ideas. Learn these and the documentation stops looking like a wall.

The model repository. This is a folder, on local disk or in cloud storage, that holds everything Triton should serve. Each model gets its own subfolder, named after the model. Inside that subfolder sits one configuration file and one or more numbered version folders containing the actual model file. The folder layout is not a suggestion; Triton finds your model by reading the structure, so spelling and nesting matter.

The backend. A backend is the piece of code inside Triton that knows how to run one kind of model. There is a backend for ONNX Runtime, one for TensorRT, one for PyTorch's TorchScript format, one for OpenVINO, one for tree-based models such as XGBoost, and one that runs arbitrary Python code that you write. When you tell Triton "this model uses the onnxruntime backend", you are telling it which engine to hand the model file to. Triton itself does not understand your neural network; the backend does.

The model configuration. Each model has a file called config.pbtxt. It is written in a text format called protobuf text, which looks a little like JSON without commas between fields. It declares the model's name, which backend runs it, what inputs it expects (names, data types, shapes), what outputs it returns, and how Triton should run it (how many copies, whether to batch). The configuration is the contract between your model and the world: clients must send inputs exactly as it describes.

The instance. A running copy of a model is called an instance. By default Triton makes one instance per model on each GPU, or one on the CPU if there is no GPU. You can ask for more. Because each instance can work on a request independently, two instances can process two requests at the same moment.

Two more terms will come up. A protocol is the language clients use to talk to the server. Triton speaks HTTP with JSON bodies and gRPC, a faster binary protocol, and both follow a community specification called the KServe v2 inference protocol. This matters because it means a client written for one KServe-compatible server largely works against another. A tensor is simply a multi-dimensional array of numbers with a fixed data type, and it is the unit Triton sends and receives. Your image is a tensor of shape three by 224 by 224. Your sentence, after tokenising, is a tensor of integers. When Triton documentation says "input", it means a named tensor.

Finally, remember three port numbers, because you will type them constantly. Port 8000 serves HTTP, port 8001 serves gRPC, and port 8002 serves metrics in a format the Prometheus monitoring system understands.

Installing and checking your setup

Triton's supported way to run is as a container image published on NVIDIA's NGC registry, so installing Triton mostly means installing Docker. If containers are new to you, read the Docker guide (/student-guides/docker) first; you only need images, docker run, port mapping, and volume mounts, all of which it teaches.

On Linux with an NVIDIA GPU, install Docker and the NVIDIA Container Toolkit, which lets containers see the GPU. Your host also needs an NVIDIA driver recent enough for the CUDA version inside the container. The 26.08 container is built on CUDA 13.4.1, so check the release notes for the minimum driver, and expect that very old GPUs are no longer supported by current containers. Turing-generation cards (compute capability 7.5) and newer are the safe baseline.

On a Mac or a laptop without an NVIDIA GPU, there is no official native Triton server for macOS. You can still learn everything in this guide, because Triton runs on the CPU too. Run the same container under Docker Desktop or Colima, which gives you a Linux virtual machine. You will not get GPU acceleration, and the tag you pull must have a build for your processor architecture, which is worth confirming on the NGC page when you pull. The Python client library installs and runs natively on a Mac.

On Windows, the practical route is WSL2 (Windows Subsystem for Linux) with Docker, then following the Linux steps. Official native Windows support is limited, so do not plan around it.

Pull the image:

docker pull nvcr.io/nvidia/tritonserver:26.08-py3

The image is large, several gigabytes, because it bundles CUDA libraries and several inference engines. Expect the first pull to take a while. The -py3 suffix means the full image with all common backends; NVIDIA also publishes slimmer variants for specific uses, and a separate -py3-sdk image that contains client programs and the Perf Analyzer benchmarking tool.

If you are on a machine with a company network that inspects TLS traffic, a pull may fail with a certificate error. That is a trust problem with your network, not with Triton; add your organisation's certificate to your container runtime rather than disabling verification.

Install the Python client library on your own machine too, so you can send requests without entering a container:

python -m venv .venv
source .venv/bin/activate
pip install "tritonclient[http]" numpy

The square-bracket part chooses which protocols to install. [http] is enough for this guide, [grpc] adds gRPC, and [all] installs both.

Checking the install is easiest if you start the server once with an empty repository, to prove the container runs:

mkdir -p models
docker run --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models

Break that command down, because you will reuse it all guide. --rm deletes the container when it stops. Each -p publishes a port from the container to your machine. -v "$PWD/models:/models" mounts your local models folder inside the container at /models, so Triton sees your files and you can edit them from outside. The image name and tag come next, and then the command to run inside the container: tritonserver with the flag --model-repository=/models telling it where to look. On a machine with a working GPU setup you would add --gpus=1 right after docker run to give the container one GPU; leave it off to run on the CPU.

With an empty folder, Triton should start, print a table of loaded models that is empty, and keep running. In a second terminal, ask whether it is ready:

curl -v localhost:8000/v2/health/ready

Look for HTTP/1.1 200 OK in the output. A 200 status means the server is up and every model it was asked to load is ready. There is also /v2/health/live, which only says that the process is running. The distinction matters later: "live" means do not restart it, "ready" means it is safe to send traffic. Stop the server with Ctrl+C in its terminal.

Your first model, built step by step

To keep this first project small and free of downloads, we will serve a model written in Python: a function that adds two vectors of four numbers. It is deliberately trivial. The point is not the arithmetic; it is the shape of everything around it, which is identical for a real neural network. Later you will swap the Python logic for an ONNX file and nothing else about the workflow changes.

Create the folder structure. The model is named add_vectors, and it has one version, numbered 1:

mkdir -p models/add_vectors/1

The result must look like this:

models/
  add_vectors/
    config.pbtxt
    1/
      model.py

Three rules about this layout. The folder name under models/ is the model's name, and clients will use it in URLs. The version folder must be an integer; a folder named latest or v1 is silently ignored. And the Python backend expects the file inside the version folder to be called model.py.

Now write the configuration:

name: "add_vectors"
backend: "python"
max_batch_size: 8

input [
  {
    name: "INPUT0"
    data_type: TYPE_FP32
    dims: [ 4 ]
  },
  {
    name: "INPUT1"
    data_type: TYPE_FP32
    dims: [ 4 ]
  }
]

output [
  {
    name: "OUTPUT0"
    data_type: TYPE_FP32
    dims: [ 4 ]
  }
]

instance_group [
  {
    count: 1
    kind: KIND_CPU
  }
]

Read it line by line. name must match the folder name. backend: "python" selects the backend that runs your model.py. max_batch_size: 8 says Triton may hand this model up to eight requests stacked together; we will explain the consequences of this number in a moment, because it changes how dims is read. Each entry in input gives a tensor name, a data type (TYPE_FP32 is a 32-bit float), and dims, the shape of one example. The output block does the same for what the model returns. Finally instance_group asks for one copy of the model running on the CPU.

That max_batch_size detail is the single most common point of confusion for newcomers, so here it is plainly. When max_batch_size is greater than zero, Triton adds an invisible leading dimension for the batch. You write dims: [ 4 ] meaning "one example is four numbers", and a client sends a tensor of shape [1, 4] for one example or [5, 4] for five. When max_batch_size is zero, batching is off, Triton adds nothing, and your dims must describe the complete shape the client sends, batch dimension included. Mixing these up gives shape errors, covered in the errors section.

Now the Python code. A Python-backend model is a class called TritonPythonModel with up to three methods Triton knows how to call:

import numpy as np
import triton_python_backend_utils as pb_utils


class TritonPythonModel:
    def initialize(self, args):
        # Runs once when the model loads. Load weights or open
        # resources here, not inside execute().
        self.model_name = args["model_name"]

    def execute(self, requests):
        # Runs for each group of requests Triton hands us.
        # We must return exactly one response per request, in order.
        responses = []
        for request in requests:
            a = pb_utils.get_input_tensor_by_name(request, "INPUT0").as_numpy()
            b = pb_utils.get_input_tensor_by_name(request, "INPUT1").as_numpy()
            out = pb_utils.Tensor("OUTPUT0", (a + b).astype(np.float32))
            responses.append(pb_utils.InferenceResponse(output_tensors=[out]))
        return responses

    def finalize(self):
        # Runs once when the model unloads. Free anything you opened.
        pass

The module triton_python_backend_utils exists only inside the Triton container; you cannot import it on your laptop, so do not worry if your editor underlines it. initialize is where expensive setup belongs, because Triton calls it once, not per request. execute receives a list of requests, and the rule that trips people up is that you must return a list of responses with the same length and order. finalize is cleanup.

Notice what the code does not contain: no web framework, no ports, no JSON parsing, no health endpoint. Triton supplies all of that. Your code is only the model.

Start the server, this time with the folder populated:

docker run --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models

Watch the startup output. Near the end Triton prints a table with columns for model, version, and status. You are looking for this line:

| add_vectors | 1       | READY  |

READY means the model loaded and can serve. If you see UNAVAILABLE with a reason in the last column, the reason is the thing to read; later sections explain the usual ones. Triton also prints the HTTP and gRPC endpoints it started and the port the metrics endpoint listens on. In the second terminal check the model itself:

curl localhost:8000/v2/models/add_vectors/ready
curl localhost:8000/v2/models/add_vectors

The first returns an empty 200 response when the model is ready. The second returns a JSON description, including the name, the versions, and the input and output tensors as Triton understands them. Compare it with your config.pbtxt; seeing your own configuration echoed back is a good confirmation that Triton read what you meant.

Sending your first request

A server is only useful once someone calls it. Triton's HTTP interface is plain JSON, so you can use curl. A request to run inference is a POST to /v2/models/<name>/infer, with a body listing each input by name, shape, data type, and flattened data:

curl -s -X POST localhost:8000/v2/models/add_vectors/infer \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [
      {"name": "INPUT0", "shape": [1, 4], "datatype": "FP32", "data": [1, 2, 3, 4]},
      {"name": "INPUT1", "shape": [1, 4], "datatype": "FP32", "data": [10, 20, 30, 40]}
    ]
  }'

Notice the details. The shape is [1, 4], one example of four numbers, because batching is enabled and the first dimension is the batch. The data type is written FP32, without the TYPE_ prefix used in config.pbtxt; the config uses the protobuf enum names and the wire protocol uses the short names. The data is a flat list in row-major order, and its length must equal the product of the shape.

The response looks like this, with some fields trimmed:

{
  "model_name": "add_vectors",
  "model_version": "1",
  "outputs": [
    {"name": "OUTPUT0", "datatype": "FP32", "shape": [1, 4], "data": [11.0, 22.0, 33.0, 44.0]}
  ]
}

That is your model, answering over the network. You wrote about twenty lines of Python and a config file.

Sending JSON from the shell is awkward for anything real, particularly images or large arrays, so most programs use the client library. Here is the same call in Python:

import numpy as np
import tritonclient.http as httpclient

client = httpclient.InferenceServerClient(url="localhost:8000")

a = np.array([[1, 2, 3, 4]], dtype=np.float32)
b = np.array([[10, 20, 30, 40]], dtype=np.float32)

inputs = [
    httpclient.InferInput("INPUT0", a.shape, "FP32"),
    httpclient.InferInput("INPUT1", b.shape, "FP32"),
]
inputs[0].set_data_from_numpy(a)
inputs[1].set_data_from_numpy(b)

result = client.infer("add_vectors", inputs)
print(result.as_numpy("OUTPUT0"))

Run it with python client.py and you should see [[11. 22. 33. 44.]]. Look at how the code maps to the concepts. An InferInput is one named input tensor, built from a name, a shape, and a data type string. set_data_from_numpy loads your array into it, and it checks that the array's dtype agrees with the declared data type, which is a helpful early error. client.infer sends the request, and as_numpy("OUTPUT0") extracts one output as a NumPy array by name.

The most important point here is a mistake worth making deliberately. Change INPUT1 to be shaped [1, 5] and rerun. The server rejects the request with a message saying the shape of the input does not match what the model expects. Triton validates every request against the configuration before your code ever runs. That is a feature: bad requests never reach your model, and the error tells the caller exactly what was expected.

There is also a gRPC client, tritonclient.grpc, with almost the same API, which connects to port 8001. gRPC is generally a better choice for high request rates and for large tensors, because it is a compact binary protocol. HTTP is easier to debug with curl. Start with HTTP, switch if profiling says to.

Understanding the model configuration

You have written one config.pbtxt. Now it is worth knowing the full vocabulary, because the configuration file is the control panel for everything Triton does.

Names and types. name identifies the model. Data types are written TYPE_BOOL, TYPE_UINT8, TYPE_INT32, TYPE_INT64, TYPE_FP16, TYPE_FP32, TYPE_FP64, and TYPE_STRING for text and byte strings. Choose the type your model was exported with; a type mismatch between client and config is rejected, not silently converted.

Dimensions and variable sizes. In dims, a value of -1 means "any size on this axis". A text model might declare dims: [ -1 ] for a sequence of token ids of unknown length. You can use -1 on any axis, but be careful: the underlying model must genuinely support variable sizes there, which many exported ONNX graphs only do if they were exported with dynamic axes.

Batching. Batching is how a GPU earns its money, so understand it properly. A GPU runs one large computation much more efficiently than many small ones, so processing eight examples together can cost little more than processing one. max_batch_size declares the largest batch the model accepts. Setting it above zero does two things: it adds the invisible batch dimension we met earlier, and it permits Triton to do the stacking for you. By itself it does not make Triton wait to collect requests; if clients already send batches, they are passed through. To have Triton combine separate requests from separate clients into one execution, you add the dynamic batcher:

dynamic_batching { }

With that empty block Triton will form batches opportunistically from whatever requests are queued while the model is busy. You can tune it with a list of preferred_batch_size values and a max_queue_delay_microseconds, the longest Triton will wait for more requests to fill a batch before running what it has. That delay is a deliberate trade of a little latency for more throughput. The documentation's own example shows eight concurrent clients going from roughly 73 to roughly 272 inferences per second after enabling it, though your numbers depend entirely on your model and hardware. Treat that as a hint that it is worth trying, not a promise. For a beginner, the lesson is only that the knob exists and where it lives; you will tune it properly in the mid-level guide.

If your model cannot be batched at all, set max_batch_size: 0. Then dims must include the batch axis and Triton will never stack or split requests.

Instance groups. The instance_group block says how many copies of the model to run and where. Two examples:

instance_group [ { count: 2 kind: KIND_GPU } ]

instance_group [ { count: 1 kind: KIND_CPU } ]

KIND_GPU places copies on GPUs, KIND_CPU on the CPU. If you omit the block entirely, Triton uses one instance per available GPU, or one CPU instance when there is none. Two instances on the same GPU let it overlap the work of two requests, which helps when a single instance cannot keep the card busy, but each copy consumes GPU memory, so more is not automatically better.

Version policy. By default Triton serves only the latest version it finds (the highest-numbered folder). The version_policy field changes this:

version_policy: { latest: { num_versions: 2 } }

version_policy: { specific: { versions: [ 1, 3 ] } }

version_policy: { all: { } }

This lets you keep an old version serving next to a new one so that you can roll back without redeploying, and clients can address a version in the URL with /v2/models/<name>/versions/<n>/infer.

Warmup. The first request to a freshly loaded model is often slow, because engines allocate memory and compile kernels lazily. A model_warmup block makes Triton run sample requests at load time, so the model is genuinely ready before it reports ready. You will want this once you serve real models; for now it is enough to know it exists.

Inspecting the real configuration. Triton may modify your configuration while loading it, for example by filling in fields it can infer. To see what is actually in effect, ask the server:

curl localhost:8000/v2/models/add_vectors/config

When behaviour surprises you, compare this output to the file you wrote. A difference between them is often the whole explanation.

Serving real model files: backends and file names

The Python backend is excellent for learning and for glue code, but most of what Triton serves in practice are exported model files executed by an optimised engine. The mechanics are the same as before; the version folder holds a model file instead of model.py, and the config names a different backend.

Triton looks for a default file name in the version folder, depending on the backend:

• ONNX Runtime: model.onnx
• TensorRT: model.plan
• TorchScript (PyTorch): model.pt
• OpenVINO: model.xml together with model.bin
• Python: model.py
• DALI (a data-preprocessing library): model.dali

If your file has a different name, set default_model_filename in the config. If you export from PyTorch, remember that Triton's PyTorch backend runs TorchScript files, produced by tracing or scripting a model, rather than a plain saved state dictionary.

Here is what a complete ONNX model looks like, using the image classifier from NVIDIA's own quickstart, called densenet_onnx:

models/
  densenet_onnx/
    config.pbtxt
    1/
      model.onnx

name: "densenet_onnx"
backend: "onnxruntime"
max_batch_size: 8
input [
  { name: "data_0" data_type: TYPE_FP32 dims: [ 3, 224, 224 ] }
]
output [
  { name: "fc6_1" data_type: TYPE_FP32 dims: [ 1000 ] }
]

Read the dimensions as you now can: one example is a three-channel image 224 pixels square, and the output is a vector of 1,000 class scores. The tensor names data_0 and fc6_1 are not arbitrary. They are the names baked into the ONNX file, and the config must use them exactly. If you are not sure what names your own file contains, open it in a viewer such as Netron, or inspect it with the onnx Python package, before writing the config. For ONNX the configuration can in fact be auto-completed from the file, but writing it out documents the contract.

NVIDIA's quickstart provides example models, including this classifier, through a script named fetch_models.sh in the docs/examples folder of the Triton server repository on GitHub. Cloning that repository and running the script is the quickest way to get a real model to experiment with; check the repository for the current location of the script, since folders occasionally move. Mount the resulting model_repository folder at /models exactly as before, and the server will load it.

Then, from the SDK container, you can classify an image without writing any client code:

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.08-py3-sdk
/workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

The flags select the model, ask for the top three classes, and choose an image-scaling convention. The output lists class scores with labels. --net=host lets the client container reach the server on localhost; this works as written on Linux, while on Docker Desktop you would instead point the client at the host address.

One backend deserves a warning at this stage: TensorRT. A TensorRT model.plan file is not a portable model format like ONNX. It is an engine compiled for a specific TensorRT version and a specific GPU architecture. A plan built on one machine will often refuse to load on another, or in a container with a different TensorRT version. If you use TensorRT, build plans inside the same container release you will serve with, on the same kind of GPU you will serve on. Triton upgrades its container monthly and bumps TensorRT along with CUDA, so plans must be rebuilt as part of upgrading. ONNX and TorchScript files do not have this limitation, which makes them the friendlier starting point.

One more feature is worth knowing exists: ensembles and business logic scripting. Real systems rarely call a lone model. They resize an image, run a classifier, and decode the result. Triton lets you declare such a pipeline in configuration (an ensemble) or code it in the Python backend (business logic scripting), so that the client makes one call and Triton wires the steps together on the server. You do not need either yet. When your client starts doing preprocessing that belongs next to the model, you will recognise the need.

Running the server: flags, logs, and model control

Until now we have used a single flag, --model-repository. A few more are worth knowing on day one, because they change how Triton behaves when things go wrong.

Where the models come from. --model-repository can be passed more than once to combine several repositories, as long as model names are unique across them. It also accepts cloud paths such as s3://bucket/path for Amazon S3, gs://bucket/path for Google Cloud Storage, and as://account/container/path for Azure. Cloud credentials come from the standard environment variables of each provider, for example the AWS access key variables or GOOGLE_APPLICATION_CREDENTIALS. Where you can, prefer a role or workload identity attached to the machine over long-lived keys pasted into environment variables.

What happens at startup failure. By default Triton exits if any model fails to load; this is the --exit-on-error flag, which defaults to true. That is a safe default for production, because a half-broken service should not accept traffic. While developing it can be annoying, since one broken model stops all the others. Set --exit-on-error=false and Triton will keep running, mark the failed model UNAVAILABLE in its table with a reason, and serve the rest. Related is --strict-readiness, true by default: the readiness endpoint reports not ready while any model has failed.

Model control mode. This setting decides how Triton reacts to changes in the repository after it starts. There are three values:

• --model-control-mode=none is the default. Triton loads everything at startup and ignores later changes. To change a model, you restart the server.
• --model-control-mode=poll makes Triton check the repository periodically, controlled by --repository-poll-secs, and reload what changed. The documentation advises against it for production use.
• --model-control-mode=explicit loads nothing unless told to, either with --load-model=<name> at startup or by calling the repository API later.

For explicit mode, the API is a pair of POST requests:

curl -X POST localhost:8000/v2/repository/models/add_vectors/unload
curl -X POST localhost:8000/v2/repository/models/add_vectors/load

and POST /v2/repository/index lists every model Triton can see, with its state. A model passes through the states UNAVAILABLE, LOADING, READY, and UNLOADING. Explicit mode is a handy way to experiment, since you can change a file and reload a single model without restarting the whole server. The security-minded advice in the official deployment guide is to run production servers in the default mode, because letting clients load models is letting them run code. We return to that in the senior guide.

Logging. Triton logs to the terminal by default. --log-verbose=1 turns on verbose logging, which is noisy but invaluable when a model refuses to load and the normal message is not enough. --log-file writes to a file. When a problem is mysterious, add --log-verbose=1, reproduce it once, and read the lines around the failure.

Switching protocols off. --allow-http=false and --allow-grpc=false disable the corresponding endpoints. Reducing what is exposed is good hygiene once you know which protocol your clients use.

A complete invocation using several of these might look like this:

docker run --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models \
               --model-control-mode=explicit \
               --load-model=add_vectors \
               --exit-on-error=false \
               --log-verbose=1

Another Docker detail will matter as soon as you use the Python backend with larger tensors: containers get a small shared-memory area by default, and the Python backend uses shared memory to pass data between processes. If you see crashes or "Bus error" messages under load, give the container more with --shm-size=1g (or more) on the docker run line.

Watching it work: health, metadata, and metrics

A service you cannot observe is a service you cannot trust. Triton ships with three observation tools, all reachable without writing code.

Health endpoints. You met /v2/health/live and /v2/health/ready. They return empty bodies with a status code, which is exactly what orchestration systems want: Kubernetes (/student-guides/kubernetes) readiness and liveness probes can point straight at them. For a particular model, /v2/models/<name>/ready answers the same question.

Metadata. GET /v2 describes the server (name, version, extensions it supports), and GET /v2/models/<name> describes a model's tensors as Triton understands them. A client can fetch the metadata at startup and build its requests from it, which is how generic tooling talks to arbitrary models.

Metrics. Port 8002 serves metrics in the Prometheus text format:

curl localhost:8002/metrics

You will get a long list. Do not read it all; learn a handful. Send a few requests first, then look for these:

• nv_inference_request_success and nv_inference_request_failure count requests that worked and requests that failed, per model.
• nv_inference_count counts individual inferences (each example in a batch counts), and nv_inference_exec_count counts model executions. If requests are batched well, the first is noticeably larger than the second. Dividing them gives the average batch size, which is a quick test of whether batching is doing anything.
• nv_inference_request_duration_us is the total time spent on requests, and nv_inference_queue_duration_us is the part spent waiting in line. If queue time dominates, the model is the bottleneck, not the network.
• nv_gpu_utilization and nv_gpu_memory_used_bytes describe the GPU when there is one.

The durations are cumulative counters in microseconds, so a single reading means little. A monitoring system such as Prometheus (/student-guides/prometheus), with dashboards in Grafana (/student-guides/grafana), turns them into rates over time. For a beginner the point is simply to know that the data is there, free, from the first moment you run the server.

Triton also includes a benchmarking tool called Perf Analyzer, shipped in the SDK image. It sends synthetic load at a model and reports throughput and latency percentiles:

perf_analyzer -m add_vectors --concurrency-range 1:4

Run it from the SDK container with host networking, as above. It is the honest way to answer "how fast is it?", because it measures the server under load rather than a single request. Compare the numbers before and after you change a configuration setting such as dynamic batching, and you have a method for improving performance that rests on measurement instead of guessing. Keep in mind that the trivial adder will be limited by the Python backend and the network, not by a GPU, so use a real model when you care about the result.

Reading errors and fixing the common ones

Most of your first hours with Triton will be spent reading error messages. They are quite good once you know where to look. The first rule is to read the startup output from the top, not the bottom: the last line of a failed start is usually a generic summary, and the cause is several lines above it.

"creating server: Internal - failed to load all models." This is the generic summary printed when any model failed to load and --exit-on-error is true. Scroll up. Immediately before it Triton prints a line per failing model saying which model and why. Fix that one, or use --exit-on-error=false while you investigate.

The model shows UNAVAILABLE. The table's last column gives the reason. Common reasons are a missing model file (the file name does not match the backend's default, such as model.onnx.bak or a file placed in the model folder instead of the version folder), a version folder with a non-numeric name, a tensor name in the config that does not exist in the model, or a Python import error inside initialize. The debugging guide's advice is worth following as a habit: before blaming Triton, confirm that the model runs in its own framework outside Triton. If it fails to run in ONNX Runtime directly, no config will fix it.

A batch-size message at load time, such as "unexpected configuration maximum batch size ... model maximum is 1". This happens most with TensorRT engines and some ONNX files that were exported with a fixed batch dimension. The model itself cannot batch, but your config claimed it could. Either set max_batch_size: 0 and put the full shape in dims, or re-export the model with a dynamic batch axis.

A request error saying the shape does not match. The message names the input, the expected shape, and the shape you sent. With batching enabled, remember that the first dimension is the batch and is not part of dims. Nearly every shape error from beginners is the max_batch_size confusion from the first-project section.

A request error naming an unexpected input. The tensor name in your client does not match the config; the message lists the allowed names. Names are case-sensitive.

A batch-size limit error. You sent more examples in one request than max_batch_size. Either send fewer or raise the limit if the model supports it.

"Request for unknown model." Either the name is misspelled in the URL or client call, or the model failed to load. Check /v2/repository/index or the startup table.

CPU-only complaints when loading a GPU model. If you start the container without --gpus but the model's config places instances on KIND_GPU, or the model uses a GPU-only backend like TensorRT, it cannot load. Either give the container a GPU or set instance_group to KIND_CPU and use a CPU-capable backend.

An engine incompatible with this TensorRT version. A TensorRT plan built elsewhere or by another release will not deserialise. Rebuild it inside the container you serve with.

"NVIDIA driver too old" style failures. The host driver must meet the CUDA requirement of the container. Updating the driver, or pulling an older container tag that matches it, fixes this. The release notes list the requirement.

Health returning a non-200 while the server runs. The server is alive but not ready: a model is still loading, or a model has failed and strict readiness is on. Large models can take minutes to load, which is why production systems use a generous startup probe.

Bus error or shared-memory failures with the Python backend. Increase --shm-size on docker run.

Permission denied reading the model folder. The container user cannot read files that were created with restrictive permissions on your machine. Make the repository readable.

All the exact message wordings can differ slightly between releases, so match on the meaning, not the characters. When the message is unclear, run with --log-verbose=1 and reproduce once.

Putting it all together

Let us build one small project end to end, using everything above. The goal: a text-length model served by Triton, a client that calls it, a metrics check, and a script that starts it all, so a teammate can reproduce your work with two commands. We will reuse the Python backend so that nothing needs downloading. The model takes a list of strings and returns each string's length.

Create the layout:

triton-demo/
  models/
    text_length/
      config.pbtxt
      1/
        model.py
  client.py
  run.sh

The configuration declares a string input of variable length and an integer output. Because strings are variable-sized byte sequences, the type is TYPE_STRING:

name: "text_length"
backend: "python"
max_batch_size: 16

input [
  { name: "TEXT" data_type: TYPE_STRING dims: [ 1 ] }
]
output [
  { name: "LENGTH" data_type: TYPE_INT32 dims: [ 1 ] }
]

dynamic_batching { }

instance_group [ { count: 1 kind: KIND_CPU } ]

Each example is one string, so dims is [ 1 ], and the client sends shape [batch, 1]. The dynamic batcher is on, so concurrent requests can be combined.

import numpy as np
import triton_python_backend_utils as pb_utils


class TritonPythonModel:
    def execute(self, requests):
        responses = []
        for request in requests:
            text = pb_utils.get_input_tensor_by_name(request, "TEXT").as_numpy()
            lengths = np.vectorize(len)(text).astype(np.int32)
            out = pb_utils.Tensor("LENGTH", lengths)
            responses.append(pb_utils.InferenceResponse(output_tensors=[out]))
        return responses

A string tensor arrives in the Python backend as a NumPy array of bytes objects, and len of a bytes value is the byte count. For non-ASCII text such as Arabic, byte length differs from character count, since Arabic letters take two bytes in UTF-8. If you wanted characters you would decode first; the example is a good reminder that data types have consequences.

The client uses the object type for strings:

import numpy as np
import tritonclient.http as httpclient

client = httpclient.InferenceServerClient(url="localhost:8000")

texts = ["hello", "triton", "مرحبا"]
data = np.array(texts, dtype=object).reshape(-1, 1)

inp = httpclient.InferInput("TEXT", data.shape, "BYTES")
inp.set_data_from_numpy(data)

result = client.infer("text_length", [inp])
for text, length in zip(texts, result.as_numpy("LENGTH").flatten()):
    print(f"{text!r}: {length} bytes")

The wire type for strings is BYTES, not STRING; remember it as another case where the config and the protocol differ in spelling. The script prints each word with its byte length, and you will see the Arabic word reports ten bytes for five characters.

Wrap it so a teammate does not need to remember the long docker run:

#!/usr/bin/env bash
set -euo pipefail

docker run --rm \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models

Now exercise the whole thing. In one terminal run bash run.sh. Wait for the READY row in the startup table. In another, run curl localhost:8000/v2/health/ready, then python client.py, then curl localhost:8002/metrics | grep text_length to find the counters for your model. Every step you took in this guide appears here: layout, configuration, a Python-backend model, a client, health, metrics, and a pinned container tag. Commit the folder to Git, and anyone on your team can reproduce it, which is precisely the benefit of describing a service as configuration. If you later containerise your own serving image or deploy to Kubernetes, the model repository folder you just wrote moves unchanged.

What you can now do, and what comes next

You started this guide with a model file and no way for anyone else to use it. You can now explain what Triton is for, in terms of idle GPUs, format sprawl, and rebuilt operations. You can name the four nouns, the repository, the backend, the configuration, and the instance, and use them to locate a problem. You can pull and run the container, lay out a model repository, write a config.pbtxt with the right dimension convention, serve both a Python model and an exported file, call it from curl and from the Python client, watch its health and metrics, and read the startup errors that stop most first attempts.

What this guide did not do is make you an operator. You have not tuned batching against real traffic, built an ensemble pipeline, served a large language model, deployed to Kubernetes, protected the server on a network, or planned upgrades across the monthly release train. Those are the next two levels.

The mid-level guide covers how the scheduler and dynamic batcher really work and how to choose instance counts, how to accelerate ONNX models with TensorRT, how to build ensembles and business logic scripts, and how to benchmark with Perf Analyzer and Model Analyzer. The senior guide covers security (Triton has no built-in authentication and must sit behind a gateway), multi-tenancy, scaling, upgrades, cost, and incident response.

Beyond this series, natural neighbours on this site are the Docker guide (/student-guides/docker) for the container skills Triton leans on, the KServe guide (/student-guides/kserve) for running Triton on Kubernetes through a standard serving layer, the BentoML guide (/student-guides/bentoml) and the Ray Serve guide (/student-guides/ray-serve) for alternative serving approaches that you can compare against, the TorchServe guide (/student-guides/torchserve) for a PyTorch-specific alternative, and the MLflow guide (/student-guides/mlflow) for tracking the models you eventually put in the repository.

One last piece of advice. Keep a small, working example like the one in this guide somewhere you can run in five minutes. When a colleague asks "how would we serve this?", answering with a running server and a twenty-line client is far more persuasive than any slide.

Sources

• Triton Inference Server documentation
• Release notes index
• Release 26.08 notes
• Quickstart
• Model repository
• Model configuration
• Model management
• Optimization and Perf Analyzer
• Metrics
• Debugging guide
• Secure deployment considerations
• Triton server repository and issues
• NGC tritonserver container catalogue

This is the interview companion to the beginner Triton guide. It assumes you have read that guide and built the small model repository in it. The aim is not to memorise phrases but to be able to explain, calmly and in your own words, what Triton is, why a team would use it, and how to get it running and unstuck. Interviewers for entry-level MLOps roles rarely expect deep tuning knowledge. They want to see that you understand the problem the tool solves, that you have actually run it, and that you know the first things to check when it misbehaves.

The thirty-second answer

Triton Inference Server is NVIDIA's open-source server for running trained models behind a network API. You put each model in a folder with a small configuration file, start Triton pointing at the folder, and it serves the models over HTTP and gRPC, with batching, multiple model formats, and Prometheus metrics built in, so you do not have to write a custom web service around each model.

If you get a follow-up asking "why not just use FastAPI", the short answer is that a hand-written service works fine at small scale, but Triton gives you server-side batching for GPU efficiency, one consistent interface across ONNX, TensorRT, PyTorch and others, and operational features such as health checks and metrics without extra code. At low traffic with one framework, the hand-written route is often the simpler choice, and it is good to say so; it shows judgement.

The model you should be able to draw

Be ready to sketch the structure on a whiteboard in under a minute. Four nouns carry the whole picture.

The model repository is a folder, local or in cloud storage, with one subfolder per model. Inside each model folder there is a config.pbtxt and one or more integer-numbered version folders holding the model file. The backend is the engine that executes a model type: ONNX Runtime, TensorRT, PyTorch for TorchScript, OpenVINO, a Python backend for custom code, and others. The configuration declares the model's name, its backend, its input and output tensors with names, types, and shapes, and how it should run. An instance is a running copy of the model, on a GPU or the CPU, and more instances mean more requests handled at once.

Then add the request path: a client sends a request over HTTP on port 8000 or gRPC on port 8001, Triton validates it against the configuration, may queue and batch it with other requests, hands it to the backend, and returns the result. Port 8002 exposes metrics.

If you can draw that and name the ports, you have covered most of what a beginner question can ask. The words to use for the protocol are worth remembering too: both interfaces follow the KServe v2 inference protocol, which is a community standard, so clients and tooling are not locked to Triton.

The commands to have ready

An interviewer who asks "how would you start Triton" wants to see that you have done it. Know this shape cold:

docker run --rm --gpus=1 -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v "$PWD/models:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models

Be able to explain each piece: the GPU flag (omit it for CPU-only), the three published ports, the volume mount that exposes your model repository at /models inside the container, the pinned image tag, and the tritonserver command with the repository flag. Explain why you pin the tag: Triton releases monthly and each container changes CUDA, TensorRT and other library versions, so an unpinned tag can change behaviour under you.

For health and inspection, know these:

curl -v localhost:8000/v2/health/ready
curl localhost:8000/v2/models/<name>
curl localhost:8000/v2/models/<name>/config
curl localhost:8002/metrics

The first returns 200 when the server and all its models are ready, the second describes a model's tensors, the third shows the configuration actually in effect, and the fourth lists metrics.

For inference, know the shape of a request body. It is JSON with an inputs list, and each input has a name, a shape, a datatype such as FP32, and flat data. The URL is /v2/models/<name>/infer. And know the Python client in outline: create an InferenceServerClient, build InferInput objects with a name, shape and datatype string, call set_data_from_numpy, call infer, and read an output with as_numpy.

You should also be able to write a minimal config from memory:

name: "mymodel"
backend: "onnxruntime"
max_batch_size: 8
input [ { name: "x" data_type: TYPE_FP32 dims: [ 3, 224, 224 ] } ]
output [ { name: "y" data_type: TYPE_FP32 dims: [ 1000 ] } ]

No one expects the exact tensor names, but they will expect you to say that the names must match the model file.

The classic traps and why they happen

Interviewers love to ask about mistakes because mistakes show whether you have really used a tool. These are the ones to know, each with its cause.

Batch dimension confusion. With max_batch_size above zero, dims leaves out the batch axis and clients send an extra leading dimension. With max_batch_size: 0 the dims are the full shape. Nearly every first-time shape error is this. A strong answer states the rule and gives the consequence in one sentence: if dims is [4] and batching is on, the client sends [1, 4].

Model file naming and layout. Triton finds models purely by structure. A version folder named anything other than an integer is ignored, and the model file must have the backend's default name, such as model.onnx, model.pt, model.plan, or model.py, unless default_model_filename overrides it. The usual result of getting it wrong is a model that is missing or UNAVAILABLE.

Tensor names that do not match the file. The config's input and output names must be the ones inside the model. A renamed tensor causes a load failure. The fix is to inspect the model, with a viewer such as Netron, first.

Config data types versus wire data types. The config uses TYPE_FP32 and TYPE_STRING; the HTTP and client protocol uses FP32 and BYTES. Mixing the spellings gives errors that look mysterious until you spot it.

TensorRT engines are not portable. A model.plan is built for a specific TensorRT version and GPU architecture. If you build it on one machine and load it in a container with a different TensorRT, or on a different GPU generation, it will not load. The fix is to build the plan in the same container release, on the same GPU type, that will serve it. ONNX and TorchScript do not have this restriction.

GPU model, CPU container. Forgetting --gpus or the NVIDIA Container Toolkit means a GPU model cannot load. The messages talk about CPU-only mode or CUDA initialisation failures.

The host driver is too old. The container's CUDA version needs a minimum host driver. The container starts but cannot use the GPU or fails on initialisation.

Server exits because one model failed. The default --exit-on-error=true means one bad model stops everything. It is a deliberate safe default; while developing you can turn it off.

Ready versus live. /v2/health/live means the process is running; /v2/health/ready means models are loaded and it can take traffic. Confusing them in a Kubernetes probe causes either restart loops, if you use readiness as liveness for a slow-loading model, or traffic sent to a server not yet ready.

Editing files and expecting a reload. In the default control mode, Triton reads the repository once at startup. A change requires a restart, or explicit model control mode with the load and unload API.

Treating the server as secure. Triton has no built-in authentication. It should sit behind a gateway or proxy and not be exposed to untrusted networks. This one is more of a mid-level topic, but mentioning it unprompted marks you as someone who thinks about production.

Common interview questions

What problem does Triton solve?

A model file is not a service. Teams need to expose models over a network with good hardware utilisation and operational features. Writing a service per model repeats the same work and tends to leave the GPU underused, because a simple handler processes one request at a time. Triton is a ready-made server that loads models from a repository, supports many frameworks through backends, can batch requests on the server, and exposes health and metrics endpoints.

A complete answer mentions all three pains: idle GPUs, a different glue layer per framework, and rebuilding operational plumbing. A weak answer says only "it makes models faster".

What is a model repository and what does it look like?

It is a directory, local or cloud, containing a folder per model. Each model folder holds config.pbtxt and numbered version folders, each containing the model file. For example models/densenet_onnx/config.pbtxt and models/densenet_onnx/1/model.onnx. Triton starts with --model-repository=<path> and can take the flag more than once to combine repositories, provided model names are unique. Cloud paths like s3://, gs://, and as:// are supported.

What is a backend?

The component that actually executes a model of a particular type. Triton itself schedules and serves; the backend runs the computation. Examples are ONNX Runtime, TensorRT, PyTorch, OpenVINO, a tree-model backend, and a Python backend for arbitrary code. You choose one per model with the backend field in the configuration.

What is in config.pbtxt and why is it needed?

It declares the model's name, backend, batching limit, input and output tensors, and runtime behaviour such as instance counts and the dynamic batcher. It is needed because the server must know the contract in order to validate requests before they reach the model, and it needs to know how to run it. For ONNX, TensorRT and OpenVINO models Triton can auto-complete much of it from the model file, but the file is still a good thing to write explicitly. To see what Triton actually uses, you call the config endpoint.

What does max_batch_size do?

It sets the largest batch Triton will pass to the model, and when it is above zero it implies a leading batch dimension on every tensor that is not written in dims. It does not on its own cause Triton to combine separate requests; that is the dynamic batcher's job. Setting it to zero disables batching and requires dims to describe the full tensor.

What is dynamic batching and why does it matter?

It is a scheduler setting, enabled with a dynamic_batching { } block, that lets Triton combine requests from different clients into one execution. GPUs process larger batches far more efficiently per example, so the throughput gain can be large, at the cost of a small amount of waiting that you can bound with max_queue_delay_microseconds. The NVIDIA documentation shows a sample where throughput rose from about 73 to about 272 inferences per second; say "in their example" to avoid overselling it. You verify the effect with Perf Analyzer rather than assuming.

What is an instance group?

It controls how many copies of a model run and on which device, for example count: 2 kind: KIND_GPU. Several instances let requests execute concurrently, which can improve GPU utilisation, but each takes memory, so there is a limit.

How do clients talk to Triton?

Over HTTP on port 8000 with JSON bodies, or gRPC on port 8001, following the KServe v2 protocol. There are client libraries for Python, C++ and Java; in Python you install tritonclient with the http, grpc, or all extra. gRPC is usually better for high request rates and large payloads, while HTTP is simpler to debug with curl.

How do you check whether Triton is healthy?

/v2/health/live for liveness and /v2/health/ready for readiness. Readiness is true only when the server is up and its models are loaded, which is what you want a load balancer or Kubernetes readiness probe to check. For a single model, /v2/models/<name>/ready. In addition the startup log prints a table of every model and its state: READY, or UNAVAILABLE with a reason.

A model shows UNAVAILABLE. What do you do?

Read the reason in the status table, scroll up in the logs to the lines for that model, and classify the failure. Is the file name right and in a numbered version folder? Do the tensor names in the config match the model? Does the model run in its own framework outside Triton? Is it a TensorRT plan built for a different version or GPU? Is it a GPU model started without a GPU? If the cause is still unclear, restart with --log-verbose=1 and reproduce once. The key signal in a good answer is the order: read the message first, then check the usual layout and naming causes, then isolate Triton from the model.

How would you monitor it?

Triton exposes Prometheus metrics on port 8002. I would scrape them and graph request success and failure counts, request and queue durations, and GPU utilisation and memory. The ratio of inference count to execution count shows how much batching is happening. Durations are cumulative counters in microseconds, so you look at rates, not raw values.

How do you measure performance?

With Perf Analyzer, which ships in the SDK container. It drives synthetic load at a model and reports throughput and latency percentiles at chosen concurrency levels. I would take a baseline, change one setting such as dynamic batching or instance count, and measure again. A single request timed by hand does not show behaviour under load.

Triton versus a FastAPI service around the model?

For one small model on CPU with modest traffic, FastAPI is simpler and gives full control. Triton earns its place when you have GPUs to keep busy, several model formats or many models, and you want server-side batching, metrics, model versioning and health endpoints without building them. The honest framing is that it is a trade between control and effort, not a question of one being better. It also costs you a configuration format and a container to learn.

Which model formats does Triton run?

TensorRT, ONNX, TorchScript through the PyTorch backend, OpenVINO, tree models through the FIL backend, DALI pipelines, custom Python code, and large language model engines through TensorRT-LLM and vLLM backends. The TensorFlow backend has been removed from recent containers, so check the release notes before relying on it. A good answer notes that TensorRT plans are tied to a version and architecture.

How do you serve two versions of a model?

Place two numbered folders under the model and set version_policy in the config, for example latest with num_versions: 2, or all, or a specific list. Clients choose a version in the URL, /v2/models/<name>/versions/<n>/infer. By default only the highest version is served.

How do you load or unload a model without restarting?

Start in explicit model control mode and use the repository API, POST /v2/repository/models/<name>/load and .../unload. The default mode ignores later changes, and poll mode watches the folder but is not recommended for production by the documentation. Explicit mode is also the safer of the dynamic options, but if loading can be requested by clients, that API must be protected.

What would you do before putting Triton on the internet?

Not put it directly on the internet. It has no built-in authentication, and the documentation states it does not provide a security sandbox for arbitrary model or backend code. It belongs behind a gateway or proxy that handles authentication and TLS. Disable protocols and endpoints you do not use, keep the default control mode, run as a non-root user, and load only models you trust, because a Python backend model is arbitrary code.

Sixty-second self-test

Cover the answers and try these aloud. If you stumble on more than two, reread the matching section of the beginner guide and build the project again.

1. In two sentences, what is Triton?
2. Draw the layout of a model repository for a model called sentiment with one version, and name the file you put in the version folder for an ONNX model.
3. Which ports serve HTTP, gRPC, and metrics?
4. With max_batch_size: 8 and dims: [ 4 ], what shape does a client send for three examples? And with max_batch_size: 0?
5. What is the difference between the readiness and liveness endpoints?
6. Name two reasons a model can show UNAVAILABLE.
7. Why must you rebuild a TensorRT plan when you upgrade the container?
8. What is the first thing you do when startup fails with "failed to load all models"?
9. Which setting makes Triton combine requests from different clients, and what does max_queue_delay_microseconds trade?
10. Why is a Triton server not safe to expose directly to the internet?

Model answers in brief: a server for trained models with HTTP and gRPC, batching, and metrics; sentiment/config.pbtxt and sentiment/1/model.onnx; 8000, 8001, 8002; [3, 4], and then whatever full shape dims declares, since there is no extra batch axis; ready means models are loaded, live means the process runs; wrong file name or version folder, tensor name mismatch, or GPU model without a GPU; plans are tied to the TensorRT version and the GPU; scroll up and read the per-model error; the dynamic batcher, trading a little latency for throughput; no built-in authentication and no sandbox.

Finish by being ready for one open question: "Tell me about something you served with it, and what went wrong." Even if your experience is the tutorial project, describe a real stumble, such as getting a shape error from the batch dimension, and what you did to find and fix it. That specific, honest story is worth more than any list of features.

This is the practical companion to the beginner Triton guide. It lists the mistakes new users actually make, each with the symptom you will see, the cause underneath, and the fix, followed by habits worth adopting on day one and a short checklist. Where the main guide teaches how things work, this page is meant to be skimmed when something has just broken. Versions referred to are Triton 2.72.0 and the 26.08 container.

The mistakes beginners actually make

Mistake 1: confusing the batch dimension

Symptom. The client gets an error saying the input shape does not match what the model expects, with numbers that look nearly right, such as expected [-1, 4] and got [4].

Cause. When max_batch_size is above zero, Triton adds a hidden leading batch axis. The dims field describes one example only, but the client must send the batch axis as well. Beginners write dims: [ 4 ] and then send a flat array of four numbers.

Fix. Send shape [1, 4] for one example. If your model genuinely does not batch, set max_batch_size: 0 and write the complete shape in dims. Keep a sticky note with the rule: batching on means dims exclude the batch, client includes it.

Mistake 2: a version folder that Triton ignores

Symptom. The server starts without errors but your model is not in the table, or a request returns "unknown model" or "no available versions".

Cause. Triton recognises only integer-named folders as versions. Folders called v1, latest, or 1.0 are skipped without complaint. A second cause is putting the model file straight inside the model folder, next to config.pbtxt, instead of inside a version folder.

Fix. The structure must be <repo>/<model>/<integer>/<file>. Look at it with find models -type f, which shows the layout in one glance.

Mistake 3: the wrong file name inside the version folder

Symptom. The model is UNAVAILABLE, and the reason mentions that it cannot find or open the model file.

Cause. Every backend expects a default name: model.onnx, model.pt, model.plan, model.py, model.dali, or model.xml with model.bin. A file named resnet50.onnx or model_final.onnx is not found.

Fix. Rename the file to the default, or add default_model_filename: "resnet50.onnx" to the configuration. Renaming is simpler and leaves less to remember.

Mistake 4: tensor names invented rather than copied

Symptom. Load failure naming an input or output that does not exist, or an "unexpected inference input" error at request time listing the names it allows.

Cause. The names in config.pbtxt must match the tensor names inside the model file. People write input and output because those seem natural.

Fix. Open the model in Netron, or list its inputs with the ONNX library, and copy the names exactly. Names are case-sensitive. If a request error lists allowed inputs, copy from that list.

Mistake 5: mixing the two spellings of data types

Symptom. Errors about an unknown or mismatched data type, either when the model loads or when the client builds an input.

Cause. The configuration uses protobuf enum names such as TYPE_FP32, TYPE_INT64, and TYPE_STRING. The wire protocol and the client library use the short names FP32, INT64, and BYTES. Strings are the biggest trap, because the config says string and the client says bytes.

Fix. Use the long form in config.pbtxt and the short form in client code. When sending text from Python, build the array with dtype=object and declare the input as BYTES.

Mistake 6: a NumPy dtype that disagrees with the declared type

Symptom. The client raises an error before sending anything, saying the data type of the array does not match the declared one, or the server's answer contains nonsense numbers.

Cause. NumPy's default floating type is 64-bit, but most models want 32-bit. An array made with np.array([1.0, 2.0]) is float64, which does not match FP32.

Fix. Create arrays with an explicit dtype, such as np.float32, or call .astype(np.float32). For integers, be equally explicit about int32 versus int64.

Mistake 7: expecting the server to notice file changes

Symptom. You edit config.pbtxt or replace the model file, send a request, and see the old behaviour.

Cause. In the default control mode, --model-control-mode=none, Triton reads the repository once at startup and ignores later changes.

Fix. Restart the server. If you need to change a model repeatedly during development, start in explicit mode and call POST /v2/repository/models/<name>/unload then .../load. Avoid poll mode for anything beyond experiments; the documentation does not recommend it for production.

Mistake 8: an unpinned or mismatched container tag

Symptom. A setup that worked last month fails after a pull, or two colleagues see different behaviour from the same repository.

Cause. Triton releases monthly, and each release changes the CUDA, TensorRT, and ONNX Runtime versions inside the container. Using a floating or locally remembered tag lets that change arrive without you choosing it.

Fix. Write the full tag, for example nvcr.io/nvidia/tritonserver:26.08-py3, in your scripts and commit it. Upgrade deliberately, reading the release notes, and rerun your checks.

Mistake 9: TensorRT plans carried between machines

Symptom. A model.plan that worked on your workstation fails on the server with a message about the engine being incompatible with this version of TensorRT, or a deserialisation failure.

Cause. A plan is compiled for a particular TensorRT version and GPU architecture. It is not a portable file.

Fix. Build the plan inside the container you will serve with, on the same GPU type, ideally as a scripted step. If portability matters more than the last bit of speed, serve ONNX and let the backend handle the optimisation.

Mistake 10: a GPU config on a machine with no GPU access

Symptom. A model fails to load with messages about CUDA initialisation, no available devices, or running in CPU-only mode.

Cause. Either the container was started without --gpus, the NVIDIA Container Toolkit is missing on the host, the host driver is older than the container's CUDA requires, or the configuration or backend assumes a GPU.

Fix. Check the host first: run nvidia-smi on the host, and then inside the container. If you are on a laptop, set instance_group [ { count: 1 kind: KIND_CPU } ] and use a CPU-capable backend. If the driver is old, upgrade it or choose an older container tag that matches it.

Mistake 11: reading only the last line of a failed start

Symptom. You see error: creating server: Internal - failed to load all models and have no idea what is wrong.

Cause. That line is the generic summary. The real reason appears several lines earlier, in the entry for the model that failed.

Fix. Scroll up. Find the model's name in the log and read the message beside it. While investigating, add --exit-on-error=false so the other models keep serving and the broken one appears as UNAVAILABLE with its reason in the table. For stubborn cases add --log-verbose=1.

Mistake 12: Docker volume mounts that mount the wrong thing

Symptom. The container starts and reports that the repository is empty, or cannot find it, even though the folder exists on your machine.

Cause. Several variations: a relative path without a leading slash, which Docker treats as a named volume; a path with a typo; a mount in a shell where $PWD is not what you think; or, on Windows with WSL2, a folder that lives on the Windows side.

Fix. Use -v "$PWD/models:/models", and verify with docker run --rm -v "$PWD/models:/models" ubuntu ls -R /models before involving Triton. On WSL2, keep the project inside the Linux file system.

Mistake 13: Python backend crashes under load with shared-memory errors

Symptom. A Python-backend model works for tiny requests, then fails or the process dies with a bus error or an inability to allocate shared memory when tensors get larger.

Cause. The Python backend moves data between processes through shared memory, and containers have a small default allocation.

Fix. Add --shm-size=1g or larger to the docker run command. Size it according to your tensors and instance count.

Mistake 14: heavy work in execute that belongs in initialize

Symptom. Every request is slow, even trivial ones, or memory grows because a file or model is loaded over and over.

Cause. In a Python-backend model, initialize runs once at load and execute runs per batch of requests. Beginners load weights or open files inside execute.

Fix. Load anything expensive in initialize and store it on self. Keep execute to the per-request work, and remember it must return exactly one response for each request, in order.

Mistake 15: missing Python dependencies inside the container

Symptom. A Python-backend model is UNAVAILABLE with a ModuleNotFoundError for a library you have installed on your laptop.

Cause. The model runs inside the container's Python, not yours. Your virtual environment is invisible to it.

Fix. For a quick experiment, build a derived image that installs the library with pip. For a cleaner approach, the Python backend supports a custom execution environment; the documentation describes packaging one. Before reaching for either, ask whether the library is really needed in the serving path, since heavy preprocessing is sometimes better done in the client.

Mistake 16: judging performance from one request

Symptom. A single request takes much longer than expected, so you conclude the setup is slow, or the opposite: one fast request leads you to ship.

Cause. The first request to a model often includes one-time work such as memory allocation or kernel setup, and one request tells you nothing about throughput under concurrency.

Fix. Use the Perf Analyzer, shipped in the SDK image, with a concurrency range, and read the throughput and latency percentiles it reports. For real models, add a model_warmup block so the cold start happens at load time rather than on a user's first request.

Mistake 17: assuming max_batch_size batches requests

Symptom. You set max_batch_size: 32 and throughput does not improve under concurrent clients.

Cause. That field only permits batches up to a size. Triton combines separate requests only when the dynamic batcher is enabled.

Fix. Add dynamic_batching { }, restart, and measure with Perf Analyzer. Check afterwards that the ratio of nv_inference_count to nv_inference_exec_count rose above one. If it stays at one, requests are not arriving concurrently or the model is idle most of the time anyway.

Mistake 18: exposing the server openly

Symptom. None, until someone else finds it. Ports 8000, 8001, and 8002 are reachable from places you did not intend.

Cause. Publishing ports with -p binds them on all interfaces by default, and Triton has no built-in authentication.

Fix. Keep it on a private network or bind to localhost with -p 127.0.0.1:8000:8000 while experimenting. In real deployments put a gateway in front for authentication and TLS, switch off protocols you do not use with --allow-http=false or --allow-grpc=false, and keep the default model control mode so that nobody can load models remotely.

Habits worth adopting on day one

Pin everything. Write the container tag and the client library version in a file. A server that behaves differently every month is expensive to debug.

Keep the model repository in version control, or build it from a script. A config.pbtxt is a contract. Reviewing a change to it in a pull request catches the batching and shape mistakes before they reach a server. The model files themselves can live in object storage and be fetched by the script, since they are large.

Test the model outside Triton first. The official debugging guide begins here for a reason. If the ONNX file fails in ONNX Runtime on its own, no configuration helps. A five-line script that loads the file and runs a dummy input saves hours.

Look at what Triton believes. After startup, fetch /v2/models/<name>/config and /v2/models/<name>. Reading the server's description of your model catches differences between what you wrote and what it loaded.

Start without optimisation, then add one change at a time. Get a correct answer from the plain configuration. Then add dynamic batching, measure, add an instance, measure. If you change three things at once you will not know which helped.

Write a smoke-test script. A short Python client that sends one known input and checks the output against the framework's own answer is the best regression test you can have. Run it after every container upgrade and every config change.

Check readiness, not just the process. Any script or probe that starts Triton should wait for /v2/health/ready to return 200 before sending traffic. A liveness check alone will report success while models are still loading.

Name tensors and models deliberately. Short, stable names appear in URLs, dashboards, and client code. Renaming later breaks every consumer. Use lowercase with underscores for models, such as sentiment_onnx.

Be explicit about data types. In client code, always create arrays with a dtype. The default of 64-bit floats does not match most models and causes avoidable errors.

Keep notes on the region and the data. If your employer has data-residency rules, write down where the server runs and where logs and metrics go. For teams in the Gulf and Egypt this is often the first compliance question, and answering it early is easier than moving a service later.

Read the release notes before upgrading. Monthly releases change CUDA, TensorRT, and library versions, and the notes list known issues. Upgrade in a test environment, rerun the smoke test and the Perf Analyzer baseline, and only then change production.

Treat Python-backend models as code you run, not data you load. They execute arbitrary Python inside the server. Load only models your team wrote or reviewed.

Quick recipes

Several small commands save time. Put them in a notes file.

Show the repository layout to catch folder mistakes:

find models -type f | sort

Wait until the server is ready before testing, from a script:

until curl -sf localhost:8000/v2/health/ready; do sleep 2; done
echo "Triton is ready"

List every model and its state when running in explicit mode:

curl -s -X POST localhost:8000/v2/repository/index

See batching efficiency from the metrics, by comparing the inference and execution counters for your model:

curl -s localhost:8002/metrics | grep -E "nv_inference_(count|exec_count)"

If the two numbers are equal, no batching is happening. If the first is several times the second, batching is working.

Run the Perf Analyzer from the SDK container against a server running on the same host:

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.08-py3-sdk \
  perf_analyzer -m add_vectors --concurrency-range 1:4

On Docker Desktop for Mac or Windows, host networking behaves differently, so point the tool at the host's address instead of relying on localhost.

Test a model's input and output names without Triton, using the ONNX library in Python:

import onnx

model = onnx.load("model.onnx")
for tensor in model.graph.input:
    print("input", tensor.name)
for tensor in model.graph.output:
    print("output", tensor.name)

That tells you what to put in the configuration, and removes the single most common cause of failed loads.

A short checklist

Run through this before you ask a colleague for help, and before you call a model served.

• The container tag is pinned, and the host driver meets the container's CUDA requirement if you use a GPU.
• The folder layout is <repo>/<model>/config.pbtxt and <repo>/<model>/<integer>/<file>.
• The file inside the version folder has the backend's default name, or default_model_filename is set.
• The tensor names in config.pbtxt match the model file exactly, including case.
• Data types are written long in the config (TYPE_FP32) and short in the client (FP32, BYTES).
• max_batch_size and dims agree with how the client shapes its arrays.
• The startup table shows READY for each model, and /v2/health/ready returns 200.
• A smoke-test client has sent a known input and compared the output with the framework's own result.
• Metrics on port 8002 respond, and the request success counter increases when you send traffic.
• A Perf Analyzer baseline is recorded before any tuning change.
• The server is not reachable from networks that should not reach it, and protocols you do not use are switched off.
• For TensorRT, the plan was built in the same container release and on the same GPU type as the server.

If a box cannot be ticked, you have found where to look. Most beginner problems with Triton are one of the first five lines, and finding them quickly is mostly the habit of reading the log from the top and comparing the server's view of the model with your own.

This is part two of three. It picks up where Beginner ended: you can start tritonserver against a model repository, write a small config.pbtxt, and send an inference request. Here the aim is to run Triton well at work, which means predicting what it does under load, measuring instead of guessing, and operating it on shared infrastructure without surprises.

Where this picks up

The version used throughout is Triton Inference Server 2.72.0, shipped as container release 26.08, tagged nvcr.io/nvidia/tritonserver:26.08-py3. Releases are monthly and tagged <yy>.<mm>, so check the release-notes index for anything newer than the tag you pin. Each section starts with the problem it solves and ends, in most cases, with a Try it you can run against a real model.

What happens to a request inside Triton

Most tuning mistakes come from a vague picture of the request path. You cannot decide where to add an instance, or why latency rose after you raised a batch size, without knowing which component holds the request at each moment. So start with the path.

A request arrives over HTTP on port 8000 or gRPC on port 8001. Both follow the KServe v2 inference protocol, so the same inference request is expressed in either. The frontend validates it against the model's configuration: the model name and version must resolve, the input names and datatypes must match, and the shapes must fit. If the model has batching enabled, the request's batch size must not exceed max_batch_size. Validation failures return an error immediately and never reach a GPU, which is why a malformed request is cheap and a slow request is not a validation problem.

A valid request then goes to the model's scheduler. The scheduler is the part that decides when and with whom the request runs. There are four kinds. The default scheduler is a passthrough: each request goes to the next free instance on its own. The dynamic batcher holds requests briefly and merges them into larger batches. The sequence batcher routes all requests of one stateful sequence to the same instance. The ensemble scheduler drives a graph of other models. Which one a model uses is a configuration choice, and it is the single biggest influence on throughput.

From the scheduler the request goes to an instance of the model. An instance is one loaded copy of the model bound to a device, and it runs one execution at a time. The instance hands the batch to the backend (TensorRT, ONNX Runtime, PyTorch, Python, and so on), which runs the actual computation and returns output tensors. Triton then splits a batched result back into per-request responses and sends each to the right caller.

Triton measures the time in each stage and exposes it as counters, which is what makes diagnosis possible later. Time spent waiting in the scheduler is queue time (nv_inference_queue_duration_us). Time copying inputs to the device, running the model and copying outputs back are the three compute durations (nv_inference_compute_input_duration_us, nv_inference_compute_infer_duration_us, nv_inference_compute_output_duration_us). The end-to-end figure is nv_inference_request_duration_us. When a service is slow, the first question is always whether the time is in the queue or in compute, because the fixes are opposite. Long queue time means requests are waiting for a free instance; add instances or batch better. Long compute time means the model itself is slow; look at the backend, precision and the model.

Two more counters matter and are easy to confuse. nv_inference_count counts inferences, meaning individual items in a batch. nv_inference_exec_count counts executions, meaning how many times the model actually ran. If you send batches of eight and the server runs them as they come, the two differ by a factor of eight. If the dynamic batcher merges requests, executions fall below request count. The ratio nv_inference_count / nv_inference_exec_count is therefore the average batch size actually achieved, which we use throughout as the measure of batching efficiency.

The model repository and configuration, in depth

Beginner showed a minimal config. At this level the configuration file is where most of your decisions live, so it is worth knowing exactly how Triton reads it and what it will fill in for you.

The repository is a directory, or a cloud storage path, containing one folder per model. Inside each model folder is a config.pbtxt and one or more integer-named version folders. A folder whose name is not an integer is ignored, which is occasionally useful for stashing a README or a scratch directory but is a common source of "my new version does not appear" when someone names it v2. Each version folder holds the model file under a default name that depends on the backend: model.plan for TensorRT, model.onnx for ONNX Runtime, model.pt for TorchScript, model.xml and model.bin for OpenVINO, model.py for the Python backend and model.dali for DALI. If your build pipeline produces a different filename, set default_model_filename in the config rather than renaming files in a post-build step.

Auto-complete is the behaviour that surprises people. For TensorRT, ONNX Runtime and OpenVINO models, Triton can read the model file and fill in the inputs, outputs and datatypes if you omit them, so a config can be nearly empty. PyTorch and Python models cannot be introspected, so they need full input and output declarations. Two consequences follow. First, the config on disk is not the config the server uses; the effective one is available from GET /v2/models/<name>/config, and that is the one to read when behaviour looks wrong. Second, for a production repository you usually want the config to be the whole truth, because a model file changing underneath you changes the effective config silently. Start the server with --disable-auto-complete-config once you have written complete configs, and the files on disk become authoritative.

# What the server actually decided, as opposed to what you wrote
curl -s localhost:8000/v2/models/densenet_onnx/config | python3 -m json.tool

# Fail fast on incomplete configs rather than letting the server guess
tritonserver --model-repository=/models --disable-auto-complete-config

The batch dimension rule is the most common source of shape errors. With max_batch_size greater than zero, Triton adds a leading batch dimension to every input and output itself, so the dims in the config exclude it: an image model taking [3, 224, 224] is sent tensors shaped [N, 3, 224, 224]. With max_batch_size: 0, batching is off and dims describe the full tensor including any leading dimension. A model whose first dimension is variable but not a batch, such as a sequence length, belongs in the second form with -1 in the dims. If you confuse the two, you get either a shape mismatch error or, worse, a model that accepts requests and interprets the wrong axis as the batch.

When one repository must serve different hardware, you have two tools. The compute-capability map cc_model_filenames chooses a different file per GPU architecture inside one config:

cc_model_filenames [
  { key: "7.5" value: "model_T4.plan" },
  { key: "8.0" value: "model_A100.plan" }
]

The second tool is a model-specific config file. Starting the server with --model-config-name=h100 makes Triton read configs/h100.pbtxt from each model directory instead of config.pbtxt, so one repository can hold tuned settings for several GPU types while the image and the model files stay identical. Prefer this when the difference is tuning (instance counts, batch sizes) and the compute-capability map when the difference is the file itself.

Dynamic batching, precisely

GPUs are throughput machines that are inefficient at batch size one. A request that uses a small fraction of the device still occupies an execution slot, so serving single requests one at a time wastes most of the hardware. The dynamic batcher fixes this on the server side: it holds incoming requests for a short window, merges those that fit, runs one larger execution, and splits the result back. Callers still send individual requests and see individual responses, so you gain throughput without changing any client.

Enabling it is one block, and the empty form already works:

max_batch_size: 8
dynamic_batching {
  preferred_batch_size: [ 4, 8 ]
  max_queue_delay_microseconds: 100
}

Read the three controls as a policy. max_batch_size is the hard ceiling and dynamic batching requires it to be greater than zero. max_queue_delay_microseconds is the most the batcher will hold a request waiting for company; it is the latency you are willing to trade for batching, and it defaults to zero, which means the batcher forms batches only from requests that are already queued at that instant. preferred_batch_size lists sizes the batcher should aim for, which matters when the backend is markedly faster at certain sizes, as with TensorRT engines optimised for specific profiles. Without a delay, a preferred size has little room to act, because nothing waits long enough for a batch to fill.

The NVIDIA optimization documentation shows the effect with a worked example: for one model under eight concurrent requests, enabling the dynamic batcher raised throughput from about 73 inferences per second to 272. Treat that as the shape of the gain, not a promise. The size of the win depends on how much your model benefits from batching, which is large for compute-light models on big GPUs and small for models that already saturate the device at batch one.

There are two traps. The first is that dynamic batching adds latency by design. Under light traffic, a nonzero queue delay means a lone request waits the full delay for company that never arrives. If your p99 matters more than your throughput, keep the delay very small and verify with a measurement. The second trap is that a model that is not batchable cannot be batched by configuration. If its inputs have different shapes per request, or if it keeps state across requests, merging produces wrong answers or errors. Stateful models need the sequence batcher, and variable-shaped inputs need ragged-batching features or padding on the client.

The way to judge whether batching is working is the ratio from earlier. Compute nv_inference_count / nv_inference_exec_count over a window in production. A ratio near one means requests are being executed alone, either because traffic is too low, the delay is too short, or the batcher is not enabled. A ratio near max_batch_size means the batcher is saturated, and raising the ceiling may help if the model and memory allow it.

# Average executed batch size over five minutes
rate(nv_inference_count{model="resnet50"}[5m])
  / rate(nv_inference_exec_count{model="resnet50"}[5m])

Instance groups and concurrent execution

Batching makes each execution efficient. Instance groups make several executions run at the same time. They solve a different problem: a single model instance runs one execution at a time, so while it computes, the next batch waits in the queue even if the GPU has spare capacity. Adding instances lets Triton overlap executions, hiding transfer and scheduling gaps and filling a device that one stream cannot.

instance_group [
  { count: 2 kind: KIND_GPU gpus: [ 0 ] },
  { count: 1 kind: KIND_CPU }
]

count is how many copies per entry, kind is KIND_GPU, KIND_CPU or, for models that manage devices themselves, KIND_MODEL, and gpus pins the copies to specific devices. If you omit the group, Triton creates one instance per available GPU, which surprises people on multi-GPU hosts: a model you thought used one card is loaded on every card, and consumes memory on all of them. Pin explicitly when the host is shared.

More instances are not free. Each instance is a full copy of the model in device memory, so weights are duplicated, and each needs its own activation memory at the batch sizes you allow. Two instances of a model that nearly fills a GPU will not load. More instances also split the same queue, which shrinks the batches the dynamic batcher can form, since requests are shared across instances. There is a real tension between "more instances to overlap" and "fewer instances to batch bigger", and it is resolved by measurement, not by rule.

instance_group [
  {
    count: 2
    kind: KIND_GPU
    rate_limiter {
      resources [ { name: "R1" count: 4 } ]
      priority: 2
    }
  }
]

Use the rate limiter when two models share a device and one must not starve the other, or when a resource such as device memory bandwidth, not compute, is the true bottleneck. For a single-model server it adds complexity for no gain.

The Perf Analyzer documentation gives a sizing rule that connects these ideas: the concurrency at which a model reaches maximum throughput is about two times the maximum batch size times the number of instances. It is a starting range for your sweep, not a law, but it tells you where to look first and why a test with concurrency of four against a model with batch size eight and two instances shows nothing interesting.

Measuring with Perf Analyzer

Every claim above is a hypothesis until you measure. Perf Analyzer (perf_analyzer) is the client that sends synthetic load to a running server and reports throughput and latency. It ships in the SDK container, tagged <xx.yy>-py3-sdk and matched to the server release, and it removes the most common reason for bad benchmarks, which is a hand-written client that is itself the bottleneck.

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.08-py3-sdk

# Inside the SDK container
perf_analyzer -m inception_onnx --percentile=95 --concurrency-range 1:4

The command sweeps concurrency from one to four outstanding requests and prints, for each level, the throughput in inferences per second and latency at the chosen percentile, broken down into client send and receive, server queue, and server compute. That breakdown is the same decomposition as the server metrics, so a result such as "queue 80 percent of latency" immediately points at instances or batching rather than the model.

Concurrency is the number of requests held in flight, which is why it is the right knob. A real service sees an arrival rate, but a closed-loop test that keeps N requests outstanding explores the same space safely: low N shows best-case latency, and rising N shows where throughput flattens and latency starts to climb. The right operating point is the knee, where more concurrency buys little throughput and a lot of latency. Per the sizing rule above, expect the knee near two times the batch size times the instance count.

TensorRT acceleration and plan compatibility

If the model is ONNX, you have a cheap way to use TensorRT without leaving the repository format: tell the ONNX Runtime backend to hand eligible subgraphs to TensorRT through an execution accelerator. It saves you from building and managing a plan file yourself.

optimization {
  execution_accelerators {
    gpu_execution_accelerator : [ {
      name : "tensorrt"
      parameters { key: "precision_mode" value: "FP16" }
      parameters { key: "max_workspace_size_bytes" value: "1073741824" }
    } ]
  }
}

precision_mode selects the numeric precision TensorRT may use, and FP16 usually raises throughput substantially with a small accuracy change that you must check on your own validation set. max_workspace_size_bytes bounds the scratch memory TensorRT may use while optimising. The optimization documentation reports about double the throughput and half the latency for its example model. Again, that is an illustration of the possible size of the change, and your model decides your result.

Mind the cost: TensorRT optimises at load, so the first load of a model with this accelerator is slower, sometimes by a lot, since the engine is built then. That is an important input to readiness and scaling behaviour, covered below. Use model_warmup so the first user request does not pay for any remaining lazy work.

The other route is to build a TensorRT engine yourself and serve the resulting model.plan with the TensorRT backend. This gives more control and typically the best performance, but it comes with a compatibility rule that causes more production incidents than any other single thing in this tool. A plan file is tied to the TensorRT version and the GPU architecture that built it. Moving it to a container with a different TensorRT version, or to a GPU of a different architecture, fails to load with an error saying the engine plan file is not compatible with this version of TensorRT. Because every monthly Triton container bumps CUDA, TensorRT and the other libraries (the 26.08 container carries TensorRT 11.2.1.2 and CUDA 13.4.1, for example), a plan built for 26.07 is not guaranteed to load in 26.08.

The consequence for process is firm. Build plans in CI inside the same container tag you deploy, for each GPU type you serve, and publish them as release artifacts keyed by container tag and architecture. Never treat a .plan as a portable model artifact the way you treat an ONNX file. The compute-capability map shown earlier is how one repository can carry several plans, one per architecture. When you upgrade Triton, the upgrade checklist starts with "rebuild every plan".

The Python backend, ensembles and Business Logic Scripting

Real services are rarely one model. They decode an image, normalise it, run a network, and post-process the output into something a caller wants. Triton offers three ways to express that, and the choice among them is a design decision with performance consequences.

The Python backend lets you write a model as Python code. It is the escape hatch for preprocessing, postprocessing and anything without a native backend. A model directory contains a model.py defining a class named TritonPythonModel with initialize, execute and finalize methods. initialize runs once at load and is where you open files and load weights. execute receives a list of requests, which is a batch when the dynamic batcher is on, and returns a list of responses, one per request. finalize runs at unload.

import json
import numpy as np
import triton_python_backend_utils as pb_utils


class TritonPythonModel:
    def initialize(self, args):
        config = json.loads(args["model_config"])
        self.scale = float(config["parameters"]["scale"]["string_value"])

    def execute(self, requests):
        responses = []
        for request in requests:
            x = pb_utils.get_input_tensor_by_name(request, "INPUT0").as_numpy()
            out = pb_utils.Tensor("OUTPUT0", (x * self.scale).astype(np.float32))
            responses.append(pb_utils.InferenceResponse(output_tensors=[out]))
        return responses

    def finalize(self):
        pass

Two things deserve care. The Python backend runs in separate processes, so a heavy execute is bounded by the Python interpreter, and a loop over requests that does per-request work serialises it. Where you can, process the whole list of requests as one array operation. And Python dependencies are a deployment problem: a ModuleNotFoundError raised in initialize means the packages you import are not in the environment the backend runs in. The remedy is to ship a packed environment and point the model at it with the EXECUTION_ENV_PATH parameter; build that environment in CI against the same container so versions match.

An ensemble wires several models into a graph declared in configuration, with no code. Triton passes tensors from the output of one step to the input of the next and runs the steps with their own schedulers and instances.

name: "image_pipeline"
platform: "ensemble"
max_batch_size: 8
input [ { name: "RAW_IMAGE" data_type: TYPE_UINT8 dims: [ -1 ] } ]
output [ { name: "SCORES" data_type: TYPE_FP32 dims: [ 1000 ] } ]
ensemble_scheduling {
  step [
    {
      model_name: "preprocess"
      model_version: -1
      input_map { key: "RAW" value: "RAW_IMAGE" }
      output_map { key: "TENSOR" value: "pre_out" }
    },
    {
      model_name: "classifier"
      model_version: -1
      input_map { key: "data_0" value: "pre_out" }
      output_map { key: "fc6_1" value: "SCORES" }
    }
  ]
}

The benefit of an ensemble is that tensors move between steps inside the server, with no network round trip and no client logic. Each step can have its own batching and instance settings, so a cheap CPU preprocess and an expensive GPU model scale separately. The limit is that an ensemble is a static graph. It cannot branch on a value, loop, or call a model a variable number of times.

Business Logic Scripting (BLS) covers exactly that gap. Inside a Python backend model you call other models in the repository programmatically and use their results to decide what to do next. That makes dynamic pipelines possible: run a detector, then run a classifier once for each detection found. The price is that the control flow now lives in your code, with the testing and error handling that implies, and the Python process becomes part of the hot path.

Model control, versions and warmup

How and when models load decides how safely you can change a running server. Triton has three model control modes, set with --model-control-mode.

In none mode, the default, Triton loads every model at startup and ignores later changes to the repository. This is the simplest and most predictable mode: what you deployed is what is running, and a change means a new container or a restart. The secure-deployment guide recommends it for production, because dynamic model updates can lead to arbitrary code execution if an attacker can reach the repository or the load API.

In explicit mode, nothing loads unless asked. You name models at start with --load-model=<name> (or --load-model=* for all) and afterwards use the repository API to load and unload them:

curl -X POST localhost:8000/v2/repository/models/densenet_onnx/load
curl -X POST localhost:8000/v2/repository/models/densenet_onnx/unload
curl -X POST localhost:8000/v2/repository/index

This mode suits a model-management service that decides what is resident, such as a system that hosts many models and cannot fit them all on a GPU. In poll mode, Triton watches the repository every --repository-poll-secs seconds and loads, unloads or reloads on change. The documentation does not recommend it for production, and the reason is practical: a half-copied model directory is read mid-write, and the failure shows up as a load error at an unpredictable moment.

Startup failure policy is the other half. By default --exit-on-error is true, so if any model fails to load the whole server exits with error: creating server: Internal - failed to load all models. This is the right default for a service that should not run half-broken, and the error is not the cause: the cause is on the lines above it, one per failing model. Setting --exit-on-error=false keeps the healthy models up with the failed one in the UNAVAILABLE state, which suits a multi-model server where one bad model should not take down the others, provided your readiness probe is aware of it. With --strict-readiness at its default of true, /v2/health/ready reports not ready until every model is ready, so a partial server will not receive traffic. Decide deliberately which behaviour you want rather than inheriting the defaults by accident.

Version policy and rollback fit together. With latest { num_versions: 2 }, the two highest versions are served, so a new version 3 goes live beside 2, and clients that pin a version keep working. Because Triton exposes versions in the URL (/v2/models/<name>/versions/<v>/infer), you can compare versions on the same server during a rollout and drop the old one when satisfied. specific { versions: [1, 3] } lets you hold a known-good version while testing another.

Warmup closes a gap that readiness probes cannot see: a model reports READY once loaded, but the first inference can still be slow because of lazy allocation, kernel compilation or autotuning. With model_warmup, Triton runs your synthetic requests before declaring the model ready.

model_warmup [
  {
    name: "warmup_sample"
    batch_size: 1
    inputs {
      key: "INPUT0"
      value: { data_type: TYPE_FP32 dims: [ 16 ] zero_data: true }
    }
  }
]

Zero-filled input is fine for many models but not all; a model with data-dependent branching may exercise different code on real data, and warmup is best given inputs of realistic shape and, if possible, realistic content via a file.

Running Triton on Kubernetes

A single container on a GPU host is a demo. In production Triton runs as a Deployment on GPU nodes, and the useful questions are about probes, resources and what scales it. The mapping from Triton concepts to Kubernetes is direct, and getting it wrong is mostly about timing. If Kubernetes itself is new to you, the Kubernetes guide and the Helm guide cover the objects used below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton
spec:
  replicas: 2
  selector:
    matchLabels: { app: triton }
  template:
    metadata:
      labels: { app: triton }
    spec:
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:26.08-py3
          args:
            - tritonserver
            - --model-repository=s3://my-models/prod
            - --model-control-mode=none
          ports:
            - { name: http, containerPort: 8000 }
            - { name: grpc, containerPort: 8001 }
            - { name: metrics, containerPort: 8002 }
          resources:
            limits:
              nvidia.com/gpu: 1
          startupProbe:
            httpGet: { path: /v2/health/ready, port: http }
            periodSeconds: 10
            failureThreshold: 60
          readinessProbe:
            httpGet: { path: /v2/health/ready, port: http }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /v2/health/live, port: http }
            periodSeconds: 10

The three probes use two endpoints on purpose. /v2/health/live answers as soon as the server process is up and should drive the liveness probe: if it fails, restart. /v2/health/ready returns success only when models are loaded (under --strict-readiness), so it belongs on readiness and on a startup probe. The startup probe matters because model loading is slow. A large model pulled from object storage, or a TensorRT engine built at load, can take minutes, and a liveness probe that starts counting at container start will kill the pod mid-load, restart it, and loop forever. Give the startup probe a generous failureThreshold times periodSeconds, and point liveness only at the live endpoint.

Scaling comes in two directions. Vertical scaling is everything covered above: instance counts, batching and precision, which raise what one GPU delivers. Horizontal scaling adds replicas behind a Service. Whatever sits in front must balance by request, and with gRPC that needs care: a single long-lived HTTP/2 connection can pin all traffic to one pod, so use a gateway or mesh that balances at the request level, or client-side balancing across endpoints.

For autoscaling, CPU utilisation is nearly useless because the work happens on the GPU. Better signals come from Triton's own metrics: the queue duration, or the pending request count (nv_inference_pending_request_count), exposed to the Horizontal Pod Autoscaler through a metrics adapter. The hard part is that the cost of a scale-up is the model load time. A new replica is not useful until it has pulled weights and become ready, which may be minutes, so scale on a leading signal and keep headroom rather than reacting to a queue that is already long. Scale-to-zero is rarely sensible for big models for the same reason.

On shared GPUs, MIG (Multi-Instance GPU) partitions a card into isolated slices, so several smaller Triton deployments can share one GPU with memory isolation. That suits many small models and multi-tenant setups better than packing models in one server.

Triton is also the runtime behind several managed services. SageMaker, Vertex AI and Azure ML all offer Triton-based serving, KServe has a Triton runtime, and the server has --allow-sagemaker and --allow-vertex-ai flags, both off by default, to speak those platforms' request conventions. If you are on a managed platform, see the SageMaker, Vertex AI, Azure ML and KServe guides for how they wrap it. In the Gulf and Egypt, the practical point is region: GPU capacity in a nearby cloud region can be scarce, and model weights loaded from object storage should sit in the same region as the pods, both for load time and to keep data within the residency boundary your employer requires.

Metrics and observability

Triton publishes Prometheus metrics at http://<host>:8002/metrics with no extra setup, which is a large part of why it is pleasant to operate. The skill at this level is knowing which metric answers which question. For the surrounding tooling, see the Prometheus and Grafana guides.

Group the metrics by the question they answer. For traffic and errors: nv_inference_request_success and nv_inference_request_failure are counters per model, so the rate of failures over the rate of total requests is your error ratio. For latency by stage: the _duration_us counters, which are cumulative microseconds, so an average over a window is the rate of the duration counter divided by the rate of the request count.

# Average time in the scheduler queue per request, in milliseconds
(rate(nv_inference_queue_duration_us{model="resnet50"}[5m])
  / rate(nv_inference_request_success{model="resnet50"}[5m])) / 1000

# Average model compute time per request, in milliseconds
(rate(nv_inference_compute_infer_duration_us{model="resnet50"}[5m])
  / rate(nv_inference_request_success{model="resnet50"}[5m])) / 1000

For backlog: nv_inference_pending_request_count is the number of requests currently waiting, a leading indicator that often moves before latency does. For hardware: nv_gpu_utilization, nv_gpu_memory_used_bytes, nv_gpu_power_usage and nv_energy_consumption come from the GPU, with nv_cpu_utilization and the pinned-memory pool metrics for the host side. For caching: nv_cache_num_hits_per_model tells you whether the response cache is earning its memory. And for batching, the ratio described earlier.

Averages hide tails. Triton can report latency as histograms or summaries, enabled through --metrics-config, for example histogram_latencies=true or summary_latencies=true, with summary_quantiles="0.5:0.05,0.9:0.01" to choose summary quantiles. Histograms let Prometheus compute percentiles across replicas, which summaries cannot, so prefer histograms when you run more than one pod. Cheaper still, you can turn the latency counters off with counter_latencies=false when you do not use them, and other flags limit what is exposed: --allow-gpu-metrics=false, --allow-cpu-metrics=false, --allow-metrics=false, --metrics-port, --metrics-address and --metrics-interval-ms.

A dashboard worth building for each model has four rows. Traffic and error ratio. Latency split into queue and compute, so a regression says which stage moved. Batching ratio and pending count, to show whether the batcher and instances match the load. And GPU utilisation and memory, to show headroom. Alert on error ratio and on queue time, not on GPU utilisation alone; a GPU at 40 percent utilisation with growing queue time means instances or batching are the limit, and a GPU at 95 percent with a flat queue is healthy.

Securing a Triton deployment

Triton's trust model is blunt, and understanding it prevents most of the serious mistakes. The official secure-deployment guide states that Triton does not provide a security sandbox for arbitrary model or backend code, and that it has no built-in user authentication or authorisation. A model is code, and the server runs it. So the boundary you build around Triton is the whole security story, and the server must never be exposed directly to untrusted networks.

Put a gateway, an API gateway or a service mesh with mutual TLS in front of it, and let that layer authenticate callers and authorise which models they may call. If you must terminate TLS on Triton itself, gRPC supports --grpc-use-ssl and --grpc-use-ssl-mutual. Then reduce what the server offers:

tritonserver \
  --model-repository=/models \
  --model-control-mode=none \
  --disable-auto-complete-config \
  --allow-http=false \
  --allow-metrics=false

Each flag removes an attack or surprise. --model-control-mode=none closes the load and unload API, which the documentation links to arbitrary code execution when exposed. --disable-auto-complete-config stops the server inferring configuration from files. --allow-http=false and --allow-grpc=false turn off a protocol you do not use, and --allow-metrics=false does the same for metrics where another mechanism collects them. If your platform scrapes metrics, keep them on but ensure only the monitoring system can reach port 8002.

When you need admin APIs available but protected, restrict them rather than disabling them. --http-restricted-api and --grpc-restricted-protocol limit named API groups, such as model-repository and shared-memory, to requests that carry a configured key in a header. That is a lightweight gate, not a substitute for a gateway, but it keeps an accidental exposure from becoming an open administration surface.

Run the container as a non-root user (docker run --user triton-server in the documentation) and treat the repository as a supply-chain input. Anyone who can write to the model repository can execute code in the server, so repository write access should be as controlled as image push access, and models should be built and published by CI, not copied by hand. The 26.08 release notes also flag known vulnerabilities in the vLLM v0 API and in Ray, with the advice to restrict network access if you use those, which is a reminder to read the known-issues section of release notes, not only the feature list.

Cloud repositories and credentials

Production repositories usually live in object storage, so that every replica loads the same artifacts and a deploy is a path change. Triton supports S3, GCS and Azure with distinct URL forms: s3://bucket/path (or s3://host:port/bucket/path for S3-compatible stores), gs://bucket/path and as://account/container/path. You can pass --model-repository more than once to combine repositories, provided model names are unique across them.

tritonserver \
  --model-repository=s3://my-models/prod \
  --model-repository=/local/overrides

Credentials come from the environment, and the guidance is to prefer an identity from the platform to a static key. For GCS, GOOGLE_APPLICATION_CREDENTIALS points to a credentials file. For S3, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION work, but on AWS an IAM role attached to the pod is better because there is no secret to leak or rotate. For Azure, AZURE_STORAGE_ACCOUNT with AZURE_STORAGE_KEY works, and AZURE_STORAGE_AUTH_TYPE=managed_identity uses the workload's identity instead. A beta option, TRITON_CLOUD_CREDENTIAL_PATH, points to a JSON file holding several credentials and picks one by longest-prefix match on the repository path, which is useful when two buckets need two identities.

Debugging beyond the basics

Most problems in a Triton deployment fall into five families: the server will not start, a model will not load, a request is rejected, a request is slow, or something breaks only under load. The method for all of them is the same and comes straight from NVIDIA's debugging guide: establish whether the model works outside Triton first. Run the model in its own framework, with ONNX Runtime, trtexec or PyTorch, on the same input and same GPU. If it fails there, Triton is not the problem, and you have saved an hour.

For a server that will not start, read upward from the final line. error: creating server: Internal - failed to load all models is a summary; the cause is one or more per-model lines above it. Common ones are the batch-size mismatch with a TensorRT plan shown earlier, a missing model file (the backend looks for the default filename and does not find it, so set default_model_filename or fix the name), and a plan built for a different TensorRT version, which reads as an incompatible engine plan. A GPU model started in a container without GPUs reports that it cannot load in CPU-only mode, which is a run-flag problem (--gpus), not a model problem. If a host driver is older than the container's CUDA requires, the container fails at GPU initialisation; the fix is a newer driver on the node or an older container tag.

For a rejected request, the error message names the check that failed: an unknown model (the name is wrong, or the model failed to load), a batch size above max_batch_size, an input name that is not in the config, or a shape that does not match the declared dims. Read the effective config from the server, then compare the request against it line by line. Remember the batch dimension rule from earlier, because a missing or extra leading dimension is the most common cause.

For a slow request, apply the stage method. Queue time high: instances, batching or an upstream pile-up. Compute high: the model, precision or backend. Neither high but client latency high: network, serialisation, or the client itself. Large tensors are a common hidden cost: serialising a big image through JSON over HTTP can dominate. Binary payloads, gRPC and the shared-memory extensions (system and CUDA shared memory, through tritonclient.utils.shared_memory and cuda_shared_memory) exist to avoid copying big tensors between client and server on the same host.

For problems that appear only under load, run Perf Analyzer at rising concurrency and compare. A failure at high concurrency often turns out to be memory: too many instances times too large a batch. Inside Docker, shared memory is another classic: the Python backend uses shared memory between processes, and a container started with the default small shared-memory allocation fails with bus errors or allocation failures as load rises. Start containers with a larger --shm-size, and pass --ulimit memlock=-1 where the documentation advises; check the debugging guide for the sizes it recommends for your backend.

Testing and CI for model serving

Serving configuration is code, and it breaks in ways unit tests cannot see: a renamed tensor, a config that disagrees with the model file, a plan built for the wrong TensorRT. A small, fast pipeline catches nearly all of it before deployment, and CI gives you the place to enforce it. For the pipeline mechanics, see the GitHub Actions guide and GitLab CI guide.

The pipeline has four layers, ordered by cost. First, a static check of the repository: every model has a config.pbtxt, version folders are integers, filenames match the backend defaults, and max_batch_size and dims agree with the exported model's signature. This runs in seconds on any machine with no GPU.

Second, a load test: start the exact container tag you will deploy against the candidate repository in CI, wait for /v2/health/ready, and fail on any model that is not ready. Run it with --disable-auto-complete-config so a config that only works because Triton guessed fails here and not in production. If your CI has no GPU, a CPU-only container can still validate repository structure and CPU models, but it cannot validate a TensorRT plan, which needs the real architecture, so plan builds and GPU load tests belong on GPU runners.

docker run -d --name triton-ci --gpus=1 -p8000:8000 \
  -v "$PWD/model_repository:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models --disable-auto-complete-config

# Wait for readiness, fail the job if it never arrives
for i in $(seq 1 60); do
  curl -sf localhost:8000/v2/health/ready && break
  sleep 5
done
curl -sf localhost:8000/v2/health/ready || { docker logs triton-ci; exit 1; }

Third, a contract test: send a fixed set of real inputs through tritonclient and compare outputs with stored expected values within a tolerance. This catches a wrong preprocessing step, a changed label order and a precision change that hurt accuracy. Keep the golden inputs small and versioned alongside the repository.

Fourth, a performance gate: run Perf Analyzer against the candidate with fixed concurrency and compare throughput and p95 latency against a stored baseline from the previous release. Fail the pipeline if either regresses beyond a margin you agree. Performance on shared CI hardware is noisy, so use dedicated GPU runners where possible, compare relative to a baseline measured on the same runner, and require a regression on two consecutive runs before blocking.

Putting it all together

Here is a realistic path through everything above, as you might run it for an image classifier going to production. It is a checklist in prose, and each step uses a section you have read.

Start with the artifact. The model is exported to ONNX and registered. In CI, a job builds a TensorRT plan inside the 26.08-py3 container for each GPU type you serve, and writes plans and the ONNX file into a versioned prefix in object storage, with a cc_model_filenames entry so a single config selects the right plan per architecture. The config.pbtxt is complete, never relying on auto-complete, with explicit instance groups.

name: "classifier"
backend: "tensorrt"
max_batch_size: 16
input [ { name: "data_0" data_type: TYPE_FP32 dims: [ 3, 224, 224 ] } ]
output [ { name: "fc6_1" data_type: TYPE_FP32 dims: [ 1000 ] } ]
instance_group [ { count: 2 kind: KIND_GPU gpus: [ 0 ] } ]
dynamic_batching { max_queue_delay_microseconds: 100 }
version_policy: { latest: { num_versions: 2 } }
model_warmup [
  {
    name: "warmup_sample"
    batch_size: 1
    inputs { key: "data_0" value: { data_type: TYPE_FP32 dims: [ 3, 224, 224 ] zero_data: true } }
  }
]

The pipeline then proves the build: it starts the container with --disable-auto-complete-config, waits for readiness, runs the contract test on stored inputs, and runs Perf Analyzer with a concurrency range around two times 16 times 2, comparing against the previous baseline. Only a pass promotes the prefix.

The Deployment references the new prefix, runs with --model-control-mode=none, and uses a startup probe sized for the observed load time, readiness and liveness on their separate endpoints, and a pod identity for storage access. A gateway in front authenticates callers and balances requests across replicas, and the Service is not reachable from outside the cluster except through it. Metrics are scraped from port 8002 only by the monitoring system.

On the dashboard, queue time and compute time sit side by side with the batching ratio and pending count. An alert fires on the error ratio and on queue time beyond an agreed bound. When it does, the on-call follows the stage method: check queue time first, then the batching ratio and instance count, then compute time, and only then suspect the model. A rollout of a new model version is a new prefix, a new ReplicaSet, and a readiness-gated switch, and the old prefix stays for rollback. At each monthly upgrade, plans are rebuilt, the contract and performance gates run again, and the release notes are read for driver and CUDA changes before the node pool is touched.

Where you are now

You can now describe how a request moves from the frontend through a scheduler to an instance and a backend, and you can use queue time, compute time and the batching ratio to say which stage is the problem. You can write complete configurations without leaning on auto-complete, choose between the passthrough, dynamic and sequence schedulers, size instance groups and prove the choice with Perf Analyzer. You know why a TensorRT plan is tied to a TensorRT version and a GPU architecture, and you have a process that rebuilds plans with every container upgrade.

You can build pipelines with ensembles, the Python backend and BLS and say when each fits. You run Triton on Kubernetes with probes that account for slow loads, you can scale on signals that mean something for GPU work, and you have a CI pipeline that fails the way production would. You can harden a deployment according to the official guidance, and you understand why the server itself provides no authentication.

Sources

• Triton Inference Server user guide
• Release notes index
• Release 26.08 notes
• Quickstart
• Model configuration
• Model repository
• Model management
• Optimization
• Metrics
• Secure deployment considerations
• Debugging guide
• Triton server source and issue tracker
• NGC tritonserver container catalogue

Part two of three. At this level an interviewer assumes you have run Triton and asks how you would make it fast, keep it up and diagnose it when it is not. The questions are scenarios and trade-offs, and the marks go to reasoning from the request path and the metrics, not to recalling flags. Read the review table first, then the scenarios, then the answers a strong candidate gives that a weak one misses.

Where this picks up

The model in one minute

A request enters over HTTP (8000) or gRPC (8001), is validated against the model's configuration, and goes to the model's scheduler. The scheduler is passthrough, dynamic batcher, sequence batcher or ensemble. It picks an instance, which is one loaded copy of the model bound to a device that runs one execution at a time. The instance calls the backend, and Triton splits batched output back into per-request responses.

Say this aloud in your own words at the start of any performance question, because it lets you place the problem. Time in the scheduler is queue time. Time in the backend is compute time. Time outside Triton is the client and the network. Each has a different fix.

The two counters to know: nv_inference_count counts items inferred, nv_inference_exec_count counts executions. Their ratio is the average batch actually achieved.

The three facts people forget: a TensorRT plan file is tied to a TensorRT version and a GPU architecture; Triton has no built-in authentication; and with --exit-on-error and --strict-readiness at their defaults of true, one failed model stops the server or keeps it unready.

Scenario questions

"Latency doubled after a traffic increase. What do you check first?"

A strong answer starts with the stage split, not with a guess. Pull the average queue duration and the average compute duration per request from the metrics: the rate of nv_inference_queue_duration_us divided by the rate of nv_inference_request_success, and the same for nv_inference_compute_infer_duration_us. Then act on what the split says.

If queue time grew and compute time did not, requests are waiting for an instance. Check nv_inference_pending_request_count, check whether the dynamic batcher is on, and look at the batching ratio. Options are more instances if the GPU has headroom, a better batching configuration, or more replicas. If compute time grew, the model is slower per execution. That can mean larger batches now being formed (compute per execution rises with batch size while per-item cost falls), a GPU throttling, or a co-located model competing for the device. If neither moved but clients see more latency, the problem is outside Triton: serialisation of large payloads, the network, or the gateway.

A weak answer says "add more GPUs" or "increase the batch size" before looking. A strong answer also notes that a rise in compute time per execution can be good news if throughput per item improved, so you look at both per-request and per-item figures.

"Throughput is lower than the GPU can do and utilisation sits at 40 percent."

The device is idle part of the time, so executions are not overlapping or batches are small. Check the batching ratio: near one means each request runs alone. Enable or tune the dynamic batcher. Then check instances: a single instance leaves gaps while it copies data and waits for the next batch, and a second instance overlaps work. Use Perf Analyzer with a concurrency sweep to find the knee, and start the sweep around two times the maximum batch size times the instance count, per the documentation's sizing rule. Also consider the client: a test or a caller that cannot keep enough requests in flight will never fill the GPU, and serialising large tensors through JSON can leave the GPU waiting on the CPU.

"The pod keeps restarting during rollout and the logs show models loading."

The liveness probe is probably killing it. Model loading can take minutes, especially with a model pulled from object storage or a TensorRT engine built at load. A liveness probe that begins counting at container start restarts the pod before it is ready, and the new pod does the same. The fix is a startup probe on /v2/health/ready with a generous failure budget, readiness on the same endpoint, and liveness on /v2/health/live only. A strong answer adds that you measure the real load time and size the budget from it, and that the same load time is the cold-start cost that limits autoscaling.

"After upgrading Triton, a model will not load."

First read the per-model error above the summary line error: creating server: Internal - failed to load all models. If the model is a TensorRT plan, the likely cause is that the engine plan is not compatible with the new TensorRT version: every monthly container bumps CUDA and TensorRT, and a plan is tied to the version and architecture that built it. The fix is to rebuild the plan inside the new container for each GPU type. Then explain the process fix: plans are built in CI in the deploy container, stored keyed by container tag and architecture, and rebuilt at every upgrade. Also mention reading the release notes for driver and CUDA changes, since a host driver older than the container requires will fail at GPU initialisation.

"A request returns an error about batch size."

Two distinct causes share the symptom. If the message says the request batch size must not exceed the model's maximum, the client sent more items than max_batch_size; either lower the client batch or raise the limit if the model and memory allow it. If the error appears at load time with a message about the configured maximum batch size and the model maximum being one, the TensorRT engine was built without a dynamic batch dimension, so the config cannot ask for batching. Set max_batch_size: 0 for that model or rebuild the engine with an explicit batch dimension. Strong candidates also mention the dims rule: with batching on, dims exclude the batch dimension; with max_batch_size: 0, they include everything.

"One bad model takes the whole server down at startup. Is that a bug?"

No, it is the default policy. --exit-on-error is true, so any load failure exits the server. That is often right for a single-purpose service, since running half-broken is worse than failing visibly. For a multi-model server where one model should not block the rest, set --exit-on-error=false; the failed model sits in UNAVAILABLE with a reason, and the others serve. The catch is readiness: --strict-readiness defaults to true, so /v2/health/ready stays not ready while a model is down, and Kubernetes will not send traffic. You choose between strict readiness and partial availability on purpose, and you alert on the unavailable model either way.

"How would you roll out a new model version safely?"

Use version directories and a version policy. Publish the new version as an integer folder beside the old one, with version_policy: { latest: { num_versions: 2 } } so both load. Clients pinning a version keep working, and you can send test traffic to /v2/models/<name>/versions/<v>/infer and compare. Gate on a contract test with stored inputs, and a performance comparison against a baseline, before shifting any production traffic. When satisfied, move the callers and drop the old version. At the platform level, a new repository prefix and a Kubernetes rolling update gated by readiness gives the same safety for the whole repository, and the old prefix stays for rollback. A strong answer notes that poll mode is not recommended for production and that in-flight sequences must finish before a stateful model is reloaded.

"A security review asks whether Triton is safe to expose to the internet."

The answer is no, not directly. Triton provides no sandbox for arbitrary model or backend code and has no built-in authentication or authorisation. A model is code, so anyone who can load one can run code. Put a gateway or a mesh with mutual TLS in front, run with --model-control-mode=none, disable protocols and endpoints you do not use (--allow-http=false, --allow-grpc=false, --allow-metrics=false), restrict administration APIs with --http-restricted-api and --grpc-restricted-protocol, run as a non-root user, and control write access to the model repository as tightly as image push access. Name the Python backend as a particular case: it runs arbitrary code.

Trade-off questions

Dynamic batching versus latency

Dynamic batching trades a little latency for throughput. The queue delay is the explicit price: a request may wait up to max_queue_delay_microseconds for company. Under heavy traffic batches fill instantly and the delay is never paid; under light traffic every request pays it. So the right setting depends on the latency objective and the traffic shape. A good answer proposes starting with the empty dynamic_batching { } block, measuring, raising the delay only if batches are too small, and stopping when throughput stops improving. It also names the limit: the batcher cannot merge requests for a model with state or with inputs that differ in shape.

More instances versus bigger batches

Both use the GPU better, but they pull in opposite directions. More instances overlap executions and hide transfer gaps, but each holds a full copy of the model in device memory and splits the queue, which shrinks batches. Bigger batches raise per-item efficiency but add waiting time and need activation memory. The resolution is measurement with Perf Analyzer across both settings, watching throughput, p95 latency and memory. A weak answer picks one by principle. A strong answer says the optimum is model-specific and shows how to find it.

TensorRT execution accelerator versus a built plan

The accelerator on an ONNX model is the cheap route: add a block to the config, choose FP16, and the ONNX Runtime backend hands eligible subgraphs to TensorRT, with no plan file to manage. The cost is a slower load, since optimisation happens then, and less control. A built plan served by the TensorRT backend usually gives the best performance and full control, but it ties you to the TensorRT version and GPU architecture, so you need a CI process that rebuilds per container tag and per GPU type. Choose the accelerator while you are learning or when the fleet is small, and a built plan when the performance gain justifies the pipeline. In both cases, FP16 changes numerics, so validate accuracy on real data.

Ensemble versus Business Logic Scripting versus a client-side pipeline

An ensemble is a static graph in configuration: tensors move between steps inside the server, each step has its own batching and instances, and there is no code to maintain. It cannot branch or loop. BLS, in the Python backend, calls other models from code, so it can handle data-dependent flow, at the cost of control flow in your own code and a Python process in the hot path. A client-side pipeline is simplest and most flexible but adds network round trips and pushes tensors across the wire between steps. Choose the ensemble when the graph is fixed, BLS when it is not, and the client when the steps are loosely coupled or owned by different teams.

One shared Triton versus one per tenant

A shared server uses GPUs efficiently, but there is no native tenant isolation: GPU memory is shared, one tenant's heavy load or out-of-memory event can affect others, and per-model access control exists only at the gateway. Separate deployments per tenant isolate cleanly and let you size and secure each independently, at higher cost from idle GPUs. MIG partitions give a middle path with hardware-level memory isolation on a card. A good answer asks about the isolation requirement and the cost tolerance before choosing.

Managed platform versus self-run Triton

Triton is also the runtime behind SageMaker, Vertex AI, Azure ML and KServe, and the server has --allow-sagemaker and --allow-vertex-ai flags for those conventions. A managed platform removes cluster operations and brings its own autoscaling and authentication, at the price of its conventions, constraints and sometimes cost. Self-running gives control over probes, GPU sharing and tuning, and makes you the owner of the gateway, the upgrades and the on-call. See the SageMaker, Vertex AI and KServe guides for how each wraps the server. For teams in the Gulf and Egypt, region availability of GPU instances and data-residency rules often decide this before technology does.

What a strong answer covers that a weak one misses

Quick-fire questions

What do the ports 8000, 8001 and 8002 serve? HTTP, gRPC and Prometheus metrics.

What do the health endpoints do? /v2/health/live says the process is up; /v2/health/ready says the models are ready, under strict readiness.

What does max_batch_size: 0 mean? No batching by Triton; dims then describe the whole tensor.

Where do you see the configuration the server really uses? GET /v2/models/<name>/config.

What is a decoupled model? One whose responses are not one per request, such as streaming or LLM output, using model_transaction_policy { decoupled: true } over gRPC streaming.

What are the model states? UNAVAILABLE, LOADING, READY and UNLOADING.

Which control mode do you run in production? none, with changes shipped as a new deployment.

How do you choose a file per GPU type? cc_model_filenames, or --model-config-name for per-GPU config files.

What sizes the concurrency range for a sweep? About two times the maximum batch size times the instance count.

Which tool searches configurations automatically? Model Analyzer.

How do you stop two models fighting over a GPU? The rate limiter, started with --rate-limit=execution_count, with named resources and priorities on instance groups.

A self-test

Answer each aloud in under thirty seconds.

1. Name the three stages a request spends time in and the metric for each.
2. What does a batching ratio of 1.1 on a model with max_batch_size: 16 tell you?
3. Why does the liveness probe not use the ready endpoint?
4. Why can a plan built for last month's container fail today?
5. What happens at startup when one model fails and --exit-on-error is at its default?
6. Name three flags you would set to harden a Triton deployment.
7. When is an ensemble not enough and you need BLS?
8. What does omitting instance_group do on a four-GPU host?

If you stumble on more than two, return to the matching section of the detailed file. Senior takes the same topics to platform scale: what breaks first as load, model count and tenant count rise, how upgrades and migrations run across a fleet, and when to choose a different serving stack.

Part two of three. At this level your models load and answer requests, so the problems change character. They stop being "why will not it start" and become "why is the GPU idle", "why did it break after the upgrade" and "why does this pod keep restarting". Start with the error table, then the practices in order of leverage, then the practice cards and operational recipes.

Common errors at this level

Cumulative: Beginner's errors still apply. These appear once things work.

The practices that pay off most, in order of leverage

1. Measure by stage before you tune

Queue time and compute time answer the question "what do I change". Compute the averages from the metrics or read them from Perf Analyzer's breakdown, and name the stage before touching a setting.

2. Prove every claim with Perf Analyzer

Run a concurrency sweep for each change, one change at a time, and keep the output. A speedup you cannot reproduce is a rumour.

3. Rebuild plans per container and per GPU, in CI

The incident that stops a Triton upgrade is nearly always an incompatible plan. Make the rebuild automatic and keyed by container tag and architecture, so an upgrade is a pipeline run and not a surprise.

4. Write complete configs and disable auto-complete

A config that works only because the server guessed is a deployment waiting to fail. Copy the effective config from GET /v2/models/<name>/config into your files, then start with --disable-auto-complete-config.

5. Give probes the right jobs

Liveness on /v2/health/live, readiness and a generous startup probe on /v2/health/ready. The budget comes from measured load time.

6. Watch the batching ratio

nv_inference_count / nv_inference_exec_count shows whether the batcher is doing its job. It belongs on the dashboard next to pending count and queue time.

7. Put a gateway in front and run none mode

Triton has no built-in authentication and runs model code. Gateway authentication, mutual TLS, --model-control-mode=none and restricted administration APIs are the baseline, not an extra.

8. Treat the repository as a release artifact

Version by prefix, promote by changing the path, and keep the previous prefix for rollback. Control write access like image push access.

Practice cards

1. Split the latencyRun a Perf Analyzer sweep, then compute average queue time and compute time per request from /metrics. Say which stage dominates and which section of the detailed file is your first move.
2. Find the kneeSweep --concurrency-range 1:16. Mark where throughput stops rising and compare to two times your batch size times your instance count.
3. Prove batchingTest with no batching, with dynamic_batching { }, and with a 100 microsecond delay. Record throughput, p95 and the batching ratio for each.
4. Price an instanceRaise count from one to two and watch nv_gpu_memory_used_bytes. Compute the memory cost per instance and the throughput gained.
5. Read the effective configFetch the config from the HTTP endpoint for a model with a minimal file. List what Triton filled in, copy it into the file, and start with --disable-auto-complete-config.
6. Break a load on purposeMisspell a model filename, then change a tensor name in the config. Record the exact messages and fix each from the effective config and the repository listing.
7. Kill a slow loaderPoint liveness at the ready endpoint on a model that takes 30 seconds to load, watch the restart loop, then fix it with a startup probe.
8. Rebuild a planBuild a TensorRT plan in one container tag, try to load it in another, and read the error. Script the rebuild.
9. Harden and compareList what is reachable on a default server, apply the hardening flags, and list again.

Operational recipes

Recipe: check what is running

curl -s localhost:8000/v2                          # server metadata and extensions
curl -s localhost:8000/v2/health/ready -o /dev/null -w '%{http_code}\n'
curl -s localhost:8000/v2/models/classifier        # versions and I/O metadata
curl -s localhost:8000/v2/models/classifier/config # the effective configuration
curl -s -X POST localhost:8000/v2/repository/index # states, in explicit mode

A 200 from the ready endpoint means every model is ready under strict readiness. The repository index lists each model with its state, which is where a model stuck in UNAVAILABLE shows its reason.

Recipe: a latency investigation in six steps

Recipe: sweep and capture a baseline

docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.08-py3-sdk \
  perf_analyzer -m classifier --percentile=95 --concurrency-range 1:16 \
  | tee sweep-$(date +%F).txt

Keep the output beside the config that produced it. After the next container upgrade, run the same command and diff the two files.

Recipe: batching ratio and stage times in PromQL

# Batching efficiency
rate(nv_inference_count{model="classifier"}[5m])
  / rate(nv_inference_exec_count{model="classifier"}[5m])

# Error ratio
rate(nv_inference_request_failure{model="classifier"}[5m])
  / (rate(nv_inference_request_success{model="classifier"}[5m])
     + rate(nv_inference_request_failure{model="classifier"}[5m]))

# Average queue milliseconds
(rate(nv_inference_queue_duration_us{model="classifier"}[5m])
  / rate(nv_inference_request_success{model="classifier"}[5m])) / 1000

The Prometheus guide covers rate() and recording rules, and the Grafana guide covers building the dashboard. The model label name depends on how your scrape is configured, so confirm it against /metrics.

Recipe: a CI readiness gate

set -euo pipefail
docker run -d --name triton-ci --gpus=1 -p8000:8000 \
  -v "$PWD/model_repository:/models" \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models --disable-auto-complete-config

ready=0
for i in $(seq 1 60); do
  if curl -sf localhost:8000/v2/health/ready >/dev/null; then ready=1; break; fi
  sleep 5
done
if [ "$ready" -ne 1 ]; then docker logs triton-ci; docker rm -f triton-ci; exit 1; fi

A server that never becomes ready dumps its log, which holds the per-model error. See the GitHub Actions guide for putting this in a workflow on a GPU runner.

Recipe: a Kubernetes probe block

startupProbe:
  httpGet: { path: /v2/health/ready, port: 8000 }
  periodSeconds: 10
  failureThreshold: 60
readinessProbe:
  httpGet: { path: /v2/health/ready, port: 8000 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /v2/health/live, port: 8000 }
  periodSeconds: 10

Set the startup budget (periodSeconds times failureThreshold) from the longest observed load time plus margin. A budget of ten minutes here is generous; reduce it once you know your own number.

Recipe: a hardened start

tritonserver \
  --model-repository=/models \
  --model-control-mode=none \
  --disable-auto-complete-config \
  --allow-http=false \
  --allow-metrics=false

Drop --allow-http=false if your clients use HTTP, and drop --allow-metrics=false if the metrics endpoint is how you monitor. Keep everything the service genuinely needs, and remove the rest. Whatever you keep stays behind a gateway.

Recipe: roll a model version

Recipe: upgrade the container

Pin the new tag in a branch. Rebuild every TensorRT plan inside it for each GPU type you serve. Run the load test and contract test with --disable-auto-complete-config. Run the Perf Analyzer comparison against the previous baseline. Read the release notes for the CUDA and driver requirements and confirm the node driver satisfies them before the node pool is touched. Roll out with readiness gating, and keep the old image tag ready for rollback.

Reading the errors that look alike

Several Triton failures produce similar symptoms but have opposite fixes, and telling them apart quickly is most of the skill at this level.

Load-time versus request-time errors. A load-time error appears in the server log while a model moves from LOADING to UNAVAILABLE, and it means the model file, the config or the environment is wrong. A request-time error comes back to the caller and means the request disagrees with a model that did load. If a model is UNAVAILABLE, nothing about the request matters yet, so fix the load first. The repository index and the model's ready endpoint tell you which case you are in before you read any log.

"Unknown model" versus "no available versions". An unknown model means the name is wrong or the model never loaded. A model with no available versions means the model directory exists but no integer version folder loaded, often because the folder was named something non-numeric, or because the version policy excluded the one you have. Compare the folders on disk with the versions listed in the model metadata.

Shape errors versus batch errors. If the server complains about a shape, compare the request with the effective config, remembering that dims exclude the batch dimension when max_batch_size is above zero. If it complains about batch size, the request carried more items than max_batch_size allows. Both look like "the input is wrong", but the first is about axes and the second is about a count.

Plan errors versus model errors. An engine that will not deserialise is a compatibility problem between the plan and the container or GPU, and no amount of editing the config helps. A model that loads but gives wrong outputs is a different class: look at preprocessing, label order and precision. Running the model in its own framework on the same input, outside Triton, separates the two in minutes, which is why the debugging guide puts it first.

Resource errors under load. If a server is healthy at low concurrency and fails as load rises, suspect memory before suspecting logic. Count instances times model size, add activation memory at your batch sizes, and watch nv_gpu_memory_used_bytes. For Python models in containers, suspect shared memory next. These failures are intermittent in a way that logic errors are not, and they correlate with concurrency, which is a strong clue.

Habits for the week after an upgrade

An upgrade is the moment when most Triton incidents begin, so it is worth having a routine. Before rollout, rebuild plans and run the gates. During rollout, watch error ratio, queue time and the batching ratio for the new pods against the old ones; a difference between old and new pods running side by side is far easier to see than a change over time. After rollout, keep the previous tag and repository prefix available for a full week, since some problems only show with the weekly traffic shape.

Read the known-issues part of the release notes, not just the enhancements. The 26.08 notes list security concerns with the vLLM v0 API and Ray, a CuPy multithreading issue with CUDA 13, and limits on vLLM tensor parallelism in explicit control mode. None of those matter to a team serving ONNX models, and all of them matter a great deal to a team that uses the feature in question. Reading takes ten minutes and turns an unexplained failure into a known one.

Finally, write down what you learned. A short runbook with the exact error text your team has seen, the cause and the fix is worth more than any dashboard during an incident at night. Keep it next to the config in version control so it changes when the system changes. Add each new incident to it as part of the postmortem, and prune what a later release has made obsolete. A runbook that is trusted gets used; one that is stale gets ignored.

A closing checklist

• Latency problems start with the queue-versus-compute split.
• Each tuning change has a Perf Analyzer sweep before and after.
• Configs are complete and the server runs with auto-complete disabled.
• Plans are built in the deploy container for every GPU type, keyed by container tag.
• Instance groups state count, kind and gpus.
• Probes: startup and readiness on the ready endpoint, liveness on the live endpoint.
• The batching ratio, pending count and queue time are on the dashboard.
• A gateway fronts the server, --model-control-mode=none is set, and the repository is write-controlled.
• CI starts the real container tag and fails on any unready model.
• The tag is pinned, and upgrades follow the checklist above.

Senior takes these further into platform ownership: what fails first at scale, how tenants share GPUs, how upgrades run across a fleet, and the review checklists you apply to other teams' serving designs.

This is part three of three. It takes every topic from Beginner and Mid one level further and adds the work you own when Triton is a platform other teams depend on, rather than a server you run for your own model.

Up to this point the questions were about making models serve fast. From here they are different: which component of the stack gives out first when traffic doubles, what can a model file do to the host that loads it, who is allowed to change what is served, what a monthly container bump does to a fleet of TensorRT engines, and how you tell the team that owns the dashboard that their p99 is a symptom and not the disease.

Where this picks up

I am starting with the failure model, because every capacity and security decision below follows from knowing how the server is put together and where it is brittle.

How Triton is put together, and where it breaks

You know from Mid that a request enters through HTTP or gRPC, waits in a scheduler queue, executes on a model instance through a backend, and returns. At platform scale it helps to see that as five layers, each with its own failure signature.

The first consequence is about blast radius. Triton loads backends as shared libraries inside the server process. A native crash in a custom backend, a segmentation fault in a library a Python model imported, or an out-of-memory condition on the GPU can take down every model on that server, including the ones that were healthy. The Python backend does run model code in separate stub processes, which contains many Python-level failures, but the official secure-deployment guidance is plain that Triton does not provide a security sandbox for arbitrary model or backend code. Treat a Triton pod as a single failure domain and decide deliberately what shares it.

The second consequence is the load path. A model is UNAVAILABLE, LOADING, READY or UNLOADING, and most platform incidents are a model stuck in the first state. With --exit-on-error=true, which is the default, one model that fails to load stops the whole server from starting, and you see error: creating server: Internal - failed to load all models after the real cause scrolled past a few lines earlier. That default is the right one for a production fleet because it turns a bad model into a failed rollout rather than a half-working replica. With --strict-readiness also true by default, /v2/health/ready stays unhealthy while any model is not ready, so Kubernetes will not send traffic to a replica missing a model it is supposed to serve. Teams new to the platform sometimes switch both off "to make it stay up", and they then discover at 2am that a replica returns Request for unknown model for a model that silently failed to load. Keep both defaults, and teach teams to read the per-model error lines.

The third consequence is that load and unload are heavy, stateful operations. Loading allocates GPU memory, may build TensorRT execution contexts, runs warmup requests if you configured model_warmup, and for cloud repositories first downloads the files. Docs warn that you must not modify files in the repository while a model is loading or unloading, and that in-flight sequences for a stateful model must finish before the model is reloaded. A platform that lets teams push files straight into a live repository will eventually corrupt a load. The remedy is structural: immutable, versioned repository snapshots, which the next sections build up.

Scaling: which component breaks first

"Add more GPUs" is the answer of last resort. The senior skill is naming the saturated resource, because the fixes differ and some of them make things worse. The order below is the order in which I check, from cheapest to most expensive.

The client and the network. Large tensors serialised as JSON over HTTP can spend more time in parsing than on the GPU. The HTTP infer body carries tensors as JSON arrays, and a 3x224x224 FP32 image is about 150,000 numbers. Binary tensor data over HTTP, or gRPC, removes most of that cost, and the system and CUDA shared memory extensions avoid copying altogether when client and server are on the same host. The tell is high nv_inference_request_duration_us with low compute durations and a low GPU utilisation: time is going somewhere other than inference.

The queue. If nv_inference_queue_duration_us dominates the request duration, requests are waiting for an instance. The cause is either too few instances, a dynamic batcher holding requests for max_queue_delay_microseconds longer than your latency budget allows, or a backend that is slower than the arrival rate. Do not guess between them. Batching efficiency, computed as nv_inference_count / nv_inference_exec_count, tells you how many requests share an execution. A ratio near 1 under heavy load means the batcher is not batching, usually because max_batch_size is 1 or the model does not support a batch dimension. A ratio near your max_batch_size with a growing queue means the GPU is genuinely saturated.

The GPU. Utilisation from nv_gpu_utilization is a coarse number: it reports whether a kernel was running during the sample, not how much of the device was used. A model can show 100 percent utilisation while using a fraction of the streaming multiprocessors. Before buying hardware, try concurrency: more instances of a small model on the same GPU let kernels overlap. The docs give a rule for the load you should test with: concurrency for maximum throughput is about two times max batch size times the instance count. Run Perf Analyzer with --concurrency-range across that value and look for the knee where throughput stops rising and latency starts to.

GPU memory. Every instance of every loaded model holds memory. Adding instances to chase throughput, then adding models, ends in an allocation failure at load time or, worse, at the first large request. Memory is the budget that limits how many models one GPU can host, and it is why a model count per node is a platform policy and not a team decision.

Model load time and horizontal scaling. Replica count is the usual answer, but a new Triton pod must pull a large container image, download the repository, load engines and warm up. For big models that is minutes. Horizontal autoscaling on request metrics therefore lags the traffic it is trying to follow. Scale on a leading signal such as nv_inference_pending_request_count or queue duration rather than on GPU utilisation, keep a floor of warm replicas sized for your normal peak, and treat scale-to-zero as acceptable only for models whose users tolerate a multi-minute cold start.

For LLM serving the scaling story changes shape. Decoupled models stream tokens over gRPC, and throughput depends on KV-cache memory and continuous batching in the backend, not on the dynamic batcher. Triton hosts the TensorRT-LLM and vLLM backends, but if LLM serving is most of your load, read the dedicated guides for TensorRT-LLM and vLLM, and read the known-issues section of each Triton release for them, because 26.08 lists constraints on tensor parallelism with explicit model control mode.

The repository as a supply chain

At small scale the model repository is a folder. At platform scale it is the most sensitive artifact you own, because Triton will load whatever is in it and run it with the server's privileges. A TensorRT plan is a binary deserialised by native code; a Python backend model.py is arbitrary Python; a custom backend is a shared library. The secure-deployment guidance says this directly: dynamic model updates can lead to arbitrary code execution, and only trusted models should be loaded. Whoever can write to the repository can run code on your GPU nodes.

That reframes the repository as a software supply chain with the same controls you would apply to a container image.

In practice that means four rules. First, repositories are produced by a pipeline, not edited by hand. The pipeline converts or exports the model, writes the version directory and config.pbtxt, runs the model in its native framework to confirm correctness (the debugging guide's own advice is to verify the model runs outside Triton first), runs Perf Analyzer against a budget, and publishes a snapshot. Experiment tracking such as MLflow is a good source of the lineage you attach to each snapshot: training run, dataset version, git commit.

Second, snapshots are immutable and addressed by something that changes when content changes: a bucket prefix with a version or content hash, or an OCI artifact. A promotion changes which snapshot an environment points at; it never edits files in place. This also answers the "do not modify files during load" warning, since a loaded snapshot is never touched.

Third, the config.pbtxt is reviewed like code. It decides instance counts, batching delay, memory footprint and, through the backend field and any Python code, what executes. Keep it in version control beside the model's conversion code and require review from the platform team for changes to instance groups, parameters and anything that sets an environment or custom execution path.

Fourth, you decide which kinds of artifacts the platform accepts. A sensible tiering is that TensorRT, ONNX and TorchScript models are accepted through the standard pipeline, Python backend models require a security review because they run arbitrary code, and custom C++ backends and repository agents are accepted only from the platform team. Writing that down is governance, and it is the cheapest security control you will implement.

Model control in production

The control mode is the knob that connects the supply-chain discussion to the running server. Triton offers three, and the platform choice has consequences for rollout, security and incident response.

--model-control-mode=none is the default. Triton loads every model at start-up and ignores later changes to the repository. The secure-deployment guide recommends it for production, and I agree: the set of models a server runs is fixed by what you started it with, so a deployment is an immutable unit, a new rollout is how anything changes, and there is no API through which a model can be swapped at runtime. This pairs naturally with Kubernetes: a new snapshot means a new Deployment revision with a rolling update gated by readiness probes.

--model-control-mode=explicit starts with nothing loaded unless you pass --load-model=<name> (or --load-model=*), and exposes the repository API: POST /v2/repository/models/<name>/load, .../unload, and POST /v2/repository/index. Its appeal is flexibility: a long-lived server can host many models and load them on demand, which helps with memory-constrained nodes serving a long tail of rarely used models. Its cost is that the repository API becomes a privileged control plane. Anyone who can call load can make the server read from whatever repository path is configured and run what it finds, which is why the docs treat dynamic updates as a route to arbitrary code execution.

If you do run explicit mode, three controls are not optional. Restrict the repository API with --http-restricted-api and --grpc-restricted-protocol so that the model-repository category requires a key header, and have only your control plane hold that key. Never expose the endpoint to teams or the internet directly. And make sure the control plane serialises operations, because reloading a stateful model while sequences are in flight is unsafe.

Finally, version management. A version_policy of latest with num_versions: 2 keeps two versions loaded and gives you an in-server rollback path; specific lets you pin versions; all loads everything and rarely belongs in production, since each loaded version consumes GPU memory. Clients choose a version in the request path, so decide whether they may. A platform convention that works is that consumers call the unversioned model name, resolving to the latest, and the platform controls what latest means through the snapshot it deploys.

The trust model and securing the endpoint

Triton has no built-in authentication or authorisation. That sentence should shape your architecture more than any other in this section. Out of the box, any client that can reach port 8000 or 8001 can list models, read their configuration, send inference requests and, depending on flags, load and unload models and register shared memory regions. The official guidance is to deploy Triton behind a gateway or proxy, never exposed directly to untrusted networks, and to restrict what the server itself offers.

Work through the flags the secure-deployment guide recommends, as a hardening baseline.

• Disable what you do not use. --allow-http=false or --allow-grpc=false if only one protocol is used; --allow-metrics=false where you do not scrape. --allow-sagemaker defaults to false and --allow-vertex-ai should stay off unless you run on those services.
• Restrict administrative APIs. --http-restricted-api and --grpc-restricted-protocol limit categories such as model-repository and shared-memory to requests that carry a configured key header. The shared-memory category matters: registering a shared memory region lets a caller point the server at memory it should not read, so keep it behind the key unless clients are co-located and trusted.
• Freeze configuration. --disable-auto-complete-config forces explicit configs, which means what you reviewed is what runs. Combined with none control mode, the runtime has no surprises.
• Keep tracing off by default. --trace-config level=off is the default, and trace output can carry data you do not want to ship around. Turn it on for an investigation and off afterwards.
• Encrypt where there is no mesh. --grpc-use-ssl and --grpc-use-ssl-mutual with the server certificate, key and root certificate options provide TLS and mTLS on the gRPC endpoint. If a mesh already provides mTLS between pods, terminate there and keep the server's plain ports on the pod network only.
• Run unprivileged. The container user triton-server exists for this: docker run --user triton-server, or runAsUser, runAsNonRoot and a read-only root filesystem in Kubernetes, with writable emptyDirs for the paths Triton needs. Test that a read-only filesystem works with your backends, since some write caches.

On top of the flags, add the controls that surround the server. A Kubernetes NetworkPolicy that allows ingress to the inference ports only from the gateway namespace and to the metrics port only from your Prometheus. No egress from the pod except to the model repository store and the telemetry collector; a compromised model that cannot call out is a much smaller incident. And do not mount the container runtime socket or cloud credentials broader than the repository needs.

Two known issues deserve explicit mention because they come from the project's own release notes. The 26.08 notes list security vulnerabilities in the vLLM v0 API and in Ray, and tell you to restrict network access. If you run those backends, that is not a footnote: put them on a tighter network segment, and track the release notes for the fix before you widen exposure. More generally, subscribe to the monthly release notes as a security feed, not only a feature list.

Secrets and cloud repositories

When models live in object storage, Triton needs credentials to read them, and that is where secrets enter the picture. Cloud paths take the forms s3://bucket/path, s3://host:port/bucket/path for S3-compatible stores, gs://bucket/path and as://account/container/path for Azure. You can pass --model-repository more than once to serve from several repositories, provided model names are unique across them.

Credentials come from the standard environment variables per provider. For Google Cloud Storage, GOOGLE_APPLICATION_CREDENTIALS. For S3, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION, or the AWS configuration and an IAM role. For Azure, AZURE_STORAGE_ACCOUNT with AZURE_STORAGE_KEY, or AZURE_STORAGE_AUTH_TYPE=managed_identity. A beta feature, TRITON_CLOUD_CREDENTIAL_PATH, points at a JSON file holding several credentials with longest-prefix matching, which is useful for a server reading from multiple buckets. Being beta, pin and test it before you depend on it.

The platform rule is simple: no long-lived keys in the pod spec. Use the identity mechanism of your cluster, whether IAM roles for service accounts, workload identity or managed identity, so the pod receives short-lived credentials scoped to read one repository prefix. Where that is not possible, mount credentials from a secret manager such as Vault rather than hard-coding them in manifests or images. Give the Triton identity read-only access: the server never needs to write to the repository, and a read-only role means that a compromised server cannot tamper with the artifacts it loads.

A subtlety is that cloud repositories are downloaded to a local directory before loading, controlled by the mount directory variables TRITON_GCS_MOUNT_DIRECTORY, TRITON_AWS_MOUNT_DIRECTORY and TRITON_AZURE_MOUNT_DIRECTORY. That local copy consumes node disk. A pod with dozens of gigabytes of engines on an undersized ephemeral volume is evicted for disk pressure during rollout, which looks like a mysterious crash loop. Size ephemeral storage for the repository, set the directory to a volume you chose, and put it in the capacity plan.

Multi-tenancy: what Triton does and does not give you

Many teams will ask to "just put our model on the shared Triton". Whether you should depends on an honest account of the isolation Triton offers, and it is thin. There is no native tenant concept: no per-tenant authentication, no per-tenant quotas, no separate address spaces. GPU memory is shared by all loaded models on a device, so one tenant's large allocation or out-of-memory failure can affect another. The server is a single process, so a crash in one tenant's backend is everyone's outage.

A workable middle path is tiering by risk. Models from the standard pipeline, in supported formats, without custom code, share servers grouped by latency class. Models that run custom Python or need special backends get dedicated servers. Anything the platform team does not control, such as a customer-supplied model, never shares a server with other tenants and may need stronger isolation than a pod, which is outside what Triton can promise.

Within a shared server, the tools you do have are these. The rate limiter (--rate-limit=execution_count with named resources in each instance group's rate_limiter block, plus a priority) limits concurrent executions across models, so you can stop one model's instances from monopolising the device. Instance groups bound each model's concurrency. MIG on supported data-centre GPUs partitions a physical GPU into isolated slices with their own memory, which is the closest thing to real GPU isolation, and you can run one Triton per slice. And at the gateway, per-consumer quotas shape who sends traffic at all. None of these makes a shared server safe for hostile tenants; they make it fair for cooperating ones.

Metrics are per model, so chargeback and noisy-neighbour analysis work at model granularity. nv_inference_count and nv_energy_consumption give you usage and energy; combined with an owner label added at scrape time from your deployment metadata, you can show each team what it consumed. The --http-header-forward-pattern option forwards matching request headers to backends as request parameters, which can carry a tenant identifier into a Python model for logging; verify that flag against your release before relying on it.

Cost: GPU utilisation is the lever

GPUs are the largest line on the bill, and most waste on a Triton platform comes from a handful of patterns. The levers, in rough order of return, are these.

Batch efficiently. A dynamic batcher that actually forms batches multiplies throughput per GPU; the official optimisation guide's example went from roughly 73 to 272 inferences per second at eight concurrent requests. The price is queue delay, so batching is a latency-for-throughput trade you set per model against its budget through max_queue_delay_microseconds. A platform should publish defaults per latency class instead of letting each team rediscover them.

Pack models onto GPUs. Several small models can share a device through concurrent execution. A single model on a GPU that sits at 15 percent utilisation is paying for 85 percent idle. The rate limiter and instance counts let you pack without letting one model starve the rest.

Reduce precision where accuracy allows. FP16 or INT8 engines roughly halve memory and often improve throughput, with the accuracy cost depending on the model. The ONNX Runtime TensorRT accelerator path (gpu_execution_accelerator with precision_mode FP16) gives a cheap first step and the official example reports about double the throughput and half the latency. Treat that figure as an example from a specific model, not a guarantee, and gate any precision change on an accuracy evaluation.

Partition with MIG when many small, steady workloads each need isolation, and cache repeated requests with the response cache where inputs repeat: the server needs --cache-config with a local cache and the model needs response_cache { enable: true }. Check the exact --cache-config syntax in the docs for your release before rolling it out, and measure nv_cache_num_hits_per_model to confirm that it earns its memory.

Cost needs an owner and a number. A useful platform metric is cost per thousand inferences per model, computed from GPU hours allocated divided by nv_inference_count. When a team's model costs ten times its neighbours, that is a conversation about batching and precision, held with evidence.

Observability and service levels

The Mid level taught you the metrics endpoint. The platform job is to turn those series into promises. Triton publishes Prometheus metrics on port 8002: success and failure counters, request, queue and compute durations split into input, infer and output phases, pending request counts, GPU and CPU utilisation, memory, power and energy, pinned memory pool usage and cache hits. Your Prometheus and Grafana setup should treat a Triton dashboard as a product.

Start from the service-level indicators users feel. Availability is the ratio of nv_inference_request_success to total requests per model; failures include model errors and invalid requests, so separate client errors (bad shapes) from server errors before you alert, since a team shipping a bad client will otherwise page you. Latency comes from nv_inference_request_duration_us. By default those are counters, and for percentile latency you enable histograms with --metrics-config histogram_latencies=true, or summaries with summary_latencies=true and summary_quantiles. Decide which you need: histograms aggregate across replicas correctly, while summaries do not, so for a fleet prefer histograms. Counter-style latency metrics, counter_latencies, remain available if you want cumulative durations for averages.

The split into queue and compute durations is your diagnostic tool. Latency from queue delay points at capacity or batching configuration; latency from compute points at the model or the GPU; latency from input and output phases points at data movement and serialisation. Put the three on the same panel for every model, and your on-call engineers will answer "why is this slow" in a minute.

Write alerts about symptoms and leading indicators. A page for sustained error ratio per model. A page for readiness failures across more than a threshold of replicas. A warning when queue duration exceeds a fraction of the latency budget for several minutes, which fires before users complain. A warning when GPU memory use approaches the device limit, since that predicts load failures. Avoid paging on GPU utilisation.

Upgrades and migrations

Triton ships monthly, with tags such as 26.08, and the server version moves with them: the 26.08 container is Triton 2.72.0. Each container bundles specific CUDA, TensorRT, ONNX Runtime, OpenVINO, PyTorch and driver requirements. At fleet scale the monthly cadence is a hazard if you follow it blindly, and a security risk if you ignore it. The discipline sits between the two.

Pin the container tag, always. Never deploy latest or a floating tag. Record the tag in the repository snapshot's metadata and in the Deployment, so an incident responder can answer "what exactly was running" immediately.

Understand what the container bump drags with it. The 26.08 notes list CUDA 13.4.1 and TensorRT 11.2.1.2 among others. A newer CUDA requires a sufficiently recent host driver; the exact minimum is on the release page and in NVIDIA's compatibility documentation, and I am not going to quote a number from memory. The practical rule is to read the "driver requirements" for the target release and compare against your node image, then upgrade drivers first in a canary node pool. GPU support also narrows over time: current containers target compute capability 7.5 and newer, so check older cards, such as Pascal or Volta fleet, before you plan a bump.

Rebuild TensorRT plans. A model.plan is tied to the TensorRT version and the GPU architecture that produced it. After a container upgrade, a plan from the previous TensorRT will typically fail to load with a message that the engine plan is not compatible with this version of TensorRT; the exact wording varies. Make the engine build a pipeline stage that runs inside the matching container on the matching GPU type, not a laptop step, and key your snapshot by container tag and architecture. A multi-GPU-type fleet needs a plan per architecture, which the cc_model_filenames configuration (a map from compute capability to filename) can select, or model-specific configs through --model-config-name.

Promote through environments with a performance gate. The upgrade test is not "does it start". It is: run the standard Perf Analyzer suite against the new container on representative models, compare throughput and p99 against the baseline, check output equivalence on a golden request set, and only then roll out. Backend upgrades can change numerics and performance; the 26.08 notes alone mention scheduler fixes, CUDA graph changes and a vLLM parsing fix, any of which could shift behaviour.

Roll out with a canary and an instant rollback. A rolling update gated by /v2/health/ready and a startup probe sized for model load time, with maxUnavailable low enough to keep capacity, plus a canary replica receiving a slice of traffic, catches most regressions. Keep the previous container tag and previous snapshots addressable so rollback is a revert of one commit, which pairs naturally with a GitOps tool such as Argo CD and packaging in Helm.

Read the known issues. Each release lists them. 26.08 mentions, for example, CuPy multithreading with the CUDA 13 device API, limits on vLLM tensor parallelism in explicit mode, and Java CAPI segfault reports. A known issue touching a feature you use is a reason to hold a release.

Cadence is a policy. A reasonable one: track every monthly release in a test environment, promote to production on a quarterly cadence, and accelerate for security fixes. Document it so teams can plan their own engine rebuilds.

Migrations out of Triton, or into it from a framework-native server like TorchServe, follow the same logic: run both in parallel, shadow traffic, compare outputs and latency, and move consumers gradually. The KServe v2 protocol that Triton speaks makes the client side of that move easier than most.

Governance

Governance is what keeps ten teams' models from becoming ten different operating models. It does not have to be heavy; it has to be written down and enforced by automation, because a wiki page nobody can break is a policy and one that depends on memory is a wish.

Define the onboarding contract: what a team must provide to get a model on the platform. A model in a supported format; a config.pbtxt that passes lint; declared input and output shapes; a latency and throughput budget; an owner and an on-call contact; a Perf Analyzer report on the target GPU type; a rollback plan. The platform provides in return: the endpoint, dashboards, alerts, scaling, and a support channel.

Define tiers, and make them visible in the repository structure or labels: a standard tier for supported formats on shared servers, a restricted tier for Python backend and custom code on dedicated servers, and a platform-only tier for custom backends and repository agents. Each tier maps to the isolation discussion above.

Enforce with policy as code in the pipeline and the cluster. Config lint in CI can reject --model-control-mode changes, missing max_batch_size decisions, unbounded instance counts and unreviewed parameters entries. An admission policy can require the pinned container tag from your registry, a non-root security context, the network policy label, and resource requests including nvidia.com/gpu. Image scanning of the Triton container and any derived images with a scanner like Trivy should be a gate with a documented exception process; the NGC image bundles a large number of libraries, and findings in them are routine, so gate on fixable high and critical issues rather than on everything.

Incident playbooks

Platform on-call is mostly pattern recognition. The playbooks below are the ones I would write first, each starting from the symptom and the evidence.

Server will not start, or crash-loops at start-up. Read the logs from the top, not the bottom: the final line failed to load all models is a summary, and the cause is an earlier per-model error. Typical causes are a missing or misnamed model file (the default names are model.plan, model.onnx, model.pt, so check default_model_filename), a config that disagrees with the model (a batch size above what a TensorRT engine without a batch dimension supports produces an error about the model maximum being 1), an engine plan from another TensorRT version or GPU architecture, GPU-only models on a CPU-only container or without --gpus, or a host driver older than the container's CUDA. First action: roll back to the last known-good container tag and snapshot, then investigate off the hot path. Before blaming Triton, run the model in its native framework to see whether the model itself is broken.

Replicas healthy but requests fail with unknown model. Check GET /v2/models/<name>/ready and the repository index. If strict readiness was disabled, a model may be down while the replica reports ready. Fix the model; restore strict readiness.

Latency spike with no error increase. Compare queue, input, compute and output durations. Rising queue time with steady compute: capacity or batching delay, check pending requests and replica count and whether an autoscaler lagged. Rising compute time: a changed model or GPU contention from a new tenant, or thermal or power limiting, visible in nv_gpu_power_usage. Rising input and output time: payload sizes changed, a client switched from binary to JSON, or the node's CPU is saturated.

GPU out of memory. Out-of-memory at load means the sum of instance footprints exceeds the device, so reduce instances, unload models, or move models. Out-of-memory at request time means a large request or batch pushed allocation past the limit; reduce max_batch_size, or add request size limits at the gateway. Check nv_gpu_memory_used_bytes before and after the incident for the trend.

Python backend failures. ModuleNotFoundError in initialize means missing dependencies; the supported route is a custom execution environment, the EXECUTION_ENV_PATH parameter pointing at a packed environment, which you should confirm in the docs for your release. Bus error or failures to allocate shared memory inside Docker usually mean the container's shared memory is too small; raise --shm-size (the docs suggest at least 1 GB for the Python backend; check your release) and consider --ulimit memlock=-1.

Suspected compromise of a model or the server. Isolate the pod's network, preserve logs and the repository snapshot hash, rotate the repository credentials, and compare the snapshot against the pipeline's recorded digest. Because Triton has no sandbox for arbitrary code, assume the pod's identity is compromised and rotate everything it could read. This is the incident where the read-only repository role and the restrictive NetworkPolicy shrink the blast radius.

After every incident, change something structural: a lint rule, an alert, a default. If the post-incident action is "be more careful", the incident will come back.

Where Triton stops

A platform owner should be able to say when not to use the tool. Triton is strongest where you have GPU inference for several framework types behind one API, performance-sensitive workloads that benefit from dynamic batching and concurrent execution, and a team willing to operate GPU infrastructure. It is weaker in other situations.

Simple, CPU-bound, single-framework models. A scikit-learn model with a few requests per second does not need a GPU server with a repository, config files and monthly container upgrades. A Python web service, or a lightweight option such as BentoML or a FastAPI app, is cheaper to operate. Triton can serve such models, including tree models through the FIL backend, but you pay a complexity price you may not recoup.

LLM serving as the primary workload. Triton hosts TensorRT-LLM and vLLM backends, but both engines can be run directly, and their own servers move faster on features. Evaluate Triton for LLMs when you want a unified front end across LLM and non-LLM models, one metrics model, and ensembles or BLS around the LLM; otherwise compare against vLLM and TensorRT-LLM directly.

Kubernetes-native model serving with a platform on top. If you want canary rollouts, autoscaling to zero and a model CRD out of the box, KServe supports Triton as a runtime and gives you that layer, and Seldon Core occupies a similar space. In that arrangement, Triton remains the engine and the standard serving layer manages deployment. Likewise, managed services such as SageMaker, Vertex AI and Azure ML can run Triton containers, which shifts operational work to the cloud at the price of their abstractions and quotas.

The test I use: if the team spends more time on the serving layer's configuration and upgrade cycle than the performance gain it buys, the tool is wrong for that workload.

Running Triton as a platform for other teams

Everything above converges on one job: making it easy to do the right thing. A platform that is secure but painful will be bypassed, and one that is easy but unsafe will be exploited. The goal is a golden path that is both.

Ship a paved road. A template repository per model: conversion script, config.pbtxt skeleton with sensible batching defaults, a Perf Analyzer job, CI that builds the snapshot, and a deploy manifest. A team adds a model by opening a pull request against that template, not by learning Triton from scratch. Reserve the freedom to leave the path, with review.

Publish the contract and the numbers. Teams should find, in one place, the supported formats and backends, the latency classes and their batching defaults, the container tag and when it will change, how to get a GPU quota, the cost per thousand inferences of existing models, and the SLOs you commit to. When those are public, arguments become comparisons.

Own the lifecycle, not just the server. Onboarding, deprecation and offboarding. Quarterly review of idle models and oversized instance counts. A calendar for the upgrade cadence, announced a month ahead, with a compatibility test teams can run on their own models. A named owner for every model in the repository, because an ownerless model is the one nobody dares to delete.

Measure the platform itself. Time to first deployment for a new team, percentage of models passing the pipeline without manual help, upgrade lead time, incident count by category, GPU utilisation fleet-wide, and cost per thousand inferences. Review it quarterly with your consumers. A platform without its own metrics cannot argue for its own budget.

The review checklist

Use this for pull requests that touch the platform and for design documents that propose a new model or serving pattern. A "no" is not a veto; it is a question that needs an answer.

• Model and repository. Is the repository snapshot produced by the pipeline and immutable? Is the model verified in its native framework and tested for equivalence? Is a TensorRT plan built in the matching container and architecture? Is there a version policy and a rollback path?
• Configuration. Is max_batch_size a deliberate decision, and are dims consistent with it? Are instance counts justified by a measured knee? Is the batching delay within the latency budget? Is warmup configured for models with slow first requests? Are auto-complete and control mode set for production?
• Capacity. Is GPU memory footprint per replica known, and does the node fit it with headroom? Is the autoscaling signal queue or pending requests, not utilisation? Is a warm floor set, and does the startup probe cover the slowest load?
• Security. Is the endpoint behind a gateway or mesh with authentication? Are admin APIs restricted or disabled? Is the container non-root with a network policy limiting ingress and egress? Are repository credentials short-lived, read-only and scoped? Does the model use Python or custom code and has it been reviewed?
• Observability. Are latency histograms enabled, and are queue and compute durations on the dashboard? Do alerts fire on symptoms and leading indicators? Is tracing sampled, and is anything sensitive logged?
• Operations. Is the container tag pinned and recorded? Is the upgrade tested with Perf Analyzer against a baseline? Is there a named owner and on-call? Is there a runbook for the failures above?
• Cost and governance. Is cost per thousand inferences known? Is the tier appropriate for the isolation needed? Is there a deprecation date or review date?

For design documents, add: what happens when a GPU node dies mid-rollout, what happens at ten times current traffic, what is the largest input a client can send, and what is the plan when this model's team leaves.

The complete picture

Put the whole series together and Triton looks like this. At the beginner level, a model repository with a configuration file, served over HTTP and gRPC. At the middle level, a tuned server: dynamic batching, instance groups, ensembles, and Perf Analyzer to prove the settings work. At the senior level, a platform: a pipeline that produces immutable, reviewed repository snapshots; a hardened, unauthenticated-by-design server placed behind identity and network controls; fixed control mode so what runs is what was deployed; capacity plans that start from the saturated resource; an upgrade process that treats each monthly container as a change to test; metrics that become service levels and costs; and governance that makes the safe path the easy one.

Where the series leaves you

You can now operate Triton for other teams: explain its architecture and where it breaks, size and scale it from evidence, secure it without relying on features it lacks, host several teams with an honest isolation story, control cost, upgrade without outages, and say when another tool fits better. The senior skill is not knowing more flags; it is knowing which decision each flag belongs to and who must approve it.

From here, the natural next steps are the neighbouring guides. For LLM workloads go deeper with TensorRT-LLM and vLLM. For a Kubernetes-native serving layer see KServe, and for the cluster itself, Kubernetes. For the observability side, Prometheus, Grafana and OpenTelemetry. Keep the monthly release notes on your calendar, and treat the first page of each one, the known issues, as required reading.

Sources

• Triton Inference Server user guide
• Release notes index
• Release 26.08 notes
• Quickstart
• Model configuration
• Model repository
• Model management
• Optimization
• Metrics
• Secure deployment considerations
• Debugging guide
• Triton on GitHub
• Triton containers on NGC

This is part three of three. At this level an interviewer is not testing whether you know the flags. They want to hear how you reason about a shared GPU serving platform: where it breaks, what you would measure before changing anything, and how you would defend a decision to people who control the budget. Expect system-design prompts, incident questions and trade-off arguments, usually in that order.

How to answer at this level

Three habits separate strong senior answers from weak ones. First, name the constraint before the solution. "It depends on whether we are bound by latency, throughput or cost" is a better opening than a list of features. Second, ask for the measurement. Almost every performance question in this guide has a metric that settles it, and saying which one shows you have run the system. Third, state the failure mode of your own proposal. Every design has one, and an interviewer trusts a candidate who volunteers it.

A fast way to anchor any Triton answer is its shape: requests come in over HTTP or gRPC, wait in a per-model scheduler queue, run on a model instance through a backend, and the whole thing is one server process sharing GPU memory. The server has no authentication of its own, loads whatever is in its model repository, and ships a new container every month. Most good answers are consequences of those sentences.

System-design prompts

Design a shared inference platform for twelve teams

What the interviewer wants: requirements first, then isolation, capacity, security, operations, and an honest account of trade-offs.

A strong answer: I would start by asking what the models are, what the latency classes look like, and how much we trust the tenants. Say the answer is twelve internal teams, mostly vision and ranking models in TensorRT, ONNX and PyTorch formats, with a few Python-backend pre-processing steps, and p99 budgets from 30 ms to a few seconds.

For isolation I would tier by risk, because Triton has no native tenant concept: no per-tenant authentication, quotas or separate address spaces, and all models on a GPU share its memory. Standard-format models without custom code share servers grouped by latency class so that a batch-tolerant model does not sit next to a latency-critical one. Anything with Python backend code or custom backends gets a dedicated server, because those run arbitrary code inside the serving pod and Triton's own security guidance says it provides no sandbox for it. Where tenants need hard GPU isolation, I would use MIG partitions with a Triton per slice.

For the supply chain, models arrive through a pipeline, not by hand. The pipeline converts the model, verifies it in its native framework, runs Perf Analyzer against the team's budget, and publishes an immutable repository snapshot to object storage. Servers run in the default --model-control-mode=none, which the secure-deployment guide recommends, so a new snapshot means a new rollout gated by /v2/health/ready, and there is no runtime API through which a model can change.

For security, Triton sits behind a gateway or mesh that authenticates callers and applies per-consumer quotas, because the server will answer anyone who can reach port 8000 or 8001. I would restrict admin categories with --http-restricted-api and --grpc-restricted-protocol, run as the non-root triton-server user, apply a NetworkPolicy so only the gateway reaches the inference ports and only Prometheus reaches metrics, and give the pod a read-only workload identity for the repository.

For capacity, each model gets a measured knee from Perf Analyzer, a replica count with a warm floor, and an autoscaler driven by pending requests or queue duration, not GPU utilisation, because model load time means scaling lags and utilisation is a coarse signal.

For operations, a pinned container tag, a monthly test of the new release, quarterly production promotion, and engine rebuilds as a pipeline stage because a TensorRT plan is tied to the TensorRT version and GPU architecture.

Failure mode I would volunteer: a shared server is one failure domain. A native crash or a GPU out-of-memory in one tenant's model can take the others down, which is why the tiering exists and why I would accept a higher idle cost for the restricted tier.

What a weak answer misses: it describes Triton features, never mentions authentication, assumes models can be isolated inside one server, and scales on GPU utilisation.

Serve a recommendation model at 20,000 requests per second with a 20 ms p99

What the interviewer wants: a sizing method, not a number.

A strong answer: I cannot give a replica count without a measurement, so I would describe how to get one. Take the real model on the target GPU type and run Perf Analyzer across a concurrency range. The docs give a starting rule that the concurrency for maximum throughput is about two times max batch size times the instance count, so I test around that value and look for the knee where throughput stops rising and p95 latency starts to climb. That gives throughput per replica at the latency budget.

Dynamic batching is the main throughput lever, and the cost is queue delay. With a 20 ms budget I would set max_queue_delay_microseconds to a small fraction of that, a few hundred microseconds to a couple of milliseconds, and verify with the batching-efficiency ratio, nv_inference_count over nv_inference_exec_count, that batches are actually forming. I would also consider several instances of the model per GPU so kernels overlap, and FP16 if accuracy holds on an offline evaluation.

Then the non-GPU parts. At that request rate, JSON tensors over HTTP can cost more than inference, so gRPC or binary tensor data, and shared memory if the client is co-located. Replicas follow from dividing target peak by measured throughput at p99 with headroom, and I would size the warm floor for the normal peak because new replicas take minutes to load.

Follow-up you should expect: "What if p99 is fine at p50 but spikes?" Answer: compare queue duration against compute duration. Queue spikes mean bursts exceeded capacity or batching delay is too generous; compute spikes mean contention or power throttling.

Migrate a TorchServe fleet to Triton without downtime

A strong answer: I would first ask why. If the answer is throughput and GPU utilisation, Triton's dynamic batching and concurrent execution are a real gain; if the answer is "everyone uses it", the migration may not pay for itself. Assuming it is justified, I run both in parallel. Export each model to a format Triton serves well, TorchScript or ONNX, or TensorRT where the gain justifies the engine build. Write the config.pbtxt with explicit input and output definitions, since PyTorch models need full I/O. Mirror production traffic to the Triton deployment and compare outputs against the old service on a golden set and live traffic, plus latency percentiles. Then shift consumers gradually behind the gateway, keeping the old service as a rollback until a full cycle has passed. Pre- and post-processing that lived in handlers has to move: into the client, an ensemble, or a Python backend model, and each has a cost I would call out.

Add LLM serving to an existing Triton platform

A strong answer: Triton hosts TensorRT-LLM and vLLM backends, and LLM models are decoupled, meaning responses are not one per request and streaming happens over gRPC. That changes scaling: throughput depends on KV-cache memory and the backend's own batching, not Triton's dynamic batcher. I would ask whether the unified front end justifies it. If LLMs are most of the load, running vLLM or TensorRT-LLM directly may move faster on features. I would also read the release notes' known issues before committing; 26.08 lists limits on vLLM tensor parallelism with explicit model control mode and security vulnerabilities in the vLLM v0 API and in Ray that require restricting network access. Those facts change the network design and the control-mode design, so they belong in the proposal, not in a footnote.

Incident-response questions

The whole fleet restarted after a deploy and no replica becomes ready

State your first move: roll back to the last known-good container tag and snapshot, because restoring service comes before diagnosis. Then diagnose from the logs read from the top. error: creating server: Internal - failed to load all models is a summary; the cause is a per-model error earlier. With --exit-on-error=true, the default, one bad model stops the server, and with strict readiness one failed model keeps the replica unready. The usual suspects are a TensorRT plan built by a different TensorRT version or GPU architecture, a changed config that disagrees with the model, a missing file under the default name, a host driver too old for the container's CUDA, or an unreachable repository because credentials rotated. Say that you would check what changed in the deploy: the container tag, the snapshot, or the node image.

p99 latency doubled, error rate flat, GPU utilisation unchanged

Split the latency using the queue, compute-input, compute-infer and compute-output durations. Queue growth with steady compute means capacity or batching: check pending requests, replica count, whether the autoscaler lagged and whether someone changed max_queue_delay_microseconds. Compute-infer growth means the model, a precision or engine change, contention from a new tenant, or power and thermal limiting visible in nv_gpu_power_usage. Input and output growth means payloads or serialisation changed, for example a client moving from binary to JSON or a CPU-saturated node. Flat utilisation is not reassuring; it is a coarse signal that can stay steady while queues build.

A team asks to upload models straight into production

Say no to the mechanism and yes to the need. Live uploads imply poll or explicit mode, and the docs advise against poll in production while dynamic updates are a route to arbitrary code execution. Poll can also observe a half-written file. Offer the alternative: a pipeline that produces an immutable snapshot with a fast promotion, so their need for speed is met within minutes without opening a hole. If a long tail of rarely used models really does require on-demand loading, offer explicit mode behind a control plane, with the repository API restricted by key and serialised operations.

You suspect a model in the repository is malicious

Isolate the pod's network, preserve logs and the snapshot hash, and compare the snapshot against the pipeline's recorded digest to find out whether the artifact was tampered with or arrived malicious. Rotate every credential the pod could read. Triton provides no sandbox for arbitrary model or backend code, so assume the pod's identity and anything mounted into it are compromised. Say what limited the blast radius: a read-only repository role, a restrictive NetworkPolicy with no unnecessary egress, a non-root user, and a gateway that kept the model's callers separate. Afterwards, tighten the tier rules for who can ship Python and custom backends.

Trade-off questions

Shared servers versus a server per team

Shared servers win on utilisation and cost per model; per-team servers win on failure isolation, upgrade independence and chargeback. Triton gives no tenant isolation, so the shared option is only safe for cooperating tenants running supported formats. My default is shared servers for the standard tier, dedicated for custom code, and a clear rule written before the first team onboards. Retrofitting later is a migration.

Explicit model control versus none

none makes the deployment an immutable unit and removes a runtime control API, at the price of a rollout for every change and all models resident. explicit allows many models on a node and on-demand loading, at the price of a privileged API that must be restricted and a control plane that must serialise operations. I choose none unless memory pressure from a long tail of models makes it impractical.

TensorRT engines versus ONNX Runtime

TensorRT usually gives the best GPU performance, but each plan is tied to the TensorRT version and GPU architecture, so every upgrade and every new GPU type needs a rebuild and a plan per architecture. ONNX Runtime is more portable and simpler to operate, and it can use the TensorRT accelerator through the optimization block, the official example showing about double the throughput. I let the measured difference decide: if the gain is under the operating cost of the engine pipeline, I stay with ONNX Runtime.

Triton versus KServe, Seldon or a managed service

They are not exact alternatives. Triton is an inference engine; KServe and Seldon Core are Kubernetes-native serving layers that can run Triton as a runtime and add rollouts and scaling; managed services such as SageMaker, Vertex AI and Azure ML can run Triton containers. If the organisation already runs a serving layer, Triton is the engine inside it. If not, Triton plus Helm and a GitOps tool is a smaller starting point, and I would adopt the layer later when rollout management becomes the bottleneck.

Arguing a trade-off to leadership

Leadership does not need Triton's internals. They need a decision framed in money, risk and time. A structure that works is: the decision, the options, the recommendation, the cost of being wrong, and what would change your mind.

Example: asking for budget to move from one Triton per team to shared servers. Lead with the outcome. "Our GPUs average 18 percent utilisation. Consolidating the standard tier onto shared servers would let us retire roughly a third of GPU nodes." Then the risk: "A shared server is a single failure domain, so custom-code models stay on dedicated servers and the consolidation covers only supported formats." Then the evidence: a measured table of per-model throughput and utilisation. Then the cost of being wrong: "If the rollout causes a latency regression, rollback is a one-commit revert because snapshots are immutable and the container tag is pinned." Then the trigger: "If p99 on any consolidated model exceeds its budget for a week, that model returns to dedicated capacity."

Example: pushing back on "we need to upgrade now". Explain that each monthly release changes CUDA and TensorRT, and engines must be rebuilt. Propose a quarterly cadence with an accelerated path for security fixes, and show the test that makes any upgrade safe. This turns "no" into "yes, on a schedule, with evidence".

Numbers that persuade: GPU utilisation, cost per thousand inferences per model, time to onboard a team, incident count and time to recover, upgrade lead time. Numbers that do not: raw request counts without a cost or latency attached.

What a strong answer covers that a weak one misses

• Measurement before change. Strong candidates name the metric: queue duration, batching efficiency, pending requests. Weak ones propose more GPUs.
• The trust model. Strong candidates say Triton has no authentication and no sandbox and build around it. Weak ones assume a private network is enough.
• The repository as a supply chain. Strong candidates talk about immutable snapshots, reviewed configs and who can write. Weak ones talk about folders.
• Upgrade realism. Strong candidates mention rebuilding TensorRT plans, driver requirements and one change per rollout. Weak ones say "pull the new tag".
• Honest limits. Strong candidates say when Triton is the wrong tool: simple CPU models, hostile tenants, an LLM-only workload. Weak ones sell the tool.
• Their own failure mode. Strong candidates volunteer what could go wrong with their design.

Sixty-second self-test

Answer each aloud without notes. If you hesitate on more than two, reread the matching section of the detailed guide.

1. Which Triton default stops the server when one model fails to load, and what keeps a replica unready while a model is down?
2. Name the metric ratio that tells you whether dynamic batching is working.
3. Why is GPU utilisation a poor autoscaling signal, and what do you use instead?
4. What does a TensorRT model.plan depend on, and what does that mean for upgrades?
5. Which control mode does the secure-deployment guide recommend for production, and why?
6. What does Triton give you for authentication?
7. What are two controls for limiting one model's share of a GPU on a shared server?
8. When would you advise a team not to use Triton?
9. How do you argue for a quarterly upgrade cadence to a manager who wants the latest release?
10. What is the first thing you do when no replica becomes ready after a deploy?

Model answers in brief: 1. --exit-on-error=true and strict readiness. 2. nv_inference_count divided by nv_inference_exec_count. 3. It is coarse and can stay flat while queues grow; use pending requests or queue duration. 4. The TensorRT version and GPU architecture; rebuild per release and architecture. 5. none, because dynamic updates can lead to arbitrary code execution. 6. Nothing built in; use a gateway or mesh. 7. The rate limiter with named resources, instance counts, and MIG for hard isolation. 8. Simple CPU-bound models, hostile tenants, or workloads where a simpler runtime is cheaper to run. 9. Each release changes CUDA and TensorRT and requires engine rebuilds, so a tested quarterly cadence with a fast path for security fixes costs less than monthly churn. 10. Roll back to the last known-good tag and snapshot, then read the logs from the top.

This is part three of three. At this level the problems are operational: a model that behaves in the test cluster and falls over under real traffic, a monthly container that quietly invalidates a fleet of engines, a shared server where one team's choice becomes everyone's outage. Start with the production traps, then the habits and the review checklists. Where a symptom has an error message, check its exact wording against your container release, since texts change between versions.

Production traps that appear at scale

Cumulative: everything from Beginner and Mid still applies. These rarely show up in a single-model proof of concept.

Three of these deserve a closer look because they repeat across organisations.

The TensorRT plan that is "just a file"

A model.plan looks like any other model file, so teams copy it between environments, commit it to a repository, or leave one built last quarter in a snapshot. A plan is tied to the TensorRT version and the GPU architecture that built it. A node pool with a different GPU, or a container bump with a new TensorRT, produces a load failure about an incompatible engine plan. The fix is not a rebuild by hand but a policy: plans are build products, never source, and the pipeline builds them inside the target container on the target GPU type. Key every snapshot by container tag and architecture so mismatches are visible in the name. Where one fleet has several GPU architectures, the cc_model_filenames setting lets one model directory carry a plan per compute capability.

The readiness switches someone turned off

When a replica is flapping because one model fails to load, an engineer under pressure sets --exit-on-error=false and relaxes readiness "just to get traffic flowing". The fleet then runs with replicas that are healthy by the probe and missing models in reality. Callers get Request for unknown model or has no available versions from some replicas and not others, which looks like intermittent routing trouble. Keep both defaults in the platform chart and make a change to them require platform review. If a team genuinely wants partial availability, give them one model per deployment so failure stays contained.

The autoscaler that watches the wrong number

GPU utilisation is easy to graph and tempting to scale on. It reports whether the GPU was busy during a sample, so it can sit at the same value while the queue grows. Meanwhile new replicas take minutes to start. The pattern that works is a leading signal, such as nv_inference_pending_request_count or queue duration against the latency budget, a floor of warm replicas sized to normal peak, and a scale-out step large enough to absorb a burst rather than one replica at a time.

Platform-ownership habits

These are the practices that pay off most, in rough order of leverage.

1. Make the repository immutable and pipeline-built. Everything else depends on it. A snapshot produced by CI, addressed by a version or hash, promoted by changing a pointer, and never edited in place removes whole categories of incident: corrupted loads, unreviewed changes, drift between replicas.

2. Pin and record the container tag. Put the tag in the Deployment, the snapshot metadata and the dashboard. During an incident the first question is "what exactly is running", and the answer should take ten seconds.

3. Test every container upgrade as a performance change. Run the standard Perf Analyzer suite on old and new containers using the same snapshot and hardware, plus a golden-request equivalence check. Change one variable per rollout so a regression has one suspect.

4. Read the release notes' known issues before every promotion. The monthly notes list things that matter for you: backend limits, security warnings, platform quirks. Hold a release if a known issue touches a feature you use.

5. Publish defaults, not options. Latency classes with batching delays, instance counts and memory budgets, chosen by measurement, so teams start from a good place. The platform's job is to reduce the number of decisions a team makes, because each decision is a chance to get it wrong.

6. Keep authentication outside the server and admin APIs restricted. Triton has no user authentication. Put identity and quotas in the gateway or mesh, restrict model-repository and shared-memory categories with a key, and do not rely on being inside the network.

7. Give every model an owner and a review date. An ownerless model costs GPU memory forever and is impossible to delete confidently. A quarterly sweep of idle models returns capacity.

8. Measure the platform as a product. Time to first deployment, percentage of models onboarded without help, cost per thousand inferences, incident count by category, upgrade lead time. Review them with your consumers.

9. Run game days. Corrupt a model file, upgrade without rebuilding an engine, starve shared memory, kill a node mid-rollout. Each exercise finds a runbook gap cheaply.

10. Change something structural after each incident. A lint rule, an alert, a default, a probe. A retro whose action is "be more careful" will be repeated.

Operational recipes

Short, copyable procedures. Verify flags against your container release before use.

Recipe: find out whether batching works

Compute the ratio of inferences to executions from the metrics endpoint, per model.

curl -s localhost:8002/metrics | grep -E 'nv_inference_(count|exec_count)\{model="resnet50'

Divide the two values. A ratio near 1 under load means requests are not being grouped. A ratio near max_batch_size with a growing queue means the GPU is the limit. Do the arithmetic in a Prometheus rate query for a trend instead of reading counters once.

Recipe: find the knee with Perf Analyzer

Run from the SDK container so the tool matches the server's release.

perf_analyzer -m resnet50 --percentile=95 --concurrency-range 1:16:2

Record throughput and p95 at each step. The knee is where throughput stops rising and latency starts to. The docs' rule of thumb for the concurrency that reaches maximum throughput is about two times max batch size times the instance count; test around it. Store the table beside the model's config in version control.

Recipe: read a failed start

Do not read from the bottom.

docker logs triton 2>&1 | grep -nE "failed to load|error|Invalid argument|UNAVAILABLE" | head -40

The final failed to load all models is a summary. The first per-model error is the cause. Then check, in order: file name and version directory, config against the model, engine plan against the container, the host driver against the container's CUDA, and repository credentials.

Recipe: hardened run for a test instance

A local approximation of the production baseline, useful for verifying that your model works under it.

docker run --gpus=1 --rm --user triton-server --read-only \
  --tmpfs /tmp --shm-size=1g \
  -p8000:8000 -p8001:8001 -p8002:8002 \
  -v /full/path/to/models:/models:ro \
  nvcr.io/nvidia/tritonserver:26.08-py3 \
  tritonserver --model-repository=/models \
  --model-control-mode=none --disable-auto-complete-config

If a backend needs to write somewhere, the failure tells you which path to give it a writable volume for. That discovery belongs in staging, not on the first production rollout. Some backends may need further writable locations, so treat the flags above as a starting point to test.

Recipe: histogram latencies for fleet percentiles

Summaries cannot be aggregated across replicas; histograms can.

tritonserver --model-repository=/models \
  --metrics-config histogram_latencies=true

Then compute quantiles with a histogram query in Prometheus over all replicas. Use summaries only for a single-instance view.

Recipe: confirm a model's state and config

curl -s localhost:8000/v2/models/resnet50/ready -o /dev/null -w "%{http_code}\n"
curl -s localhost:8000/v2/models/resnet50/config
curl -s -X POST localhost:8000/v2/repository/index

The first prints 200 when the model is ready. The second shows the configuration Triton resolved, including anything auto-completed. The third lists models and their states with reasons for any that are unavailable. If you restricted the repository API, the index call needs the key header you configured.

Recipe: a pre-upgrade checklist in commands

docker pull nvcr.io/nvidia/tritonserver:26.08-py3
docker pull nvcr.io/nvidia/tritonserver:26.08-py3-sdk

Pull the server and the matching SDK image, read the release's notes for driver and known issues, compare the required driver with your node image, rebuild engines in the new container on each GPU type, run the Perf Analyzer suite and golden requests, then canary. Substitute the release you are moving to.

Review checklists

For a pull request that adds or changes a model

• Is the snapshot built by the pipeline, and does its metadata record the container tag, architecture and source commit?
• Was the model verified in its native framework and compared against a golden request set?
• Is max_batch_size a deliberate decision, and are dims consistent with it?
• Is the instance count justified by a measured knee, not copied from another model?
• Is max_queue_delay_microseconds below the share of the latency budget you allow for batching?
• Is warmup configured if first requests are slow?
• Does the model use Python backend code or a custom backend, and if so has the platform security review happened?
• Does the version_policy load only what is needed?
• Is the per-replica GPU memory footprint known and within the node's headroom?
• Is there an owner, an on-call contact and a review date?

For a pull request that changes platform configuration

• Does it touch the control mode, readiness behaviour, restricted APIs or auto-complete? If so, who from the platform team approved?
• Does the container run non-root with a restrictive security context?
• Do NetworkPolicies still limit ingress to the gateway and metrics to Prometheus, and limit egress?
• Are repository credentials short-lived, scoped and read-only?
• Is the probe configuration sized for the slowest model load?
• Are the autoscaling signals pending requests or queue duration?
• Is the container tag pinned, and has it passed the performance suite?

For a design document proposing a new serving pattern

• What is the latency budget, and where in it does the queue delay sit?
• What happens at ten times traffic, and what saturates first?
• What happens when a GPU node dies during a rollout?
• What is the largest input a caller can send, and where is it limited?
• Who can write to the repository, and how would a tampered artifact be detected?
• Which tier of the platform does it belong to, and why?
• What is the cost per thousand inferences, and who pays?
• What would make us leave Triton for this workload?
• What is the rollback plan, and how long does it take?

Practice cards

Use these to rehearse. Each has a prompt and the shape of a good answer.

Card 1: the unready fleet. After a deploy, every replica stays unready. Good answer: roll back to the last known tag and snapshot; read logs from the top; check engine compatibility, config mismatch, filenames, driver and credentials; keep strict defaults.

Card 2: the batch that never forms. A team says dynamic batching "does nothing". Good answer: compute nv_inference_count over nv_inference_exec_count; check max_batch_size and the batch dimension; check whether load is too low to form batches; check the queue delay; rerun Perf Analyzer at higher concurrency.

Card 3: the neighbour problem. Latency for a critical model rises whenever another team runs a batch job. Good answer: look at the rate limiter, instance counts and gateway quotas; consider moving the batch model to a separate server or a MIG slice; define latency classes so they do not share.

Card 4: the upgrade request. A manager wants the newest release next week. Good answer: explain engine rebuilds, driver requirements and known issues; propose the performance suite and a canary; offer a date; accelerate only for security fixes.

Card 5: the cheap model. A scikit-learn model with five requests per second has its own Triton deployment. Good answer: question whether it needs one; consider a simpler runtime; if it stays, put it on a shared CPU tier.

Card 6: the open endpoint. A scan shows the inference port reachable across the cluster. Good answer: authentication at the gateway or mesh; NetworkPolicy; restricted admin APIs; disable unused protocols; verify with a call from another namespace.

Habits worth keeping

Keep a short "what changed" log for the platform: container tags, defaults, flags. Most incidents are explained by the last change, and the log makes it findable. Keep the Perf Analyzer suite runnable by anyone in one command, since a test only one person can run will not run. Keep the metrics dashboard identical for every model, so that the on-call engineer reads one layout, and the same panels, whichever model pages them. Keep a small number of golden requests per model with expected outputs, because they catch the change that returns plausible but wrong numbers, which latency graphs never will.

And keep reading. The official documentation changes monthly with the containers. Read the release notes the week they appear, and keep the neighbouring guides close: TensorRT-LLM and vLLM for LLM workloads, KServe for a Kubernetes-native layer, Prometheus and Grafana for the dashboards you will live in.

Final checklist

• The repository is immutable, pipeline-built and reviewed
• Control mode is none in production, with strict readiness and exit-on-error left on
• Authentication, quotas and TLS live in a gateway or mesh
• Admin APIs are restricted, and the pod is non-root with a NetworkPolicy
• Repository credentials are short-lived and read-only
• The container tag is pinned and recorded, upgrades are tested as performance changes
• TensorRT plans are built in the target container on the target GPU
• Autoscaling uses queue or pending requests, with a warm floor
• Latency histograms, queue and compute splits and the batching ratio are on every model's dashboard
• Cost per thousand inferences, an owner and a review date exist for every model