The Complete Text Generation Inference Guide

This is part one of three. It covers everything you need to start a Text Generation Inference server, talk to it, and understand what it prints when something goes wrong. By the end you can launch a model in a container, check that it is healthy, send it prompts through three different APIs, stream tokens back as they are produced, read the startup logs and the common errors, and hand a colleague one script that brings the whole thing up. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. A language-model server rewards hands-on work more than most tools, because the interesting behaviour (the wait while weights load, the memory errors, the way tokens stream) only becomes real once you have watched it happen on your own machine or a rented GPU.

One piece of context comes before any command, because it changes how you should read the rest of this guide.

What TGI is, and what you must know before you start

Text Generation Inference, usually shortened to TGI, is a server for large language models. You point it at a model from the Hugging Face Hub, it loads the weights onto one or more GPUs, and it exposes an HTTP API. Your application sends a prompt to that API and gets generated text back, either all at once or token by token as the model produces it. It is built by Hugging Face, written mostly in Rust (the web layer) and Python (the model layer), and licensed under Apache 2.0.

Now the important caveat. TGI is in maintenance mode, and its repository is archived. Hugging Face moved the project into maintenance mode on 11 December 2025, accepting only minor bug fixes and documentation fixes from then on, and archived the GitHub repository (made it read-only) on 21 March 2026. The final release is v3.3.7, published on 19 December 2025. No further releases are expected, which means no new model architectures and no new security patches.

That does not make TGI a waste of your time, and the reasons matter:

• It is still widely deployed. Plenty of production systems, tutorials, cloud templates and job descriptions mention it, so you will meet it at work whether or not you would start a new project on it.
• It is frozen, which is a kind of stability. The behaviour in v3.3.7 will not shift under you. A guide written today stays true.
• Its ideas transfer directly. Continuous batching, a paged KV cache, prefix caching, chunked prefill, and an OpenAI-compatible chat endpoint are the core concepts of every modern inference engine. Hugging Face itself now recommends vLLM and SGLang, as well as local engines such as llama.cpp and MLX, for new work. Learning TGI properly means learning the vocabulary those tools share. If you want to compare, the catalogue has guides for vLLM and SGLang.

The rest of this guide treats TGI as what it is: a stable, finished product at version 3.3.7. Where a tutorial you find online disagrees with this guide, check its date; many were written for versions 1.x or 2.x, and some defaults and APIs have changed since.

The problem TGI solves, and what came before it

To see why a dedicated server exists, start with the simplest way to run a language model: load it in a Python script with the transformers library and call generate(). That works well for one person in a notebook. It falls apart as soon as several users share the model, for three reasons.

First, a language model produces text one token at a time. A token is a small chunk of text, often a word or part of a word. To write a 200-token answer, the model runs a full forward pass 200 times, each time using everything generated so far. A naive script processes one request at a time, so the second user waits for the first to finish completely.

Second, a naive batch wastes the GPU. The obvious fix is to group several requests into a batch and run them together, since a GPU is far more efficient doing many things at once. But requests are different lengths. In a static batch, everyone waits for the longest answer, and the GPU sits idle on the short ones that finished early. The user whose reply needed five tokens waits as long as the one who asked for an essay.

Third, the memory is the real limit. While generating, the model keeps the attention keys and values of every previous token so it does not recompute them. This is called the KV cache, and it grows with every token of every active request. Managing it badly means the GPU runs out of memory long before the compute is exhausted.

TGI solves these problems in the following ways, each of which you will meet properly later:

• Continuous batching. New requests join the running batch between steps, and finished requests leave it immediately. Nobody waits for a slow neighbour.
• Paged KV cache. Memory is split into fixed-size blocks that need not be contiguous, so there is little waste and blocks can be shared between requests with a common prefix.
• Optimised kernels and tensor parallelism. Fast attention implementations, and the ability to split one large model across several GPUs.
• Streaming and a real HTTP API. Tokens flow back to the client as they are produced, over Server-Sent Events, with the same API shape you already know from hosted providers.
• Production plumbing. Health checks, Prometheus metrics, OpenTelemetry tracing, and request validation.

Before servers like this existed, teams wrote their own Flask or FastAPI wrapper around transformers, handled concurrency with locks or a queue, and discovered the problems above in production. TGI was Hugging Face's answer, first built to power its own hosted inference and the HuggingChat product, and later released for everyone.

A note on terms you will see constantly. Inference means using a trained model to produce output, as opposed to training it. Serving means running a model behind an API so other programs can call it. Latency is how long one request takes; throughput is how many tokens per second the server produces across all users. The two pull against each other, and most settings in an inference server are a trade between them.

The mental model: three processes and a handful of nouns

Almost every failure in TGI makes sense once you know that it is three cooperating programs, not one. The Docker image starts them for you, but when something breaks, the logs are labelled by which one spoke.

The launcher (text-generation-launcher, written in Rust) is the program the container runs when it starts. It reads your flags, reads the model's config.json, downloads and converts the weights, starts one model-server process per GPU, then starts the router, and finally supervises all of them. If any child dies, the launcher shuts everything down. That is why a crash in one place shows up as "Shard crashed" or "Webserver crashed" followed by the container exiting.

The router, also called the web server (text-generation-router, Rust), is the part you talk to. It accepts HTTP requests, tokenizes the text, validates it against the limits, puts it in a queue, and builds batches. It talks to the model server over gRPC on a Unix socket, and streams tokens back to you. It has no configuration file; everything is a flag or an environment variable.

The model server (text-generation-server, Python and PyTorch) loads the weights, owns the KV cache, and runs the forward passes on the GPU. With several GPUs, there is one process per GPU, called a shard, and the shards coordinate through NCCL, NVIDIA's library for GPU-to-GPU communication.

The vocabulary you need on day one:

• Prefill. The first pass over your whole prompt. It fills the KV cache and produces the first token. It is compute-heavy and the most memory-hungry step.
• Decode. Every later step: one new token per active request. It is limited by memory bandwidth rather than compute.
• KV cache. The stored attention state of past tokens. Its size decides how many requests the server can hold at once.
• Continuous batching. New requests join the running batch between decode steps.
• Shard. One slice of the model on one GPU, when the model is split across GPUs (tensor parallelism).
• Warmup. At startup, the server runs a worst-case prefill to measure memory, then reserves the KV cache and prepares its kernels. Most out-of-memory failures happen here, which is good: you learn about them before users do.
• Token. The unit of text the model reads and writes. Limits in TGI are all in tokens, not characters or words. A rough rule for English is about three-quarters of a word per token, but code and non-English text, Arabic included, can use noticeably more tokens for the same meaning.

One more concept: TGI serves one model per server. There is no switching between models inside a running instance. If you need two models, you run two containers. The one exception is multi-LoRA serving (small fine-tuned adapters on one base model), which later guides cover.

What you need before you install

TGI is a Linux and NVIDIA GPU server. That sentence saves a lot of confusion, so here is the full picture.

The supported path is a Linux machine with an NVIDIA GPU, a driver that supports CUDA 12.2 or newer, Docker, and the NVIDIA Container Toolkit (the piece that lets containers see the GPU). The images for 3.3.1 and later are built on Torch 2.7 and CUDA 12.8, so a recent driver is the safest choice. The optimised kernels are officially tested on H100, A100, A10G and T4 cards. On other NVIDIA GPUs TGI still runs and still batches continuously, but some operations such as flash attention and paged attention may not be used, so it can be slower. Flash attention itself needs an Ampere-generation GPU or newer; older cards fall back to a different path.

Other accelerators exist but are a separate topic: AMD ROCm images, Intel GPU and CPU images, Intel Gaudi, AWS Inferentia2 and Trainium, and TensorRT-LLM and llama.cpp backends. The docs say plainly that not all variants offer the same features. This guide uses the NVIDIA path throughout.

On a Mac, there is no accelerated path. None of the images support Apple Silicon GPUs, and Docker Desktop on a Mac cannot pass a GPU through to a container. The only documented macOS steps are for building from source on the CPU, which the docs themselves say is not the recommended usage. If you are on a Mac, do your hands-on work on a rented Linux GPU machine (any cloud provider offers an hourly one, including regions in the Gulf and Europe), or use Hugging Face Inference Endpoints. For running models locally on a Mac, the sensible tools are MLX or llama.cpp directly, and the catalogue's llama.cpp guide and Ollama guide cover that ground.

On Windows, the TGI docs say nothing. The practical route, which comes from Docker's and NVIDIA's own documentation rather than TGI's, is WSL2 with Docker Desktop's WSL2 backend and an NVIDIA driver that supports CUDA in WSL. Then the Linux commands work unchanged. Treat it as community-supported at best. Native Windows builds are not supported.

How big a GPU do you need? A rule of thumb: the weights alone take roughly two bytes per parameter in half precision. A 7-billion-parameter model needs about 14 GB for weights, and then more for the KV cache, so a 24 GB card is comfortable and a 16 GB card is tight. Smaller models, such as those with one to three billion parameters, run on modest cards and are the best choice for learning, because they download and start in minutes rather than tens of minutes.

Check your setup before pulling a multi-gigabyte image. Run these two commands on the Linux machine:

nvidia-smi
docker run --rm --gpus all ubuntu nvidia-smi

The first shows your GPU, driver version and the highest CUDA version it supports. The second proves that Docker can pass the GPU into a container. If the second fails with an error about a missing runtime or driver, the NVIDIA Container Toolkit is not installed or Docker was not restarted after installing it. Fix that first; nothing else in this guide will work without it.

Running your first server

The official way to run TGI is a single docker run command. Here it is, with a small model and a version pinned to the final release:

model=HuggingFaceH4/zephyr-7b-beta
volume=$PWD/data

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:3.3.7 \
    --model-id $model

That command is dense, so take it apart. Everything before the image name is a Docker option; everything after it is passed to TGI's launcher.

• --gpus all gives the container access to every GPU. TGI uses all visible GPUs by default, one shard each.
• --shm-size 1g gives the container 1 GB of shared memory. Docker's default is only 64 MB, and the GPU shards use shared memory to talk to each other through NCCL. With one GPU you may get away without it; with several, the lack of it causes hangs or errors. Always include it.
• -p 8080:80 publishes the container's port 80 on your machine's port 8080. The image sets its listening port to 80 inside the container, so the right-hand number is 80; the left-hand number is your choice. (Outside Docker, the launcher's own default port is 3000, which is a common source of confusion.)
• -v $volume:/data mounts a folder from your machine at /data in the container. The image stores downloaded weights there (it sets the Hugging Face home directory to /data). Without this mount, the weights live inside the container and are downloaded again every time you start a fresh one. For a multi-gigabyte model that is the difference between a thirty-second start and a twenty-minute one.
• ghcr.io/huggingface/text-generation-inference:3.3.7 is the image from GitHub's container registry. Pin the version. The latest tag exists, but because the project is frozen, a pinned tag is the only way you can state, a year from now, exactly what you ran. Older tutorials pull from other registries, such as an Azure one; those images are no longer pushed, so use ghcr.io.
• --model-id $model is the first TGI flag: which model to serve. It is a Hugging Face Hub identifier of the form organisation/name. If you leave it out, the launcher defaults to a tiny placeholder model, bigscience/bloom-560m, which is fine for a smoke test but useless for real work.

What happens next takes a while, and it is worth knowing the stages so you do not think it has hung. The launcher downloads the weights from the Hub into /data (minutes for a large model, and only the first time), converts them if needed, starts the shard, loads the weights onto the GPU, runs warmup, and finally starts the router. Only then does the server answer requests. The first start of a 7B model on a good connection commonly takes several minutes.

Run it in the foreground at first, so you can watch the logs. Once you are comfortable, add -d to detach and --name tgi to name the container, then follow logs with docker logs -f tgi.

Reading the startup logs and confirming the server is healthy

Do not treat the startup output as noise. It tells you which attention method was chosen, how the memory was divided, and where it failed if it did. Here is the sequence you should see, in order, with what each line means.

You will see the launcher print its parsed arguments, then lines about downloading. A line saying Successfully downloaded weights for followed by the model name means the files are on disk. Then:

Waiting for shard to be ready...
Shard ready in 31.2s

This means the model server has loaded the weights onto the GPU. Next, a line similar to Using attention flashinfer - Prefix caching true tells you the launcher's choices. Flashinfer is the default attention implementation. Prefix caching being on means that if two requests begin with the same text, such as a shared system prompt, the work for that shared part is reused. You do not need to understand more yet; just notice that these decisions are visible in the log.

Then comes Default max_batch_prefill_tokens to ..., and warmup lines from the router. When the router prints that it is connected and listening, the server is ready. The exact wording varies slightly between versions, so look for the sequence rather than an exact string: downloaded, shard ready, attention chosen, warmup, connected.

Now confirm from outside. Run this in a second terminal:

curl -s localhost:8080/health -o /dev/null -w '%{http_code}\n'

The -s silences the progress meter, -o /dev/null discards the body, and -w prints just the HTTP status code. You should see 200. Before the server is ready you will see a connection error, or 503 with a body of {"error":"unhealthy","error_type":"healthcheck"}. Keep re-running it, or put it in a loop, until you get 200.

Next, ask the server what it decided:

curl -s localhost:8080/info

This returns a JSON document describing the running server. The fields to look at first:

• model_id and model_sha: which model and which exact revision you are serving.
• max_input_tokens and max_total_tokens: the longest prompt accepted and the longest prompt-plus-answer allowed. In TGI version 3, you normally do not set these; the server computes them from your hardware and the model.
• version: the TGI version, which should say 3.3.7.

Those token limits matter more than any other numbers on this page, and we return to them in the configuration section. Looking at /info right after startup is the best habit a beginner can build, because it shows you what the server actually chose rather than what you assumed.

There is also an interactive API reference. Open http://localhost:8080/docs in a browser and you get a Swagger page listing every route, with a "Try it out" button for each. It is a good way to see which fields exist before writing code.

Your first generation with /generate

TGI has its own native endpoint, POST /generate. You send a JSON body with the prompt in inputs and the options in parameters, and you get JSON back. It is the simplest possible interaction, so start here.

curl localhost:8080/generate -X POST \
    -H 'Content-Type: application/json' \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}'

The reply looks like this:

{"generated_text":" Deep learning is a subset of machine learning that uses neural networks with many layers to"}

Notice three things. First, the answer starts mid-sentence and stops abruptly: max_new_tokens is 20, so the model was cut off after twenty tokens. That is a limit you set, not an error. Second, the text begins with a space; the model continues from your prompt, token by token, so what you get is the continuation, not a tidy reply. Third, there was no chat structure: /generate hands your string to the model as-is. For a base model that is exactly right; for an instruction-tuned model, which expects a conversation in a specific format, you usually want the chat endpoint described in the next section.

The parameters object controls how the text is produced. The ones to know now:

• max_new_tokens: the most tokens to generate. In TGI 3, if you omit it, the server picks a value automatically, up to what fits in max_total_tokens minus your prompt length. Setting it explicitly is still good practice, because it bounds your cost and latency.
• temperature: how random the output is. Higher values give more varied text. It must be strictly greater than zero.
• top_p and top_k: restrict sampling to the most likely tokens. top_p takes a value between 0 and 1, exclusive; top_k takes a positive integer.
• do_sample: whether to sample at all. With false, the model always takes its most likely token, which is deterministic ("greedy") and what you want for a repeatable test.
• stop: a list of strings that end generation when they appear. The server allows up to four by default.
• seed: a number that makes sampled output repeatable.
• repetition_penalty: values above 1 discourage repeating text.
• details: when true, the response also includes details such as the finish reason and the token count.

Try the details flag, because it teaches you how to see why generation ended:

curl localhost:8080/generate -X POST \
    -H 'Content-Type: application/json' \
    -d '{"inputs":"Name three colours:","parameters":{"max_new_tokens":30,"details":true,"do_sample":false}}'

The response now includes a details object with a finish_reason. You will see length when the token limit stopped it, eos_token when the model decided it was done, and stop_sequence when one of your stop strings matched. Reading the finish reason is the quickest way to tell "the model finished" from "I cut it off".

The chat endpoint and the OpenAI-compatible API

Most models you will serve are instruction-tuned: trained to follow a conversation made of messages with roles, such as a system message, user messages and assistant replies. Each model family expects those messages wrapped in its own special format, called a chat template, which lives in the model's tokenizer_config.json. Writing that by hand is error-prone.

TGI does it for you. Since version 1.4 it has served the Messages API, an endpoint compatible with the OpenAI Chat Completions format, at POST /v1/chat/completions. In version 3 it is always on; you no longer need a flag. The server applies the model's own chat template to your messages and then generates.

curl localhost:8080/v1/chat/completions \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "tgi",
      "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain what a GPU does in one sentence."}
      ],
      "max_tokens": 60
    }'

The model field is informational: TGI serves one model, so the value is not used to choose anything, and the docs use the placeholder "tgi". The reply has the familiar OpenAI shape, with the text at choices[0].message.content and token counts under usage.

Because the shape matches OpenAI's, you can use OpenAI's own Python library by pointing it at your server. Install it with pip install openai, then:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1/", api_key="-")

reply = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain what a GPU does in one sentence."},
    ],
    max_tokens=60,
)
print(reply.choices[0].message.content)

The api_key="-" is a dummy: the library insists on a value, and TGI ignores it unless you started the server with --api-key. Everything else, including the base_url ending in /v1/, is how you redirect any OpenAI-compatible tool to your own server. This is a major practical benefit: code written against a hosted provider can be switched to your TGI server by changing one URL, and switched again later when you move to another engine.

Hugging Face's own client, huggingface_hub, also works and is the one TGI's documentation recommends:

from huggingface_hub import InferenceClient

client = InferenceClient(base_url="http://localhost:8080/v1/")
out = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Say hello in Arabic."}],
    max_tokens=40,
)
print(out.choices[0].message.content)

You will see an older package, text-generation (installed with pip install text-generation), in many tutorials. Its own README now carries a legacy warning and recommends huggingface_hub instead. Avoid it in new code.

Two chat-specific traps are worth knowing now. Some models' templates do not allow a system role, and sending one produces an error starting Template error:; the fix is to fold your instructions into the first user message. And if a model has no chat template at all (typically a base model rather than an instruct model), the chat endpoint cannot work, and you should use /generate instead.

There is also POST /v1/completions, the older OpenAI completions shape, and GET /v1/models, which lists the single served model. You will rarely need either.

Streaming tokens as they are produced

A model that takes eight seconds to write a long answer feels broken if the user sees nothing for eight seconds, and fine if words appear immediately. Streaming is how every chat product achieves that. The server sends each token the moment it is generated, over Server-Sent Events (SSE), a simple web standard where one HTTP response stays open and delivers a series of small data: messages.

With the chat endpoint, add "stream": true:

curl -N localhost:8080/v1/chat/completions \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "tgi",
      "messages": [{"role": "user", "content": "Write a haiku about the desert."}],
      "max_tokens": 60,
      "stream": true
    }'

The -N flag turns off curl's output buffering; without it you would see nothing until the end, which defeats the purpose. The output is a run of lines each beginning with data: followed by a JSON chunk. In every chunk, the new piece of text sits at choices[0].delta.content, and the final chunk carries a finish reason. The stream ends with a closing message, and the connection closes.

In Python, the OpenAI client makes this a loop:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1/", api_key="-")

stream = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Write a haiku about the desert."}],
    max_tokens=60,
    stream=True,
)
for chunk in stream:
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)
print()

The if piece: guard matters: some chunks, such as the first (which carries the role) and the last, have no text, and printing None would be a bug. The flush=True makes the characters appear immediately instead of waiting for the terminal's buffer.

On the native API, streaming has its own route, POST /generate_stream, with the same request body as /generate. Each event carries one token and, on the last event, the full generated text. Alternatively POST / accepts the same body and chooses between streaming and not based on a stream field.

Why streaming matters beyond looks: it changes what you measure. Time to first token is how long until the first piece appears, and it is dominated by queueing and the prefill step. Inter-token latency is the gap between later tokens, set by the decode step. Users judge a chat by the first and feel the second as reading speed. When you start tuning a server, these are the two numbers to watch, and streaming is the only way to see them from the client.

One limitation: the best_of parameter, which generates several candidates and returns the best, is not allowed with streaming. The server rejects it with a 422 error saying best_of is not supported when streaming tokens.

Gated models, tokens, local models and the weights cache

Not every model downloads freely. Many popular ones, such as the Llama and Gemma families, are gated: you must log in to the Hugging Face Hub, read and accept the model's licence on its page, and then authenticate when you download. If you launch such a model without credentials, the download fails and the launcher exits with Error: DownloadError, with a message about a 401, 403 or gated repository a few lines above.

The fix has two parts, and both are required. On the Hub, visit the model page while logged in and accept its terms. Then create an access token in your account settings, choosing a read token (the least powerful kind, which is all you need), and pass it to the container:

export HF_TOKEN=hf_your_read_token_here

docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \
    -e HF_TOKEN=$HF_TOKEN \
    ghcr.io/huggingface/text-generation-inference:3.3.7 \
    --model-id meta-llama/Llama-3.2-1B-Instruct

The environment variable is HF_TOKEN. You will see an older name, HUGGING_FACE_HUB_TOKEN, in older material; the router still falls back to it, but HF_TOKEN is current. Never write the token into a script you commit, and never bake it into an image. Read it from your shell, a secrets manager, or a file that is in your .gitignore.

Three related ideas make downloads manageable:

• Pin a revision. The --revision flag takes a commit hash, branch or tag from the model's repository. Without it you get the default branch, which the model's authors can change at any time. For anything you want to reproduce, pin the commit hash.
• Use a local model. If you have a model directory on disk (for example one saved with save_pretrained), mount it under /data and pass its in-container path: --model-id /data/my-model. The folder must contain the weights, the tokenizer files and the config.
• Understand the cache layout. Since version 2.3, downloaded models live under /data/hub/models--organisation--name inside the container, which is ./data/hub/... on your side. If you reuse a cache volume from an old tutorial that stored models directly under /data, TGI will not find them and will download again.

A note on file formats, since it causes real confusion. Model weights come as safetensors files (a safe, pickle-free format) or as older .bin files, which are Python pickles. Pickles can execute arbitrary code when loaded, so since TGI 2.0 it does not convert them unless you pass --trust-remote-code. Always prefer a repository that has safetensors. Tensor parallelism across several GPUs also requires safetensors.

For teams in the Gulf or Egypt with data-residency rules, one more property is useful: once the weights are in your volume, you can run the container with HF_HUB_OFFLINE=1 and no token at all, and it never contacts the Hub. Prompts never leave your infrastructure because the model runs on your own GPU, which is often the reason an organisation chooses to self-host.

Configuration: flags, environment variables and the token budget

Every launcher flag has an environment-variable twin: the flag --max-total-tokens is the variable MAX_TOTAL_TOKENS. In Kubernetes and managed platforms you will set variables; in a hand-run command you will use flags. They do the same thing. To see every flag for your version, run:

docker run --rm ghcr.io/huggingface/text-generation-inference:3.3.7 --help

The list is long, and a beginner needs only a few of them. Start with these:

The most useful idea in this whole section is the token budget. Four numbers govern memory, and they stand in a fixed relationship:

• max_input_tokens: the largest prompt.
• max_total_tokens: the prompt plus the generated answer, per request. The docs call it the most important value to set, because it is the memory budget of one request.
• max_batch_prefill_tokens: the total prompt tokens the server will process in one prefill step.
• max_batch_total_tokens: all the tokens held in the KV cache across the whole batch.

The rules: the input limit must be smaller than the total limit, the prefill budget must be at least the input limit, and the total limit must not exceed the batch total.

Here is the good news. Since TGI 3.0, you should leave all four unset. The release notes call it "zero config": the server inspects your GPU and your model and chooses these numbers itself, and the official advice is to remove your flags and let it decide. You can see what it chose in /info. Many old tutorials set --max-input-length, --max-batch-prefill-tokens and friends by hand; copying them into version 3 often makes things worse, and the old name --max-input-length is deprecated in favour of --max-input-tokens (setting both is an error).

When do you intervene? When you want to trade context for concurrency. The KV cache is a fixed pool. If you allow every request up to 32,000 tokens, fewer requests fit at once than if you allow 4,000. If your application never sends more than 4,000-token prompts, lowering --max-total-tokens frees memory for more simultaneous users. That is the first real tuning decision you will make, and Mid-level covers it properly.

Quantization deserves a short mention. Quantization stores weights with fewer bits, shrinking memory use so a bigger model fits on a smaller card, at some cost to quality. For a beginner, the safe rule is: if a model repository is already quantized (for example AWQ or GPTQ), TGI reads that from the model's config.json and you pass nothing. If you need TGI to shrink a normal model on the fly, --quantize eetq is the drop-in 8-bit choice, and --quantize fp8 suits H100-class cards. The older --quantize bitsandbytes is deprecated and also disables CUDA graphs, so avoid it.

Finally, two operational flags. --api-key makes the main endpoints require an Authorization: Bearer <key> header, but it does not protect /health, /info, /metrics or /docs, and it provides one shared key, not per-user keys. It is a basic gate, not an authentication system; put TGI behind a gateway if the server is reachable by anyone you do not trust. And TGI sends anonymous usage statistics by default when it runs inside Docker; if your organisation forbids that, add --usage-stats off.

Common errors and how to read them

When TGI fails at startup, the last line is usually unhelpful: Error: ShardCannotStart, Error: WebserverFailed or Error: DownloadError. These are only the launcher's name for which stage failed. The real cause is a few lines above, usually under a heading such as "Shard complete standard error output". Always scroll up before searching the web.

Here are the errors beginners meet most, with their meaning.

Out of memory during warmup. The message reads Not enough memory to handle {N} prefill tokens. You need to decrease --max-batch-prefill-tokens, followed by Unable to warmup the Python model shards. The weights loaded, but the worst-case prefill did not fit in the remaining memory. Causes: a manually set --max-batch-prefill-tokens or --max-input-tokens that is too large, another process holding the GPU (check nvidia-smi), or a model that is simply too big for the card. Fixes, in order: remove manual limits, lower the prefill budget, use a quantized model, use a smaller model, or use more GPUs.

A second variant. Not enough memory to handle max_total_tokens={N} means one request's full context does not fit in the KV cache left over. Lower --max-total-tokens, or add memory.

Invalid token settings. Messages such as max_input_tokens (X) must be < max_total_tokens (Y) tell you your numbers break the rules from the previous section. Fix the numbers, or better, delete them.

GPUs not found. sharded is true but only found 1 CUDA devices or an error about not having enough CUDA devices means the container cannot see the GPUs you asked for. Check --gpus all and --num-shard. If the second check from the prerequisites section fails, fix Docker's GPU access first.

NCCL or hang with several GPUs. An error mentioning NCCL, or startup that never finishes, usually means /dev/shm is too small. Add --shm-size 1g. As a last resort, NCCL_SHM_DISABLE=1 avoids shared memory at a performance cost.

Flash attention on an old GPU. FlashAttention only supports Ampere GPUs or newer. means your card is too old for that kernel. Older cards generally fall back automatically; if not, use --disable-custom-kernels or newer hardware.

Download failures. Error: DownloadError with a 401, 403 or gated message: a token or licence problem, as above. Without those words, check network access and free disk space in the volume.

A model with only .bin files. Pickle weights are not converted without --trust-remote-code. Prefer a safetensors version of the model. Trust remote code only for repositories you have vetted, and pin a revision when you do, because it runs Python from the repository.

Then there are the HTTP errors your client sees once the server is up. They arrive as JSON with an error message and an error_type.

• 422, validation. Your request broke a rule. The message tells you which: inputs must have less than {max} tokens, inputs tokens + max_new_tokens must be <= {max_total}, temperature must be strictly positive, top_p must be > 0.0 and < 1.0, or stop supports up to 4 stop sequences. Shorten the prompt, lower max_new_tokens, or fix the parameter.
• 429, "Model is overloaded". More requests are in flight than --max-concurrent-requests allows (128 by default). Retry with backoff, or add replicas.
• 424, generation failed. Something broke inside a shard mid-generation, often a CUDA error. Read the server logs.
• 401 with an empty body. The server runs with --api-key and your header is missing or wrong.
• 413, payload too large. Your request body exceeds --payload-limit, 2 MB by default. This mostly happens with large base64 images.

Last, a problem that is not an error: the first request is slow when it uses grammar, response_format or tools. TGI compiles the constraint on first use and caches it. Send one warm-up request after a deploy.

Putting it all together

Here is one small end-to-end project: a script that starts a server, waits until it is ready, runs a short chat with streaming, and cleans up. It uses only what this guide has covered.

First, a start script. Save it as start-tgi.sh:

#!/usr/bin/env bash
set -euo pipefail

MODEL="${MODEL:-HuggingFaceTB/SmolLM2-1.7B-Instruct}"
IMAGE="ghcr.io/huggingface/text-generation-inference:3.3.7"

mkdir -p "$PWD/data"

docker run -d --name tgi --gpus all --shm-size 1g \
    -p 8080:80 -v "$PWD/data:/data" \
    -e HF_TOKEN="${HF_TOKEN:-}" \
    "$IMAGE" --model-id "$MODEL" --usage-stats off

echo "Waiting for the server (first start downloads weights)..."
until curl -sf localhost:8080/health > /dev/null; do
    if ! docker ps --format '{{.Names}}' | grep -q '^tgi$'; then
        echo "Container exited. Last logs:"; docker logs --tail 40 tgi; exit 1
    fi
    sleep 5
done
echo "Ready:"; curl -s localhost:8080/info | head -c 400; echo

Notice the loop does two jobs: it waits for health, and it notices if the container died, so a failed start prints the last logs instead of waiting forever. The model default is a small instruction-tuned model; change it with the MODEL variable.

Second, a client script. Save it as chat.py:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1/", api_key="-")

history = [{"role": "system", "content": "You are a concise assistant."}]

while True:
    question = input("you> ").strip()
    if not question:
        break
    history.append({"role": "user", "content": question})

    answer = ""
    stream = client.chat.completions.create(
        model="tgi", messages=history, max_tokens=300, stream=True
    )
    for chunk in stream:
        piece = chunk.choices[0].delta.content
        if piece:
            answer += piece
            print(piece, end="", flush=True)
    print()
    history.append({"role": "assistant", "content": answer})

This keeps the conversation in a list and resends it on every turn. That is how every chat client works with a server like this: the server remembers nothing between requests. The history is yours. Every turn sends a longer prompt, which is why prefill grows, why long chats get slower, and why prefix caching (reusing the shared start of each prompt) helps so much for exactly this pattern.

Run it:

chmod +x start-tgi.sh
./start-tgi.sh
python chat.py

When you are done, stop and remove the container; the weights stay in ./data for next time:

docker rm -f tgi

Now extend it. Add a --max-total-tokens value to the start script and watch /info change. Point chat.py at a different model by changing MODEL. Time the first token. Break something on purpose, such as an invalid token limit, and practise reading the failure from the logs. A beginner who can start a server, diagnose a failed start, and write a streaming client has done the whole job at this level.

What you can now do, and what comes next

You can now do the whole beginner loop. You know that TGI is a Linux and GPU server made of a launcher, a router and a model server, and that it is a frozen, archived product at version 3.3.7. You can start it with a pinned image and a persistent volume, wait for it with a health check, and read /info to see the token limits it chose. You can call it three ways: the native /generate endpoint, the OpenAI-compatible chat endpoint, and streaming. You can supply a Hub token for gated models, choose a revision, and use a local model. And you can read a failed startup, telling a memory problem from a download problem from a GPU-visibility problem.

What the other levels add:

• Mid-level goes under the hood: how continuous batching and the paged KV cache decide capacity, quantization, multi-GPU tensor parallelism, structured output with grammars, tool calling, monitoring with Prometheus, and running TGI on Kubernetes.
• Senior treats the server as a platform: capacity planning, the security model (including what --api-key does and does not protect), upgrade policy for a frozen project, and a migration plan to vLLM or SGLang.

Where to look next in this catalogue: vLLM and SGLang are the engines Hugging Face recommends going forward, and the concepts here map across directly. If you want a gentler local-first path, try Ollama or llama.cpp. To package or orchestrate the container you now run by hand, the Docker and Kubernetes guides are the natural follow-ups, and LiteLLM is a common gateway to put in front of a server like this.

One habit to carry away: write down the version, the image tag and the model revision of anything you deploy. With a frozen project that is easy to do and very valuable, because it is the information you will need on the day you migrate.

Sources

• Text Generation Inference documentation
• Quick tour
• Using TGI with NVIDIA GPUs
• Installation
• Consuming TGI
• Launcher reference
• Using the TGI CLI
• Gated model access
• Preparing the model
• Architecture
• Usage statistics
• API reference (Swagger)
• TGI on Hugging Face Inference Endpoints
• GitHub repository and releases

Part one of three. A twenty-five-minute review of the Beginner Detailed track, then the questions interviewers ask about Text Generation Inference. Mid-level reviews this plus its own material; Senior reviews all three.

One framing note before anything else, because interviewers in 2026 will test it: TGI is in maintenance mode (since 11 December 2025) and its repository is archived (since 21 March 2026). The last release is v3.3.7. A candidate who describes TGI as actively developed signals that they have not looked recently. A candidate who explains what it did well, why it is frozen, and what carries over to vLLM and SGLang sounds like someone who has run it.

The thirty-second answer

Text Generation Inference is Hugging Face's server for large language models. It loads a model onto one or more GPUs and serves it over HTTP, with continuous batching, a paged KV cache and streaming, so many users can share one model efficiently. It exposes a native /generate API and an OpenAI-compatible /v1/chat/completions endpoint. It is now frozen in maintenance mode, and Hugging Face recommends vLLM or SGLang for new projects.

Then the line that shows you have used it: "the launcher starts one model-server shard per GPU plus a router, and when something fails the real error is a few lines above the launcher's last message."

The model

The commands to have ready

You should be able to write these from memory, and explain every flag.

docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \
    ghcr.io/huggingface/text-generation-inference:3.3.7 \
    --model-id HuggingFaceH4/zephyr-7b-beta

Be ready to say why each piece is there: --gpus all exposes the GPUs, --shm-size 1g gives the shards shared memory for NCCL (Docker's default is 64 MB), -p 8080:80 maps to the container's port 80, -v ... :/data persists downloaded weights, and the tag is pinned rather than latest.

curl -s localhost:8080/health -o /dev/null -w '%{http_code}\n'   # 200 when ready
curl -s localhost:8080/info                                       # what the server chose
curl localhost:8080/generate -X POST -H 'Content-Type: application/json' \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":20}}'

And the chat call from Python:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1/", api_key="-")
r = client.chat.completions.create(
    model="tgi", messages=[{"role": "user", "content": "Hi"}], max_tokens=50)
print(r.choices[0].message.content)

Know the endpoints: /generate, /generate_stream, /v1/chat/completions, /v1/completions, /v1/models, /health, /info, /metrics, and /docs for Swagger.

The classic traps and why they happen

The server seems to hang at startup. It is not hanging. It is downloading weights, loading them, then running warmup. The first start of a large model takes many minutes. The fix is to watch the logs and poll /health, not to restart.

Restarts are slow every time. The weights were downloaded into the container's writable layer, because there was no -v volume at /data. Removing the container removed the weights.

Error: DownloadError on a Llama or Gemma model. The model is gated. You need to accept the licence on the Hub and pass -e HF_TOKEN=... with a read token. It is an authorisation problem, not a GPU problem.

Out of memory during warmup. The weights fit, but the worst-case prefill did not. People make this worse by copying old tutorials that set --max-batch-prefill-tokens by hand. In v3 the correct first move is to remove manual limits.

Hangs or NCCL errors on several GPUs. /dev/shm is too small. Add --shm-size 1g.

temperature: 0 returns 422. The server requires temperature strictly greater than zero. For deterministic output, send do_sample: false.

Answers end mid-sentence. max_new_tokens or max_tokens was reached. Check finish_reason: length means you cut it off; eos_token means the model finished.

The first structured-output request is very slow. The grammar compiles on first use and is cached afterwards. Not a bug.

The system role errors. Some chat templates reject a system message; the server returns Template error. Fold the instructions into the user message.

Common interview questions

What problem does TGI solve that a plain transformers script does not?

A script handles one request at a time, wastes the GPU on static batches where everyone waits for the longest answer, and manages KV-cache memory badly. TGI adds continuous batching so requests join and leave the batch between steps, a paged KV cache that wastes little memory and allows prefix sharing, optimised attention kernels, tensor parallelism for large models, token streaming, request validation and metrics. In short, it turns a model into a multi-user service.

Explain prefill and decode.

Prefill processes the entire prompt in one pass, fills the KV cache and yields the first token; it is compute-bound and the most memory-hungry step. Decode then generates one token per step per active request, reading the cache; it is memory-bandwidth-bound. Time to first token is mostly queueing plus prefill; the gap between later tokens is decode.

What is continuous batching?

Instead of forming a fixed batch and waiting for all members to finish, the server lets new requests join between decode steps and removes finished requests immediately. A short request never waits for a long neighbour, and the GPU stays busy. The cost is that admitting newcomers requires a prefill that briefly pauses the running batch; settings such as --waiting-served-ratio and --max-waiting-tokens balance that.

What is the KV cache and why does it matter?

It stores the attention keys and values of past tokens so they are not recomputed. It grows with every token of every active request, so its size, not compute, usually decides how many requests fit at once. TGI uses a paged layout: fixed-size blocks that need not be contiguous, which reduces waste and lets blocks be shared across requests with a common prefix.

What are the three processes inside TGI?

The launcher, which parses flags, downloads and converts weights, starts the others and supervises them; the router, a Rust web server that handles HTTP, validation, queueing and batching and talks gRPC over a Unix socket to the shards; and the model server, Python and PyTorch, one shard per GPU, holding the weights and the KV cache. If any child dies, the launcher shuts everything down.

How do you check that a server is up and what it is doing?

Poll GET /health for a 200 (503 with an unhealthy body before it is ready). Then GET /info shows the model id and revision, the version, and the max_input_tokens and max_total_tokens that zero-config chose. The logs show the attention implementation and whether prefix caching is on.

Why mount a volume at /data?

The image stores Hub downloads there. A volume persists the weights across container restarts and lets you share them between containers. Without it every fresh container re-downloads gigabytes.

What changed in TGI version 3 regarding configuration?

Zero config. The server computes the four token-budget values from the GPU and the model, and the official advice is to remove the flags you used to set. Prefix caching and chunked prefill are on by default. max_new_tokens defaults automatically when omitted. Prompt logprobs are off by default and need --enable-prefill-logprobs.

How would you call TGI from an application?

Through the OpenAI-compatible endpoint using the openai SDK with base_url pointing at /v1/, or through huggingface_hub.InferenceClient. The older text-generation Python package is legacy. Using the OpenAI shape means the application can later be pointed at vLLM or SGLang with a URL change.

How do you stream a response?

Set stream: true on the chat endpoint, or call /generate_stream on the native API. The server returns Server-Sent Events, one chunk per token. In the OpenAI client, you iterate over chunks and read choices[0].delta.content, skipping empty chunks. Streaming improves perceived latency and is the only way a client can measure time to first token.

What is a gated model and how do you serve one?

A Hub model whose licence you must accept before downloading. Accept it on the model page, create a read-scope token, and pass HF_TOKEN to the container. Never bake the token into an image or commit it.

What does --api-key protect?

A single shared bearer token on the main inference routes: /generate, /generate_stream, /v1/chat/completions, /v1/completions and a few more. It does not protect /health, /info, /metrics, /v1/models or /docs, and there are no per-user keys. So the server should sit behind a gateway or private network.

Which hardware does TGI support, and what about Macs?

Its main path is Linux with NVIDIA GPUs (CUDA 12.2 or newer; kernels tested on H100, A100, A10G, T4). There are separate images for AMD ROCm, Intel, Gaudi and AWS Neuron, plus TensorRT-LLM and llama.cpp backends, with feature differences. There is no accelerated macOS path: Docker on a Mac has no GPU passthrough. On a Mac you would use MLX or llama.cpp, or a remote Linux GPU machine.

TGI is archived. Would you still use it?

For a new project, no: the maintainers recommend vLLM or SGLang, and no new security fixes or model architectures will ship. For an existing deployment, it is stable and usable, but I would pin the image, scan it, keep it on a private network, and plan a migration. Because TGI serves the OpenAI-compatible API, moving clients is cheap; what needs re-validation is TGI-specific payloads, grammar syntax, metric names and health probes.

An engineer says the server is "stuck" ten minutes after start. What do you check?

Read the logs for the stage: downloading, loading, or warmup. Look at disk growth in the volume, and nvidia-smi for GPU memory activity. A multi-GB model on the first run legitimately takes that long. If the log shows an error, scroll above the final Error: line for the real cause.

What does a 429 mean?

Model is overloaded: more requests are in flight than --max-concurrent-requests (128 by default). Clients should retry with backoff; operators add replicas.

What does a 422 mean?

The request failed validation. The message names the rule: prompt too long, inputs plus max_new_tokens over the total limit, an invalid temperature or top-p, too many stop sequences, and so on. Fix the request; the server is healthy.

More questions that come up

Why does TGI use a Unix socket and gRPC between the router and the shards?

The router and the model server are separate programs written in different languages, Rust and Python. A local socket with gRPC gives them a typed, fast channel without a network hop. The practical consequence for a beginner is that a crash in either process is visible as a launcher message, and that the router is not something you scale independently in a normal deployment.

A user complains that long conversations get slower. Why?

The server is stateless, so the client resends the whole history every turn. Each turn's prompt is longer, which makes prefill longer and fills more of the KV cache. Prefix caching helps because the start of each prompt matches the previous one, but the cost still grows. The fixes are trimming or summarising old turns, and keeping the prompt under max_input_tokens.

What is the difference between /generate and /v1/chat/completions?

/generate is TGI's native route. It takes a raw string in inputs and continues it, so you must format any chat structure yourself. The chat route takes a list of role-tagged messages and applies the model's own chat template. For instruction-tuned models, use the chat route; for base models without a template, use /generate.

What is quantization, in one answer?

Storing weights with fewer bits so a model needs less memory, at some cost in quality. If a repository is already quantized, TGI detects that from its config. On-the-fly options include eetq for 8-bit and fp8 on H100-class GPUs; bitsandbytes 8-bit is deprecated.

Why is max_total_tokens described as the most important setting?

It is the memory budget of a single request, prompt plus answer. The KV cache is a fixed pool, so a larger per-request budget means fewer simultaneous requests. Lowering it, when your application does not need long contexts, is the simplest way to raise concurrency.

How do you keep a Hub token safe?

Use a read-scope token, pass it as HF_TOKEN from the environment or a secret store, never put it in an image or a repository, and once the weights are cached you can run with HF_HUB_OFFLINE=1 and no token at all.

What would you tell a team that asks whether to adopt TGI today?

I would say that it works and is stable at 3.3.7, but it will not receive new features or security fixes, and Hugging Face points new projects at vLLM or SGLang. If they already run it, pin it, isolate it and plan the move. If they are starting fresh, start on a maintained engine and keep their application on the OpenAI-compatible API either way.

What interviewers listen for

A strong beginner answer does four things that a weak one does not.

• Names the layers. It says launcher, router and shard, and uses them to place a failure. A weak answer says "the server crashed".
• Separates startup from serving. It knows warmup is where memory errors surface and that the first start is slow by design.
• States the current status honestly. It mentions maintenance mode and the move to vLLM and SGLang without being asked, and says what concepts transfer.
• Pins things. It mentions a pinned image tag, a pinned model revision, and a persistent volume. That is the production instinct they are checking for.

A common weak pattern is reciting flags without saying what problem each solves. If you are asked about --shm-size, the answer is "NCCL needs shared memory and Docker's default is 64 MB", not "it sets shared memory size".

Sixty-second self-test

Answer out loud, without scrolling up. Anything you hesitate on is your revision list.

1. What are the three processes, and which one reports a CUDA out-of-memory error?
2. Why does the Docker command have --shm-size, -v and a pinned tag?
3. Which port does the container listen on, and which is TGI's default outside Docker?
4. Define prefill and decode in one sentence each.
5. What does continuous batching change compared with a static batch?
6. What does zero config mean in version 3, and what should you do with old token flags?
7. Which endpoint do you poll to know it is ready, and which one shows the chosen token limits?
8. A request with temperature: 0 fails. What status code, and what do you send instead?
9. What is the cause of Error: DownloadError on a Llama model, and how do you fix it?
10. Name two endpoints that --api-key does not protect.
11. What is the project's status today, and what does Hugging Face recommend for new work?
12. Why is the OpenAI-compatible endpoint a good choice for application code?

If you can answer ten of twelve fluently, you are ready for the Mid-level track. If not, reread the sections on the mental model, the startup logs, and common errors; those carry most of the interview weight.

Part one of three. Most beginner problems with Text Generation Inference come from three misunderstandings: treating startup as instant when it is a multi-stage process, copying old tutorials that predate version 3, and forgetting that the server remembers nothing between requests. Start with the table, then the longer explanations, the habits and the checklist.

A reminder that frames every tip: TGI is in maintenance mode and the repository is archived. The final release is v3.3.7. Tips that mention versions refer to that release.

Common errors at this level

The mistakes in detail

Assuming a slow start means a broken server

Symptom. You run the container, nothing answers for five minutes, and you press Ctrl-C and start over. The second attempt does the same.

Cause. Startup is a pipeline with several slow stages: the launcher downloads weights (gigabytes on the first run), converts them if necessary, each shard loads them onto the GPU, and then warmup runs a worst-case prefill and prepares kernels. The server answers requests only after all of that. Interrupting it restarts the pipeline, and without a volume, even the download.

Fix. Run in the foreground at first and read the log sequence: weights downloaded, shard ready, attention implementation chosen, warmup, connected. Poll /health until it returns 200. In scripts use a loop with curl -sf rather than a fixed sleep. Use a small model while learning, so each cycle takes minutes instead of tens of minutes.

Forgetting the volume

Symptom. Starting the server is slow every single time, even though you used the same model yesterday.

Cause. The image keeps its Hugging Face cache at /data. If you do not mount a host folder there, the cache lives in the container's writable layer and disappears when the container is removed. A related variant: you reuse an old cache folder created by a tutorial for TGI 2.2 or earlier, and TGI 2.3 and later look under /data/hub/, so it downloads again.

Fix. Always include -v $PWD/data:/data. After the first run, look inside ./data/hub/ and confirm a models--organisation--name folder exists. If you must reuse an old cache, check its layout.

Copying flags from an old tutorial

Symptom. Your server refuses to start with a message about max_input_length being deprecated, or it starts but runs slower or runs out of memory when the tutorial said it would not.

Cause. TGI version 3.0 introduced zero config. The server now computes max_input_tokens, max_total_tokens, max_batch_prefill_tokens and max_batch_total_tokens from the GPU and model. Tutorials from the 1.x and 2.x era tell you to hand-tune them. Hand-set values can be worse than the automatic ones and can break the rules the launcher checks. The name --max-input-length is deprecated, and giving both it and --max-input-tokens is an error.

Fix. Start with no token flags at all. Read /info to see what the server chose. Only set --max-total-tokens when you have a reason, for example to give up context in exchange for more concurrent users. When you read a tutorial, check its date and the TGI version in its commands.

Using the latest tag

Symptom. A colleague's server behaves differently from yours, or a setup that worked last month fails.

Cause. An unpinned tag is a moving target. TGI is now frozen, which makes pinning easy and valuable: the image ghcr.io/huggingface/text-generation-inference:3.3.7 will not change under you, while latest is simply whatever was newest when it was last pushed. Some other tags have gaps; for example the AMD ROCm tag only goes up to 3.3.6-rocm.

Fix. Pin an explicit tag in every script and document. Pin the model, too, with --revision <commit hash>, because a model's default branch can change.

Mistaking a gated-model failure for a hardware problem

Symptom. Launching a Llama or Gemma model fails with Error: DownloadError, and you spend an hour checking drivers.

Cause. The model is gated. The Hub refuses the download unless you have accepted its licence and present a token. The real cause (a 401, 403 or "gated" message) is printed a few lines above the final error.

Fix. Log in on the Hub, open the model page and accept the terms, create a read token, and pass it with -e HF_TOKEN=.... The older variable HUGGING_FACE_HUB_TOKEN still works as a fallback, but use the current name. Keep the token out of version control and out of images.

Not reading above the last error line

Symptom. You search the web for Error: ShardCannotStart and find nothing useful.

Cause. The last line is only the launcher's name for the stage that failed (ShardCannotStart, ShardFailed, WebserverFailed, DownloadError). The informative text is above it, often under "Shard complete standard error output".

Fix. Scroll up to the first genuine error message and search for that. Almost every common cause is in the table above: memory, shared memory, GPU visibility, authorisation.

Setting temperature to zero

Symptom. A request returns a 422 saying the temperature must be strictly positive.

Cause. In TGI, temperature must be above zero. Many people reach for zero to mean "no randomness", a convention from other APIs.

Fix. Send "do_sample": false for greedy, repeatable output. When you do sample, use a small positive temperature. If you need repeatable sampled output, set seed.

Believing the server remembers the conversation

Symptom. Your chat bot answers the second question as if the first never happened.

Cause. TGI is stateless between requests. Each call carries everything the model should see. A conversation is a list of messages that your code builds and resends.

Fix. Keep a history list, append the user message, send the whole list, then append the assistant reply. Be aware that the prompt grows every turn, so latency grows, and the history eventually exceeds max_input_tokens, which produces a 422 error. Trim or summarise old turns.

Mapping the wrong port

Symptom. curl: (7) Failed to connect even though the container is running.

Cause. The Docker image listens on port 80, set by its PORT variable, whereas the launcher's own default outside Docker is 3000. Mapping -p 8080:3000 therefore points at nothing.

Fix. -p 8080:80 for the standard image. Verify with docker ps and read the mapping column.

Putting the server on the open internet

Symptom. The server works, so you expose it with a public IP, and a week later your GPU bill spikes.

Cause. TGI has almost no built-in security. --api-key is one shared token that protects only the inference routes; /health, /info, /metrics and /docs stay open. There is no TLS, no per-user keys and no rate limiting. And since the project is archived, no new security patches will arrive.

Fix. Keep it on a private network, behind a gateway or reverse proxy that handles TLS, authentication and quotas. If you do use --api-key, remember that the launcher logs its arguments at startup, so treat the logs as sensitive.

Expecting TGI to run on a Mac

Symptom. You try the Docker command on a MacBook and it is extremely slow or fails to find a GPU.

Cause. None of the images support Apple Silicon GPUs, and Docker Desktop on a Mac has no GPU passthrough. The only documented macOS path is a CPU-only source build, which the docs do not recommend.

Fix. Do the exercises on a rented Linux GPU machine, or use another engine for local work. The catalogue's llama.cpp and Ollama guides cover local, Mac-friendly serving.

Sending huge images or prompts without checking limits

Symptom. A vision-language request fails with a 413, or a long prompt returns a 422 about too many tokens.

Cause. The server enforces a request body size (--payload-limit, 2 MB by default) and the token limits you saw in /info. Base64 images blow past the body limit quickly. In v3.3.7 there is also a cap on how large an image the server will fetch from a URL.

Fix. Read /info to know your limits before designing a client. Send image URLs instead of base64 where possible, or raise --payload-limit.

Habits worth adopting on day one

A few more habits, in the order they pay off.

Use the OpenAI-compatible endpoint in application code. Writing against /v1/chat/completions with the openai or huggingface_hub client keeps your application portable. Because TGI is frozen and Hugging Face recommends vLLM and SGLang, the day will come when the server is swapped; clients that spoke the common API need only a new URL. Keep any use of TGI-specific features, such as /generate payloads or grammar parameters, behind a thin wrapper so the dependency is in one place.

Set max_tokens on every request. Leaving it out is allowed in v3, but an explicit limit bounds cost, latency and the chance of a runaway generation. Pick a number based on what your feature needs, not the maximum.

Log the finish reason. length, eos_token and stop_sequence tell you whether answers are being cut off. A rise in length finishes is the earliest sign that your limit is too low or that the model is rambling.

Stream by default for anything a human reads. The perceived speed difference is large, and streaming is the only way to measure time to first token from the client.

Warm the server after each deploy. Send one ordinary request, and one request for each structured-output feature you use. The first grammar compile is slow, and you would rather pay that cost than let a user do it.

Turn off usage statistics if your policy requires it. TGI sends anonymous usage data by default when it runs in Docker. Add --usage-stats off in regulated environments. Decide this deliberately rather than discovering it in a network audit.

Check what is on the GPU before you start. Run nvidia-smi. A leftover process from an earlier run is a classic cause of a memory error at warmup that seems to come from nowhere.

Keep the launch command in a file. A shell script or a Compose file under version control, not a command in your shell history. When something works, you can repeat it; when it breaks, you can diff it.

Practice cards

Work through these on a small model. Each takes about ten minutes.

A short checklist

Before you call a TGI setup done, tick every line.

• The machine is Linux with an NVIDIA GPU, and docker run --rm --gpus all ubuntu nvidia-smi works.
• The image tag is pinned (for example 3.3.7), not latest.
• The model id is pinned with --revision if you need reproducibility.
• --shm-size 1g is present.
• -v ...:/data is present, and the cache folder exists on the host.
• The port mapping is host:80.
• For gated models, the licence is accepted and HF_TOKEN is a read token passed from the environment.
• No hand-set token limits unless you have a reason, and /info was read after startup.
• A health poll waits for 200 before any client runs.
• The client uses the OpenAI-compatible endpoint, sets max_tokens, and streams if a human is watching.
• Conversation history is kept by the client, and trimmed before it exceeds the input limit.
• The server is not reachable from the public internet, and the logs are treated as sensitive if --api-key is used.
• --usage-stats off is set if your policy needs it.
• The launch command lives in a versioned script.
• You have written down, somewhere, that TGI is in maintenance mode and which engine you would migrate to.

When every box is ticked, you have a server that a colleague could start, a failure you could diagnose, and a client you could point somewhere else tomorrow. That is the beginner standard, and the Mid-level track builds directly on it.

This is part two of three. It assumes you can already launch the container, call /generate and /v1/chat/completions, and read a startup log. Here we explain why the server behaves the way it does, so that you can predict an out-of-memory failure, a latency spike or a 429 before it happens, and so that you can run TGI properly on a cluster.

One framing note before anything else. Hugging Face put TGI into maintenance mode on 11 December 2025 and archived the repository on 21 March 2026. The final release is v3.3.7. Nothing in this guide will get newer upstream features, and no security patches will ship. That does not make the knowledge wasted: the router and shard split, continuous batching, paged KV cache, prefix caching and chunked prefill are exactly the concepts you will meet again in vLLM and SGLang. Treat TGI as a stable, frozen stack that many teams still run, learn how to run it well, and plan the move off it. Every production section below ends with a migration thought for that reason.

Where this picks up

Each section opens with the problem it solves and ends with a Try it you can run against a real deployment. Examples use the pinned image ghcr.io/huggingface/text-generation-inference:3.3.7. The official docs still show 3.3.5; replace it. Never use latest in production, because a moving tag turns a rollback into guesswork.

The three processes, and why it matters which one broke

Beginner said TGI is "a server". It is really three cooperating programs, and nearly every failure becomes easy to diagnose once you know which one died.

The launcher (text-generation-launcher, written in Rust) is the container's entrypoint: the image runs /tgi-entrypoint.sh, which does exec text-generation-launcher $@. The launcher parses your flags, reads the model's config.json, decides the attention implementation and whether prefix caching is allowed, downloads and converts the weights by calling the Python server's download-weights subcommand, starts one model-server process per GPU, and finally starts the router with compatible arguments. It then supervises all of them. If any child dies, the launcher shuts everything down, which is why the log ends with lines such as "Shard 0 crashed" or "Webserver Crashed" and the container exits.

The router (text-generation-router, Rust, built on the axum web framework) is the part clients talk to. It accepts HTTP, tokenizes and validates requests on a small pool of validation workers (--validation-workers, default 2), queues them, builds batches, owns the block allocator and the prefix-cache lookup, and streams tokens back over server-sent events. It talks to the shards over gRPC on a Unix domain socket, by default /tmp/text-generation-server. The router has no configuration file; everything is flags and environment variables.

The model server (text-generation-server, Python and PyTorch) is a shard: it holds its slice of the weights, owns the GPU's KV cache, and runs the prefill and decode forward passes. With tensor parallelism the shards synchronize through NCCL, the GPU collective-communication library. Its subcommands are download-weights, quantize and serve.

This split gives you a practical diagnostic rule. A validation error (HTTP 422) is decided in the router before any GPU work happens. A 429 is the router's admission control. A 424 means the request got to a shard and the shard failed. A container that exits during startup almost always has the real cause in the block headed "Shard complete standard error output", a few lines above the final Error: ShardCannotStart or Error: WebserverFailed. Read upward from the last line, not at it.

Every flag has an environment-variable twin: --max-total-tokens is MAX_TOTAL_TOKENS. In Kubernetes and SageMaker you will set environment variables, and some router-only settings, such as MAX_IMAGE_FETCH_SIZE, exist only that way. The launcher starts the router with the parent environment, so the router sees them.

Because the three programs are separate, the container has a surprising property: there is no multi-model serving. One container is one launcher, one model, N shards and one router. The only "many models in one server" feature is multi-LoRA over a single base model, which we reach later. If you need three models, you run three deployments.

The request lifecycle and the scheduler

Understanding what happens to a request between the socket and the first token explains almost every latency question you will be asked.

1. The client sends a request. The router tokenizes it and checks it against the limits. A violation returns 422 with error_type: "validation".
2. Admission control: if more than --max-concurrent-requests (default 128) are in flight, the router returns 429 Model is overloaded. Otherwise the request enters the queue.
3. The batcher takes requests from the queue that fit the block budget and sends prefill(batch) to every shard, which run it in lockstep. The first token is streamed back.
4. The router then loops decode over the running batch, one token per sequence per step.
5. When new requests arrive, the router has to pause the running batch and run a prefill for the newcomers, then resume decoding the merged batch.
6. Finished requests, and requests whose client disconnected, are filtered out of the batch and their cache blocks are released.

Two phases deserve precise names. Prefill is the first forward pass over the whole prompt. It fills the KV cache (the stored attention keys and values for all earlier tokens) and produces the first token. It is compute-bound, and it is the most memory-hungry step. Decode is one forward pass per generated token. It is bound by memory bandwidth, because every step reads all the weights and the cache. This is why time-to-first-token (TTFT) depends mostly on prompt length and queueing, while inter-token latency depends on batch size and model size.

Continuous batching is the technique of letting new requests join between decode steps instead of waiting for the entire batch to finish. It is why TGI's throughput holds up under mixed traffic. Its cost is the pause described in step 5: every time newcomers are admitted, running requests stall for the duration of a prefill. Two knobs balance this:

• --waiting-served-ratio (default 0.3 in the launcher) decides how much waiting work justifies interrupting the running batch.
• --max-waiting-tokens (default 20) is the number of decode steps after which waiting requests are forced in regardless.

Raising the ratio makes the server less willing to interrupt running requests, which improves inter-token latency and worsens TTFT for newcomers. Lowering it does the opposite. Most teams never touch it, because the defaults suit chat. Touch it only after measurement shows the specific symptom: long stalls mid-stream under bursts point at too-eager admission, and long waits before the first token under steady load point at too-lazy admission.

Streaming is tied to this loop. /generate_stream and chat with "stream": true send each token as a server-sent event the moment the router receives it. A client that disconnects frees its slot at the next filter step, so abandoned streams are cheap.

One more subtlety. If max_new_tokens is omitted, v3 computes it as max_total_tokens minus the input length, and internally re-queues the request every DEFAULT_GENERATION_LENGTH tokens so one user cannot take up a slot forever. Rather than trusting that, set max_tokens explicitly in your application. A bounded request is a request whose cost you can reason about.

The memory budget: warmup, token limits and zero config

Nearly every capacity problem on a GPU is a memory-budget problem, and TGI gives you four numbers to describe that budget. Knowing them precisely is the single most valuable thing at this level.

• max_input_tokens: the longest prompt accepted.
• max_total_tokens: prompt plus generated tokens for one request. The docs call it the most important value to set, because it is one request's memory budget.
• max_batch_prefill_tokens: the total prompt tokens the server will process in one prefill step. Prefill is the memory spike, so this caps the peak.
• max_batch_total_tokens: every token held in the KV cache across the whole batch. It is inferred from the free VRAM after the weights are loaded and a warmup pass has run. Do not set it by hand unless you must.

The launcher and router enforce four invariants between them:

max_input_tokens < max_total_tokens
max_batch_prefill_tokens >= max_input_tokens
max_batch_prefill_tokens <= max_batch_total_tokens
max_total_tokens <= max_batch_total_tokens

When you break one you get messages like Invalid argument: `max_input_tokens`(X) must be < `max_total_tokens`(Y). A separate trap: --max-input-length is a legacy alias of --max-input-tokens, and the launcher refuses to start if both are set. Use only the tokens spelling.

Since v3.0.0 these are chosen automatically, which the release called "zero config". If you leave all four unset, TGI sets max_total_tokens to the smaller of what fits in memory and the model's max_position_embeddings, and max_input_tokens to one less. The official advice is blunt: remove all the flags you are using and you are likely to get the best performance. Manually pinning these numbers on a v3 deployment is the most common self-inflicted wound, usually inherited from a v1 tutorial.

How does it measure? At startup the router asks every shard to run a warmup: a maximal prefill at the configured limits. That reveals peak memory. The shards then allocate as many KV-cache blocks as the remaining memory allows and capture CUDA graphs, which are pre-recorded decode kernels for batch sizes given by --cuda-graphs (default 1,2,4,8,16,32, and 0 disables them). Out-of-memory errors at startup nearly always happen here, which is a good property: a configuration that survives warmup is a configuration with headroom measured, not guessed.

curl -s localhost:8080/info | python3 -m json.tool

/info returns model_id, model_sha, max_input_tokens, max_total_tokens, the version, and more. It is the ground truth for what zero config chose, and it should be the first thing you paste into an incident channel.

The startup failures map cleanly to the budget:

If the GPU is shared with another process on purpose, --cuda-memory-fraction (default 1.0) tells TGI to take only a share. If it is shared by accident, no flag fixes that.

KV cache, prefix caching and chunked prefill

The KV cache is where your concurrency lives, and three v3 features decide how efficiently it is used.

PagedAttention splits the cache into fixed-size blocks reached through a lookup table, so a sequence's cache need not be contiguous. TGI's original kernels came from vLLM. The block structure matters beyond fragmentation, because blocks can be shared between sequences, and sharing is what makes prefix caching possible.

Prefix caching reuses the blocks of a prompt prefix the server has already seen: a shared system prompt, a long document asked about repeatedly, or the earlier turns of a conversation. The router looks the prefix up in microseconds, so the cost of a miss is negligible and the benefit of a hit is skipping most of the prefill. It has been on by default since v2.3 and is controlled by the PREFIX_CACHING environment variable. Hugging Face's own December 2024 benchmark claimed that on 200,000-token re-queries TGI answered in 2 seconds against 27.5 seconds for vLLM. That is a vendor measurement from the time, and the other engines have moved on since, so quote it as a claim and not as a current comparison. The mechanism, not the number, is what to remember.

The launcher switches prefix caching off automatically in four cases: vision-language models, encoder-decoder (seq2seq) models, deployments with LoRA adapters, and when the paged attention implementation is in use. Nothing errors. You simply get no benefit, and a RAG service with a 6,000-token system prompt gets slower the day somebody adds an adapter. Check the startup line that says Using attention flashinfer - Prefix caching true and treat false as a finding.

Two operational facts follow from "the cache is local to one replica". First, a load balancer that spreads one conversation across replicas will see a lower hit rate than the benchmarks suggest. The v3 docs suggest sticky sessions, hashing on a user or conversation ID, only if the hit rate matters to you, and add "do not apply this blindly": stickiness fights even load distribution. Second, a freshly restarted replica starts with an empty cache, so its first minutes look slower than steady state.

Chunked prefill processes a long prompt in pieces. Peak VRAM falls, because the prefill spike is bounded, and decode steps for other requests can proceed between chunks. It is on by default (PREFILL_CHUNKING defaults to 1) and was extended to vision-language models in v3.3.0.

Attention implementation. The launcher chooses one of three and logs it:

You can force one with the ATTENTION environment variable, but forcing paged also disables prefix caching, so do it knowingly. Flash attention needs Ampere or newer; on older cards you get FlashAttention only supports Ampere GPUs or newer. on a path that needs it. The kernels are tested on H100, A100, A10G and T4. On other NVIDIA GPUs continuous batching still applies but some operations, flash attention and paged attention, will not run as optimized kernels.

Models TGI does not implement natively fall back to the transformers library's AutoModelForCausalLM or AutoModelForSeq2SeqLM, loaded with device_map="auto". That fallback is functional but slow: no tensor parallelism, no flash attention, no paged attention. The practical rule is to check the supported-models page before you choose a model, because a fallback model will quietly cost you the features this whole section describes.

Quantization, data types and the KV cache dtype

Quantization trades a little accuracy for a lot of memory, and memory is concurrency. TGI's --quantize flag covers two very different families, and mixing them up is a common source of confusion.

Pre-quantized weights, where the model repository already contains quantized tensors and TGI just loads them: awq, gptq, marlin, exl2 and compressed-tensors. For these repositories, TGI reads the quantization method from config.json automatically, so usually you do not pass the flag at all. GPTQ uses exllama kernels when possible and falls back to triton. EXL2 cannot use tensor parallelism, and the launcher says so: Sharding is currently not supported with exl2 quantization. On AMD ROCm, AWQ checkpoints are not supported.

Quantization at load time, where TGI converts full-precision weights as it loads them: eetq (8-bit), fp8 (e4m3, for H100 and newer), bitsandbytes-nf4 and bitsandbytes-fp4. The older bitsandbytes 8-bit option is deprecated: the launcher warns "Bitsandbytes is deprecated, use eetq instead, which provides better latencies overall and is drop-in in most cases", and it also disables CUDA graphs. Replace it with eetq, or with fp8 on H100-class hardware.

If you need your own GPTQ checkpoint, the server image includes a subcommand:

text-generation-server quantize <model> <out_dir> [--upload-to-model-id ...]

Choosing between them is a matter of constraints. If the model repo already ships an AWQ or GPTQ variant, use it, because it avoids load-time work and gives predictable memory. If you only have the base model and an H100, --quantize fp8 is the least effort. If you have older Ampere GPUs, eetq is the drop-in. Always measure quality on your own evaluation set after switching; the memory saving is guaranteed, the quality cost is not zero.

Two more flags are worth knowing. --dtype float16|bfloat16 sets the compute dtype. And --kv-cache-dtype fp8_e4m3fn|fp8_e5m2 stores the KV cache itself in 8 bits on CUDA, which roughly doubles how many cached tokens fit without touching the weights. This is distinct from weight quantization and often a cheaper first step for long-context services, since the cache, not the weights, is what limits concurrency at 32k contexts.

Multiple GPUs: tensor parallelism, shared memory and replicas

When a model does not fit on one GPU, tensor parallelism (TP) splits each weight matrix across GPUs, one shard process per GPU, with NCCL exchanging partial results every layer. By default TGI uses all visible GPUs. You control it with --num-shard N or --sharded true|false. TP works only for natively supported architectures (not the transformers fallback), it needs safetensors weights, and it is intra-node: it depends on fast links between GPUs. Multi-node TP is possible with --master-addr and --master-port (default localhost:29500) but is uncommon.

The most common multi-GPU failure is not a model problem but a Docker one. NCCL uses shared memory for communication, and Docker's default /dev/shm is 64 MB. Symptoms are an NCCL error ... unhandled system error or a hang during shard initialization. The fix is --shm-size 1g (the NVIDIA install page uses 64g), and in Kubernetes a memory-backed emptyDir mounted at /dev/shm. NCCL_SHM_DISABLE=1 is a last resort that works but slows communication.

The count errors are equally mechanical: `sharded` is true but only found 1 CUDA devices means the container is not seeing the GPUs you asked for, and `num_shard` cannot be < 1 is a typo. Before v3.3.7, NVIDIA_VISIBLE_DEVICES=all or void could miscount devices, one of the reasons to be on the final release.

Do you actually want TP? Not always. TP buys capacity for a model too big for one GPU and reduces latency, but it adds synchronization at every layer and makes the whole group fail together. If the model fits on one GPU, throughput per dollar is usually better with independent replicas, each with its own GPU. On a four-GPU machine you can run two two-GPU replicas by starting two containers, each with its own CUDA_VISIBLE_DEVICES and --num-shard 2, on different ports, and putting a load balancer in front. Scaling out in TGI always means more independent replicas; there is no built-in cluster manager, no pipeline parallelism and no disaggregated prefill and decode.

docker run -d --name tgi-a --gpus '"device=0,1"' --shm-size 1g -p 8080:80 \
  -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.7 \
  --model-id meta-llama/Llama-3.1-70B-Instruct --num-shard 2 --quantize fp8
docker run -d --name tgi-b --gpus '"device=2,3"' --shm-size 1g -p 8081:80 \
  -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:3.3.7 \
  --model-id meta-llama/Llama-3.1-70B-Instruct --num-shard 2 --quantize fp8

That example needs a gated-model token (-e HF_TOKEN=...) and accepted licence terms, which we cover under debugging. Both containers share the same weights volume, so the download happens once.

Guidance, tools, LoRA and speculation

These four features change what the server does per request, and each has a cost you should know in advance.

Guided generation constrains the output with the outlines library. You can use it three ways: grammar on /generate ({"type": "json" | "regex" | "json_schema", "value": ...}), response_format on the chat endpoint, and tools with tool_choice. The first request with a new grammar compiles it, which is slow, and later requests hit a cache. So a latency alarm on the first call after a deploy is expected, and a warm-up request per grammar belongs in your readiness routine. If you want none of it, --disable-grammar-support turns it off and requests then fail with grammar is not supported.

from openai import OpenAI
client = OpenAI(base_url="http://tgi:8080/v1/", api_key="-")
resp = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Extract: Ali, 34, Cairo"}],
    response_format={"type": "json_schema", "value": {
        "name": "person", "strict": True,
        "schema": {"type": "object",
                   "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
                   "required": ["name", "age"]}}},
    max_tokens=100,
)

Tool calls changed shape in v3.2.0. The response now follows OpenAI's format: tool_calls is a list and function.arguments is a JSON string, not an object. Code written against older TGI that reads the arguments as a dictionary breaks on upgrade with a confusing type error. Always json.loads(tool_call.function.arguments). Be wary of the older doc examples that show "parameters" objects; they are pre-3.2 output.

LoRA adapters are loaded at startup with --lora-adapters (for example repo/a,repo/b@rev,name=/local/path) and chosen per request with the adapter_id parameter. Note that one docs page says lora_model; the example and the API use adapter_id. Adapters share one base model, so this is how you serve many fine-tunes on one deployment, with logical isolation only. Remember that enabling adapters turns off prefix caching.

Speculative decoding guesses several tokens ahead and verifies them in one pass, which lowers latency when guesses are right. Medusa heads are used automatically when the model repository ships them. The n-gram method (--speculate N) proposes tokens by repeating earlier ones, so it helps most on code and repetitive text and does little for free-form prose. The metric tgi_request_skipped_tokens counts what speculation saved, which is how you tell whether it is earning its place.

Running it on Kubernetes

There is no official Helm chart in the repository, so you write the manifest yourself, and the container's behaviour dictates what it must contain. It must request GPUs (nvidia.com/gpu), mount a memory-backed /dev/shm, mount a persistent volume at /data where the Hugging Face cache lives, take its configuration from environment variables, and use /health for probes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-llama
spec:
  replicas: 2
  selector:
    matchLabels: { app: tgi-llama }
  template:
    metadata:
      labels: { app: tgi-llama }
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: tgi
          image: ghcr.io/huggingface/text-generation-inference:3.3.7
          args: ["--model-id", "meta-llama/Llama-3.1-8B-Instruct", "--json-output"]
          env:
            - name: HF_TOKEN
              valueFrom: { secretKeyRef: { name: hf-token, key: token } }
            - name: API_KEY
              valueFrom: { secretKeyRef: { name: tgi-api-key, key: key } }
            - name: USAGE_STATS
              value: "off"
          ports:
            - { containerPort: 80, name: http }
            - { containerPort: 9000, name: metrics }
          resources:
            limits: { nvidia.com/gpu: 1 }
          startupProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 5
          volumeMounts:
            - { name: shm, mountPath: /dev/shm }
            - { name: cache, mountPath: /data }
      volumes:
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 1Gi }
        - name: cache
          persistentVolumeClaim: { claimName: tgi-weights }

Each choice answers a failure you have seen or will. The startup probe exists because startup is long: download, load, warmup and CUDA-graph capture can take many minutes for a 70B model, and a liveness probe tuned for a web service kills the pod mid-warmup and restarts the whole sequence, which is the "pod killed during startup" loop. A startup probe with a generous failure threshold gives it time, and only afterwards do the readiness checks take over. The /health endpoint returns 200 when healthy and 503 with {"error":"unhealthy","error_type":"healthcheck"} otherwise.

The terminationGracePeriodSeconds must exceed the launcher's --graceful-termination-timeout, which is 90 seconds by default. On SIGTERM the launcher stops accepting new work and waits for in-flight requests; if Kubernetes kills the pod at its default 30 seconds, streams are cut mid-answer on every rollout.

The weights volume is the cold-start fix. Cold start is image pull (several gigabytes) plus weight download plus warmup, and only the download is under your control. With the cache on a persistent volume, later pods skip it, and after the first successful run you can set HF_HUB_OFFLINE=1 so production pods need no Hub access or token at all. Be aware of the layout change in v2.3: weights now live under /data/hub/models--org--name, so an old volume with the pre-2.3 layout re-downloads.

Secrets come from Kubernetes Secrets, never from the image or the manifest in plain text. The read-scope HF_TOKEN is the current name; HUGGING_FACE_HUB_TOKEN is the legacy one.

Telemetry is on by default when running in Docker, and it sends start and stop events, a heartbeat, the configuration and hardware information, and error stack traces. In regulated or Gulf-region environments with data-residency rules, set USAGE_STATS=off (the manifest above does), or no-stack if you want events without stack traces.

Scaling replicas is a matter of the Horizontal Pod Autoscaler reading a good signal, which we cover under observability. The pool is only as good as the traffic distribution in front of it: use a Service with an ingress or gateway, and remember the sticky-session trade-off from the prefix-caching section.

Security: what TGI does not do for you

TGI's security model is minimal, and understanding the gaps tells you what must sit around it.

Authentication is a single shared bearer token. --api-key (or API_KEY) makes the router require Authorization: Bearer <key> on /, /generate, /generate_stream, /v1/chat/completions, /v1/completions, /vertex, /invocations and /tokenize. It does not protect /health, /ping, /info, /metrics, /v1/models, /chat_tokenize, /docs or /api-doc/openapi.json. There are no per-user keys, no per-tenant quotas, no rate limiting and no TLS termination. An unauthenticated /info and /metrics leak the model name and traffic shape, so keep TGI on a private network, and put a gateway or ingress in front for TLS, identity, quotas and routing. Because TGI speaks the OpenAI API, gateways such as LiteLLM work naturally.

Supply chain. --trust-remote-code executes Python code from the model repository, and the launcher warns about it. Since v2.0, pickle (.bin) weights are not converted unless you pass that flag, so the safe route is a safetensors revision. If you must trust remote code, pair it with --revision <commit sha> so the code you reviewed is the code you run. Pin --revision for every production model anyway, since branches move.

Image and CVEs. Because the repository is archived, no further security patches will ship. Scan the image you run (it bundles Python, Torch and many kernels, built on an NVIDIA CUDA Ubuntu base), pin its tag, and keep a migration plan with a date.

Vision-language models fetch image URLs on the server side, which is a server-side request forgery surface. Version 3.3.7 added MAX_IMAGE_FETCH_SIZE (default 1 GiB) to bound the size of a fetched image, and the error you see is Image size {n} bytes exceeds maximum allowed size of {max} bytes. Combine it with an egress network policy and --payload-limit (default 2,000,000 bytes). Set the environment variable on the container; it is a router option and not a launcher flag.

Leave off the --ngrok flags in production, because they expose the server publicly. And keep --cors-allow-origin unset unless browsers call TGI directly.

Observability and autoscaling signals

TGI exposes Prometheus metrics at /metrics on the main port, and since v3.3.0 on a dedicated listener controlled by --prometheus-port (default 9000). The dedicated port lets you scrape metrics without exposing the API port, and it is the reason the manifest above names a metrics port.

The series fall into groups, and a few carry most of the value:

Many series carry a method label separating prefill from decode. The duration series are histograms, so alert on percentiles rather than means: p95 of tgi_request_queue_duration rising while tgi_batch_current_size is flat at its ceiling is the textbook sign of saturation.

For autoscaling, utilization is a poor signal on GPU inference because a GPU can look busy while the queue is empty, or look idle while requests wait on memory. Better candidates are tgi_queue_size (are people waiting?), p95 queue duration (how long?) and tgi_batch_current_size against what you measured as your replica's comfortable ceiling. Feed one of them to the HPA through Prometheus Adapter or KEDA. These signals tie directly to the guides on Prometheus and Grafana: scrape on the metrics port, and import the official dashboard assets/tgi_grafana.json from the repository as a starting point.

Tracing uses OpenTelemetry: set --otlp-endpoint to your collector's gRPC address and --otlp-service-name if the default text-generation-inference.router does not suit you. Since v3.3.0 the router takes the trace id from incoming request headers, so spans from your gateway and TGI join one trace, which is the whole point. See OpenTelemetry for the collector side. Logs: --json-output produces structured logs, which any log shipper can parse, and LOG_LEVEL controls verbosity. If you are shipping them, remember the API-key leak above.

Benchmarking and capacity planning

A capacity number you did not measure is a rumour. TGI ships a tool and there are others, and they answer different questions.

text-generation-benchmark --tokenizer-name <id> is a text-user-interface microbenchmark. It talks to the model server over its Unix socket and bypasses the router, so it measures raw prefill and decode latency and throughput by batch size. That is useful for comparing GPUs, quantization choices or --kv-cache-dtype, and useless for predicting what users experience, because it includes no queueing, no tokenization and no HTTP. For end-to-end numbers use Hugging Face's inference-benchmarker or the k6 scripts in the repository's load_tests directory, aimed at the real endpoint.

A defensible method has five parts:

1. Use prompts shaped like production: realistic length distribution, realistic output length, realistic share of repeated prefixes.
2. Run a fixed number of requests, not a fixed duration, so runs are comparable.
3. Run twice, the second time with the prefix cache warm, and report both.
4. Run on the hardware you will deploy, not a bigger card.
5. Report TTFT, inter-token latency and throughput together, at several concurrency levels, and find the knee where latency starts to climb.

Turn that into capacity: the replica count equals peak concurrent requests divided by the concurrency at the knee, plus headroom for a failed replica and for cold-start lag. Check the figure against the KV-cache estimate from the memory-budget section; if the two disagree by a large factor, one of your assumptions about prompt length is wrong.

Debugging beyond the basics

Most TGI incidents fall into a handful of families. Reading the right place first saves an hour.

It will not start. The launcher's final line (Error: ShardCannotStart, Error: DownloadError, Error: WebserverFailed) is a category, and the cause is above it. For DownloadError, look for 401 or "gated repo" text: the model needs HF_TOKEN with a read scope, and you must have accepted the model's terms on the Hub. In air-gapped clusters, pre-populate the volume and set HF_HUB_OFFLINE=1. Also check that /data is not full.

It starts, then answers badly or slowly. Compare /info with what you expected. If max_total_tokens is lower than your prompts need, requests fail with Input validation error: `inputs` tokens + `max_new_tokens` must be <= {max_total}. Fix with a shorter prompt, truncate, a smaller max_tokens, or a bigger budget.

A model fails in a way you cannot explain. If it has only .bin weights, it will refuse to convert without --trust-remote-code; prefer a safetensors revision, which the safetensors/convert Space can open as a pull request on the model (use --revision refs/pr/<ID>). If a chat request returns Template error, the chat template rejected your messages, often a system role the template does not allow; override with --tokenizer-config-path.

Status codes are signals. The table summarizes them.

A 424 deserves special attention because it means the GPU side hit a CUDA error or an out-of-memory during decode, not at warmup. If you see them under load, you pushed past what warmup measured, so lower the concurrency or token budgets.

Slow first request after deploy. Usually grammar compilation or CUDA-graph and kernel warmup. Send a warm-up request in your rollout routine.

Silent feature loss. Prefix caching off because of LoRA or a VLM, paged attention on old GPUs, CUDA graphs off because of bitsandbytes. None of these produce errors. They produce slower service, and the evidence is in the startup log.

Tool-call parsing breaks after an upgrade. function.arguments is a string since 3.2.

Configuration management and testing

A TGI deployment has no configuration file, so its configuration is the set of flags and environment variables you pass, and that set deserves the same discipline as code. Keep it in version control as a single manifest, Helm values file or Compose file. Pin three things explicitly in it: the image tag, the model --revision as a commit hash, and any value you override from zero config. For each override, write a comment saying which measurement justified it. Overrides without a recorded reason are the ones that survive long after the hardware changed and quietly cap performance.

Prefer environment variables over command-line arguments in Kubernetes. They are easier to patch per environment with Kustomize or Helm, they are what SageMaker and Inference Endpoints expose, and the variable names mirror the flags, so a reviewer can read MAX_TOTAL_TOKENS and know what it is. Keep secrets, the Hub token and the API key, out of that file and in a secret store. Related guides cover this well: see Helm for templating the values per environment and Kustomize for overlays.

Testing a model server is different from testing a web service, because the interesting failures depend on the GPU. A useful test ladder has four rungs.

1. Contract test, runs anywhere. Your client code against a recorded or mocked response in the OpenAI shape. This catches the post-3.2 tool-call change: assert that arguments is parsed with json.loads.
2. Smoke test, runs on a GPU after every deploy. Wait for /health to return 200, read /info and assert the model id, revision and the token limits you expect, send one generation, and send one request for each guided feature you use so grammar compilation happens before real traffic.
3. Golden prompts, runs on a GPU before promoting a change. A fixed set of prompts with do_sample: false or a fixed seed, with assertions on structure (valid JSON, expected keys) rather than exact wording. Use it to approve a new quantization, a new revision or a changed KV dtype. Exact-text assertions are fragile because different kernels and dtypes change outputs slightly.
4. Load test, runs on a schedule. The benchmark from the previous section, with a threshold on the knee, so a regression in latency appears in CI instead of in production.

#!/usr/bin/env bash
set -euo pipefail
BASE=${1:-http://localhost:8080}
for i in $(seq 1 120); do
  [ "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/health")" = 200 ] && break
  sleep 5
done
curl -sf "$BASE/info" | python3 -c "import sys,json; i=json.load(sys.stdin); print(i['model_id'], i['max_total_tokens'])"
curl -sf "$BASE/generate" -X POST -H 'Content-Type: application/json' \
  -d '{"inputs":"Say hello","parameters":{"max_new_tokens":8,"do_sample":false}}'

Notice the polling loop. Because startup is slow, a test that assumes readiness after the container starts fails randomly, and the fix is always to poll /health with a long budget.

For CI pipelines on the managed runners most teams use, GPU tests are expensive. A reasonable compromise is to run the contract tests on every pull request, and run the smoke and golden tests on a GPU runner only when the manifest, the model revision or the client prompts change. See GitHub Actions or GitLab CI for conditional job triggers.

Planning the migration off TGI

Hugging Face recommends vLLM or SGLang for new work, and Inference Endpoints publishes a migration guide: you cannot change the engine of an existing endpoint, so you create a new endpoint on the new engine, test it, move traffic, then pause or delete the old one. The same blue-green habit works for your own clusters.

Migration is cheap in proportion to how disciplined your clients are. Applications that use the OpenAI-compatible /v1/chat/completions port with little more than a changed base URL. Applications that call native /generate, use TGI-specific grammar syntax, or depend on adapter_id semantics need real work. Dashboards and alerts built on tgi_* metric names need rewriting, and probes need the new health paths. So the habits to adopt now, while you still run TGI, are: talk to it through the OpenAI interface, keep prompts and schemas in your repository, wrap the client in one module so the engine is a config value, and keep a benchmark script you can aim at both engines. You may already know vLLM and SGLang from this catalogue; the concepts in this guide will map almost one to one.

Putting it all together

Here is the order in which a careful engineer brings a new TGI deployment to production.

1. Choose the model and check it is natively supported, not a transformers fallback. Pin the --revision to a commit.
2. Pin the image to 3.3.7 (or 3.3.6-rocm on AMD).
3. Launch with no token flags and read /info. Accept zero config unless a measurement says otherwise.
4. Decide quantization and KV dtype from the memory budget; evaluate quality on your own prompts.
5. Decide TP versus replicas: use TP only when the model needs it.
6. Benchmark end to end at several concurrencies and find the knee.
7. Write the manifest: startup probe, /dev/shm, weights volume, grace period above 90 seconds, secrets, USAGE_STATS=off.
8. Put a gateway in front for TLS, authentication and quotas; keep /metrics and /info private.
9. Scrape metrics, alert on queue duration and failure counts, autoscale on queue size, propagate trace context.
10. Write the runbook of status codes and startup failures, and a dated plan to migrate.

Where you are now

You can explain TGI as a launcher, a router and shards, and use that to place a failure. You can predict how the four token budgets are derived, read /info and warmup output, and tell which of prefix caching, chunked prefill and CUDA graphs are actually active. You can choose between quantization methods, decide between tensor parallelism and replicas, run the server on Kubernetes without restart loops, and put the missing security layers around it. You know which metrics matter, how to benchmark honestly, and why the migration to vLLM or SGLang should be planned now. The Senior level covers running TGI as a platform for other teams, including multi-tenancy, upgrades and incident playbooks, and the conditions under which you should stop running it.

Sources

• Text Generation Inference documentation
• Architecture
• Launcher reference
• Metrics reference
• Monitoring tutorial
• Quantization
• Tensor parallelism
• PagedAttention
• Chunking
• Speculation
• Guidance
• LoRA
• Usage statistics
• Using the CLI
• Inference Endpoints: TGI engine and migration notes
• GitHub repository and releases

This is part two of three. Mid-level interviews for TGI stop asking what the tool is and start asking what you would do when it misbehaves. The questions below are scenarios and trade-offs, each with a reasoned answer, followed by what separates a strong answer from a weak one. One fact belongs in every answer where it is relevant: TGI has been in maintenance mode since 11 December 2025, its repository was archived on 21 March 2026, and v3.3.7 is the final release. An interviewer who sees you volunteer that, and the migration implication, will trust the rest of what you say.

Where this picks up

At the beginner level you could describe TGI as an inference server and run it in Docker. Here you are expected to reason about its internals well enough to diagnose problems from symptoms, to size a deployment, and to explain choices. A good frame to keep in your head is the three processes: the launcher supervises, the router validates, queues and batches, and the shards run the model on the GPUs. Nearly every answer below becomes simpler if you say which of the three is involved.

The sixty-second mental model

If you are asked to explain how TGI serves a request, a compact answer runs like this. The client talks to the router over HTTP. The router tokenizes and validates, rejecting bad requests with a 422, and admits the request to a queue unless more than 128 are already in flight, in which case it returns a 429. The batcher forms a batch that fits the KV-cache block budget and sends a prefill to every shard, which fills the cache and yields the first token. Decoding then proceeds one token per sequence per step, and new arrivals are folded in between steps by pausing the running batch for their prefill, which is continuous batching. Finished or disconnected requests are removed and their blocks freed. Prefix caching lets a repeated prompt prefix skip most of its prefill, and chunked prefill keeps long prompts from dominating memory.

The point of memorizing this is that every operational question maps onto a step in it.

Scenario questions

The container exits during startup with an out-of-memory message. What do you do?

Start by reading upward from the final Error: line. The launcher's last line is only a category, such as ShardCannotStart or WebserverFailed, and the real cause sits in the shard's standard-error output above it. If the message says Not enough memory to handle {N} prefill tokens. You need to decrease --max-batch-prefill-tokens, the warmup pass, which runs a maximal prefill to measure peak memory, did not fit next to the weights.

Then ask whether anyone set the token flags by hand. Since v3.0.0 TGI computes max_input_tokens, max_total_tokens, max_batch_prefill_tokens and max_batch_total_tokens from the hardware, and Hugging Face's advice is to remove the flags you are using. A manual --max-batch-prefill-tokens copied from an older tutorial is the most common cause. If it persists with defaults, the options are: quantize (--quantize eetq or fp8 on H100-class GPUs, or a pre-quantized AWQ or GPTQ repository), add shards, lower --max-input-tokens, or free the GPU. If another process legitimately shares the card, --cuda-memory-fraction reserves a share.

A different message, Not enough memory to handle max_total_tokens={N}, means one full-length request cannot fit in the cache that remains, and the fix is a lower --max-total-tokens or more VRAM.

Users report that responses start slowly under load, but the GPU is not saturated. How do you investigate?

I would separate queueing from compute. Time-to-first-token is the sum of waiting in the queue, validation and prefill. Look at tgi_request_queue_duration and tgi_queue_size first. If queue time is high while the batch is below its ceiling, something is limiting admission: a low --max-concurrent-requests, or prefill budget limits. If queue time is low but prefill time is high, the prompts are long, and the cause is the prompt size, which prefix caching and chunked prefill help with.

Also check whether prefix caching is actually on. The startup line Using attention ... Prefix caching true tells you. It is silently disabled for vision-language models, encoder-decoder models, LoRA deployments and paged attention. A team that added a LoRA adapter to a RAG service and then saw slower first tokens has exactly this problem.

Finally, consider the scheduler: when newcomers arrive, running requests pause for a prefill. A steady stream of long prompts therefore hurts inter-token latency of everyone else, and --waiting-served-ratio and --max-waiting-tokens trade one against the other. I would only change them after the metrics showed that specific symptom.

Clients receive 429 responses. Is that a bug?

No, it is admission control. The router returns Model is overloaded with an overloaded error type when more than --max-concurrent-requests (default 128) are in flight. The right responses are on the client side and the capacity side. Clients should retry with exponential backoff and jitter. On capacity, either add replicas or reconsider the limit, but raising the limit alone does not create GPU memory: the queue still needs KV space, so it can convert fast 429s into slow timeouts and 424 failures. I would scale on tgi_queue_size rather than on GPU utilization, and put a gateway in front that can shed load per tenant.

Some requests fail with HTTP 424. What does that tell you?

A 424, Request failed during generation, means the request passed validation and admission and reached a shard, and the shard failed during generation. It is different from a 422, which is the router rejecting the request, and different from startup OOM, which happens at warmup. Typical causes are a CUDA error or an out-of-memory during decode under load, which means traffic pushed past what warmup measured. I would read the shard logs around the timestamps, check concurrency and prompt-length distributions, and lower token budgets or add capacity. If it appears only on one replica, I would suspect a bad GPU or node.

After upgrading from 3.1 to 3.3, your agent code throws a type error when it reads tool-call arguments. Why?

The 3.2.0 release made tool calling follow OpenAI's response format: tool_calls is a list and function.arguments is a JSON string rather than an object. Code that treated it as a dict now fails. The fix is json.loads(tool_call.function.arguments). A strong answer also notes that some older documentation examples still show the pre-3.2 shape with a parameters object, so copying from them reintroduces the bug, and that this is an example of a breaking change in a minor version, which is why you pin tags and canary upgrades.

A multi-GPU container hangs or errors at shard startup. What is the first thing you check?

Shared memory. NCCL communicates between shards through /dev/shm, and Docker's default is 64 MB. The symptoms are an NCCL unhandled system error or a hang during shard initialization. The fix is --shm-size 1g or more in Docker, and a memory-backed emptyDir mounted at /dev/shm in Kubernetes. NCCL_SHM_DISABLE=1 works but costs performance, so it is a last resort. The second check is that the container sees the GPUs you asked for: sharded is true but only found 1 CUDA devices means the device exposure is wrong, and versions before 3.3.7 could miscount when NVIDIA_VISIBLE_DEVICES was all or void.

A pod is repeatedly killed in Kubernetes while starting a 70B model. What is wrong?

The probe configuration is shorter than the startup sequence. Startup comprises image pull, weight download, loading, a warmup pass and CUDA-graph capture, which can take many minutes. A liveness probe designed for web services kills the pod during warmup and starts the sequence again. I would add a startupProbe on /health with a high failureThreshold, keep readiness separate, put the weights on a persistent volume so later pods skip the download, and for production consider setting HF_HUB_OFFLINE=1 once the cache is populated. On ROCm, remember that TunableOp adds one to two minutes of warmup.

Rollouts cut users' streams in the middle of an answer. Why, and how do you fix it?

On SIGTERM the launcher stops taking new work and waits up to --graceful-termination-timeout, 90 seconds by default, for in-flight requests. Kubernetes' default terminationGracePeriodSeconds is 30, so the pod is killed before the server finishes. Set the pod's grace period above the launcher's timeout, for example 120 seconds, and make sure the load balancer stops sending new requests when the pod becomes not-ready.

Trade-off questions

Tensor parallelism or independent replicas?

If the model fits on one GPU, I prefer replicas. Tensor parallelism splits each matrix across GPUs and synchronizes with NCCL at every layer, which adds communication overhead and ties the group's fate together: if one shard dies, the launcher shuts the whole group down. Replicas scale throughput more linearly, fail independently, and avoid the NVLink and shared-memory requirements. Tensor parallelism earns its place when the model does not fit on one GPU or when single-request latency matters more than cost. On a four-GPU machine, two two-GPU replicas, each started with its own CUDA_VISIBLE_DEVICES and --num-shard 2, is a common arrangement. Restrictions to mention: it needs natively supported architectures and safetensors weights, EXL2 cannot use it, and fallback transformers models get none.

Which quantization method should you choose?

It depends on what exists and on the hardware. If the model already has an AWQ, GPTQ or Marlin repository, use it because it avoids load-time work. If you only have the base model and an H100, fp8 is the least effort. On older hardware eetq is the drop-in replacement for the deprecated 8-bit bitsandbytes, which also disables CUDA graphs. For long-context workloads, the KV cache, not the weights, is what limits concurrency, so --kv-cache-dtype fp8_e4m3fn on CUDA may give more headroom than quantizing the weights. In every case the memory gain is guaranteed and the quality cost must be measured on your own evaluation set.

Zero config or hand-tuned limits?

Zero config by default. The v3 release derives the four token budgets from the actual hardware and model, after measuring with a warmup pass, and manual limits tend to be stale. Hand-tune only when you have a business reason, such as capping --max-input-tokens to double concurrency for a model whose full context you never use, or --max-total-tokens to bound per-request cost. Keep the invariants in mind: input must be below total, prefill must be at least input, and total must be at most the batch total. A strong answer explains that overriding is a deliberate trade of per-request capability for concurrency, not a way to tune "performance" in general.

Sticky sessions for prefix caching?

Prefix caching is per replica, so spreading a conversation over several replicas lowers the hit rate. Hashing on a user or conversation ID raises it, but at the price of uneven load and more complicated failover. The official guidance is to apply it only when the hit rate matters and not blindly. I would measure first: compare latency for repeated prefixes across replicas, and decide if the saving justifies the complexity. For short prompts it will not.

Should you run TGI in production today?

It can be reasonable for an existing, stable deployment, where the version is pinned and scanned, the team understands the failure modes, and there is a dated plan to migrate. For a new project, the maintainers recommend vLLM or SGLang, and the repository being archived means no new model support and no security patches. So I would not start a new service on it, and for existing services I would reduce migration cost: use the OpenAI-compatible API, keep configuration in version control, and keep a benchmark that can target both engines. For the migration itself, the pattern from Inference Endpoints applies: build the new deployment next to the old one, test, shift traffic gradually, then retire the old.

Native /generate or the OpenAI-compatible endpoint?

The OpenAI-compatible /v1/chat/completions is the better default: it applies the model's chat template, works with the OpenAI SDKs and other tooling, and is portable to vLLM and SGLang. The native /generate gives access to TGI-specific parameters such as best_of, decoder_input_details and top_n_tokens, but locks you in. Use native only when you need those, and isolate it behind one function.

Troubleshooting under pressure

How do you tell, from a log, which component failed?

The launcher prints which child exited. "Shard N crashed" points at the model server, and its stderr is in the preceding block. A webserver failure points at the router: it can be a bad router argument, a port conflict, or the router's own warmup failing. Error: DownloadError is a download problem, usually authentication or disk. Error: ArgumentValidation is a flag problem the launcher caught itself. Mentioning the mapping from the final line to a component is what a strong answer does.

A request fails with a validation error mentioning max_new_tokens. What happened?

The 422 text, Input validation error: `inputs` tokens + `max_new_tokens` must be <= {max_total}, means prompt tokens plus requested output exceed max_total_tokens. The remedies are to shorten the prompt, pass truncate, request fewer tokens, or increase the server budget if memory allows. A related one, temperature must be strictly positive, comes from sending temperature: 0; send do_sample: false for greedy decoding.

What would you log or alert on?

Alert on p95 tgi_request_queue_duration, on a rising tgi_queue_size, on the failure counter by error type, on 5xx and 424 rates at the gateway, and on /health failures. Autoscale on queue size or queue duration. Watch tgi_request_mean_time_per_token_duration for inter-token latency regressions, and tgi_request_input_length to notice when prompt lengths drift, because that changes capacity.

Security and operations questions

Is --api-key enough to secure TGI?

No. It is one shared bearer token that protects the generation, chat, completion, tokenize and SageMaker and Vertex-compatible routes, but not /health, /info, /metrics, /v1/models, /chat_tokenize or the docs routes. There are no per-user keys, no tenant quotas, no rate limiting and no TLS. I would keep TGI on a private network and put a gateway in front for TLS, identity, quotas and routing. A strong answer also mentions that the key appears in the launcher's startup log and in the router's process arguments, so logs are sensitive.

What are the supply-chain risks?

--trust-remote-code executes Python from the model repository; always combine it with a pinned --revision commit. Pickle weights are not converted without that flag, so prefer safetensors. The image bundles a large Python and CUDA stack and will not get new patches now that the project is archived, so scan it and pin the tag. For vision models, the router fetches image URLs itself, so bound MAX_IMAGE_FETCH_SIZE (v3.3.7), --payload-limit and egress.

What would you do about telemetry in a regulated environment?

Usage statistics are on by default inside Docker and include the configuration, hardware information and error stack traces. Set --usage-stats off, or the USAGE_STATS=off variable, and mention it in the deployment review. Regions with data-residency rules, common among Gulf and Egyptian employers, make this a routine checklist item.

What a strong answer covers that a weak one misses

Questions to ask the interviewer

Strong candidates ask back. Useful questions are: which TGI version and tag is pinned, and is there a plan and date to migrate to vLLM or SGLang; whether prefix caching is actually enabled on the production workload; what the request length distribution looks like; how the service scales and on what signal; how the weights are delivered to new pods; and where authentication happens, since TGI itself offers only a shared key. These questions show you think about the deployment as a system.

Quick reference to have ready

• Image: ghcr.io/huggingface/text-generation-inference:3.3.7. On AMD, 3.3.6-rocm. No 3.3.7-rocm or 3.3.7-neuron exists.
• Health: /health (200 or 503), /info for the chosen limits, /metrics for Prometheus, with a separate listener on --prometheus-port (9000).
• Defaults worth knowing: 128 concurrent requests, 90 second graceful termination, CUDA graphs 1,2,4,8,16,32, waiting-served ratio 0.3, max waiting tokens 20, payload limit 2,000,000 bytes.
• Status codes: 422 router validation, 429 overloaded, 424 generation failure, 413 payload, 401 key.
• Deprecated or legacy: --max-input-length, --quantize bitsandbytes, HUGGING_FACE_HUB_TOKEN, the text-generation Python client, Azure registry images, pre-3.2 tool-call shape.
• Related guides: vLLM, SGLang, Kubernetes, Prometheus.

Where you are now

You can handle the questions that follow "I have used TGI at work": diagnose startup and runtime failures from the messages, size and tune a deployment from its memory budget, argue the trade-offs between parallelism, quantization and limits, and describe its security gaps and its end-of-life status honestly. The Senior interviews build on this with system-design prompts, incident response and arguing a trade-off to leadership.

This is part two of three. At the beginner level the mistakes were about getting the container to start. At this level they are subtler: the server runs, and runs worse than it should, or fails only under load or after an upgrade. This file lists the errors that appear once you use TGI at work, the practices with the best return, practice cards you can run, and operational recipes. One thing is true of all of them: TGI is in maintenance mode, the repository is archived, and v3.3.7 is the final release, so the habits that keep a frozen system healthy matter more than chasing new features.

Where this picks up

You already know the launcher, the router and the shards, you can hit the endpoints, and you have seen zero config choose limits for you. This level is about the failure modes and the habits that avoid them. Each entry below gives the symptom, the cause and the fix, because the symptom is what you will actually see at two in the morning.

Errors at this level

Hand-set token limits copied from an old tutorial

Symptom. The server starts with lower concurrency than expected, or fails with an out-of-memory error during warmup. Cause. Since v3.0.0 TGI computes max_input_tokens, max_total_tokens, max_batch_prefill_tokens and max_batch_total_tokens from the hardware. Values from a v1 or v2 tutorial are stale and often wrong for your GPU. Fix. Delete the flags, restart, and read /info. Add a limit back only with a measurement and a comment saying why. Remember the invariants: input below total, prefill at least input, total at most the batch total.

Setting both --max-input-length and --max-input-tokens

Symptom. The launcher refuses to start and complains that max_input_length is deprecated for naming consistency. Cause. --max-input-length is a legacy alias, and setting both is an error. Fix. Use --max-input-tokens only, and search your manifests for the old name.

Using latest, or copying the docs' 3.3.5 pin

Symptom. A rollout changes behaviour with no change in your repository, or two environments differ. Cause. latest moves, and the docs still show 3.3.5 even though the final release is 3.3.7. Fix. Pin ghcr.io/huggingface/text-generation-inference:3.3.7 for NVIDIA and 3.3.6-rocm for AMD, since no 3.3.7-rocm exists. Do not pull from the Azure registry, which stopped receiving images in v3.3.6.

A multi-GPU container that hangs, or fails with an NCCL error

Symptom. Shard initialization stalls or prints NCCL error ... unhandled system error. Cause. /dev/shm is too small; Docker's default is 64 MB. Fix. --shm-size 1g or more, and a memory-backed emptyDir at /dev/shm in Kubernetes. Keep NCCL_SHM_DISABLE=1 for emergencies, because it costs performance.

Expecting tensor parallelism to work for everything

Symptom. Sharding is currently not supported with exl2 quantization, or a model that runs on one GPU and ignores the others. Cause. EXL2 cannot use tensor parallelism, and models served through the transformers fallback get no tensor parallelism, no flash attention and no paged attention. Fix. Check the supported-models list before choosing a model, and pick a different quantization format when you need sharding.

Using the deprecated 8-bit bitsandbytes

Symptom. A warning that bitsandbytes is deprecated, plus lower throughput than expected. Cause. That option also disables CUDA graphs. Fix. Use --quantize eetq, or fp8 on H100-class GPUs, or a pre-quantized AWQ or GPTQ repository.

Prefix caching silently off

Symptom. No error, but long shared prompts are slow on every request. Cause. The launcher turns prefix caching off for vision-language models, encoder-decoder models, LoRA adapters and paged attention. Fix. Look at the startup line that reports prefix caching. If it says false and your workload relies on shared prefixes, change the architecture of the deployment, for example by serving adapters separately.

Grammar or tool requests that look hung

Symptom. The first structured-output request after a deploy takes far longer than the rest. Cause. The outlines grammar is compiled on first use and cached afterwards. Fix. Send a warm-up request per schema in your rollout, keep schemas small and flat, and reuse them. If you do not use guidance at all, --disable-grammar-support turns it off, and requests that try it will fail with grammar is not supported.

Tool-call arguments treated as a dictionary

Symptom. A type error in agent code after upgrading past 3.1. Cause. From v3.2.0, tool_calls is a list and function.arguments is a JSON string. Fix. json.loads(tool_call.function.arguments), and do not copy pre-3.2 examples that show a parameters object.

temperature: 0

Symptom. HTTP 422: temperature must be strictly positive. Cause. The server validates it. Fix. For greedy decoding send do_sample: false.

Pods killed during startup

Symptom. The pod restarts repeatedly and never becomes ready, especially for large models. Cause. A probe that is shorter than download, load, warmup and CUDA-graph capture. Fix. A startupProbe on /health with a high failure threshold, a persistent weights volume, and a terminationGracePeriodSeconds above the 90-second graceful timeout.

Cutting off streams at rollout

Symptom. Users see half an answer when a pod is replaced. Cause. The pod is killed before the launcher finished draining. Fix. A longer grace period, and readiness failing first so new requests stop arriving.

Trusting --api-key alone

Symptom. An audit finds /info and /metrics open to anyone who can reach the service. Cause. The key does not protect /health, /ping, /info, /metrics, /v1/models, /chat_tokenize, /docs or the OpenAPI JSON. Fix. Keep TGI on a private network and put a gateway in front. Also remember the key is written to the startup log and shows in the router's command line.

A cold node pulling the image and the weights at the same time

Symptom. The first replica on a new node takes ten minutes or more to become ready. Cause. The image is several gigabytes and the weights must download too. Fix. Pre-pull the image onto GPU nodes, cache weights on a persistent volume, and set HF_HUB_OFFLINE=1 once cached. An old volume with the pre-2.3 layout (/data/models--...) re-downloads, because weights now live under /data/hub.

The practices that pay off most, in order of leverage

1. Read /info after every start. It tells you the actual model, revision and token limits. Most "performance mysteries" are an unexpected number here.
2. Keep zero config until a measurement says otherwise. The cheapest optimization is deleting flags.
3. Pin everything: image tag, model --revision as a commit hash, client library versions.
4. Put the weights on a volume. Cold start is the largest controllable cost, and it also removes the Hub dependency in production.
5. Give the pod the right lifecycle: startup probe, readiness, a grace period above 90 seconds.
6. Measure end to end and find the knee. The microbenchmark text-generation-benchmark bypasses the router, so use it for comparing hardware or quantization, and use inference-benchmarker or k6 for capacity.
7. Scale on queue size, not GPU utilization.
8. Put a gateway in front. Authentication, TLS, quotas and routing are not TGI's job.
9. Use the OpenAI-compatible API in application code, so the engine is a configuration value.
10. Date your migration. The sooner you can run vLLM or SGLang next to TGI on the same benchmark, the cheaper the move.

Practice cards

Each card is a short exercise. Do them on a real GPU box or a development cluster.

Operational recipes

Recipe: a deploy smoke test

After every rollout, wait for readiness, check /info, send one generation, and send one request for each guided feature you use.

BASE=http://tgi.internal:8080
until [ "$(curl -s -o /dev/null -w '%{http_code}' $BASE/health)" = 200 ]; do sleep 5; done
curl -s $BASE/info | python3 -c "import sys,json; i=json.load(sys.stdin); print(i['model_id'], i['model_sha'], i['max_input_tokens'], i['max_total_tokens'])"
curl -s $BASE/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"tgi","messages":[{"role":"user","content":"ping"}],"max_tokens":8}'

The loop matters: do not assume readiness the moment the container starts. If --api-key is set, add -H "Authorization: Bearer $KEY" to the generation call.

Recipe: two replicas on one four-GPU machine

Give each container its own devices and port, and the same weights volume.

docker run -d --gpus '"device=0,1"' --shm-size 1g -p 8080:80 -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.7 --model-id <model> --num-shard 2
docker run -d --gpus '"device=2,3"' --shm-size 1g -p 8081:80 -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.7 --model-id <model> --num-shard 2

Put a load balancer in front, and health-check each on /health.

Recipe: air-gapped production

1. On a connected machine, run the container once with HF_TOKEN set so the weights land in /data/hub.
2. Copy the volume, or snapshot the persistent volume, into the cluster.
3. Run with -e HF_HUB_OFFLINE=1 and no token.

Production pods now hold no Hub credential at all, which is both safer and faster to start.

Recipe: replacing a pickle-only model

If a model has only .bin files, TGI will not convert them without --trust-remote-code. Use the safetensors/convert Space to open a pull request on the model repository, then launch with --revision refs/pr/<ID>. Pin to the resulting commit once merged, and never combine trust-remote-code with an unpinned revision.

Recipe: the incident one-pager

Keep this where on-call can find it.

Recipe: shipping logs safely

Turn on --json-output for structured logs. Because the startup arguments, including the API key, are logged without redaction, filter the key at the shipper, or drop the startup block entirely, and rotate the key if it has already reached a shared index. For OpenTelemetry, set --otlp-endpoint, and since v3.3.0 the router continues the trace id from incoming headers, so your gateway's trace and TGI's join up. See OpenTelemetry for the collector side.

Recipe: a migration rehearsal

Run vLLM next to TGI on the same model and the same benchmark. Send the same OpenAI-format requests to both and compare latency at several concurrencies, structured-output behaviour and tool-call shape. List every client call that uses native /generate, TGI-specific grammar syntax or adapter_id, because those are the parts that need changes. See vLLM for the target side.

Habits for your first week on an inherited deployment

Most people at this level do not build a TGI deployment from scratch; they inherit one. The first week is best spent on a short audit rather than on tuning, because the problems you find will usually be configuration drift rather than software defects.

Start with identity. Run docker inspect or read the manifest to learn the exact image tag, and compare it with version and sha from /info. Then check the model revision. If either is a moving reference, such as latest or a branch name, your first change is to pin it, because you cannot reason about a system whose contents change without a commit.

Next, read the full startup log once, slowly. You are looking for five lines: the attention implementation and prefix-caching state, the quantization in use, the token budgets that were chosen or overridden, the CUDA-graph setting, and any deprecation warning. A warning about bitsandbytes or max_input_length is a free fix. A line showing that prefix caching is off is a conversation to have with whoever added the adapter or the vision model.

Then compare flags with intent. For every non-default flag in the manifest, ask who added it and why. Many will be historical, copied from a tutorial written before v3.0.0 made limits automatic. Remove them in a staging environment, run the smoke test and the benchmark, and keep only the ones that measurably help. Do this one at a time so you can attribute any change.

After that, look at the edges of the system. Which routes are reachable from outside the cluster? Is /metrics scraped, and is it also public? Does the gateway handle authentication, or is the shared --api-key the only protection? Do logs containing the key go to a shared index? These questions take an hour and often reveal the highest-risk finding of the week.

Finally, write down two numbers: the concurrency at which latency starts to climb, and the time from pod creation to ready. The first tells you how many replicas peak traffic needs, and the second tells you how far ahead of a spike you must scale. Both go into the runbook, together with a dated plan to evaluate vLLM or SGLang, since a deployment that cannot say when it will move is a deployment that will be moved in an emergency.

A short checklist before you call it production

• Image pinned to 3.3.7 (or 3.3.6-rocm), model revision pinned to a commit.
• Token limits left to zero config, or each override justified in a comment.
• /info checked and saved after deploy.
• Startup probe, readiness probe, and a grace period above 90 seconds.
• /dev/shm mounted as memory, weights on a persistent volume.
• Prefix-caching state known and expected.
• Quantization choice evaluated on your own prompts.
• Gateway in front; /info and /metrics not publicly reachable; logs treated as sensitive.
• USAGE_STATS=off where policy requires it.
• Metrics scraped, queue-based alert and autoscaling in place.
• Benchmark results with a recorded knee.
• A runbook with the status-code table, and a dated migration plan.

Where you are now

You can recognise the mid-level failure modes by their messages, apply the small set of habits that prevent most of them, and run exercises that make the behaviour visible rather than theoretical. The Senior level takes this to platform ownership: multi-tenant setups, upgrade and migration strategy, incident playbooks and the point at which TGI is the wrong tool.

This is part three of three. The earlier levels taught you to run TGI and to tune it. This one is about owning it as a platform that other teams build products on, which changes the questions: what breaks first, who is allowed to call it, what an upgrade means when there are no more upgrades, and when the honest answer is to move to a different engine.

One fact frames everything below. Hugging Face put Text Generation Inference into maintenance mode on 11 December 2025, and the GitHub repository was archived, read-only, on 21 March 2026. The final release is v3.3.7, published 19 December 2025. Nothing in this guide assumes new features, new model architectures or new security patches. You are operating a frozen, mature, widely deployed serving stack, and a large part of senior ownership is deciding how long to keep doing so.

Where this picks up

I will start with the architecture, because every failure mode in this guide is a statement about one of three processes.

Architecture and failure modes

You already know the three components: the launcher supervises, the router admits and batches, the model server shards hold the weights and the KV cache. The platform view adds a fourth thing, the container boundary, because that is the unit your orchestrator restarts. One container is one model and one launcher. There is no multi-model serving inside an instance, no cluster manager, no pipeline parallelism and no disaggregated prefill and decode. Scaling out means more independent replicas behind a load balancer, and that single design fact explains most of what follows.

Think of failures by blast radius. A bad request, such as a prompt over the limit, a stop list longer than four entries or a template error, fails alone with a 422 and costs the rest of the platform nothing. A queue that grows past --max-concurrent-requests (128 by default) makes the router answer 429 with "Model is overloaded"; this is a healthy, loud refusal and a signal to scale out, not a fault. A failure during generation, usually a CUDA error or an out-of-memory in decode, surfaces as HTTP 424 with Request failed during generation; the in-flight requests on that batch are lost, and whether the process survives depends on the error.

The failure that matters most for a platform owner is the coupling between the children. If any shard dies, the launcher shuts everything down and prints Shard N crashed or Webserver Crashed. That is the correct behaviour for tensor parallelism, since a missing shard cannot be replaced mid-flight, but it means one GPU fault on an eight-GPU node takes the whole replica, every in-flight stream and the warm KV cache with it. Your orchestrator then restarts the container, and the restart costs a full warmup. The recovery time is the property you must measure, because it decides how much spare capacity you need to hide a single replica loss.

The second coupling is startup. Before a replica serves, the launcher downloads and converts weights, loads shards, and the router runs a maximal warmup prefill to measure free memory, then allocates the KV blocks and captures CUDA graphs. Most out-of-memory errors happen here, with messages such as Not enough memory to handle {N} prefill tokens or Not enough memory to handle max_total_tokens. A replica that loops through crash and restart on those errors is a configuration problem rather than a hardware problem: someone set a token budget by hand that the GPU cannot honour. Zero config, the v3 default, exists to prevent exactly this, so any manual override on your platform should need a recorded reason.

The router and shards talk over gRPC on a Unix domain socket, which means they are designed to share a host. The Architecture page says they need not be deployed together, but the launcher wires them with a socket path, so treat a split topology as a custom build you would own entirely. For a platform, the supported shape is the plain one: a container, a launcher, a router, N shards.

Capacity and what breaks first

At small scale the thing that breaks first is GPU memory, specifically the KV cache. At platform scale there is an ordered list, and knowing the order lets you instrument the right things.

First, KV cache memory bounds concurrency. Roughly, concurrent requests equal max_batch_total_tokens divided by the average prompt plus generated tokens. The official example is a 128k-context 70B model on hardware that fits the context only three times: it serves three full-context requests at once, and capping --max-input-tokens at 64k doubles that. The lesson for a platform is that context length is a priced resource. A tenant who is allowed 128k tokens is consuming an order of magnitude more capacity than one who is allowed 8k, and a flat per-request price hides that.

Second, the queue. Admission is bounded by --max-concurrent-requests, but the real constraint is that queued requests still need KV blocks before they run. Queue depth, tgi_queue_size, and the 95th percentile of tgi_request_queue_duration tell you that demand has outrun supply. These are better autoscaling signals than GPU utilisation, because a saturated decode loop shows high utilisation at both healthy and unhealthy load.

Third, prefill and decode interference. Continuous batching admits newcomers by pausing the running batch for a prefill. Long prompts therefore show up as inter-token latency spikes for everyone else on that replica. Chunked prefill, on by default in v3, softens this by splitting long prompts, and --waiting-served-ratio (launcher default 0.3) and --max-waiting-tokens (default 20) trade time to first token for newcomers against smoothness for running streams. Treat those two flags as a policy dial and change them only against a measured latency objective.

Fourth, the single-model replica model. Because there is no multi-model serving, a platform with ten models runs ten deployments, each with its own warm pool. Low-traffic models waste whole GPUs. This is the point where multi-LoRA over one base model is attractive, and also the point where scale-to-zero starts to look tempting despite its cold starts.

The hard limits are worth keeping on one page, because they decide what you can promise. EXL2 cannot use tensor parallelism. Models that fall back to the transformers path, those TGI does not implement natively, get no tensor parallelism and no flash or paged attention. Vision-language models, encoder-decoder models and LoRA turn prefix caching off. Neuron needs static shapes. TensorRT-LLM engines are fixed at compile time to a batch size, input and output lengths and a GPU architecture. ROCm has no AWQ and no sliding-window kernel.

For load testing use inference-benchmarker from Hugging Face or the k6 scripts under load_tests/ in the repository for end-to-end behaviour. The text-generation-benchmark TUI talks to the model server over the Unix socket and bypasses the router, so it measures the shards alone and says nothing about queueing or validation.

Reading the router, not just the engine

Senior ownership means diagnosing from metrics before you read logs. The series you rely on come from GET /metrics on the main port, and since v3.3.0 from a dedicated listener on --prometheus-port (default 9000). That separation matters for a platform: you can scrape the metrics port from inside the cluster while the API port sits behind a gateway, so the metrics endpoint is never exposed to tenants.

Group the series by the question they answer. Batch series, tgi_batch_current_size, tgi_batch_current_max_tokens, tgi_batch_forward_duration, tgi_batch_decode_duration and tgi_batch_filter_duration, describe the engine. Request series, tgi_request_count, tgi_request_duration, tgi_request_queue_duration, tgi_request_validation_duration, tgi_request_generated_tokens and tgi_request_mean_time_per_token_duration, describe the user's experience. tgi_queue_size is the pressure gauge. The code also emits tgi_request_failure with an err label such as overloaded or validation, which is how you separate tenant mistakes from platform faults.

A service-level objective for an LLM platform has at least three parts. Time to first token is largely queue duration plus prefill. Inter-token latency is tgi_request_mean_time_per_token_duration. Availability is the share of requests not ending in 424 or 5xx, with 422 counted as the caller's fault and 429 counted according to your own contract. Write down whether a 429 is a breach, because capacity planning flips depending on the answer.

Alert on causes. A rising tgi_queue_size with flat tgi_batch_current_size means decode is saturated and you should add replicas. A rising tgi_request_validation_duration is unusual and points to the tokenizer workers (--validation-workers, default 2) being starved by very large inputs. A climbing 424 count with healthy queue depth points to the GPU or the driver. Add the replica-restart count from your orchestrator, because a crash loop looks like a latency problem from the client side.

Tracing completes the picture. Set --otlp-endpoint to a collector and --otlp-service-name to something unique per deployment, and since v3.3.0 the router continues the trace context from incoming headers instead of starting a new trace, so a request can be followed from your gateway through TGI. For structured logs, --json-output. Pair this with the Prometheus and Grafana guides: the official dashboard is assets/tgi_grafana.json in the repository, and community dashboards exist as Grafana IDs 19831 and 20246. See Prometheus, Grafana and OpenTelemetry.

The trust model

TGI's security posture is small and you should state it plainly to every team that consumes the platform: there is a single shared bearer token and nothing else. --api-key (or API_KEY) makes the router compare Authorization: Bearer <key>, case-insensitively, on a specific list of routes: /, /generate, /generate_stream, /v1/chat/completions, /v1/completions, /vertex, /invocations and /tokenize. It does not protect /health, /ping, /info, /metrics, /v1/models, /chat_tokenize, /docs or /api-doc/openapi.json. There are no per-user keys, no roles, no per-tenant rate limits, no audit trail and no TLS termination.

Two consequences follow. First, anything reachable on the API port can read /info and /metrics without a key, which reveals model identity, versions, limits and traffic patterns. Second, a key shared by everyone cannot be revoked for one team. Put TGI on a private network, put a gateway in front of it that terminates TLS and authenticates callers, and give the gateway a single backend key that never leaves your platform.

Other items belong in the trust model. CORS is controlled by --cors-allow-origin; leave it unset unless browsers call TGI directly, and they should not. The flags --ngrok, --ngrok-authtoken and --ngrok-edge expose the server publicly and have no place in production. Vision models make the router fetch image URLs itself, which is a server-side request forgery surface: the fetch size is capped by MAX_IMAGE_FETCH_SIZE (added in v3.3.7, default 1 GiB, set as a container environment variable because it lives on the router and not on the launcher), and you should add egress network policy so the router cannot reach internal endpoints or cloud metadata addresses, plus a tight --payload-limit for base64 images.

Usage telemetry deserves a decision, not a default. Inside Docker, TGI sends start and stop events, a heartbeat every fifteen minutes, its configuration, GPU and CPU information and error stack traces. --usage-stats off (or USAGE_STATS=off) disables it, and no-stack keeps events but drops stack traces. In regulated Gulf and Egyptian environments with data-residency rules, set it to off and say so in the platform's data-flow document.

Secrets, weights and the supply chain

The credential inventory is short: HF_TOKEN for the Hub (read scope only; HUGGING_FACE_HUB_TOKEN is the legacy name that the router still falls back to) and API_KEY for the server. Inject both from a secret store, such as a Kubernetes Secret fed by Vault or your cloud's secret manager, and never bake them into an image. The strongest pattern is to remove the Hub credential from production entirely: download weights once in a controlled job into a volume, then run the serving pods with HF_HUB_OFFLINE=1 and no token. Those pods hold no Hub credential to steal.

Model weights are an executable supply chain. Three controls apply. Pin --revision to a commit SHA so a model repository cannot change underneath you. Prefer safetensors: since v2.0, pickle .bin weights are not converted unless you pass --trust-remote-code, and that is a good default to keep. If a repo ships only pickle, use the Hub's safetensors conversion Space, which opens a pull request, and serve it with --revision refs/pr/<ID>. And treat --trust-remote-code as arbitrary code execution from the model repository: allow it only for reviewed repos at a pinned SHA, and reject it in platform admission policy otherwise.

The container image is the other half. It is based on an NVIDIA CUDA Ubuntu image and bundles Python, Torch and many kernels, which is a large attack surface, and because the repository is archived, no one will ship patches for it. Your obligations are therefore ones upstream used to share. Pin an exact tag, 3.3.7 for NVIDIA and 3.3.6-rocm for AMD (no 3.3.7-rocm exists), and never latest. Mirror the image into your own registry so a deleted or moved upstream cannot stop a deployment; note that Azure registry images stopped being pushed in v3.3.6, so ghcr.io is the source. Scan the mirrored image on a schedule with Trivy, record the results, and set a policy for what happens when a critical finding has no upstream fix, since it never will. Options are accepting the risk with an expiry, mitigating with network and runtime controls, or accelerating the migration.

Multi-tenancy in a server with no tenants

A platform serves many teams and TGI knows about none of them. Isolation is therefore something you assemble around it, and it is worth being exact about which layer gives which guarantee.

Identity and quota live at the gateway. TGI speaks the OpenAI API, so a gateway such as LiteLLM or your API gateway can issue per-team keys, enforce token and request quotas, record usage per key and route by model name to the right TGI deployment. That gateway is also where you put a maximum context per tenant, which is the single most effective capacity control, since context length drives KV consumption.

Fairness inside a replica comes from the scheduler. max_waiting_tokens and waiting_served_ratio govern how aggressively newcomers interrupt running streams, and long max_new_tokens requests are re-queued in chunks so one caller cannot hold all slots. This is global fairness between requests, not between tenants. A single tenant sending thousands of concurrent requests still fills the queue, which is why per-tenant concurrency limits belong in the gateway.

Isolation of weights and data is by deployment. If two teams must not share a model or a log stream, give them separate deployments, because everything inside one replica shares a process, a cache and logs. Multi-LoRA lets several fine-tunes share a base model: adapters are loaded at start with --lora-adapters and selected per request with adapter_id, and the isolation is logical only. Remember that LoRA turns prefix caching off, so you trade cache hits for density. The official LoRA page says lora_model where the API actually takes adapter_id; do not copy that error into your platform docs.

The prefix cache is shared across tenants within a replica. That is documented as a performance feature; whether it creates a timing side channel for sensitive workloads is an inference, not a documented claim, so decide it as a risk question. Where tenants have different sensitivity levels, separate deployments are the conservative answer.

Prefix caching is also per replica, not cluster-wide. The v3 documentation suggests sticky sessions, hashing on a user or conversation ID, only where cache hits matter, with the explicit warning not to apply this blindly. Sticky routing concentrates load unevenly, so measure the hit benefit before paying the balance cost.

Deploying it on Kubernetes

There is no official Helm chart in the repository, so the platform team owns the manifest. The requirements follow from the architecture. Request nvidia.com/gpu. Mount /dev/shm as a memory-backed emptyDir with a size limit, because Docker's default 64 MB of shared memory causes NCCL errors under tensor parallelism. Mount a persistent volume or node cache at /data, where the image keeps HF_HOME. Configure through environment variables, since every launcher flag has an upper-case twin. Use /health for probes, remembering that it returns 200 only when healthy and 503 with {"error":"unhealthy","error_type":"healthcheck"} otherwise.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi-llama
spec:
  replicas: 2
  selector:
    matchLabels: { app: tgi-llama }
  template:
    metadata:
      labels: { app: tgi-llama }
    spec:
      terminationGracePeriodSeconds: 120
      containers:
        - name: tgi
          image: registry.example.com/mirror/text-generation-inference:3.3.7
          args: ["--model-id", "/data/llama", "--usage-stats", "off", "--json-output"]
          env:
            - { name: HF_HUB_OFFLINE, value: "1" }
            - name:  API_KEY
              valueFrom: { secretKeyRef: { name: tgi-backend, key: api-key } }
          ports:
            - { name: http, containerPort: 80 }
            - { name: metrics, containerPort: 9000 }
          resources:
            limits: { nvidia.com/gpu: 1 }
          startupProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            failureThreshold: 90
          readinessProbe:
            httpGet: { path: /health, port: http }
          volumeMounts:
            - { name: weights, mountPath: /data }
            - { name: shm, mountPath: /dev/shm }
      volumes:
        - name: weights
          persistentVolumeClaim: { claimName: llama-weights }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 1Gi }

Three details deserve emphasis. The startup probe must be generous: download, load, warmup and CUDA graph capture can take many minutes for large models, and a liveness check shorter than that kills pods in a loop that looks like a bug. On shutdown, the launcher stops accepting work and waits up to --graceful-termination-timeout (90 seconds by default) for in-flight requests, so set terminationGracePeriodSeconds above it, as above. And the container listens on port 80 because the image sets PORT=80, while the launcher's default outside Docker is 3000, a difference that wastes an afternoon whenever someone ports a command line between the two.

Autoscaling should use the queue signals. A horizontal autoscaler driven by tgi_queue_size or a queue-duration percentile through the Prometheus adapter reacts to demand, while CPU is meaningless here. Remember that a new replica is useful only after a cold start made of image pull (many gigabytes), weight load and warmup, so keep a floor of warm replicas and scale on trend rather than on instantaneous spikes. See Kubernetes and Helm for packaging this in your own chart, and Argo CD for reconciling it.

Hardware backends as a platform decision

TGI shares its router, API and launcher experience across several model-server backends, and the choice of backend is a platform decision with long consequences, because "not all variants provide the same features". CUDA is the default and the best exercised path. ROCm serves AMD MI210, MI250 and MI300 parts from the 3.3.6-rocm image, with the PyTorch TunableOp tuning on by default, which adds a minute or two to warmup, and with no AWQ checkpoints and no sliding-window attention kernel. Intel GPU and CPU images, built on IPEX, run with --cuda-graphs 0. Gaudi uses the habana runtime. AWS Inferentia2 and Trainium run through the Neuron backend, which needs a model exported with static shapes or a hit in the aws-neuron/optimum-neuron-cache model cache, so a request limit is baked in at export time. TensorRT-LLM needs engines compiled ahead of time with Optimum-NVIDIA and tied to a GPU architecture and shape limits. The llama.cpp backend has no published image at all and must be built from Dockerfile_llamacpp, then run on the same CPU architecture it was built on.

For a platform, the practical rule is to standardise on one backend per fleet and to treat each additional backend as a separate product with its own images, its own runbooks and its own test suite. A tenant who asks for "the same model on cheaper hardware" is asking for a new product, not a flag. Also check tags before promising anything: the newest AMD and Neuron tags are one patch behind the NVIDIA ones, so a fleet that mixes vendors cannot run a single version everywhere.

Optimised kernels are officially tested on H100, A100, A10G and T4. On other NVIDIA GPUs, continuous batching still applies but some operations, such as flash attention and paged attention, are not executed, and flash attention needs Ampere or newer. If your fleet contains older cards, label those nodes so schedulers do not place large models on them by accident, and expect the launcher to select paged attention for compute capability below 8.

Structured output and tool calling as platform features

Guidance is constrained decoding built on outlines. Consumers reach it as grammar on /generate, as response_format on /v1/chat/completions, or through tools and tool_choice. For a platform owner, three behaviours matter more than the syntax.

The first request with a new grammar compiles it, which is slow, and later requests hit a cache. A team that deploys a new schema at noon and sees a latency spike on its first calls is seeing compilation, not a regression. Offer a warm-up step in your deployment tooling that sends one request per schema, and tell consumers about it. A schema that changes on every request defeats the cache entirely, so treat dynamic schemas as a design smell and raise it in review.

Server-side, --disable-grammar-support turns the feature off, after which requests that carry a grammar fail with grammar is not supported. Decide per deployment whether you want it, since it adds a compilation workload that a latency-sensitive tenant may not want sharing a replica.

Tool calling changed shape in v3.2.0, and your consumer documentation should show the current form: tool_calls is a list and function.arguments is a JSON string that the client must parse. The tools section of the official guidance page still shows the older shape, so do not paste from it. Include a tool-calling case in the contract tests, because it is the feature most likely to differ between engines during migration.

Tokens, templates and onboarding a new model

New-model intake is where platforms accumulate hidden risk, so make it a procedure. First, check the architecture against the supported-models list. A model TGI does not implement natively falls back to the transformers path with device_map="auto", which means no tensor parallelism and no flash or paged attention; that is acceptable for a small experiment and not for a production tenant. Second, check the weights format. Safetensors is required for tensor parallelism and is the safe default; a pickle-only repo needs a converted revision. Third, check the chat template. The Messages API applies the template from tokenizer_config.json, and a template that rejects a system role produces Template error: ... for every request that includes one. You can override it with --tokenizer-config-path, but an override is an exception to register, because it changes the exact text the model sees.

Fourth, decide limits. Read what zero config chose from /info, then lower the context at the gateway to what the tenant needs. Fifth, record the licence and the pinned revision. Sixth, run the evaluation the tenant cares about against the quantized variant you intend to deploy, not against the unquantized original. Finally, load test with a realistic prompt length mix, cold and warm cache, and record the concurrency the replica sustains within the latency objective. That number becomes the capacity figure that autoscaling and quotas use.

Remember the multi-region angle: many organisations in the Gulf and Egypt need weights and logs to stay in country. Store the weights in a bucket in your region, mirror images in the same place, and keep the router's OTLP traces and JSON logs in region too.

Cost, cold starts and utilisation

The cost driver is GPU hours, and the platform metric that matters is useful utilisation: tokens served per GPU-hour at your latency objective. A cluster with ten percent of its GPU time spent serving is expensive regardless of how well each replica is tuned.

The levers, from cheapest to most invasive. Right-size --max-total-tokens and enforce it per tenant, since long contexts reduce concurrency for everyone. Quantize: pre-quantized AWQ, GPTQ or Marlin weights, eetq on the fly as the drop-in for the deprecated 8-bit bitsandbytes path, and fp8 on H100-class hardware, with --kv-cache-dtype fp8_e4m3fn or fp8_e5m2 (CUDA only) to hold more concurrent context. Use prefix caching where prompts share a long prefix. Use speculation, Medusa heads when the repo has them or n-gram with --speculate N, when latency rather than throughput is the goal. Consider different hardware: the repository ships Inferentia2 and Trainium images through the Neuron backend, Gaudi and Intel images, though note the newest Neuron and ROCm tags are 3.3.6-neuron and 3.3.6-rocm.

Cold starts decide whether scale-to-zero is viable. A start is image pull plus weight download plus warmup, plus on ROCm the PyTorch TunableOp tuning that adds a minute or two unless you set PYTORCH_TUNABLEOP_ENABLED=0. Mitigate with pre-pulled images on GPU nodes, a weights volume so nothing downloads at start, and HF_HUB_ENABLE_HF_TRANSFER=1, which the image already sets. Scale-to-zero is reasonable for internal, low-traffic models whose users tolerate minutes of wait, and wrong for interactive products.

For a region note: GPU availability and egress to the Hub differ across Middle East cloud regions, and a mirror of both image and weights inside your region is usually faster and removes an external dependency at the worst possible moment, which is when you are scaling up in an incident.

Upgrades, and the migration that replaces them

For most tools this section would describe a rolling upgrade process. For TGI it describes the end of one. Version 3.3.7 is final. There is nothing to upgrade to, so what you own is a migration.

Still, the history teaches how to run any engine change, because breaking behaviour shipped in minor versions. Version 2.0.0 stopped converting pickle weights without --trust-remote-code and turned CUDA graphs on by default. Version 2.3.0 moved the hub cache from HUGGINGFACE_HUB_CACHE to HF_HOME, so a weights volume laid out as /data/models--... became /data/hub/models--... and silently re-downloaded, and it turned prefix caching on. Version 3.2.0 changed tool calling to the OpenAI shape: tool_calls is a list and function.arguments is a JSON string, so code that treated arguments as a dictionary broke. Version 3.0.0 made token budgets automatic and turned prompt logprobs off unless --enable-prefill-logprobs is set. The platform habit is the same each time: read the notes, pin the tag, canary, and run a contract test suite against the new version.

Hugging Face's own guidance is to move to vLLM or SGLang, with llama.cpp or MLX for local use. Inference Endpoints now recommends vLLM or SGLang and publishes a migration guide, and because you cannot change the engine of an existing endpoint, the process there is to create a new endpoint, test it, move traffic and then pause the old one. That pattern, parallel run then cutover then retire, is the right shape for your own platform too.

What transfers cleanly is everything built on the OpenAI-compatible /v1/chat/completions, which vLLM and SGLang also serve. What you must re-validate is a short list: native /generate payloads, which are TGI-specific; grammar and response_format syntax; LoRA adapter_id semantics; Prometheus metric names, since the tgi_* series will not exist; health-probe paths; chat template behaviour; and the exact token budgets, because the memory model differs. Write the contract tests against the gateway's public API now, run them against TGI, and rerun them unchanged against the candidate. See vLLM and SGLang for the targets.

Governance and incident playbooks

Governance for a frozen engine is lightweight but real. Keep a decision register with three entries per model deployment: the pinned image tag and digest, the pinned model revision, and the owner. Keep an exception register for anything that deviates from the platform default, for example a manual --max-total-tokens, --trust-remote-code, a disabled --usage-stats override or a non-default quantization, each with a reason and an expiry date. Review both quarterly, and review the migration plan at the same time, because the cost of delaying it rises as the pinned image ages.

Write playbooks before you need them. Four cover most incidents.

Replicas crash-looping at startup. Read the lines above Error: ShardCannotStart or WebserverFailed, under "Shard complete standard error output". If you see Not enough memory to handle a number of prefill tokens, someone set --max-batch-prefill-tokens or --max-input-tokens by hand or another process holds the GPU: remove manual limits, lower the prefill budget, quantize or add shards. If you see Error: DownloadError with 401 or 403 text, the token or the gated-model terms are the problem, or a disk is full. If NCCL errors or hangs appear under tensor parallelism, check shared memory first.

Latency rising with healthy GPUs. Check tgi_queue_size and queue duration. Rising queues mean add replicas or cut per-tenant context limits. If queues are flat but inter-token latency spikes, look for long prompts arriving, which show up in tgi_request_input_length, and consider --waiting-served-ratio and chunked prefill behaviour.

A flood of 429. The replica is at --max-concurrent-requests. Scale out, fix client retry storms with backoff and budgets, and only then consider raising the limit, since queued requests still consume memory.

Tool-calling clients broke after a deploy. If anything moved across 3.2.0, function.arguments is now a string; clients must json.loads it.

Each playbook should end with an owner, an escalation route and a statement of what evidence to capture. For bug reports and for your own records, text-generation-launcher --env prints runtime environment information. Because there will be no upstream to report to, your evidence is for your own post-incident review and for the team building the replacement.

Where TGI stops

Pick something else, now or soon, when one of these is true.

You need features TGI will never receive. New model architectures will not be added. If a model you must serve is not on the supported list, you are in the transformers fallback with no tensor parallelism and no flash attention, which is rarely acceptable for large models.

You need cluster-level serving features. There is no disaggregated prefill and decode, no pipeline parallelism, no multi-model serving in one instance and no tenant-aware scheduling. Teams with these needs are already looking at vLLM or SGLang, and the TensorRT-LLM guide covers a compile-ahead alternative; see TensorRT-LLM.

You need security patches. A frozen image carries known vulnerabilities forever. Where policy requires a supported base, TGI fails the requirement regardless of how stable it feels.

You need a different environment. Local and desktop use belongs to llama.cpp or MLX, and the llama.cpp backend inside TGI has no published image and must be built from Dockerfile_llamacpp. macOS has no accelerated runtime path for TGI at all, and Windows is community and unsupported through WSL2.

Equally, know when to stay. A stable service on a supported architecture, with traffic that fits its capacity, a gateway in front, scanning in place and a funded migration plan, does not need to be moved on a deadline. The risk is not that TGI stops working. The risk is that nobody owns the decision, so the image ages for three years and the migration happens in an emergency. State the exit criteria, a date or a trigger, in your decision register.

Running TGI as a platform for other teams

Consumers care about a handful of things, and the platform's job is to make each one a stated, tested promise rather than folklore.

A stable contract. Publish the OpenAI-compatible endpoint as the supported interface and discourage the native /generate API, because the former survives an engine change and the latter does not. Document the supported request fields, the context limit per model and the exact meaning of the error codes: 422 is the caller's request, 429 is capacity and should be retried with backoff, 424 is a platform fault.

Known limits per model. Publish max_input_tokens and max_total_tokens read from /info, plus the tenant's own quota. Note that temperature: 0 is rejected, since the server needs do_sample: false instead, and that stop accepts up to four sequences by default; these are the things consumers hit first.

Structured output with a caveat. Grammar-constrained decoding through response_format, grammar or tools works, but the first request with a new grammar compiles it and is slow, with later ones served from a cache. Tell consumers to warm schemas at deploy time.

A client library story. The text-generation Python package is legacy; steer teams to the openai SDK or huggingface_hub.InferenceClient.

A change policy. Announce image or model changes, run the contract suite in a canary first, and keep the previous deployment available for rollback. With the engine frozen, most changes are now model or configuration changes, and those deserve the same discipline.

A self-service path with guardrails. Teams will ask for new models. A good intake form asks for the licence, the model's architecture against the supported list, the context length they truly need, the expected traffic and the owner. Defaults to zero config, safetensors, a pinned revision and quantization where it passes an evaluation.

The review checklist

Use this for design documents and pull requests that change a TGI deployment. Each line is a question with a reason.

• Image: is the tag pinned to 3.3.7 (or 3.3.6-rocm, 3.3.6-neuron, as applicable), mirrored in our registry, with a recorded digest and a recent scan?
• Weights: is --revision a commit SHA, are the weights safetensors, and are serving pods offline with no Hub token?
• Remote code: is --trust-remote-code absent, or listed in the exception register with a reviewed, pinned repo?
• Token budgets: are max_input_tokens and friends left to zero config, or is there an exception with measurements?
• Limits: is there a per-tenant context limit enforced at the gateway?
• Auth: are the routes outside --api-key protection (/health, /info, /metrics, /v1/models, /docs) unreachable by tenants?
• Secrets: is the API key kept out of shared logs, and can we rotate it without an outage?
• Telemetry: is --usage-stats set deliberately?
• Probes: does the startup probe tolerate a real cold start, and does terminationGracePeriodSeconds exceed --graceful-termination-timeout?
• Shared memory: is /dev/shm a memory-backed volume for multi-GPU?
• Autoscaling: does it scale on queue signals, with a warm floor?
• Observability: are /metrics or the metrics port scraped, dashboards present, alerts tied to causes?
• Vision: are MAX_IMAGE_FETCH_SIZE, --payload-limit and egress policy set for image traffic?
• Exit: does the document state the migration trigger and owner?

The complete picture

Across the three levels, the picture of TGI is a coherent one. At the bottom is a Rust launcher that supervises a Rust router and a Python model server that runs one shard per GPU. The router validates, queues and batches continuously, with a paged KV cache, prefix caching and chunked prefill turned on by default and with token budgets computed for you in v3. Clients speak either the native /generate API or the OpenAI-compatible Messages API, and structured output, LoRA adapters, speculation and quantization are options you switch on with flags or their environment twins.

Around that engine you build a platform: a gateway that supplies identity, quotas and routing that TGI lacks, a deployment that respects startup and shutdown behaviour, metrics that expose queueing before users feel it, a supply chain that you own because upstream will not patch it, and a decision register that makes the migration an explicit programme. Almost everything transferable here, continuous batching, paged attention, prefix caching, chunked prefill and the OpenAI API, carries directly to vLLM and SGLang, so the skills outlive the tool.

Where the series leaves you

You can now run TGI, tune it, observe it, secure it as far as its model allows, and operate it for other teams. More valuable than any flag is the habit this guide has tried to build: for each component, ask what its failure looks like, who detects it, and who decides what happens next.

Your next steps are concrete. Stand up the gateway if you do not have one. Write the contract tests. Mirror and scan the image. Put the migration trigger in writing. Then continue with vLLM or SGLang, the engines Hugging Face itself recommends, and with Langfuse for application-level observability above the engine.

Sources

• Text Generation Inference documentation index
• Architecture
• Launcher reference
• Metrics reference
• Monitoring tutorial
• Usage statistics
• Safety tutorial
• Visual language models
• LoRA concepts
• Chunking concepts
• Quantization concepts
• Inference Endpoints: TGI engine and migration guidance
• TGI GitHub repository (archived)
• TGI release notes
• Container images on GHCR

Part three of three. Senior interviews about TGI are rarely about flags. They test whether you can design a serving platform, run it through an incident, and argue honestly about a tool that is in maintenance mode. About fifty minutes of material. Read the opening framing first, then work through the design prompts, the incident questions and the leadership conversations.

The framing you must get right before anything else

Open any senior answer with the lifecycle fact, because interviewers use it to separate people who have operated TGI from people who have read a tutorial. Hugging Face put TGI into maintenance mode on 11 December 2025, the repository was archived on 21 March 2026, and the final release is v3.3.7 from 19 December 2025. That does not make the tool unusable. It makes every platform decision a statement about how long you will keep it.

A strong opening for almost any question: "TGI is a stable, frozen, widely deployed engine. Its ideas, continuous batching, paged KV cache, prefix caching, chunked prefill and the OpenAI-compatible API, carry straight over to vLLM and SGLang, so I design the platform around those contracts and treat the engine as replaceable."

The thirty-second architecture answer

"One container is one model. A launcher supervises a router and one Python shard per GPU. The router validates, queues and batches continuously, and talks to shards over gRPC on a Unix socket. Shards hold weights and the KV cache and synchronise through NCCL under tensor parallelism. Scaling out means independent replicas behind a load balancer. There is no multi-model serving, no cluster manager, and only one shared API key, so a platform needs a gateway in front for identity and quotas."

Practise saying it in under thirty seconds, then be ready to defend every clause.

System design prompts

Prompt one: an internal LLM platform for twelve teams

Design the serving layer for twelve product teams that want hosted open-weight models. Some are interactive, some batch.

A strong answer starts from requirements rather than boxes. Ask about latency objectives, since an interactive chat tool and an overnight summarisation job need different things; about data sensitivity, since that decides whether tenants may share a replica; about the model list; and about the expected context lengths, because context length drives GPU cost.

Then structure the answer in layers. At the edge, a gateway that authenticates callers with per-team keys, enforces token and concurrency quotas and a maximum context per tenant, records usage and routes by model name. TGI speaks the OpenAI API, so a gateway such as LiteLLM fits without custom work. Behind it, one deployment per model, each a set of identical replicas, because TGI has no multi-model serving. Each replica is a container with GPUs requested through nvidia.com/gpu, a memory-backed /dev/shm, a weights volume at /data, configuration through environment variables, and /health for probes. Weights are downloaded once by a controlled job, with HF_HUB_OFFLINE=1 in serving pods so they hold no Hub token.

Say what you would leave alone. You would keep zero config for the token budgets, because since v3 TGI derives max_input_tokens, max_total_tokens and max_batch_prefill_tokens from the hardware, and the official advice is to remove manual flags. You would segregate interactive and batch traffic into different deployments, so a long batch prompt cannot cause inter-token latency spikes for a chat user: continuous batching admits newcomers by pausing running streams for a prefill, and long prompts make that pause visible.

What a weak answer misses: authentication (TGI's --api-key is a single shared token that does not even cover /health, /info or /metrics), capacity thinking (concurrency is bounded by the KV cache, not by GPU count), and the exit plan.

Follow-up you should expect: "How do you scale it?" Answer with signals, not CPU. Scale on tgi_queue_size or a high percentile of tgi_request_queue_duration, keep a warm floor because a cold start is image pull plus weight load plus warmup, and size spare capacity from your measured replica restart time.

Prompt two: serving a 70B model with long contexts

A team needs a 70B model with 128k-token contexts. How many GPUs, and what do you worry about?

The point of the question is whether you reason from the KV cache. Weights take a fixed amount of memory split across shards under tensor parallelism. What remains after weights, activations and warmup is the KV budget. Concurrency is roughly that budget divided by the average prompt plus generated tokens. The official documentation gives a sharp example: on hardware that fits the full context only three times, a 128k-context 70B model serves three full-length requests at once, and capping --max-input-tokens at 64k doubles it.

So the answer is a set of questions and levers. How many requests truly use the full context? If few, cap context per tenant at the gateway and keep the rest cheap. Can the weights be quantized to fit more cache, using AWQ, GPTQ, Marlin or fp8 on H100-class hardware, and can the cache itself shrink with --kv-cache-dtype fp8_e4m3fn or fp8_e5m2, which is CUDA only? Is prefix caching likely to help, for example a shared long document across a conversation? Tensor parallelism is intra-node over NVLink or PCIe through NCCL, and multi-node uses --master-addr and --master-port and is uncommon, so prefer a node with enough GPUs.

Watch for the traps: EXL2 cannot use tensor parallelism, and sharded configurations need safetensors weights. Raise the startup probe tolerance, since a 70B load and warmup takes many minutes. Set --shm-size or a memory emptyDir for NCCL. And say you would validate by load testing with inference-benchmarker or the repository's k6 scripts, with the prefix cache both cold and warm.

Prompt four: moving off TGI

You inherit twenty TGI deployments. Leadership asks for a plan to move to vLLM within two quarters.

Do not start with the engine; start with the contract. Inventory what clients actually call. Anything on /v1/chat/completions ports with little change, because vLLM and SGLang serve that API. Native /generate payloads are TGI-specific and must be rewritten. Grammar and response_format syntax, LoRA selection, tool-calling output, chat templates and token limits all need checking. Also list the operational surface: the tgi_* metric names, dashboards and alerts, probe paths, the weights cache layout and the environment variables.

Then propose the method. Insert or confirm a gateway so routing is a configuration change. Write contract tests against the public API and run them on TGI first, so you have a baseline. Stand up the new engine for one low-risk model, replay captured traffic, compare latency, error rates and output quality, then ramp a small percentage and widen. Retire old deployments after a soak period. This mirrors the Hugging Face guidance for Inference Endpoints: you cannot change the engine of an existing endpoint, so you create a new one, test it, move traffic and pause the old one.

Name the risks honestly: memory models differ, so token budgets that were automatic on TGI need re-derivation; prefix caching and chunked prefill behave differently; and output can differ slightly even at the same settings, so quality evaluation belongs in the plan and not only latency.

Incident-response questions

Interviewers want to hear that you triage by component, read evidence in order and communicate while you do it. Structure every incident answer as: stabilise, diagnose, fix, prevent.

The replicas will not start

A deploy rolled out and all new pods crash-loop. What do you do?

First, stop the bleeding: if the old replicas are still serving, halt the rollout so capacity is not drained. Then read the logs from the top, because the launcher's final message, Error: ShardCannotStart or Error: WebserverFailed, is only a summary. The cause is in the lines labelled "Shard complete standard error output". Three families cover most cases.

Not enough memory to handle {N} prefill tokens. You need to decrease --max-batch-prefill-tokens means warmup tried a maximal prefill that does not fit the free memory. Typical causes: someone set --max-batch-prefill-tokens or --max-input-tokens by hand, a different process holds the GPU, or the new model is bigger than the old. Fixes are to remove the manual limits and let zero config choose, lower the budget, quantize, or add shards. Not enough memory to handle max_total_tokens means a single full-length request cannot fit, so lower --max-total-tokens or add memory.

Error: DownloadError with 401, 403 or gated-repo text points to a missing or invalid HF_TOKEN, license terms not accepted, no network in an air-gapped cluster or a full disk.

NCCL errors or a hang during shard start under tensor parallelism point to shared memory: Docker's default /dev/shm of 64 MB is too small.

Also check the boring cause, the probes. If the pod is healthy but killed, the startup or liveness probe is shorter than download, load and warmup. Say that you would confirm by watching for Shard ready and the router's Connected before concluding it is a TGI fault.

Latency doubled, GPUs look fine

Users report slow responses. GPU utilisation is high and nothing is crashing.

High utilisation tells you nothing, because a saturated decode loop looks the same at healthy and unhealthy load. Go to the queue. If tgi_queue_size and the 95th percentile of tgi_request_queue_duration are rising, demand has exceeded capacity: scale out, and look for a client retry storm or a new tenant. If the queue is flat but tgi_request_mean_time_per_token_duration has risen, look for long prompts, visible in the tgi_request_input_length distribution: continuous batching pauses running streams to prefill newcomers, so long prompts hurt everyone on that replica. Check whether a recent deploy changed --waiting-served-ratio or --max-waiting-tokens, whether prefix caching was disabled by adding LoRA or vision, and whether a new tenant raised the average context.

Fixes are structural: per-tenant context caps, separate deployments for batch and interactive traffic, more replicas, and a latency objective on the scheduler flags rather than a hunch.

A wave of 429s

Clients are getting "Model is overloaded".

State that 429 is the router refusing admission beyond --max-concurrent-requests, which defaults to 128, and that this is the platform protecting itself. The wrong reflex is raising the limit: queued requests still need KV blocks, so a larger limit moves the pain to memory and latency. The right sequence is to confirm via tgi_request_failure with err="overloaded", scale out, check clients for retry storms and add jittered backoff with a retry budget, and add per-tenant concurrency limits at the gateway so one team cannot starve the rest.

Tool-calling clients broke after a deploy

Agents that worked yesterday now crash parsing tool calls.

Recognise the v3.2.0 change: TGI now returns the OpenAI shape, with tool_calls as a list and function.arguments as a JSON string rather than an object. Code that treated arguments as a dictionary fails. The fix is json.loads(tool_call.function.arguments). The deeper answer is process: that was a breaking change shipped in a minor version, so pin tags, canary every upgrade and keep contract tests that include tool calling. Add that old documentation examples still show the pre-3.2 shape, so tell teams not to trust them.

The API key leaked

A key appears in a log aggregator. Walk me through it.

Rotate first, then investigate. The shared backend key protects the TGI replicas, so rotating means issuing a new secret, rolling the deployment and updating the gateway, ideally without downtime by supporting a rolling restart. Then find how it leaked. The launcher logs its parsed arguments at startup with no redaction, so the key is in the first lines of container logs, and it is also passed to the router as a command-line argument visible to ps inside the container. Fix by redacting at the log pipeline, restricting who reads those logs, and keeping the key off the command line where possible by delivering it through the environment. Remember too that /info and /metrics are unauthenticated by design, so the real control is network placement: tenants must reach only the gateway.

A critical CVE in the image

A scanner flags a critical vulnerability in the TGI image. Upstream will not fix it.

This is the question that distinguishes owners. Because the repository is archived, no patched image is coming. Assess exploitability: is the vulnerable component reachable, given that TGI sits on a private network behind a gateway? Mitigate with network policy, runtime restrictions and reducing exposed routes. Record a time-boxed exception with an owner. Consider rebuilding the image yourself from the final source if the fix is a dependency bump you can validate, understanding that you now own that fork. And treat the finding as evidence for accelerating the migration.

Trade-off questions

Stay on TGI or migrate

A balanced answer refuses both extremes. Staying is justifiable when the service is stable, the architecture is supported, the traffic fits, and the risk register has an owner. Migration is justified by missing capabilities (new architectures, disaggregated serving, tenant-aware scheduling), unpatched vulnerabilities, or the desire to align with where the ecosystem and Hugging Face itself are heading. The deciding factor is rarely performance; it is the support horizon. Offer a rule: migrate on a trigger, such as a required model the engine lacks, an unmitigable CVE, or a date, and write it down.

Managed endpoint or self-hosting

Managed platforms such as SageMaker, through the Hugging Face LLM container, remove cluster work but cost more per GPU hour and constrain configuration. Inference Endpoints now recommends vLLM or SGLang over TGI for new work. Self-hosting gives control over data residency, which matters for many Gulf and Egyptian employers with in-country data rules, and over cost at steady high utilisation. Mention scale-to-zero as the managed advantage for spiky traffic, with its long cold start as the price.

How to argue a trade-off to leadership

Leadership does not want a flag tour. They want a decision, a cost, a risk and a date. Use a fixed shape.

State the decision in one sentence. "I recommend we keep the twenty TGI deployments in place this quarter, put a gateway in front of them, and migrate the three highest-risk models to vLLM next quarter."

Quantify the options. Cost per million tokens now and after quantization. GPU count for each option. Engineer-weeks for the migration. Show the cost of doing nothing: an image that ages, vulnerabilities with no upstream fix, and a growing list of models TGI cannot serve.

Name the risks and their owners. The unpatched image is a security risk owned by the platform team with a mitigation and an expiry date. The migration risk is behavioural drift between engines, owned by whoever runs the contract tests.

Offer a reversible path. The gateway makes every step reversible: route one percent, compare, widen, and roll back with a configuration change. Leadership approves reversible plans faster.

Define the trigger. Say what would change your mind: a model requirement, a vulnerability, an engine capability. This shows you are not attached to the tool.

Rapid-fire facts to have ready

Sixty-second self-test

Say these aloud without notes. If you stall on more than two, reread the matching section of the detailed guide.

1. State the maintenance-mode date, the archive date and the final version.
2. Describe launcher, router and shard in two sentences, and say what happens when one shard dies.
3. Explain why concurrency is bounded by the KV cache, with the 70B example.
4. List four routes that --api-key does not protect.
5. Name the three autoscaling signals you would use instead of CPU.
6. Explain what 422, 429 and 424 each mean and who owns the fix.
7. Describe the v3.2.0 tool-call change and the one-line client fix.
8. Say how you would remove the Hub token from production pods.
9. Describe the migration method: gateway, contract tests, canary, ramp, retire.
10. State your trigger for leaving TGI.

Part three of three. At this level the problems are operational and slow-burning: a replica fleet nobody can safely restart, an image that has not been patched in a year because it cannot be, a shared key that leaked into a log search. Start with the production traps, then the habits of someone who owns the platform, then the checklists for reviews.

Production traps that only appear at scale

Cumulative: everything from Beginner and Mid still applies. These do not break a demo. They break a platform at the third team or the thousandth user.

Several of these deserve a paragraph, because the table hides the reasoning.

The coupling you cannot see until it bites

The launcher supervises the router and every shard, and if any child dies it stops all of them. On a single GPU that is invisible. On an eight-GPU tensor-parallel replica, one GPU error turns into a full cold start: reload weights, run warmup, capture CUDA graphs. If your capacity plan assumes replicas fail independently and instantly recover, one fault becomes a brownout. Measure the real recovery time in a game day by killing a shard on purpose, and set your spare-capacity margin from that number.

Zero config is a feature you can quietly turn off

Since v3.0.0, TGI picks max_input_tokens, max_total_tokens, max_batch_prefill_tokens and max_batch_total_tokens from the hardware and the model, and the official advice is to remove the flags you are using. The trap is inherited configuration: a Helm values file from the v2 era carrying hand-tuned numbers that were right for one GPU and model and are wrong for the next. Every manual token flag in your platform should either be deleted or carry a comment and an entry in the exception register explaining what it fixes and how it was measured.

Metrics are not behind the API key

The key covers generation routes only. /health, /ping, /info, /metrics and /v1/models answer anyone who can reach the port, and so do /docs and /api-doc/openapi.json. That is not a bug you can configure away; it is the model. The fix is network placement. Use --prometheus-port (default 9000, since v3.3.0) for scraping from inside the cluster, expose only the API port through the gateway, and make sure the gateway does not forward the unauthenticated paths.

The prefix cache is per replica

If your product depends on cache hits, such as long shared system prompts or document chat, a load balancer that spreads one conversation across replicas throws the benefit away. Sticky sessions help, and the official guidance says not to apply them blindly. Measure time to first token with and without stickiness on real traffic. If stickiness raises hit rates but creates hot replicas, you may do better with a smaller cache benefit and smoother load.

What stopped being true while you were not looking

Teams carry old tutorials into new clusters. Check these against your manifests: the cache is under HF_HOME and /data/hub, not HUGGINGFACE_HUB_CACHE; HF_TOKEN is the current name and HUGGING_FACE_HUB_TOKEN the legacy one; --max-input-length is a deprecated alias that errors if set together with --max-input-tokens; 8-bit bitsandbytes is deprecated in favour of eetq; prompt logprobs are off unless --enable-prefill-logprobs; and the text-generation Python client is legacy. None of these will page you, which is why they survive.

Supply-chain traps for a frozen engine

The usual advice is to keep dependencies current. Here you cannot, so the traps are different.

Floating tags. latest resolves to whatever the registry says today, and there are exactly two defensible states for a TGI tag: an exact version and a digest. Pin 3.3.7 for NVIDIA. For AMD the newest tag is 3.3.6-rocm, because no 3.3.7-rocm exists, and the same is true for 3.3.6-neuron. People who bump a tag by pattern, changing the version number and expecting the suffix variant to exist, get a pull failure at the worst time.

Registry sources. Images moved to ghcr.io and the Azure registry stopped receiving pushes in v3.3.6. A manifest that still points at an Azure path will keep pulling an old image and never fail, which is worse than failing. Mirror the image to your own registry and deploy only from the mirror, so an upstream deletion cannot stop a scale-up in the middle of an incident.

Model weights. --revision set to a branch name is a moving target. Pin a commit SHA. Prefer safetensors, because pickle .bin weights are not auto-converted unless --trust-remote-code is set, and that flag runs Python from the model repository. If you must use it, restrict to reviewed repos at a pinned revision and enforce that in admission policy.

The Hub token in production. A read token in every serving pod is a standing credential. The better pattern is to download once in a controlled job into a volume and run serving pods with HF_HUB_OFFLINE=1 and no token at all. Then there is nothing for a compromised pod to steal.

Scanner noise versus real risk. The image bundles a CUDA base, Python, Torch and many kernels, so scanners will report a long list. Triage by reachability: TGI behind a gateway on a private network exposes far less than the raw package list suggests. Record decisions with an expiry date rather than ignoring the report, and review the register when the migration plan is reviewed. See Trivy for the scanning workflow and Vault for secret delivery.

Telemetry and data residency

TGI sends usage statistics by default when running inside Docker: start and stop events, a fifteen-minute heartbeat, the configuration, GPU and CPU details and error stack traces. For many Gulf and Egyptian employers, data-residency and outbound-traffic rules make this a compliance item rather than a curiosity. Set --usage-stats off (or USAGE_STATS=off) in the platform default and note it in your data-flow document. The no-stack mode keeps events and drops stack traces and error types, which is a middle path if your policy allows some telemetry. Also check what else reaches outward: Hub downloads at start unless you run offline, and image fetches for vision requests.

Platform-ownership habits

These are ordered by leverage. The first three give most of the benefit.

One: put a gateway in front, always

Everything TGI lacks, identity, per-tenant quotas, rate limiting, TLS, audit, model routing, a gateway supplies. It also makes the engine replaceable, because clients depend on your endpoint and not on the engine's quirks. Because TGI speaks the OpenAI API, an off-the-shelf gateway such as LiteLLM works without custom code. If you do only one thing from this guide, do this.

Two: write the contract before the tests, and the tests before the migration

Define what you promise: supported request fields, per-model context limits, the meaning of 422, 429 and 424, and streaming behaviour. Write tests that exercise the gateway's public API, including streaming, stop sequences, structured output and tool calls. Run them against TGI today and against any candidate engine tomorrow. This is the single most valuable asset in a migration, and it costs a few days.

Three: own the exit plan

A frozen engine needs a written trigger and an owner. The trigger can be a date, a required model, or a vulnerability you cannot mitigate. The plan names the target (usually vLLM or SGLang), the first model to move, the canary percentage and the rollback. Reviewing it quarterly costs minutes and prevents an emergency migration.

Four: keep a decision register and an exception register

For every deployment, record the pinned image tag and digest, the model revision and the owner. For every departure from defaults, such as a manual token budget, --trust-remote-code, a custom quantization or a disabled safeguard, record the reason and an expiry date. Registers sound bureaucratic until the person who set a flag has left and a replica will not start.

Five: measure the cold start and the restart

Time from pod creation to the first successful /health. Break it down: image pull, weight load, warmup, CUDA graph capture, and on ROCm the PyTorch TunableOp tuning, which adds a minute or two unless PYTORCH_TUNABLEOP_ENABLED=0. These numbers set your probe thresholds, your autoscaler's lead time and your scale-to-zero policy. Re-measure when the model, the GPU type or the registry changes.

Six: alert on causes

Page on queue depth and queue duration trending up, restart counts, a rising 424 rate and a rising rate of tgi_request_failure. Do not page on GPU utilisation alone. Add a ticket-level alert on image age and on scan results against the pinned image, which for a frozen engine is a slow-moving risk indicator. See Prometheus, Grafana and OpenTelemetry.

Seven: run a game day twice a year

Kill a shard, fill the queue until it returns 429, expire the Hub token, roll the API key, and fail over to the mirror registry. Each exercise should produce one concrete change to a playbook. Teams that rehearse find the unwritten assumptions, such as a probe that kills slow starts, while nothing is on fire.

Eight: publish consumer documentation

Short and specific. The endpoint, the limits per model read from /info, the error codes and retry guidance, the change policy, and a note that temperature: 0 is rejected (send do_sample: false) and that stop takes at most four sequences by default. Tell teams to warm structured-output schemas at deploy time because the first grammar compile is slow, and to use the openai SDK or huggingface_hub.InferenceClient rather than the legacy text-generation package.

Practice cards

Use these as short drills with a teammate. Each takes a few minutes.

Card: read the startup log. Start a replica and find, in order: the weights download line, Waiting for shard to be ready..., Shard ready, the attention and prefix-caching line, the default max_batch_prefill_tokens line and the router's Connected. Explain what each stage does and which one would be slow with a cold volume.

Card: break it on purpose. Set --max-input-tokens above --max-total-tokens and read the invariant error. Then set --max-input-length together with --max-input-tokens and read the deprecation error. Knowing the message wording makes the real incident faster.

Card: the 422 tour. Send a request with an empty inputs, a temperature of 0, five stop sequences, and a prompt over the input limit. Match each response to the validation message and the fix.

Card: capacity sums. Given a KV budget of 600,000 tokens and an average request of 3,000 prompt tokens plus 500 generated, estimate concurrency. Repeat with a 20,000-token average and see why context caps matter.

Card: the metric map. Draw the request lifecycle (validate, queue, prefill, decode, stream) and place each tgi_* metric on it.

Card: the rotation drill. Rotate the backend API key with zero client-visible errors. Write the steps, then run them in a test environment.

Operational recipes

Recipe: confirm what zero config chose

curl -s http://localhost:8080/info | python3 -m json.tool

Read max_input_tokens, max_total_tokens, version and docker_label. Record them in the decision register, and compare against the previous deployment before and after any model or hardware change.

Recipe: a health check that matches reality

curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/health

200 means healthy. A 503 with {"error":"unhealthy","error_type":"healthcheck"} means not. Use it for startup and readiness probes, and allow a long failure threshold for large models.

Recipe: scrape metrics on the dedicated port

scrape_configs:
  - job_name: tgi
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: tgi-llama
        action: keep
      - source_labels: [__address__]
        regex: '([^:]+)(?::\d+)?'
        replacement: '$1:9000'
        target_label: __address__

This points the scrape at port 9000, the --prometheus-port default, so the API port never needs to expose metrics to anyone but the gateway.

Recipe: a safe production launch line

docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \
  -e HF_HUB_OFFLINE=1 -e API_KEY="$TGI_API_KEY" \
  registry.example.com/mirror/text-generation-inference:3.3.7 \
  --model-id /data/llama --usage-stats off --json-output

This pins a mirrored tag, runs offline from a pre-populated volume, disables telemetry and emits structured logs. Passing the key through the environment avoids typing it into shell history, though remember the launcher still hands it to the router as an argument.

Recipe: a rolling restart without dropping streams

Set terminationGracePeriodSeconds above --graceful-termination-timeout (90 seconds by default), roll one replica at a time with a readiness gate on /health, and keep the old replica serving until the new one is ready. Verify by holding a long streamed request open across the restart and confirming it finishes.

Recipe: check the weights volume layout

After v2.3, expect /data/hub/models--org--name. If a volume still holds /data/models--..., the next start will download again. List the volume before changing images, not after the pod is Pending on a download.

Review checklists

For a pull request that changes a deployment

• Is the image tag exact, from the mirror, with a digest recorded?
• Did any --max-* or MAX_* token setting appear or change? If so, where is the measurement?
• Is --trust-remote-code present? If so, is the revision a commit SHA and the repo reviewed?
• Does the startup probe still exceed the measured cold start?
• Is terminationGracePeriodSeconds larger than the graceful termination timeout?
• Is /dev/shm memory-backed for multi-GPU, and is the weights volume mounted at /data?
• Are secrets from a secret store, not literals, and is the Hub token absent from serving pods?
• Is --usage-stats explicit?
• Did the model type or adapters change in a way that disables prefix caching?
• Do the contract tests pass against the canary?

For a design document

• Does it say who calls TGI and through what? A gateway should appear on the diagram.
• Does it state the capacity model: KV budget, average and maximum context, expected concurrency?
• Does it describe how tenants are isolated, and say honestly that isolation inside a replica is logical?
• Does it list the unauthenticated routes and how they are kept private?
• Does it describe failure of one shard, one replica and one zone?
• Does it include cold-start time and the autoscaling signal?
• Does it name the migration trigger, the target engine and an owner?
• Does it call out data-residency choices such as telemetry and weight and image mirrors in region?
• Does it avoid promising features, new models or security fixes TGI will not receive?

What to carry forward

Three ideas outlast this tool. Capacity is a memory question before it is a compute question, so measure the KV budget and price context. A platform is the sum of what sits around the engine, the gateway, the registers, the probes and the playbooks, and that is where senior work lives. And a frozen dependency is not a crisis if someone owns the decision to leave it, and a slow disaster if nobody does.

Next, continue with vLLM and SGLang, where these habits apply directly, and with Kubernetes for the orchestration side.